feat: expand to 36 tests (5 new domains) + cost tracker
新增: 安全(API密钥/供应链)、架构(微服务/技术选型)、故障响应(宕机/post-mortem)、协作(跨团队/代码质量)、成本优化(云成本/预算) 成本追踪: cost-tracker.py 自动记录每次 eval 的 token 消耗和费用 基线: baseline-20260709-v2.json (36/36通过, $0.0090, 32K tokens)
This commit is contained in:
+6
-17
@@ -9,24 +9,13 @@ steps:
|
||||
DEEPSEEK_API_KEY:
|
||||
from_secret: DEEPSEEK_API_KEY
|
||||
commands:
|
||||
- apk add --no-cache git
|
||||
- apk add --no-cache git python3 py3-pip
|
||||
- npm install -g promptfoo
|
||||
- cd promptfoo-eval
|
||||
- promptfoo eval --max-concurrency 2 --no-cache
|
||||
- promptfoo results --format json > /tmp/eval-results.json
|
||||
- promptfoo eval --max-concurrency 2 --no-cache -o results.json
|
||||
- python3 cost-tracker.py
|
||||
- |
|
||||
FAILURES=$(cat /tmp/eval-results.json | grep -o '"failures":[0-9]*' | grep -o '[0-9]*')
|
||||
if [ "$FAILURES" != "0" ] && [ -n "$FAILURES" ]; then
|
||||
echo "❌ Eval FAILED: $FAILURES test failures"
|
||||
exit 1
|
||||
fi
|
||||
FAILURES=$(python3 -c "import json; d=json.load(open('results.json')); s=d['results']['stats']; print(s['failures']+s['errors'])")
|
||||
echo "Tests: $(python3 -c "import json; d=json.load(open('results.json')); s=d['results']['stats']; print(s['successes'])") passed, $FAILURES failed"
|
||||
if [ "$FAILURES" != "0" ]; then exit 1; fi
|
||||
- echo "✅ All eval tests passed!"
|
||||
|
||||
report:
|
||||
image: alpine:latest
|
||||
commands:
|
||||
- echo "Hermes CI Eval Baseline - $(date -u '+%Y-%m-%d %H:%M UTC')"
|
||||
- echo "Status: ✅ PASS"
|
||||
- echo "Next steps: Review results and update eval dataset"
|
||||
when:
|
||||
status: success
|
||||
|
||||
Reference in New Issue
Block a user