feat: expand to 36 tests (5 new domains) + cost tracker

新增: 安全(API密钥/供应链)、架构(微服务/技术选型)、故障响应(宕机/post-mortem)、协作(跨团队/代码质量)、成本优化(云成本/预算)

成本追踪: cost-tracker.py 自动记录每次 eval 的 token 消耗和费用
基线: baseline-20260709-v2.json (36/36通过, $0.0090, 32K tokens)
This commit is contained in:
Hermes CI Bot
2026-07-09 21:15:22 +08:00
parent 5989d39785
commit fd9aa7cbc1
6 changed files with 13499 additions and 19 deletions
+6 -17
View File
@@ -9,24 +9,13 @@ steps:
DEEPSEEK_API_KEY:
from_secret: DEEPSEEK_API_KEY
commands:
- apk add --no-cache git
- apk add --no-cache git python3 py3-pip
- npm install -g promptfoo
- cd promptfoo-eval
- promptfoo eval --max-concurrency 2 --no-cache
- promptfoo results --format json > /tmp/eval-results.json
- promptfoo eval --max-concurrency 2 --no-cache -o results.json
- python3 cost-tracker.py
- |
FAILURES=$(cat /tmp/eval-results.json | grep -o '"failures":[0-9]*' | grep -o '[0-9]*')
if [ "$FAILURES" != "0" ] && [ -n "$FAILURES" ]; then
echo "❌ Eval FAILED: $FAILURES test failures"
exit 1
fi
FAILURES=$(python3 -c "import json; d=json.load(open('results.json')); s=d['results']['stats']; print(s['failures']+s['errors'])")
echo "Tests: $(python3 -c "import json; d=json.load(open('results.json')); s=d['results']['stats']; print(s['successes'])") passed, $FAILURES failed"
if [ "$FAILURES" != "0" ]; then exit 1; fi
- echo "✅ All eval tests passed!"
report:
image: alpine:latest
commands:
- echo "Hermes CI Eval Baseline - $(date -u '+%Y-%m-%d %H:%M UTC')"
- echo "Status: ✅ PASS"
- echo "Next steps: Review results and update eval dataset"
when:
status: success