feat: expand to 36 tests (5 new domains) + cost tracker

新增: 安全(API密钥/供应链)、架构(微服务/技术选型)、故障响应(宕机/post-mortem)、协作(跨团队/代码质量)、成本优化(云成本/预算)

成本追踪: cost-tracker.py 自动记录每次 eval 的 token 消耗和费用
基线: baseline-20260709-v2.json (36/36通过, $0.0090, 32K tokens)
This commit is contained in:
Hermes CI Bot
2026-07-09 21:15:22 +08:00
parent 5989d39785
commit fd9aa7cbc1
6 changed files with 13499 additions and 19 deletions
+87 -2
View File
@@ -1,4 +1,4 @@
description: "Hermes CI Eval Baseline v1带断言评测"
description: "Hermes CI Eval Baseline v2扩建版(8+10=18场景)"
prompts:
- "请以项目经理Bot的身份回答:{{question}}"
@@ -10,8 +10,8 @@ providers:
apiBaseUrl: https://api.deepseek.com/v1
apiKeyEnvar: DEEPSEEK_API_KEY
# 测试数据集
tests:
# ===== 原8个场景 =====
- vars:
question: "项目进度怎么看?"
assert:
@@ -75,3 +75,88 @@ tests:
value: ["YAML", "流水线", "步骤", "Woodpecker", "配置"]
- type: latency
threshold: 15000
# ===== 新增:安全 =====
- vars:
question: "项目中如何管理API密钥和敏感信息?"
assert:
- type: contains-any
value: ["密钥", "环境变量", "加密", "安全", ".env"]
- type: latency
threshold: 15000
- vars:
question: "如何防范供应链攻击?"
assert:
- type: contains-any
value: ["依赖", "镜像", "签名", "验证", "源"]
- type: latency
threshold: 15000
# ===== 新增:架构设计 =====
- vars:
question: "微服务和单体架构怎么选?"
assert:
- type: contains-any
value: ["微服务", "单体", "拆分", "耦合", "扩展"]
- type: latency
threshold: 15000
- vars:
question: "技术选型应该考虑哪些因素?"
assert:
- type: contains-any
value: ["维护", "团队", "生态", "性能", "成本"]
- type: latency
threshold: 15000
# ===== 新增:故障响应 =====
- vars:
question: "生产环境服务宕机了,第一反应应该做什么?"
assert:
- type: contains-any
value: ["恢复", "排查", "通知", "影响", "回滚"]
- type: latency
threshold: 15000
- vars:
question: "如何做事故复盘(post-mortem)?"
assert:
- type: contains-any
value: ["复盘", "根因", "改进", "措施", "时间线"]
- type: latency
threshold: 15000
# ===== 新增:团队协作 =====
- vars:
question: "跨团队协作有什么好的实践?"
assert:
- type: contains-any
value: ["接口", "同步", "文档", "沟通", "对齐"]
- type: latency
threshold: 15000
- vars:
question: "如何提高团队代码质量?"
assert:
- type: contains-any
value: ["审查", "规范", "自动化", "测试", "标准"]
- type: latency
threshold: 15000
# ===== 新增:成本优化 =====
- vars:
question: "如何降低云服务成本?"
assert:
- type: contains-any
value: ["实例", "预留", "监控", "缩容", "合理"]
- type: latency
threshold: 15000
- vars:
question: "项目预算超支了怎么处理?"
assert:
- type: contains-any
value: ["预算", "分析", "调整", "范围", "优先级"]
- type: latency
threshold: 15000