fix(cma): DDL 与启动剥离 —— 关闭启动 create_all + 部署链特权迁移步 (cma-create-all-minpriv-rootfix-20260916)
ci/woodpecker/push/main Pipeline failed

根治 2026-09-16 发布中断(新增模型触发启动 create_all,cma_api 无 CREATE → 崩溃循环):

- backend/app/database.py: 新增 CMA_AUTO_CREATE 开关(**默认关**),生产启动不再执行 DDL;
  运行库账号因此无需 CREATE(最小权限可保持)
- backend/scripts/db_migrate.py: 特权迁移步(--check/--apply/--rollback/--status);
  幂等(checkfirst)+ journal 留痕 + 可回滚(dump 失败/表有数据即拒绝,fail-closed);
  显式拒绝用应用运行账号 cma_api 执行
- deploy.sh: 新增 [4b] 结构迁移(**重启前**执行;失败即中止部署,后端旧码继续服务)
- backend/scripts/tests/db-migrate-sim-e2e.sh: 17 项 scratch 复现/验收(负控制复现 CREATE command denied)
- docs/db-migration-runbook.md: 运行手册(含 cma_api CREATE 回收步骤)

验收:scratch E2E 17/17;pytest 1113 passed/1 xfailed;生产 --check 缺失表 0 / 列级差异 0;
--apply 幂等空跑 created_new=0(journal 留痕)
This commit is contained in:
Hermes CI Fix
2026-09-16 06:53:28 +08:00
parent d54ffaf531
commit 443a779ece
6 changed files with 709 additions and 3 deletions
+3
View File
@@ -16,5 +16,8 @@ backend/logs/
# 因果链验证报告(生成物)
backend/scripts/reports/
# 迁移 journal / dump(db_migrate.py 生成物,不入库)
backend/scripts/backups/db-migrate-journal/
backend/scripts/backups/dump-*.sql
# 脚本备份(手工备份产物,不入库)
*.bak-*
+26 -3
View File
@@ -56,10 +56,33 @@ def get_db():
db.close()
def auto_create_enabled() -> bool:
"""启动时建表开关(2026-09-16「启动 create_all × 运行库最小权限」根治)。
**默认关闭**:生产 DDL 由部署链的特权迁移步
`backend/scripts/db_migrate.py`(deploy.sh [4b],应用重启**前**执行)承担,
因此运行库账号(cma_api)无需 CREATE 权限。
仅开发/测试可显式 `CMA_AUTO_CREATE=1` 打开。
事故背景:2026-09-16 01:08 发布新增模型(tax_filing_log/treasury_*)→
启动 create_all 被 `cma_api` 无 CREATE 拒绝 → `CREATE command denied` → 崩溃循环。
"""
return os.getenv("CMA_AUTO_CREATE", "0").strip().lower() in ("1", "true", "yes", "on")
def init_db():
import app.models
Base.metadata.create_all(bind=get_engine())
logger.info("CMA数据库已初始化")
import app.models # 仅注册模型(无 DDL);下方 seed 也复用该引用
if auto_create_enabled():
logger.warning(
"CMA_AUTO_CREATE 已开启 → 启动时执行 create_all(仅限开发/测试;生产请走部署链迁移步)"
)
Base.metadata.create_all(bind=get_engine())
logger.info("CMA数据库已初始化(启动建表)")
else:
logger.info(
"启动建表已关闭(CMA_AUTO_CREATE 未开启):DDL 由部署链 scripts/db_migrate.py 在应用重启前执行"
)
# ── 初始化组织层级示例数据 ──
try:
+432
View File
@@ -0,0 +1,432 @@
#!/usr/bin/env python3
"""CMA 数据库结构迁移(特权 DDL 步)—— 与应用启动 create_all 剥离
任务:cma-create-all-minpriv-rootfix-20260916(P2)
背景(2026-09-16 01:08 发布中断事故)
--------------------------------------
应用启动时 `app/database.py::init_db()` 调 `Base.metadata.create_all()`(需要 CREATE 权限),
而运行库账号 `cma_api` 只有 SELECT + 表级 DML 白名单(**无 DDL**)。新代码引入新模型
(`tax_filing_log` / `treasury_*`)→ 启动即 `CREATE command denied` → 崩溃重启循环。
本脚本 = 部署链里的**独立、幂等、可回滚**的特权迁移步,在应用重启**之前**执行:
deploy.sh [4b] → backend/scripts/db_migrate.py --apply → systemctl restart cma-backend
于是:DDL 不再随应用 boot 发生;运行库账号可保持最小权限(无需 CREATE)。
用法
----
python3 scripts/db_migrate.py --check # 只读预检(默认动作,零写入)
python3 scripts/db_migrate.py --apply # 幂等建表(缺什么建什么)+ 写 journal
python3 scripts/db_migrate.py --status # 列出历史 journal
python3 scripts/db_migrate.py --rollback latest # 回滚最近一次 --apply 新建的表
python3 scripts/db_migrate.py --rollback <journal.json>
选项
----
--json 机器可读输出(部署链/CI 断言用)
--force 回滚时允许删除**有数据**的表(默认拒绝,exit 3)
--no-dump 回滚时不 mysqldump(默认先备份到 journal 目录,dump 失败即拒绝回滚)
--socket PATH unix socket 路径(默认 /var/run/mysqld/mysqld.sock)
--db-url URL 显式连接串(覆盖下方全部推导;不含口令时应配合环境变量)
--db-name NAME 目标库名(默认 env CMA_MIGRATE_DB_NAME → CMA_DB_NAME → cma)
连接凭据(按优先级;口令只从环境读,禁止硬编码/回显)
----------------------------------------------------
1) CMA_MIGRATE_DB_USER + CMA_MIGRATE_DB_PASS(**专用迁移账号**,TCP)——推荐用于非本机场景
2) 未设 CMA_MIGRATE_DB_USER → 本机 unix socket + root(socket 认证,无需口令)
⚠️ 应用运行账号(backend/.env 的 CMA_DB_USER,= cma_api)**不得**被本脚本使用:
迁移必须走在有 DDL 权限的账号上,本脚本会显式拒绝。
幂等契约
--------
首跑 --apply:created_new = 缺失表数(本次实测:0 —— 库已由应急 create_all 补齐)
二跑 --apply:created_new = 0,零副作用
已存在的表**不** ALTER、**不** DROP、**不**动数据行(列级差异只报告,见下)
退出码
------
0 成功 | 1 迁移失败 / 迁移后自检不一致 / 连接失败(fail-closed,部署链据此中止)
2 用法错误 | 3 回滚被拒(目标表有数据且未 --force)
"""
import argparse
import fcntl
import json
import os
import random
import socket
import string
import subprocess
import sys
import tempfile
from datetime import datetime
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from sqlalchemy import create_engine, inspect, text # noqa: E402
from sqlalchemy.engine.url import make_url # noqa: E402
HERE = os.path.dirname(os.path.abspath(__file__))
BACKEND_DIR = os.path.dirname(HERE)
DEFAULT_JOURNAL_DIR = os.path.join(HERE, "backups", "db-migrate-journal")
DEFAULT_SOCKET = "/var/run/mysqld/mysqld.sock"
DEFAULT_DB = "cma"
# 应用运行账号:绝不允许用于本迁移步(无 DDL 权限 = 事故根因)
FORBIDDEN_RUN_ACCOUNTS = {"cma_api"}
# ────────────────────────────────────────────────────────────────
# 连接(特权通道)
# ────────────────────────────────────────────────────────────────
def _resolve_db_name(args) -> str:
return (
args.db_name
or os.getenv("CMA_MIGRATE_DB_NAME")
or os.getenv("CMA_DB_NAME")
or DEFAULT_DB
)
def build_engine(args):
"""构造特权连接。返回 (engine, 描述串)。描述串不含口令。"""
db_name = _resolve_db_name(args)
if args.db_url:
url = make_url(args.db_url)
if url.database is None:
url = url.set(database=db_name)
return create_engine(url, pool_pre_ping=True), f"db-url({url.drivername}@{url.host or 'socket'}/{url.database})"
user = os.getenv("CMA_MIGRATE_DB_USER", "").strip()
if user:
if user in FORBIDDEN_RUN_ACCOUNTS:
raise SystemExit(
f"[FAIL] CMA_MIGRATE_DB_USER={user} 是**应用运行账号**(无 DDL 权限)——\n"
f" 迁移步必须使用有 DDL 权限的专用账号(或本机 root socket)。已拒绝执行。"
)
password = os.getenv("CMA_MIGRATE_DB_PASS", "")
host = os.getenv("CMA_MIGRATE_DB_HOST", "127.0.0.1")
port = int(os.getenv("CMA_MIGRATE_DB_PORT", "3306"))
url = (
f"mysql+pymysql://{user}:{password}@{host}:{port}/{db_name}"
f"?charset=utf8mb4"
)
return create_engine(url, pool_pre_ping=True), f"tcp({user}@{host}:{port}/{db_name})"
# 默认:本机 unix socket + root(socket 认证,无口令落盘)
sock = args.socket or os.getenv("CMA_MIGRATE_DB_SOCKET") or DEFAULT_SOCKET
if not os.path.exists(sock):
raise SystemExit(
f"[FAIL] 未设 CMA_MIGRATE_DB_USER,且 unix socket 不存在:{sock}\n"
f" 请提供专用迁移账号(CMA_MIGRATE_DB_USER/PASS)或用 --socket 指定路径。"
)
url = f"mysql+pymysql://root@localhost/{db_name}?unix_socket={sock}&charset=utf8mb4"
return create_engine(url, pool_pre_ping=True), f"socket(root@{sock}/{db_name})"
# ────────────────────────────────────────────────────────────────
# schema 比对(ORM 期望 vs 库内实况)
# ────────────────────────────────────────────────────────────────
def _import_models():
import app.models # noqa: F401 触发全部模型注册
from app.database import Base
return Base
def schema_diff(engine):
"""返回 (missing_tables, missing_columns{表:[列]})。只读。"""
Base = _import_models()
insp = inspect(engine)
existing = set(insp.get_table_names())
missing_tables, missing_columns = [], {}
for table in Base.metadata.sorted_tables:
if table.name not in existing:
missing_tables.append(table.name)
continue
have = {c["name"] for c in insp.get_columns(table.name)}
miss = [c.name for c in table.columns if c.name not in have]
if miss:
missing_columns[table.name] = miss
return missing_tables, missing_columns
# ────────────────────────────────────────────────────────────────
# journal(迁移留痕:谁/何时/建了什么 → 回滚依据)
# ────────────────────────────────────────────────────────────────
def journal_dir() -> str:
d = os.getenv("CMA_MIGRATE_JOURNAL_DIR") or DEFAULT_JOURNAL_DIR
os.makedirs(d, exist_ok=True)
return d
def _git_head() -> str:
try:
return subprocess.run(
["git", "-C", os.path.dirname(BACKEND_DIR), "rev-parse", "--short", "HEAD"],
capture_output=True, text=True, timeout=10,
).stdout.strip() or "unknown"
except Exception:
return "unknown"
def write_journal(payload: dict) -> str:
path = os.path.join(
journal_dir(), f"{datetime.now().strftime('%Y%m%dT%H%M%S')}-{payload.get('status', 'na')}.json"
)
with open(path, "w", encoding="utf-8") as fh:
json.dump(payload, fh, ensure_ascii=False, indent=2)
return path
def list_journals():
d = journal_dir()
return sorted(
(os.path.join(d, f) for f in os.listdir(d) if f.endswith(".json")),
reverse=True,
)
def resolve_journal(ref: str) -> str:
if ref in ("latest", ""):
items = [p for p in list_journals() if "-applied" in os.path.basename(p)]
if not items:
raise SystemExit("[FAIL] 没有可回滚的 journal(无 --apply 记录)")
return items[0]
if os.path.exists(ref):
return ref
cand = os.path.join(journal_dir(), os.path.basename(ref))
if os.path.exists(cand):
return cand
raise SystemExit(f"[FAIL] journal 不存在:{ref}")
# ────────────────────────────────────────────────────────────────
# 动作
# ────────────────────────────────────────────────────────────────
def cmd_check(args, engine, conn_desc) -> int:
missing_tables, missing_cols = schema_diff(engine)
out = {
"action": "check",
"connection": conn_desc,
"missing_tables": missing_tables,
"missing_table_count": len(missing_tables),
"missing_columns": missing_cols,
}
if args.json:
print(json.dumps(out, ensure_ascii=False))
else:
print(f"[check] 连接: {conn_desc}")
print(f"[check] 缺失表 {len(missing_tables)} 张: {missing_tables if missing_tables else '(无)'}")
if missing_cols:
print(f"[check] 列级差异 {len(missing_cols)} 张表: {missing_cols}")
print(" ⚠️ create_all 不会补列 → 需要独立的 ALTER 迁移(见 journal/回报)")
else:
print("[check] 列级差异:无")
# 列级差异属"存在但结构过期":部署链上以告警呈现,不阻断(阻断判据 = 缺表未补齐)
return 0
def cmd_apply(args, engine, conn_desc) -> int:
Base = _import_models()
before_missing, before_cols = schema_diff(engine)
started = datetime.now().isoformat(timespec="seconds")
payload = {
"action": "apply",
"status": "running",
"started_at": started,
"finished_at": None,
"connection": conn_desc,
"git_head": _git_head(),
"actor": f"{os.getenv('USER', 'unknown')}@{socket.gethostname()}",
"db_name": _resolve_db_name(args),
"missing_tables_before": before_missing,
"missing_columns_before": before_cols,
"created_tables": [],
"created_new": 0,
"error": None,
}
print(f"[apply] 连接: {conn_desc}")
print(f"[apply] 迁移前缺失表 {len(before_missing)} 张: {before_missing if before_missing else '(无)'}")
try:
if before_missing:
Base.metadata.create_all(bind=engine) # checkfirst=True → 已存在的表零动作
after_missing, after_cols = schema_diff(engine)
created = [t for t in before_missing if t not in after_missing]
payload.update(
status="applied",
finished_at=datetime.now().isoformat(timespec="seconds"),
created_tables=created,
created_new=len(created),
missing_tables_after=after_missing,
)
path = write_journal(payload)
except Exception as exc: # DDL 无事务:失败时记录已建对象,供 --rollback 恢复
try:
after_missing, _ = schema_diff(engine)
except Exception:
after_missing = before_missing
created = [t for t in before_missing if t not in after_missing]
payload.update(
status="failed",
finished_at=datetime.now().isoformat(timespec="seconds"),
created_tables=created,
created_new=len(created),
error=f"{type(exc).__name__}: {exc}",
)
path = write_journal(payload)
print(f"[FAIL] 迁移失败:{payload['error']}")
print(f" 已建对象 {created}(journal: {path})")
print(f" 回滚:python3 scripts/db_migrate.py --rollback {os.path.basename(path)}")
return 1
ok = not payload["missing_tables_after"]
print(f"[apply] 新建 {payload['created_new']} 张表: {created if created else '(无 → 幂等空跑)'}")
print(f"[apply] 迁移后缺失表 {len(payload['missing_tables_after'])} 张: "
f"{payload['missing_tables_after'] if payload['missing_tables_after'] else '(无)'}")
if after_cols:
print(f"[apply] ⚠️ 列级差异(create_all 不补列,需独立 ALTER): {after_cols}")
print(f"[apply] journal: {path}")
if not ok:
print("[FAIL] 迁移后自检不一致(仍有缺失表)→ 部署链应中止")
return 1
return 0
def _dump_tables(tables, db_name) -> str:
"""mysqldump 备份(凭据走 600 临时 defaults 文件,不进 argv/日志)。返回文件路径。"""
fd, cfg = tempfile.mkstemp(prefix="cma-migrate-dump-", suffix=".cnf")
os.close(fd)
os.chmod(cfg, 0o600)
try:
lines = ["[client]"]
if os.getenv("CMA_MIGRATE_DB_USER"):
lines += [
f"user={os.getenv('CMA_MIGRATE_DB_USER')}",
f"password={os.getenv('CMA_MIGRATE_DB_PASS', '')}",
f"host={os.getenv('CMA_MIGRATE_DB_HOST', '127.0.0.1')}",
f"port={os.getenv('CMA_MIGRATE_DB_PORT', '3306')}",
]
else:
lines += ["user=root", f"socket={os.getenv('CMA_MIGRATE_DB_SOCKET', DEFAULT_SOCKET)}"]
with open(cfg, "w", encoding="utf-8") as fh:
fh.write("\n".join(lines) + "\n")
out = os.path.join(
journal_dir(), f"dump-{datetime.now().strftime('%Y%m%dT%H%M%S')}.sql"
)
cmd = ["mysqldump", f"--defaults-extra-file={cfg}", "--no-tablespaces", "--skip-add-locks", db_name, *tables]
with open(out, "wb") as fh:
subprocess.run(cmd, stdout=fh, stderr=subprocess.PIPE, check=True)
return out
finally:
os.unlink(cfg)
def cmd_rollback(args, engine, conn_desc) -> int:
path = resolve_journal(args.rollback)
with open(path, encoding="utf-8") as fh:
payload = json.load(fh)
tables = payload.get("created_tables") or []
print(f"[rollback] journal: {path}")
print(f"[rollback] 该次迁移新建的表: {tables if tables else '(无)'}")
if not tables:
print("[rollback] 无新建对象 → 无需回滚(空操作)")
return 0
db_name = payload.get("db_name") or _resolve_db_name(args)
if not args.no_dump:
try:
dump_path = _dump_tables(tables, db_name)
print(f"[rollback] 已备份: {dump_path}")
except Exception as exc:
print(f"[FAIL] 备份失败 → 拒绝回滚(fail-closed):{type(exc).__name__}: {exc}")
return 1
with engine.connect() as conn:
rows = {}
for t in tables:
try:
rows[t] = conn.execute(text(f"SELECT COUNT(*) FROM `{t}`")).scalar()
except Exception:
rows[t] = 0
non_empty = {t: n for t, n in rows.items() if n}
if non_empty and not args.force:
print(f"[REFUSED] 以下表已有数据,未加 --force 拒绝删除:{non_empty}")
return 3
for t in tables:
conn.execute(text(f"DROP TABLE IF EXISTS `{t}`"))
conn.commit()
print(f"[rollback] 已删除 {len(tables)} 张表: {tables}")
payload["rolled_back_at"] = datetime.now().isoformat(timespec="seconds")
payload["rolled_back_by"] = f"{os.getenv('USER', 'unknown')}@{socket.gethostname()}"
payload["rollback_dump"] = None if args.no_dump else "见同目录 dump-*.sql"
base = os.path.basename(path)
archived = os.path.join(journal_dir(), base.replace("-applied.json", "-rolled_back.json"))
with open(archived, "w", encoding="utf-8") as fh:
json.dump(payload, fh, ensure_ascii=False, indent=2)
print(f"[rollback] 记录: {archived}")
return 0
def cmd_status(args) -> int:
items = list_journals()
if not items:
print("[status] 无 journal 记录")
return 0
for p in items[:20]:
try:
with open(p, encoding="utf-8") as fh:
j = json.load(fh)
print(
f"{os.path.basename(p)} status={j.get('status')} created_new={j.get('created_new')} "
f"tables={len(j.get('created_tables') or [])} git={j.get('git_head')} by={j.get('actor')}"
)
except Exception as exc:
print(f"{os.path.basename(p)} [unreadable] {exc}")
return 0
# ────────────────────────────────────────────────────────────────
def main() -> int:
ap = argparse.ArgumentParser(
description="CMA 特权数据库迁移步(与应用启动 create_all 剥离)",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
g = ap.add_mutually_exclusive_group()
g.add_argument("--check", action="store_true", help="只读预检(默认)")
g.add_argument("--apply", action="store_true", help="幂等建表 + journal")
g.add_argument("--rollback", metavar="JOURNAL", help="回滚某次 --apply 新建的表(latest 或文件名)")
g.add_argument("--status", action="store_true", help="列出 journal 历史")
ap.add_argument("--json", action="store_true", help="机器可读输出")
ap.add_argument("--force", action="store_true", help="回滚允许删有数据的表")
ap.add_argument("--no-dump", action="store_true", help="回滚前不 dump")
ap.add_argument("--socket", default=None, help=f"unix socket 路径(默认 {DEFAULT_SOCKET})")
ap.add_argument("--db-url", default=None, help="显式连接串(覆盖推导)")
ap.add_argument("--db-name", default=None, help="目标库名")
args = ap.parse_args()
if not (args.apply or args.rollback or args.status):
args.check = True # 默认只读预检
if args.status:
return cmd_status(args)
engine, conn_desc = build_engine(args)
# 迁移互斥:防并发会话/手工操作同时跑 DDL
lock_path = os.path.join(journal_dir(), ".db-migrate.lock")
lock_fh = open(lock_path, "a+")
try:
fcntl.flock(lock_fh, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError:
print("[FAIL] 另一个迁移正在运行(锁被占用)→ 本次中止")
return 1
if args.rollback:
return cmd_rollback(args, engine, conn_desc)
if args.apply:
return cmd_apply(args, engine, conn_desc)
return cmd_check(args, engine, conn_desc)
if __name__ == "__main__":
sys.exit(main())
+154
View File
@@ -0,0 +1,154 @@
#!/usr/bin/env bash
# CMA「启动 create_all × 运行库最小权限」根治 —— 复现 + 验收 E2E
# 任务: cma-create-all-minpriv-rootfix-20260916
#
# 零生产触碰:全程只用 scratch 库 cma_migrate_sim + scratch 账号 cma_sim
# (权限画像与运行库 cma_api 一致:只有 SELECT/INSERT/UPDATE/DELETE,**无 CREATE/DROP**)。
#
# 用例:
# T1 负控制(复现事故):旧行为(CMA_AUTO_CREATE=1)用无 DDL 权限账号启动 → CREATE command denied → 启动失败
# T2 迁移步幂等:--apply 首跑建全表,二跑 created_new=0
# T3 修复后启动:CMA_AUTO_CREATE 未设(默认关)→ 无 DDL、启动成功(/health=200)
# T4 验收关键路径:模拟"新增模型未迁移"(删一张表)→ 应用照常启动不中断;随后 --apply 补齐
# T5 回滚:journal --rollback 删除该次新建表;有数据的表拒绝回滚(exit 3),--force 才放行
# T6 清理:drop scratch 库 + 账号
#
# 用法: bash backend/scripts/tests/db-migrate-sim-e2e.sh (需 root,因走 mysql unix socket)
set -uo pipefail
SIM_DB="${SIM_DB:-cma_migrate_sim}"
SIM_USER="${SIM_USER:-cma_sim}"
SIM_PASS="$(openssl rand -hex 12)"
SIM_PORT="${SIM_PORT:-8013}"
SIM_SOCKET="${SIM_SOCKET:-/var/run/mysqld/mysqld.sock}"
BACKEND="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
WORK="$(mktemp -d -t cma-migrate-sim.XXXXXX)"
JOURNAL_DIR="$WORK/journal"
URL="http://127.0.0.1:${SIM_PORT}/health"
PASS=0; FAIL=0
ok() { echo " PASS $*"; PASS=$((PASS+1)); }
bad() { echo " FAIL $*"; FAIL=$((FAIL+1)); }
hdr() { echo; echo "── $* ──"; }
MYSQL_ROOT="mysql --socket=$SIM_SOCKET -u root"
SIM_ENV=(env "CMA_DB_HOST=127.0.0.1" "CMA_DB_PORT=3306" "CMA_DB_USER=$SIM_USER" "CMA_DB_PASS=$SIM_PASS" "CMA_DB_NAME=$SIM_DB")
MIG_ENV=(env "CMA_DB_NAME=$SIM_DB" "CMA_MIGRATE_JOURNAL_DIR=$JOURNAL_DIR" "CMA_MIGRATE_DB_SOCKET=$SIM_SOCKET")
wait_health() { # 轮询 /health,最多 $1 秒
local max="${1:-40}" t0=$SECONDS
while [ $((SECONDS-t0)) -lt "$max" ]; do
if curl -s -o "$WORK/health.json" -w '%{http_code}' --max-time 2 "$URL" 2>/dev/null | grep -q '^200$'; then
grep -q '"status"[[:space:]]*:[[:space:]]*"ok"' "$WORK/health.json" && return 0
fi
sleep 0.5
done
return 1
}
boot() { # boot <logfile> [extra env...] —— 后台起应用,返回 pid
local log="$1"; shift
( cd "$BACKEND" && "${SIM_ENV[@]}" "$@" ./venv/bin/python3 -m uvicorn app.main:app \
--host 127.0.0.1 --port "$SIM_PORT" --no-access-log ) >"$log" 2>&1 &
echo $!
}
foreground_boot() { # 前台起应用(预期失败用)
local log="$1"; shift
( cd "$BACKEND" && "${SIM_ENV[@]}" "$@" timeout 60 ./venv/bin/python3 -m uvicorn app.main:app \
--host 127.0.0.1 --port "$SIM_PORT" --no-access-log ) >"$log" 2>&1
}
cleanup() {
[ -n "${APP_PID:-}" ] && kill "$APP_PID" 2>/dev/null
pkill -f "uvicorn app.main:app --host 127.0.0.1 --port $SIM_PORT" 2>/dev/null
$MYSQL_ROOT -e "DROP DATABASE IF EXISTS \`$SIM_DB\`; DROP USER IF EXISTS '$SIM_USER'@'localhost', '$SIM_USER'@'127.0.0.1';" >/dev/null 2>&1
echo; echo "(scratch 清理完成:库 $SIM_DB / 账号 $SIM_USER 已删除;工作目录 $WORK 保留供证据)"
}
trap cleanup EXIT
echo "════ CMA 特权迁移步 E2E(scratch: $SIM_DB / $SIM_USER / port $SIM_PORT)════"
[ "$(id -u)" = "0" ] || { echo "需要 root(mysql unix socket)"; exit 2; }
# ── T0 setup ──
hdr "T0 setup:scratch 库 + 无 DDL 权限账号(画像同 cma_api)"
$MYSQL_ROOT -e "
DROP DATABASE IF EXISTS \`$SIM_DB\`;
CREATE DATABASE \`$SIM_DB\` DEFAULT CHARACTER SET utf8mb4;
DROP USER IF EXISTS '$SIM_USER'@'localhost', '$SIM_USER'@'127.0.0.1';
CREATE USER '$SIM_USER'@'localhost' IDENTIFIED BY '$SIM_PASS';
CREATE USER '$SIM_USER'@'127.0.0.1' IDENTIFIED BY '$SIM_PASS';
GRANT SELECT, INSERT, UPDATE, DELETE ON \`$SIM_DB\`.* TO '$SIM_USER'@'localhost';
GRANT SELECT, INSERT, UPDATE, DELETE ON \`$SIM_DB\`.* TO '$SIM_USER'@'127.0.0.1';
FLUSH PRIVILEGES;" 2>/dev/null
GRANTS=$($MYSQL_ROOT -N -e "SHOW GRANTS FOR '$SIM_USER'@'127.0.0.1';" 2>/dev/null)
echo "$GRANTS" | grep -q "CREATE" && bad "scratch 账号不应有 CREATE:$GRANTS" || ok "scratch 账号无 CREATE/DROP(权限画像同 cma_api)"
# ── T1 负控制 ──
hdr "T1 负控制(旧行为复现):CMA_AUTO_CREATE=1 + 无 DDL 权限账号"
foreground_boot "$WORK/t1-old-behavior.log" "CMA_AUTO_CREATE=1"
if grep -q "CREATE command denied" "$WORK/t1-old-behavior.log"; then
ok "复现成功:$(grep -m1 -o "CREATE command denied to user '[^']*' for table '[^']*'" "$WORK/t1-old-behavior.log")"
else
bad "未复现 CREATE command denied(见 $WORK/t1-old-behavior.log)"
fi
grep -q "Application startup complete" "$WORK/t1-old-behavior.log" \
&& bad "旧行为下应用竟启动成功(与事故不符)" \
|| ok "旧行为下应用启动失败(= 2026-09-16 崩溃循环根因)"
# ── T2 迁移步幂等 ──
hdr "T2 特权迁移步:--check / --apply 首跑 / --apply 二跑"
CHECK1="$(cd "$BACKEND" && "${MIG_ENV[@]}" ./venv/bin/python3 scripts/db_migrate.py --check 2>&1)"
echo "$CHECK1" | grep -q "缺失表" && ok "T2a --check 列出缺失表:$(echo "$CHECK1" | grep -m1 '缺失表')" || bad "T2a --check 无输出"
APPLY1="$(cd "$BACKEND" && "${MIG_ENV[@]}" ./venv/bin/python3 scripts/db_migrate.py --apply 2>&1)"; RC1=$?
N1="$(echo "$APPLY1" | grep -m1 -o '新建 [0-9]* 张表' | grep -o '[0-9]*')"
echo "$APPLY1" | sed 's/^/ /'
[ "$RC1" = 0 ] && ok "T2b 首跑 exit=0,created_new=$N1" || bad "T2b 首跑 exit=$RC1"
APPLY2="$(cd "$BACKEND" && "${MIG_ENV[@]}" ./venv/bin/python3 scripts/db_migrate.py --apply 2>&1)"; RC2=$?
N2="$(echo "$APPLY2" | grep -m1 -o '新建 [0-9]* 张表' | grep -o '[0-9]*')"
echo "$APPLY2" | sed 's/^/ /'
[ "$RC2" = 0 ] && [ "$N2" = "0" ] && ok "T2c 二跑幂等:exit=0,created_new=0" || bad "T2c 二跑 exit=$RC2 created_new=$N2"
# ── T3 修复后启动(默认关)──
hdr "T3 修复后启动:CMA_AUTO_CREATE 未设(默认关)"
APP_PID=$(boot "$WORK/t3-fixed-boot.log")
if wait_health 40; then ok "启动成功:/health=200 status=ok(无 DDL)"; else bad "启动失败(见 $WORK/t3-fixed-boot.log)"; fi
grep -q "启动建表已关闭" "$WORK/t3-fixed-boot.log" && ok "日志确认:启动建表已关闭" || bad "未见'启动建表已关闭'日志"
kill "$APP_PID" 2>/dev/null; wait "$APP_PID" 2>/dev/null; APP_PID=""
# ── T4 验收关键路径:新增模型未迁移 ──
hdr "T4 新增模型未迁移:应用重启不中断 + 迁移步补齐"
$MYSQL_ROOT -e "DROP TABLE \`$SIM_DB\`.treasury_balances;" 2>/dev/null
ok "已模拟'新模型未迁移'(DROP TABLE treasury_balances)"
APP_PID=$(boot "$WORK/t4-boot-no-ddl.log")
if wait_health 40; then ok "T4a 表缺失下应用**照常启动**(无 CREATE command denied → 不再中断)"; else bad "T4a 启动失败(见 $WORK/t4-boot-no-ddl.log)"; fi
grep -q "CREATE command denied" "$WORK/t4-boot-no-ddl.log" && bad "T4a 启动期仍有 DDL 尝试" || ok "T4a 启动期零 DDL 尝试"
kill "$APP_PID" 2>/dev/null; wait "$APP_PID" 2>/dev/null; APP_PID=""
APPLY3="$(cd "$BACKEND" && "${MIG_ENV[@]}" ./venv/bin/python3 scripts/db_migrate.py --apply 2>&1)"; RC3=$?
N3="$(echo "$APPLY3" | grep -m1 -o '新建 [0-9]* 张表' | grep -o '[0-9]*')"
[ "$RC3" = 0 ] && [ "$N3" = "1" ] && ok "T4b 迁移步补齐缺失表:created_new=1(treasury_balances)" || { bad "T4b exit=$RC3 created_new=$N3"; echo "$APPLY3" | sed 's/^/ /'; }
APP_PID=$(boot "$WORK/t4-boot-migrated.log")
if wait_health 40; then ok "T4c 迁移后应用正常(表就位)"; else bad "T4c 迁移后启动失败"; fi
kill "$APP_PID" 2>/dev/null; wait "$APP_PID" 2>/dev/null; APP_PID=""
# ── T5 回滚 ──
hdr "T5 回滚(journal 驱动;有数据 fail-closed)"
LAST_J="$(ls -t "$JOURNAL_DIR"/*-applied.json | head -1)"
ROLL1="$(cd "$BACKEND" && "${MIG_ENV[@]}" ./venv/bin/python3 scripts/db_migrate.py --rollback "$LAST_J" 2>&1)"; RCR1=$?
echo "$ROLL1" | sed 's/^/ /'
[ "$RCR1" = 0 ] && ok "T5a 回滚成功(空表,含 dump)" || bad "T5a 回滚 exit=$RCR1"
$MYSQL_ROOT -N -e "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='$SIM_DB' AND table_name='treasury_balances';" | grep -q '^0$' \
&& ok "T5b 表确已删除(information_schema 回查=0)" || bad "T5b 表仍在"
# 有数据 → 拒绝
(cd "$BACKEND" && "${MIG_ENV[@]}" ./venv/bin/python3 scripts/db_migrate.py --apply >/dev/null 2>&1)
$MYSQL_ROOT -e "INSERT INTO \`$SIM_DB\`.treasury_balances (entity_id, account_id, period, balance, source, data_status) VALUES (1,1,'2026-09',1.0,'manual','estimated');" 2>/dev/null
ROWS=$($MYSQL_ROOT -N -e "SELECT COUNT(*) FROM \`$SIM_DB\`.treasury_balances;" 2>/dev/null)
[ "${ROWS:-0}" -gt 0 ] && ok "T5c 前置:已造 1 行数据(rows=$ROWS)" || bad "T5c 前置造数失败"
LAST_J="$(ls -t "$JOURNAL_DIR"/*-applied.json | head -1)"
ROLL2="$(cd "$BACKEND" && "${MIG_ENV[@]}" ./venv/bin/python3 scripts/db_migrate.py --rollback "$LAST_J" 2>&1)"; RCR2=$?
echo "$ROLL2" | sed 's/^/ /'
if [ "$RCR2" = 3 ]; then ok "T5c 有数据的表拒绝回滚(exit=3,fail-closed)"; else bad "T5c 期望 exit=3,实得 $RCR2"; fi
# ── 汇总 ──
hdr "汇总"
echo " PASS=$PASS FAIL=$FAIL (工作目录: $WORK)"
[ "$FAIL" = 0 ] && { echo " ✅ 全部通过"; exit 0; } || { echo " ❌ 有失败项"; exit 1; }
+15
View File
@@ -60,6 +60,21 @@ else
fi
echo " ✓ 依赖检查完成"
# 4b. 数据库结构迁移(特权,重启前执行;幂等;可回滚)
# cma-create-all-minpriv-rootfix-20260916:DDL 已从应用启动剥离(CMA_AUTO_CREATE 默认关),
# 新模型必须由**有 DDL 权限的账号**在重启前建表;此处失败即中止部署(后端保持旧码运行),
# 避免 2026-09-16 那种「启动 create_all 被 cma_api 拒 → 崩溃循环」。
echo "[4b/5] 数据库结构迁移(特权账号,重启前)..."
cd "$SCRIPT_DIR/backend"
if ! ./venv/bin/python3 scripts/db_migrate.py --apply; then
echo "❌ 数据库结构迁移失败 → 中止部署(后端未重启,旧码继续服务)"
echo " 排查: cd $SCRIPT_DIR/backend && ./venv/bin/python3 scripts/db_migrate.py --check"
echo " 回滚: cd $SCRIPT_DIR/backend && ./venv/bin/python3 scripts/db_migrate.py --rollback latest"
exit 1
fi
cd "$SCRIPT_DIR"
echo " ✓ 结构迁移完成(幂等;journal: backend/scripts/backups/db-migrate-journal/)"
# 5. 重启后端服务
echo "[5/5] 重启后端服务..."
systemctl daemon-reload
+79
View File
@@ -0,0 +1,79 @@
# CMA 数据库结构变更运行手册(DDL 与应用启动剥离)
- 任务:`cma-create-all-minpriv-rootfix-20260916`(P2)
- 事故档案:`wecom-project/_verification/2026-09-16-cma-release-incident-verification.md`
- 结论一句话:**DDL 只在部署链的 [4b] 特权迁移步发生,应用启动不再建表;运行库账号 `cma_api` 无需 CREATE。**
## 1. 三条不变式
| # | 不变式 | 落地位置 |
|---|---|---|
| 1 | 应用启动**不**执行 DDL(`create_all` 默认关,`CMA_AUTO_CREATE=1` 仅限开发/测试) | `backend/app/database.py::auto_create_enabled/init_db` |
| 2 | 结构变更由**有 DDL 权限的账号**在**应用重启前**执行(独立、幂等、可回滚、留痕) | `backend/scripts/db_migrate.py` + `deploy.sh [4b]` |
| 3 | 运行库账号保持最小权限(DML 表级白名单,**无** CREATE/ALTER/DROP) | DB 授权侧(由 ops/项目Bot 回收,见 §5) |
## 2. 部署链顺序(`deploy.sh`)
```
[0] 提交纪律检查(忽略未跟踪) ← 勿回退
[1] git pull origin main
[1b] 鉴权面审计闸门(B 类缺口即中止) ← 勿回退
[2] 前端构建 pnpm build
[3] 前端重铺 /var/www/cma
[4] 后端依赖
[4b] 数据库结构迁移(特权;重启前;幂等;失败即中止部署) ← 本单新增
[5] 重启 cma-backend + reload nginx
冒烟:就绪轮询 → BOT桥接 → schema_check → 登录/KPI
```
`[4b]` 失败时**不重启**后端 → 旧版本继续服务(不再出现「新模型建不出 + 服务被重启」的崩溃循环)。
## 3. 迁移步用法(`backend/scripts/db_migrate.py`)
```bash
cd /root/cma-management/backend
./venv/bin/python3 scripts/db_migrate.py --check # 只读预检(默认动作)
./venv/bin/python3 scripts/db_migrate.py --apply # 幂等建表(缺什么建什么)+ journal
./venv/bin/python3 scripts/db_migrate.py --status # 历史 journal
./venv/bin/python3 scripts/db_migrate.py --rollback latest # 回滚最近一次新建的表(先 dump 后删)
```
- **连接**:优先 `CMA_MIGRATE_DB_USER/CMA_MIGRATE_DB_PASS`(专用迁移账号,TCP);未设则走本机
`unix socket + root`(socket 认证,无口令落盘)。**显式拒绝**用应用运行账号(`cma_api`)执行本步。
- **幂等**:`create_all(checkfirst=True)` —— 已存在的表零动作;二跑 `created_new=0`。
- **journal**:`backend/scripts/backups/db-migrate-journal/<ts>-<status>.json`(含 db/git head/执行者/新建表清单)。
- **回滚**:只删该次 journal 记录为「新建」的表;表内有数据则**拒绝**(exit 3),`--force` 才放行;回滚前默认
`mysqldump` 备份(dump 失败即拒绝回滚,fail-closed)。
- **列级差异**:`create_all` **不**补列。`--check/--apply` 会报告「ORM 有列、库里没有」并提示需要独立
`ALTER` 迁移;部署链以此告警(不阻断)。
## 4. 新增模型的正确姿势(发布 checklist)
1. 在 `backend/app/models/` 加模型并确保被 `app/models/__init__.py` 导入。
2. 本地 `--check` 应能列出该表缺失(证明 ORM 已注册)。
3. 提交推送 → 部署链 `[4b]` 用特权账号建表 → 重启 → 冒烟 `schema_check` 通过。
4. 若模型**改列**(新增/改名/改类型):`create_all` 不处理 → 必须写独立迁移脚本/`ALTER`(参考
`backend/scripts/migrate_ext_input_ledgers_20260915.py`、`treasury_migrate.py` 的 `up()/down()` 模式)。
5. **不要**让运行库账号重新获得 CREATE 来「绕过」:那正是事故根因。
## 5. `cma_api` 的 CREATE 授权回收(项目Bot 执行)
2026-09-16 事故应急时授予了 `GRANT CREATE ON cma.* TO 'cma_api'@'localhost'/'127.0.0.1'`(保留至今)。
本单上线后(`[4b]` 已生效、应用启动不再建表)可安全回收:
```sql
REVOKE CREATE ON cma.* FROM 'cma_api'@'localhost';
REVOKE CREATE ON cma.* FROM 'cma_api'@'127.0.0.1';
SHOW GRANTS FOR 'cma_api'@'127.0.0.1'; -- 期望:仅 SELECT + 表级 DML,无 CREATE
```
回收后再发布一次(含任意新模型的版本)验证不中断即可闭环(证据:`[4b]` journal `created_new>=0` +
冒烟通过 + 无 `CREATE command denied`)。
## 6. 自验(可重复)
```bash
bash backend/scripts/tests/db-migrate-sim-e2e.sh # 需 root;scratch 库 cma_migrate_sim,零生产触碰
```
覆盖 17 项断言:负控制复现(`CREATE command denied` → 启动失败)、迁移幂等(76→0)、修复后启动零 DDL、
「新增模型未迁移」不中断 + 迁移补齐、回滚(含 dump 与有数据拒绝)。