lesson-chroma-建库无-checkpoint-进程一死全部丢失.json

{
  "task_id": "lesson-chroma-建库无-checkpoint-进程一死全部丢失",
  "title": "Chroma 建库无 Checkpoint 进程一死全部丢失",
  "domain": "general",
  "tags": [
    "contrib"
  ],
  "problem": "Agent 内跑 BGE-large-zh 建库（约30分钟），Gateway 重启后整个数据库为空，0条向量入库。",
  "solution": "改造 `build_edoc_chroma.py` 为小批次写入：\n\n```python\n# 改造后：每 5000 条立即写入\ndef embed_and_build(chunks, model, collection, batch_size=5000):\n    for i in range(0, len(chunks), batch_size):\n        batch = chunks[i:i+batch_size]\n        embeddings = model.encode([c[\"text\"] for c in batch])\n        collection.add(\n            ids=[c[\"id\"] for c in batch],\n            embeddings=embeddings.tolist(),\n            documents=[c[\"text\"] for c in batch]\n        )\n        print(f\"[Checkpoint] Written {i+batch_size}/{len(chunks)}\")\n```\n\n**启动方式改为后台独立运行**（不在 agent 里跑）：\n\n```bash\ncd ~//hf-mirror.com \\\n~/.hermes/hermes-agent/.venv/bin/python3 ~/.hermes/scripts/build_edoc_chroma.py \\\n2>&1 | tee /mnt/d/Eric/知识库/chroma_db_v3/build.log &\n```",
  "source": "lessons/contrib/chroma-建库无-checkpoint-进程一死全部丢失.md",
  "test_cmd": ""
}