agentshield/public/comparisons/braintrust.html at feat/agent-devtools-emitter · ghzhost/agentshield · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>AgentShield vs Braintrust: Enforcement vs Evaluation</title>
<meta name="description" content="Braintrust evaluates AI outputs. AgentShield blocks runaway spending. Eval vs enforcement — why you need both for production AI agents.">
<style>
:root { --bg:#0a0a0a; --surface:#141414; --accent:#00d4aa; --text:#e8e8e8; --muted:#888; --danger:#ff4757; --green:#22c55e; }
* { margin:0; padding:0; box-sizing:border-box; }
body { font-family:-apple-system,BlinkMacSystemFont,system-ui,sans-serif; background:var(--bg); color:var(--text); line-height:1.6; }
.container { max-width:900px; margin:0 auto; padding:24px; }
h1 { font-size:2.2em; margin-bottom:8px; }
h2 { font-size:1.4em; margin:32px 0 12px; color:var(--accent); }
h3 { margin:20px 0 8px; }
p { margin-bottom:12px; color:#ccc; }
.nav { background:var(--surface); padding:12px 24px; display:flex; justify-content:space-between; align-items:center; }
.nav a { color:var(--accent); text-decoration:none; }
.cta { display:inline-block; background:var(--accent); color:#0a0a0a; padding:12px 28px; border-radius:8px; text-decoration:none; font-weight:700; margin:16px 0; }
table { width:100%; border-collapse:collapse; margin:20px 0; }
th,td { padding:12px 16px; text-align:left; border-bottom:1px solid #2a2a2a; }
th { background:var(--surface); color:var(--accent); font-size:0.9em; text-transform:uppercase; }
.feature-check { color:var(--green); font-size:1.2em; }
.feature-x { color:var(--danger); font-size:1.2em; }
.callout { background:rgba(0,212,170,0.08); border:1px solid rgba(0,212,170,0.3); border-radius:8px; padding:20px; margin:24px 0; }
.footer { text-align:center; padding:32px; color:var(--muted); font-size:0.85em; }
</style>
</head>
<body>
<nav class="nav">
<a href="https://agentshield.fly.dev">🛡️ AgentShield</a>
<a href="/">Home</a>
</nav>
<div class="container">
<h1>AgentShield vs Braintrust: Enforcement vs Evaluation</h1>
<p style="font-size:1.1em;color:var(--muted);margin-bottom:32px">Braintrust evaluates AI outputs. AgentShield blocks runaway spending. Eval vs enforcement — why you need both for production AI agents.</p>

<h2>What Braintrust Does</h2>
<p>Braintrust is an AI evaluation platform. It helps teams test prompts, run evals, log traces, and track accuracy over time. It's built for the experimentation and quality-assurance phase of AI development.</p>

<h2>What AgentShield Does</h2>
<p>AgentShield is a runtime spend-control firewall. It operates during production — not during evaluation. When your agent is live and making real API calls with real money, AgentShield is the safety net.</p>

<h2>Feature Comparison</h2>
<table>
<thead><tr><th>Capability</th><th>AgentShield</th><th>Braintrust</th></tr></thead>
<tbody>
<tr><td>Blocks transactions before execution</td><td><span class="feature-check">✅ <1ms eval</span></td><td><span class="feature-x">❌ Post-hoc analysis</span></td></tr>
<tr><td>Transaction limit rules</td><td><span class="feature-check">✅ Per-call caps</span></td><td><span class="feature-x">❌ Not a firewall</span></td></tr>
<tr><td>Daily spend caps</td><td><span class="feature-check">✅ Per-agent caps</span></td><td><span class="feature-x">❌ Not available</span></td></tr>
<tr><td>AI output evaluation</td><td><span class="feature-check">❌ Runtime enforcement</span></td><td><span class="feature-x">✅ Core feature</span></td></tr>
<tr><td>Prompt testing / experimentation</td><td><span class="feature-check">❌ Not an eval tool</span></td><td><span class="feature-x">✅ Full eval suite</span></td></tr>
<tr><td>Traces and logging</td><td><span class="feature-check">✅ Decision audit trail</span></td><td><span class="feature-x">✅ Full trace logging</span></td></tr>
<tr><td>Real-time SSE alerts</td><td><span class="feature-check">✅ Instant block notifications</span></td><td><span class="feature-x">✅ Webhooks</span></td></tr>
<tr><td>Zero dependencies</td><td><span class="feature-check">✅ Python stdlib only</span></td><td><span class="feature-x">❌ Requires SDK</span></td></tr>

</tbody>
</table>

<h2>When to Use Braintrust</h2>
<ul style="color:#ccc;margin-bottom:20px;padding-left:20px">
<li>You are testing prompts and iterating on quality</li><li>You need systematic AI output evaluation</li><li>You want trace-level debugging of AI behavior</li>
</ul>

<h2>When to Use AgentShield</h2>
<ul style="color:#ccc;margin-bottom:20px;padding-left:20px">
<li>Your AI agents are in production making real API calls</li><li>You need hard financial limits, not quality scores</li><li>You want to prevent $2,800 mistakes, not evaluate them</li>
</ul>

<h2>Use Both Together</h2>
<p>Use Braintrust during development to evaluate prompt quality. Deploy with AgentShield in production to enforce spend limits. Different phases, different tools.</p>
<div class="callout">
<h2 style="margin-top:0">The $2,800 Test</h2>
<p>If an AI agent makes 21 API calls at $133 each in 60 seconds at 3 AM:</p>
<p><strong style="color:var(--danger)">Braintrust</strong> logs it. You see the damage in the morning dashboard.</p>
<p><strong style="color:var(--accent)">AgentShield</strong> blocks it in under 1ms. The agent never spends the money.</p>
<p style="margin-top:12px"><strong>Prevention costs $19/month. A single runaway agent costs $2,800 in 60 seconds.</strong></p>
<a href="/dashboard" class="cta">Start 14-Day Free Trial →</a>
</div>
</div>
<footer class="footer">AgentShield — The Safety Layer for Autonomous AI Agents. <a href="/" style="color:var(--accent)">Home</a> · <a href="/comparisons/helicone" style="color:var(--muted)">vs Helicone</a> · <a href="/comparisons/langsmith" style="color:var(--muted)">vs LangSmith</a></footer>
</body>
</html>