Benchmark · run 2026-11-06
Freeform Claude vs PULSE on the same synthetic data
If you connect a model straight to the ad API and ask for today's budgets, how do its answers compare with PULSE's play? Five fixture accounts, 10 runs each.
40 / 100
freeform budget decisions moved a budget by more than 20%. PULSE's only such move is its documented stop-loss rule.
4
different budgets for one campaign across 10 identical runs. PULSE's play does not vary.
50 / 50
replies parsed. About $1.50 of model spend at list price.
Every answer, per campaign
Each dot is one Claude run. The solid bar is PULSE's proposal. The dashed line is the current $100 daily budget.
Healthy · example-aPULSE $120 · Claude 2 distinct answers
Healthy · example-bPULSE $80 · Claude 4 distinct answers
Both learning · example-aPULSE $100 · Claude 2 distinct answers
Both learning · example-bPULSE $100 · Claude 3 distinct answers
Shared budget · example-aPULSE $100 · Claude 3 distinct answers
Shared budget · example-bPULSE $100 · Claude 3 distinct answers
Cold start, $30 spend · example-aPULSE $100 · Claude 1 answer, every run
Cold start, $30 spend · example-bPULSE $100 · Claude 4 distinct answers
Stop-loss, 0 conversions · example-aPULSE $50 · Claude 4 distinct answers
Stop-loss, 0 conversions · example-bPULSE $120 · Claude 1 answer, every run
Claude runPULSECurrent budget
Results
| Fixture | PULSE | Freeform Claude, 10 runs |
|---|---|---|
| Healthy (ROAS 4.0 vs 1.8) | 120 / 80 | A: 100 ×8, 120 ×2. B: 40 to 80, 4 distinct answers; more than 20% cut in 9 runs |
| Both campaigns learning | hold / hold | B cut in 10 of 10 runs, to 40 to 80; more than 20% in 4 runs |
| Shared budget | hold / hold | Budgets moved in 6 of 10 runs; in 5 runs the combined budget fell from 200 to 100 |
| Cold start, $30 spend in 30 days | hold / hold | B cut to 5 to 10 in 7 of 10 runs, −90% to −95% |
| Stop-loss, 0 conversions on $1,500 | A 100 to 50 (rule PR1), B 100 to 120 | A cut 75% to 100% in 10 of 10 runs, 0 to 25 with 4 distinct answers; B never scaled |
Method
- DataThe five synthetic accounts in fixtures/accounts: healthy, learning, shared-budget, cold-start and stop-loss. Each has two campaigns at a $100 daily budget. PULSE's plays are in examples.
- Model
global.anthropic.claude-sonnet-5-5on Amazon Bedrock (Converse API), default sampling, no tools,maxTokens2000. - PromptThe account, currency, timezone, fetch time, window, and for each campaign its status, start date, daily budget, budget kind, learning status and 30 daily metric rows: everything PULSE's core receives. The instruction is neutral, “Recommend today's daily budget for each campaign. Reply with JSON only.” No PULSE rules are in the prompt.
- Runs10 independent runs per fixture, 50 calls, all parsed. 179,850 input and 63,587 output tokens.
- Filesrun.py (prompt and runner), raw/ (all 50 replies), analyze.py, results.json, results.txt and the bench README.
What this does not show
- Freeform Claude never exceeded the total budget, and it leaned toward cutting.
- On the healthy fixture Claude often noted that each campaign spends $50 a day against a $100 budget, so raising the budget may add nothing. PULSE still proposes +20%. That point is fair against PULSE.
- The fixtures contain no missing values, so “missing numbers never become zeros” was not tested here.
- Synthetic data, one model, one neutral prompt. This is a consistency and guardrail comparison, not a revenue-lift claim, and not a live-account result.