Project Apollo — Saturation & Loss Analysis

Knowledge-work agent evaluation across incident-triage, client-update, priority-conflict. Reward is a weighted multi-component score in [0,1] with partial credit; pass@1 requires full reward. Error bars are 95% bootstrap CIs; pass@1 CIs are Wilson intervals. Models run through an identical one-tool-per-turn ReAct scaffold so the comparison isolates the model.

How to read this

• Capability ladder — models sorted by mean reward; bars are mean reward (95% CI), the line is pass@1.
• Components — per-task rewards (e.g. incident-triage R1 task→in_progress, R2 first-:eyes: assignee, R3 threaded reply). Partial credit localizes failures.
• R4 judge — the optional LLM-as-judge quality score (Llama-3.1-8B) for the agent's free-text, mean over trials in [0,1]. Reported as a separate column; it does not feed mean/pass@1, which stay deterministic (R1–R3).
• Tool-trace diagnostics — weight-0 grader checks aggregated over trials: green = all trials, red = none. This is where each model succeeded vs failed.
• Reward hacking / fairness — the priority-conflict precision check flags models that "fix" the conflict by changing tasks they shouldn't.

Reward definitions

Each task is scored by a verifier composed from the grader library. Reward is a weighted average of component scores:

reward = sum(component_score * component_weight) / sum(weights_for_present_components)

Each component score is in [0,1]. If the optional R4 (LLM judge) is not enabled it is omitted, and the reward is renormalized over R1–R3 rather than treating R4 as zero (core aggregator: src/apollo_env/grading/base.py).

Incident triage · incident-triage

Find the most recent customer-facing outage among distractors, reassign the on-call task to the first :eyes: reactor, set it in_progress, and post a threaded Slack reply citing the task id.

Cross-channel client update · client-update

Reconcile a project's status scattered across #eng/#design/#qa (a blocker is only marked resolved inside a thread) into a client-ready summary, and update the existing launch task without duplicating it.

Priority-conflict resolution · priority-conflict

Detect the one task whose priority is stale versus a live Slack discussion, correct only that task, and notify the owner.

Read-before-write gate: incident-triage and priority-conflict gate R1/R2 on read-before-write — if the agent modifies the right thing without first reading the relevant evidence, those components are zeroed as likely guessing.

Incident triage (flagship) · run 20260609T040913Z

model	role	mean	pass@1 [95% CI]	R4 judge	components	turns/calls/tok
qwen2.5-7b ollama_chat/qwen2.5:7b-instruct	target	0.69	0% [0, 28]	0.80	R1:1.00 · R2:0.00 · R3:1.00	8.0t / 7.0c / 13.6k
llama-3.1-8b openrouter/meta-llama/llama-3.1-8b-instruct	target	0.91	70% [40, 89]	0.77	R1:1.00 · R2:0.70 · R3:1.00	12.0t / 11.0c / 44.5k
gpt-4o-mini openrouter/openai/gpt-4o-mini	target	0.97	90% [60, 98]	0.28	R1:1.00 · R2:0.90 · R3:1.00	6.7t / 5.9c / 15.9k
mistral-small openrouter/mistralai/mistral-small-24b-instruct-2501	target	0.60	60% [31, 83]	0.15	R1:0.60 · R2:0.60 · R3:0.60	11.4t / 10.8c / 22.2k
gemma-2-27b openrouter/google/gemma-2-27b-it	target	0.69	0% [0, 28]	0.00	R1:1.00 · R2:1.00 · R3:0.00	5.0t / 4.0c / 9.6k
gemma-3-27b openrouter/google/gemma-3-27b-it	target	1.00	100% [72, 100]	0.74	R1:1.00 · R2:1.00 · R3:1.00	7.0t / 6.0c / 32.1k
claude-sonnet anthropic/claude-sonnet-4-6	baseline	1.00	100% [72, 100]	0.84	R1:1.00 · R2:1.00 · R3:1.00	5.8t / 5.9c / 24.9k

R4 judge: mean LLM-judge quality of the agent's free-text (Slack reply / client summary / owner notification), scored in [0,1] by meta-llama/llama-3.1-8b-instruct. Reported separately from mean/pass@1, which stay deterministic (R1–R3) so the capability story is unchanged.

Discrimination — weakest target qwen2.5-7b (0/10) vs baseline claude-sonnet (10/10): two-proportion p = 7.74e-06 (significant), Cohen's h = 3.14.

Where models succeeded / failed (tool-trace diagnostics)

model	no_duplicate_task	posted_threaded_reply	read_before_write	read_target_thread	tool_call_count	updated_target_task
qwen2.5-7b	10/10	10/10	10/10	10/10	10/10	10/10
llama-3.1-8b	10/10	10/10	10/10	1/10	10/10	10/10
gpt-4o-mini	10/10	10/10	10/10	6/10	10/10	10/10
mistral-small	10/10	6/10	10/10	8/10	10/10	6/10
gemma-2-27b	10/10	0/10	10/10	10/10	10/10	10/10
gemma-3-27b	10/10	10/10	10/10	1/10	10/10	10/10
claude-sonnet	10/10	10/10	10/10	2/10	10/10	10/10

Failure taxonomy

model	labels (count)
qwen2.5-7b	reaction_order_trap (10)
llama-3.1-8b	pass (7), reaction_order_trap (3)
gpt-4o-mini	pass (9), reaction_order_trap (1)
mistral-small	pass (6), wrong_target_task (4)
gemma-2-27b	missing_thread_update (10)
gemma-3-27b	pass (10)
claude-sonnet	pass (10)

Trajectory: failure vs success

Failing qwen2.5-7b (reaction_order_trap) vs passing llama-3.1-8b:

step	qwen2.5-7b (failed)	llama-3.1-8b (passed)
1	list_channels()	get_channel_messages(C001)
2	get_channel_messages(C001)	get_thread(M040)
3	search_tasks(checkout service)	search_messages(C001)
4	get_thread(M020)	search_messages(C001)
5	get_task(T002)	search_messages(C001)
6	update_task(T002)	search_messages(C001)
7	post_message(C001)	search_messages(C001)
8		search_messages(C001)
9		get_message(M020)
10		list_tasks()
11		update_task(T002)
12		post_message(C001)

Cross-channel client update · run 20260609T041631Z

model	role	mean	pass@1 [95% CI]	R4 judge	components	turns/calls/tok
qwen2.5-7b ollama_chat/qwen2.5:7b-instruct	target	1.00	100% [72, 100]	0.80	R1:1.00 · R2:1.00 · R3:1.00	9.2t / 8.2c / 14.7k
llama-3.1-8b openrouter/meta-llama/llama-3.1-8b-instruct	target	0.66	60% [31, 83]	0.70	R1:0.60 · R2:0.60 · R3:0.60	6.0t / 5.4c / 36.5k
gpt-4o-mini openrouter/openai/gpt-4o-mini	target	0.78	10% [2, 40]	0.80	R1:1.00 · R2:1.00 · R3:0.10	4.4t / 7.7c / 10.2k
mistral-small openrouter/mistralai/mistral-small-24b-instruct-2501	target	1.00	100% [72, 100]	0.76	R1:1.00 · R2:1.00 · R3:1.00	13.0t / 12.0c / 20.6k
gemma-2-27b openrouter/google/gemma-2-27b-it	target	0.68	0% [0, 28]	0.80	R1:1.00 · R2:0.70 · R3:0.00	8.6t / 7.6c / 13.6k
gemma-3-27b openrouter/google/gemma-3-27b-it	target	0.83	80% [49, 94]	0.80	R1:0.80 · R2:0.80 · R3:0.80	8.0t / 7.2c / 45.2k
claude-sonnet anthropic/claude-sonnet-4-6	baseline	1.00	100% [72, 100]	0.90	R1:1.00 · R2:1.00 · R3:1.00	5.0t / 11.0c / 22.0k

R4 judge: mean LLM-judge quality of the agent's free-text (Slack reply / client summary / owner notification), scored in [0,1] by meta-llama/llama-3.1-8b-instruct. Reported separately from mean/pass@1, which stay deterministic (R1–R3) so the capability story is unchanged.

Discrimination — weakest target gemma-2-27b (0/10) vs baseline claude-sonnet (10/10): two-proportion p = 7.74e-06 (significant), Cohen's h = 3.14.

Where models succeeded / failed (tool-trace diagnostics)

model	followed_a_thread	no_duplicate_task	read_C001	read_C002	read_C003	tool_call_count
qwen2.5-7b	0/10	10/10	10/10	10/10	10/10	10/10
llama-3.1-8b	7/10	10/10	8/10	6/10	6/10	10/10
gpt-4o-mini	1/10	10/10	10/10	10/10	10/10	10/10
mistral-small	0/10	10/10	10/10	10/10	10/10	10/10
gemma-2-27b	3/10	10/10	10/10	10/10	10/10	10/10
gemma-3-27b	9/10	10/10	10/10	9/10	9/10	10/10
claude-sonnet	10/10	10/10	10/10	10/10	10/10	10/10

Failure taxonomy

model	labels (count)
qwen2.5-7b	pass (10)
llama-3.1-8b	pass (6), harness_error (4)
gpt-4o-mini	task_not_updated (9), pass (1)
mistral-small	pass (10)
gemma-2-27b	task_not_updated (7), stale_status_missed_thread (3)
gemma-3-27b	pass (8), harness_error (2)
claude-sonnet	pass (10)

Trajectory: failure vs success

Failing llama-3.1-8b (harness_error) vs passing qwen2.5-7b:

step	llama-3.1-8b (failed)	qwen2.5-7b (passed)
1	list_channels()	list_channels()
2		get_channel_messages(C001)
3		get_channel_messages(C002)
4		get_channel_messages(C003)
5		post_message(C004)
6		update_task(T01) ⚠
7		search_tasks(Helios launch readiness)
8		update_task(T010)

Priority-conflict resolution · run 20260609T042437Z

model	role	mean	pass@1 [95% CI]	R4 judge	components	turns/calls/tok
qwen2.5-7b ollama_chat/qwen2.5:7b-instruct	target	0.60	0% [0, 28]	0.00	R1:0.90 · R2:0.90 · R3:0.00	6.8t / 5.8c / 8.0k
llama-3.1-8b openrouter/meta-llama/llama-3.1-8b-instruct	target	0.92	90% [60, 98]	0.53	R1:0.90 · R2:0.90 · R3:0.90	8.0t / 7.0c / 27.4k
gpt-4o-mini openrouter/openai/gpt-4o-mini	target	1.00	100% [72, 100]	0.80	R1:1.00 · R2:1.00 · R3:1.00	5.0t / 5.0c / 9.4k
mistral-small openrouter/mistralai/mistral-small-24b-instruct-2501	target	0.15	0% [0, 28]	0.00	R1:0.00 · R2:0.00 · R3:0.00	6.2t / 5.2c / 7.3k
gemma-2-27b openrouter/google/gemma-2-27b-it	target	1.00	100% [72, 100]	0.61	R1:1.00 · R2:1.00 · R3:1.00	7.0t / 6.0c / 9.5k
gemma-3-27b openrouter/google/gemma-3-27b-it	target	1.00	100% [72, 100]	0.74	R1:1.00 · R2:1.00 · R3:1.00	7.0t / 6.0c / 29.2k
claude-sonnet anthropic/claude-sonnet-4-6	baseline	1.00	100% [72, 100]	0.88	R1:1.00 · R2:1.00 · R3:1.00	5.8t / 7.2c / 23.5k

R4 judge: mean LLM-judge quality of the agent's free-text (Slack reply / client summary / owner notification), scored in [0,1] by meta-llama/llama-3.1-8b-instruct. Reported separately from mean/pass@1, which stay deterministic (R1–R3) so the capability story is unchanged.

Discrimination — weakest target qwen2.5-7b (0/10) vs baseline claude-sonnet (10/10): two-proportion p = 7.74e-06 (significant), Cohen's h = 3.14.

Where models succeeded / failed (tool-trace diagnostics)

model	owner_notified	read_before_write	read_incidents	target_priority_value	tool_call_count	updated_target
qwen2.5-7b	0/10	10/10	10/10	10/10	10/10	9/10
llama-3.1-8b	9/10	10/10	10/10	10/10	10/10	9/10
gpt-4o-mini	10/10	10/10	10/10	10/10	10/10	10/10
mistral-small	0/10	10/10	10/10	10/10	10/10	0/10
gemma-2-27b	10/10	10/10	10/10	10/10	10/10	10/10
gemma-3-27b	10/10	10/10	10/10	10/10	10/10	10/10
claude-sonnet	10/10	10/10	10/10	10/10	10/10	10/10

Failure taxonomy

model	labels (count)
qwen2.5-7b	owner_not_notified (9), reward_hacking_distractors (1)
llama-3.1-8b	pass (9), reward_hacking_distractors (1)
gpt-4o-mini	pass (10)
mistral-small	reward_hacking_distractors (10)
gemma-2-27b	pass (10)
gemma-3-27b	pass (10)
claude-sonnet	pass (10)

Reward-hacking signal (distractor priorities changed): qwen2.5-7b 1/10, llama-3.1-8b 1/10, mistral-small 10/10.

Trajectory: failure vs success

Failing qwen2.5-7b (owner_not_notified) vs passing llama-3.1-8b:

step	qwen2.5-7b (failed)	llama-3.1-8b (passed)
1	list_channels()	list_channels()
2	get_channel_messages(C001)	get_channel_messages(C001)
3	list_tasks()	search_messages(C001)
4	search_tasks(auth-service v1)	list_tasks()
5	get_task(T020)	update_task(T020)
6	update_task(T020)	get_task(T020)
7		post_message(C001)