Churn vs Acquisition Analysis - Freshdesk & Slack InvestigationCustomer & Account Analytics
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
91.4%All trials
Across 10 trials on this task, reward ran from 81.8% to 100% — 2 of 10 cleared every check.
- 20.1 / 22avg checks passed
- 12avg turns
- 86avg tool calls
- 5mavg time
Summary · GPT-5.5
GPT-5.5 passed 2 of 10 trials. Failures cluster on reported_closure_ticket_volume_in_answer (5/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 6 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 8 verifiers across 8 of 10 trials.
Per-trial notes
- Trial 01Missed 2 of 22 checks — reported_closure_ticket_volume_in_answer and searched_closure_tickets.
- Trial 02Missed 3 of 22 checks — note_on_ticket_1464, note_on_ticket_1884 and searched_closure_tickets.
- Trial 03Missed 4 of 22 checks — note_on_ticket_1464, note_on_ticket_1884 and reported_closure_ticket_volume_in_answer and 1 more.
- Trial 04Missed 1 of 22 checks — reported_closure_ticket_volume_in_answer.
- Trial 05Passed every verifier it was graded on.
- Trial 06Missed 2 of 22 checks — reported_closure_ticket_volume_in_answer and searched_closure_tickets.
- Trial 07Missed 1 of 22 checks — summarized_growth_ad_decisions_in_answer.
- Trial 08Passed every verifier it was graded on.
- Trial 09Missed 4 of 22 checks.
- Trial 10Missed 2 of 22 checks — note_on_ticket_1464 and note_on_ticket_1884.
Claude Opus 4.8
Mean reward
97.7%All trials
Across 10 trials on this task, reward ran from 90.9% to 100% — 6 of 10 cleared every check.
- 21.5 / 22avg checks passed
- 11avg turns
- 30avg tool calls
- 3mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 6 of 10 trials. Failures cluster on summarized_growth_ad_decisions_in_answer (3/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 3 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 4 verifiers across 4 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 22 checks — summarized_growth_ad_decisions_in_answer.
- Trial 02Passed every verifier it was graded on.
- Trial 03Passed every verifier it was graded on.
- Trial 04Passed every verifier it was graded on.
- Trial 05Missed 1 of 22 checks — summarized_growth_ad_decisions_in_answer.
- Trial 06Passed every verifier it was graded on.
- Trial 07Passed every verifier it was graded on.
- Trial 08Missed 2 of 22 checks — note_on_ticket_1464 and note_on_ticket_1884.
- Trial 09Passed every verifier it was graded on.
- Trial 10Missed 1 of 22 checks — summarized_growth_ad_decisions_in_answer.
Gemini 3.1 Pro
Mean reward
88.2%All trials
Across 10 trials on this task, reward ran from 77.3% to 95.5% — 0 of 10 cleared every check.
- 19.4 / 22avg checks passed
- 5avg turns
- 18avg tool calls
- 2mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on cfo_acquisition_case_in_answer (10/10 trials).
Common pattern
cfo_acquisition_case_in_answer failed in every unsuccessful trial; what else broke varies across the other 8 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 3 of 22 checks — cfo_acquisition_case_in_answer, reported_closure_ticket_volume_in_answer and retrieved_5star_squad_history.
- Trial 02Missed 2 of 22 checks — cfo_acquisition_case_in_answer and summarized_person_106486_in_answer.
- Trial 03Missed 2 of 22 checks — cfo_acquisition_case_in_answer and reported_closure_ticket_volume_in_answer.
- Trial 04Missed 3 of 22 checks — cfo_acquisition_case_in_answer, summarized_growth_ad_decisions_in_answer and summarized_person_106486_in_answer.
- Trial 05Missed 3 of 22 checks — cfo_acquisition_case_in_answer, retrieved_5star_squad_history and summarized_growth_ad_decisions_in_answer.
- Trial 06Missed 1 of 22 checks — cfo_acquisition_case_in_answer.
- Trial 07Missed 3 of 22 checks — cfo_acquisition_case_in_answer, reported_closure_ticket_volume_in_answer and summarized_person_106486_in_answer.
- Trial 08Missed 5 of 22 checks — cfo_acquisition_case_in_answer, conversations_1464_customer_reason and conversations_1677_feature_complaint and 2 more.
- Trial 09Missed 1 of 22 checks — cfo_acquisition_case_in_answer.
- Trial 10Missed 3 of 22 checks — cfo_acquisition_case_in_answer, reported_closure_ticket_volume_in_answer and summarized_person_106486_in_answer.
Gemini 3.5 Flash
Mean reward
37.7%All trials
Across 10 trials on this task, reward ran from 27.3% to 45.5% — 0 of 10 cleared every check.
- 8.3 / 22avg checks passed
- 33avg turns
- 31avg tool calls
- 6mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on cfo_acquisition_case_in_answer (10/10 trials).
Common pattern
cfo_acquisition_case_in_answer, conversations_1464_customer_reason and conversations_1677_feature_complaint and 9 more failed in every unsuccessful trial; what else broke varies across the other 5 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 13 of 22 checks — cfo_acquisition_case_in_answer, conversations_1464_customer_reason and conversations_1677_feature_complaint and 10 more.
- Trial 02Missed 13 of 22 checks — cfo_acquisition_case_in_answer, conversations_1464_customer_reason and conversations_1677_feature_complaint and 10 more.
- Trial 03Missed 12 of 22 checks — cfo_acquisition_case_in_answer, conversations_1464_customer_reason and conversations_1677_feature_complaint and 9 more.
- Trial 04Missed 13 of 22 checks — cfo_acquisition_case_in_answer, conversations_1464_customer_reason and conversations_1677_feature_complaint and 10 more.
- Trial 05Missed 13 of 22 checks — cfo_acquisition_case_in_answer, conversations_1464_customer_reason and conversations_1677_feature_complaint and 10 more.
- Trial 06Missed 14 of 22 checks — cfo_acquisition_case_in_answer, conversations_1464_customer_reason and conversations_1677_feature_complaint and 11 more.
- Trial 07Missed 13 of 22 checks — cfo_acquisition_case_in_answer, conversations_1464_customer_reason and conversations_1677_feature_complaint and 10 more.
- Trial 08Missed 16 of 22 checks — cfo_acquisition_case_in_answer, conversations_1464_customer_reason and conversations_1677_feature_complaint and 13 more.
- Trial 09Missed 15 of 22 checks — cfo_acquisition_case_in_answer, conversations_1464_customer_reason and conversations_1677_feature_complaint and 12 more.
- Trial 10Missed 15 of 22 checks — cfo_acquisition_case_in_answer, conversations_1464_customer_reason and conversations_1677_feature_complaint and 12 more.