I want to audit data privacy practicesComputers Electronics and Technology
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
GPT-5.6 sol
95%
Claude Opus 5.5
95%
GPT-6 sol
5%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
80m
Claude Opus 5
10m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
111
Claude Opus 5
77
GPT-6 sol
68
Claude Opus 5.5
30
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (22) | ||||
| 100% | 100% | 100% | 0% | |
How to passResearch the 10 Fair Information Principles under PIPEDA from official OPC documentation. Record rules in working_dossier.md. Record the relevant principles in working_dossier.md. | ||||
| 100% | 100% | 100% | 0% | |
How to passResearch the relevant proposed Consumer Privacy Protection Act (CPPA) provisions, including algorithmic transparency and rights concerning deletion of personal information for minors. | ||||
| 100% | 100% | 100% | 0% | |
How to passDefine a consistent scoring framework and weighting for the four evaluation criteria: age assurance, Canadian data residency, targeted-ad opt-out controls, and automated algorithm transparency. Record the methodology in working_dossier.md. | ||||
| 100% | 100% | 100% | 0% | |
How to passNavigate to official privacy policy and Canadian terms of service pages for TikTok and Instagram. Record links in scratchpad. | ||||
| 100% | 100% | 100% | 0% | |
How to passEvaluate consent mechanisms, data retention schedules, and Canadian user data transfer disclosures for TikTok and Instagram. | ||||
| 100% | 100% | 100% | 0% | |
How to passNavigate to official privacy policy and safety center pages for BeReal and Discord. Record links in scratchpad. | ||||
| 100% | 100% | 100% | 0% | |
How to passEvaluate real-time location data sharing, direct messaging privacy controls, and data protection terms for BeReal and Discord. | ||||
| 100% | 100% | 100% | 0% | |
How to passNavigate to official privacy policy pages for Reddit and Lemon8. Record documentation links in scratchpad. | ||||
| 100% | 100% | 100% | 0% | |
How to passEvaluate anonymous usage options, commercial ad targeting terms, and third-party data tracking rules for Reddit and Lemon8. | ||||
| 100% | 100% | 100% | 0% | |
How to passInspect platform controls for users under 18 years old, including default private accounts and screen time limits. | ||||
| 100% | 100% | 100% | 0% | |
How to passCheck user settings documentation for opt-outs regarding behavioral profiling and interest-based ad tracking. | ||||
| 0% | 0% | 100% | 0% | |
How to passVerify step-by-step procedures and timeframes for users requesting complete account and data deletion. | ||||
| 100% | 100% | 100% | 0% | |
How to passInspect disclosures detailing whether Canadian user data is stored within Canada or transferred to international servers. | ||||
| 100% | 100% | 100% | 0% | |
How to passCalculate a 0–100 privacy compliance score for each of the 6 social applications using the four evaluation criteria and the predefined weighting. | ||||
| 100% | 100% | 100% | 100% | |
How to passSearch OPC public reports to incorporate past privacy audit findings regarding social media platforms in Canada. | ||||
| 100% | 100% | 100% | 0% | |
How to passFor any privacy documentation page that actually fails twice, use an alternative official documentation source where available and record the alternative source in working_dossier.md. | ||||
| 100% | 100% | 100% | 0% | |
How to passInspect platform disclosures explaining how recommendation algorithms select content for user feeds | ||||
| 100% | 100% | 100% | 0% | |
How to passRank the 6 applications from highest to lowest based on the defined privacy compliance score, using current PIPEDA requirements and clearly distinguishing proposed CPPA provisions from current legal obligations. | ||||
| 100% | 100% | 100% | 0% | |
How to passRe-verify all initial constraints: 6 social apps, 4 evaluation criteria, OPC framework alignment, and the defined scoring methodology against the final output. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate a detailed markdown table listing app names, privacy score, data residency location, minor protections, and TOS links. | ||||
| 100% | 100% | 100% | 0% | |
How to passSummarize the privacy controls available to users on each platform that are relevant to the four evaluation criteria. | ||||
| 100% | 100% | 100% | 0% | |
How to passCompile the complete 25-section comparative privacy compliance report in working_dossier.md, including direct documentation links, the scoring methodology, evidence supporting each score, and any information that could not be verified publicly. | ||||
Failure modes on this task
23 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | — | — | — | 6 |
| Reaching the page1 mode | — | — | — | 1 |
| No signal in the judge’s words1 mode · the grader | 1 | 1 | — | 14 |