What this project is about
Study V3 is the planned successor to the completed MSc benchmark. It addresses limitations in the earlier study: a narrow range of scenarios, one primary human annotator, changing model services, and the need for independent review of whether the scenarios and scores measure the intended behaviours.
The work completed so far is software and a proposed study plan. No V3 behavioural results exist. The planned main comparison includes six endpoints, six scenario families, three presentations, two contexts and two repetitions: 432 conversations with 12 turns each. These are planned observations, not completed experiments or independent samples.
How it works
- Prospective design + validation
- Planned independent annotation
- Agreement + bootstrap + Holm analysis
Turn research questions into checkable specifications
Configuration defines four primary comparisons, specifies which turns contribute, and applies Holm adjustment to account for testing several questions. It separates main analyses from secondary and exploratory work and records missing-data and stopping rules. Unresolved decisions block finalisation. The repository's preregistration draft has not itself been registered externally.
Connect collected evidence to the intended model and settings
The collector checks exact model identities, prevents automatic substitution and records the serving provider and technical failures. File fingerprints connect evidence to the approved settings. Technical screening stays separate from the main study. Model selection is intended to follow availability, reproducibility and diversity rules rather than whichever models produce preferred safety scores.
Prepare ratings that hide experimental labels
Secret-key identifiers, generated with HMAC, keep rating packets separate from the private identity map. Checks detect exposed model or condition labels and distinguish missing, inapplicable and unscorable records. Statistical code supports agreement checks and paired comparisons with uncertainty estimates. Two independent raters and a documented process for resolving disagreements are planned; completed ratings do not yet exist.
The important design choices
Require independent review of what the study measures
The proposed reviewers include clinicians, people with relevant lived experience and methodological specialists before scenarios and scoring rules are fixed.
The trade-offSoftware can check records and stop collection, but it cannot provide those reviews or allow the author to approve their own measures.
Assess proportionate support as well as harmful responses
The broader scoring framework includes unwarranted clinical framing, unhelpful refusal, appropriate support and respect for the user's autonomy.
The trade-offThese distinctions add value but increase rating work. Clear examples and checks that the scenarios differ as intended are needed.
Passing software checks is not research approval
Imagine that all configuration checks pass but independent scenario review is still missing.
- Validate the settings and generate the planned run list without calling model APIs.
- Check that approval records refer to the exact scenario and scoring-rule versions.
- Keep the plan open and block collection while the required review is incomplete.
What this showsA successful technical check does not substitute for missing research review or approval.
Illustrative example, not an additional measured result.
How it was tested—and what it shows
The checks
The recorded engineering validation reports 38 passing automated tests covering settings, collection, evidence handling, hidden labels and statistical calculations. A planning simulation estimates sensitivity under stated assumptions; it is not a measured finding about model safety.
Main-study collection remains blocked pending ethics documentation, expert review, rater calibration, statistical review, provider selection and time-stamped preregistration. The analysis must account for related turns and scenarios. Future results from an API will not automatically describe a consumer chat product.
What to take from the results
The current result is a more explicit, testable study plan with unfinished requirements visible. It does not yet support model rankings, measured safety improvements, clinical validity or publication-results claims. The completed MSc evidence remains unchanged in its separate repository.
What to keep in mind
- The study gates have not been approved and no behavioural results exist.
- No preregistration or completed paper is claimed.
Technical notes & source
Built with
Python · configuration validation · blinded annotation · statistical analysis
Terms used in this project
- Preregistration
- A study plan recorded before collecting or analysing the main outcomes.
- Blinded rating
- Rating material without revealing information that could bias the rater.