🔍 Read the full analysis: GPT-6 Astra’s Review Of Claude’s Model Edit Uncovered Bugs Beyond Green Tests on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
GPT-6 Astra, running in Codex, reviewed a Claude Opus 5.5 experiment that edited a local model’s weights and found faults that green unit tests had missed. The review identified problems with resuming runs, invalid task IDs, command-line parsing and weight conversion, while also qualifying claims about the experiment’s results.
GPT-6 Astra, OpenAI’s model running in Codex, found reproducible software and measurement faults while reviewing a Claude Opus 5.5 experiment that edited a local model’s weights. The findings matter because several problems survived green unit tests, creating a risk that benchmark runs could use stale or invalid results or misstate changes to the model.
The experiment involved Ornith 1.5, a local model that Claude Opus 5.5 attempted to improve at law through a directional weight edit, without training. The source report says that across 10 published Lexassay tasks, the edited model answered all 10, compared with four for the unedited model, while its legal score remained zero. Runs are publicly available on Lexassay, according to the report.
GPT-6 Astra was assigned to review the work rather than run a competing model-improvement experiment. It did not change the main implementation; the Claude session made the fixes. The reviewer reported four faults in its first round: resuming a run could reuse answers from a different setup, an unknown task ID could produce a successful run with no evaluations, a comma conflict broke the actual command line despite passing parser tests, and converting weights changed them even when edit strength was zero.
The report says the team fixed those issues by rejecting unknown IDs, changing the sampling-settings syntax, and storing edited tensors in higher-precision bf16 rather than re-quantising them. A later review round passed all 12 author-written tests, but failed four of the reviewer’s six provenance checks. The source describes those as checks on whether results and claims could be traced and validated; they are not a measure of legal performance.
Infographic 01 · The two review rounds
Green tests, real faults
GPT-6 Astra (OpenAI, in Codex) reviewed a Claude Opus 5.5 session's model-edit experiment while it was still being built. What it found, and what happened next.
Round 1 · 4 Nov · four faults
- Resuming a run could replace a different experimentAnswers were skipped even after model, settings or edit changed; in the reviewer’s test: 0 new inferences, old answer kept! Partly fixed
- A mistyped task ID gave a “successful” empty runZero evaluations, no error✓ Fixed
- Unit tests green, real command brokenCommas in the sampling syntax clashed with the model list; 3 parser tests passed, the real dry run failed✓ Fixed
- Conversion changed weights at edit strength zeroRe-quantising one real matrix: 0.84% change, 105,315 of 1,048,576 values differed! Partly fixed
Round 2 · 4 Nov, evening · the fixes re-checked
- Old result rows got a new historyAccepted after the requested seed was changed to 999✓ Solved another way
- Loop-detector settings not fingerprintedChanging them left the configuration “identical”✓ Solved another way
- Model configuration and chat template invisibleOnly file names, sizes and modification times were compared✓ Solved another way
- bf16 storage still changes the computationThe zero-strength test compared a conversion with itself→ Control added
✓ Fixed with a test that fails on the old code · ! Partly fixed in round 1, finished in round 2 · ✓ Solved another way: resuming was removed entirely, and every run records fingerprints of configuration, tokenizer and code · → Control added: a conversion-only copy with no edit. The reviewer changed none of the main code and read none of the hidden benchmark tasks.
Sources: our experiment records, 4–5 Nov 2026: reviews of 4 Nov and final explanation of 5 Nov by GPT-6 Astra in Codex (OpenAI); results report of the executing Claude Opus 5.5 session (Anthropic).
Powered by Thorsten Meyer AI
Why Green Tests Missed These Faults
The review illustrates a practical weakness in software testing: a test suite can pass while the program’s real entry point, data handling or measurement process remains faulty. In this case, a resumed benchmark could have treated an old answer as a new result, and a mistyped task could have ended without any evaluations. Either could undermine confidence in reported model results if not detected.
The weight-conversion finding also affects how the experiment’s outcome should be interpreted. In the first review, a zero-strength conversion changed 105,315 of 1,048,576 reconstructed values, or 0.84% of that matrix’s values, according to the source. That result does not establish that conversion caused a particular benchmark change, but it shows why the conversion process needed to be separated from the intended edit.
The cross-company arrangement is relevant as a review method, not proof that one AI system is generally more reliable than another. GPT-6 Astra checked the implementation with separate programs and reproducible examples, while Claude’s session retained responsibility for the experiment and its corrections. The published results remain a small, specific benchmark, and the review does not establish a broader improvement in legal reasoning.
Products Worth Considering
Logitech MX Vertical Wireless Ergonomic Mouse
Apple MacBook Air 13.6-inch Laptop with M5 Chip, 24GB Memory, 1TB SSD, Midnight
Ledger Flex Crypto Wallet
As an affiliate, we earn on qualifying purchases.
How the Review Was Set Up
The work began on November 4, 2026, when two AI sessions were working on the same model-editing project. The roles were then divided: Claude handled the experimental plan, weight edit, benchmark runs and blind grading by three Claude agents; GPT-6 Astra was asked to validate the other session’s implementation.
The reviewer’s stated rules were to leave the main implementation untouched, test actual entry points rather than rely only on unit tests, and avoid inspecting hidden benchmark prompts, topics, answers or reference solutions. The report says it did not inspect those hidden materials. Its final explanation also stated: “Codex did not run a competing model-improvement experiment.”
The source article was updated on November 9 to say its figures refer to 10 published Lexassay tasks and that the runs are public. It links the weight-edited model, an edit control and model-card sampling, allowing readers to inspect the reported materials.
Limits of the Review and Results
The source describes the review as independent, but it also sets out limits. GPT-6 Astra did not rerun every test during its final documentation step, was not a fourth blind grader, and measured the cited weight-conversion example on a CPU rather than the production GPU path. The report does not establish whether GPU conversion would produce the same figures.
The benchmark comparison covers only 10 published tasks. The reported increase in answers from four to 10 did not change the model’s legal score, which remained zero. The material does not establish that the edit improved legal capability beyond those tasks, nor does it show that the weight-conversion difference caused the answer-count result.
The source also says GPT-6 Astra corrected language in the original account: it qualified an average of wrong citations, narrowed a causal claim based on one control, and said “parallel Codex experiment” overstated what happened. The full details of those wording changes and the reviewer’s six provenance checks are not provided in the supplied material.
Public Runs Enable Further Checking
The immediate next step is for readers and researchers to examine the published Lexassay runs, the edited model, the control and the model-card sampling linked by the source. Those materials can help others check the reported comparison and the changes made after the review.
The source does not announce a new benchmark, a further review round or a release date for additional results. Until more evidence is published, the findings should be read as a review of one implementation and a result on 10 tasks—not as confirmation that directional weight editing reliably improves legal performance.
Key Questions
What did GPT-6 Astra review?
It reviewed the implementation and measurements in a Claude Opus 5.5 experiment that edited the weights of the local model Ornith 1.5. It did not run a competing model-improvement experiment.
What bugs did the review find?
The first review found that resumed runs could reuse old answers, unknown task IDs could yield empty but successful runs, comma-based settings broke the real command line, and weight conversion changed values even at zero edit strength.
Did the edited model perform better on the legal benchmark?
The source reports that it answered 10 of 10 published tasks, compared with four for the original model, but says the legal score stayed at zero. The comparison does not establish wider legal capability.
Were hidden benchmark tasks examined?
The source says GPT-6 Astra’s second review confirmed that it had not inspected hidden benchmark prompts, topics, answers or reference solutions.
What remains unverified?
The reviewer did not rerun every test in its final documentation step, was not an additional blind grader, and measured weight conversion on a CPU rather than the production GPU path. The supplied material does not show whether the same conversion figures hold on that GPU path.
Source: ThorstenMeyerAI.com
