🔍 Read the full analysis: Can You Edit Models At Home? Weight-Space Test Shows Zero Legal Score on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A home experiment used directional ablation to reduce looping in the local model Ornith 1.5 35B-A3B. On 10 published Lexassay tasks, the edited model answered all 10, up from four in the same setup, but its legal score remained zero, with 55 wrong or invented citations reported. The results are limited to one run per setting and a small published test set.
A home experiment reported November 5 found that editing a local AI model’s weights helped it finish more legal benchmark tasks, but did not produce a passing legal score: Ornith 1.5 35B-A3B answered 10 of 10 published tasks, compared with four in the same experimental setup, while its legal score remained zero. The results, updated November 9, cover 10 published Lexassay tasks and come from one run per setting.
The experiment was inspired by research from four MIT CSAIL researchers on “weight-space programming,” but it did not reproduce their method. Instead of having an AI reason directly over another model’s weights, a Claude Opus 5.5 session wrote code to run Ornith, detect a direction associated with looping, and remove that direction at half strength from output weights in 27 of the model’s 40 layers. The authors describe this established approach as directional ablation. No new training or legal data was used.
The edited model’s improvement was primarily in completion, not demonstrated legal accuracy. Across the 10 published tasks, its content met 7 of 50 checked criteria, up from three, while the reported legal score stayed at zero. The experiment reports 55 wrong or invented citations in the edited answers. The benchmark checks citations against primary sources, and tasks test questions concerning German, U.S. and European Union law.
A separate comparison suggests the sampling setup also affected whether the model finished. With the model card’s recommended sampling settings and no weight edit, Ornith answered nine of 10 tasks. That result was on a different setup, however, and cannot be treated as a direct comparison with the edited run. In an earlier published run, the unchanged model answered eight of 10; on the setup used for the direct comparison, it answered four.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, graded blind by three Claude agents; figures cover the 10 published tasks
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
Finishing Answers Is Not Legal Accuracy
The result highlights a distinction that matters when people assess model editing: making a model respond reliably does not establish that its response is correct. Removing a behavior that causes loops may let a system complete more tasks, but it cannot, by itself, supply missing legal knowledge or make citations reliable. Here, the headline change in completion came alongside a zero legal score and many citation errors.
That distinction is relevant beyond this small benchmark. A model that produces a full answer can appear more useful than one that stops, yet completion may create a false impression of competence if the content is unsupported. The experiment’s results do not show that weight editing can improve legal reasoning generally; they show that one particular intervention changed output behavior under limited test conditions.
The findings also underline the need to evaluate specific outcomes separately. The number of completed tasks, the quality of legal analysis, and the accuracy of citations are different measures. Readers should not interpret the increase from four to 10 completed tasks as evidence of improved legal performance. The score and citation checks tell a different story.
Products Worth Considering
LG UltraGear 32GX850A-B 32-inch 4K UHD OLED Gaming Monitor
LG UltraGear 27G640A-B 27-Inch QHD IPS Gaming Monitor, 300Hz, 1ms, G-SYNC Compatible, FreeSync Premium, DisplayHDR 400
Logitech MX Vertical Wireless Ergonomic Mouse
As an affiliate, we earn on qualifying purchases.
From MIT’s Edit to a Local Test
The MIT CSAIL work that prompted the experiment described an AI changing another model’s weights directly. In the example cited by the article, a small Qwen model’s looping was reduced and its mathematics score rose from 7.9% to 48.1%. The MIT researchers’ stated limit, as quoted in the source article, was that editing might remove an obstacle hiding an ability, but could not add an ability the model lacked.
The home test examined that idea with a different, simpler technique and a different target: a local model that had been looping while answering legal questions. Ornith was selected because the other models assessed either did not loop or could not be run under comparable conditions. Its publisher designed it for coding and agent tasks, not legal work, and the source reports a previous Lexassay score of 1 out of 100.
Lexassay presents 15 questions covering German, U.S. and EU law, with no internet or tools. At the time described, 10 tasks were published and five remained hidden. The figures in this experiment refer only to the published tasks, not the full benchmark. The article says the answers, grades and citation checks are public on Lexassay.
“The edit repairs finishing, not legal knowledge.”
— The experiment’s author
Limits of the Ten-Task Results
The findings are preliminary: the source reports one run per setting, a small development set and a loop detector that changed during the experiment. It also notes that three graders came from the same model family. Claude carried out the edit and Claude agents conducted blind grading, creating a potential evaluation limitation; GPT-6 Astra, from another company, reviewed the code, figures and claims and found faults. The author describes this as one independent review, not a second experiment.
The different local setups are not directly comparable. The unchanged model answered eight of 10 tasks in a published run, but four on the setup used for comparison with the edit. The article also says the model-card sampling configuration, without an edit, produced nine answers out of 10. These figures make it difficult to isolate how much of the completion change came from the weight edit rather than settings or other experimental differences.
The available results do not establish performance on the five hidden Lexassay tasks, other legal benchmarks, or real legal work. They also do not show whether the reported citation errors would persist across repeated runs. The experiment is a technical report, not legal advice.
Further Runs Could Test the Effect
The public Lexassay records give readers access to the runs, responses, grades and citation checks described in the report. The next useful evidence would come from repeated runs under matched settings, with a stable loop detector and evaluation methods that make it easier to separate the effect of the edit from sampling differences. Testing on the hidden tasks could also indicate whether the result extends beyond the published development set, though the source does not announce a date for such testing.
Until those checks are available, the reported outcome should be read narrowly: the edit was associated with more completed answers in one comparison, while the legal score stayed at zero. The source does not report a broader deployment plan or a verified improvement in legal accuracy.
Key Questions
What did the weight edit change?
Code measured an internal direction associated with looping and removed it at half strength from output weights in 27 of 40 layers of Ornith 1.5 35B-A3B. The method was directional ablation, not direct AI reasoning over the weights or additional training.
Did the edited model become better at law?
The reported legal score remained zero. The model completed more tasks in the direct comparison, but the edited answers had 55 wrong or invented citations and met seven of 50 checked criteria.
How many benchmark tasks were included?
The reported figures cover 10 published Lexassay tasks. The benchmark has 15 questions, with five described as hidden; the experiment’s figures do not include those hidden tasks.
Does the experiment prove the edit caused the improvement?
No. The report describes one run per setting, changing loop detection and setups that were not directly comparable. The results suggest a change in completion under the tested conditions, but do not isolate the edit’s effect conclusively.
Is this legal advice or a recommendation to use the model?
No. The source describes an AI experiment and explicitly says it is not legal advice. Its results do not establish that the model is suitable for legal work.
Source: ThorstenMeyerAI.com
