all research

Architecture & Predictive Design · 8 min read

A 7B model that beat Claude and GPT-5.2 at reading building regulations

Fine-tuning plus reinforcement learning on a narrow regulatory task outperformed frontier models zero-shot. The argument for training rather than prompting, and for keeping regulation on your own hardware.

−23.8%

Tree edit distance vs SFT baseline

−38.6%

Token-level Levenshtein distance

7B

Parameters, locally hostable

Analysis of published research

Reinforcement learning to improve large language model-based automated code compliance systems

Jack Wei Lun Shi, Minghao Dang, Wawan Solihin, Leong Hien Poh, Justin K. W. Yeoh · arXiv, National University of Singapore · 21 June 2026

Automated code compliance means turning regulatory text into something a computer can execute against a model. The obvious approach is to ask a large language model to do the translation.

The obvious approach fails in a specific and dangerous way. The models generate rules that look right and are wrong, or invent requirements that do not exist. In compliance, a plausible-but-wrong rule is worse than no rule at all, because no rule gets noticed and a wrong rule gets trusted.

Two stages, doing two different jobs

P4IR, the framework in this paper, splits the problem. Supervised fine-tuning first, to instil the domain knowledge and get the semantics right. Then Group Relative Policy Optimization, a reinforcement learning method, to improve the structure of what gets generated and cut hallucination.

The intermediate representation is a code skeleton: the classes, functions and parameters an executable rule would need, without the implementation. The reward signal is Jaccard similarity between the generated skeleton and a reference one, measured over those three components.

Two details I found interesting. The base model is Mistral 7B Instruct v0.3, chosen specifically because it is open source. And the GRPO stage deliberately omits chain-of-thought reasoning steps, which is against current fashion and evidently worked.

Improvement over supervised fine-tuning alone, from the paper's abstract
percent reduction in distance, higher is better
Token-level Levenshtein distance38.6
Tree edit distance23.8

The result that should change how firms buy

In zero-shot comparison, this 7B model outperformed Claude Opus, Sonnet 4.5, GPT-5.2 and Qwen on both code structure and semantics.

Sit with that. A model small enough to run on a workstation, trained on the right domain data, beat the frontier on a narrow regulatory task. The GRPO stage also produced a small but statistically significant reduction in false positives, which in compliance is the metric that decides whether anyone keeps using the system.

On a narrow task with good training data, the question stops being which frontier model you subscribe to.

Why this matters commercially, not just technically

Two arguments follow, and they compound.

The second point lands harder in Ireland and the UK than the paper's authors probably intended. Every AI conversation with a public sector client or a large contractor eventually reaches data residency, and the honest answer for a frontier-API architecture is complicated. For a locally hosted 7B model it is one sentence.

Where this stops

Singapore regulations only. Building codes are not transferable, and a model trained on Singaporean requirements tells you nothing about how it performs on Irish Technical Guidance Documents or UK Approved Documents. The training data is the moat and it does not travel.

More importantly, the metrics measure similarity to reference skeletons rather than whether the resulting checks are correct when run against a real model. That is a meaningful gap. Producing a well-structured rule and producing a rule that catches the right violations are different achievements, and only the first is measured here.

Work with me

Run this model against your own project

I am Kanishk Kapoor, Technical Accounts Manager at AI Institute in Dublin. I build agentic AI systems with built-environment teams across Ireland and the UK. If any figure here looks wrong for your business, that is the useful conversation. Send me your assumptions and I will re-run it.

Continue reading