Back in the Race: US Open-Weight Models Closing the Performance Gap

For many defense and industrial software teams, choosing an AI model starts with a hard constraint: sensitive code has to stay on infrastructure they control. Some work on networks disconnected from the internet. Others must use US-developed models.

That makes the strength of US-developed open-weight models a practical question. Chinese developers have set the pace in the global open-weight race, leading US peers on major capability benchmarks and model downloads. But US-developed models are gaining ground. With the recent release of Meta’s Muse Spark Open, we wanted to understand whether the gap is beginning to close, particularly on a task central to our work at Code Metal: translating code across languages while preserving its behavior.

We evaluated Meta’s Muse Spark Open, Reflection AI’s Beam, NVIDIA’s Nemotron 3 Ultra, Thinking Machines Lab’s Inkling and OpenAI’s gpt-oss-120b on an internal benchmark translating C++ to Rust. We also tested five Chinese-developed models as references: GLM-5.3, GLM-5.3 Flash, DeepSeek V4.1 Flash, Qwen3.8 27B, and MiMo v2.6 Pro.

On this workload, Muse Spark Open matched the strongest reference models on judged function-body correctness. Four US-developed model families exceeded 90%: Muse Spark Open, Beam, Nemotron 3 Ultra and Inkling. The results weren’t uniform: gpt-oss-120b reached 42% at its best tested setting. But strong performance across four model families points to a broader set of promising US-developed options.

For teams that need to keep sensitive code on their own infrastructure, that breadth matters. It gives them more models to evaluate within their security and deployment constraints. Judged function-body correctness does not establish a buildable or verified translation; turning these results into reliable software still requires testing and verification. But the pool of models worth putting through that process is growing.

What we measured and how

The internal benchmark used a C++ library covering 14 algorithms, including custom iterators, red-black tree-map functions and fixed-capacity vectors. Here’s how we ran the benchmark and assessed the generated code:

• Fixed agent harness. Each session can read and search C++, edit the Rust crate, run cargo check and look up project conventions. There is no shell or human intervention. A session ends when the model stops or reaches 80 tool rounds, with no external repair loop afterward.
• Repeated tasks. Most settings have 12 sessions and 108 planned function-body evaluations, including repeats of the same tasks. Nemotron has 11 sessions and 94 evaluations; Beam at max has 11 and 104 because their request budgets ran out.
• Blinded assessment. Two runs of the same AI judge compared each written body with its C++ source without the model’s identity. Separate adjudication resolved disagreements. Planted calibration items met a preset agreement threshold. Unwritten bodies count against the score; compilation is measured separately.
• Reasoning settings. We evaluated settings supported by each serving endpoint, omitting some that probing did not distinguish. Setting names are vendor-specific and are not equivalent across models.

Muse Spark Open reaches the top tier

At medium, all 108 planned Muse Spark Open function-body evaluations were judged correct. GLM-5.3 Flash reached 100%, GLM-5.3 99%, and DeepSeek V4.1 Flash’s 1017 checkpoint 98–99% at their strongest tested settings. These small differences do not establish a reliable ranking within that group.

Five of Muse’s 12 medium-setting sessions also passed cargo check. A correct judgment on individual bodies does not establish a buildable or verified translation; the production pipeline still has work to do.

Serving performance shapes the result

We estimate time per task as completion tokens divided by the endpoint’s measured decode speed at eight concurrent requests, plus tool time. This is not elapsed wall-clock time or a comparison on identical hardware. Muse’s measured wall time was 3.4–6.8 times its normalized time because of waits before the first token. Other traffic may have inflated the estimates for Nemotron and GLM-5.3.

Muse at medium used a median 99 normalized seconds per task, compared with 136 for GLM-5.3 Flash and 260 for GLM-5.3 at low. DeepSeek was comparable at low, at 98 seconds, and reached 98% judged correct in 61 seconds with thinking off. These estimates describe the deployments tested.

Muse generated about 8,700 completion tokens per task, the fewest among settings reaching at least 98% judged correct. Its endpoint decoded at 106 tokens per second per session, compared with DeepSeek’s 255. That makes serving performance a useful next experiment. Whether faster serving reduces full-library runtime must be measured with compilation, repair and verification included.

More reasoning is not always better

Muse reached 100% judged correct at medium and xhigh, but xhigh used 2.5 times the normalized time and nearly three times the tokens. At their highest tested settings, GLM-5.3 Flash fell to 34% and GLM-5.3 to 11% of planned bodies judged correct. They wrote fewer bodies, though nearly all the bodies they did write were judged correct. Some DeepSeek sessions exhausted their reasoning budget before writing code. The useful setting depends on the task, harness and budget.

How to read these results

The benchmark measures a constrained generation task. It does not measure a complete library migration or establish that a model is ready for a particular deployment.

• Limited coverage. Results come from one codebase and one language pair. The map tasks cover construction, copying and indexed access, not insertion, erasure or rebalancing. Repeated tasks and small session counts limit generalization.
• Judgment uncertainty. Two Muse code patterns split the initial judges before adjudication ruled them correct. Reversing those decisions would put medium at 96–97%. Reversing one DeepSeek thinking-off decision would put it at 97%. A 100–100 bootstrap interval reflects identical sampled scores, not certainty about correctness.
• Serving and precision. Qwen3.8’s server errors ended several sessions, and MiMo v2.6 Pro was available only with thinking on. Their scores reflect those conditions. Muse was tested in NVFP4, a 4-bit build. Beam was served in BF16 on eight NVIDIA B300 GPUs.
• Model coverage. We did not test Kimi K3. The DeepSeek results highlighted here use checkpoint 1017.

What this means for deployment

US-developed open-weight models now offer substantially better generation quality on this benchmark than our gpt-oss-120b baseline. Choosing among them still requires checking license terms, hardware capacity and performance in the intended environment. Our platform’s model-agnostic design lets us evaluate new releases against the same tasks and acceptance criteria.

The next step for Muse Spark Open is a complete library run through the production pipeline, with compile feedback, repair, tests and verification. We will also evaluate serving changes and tune the tools, prompts and reasoning settings around the model. That will show whether the single-session gains translate into shorter, successful migrations.

At Code Metal, the model is one component of a translation system. Our pipeline compiles, tests and compares generated Rust against the original C++ before release. Better models make that bar easier to clear. They don’t remove it.

Appendix of key results

Correctness is the share of planned function bodies judged correct, including repeated tasks. Time and completion tokens are per-session medians. Compilation means the session wrote at least one body and the crate passed cargo check; it does not imply that all planned bodies were completed.