Germany has released what it claims is Europe's best open-weight language model. It also accidentally proved that the open-source approach to AI — transparency, public data, community scrutiny — isn't just a philosophy. It's the only thing that catches you when you mess up.
Soofi S, a 30 billion parameter model from a research consortium coordinated by the KI Bundesverband (German AI Association), topped benchmarks in both English and German among fully open models. Then a researcher named Elie Bakouch spotted something troubling in the publicly available training data: the model had been trained on paraphrased questions from the GPQA evaluation benchmark — including the notoriously difficult GPQA-Diamond variant.
Seven days later, the consortium confirmed it. They didn't hide it. They didn't spin it. They published a revised report, removed the affected scores, and credited the community for catching it.
The Model That Could Have Been Europe's Answer
Soofi S is one of the first large language models trained entirely on Deutsche Telekom's Industrial AI Cloud in Munich — sovereign European infrastructure running on 512 Nvidia B200 GPUs, powered by renewable energy, cooled with water from the Eisbach canal, and feeding waste heat into the surrounding neighborhood.
It uses a hybrid Mamba-Transformer architecture with 31.6 billion total parameters but activates only ~3.2 billion per token. That puts its compute cost closer to a 3B model than a conventional 30B model. The consortium adopted Nvidia's Nemotron architecture without modification, combining Mamba-2 layers with standard attention layers.
The key architectural advantage? Memory behavior. In conventional transformers, the KV cache grows linearly with context length. Only 6 of Soofi S's 52 layers maintain such a cache. The practical payoff: at 40,000-token context with 32 parallel requests, Soofi S generates roughly eight times more tokens per second per GPU than dense models in the 14-24 billion parameter range. Throughput stays nearly flat from 4,000 to 256,000 tokens.
The training mix was deliberately weighted toward German: 7.2% in phase one, rising to 15.3% in phase two. For comparison, Nvidia's Nemotron reference recipe allocates all non-English languages combined to just about 5%. The consortium processed ~27 trillion tokens across three phases, totaling about 253,000 GPU-hours between March and May.
The Benchmarks: Impressive Until They Weren't
In evaluations against 16 other open models, Soofi S led all fully open models on aggregate scores for both English and German. That includes OLMo 3 32B from Allen Institute for AI and Apertus 70B from ETH Zurich and EPFL. Against every European sovereign baseline, it came out ahead on all German benchmarks, sometimes by double-digit margins.
Code benchmarks were strong: 73.8% on HumanEval, 70.2% on MBPP, and 84.2% on the German MBPP variant. On INCLUDE-DE, a test for Germany-specific regional knowledge, it tied for first place at 61.2 points with Alibaba's Qwen3.5 35B-A3B.
But the GPQA-Diamond score — the one that made headlines — climbed from 32.3 to 43.4 points during training. The team initially saw this as genuine capability gain. It wasn't. The test questions had been paraphrased into the training data because GPQA publishes its entire evaluation set as a single Hugging Face split labeled "train." Soofi's pipeline selected content by label name rather than semantics, ingesting evaluation items as if they were training data.
How the Contamination Happened
Here's the quirk: GPQA has no separate training set on Hugging Face. All of its test material sits under the default label "train." Because the data pipeline selected content based on these split names, the test set ended up mixed in with practice questions.
A full audit triggered by the discovery turned up three more benchmarks with the same pattern: TruthfulQA, BLiMP, and Inverse Scaling. None of those are used in Soofi-S's evaluation, so they don't affect the results — but the consortium disclosed them anyway so independently produced scores wouldn't be read as genuine performance.
The contamination went undetected during training because the model's improvement on GPQA tasks was gradual — climbing from 32.3 to 43.4 across annealing checkpoints. This fell within the normal range of progress and looked no different from legitimate gains on other tests. There was no sudden spike that could have served as a warning sign. As the authors noted: "slow inflation that is indistinguishable, at the trajectory level, from genuine capability gains."
The Open Source Advantage: Community Caught It
Bakouch posted his finding on X on July 15 after examining the publicly available QA-base dataset. The consortium didn't respond for seven days, then published version 3.0 of their pretraining report acknowledging the contamination.
"We state this as an explanation of the failure mode, not as an excuse," the report reads. "Verifying that a split labeled train is in fact a training split was our responsibility."
Nicolas Flores Herr from Fraunhofer IAIS pointed to the incident as evidence that the open approach works: the community could only find the problem because the data was publicly available. The team has made roughly 152,000 individual evaluation results available for verification. A separate evaluation by consortium partner Ellamind, using deliberately withheld test data the model was guaranteed not to have seen during training, confirmed the remaining results.
The corrected QA-base dataset is now live on Hugging Face with items from all four affected benchmarks removed. Future mixtures will get allowlist validation plus n-gram screening against the complete evaluation suite.
What It Means for European AI Sovereignty
Soofi S was designed as part of European sovereignty projects — a model that can run on sovereign infrastructure, be tested in industrial applications, and compete with the highest-performing international open-weight models. The project is funded by the German Federal Ministry for Economic Affairs and Energy as part of the European IPCEI-CIS program.
Participants include Fraunhofer Institutes IAIS and IIS, DFKI, TU Darmstadt, University of Würzburg, L3S Research Center, Berlin University of Applied Sciences, and companies Ellamind and Merantix Momentum.
The consortium is now looking for industry partners for the next phase to test the model in applications involving technical documents, code generation, and agent-based systems. The fine-tuned instruct and reasoning variants are in beta testing and expected to ship under a permissive license in coming weeks. The larger Soofi L model is already in training.
The European context matters here. After Washington briefly cut off Europe from Anthropic's models in June, EU technology chief Henna Virkkunen declared AI a geopolitical weapon and began racing to build Europe's own stack. Soofi S represents exactly that effort — a model trained on sovereign infrastructure, with data sources auditable by any researcher who wants to look. That's different from the opaque training pipelines of American labs, where even the model cards are incomplete guesses at best.
Notably, the consortium doesn't meet a stricter proposal for a European open-data definition because 1.3% of the training mix comes from Genios, a commercially licensed corpus of 193 million newspaper articles. About 99% of the training mix can be independently reconstructed.
🔥 Hot Takes
1. The GPQA contamination is a feature, not a bug, of open-source AI. If Soofi S had been a closed model from Anthropic or OpenAI, nobody would have inspected its training data. Nobody would have cross-referenced it against GPQA's test set. The contamination happened because of a simple labeling quirk — a Hugging Face split called "train" that was actually a test set. But the fix happened because the data was public. Open source doesn't prevent mistakes. It makes them correctable. That's the entire argument for openness in one incident.
2. Europe's AI sovereignty play is real — and it needs to scale. Soofi S proves the technical capability exists: a European consortium, European infrastructure, European funding, producing a model that competes with the best open-weight models globally. The hybrid Mamba-Transformer architecture delivers 8x throughput over dense models at long contexts. The German-language focus gives it an edge US models can't match. But 30B parameters is a drop in the bucket compared to the trillion-parameter arms race. If Europe wants true sovereignty, it needs bigger models, more GPUs, and faster iteration. The infrastructure is there — Deutsche Telekom's Munich facility ran the training. Now it needs investment at scale.
3. The "overtrained" criticism misses the point of MoE architectures. Critics pointed to Soofi S's training ratio — several hundred to one tokens per parameter — as violating Chinchilla scaling laws. But Michael Fromm correctly notes those laws apply to dense models, not Mixture-of-Experts. In MoE architectures, individual experts benefit from seeing the same documents repeatedly. The real comparison isn't Chinchilla's sweet spot; it's whether the model performs well on benchmarks it hasn't seen. And Soofi S does — except for GPQA, which it did see. Oops.
4. The commercial licensing loophole is Europe's Achilles heel. 1.3% of Soofi S's training data is commercially licensed Genios newspaper content. That's small, but it breaks the "fully open" claim and means the model can't meet stricter European open-data definitions. If Europe wants truly sovereign AI, it can't rely on commercial licenses for even tiny fractions of its training data. It needs to build its own open corpora at scale — European web text, European academic papers, European government documents. The Genios dependency is a reminder that Europe still relies on commercial data pipelines that may not align with sovereignty goals.