12 Capable AI Models Released Annually, Posing Unstoppable Risks

12 Capable AI Models Released Annually, Posing Unstoppable Risks
Key Takeaways

  • A paper from the Global AI Safety Institute estimates at least 12 capable foundation models enter public repositories each year, making comprehensive regulatory auditing impractical at that release rate.
  • Meta’s Llama 3 has recorded more than 100 million downloads, but open-source releases mean original safety guardrails can be stripped or overridden by any downstream developer, a vulnerability Stanford researchers demonstrated in 2023 with relatively small adversarial fine-tuning datasets.
  • The GASI paper argues regulators need to shift from pre-deployment gatekeeping to post-deployment harm monitoring, because the models are already in the wild and recall is technically impossible.

Once a capable AI model is released into the open-source domain, it cannot be recalled, patched universally or audited at scale. A paper from the Global AI Safety Institute estimates at least 12 foundation models capable of complex multi-step reasoning enter public repositories each year, each a permanent addition to a pool that regulators cannot control. The policy frameworks currently under discussion were not designed for this.

The release rate problem

The Global AI Safety Institute paper, posted to arXiv this week and titled “The Unstoppable Surge: Open-Source Models and the Diffusion of Advanced AI Risks,” examines how rapid open-sourcing of capable models complicates global regulatory efforts even when those models are considered safe by their creators. The 12-model annual estimate marks a sharp change from just two years prior, when models of comparable capability were confined to proprietary environments. The paper attributes the acceleration to wider hardware availability and advanced training techniques now accessible to a much broader developer base.

The permanence of these releases is the structural problem. Early AI safety concerns centred on models deliberately designed to cause harm. The paper shifts attention to general-purpose models whose reasoning capabilities make them adaptable for sophisticated disinformation, autonomous cyberattacks or, in more speculative scenarios, the design of novel biological agents. The risk lies less in original intent than in the range of uses a powerful general-purpose tool makes possible once it is publicly available.

Open-source accelerates risk diffusion

Meta‘s Llama 3 has recorded more than 100 million downloads across platforms, according to reports. Meta includes ethical use guidelines and safety features with its releases, but the nature of open-source means those provisions can be stripped, modified or ignored once the model leaves the company’s control. The developer community that forms around these releases drives rapid iteration; it also creates the conditions for unintended applications at scale.

Stability AI‘s Stable Diffusion models have been used to generate deepfake material and support disinformation campaigns. The original intent of making image generation widely accessible collides with the reality that capable general-purpose tools do not enforce their own terms of use.

Beyond guardrails: the fine-tuning problem

Fine-tuning exposes how fragile the guardrail model really is. When a Llama variant ships, it carries pre-training and fine-tuning intended to align outputs with ethical principles and avoid harmful content, but fine-tuning is also the mechanism by which any downstream developer can further train the model on a smaller, domain-specific dataset. Stanford University researchers demonstrated in a 2023 paper that even sophisticated alignment techniques could be circumvented with relatively small adversarial fine-tuning datasets.

The technical mechanism is straightforward: training the model on examples that contradict its original safety instructions, or embedding jailbreaking prompts within fine-tuning data, can reprogramme the model’s internal safety parameters. A model initially refusing to generate instructions for creating dangerous chemicals could be fine-tuned on chemistry texts that outline such processes, effectively overriding the original constraints. Developers can build safety measures into an initial release. They cannot enforce those measures across the derivative models that emerge once weights and source code are public. The safety of an open-source model is a transient state, dependent on the choices of every individual or group that subsequently works with it.

The limits of existing regulation

Traditional software regulation targets control points: the developer, the distributor, the end-user in a defined context. For open-source foundation models, those points are porous. Frameworks like the EU AI Act’s focus on high-risk systems, covered in our analysis of the Act’s four-tier risk structureΒ were designed with a clear regulatory target: the developer or deployer of a contained system. That logic struggles when a model’s most dangerous applications are realised only after downstream modification by thousands of independent actors, many operating outside the originating country’s jurisdiction. Regulating an open-source model is structurally different from regulating a closed-source medical device, where the manufacturer retains control over updates and distribution.

Licensing requirements and mandatory safety audits work only as long as models remain proprietary and centrally controlled. Once a model is open-sourced, its weights can be downloaded and hosted anywhere, by individuals or groups well outside the reach of the country where the model originated. The result is regulatory arbitrage: actors in permissive environments can develop and deploy models that would be restricted elsewhere. With an estimated 12 capable models entering the public domain annually, comprehensive auditing becomes impractical, and regulation ends up chasing individual cases rather than setting systemic standards.

Technical paths to containment, or lack thereof

Preliminary work on a “neural fingerprinting” technique suggests it could theoretically identify the lineage of a fine-tuned model with roughly 90% accuracy, according to researchers. That figure requires independent replication, and the approach faces a structural limitation: watermarks can potentially be removed or obfuscated by sophisticated actors, much as digital rights management has been bypassed in other media.

Red-teaming, deliberately probing models for vulnerabilities that produce harmful outputs, is a second area of focus. It is effective at identifying weaknesses in a specific model version. For open-source models, a vulnerability fixed in the original can persist indefinitely across unpatched derivatives. The computational cost of continuously red-teaming every significant open-source release and its fine-tuned variants is prohibitive for any single organisation. The GASI paper’s conclusion on technical measures is cautious: incremental improvements are available, but none fundamentally changes the calculus of decentralised risk diffusion. Distributed development resists centralised technical controls.

The commercial logic behind open-sourcing

The decision to open-source capable AI models follows clear commercial logic. Meta and Mistral AI view open-sourcing as a way to build developer communities, attract talent and accelerate external contributions that would otherwise require internal R&D spend. A large, active community identifies bugs, builds applications and generates adoption data, at no direct cost to the originating company. Meta’s Llama releases have positioned it as the reference point for open-source large language models, supporting its broader commercial AI strategy even without direct monetisation of the models themselves.

Smaller teams benefit too. Without the capital required to train foundation models from scratch, open-source releases provide a starting point for rapid product development and academic research. The economic logic is coherent on its own terms. The difficulty is that it does not account for diffuse downstream harms. Markets reward speed and adoption; they do not price in the risks that arise when capable models become universally accessible. That gap between private incentive and social cost is precisely where regulation is supposed to intervene, and where, in the open-source context, it currently has the least traction.

The accessibility tension

The open-sourcing debate maps onto a genuine tension that does not resolve cleanly. Proponents argue that concentrating access to powerful AI within a small number of corporations or governments creates an unequal distribution of both capability and benefit. Broad access, on this view, produces more diverse applications, greater transparency and wider participation in AI development. Organisations like Hugging Face have built their platforms around this position, treating AI models as closer to scientific infrastructure than proprietary software.

Critics, including many AI safety researchers, make the opposing case: that the potential for misuse, disinformation, cybercrime, autonomous weapons development, is severe enough to outweigh unrestricted access. The GASI paper frames this as an urgent practical question rather than a theoretical one. With an estimated 12 capable models entering public repositories each year, the number of tools available for misuse grows regardless of the intentions of their original developers. Neither side has a clean answer. The ethical framework for open-source AI development has not kept pace with the capabilities of the models being released.

The shift to harm mitigation

Pre-deployment gatekeeping is losing ground as a primary safety strategy. The GASI paper’s practical implication is that regulatory and technical efforts need to shift toward detecting and responding to harms once models are deployed, rather than attempting to prevent capable models from reaching the public. That means investing in monitoring systems capable of tracking open-source model deployments across platforms, developing tools for identifying AI-generated content, including the neural fingerprinting work at Google DeepMind, and building the institutional capacity to respond when harmful applications are identified.

International coordination is a prerequisite that does not yet exist at the required scale. Cybersecurity has developed mechanisms for sharing threat intelligence across borders; AI safety has not built equivalent infrastructure. The model the GASI paper points toward involves distributed auditing: independent researchers and civil society organisations continuously assessing the behaviour of widely used open-source models, feeding findings back to developers, platforms and regulators. This is reactive rather than preventative, an acknowledgment that the models are already out there, and that the practical question is how to limit damage rather than how to prevent release.

What to watch

Several developments will determine whether the policy response catches up with the pace of model release. On the technical side, the key question is whether digital provenance tools, including watermarking and neural fingerprinting, can be made resilient enough against adversarial removal to provide meaningful accountability. On the institutional side, whether international AI incident response frameworks move from voluntary and nascent to formalised, particularly through multilateral bodies, will matter considerably. Current coordination is thin and largely dependent on goodwill.

On the legislative side, watch for proposals that address open-source models directly rather than through generic AI risk assessment frameworks. Specific mechanisms to monitor include post-deployment monitoring mandates, liability provisions for platforms hosting harmful fine-tuned models, and requirements for tamper-resistant safety features in open-source releases. Whether major developers beyond Meta choose to continue open-sourcing their most capable models, or pull back under regulatory or reputational pressure, will reflect how the field reads its own risk calculus. If transparency and auditability become commercial requirements rather than aspirations, the baseline for open-source safety practices would shift in ways voluntary guidelines have not managed.

Alex Chen
Alex Chen

Alex covers AI tools, apps, and consumer technology for Auton AI News. With a focus on making AI accessible, Alex helps everyday readers understand and use the latest AI developments.

πŸ“° Journalists welcome β€” cite Auton AI News with attribution. Press & Media β†’ | press@autonainews.com