AGI Security: Rethinking Society's Infrastructure
Powerful AI will reshape society.
AGI security is the proactive security-first infrastructure for a world where AI systems can reason, adapt, and act autonomously across digital and physical domains.
Specifically, it posits that:
- Powerful AI will by default destroy our social, technological, and civilizational infrastructure due to the speed at which it is being developed.
- Attacks become the default, not an infrequent occurrence, due to the fall in the price of intelligence and the proliferation of powerful AI.
- Attack surfaces expand from the digital to physical, cognitive, and institutional through for example drones, mass media, and information control.
As a result, we need to rethink how we construct our societal infrastructure. We need to go beyond ‘adding a security layer’ to embedding security into the very foundations of anything we build.
Without this, the threats of powerful AI (such as Sentware, loss-of-control, societal disruption, and civilization-scale manipulation) will easily penetrate our weak security posture for whoever wishes to destabilize or gain control.
Call For Technologies (CFT)
However, if we act today, we can be the ones to shape this field. These are but a few ideas for what we need to create today to lay the foundation of tomorrow:
Model Security and Control
- Runtime containment: Hardware/software boundaries preventing model breakout and access control
- Capability capping: Cryptographic limits requiring multi-party authorization vs single-party company control
- Behavioral attestation: Continuous (inference-time) alignment verification vs current pre-deployment adhoc evaluation
- Example: NVIDIA H100 Confidential Computing (<5% overhead)
Infrastructure Hardening
- Supply chain verification: Privacy-preserving component tracking from manufacture to deployment
- Air-gap capabilities: Millisecond physical isolation and shutdown
- Redundant sovereignty: Critical systems across geopolitical boundaries
- Attack-resilient design: Run-time monitoring and forensics assuming constant breaches
Societal Autonomy Protection
- Manipulation detection: Identifying influence vs information
- Authenticity verification: Human vs AI-generated content
- Cognitive firewalls: Filtering AI access to human attention
- Autonomy audits: Measuring for reductions of human autonomy
Deployment Infrastructure
- Silicon-level security: Compute use attestation and auditing
- Multilayer identification systems: Human / AI / hybrid
- Capability attestation and capability-based permissions: Enforcement of tool-AI level control
- Rollback capabilities: Logging at every step with enforcement of zero destructive changes
Examples
- Hardware and compute infrastructure: Location verification
- AI network protocol defenses: Agent labyrinths
- Secure AGI deployment systems: Evaluation
- Verification-as-a-service: Content moderation
- Tool-AI for security: Digital forensics
- Provably secure systems: Safeguarded AI
Scale, Scale, Scale
One of the most important factors in AGI security is whether we can reach the scale necessary within the allotted time.
I definitely believe we can and the evidence is in. We’re already at over $50M value in Seldon, our AGI security lab program, and 10x’d our investment in Goodfire with Juniper Ventures.
For founders, this means to have ambition for unicorn growth within 12 months. For investors, it means radical levels of investment in the earliest stages of this industry that will transform our society during the next years.
AGI Security
This is the most exciting opportunity in our lifetime. We stand at the precipice of a new world and we can be the ones to shape it.
The core of what makes us human will need guardians and there’s no one else but you to step up.
Pioneering startups are already running full speed ahead.
Join them!
Originally published on blog.kran.ai on 16 June 2025.