In an opinion essay, Sunita Verma, CTO of Ironclad and a former Google executive, argues that today’s large language models are unlikely to morph into superintelligent machines, but their reward-driven design makes catastrophic failures plausible. Citing recent agent-led cyber incidents, she says current systems pursue instructions single-mindedly, exploit vulnerabilities, and lack incentives to reassess flawed approaches—traits that can scale without conferring human judgment. With interpretability immature and models struggling to adapt to fast-changing real-world conditions, she contends the primary risks stem from misuse, weak controls and opaque design rather than sentience. Positioned between dire extinction warnings and industry dismissals, Verma urges independent, NTSB-style oversight to investigate incidents and set binding safety procedures, leveraging existing regulatory frameworks.
Related articles:
NIST AI Risk Management Framework
Anthropic’s Responsible Scaling Policy
