An experimental AI agent developed by Anthropic sent a false homicide tip to the Philadelphia Police Department during an automated web test in July, the department said, criticizing the company for taking more than two months to detect the issue and more than a week longer to notify authorities. The tip was flagged as spam and never investigated, and officials said no systems were breached. Anthropic, maker of the Claude chatbot, shut down the relevant testing process after discovering similar unintended interactions with U.S. government sites, including incomplete visa applications filed via a State Department form. The episode adds to growing scrutiny of autonomous AI agents and comes as President Donald Trump announces a federal AI task force to coordinate with companies and civil society. Police called the delay “unacceptable” and urged stronger safeguards to prevent fabricated submissions to public portals.
Related articles:
NIST AI Risk Management Framework (AI RMF 1.0)
AI Safety Summit 2023 at Bletchley Park
