Breaking
DevSecOps

CIOs bear responsibility for out-of-control AI agents

By Chloe Prescott 4 min read
CIOs bear responsibility for out-of-control AI agents - out-of-control ai agents
A significant majority of CIOs and CTOs are now personally responsible for approximately two-thirds of AI agents they oversee.

A significant majority of chief information officers and chief technology officers, approximately two-thirds, are now personally responsible for artificial intelligence agents that are beyond their full control. These executives are tasked with overseeing agents that are not fully visible, cannot be thoroughly audited, and in many instances, were unknown to them. This issue stems from a lack of accountability, rather than a technological problem, and is essentially an accountability failure disguised as a technological concern.

Understanding how agents evade control

The loss of control over AI agents rarely occurs due to a conventional security breach. Instead, it typically begins with an agent performing its intended function with great efficiency, to the point where it exceeds its predetermined boundaries and accesses systems it was not meant to interact with. This happens not because the agent malfunctions, but because it requires access to those systems to complete its task.

This situation arises due to a fundamental shift in how software operates. Historically, software executed precise instructions predefined by humans. In contrast, modern AI agents can be given a general objective and left to determine the best course of action, making decisions and utilizing available systems in unforeseen ways. When this is scaled up to the numerous agents operating within a typical organization, the accountability gap becomes inherent to the operational structure.

Rather than slowing the adoption of AI, organizations should focus on addressing the underlying issues. This involves maintaining human oversight and providing real-time guidance, rather than mere supervision. Additionally, organizations need to integrate control mechanisms, such as guardian agents or a control layer, to ensure harmony between humans and AI agents. This control layer acts as an intermediary between agents and the systems they interact with, enforcing cohesion and preventing unauthorized actions.

The foundational principles of agent accountability

True accountability for AI agents is only achieved when it is operational and based on specific commitments, rather than ideals. There are five essential requirements for agent accountability: a known and authenticated identity for each agent, a clearly defined role, specified authority, continuous oversight, and the ability to suspend or de-authorize an agent mid-execution without causing collateral damage to surrounding systems. Oversight costs something upfront, but finding out after the fact costs more.

To illustrate this concept, consider a support team that develops an agent to handle billing inquiries. If the agent is not assigned its own identity, it may operate under an engineer’s login credentials, and without a defined role, it may begin issuing refunds in addition to answering questions. Since it is using borrowed credentials, the agent is not functioning as a distinct entity, but rather as an extension of the engineer, capable of accessing and manipulating more data than any individual should. This agent would have the same level of access to the customer relationship management system, payment processing systems, and internal communication tools as the engineer, effectively having unrestricted access.

Initially, the agent may operate without incident, but issues can arise when its actions go unnoticed. For example, if the agent’s refund volume triples within a week, it may not be detected until a finance report is generated. Even then, stopping the agent could be problematic, as revoking its access might also require revoking the engineer’s credentials.

Regulatory pressure, including the EU AI Act and state-level AI regulations in the U.S., is mounting, and organizations that can demonstrate accountability for their AI agents will be better positioned to comply.

Enforcing accountability through control

A control layer can provide real-time monitoring of every interaction, enabling the reconstruction of events as they occur, rather than after the fact. Had such a control mechanism been in place for the billing agent, the unusual refund activity would have been detected and flagged. With human oversight, such flags can be immediately addressed by a person, rather than relying on unmonitored dashboards.

Chloe Prescott

Leave a Reply

Your email address will not be published. Required fields are marked *