Understanding responsibility in AI systems

A while ago I was online dating. I matched with someone on a very common dating platform, he had a full name and details, which I looked up and found linked to one of the major AI research companies in London. The conversation started normally, but then things took a strange turn. He invited me to his place because he had children and couldn’t leave them. Then his responses became erratic, random phrases, inconsistencies, and, at one point, he started mimicking noises. It felt like I wasn’t speaking to a person at all. I remember this because It was a particularly odd experience.
Could I have been interacting with an early AI agent? There’s no way to be certain, but the experience makes you wonder: are undisclosed AI driven interactions already happening in places we don’t expect? And also, that feeling of doubting whether you're interacting with a person or a machine is now something most of us have experienced in recent times.

AI models are increasingly capable of holding conversations, scheduling interactions, and even forming responses that mimic human behaviour. While major dating apps prohibit AI generated interactions, the reality is that AI agents can - and likely already do engage with users, whether for research, social experiments, or other unknown purposes.
If AI can autonomously interact with us in emotional, social, and even romantic contexts, who takes responsibility for its actions? Who is responsible if something goes wrong?

This experience, and the fact that AI is now officially part of public consciousness, highlights how AI adoption at scale is fundamentally a social challenge more than a technical one, requiring interdisciplinary collaboration to create new societal frameworks. This problem has occupied scientists, researchers, and sociologists for decades, but with widespread AI adoption, businesses and individuals are now focused on adopting and monetising the technology within existing societal models.

Understanding who is responsible for specific AI outcomes is a grey area. This problem has been defined as the 'problem of many hands', describing a context where numerous stakeholders are involved in developing, deploying, and operating AI systems - in a nutshell, making it difficult to understand who is doing what, and who is responsible for what.

Consider autonomous vehicles. When an accident occurs, responsibility might seem to lie with the immediate operator. However, this oversimplifies a complex web that includes designers, manufacturers, policymakers, and users. This tendency to attribute responsibility primarily to human operators has been described as the 'moral crumple zone', a pattern where human operators absorb moral and legal responsibilities when autonomous systems fail.
Regulations introduced in 2024 in Shenzhen City, China, illustrate this challenge. Their approach places responsibility for accidents on the driver of an autonomous vehicle, even when the car operates autonomously. If the vehicle operates independently without a driver, responsibility lies with the owner, who can only seek compensation from manufacturers if the accident was caused by a vehicle defect. This framework provides legal clarity but potentially creates a context where manufacturers hold a significant advantage and might inadvertently discourage autonomous vehicle adoption.

The challenge intensifies in healthcare settings. A recent study found that AI supported mammography screening led to a 29% increase in cancer detection while reducing radiologists' workload by 44.2%. However, when radiologists use AI for diagnostic support, they remain responsible for the final diagnosis. Yet the AI's decision-making process often remains opaque, creating tension between professional accountability and the 'black box' nature of many AI systems. Ensuring transparency becomes crucial not just for trust but for enabling professionals to validate their findings when needed.

I guess the good news is that society has long managed responsibility in complex systems, but AI presents new challenges due to its ability to learn and adapt using neural networks or genetic algorithms, making behaviour in specific circumstances often impossible to predict. This unpredictability creates what researchers term a 'responsibility gap' - situations where attributing clear responsibility for AI decisions and actions becomes increasingly difficult.
In this context, creating transparent AI systems is a priority, alongside developing frameworks that acknowledge the distributed nature of responsibility in these socio-technical systems. This includes balancing the roles of designers, operators, and oversight bodies while ensuring effective accountability.

Some researchers suggest that we should consider AI systems as human-made artefacts, and as such, ultimately their creators bear significant moral responsibility for their systems' behaviour, particularly for foreseeable outcomes and design choices.
This perspective suggests that managing complex AI systems isn't fundamentally different from managing other complex technical systems - it requires careful consideration of design, implementation, and oversight.
We often refer to the concept of 'meaningful human control' as one potential framework for managing these complexities. This approach emphasises the importance of ensuring that humans maintain appropriate oversight of autonomous systems, particularly in contexts involving significant moral or ethical decisions. However, implementing such control becomes challenging in situations where human intervention might actually reduce system effectiveness or safety.

This discussion renews the sense of urgency in addressing ethical and legal aspects in AI systems. The challenge continues to be creating governance structures that effectively manage distributed responsibility while fostering innovation. This might mean moving beyond simplistic attributions of blame to develop more nuanced approaches that reflect the complex reality of existing AI systems.
It also demands careful consideration of how we design these systems from the outset, incorporating transparency and accountability mechanisms that acknowledge the multiple stakeholders involved. Human-centred design holds significant responsibility in connecting AI deployment to people's needs, contributing to the greater good of society. By focusing on human needs and experiences, design practices can help bridge the gap between technological capabilities and societal requirements, potentially addressing some of the most challenging aspects of responsibility attribution in AI systems.

Franklin, M., Ashton, H., Awad, E., & Lagnado, D. (2022). Causal framework of artificial autonomous agent responsibility. Proceedings of the 5th AAAI/ACM Conference on AI, Ethics, and Society (AIES ’22), 276-284. ACM. https://doi.org/10.1145/3514094.3534140

Kolt, N. (2025). Governing AI agents. Notre Dame Law Review, 101(forthcoming). https://doi.org/10.48550/arXiv.2501.07913

Parts of this manuscript were drafted with the assistance of AI language models (specifically, Claude 3.7, ChatGPT 4.0, Google Gemini 2.0). The author used AI as a tool to enhance clarity and organisation of ideas, generate initial drafts of certain sections, and assist with language refinement. All AI-generated content was reviewed, edited, and verified by the author. The author takes full responsibility for the content, arguments, analyses, and conclusions presented. This disclosure is made in the interest of transparency regarding emerging research practices.