Human - AI collaboration, cognitive shortcuts and mental models formation

Until a couple of years ago or so, AI was deployed mostly behind the scenes. In military and defence AI is embedded in surveillance systems, autonomous drones and cyber warfare tools. In healthcare it detects anomalies in scans, delivers personalised treatment plans and manages lots of the admin work. Now that the technology is embedded in many consumer facing tools that we can all use. However, our understanding of how people and AI models collaborate is still evolving.

When we interact with complex systems, we develop mental models that help us make sense of them. These mental constructs enable to predict what is coming next interacting with our surroundings. Mental models are mostly based on our experiences, they can be flexible and inaccurate, simply providing enough insight so that we can use an object, orient ourselves in space, interact with a technology, talk to a person.
Forming an accurate mental model of an AI system (or anything else that is not super familiar to us) is a process that takes time, particularly as the technology continues to evolve. Research suggests that even small updates to an AI model can disrupt existing mental maps, weakening trust and impacting decision-making effectiveness. 

Some suggest that in the absence of well-formed mental models, we often resort to heuristics and biases as navigation aids. These mental shortcuts are natural components of human cognition. We use them to manage uncertainty, minimise cognitive effort and understand the complexities of the world. Three common heuristics emerge when interacting with AI systems (those can be applied to any tech that we don’t know very well):

The anchoring heuristic refers to the cognitive tendency to rely too heavily on the first piece of information encountered (the anchor) when making decisions or judgments. We might anchor our trust in an AI system based on our experience with similar technologies, regardless of their actual similarities or differences. Perhaps you have used voice assistants such as Alexa or Siri and now you are perfectly at ease using the AI voice assistant that comes with your new car, expecting the same experience even if the underlying tech is different.

The availability heuristic refers to our tendency to judge the likelihood or frequency of events based on how easily examples come to mind. A single dramatic failure experienced recently  might disproportionately influence our willingness to trust similar systems in the future. Maybe you experienced hallucinations last time you used your LLM of choice and you decided that you won’t use those tools again.

The representativeness heuristic occurs when we decide if something fits into a category based on how much it looks like our mental picture of that category, instead of looking at actual statistics or facts. Sometimes we assess AI outputs based on how closely they match our expectations rather than through objective evaluation of the evidence. For example, maybe you are using an AI tool to screen potential candidates for a role that you are hiring for, and you end up selecting recommendation of those candidates that attended your old uni, because their profile instantly feels right.

These mental shortcuts can lead to two problematic reliance patterns: automation bias and algorithm aversion. 

Automation bias occurs when decision-makers fully trust a technology, potentially disregarding contradictory evidence or important contextual information. This partially results from anchoring on pre-existing information regardless of subsequent data. Conversely, algorithm aversion develops when we become biased against a technology that previously provided inaccurate results. This leads to disregarding algorithmic suggestions regardless of their potential accuracy, even when evidence shows that algorithmic mistakes may be less severe than human ones.

Both patterns can disrupt effective AI–human collaboration and complicate trust calibration, intended as our ability to fluidly align our trust with actual technological capabilities. Trust evolves with time, knowledge and experience, constantly adjusting to context. People can distinguish different performance levels in AI systems after multiple interactions. 
For example, a recent study found that trust and usage increased at accuracy levels above 70% and decreased below this threshold. It seems that we can form flexible mental models through interaction and develop appropriate reliance based on the operational context.
Developing accurate mental models helps balance collaboration between people and complex systems, encouraging reliance strategies based on genuine understanding of the technology. For instance, better comprehension of AI error boundaries enables more calibrated trust and reliance. 
Ideally, mixed teams of humans and AI should aim for complementary partnerships where individual skills work together for superior outcomes that neither could achieve independently. Increasing transparency in AI design can help users form reliable mental models, rather than forcing them to resort to subjective heuristics.

The challenge remains significant. Many inteligent systems lack sufficient transparency for users to form accurate mental maps. This becomes particularly evident during covariate shifts, where AI models encounter input data different from their training sets, resulting in prediction accuracy drops that users struggle to account for in their reliance strategies.

AI systems are becoming increasingly common in high-stakes contexts. While their computational power has the potential to transform healthcare services, adoption still presents complex challenges, including ethical concerns, the development of appropriate reliance on diagnostic tools, and the management of biases. These unresolved issues are layered on top of already complex systems with low adaptability, making consistent technology adoption difficult.
As AI becomes more integrated into decision-making processes, understanding how we form and apply mental models is important. This understanding could inform the design of systems that better support appropriate reliance and collaborative decision-making, particularly in high-stakes environments.