Skip to content
PHAROS
← Journal
04 / Practical inquiry

Who is allowed to say the model is wrong?

Human oversight means little if the person closest to the mistake cannot slow, challenge or reverse the decision.

A hand uses an inspection lens to examine a fracture in a ceramic test piece.
PHAROS JOURNAL / 04Room for a challenge · Film still

A human is in the loop

Consider a hypothetical case. A manager receives an AI-generated risk score. The employee closest to the work says the score relies on an old record, misses a recent change and treats an exception as a pattern.1

The manager is still technically in control. A person will click the button. The organisation can therefore say that a human was in the loop.

But can the employee stop the decision? Can the manager reject the score without becoming responsible for ignoring the system? Will the correction change anything beyond this one case?

Those questions reveal the difference between human presence and human authority.

Oversight is a job, not a location

The NIST AI Risk Management Framework says human roles in AI decision-making and oversight need to be clearly defined and differentiated. Its playbook goes further: organisations can document oversight, overrides, reported errors, complaints, escalations and accountable go or no-go decisions.23

For covered high-risk systems, Article 14 requires providers to design systems so assigned people can effectively oversee their use, including understanding limitations, guarding against over-reliance, interpreting output, disregarding or reversing it, and interrupting the system where applicable. Article 26 separately requires deployers to assign oversight to people with the necessary competence, training and authority, and with appropriate support.4

That list matters because interface access is not enough. A reviewer may see the recommendation and still lack time, information, status or permission to challenge it.

The law does not apply to every AI use. The operating lesson travels further than the legal rule: if a person is expected to catch a mistake, the organisation has to make catching it possible.

The social cost of being the exception

Long before the current AI wave, Amy Edmondson studied 51 work teams and found that psychological safety was associated with learning behaviour. Asking for help, discussing errors and seeking feedback all carry interpersonal risk.5

AI can add another layer to that risk. The recommendation arrives with the appearance of calculation. Challenging it may feel less like questioning a draft and more like questioning the organisation's investment, the data team, a senior sponsor and the promise of efficiency at once.

An experiment with public-sector decision-makers found patterns of automation bias and selective adherence to algorithmic advice. The result does not tell us how every workplace behaves, but it unsettles the comforting idea that putting a person beside a recommendation automatically produces independent judgment.6

A challenge route that depends on an unusually confident employee is not a control. It is a hope.

Five questions for a real challenge route

Pharos's synthesis of the evidence is a five-question test for whether oversight can survive a real disagreement:7

  • Who can challenge the output, including the person affected by it?
  • What evidence can they see, and what information can they add?
  • Who has authority to pause, reverse or escalate the decision?
  • What happens to the challenger while the disagreement is reviewed?
  • How does a valid correction change the workflow, guidance or model use afterwards?

What the disagreement can teach

Not every objection should win. People can be mistaken, self-interested or attached to a familiar process. A challenge route is not a promise that human judgment outranks evidence.

Its purpose is to keep the evidence open long enough for a consequential disagreement to be examined by someone with the authority and competence to decide. The decision may uphold the model, reject it or reveal that the system and the human were answering different questions.

The valuable record is not simply that a person overrode an answer. It is why the answer failed, what evidence changed the decision and whether the boundary should move for the next case.

If nobody can challenge the model without paying a social or procedural price, the organisation does not have human oversight. It has a human witness.

Sources

Source record updated 5 November 2026. External findings stay within their published scope. First-party Pharos observations and interpretations are identified.

  1. 1

    Pharos. Hypothetical AI-supported risk decision. Constructed situation, 2026.

    The opening is illustrative. It is not a customer case, employment claim or report of a specific system.
  2. 2

    National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST, 2023.

    Voluntary framework. It states that human roles and responsibilities in decision-making and oversight should be clearly defined and differentiated.
  3. 3

    National Institute of Standards and Technology. AI RMF Playbook. NIST AI Resource Center, Living resource.

    Suggested actions include documenting oversight, overrides, errors, complaints, escalations and accountable go or no-go decisions. The playbook is not binding law.
  4. 4

    European Union. Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence. Official Journal of the European Union, 2024.

    Articles 14 and 26 concern human oversight and deployer duties for covered high-risk systems. Applicability depends on the system, actor, context and date; current legal advice is required.
  5. 5

    Amy C. Edmondson. Psychological Safety and Learning Behavior in Work Teams. Administrative Science Quarterly 44(2), 1999.

    Multimethod field study of 51 work teams in one manufacturing company. Psychological safety was associated with learning behaviour; this is not an AI study.
  6. 6

    Saar Alon-Barkat and Madalina Busuioc. Human-AI Interactions in Public Sector Decision Making: Automation Bias and Selective Adherence to Algorithmic Advice. Journal of Public Administration Research and Theory 33(1), 2023.

    Three experiments in the Netherlands, including a study of 605 participants. Public-sector experimental findings do not establish a universal workplace effect.
  7. 7

    Pharos. Five-question challenge-route test. Proposed operating method, 2026.

    Pharos synthesis informed by the cited oversight, organisational-learning and human-AI decision research. It has not been validated as a standard or predictor of outcomes.