Skip to content

Evidence

What the research says, caveats included

We say AI-powered audits are faster, cheaper and more thorough than a traditional hand audit. Here is the evidence for each part of that claim, how strong it is, and where it runs out.

Read this first

Most studies below test an AI model doing a task itself, or helping a specialist do it, as in breast screening. We work differently. AI helps us build and test a rule engine, the engine runs the audit, and the two of us read every finding.

So the studies show two things. AI can read detailed rules and work through records quickly and carefully, which is why the engine can be built fast. And a model left to compute answers on its own drifts, varies from run to run and sometimes makes things up, which is why we never let a chatbot calculate holiday pay.

Comparison

A traditional hand audit and AI Auditors, side by side

Each row gives the traditional position, ours, what the evidence says and how strong it is. Sources marked “against” cut the other way and are listed so you can weigh both.

Speed

Elapsed time to an answer

Traditional hand audit
Weeks to years. Health NZ's Holidays Act remediation began in 2016-17 and is still paying former staff in 2027. A Christchurch City Council payroll review took 18 months. Remediation providers say the work can take over a year.
AI Auditors
The engine runs over the full export in minutes. The two of us review the findings within days. “Days, not months” is our own target, not a study result.
What the evidence says
In a randomised breast-screening trial of more than 100,000 women, AI that sorted every scan cut the radiologists' reading workload by 44%. Legal AI tools were 6 to 80 times faster than lawyers in an independent benchmark, and a peer-reviewed trial found AI made legal work faster and better. Once expert review and fixes are counted the gain is smaller: about 1.1 to 1.4 times in OpenAI's study of 44 occupations, and one randomised trial found experienced software developers 19% slower with AI. At task level, a 2026 professional-services task benchmark measured under 10 minutes for a frontier model, against 30 minutes to three hours for licensed professionals.
Strength
Strong at task level. Moderate at engagement level.

For: MASAI trial, 2025 (opens in a new tab)Vals Legal AI Report, 2025 (opens in a new tab)Schwarcz et al., 2026 (opens in a new tab)GDPval (OpenAI), 2025 (opens in a new tab)Health NZ (opens in a new tab)RNZ, June 2026 (opens in a new tab)Star News / ODT, 2020 (opens in a new tab)KPMG NZ (opens in a new tab)Mercor, 2026 (opens in a new tab)

Against: METR, 2025 (opens in a new tab)

Accuracy

Finding the errors that are there

Traditional hand audit
People miss things under volume and time pressure. In Stockholm breast screening, two radiologists reading every scan found 250 cancers; one radiologist working with AI found 261. Payroll errors go unnoticed for years: Health NZ's go back to 2010, and in 2026 the Financial Markets Authority found it had used one anniversary date for every employee.
AI Auditors
The same tested calculation is applied to every record. The engine is checked against payroll files with planted errors before it runs on yours. Accuracy depends on the rules being right, which is why two people read every finding.
What the evidence says
Where it has been tested hardest, AI plus a person finds more than people alone. In a randomised trial, AI-supported breast screening found 29% more cancers than two radiologists, with no rise in false positives. Across 463,094 women screened in Germany, radiologists using AI found 17.6% more. A customs model that scored every import found illegal cases with 92.7% precision while inspecting 1% of shipments. On long, open-ended or judgement work AI still drifts and invents things. No independent study tests our exact method, so we publish our own test results.
Strength
Strong in screening medicine. Mixed elsewhere. Thin for our exact method.

For: MASAI trial, 2025 (opens in a new tab)MASAI trial, 2026 (opens in a new tab)Eisemann et al., 2025 (opens in a new tab)Dembrower et al., 2023 (opens in a new tab)Liu et al., 2019 (opens in a new tab)Kim et al., 2020 (opens in a new tab)US Treasury, 2024 (opens in a new tab)Vals Legal AI Report, 2025 (opens in a new tab)Vals Legal Research, 2025 (opens in a new tab)LawGeex, 2018 (opens in a new tab)Health NZ (opens in a new tab)HCA Magazine NZ, 2026 (opens in a new tab)

Against: Freeman et al., 2021 (opens in a new tab)Stanford HAI, 2024 (opens in a new tab)ACS Information Age, 2025 (opens in a new tab)Dell'Acqua et al., 2023 (opens in a new tab)TIGTA, 2025 (opens in a new tab)

Cost

What the work costs the client

Traditional hand audit
Large. Health NZ has spent more than $191m on the fix alone, including $64m to its remediation partner. The former DHBs paid $25m to consultants over four years.
AI Auditors
A fixed fee, quoted before we start. The price is our decision, not a study result.
What the evidence says
Machine time is far cheaper: in a 2024 contract-review study a model cost about US$0.25 per contract, against about US$76 for a senior lawyer. What a client pays depends on review time: 1.2 to 1.6 times cheaper in OpenAI's measurement once an expert reviews and fixes the work. Fewer false alarms also save review time: up to 80% fewer in a central-bank money-laundering test on simulated data.
Strength
Moderate.

For: Martin et al., 2024 (opens in a new tab)GDPval (OpenAI), 2025 (opens in a new tab)BIS Project Aurora, 2023 (opens in a new tab)RNZ, June 2026 (opens in a new tab)Newsroom, 2022 (opens in a new tab)

Coverage

A sample, or every record

Traditional hand audit
Audits sample by design. ISA 530 defines sampling as testing fewer than 100% of items, and names the risk that the sample gives a different answer from the whole.
AI Auditors
Every employee, every pay line and every leave transaction in the export period.
What the evidence says
Sampling is standard in audit, and regulators now see firms testing full populations as AI use grows. Other fields already check everything and let people judge what is flagged. In the Swedish screening trial, AI scored every scan before a radiologist read it. The UK's National Fraud Initiative matches whole public-sector datasets, payroll included, and found or prevented £443 million of fraud and error in two years. US checks of pandemic loan applications against Social Security records flagged US$5.4 billion of potential identity fraud. No study yet shows that checking every record finds more errors than sampling. It is logical, but unmeasured.
Strength
Strong on practice. Thin on measured outcomes.

For: ISA 530 (opens in a new tab)PCAOB SEIAG, 2025 (opens in a new tab)MASAI trial, 2025 (opens in a new tab)UK National Fraud Initiative, 2022 (opens in a new tab)PRAC, 2023 (opens in a new tab)Labour Inspectorate, 2016 (opens in a new tab)

Consistency

Same data, same answer

Traditional hand audit
Results vary from person to person. Lawyers reviewing the same NDAs scored anywhere from 67% to 94% (a vendor study).
AI Auditors
Deterministic code: the same data gives the same result every time, and the rules are versioned.
What the evidence says
Repeatability is a property of deterministic code, not a study finding. The evidence against letting an AI agent do the work itself is strong: in a customer-service benchmark, the best agent solved fewer than a quarter of retail tasks in all eight of eight attempts.
Strength
Thin as direct evidence.

For: LawGeex, 2018 (opens in a new tab)Yao et al., 2024 (opens in a new tab)

Explainability

Can you see how the answer was reached

Traditional hand audit
Working papers and reviewer notes. Quality varies by firm and by person.
AI Auditors
Each finding shows the calculation steps, the inputs and the section of the Act relied on.
What the evidence says
This is a design feature, not a study result. There is strong evidence that free-text AI explanations cannot be trusted on their own: legal AI tools hallucinated in 17% to 34% of queries, and unsupervised AI reports have failed publicly. That is why we show the calculation, not AI prose.
Strength
None for our claim. Strong against AI prose.

For: Stanford HAI, 2024 (opens in a new tab)ACS Information Age, 2025 (opens in a new tab)Dell'Acqua et al., 2023 (opens in a new tab)

Human oversight

Who is accountable for the result

Traditional hand audit
Partner and manager review. People do every step.
AI Auditors
Two senior people scope the work, read every finding and sign the report.
What the evidence says
Regulators expect a “human-in-the-lead” model. Review is needed: AI users were 19 percentage points less often correct on tasks outside AI's competence. Review can also fail: when an AI suggested the wrong answer, even very experienced radiologists rated only 45.5% of mammograms correctly, against 82.3% when it was right. So every finding shows its working, and our review checks the working, the scope and the context rather than trusting a verdict.
Strength
Moderate.

For: PCAOB SEIAG, 2025 (opens in a new tab)Dell'Acqua et al., 2023 (opens in a new tab)Dratsch et al., 2023 (opens in a new tab)GDPval (OpenAI), 2025 (opens in a new tab)

Honesty

How we measure ourselves

What we claim

  • Faster and cheaper on well-defined, rule-based work, with the size of the gain stated honestly: large on machine time, much smaller once expert review is counted.
  • More thorough, because every line is checked and re-checked by multiple AI agents.
  • Repeatable and explainable, because the engine is code and every finding shows its working. These are design choices, not study results.

What we do not claim

  • That AI is more accurate than people at everything. On long, open-ended and judgement-heavy work the research says otherwise.
  • Any “a hundred times faster” figure. Numbers like that come from machine time alone.
  • That we find every error or guarantee compliance. The engine is only as good as the data and the rules, which is why two people read every finding.
  • That most New Zealand employers are non-compliant. No current statistic says so.

A result we take seriously

In a 2023 radiology experiment, when an AI system suggested the wrong answer, even very experienced radiologists rated only 45.5% of those mammograms correctly, against 82.3% when its suggestion was right. People lean on a confident machine. We treat that as a design rule: every finding shows its calculation and the section of the Act, so our review checks the working rather than trusting a verdict, and the engine is tested against planted errors before it runs on your data.

Source: Dratsch et al., 2023 (opens in a new tab)

Our own results

Our test results

No independent study tests an AI-built payroll rule engine against human payroll auditors, so the evidence for our exact method has to be our own. We test the payroll engine against ten worked employee records with planted Holidays Act and Employment Leave Act readiness errors. It reproduces every planted amount to the cent and raises no finding on the two clean records (EMP-01 and EMP-06). The test code and fixtures are in the repository. We will publish results on real client data as engagements allow.

Sources

Every source on this site

Read on 8 October 2026. “Opened” means we read the figure at the link. “Reported” means we read it in a news or law-firm report of the original. Vendor and industry interests are noted.

AI-assisted review in medicine, law, government and other fields

  • MASAI trial, 2025

    Hernström, V. et al., Screening performance and characteristics of breast cancer detected in the Mammography Screening with Artificial Intelligence trial (MASAI), Lancet Digital Health 7(3):e175-e183, 2025. A randomised trial of 105,934 women in Sweden.

    OpenedPeer reviewedPubMed abstract (opens in a new tab)Journal article (DOI) (opens in a new tab)

  • MASAI trial, 2026

    Gommers, J. et al., Interval cancer, sensitivity, and specificity comparing AI-supported mammography screening with standard double reading without AI in the MASAI study, Lancet 407(10527):505-514, 2026.

    OpenedPeer reviewedPubMed abstract (opens in a new tab)Journal article (DOI) (opens in a new tab)

  • Eisemann et al., 2025

    Eisemann, N. et al., Nationwide real-world implementation of AI for cancer detection in population-based mammography screening (PRAIM), Nature Medicine 31(3):917-924, 2025. 463,094 women in Germany; observational.

    OpenedPeer reviewedPubMed abstract (opens in a new tab)Journal article (DOI) (opens in a new tab)

    Three authors work for Vara, the AI vendor.

  • Dembrower et al., 2023

    Dembrower, K. et al., Artificial intelligence for breast cancer detection in screening mammography in Sweden: a prospective, population-based, paired-reader, non-inferiority study (ScreenTrustCAD), Lancet Digital Health 5(10):e703-e711, 2023. 55,581 women.

    OpenedPeer reviewedPubMed abstract (opens in a new tab)Journal article (DOI) (opens in a new tab)

    Part-funded by Lunit, the AI vendor.

  • Liu et al., 2019

    Liu, X. et al., A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis, Lancet Digital Health 1(6):e271-e297, 2019.

    OpenedPeer reviewedPubMed abstract (opens in a new tab)Journal article (DOI) (opens in a new tab)

  • Schwarcz et al., 2026

    Schwarcz, D.B. et al., AI-Powered Lawyering: AI Reasoning Models, Retrieval Augmented Generation, and the Future of Legal Practice, Journal of Law and Empirical Analysis 3:220, 2026. A randomised trial with 127 law students.

    OpenedPeer reviewedUniversity of Minnesota (abstract) (opens in a new tab)Working paper (PDF) (opens in a new tab)

  • Vals Legal AI Report, 2025

    Vals AI, Vals Legal AI Report, 27 February 2025. Some vendors tested were Vals customers (disclosed in the report).

    OpenedVals AI (opens in a new tab)

  • Vals Legal Research, 2025

    Vals AI, VLAIR: Legal Research, October 2025.

    ReportedVals AI (opens in a new tab)LawSites (opens in a new tab)

  • Martin et al., 2024

    Martin, L. et al., Better Call GPT, Comparing Large Language Models Against Lawyers, arXiv:2401.16212, January 2024. Ten procurement contracts.

    OpenedarXiv (opens in a new tab)

    Vendor-affiliated: the authors work for Onit, which sells legal technology. A preprint.

  • LawGeex, 2018

    LawGeex, AI vs Lawyers NDA study, February 2018, as reported by Artificial Lawyer, 26 February 2018.

    ReportedArtificial Lawyer (opens in a new tab)

    Vendor study: LawGeex sells contract-review AI.

  • US Treasury, 2024

    US Department of the Treasury, Treasury Announces Enhanced Fraud Detection Processes, Including Machine Learning AI, Prevented and Recovered Over $4 Billion in Fiscal Year 2024, 17 October 2024. Self-reported.

    OpenedUS Treasury (opens in a new tab)

  • Kim et al., 2020

    Kim, S. et al., DATE: Dual Attentive Tree-aware Embedding for Customs Fraud Detection, Proceedings of KDD 2020, pp. 2880-2890. Five years of import data, with the World Customs Organization.

    OpenedPeer reviewedAuthors' summary (GitHub) (opens in a new tab)Conference paper (DOI) (opens in a new tab)World Customs Organization (opens in a new tab)

  • BIS Project Aurora, 2023

    BIS Innovation Hub Nordic Centre, Project Aurora: the power of data, technology and collaboration to combat money laundering across institutions and borders, 31 May 2023. Simulated data.

    OpenedBIS (opens in a new tab)Report (opens in a new tab)

    Built with Lucinity, an anti-money-laundering software vendor.

  • PRAC, 2023

    Pandemic Response Accountability Committee, Fraud Alert PRAC-2023-02, 30 January 2023: $5.4 billion in potentially fraudulent pandemic loans linked to over 69,000 questionable Social Security numbers. Data matching, not AI.

    OpenedOversight.gov (opens in a new tab)

  • UK National Fraud Initiative, 2022

    Cabinet Office, National Fraud Initiative Report, December 2022. Data matching, not AI.

    OpenedGOV.UK (PDF) (opens in a new tab)

  • GDPval (OpenAI), 2025

    Patwardhan, T. et al. (OpenAI), GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks, arXiv:2510.04374, 5 October 2025. 44 occupations.

    OpenedarXiv (opens in a new tab)

    Vendor research: OpenAI makes AI models.

  • Dell'Acqua et al., 2023

    Dell'Acqua, F. et al., Navigating the Jagged Technological Frontier, HBS Working Paper 24-013, September 2023, as reported by MIT Sloan, 19 October 2023.

    ReportedMIT Sloan (opens in a new tab)

  • Freeman et al., 2021

    Freeman, K. et al., Use of artificial intelligence for image analysis in breast cancer screening programmes: systematic review of test accuracy, BMJ 374:n1872, 2021.

    OpenedPeer reviewedPubMed abstract (opens in a new tab)Journal article (DOI) (opens in a new tab)

  • Dratsch et al., 2023

    Dratsch, T. et al., Automation Bias in Mammography: The Impact of Artificial Intelligence BI-RADS Suggestions on Reader Performance, Radiology 307(4):e222176, 2023. 27 radiologists.

    OpenedPeer reviewedPubMed abstract (opens in a new tab)Journal article (DOI) (opens in a new tab)

  • Stanford HAI, 2024

    Magesh, V. et al., Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, Journal of Empirical Legal Studies 22:216-242, 2025. Summary by Stanford HAI, 23 May 2024.

    OpenedPeer reviewedStanford HAI (opens in a new tab)

  • ACS Information Age, 2025

    ACS Information Age, 7 October 2025: a government assurance review drafted with generative AI was found to contain fabricated references, and part of the fee was repaid.

    OpenedACS Information Age (opens in a new tab)

  • METR, 2025

    METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, 10 July 2025 (arXiv:2507.09089), with an update of 24 February 2026. A randomised trial with 16 experienced developers.

    OpenedMETR (opens in a new tab)METR update, 2026 (opens in a new tab)

  • Yao et al., 2024

    Yao, S. et al., τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, arXiv:2406.12045, June 2024.

    OpenedarXiv (opens in a new tab)

    AI-industry authors: Sierra builds customer-service AI agents.

  • TIGTA, 2025

    Treasury Inspector General for Tax Administration, report 2025-308-022 on the IRS use of AI to select tax returns for examination, 19 May 2025.

    OpenedTIGTA (PDF) (opens in a new tab)

  • Mercor, 2026

    Barton, A. (Mercor Economics), Human Baselines for Benchmarks, October 2026. A task benchmark: twelve licensed US professionals on four realistic month-end tasks.

    OpenedMercor blog (opens in a new tab)Paper (PDF) (opens in a new tab)

    AI-industry source: Mercor sells AI evaluation and training data. Not peer reviewed.

Audit sampling: the standard and the regulator

New Zealand payroll compliance

AI use and AI risk

Microsoft 365 security

  • Microsoft, 2025

    Microsoft On the Issues, Microsoft Digital Defense Report 2025, 16 October 2025.

    OpenedMicrosoft (opens in a new tab)

    Vendor report: Microsoft sells the platform and its security tools.

  • Practical 365, 2024

    Practical 365, Entra ID multifactor authentication reaches 38%, 19 February 2024, quoting Microsoft.

    ReportedPractical 365 (opens in a new tab)

  • FBI IC3, 2026

    FBI Internet Crime Complaint Center, 2025 IC3 Annual Report, April 2026. US reports only.

    OpenedIC3 (PDF) (opens in a new tab)

  • NCSC NZ, 2026

    NCSC NZ, Quarter One sees significant cyber incidents, 22 June 2026.

    OpenedNCSC NZ (opens in a new tab)

  • NCSC NZ SME tracker, 2026

    NCSC NZ SME Cyber Security Behaviour Tracker 2026, as reported by Beinsure, 24 September 2026.

    ReportedBeinsure (opens in a new tab)

  • Microsoft Security, 2026

    Microsoft Security, Inside an AI-enabled device code phishing campaign, 6 April 2026.

    OpenedMicrosoft Security (opens in a new tab)

    Vendor report: Microsoft sells the platform and its security tools.

  • The Hacker News, 2026

    The Hacker News, Device Code Phishing Hits 340+ Microsoft 365 Orgs Across Five Countries, 25 March 2026.

    OpenedThe Hacker News (opens in a new tab)

Talk to us about an audit

A short call to scope it. A fixed fee before we start. Big Four experience, without the junior team or the Big Four fee, in days not months.