There’s a strong likelihood that everyone you run into these days is ‘AI-enabled.’ But it often feels like ‘AI-enabled’ is code for ‘human-disabled.’ At a recent tradeshow, another fund administrator told me they’d automated their entire NAV calculation process via AI. When I started asking pointed questions like, “How does your team handle edge cases?” I realised they had no idea how the underlying logic worked. My guess is AI was brought in just to populate their marketing messages with the latest tech buzzword.

Unvalidated ‘AI-enabled’ claims introduce invisible risks — cognitive debt, oversight effects, compounding errors — which you may unknowingly absorb as a client. Here are five questions you can ask a vendor to determine if they have thoughtfully adopted AI or if they’re just jumping on the AI bandwagon. 

1.    Can you explain your validation process? 

It’s easy for a vendor to claim their models are highly accurate, but compared to what? 

A strong validation protocol runs AI systems parallel to existing human workflows before deployment. Add the AI, keep the humans, compare the outputs, and investigate the mismatch. Over multiple iterations, human teams essentially teach the model where to look for specific data, creating a continuous improvement loop versus a one-time correction. 

Providers also need a validation process specifically designed to catch hallucinations. A UCSD study found that LLMs hallucinate 60% of the time when summarising product reviews, but users still overwhelmingly chose the AI-generated summary. We are psychologically predisposed to trust AI due to its confidence and fluency. 

Your vendors should be able to walk you through their validation process, what specific mechanisms exist to catch hallucinations, and how their confidence intervals are designed to counteract human bias towards AI output.

2.    When AI fails, who’s accountable, and at what cost? 

Humans can explain their reasoning, own their mistakes, and course correct. AI cannot. 

In a fiduciary context, human accountability is critical. 

Each AI function should have a clear owner and escalation process. But it’s also important to ask how the humans are coping. Harvard Business Review studied the impact of AI usage on mental fatigue and burnout. It found that the more AI output demands review and supervision, the more stress it causes — leading to higher decision fatigue, poorer decision making, and increased employee turnover. 

Journalist Cory Doctorow introduced the idea of the “reverse centaur” in a recent Medium column. For people who use AI to boost their own productivity, AI can become a valuable tool — strong and tireless, directed by human judgment. Doctorow calls these users ‘centaurs.’ Reverse centaurs are the opposite: humans trying to monitor heads of endless AI output with fallible human attention. 

Service providers should build centaurs. You don’t want to work with a vendor who points to the model itself for accountability or describes an extensive oversight structure that creates burnout. AI should be used to take over repetitive tasks to support staff, not create a new category of exhausting work. 

3.    What’s the real ROI?

How much ROI is AI really delivering? MIT NANDA recently published a study that rattled Wall Street, showing that of the 80% of organisations that have implemented AI programmes, 95% have achieved zero return. Despite the billions invested into GenAI, a meaningful return remains elusive. 

Your vendors should be focused on AI initiatives that make a real difference for clients, like adding new features and functionality and significantly cutting service delivery timeframes. A lot of the hype around AI was that it would save money by reducing headcount. But we’ve seen the hidden costs there — poor user experience when people have to talk to bots versus a human account manager, and poor quality “AI slop” that feels low-thought and low-effort. As use cases are more defined and token costs rise, it’s becoming clear that AI’s real value is to complement human talent for better client service. 

4.    Where specifically are you not using AI, and how is data protected?

Where AI is not being used is just as important as where it is. AI should be focused on repetitive and routine tasks, not creative work or anything that requires judgement. MIT’s Media Lab ran brain scans on students using ChatGPT to write essays and found that participants had significantly poorer recall of what they produced. Critical thinking remains a key competitive edge and shouldn’t be outsourced. 

Most fund administration calculations are deterministic. The formulas for fee computations, allocations, and P&L calculations are historically reliable and easy to write, understand, and debug. If your vendor is automating these with AI, ask why. They are probably unnecessarily introducing risks into a process that previously had none. 

Data privacy deserves equal scrutiny. Can AI access your investor data, especially in third-party systems? A vendor should conduct rigorous data privacy and security audits and obtain enterprise-grade licenses when available. Make sure client data isn’t used as part of model training that can be accessed by other third parties. If your service provider is operating AI systems but cannot explain how the underlying architecture actually handles your data, you might be handing sensitive information to a black box.

5.    How do you measure success, and what’s your baseline? 

Take AI productivity claims with a pinch of salt. A recent METR study found that developers using AI actually built new features and resolved bug fixes at a slower pace with AI, but they thought they were working faster. 

Individual productivity doesn’t always translate to organisational productivity either. I may be able to write an email in two minutes instead of four with AI, but because it’s verbose, it takes twice as long to read.

If your vendor doesn’t share measurable data, instead gauging success by how users “feel” about the tools, you may be hearing an inaccurate story. Ask the vendor how AI saves time in end-to-end processes versus at the individual level. Outcome metrics should be tied to client-facing results and measured against pre-AI benchmarks.

Final thoughts

None of this is to say that AI isn’t useful. It’s an impressive technology that works well in specific applications: reading documents, extracting data, managing operations, and reducing repetitive tasks. The companies that get it right match the tool to the problem with clear acceptance criteria and measurable outcomes. 

Those that don’t quite get it are the ones chasing mandates and measuring success by perception, hoping no one asks the hard questions.

AI delivers output, but it’s people who give you accountability. Before you sign off on any vendor’s AI pitch, make sure they can tell you who stands behind their work and not just which AI model runs it.