Buying AI has become fairly easy for law firms. Most vendors can have a pilot running within weeks, and their demonstrations are persuasive, whether that means a long agreement summarized in seconds or a polished first draft produced from a few bullet points.
What remains much harder to establish is what the firm is actually buying, and how that product will behave once it is working on real matters under real deadlines.
In legal work, output that looks right is a low bar. Firms handle privileged information and specialized terminology, and they produce documents where a single altered word can shift an obligation. An answer that is 95 per cent correct might be perfectly acceptable in marketing copy, but in a share purchase agreement the remaining 5 per cent is where the liability sits.
I began my career as a court interpreter, and I learned early how much can turn on one word. That lesson has stayed with me through more than two decades of building language and AI services for law firms, banks and securities regulators, and it shapes how I think firms should evaluate AI vendors.
Before any tool touches client work, I would want answers to six questions.
1. What happens to our data?
Every vendor will say client information is secure. Firms should ask for the specifics behind that claim: where data is processed and stored, whether any of it is retained, whether it can be used to train a model, who can access it and what happens to it when the contract ends.
It is also worth asking about the full chain of providers involved in producing a result. Many enterprise AI products are built on foundation models from other companies, combined with the vendor’s own technology and data. That is a sensible way to build, and often the most effective one.
The firm still needs a clear view of the architecture, including every third party that handles its information along the way. No firm would pass confidential client files to an unfamiliar subcontractor because a supplier vouched for them, and an AI product deserves the same scrutiny.
2. How do you know when the AI is wrong?
Generative AI almost always produces an answer, and incorrect answers tend to look as polished as correct ones.
In legal documents, the errors that matter are rarely dramatic. A dropped qualifier, a defined term used inconsistently or “shall” becoming “may” can change the meaning of a clause while escaping the notice of anyone who is not reading as a specialist.
Firms should ask the vendor how it measures accuracy for the specific work its system performs, what standard it tests against, which kinds of errors occur most often and where the technology performs poorly. A credible vendor will discuss the limits of its product as readily as its strengths.
One question I find especially revealing is when the vendor would advise a firm not to use its tool at all. A thoughtful answer to that tells a buyer more than a second demonstration will.
3. What happens when we correct it?
This question gets far less attention than it deserves. If an associate made the same mistake repeatedly and never absorbed a partner’s corrections, the firm would treat it as a performance problem. Firms should hold the AI systems they adopt to a similar expectation.
For specialized professional work, an AI system becomes more valuable as it reflects what the firm knows, including its terminology, precedents, client preferences and internal standards. Corrections made by trusted lawyers are among the most valuable of those inputs.
Firms should ask what happens after someone fixes an output, whether that correction becomes part of an approved body of knowledge that improves later results, and who decides which feedback the system learns from. The answers will show how much of the product’s value depends on the underlying model and how much comes from what the firm contributes over time.
4. Where are the humans?
Discussions about AI in law often treat people and technology as alternatives, as though each task must go to one or the other. In practice, the more useful question is which parts of a workflow can be automated, which require professional judgment and how the two should hand work to each other.
The answer varies with the work. Some tasks are low-risk and repeatable enough to automate almost entirely. Others suit an arrangement where AI produces the first pass and a qualified professional reviews it. Some work carries enough risk or complexity that it should stay primarily in human hands, and a good vendor will say so.
Many vendors describe their products as having a “human in the loop”. Firms should find out who those people are, what qualifications they hold, at what point they become involved and whether they have the authority to override the system. The aim is the best result at an acceptable level of risk, and the share of work handled by AI matters only to the extent it serves that aim.
5. Who will help us put it to work?
A new tool delivers value only once lawyers and staff are using it well, and getting there usually takes more effort than firms expect. Workflows have to be adjusted and users trained in ways that reflect how each practice group actually works, and people need support during the first months while new habits take hold. That internal change management can place real demands on a firm’s time and people.
Firms should ask whether the vendor will help plan the rollout, train users and support practice groups as they bring the tool into daily work. It is worth being specific about who on the vendor’s side is responsible for that support, how long it continues and which parts of the work the firm is expected to handle itself. When those expectations are unclear, each side tends to assume the other is leading adoption, and a capable tool ends up underused.
The vendors I would trust with client work treat implementation as part of what they deliver, alongside the technology itself. They measure their success by how well the firm is using the tool months after launch.
6. What happens when the technology changes?
The AI product a firm buys this year will be different in two years, and possibly in six months. Vendors swap underlying models, new capabilities arrive, pricing shifts and regulation develops. The firm’s own use of the tools will mature as well. For these reasons, AI adoption does not end at installation and training the way many software rollouts do.
Firms should ask how the vendor evaluates a model change before it reaches their workflows, how performance is retested after a switch and whether the knowledge and corrections the firm has built up survive the transition. They should then ask who, on both sides of the relationship, is responsible for managing all of this on an ongoing basis. That question points to a larger issue many firms are only beginning to address, which is that governing AI over time demands considerably more of a firm than the initial purchase.
Looking past the demonstration
AI has real potential to improve how legal work gets done, and firms that adopt it carefully are already seeing the benefit. Remember that a slick demonstration shows what a tool can do under favourable conditions.
The questions above test how a tool will perform on a firm’s actual matters, for its actual clients and over years of use. The more consequential the work, the more those answers matter. Vendors worth trusting will welcome scrutiny of their tools and processes, and a profession that manages risk for a living is well placed to apply it.
Gary Kalaci is the founder and CEO of Apertera (formerly Alexa Translations), which provides AI-enabled language solutions to legal, financial and regulated organizations. A lawyer by training, he began his career as a translator and court interpreter. Since founding the company in 2002, he has built it into a provider serving most of Canada’s major national law firms, all of its major banks and leading securities regulators.
