Stephen Canterbury

This is part of a blog post series focusing on the AI usage gaps identified within the MIT NANDA report, The GenAI Divide: State of AI in Business 2025.

After working directly with enterprises to deploy, monitor, optimize and scale a generative AI agent, I’m struck by how much of a partnership the work is—and how critical the right expertise becomes at specific steps in the process. 

So often, the first major decision point an enterprise reaches in their AI agent initiative is whether to buy a production-ready solution or build their own. But that simple either/or question glosses over the deeper and more complex challenges of getting from opportunity to pilot to production. The deciding factor for successful AI deployment is almost always a matter of well-directed expertise, not just in spinning up a passable AI agent, but in driving the project toward value realization.

Typically, the necessary expertise is a joint venture, a partnership between the enterprise and a vendor with deep experience in applying AI in the real world.

A partnership built on expertise

The recent MIT NANDA study, The GenAI Divide, found that partnering with a vendor doubles the likelihood of a successful outcome compared to trying to build a solution internally.

That doesn’t surprise me. Deploying an AI agent requires such a broad set of skills and knowledge that a single enterprise is very unlikely to have all of it in-house.

It’s true that no AI agent vendor will understand your customers or your business as well as you do. They can’t replace your knowledge of internal processes and workflows, or your insight into what’s working in your customer journeys and what’s causing unnecessary friction. And your IT team already knows the intricacies (and complexity) of your CX tech stack better than a vendor ever will. That makes your internal resources invaluable.

It’s not that a dedicated internal AI team can’t build a custom in-house solution. More so that an AI vendor likely has more experience launching an AI agent for customer service, monitoring its performance, and optimizing to drive value in a repeatable fashion. Their expertise goes well beyond the ability to build a reliable solution. And that makes partnering with the right vendor a smart decision.

Here are some of the crucial capabilities an experienced partner brings to a generative AI agent deployment.

1. Driving value with the right use cases

Everyone from your frontline agents to the C-suite probably has ideas about which use cases you should prioritize for automation with a generative AI agent. Some are high-volume, some eat up a lot of agent hours with long handle times, and some are too complex for traditional automation to deal with. A few are likely just pet peeves for people in your organization. 

But choosing the right use cases to target first should be a multi-step process that considers a range of key factors. And it must be data-driven. A partner with deep expertise can help you avoid the common pitfalls, like choosing use cases that are too low-volume, too complex for your initial launch, or lacking required APIs or knowledge content. Choosing the wrong use cases can lead to months of costly tinkering before you can even think about ROI.

The right partner will ensure that you’re applying generative AI to problems where it excels—and that you’re choosing use cases that will deliver larger and faster returns. 

2. Designing efficient and effective testing strategies

With a generative AI agent, testing is more than a simple quality assurance step. It’s a strategic function that verifies the AI agent’s ability to handle the complexity and variability of customer service interactions. It’s also very different from traditional software testing. 

The kind of test scripts you’ve probably used with deterministic bots and virtual agents won’t address the multivariable complexity and edge cases a generative AI agent is designed to handle. To thoroughly test your agent, you’ll need to simulate a range of realistic scenarios with variation in the conversation flows, customer personalities, and specific issues to be resolved.  

The right partner can design a testing methodology that mirrors real-world complexity, stress-testing the agent's knowledge base retrievals, API dependencies, reasoning, policy adherence, and conversational flow in a way that accounts for the variability of actual customer interactions.

A partner who’s experienced in this type of testing will help you zero in on the right scenarios and variations to test so you can launch with confidence—without spending weeks on end testing every scenario you can dream up. The ability to test the right scenarios at scale is crucial to reducing risk as you launch your AI agent. 

3. Pinpointing and diagnosing complex issues quickly

Diagnosing issues is inherently more complex with a generative AI agent than debugging a deterministic bot. Inaccurate responses and other failures often stem from a combination of elements, such as a weak knowledge article, a problem with a required API, unclear instruction, and an ambiguous customer question or comment. So, even if you can monitor or flag all conversations for annotation and review, diagnosing the underlying causes could be a challenge. You might be left knowing what happened, but not why.

Uncovering the root causes of an AI agent’s performance issues is a specialized skill that takes time and real-world experience to develop. A partner that has deployed and optimized generative AI agents in multiple enterprise contact centers will likely be more adept at diagnosing issues than your in-house team. And they’ll know the inner workings of their own solution, which can help them pinpoint issues and propose solutions more quickly. That allows for precise modifications rather than larger, potentially destabilizing rollbacks.

4. Sustaining post-launch performance and ROI

When it comes to boosting returns on your investment, addressing the obvious problems with an AI agent’s performance, accuracy, and reliability is just the beginning. If your AI agent is only doing a decent job of resolving customer issues, you’re leaving a lot of value on the table. 

Over time, you’ll find that many of the modifications that improve outcomes are small, targeted, and nuanced. For example, recognizing that key information is missing from your knowledge base is easy. But knowing how to improve your AI agent’s accuracy by restructuring your knowledge articles takes much more know-how. Similar changes, like modifying task instructions, improving memory management, adjusting API data pulls, and incorporating human judgment as a feedback loop, only drive value when they’re rooted in deep expertise.

Understanding what to tweak requires a kind of judgment that’s built on a lot of experience tuning an AI agent that’s live with customers. The right partner adds that experience to the equation.

5. Staying ahead of the innovation race

As quickly as AI technology is evolving, keeping up with every improvement, model, and new technique for applying it in the real world is a full-time project. A vendor with a deep bench of AI researchers, engineers and strategists focused on innovation and application in the enterprise can continue to enhance your AI agent’s capabilities and performance. That ensures your solution remains on the leading edge of what’s possible instead of falling behind.

Even as many enterprises are just now adopting generative AI agents, the AI for CX industry is already moving toward full customer experience platforms that orchestrate a network of AI agents. In the near future, these agents will constantly listen to and analyze interactions, engage with customers proactively, and gather new intelligence on customer journeys. 

These emerging capabilities will fundamentally change how your enterprise engages with customers. That’s a lot bigger than simply building a decent AI agent that automates some of your interactions. And it’s a lot to miss out on if you don’t partner with an AI solution provider that’s already building this future state. 

The bottom line is… the bottom line

The whole point of adding a generative AI agent to your contact center is to lower the cost to serve while improving your customer service delivery. On its own, that’s a tough balance to strike. It’s also just the baseline for what’s possible with an AI agent. Building and deploying your AI agent is only a small part of the process. The rest is where your biggest opportunities for value realization lie. And it requires deep expertise beyond the technical work of building the solution. That’s why partnerships beat internal builds.

Let’s be honest. A partner with deep AI expertise comes with a price. But the lack of that expertise is much more costly, especially long-term.

When talking to Contact Center leaders, we constantly hear the refrain, “How do I bring down handle times?” As long as contact centers have been around, the standard measure of agent efficiency has been how quickly they can handle a customer’s inquiry and move on to the next customer. In the phone world, the simplest way to improve the output of a phone agent is to shorten their call time. Thus handle time (or AHT) has long been our key metric.

With the onset of digital channels, and increasing adoption of these channels by both customers and companies, handle time is no longer an adequate measure of agent efficiency.

As a Success Manager at ASAPP, I work with our customers to drive value and optimize contact center operations for digital and voice programs. A key part of our effort is identifying the right metrics for analysis to measure program success. We are continuously evolving what we measure and the recommendations made based on changes in the way companies interact with customers (over multiple channels and sometimes asynchronously, for example.) And, on the specific goals of the individual company.

In order to best evaluate program performance, particularly agent efficiency, we need to look beyond AHT for three key reasons:

1. Digital automation is changing the average conversation that gets to a live agent. The simplest customer inquiries are able to be solved by automation, and even those issues that used to be too complex for automation can often be solved through integrations to company systems. For example, an internet service customer can check if they are in an outage, troubleshoot their modem, send diagnostic information, AND schedule a service appointment, without ever speaking to a live agent. This expansion of automation causes only the most complex issues to get to agents, driving up the average time it takes to handle those inquiries.
2. Digital agents may have a more expansive set of tools than ever before. It’s not just about being able to handle more than one chat at a time, digital agents can rely on automation and augmentation to maximize the number of customers they handle at once.
3. Voice and digital conversations just aren’t the same, and we need to identify a way to benchmark them at a programmatic level. The handle time of a voice call is all of the time the agent and the customer are spending on that one call, while the handle time for a digital conversation is all of the time the customer is on that conversation. Digital agents are likely to be handling multiple conversations concurrently. And, a digital conversation may stop and start over multiple sessions, as customers handle their inquiry asynchronously.

Customer Experience vs. Agent Efficiency

This isn’t to say that AHT is irrelevant, in fact it is still very relevant, but not in regards to agent performance. AHT is most relevant for the customer experience. Companies must still be concerned with how long the experience is for a customer to handle their issue. What I have started pushing for is the shift in perspective, AHT is a KPI for the customer experience, but when evaluating front line agent efficiency/output, we have better measures.

From AHT to Throughput

In order to find the best opportunities to maximize workforce efficiency, modern CX teams have shifted focus from AHT to Throughput. Throughput can be measured a number of different ways, but simply it is a measure of an agent’s output in a certain time period, normally an hour.

‍

Throughput measures an agent’s output in a way that works for both voice and digital engagement, including asynchronous interactions.

Stephen Canterbury

‍

In most cases, organizations are measuring this as resolutions/assignments per utilized/logged-in hour. This measure can easily be translated into a simplified cost per chat metric and overall, gives a holistic picture of how much can one front line team member do. Throughput also helps to avoid the biases of handle time, because it can be based off of total time that an agent is working, potentially highlighting absenteeism or other addressable gaps.

Take the below example of a way that we are seeing organizations start to shift their thinking:

Agent A: On the clock for 5 hours, handles 20 customer issues, Average Handle Time = 20 minutes.

Agent B: On the clock for 5 hours, handles 25 customer issues, Average Handle Time = 20 minutes

Assuming both agents are working the same shift, we would obviously want Agent B’s performance over Agent A’s. Agent B will handle an assignment more per hour than Agent A, while the customer experiences the same handle time. Combining this analysis with Occupancy and Customer Satisfaction enable an organization to get a complete picture of the body of work of an agent. Throughput becomes the measure to assess efficiency. Those agents that can handle more customers per hour, while staying within CSAT and business (occupancy) targets, are the top performers. We can use their behaviors as a template to better train and support the rest of the agent base.

Where do we go from here?

Technology advancements are continuing to push the contact center industry forward. Automation is becoming easier to build and integrate with company systems. Only the most complex customer issues will need live support. Artificial Intelligence is getting smarter and more helpful in supporting live agent conversations. The lines between live agent and automation blur as agents are supported by recommendations, automated flows, and more. Throughput will be a metric that can scale with the changing landscape, better than any other measure of agent efficiency.

Efficiency at 30,000ft—Organizational Throughput

Even still, some forward-thinking organizations are looking beyond agent throughput to a broader program efficiency view. The question we are starting to hear is “How efficient is my Customer Experience program?” Companies are leaning into throughput, but viewing it in a programmatic lens. When you combine both the automated and live aspects of your program, how effective is your investment? Program effectiveness is being measured when looking at all automated AND handled customers per staffed agent.

Organizational Throughput is helping to view the program as a whole, tracking improvements or degradations in automation and live support on the same playing field. As the worlds of automation and live support become more intertwined, it only makes sense for organizations to start looking at these two separate entities’ performance together.