Table of Contents
Get past the vague language
Every vendor who sells a generative AI agent for contact centers makes the same big claims about what you can achieve with their product—smarter automation, increased productivity, and satisfied customers. That language makes all the solutions sound pretty much the same, which makes a fair comparison more difficult than it ought to be.
If you want to get past the vague language, take control of the conversation by asking these key questions. The answers will help you spot the differences between solutions and vendors so you can make the right choice for your business.
1. What exactly does your AI agent do?
The most capable AI agents can handle customer interactions from start to finish across voice and digital. They can listen to the customer, understand their intent, and take action to resolve the issue.
Some AI agents simply automate specific processes or serve up information and other guidance to human agents, while others can operate independently to talk to customers, assess their needs and take action to resolve their issues. Ask these questions to distinguish between them.
- Can your AI agent handle customer interactions from start to finish on its own? Or does it simply automate certain processes?
- How do your agents use generative AI?
- What channels does your AI agent support?
Look for an AI agent solution that can support the channels your customers use and take action to resolve issues, not just provide information. The ability to understand requests, access relevant information, and complete tasks is often what separates an AI agent from a conversational interface.
2. How do you ensure the AI works in production, at scale?
Not every AI agent that performs well in a demo will perform well in production. Production-ready AI agents should have a proven approach to testing, monitoring, and improving performance in real customer environments.
Enterprise deployments must handle changing business rules, complex customer interactions, and large volumes of requests. Ask these questions to understand whether a vendor can successfully support AI beyond the pilot stage.
- What evidence can you provide that the solution operates successfully at enterprise scale?
- What business outcomes have customers achieved?
- How do you support customers as they move from pilot to production?
- What resources and expertise are available after deployment?
- How long does a typical deployment take?
Look for a vendor that can demonstrate a repeatable, reliable approach to deploying AI at scale, with evidence that their solution performs successfully in real-world environments and that adequate support and expertise is available to help your team adopt it beyond the pilot phase.
3. How will your solution protect our data (and our customers’ data)?
Enterprise AI solutions should protect customer data through strong access controls, security boundaries, authentication, and data privacy safeguards.
Security is always a top concern, and generative AI adds some new risks into the mix, such as prompt injection, which could allow a bad actor to manipulate the AI into leaking sensitive data, granting access to restricted systems, or saying something it shouldn’t. Any AI vendor worth considering should have strong, clear answers to these security questions.
- How do you ensure that the AI agent cannot be exploited by a bad actor to gain unauthorized access to data or systems?
- How do you ensure that the AI agent cannot retrieve data it is not authorized to use?
- How does your solution maintain data privacy during customer interactions?
- Will customer data be used to train AI models?
Look for a solution that can detect when someone is trying to exploit the system by asking it to do something it should not. It should also have strong security boundaries that limit the AI agent’s access to data (yours and your customers’). Security and authentication in the API layer are especially critical for protecting data. And all personal identifiable information (PII) should be redacted before data is stored.
4. How do you keep your AI agent from ticking off my customers or damaging my brand?
AI agents should provide accurate information, stay within approved boundaries, and behave predictably during customer interactions.
We’ve all heard stories of bots that spouted offensive language, agreed to sell pricey products for a pittance, or encouraged people to do unsafe things. Solution providers worth considering should have robust safety mechanisms built in to ensure that the AI agent stays on task, produces accurate information, and operates within defined business and compliance requirements. Get the details on how a vendor approaches AI safety with these questions.
- How do you mitigate and manage hallucinations?
- How do you prevent the AI agent from sharing misinformation with our customers?
- How do you prevent jailbreaking?
- How do you ensure the AI agent stays within approved business rules and policies?
Look for a solution that grounds the AI agent in information specific to your business, such as your knowledge base, and includes automated QA mechanisms that evaluate output to catch harmful or inaccurate responses before they are communicated to your customer. The solution should also incorporate a variety of guardrails to protect against people who want to exploit the AI agent (jailbreaking). These measures should include prompt filtering, content filtering, models to detect harmful language, and mechanisms to keep the AI agent within scope.
5. How does your solution keep a human in the loop?
The best AI agents don't eliminate human involvement—they incorporate human expertise where it adds the most value. Solution providers acknowledge the importance of keeping a human in the loop. But that doesn’t mean they all agree on what that human should be doing or how the solution should accommodate and enable human involvement.
These questions will help you assess how thoroughly the vendor has planned for a human in the loop, and how well their solution will support a cooperative relationship between the AI and your team.
- What role(s) do the humans in the loop play? Are they involved primarily during deployment and training, or are they also involved during customer interactions?
- When and how does your AI agent involve a human expert?
- Can the AI agent ask the human agent for the input it needs to resolve the customer’s issue without handing over the interaction to the human?
- What kind of concurrency can we expect with a human in the loop?
Look for a solution that allows AI agents to seek guidance from human experts, receive input and approvals, and continue resolving the customer's issue without unnecessarily handing off the interaction. Human involvement should be a deliberate design choice, not just a fallback when the AI gets stuck.
6. How will we know what the AI is doing—and why?
AI agents should provide visibility into the information, tools, systems, and actions used during each customer interaction. This visibility helps teams understand performance, investigate issues, and improve how the AI agent operates over time.
When a human agent performs exceptionally well—or makes a mistake—you can ask them to explain their reasoning. That’s often the first step in improving performance and ensuring they’re aligned with your business goals. It’s equally important to understand how an AI agent is making decisions. Use these questions to learn how a solution offers insight into the AI’s reasoning and decision-making.
- How will we know what specific tools and data the AI agent is using for each customer interaction?
- In what ways do you surface information about how the AI agent is reasoning and making decisions?
- How can our team review past interactions, actions, and outcomes?
Look for a vendor who provides a high degree of transparency and explainability in their solution. The AI agent should generate an audit trail that lists all systems, data, and other information sources it has accessed with each interaction. This record should also make it easy to understand what the AI agent did, why it took specific actions, and where teams may need to investigate or improve performance.
7. How do you evaluate, test, and improve AI performance?
AI agents should be regularly evaluated and improved to ensure they continue meeting business and customer expectations.
Like human agents, AI agents need ongoing review. Without a clear process for evaluation and improvement, performance can drift over time as customer behavior, business policies, and knowledge sources change. Use these questions to understand how a vendor approaches testing, measurement, and continuous improvement.
- How do you test and validate AI performance before deployment?
- How do you monitor AI performance after launch?
- How do you measure AI performance and identify opportunities for improvement?
- What tools do you provide to help teams improve AI performance over time?
- How do you evaluate changes before they are deployed to customers?
Look for a solution that provides a structured approach to testing, monitoring, and improving AI performance. Teams should be able to evaluate interactions, identify areas for improvement, validate changes before deployment, and continuously optimize how the AI agent performs over time.
8. How much effort is required to use and maintain the solution?
The best AI agent solutions allow business teams to manage updates and improvements without relying on developers for every change.
Conditions in a contact center can change quickly. Product updates, new service policies, modified workflows, revised knowledge base content, and even shifts in customer behavior can require your agents to adapt—including your AI agents. Ask these questions to find out how well a solution empowers your team to handle simple tasks on their own, without waiting on technical resources.
- What kinds of changes and updates can our contact center team make to the solution without pulling in developers or other technical resources?
- What will it take to train our supervisors and other CX team members to work with this solution?
- How are updates tested and approved before they are deployed?
Look for a vendor who has invested in user experience research to ensure that their solution’s interfaces and workflows are easy to use. The solution should have an intuitive console that empowers non-technical business users with no-code tools to manage changes and updates on their own.
9. Why should we trust your team?
Trust starts with expertise. What you really need to know is whether a vendor has the expertise to deliver a reliable solution now and continue improving it as AI technologies evolve. These questions will help you determine which solution providers are best equipped to keep up with the pace of innovation and support your business over the long term
- What components of your solution were developed in-house vs. acquired from third parties?
- How do you help customers evaluate and validate the solution before full deployment?
- What independent validation can you share from customers, analysts, or industry experts?
- Can you point me to your team’s research publications and patents?
Look for a vendor with a strong track record of in-house development and AI innovation. Experience building and operating AI systems at scale is a strong indicator of the vendor’s future success. Vendors should also be able to demonstrate a practical approach, such as a fast proof of value, helping customers validate performance and build confidence before making a large-scale commitment.
10. What business outcomes should we expect?
AI agents should improve both operational efficiency and customer outcomes. Understanding how a vendor measures success can help you determine whether the solution aligns with your business goals and priorities.
Every organization has different objectives, whether that's reducing costs, increasing resolution rates, improving customer satisfaction, or helping agents work more effectively. Ask these questions to understand how a vendor defines success and measures results.
- What metrics do your customers typically use to measure success?
- What business outcomes have customers achieved with your solution?
- How long does it typically take customers to see results?
- How do you measure the impact of the AI agent on customer experience?
- How do you measure the impact of the AI agent on operational performance?
Look for a vendor that can clearly explain how success is measured and what outcomes customers typically achieve. The strongest solutions should be able to demonstrate improvements in both customer experience and operational efficiency.
This list of questions is not exhaustive. There’s a lot more you could—and should—ask. But it’s a good start for rooting out the details you’ll need to make a fair comparison of generative AI agents.
Want to ask ASAPP some questions about its Customer Experience Platform (CXP) and GenerativeAgent? Get in touch. We’ve got answers.
Frequently asked questions
What is a customer service AI agent?
A customer service AI agent is software that can interact directly with customers, understand their intent, reason through requests, retrieve information, and take action to help resolve issues. Unlike traditional chatbots, the most capable AI agents can handle entire interactions across voice and digital channels while following business rules and policies.
How is an AI agent different from a chatbot?
Traditional chatbots typically follow predefined rules and decision trees to answer questions or guide customers through simple tasks. AI agents use generative AI to understand customer requests, adapt to different situations, reason through problems, and complete more complex interactions. The most advanced AI agents can also take action in business systems to help resolve customer issues.
Can AI agents take action in enterprise systems?
Some AI agents can do more than provide information. They can securely access enterprise systems to complete tasks, update records, initiate workflows, and help resolve customer issues. When evaluating a solution, it's important to understand what actions the AI can perform, what systems it can access, and when human approval is required.
When should an AI agent involve a human?
AI agents should involve human experts when additional judgment, approval, or specialized expertise is needed. The best solutions allow AI agents to seek guidance, obtain approvals, and continue working toward resolution without unnecessarily handing off the interaction. Human involvement should be an intentional part of the workflow, not simply a fallback when the AI encounters a challenge.
How do you evaluate an enterprise AI agent platform?
Evaluating an enterprise AI agent platform requires looking beyond demos and conversational quality. Organizations should assess the platform's ability to resolve customer issues, protect sensitive data, operate safely, involve humans when appropriate, provide visibility into its actions, support continuous improvement, and deliver measurable business outcomes. Just as importantly, buyers should evaluate the vendor's expertise, innovation, and ability to support the solution over time.
