Chatbots and human live chat agents both share a (broad) common purpose: to support website visitors and provide quick, convenient customer service online. They both interact with customers in the same channel. They both operate within the same contact centre team.
So, it is perhaps not too surprising that chatbots and their human counterparts often end up measured against the same KPIs.
But that is a mistake. What if we looked at chatbot KPIs through a different lens?
The mistake of identical human and chatbot KPIs
For all their similarities, using the same KPIs for chatbots and human teams is a bad practice.
Chatbots, to put it simply, aren’t humans. To list just a few considerations:
- • They don’t work the same way. Chatbots recognise and respond to keywords; humans specialise in context.
- • They (usually) don’t answer the same issues. Typically, chatbots field FAQ while humans deal with more complex queries.
- • Targets differ between bots and human agents. For example, a chatbot will be expected to push out a response automatically. A human, meanwhile, may need a few minutes to compose a message.
- • They have different capabilities and permissions. Agents may have access to more features – such as co-browsing, or omnichannel calling, for instance. Such factors impact the type, tone, and timing of interactions.
- • The details of their goals may differ. Though they both interact with customers, chatbots may slot in specifically to help a company offer out of hours service, or to take the place of web forms.
These factors (and more) mean that chatbot KPIs and human agent KPIs don’t necessarily align.
There are, of course, overlaps. No matter the tool or the channel, you want to measure metrics such as customer satisfaction and first time resolution rate, for instance. But beyond these overarching benchmarks, chatbots and humans need to be viewed and evaluated through different lenses.
Bad chatbot KPIs
Contact centres often apply common live chat KPIs to chatbots as a means to gauge their success. Unfortunately, many of these common KPIs aren’t that helpful where chatbots are concerned. Such examples include:
• Sentiment scores
Sentiment scores are great for monitoring the performance of human agents. Human agents not only know how to recognise sentiment, but how to respond appropriately to get the right scores.
Chatbots, on the other hand, don’t typically deal with issues that customers are emotional about. Additionally, some chatbots require or prompt customers to give one-word/set answers — leaving no room for accurate sentiment scoring.
• Chat transfers
With human agents, the number of chat sessions transferred can point to an issue. That might be an issue with your chat routing rules, or an agent that needs extra training, for example.
With chatbots, transfers aren’t so cut and dry. For some chatbots, transfers are the goal. Here, the chatbot is in place to assist with chat routing in a conversational format. So, a transfer is an integral part of the interaction.
Other times, the chatbot may be asked a question more suited to a human agent. Transferring doesn’t represent a bot failure, but rather a bot doing its job by escalating appropriately.
Other times still, customers may simply not wish to talk to the bot. This leads to a transfer that has nothing to do with bot performance, and everything to do with customer preference.
Even if you were to try to use this metric to see where the bot has failed, transfer rate doesn’t show abandoned conversations. That is, the conversations where customers have given up entirely.
So, chat transfers are one of the more ineffective chatbot KPIs.
• NPS – net promoter score
The NPS metric asks customers to rate how likely they are to recommend your services or products to others.
With human agents, this gets asked at the end of a (hopefully) successful support interaction. With chatbots, many customers will never make it to the NPS score question. That’s because they often opt to leave the chat early, once they’ve received the quickfire answer to their FAQ. So, NPS wouldn’t include the opinions of customers that have bounced — whether they did so happily or in frustration.
As you can see, there are everyday KPIs that work great for human agents, but do little to shine a light on chatbot performance.
Good chatbot KPIs
So, what are the chatbot KPIs that you should be using?
• Target audience session volume
Is your chatbot attracting the audience you’re targeting with it? Measuring this can tell you if you’re achieving your marketing goals.
• Fallback rate
Fallback refers to the times that your chatbot doesn’t understand a message it has received. And so, it falls back to its previous question. A high fallback rate will tell you that customers are asking for something your chatbot hasn’t been equipped to handle. The higher the fallback rate, the more likely your chatbot needs some more tuning to be as helpful as possible.
• Volume of users/interactions
Chat volume tells you the adoption rate and popularity of your chatbot. A bot with a high volume is popular — and therefore it is likely proving useful to customers. A low volume, meanwhile, can sometimes suggest that the bot isn’t as helpful as it could be.
• FAQs
What are the most common questions your chatbot gets asked? Monitoring this can point to shortfalls with your product or your website. If there’s a common issue being asked about, you know to fix it.
Additionally, if the chatbot is regularly getting a question that leads to bounce or transfer, it suggests it’s a question you need to teach your chatbot to answer.
• Bounce rate
The bounce rate for a chatbot is the same as it is for, say, a website page. It measures how often customers quit the chat session when talking to your bot.
A low bounce rate tells you that the chatbot is performing well and being helpful.
A high bounce rate, meanwhile, suggests customers are getting frustrated with the bot, or it isn’t proving useful to them.
• Goal completion rate
Chatbots can have different criteria for success than humans.
FAQ bots succeed by successfully answering a question. Routing bots, meanwhile, succeed by transferring a customer to the right department. Guide bots succeed by finding the right web page or product that a customer is looking for, and so on.
As such, the goal completion rate measures how often a chatbot is successful in achieving these goals. This is a bit like the resolution rate — how often a human closes a ticket with a happy customer behind them. It tells you that your bot is performing as intended.
Chatbot KPIs
One of the biggest pitfalls when it comes to chatbots is forgetting that they aren’t human. They can’t think on their feet. They don’t truly understand the emotion behind customer messages.
Why, then, would we use the same KPIs for chatbots that we do for humans?
There’s a need for organisations to adapt as the age of automation and AI-powered tools — such as chatbots — continues unabated. So, which chatbot KPIs do you use?
Useful links
Reducing contact centre expenditure with a chatbot for FAQ
Customer understanding: getting on the same (web) page as your customers