How successful is your chatbot? Like all contact channels, you need to track certain metrics to measure the performance of your bot. These chatbot analytics will show you what your bot is doing well and the areas where it can be improved. 

In this article we look at the top seven metrics you should be tracking in your chatbot analytics. How do we know these are the most important? We’re speaking from experience! These are the metrics we measure for our own successful chatbot. We also recommend all our customers track these too. 

As well as performance metrics, this list also includes two cost-related metrics that are vital to consider when developing and deploying your bot.

Performance analytics

Let’s start with performance metrics. To measure the success of your chatbot, you need to track five key performance related metrics. 

Success means different things for different companies. To get the most out of your chatbot analytics and determine if your bot is ‘successful’, make sure you set clear goals and KPIs to compare your monthly data against.

1. Deflection

This metric refers to the percentage of customer queries that are handled by your chatbot without any human intervention. It’s a clear indication of how well your bot is doing its job. 

In most customer service scenarios, the role of a chatbot is to reduce the demand on live agents. The bot handles FAQs from start to finish, only handing over to an agent if it can’t deal with the customer request. 

If your chatbot analytics show a low level of deflection, it means your bot is not adequately handling a good portion of customer queries. This could mean it’s actually creating more work for agents, rather than helping them.

Firstly, as chatbots act as an additional contact channel for customers, they tend to increase the number of interactions. This is great for customer engagement, but if your chatbot isn’t handling a lot of these additional interactions, it’s just more queries for your agents to deal with.

Secondly, if your bot can’t answer simple questions, customers will grow frustrated. When they are transferred to a live agent, the agent has to deal with their frustration as well as their initial query. This makes it a lot harder to ensure the customer enjoys a  good customer experience. 

Of course, your chatbot analytics won’t show a 100% deflection rate, and this should never be the aim. Bots are not designed to replace human-driven customer service – they’re just another tool to help improve the overall customer experience. 

A good level of deflection will depend on your company and the type of customer queries you receive. Studies have shown that aiming to deflect anywhere between 40% to 80% of queries is a realistic goal. 

In general, you should train your chatbot to handle common customer requests by building pre-defined conversation paths. When set up correctly, your chatbot will be able to handle these specific queries from start to finish, automating your FAQ customer service. If a customer has a complicated or unusual issue, your chatbot should be programmed to hand over to a live agent. 

2. Unrecognised customer queries

Another important metric to keep an eye on in your chatbot analytics is the percentage of customer inputs that fall outside your chatbot’s prescribed workflow. In layman terms: what percentage of customer questions does your chatbot not understand? 

‍

This metric is only relevant if you use a ‘free input’ chatbot, i.e. you let customers freely type in their questions rather than clicking predefined options with a rule-based bot.

When you build a chatbot, the aim is to design complete, human-like conversations. Let’s look at an example. One of your most common customer queries is: ‘I can’t log into my account’, so you build an appropriate conversation path for your bot.

As long as the customer follows one of your pre-built conversation paths, the chatbot can handle the customer request. 

If a customer asks a question you have not trained your chatbot to answer, it won’t understand the query and won’t be able to answer it. In this case, it will need to transfer the customer to a live agent. 

You will never be able to predict every single question a customer can ask you, but the more questions you prepare your chatbot for, the better. Try to map out all likely customer conversations and cover as many eventualities as possible. 

Combined with the deflection metric we discussed above, these two chatbot analytics will help you understand how in-sync your bot is with your customers. The aim should be to get it efficiently answering all of your FAQs. 

3. Average handling time (AHT)

A good chatbot should help your agents’ workflows and provide efficient customer service. This means it should reduce your average interaction handling time. 

One of the driving factors for companies adopting chatbots is the speed in which they can answer customer questions. Today, a timely response is one of the most valued aspects of customer service. Chatbots answer this demand. 

When it comes to FAQs, bots will always be faster than agents. They can handle multiple requests at once, meaning no chat queues and, without the need to type or ‘think’, they can provide immediate responses. 

Even your most efficient agent will be slower than a chatbot in an FAQ scenario. This is why your average handling time should be reduced after introducing a chatbot.

A reduction in AHT is another chatbot analytic that shows your bot is doing its job well. It’s taking care of common queries quickly and efficiently, while also freeing up agents to deal with more complex issues. 

Average handling time is a metric that any good chat and chatbot analytics system will track automatically. You can track bot handling times and live chat handling times separately, but to truly know if your bot is making a difference, we recommend tracking an overall chat enquiries AHT too.   

4. Customer satisfaction (CSAT)

Like all contact channels, customer satisfaction is a key metric to keep track of in your chatbot analytics. After all, every business wants satisfied and loyal customers.

The easiest way to measure customer satisfaction with your chatbot is to get feedback. This can be as simple as asking customers to click a ‘thumbs up’ or ‘thumbs down’ button after an interaction, or to rate the experience out of 10. 

For more detailed feedback you can give customers a short post-conversation questionnaire. By asking a few simple questions, you can find out what customers like and dislike about their experience with the bot, giving you a clearer picture of where you can improve. 

That said, it’s all well and good measuring overall customer satisfaction with your chatbot, but it’s more useful if you have other data to compare it with. As a result, we recommend tracking two customer satisfaction related metrics in your chatbot analytics.

First, you should track chatbot customer satisfaction vs agent customer satisfaction. If CSAT scores are a lot lower for your bot compared to live agents, your bot needs improving. If it’s a lot higher, encourage more engagement with your bot (and maybe consider retraining your agents!).

Secondly, you should also track your customer satisfaction levels from before and after introducing your chatbot. Chatbots are supposed to improve overall customer satisfaction – if your satisfaction rates are lower after introducing your bot, it is not serving its purpose. 

5. Lead generation

Is your chatbot generating leads? To know this you need to include lead generation in your chatbot analytics. 

For sales and ecommerce chatbots, this is one of the most important metrics to track as these bots should be capturing high quality leads for your business. If they’re not, it probably means your bot is not adequately engaging customers. 

If your bot is principally for customer support rather than sales, lead generation might be less important for you. However, customer data collection (such as names, contact details, and personal preferences) is always useful for targeted marketing campaigns, meaning lead generation can still be worth keeping an eye on. 

What constitutes a lead can be defined by your business. Is it simply collecting a customer’s name and contact details? Or do you want a bit more information to qualify the lead, such as age range, product interest, company and role (for B2B)? 

When you have defined what makes a lead, you can set up tagging in your chatbot analytics. With this in place, when a chatbot interaction fulfils the parameters of a lead, it will be tagged as an inbound lead. 

To really know if your chatbot is succeeding at generating more leads, compare the number of inbound leads generated month by month before and after you introduced the bot. Ideally, you want to see a significant increase in leads after the chatbot has been put in place. 

Cost analytics

When researching the right bot for you, there are important cost-related chatbot analytics you should take into account. Price is always a factor when introducing new tech and features to your company, but time-related costs are equally important. 

6. Development and configuration cost

Unless you have the expertise in-house to build your own chatbot, you will need to look to a third-party to both develop and set up a bot on your website. In this case, pricing always needs to be carefully considered. 

Price will vary largely based on the complexity of the chatbot. For example, a rule-based bot that caters for a handful of FAQs will cost a lot less than a free text input AI chatbot. 

The easiest way to ensure you are getting the best deal for you is to compare like for like. Know what you want your bot to do and request pricing quotes from multiple companies. Remember to get the cost for:

1. Developing the bot
2. Configuring bot on website e.g. customising the UX and chat widget (if required)
3. Integrating bot into existing workflows e.g. CRMs and chat systems

It’s also good to know how much support you’ll get with the price. Are you on your own after configuration or will you get continued expert support? 

7. Time to go live

Another important factor to consider when choosing a chatbot company is the time it will take to develop your bot and get it up and running. 

Much like price, this will be influenced by the complexity of your chatbot, but ideally you want a quick turn around.  

A standard FAQ chatbot shouldn’t take long to develop and configure. With Talkative, for example, we can build the bot and get it live within a few days. 

Consider how long you’re prepared to wait for your new chatbot and ask companies for an accurate timescale of development through to deployment. 

The quicker your bot is up and running, the sooner it can start helping agents, improving the customer experience, and generating leads. 

Chatbot analytics round up

To sum up, the most important chatbot analytics to track are:

1. Deflection
2. Unrecognised customer queries
3. Average handling time
4. Customer satisfaction
5. Lead generation
6. Deployment and configuration cost
7. Time to go live (development to deployment timescale) 

Do you use any other metrics to measure the success of your chatbot?

This blog post was first published by Talkative.