It’s pretty dang hard to get around the internet these days without hearing about chatbots and ‘the future of AI’. It’s a hot topic and something we’re pretty excited about.

I’m pretty strict about what I subscribe to and here is my inbox filtering for AI (so not including emails using chatbot specifically).

We’ve been learning more about what customers need and expect from their online experience, and helping clients run a chatbot test to see how the channel converts. It can serve as a great user acquisition channel, depending on your target audience.

In this post, we’ll briefly cover what opportunities a chatbot can help you take advantage. But, the true intention is to give you a framework and template for running a chatbot test on your own to see if it works well for your customers as well as employees.

Tell me more about chatbots…

Chatbots are Artificial Intelligence (AI) programs that can process and respond to simple queries from your audience like an interactive FAQ. You’ve seen them across the internet, often in the bottom right corner of a web page. It looks similar to live chat but with programmed data on the other end.

When a company is utilizing a chatbot, they’ve taken the time to teach the chatbot the answers to questions they expect a customer to ask.

It works a bit like a flow chart:

Examples of companies running chatbots…

• Duolingo uses chatbots inside their language teaching app and allows users to practice conversation by simulating text exchanges. [Learn more about the bot here]
• Apartment Ocean is a chatbot built to help real estate agents qualify leads and learn more about potential customers. [Learn more about the bot here]
• Pizza Hut allows people to order pizzas and reorder their favorites via a Facebook messenger chatbot. [Test out their bot here]
• Casper, a mattress company, has a purely promotional bot that is active from 11pm to 5am to ‘keep you company when you just can’t fall asleep’. While it can chat on many topics from Stranger Things to Seinfeld, it also takes the opportunity to plug their mattresses from time to time. [Learn more about the bot here]

Why should I run a chatbot test?

Do you ask customers to fill out a form?
Chatbots have helped create a shift toward something being coined as ‘conversational marketing’. You can use chatbots to replace long forms with more intuitive and natural conversations. You can set up a bot to ask those same qualifying leads your forms are searching for. Depending on the potential customer’s answers, the chatbot will send them through the right flow.

Looking to help your sales agents save time and close more leads?
A chatbot can automatically qualify leads and get them to the right agents. As research from InsideSales.com and the Harvard Business Review shows, even if you wait just five minutes to respond after a lead first reaches out, there’s a 10x decrease in your odds of actually getting in touch with that lead. After 10 minutes, there’s a 400% decrease in your odds of qualifying that lead. By automating this crucial step, your chatbot can quickly disqualify leads and get the most promising ones quickly to your agents.

Want to improve your customer experience?
There’s never going to be a future where Artificial Intelligence totally takes over because humans and chatbots are good at different things. Leaning on our strengths and the strengths of chatbots can make for a power team. When a chatbot pilot program was initiated in a telco company, it could handle 82% of common queries in customer service. After 5 weeks of tweaking, analyzing, and optimizing by human agents, its success increased to 88%, according to Accenture.

Let’s check out the data

In the 2018 State of Chatbots study from Drift and friends, they surveyed over 1,000 internet users in the United States and made sure to match their audience to represent the U.S. adult online population. Here’s what the group had to say about their current online experiences:

In addition to helping people get quicker answers, you can use the chatbot data to make changes to your website and try to eliminate the root cause of the most frequently asked questions.

Do you have customers spanning multiple generations?: One preconceived notion I had about chatbots is that they were better suited for companies with younger target audiences. I was excited to find data saying quite the opposite. In that same 2018 State of Chatbots Study, Drift found that Baby Boomers (age 55+) were 24% more likely than Millennials (age 18-34) to expect benefits from chatbots in five of the nine following categories:

I’m in. How do I run a chatbot test?…

We feel really great about the future of chatbots and their ability to improve customer experience and to deliver higher quality leads. That being said, before jumping in with both feet we suggest running a chatbot test to validate or invalidate whether it works well for customers and team.

We’ve organized a super simple experiment to help sales teams run a chatbot test and see if it could work for them:

• Step 1: Develop a single hypothesis about what the chatbot will deliver
• Step 2: Explicitly Identify the metric that will help you validate or invalidate your hypothesis
• Step 3: Get benchmark data for that metric. You may be able to pull this from your current process or will need to build time into your experiment to capture it.
• Step 4: Test chatbot
• Step 5: Compare the two data sets to see if your hypothesis is valid or invalid.

Here is an example chatbot test:

• Step 1 – Hypothesis: Implementing a chatbot will decrease the amount of time it takes to make first contact with a lead.
• Step 2 – Metric: Hours from form filled to first contact.
• Step 3 – Benchmark data: Depending on the size of your sales team, pull the data from at least 10% of your sales agents. If you are already measuring this metric, awesome! Just pull it and proceed to Step 4. If not, spend 10 days (without the agents’ knowledge) measuring this.
• Step 4 – Test: There are a number of chatbot solutions out there. We recommend Intercom’s 14-day free trial because it’s a lean and easy way to get started and they have great analytics. Have the same agents you used for benchmark data spend 10 days using Intercom’s chatbot to qualify and make initial contact with leads.
• Step 5 – Compare: In a spreadsheet, take the data from your two sets of 10 days and compare your metric, ‘hours from form filled to first contact’. Which one has a lower average? Does one have a better conversion rate?

Continued reading about Chatbots:

• How Chatbots are Revolutionizing the Sales Process
• The Future of Self-Service
• The 2018 State of Chatbots Report
• How to Use Chatbots to Push Customers Down the Sales Funnels
• Announcing Operator: The Bot For Better Customer Experiences

Ready to run a chatbot test?

If you’re interested in testing out a chatbot for your sales, customer success, or customer support team, we hope this experiment helps. We’re also always available to talk through ideas and implement an experiment tailored to your team and goals. Check out a

with Tuff.

Mary Jantsch

Mary is a growth marketer at Tuff. She specializes in Customer Experience, Email Strategy, Copywriting and Chat. She also runs on coffee and peanut butter pretzels.