Chatbot Evaluation: Putting Banking Chatbots to the Test (part 1)

Master Of Code Global
Chatbots Life
Published in
9 min readFeb 8, 2023

--

At Master of Code, as a Conversational AI company, we developed a Chatbot Analysis Framework for businesses to review and evaluate their Conversational AI solutions in terms of the user experience and the solution’s maturity and complexity. It reviews user-facing elements that are critical to the overall customer experience. With this framework, analyzers were able to identify the solution’s strengths, any gaps in the user experience, and opportunities for new features.

To start, we picked an exciting sector to dive into: Conversational AI in Finance. These banking chatbots are high volume and serve the needs of customers seeking finance-related support and services, which makes way for the opportunity to develop or enhance contextual and transactional experiences. Given how personalized the banking web experiences are today, we can now translate them into a more conversational one.

Evaluating Canadian Banking Chatbots Based on Top Success Metrics

We reviewed 6 banks using our Chatbot Analysis Framework and scored their conversational AI solution. Each line item in the table was rated with chatbot evaluation metrics during analysis by giving:

  • A full point if the chatbot met the feature’s best practice standards.
  • A partial point if the solution displayed only part of the described feature.
  • A zero if none of the features were currently present in the solution at all.

These scores were then added up out of a total score of 20 and reviewed against the chatbot evaluation metrics table below.

Want to evaluate your Conversational AI solution? Download our Chatbot Analysis Framework!

Let’s take a deeper look at the results of each of the 6 banks.

TD Bank Chatbot

First, we looked at TD Bank’s chatbot, Clari, which exists on the TD app only and can answer basic FAQs and account questions from customers.

TD’s Banking Chatbot Strengths:

  • Provides bot prescription
  • Utilizes structured content capabilities
  • Meets accessibility standards

TD Bank’s Clari does well to identify itself as a Conversational AI assistant and has the option to educate users on what it can do as a bot, which helps set expectations for the end-users up front.

Clari utilizes the app channel capabilities, such as structured content, and buttons, and has app-specific UI blocks where users can tap the call-to-action button to bring up a pop-up window with more information and a link to a web landing page. Another best practice Clari demonstrates is that its UI meets the standards of the Website Content Accessibility Guidelines (WCAG).

TD’s Banking Chatbot Opportunities

  • Offer both unauthenticated and authenticated experiences
  • Implement transactional use cases
  • Provide more personalized responses
  • Live agent handover
  • Shortening long-tailed actions on the web into faster conversational flows
Banking chatbot evaluation metric: Contextual conversational experience

Clari provides simple answers to common financial support questions but is connected to the customer’s account, as you have to be logged into the app to talk to the chatbot. There’s an opportunity here to include transactional use cases and make the conversational experience more contextual by pulling that data into the banking chatbot. Clari also has a few intents that leverage customer data to add personalization (i.e., recent transaction questions), but why stop there? As an example of a conversational ai for customer experience, adding customers’ names to the dialogue would not only make the experience more personal but strengthen trust as well.

Boost your Customer Experience based on these 2023 Insights. Download Conversational AI in Customer Experience eBook now!

Additionally, having the option of live agent handover would greatly benefit the user experience. Currently, Clari only offers customers the option to call if they need live support assistance. Introducing the ability to transfer to a customer support agent within the bot will boost customer CSAT, and reduce the friction of having to switch channels.

Banking chatbot evaluation metric: live agent handoff availability through chat

One of the main use cases Clari has is viewing and breaking down recent transactions. While this is something banking customers like to do often, this can be easily accessed on the dashboard screen when they open the app. Therefore, it isn’t clear what pain point this use case is solving as it’s just as, if not more, efficient on the web & app user flows. Reviewing customer feedback and utterances would be helpful in figuring out how to improve this experience even further.

Read Also: Use Cases for Chatbots in Banking

Banking chatbot evaluation metric: what pain point this banking use case is solving

Banking Chatbot Results within Framework: Minimal use of automation.

RBC Bank Chatbot

Next, we reviewed RBC Bank’s chatbot, Ask NOMI. It’s only available on the RBC mobile app and can help support frequently asked questions, including inquiries about a recent transaction, canceling an e-transfer to an individual, or helping customers navigate the banking app.

RBC’s Banking Chatbot Strengths

Similar to TD Bank’s chatbot Clari, Ask NOMI helps set customers' expectations by introducing itself and providing menu options once the chat is initiated. We found that Ask NOMI also has a trustworthy and personable tone of voice consistent throughout the bot, which is beneficial for customer engagement and retention.

Ask NOMI does well to leverage app capabilities such as quick replies, structured content, and rich links and images (i.e., to show recent transactions, etc.). It responds to text and voice queries and handles free-form natural language understanding (NLU) in 12+ languages. It provides concise content that is easily digestible, reducing any heavy cognitive load for customers, and meets the WCAG standards.

Chatathon by Chatbot Conference

Also Read: Attachment Detection Strategies for Virtual Assistants: photo, video, audio messages

RBC’s Banking Chatbot Opportunities

  • Pull existing tools into the chatbot
  • Turn wayfinding use cases into transactional ones
  • Live agent handoff within call center automation option
  • Include global commands for users

While Ask NOMI utilizes customer data to provide insights such as recent transactions, there is an opportunity for it to turn its wayfinding use cases into transactional ones allowing users to complete more tasks in a shorter amount of time. Ask NOMI is part of a larger feature, simply called NOMI, that includes other tools called NOMI Insights, NOMI Find & Save, and NOMI Budgets. These tools could be pulled into the bot where customers can access everything in one place while still engaging in a conversational experience.

Banking chatbot evaluation metric: clear wayfinding option

Ask NOMI seems to have no live agent handoff via chat. Instead, it provides a link to a page for escalation via the Advice Centre, which is only accessible by phone. Like TD Bank’s Clari, we recommend implementing live agent escalation for banking chatbots because it will reduce friction for customers who need additional help and open another channel for support. Customers who are not able to talk or use the phone due to a disability or other reasons are currently blocked from this experience.

Banking chatbot evaluation metric: live agent handoff availability through chat

Lastly, it was observed that Ask NOMI’s NLU wasn’t able to comprehend phrases like “main menu” or “go back to the main menu.” For customers who want to see what else the bot can help with or simply exit a flow, this global command is very important in chatbots according to best practice standards.

Banking Chatbot Results within Framework: Good use of automation.

Free Download: The Ultimate Guide to Conversational AI in Finance

Scotiabank Chatbot

The Scotiabank Chatbot is an FAQ-type virtual assistant available within their mobile app experience that answers questions about everyday banking and connects you with a customer care advisor if you want to chat with a person. It also leverages the app for transactional capabilities (i.e. directing the customer to go to a section within the app to make a bill payment).

Scotiabank’s Banking Chatbot Strengths

  • An excellent welcome message with personalization & prescription
  • Leverages app-structured content capabilities
  • Content is easily digestible
  • Includes live agent handoff

The conversational experience starts with a short introduction, where the bot introduces itself as a ‘chatbot’, presents multiple options on what the bot can handle, and refers to the customer by name in the introduction. This is an example of a good welcome message for a banking chatbot. It’s short and concise, inviting the customer by providing personalization and setting expectations. Additionally, the bot continues to keep a friendly tone throughout the experience.

This banking chatbot does well to utilize rich links, structured carousel menus, and buttons to help guide the customer on what they can do next. The content in the bot is easily digestible as well, keeping the cognitive load to a minimum.

The Scotiabank Chatbot appears to have live agent handoff, but while reviewing, only one instance was found where it offered to “chat with a person”. Additionally, it informs the customers of other live support options after hours.

Scotiabank’s Banking Chatbot Opportunities

  • Utilize APIs to have contextual & transactional experiences inside the bot. This option available with the help of conversational ai integration
  • Review and update training data & content.
  • Improve UI to meet accessibility standards

Apart from the personalization in the welcome message, we could not find other areas in the bot that give indications of APIs or other integrations. As mentioned above, the chatbot leverages the app for transactional capabilities; however it removes the customer from the experience and relocates them to a page in the app. There’s a big opportunity here to enhance the flows by using customer data to create more contextual and transactional conversations (within the bot) as the customer is already authenticated.

Scotiabank FAQ Chatbot

Furthermore, we identified that the chatbot seemed not to recognize common financial terms like “mortgage payment” and “interest rates.” It would be worth reviewing such keywords to understand the utterance volume to see if new intents or use cases need to be implemented. If they’re low volume, updating the dialog to help set expectations that such content isn’t available in the bot or offering alternative options would be a recommended experience.

https://masterofcode.com/blog/putting-banking-chatbots-to-the-test
Banking chatbot evaluation metric: financial term recognition

In regards to accessibility standards, there’s an opportunity to increase the font size and improve the contrast of the buttons to make them more legible for those with visual impairments.

Banking Chatbot Results within Framework: Good use of automation.

Banking Chatbots Evaluation Based on Top Success Metrics Conclusion

After putting these banking chatbots to the test using our Chatbot Analysis Framework, we found they shared some strengths, such as meeting accessibility standards, leveraging the appropriate channel capabilities, and having a consistent bot persona or tone of voice. However, they had differing results for opportunities which included leveraging authentication, adding more transactional use cases, implementing live agent handoff, and providing better prescriptions. If you’d also like to make chatbot analytics, run these chatbot evaluation metrics, and learn how you can optimize your virtual assistant to demonstrate strong use of automation, download our Bot Analysis Framework or get in touch with us.

Check out part 2, where we’ll review another three banking virtual assistants using our Chatbot Analysis Framework with the most important evaluation metrics, share more insight about the top Canadian banking chatbots, and discuss valuable use cases for financial services bots.

Explore the ways how to improve your customer engagement within Conversational AI! CONTACT US

Get Certified in ChatGPT + Conversational UX + Dialogflow

--

--

A service company with a product mindset developing custom digital experiences for web, mobile, as well as AI-based conversational chat and voice solutions.