Convert Assist, a Gen-AI product, launched in summer of 2024 to a lot of fanfare. Engagement tanked 95% within the first 60 days. Research-led repositioning and pragmatic prioritization turned it around.
Context
What is CallRail?
To put it simply, the platform helps businesses track where their leads are coming from. And this helps them understand what the ROI on their marketing spend is.
Why AI?
Convert Assist came about from a third party research study done by the Marketing team. It identified a gap for the SMB segment — guidance and coaching that helped convert leads at the price point CallRail offered. With this guidance, the team designed Convert Assist: a combination of three R&D AI experiments, Coaching, Action Plan, and Smart Follow Up.
Measuring Success
A part of the issue was that the team hadn’t defined concrete success metrics leading up to launch. Each function was chasing their own metrics, resulting in disparate strategies.
Aligning Product, Marketing, Sales and leadership required a handful of workshops, and, presentations, but we got there!
# In-app clicks on Convert Assist
This metric traced to discoverability of the Convert Assist feature.
# Paying subscribers over time
Aiming for a steady increase, tracking paying subscribers would speak to value.
$ Generated by overage charges
If Convert Assist was continuous used, the overage charges per account would increase month over month, becoming a positive indicator for usefulness.
Affecting Success
Solution Hypothesis: Positioning Convert Assist in high traffic locations will result in high usage.
Positioning
The Timeline: 3rd most visited page in the entire application, after Dashboard and Call Log.
Primary Task: Review lead history
Secondary Task: Identify next steps
(view larger)
Post-launch metrics
30 Days
1509
# In-app clicks on Convert Assist
155
# Paying subscribers over time
$4k
$ Generated by overage charges
60 Days
74 -95%
# In-app clicks on Convert Assist
185 +19%
# Paying subscribers over time
~$16k +292%
$ Generated by overage charges
Diagnosing the engagement problem
Session Reviews
Despite Convert Assist being born from market research, the drop in engagement was concerning.
Observations
Total sessions were reducing for paying customers
Clicks on Convert Assist button steadily decreased
Explanations
Users couldn’t find Convert Assist (Discoverability)
It wasn’t useful/valuable enough.
- Q4 ’24 Cohort
- Q1 ’25 Cohort
| Funnel step | Q4 ’24 Cohort | Q1 ’25 Cohort |
|---|---|---|
| Visited Call Log | 3,530 | 4,830 |
| Visited Timeline | 1,040 | 1,380 |
| Clicked CA CTA | 238 | 15 |
Customer Interviews
A round of interviews with active, paying subscribers of Convert Assist helped uncover the qualitative aspects of the engagement problem.
30+
Completed customer interviews
7
Marketing agencies
13
Small-medium businesses
Key Findings
A round of interviews with active, paying subscribers of Convert Assist helped uncover the qualitative aspects of the engagement problem.
01
The agent end user finds Convert Assist to be most useful. Integrating the insights with their source of truth, CRMs, was important. Managers/Owners personas found little value in call-by-call insights. Aggregations or summaries were more useful to these personas since they oversaw teams or in some cases, the entire business.
02
Convert Assist cost too much to be viable for SMBs. Because it ran automatically on every call logged in CallRail, the usage costs were ballooning to unsavory levels for the subscribers.
03
Despite Convert Assist’s position in the 3rd most interacted with page, it wasn’t as discoverable as it could have been.
I love the coaching and follow up. I’d rather see it in HubSpot when I’m emailing the lead.
I don’t review calls one by one. I want to know when a good quality lead wasn’t handled properly.
Goals and solutions
Improving Discoverability
User interviews confirmed the value of the insights generated by Convert Assist. Insights hidden behind a click in a panel were brought out to the main call card.
Position Convert Assist in agent workflows
In addition to building an API end point for insights that could feed into any CRM, Convert Assist’s Smart Follow Up was placed in CallRail’s Messenger. This helped increase the features’ utility for customer-facing Agent personas.
Convert Assist for Managers/Owners
Aggregates and summaries are extremely coveted by busy Managers and Business Owners. CallRail has two vectors for delivering these that perform extremely well, both emails: Multi Conversation Insights and Call Notification Summary.
01 Emails
With nearly perfect open rates, Multi Conversation Insight and Call Summary Notification got Convert Assist.
-
(view larger)
Call Notification Summary
02 In-app
Adding an in-app experience to support Owners/Managers reviewing insights in-depth complimented the email upgrades.
Cost management
To make the costs more appealing, controls to manage where and when Convert Assist ran were added to the admin experience.
(view larger)
Results
May 2025
The solutions made a positive impact- overage charge increase in particular indicates continual use. However, this change wasn’t enough to keep Convert Assist as a product line. Instead, its parts were broken up and absorbed by other product lines offered by CallRail.
60 -97%
# In-app clicks on Convert Assist
*Removed the CTA from the Timeline.
579 +212%
# Paying subscribers over time
~$41k +156%
$ Generated by overage charges
Reflections
01
The gap between what research indicates and what is actually useful is a thin line- following research data, especially from initiatives as broad as what CallRail did, led to a chasm that changes couldn’t bridge.
02
Trust in AI-generated outputs is earned in drops, and lost in buckets. By hiding the AI processes and explanations, Convert Assists’ reliability and accuracy were constantly in question.
03
Convert Assist fell victim to opinions and preferences of those who didn’t stand to benefit from it or use it- internal CallRail employees. Working on Convert Assist strengthened my persuasion skills.