The Experience API (xAPI) and analytics complement each other nicely. You can use the xAPI to track any type of user/system interaction and then use analytics tools to compile and interpret xAPI data. The interactions you track may result from learning systems and programs or from actual work systems and processes.
Applying xAPI and analytics to measure learning and work outcomes enables you to go beyond front-end analysis to continuously analyze your learning and performance solution’s impact on business results. This approach enables you to adapt your learning and performance solutions in response to the ever-changing needs and priorities of the business.
Following is a fictitious scenario illustrating a mission-critical organizational performance problem. We will explain how you can use a disciplined xAPI-driven analytics approach to address the problem.
The business challenge
Business is slow at Imaginary Widgets. In the past, that has often meant budget cuts to the cost centers, including L&D. This time, the EVP of sales is asking for help from the L&D Director.
Imaginary Widgets, which we will call IW, is a B2B company with six major product lines. The life cycle for each product line is about eighteen months after which they redesign the entire line. The company staggers the process so that one product line is overhauled every quarter. While IW is pretty good at marketing, the EVP of sales thinks they are not preparing their sales force well enough. Sales reps seem to be overwhelmed with the quarterly changes and revenue is significantly below targets for the first three to six months after introducing a product line update.
The L&D organization has had difficulty getting information about the product line changes far enough in advance to have their courses ready when the product is released. Once the courses are ready, it takes time to get the entire sales force through all the material. L&D must find a better way.
The front-end analysis
The L&D director assigns the problem to a team of her most talented instructional systems designers. The ISD team decides that a front-end analysis is needed to learn more about the problem. They start by meeting with each of the regional sales directors.
Several of the regional sales directors suspect that, for some time after product launch, sales reps are facing increasing difficulty moving customers through the sales funnel, which has five stages: prospect, contact, in negotiation, offer, and close. The regional directors are not sure, however, at what stages customers are dropping out compared to more successful points in the product life cycle.
Figure 1: Imaginary Widgets’ sales funnel
Sales funnel states and revenue are tracked in IW’s customer relationship management (CRM) system. Looking at CRM reports, the ISD team confirms that sales revenue drops significantly below target for three to six months after each product line update. The team also discovers that, during that time period, there are far fewer prospects and contacts combined with a higher percentage of dropouts after those funnel stages.
In an effort to identify the key challenges faced by sales representatives during the first three to six months after each product line update, the ISD team conducts several focus groups with sales managers.
Human performance issues contributing to the business challenge
The focus groups reveal two key performance issues:
- Key information needed to sell the updated products is extremely hard to find. This includes market research, competitor comparisons, product specifications, and collateral sales material.
- The product-line training takes too much time. The training itself is too long, costing valuable customer face-time. Many sales reps must sell untrained for days or even weeks before they can complete the training.
Workarounds
Focus groups also reveal that managers try to assist their sales teams with a variety of workarounds, some of which are more effective than others. Two workarounds were used by the higher-performing sales teams:
- Some sales managers have informal relationships with product managers, whom they call for product information.
- One sales manager, whose high-performing sales team’s revenues are consistently higher, holds weekly team meetings to share product information and customer stories. Interestingly, the sales reps on the high-performing sales team spend less time researching and training and more time selling than sales reps in other teams.
Findings
Wrapping up the front-end analysis, the ISD team presents their findings and recommendations to the L&D Director. First, the analysis revealed business problems, goals, and metrics (Table 1), which comprised direct success measures.
Table 1: Direct success measures
|
Business problem |
Goal |
Business metric to track |
|
Revenue below targets for three to six months after launch of updated product line |
Consistently meet or exceed revenue targets |
Product line sales revenue tracked in CRM system |
|
Lower numbers of prospects and contacts in sales funnel for three to six months after launch of updated product line |
Maintain consistent numbers of prospects and contacts through all 18 months of product life cycle |
Sales funnel counts tracked in CRM system |
|
Higher percentage of customers dropping after prospect and contact stages for three to six months after launch of updated product line |
Maintain consistently lower percentage of customers who drop after prospect and contact stages through all 18 months of product life cycle |
Sales funnel counts tracked in CRM system |
|
Time is allocated less efficiently for most sales teams compared to the one high-performing sales team |
Get other sales teams to use time efficiently like high-performing team does |
Percentage of time spent in four categories (selling, researching, training, other activities) tracked in time-tracking system |
Recommended solution
Based on these findings, the ISD team recommends a learning and performance ecosystem solution that includes three key components:
- A searchable product knowledge base where sales representatives can find relevant information quickly. Product managers and marketing experts will be responsible for publishing information the sales force needs.
- Online access to product management experts that allows sales representatives to discuss product questions and issues.
- A formalized structure and guidance for sales managers to conduct the same types of weekly information-sharing meetings that seem to be making the high-performing sales team successful.
Each of the learning and performance solution components includes a set of ongoing questions to answer through xAPI activity tracking. The questions and activity tracking comprise indirect success measures (Table 2).
Table 2: Indirect success measures
|
Learning and performance solution component |
Question to answer |
Activity to track |
|
Searchable product knowledge base |
What types of information are salespeople seeking and either finding or not finding? |
• Sales rep searches where results were selected and accessed •Sales rep searches where results were not selected or accessed |
|
What types of information are salespeople using? |
Sales rep consumption of information by: • Product (i.e., each product in the product line) • Type (i.e., market research, competitor comparisons, product specifications, sales collateral) |
|
|
What types of information are being populated? |
Product manager and marketing expert contributions by: • Product (i.e., each product in the product line) • Type (i.e., market research, competitor comparisons, product specifications, sales collateral) |
|
|
Online access to product management experts |
Which product management experts are sales people contacting? |
Number of contacts per product manager |
|
Which product management experts are sales people getting the most value from? |
Sales rep ratings of product manager interactions |
|
|
Weekly sales team meetings |
Which sales managers are conducting meetings? |
Whether weekly meetings occurred per sales manager |
|
Are sales reps attending the meetings? |
Number of sales reps in attendance |
The recommendations also include some sales representative attributes that are thought to impact success (Table 3).
Table 3: Sales representative attributes
|
Question to answer |
Profile data to track |
|
Does experience impact success? |
|
|
Do certifications and/or education impact success? |
|
|
How does the percentage of time spent on sales vs. research impact success? |
|
Need for analytics and the xAPI
Once implemented, the impact of the recommended learning and performance solution must be continually evaluated, adjusted, refined, and improved. Key to the success of this solution will be the application of xAPI and analytics.
Learning and performance activities (indirect success measures) and sales representative attributes will be tracked and compared with business metrics (direct success measures) using a dashboard. This approach will enable L&D to continue its analysis in a steady state and tweak the learning and performance solution as needed while sales leadership monitors the solution’s impact on revenue.
To assemble the dashboard elements, you must define an overarching xAPI tracking model. Before we review IW’s xAPI model, it is helpful to touch on how the xAPI works.
xAPI statements
The basic unit of communication in xAPI is the statement. The statement describes an experience within a specific activity, which provides the foundation for analytics. Statements rely on an actor-verb-object structure.
Take the statement:
Jack accessed the /widget2Description
This statement indicates that Jack (actor) has accessed (verb) the product description for Widget2 (object). The xAPI statement is built into the knowledge base system and is triggered by a specific action, in this case Jack accessing a document. Once triggered, the statement generates and stores tracking data to a learning record store (LRS). Later, analytics tools can be used to analyze all the xAPI data collected in the LRS.
So, with this in mind, the IW ISD team puts together an xAPI tracking model for Imaginary Widgets.
The xAPI tracking model
The first step is to define the xAPI data we will use. Figure 2 lists the direct and indirect success factors that the systems currently available to IW can measure.
Figure 2: Success factors
to track
Verbs in xAPI provide meaning to what is going on within a learning experience. When applied to the overall context of sales, these verbs become powerful ways of storing and querying data. Each verb describes an activity performed by a sales rep that results in a measure described in Tables 1-3, above.
IW considers each customer engagement as a sales opportunity. There are verbs for each stage of the sales funnel in an opportunity, which culminates (hopefully) with a close. Since sales of an individual product and products sold in a bundle are both important to track, two separate verbs are used—closed, which reflects all product sold in an opportunity, and sold, which is used to track each individual product. The verb, sold, retains the context of the opportunity, but is used to get more granular information on a single product.
Some of the data to analyze comes from a user profile rather than an activity statement. To handle this type of static data, the xAPI includes a State API. You can update the State API as often as desired to store information about a particular user. Figure 3 lists the data that the State API will track.
The xAPI allows hierarchies, dependencies, and other structural components, especially within activities. A robust xAPI model would make use of those capabilities, for instance to define higher-order activity types. For the sake of simplicity in this example, we will not get into these advanced structures.
Figure 3: Sales rep
attributes in a user profile
The xAPI model helps the ISD team plan how data they are tracking will feed the analytics dashboard. Direct success statements will produce counts, averages, percentage breakdowns, and measurements of the time it takes to progress through the various stages of the sales funnel. Indirect success statements will produce counts of experts contacted, knowledge base content published, and accessed. Indirect success statements will also identify things like search success rates, most-used resources and experts, most highly rated resources and experts, and whether weekly sales team meetings are occurring.
The xAPI tracking model will enable analysts to dig deeper to identify successful behaviors that drive high performing individuals and groups. Some examples of these behaviors may include sets of multiple activities that are more likely to result in sales success when performed in a specific combination or sequence.
The sales representative attributes, which appear in the user profile, will be available as optional filters for the direct and indirect success data so that further analysis can be done on what makes a high-performing sales team or individual.
After defining the verbs to be used in xAPI statements, the ISD team lists measures and maps relationships in a high-level diagram shown in Figure 4.
Figure 4: xAPI tracking model
Creating statements
Now that they have created the xAPI tracking model, the ISD team is ready to define xAPI statements. Most of the statements generated within this learning and performance solution will be structured like the example shown in Figure 5, below, which shows a code example of the statement “Sondra Greene accessed the Widget 565 Product Video.”
As far as the coding details go, the actor structure contains an agent declaration, the sales rep’s name (Sondra Greene), and the sales rep’s email address, which is used as a unique identifier, in this case sondra.greene@imaginarywidgets.com.
Next, a verb structure is used. The id is extremely important because, when querying verbs in the system, the verb will separate the activities. This company MUST use the same id for all resources accessed if it wants to effectively capture them all with a single query argument. The display is simply how that verb is shown on any sort of UI. Multiple languages are available, but only US English is shown here.
Finally, the object (activity if not declared otherwise), has an id corresponding to the resource (in a perfect implementation, the video would also be located at the id’s resolved URL: http://www.imaginarywidgetcompany.com/producttrainingvideos/sl1859f). You can include an optional definition as metadata. Although not required, inclusion of a definition is strongly recommended to provide more context for the analytics.
Figure 5: Coding example
1
For direct sales and other statements with a result object, take a look at Figure 6 below, which is the code for the statement “Sondra Greene sold one unit of Widget 17 for 10,000 USD as a part of sale widgetbuyer1021.” As you remember sold is used to identify an individual product sold as a part of a closed sale. xAPI uses context to identify the opportunity, named widgetbuyer1021, in which this particular sale took place.
The result portion is important for tracking outcomes of statements. While the verb part of the statement keeps track of an action, we still need a way to record whatever resulted from the action. The result section describes things we need to track resulting from the sold verb, including product ids, quantity sold, amount of revenue gained, and the monetary unit we are using.
Given that IW may implement additional future sales-related applications of xAPI, it is important for the company to document its standards and a common vocabulary for the sales-use model. IW has defined an enterprise-wide taxonomy in the form of an xAPI sales profile that contains the verbs and extensions (shown in Figure 6) that reflect a sale.
Figure 6: Coding example 2
Dashboard
Finally, the ISD team designs the analytics dashboard. The granularity of statements has made this relatively easy to do. They can run xAPI queries on specific actor-verb-object combinations to produce the dashboard (Figure 7).
Figure 7: Analytics dashboard
Conclusion
Over the coming months, IW’s sales leadership will pay close attention to the dashboard to determine whether sales revenue is improving during the first weeks of a quarterly product line update. L&D will be monitoring the dashboard to determine which aspects of the learning and performance solution seem to be having an impact.
The ISD team already has some ideas about how to expand their use of xAPI to make the learning and performance solution even more impactful. They plan to introduce a “Did you know?” page where a sales rep can get personalized performance improvement tips like “Did you know that you are 40 percent more successful when you spend five minutes researching the product immediately before a call?” and other trends that are now possible to identify through the xAPI and analytics. The team plans to clean up the knowledge base by archiving poor performing resources based on xAPI usage and rating data. They have also started to tag sales team meetings with product topics that were discussed to gain more insight into how these affect sales results. With so many possibilities, the application of xAPI and analytics to improve sales at IW has just begun.
All Contributors
