With the financial advantages of big data gaining global attention, C-level managers and senior business leaders are beginning to demand better reporting and improved accountability across all organizational divisions. While the goal of measuring learning seems unreachable for most in the training division, the reality is that these technological advancements mean that disruption in the training industry is inevitable. These innovations will enable unprecedented reporting of training effectiveness, business impact, and return on training investment.

In this session you will learn about the looming industry disruption, including the advantages and the benefits. You will explore the tools to develop an overarching strategy to prepare your organization for the change and provide tools to communicate with your training team, peers, and other business leaders. You will discuss strategies for planning a long-term change within your organization. This session will examine what’s happening with analytics in a logical and easy-to-understand way. You will leave this session understanding what you need to know about analytics, what you need to do now, and how to do it.

In this session, you will learn:

• True learning measurement and how to separate useful and meaningful data from the noise
• The effects of big data on the way training is currently done
• Strategy for communicating to other business leaders, peers, and teams
• Tools for developing a roadmap for change within your organization

Audience:
Intermediate and advanced managers, directors, and executives with an interest in the future of eLearning, decision-making authority, and/or the responsibility to manage change.

Technology discussed in this session:
Enterprise Infrastructure, learning management systems, HRIS, learning record stores, the cloud, and analytics dashboards.

Many training professionals find it challenging to know if training was effective on the job or if part of the organizational system is not supporting the training delivered. While many training departments collect data under Kirkpatrick’s levels one or two, few are able to track data beyond that. Training departments need a methodology for finding out training needs without on the job observation or having trainees return for a test.

In this session participants will explore the problem-based inquiry (PBI), a method used to systematically mine data that already exists at the organizational level that provides insight into what may need addressing with training. You will learn how this information is mined from the organizational help desk’s database of actions workers struggle with on the job. You will explore the tools and methodologies needed to use the PBI method. You will discover how inexpensive the PBI method is to employ and how it utilizes data already captured by the organization by the help desk.

In this session, you will learn:

• How to indirectly evaluate training programs
• How to use the problem-based inquiry method
• The benefits of indirect training evaluation
• How to use data already captured to improve training

Audience:
Novice to advanced designers, developers, project managers, managers, and directors with a basic appreciation for the need to assess training effectiveness.

Technology discussed in this session:
The problem-based inquiry methodology.

In the world of Big Data, organizations are increasingly using data and technology to better understand the value of their investments and efforts. As training professionals, it is imperative that we also use data to communicate the impact of our work. Many learning professionals avoid data and measurement because it seems like a daunting task, but it doesn’t have to be.

In this session, you will examine eight smooth steps for involving stakeholders in developing an evaluation framework for your eLearning initiatives. You will discover how this framework can be foundational for creating a common language for stakeholders. You will learn how the use of this framework can minimize the number of implementation issues, and provide a structure for the ongoing assessment of eLearning programs.

In this session, you will learn:

• The importance of evaluation for eLearning
• About an evaluation framework for implementing eLearning
• Ways to foster eLearning evaluation
• The importance of setting a common language with stakeholders

Audience:
Novice to advanced designers, developers, project managers, and managers.

The availability of (and demand for) data around learning has grown dramatically. Tools like the Experience API (xAPI) make it increasingly easy to acquire data about learner’s activities, but this will provide little benefit to instructional designers or learners if we do not design to acquire meaningful data, know how to interpret that data, or know how to improve our learning design based on that data.

In this session participants will explore the use of data in the context of learning systems design. You will examine some basic principles surrounding the effective use of data and how to design to provide meaningful feedback. You will discuss concepts including comparative mapping of novice to expert practices and learner experiences with formal courses and other activities. You will explore the application of principles from UXD, web analytics, and business intelligence to design for meaningful (and actionable) contextual data. Examples of how using data generated by the xAPI that can be used for predictive analytics to make interventions seamless for the end user will also be discussed.

In this session, you will learn:

• How to apply principles from fields including business intelligence and web analytics to learning design
• How to design to gather meaningful data within the context of course goals and overall performance objectives
• How to use data analytics to improve course design
• Potential pitfalls of data interpretation

Audience:
Novice, intermediate, and advanced designers, developers, project managers, and managers. A conceptual knowledge of the xAPI is useful, but not required.

Technology discussed in this session:
The xAPI, learning record stores.

eLearning has a broken feedback loop, and it’s holding us back as a field. Because we usually can’t see our products being used, we lack the most basic information necessary to improve what we do. Traditional evaluation at best is costly and difficult to measure, and at worst either ignored all together or implemented in such a superficial way that it’s meaningless. Even good evaluation measures are not granular enough to inform future design decisions.

In this session you will explore the need for designers and developers to be able to see the impact of our creations in a meaningful way, so we can grow as a field. You will discuss the importance of this need being addressed in a way that is both inexpensive and easy to implement. You will examine the concept of “guerrilla evaluation” methods and what we can learn from the field of software usability. You will leave this session with a list of practices we need to add to that will ensure successful eLearning design.

In this session, you will learn:

• How to recognize when you have a broken feedback loop
• How to use user-experience best practices
• How to conduct a guerrilla evaluation
• How to ensure you are moving forward as a practitioner

Audience:
Novice to advanced designers, developers, project managers, managers, and directors.

Technology discussed in this session:
User-experience practices and guerrilla evaluation methods.

Everyone’s talking about Big Data, but learning systems already generate piles of data and it’s a mess. How can we move on to a more in-depth analysis of learner behavior if we can’t even straighten out the basics? Our technicians roll their eyes and complain about poor data quality in the upstream feeds. The report designers complain that the stakeholders keep changing their requirements, and our teams are burning precious hours managing unwieldy Excel spreadsheets. We don’t need a new technology. We need a new approach.

In this session participants will examine a framework for understanding how to efficiently work with learning data. You will explore an overarching approach that you can use to communicate to both technical staff and business stakeholders. Using humor and visualization, this session will introduce you to techniques, tools, and concepts that are not usually thought of by database professionals outside of the learning industry. 

In this session, you will learn:

• How to clarify the business drivers for gathering data and use them to inform your strategy
• To look at learning data holistically so that you can find solutions to data problems
• To discover techniques to use existing data to generate the data needed for reporting
• How to put data limitations into a perspective that business stakeholders understand
• How to create low-effort, repeatable, and flexible processes for collecting, cleaning, and reporting on learning data

Audience:
Novice to advanced designers, developers, project managers, and managers.

When learners are geographically dispersed, and the training is completed, it is difficult to know if the skills were transferred back on the job. In addition, it is difficult to observe the learner performing a new task, and to provide feedback for improvement and/or reinforcement of what was done well. We need to find ways to use technology to enable us to effectively evaluate learners’ ability to apply what they learn in training.

In this session participants will explore two real-world case studies, demonstrating the design and technology considerations applied to implement two 360-degree practicums. You will discuss key considerations for getting buy-in from evaluators. You will examine the communications used, review examples of templates and how-to aids on using software applications, and see samples of the resources provided to both learners and evaluators.

In this session, you will learn:

• Creative ways to leverage technology in order to implement remote skill assessment
• Considerations for getting buy-in and involvement from the evaluators
• A framework to give to the evaluators so they are confident with how and what to assess
• Planning steps for how to support the knowledge needed by the learners for them to be successful in the assessment
• How to measure success

Audience:
Intermediate designers and developers with some experience with online survey tools, HTML coding, and awareness of Kirkpatrick levels of evaluation.

Technology discussed in this session:
SurveyGizmo, video conferencing (Lync), mobile phones, LMS.

Instructional designers are seeking more than just knowledge retention from learners; we are seeking positive, quantifiable, and repeatable changes in behavior—the gold standard of eLearning results in the business world. However, these behavior changes are challenging to facilitate, and even more challenging to track.

In this session you will learn how to credibly answer the age-old measurement question: Do retail salespeople who complete online training on specific brands sell more than those who don’t complete the training? You will examine the creation and results of a study comparing point-of-sale data to sales associates’ engagement with eLearning courses in the categories and brands they sell. You will discuss the factors that were used to isolate the effects of the eLearning courses from other factors that affect sales. You will explore how you can adapt this case study in your own organization.

In this session, you will learn:

• How eLearning really can make a difference in business
• How leveraging common goals with your partners can advance your business
• How creative problem-solving can help you find data that will help drive your business
• Why successful people thrive on training that can help them do their job more effectively

Audience:
Novice to advanced designers, developers, and managers.

Technology discussed in this session:
HTML5.