|
LEVEL
ONE -
Reaction
Did the learners like the presentation? |
| TOOL |
BENEFIT |
BARRIER |
Online
Evaluation or
Survey
Determine
learners subjective reaction to navigation, structure, media,
content |
-
Immediate
feedback - modules can be refined without delay for data
entry and report generation
-
Less expensive - Automate collection of results
-
Increased
Accuracy - no entry error or re-entry overhead
-
Higher
Completion rates - quicker and easier than paper based
-
Higher
Validity - perception of anonymity
|
- May
be difficult to align evaluation with a fully dynamic module
- What is the business value of "liking?" - None
- Still evaluating an "event" not process
|
Web
Log Reports
Objective
reporting about how and when learners are accessing content. |
Log
management software available for free or low cost to report...
WEB
TRENDS EXAMPLE
- Number of requests - indicates usefulness of content
- Number of distinct hosts - indicates spread of "referrals"
- Visitors
Over Time - Detail by day with Avg Length
in minutes and Total
- Referring Site Report - indicates linear or non-linear access
- Top
Search Keywords - indicates what learners are looking for and if they're finding
it.
-
Hits
by Hour of the Day - Detail
by Hour - Business and Non-Business Hours (according to where
server is located)
- Paths
- where did people go within the site?
- User Report - who's using what - when
- Entry point pages - starting point
- Exit point pages - end point
- Geographic Breakdown - by state, country and continent
Some software also allows you to track the path a learner took
through the site to see how they linked concepts. |
Amount
of information can be overwhelming. |
Incomplete
Lessons
Incorrectly
used to indicate that eLearning is ineffective |
There
are many reasons a learner may not complete a lesson.
They may have learned what they needed, it may be a judgment of
the content. There's no way to know by only tracking this statistic. |
Using
old metrics to measure new processes |
|
LEVEL
TWO -
Learning
Did the learners learn anything? |
| USE |
BENEFIT |
BARRIER |
Online
Pre - Post Testing
Objectively
determine impact of learning content
|
- Allows
learners to "test out" of required modules
- Information
can be summarized from all learners to indicate undefined gaps
- Can
be used to evaluate effectiveness of specific content or objects
|
- Baseline
information may be unavailable
- Some
people can do the tasks but do poorly on "tests"
- No
indication of usefullnes of what they learned
- Event
focused
|
Email
follow up
Subjective
determination of effectiveness |
Can
be automated for multiple contacts to determine retention over
time
Reduced
halo effect by being separated from presentation. |
Can
be difficult to summarize |
Post
Learning Survey
Determine
learners subjective opinion |
Allows
time for learner to reflect |
Timing
is crucial |
| LEVEL
THREE -
Transfer
Did the learners apply what they learned
to their job? |
| USE |
BENEFIT |
BARRIER |
360'
Manager, Peer, Client Assessments
Gather
subjective evaluations from learner's contact sphere |
- Multiple
viewpoints
- Can
be personalized based on learning content accessed
- Can
be used for overall performance analysis
|
Learners
may be intimidated knowing that others are evaluating them. |
Moderated
Email Forum
Solicit
comments and stories from learners |
Validates
learners experience |
Organizing
and summarizing information |
Cookies
and other site tracing tools
Gather
objective information |
Used
to link learner's interaction with content to their performance |
- May
be security or privacy issues
- Site
hosting tools may be inadequate
|
Amazon
Approach
Determine
development of SME or guru resources |
Allows
Learners to "rate" resources, either people or content,
with quality or usefulness ratings. |
- Requires
a robust infrastructure
- Many fears about allowing anyone to rate the resources - no control.
|
| LEVEL
FOUR -
Organizational
Impact
How did the organization benefit from the learner's increased
performance? |
| USE |
BENEFIT |
BARRIER |
BLOGS
- Web Logs
Collect
comments and comments about other's comments about content, presentation,
usability, strategic impact and other issues. |
-
Can be used to develop communities of interest as a mini-portal
- Easy to use
- Requires less bandwidth than email forums.
- Informal - less hidden agendas
|
A
new communication tool that may not be understood |
Spend
$100
Allow
managers to "spend" $100 in any combination of modules
or courses |
Generates
an easy to calculate summary of which modules or courses are valuable
to management and determine the priorities from a management point
of view. |
Managers
have to take time to reflect on their choices |
Drop
In Rates
Show
Depth and Breadth of Module Penetration and Need |
- Compare
difference between number of target attendees vs true attendees.
- Modules that have great word of mouth will show more attendees
than anticipated
|
Most
organizations measure for failure not success.
Many systems only
flag areas that appear to be under-performing, not over-performing |
Time
Based Analysis
Determine
direct impact on a functional group |
- Can
show a direct impact on results
- Results can then be extrapolated over time to show ROI
|
There
may be other variables affecting the levels of performance during
a particular time. |
Control
Group
Objectively
compare performance of those who have vs have not accessed learning
modules |
Very
clear, easy to present, numerical results about performance impact
in whatever ever metrics are important to that function. |
It's
difficult to create control groups that are equal except for the
amount of learning they've accessed. |
