We Measure What's Easy, Not What Matters
Completion rates tell you that someone was in the room. Post-training satisfaction scores tell you that they enjoyed the experience. Neither tells you whether they can do anything differently on Monday morning.
The fundamental measurement problem in learning design is that the outcome we care about — changed behaviour in real work contexts — is the hardest thing to measure, so we substitute proxies that are easy to collect and call it evaluation.
What I Actually Measure — and When
I use an adapted Kirkpatrick model as the structural spine, modified for the realities of East African programme delivery: variable connectivity, sparse management oversight infrastructure, and learner populations who don't have time for lengthy evaluation instruments.
Learner Experience & Engagement
"Did the programme create the conditions for learning? Would learners engage again?"
- Session-by-session engagement (not end-of-programme)
- Perceived relevance to real work context
- Psychological safety in the learning environment
- Facilitator quality from learner perspective
- 4-question pulse check after each module (paper or SMS)
- One open-ended reflection prompt per session
- Facilitator observation notes for disengagement patterns
Skill & Knowledge Acquisition
"Can learners demonstrate the target competency under controlled conditions?"
- Performance on scenario-based assessments (not recall tests)
- Pre/post competency self-assessment against observable indicators
- Peer assessment quality during live facilitation practice
- Embedded scenario activities with structured observation rubrics
- Micro-facilitation observation (not role play — real practice)
- Pre/post competency indicator checklist completed by learner and peer
Transfer to Real Work Context
"Are trainers actually facilitating differently 60 and 90 days after the programme?"
- Observable facilitation behaviour vs. pre-programme baseline
- Use of programme-developed tools in live delivery
- Adaptation decisions trainers make in the field
- Peer feedback quality given to other trainers
- 60-day structured check-in (phone or WhatsApp voice note protocol)
- Field supervisor observation form (4 observable indicators)
- Trainer self-review protocol (built into programme, not added later)
- Tool usage documentation (trainers photograph their toolkit use)
Organisational & Programme Impact
"Is the organisation getting what it needed from this investment in trainer capacity?"
- Programme reach — sessions delivered post-certification
- Downstream learner feedback from trained trainers' sessions
- Trainer retention and continued programme engagement
- Reduction in facilitation quality incidents
- Programme delivery log (simple, trainer-maintained)
- Spot-sample of downstream learner feedback (quarterly)
- Annual programme coordinator review against baseline
The Measurement Timeline
The most common M&E failure is collecting all evaluation data at the end of the programme. By then, it's too late to adapt delivery, and the data serves reporting rather than improvement. Here's how I distribute collection across the programme cycle.
| Phase | Timing | What Gets Collected | Purpose |
|---|---|---|---|
| Baseline | Before Day 1 | Competency self-assessment against programme indicators; prior training inventory | Benchmark establishes where learners actually start |
| During Delivery | After each module | 4-question pulse check; facilitator observation notes; activity performance data | Adaptive allows real-time programme adjustment |
| Programme Close | Final day | Post-competency self-assessment; peer facilitation observation; programme reflection | Learning Level 2 measurement against baseline |
| 30-Day Follow-up | One month post | Brief check-in on first facilitation attempts; tool usage; early challenges | Transfer early signal on behaviour application |
| 60–90 Day Review | Two–three months post | Field supervisor observation; structured trainer self-review; peer feedback documentation | Transfer primary Level 3 data collection point |
| Annual Review | 12 months post | Programme delivery log review; downstream learner spot-sample; coordinator debrief | Results Level 4 — impact on organisational outcomes |
What the Instruments Actually Look Like
In East African field programme contexts, evaluation instruments have to survive the same constraints as learning content: low connectivity, limited time, mixed literacy, and no administrative infrastructure to chase up non-responses. Every instrument I design meets this test: can a field trainer complete this in under 5 minutes with a basic smartphone or paper form?
Module Pulse Check
After each module — 3 minutes
Four questions on a 1–5 scale: relevance to my real work, clarity of the content, quality of facilitation, one thing I'll do differently. Delivered via paper card or WhatsApp poll for digital cohorts. Aggregate reviewed before next day's session.
Competency Indicator Checklist
Pre-programme and programme close
12 observable indicators mapped directly to programme objectives — stated as visible behaviours, not knowledge claims. Completed by the learner and by a peer. The gap between self-assessment and peer observation is often the most useful data point.
Facilitation Observation Rubric
During live practice and 60–90 day field review
4 observable dimensions: group engagement management, content adaptation, pacing judgment, feedback quality. Scored by observer on a 3-point scale with a required evidence note. Designed to be usable by field supervisors who are not instructional designers.
60-Day Structured Check-in
Two months post-programme
6-question voice note or text protocol sent to trainers via WhatsApp. Questions focus on: sessions delivered, tools used, hardest facilitation moment, what they'd change about their own delivery. Responses recorded and coded against Level 3 indicators.
Trainer Self-Review Protocol
After each facilitation session — ongoing
A one-page form built into the facilitation toolkit itself, not a separate evaluation document. Trainers complete it immediately post-session as part of their facilitation workflow. Questions: what worked, what didn't, what I'd adapt next time.
Programme Delivery Log
Maintained by trainer — ongoing
Simple grid: date, location, topic, number of participants, one challenge noted. Collected quarterly. This is the primary Level 4 data source — it tells you what reach the training investment actually produced over time.
Closing the Loop: Data → Decision → Change
Collecting evaluation data only matters if it feeds programme decisions. Here's how I build the review cycle into programme management so data doesn't just accumulate — it changes things.
During & post-delivery
Patterns & gaps
Design implications
What changes
Update programme
Next cycle
The MATTERS CBE programme completed one full cycle of this loop between cohort 1 and cohort 2. The primary change: the peer observation protocol was restructured after cohort 1 data showed inconsistent feedback quality between observer pairs. Cohort 2 received a calibration session before peer observation began.
How This Framework Maps to a Real Programme
Here's how each framework element was applied — or is planned for application — in the MATTERS CBE Training of Trainers programme.
What Was Implemented
Module pulse checks after each of the five modules. Pre/post competency self-assessment. Live facilitation observation using the rubric during residential sessions. Trainer self-review protocol integrated into the facilitation kit.
What Cohort 1 Data Changed
Peer observation calibration session added for cohort 2. Module 3 (Group Dynamics) extended by half a day based on facilitator observation notes showing insufficient practice time for the target skills. Pulse check format simplified after cohort 1 reported it felt too long.
What's In Progress
60-day structured check-in protocol being rolled out for cohort 2. Field supervisor observation form distributed to programme coordinators. Programme delivery log designed and in use with 6 certified trainers from cohort 1.
What's Not Yet Done
Annual review cycle not yet completed (programme is in year 2). Downstream learner spot-sampling not yet systematised. Level 4 impact data is currently anecdotal. This is the honest status — and the next design priority.
Honest Assessment
The measurement infrastructure for Levels 1 and 2 is solid. Level 3 (behaviour transfer) is partially in place and producing early data. Level 4 (organisational results) is the honest gap — we're collecting the right leading indicators, but the longitudinal data set is still being built. I'll update this page as that data matures.