A board member asks the question at the end of the meeting, after the slides, after the photos of smiling participants: "So, did it work?" The program director says enrollment is up, satisfaction scores are high, and the team is proud of the launch. All of that can be true, and none of it answers the question. Things getting better after a program is not the same as the program making things better. Telling those two apart is the whole job of evaluation, and it's a job you can do well on a modest budget if you plan for it early.
I trained in evaluation methods at ICPSR's Summer Program and my doctorate concentrated in research, assessment, and program evaluation. Most of the evaluations I see go wrong for the same reason: nobody decided, before the program started, how they'd know whether it worked. This guide walks through the questions that decide that, in plain language, so you can commission or run an evaluation that a skeptical funder will actually believe.
First, decide what you're really asking
"Did it work" hides at least two different questions, and confusing them wastes money.
- Did we run it well? This is process evaluation. Did the program reach the people it was meant to reach, at the dose it was designed for, delivered the way it was written? A tutoring program that only met half as often as planned didn't fail to help; it was never really tested. You cannot judge an outcome you never delivered.
- Did it change anything? This is outcome evaluation, and it's the one the board is asking about. Did the people who went through the program end up better off than they otherwise would have been, on something you care about and measured?
There's also a timing split worth naming. A formative evaluation runs while the program is still moving, so you can fix it. A summative evaluation asks the up-or-down verdict at the end. A common and expensive mistake is to spend the whole budget on a summative verdict and learn, too late, about a delivery problem a formative check would have caught in month two.
The hardest question is "compared to what?"
Here's the trap at the center of every evaluation. You measure participants before the program and after, they improved, and it feels like proof. But you never get to see the other timeline: what those same people would have done without the program. That invisible comparison, the counterfactual, is the thing every credible design is trying to stand in for.
Things getting better after a program is not the same as the program making things better. The entire craft is estimating what would have happened anyway.
Plenty of things move outcomes on their own. People who sign up for a job-training program are often the ones already looking hardest for work, so they'd have found jobs at a higher rate regardless. A reading score measured right after a rough baseline tends to drift back up on its own, a pattern called regression to the mean, whether or not you intervened. The economy improves, a season changes, a policy elsewhere shifts. A good evaluation isolates the program's effect from all of that. A weak one takes credit for the tide.
Designs you can actually afford
You do not need a laboratory. You need a defensible answer to "compared to what," and there's a ladder of options from strongest to most practical.
Randomized assignment, when it's possible
If more people want the program than you can serve at once, a lottery is both fair and rigorous. Randomly assigning who gets in now and who gets in later creates two groups that are alike on average, so any later difference between them is the program's effect. This is the cleanest design, and it's often available in exactly the situations people assume it isn't: a waitlist is a randomization opportunity hiding in plain sight.
A comparison group you didn't randomize
When you can't randomize, find a group as similar as possible to your participants who didn't go through the program: another site, a later cohort, applicants who didn't enroll for reasons unrelated to their outcomes. The risk is that the groups differ in ways you can't see, so methods like matching and difference-in-differences try to level the field by comparing changes over time rather than raw levels. It's not as airtight as a lottery, but a well-chosen comparison group is far more convincing than a single group measured twice.
Interrupted time series
When a program or policy switches on at a known date and you have data from well before and well after, you can use the program's own history as the comparison. You establish the trend line the outcome was already following, then test whether the line shifts, in level or in slope, right when the program started. I've used this approach to study enrollment policy, where you often can't assign students to conditions but you do have years of records. Its strength is that the pre-program trend does a lot of the work of ruling out "it was already heading there." Its weakness is anything else that changed at the same moment, so you note those threats honestly rather than pretending they don't exist.
Before-and-after, used honestly
Sometimes a simple pre-and-post measure on one group is all the budget allows. That's fine, as long as you say plainly what it can and can't show. It can tell you whether participants changed. It cannot, on its own, tell you the program caused the change. Reported with that caveat, a pre-post result is honest evidence worth having. Reported as proof of impact, it's the most common way evaluations mislead the people who paid for them.
Numbers tell you whether, words tell you why
A purely quantitative evaluation can show that scores moved without explaining what happened inside the program to move them, and that gap matters when you want to repeat or scale the work. This is where mixed methods earn their keep. Interviews, focus groups, and open-ended responses tell you which parts participants actually used, where the program broke down, and why a number went the way it did. I analyze this kind of material in tools like Atlas.ti and pair it with the quantitative side in STATA, SPSS, or R. The two halves check each other: if the numbers say it worked but participants can't point to anything that changed for them, that's a signal to look harder before you celebrate.
Ways an evaluation quietly goes wrong
Most misleading evaluations aren't dishonest. They're the result of a few predictable errors.
- Measuring the easy thing instead of the real thing. Attendance and satisfaction are simple to collect, so they stand in for outcomes nobody actually measured. A workshop can be well attended and warmly reviewed and change no behavior at all.
- Teaching to the metric. Once a single number becomes the target, people optimize the number rather than the thing it was supposed to represent. Pick measures that are hard to game and, where you can, more than one.
- Fishing for a positive result. If you test the program against twenty outcomes and report the two that came out well, you've found noise, not effects. Decide your main outcomes before you look at the data, and say how many things you tested.
- Confusing statistical significance with mattering. A result can be statistically detectable and too small to justify the cost, or genuinely important but measured on a sample too small to confirm. Report the size of the effect and its uncertainty, not just whether a p-value cleared a line.
What a useful evaluation report looks like
The best report I can write for a funder is one they can act on, not one that hides behind method. It states the question in plain language, shows what was actually delivered before claiming any effect, gives the size of the result with an honest range around it, and separates what the design can support from what it can't. It says what surprised us. It names the threats we couldn't fully rule out. And it ends with a decision the reader can make: keep it, fix this specific part, test it at larger scale, or stop. A report that can only conclude "the program was a success" usually wasn't designed to be able to conclude anything else.
The takeaway
Whether a program worked is a real, answerable question, but only if you decide how you'll answer it before the program starts. The design doesn't have to be expensive; it has to have a credible "compared to what." Build that in early, measure what matters rather than what's easy, and be honest about uncertainty, and you'll end up with evidence a board trusts and a program you can actually improve.
Program evaluation and policy analysis is part of what I do. If your agency, institution, or nonprofit needs an evaluation designed or an existing one made defensible, get in touch and we can scope it.