- The shift: from “what did learners say” to “which version worked better”
- Stage 1: Get the open feedback into a state where AI can read it
- Stage 2: Theme each cohort independently first
- Stage 3: Run the comparison with the variable named
- Stage 4: Validate with a hard metric
- A worked example, in numbers
- The failure mode: pattern-finding in noise
- What this looks like at scale
The reason most training programs don’t visibly improve from one cohort to the next is not that the L&D team isn’t reading the feedback. It’s that reading 200 open-ended survey responses by hand and then writing a stakeholder report on what to change is a 10-hour task, which means it gets done badly, late, or not at all. AI changes that economics. The smarter teams are not just summarising feedback faster. They are running structured A/B tests across cohorts and using AI to isolate what actually moved the needle.
LinkedIn’s 2025 Workplace Learning Report finds around 47% of L&D functions have generative AI fully embedded, with personalised pathways and content authoring leading. Feedback analysis is the operations layer underneath both. Without it, every “improvement” is a guess. With it, the next cohort is measurably better than the last one for a reason you can name.
The shift: from “what did learners say” to “which version worked better”
The default approach to training feedback is descriptive. You collect comments, cluster the themes, write a summary, propose changes, ship the next version. The problem is you cannot tell whether the changes you made caused the changes you see in the next cohort, because you changed too many things at once.
The smarter approach borrows from product. Pick one variable per cohort. Change it deliberately. Measure the difference. Use AI to isolate signal from noise. Q1 cohort runs the original leadership module. Q2 cohort runs the same module with one change: the practice activity moved from the end of section three to the start of section one. Everything else identical. The question is no longer “what did learners think.” It is “did the structural change produce a different pattern of feedback, and did the right metrics shift.”
Stage 1: Get the open feedback into a state where AI can read it
Use SurveyMonkey, Typeform, or Google Forms for collection. Export each cohort’s open-ended responses as a single text dump, one comment per line, with no learner identifying information. Two text files, one per cohort. That is all the prep you need.
The reason to strip identifiers is not just privacy: it is because the model is more honest in its analysis when it is not pattern-matching to perceived seniority or department. The signal you want is in the text.
Stage 2: Theme each cohort independently first
Do not start with a comparison. Theme each cohort’s feedback on its own, then compare the themes. Mixing the two corpora in the same prompt produces a soft averaged summary that hides the difference.
“You are a learning analyst. Below are open-ended responses from learners who completed [MODULE NAME]. Theme the comments. For each theme, give: the theme name (4 to 6 words), the rough share of comments that touched on it (low, medium, high), the dominant sentiment (positive, mixed, negative), and three representative verbatim quotes. Identify any theme that fewer than 5% of learners mentioned but that, if true, would be worth investigating (a quiet but high-signal concern). Output as a table. RESPONSES: [PASTE].”
Run that prompt twice, once per cohort. You now have two structured theme sets you can compare cleanly.
Stage 3: Run the comparison with the variable named
Now ask the model to compare the theme sets, with the variable you changed named explicitly. Naming the variable is the entire point. Without it, the model will compare on every dimension and produce a noisy answer. With it, the model will tell you whether the variable produced a real shift or whether the noise dominates.
“Below are two theme sets from the same training module run with two different cohorts. Cohort A ran the original version. Cohort B ran a version where the only deliberate change was [DESCRIBE THE CHANGE PRECISELY, e.g. ‘the hands-on practice activity moved from the end of section three to the start of section one’]. Compare the two theme sets. Tell me: which themes shifted in volume between the cohorts, which themes shifted in sentiment, whether the change I made plausibly explains the shift, and which observed differences are more likely to be cohort noise than a real effect. Be specific about uncertainty. THEME SET A: [PASTE]. THEME SET B: [PASTE].”
The output is a structured read on whether your A/B test produced a signal. The honest answer is sometimes no. AI will tell you that, if you ask it to.
Stage 4: Validate with a hard metric
Sentiment analysis on its own is not enough. Pair it with a hard outcome: post-module quiz score, manager-observed behaviour change at 30 days, completion rate, time-to-application. Power BI handles the dashboarding. The AI-derived sentiment shift is corroborating evidence. The hard metric is the proof. A version that scores higher on sentiment but lower on quiz scores is not a better module: it is a more comfortable module, and “comfortable” is a failure mode for training, not a goal.
A worked example, in numbers
Take a 60-minute compliance module. Cohort A runs the original. Cohort B runs the version with the practice activity moved earlier. Cohort A’s themes: “video too long” (high), “didn’t see the relevance until the quiz” (medium). Cohort B’s themes: “the practice activity made the rest of the module make sense” (high), “I’d run this again” (medium). The hard metric: Cohort B’s quiz pass rate goes up, completion rate goes up, time-to-first-application drops. The change held. One variable, one cohort, one defensible improvement. Repeat the loop quarterly and the training program compounds.
The failure mode: pattern-finding in noise
The single biggest mistake L&D teams make when they discover AI-assisted analysis is to over-read small differences. With 25 learners per cohort, themes will shift between cohorts simply because two outspoken people had different days. The model will dutifully identify the shift, because that is what it was asked to do.
The fix is structural. Set a threshold below which you ignore the difference. A theme that shifted from “low share” to “medium share” in a 25-person cohort is mostly noise. A theme that shifted from “absent” to “high share” in the same cohort is signal. Tell the model the threshold:
“When comparing the two theme sets above, ignore any difference that could plausibly be explained by 2 or 3 outspoken individuals in a 25-person cohort. Report only the shifts that would survive a 50-person sample. For shifts you cannot rule out as noise, label them as ‘inconclusive at this sample size, worth re-running.’”
That single instruction is the difference between an AI-assisted analysis that improves your training and one that produces confident-sounding nonsense from random variance.
What this looks like at scale
The 47% of L&D functions running AI in production are mostly not running A/B tests yet. The ones who do are an order of magnitude rarer, and they are also the teams whose stakeholder reports get read. Senior leaders do not respond to “we collected feedback and made changes.” They respond to “we changed one variable, observed a 14-point improvement in pass rate, and here is the next test we are running.” AI does not produce that report on its own. It produces the analysis that lets you write that report in 30 minutes instead of two days.
The smarter way to use AI for training improvement is not to summarise feedback faster. It is to design experiments your training program is now cheap enough to run, then use AI to read the results without drowning. The teams compounding the quality of their training over the next year are the ones who have crossed that line.