The things algorithms cannot see in education

A few days ago I went to an education technology meet-up about how AI can help students learn. The discussion was richer than I expected. Some people are using reinforcement learning to optimise question setting and teaching paths; others are using large language models to track what a student knows. One speaker mentioned a hugely ambitious project: a team trying to collect one million recordings of real lessons to build an open dataset. You could feel the field seriously exploring what is possible.

But when it was over, one basic question surfaced in my head: what exactly are the algorithms and the data measuring?

What does "getting it" mean?

One concept that came up is called knowledge tracing: using an algorithm to judge whether a student has mastered a particular idea. The logic of the traditional models is simple. Answer correctly and you know it; answer wrongly and you do not. Now people are starting to use large language models for this, feeding a student's answer history to the model and asking it to judge whether the student has really understood. It sounds more intelligent. I still have my doubts.

One student can use the formula to work out the time of free fall and pass the exam. But ask him why a feather and an iron ball land at the same moment in a vacuum, and he may have no answer, because all he learned was to substitute the numbers. He never built a real understanding of gravity. Another student may calculate a little more slowly, but she can explain the principle behind it, and when the question changes shape she finds a way through. These two kinds of "knowing" are completely different things. In front of a standardised test, their scores can be identical.

Technology can capture behaviour, but it struggles to reach the texture of understanding. A large language model processes language. It can analyse what a student said, but it cannot necessarily understand what the student was thinking. Because language can bypass thought. A paragraph recited from memory and a paragraph genuinely understood can sound exactly the same. We have all seen it: the retelling is fluent, and two follow-up questions later it falls apart.

Guiding is not a cure-all

The meet-up also touched on how many companies, Google and OpenAI among them, are training "Socratic" teaching AIs that do not give the answer directly, but use questions to guide students to discover it themselves. It sounds ideal. In practice it is not so simple.

Socratic guided dialogue has a hidden premise: the student already has the relevant foundations in their head, and only needs them drawn out. But what about a genuine beginner? Ask someone who has never written code, "how do you think you would get a computer to repeat something?", and they have no framework for thinking about the question at all. They do not know what a variable is. They do not know what a loop is. At that point, further questioning is not guidance, it is torment. Telling them directly, "this is called a loop, and here is the syntax," may be the real help.

Genuinely good teaching should be able to judge: does this student, at this moment, need to be told directly or guided to explore? What that requires is not a fixed "teaching persona" but a live choice of strategy. And that judgement is very hard for an algorithm, because it requires sensing a person's state. Right now, are they curious or defeated? Stuck on the concept, or just forgetting one detail? If all you analyse is answers, you are still a long way from education.

Can education be standardised?

One more discussion kept me thinking for a long time. Someone said the education AI field needs a unified evaluation standard, the way image recognition has the famous ImageNet dataset: everyone measuring with the same ruler, so different systems can be compared.

But "is this a cat or a dog" can have an objective answer. What about "what is good education"? Exam scores? Creativity? Critical thinking? A lifelong appetite for learning? These goals even pull against one another. A system that chases scores with all its might tends to sacrifice intrinsic motivation and creativity. And the reverse is true too.

My own view is that the aims of education should themselves be argued over, not standardised. Different philosophies of education should have their own evaluation frameworks, competing with and criticising one another, rather than having their diversity wiped out by one unified standard. Once everyone optimises toward a single metric, the things that cannot be measured are systematically ignored. Is that not exactly the trap exam-driven education is in today?

The limits of the technology

None of this is to say technology is useless, or that exams should be abolished. I think precisely that they should have their own place. Technology can improve efficiency, extend scale, and personalise recommendations. That value is real. But we have to face one thing honestly: the subject of education is a human being, and a human being cannot be fully quantified.

One speaker also mentioned a figure: fewer than 5% of the education technology products on the market have ever validated their effectiveness. Not tested and found wanting; simply never tested at all. Perhaps that is exactly where the problem lies. "Learning better" is too hard to define, so everyone turns to what is easy to measure: time in the app, completion rate, correctness rate. Slowly, these metrics turn from means into ends. The product "succeeds," and whether the student actually learned anything, nobody knows.

Perhaps this is the humility education technology needs to keep: admitting that some of the things that matter most are exactly the ones we do not yet know how to measure. And that is not only education technology's problem.