Mandarin Learning App Pilot

So we built an app. It teaches Chinese characters. We thought it was pretty good.

That last sentence is the problem. "We thought it was pretty good" is not evidence. It's a vibe. And if you want to know whether a learning tool actually teaches anyone anything, you eventually have to stop guessing and go find out. That's what a pilot study is: a small, honest first test, run before you've sunk a year into building the wrong thing.

Here's how we did ours at E-Bridge, and the stuff I wish somebody had told me on day one.

Step 1: Ask a question small enough to answer

My first research question was basically "does our app work?" Which is a terrible question. Work how? For whom? Compared to what?

The version we ended up with was much narrower: does grouping vocabulary by meaning help learners remember it better? So instead of teaching characters in a random order, we sorted them into semantic categories — food, animals, family, that kind of thing — and looked at whether category grouping changed retention.

That's a question you can actually answer with a small group of people in a few weeks. "Does our app work" is a question you answer with a decade and a lab.

Rule of thumb: if you can't picture the graph before you collect the data, your question is still too big.

Step 2: Know exactly who you're studying

We worked with heritage learners — kids who grow up hearing Mandarin at home but often can't read or write it. This matters enormously. A heritage learner already has the sounds and a lot of the vocabulary in their head; what they're missing is the writing system. They are not beginners, and if you test your app on true beginners and then claim it works for heritage learners, you've learned nothing about the people you actually built it for.

Write down your population before you start. In one sentence. If you can't, you don't have a study yet.

Step 3: Design the thing carefully, then design it again

Ours ended up as: 36 single-character words, split across 6 semantic categories, taught through the app, with retention measured over repeated sessions.

Some choices that seem small but really aren't:

• Same number of items per category. Six categories, six words each. If one category has ten words and another has three, you can't tell whether the category mattered or the amount of practice did.

• Single characters, not compounds. Fewer moving parts. In a pilot, simple beats realistic.

• Everyone sees everything. Each learner practiced words in both arrangements — grouped by meaning, and mixed together. That's called a within-subjects design, and with a small group it's a lifesaver: every kid becomes their own comparison, so you're never stuck trying to argue that Group A and Group B were equally good at Chinese to begin with.

Step 4: Accept that your sample is going to be small

We ended up with 13 learners who used the app more than five times. Not 13 signups — 13 who actually kept showing up. That distinction matters, and I'd encourage you to decide what counts as a real participant before you look at the data, so you're not tempted to draw the line wherever it flatters you.

Thirteen is small. I spent a while feeling bad about that, and then I got over it, because a pilot study is not supposed to prove anything to the world. It's supposed to tell you whether the thing is worth scaling up. With 13 kids and three-plus weeks of practice, we learned:

• whether the app actually ran on real devices in a real classroom (partly!)

• whether kids would use it without being nagged (sometimes!)

• whether our data logging captured what we needed (no, and thank god we found out then)

That last one is worth the whole study by itself. Discovering your app doesn't record the timestamp you needed is annoying with 13 users and catastrophic with 500.

Step 5: Collect data on purpose, not by accident

Decide what you're logging before launch. We wanted: which word, which category, correct or not, how long the answer took, and when the session happened. Then we ran a fake session ourselves and checked that all of it landed in the database in a form we could actually analyze.

Also: three-plus weeks of practice. Anything shorter and you're measuring short-term memory, which is not the interesting kind. The whole point of character learning is what survives a week later.

What I'd actually tell you

A pilot study is not a mini version of a real study. It's a different thing with a different job: finding out everything that's broken while it's still cheap to fix.

Ours told us our logging was incomplete, that classroom time is chaotic, and that a group of motivated kids will absolutely keep using an app for weeks if it isn't annoying. None of that is publishable. All of it made the next version better.

Start smaller than feels respectable. Write the question down. Check your data pipeline with fake data. Use math you can explain to a friend. Then go find out whether the thing you built does what you think it does — because "we thought it was pretty good" is still not evidence.