Academic Study Shows Motivated Babbel Learners Improve Speaking Skills, Grammar and Vocabulary

Despite the commercial success and growing ubiquity of language learning apps, little information is available on app users’ real-world learning outcomes. Babbel is committed to providing transparency on how dedicated learners can improve their speaking skills. This is why we recently teamed up with second language acquisition experts at Michigan State University (MSU), including Dr. Shawn Loewen (pictured at right). The resulting study found that participants on the whole improved their speaking skills, grammar and vocabulary after approximately 12 weeks of learning with Babbel.

What sets Babbel apart from other language learning apps and vocabulary trainers is our focus on developing practical, real-life conversation skills. We believe our method instills confidence and competence in using a new language outside of the app – and not just at an advanced level, but even after the first few lessons. In order to establish the efficacy of Babbel’s method and discover ways we could improve, we regularly collaborate with applied linguists who study second language acquisition. On a recent project, my Babbel colleagues and I had the privilege to work with PhD candidate Daniel Isbell  and Dr. Shawn Loewen, Director of MSU’s Second Language Studies program.

We recruited English-speaking undergraduates at MSU to study Spanish with Babbel. At the beginning of the project, they took tests assessing their existing oral proficiency in Spanish, as well as vocabulary and grammar knowledge. All participants who completed the project then took the same tests after learning Spanish with Babbel over approximately 12 weeks. As a group, participants made statistically significant gains in oral proficiency, grammar knowledge, and / or vocabulary knowledge. Learners scored improvements correlated with the amount of time they invested in using the app. 96% of those participants who logged a minimum of 10 hours with Babbel saw better test scores on grammar and vocabulary tests and 73% became better speakers. The full report is available here, and the research team at MSU plans to publish the results in an academic journal.

This type of collaboration between an academic institution and a commercial language learning software is, unfortunately, still quite rare. I interviewed my co-authors to find out what they had to say about working with Babbel and how they interpret the study’s results.

What first got you interested in studying app-based language learning?

Dan (seen at right): What really got me interested in the field was realizing that there was so little academic research on learning in this context despite there being hundreds of millions of people using apps – some of them using apps exclusively – to learn another language. Especially with widely-used commercial apps, there are extremely few rigorous academic studies like ours. At the same time, as someone interested in the science of language learning, I couldn’t help but be skeptical of some of the claims I had seen various companies make about their apps’ effectiveness and learning speed. Why not try to fill the gap and test those claims?

Shawn: Typically, in the field of second language acquisition, researchers don’t investigate the effectiveness of commercial products. In part, I think it’s because language apps and other programs are not viewed as serious enough for scientific inquiry.

This study is a fairly rare example of a traditional academic institution and a language app working together to undertake a rigorous study of a platform’s learning outcomes. For Babbel, these results are incredibly useful. They provide insights used by our product team to reinforce learner motivation and guide them towards successful habits. What are the benefits of such a collaboration from your perspective?

Shawn: From a research standpoint, one of the primary benefits of working with Babbel was having access to the data metrics regarding our study participants’ learning time and how they actually studied. Without Babbel’s input, we would not have been able to accurately track the amount of time learners spent on the app, how frequently they accessed it, or which lessons they studied.

From a more personal standpoint, it was very interesting to hear the inside perspective from a language instruction company. Researchers are often dismissive of what we view as extravagant claims about how fast and easy somebody can learn a language. However, from a business perspective, you’re not going to get someone to buy your product if you tell them that it won’t actually help them. So while I’m still cautious in what I would claim in terms of language learning, I agree that it’s better to have people study a language than not.

Dan: I really appreciated the workflow we were able to establish related to participant data. We had access to accurate Babbel analytics for participants while keeping their corresponding objective measures—the vocabulary, grammar, and speaking test results—exclusive to the MSU side during the research. This was really key for ensuring the independence and rigor of the research. I really appreciated Babbel’s support of our independence in doing this study.

On the whole, participants in this study improved their oral proficiency, as measured by a score increase on a speaking test called the Oral Proficiency Interview computer-version . How would you define oral proficiency?

Shawn: I would say it means being able to engage in a conversation with a speaker of the language.

Dan: I think the best way to think of it is “how much you can do with the language” when speaking. People with high proficiency can handle a variety of simple and complicated tasks and situations without much difficulty in getting things done.

This project has been incredibly valuable for Babbel’s product and language teams. The results enable us to track participants’ app usage to an external measure of oral proficiency, going beyond anecdotal reports from our learners.  Generally speaking, is it true that speaking skills are both more difficult to develop as well as to assess than reading, listening and writing? If so, why?

Shawn: Speaking and listening skills are more difficult to develop because they are used in real time, and learners are not able to take time to think about the language that they need to use. It needs to be more or less automatic. With reading and writing, learners typically can take additional time to think about the language they want to use, or are reading. This extra time allows learners to think about the rules they know or any other explicit information they have about the language.

Dan: Speaking is definitely difficult to assess. One reason is practical—recording audio to be evaluated later or having a trained professional available to immediately evaluate each speaking test is considerably more resource-intensive than assessing the other skills. Another challenge with speaking is capturing the interactive side of it. Most of the time that we speak, we are speaking to people who are going to respond and co-construct a conversation with us. That’s hard to capture in a testing situation. Compare this to reading, for instance, where reading a short article by yourself is a pretty good representation of one of the most common ways we actually use our reading skills in the real world.

On average, study participants used Babbel for approximately 9.6 hours over approximately 12 weeks. However, some participants who studied less than that managed to improve their speaking, grammar and vocab scores, while others who studied more didn’t improve across all three. Yet, it’s clear that Babbel study time predicted vocab, grammar and speaking score gains. What might account for this variation?

Dan: The simple explanation is that people are just different – they all bring different skills, experiences and attitudes to the task of language learning. People have different memory capacities, for example – some people remember things more easily than others. Another way that people differ is in their listening abilities. Learning new speech sounds, for adults especially, is a tricky task, but some people just tend to have an easier time of it than others.

The other important thing to remember was that many of the participants had some previous experience with Spanish. Spanish is the most commonly-taught foreign language in the U.S., where it’s also the most commonly spoken non-English language. So participants with more previous experience might have been able to re-awaken or reinforce some long-dormant Spanish skills by completing Babbel lessons and use that to make more impressive learning gains.

One of the other interesting findings from the study was how influential participants’ self-reported motivation levels appear to be on their learning outcomes. Generally speaking, what can we say about the impact of motivation on learners’ progress or app usage?

Shawn: Motivation has been found to be an important ingredient for successful language learning, and I think it is even more important for learners who are using an app for self-study. Learners need to be intrinsically motivated in order to stick with it because they don’t have a teacher or class forcing them to study.

Dan: Motivation has pretty consistently been found to be one of the biggest influences on language learning achievement. So this really isn’t surprising, but good to know that it also applies in app-based language learning. One big part of language learning is just sticking with it, and being motivated can lead to better language learning habits and discipline.

Shawn: For me, one of the biggest surprises about this study was that a language learning app could help learners develop some oral proficiency (speaking ability). Maybe I wasn’t as skeptical as some researchers, but I was still somewhat doubtful that Babbel, or any other app, could help learners develop their speaking ability. That’s why it was important to me to have a recognized, high-quality assessment of learners’ listening and speaking abilities, namely the Oral Proficiency Interview. I assumed that learners would make progress on grammar knowledge and vocabulary, but that type of explicit knowledge isn’t always so useful for speaking and listening because learners don’t have time to think about rules or probe their memories for vocabulary items.

Thank you both for taking the time to answer my questions.