mLearning: its future and challenges

Mobile learning, or mLearning, means learning on a portable electronic device, such as a smartphone or tablet. Its main advantages include easy access anytime and anywhere, as well as the use of content developed through microlearning.

At the Google Developers Summit 2019 conference, Jingtao Wang, a Research Scientist at Google, presents the future and challenges of mobile learning. He shares smartwatch case studies and how artificial intelligence can monitor their use, along with physiological studies of how students learn.

Jingtao Wang is a research scientist at Google whose research areas include intelligent mobile interfaces, machine learning, and educational technology. Wang has received several awards, including Microsoft Azure for Research, the Google Faculty Research Award, and ACIE Innovation in Education.

Watch the video

Video transcript

You can turn on captions directly on YouTube under Settings > Subtitles/CC, or read the transcript below.

Hello, everyone!

My name is Jingtao Wang, and I am a researcher at Google. Before joining Google full time, I was a researcher at the University of Pittsburgh.

I have two main research interests. The first is machine learning on electronic devices. The second is applying machine learning in education and human learning. Today, I would like to share some of my ongoing projects with you.

I am working on projects that connect these two interests.

As we all know, human learning is both challenging and rewarding. Imagine the opportunities that open up when someone learns a new language, learns to play a new musical instrument, or learns how to use emerging technologies such as TensorFlow.

Meanwhile, worldwide, learning remains as challenging as it is important. Surveys show that more than 10 million people are registered for MOOCs such as Coursera, but fewer than 7% complete their courses. That means learning something new is neither trivial nor easy.

At the same time, people need to learn not only in classrooms but also in other, informal environments.

Today, I would like to show you some examples of how to use machine learning with a digital tool you wear on your wrist, hold in your palm, and use while walking: the smartwatch.

SmartRSVP

The first project I will show you is called SmartRSVP. The idea is how to turn your smartwatch into an intelligent learning device.

The first challenge was its small screen. A typical smartwatch has a 1.5-inch screen and displays only two or three words at a time, barely enough to read a sentence.

So how can we enable effective learning?

It depends on the task. What if we take small samples and break the task into bite-sized pieces? Learning delivered to a watch on your wrist can provide timely reminders, prompts, and checks to see whether you understand the content displayed.

Another question we want to address is: how can we read long sentences on such a small watch?

The solution is a technology called RSVP. Its idea is to display one word at a time, in sequence. Each word can be larger, the display can move faster, and you can read without moving your gaze.

Here we introduce two technologies that make this kind of rapid reading easier on smartwatches. First, we detect your attention: we display the next word only when you are looking at the screen. In other words, we pause when you look away.

Second, we monitor the user's heart rate in real time to understand their attention and whether it increases during a faster-paced test.

Interaction in large classrooms

The other project I will share is about improving interaction in large classrooms.

These are quite common in undergraduate education; classes with more than 200 students are not unusual. How can we improve interaction in that setting? An intelligent mobile application called Kazmir offers three important features to meet this need.

The first feature automatically sends a reminder after each lecture, asking students about the material, presenting questions about the content, or asking whether they have doubts they would like to explore or clarify with their instructor. This also naturally encourages dialogue, allowing deeper, more specific questions.

This system has been implemented in classrooms in the United States and abroad. More than 400 students have used the pilot system I have been working on for over five years.

For MOOCs—massive open online courses—learning more efficiently on mobile devices has two components:

Using a camera to track learners' facial expressions, and using the rear camera to track physiological signals. This is difficult because of the variety of students taking these courses. We need to understand what students learn while watching course videos, paying particular attention to the moments when they become distracted.

To do this, we use adaptive intervention techniques to help students remain attentive and calm while watching the lesson.

Living Jiagu: learning about the creation and evaluation of language with artificial intelligence

Now I will introduce the project I am most excited about: language learning with artificial intelligence. This project explores the intersection of ancient pictorial art and artificial intelligence.

In this image, we see a script called “oracle bone” on a turtle shell. It is the origin of Chinese writing and was used by people 3,000 years ago.

This distinctive script preceded most other Asian writing systems. When it died out, a new language appeared: today's Chinese, whose origins and evolution can be traced to this shell.

Understanding the origins of a language can help us understand languages more deeply. An interactive exhibition can take the experience further, allowing people learning a new language to do so through everyday objects.

Here are some technical details

We use a multilayer neural network to connect the character a person creates with the image shown on the screen. At the same time, we use an algorithm to determine, for example, whether each part of the character has the correct meaning. It is then animated, seeking to connect each object with the meaning represented in the Asian language.

This project launched at Google in China in 2019. More than 1,000 participants experienced the demonstration, which included two parts, the first designed for large touchscreens.

One task involves imitating the Chinese ancestors who created pictographs. Creating characters that help animate and compose scenes on a screen lets us build a connection between everyday objects and their corresponding meanings as we learn.

Machine learning can support learning and knowledge more efficiently and engagingly, because learning does not necessarily happen in classrooms. It can also happen on your wrist or in the palm of your hand.

Thank you!