Enhancing STEM Education using Augmented Reality and Machine Learning

Abstract

Learning Science, Technology, Engineering andMathematics (STEM) in the 21st century has been evolved from the conventional textbook to the interactive platform using electronic devices. This paper presents the implementation of a mobile application system, named AUREL (Augmented Reality Learning) in enhancing the learning experience by projecting Augmented Reality (AR) objects onto 2D images. This AR visualization is used to improve the understanding of STEM subjects and increases the enthusiasm of students towards STEM subjects. In this implementation, Google’s Cloud Tensor Processing Units (TPUs) are used to train specific datasets alongside Cloud Vision API to detect a wide range of objects. MLKit for Firebase is used to host the custom TensorFlow Lite models for specific use cases for better accuracy. On the other hand, Google Cloud Platform (GCP) is used to harvest STEM data, manage STEM 3D information and data processing. Subsequently, the processed information will be displayed in AR in the mobile application using ARCore’s Sceneform SDK. The application of AUREL could be extended to all science subjects so that students can learn using an interactive platform.

Application Pipeline

Multistage Inferencing Approach

Experiments and Results

Reference

Conference paper is available here Journal extension is available here

If you find this work useful, please cite us:

@inproceedings{ang2019enhancing,
  title = {Enhancing STEM education using augmented reality and machine learning},
  author = {Ang, Ivan and Lim, King Hann},
  booktitle = {2019 7th International Conference on Smart Computing \& Communications (ICSCC)},
  pages = {1--5},
  year = {2019},
  organization = {IEEE},
}