To build a deep learning model that can generalize well to new environments (novel data distributions), we need to ensure that two things happen. First, the model should be complex enough to capture the characteristics of every new environment. And second, the training data should be a diverse dataset representing the properties of different data distributions.
However, in some applications, we cannot meet these two conditions because of some constraints. For instance, some memory limitations apply when working with mobile and edge devices. Moreover, labeling data for every new environment imposes data annotation costs. In such cases, we need an annotation-free approach to train a lightweight model for every novel data distribution. Here is where Adaptive Learning comes to play.
How it works
Adaptive learning is a teacher-student method that trains a student network for each new environment based on the labels generated by a teacher model. This model builds a student model that adapts to new environments.
In this method, the teacher model is a deep neural network trained on a rich dataset and can generalize well to new environments. We feed the data gathered from the new domain to the teacher model to get the predicted labels. Then, we use these labels to train a student model. Finally, we can use the lightweight student model for inference on edge devices.
Adaptive learning can increase model performance significantly. By applying adaptive learning to the Smart Social Distancing application as a use case, we have achieved a 72.20 mAP score compared to a 25.02 baseline. Since this approach uses automatically generated labels, it cuts data annotation costs. Also, adaptive learning builds a lightweight model that is deployable on edge devices.
You can now try adaptive learning on several platforms, such as:
- X86 CPU
- X86 GPU
Support for a variety of edge devices is coming soon.
We faced several challenges, such as catastrophic forgetting and some network instabilities when we wanted to apply adaptive learning to the Smart Social Distancing application. You can read the adaptive learning article by Neuralet to learn more about how adaptive learning works and see how we solved the challenges along the way. The source code is available on our GitHub repository.