Neuralet > Neuralet Adaptive Learning Service Tutorial

    Neuralet Adaptive Learning Service Tutorial

    Neuralet Adaptive Learning Edge vision

    Adaptive Learning; An Automatic Model-Personalization Tool

    As explained in our “Adaptive Learning” article, adaptive learning is a technique that brings about personalized and efficient deep neural networks with accurate outputs that can adapt to new environments and fresh (formerly unseen) data sets without any extra manual data annotation.

    For instance, if you need to deploy an object detection model for your company’s production line, danger zones, etc., you can use a generalizable heavy model with a highly accurate performance in different conditions and environments or employ a lightweight model with real-time performance that is not as generalizable and accurate. In both scenarios, there are some constraints and benefits.

    If you choose the first option – the heavy model, you will have a pretty generalized network with high accuracy at the cost of computational complexity and low speed. You will also not be able to deploy your model on edge devices that have major resource limitations. 

    In the second option, by employing a lightweight network, you will have a model with higher speed and relatively much lower computational costs, which can be deployed on edge devices, such as Google Edge TPU or Nvidia’s Jetson family of devices for real-time applications. However, the results are less accurate, and can-not be generalized on the unseen data from a different environment.

    Traditionally, if someone wanted accuracy and speed, they had to gather and annotate data from scratch for each new environment and then use this dataset to train a lightweight model specifically for that environment. However, the data annotation is not an automatic process and is done by humans, making it a highly time-consuming process. The Adaptive Learning method solves this problem by using two kinds of networks: Teacher and Student. We train a lightweight student network for each new environment using the labels generated by a complex network (teacher) trained on rich datasets that can generalize well to new environments. This method significantly increases the student model accuracy. In conclusion, we can infer that adaptive learning is a favorable solution to the stated problem, leaning on many reasons. Such as; No necessity for data annotation, Efficient specialized models (deployable on edge devices), and Accessibility; since its lightweight model runs on any edge device that is accessible everywhere. 

    Neuralet Adaptive Learning API

    Neuralet Adaptive Learning Service is designed to customize the object detection models to datasets and environments as provided by customers. Considering user constraints on running our previous adaptive learning module, we decided to provide a service that eliminates the complexities and automatically carries out the configurations. In this service, we resolved resource limitations by providing the training machines, so users do not need any further local training resources such as GPUs. This means that the user can send a video, specify requirements, and request model adaption to the dataset. After the training process, the user can download the trained student model.

    Alongside the Adaptive Learning API, we have the Neuralet Edge Object Detection, which is an open-source module containing supplementary codes created to take actions on your trained model. We have provided the required scripts for exporting your models or run inferences on various edge devices. Moreover, in this module, we have created a client code that is, in fact, an interface for users to connect with the provided Adaptive Learning API. Using this client code written in Python, you can call the API’s endpoints, such as uploading a video, requesting status, or downloading the model by running a few simple commands.

    Adaptive Deep Learning by Neuralet
    Figure1. Neuralet Adaptive Learning Framework

    How to run my Adaptive Learning Object Detection model

    There are two methods to call different endpoints of Adaptive Learning API. The first uses the curl’s command-line tool in which the user should type raw commands, and the other is to use our simple Python Client interface.

    To use our client interface, you need to have python on your machine, clone our Github repository to your system, and go to the provided directory using the commands below.

    git clone https: https://github.com/neuralet/edge-object-detection.git
    cd edge-object-detection/services/adaptive-learning/client

    Either you prefer to use curl or our interface. Below is a step-by-step guide to run Adaptive Learning using each of the methods.

    Neuralet Adaptive Learning Pipeline
    Figure2. Neuralet Adaptive Learning Edge vision Pipeline

    Step 0: Signing up in Neuralet Adaptive Learning API

    In the beginning, you need to go to our API home page and sign up using your email address and password. By clicking on the sign-up button, a verification link will be sent to your email address. After verifying your account, you can sign in to Neuralet’s Adaptive Edge Vision API. Now you are able to get your token using the provided section and keep it for the next steps.

    Step 1: Preparing the Video File

    In the first step, you upload the video file. Since our API only accepts zip files as the input format, you need to create a compressed zip file from your video file, preferably using the name of input.zip.

    zip -j input.zip PATH_TO_Your_VIDEO_FILE

    Step 2: Uploading Video

    After creating the input.zip file, you upload it to Neuralet’s Servers. Subsequently, you will receive a unique id (UUID), which will be required later in the config file step. You can upload your file using one of the following methods:

    Client Code:

    If you are using the client code, you should first save your token (provided to you in step0) as a text file and type the path to this file instead of TOKEN_PATH in the python command below. The FILE_PATH is the address to the input.zip file that you have created in step1. After running the python command successfully, the unique id (UUID) will show up.

    python3 client.py --token TOKEN_PATH upload_file --file_path FILE_PATH

    Curl Commands:

    If you prefer to work with the curl, you need to follow these two stages to upload your video and get your unique id.

    • Get Upload URL:

    In this part, you should only copy your token (provided to you in step0) and paste it instead of TOKEN in the command. 

    curl -X GET "https://api.neuralet.io/api/v1/file/upload/" -H  "accept: application/json" -H "Authorization: Bearer TOKEN"

    After running this command, it will return a json containing two items. The first one is name, which has your unique id (UUID), and you should keep it for the config step. The second one is upload_link that you should use as UploadURL in the next part.

    • Upload File:

    In this stage, you must put the UploadLink you have copied in the last part in place of UploadURL. For the FILE_PATH, you need to input the path to the input.zip file you have created in the first step.

    curl "UploadURL" --upload-file FILE_PATH

    Step 3: Configure Your Training

    In order to start the adaptive learning process on your file, you should tune and modify the sample_config.json file presented in our repository.

    There are two mandatory fields in the sample_config file, which you are required to set. First, you must copy-paste your unique id (UUID provided to you in the previous step) in front of the UploadUUID field. The second field is VideoFile, in which you should put your video file’s name against it. (Please pay attention that this is the name of your original video, e.g., softbio.mp4)

    UploadUUID = Your_Unique_ID
    VideoFile = Your_video_File_Name.mp4

    In addition, there are a few more fields presented in the sample config file that you can modify based on your requirements. For example, in the config file’s Classes field, you can choose between 90 different Object Categories of COCO’s dataset by writing your desired classes’ names with a comma-separated format to train your model. Notice that the default value ("coco") will train all of the 90 object categories. You can find the 90 classes of COCO’s dataset in their original research paper. Furthermore, it is possible to change the QuantizedModel value for the Student network.

    To do this, you need to adjust the sample config file on the configs/ directory. Thus, we have prepared a brief explanation for each of the config files’ parameters and options in the following table. You can also use the sample config file in configs/sample_config.json

    ParameterOptionsComments
    Teacher | UploadUUID
    UUIDUnique id of uploaded input.zip file.
    Teacher | VideoFile
    StringName of the video you have zipped and uploaded.
    Teacher | ClassesComma-Separated strings without spaceA list of class names that you want to train your model on. These classes should be a subset of COCO classes. You can find the COCO’s category names in their original paper. To train on all of the 90 COCO classes, just put "coco".
    Student | QuantizedModeltrue or falseThis parameter indicates whether to train the student model with a quantization-aware strategy or not. This is especially useful when you want to deploy the final model on an edge device that only supports Int8 precision like Edge TPU. By applying quantization-aware training, the App will export a tflite too.

    Step 4: Start a Training Job

    In this step, you can request to train your adaptive learning model by running the command using either the Client code or Curl. Then, you will get a Job id that you should save for monitoring your training status in the next steps.

    Client Code:

    Same as the second step, you need to input the path to your token text file instead of TOKEN_PATH and the address of your config file in the config_path field.

    python3 client.py --token TOKEN_PATH train --config_path CONFIG_PATH

    Curl Commands:

    Again, similar to the second step, you should copy-paste the token we have provided to you at the beginning instead of TOKEN. Additionally, you must give the path to your config file in the JSON_CONFIGFILE_PATH field.

    curl -X POST "https://api.neuralet.io/api/v1/model/train/" -H "accept: application/json" -H "Content-Type: application/json" -H "Authorization: Bearer TOKEN" -d @JSON_CONFIGFILE_PATH

    Step 5: Get Job Status

    At this moment, your model is training on the Neuralet’s servers, which may take from a few hours to a couple of days to finish depending on the video length. If you want to know your model’s status use this command.

    In this stage, you can request a job status using the Job id generated in the last step to observe the operation progress.

    Client Code:

    Enter the address to your token text file and your Job id, respectively, in the provided TOKEN_PATH and JOBID fields of the command and run it.

    python3 client.py --token TOKEN_PATH get_status --job_id JOBID

    Curl Commands:

    You only need to repeat the previous step and copy-paste your token in the TOKEN field, and input your job id in the given field for JOB_ID.

    curl -X POST "https://api.neuralet.io/api/v1/model/status/" -H  "accept: application/json" -H  "Content-Type: application/json" -H "Authorization: Bearer TOKEN" -d "{\"job_id\":\"JOB_ID\"}"

    By running the command and sending your request to our API, you may get one of the following messages for either the Teacher or Student models each time you request for the status:

    • Allocating Resource: We are Allocating Resources (e.g., a computing machine) to your job.
    • Building: We have allocated the resources, and the program is Building an environment (installing the required packages) to start your job.
    • Training: The Training process has started. An Adaptive Learning Job is Running.
    • Wrapping Up: Your training is about to finish and is Saving data and completing the job.
    • Finished: The job has been finished successfully.
    • Failed: If the process faces an infrastructural or hardware problem such as Neuralet’s server failure, you will see this message.
    • Not Reached Yet: It usually appears as the student model’s status, which means the job’s workflow has not reached the student model’s training phase yet (i.e., while the teacher model is running, the student model’s status will be Not Reached Yet).
    • Unexpected Error: An internal error has occurred.

    Step 6: Download your model

    You can download your Adaptive Learning’s trained student model by running one of the following commands based on your preference. You will receive a file named output.zip that we will explain the contents in the next section.

    Client Code:

    You should insert the address to your token file in the  TOKEN_PATH field and replace your job id with JOBID, just as in step five.

    python3 client.py --token TOKEN_PATH download_file --job_id JOBID

    Curl Commands:

    If you are using the curl, there are two stages to finally get your output file:

    • Get your upload link:

    As in step five, you need to copy-paste the token and job id you saved in previous steps in the TOKEN and JOB_ID fields. Running this command will return an upload_link which you need in the next part.

    curl -X POST "https://api.neuralet.io/api/v1/file/download/" -H "accept: application/json" -H "Authorization: Bearer TOKEN" -H "Content-Type: application/json" -d "{\"job_id\":\"JOB_ID\"}"
    • Download your file:

    Now by putting the upload_link that you have received in the previous step against the provided field and running the command, your output file’s download process will start.

    wget "upload_link" -O output.zip

    What does the output.zip file contain?

    After extracting the output.zip file in your computer, you will see the main directory of this zip file, named train_outputs, which contains all of the Adaptive Learning files and directories. Here we will walk through the files and directories inside the train_ouputs and present a brief explanation of their contents.

    First, we are going to introduce the most important files inside the train_ouputs:

    validation_vid.mp4 :

    This is a video with a maximum length of 40 seconds, which compares the results of running an SSD-MobileNet-V2 model trained on COCO (Baseline model) and the Adaptive Learning trained model (Student) on a validation set video (Not used in the training process). 

    label_map.pbtxt :

    This pbtxt file contains a series of mappings that connects a set of class IDs with the corresponding class names. To run the inference module of our Edge Object Detection Repo, you should pass this file to the script to classify each object with the right name.

    events.out.tfevents :

    If you want to monitor and analyze your training process, you can open this file using TensorBoard and observe each step of the Adaptive Learning model training process.

     

    So far, we have introduced the most important files in the train_outputs directory. Now we are going to explain the contents of the train_outputs/frozen_graph directory. 

    train_outputs/frozen_graph :

    This is the main directory of our trained model, which contains the required files for inferencing and exporting to the edge devices.

    train_outputs/frozen_graph/frozen_inference_graph.pb :

    To run your model on Jetson, you should pass this file to the export module that we have built for edge object detection. This will export and create a TensorRT engine for you.

    train_outputs/frozen_graph/detect.tflite : 

    If you set your QuantizedModel as true in the config file, this file would be available to you inside the frozen_graph directory.

    The importance of this file is for exporting your model to the EdgeTPU. In this case, our EdgeTPU exporter accepts this detect.tflite file as an input to create an EdgeTpu compiled tflite file.

    train_outputs/frozen_graph/saved_model :

    The frozen_graph/saved_model contains a TensorFlow saved-model for inferencing on X86s.

    Conclusion

    This article introduced the Neuralet Adaptive Learning API and its features. Using this service, you can customize your object detection tasks for any specific environment by implementing the teacher-student configuration without any training resources or strong technical knowledge. Below, there is an example video of the results on an object detector using an adaptive learning method (in the bottom) compared to baseline (in the top).

    Do not forget to share your opinions and suggestions with us in the comment section below or through the hello [at] neuralet [dot] com.

    We would be happy to see your results on our Adaptive Learning model.

    Leave a Reply

    Your email address will not be published.

    Get In Touch

    Have a question? Send us a message and we will respond as soon as possible.