This will serve as the documentation for the REST API and the algorithm part of the Salamander Identification Application system, developed during the Spring of 2021 as a Bachelor's project proposed by the Norwegian Institute of Nature Research (NINA). The application is meant to aid the researchers at NINA in estimating the populations of Northern crested newts and Smooth newts.
### Developers:
* Anders Sundstedt Langlie
* Andrea Magnussen
* Eirik Martin Danielsen
* Herman Andersen Dyrkorn
## Devlopment Environment
To be able to run the REST API combined with the algorithm in a development environment, several steps are required.
### Setup
1. A python development enviorment with a python 3.7 interpreter needs to configured. (we used PyCharm, python 3.7 and pip).
2. Clone the project where you want the project to be located on your computer: ```git clone <link>```
3. Run ```pip install -r requirements.txt``` in the projects root folder to install all required libraries.
4. Place the dlc_model inside the algorithm root folder.
5. Edit the project_path in the config.yaml file inside the dlc_model to the path where it is located.
6. To run the development server, run the file called "run.py" in the projects root folder.
7. The default port the development server will run on is port 5000. The system will be exposted to the network the application is running on by default. To access it from another network, the router need to have port forwarding.
## System
In this section, we will go through the framework and libraries used, as well as how the backend/server works and what it consists of.
The server is currently a thick server, and does most of the processing. The image below shows a visualization of the system architecture.
The backend is written in the programming language Python. The main library we used for developing the REST API is Flask. The main library we used for developing the algorithm is DeepLabCut combined with OpenCV. We used several additional libraries to develop the backend.
#### Flask
The documentation for Flask can be found [**here**](https://flask.palletsprojects.com/en/2.0.x/).
#### DeepLabCut
The documentation for DeepLabCut can be found [**here**](https://github.com/DeepLabCut/DeepLabCut).
#### OpenCV
The documentation for OpenCV can be found [**here**](https://docs.opencv.org/master/).
#### Libraries
***Flask-RESTful**: Flask-RESTful is an extension for Flask that adds support for quickly building REST APIs. Documentation can be found [**here**](https://flask-restful.readthedocs.io/en/latest/).
***Flask-Bcrypt**: Flask-Bcrypt is a Flask extension that provides bcrypt hashing utilities. We used it for hashing and salting passwords and compare the hash to a string. Documentation can be found [**here**](https://flask-bcrypt.readthedocs.io/en/latest/).
***Flask-JWT-Extended**: We used this library for user authentication. We used create_access_token() to make JSON Web Tokens, jwt_required() to protect routes, and get_jwt_identity() to get the identity of a JWT in a protected route. Documentation can be found [**here**](https://flask-jwt-extended.readthedocs.io/en/stable/).
***Flask-Limiter**: Flask-Limiter provides rate limiting features to flask routes. We have set different limits to endpoint. This is to prevent DOS-attacks. Documentation can be found [**here**](https://flask-limiter.readthedocs.io/en/stable/).
***Flask-SQLAlchemy**: Flask-SQLAlchemy is an extension for Flask that adds support for SQLAlchemy to your application. It aims to simplify using SQLAlchemy with Flask by providing useful defaults and extra helpers that make it easier to accomplish common tasks. The documentation can be found [**here**](https://flask-sqlalchemy.palletsprojects.com/en/2.x/).
***DeepLabCut**: Used for predicting six points on a salamander image. Four points down the spine, and two points for the front shoulders. The documentation can be found [**here**](https://github.com/DeepLabCut/DeepLabCut).
***OpenCV**: Used to brute force match processed images of salamander abdomen patterns. The documentation can be found [**here**](https://docs.opencv.org/master/).
***Image-Encoder**: Used to encode images to base64 strings. The documentation can be found [**here**](https://pypi.org/project/image-encoder/).
### Modules
We divided the backend into two modules; REST API and algorithm. They are both located in the root folder of the project. The "api" folder contains all the code for the REST API and the "algorithm" folder contains all the code for the algorithm.
#### REST API
The REST API was implementet so that the mobile application could send requests to the server that would store data, process images and match them.
##### Endpoints
The API consists of 11 different endpoint with 21 different ways of accessing them. Some of the endpoints only has the http methods GET or POST, and some of them has GET, POST, PUT and DELETE.
The complete API documentation can be found [**here**](https://documenter.getpostman.com/view/8860712/TzJvcbk9).
All endpoints are registered in the "_\_init.py__" file. All the classes are imported and registered via the add_resource() function.
##### Database and Image Storage
For storing metadata about users, locations and salamanders, we use a local sqlite database located in the api/database folder. The image below displays how the different tables are connected.
To open the database, CD into the database folder use the command ```sqlite3```. Then use the command: ```.open database.db```. When the database is opened, you may start to type queries.
For storing the original and processed images of salamanders, we use a file hierarchy structure. The root of this file structure is called "images". The image below displays how this file structure looks like.
<palign="center">
<img src="documentation/stored_images.png"
alt="salamander" width="300"/>
</p>
When a user uploads an image it will be temporarily be stored in a folder called "temp_images". The original image the user uploaded will be named id.mime (.jpg or .png) and the processed image will be named id_str.jpg.
#### Algorithm
##### Labeling and Training
Use DeeplabCut's GUI tool to label the images. We recommend using the anaconda environments provided by DeepLabCut as they contain all necessary dependencies. An explanation of how to label the images is shown below. If you wish to use the previously labeled images your DeepLabCut has to be created with Project name ```Salamander-Abdomen``` and experimenter name ```Experiment```. This is important as the files with your labels/annotations will be named depending on these factors. After all images have been labeled a training set can be generated. When training a model you can continue using the GUI version of DeepLabCut as long as you have an Nvidia GPU with Cuda 10 or lower compatibility. If you have an Nvidia GPU which requires Cuda 11 you will have to use DeepLabCut-core as it supports Cuda 11 and Tensorflow 2.x. We used DeepLabCut-core both for training and for estimating images inside our working application. The command ```pip install git+https://github.com/DeepLabCut/DeepLabCut-core``` will install DeepLabCut-core. As this library gets updated frequently we recommend looking at the newest documentation for installation and usage.
[Legg inn bilde av lablet salamander!!!]
In the above image, a labeled salamander is shown. It is important to define the same body parts as in our project. Inside the ```config.yaml``` file in our project the body part definitions are found. Feel free to copy these to your own project. Bodyparts 1 through 4 represent the spine of the salamander and begin at the chest down to the pelvis. Make sure to label the chest first, and move down to the pelvis! The last two points represent the shoulders where the left and right points represent the salamander's left and right shoulder.
##### Models
DeepLabCut uses resnet for their models. You can choose between using resnet50, resnet101 and resnet152 for model generation. We used resnet152 as we got better results, but feel free to change the network depth when training future models.
##### Process
When a user uploads an image of a salamander, DeepLabCut will start to process it. If the predicted points are good, there will be a line interpolated between the points to straighten the pattern and crop the image. The two shoulder points is used to detremine the width of the pattern. The pattern is then returned back to the user for aproval and stored in the temp_images folder on the server.
The figure above shows a visualization of the process. This is a succesful straightening and returns a good pattern, which should be accepted by the user. Below is an example of a failed straightening, which should be cancelled by the user. It is important that this pattern should not be matched against the other salamanders in the system, as this will most definitly create a false negative, in other words, a new salamander entry. This will disturb the estimation of the population.
If the user approved the image, the matching process starts. The image will only be matched against other salamanders located in the same location with the same species and sex. If a match is registered, the images from the temp_folder is moved into the correct salamander id folder inside the location. If there was no match, a new folder will be generated with the new id of the salamander and it will be placed there.
