The History and Persister services can be considered as "siblings", History has the role of retrieving the historical data that was recorded in MongoDB, but in order for these data to reach MongoDB it was necessary for Persister to take them there.
To better understand the scenario where these two services are employed, let's exemplify the data flow:
- The devices send their measurement data to the dojot platform through some protocol, for example, MQTT;
- The data reaches a Broker* designed to work with the same protocol as the device;
- After receiving the data, the Broker forwards it to Kafka so that the interested services can consume it;
- The Persister service comes on the scene, which is notified by Kafka when there is device data to be persisted in MongoDB to create a historical basis of measurements;
- Once Persister has saved the data to MongoDB, the History service is able to see it;
- History service provides an endpoint (REST API) so that platform users can consult the historical data of measurements made by the devices, so through the History it is possible to retrieve any historical data.
(*) A Broker is a component that allows a group of devices to send their data and be managed by it, using their own native protocols.
- History and Persister services
- Table of Contents
- History service
- Persister service
- Tests
- Dependencies
- Documentation
- Issues and help
The History service is used when it is necessary to obtain historical data from devices. Through its REST interface it is possible to apply filters to obtain only the desired data from a device, like the attributes to be returned, the time period in which the data was measured, the quantity and order in which the data was returned.
See the skeleton of the request that can be made to History service:
http://{host}:{port}/device/{device_id}/history?attr={attr}&dateFrom={dateFrom}&dateTo={dateTo}&lastN|firstN={lastN|firstN}The settings are made through environment variables, so we have the following:
| Environment variable | Purpose | Default Value |
|---|---|---|
| AUTH_URL | Auth url address | "http://auth:5000" |
| DEVICE_MANAGER_URL | Device Manager url address | "http://device-manager:5000" |
| HISTORY_DB_ADDRESS | History database's address | "mongodb" |
| HISTORY_DB_PORT | History database's port | 27017 |
| HISTORY_DB_REPLICA_SET | History database's replica set address | None |
| LOG_LEVEL | Sets the log level | "INFO" |
The service is written in Python 3.6 and to run it in standalone mode you need to install the libraries that the service depends on. A very common problem is when we need to use different versions of the same library in different Python projects. This can lead to conflicts between versions and a lot of headaches for the developer. To solve this problem, the most correct thing is to create a virtual environment for each project. Basically, a virtual environment packages and stores in a specific directory all the dependencies that a project needs, making sure that no packages are installed directly on the operating system. Therefore, each project can have its own environment and, consequently, its libraries in specific versions.
There are several tools that create Python virtual environments, one of the most well-known is virtualenv. This is what we are going to use in this example.
We will use pip (Package Installer for Python) to install virtualenv, in this case, we will use pip in version 19.0.3.
Make sure to install the pip according to your operating system, then install virtualenv. That done, we will be able to create and manage our virtual environment to run the service.
First let's create a new virtual environment using the following command:
# The most common is to create virtualenv at the
# root of the project that it will belong to.
$> virtualenv ./venvAfter creating a virtual environment, we need to activate it so that we can install the necessary project packages. For this, we use the following command:
$> source ./venv/bin/activateNOTE THAT to exit the virtual environment we use the deactivate command,
as we can see below:
# When you no longer want to run the History service,
# you need to leave the virtual environment created for it:
$> deactivate ./venvTo install all dependencies, execute the following commands.
# Some libraries depend on the specific version of the pip:
$> python -m pip install pip==19.0.3
$> pip install versiontools
$> pip install -r ./requirements/requirements.txtThe next and last command will generate a
.egg file and install
it into your virtual environment:
$> python setup.py installAnother alternative is to use Docker to run the service. To build the container, from the repository's root:
# you may need sudo on your machine:
# https://docs.docker.com/engine/installation/linux/linux-postinstall/
$> docker build -t <tag-for-history-docker> -f docker/history.docker .NOTE THAT you can use the official images provided by dojot in its DockerHub page.
Beforehand, you need an already running dojot instance in your machine. Check out the dojot documentation for more information on installation methods.
To run the History service on standalone mode, just set all needed environment variables and execute:
# from the repository's root:
$> ./docker/falcon-docker-entrypoint.sh startTo run the service on docker, you can use:
# may need sudo to run
$> docker run -i -t <tag-for-history-docker>The History service (as we saw earlier) is only capable of retrieving data from the MongoDB instance, it is not able to store any data. It is the Persister's duty to make this happen: it is notified whenever a new message has arrived in Kafka and stores it in MongoDB.
The settings are made through environment variables, so we have the following:
| Environment variable | Purpose | Default Value |
|---|---|---|
| AUTH_URL | Auth url address | "http://auth:5000" |
| DATA_BROKER_URL | Data Broker address | "http://data-broker" |
| DEVICE_MANAGER_URL | Device Manager address | "http://device-manager:5000" |
| LOG_LEVEL | Define minimum logging level | "INFO" |
| PERSISTER_PORT | Port to be used by persister sevice's endpoints | 8057 |
| Environment variable | Purpose | Default Value |
|---|---|---|
| DOJOT_SERVICE_MANAGEMENT | Global service to use when publishing dojot management events | "dojot-management" |
| DOJOT_SUBJECT_TENANCY | Global subject to use when publishing tenancy lifecycle events | "dojot.tenancy" |
| DOJOT_SUBJECT_DEVICES | Global subject to use when receiving device lifecycle events | "dojot.device-manager.device" |
| DOJOT_SUBJECT_DEVICE_DATA | Global subject to use when receiving data from devices | "device-data" |
| KAFKA_ADDRESS | Kafta address | "kafka" |
| KAFKA_PORT | Kafka port | 9092 |
| KAFKA_GROUP_ID | Group ID used when creating consumers | "history" |
| Environment variable | Purpose | Default Value |
|---|---|---|
| HISTORY_DB_ADDRESS | History database's address | "mongodb" |
| HISTORY_DB_PORT | History database's port | "27017" |
| HISTORY_DB_REPLICA_SET | History database's replica set address | None |
| HISTORY_DB_DATA_EXPIRATION | Time (in seconds) that the data must be kept in the database | "604800" (7 days) |
| MONGO_SHARD | Activate the use of sharding or not | False |
The Persister service is installed in exactly the same way as the History service, just note that what changes is the virtual environment to be created, in this case, the directory name of the virtual environment must refer to the Persister service. If you want, you can also use the same virtual environment as History service, no problem, but be aware of that.
Like the standalone mode, the Persister on Docker is identical to the History service's one, paying attention only to the name of the docker image to be generated. In this case, it must refer to the Persister service.
NOTE THAT you can use the official images provided by dojot in its DockerHub page.
Beforehand, you need an already running dojot instance in your machine. Check out the dojot documentation for more information on installation methods.
To run the Persister, after setting the environment variables, execute:
$> python -m history.subscriber.persisterHistory has some automated test scripts. We use [Dredd] (http://dredd.org/en/latest/) to execute them:
# you may need sudo for this:
$> npm install -g [email protected] --no-optionalAlso install the python test dependencies:
$> pip install -r ./tests/requirements.txt
$> pip install codecovRun Dredd automated test scripts:
$> ./tests/start-test.shRun the pytest module to run the coverage tests:
$> python -m pytest --cov-report=html --cov=history tests/The service dependencies are listed in the next topics.
- Dojot Services: They are dojot services;
- Others Services: They are external services;
- Kafka (tested using Kafka version 2.12);
- MongoDB (tested using MongoDB version 3.2);
Check the documentation for more information:
- Latest History API documentation
- Development History API documentation
- Latest dojot platform documentation
The API documentation for this service is written as API blueprint. To generate a simple web page from it, you must execute the commands below:
# you may need sudo for this:
$> npm install -g aglio
# static webpage
$> aglio -i ./docs/history.apib -o ./docs/history.html
# serve apis locally
$> aglio -i ./docs/history.apib -sIf you found a problem or need help, leave an issue in the main dojot repository and we will help you!