11. How it works ?ΒΆ

Circus is a Python program in order to monitor and control processes and sockets.

Redis is an in-memory data structure store, used as a database, cache and message broker.

Telegraf is a plugin-driven server agent for collecting and sending metrics and events from databases, systems, and IoT sensors.

directory_observer is a Metwork tool that allows you to monitor activity on various directories and push the corresponding events to a Redis queue (a list). For example, you may create a file inside one of your monitored directories, the creation event will be pushed to the Redis queue.

conf_monitor is a Metwork tool in order to monitor the configuration files.

jsonlog2elasticsearch is a daemon to send json logs read from a log file to elasticsearch.

mflog2mfadmin (based on jsonlog2elasticsearch) on is a daemon to send mflog logs to elasticsearch.

MFDATA Overall architecture

Circus acts as a process watcher and runner. You may check the full circus.ini configuration file in the /home/mfdata/tmp/config_auto/ directory. Check the Circus architecture

The directory_observer.ini configuration file defines, among others, the following keys for each section (one section per plugin step):

  • active : whether or not a directory should be scanned

  • directory : which directory is monitored

  • queue : the name of the Redis queue where the events will be pushed.

You may check the full directory_observer.ini configuration file in the /home/mfdata/tmp/config_auto/ directory.

directory_observer scans the directories configured in the directory_observer.ini file.

step-xxx is the step defines in your plugin. There are as many step-xxx as plugins steps.

Once the monitoring of the directories is started, any action on the monitored directory is noticed and pushed to the Redis queue. The message contains:

  • the directory in which the event happened.

  • the file which brings about the event (the file which was created, moved or renamed).

  • the associated event.

  • the event timestamp

Depending on the event, the corresponding plugin step is executed. The event is popped from the Redis queue.

If the MFDATA plugin is configured for monitoring, the metrics are send via Telegraf to the InfluxDB database on the MFADMIN server.

See also

MFADMIN Documentation
mfadmin:mfadmin_monitoring_plugins
mfadmin:mfadmin_miscellaneous:Circus hooks