Skip to content

Tuning, Monitoring and Dashboard

.. index:: tuning

1. Tuning

.. index:: number of processes

1.1 Number of processes

A plugin is able to run multiple processes simultaneously.

You may change the number of process allocated for each step of each plugin. In order to configure this number of processes, add or update the numprocesses parameter in the [step....] section of the config.ini plugin configuration file, e.g.: 

# Number of process allocate to the step. Default value is 1
numprocesses = 3
.. index:: limits, resource

1.2 Resource limits

You are able to control resource limits for each step of a plugin by setting the following parameters in the [step....] section of the config.ini plugin configuration file, e.g.:

# resource limit for each step process
# rlimit_as => maximum area (in bytes) of address space which may be taken by the process.
# rlimit_nofile => maximum number of open file descriptors for the current process.
# rlimit_stack => maximum size (in bytes) of the call stack for the current process.
#     This only affects the stack of the main thread in a multi-threaded process.
# rlimit_fsize =>  maximum size of a file which the process may create.
# (empty value means no limit)
rlimit_as = 1000000000
rlimit_nofile = 1000
rlimit_nproc = 100
rlimit_stack = 10000000
rlimit_fsize = 100000000

For further information about Linux resource limits, check Linux documentation.

1.3 Miscellaneous

You are able to act on others tuning parameters for each step of a plugin.

# The number of seconds to wait for a step to terminate gracefully
# before killing it. When stopping a process, we first send it a TERM signal.
# A step may catch this signal to perform clean up operations before exiting.
# If the worker is still active after graceful_timeout seconds, we send it a
# KILL signal. It is not possible to catch a KILL signal so the worker will stop.
# If you use the standard Acquisition framework to implement your step, the
# TERM signal is handled like this: "we don't process files anymore but we
# try to end with the current processed file before stopping". So the
# graceful_timeout must by greater than the maximum processing time of one file.
# Default value is 600 seconds
graceful_timeout = 600

# If set then the step will be restarted sometime after max_age seconds.
# Default value is 310 seconds
max_age = 310

# If max_age is set then the step will live between max_age and
# max_age + random(0, max_age_variance) seconds.
# This avoids restarting all processes for a step at once.
# Default value is 300 seconds
max_age_variance = 300

.. index:: monitoring, MFADMIN, dashboard

2. Monitoring and Dashboards

.. seealso:: | :doc:MFADMIN Documentation <mfadmin:index> | :doc:mfadmin:mfadmin_monitoring_plugins.

2.1 Monitor a plugin

Through Metwork MFADMIN module, you may monitor the MFDATA plugins: refer to the :doc:related MFADMIN topic <mfadmin:mfadmin_monitoring_plugins>.

.. index:: Grafana dashboards, Kibana dashboards

2.2 Dashboards

:index:Time-series dashboards are available from :ref:MFADMIN Grafana GUI Interface <mfadmin:mfadmin_monitoring_plugins:Grafana Time-series dashboards>.

:index:Logs dashboards, i.e. mflog logs and Nginx access logs are available from :ref:MFADMIN Kibana GUI Interface <mfadmin:mfadmin_monitoring_plugins:Kibana dashboards>.

You may also create your own dashboards: refer to the :ref:MFADMIN topic <mfadmin:mfadmin_miscellaneous:Create specific dashboards> documentation.

Check :ref:MFADMIN architecture <mfadmin:mfadmin_intro:How it works?> and :ref:MFADMIN tips <mfadmin:mfadmin_miscellaneous:Useful links and Tips> documentation.