Plugins guide

This document is about "MetWork plugins" which are available in mfserv, mfdata and mfbase MetWork modules (only).

1. Concepts

1.1 What is a plugin?

In MetWork framework, the plugin is the place where you put your code.

For example with mfserv, if you want to develop a new webservice or website, you develop it in a plugin.

A plugin can contain:

some metadata (including versioning)
one or several applications
some custom configuration
some base initialization
some extra nginx configuration (in mfserv only)
one or several daemons (which will be automatically launched/managed)
some execution limits (number of files, address space...)
a cron configuration
a dedicated virtualenv (python) or node_modules (nodejs)
[...]

You can have some "plugins dependencies" but in most cases, a plugin is self-contained in a single directory called the plugin directory or the plugin home.

Everything is packaged by the framework during the release process in a single .plugin file.

1.2 What is a plugin template?

A plugin template is just a way to start a new plugin faster. Instead of starting your new plugin from an empty directory, you can start from a provided plugin template.

But of course, it is not mandatory.

Info

For example, in the mfserv MetWork module, you will find plugin templates for boostrapping new plugins with nodejs or python3 async with tornado, python3 with WSGI...

The process of starting a new plugin from a plugin template is called bootstrapping (see workflow).

1.3 Roles

In the classic MetWork workflow, we have two roles (of course it can be the same person):

the developer
the administrator

1.3.1 The developer (DEV)

The developer build plugins (eventually from plugin templates). He probably needs internet access on his development computer.

He tests his plugin mainly in development mode or devlink mode.

Last, he releases his plugin as a single .plugin file.

1.3.2 The administrator (OP)

The administrator work starts with some released .plugin files. He installs them with a single command (or with the provisioning feature) and optionally changes their configuration.

On the deployment machine, no internet access is necessary because .plugin files are completely self-contained (even for virtualenv or node_modules).

2. The plugin workflow

2.1 DEV actions

2.1.1 Bootstrapping

The developer can bootstrap a plugin from a plugin template.

Use the command:

bootstrap_plugin.py list

to see available plugin templates.

To create a plugin directory from a template, use the command:

bootstrap_plugin.py create --template={template_name} {your_plugin_name}

Info

If you don't set the --template= option in the previous command, you will use the default plugin template.

2.1.2 Working in development mode

After bootstrapping, the developer can work in development mode (or devlink mode).

To install a plugin in this development mode, go to the plugin directory (you must find a Makefile file in it) and use:

make develop

In this mode, the plugin is restarted automatically when the code is edited in the plugin directory.

2.1.3 Updating the `virtualenv` or the `node_modules`

If the developer want to add some custom python or node libraries not included in MetWork layers, he can add this dependency in:

{plugin_dir}/python3_virtualenv_src/requirements-to-freeze.txt (for python3 plugins)
{plugin_dir}/packages.json (for nodejs plugins)

After that, use make in the plugin directory to rebuild the virtualenv.

When should I rebuild the virtualenv/node_modules manually?

When you install a plugin (in normal or devlink mode), the corresponding virtualenv/node_modules is automatically built. So you don't need to rebuild it manually. This operation is only required when you modify the virtualenv/node_sources after the installation. Just use make to do that inside the plugin directory.

What is the difference between requirements-to-freeze.txt and requirements3.txt (or requirements2.txt) files?

The second one is automatically generated (and overwritten!) from the first one (when it is more recent in terms of last modification date)

=> So don't modify requirements3.txt manually unless you know exactly what you're doing!

In the first one (requirements-to-freeze.txt) you should describe your PyPi requirements with the PEP508 format. The idea is to describe only your requirements (probably with some flexibility in versions) and not necessary all exact requirements.

Let's take an example. You want to use Django in your plugin. You can put Django >= 3.0,<3.1 in requirements-to-freeze.txt to say "I want the latest Django 3.0.x package". During the virtualenv build, MetWork will request PyPi to download your requirements (Django here) but also all requirements (recursively). In the generated requirements3.txt, you will find all requirements with exact versions. In this example, something like:

asgiref==3.2.9
certifi==2019.3.9
Django==3.0.7
lazy-import==0.2.2
sqlparse==0.3.1
tuna==0.4.5

If you commit and distribute this requirements3.txt file, you are sure to have exactly these versions when you deploy on another machine (for example). But, if you delete this requirements3.txt file or if you modify the requirements-to-freeze.txt file (to add something for example), the file will be generated automatically again (potentially with newer versions).

Note: same idea for requirements2.txt file (but for Python2 plugins)

2.1.4 Plugin env

Your plugin code runs in a plugin env. A plugin env is a bunch of PATH, LD_LIBRARY_PATH, PYTHONPATH, NODE_PATH env var dynamic changes to run your code inside a specific environment with your specific binaries, libraries and dependencies.

With this feature, you can have a "1.0 library/dependency" running in your plugin foo and a "1.1 library" running in another plugin.

It's a generalization of the Python concept of virtualenv.

It works for Python and NodeJS exactly in the same way. But you can also provide binaries (in any language) in the bin/ subdirectory of your plugin (be sure that these files are executable) or C/C++ libraries in the lib/ subdirectory. They will be available in the PATH and/or in the LD_LIBRARY_PATH when you are inside the plugin_env.

can I use pip for my Python plugins?

Yes, you can use pip command inside your plugin env.

But, be careful : the pip command can change python packages in your plugin env but without reflecting these changes in the requirements-to-freeze.txt file. It's great for testing but if you want to release your plugin with its new dependencies, don't forget to also change the requirements-to-freeze.txt file!

can I use npm for my NodeJS plugins?

Yes, you can use npm command inside your plugin env.

When your plugin/code is executed by the MetWork framework, it's automatically run in the corresponding plugin_env. But, if you want to test things interactively in a terminal, you have to enter the plugin_env manually.

To do that, use plugin_env with no argument if you are in the plugin directory or plugin_env {the_plugin_name} anywhere else.

plugin_wrapper

In some cases, you could want to execute a single command in a plugin_env (for example from a crontab. For this, you can use the plugin_wrapper {plugin_name} -- command [command_args] utility.

2.1.5 Defining configuration options

To define configuration options, add some key/values at the end of the config.ini file of your plugin in the [custom] configuration group.

Thanks to the library used, you can also use configuration variants in this file (for different profiles of machine). See [the documentation of opinionated_configparser library] (https://github.com/metwork-framework/opinionated_configparser/blob/master/README.md) for details about this feature (note that the configuration name is set in /etc/metwork.config file in MetWork framework).

Warning

Don't parse the config.ini file by yourself. This file and the plugin framework will automatically generate some environment variables from the file with the value corresponding to your configuration variant.

For example, if you have in your plugin config.ini:

[custom]
foo=value1
bar=value2

=> you will get two environment variables in your plugin env:

MFDATA_CURRENT_PLUGIN_CUSTOM_FOO=value1
MFDATA_CURRENT_PLUGIN_CUSTOM_BAR=value2

The naming schema if (of course) always the same:

{METWORK_MODULE}_CURRENT_PLUGIN_CUSTOM_{CONFIGURATION_KEY_IN_UPPERCASE}

Warning

VERY IMPORTANT: always read the environment to get your configuration values as these values can be overriden with other files by an administrator during deployment (see further).

2.1.6 Configuring cron jobs

In every plugin, you will find a crontab file dedicated to your plugin.

This file can be used to define cron jobs specific to your plugin.

They will be automatically injected into the ${MFMODULE_RUNTIME_USER} crontab when installing the plugin and removed when uninstalling the plugin.

Don't play with the crontab command by yourself or other system configurations!

All you need is this crontab file in your plugin directory.

If you are not familiar with the syntax of crontab files, please have a look at the internet as they are plenty of beginners guide about that.

BUT, there is ONE thing specific to MetWork Framework usage:

You have to prefix all your cronjobs command with:

{{MFDATA_HOME}}/bin/cronwrap.sh --lock --log-capture-to your_command.log -- plugin_wrapper {{MFDATA_CURRENT_PLUGIN_NAME}} --

It will:

replace automatically {{MFDATA_HOME}} }}, {{MFDATA_CURRENT_PLUGIN_NAME}}, {{MFDATA_CURRENT_PLUGIN_DIR + " }}, {{MFDATA_CURRENT_PLUGIN_CUSTOM_*}} variables (to avoid hardcoding things) (full list of available variables)
load the MetWork environment
define an execution timeout of 3600 seconds (you can change this with a --timeout option after --lock for example)
avoid multiple execution of the same command (thanks to the --lock option you can remove if you don't want this and if you know exactly what you are doing)
capture all logs (stdout/stderr) and redirect them to ${MFMODULE_RUNTIME_HOME}/log/your_command.log (of course you can change this filename)
load the plugin_env of the current plugin (very useful if you installed some extra libraries in your plugin or if you want to use some custom env variables)
then execute your command!

This file will be injected into the ${MFMODULE_RUNTIME_USER} crontab.

How to debug?

Use crontab -l as ${MFMODULE_RUNTIME_USER} to see injected cronjobs for all plugins and mfdata module.

If you don't have your lines, control output of _make_and_install_crontab.sh command.

If you have your lines but if your cron don't work well, check your cron log file as specified with ... --log-capture-to your_command.log ... part of the line.

more options?

There are plenty of other interesting options (nice, ionice...). Have a look at cronwrap.sh --help output

2.1.7 Configuring "extra-daemons"

In every plugin, you can configure some extra-daemons. An extra-daemon is a custom daemon that will be launched automatically and managed by the MetWork module (number of processes, autorestart, memory limits, log rotation...) when your plugin is installed.

You can configure several extra-daemons in a plugin.

To add an extra-daemon to your plugin, add or uncomment a block like this in your plugin config.ini:

[extra_daemon_foo]
_cmd_and_args = /your/foreground/command command_arg1 command_arg2
numprocesses=1
graceful_timeout = 30
rlimit_as = 1000000000
rlimit_nofile = 1000
rlimit_stack = 10000000
rlimit_fsize = 100000000
log_split_stdout_stderr=AUTO
log_split_multiple_workers=AUTO
max_age=0

The foo part in the [extra_daemon_foo] group name is mainly just a name to identify your extra-daemon on logs (you can use what you want but the configuration group must starts with [extra_daemon_.

Like with crontab (see above), you can use variables in this block. But you don't need to prefix your command as your command will be already executed in a wrapper to load your plugin environment.

documentation of these parameters

All parameters have the same meaning as their equivalents in the rest of the file (and are documented there)

logs

Logs of your extra-daemon will be catched and redirected to ${MFMODULE_RUNTIME_HOME}/log/extra_{your_plugin_name}_{your_extra_daemon_name}.log.

Log rotation is automatic.

2.1.8 Releasing

When the development is done, the developer can release his plugin with:

make release

He will get a self-contained .plugin file with everything needed inside.

After that, no internet connection is needed for deploying it.

2.2 OP actions

2.2.1 Installing a plugin

2.2.1.1 Manually

To install a released plugin manually, the administrator use:

plugins.install /path/to/the/dot/plugin/file.plugin

as mfdata user (or in corresponding MetWork env for custom installations).

After that, the .plugin is no longer useful.

2.2.1.2 Automatically with the provisioning feature

To automatically install a released plugin with the provisioning feature, just put the .plugin file in /etc/metwork.config.d/mfdata/plugins/ directory.

Then, restart the corresponding module by restarting the corresponding service with your favorite system services manager or with /etc/rc.d/init.d/metwork restart mfdata (as root user).

2.2.2 Configuring a plugin

When a plugin is installed, some configuration options can be changed by OPs (after installation). Available keys are keys in plugin config.ini which don't start with underscore.

2.2.2.1 Manually

These keys are available in ${MFMODULE_RUNTIME_HOME}/config/plugins/{plugin_name}.ini. So for example in ~mfdata/config/config.ini (for a standard MFDATA module). By default, all keys are commented, so standard values (defined by the developer) are used. If you want (as an OP) to override these default values, uncomment the corresponding key in ${MFMODULE_RUNTIME_HOME}/config/plugins/{plugin_name}.ini and change its value.

Note: this file itself can be overriden by the provisioning feature (see further).

2.2.2.2 Automatically with the provisioning feature

If you create a file in /etc/metwork.config.d/mfdata/plugins/{plugin_name}.ini with a configuration part you want to override:

For example:

[custom]
foo=new_value

to override (only) foo key of the [custom] configuration group.

2.3 Removing a plugin

To remove a plugin, use:

plugins.uninstall {plugin_name}

Tip

The plugin name is given in the first column of the plugins.list output.

Note

If you made some changes in the ${MFMODULE_RUNTIME_HOME}/config/plugins/{plugin_name}.ini configuration override file, the uninstallation process won't delete it. If you are sure you don't need it anymore, you can delete this file manually or add the --clean option with the plugins.uninstall command.

Note

The uninstallation process (with or without the --clean option) can't delete the /etc/metwork.config.d/mfdata/plugins/{plugin_name}.ini configuration override root file (as this file belongs to the root account).

2.4 Updating a plugin

2.4.1 Manually

To update a plugin manually, there is no specific command. You have to remove it. Then, reinstall it.

2.4.2 Automatically with the provisioning feature

To automatically update a released plugin with the provisioning feature, just put the .plugin file in /etc/metwork.config.d/mfdata/plugins/ directory.

Then, restart the corresponding module by restarting the corresponding service with your favorite system services manager or with /etc/rc.d/init.d/metwork restart mfdata (as root user).

The previous version of the plugin (identified by it's name) will be uninstalled first, then the released plugin will be installed.

Note

A plugin hash is used to compare the released plugin with the installed plugin with the same name. If the hash is the same, then nothing happen. If the hash is different the already installed plugin is uninstalled then the released plugin is installed, whatever the version.

!!! The plugin will not be updated if a plugin with the same name (and even if plugin hash is different) is already installed in development mode (or devlink mode). The dev-linked plugin is kept and nothing happen.

Tip

Don't use the --clean option during plugins.uninstall to keep your configuration overrides.

hotswapping with mfserv

With mfserv, there is another way to replace a plugin with a different version called "hot-swapping". This feature is a bit slow but this guarantees that you don't loose any HTTP request during the switch.

Let's take an example:

# Standard install of the foo.plugin (version 1)
plugins.install /path/foo-1.plugin

# [...]

# Hotswapping the installed version 1 by the version 2 of the same plugin
plugins.hotswap /path/foo-2.plugin

2.5 (advanced) Installing several times the same plugin

You can't install several plugins with the same name.

But sometimes, it can be useful to install the same plugin several times. For example:

to test a new version
to use the same plugin but with a different configuration

If the plugin didn't hardcode its name in its code, you can use an advanced option to install a plugin file with another name:

# Standard install of the foo.plugin
plugins.install /path/foo.plugin

# Install of the same plugin file but with another name
plugins.install --new-name bar /path/foo.plugin

Warning

It can't work if the plugin hardcodes its name in its code. Developers should use MFDATA_CURRENT_PLUGIN_NAME env var to avoid that.

2.6 (advanced) Repackaging a plugin

If you have installed a plugin with another name (see above) or if you change the configuration (as an administrator), you can be interested in producing a new corresponding .plugin file (including name and configuration overrides) without going back through the development role.

To do that, you can use:

plugins.repackage {name_of_the_installed_plugin}

And you will get a fresh new .plugin file.

3. Plugin reference

3.1 dev commands

Command	Description
`bootstrap_plugin.py list`	list available plugin templates
`bootstrap_plugin.py create --template={TEMPLATE} {PLUGIN_NAME}`	bootstrap a plugin directory `{PLUGIN_NAME}` from the given template
`make develop`	install the current plugin in "development mode" (devlink)
`make release`	release the current plugin as a `.plugin` file
`make`	refresh the `virtualenv` or `node_modules` from requirements file
`make clean`	"clean" the current plugin and keep only "non generated" files and directories (you should commit the remaining ones to your favorite version control system) ; after that, use `make` to regenerate the `virtualenv` or `node_modules`
`make superclean`	same as `clean` target but also drop `requirements2.txt`, `requirements3.txt` and/or `package-lock.json` which can lead to a dependencies update (they are not frozen anymore) during next `make` call

3.2 interesting files inside the plugin directory

Relative path	Description
`config.ini`	main plugin configuration file
`Makefile`	build configuration file (you probably don't need to touch this unless you have specific build directives to add to the `custom::` target)
`local/`	local subdirectory (it mainly holds the python virtualenv), never touch this it's automatically generated)
`bin/`	if you put an executable in this directory, it will be available in `PATH` (in your plugin environment)
`lib/`	this library directory will be available in `LD_LIBRARY_PATH` and in `PYTHONPATH` (in your plugin environment), so you can put here shared libraries or python files you want to include easily
`postinstall`	if this executable file is present during plugin installation, it will be automatically executed in the plugin environment just after the installation
`python3_virtualenv_sources/` `requirements-to-freeze.txt`	main requirements file for python3 plugins (you shouldn't freeze versions here).
`python3_virtualenv_sources/` `requirements3.txt`	frozen requirements file for python3 plugins (generated from `requirements-to-freeze.txt` file, deleted by `make superclean`, commit this file to your VCS to freeze your dependencies).
`python3_virtualenv_sources/` `allow_binary_packages`	file to delete if you don't want `pip` to use binary packages (it will try to compile them during install)
`python3_virtualenv_sources/` `override`	file to delete if you don't want `pip` to override packages available in mfext (it will try to use the version of mfext packages)
`.layerapi2_label`	layerapi2 file to hold the plugin name as `plugin_{plugin name}@module_in_lowercase`
`.layerapi2_dependencies`	layerapi2 file to hold the layers to load when entering the plugin environment (you can also put some plugins with the syntax `plugin_{other plugin name}@module_in_lowercase` to inherit from another plugin
`.layerapi2_extra_env`	can be used to define extra environment variables in your plugin environment (see layerapi2 documentation)
`.autorestart_includes`	file patterns (`gitignore` syntax) scanned for changes to trigger a plugin autorestart
`.autorestart_excludes`	file patterns (`gitignore` syntax) to exclude for scanning (see above)
`.plugin_format_version`	version of the framework used to bootstrap the plugin (don't change this, this is used for backward compatibility)
`.releaseignore`	file patterns (`gitignore` syntax) to ignore in the release `.plugin` file

Note: mandatory files are in bold, all these files are not created by default (it mainly depends on the template you used) but you can create them afterwards.

3.3 management commands

Command	Description
`plugins.list`	list installed plugins
`plugins.install {/full/path/file.plugin}`	install the given plugin file
`plugins.uninstall {plugin_name}`	uninstall the given plugin name (the "plugin name" is given in the first column of the `plugins.list` output)
`plugins.info {plugin_name}`	get some informations about the given plugin name (must be installed)
`plugins.info {/full/path/file.plugin}`	get some informations about the given plugin file (does not need to be installed)
`plugin_env {plugin_name}`	enter (interactively) in the given plugin environment
`plugin_wrapper {plugin_name} {YOUR_COMMAND}`	execute the given command in the given plugin environment (without changing anything to your current environment), useful for cron jobs

Plugins guide

1. Concepts

1.1 What is a plugin?

1.2 What is a plugin template?

1.3 Roles

1.3.1 The developer (DEV)

1.3.2 The administrator (OP)

2. The plugin workflow

2.1 DEV actions

2.1.1 Bootstrapping

2.1.2 Working in development mode

2.1.3 Updating the virtualenv or the node_modules

2.1.4 Plugin env

2.1.5 Defining configuration options

2.1.6 Configuring cron jobs

2.1.7 Configuring "extra-daemons"

2.1.8 Releasing

2.2 OP actions

2.2.1 Installing a plugin

2.2.1.1 Manually

2.2.1.2 Automatically with the provisioning feature

2.2.2 Configuring a plugin

2.2.2.1 Manually

2.2.2.2 Automatically with the provisioning feature

2.3 Removing a plugin

2.4 Updating a plugin

2.4.1 Manually

2.4.2 Automatically with the provisioning feature

2.5 (advanced) Installing several times the same plugin

2.6 (advanced) Repackaging a plugin

3. Plugin reference

3.1 dev commands

3.2 interesting files inside the plugin directory

3.3 management commands

2.1.3 Updating the `virtualenv` or the `node_modules`