.. role:: raw-html-m2r(raw) :format: html .. GENERATED FILE, DO NOT EDIT (edit mfdata_quick_start.md instead) Quick Start =========== Create your first plugin ------------------------ **This step will teach you how to create, run, release and deploy a simple MFDATA plugin with a MFDATA template.** We are going to make a ``move`` plugin, we will called ``move_image``. This plugin will simply moves incoming JPEG image files to a set destination folder, and ignore all other file types. Log in as the mfdata user: .. code-block:: bash su - mdfata If it's the first time you log in as mfdata, you have to set a password before (\ ``passwd mfdata`` or ``sudo passwd mfdata``\ ). To make our ``move_image`` plugin work, we need the ``switch`` plugin, this plugin is installed by default with MFDATA. It **guides incoming data files towards the appropriate plugin** depending on switch conditions defined by each plugins. Create the plugin ^^^^^^^^^^^^^^^^^ First, list the available templates, the following command: .. code-block:: bash bootstrap_plugin.py list will show you : .. code-block:: console List of available plugin templates: * archive * default * delete * fork * ftpsend * move Use the ``move`` template to create the plugin : run the following command: .. code-block:: bash bootstrap_plugin.py create --template=move move_image Once you have entered this command, you will be asked to fill in some fields to configure and customize your plugin: for now, press ``[ENTER]`` to set the default value, you will be able to modify you plugin configuration anytime later. The ``/home/mfdata`` directory now contains a ``move_image`` sub-directory with files: .. config.ini : configuration file of the plugin\ :raw-html-m2r:`
` main.py : python entry point of the plugin\ :raw-html-m2r:`
` Makefile : directives to build the plugin Set the destination directory ``dest-dir`` parameter: edit the ``move_image/config.ini`` file, and set the ``arg_dest-dir`` parameter (of the ``step_main`` section) with your destination directory (e.g. ``/tmp/my-jpeg-file``\ ) .. code-block:: cfg # dest-dir : target directory to move to arg_dest-dir = /tmp/my-jpeg-file You will notice the syntax to set argument in the config.ini file: ``arg_[my-arg]`` with ``[my-arg]`` the argument name, e.g. ``arg_dest-dir`` means ``--dest-dir`` argument. We vill now set the condition that allow to move only JPEG files in a destination directory. To do this, set the ``switch_logical_condition`` parameter in the ``move_image/config.ini`` : **it represents the condition that must be respected so the switch directs the data to your plugin**. In other words, it represents what kind of data will be orientated towards your plugin by the switch plugin: .. code-block:: cfg switch_logical_condition = ( x['latest.switch.main.system_magic'].startswith(b'JPEG image') ) where ``latest.switch.main.system_magic`` is a tag attribute of the file processed by the switch plugin. The ``switch_logical_condition`` condition have to be written **as Python syntax**. **This condition must be a boolean expression that evaluates as** ``True`` **for the type of data you are interested in**. :raw-html-m2r:`Note`\ : every time you change the plugin configuration, this confiuration will be always automatically reloaded. Let's now **install (as dev build)** the plugin by entering the command (from the ``move_image`` plugin directory): .. code-block:: bash make develop Check the plugin is installed: .. code-block:: bash plugins.list You should show the plugin is installed as dev build, i.e. dev_link .. code-block:: none Output: | NAME | VERSION | RELEASE --------------------------------------------------------------------------- | switch | master.ci77.f0d7991 | 1 | move_image | dev_link | dev_link Run the plugin ^^^^^^^^^^^^^^ Let's now run the plugin by :raw-html-m2r:`injecting a JPEG file` in the incomming directory of the switch plugin. Enter the command below to inject the file into the configured directory listened by the switch plugin .. code-block:: bash inject_file --incomming [my_jpeg_file] :raw-html-m2r:`Note`\ : by default : * the root data directory of MFDATA is ``/home/mfdata/var/in`` (MFDATA_DATA_IN_DIR environment variable) * the directory listened by the switch plugin is ``${MFDATA_DATA_IN_DIR}/incoming``\ , where ``incoming`` is configured in the ``${HOME}/config/config.ini`` file (where ``${HOME}`` is the home directory of the ``mfdata`` user): .. code-block:: cfg # directories listened by the switch plugin # (separated by ";") # switch_directories=incoming Another way to inject the JPEG file is to copy it in the ``${MFDATA_DATA_IN_DIR}/incoming`` .. code-block:: bash cp [my_jpeg_file] /home/mfdata/var/in/incoming/ Then check the JPEG file has been copied in the destination directory (i.e ``/tmp/my-jpeg-file``\ ). You can also :raw-html-m2r:`check the log files` of the swicth plugin and the move_image plugin. log files are stored in the ``${HOME}/log`` directory : * step_switch_main.stdout: .. 2019-03-07T07:19:52.831304Z [INFO] (mfdata.switch.main#14459) Start the processing of /home/mfdata/var/in/incoming/penguins.jpg...\ :raw-html-m2r:`
` 2019-03-07T07:19:52.838518Z [INFO] (mfdata.switch.main#14459) 1 selected directories (0 by hardlinking)\ :raw-html-m2r:`
` 2019-03-07T07:19:52.839523Z [INFO] (mfdata.switch.main#14459) File /home/mfdata/var/in/tmp/switch.main/2fb4d3b152cb4a57834ff0e328cf4197 moved to /home/mfdata/var/in/step.move_image.main/df7e8956cdb94fd3a835e301be332166\ :raw-html-m2r:`
` 2019-03-07T07:19:52.839718Z [INFO] (mfdata.switch.main#14459) End of the /home/mfdata/var/in/incoming/penguins.jpg processing after 5 ms * step_switch_main.stderr : empty * step_move_image_main.stdout: .. 2019-03-07T07:19:53.471712Z [INFO] (mfdata.move_image.main#14491) Start the processing of /home/mfdata/var/in/step.move_image.main/df7e8956cdb94fd3a835e301be332166... \ 2019-03-07T07:19:53.477857Z [INFO] (mfdata.move_image.main#14491) /home/mfdata/var/in/tmp/move_image.main/82d41b17a8af4081be010f2bcbae61b7 copied into /tmp/my_jpeg_file/penguins.jpg \ 2019-03-07T07:19:53.478069Z [INFO] (mfdata.move_image.main#14491) End of the /home/mfdata/var/in/step.move_image.main/df7e8956cdb94fd3a835e301be332166 processing after 3 ms \ * step_move_image_main.stderr : empty The diagram below shows the data flow: .. image:: images/move_image_flow.jpg :target: images/move_image_flow.jpg :alt: archive_image_flow #. The JPEG file is processed by ``switch`` from the MFDATA incoming directory #. The ``move_image`` plugin process the JPEG file (the ``switch_logical_condition`` is ``True``\ ) Let's now inject a file :raw-html-m2r:`**which is not a JPEG one**` use inject_file command or cp command). * Check the ``step_switch_main.stdout`` log file: .. 2019-03-07T08:40:43.142957Z [INFO] (mfdata.switch.main#23646) Start the processing of /home/mfdata/var/in/incoming/test.txt... \ 2019-03-07T08:40:43.148140Z [INFO] (mfdata.switch.main#23646) 0 selected directories (0 by hardlinking) \ 2019-03-07T08:40:43.148236Z [INFO] (mfdata.switch.main#23646) No condition matched for /home/mfdata/var/in/tmp/switch.main/c0a645867f5b494b8d30d7d76b4eba58 \ 2019-03-07T08:40:43.151121Z [INFO] (mfdata.switch.main#23646) End of the /home/mfdata/var/in/incoming/test.txt processing after 5 ms \ * The injected file doesn\'t match any switch condition, it is deleted (the default "no match" policy). :raw-html-m2r:`Note` : you may change the "no match" policy, by changing the ``switch_no_match_policy`` parameter of the ``${HOME}/config/config.ini`` file: .. code-block:: cfg # "no match" policy for the switch plugin: # delete => we delete the file # keep => we keep the file in trash # move => we move the file in another directory (see below) switch_no_match_policy=delete If you set the ``move`` "no match" policy, you have to set the ``switch_no_match_move_policy_dest_dir`` parameter to the directory you want the switch plugin move the 'no match" file: .. code-block:: cfg # if switch_no_match_policy = move, set the destination directory switch_no_match_move_policy_dest_dir=/tmp/nomatch :raw-html-m2r:`IMPORTANT:` every time you change the MFDATA configuration file (\ ``${HOME}/config/config.ini``\ ), you have to stop and start MFDATA to reload the configuration, by entering the commands : * either .. code-block:: bash mfdate.stop mfdata.start * or (as root user) .. code-block:: bash service metwork restart mfdata Release the plugin ^^^^^^^^^^^^^^^^^^ We wll now release the move_image plugin (considering it as a stable version). Releasing a plugin makes it production-ready, ready to be deployed. To do this, go to the ``move_image`` plugin directory and enter the command: .. code-block:: bash make release This will create a ``.plugin`` file that will be used to deploy the plugin, e.g. ``move_image-[version]-1.metwork.mfdata.plugin`` where ``[version]`` is the value of the ``version`` parameter of the move_image ``config.ini`` you enter when you create the plugin (default value is ``${MODULE_VERSION}``\ ). You may change it: .. code-block:: cfg # Version of the plugin (X.Y.Z) # If the value is [MODULE_VERSION], # the current module version is used version=1.0.0 Deploy the plugin ^^^^^^^^^^^^^^^^^ Let's now deploy the ``move_image`` plugin on a production environment. :raw-html-m2r:`Prerequisites`\ : * Metwork have to be installed on this environment (at least MFDATA and its dependencies MFEXT and MFCOM). * You be logged in as mfdata user In order to deploy the plugin on a production environment, put down the ``move_image-[version]-1.metwork.mfdata.plugin`` in a directory on this target environement, e.g. ``/home/mfdata/released_plugins`` :raw-html-m2r:`Note`\ : for this tutorial, if your production environment is the same as your development environment, you have to unstall the ``move_image`` plugin which is already installed (by the ``make develop`` command). To uninstall the plugin, enter: .. code-block:: bash plugins.uninstall move_image Then, check the ``move_image`` plugin is no longer installed, enter: .. code-block:: bash plugins.list You should not show the ``move_image`` plugin .. code-block:: none Output: | NAME | VERSION | RELEASE --------------------------------------------------------------------------- | switch | master.ci77.f0d7991 | 1 Let's now install the plugin on the production environment, enter: .. code-block:: bash plugins.install /home/mfdata/released_plugins/move_image-1.0.0-1.metwork.mfdata.plugin Then, check the ``move_image`` plugin is no longer installed, enter: .. code-block:: bash plugins.list In practice, the plugins are installed in the ``${HOME}/var/plugins`` directory. :raw-html-m2r:`Note` : another way to install the MFDATA ``move_image`` plugin in a production environment is to put down the ``.plugin`` file into ``/etc/metwork.config.d/mfdata/external_plugins`` directory. Then to install the plugin, you just restart the MFDATA service by entering ``service metwork restart`` mfdata command (as root user). Use of the ``ungzip`` plugin -------------------------------- The ``ungzip`` plugin allows you to unzip ``.gz`` files. This plugin automatically handles ``.gz`` files and returns the unzipped file to the switch. The ``ungzip`` is **NOT** installed by default. **First, let's create an** ``archive_image`` **plugin from the MFDATA template ``archive``.** This template allows you to **archive data**\ , i.e. moving a file to an associated archiving directory. Enter the command: .. code-block:: bash bootstrap_plugin.py create --template=archive archive_image Configure the ``switch_logical_condition`` parameter in the ``archive_image/config.ini`` that allow to archive only PNG files. .. code-block:: cfg switch_logical_condition = ( x['latest.switch.main.system_magic'].startswith(b'PNG image') ) Configure the archive directory ``arg_dest-dir`` parameter in the ``archive_image/config.ini`` set the ``arg_dest-dir`` parameter : .. code-block:: cfg arg_dest-dir = /tmp/my_archive_image Install (as dev build) the plugin by entering the command from the ``archive_image`` plugin directory: .. code-block:: bash make develop Check the plugin is installed, by running ``plugins.list`` Run the plugin: inject a PNG file : .. code-block:: bash inject_file --incomming /tmp/my_png_file.png Check the PNG file is 'archived'. Go to the ``/tmp/my_archive_image/[YYYYMMDD]`` directory where ``[YYYYMMDD]`` is the current date: you will see your PNG file is archived with a 'RANDOM_ID' name: e.g. ``2b199aac9ce2489b9f81f5a274ce7bf0`` Notice, you may change the archive file name, by changing the defaut value of the arg_strftime-template in the ``archive_image/config.ini``\ , e.g.: .. code-block:: cfg # strftime-template : template inside above archive directory (strftime # placeholders are allowed, / are allowed to define subdirectories, # {ORIGINAL_BASENAME}, {ORIGINAL_UID}, {ORIGINAL_DIRNAME}, {RANDOM_ID}, # {STEP_COUNTER} and {STEP_COUNTER_MINUS_1} are also available # Default is : "%Y%m%d/{RANDOM_ID}" arg_strftime-template = %Y%m%d/{ORIGINAL_BASENAME}_{RANDOM_ID} Now we would also like to archive the compressed PNG images. **Let's now inject a compressed PNG file (.gz).** Compress your PNG file (\ ``my_png_file.png.gz``\ ) and inject it: .. code-block:: bash inject_file --incomming /tmp/my_png_file.png.gz If the ``ungzip`` plugin is not installed, you will see the image is not archived in the ``/tmp/my_archive_image`` directory, because it's not a PNG file but a GZIP file. **Let's now Install the** ``ungzip`` **plugin (if not already installed).** In order to install this plugin, enter the command: .. code-block:: bash plugins.install /opt/metwork-mfdata/share/plugins/ungzip-[[version]-1.metwork.mfdata.plugin where ``[version]`` is the value of the ``version`` of the ``ungzip`` plugin, e.g. plugins.install /opt/metwork-mfdata/share/plugins/ungzip-master.ci53.508288c-1.metwork.mfdata.plugin Then, check the ``ugzip`` plugin is installed, enter: .. code-block:: bash plugins.list You should show the ``ungzip`` plugin is installed .. code-block:: none Output: | NAME | VERSION | RELEASE --------------------------------------------------------------------------- | switch | master.ci77.f0d7991 | 1 | ungzip | master.ci53.508288c | 1 | archive_image | 1.0.0 | 1 **Now, inject again the compressed PNG file.** You will see the PNG file is now archived in the ``/tmp/my_archive_image``\ , according to the ``arg_strftime-template`` parameter, as an uncompressed PNG file. The diagram below shows the data flow: .. image:: ./images/archive_image_flow.jpg :target: ./images/archive_image_flow.jpg :alt: archive_image_flow #. The GZIP file is processed by the ``switch`` plugin from the MFDATA ``incoming`` directory #. The ``gunzip`` plugin uncompresses the file #. The ``gunzip`` plugin puts the PNG file in the ``incoming`` directory. It will be process by the ``switch`` plugin #. The ``archive_image`` plugin processes the PNG file (the ``switch_logical_condition`` is ``True``\ ) Note : the ``archive`` template allows to keep tags/attributes, about the data flow, into another file. The ``.tags`` file is stored in the same directory as the archived file directory (same filename with extension ``.tag``\ ): .. 0.switch.main.enter_step = 2019-03-12T06:39:49:234958\ :raw-html-m2r:`
` 0.switch.main.exit_step = 2019-03-12T06:39:49:238562\ :raw-html-m2r:`
` 0.switch.main.process_status = ok\ :raw-html-m2r:`
` 0.switch.main.system_magic = gzip compressed data, was "temp.png", from Unix, last modified: Mon Mar 11 14:21:31 2019\ :raw-html-m2r:`
` 1.ungzip.main.enter_step = 2019-03-12T06:39:49:244102\ :raw-html-m2r:`
` 1.ungzip.main.exit_step = 2019-03-12T06:39:49:244923\ :raw-html-m2r:`
` 1.ungzip.main.process_status = ok\ :raw-html-m2r:`
` 2.switch.main.enter_step = 2019-03-12T06:39:49:250833\ :raw-html-m2r:`
` 2.switch.main.exit_step = 2019-03-12T06:39:49:251946\ :raw-html-m2r:`
` 2.switch.main.process_status = ok\ :raw-html-m2r:`
` 2.switch.main.system_magic = PNG image data, 120 x 165, 8-bit/color RGBA, non-interlaced\ :raw-html-m2r:`
` 3.archive_image.main.enter_step = 2019-03-12T06:39:49:258391\ :raw-html-m2r:`
` first.core.original_basename = temp.png.gz\ :raw-html-m2r:`
` first.core.original_dirname = incoming\ :raw-html-m2r:`
` first.core.original_uid = e93932b0e2424e9d988a54c833266be5\ :raw-html-m2r:`
` latest.core.step_counter = 3\ :raw-html-m2r:`
` latest.switch.main.system_magic = PNG image data, 120 x 165, 8-bit/color RGBA, non-interlaced