.. GENERATED FILE, DO NOT EDIT (edit layerapi2.md instead)
Layerapi2
=========
Overview
--------
``layerapi2`` module is a library and a set of cli tools to manage a layered environnement
system.
A similar system is `environnement modules `_. ``layerapi2`` module
is more simple, probably more opinionated and reliable but less HPC oriented and deals with only one
compiler.
The library is designed to be not metwork specific and should be released
as an independent product.
Main concepts
-------------
A layer
^^^^^^^
A layer is defined by:
* a layer label (a string, not necessarily unique)
* a layer home (a full path to a directory)
Optionnaly, a layer definition can include:
* some dependencies (just a list of other layer labels)
* some conflicts (just a list of other layer labels)
* some environnement variables to set/unset durint layer load/unload
* some extra interactive profile to load/unload during layer load/unload
So concretely, a layer is a directory with the following structure:
.. code-block:: none
/path/to/my/layer/
/.layerapi2_label
/.layerapi2_dependencies
/.layerapi2_conflicts
/.layerapi2_extra_env
/.layerapi2_interactive_profile
/.layerapi2_interactive_unprofile
The only mandatory file is ``.layerapi2_label``. It contains the layer label on its first and
only line.
A layers path
^^^^^^^^^^^^^
The environnement variable ``METWORK_LAYERS_PATH`` contains a ":" separated list
of directories full paths.
When we search a given layer, we iterate the list from the beginning and for each
directory full path, we test:
* if the selected directory is a layer by itself (ie. it contains a ``.layerapi2_label`` file)
* if immediate subdirectories are layers
Consider the folowing example:
.. code-block:: none
/path1/layer1/
/path1/layer1/.layerapi2_label (containing "layer1label")
/path2/layers/
/path2/layers/layer2/
/path2/layers/layer2/.layerapi2_label (containing "layer2label")
/path3/layers/layer3/
/path3/layers/layer3/.layerapi2_label (containing "layer3label")
If the value of ``METWORK_LAYERS_PATH`` is ``/path1/layer1:/path2/layers:/path3``\ :
* we will find (by its label) the layer "layer1label" because it's directly pointed
by the ``/path1/layer1`` value
* we will find (by its label) the layer "layer2label" because ``/path2/layers/layer2`` (its home)
is an immediate subdirectory of the ``/path2/layers`` value
* we won't find (by its label) the layer "layer3label" because ``/path3/layers/layer3`` (is home) is not an immediate subdirectory of the ``/path3`` value
Notes:
* relative paths in ``METWORK_LAYERS_PATH`` are ignored
* if there are several layer homes for a given label (ie. multiple directories with the same value for ``.layerapi2_label`` file),
the first occurence is returned when searching by label
(so the order of entries in METWORK_LAYERS_PATH can be important).
Installation / Loading / Unloading
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
We consider that a layer is *installed* if we can found it by its label through the layers path.
When a layer is *installed*\ , nothing is done automatically. It's just available for loading.
Then a layer can be loaded. When the layer is loaded, the environnement is modified. We will see
that in more detail a little further.
When a layer is loaded, it can be unloaded. Then, the corresponding environnement modification is reversed.
Technical details
-----------------
What is done during layer loading ?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When you load a layer, following actions are done (in this particular order):
* first if the layer is already loaded, we do nothing more
* we iterate in the "conflicts list" of the layer and we unload each referenced layer (if loaded)
* we iterate in the "dependencies list" of the layer and we load each referenced layer (if not loaded)
* if a dependent layer is not installed (so it can't be loaded), we give up the layer loading (unless this particular dependency is marked as optional)
* then we load concretly the layer (we modify the current environnement)
Following modifications are done to the current environnement:
* we prepend to ``PATH``\ : ``{LAYER_HOME}/local/bin`` and ``{LAYER_HOME}/bin`` (if corresponding directories exist)
* we prepend to ``LD_LIBRARY_PATH``\ : ``{LAYER_HOME}/local/lib`` and {LAYER_HOME}/lib` (if corresponding directories exist)
* we prepend to ``PKG_CONFIG_PATH``\ : ``{LAYER_HOME}/local/lib/pkgconfig`` and ``{LAYER_HOME}/lib/pkgconfig`` (if corresponding directories exist)
* we prepend to ``PYTHONPATH``\ : ``{LAYER_HOME}/local/lib/python{PYTHON2_SHORT_VERSION}/site-packages`` and ``{LAYER_HOME}/lib/python{PYTHON2_SHORT_VERSION}/site-packages`` (if corresponding directories exist)
* we prepend to ``PYTHONPATH``\ : ``{LAYER_HOME}/local/lib/python{PYTHON3_SHORT_VERSION}/site-packages`` and ``{LAYER_HOME}/lib/python{PYTHON3_SHORT_VERSION}/site-packages`` (if corresponding directories exist)
* we add extra environnement variables listed by ``{LAYER_HOME}/.layerapi2_extra_env`` (if the file exists)
* we load/source the bash file ``{LAYER_HOME}/.layerapi2_interactive_profile`` file for interactive usage only (if the file exists)
* we set a special environnement variable ``METWORK_LAYER_{HASH}_LOADED`` to memorize that the layer is loaded (\ ``HASH`` is a hash of the full layer home).
What is done during layer unloading ?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When you unload a layer, following actions are done (in this particular order):
* we remove from ``PATH``\ , ``LD_LIBRARY_PATH``\ , ``PKG_CONFIG_PATH``\ , ``PYTHONPATH`` all paths
which starts with ``{LAYER_HOME}/``
* we load/source the bash file ``{LAYER_HOME}/.layerapi2_interactive_unprofile`` file for
interactive usage only (if the file exists)
* we unset the special environnement variable ``METWORK_LAYER_{HASH}_LOADED`` to memorize that
the layer is not loaded any more
* we remove extra environnement variables listed in ``{LAYER_HOME}/.layerapi2_extra_env`` (if the file exist)
* we (recursively) unload all layers which depends on this one
Syntax of ``.layerapi2_*`` files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
General
~~~~~~~
all files are plain text files and must be located exactly in the layer home. We will
use ``{LAYER_HOME}`` syntax to point out this layer home in the following.
.. note::
In all `.layerapi2_*` files, you can embed this particular syntax:
`{ENVIRONNEMENT_VARIABLE_NAME}` (with opening/closing braces), it will
be dynamically substituted by its value (at loading time).
.. warning::
Do not mix with `{LAYER_HOME}` which is just a syntax for this documentation.
``{LAYER_HOME}/.layerapi2_label``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The only mandatory file is ``layerapi2_label``. It is a plain text file with just one line
containing the layer label. Valid characters for layer labels are:
* basic letters of the English alphabet (A through Z and a through z)
* digits (0 though 9)
* space (but not at the beginning or at the end)
* following characters: ``% & + , - . : = _ @`` (but not at the beginning or at the end)
Example of ``.layerapi2_label`` file:
.. code-block:: none
valid_label_for_a_layer
``{LAYER_HOME}/.layerapi2_dependencies`` and ``{LAYER_HOME}/.layerapi2_conflicts``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Then you have ``layerapi2_dependencies`` and ``layerapi2_conflicts`` which follow the same syntax.
They are plain text files with each line is another valid layer label (see restrictions about layer names above).
Example of ``.layerapi2_dependencies``\ /\ ``.layerapi2_conflicts`` file:
.. code-block:: none
label of layer1
layer2_label
valid_label_for_a_layer
-optional_dependency1
-optional_dependency2
.. note::
If the label starts with `-`, it means that it is an optional dependency.
``{LAYER_HOME}/.layerapi2_extra_env``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ̀.layerapi2_extra_env` is different. It's a plain text files with several lines:
* spaces at the beginning/end of each lines are ignored
* lines which start with ``#`` are comments (they do nothing)
* empty lines are ignored
* ``ENV_VAR=ENV_VALUE`` lines mean "set ENV_VALUE into environnement variable named ENV_VAR" (no escaping is done, youd don't need quotation marks, the ``=`` character just delimits the variable name and its value)
Example of ``.layerapi2_extra_env`` file:
.. code-block:: none
PYTHON={MFEXT_HOME}/opt/python3_core/bin/python3
METWORK_PYTHON_MODE=3
PYTHON_SHORT_VERSION={PYTHON3_SHORT_VERSION}
PYTHONUNBUFFERED=x
.. note::
In this file, you have an example of `{ENVIRONNEMENT_VARIABLE_NAME}` syntax
usage (see above).
``{LAYER_HOME}/.layerapi2_interactive_profile`` and ``{LAYER_HOME}/.layerapi2_interactive_unprofile``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``.layerapi2_interactive_profile`` and ``.layerapi2_interactive_unprofile`` are plain ``bash`` files. The first one is sourced/loaded when the corresponding layer is loaded. But it works
only in interactive mode. For example, it won't work with ``layer_wrapper`` very important utility.
.. warning::
These files should be removed in a future version of `layerapi2` and replaced by another
system/syntax. The main reason is that we can't be sure that the
`.layerapi2_interactive_unprofile` revert what the `.layerapi2_interactive_profile`
has changed.
**Because of above warning, please don't use this feature a lot and limit bash commands to only
set aliases for interactive usage.**
Utilities
^^^^^^^^^
``layers``
~~~~~~~~~~~~~~
The ``layers`` utility list installed layers. You can also filter the output to get:
* only loaded layers
* only not loaded (but installed) layers
If you don't see your layer in ``layers`` output, check your ``METWORK_LAYERS_PATH`` environnement
variable and if there is a ``.layerapi_label`` in your layer home.
Full documentation:
.. code-block:: none
Usage:
layers [OPTION...] - list installed layers
Help Options:
-h, --help Show help options
Application Options:
-r, --raw raw output
-m, --loaded-filter Loaded layer filter (default: no filter, possible values: yes, no)
In the default output:
* you have the layer label, then the layer home
* you have ``(*)`` before the layer label if the corresponding layer is already loaded
You can also filter only "not loaded" (but installed) layers with the following call:
.. code-block:: none
layers --loaded-filter=no
``is_layer_installed``\ , ``is_layer_loaded``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
These two little utilities output ``1`` is the layer given as argument is installed/loaded.
.. code-block:: none
Usage:
is_layer_installed [OPTION...] LAYER LABEL - output 1 is the given layer is installed
Help Options:
-h, --help Show help options
.. code-block:: none
Usage:
is_layer_loaded [OPTION...] LAYER LABEL OR LAYER HOME - output 1 is the given layer is already loaded
Help Options:
-h, --help Show help options
``bootstrap_layer.sh``
~~~~~~~~~~~~~~~~~~~~~~~~~~
This little utility can be used to bootstrap an empty layer.
Details are given in the help message:
.. code-block:: none
usage: bootstrap_layer.sh LAYER_LABEL LAYER_HOME
=> bootstrap a new layer in the given LAYER_HOME (directory path)
with the given LAYER_LABEL
=> the directory is automatically created (if it does not exist)
and the .layerapi2_label file is created (if it does not exist)
=> a bin/, lib/ and lib/pkgconfig/ subdirectories are also created
=> if ${METWORK_PYTHON_MODE} environnement variable == 2,
a lib/python${PYTHON2_SHORT_VERSION}/site-packages is created
=> if ${METWORK_PYTHON_MODE} environnement variable == 3,
a lib/python${PYTHON3_SHORT_VERSION}/site-packages is created
``layer_wrapper``
~~~~~~~~~~~~~~~~~~~~~
This is probably the most interesting and the most useful utility.
First, let's have a look at full options:
.. code-block:: none
Usage:
layer_wrapper [OPTION...] -- COMMAND [COMMAND_ARG1] [COMMAND_ARG2] [...] - wrapper to execute the given command in a process with some specific layers loaded
Help Options:
-h, --help Show help options
Application Options:
-d, --debug debug mode
-e, --empty unload all layers before
-c, --cwd change working directory to the last layer home
-x, --extra-env-prefix if set, add three environnement variables {PREFIX}_NAME, {PREFIX}_LABEL and {PREFIX}_DIR containing the last layer name, label and the last layer home
-E, --empty-env empty environnement (imply --empty)
-k, --empty-env-keeps coma separated list of env var to keep with --empty-env
-l, --layers coma separated list of layers labels/homes ('-' before the name of the layer means 'optional dependency')
-p, --prepend-env ENV_VAR,VALUE string to prepend VALUE in : separated ENV_VAR (like PATH) (can be used multiple times)
-f, --force-prepend do not check existing paths in prepend
This command can be used to launch another command in a new process but within a context
where some additional layers are loaded. The original context won't be modified.
For example:
.. code-block:: none
$ is_layer_loaded foo
0
=> The layer "foo" is not loaded
$ layer_wrapper --layers=foo -- is_layer_loaded foo
1
=> We launched "is_layer_loaded foo" in a new process/context
within the layer "foo" is loaded
$ is_layer_loaded foo
0
=> The original context is not modified
Another more complex example:
.. code-block:: none
$ layers
- (*) layer1 [/tests/layer1]
- layer2 [/tests/layer2]
- layer3 [/tests/layer3]
=> We have 3 layers installed, only the first one is loaded
$ layer_wrapper --debug --empty --layers=layer2,layer3 -- layers
[DEBUG]: unloading layer1[/tests/layer1]
[DEBUG]: loading layer2[/tests/layer2]
[DEBUG]: loading layer3[/tests/layer3]
- layer1 [/tests/layer1]
- (*) layer2 [/tests/layer2]
- (*) layer3 [/tests/layer3]
=> We launched the "layers" command in a new context with first all layers
unloaded then layer2 and layer3 loaded
$ layers
- (*) layer1 [/tests/layer1]
- layer2 [/tests/layer2]
- layer3 [/tests/layer3]
=> the original context is not modified
``layer_load_bash_cmds``\ , ``layer_unload_bash_cmds``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Two very importants utilities are ``layer_load_bash_cmds`` and ``layer_unload_bash_cmds``.
They output bash commands to source/eval in order to change the current context with the given
layer loaded/unloaded (included all dependencies management).
.. code-block:: none
Usage:
layer_load_bash_cmds [OPTION...] LAYER_LABEL OR LAYER_HOME - output bash commands to eval to load the given layer
Help Options:
-h, --help Show help options
Application Options:
-d, --debug debug mode
-f, --force-prepend do not check existing paths in prepend
.. code-block:: none
Usage:
layer_unload_bash_cmds [OPTION...] LAYER_LABEL OR LAYER_HOME - output bash commands to eval to unload the given layer
Help Options:
-h, --help Show help options
Application Options:
-d, --debug debug mode
We recommend to define in your bash environnement two bash functions like this:
.. code-block:: bash
function layer_load() {
eval "$(layer_load_bash_cmds --debug "$1")"
}
function layer_unload() {
eval "$(layer_unload_bash_cmds --debug "$1")"
}
# Note: you can of course remove the "--debug" string if you don't want it
And use these two bash functions instead of ``layer_load_bash_cmds``\ , ``layer_unload_bash_cmds``
binaries directly. See full tutorial for more details.
Full tutorial
-------------
Starting point
^^^^^^^^^^^^^^
.. code-block:: bash
# we prepare an empty directory for the demo
rm -Rf ~/metwork/layerapi2_demo
mkdir -p ~/metwork/layerapi2_demo
cd ~/metwork/layerapi2_demo
.. code-block:: bash
# we unset METWORK_LAYERS_PATH
unset METWORK_LAYERS_PATH
.. code-block:: bash
# just to have an international language for error messages
export LANG=C
.. code-block:: bash
# define two bash functions to be able to modify the current context
function layer_load() {
eval "$(layer_load_bash_cmds --debug "$1")"
}
function layer_unload() {
eval "$(layer_unload_bash_cmds --debug "$1")"
}
.. code-block:: bash
echo $METWORK_LAYERS_PATH
note: the environnement variable is not set
.. code-block:: bash
# let's see installed layers
layers
note: of course, no layer installed (because METWORK_LAYERS_PATH is empty)
Bootstraping
^^^^^^^^^^^^
.. code-block:: bash
# Let's bootstrap a first layer (label: layer1_label) in the "layer1" subdir
bootstrap_layer.sh layer1_label layer1
.. code-block:: bash
# Let's bootstrap a second layer (label: layer2_label) in the "layer2" subdir
bootstrap_layer.sh layer2_label layer2
.. code-block:: bash
find . -type d |grep layer
.. code-block:: none
Output:
./layer1
./layer1/bin
./layer1/lib
./layer1/lib/python3.5
./layer1/lib/python3.5/site-packages
./layer1/lib/pkgconfig
./layer2
./layer2/bin
./layer2/lib
./layer2/lib/python3.5
./layer2/lib/python3.5/site-packages
./layer2/lib/pkgconfig
we have two empty layer structures
.. code-block:: bash
find . -type f |grep layer
.. code-block:: none
Output:
./layer1/.layerapi2_label
./layer2/.layerapi2_label
.. code-block:: bash
cat layer1/.layerapi2_label
.. code-block:: none
Output:
layer1_label
.. code-block:: bash
cat layer2/.layerapi2_label
.. code-block:: none
Output:
layer2_label
note: label files are also bootstrapped
.. code-block:: bash
# let's say that layer2 depends on layer1
echo "layer1_label" >layer2/.layerapi2_dependencies
.. code-block:: bash
# let's see installed layers
layers
note: still nothing because METWORK_LAYERS_PATH is still empty
Basic features
^^^^^^^^^^^^^^
.. code-block:: bash
# Let's set a METWORK_LAYERS_PATH
export METWORK_LAYERS_PATH=/home/fab/metwork/layerapi2_demo
.. code-block:: bash
# let's see installed layers
layers
.. code-block:: none
Output:
- layer1_label [/home/fab/metwork/layerapi2_demo/layer1]
- layer2_label [/home/fab/metwork/layerapi2_demo/layer2]
note : our two layers are found
.. code-block:: bash
layers --loaded-filter=yes
note: nothing because our two layers are not loaded
.. code-block:: bash
# same thing but with other utilities
is_layer_installed layer1_label
is_layer_loaded layer1_label
.. code-block:: none
Output:
1
0
.. code-block:: bash
# let's add a tool in the layer1
cat >layer1/bin/mytool < loading layer1_label
[DEBUG]: loading layer1_label[/home/fab/metwork/layerapi2_demo/layer1]
[DEBUG]: loading layer2_label[/home/fab/metwork/layerapi2_demo/layer2]
.. code-block:: bash
layers
.. code-block:: none
Output:
- (*) layer1_label [/home/fab/metwork/layerapi2_demo/layer1]
- (*) layer2_label [/home/fab/metwork/layerapi2_demo/layer2]
note: both layers are loaded
.. code-block:: bash
# let's unload the layer1
layer_unload layer1_label
.. code-block:: none
Output:
[DEBUG]: unloading layer1_label[/home/fab/metwork/layerapi2_demo/layer1]
[DEBUG]: layer layer2_label[/home/fab/metwork/layerapi2_demo/layer2] depends on missing dependency on layer layer1_label[/home/fab/metwork/layerapi2_demo/layer1] => unloading layer2_label
[DEBUG]: unloading layer2_label[/home/fab/metwork/layerapi2_demo/layer2]
.. code-block:: bash
layers
.. code-block:: none
Output:
- layer1_label [/home/fab/metwork/layerapi2_demo/layer1]
- layer2_label [/home/fab/metwork/layerapi2_demo/layer2]
note: the layer2 is also unloaded (because it depends on layer1)
Conflicts
^^^^^^^^^
FIXME