5. Layerapi2

5.1. Overview

layerapi2 module is a library and a set of cli tools to manage a layered environment system.

A similar system is environment modules. layerapi2 module is more simple, probably more opinionated and reliable but less HPC oriented and deals with only one compiler.

The library is designed to be not Metwork specific and should be released as an independent product.

5.2. Main concepts

5.2.1. A layer

A layer is defined by:

  • a layer label (a string, not necessarily unique)

  • a layer home (a full path to a directory)

Optionally, a layer definition can include:

  • some dependencies (just a list of other layer labels)

  • some conflicts (just a list of other layer labels)

  • some environment variables to set/unset during layer load/unload

  • some extra interactive profile to load/unload during layer load/unload

So concretely, a layer is a directory with the following structure:

/path/to/my/layer/
                 /.layerapi2_label
                 /.layerapi2_dependencies
                 /.layerapi2_conflicts
                 /.layerapi2_extra_env
                 /.layerapi2_interactive_profile
                 /.layerapi2_interactive_unprofile

The only mandatory file is .layerapi2_label. It contains the layer label on its first and only line.

5.2.2. A layers path

The environment variable METWORK_LAYERS_PATH contains a “:” separated list of directories full paths.

When we search a given layer, we iterate the list from the beginning and for each directory full path, we test:

  • if the selected directory is a layer by itself (ie. it contains a .layerapi2_label file)

  • if immediate subdirectories are layers

Consider the following example:

/path1/layer1/
/path1/layer1/.layerapi2_label (containing "layer1label")
/path2/layers/
/path2/layers/layer2/
/path2/layers/layer2/.layerapi2_label (containing "layer2label")
/path3/layers/layer3/
/path3/layers/layer3/.layerapi2_label (containing "layer3label")

If the value of METWORK_LAYERS_PATH is /path1/layer1:/path2/layers:/path3:

  • we will find (by its label) the layer “layer1label” because it’s directly pointed by the /path1/layer1 value

  • we will find (by its label) the layer “layer2label” because /path2/layers/layer2 (its home) is an immediate subdirectory of the /path2/layers value

  • we won’t find (by its label) the layer “layer3label” because /path3/layers/layer3 (is home) is not an immediate subdirectory of the /path3 value

Notes:

  • relative paths in METWORK_LAYERS_PATH are ignored

  • if there are several layer homes for a given label (ie. multiple directories with the same value for .layerapi2_label file), the first occurrence is returned when searching by label (so the order of entries in METWORK_LAYERS_PATH can be important).

5.2.3. Installation / Loading / Unloading

We consider that a layer is installed if we can found it by its label through the layers path.

When a layer is installed, nothing is done automatically. It’s just available for loading.

Then a layer can be loaded. When the layer is loaded, the environment is modified. We will see that in more detail a little further.

When a layer is loaded, it can be unloaded. Then, the corresponding environment modification is reversed.

5.3. Technical details

5.3.1. What is done during layer loading ?

When you load a layer, following actions are done (in this particular order):

  • first if the layer is already loaded, we do nothing more

  • we iterate in the “conflicts list” of the layer and we unload each referenced layer (if loaded)

  • we iterate in the “dependencies list” of the layer and we load each referenced layer (if not loaded)

  • if a dependent layer is not installed (so it can’t be loaded), we give up the layer loading (unless this particular dependency is marked as optional)

  • then we load concretely the layer (we modify the current environment)

Following modifications are done to the current environment:

  • we prepend to PATH: {LAYER_HOME}/local/bin and {LAYER_HOME}/bin (if corresponding directories exist)

  • we prepend to LD_LIBRARY_PATH: {LAYER_HOME}/local/lib and {LAYER_HOME}/lib` (if corresponding directories exist)

  • we prepend to PKG_CONFIG_PATH: {LAYER_HOME}/local/lib/pkgconfig and {LAYER_HOME}/lib/pkgconfig (if corresponding directories exist)

  • we prepend to PYTHONPATH: {LAYER_HOME}/local/lib/python{PYTHON2_SHORT_VERSION}/site-packages and {LAYER_HOME}/lib/python{PYTHON2_SHORT_VERSION}/site-packages (if corresponding directories exist)

  • we prepend to PYTHONPATH: {LAYER_HOME}/local/lib/python{PYTHON3_SHORT_VERSION}/site-packages and {LAYER_HOME}/lib/python{PYTHON3_SHORT_VERSION}/site-packages (if corresponding directories exist)

  • we add extra environment variables listed by {LAYER_HOME}/.layerapi2_extra_env (if the file exists)

  • we load/source the bash file {LAYER_HOME}/.layerapi2_interactive_profile file for interactive usage only (if the file exists)

  • we set a special environment variable METWORK_LAYER_{HASH}_LOADED to memorize that the layer is loaded (HASH is a hash of the full layer home).

5.3.2. What is done during layer unloading ?

When you unload a layer, following actions are done (in this particular order):

  • we remove from PATH, LD_LIBRARY_PATH, PKG_CONFIG_PATH, PYTHONPATH all paths which starts with {LAYER_HOME}/

  • we load/source the bash file {LAYER_HOME}/.layerapi2_interactive_unprofile file for interactive usage only (if the file exists)

  • we unset the special environment variable METWORK_LAYER_{HASH}_LOADED to memorize that the layer is not loaded any more

  • we remove extra environment variables listed in {LAYER_HOME}/.layerapi2_extra_env (if the file exist)

  • we (recursively) unload all layers which depends on this one

5.3.3. Syntax of .layerapi2_* files

5.3.3.1. General

all files are plain text files and must be located exactly in the layer home. We will use {LAYER_HOME} syntax to point out this layer home in the following.

Note

In all .layerapi2_* files, you can embed this particular syntax: {environment_VARIABLE_NAME} (with opening/closing braces), it will be dynamically substituted by its value (at loading time).

Warning

Do not mix with {LAYER_HOME} which is just a syntax for this documentation.

5.3.3.2. {LAYER_HOME}/.layerapi2_label

The only mandatory file is layerapi2_label. It is a plain text file with just one line containing the layer label. Valid characters for layer labels are:

  • basic letters of the English alphabet (A through Z and a through z)

  • digits (0 though 9)

  • space (but not at the beginning or at the end)

  • following characters: % & + , - . : = _ @ (but not at the beginning or at the end)

Example of .layerapi2_label file:

valid_label_for_a_layer

5.3.3.3. {LAYER_HOME}/.layerapi2_dependencies and {LAYER_HOME}/.layerapi2_conflicts

Then you have layerapi2_dependencies and layerapi2_conflicts which follow the same syntax. They are plain text files with each line is another valid layer label (see restrictions about layer names above).

Example of .layerapi2_dependencies/.layerapi2_conflicts file:

label of layer1
layer2_label
valid_label_for_a_layer
-optional_dependency1
-optional_dependency2

Note

If the label starts with -, it means that it is an optional dependency.

5.3.3.4. {LAYER_HOME}/.layerapi2_extra_env

The  ̀.layerapi2_extra_env` is different. It’s a plain text files with several lines:

  • spaces at the beginning/end of each lines are ignored

  • lines which start with # are comments (they do nothing)

  • empty lines are ignored

  • ENV_VAR=ENV_VALUE lines mean “set ENV_VALUE into environment variable named ENV_VAR” (no escaping is done, youd don’t need quotation marks, the = character just delimits the variable name and its value)

Example of .layerapi2_extra_env file:

PYTHON={MFEXT_HOME}/opt/python3_core/bin/python3
METWORK_PYTHON_MODE=3
PYTHON_SHORT_VERSION={PYTHON3_SHORT_VERSION}
PYTHONUNBUFFERED=x

Note

In this file, you have an example of {environment_VARIABLE_NAME} syntax usage (see above).

5.3.3.5. {LAYER_HOME}/.layerapi2_interactive_profile and {LAYER_HOME}/.layerapi2_interactive_unprofile

The .layerapi2_interactive_profile and .layerapi2_interactive_unprofile are plain bash files. The first one is sourced/loaded when the corresponding layer is loaded. But it works only in interactive mode. For example, it won’t work with layer_wrapper very important utility.

Warning

These files should be removed in a future version of layerapi2 and replaced by another system/syntax. The main reason is that we can’t be sure that the .layerapi2_interactive_unprofile revert what the .layerapi2_interactive_profile has changed.

Because of above warning, please don’t use this feature a lot and limit bash commands to only set aliases for interactive usage.

5.3.4. Utilities

5.3.4.1. layers

The layers utility list installed layers. You can also filter the output to get:

  • only loaded layers

  • only not loaded (but installed) layers

If you don’t see your layer in layers output, check your METWORK_LAYERS_PATH environment variable and if there is a .layerapi_label in your layer home.

Full documentation:

Usage:
  layers [OPTION?] - list installed layers

Help Options:
  -h, --help              Show help options

Application Options:
  -r, --raw               raw output
  -m, --loaded-filter     Loaded layer filter (default: no filter, possible values: yes, no)

In the default output:

  • you have the layer label, then the layer home

  • you have (*) before the layer label if the corresponding layer is already loaded

You can also filter only “not loaded” (but installed) layers with the following call:

layers --loaded-filter=no

5.3.4.2. is_layer_installed, is_layer_loaded

These two little utilities output 1 is the layer given as argument is installed/loaded.

Usage:
  is_layer_installed [OPTION?] LAYER LABEL - output 1 is the given layer is installed

Help Options:
  -h, --help       Show help options

Usage:
  is_layer_loaded [OPTION?] LAYER LABEL OR LAYER HOME - output 1 is the given layer is already loaded

Help Options:
  -h, --help       Show help options

5.3.4.3. bootstrap_layer.sh

This little utility can be used to bootstrap an empty layer.

Details are given in the help message:

usage: bootstrap_layer.sh LAYER_LABEL LAYER_HOME
  => bootstrap a new layer in the given LAYER_HOME (directory path)
     with the given LAYER_LABEL
  => the directory is automatically created (if it does not exist)
     and the .layerapi2_label file is created (if it does not exist)
  => a bin/, lib/ and lib/pkgconfig/ subdirectories are also created
  => if ${METWORK_PYTHON_MODE} environnement variable == 2,
      a lib/python${PYTHON2_SHORT_VERSION}/site-packages is created
  => if ${METWORK_PYTHON_MODE} environnement variable == 3,
      a lib/python${PYTHON3_SHORT_VERSION}/site-packages is created

5.3.4.4. layer_wrapper

This is probably the most interesting and the most useful utility.

First, let’s have a look at full options:

Usage:
  layer_wrapper [OPTION?] -- COMMAND [COMMAND_ARG1] [COMMAND_ARG2] [...] - wrapper to execute the given command in a process with some specific layers loaded

Help Options:
  -h, --help                 Show help options

Application Options:
  -d, --debug                debug mode
  -e, --empty                unload all layers before
  -c, --cwd                  change working directory to the last layer home
  -x, --extra-env-prefix     if set, add three environnement variables {PREFIX}_NAME, {PREFIX}_LABEL and {PREFIX}_DIR containing the last layer name, label and the last layer home
  -E, --empty-env            empty environnement (imply --empty)
  -k, --empty-env-keeps      coma separated list of env var to keep with --empty-env
  -l, --layers               coma separated list of layers labels/homes ('-' before the name of the layer means 'optional dependency')
  -p, --prepend-env          ENV_VAR,VALUE string to prepend VALUE in : separated ENV_VAR (like PATH) (can be used multiple times)
  -f, --force-prepend        do not check existing paths in prepend

This command can be used to launch another command in a new process but within a context where some additional layers are loaded. The original context won’t be modified.

For example:

$ is_layer_loaded foo
0
    => The layer "foo" is not loaded

$ layer_wrapper --layers=foo -- is_layer_loaded foo
1
    => We launched "is_layer_loaded foo" in a new process/context
       within the layer "foo" is loaded

$ is_layer_loaded foo
0
    => The original context is not modified

Another more complex example:

$ layers
- (*) layer1 [/tests/layer1]
- layer2 [/tests/layer2]
- layer3 [/tests/layer3]
    => We have 3 layers installed, only the first one is loaded

$ layer_wrapper --debug --empty --layers=layer2,layer3 -- layers
[DEBUG]: unloading layer1[/tests/layer1]
[DEBUG]: loading layer2[/tests/layer2]
[DEBUG]: loading layer3[/tests/layer3]
- layer1 [/tests/layer1]
- (*) layer2 [/tests/layer2]
- (*) layer3 [/tests/layer3]
    => We launched the "layers" command in a new context with first all layers
       unloaded then layer2 and layer3 loaded

$ layers
- (*) layer1 [/tests/layer1]
- layer2 [/tests/layer2]
- layer3 [/tests/layer3]
    => the original context is not modified

5.3.4.5. layer_load_bash_cmds, layer_unload_bash_cmds

Two very important utilities are layer_load_bash_cmds and layer_unload_bash_cmds.

They output bash commands to source/eval in order to change the current context with the given layer loaded/unloaded (included all dependencies management).

Usage:
  layer_load_bash_cmds [OPTION?] LAYER_LABEL OR LAYER_HOME - output bash commands to eval to load the given layer

Help Options:
  -h, --help              Show help options

Application Options:
  -d, --debug             debug mode
  -f, --force-prepend     do not check existing paths in prepend

Usage:
  layer_unload_bash_cmds [OPTION?] LAYER_LABEL OR LAYER_HOME - output bash commands to eval to unload the given layer

Help Options:
  -h, --help       Show help options

Application Options:
  -d, --debug      debug mode

We recommend to define in your bash environment two bash functions like this:

function layer_load() {
    eval "$(layer_load_bash_cmds --debug "$1")"
}

function layer_unload() {
    eval "$(layer_unload_bash_cmds --debug "$1")"
}

# Note: you can of course remove the "--debug" string if you don't want it

And use these two bash functions instead of layer_load_bash_cmds, layer_unload_bash_cmds binaries directly. See full tutorial for more details.

5.4. Full tutorial

5.4.1. Starting point

# we prepare an empty directory for the demo
rm -Rf ~/metwork/layerapi2_demo
mkdir -p ~/metwork/layerapi2_demo
cd ~/metwork/layerapi2_demo

# we unset METWORK_LAYERS_PATH
unset METWORK_LAYERS_PATH

# just to have an international language for error messages
export LANG=C

# define two bash functions to be able to modify the current context
function layer_load() {
    eval "$(layer_load_bash_cmds --debug "$1")"
}

function layer_unload() {
    eval "$(layer_unload_bash_cmds --debug "$1")"
}

echo $METWORK_LAYERS_PATH

note: the environnement variable is not set

# let's see installed layers
layers

note: of course, no layer installed (because METWORK_LAYERS_PATH is empty)

5.4.2. Bootstraping

# Let's bootstrap a first layer (label: layer1_label) in the "layer1" subdir
bootstrap_layer.sh layer1_label layer1

# Let's bootstrap a second layer (label: layer2_label) in the "layer2" subdir
bootstrap_layer.sh layer2_label layer2

find . -type d |grep layer

Output:

./layer1
./layer1/bin
./layer1/lib
./layer1/lib/python3.5
./layer1/lib/python3.5/site-packages
./layer1/lib/pkgconfig
./layer2
./layer2/bin
./layer2/lib
./layer2/lib/python3.5
./layer2/lib/python3.5/site-packages
./layer2/lib/pkgconfig

we have two empty layer structures

find . -type f |grep layer

Output:

./layer1/.layerapi2_label
./layer2/.layerapi2_label

cat layer1/.layerapi2_label

Output:

layer1_label

cat layer2/.layerapi2_label

Output:

layer2_label

note: label files are also bootstrapped

# let's say that layer2 depends on layer1
echo "layer1_label" >layer2/.layerapi2_dependencies

# let's see installed layers
layers

note: still nothing because METWORK_LAYERS_PATH is still empty

5.4.3. Basic features

# Let's set a METWORK_LAYERS_PATH
export METWORK_LAYERS_PATH=/home/fab/metwork/layerapi2_demo

# let's see installed layers
layers

Output:

- layer1_label [/home/fab/metwork/layerapi2_demo/layer1]
- layer2_label [/home/fab/metwork/layerapi2_demo/layer2]

note : our two layers are found

layers --loaded-filter=yes

note: nothing because our two layers are not loaded

# same thing but with other utilities
is_layer_installed layer1_label
is_layer_loaded layer1_label

Output:

1
0

# let's add a tool in the layer1
cat >layer1/bin/mytool <<EOF
#!/bin/bash

echo "it works"
EOF
chmod +x layer1/bin/mytool

# let's test it manually
layer1/bin/mytool

Output:

it works

# let's test it through the PATH
mytool

Output:

bash: mytool: command not found

note: “not found” because mytool is not in the current PATH

# let's load the layer1
layer_load layer1_label

Output:

[DEBUG]: loading layer1_label[/home/fab/metwork/layerapi2_demo/layer1]

layers

Output:

- (*) layer1_label [/home/fab/metwork/layerapi2_demo/layer1]
- layer2_label [/home/fab/metwork/layerapi2_demo/layer2]

note: the layer1 is loaded

# let's test it through the PATH
mytool

Output:

it works

# let's unload the layer1
layer_unload layer1_label

Output:

[DEBUG]: unloading layer1_label[/home/fab/metwork/layerapi2_demo/layer1]

# let's test
mytool

Output:

bash: mytool: command not found

# if we want to execute mytool without changing the current context
layer_wrapper --layers=layer1_label -- mytool

Output:

it works

is_layer_loaded layer1_label

Output:

0

note: the current context is not changed, the layer is not loaded

5.4.4.  Dependencies

layers

Output:

- layer1_label [/home/fab/metwork/layerapi2_demo/layer1]
- layer2_label [/home/fab/metwork/layerapi2_demo/layer2]

note: nothing is loaded

cat layer2/.layerapi2_dependencies

Output:

layer1_label

note: the layer2 depends on layer1

# let's load the layer2 in the current context
layer_load layer2_label

Output:

[DEBUG]: layer layer2_label[/home/fab/metwork/layerapi2_demo/layer2] depends on not loaded layer layer1_label[/home/fab/metwork/layerapi2_demo/layer1] =>                             loading layer1_label
[DEBUG]: loading layer1_label[/home/fab/metwork/layerapi2_demo/layer1]
[DEBUG]: loading layer2_label[/home/fab/metwork/layerapi2_demo/layer2]

layers

Output:

- (*) layer1_label [/home/fab/metwork/layerapi2_demo/layer1]
- (*) layer2_label [/home/fab/metwork/layerapi2_demo/layer2]

note: both layers are loaded

# let's unload the layer1
layer_unload layer1_label

Output:

[DEBUG]: unloading layer1_label[/home/fab/metwork/layerapi2_demo/layer1]
[DEBUG]: layer layer2_label[/home/fab/metwork/layerapi2_demo/layer2] depends on missing dependency on layer layer1_label[/home/fab/metwork/layerapi2_demo/layer1]                                 => unloading layer2_label
[DEBUG]: unloading layer2_label[/home/fab/metwork/layerapi2_demo/layer2]

layers

Output:

- layer1_label [/home/fab/metwork/layerapi2_demo/layer1]
- layer2_label [/home/fab/metwork/layerapi2_demo/layer2]

note: the layer2 is also unloaded (because it depends on layer1)

5.4.5. Conflicts

By default, python3@mfext is loaded. We want to use Python 2, so we will load python2@mfext

# let's load python2@mfext
layer_load python2@mfext

Output:

[DEBUG]: layer python2@mfext[/opt/metwork-mfext-master/opt/python2] depends on not loaded layer python2_core@mfext[/opt/metwork-mfext-master/opt/python2_core] =>                             loading python2_core@mfext
[DEBUG]: layer python2_core@mfext[/opt/metwork-mfext-master/opt/python2_core] conflicts with already loaded layer python3_core@mfext[/opt/metwork-mfext-master/opt/python3_core] =>                         unloading python3_core@mfext
[DEBUG]: unloading python3_core@mfext[/opt/metwork-mfext-master/opt/python3_core]
[DEBUG]: layer python3@mfext[/opt/metwork-mfext-master/opt/python3] depends on missing dependency on layer python3_core@mfext[/opt/metwork-mfext-master/opt/python3_core]                                 => unloading python3@mfext
[DEBUG]: unloading python3@mfext[/opt/metwork-mfext-master/opt/python3]
[DEBUG]: layer python3_devtools@mfext[/opt/metwork-mfext-master/opt/python3_devtools] depends on missing dependency on layer python3@mfext[/opt/metwork-mfext-master/opt/python3]                                 => unloading python3_devtools@mfext
[DEBUG]: unloading python3_devtools@mfext[/opt/metwork-mfext-master/opt/python3_devtools]
[DEBUG]: layer python3_scientific@mfext[/opt/metwork-mfext-master/opt/python3_scientific] depends on missing dependency on layer python3@mfext[/opt/metwork-mfext-master/opt/python3]                                 => unloading python3_scientific@mfext
[DEBUG]: unloading python3_scientific@mfext[/opt/metwork-mfext-master/opt/python3_scientific]
[DEBUG]: layer monitoring@mfext[/opt/metwork-mfext-master/opt/monitoring] depends on missing dependency on layer python3@mfext[/opt/metwork-mfext-master/opt/python3]                                 => unloading monitoring@mfext
[DEBUG]: unloading monitoring@mfext[/opt/metwork-mfext-master/opt/monitoring]
[DEBUG]: layer python3_devtools_jupyter@mfext[/opt/metwork-mfext-master/opt/python3_devtools_jupyter] depends on missing dependency on layer python3_devtools@mfext[/opt/metwork-mfext-master/opt/python3_devtools]                                 => unloading python3_devtools_jupyter@mfext
[DEBUG]: unloading python3_devtools_jupyter@mfext[/opt/metwork-mfext-master/opt/python3_devtools_jupyter]
[DEBUG]: loading python2_core@mfext[/opt/metwork-mfext-master/opt/python2_core]
[DEBUG]: loading python2@mfext[/opt/metwork-mfext-master/opt/python2]

Python 2 and its dependencies are loaded. Python 3 is in conflict wih Python 2. Python 3 and its dependencies are unloaded.