Layerapi2¶

Overview¶

layerapi2 module is a library and a set of cli tools to manage a layered environnement system.

A similar system is environnement modules. layerapi2 module is more simple, probably more opinionated and reliable but less HPC oriented and deals with only one compiler.

The library is designed to be not metwork specific and should be released as an independent product.

Main concepts¶

A layer¶

A layer is defined by:

a layer label (a string, not necessarily unique)
a layer home (a full path to a directory)

Optionnaly, a layer definition can include:

some dependencies (just a list of other layer labels)
some conflicts (just a list of other layer labels)
some environnement variables to set/unset durint layer load/unload
some extra interactive profile to load/unload during layer load/unload

So concretely, a layer is a directory with the following structure:

/path/to/my/layer/
                 /.layerapi2_label
                 /.layerapi2_dependencies
                 /.layerapi2_conflicts
                 /.layerapi2_extra_env
                 /.layerapi2_interactive_profile
                 /.layerapi2_interactive_unprofile

The only mandatory file is .layerapi2_label. It contains the layer label on its first and only line.

A layers path¶

The environnement variable METWORK_LAYERS_PATH contains a “:” separated list of directories full paths.

When we search a given layer, we iterate the list from the beginning and for each directory full path, we test:

if the selected directory is a layer by itself (ie. it contains a .layerapi2_label file)
if immediate subdirectories are layers

Consider the folowing example:

/path1/layer1/
/path1/layer1/.layerapi2_label (containing "layer1label")
/path2/layers/
/path2/layers/layer2/
/path2/layers/layer2/.layerapi2_label (containing "layer2label")
/path3/layers/layer3/
/path3/layers/layer3/.layerapi2_label (containing "layer3label")

If the value of METWORK_LAYERS_PATH is /path1/layer1:/path2/layers:/path3:

we will find (by its label) the layer “layer1label” because it’s directly pointed by the /path1/layer1 value
we will find (by its label) the layer “layer2label” because /path2/layers/layer2 (its home) is an immediate subdirectory of the /path2/layers value
we won’t find (by its label) the layer “layer3label” because /path3/layers/layer3 (is home) is not an immediate subdirectory of the /path3 value

Notes:

relative paths in METWORK_LAYERS_PATH are ignored
if there are several layer homes for a given label (ie. multiple directories with the same value for .layerapi2_label file), the first occurence is returned when searching by label (so the order of entries in METWORK_LAYERS_PATH can be important).

Installation / Loading / Unloading¶

We consider that a layer is installed if we can found it by its label through the layers path.

When a layer is installed, nothing is done automatically. It’s just available for loading.

Then a layer can be loaded. When the layer is loaded, the environnement is modified. We will see that in more detail a little further.

When a layer is loaded, it can be unloaded. Then, the corresponding environnement modification is reversed.

Technical details¶

What is done during layer loading ?¶

When you load a layer, following actions are done (in this particular order):

first if the layer is already loaded, we do nothing more
we iterate in the “conflicts list” of the layer and we unload each referenced layer (if loaded)
we iterate in the “dependencies list” of the layer and we load each referenced layer (if not loaded)
if a dependent layer is not installed (so it can’t be loaded), we give up the layer loading (unless this particular dependency is marked as optional)
then we load concretly the layer (we modify the current environnement)

Following modifications are done to the current environnement:

we prepend to PATH: {LAYER_HOME}/local/bin and {LAYER_HOME}/bin (if corresponding directories exist)
we prepend to LD_LIBRARY_PATH: {LAYER_HOME}/local/lib and {LAYER_HOME}/lib` (if corresponding directories exist)
we prepend to PKG_CONFIG_PATH: {LAYER_HOME}/local/lib/pkgconfig and {LAYER_HOME}/lib/pkgconfig (if corresponding directories exist)
we prepend to PYTHONPATH: {LAYER_HOME}/local/lib/python{PYTHON2_SHORT_VERSION}/site-packages and {LAYER_HOME}/lib/python{PYTHON2_SHORT_VERSION}/site-packages (if corresponding directories exist)
we prepend to PYTHONPATH: {LAYER_HOME}/local/lib/python{PYTHON3_SHORT_VERSION}/site-packages and {LAYER_HOME}/lib/python{PYTHON3_SHORT_VERSION}/site-packages (if corresponding directories exist)
we add extra environnement variables listed by {LAYER_HOME}/.layerapi2_extra_env (if the file exists)
we load/source the bash file {LAYER_HOME}/.layerapi2_interactive_profile file for interactive usage only (if the file exists)
we set a special environnement variable METWORK_LAYER_{HASH}_LOADED to memorize that the layer is loaded (HASH is a hash of the full layer home).

What is done during layer unloading ?¶

When you unload a layer, following actions are done (in this particular order):

we remove from PATH, LD_LIBRARY_PATH, PKG_CONFIG_PATH, PYTHONPATH all paths which starts with {LAYER_HOME}/
we load/source the bash file {LAYER_HOME}/.layerapi2_interactive_unprofile file for interactive usage only (if the file exists)
we unset the special environnement variable METWORK_LAYER_{HASH}_LOADED to memorize that the layer is not loaded any more
we remove extra environnement variables listed in {LAYER_HOME}/.layerapi2_extra_env (if the file exist)
we (recursively) unload all layers which depends on this one

Syntax of `.layerapi2_*` files¶

General¶

all files are plain text files and must be located exactly in the layer home. We will use {LAYER_HOME} syntax to point out this layer home in the following.

Note

In all .layerapi2_* files, you can embed this particular syntax: {ENVIRONNEMENT_VARIABLE_NAME} (with opening/closing braces), it will be dynamically substituted by its value (at loading time).

Warning

Do not mix with {LAYER_HOME} which is just a syntax for this documentation.

`{LAYER_HOME}/.layerapi2_label`¶

The only mandatory file is layerapi2_label. It is a plain text file with just one line containing the layer label. Valid characters for layer labels are:

basic letters of the English alphabet (A through Z and a through z)
digits (0 though 9)
space (but not at the beginning or at the end)
following characters: % & + , - . : = _ @ (but not at the beginning or at the end)

Example of .layerapi2_label file:

valid_label_for_a_layer

`{LAYER_HOME}/.layerapi2_dependencies` and `{LAYER_HOME}/.layerapi2_conflicts`¶

Then you have layerapi2_dependencies and layerapi2_conflicts which follow the same syntax. They are plain text files with each line is another valid layer label (see restrictions about layer names above).

Example of .layerapi2_dependencies/.layerapi2_conflicts file:

label of layer1
layer2_label
valid_label_for_a_layer
-optional_dependency1
-optional_dependency2

Note

If the label starts with -, it means that it is an optional dependency.

`{LAYER_HOME}/.layerapi2_extra_env`¶

The ̀.layerapi2_extra_env` is different. It’s a plain text files with several lines:

spaces at the beginning/end of each lines are ignored
lines which start with # are comments (they do nothing)
empty lines are ignored
ENV_VAR=ENV_VALUE lines mean “set ENV_VALUE into environnement variable named ENV_VAR” (no escaping is done, youd don’t need quotation marks, the = character just delimits the variable name and its value)

Example of .layerapi2_extra_env file:

PYTHON={MFEXT_HOME}/opt/python3_core/bin/python3
METWORK_PYTHON_MODE=3
PYTHON_SHORT_VERSION={PYTHON3_SHORT_VERSION}
PYTHONUNBUFFERED=x

Note

In this file, you have an example of {ENVIRONNEMENT_VARIABLE_NAME} syntax usage (see above).

`{LAYER_HOME}/.layerapi2_interactive_profile` and `{LAYER_HOME}/.layerapi2_interactive_unprofile`¶

The .layerapi2_interactive_profile and .layerapi2_interactive_unprofile are plain bash files. The first one is sourced/loaded when the corresponding layer is loaded. But it works only in interactive mode. For example, it won’t work with layer_wrapper very important utility.

Warning

These files should be removed in a future version of layerapi2 and replaced by another system/syntax. The main reason is that we can’t be sure that the .layerapi2_interactive_unprofile revert what the .layerapi2_interactive_profile has changed.

Because of above warning, please don’t use this feature a lot and limit bash commands to only set aliases for interactive usage.

Utilities¶

`layers`¶

The layers utility list installed layers. You can also filter the output to get:

only loaded layers
only not loaded (but installed) layers

If you don’t see your layer in layers output, check your METWORK_LAYERS_PATH environnement variable and if there is a .layerapi_label in your layer home.

Full documentation:

Usage:
  layers [OPTION...] - list installed layers

Help Options:
  -h, --help              Show help options

Application Options:
  -r, --raw               raw output
  -m, --loaded-filter     Loaded layer filter (default: no filter, possible values: yes, no)

In the default output:

you have the layer label, then the layer home
you have (*) before the layer label if the corresponding layer is already loaded

You can also filter only “not loaded” (but installed) layers with the following call:

layers --loaded-filter=no

`is_layer_installed`, `is_layer_loaded`¶

These two little utilities output 1 is the layer given as argument is installed/loaded.

Usage:
  is_layer_installed [OPTION...] LAYER LABEL - output 1 is the given layer is installed

Help Options:
  -h, --help       Show help options

Usage:
  is_layer_loaded [OPTION...] LAYER LABEL OR LAYER HOME - output 1 is the given layer is already loaded

Help Options:
  -h, --help       Show help options

`bootstrap_layer.sh`¶

This little utility can be used to bootstrap an empty layer.

Details are given in the help message:

usage: bootstrap_layer.sh LAYER_LABEL LAYER_HOME
  => bootstrap a new layer in the given LAYER_HOME (directory path)
     with the given LAYER_LABEL
  => the directory is automatically created (if it does not exist)
     and the .layerapi2_label file is created (if it does not exist)
  => a bin/, lib/ and lib/pkgconfig/ subdirectories are also created
  => if ${METWORK_PYTHON_MODE} environnement variable == 2,
      a lib/python${PYTHON2_SHORT_VERSION}/site-packages is created
  => if ${METWORK_PYTHON_MODE} environnement variable == 3,
      a lib/python${PYTHON3_SHORT_VERSION}/site-packages is created

`layer_wrapper`¶

This is probably the most interesting and the most useful utility.

First, let’s have a look at full options:

Usage:
  layer_wrapper [OPTION...] -- COMMAND [COMMAND_ARG1] [COMMAND_ARG2] [...] - wrapper to execute the given command in a process with some specific layers loaded

Help Options:
  -h, --help                 Show help options

Application Options:
  -d, --debug                debug mode
  -e, --empty                unload all layers before
  -c, --cwd                  change working directory to the last layer home
  -x, --extra-env-prefix     if set, add three environnement variables {PREFIX}_NAME, {PREFIX}_LABEL and {PREFIX}_DIR containing the last layer name, label and the last layer home
  -E, --empty-env            empty environnement (imply --empty)
  -k, --empty-env-keeps      coma separated list of env var to keep with --empty-env
  -l, --layers               coma separated list of layers labels/homes ('-' before the name of the layer means 'optional dependency')
  -p, --prepend-env          ENV_VAR,VALUE string to prepend VALUE in : separated ENV_VAR (like PATH) (can be used multiple times)
  -f, --force-prepend        do not check existing paths in prepend

This command can be used to launch another command in a new process but within a context where some additional layers are loaded. The original context won’t be modified.

For example:

$ is_layer_loaded foo
0
    => The layer "foo" is not loaded


$ layer_wrapper --layers=foo -- is_layer_loaded foo
1
    => We launched "is_layer_loaded foo" in a new process/context
       within the layer "foo" is loaded


$ is_layer_loaded foo
0
    => The original context is not modified

Another more complex example:

$ layers
- (*) layer1 [/tests/layer1]
- layer2 [/tests/layer2]
- layer3 [/tests/layer3]
    => We have 3 layers installed, only the first one is loaded

$ layer_wrapper --debug --empty --layers=layer2,layer3 -- layers
[DEBUG]: unloading layer1[/tests/layer1]
[DEBUG]: loading layer2[/tests/layer2]
[DEBUG]: loading layer3[/tests/layer3]
- layer1 [/tests/layer1]
- (*) layer2 [/tests/layer2]
- (*) layer3 [/tests/layer3]
    => We launched the "layers" command in a new context with first all layers
       unloaded then layer2 and layer3 loaded

$ layers
- (*) layer1 [/tests/layer1]
- layer2 [/tests/layer2]
- layer3 [/tests/layer3]
    => the original context is not modified

`layer_load_bash_cmds`, `layer_unload_bash_cmds`¶

Two very importants utilities are layer_load_bash_cmds and layer_unload_bash_cmds.

They output bash commands to source/eval in order to change the current context with the given layer loaded/unloaded (included all dependencies management).

Usage:
  layer_load_bash_cmds [OPTION...] LAYER_LABEL OR LAYER_HOME - output bash commands to eval to load the given layer

Help Options:
  -h, --help              Show help options

Application Options:
  -d, --debug             debug mode
  -f, --force-prepend     do not check existing paths in prepend

Usage:
  layer_unload_bash_cmds [OPTION...] LAYER_LABEL OR LAYER_HOME - output bash commands to eval to unload the given layer

Help Options:
  -h, --help       Show help options

Application Options:
  -d, --debug      debug mode

We recommend to define in your bash environnement two bash functions like this:

function layer_load() {
    eval "$(layer_load_bash_cmds --debug "$1")"
}

function layer_unload() {
    eval "$(layer_unload_bash_cmds --debug "$1")"
}

# Note: you can of course remove the "--debug" string if you don't want it

And use these two bash functions instead of layer_load_bash_cmds, layer_unload_bash_cmds binaries directly. See full tutorial for more details.

Full tutorial¶

Starting point¶

# we prepare an empty directory for the demo
rm -Rf ~/metwork/layerapi2_demo
mkdir -p ~/metwork/layerapi2_demo
cd ~/metwork/layerapi2_demo

# we unset METWORK_LAYERS_PATH
unset METWORK_LAYERS_PATH

# just to have an international language for error messages
export LANG=C

# define two bash functions to be able to modify the current context
function layer_load() {
    eval "$(layer_load_bash_cmds --debug "$1")"
}

function layer_unload() {
    eval "$(layer_unload_bash_cmds --debug "$1")"
}

echo $METWORK_LAYERS_PATH

note: the environnement variable is not set

# let's see installed layers
layers

note: of course, no layer installed (because METWORK_LAYERS_PATH is empty)

Bootstraping¶

# Let's bootstrap a first layer (label: layer1_label) in the "layer1" subdir
bootstrap_layer.sh layer1_label layer1

# Let's bootstrap a second layer (label: layer2_label) in the "layer2" subdir
bootstrap_layer.sh layer2_label layer2

find . -type d |grep layer

Output:

./layer1
./layer1/bin
./layer1/lib
./layer1/lib/python3.5
./layer1/lib/python3.5/site-packages
./layer1/lib/pkgconfig
./layer2
./layer2/bin
./layer2/lib
./layer2/lib/python3.5
./layer2/lib/python3.5/site-packages
./layer2/lib/pkgconfig

we have two empty layer structures

find . -type f |grep layer

Output:

./layer1/.layerapi2_label
./layer2/.layerapi2_label

cat layer1/.layerapi2_label

Output:

layer1_label

cat layer2/.layerapi2_label

Output:

layer2_label

note: label files are also bootstrapped

# let's say that layer2 depends on layer1
echo "layer1_label" >layer2/.layerapi2_dependencies

# let's see installed layers
layers

note: still nothing because METWORK_LAYERS_PATH is still empty

Basic features¶

# Let's set a METWORK_LAYERS_PATH
export METWORK_LAYERS_PATH=/home/fab/metwork/layerapi2_demo

# let's see installed layers
layers

Output:

- layer1_label [/home/fab/metwork/layerapi2_demo/layer1]
- layer2_label [/home/fab/metwork/layerapi2_demo/layer2]

note : our two layers are found

layers --loaded-filter=yes

note: nothing because our two layers are not loaded

# same thing but with other utilities
is_layer_installed layer1_label
is_layer_loaded layer1_label

Output:

1
0

# let's add a tool in the layer1
cat >layer1/bin/mytool <<EOF
#!/bin/bash

echo "it works"
EOF
chmod +x layer1/bin/mytool

# let's test it manually
layer1/bin/mytool

Output:

it works

# let's test it through the PATH
mytool

Output:

bash: mytool: command not found

note: “not found” because mytool is not in the current PATH

# let's load the layer1
layer_load layer1_label

Output:

[DEBUG]: loading layer1_label[/home/fab/metwork/layerapi2_demo/layer1]

layers

Output:

- (*) layer1_label [/home/fab/metwork/layerapi2_demo/layer1]
- layer2_label [/home/fab/metwork/layerapi2_demo/layer2]

note: the layer1 is loaded

# let's test it through the PATH
mytool

Output:

it works

# let's unload the layer1
layer_unload layer1_label

Output:

[DEBUG]: unloading layer1_label[/home/fab/metwork/layerapi2_demo/layer1]

# let's test
mytool

Output:

bash: mytool: command not found

# if we want to execute mytool without changing the current context
layer_wrapper --layers=layer1_label -- mytool

Output:

it works

is_layer_loaded layer1_label

Output:

0

note: the current context is not changed, the layer is not loaded

Dependencies¶

layers

Output:

- layer1_label [/home/fab/metwork/layerapi2_demo/layer1]
- layer2_label [/home/fab/metwork/layerapi2_demo/layer2]

note: nothing is loaded

cat layer2/.layerapi2_dependencies

Output:

layer1_label

note: the layer2 depends on layer1

# let's load the layer2 in the current context
layer_load layer2_label

Output:

[DEBUG]: layer layer2_label[/home/fab/metwork/layerapi2_demo/layer2] depends on not loaded layer layer1_label[/home/fab/metwork/layerapi2_demo/layer1] =>                             loading layer1_label
[DEBUG]: loading layer1_label[/home/fab/metwork/layerapi2_demo/layer1]
[DEBUG]: loading layer2_label[/home/fab/metwork/layerapi2_demo/layer2]

layers

Output:

- (*) layer1_label [/home/fab/metwork/layerapi2_demo/layer1]
- (*) layer2_label [/home/fab/metwork/layerapi2_demo/layer2]

note: both layers are loaded

# let's unload the layer1
layer_unload layer1_label

Output:

[DEBUG]: unloading layer1_label[/home/fab/metwork/layerapi2_demo/layer1]
[DEBUG]: layer layer2_label[/home/fab/metwork/layerapi2_demo/layer2] depends on missing dependency on layer layer1_label[/home/fab/metwork/layerapi2_demo/layer1]                                 => unloading layer2_label
[DEBUG]: unloading layer2_label[/home/fab/metwork/layerapi2_demo/layer2]

layers

Output:

- layer1_label [/home/fab/metwork/layerapi2_demo/layer1]
- layer2_label [/home/fab/metwork/layerapi2_demo/layer2]

note: the layer2 is also unloaded (because it depends on layer1)

Conflicts¶

FIXME

Layerapi2¶

Overview¶

Main concepts¶

A layer¶

A layers path¶

Installation / Loading / Unloading¶

Technical details¶

What is done during layer loading ?¶

What is done during layer unloading ?¶

Syntax of .layerapi2_* files¶

General¶

{LAYER_HOME}/.layerapi2_label¶

{LAYER_HOME}/.layerapi2_dependencies and {LAYER_HOME}/.layerapi2_conflicts¶

{LAYER_HOME}/.layerapi2_extra_env¶

{LAYER_HOME}/.layerapi2_interactive_profile and {LAYER_HOME}/.layerapi2_interactive_unprofile¶

Utilities¶

layers¶

is_layer_installed, is_layer_loaded¶

bootstrap_layer.sh¶

layer_wrapper¶

layer_load_bash_cmds, layer_unload_bash_cmds¶

Full tutorial¶

Starting point¶

Bootstraping¶

Basic features¶

Dependencies¶

Conflicts¶

Syntax of `.layerapi2_*` files¶

`{LAYER_HOME}/.layerapi2_label`¶

`{LAYER_HOME}/.layerapi2_dependencies` and `{LAYER_HOME}/.layerapi2_conflicts`¶

`{LAYER_HOME}/.layerapi2_extra_env`¶

`{LAYER_HOME}/.layerapi2_interactive_profile` and `{LAYER_HOME}/.layerapi2_interactive_unprofile`¶

`layers`¶

`is_layer_installed`, `is_layer_loaded`¶

`bootstrap_layer.sh`¶

`layer_wrapper`¶

`layer_load_bash_cmds`, `layer_unload_bash_cmds`¶