18. ACQUISITION API

18.1. AcquisitionStep class

class acquisition.AcquisitionStep

Abstract class to describe an acquisition step.

You have to override this class.

stop_flag

if True, stop the daemon as soon as possible.

Type

boolean

debug_mode_allowed

if True, the debug mode is allowed.

Type

boolean

failure_policy

failure policy (“move”, “delete” or “keep”).

Type

string

failure_policy_move_dest_dir

destination directory when failure policy is move.

Type

string

failure_policy_move_keep_tags

keep tags into another file when failure policy is move.

Type

boolean

failure_policy_move_keep_tags_suffix

suffix to add to the filename to keep tags when failure policy is move.

Type

string

step_limit

maximum step number (to avoid some loops).

Type

int

Constructor.

__init__()

Constructor.

add_extra_arguments(parser)

Add some extra argument to commande line parsing.

If you have to add some, you have to override this method.

Parameters

parser – an ArgumentParser object (with default options added).

after(status)

Method called after the process execution.

It’s called event in case of exceptions during process.

copy_to_plugin_step(xaf, plugin_name, step_name, keep_original_basename=False)

Copy a XattrFile (with tags) to another plugin/step.

Parameters

• xaf (XattrFile) – XattrFile to move.

• plugin_name (string) – plugin name.

• step_name (string) – step name.

• keep_original_basename (boolean) – if True, we keep the original basename of xaf.

Returns

True if ok

Return type

boolean

critical(msg, *args, **kwargs)

Log a critical message.

debug(msg, *args, **kwargs)

Log a debug message.

destroy()

Destroy what you want just before exiting.

No file will be processed after calling this method.

error(msg, *args, **kwargs)

Log an error message.

error_and_die(msg, *args, **kwargs)

Log an error message and exit immediatly.

exception(msg, *args, **kwargs)

Log an error message with current exception stacktrace.

This method should only be called from an exception handler.

get_original_basename(xaf)

Return the original basename of the file.

The original basename is the not modified basename before step0 execution. “unknown” is returned if we can’t find the basename.

Parameters

xaf (XattrFile) – file to consider.

Returns

original basename.

Return type

string

get_original_dirname(xaf)

Return the original dirname of the file.

The original dirname is the not modified basename before step0 execution. “unknown” is returned if we can’t find the dirname.

Parameters

xaf (XattrFile) – file to consider.

Returns

original dirname.

Return type

string

get_original_uid(xaf)

Return the original uid of the file.

The original uid is the first unique id given before step0 execution. “unknown” is returned if we can’t find the original uid.

Parameters

xaf (XattrFile) – file to consider.

Returns

original uid.

Return type

string

get_plugin_directory_path()

Return the plugin directory (fullpath).

Returns

(string) the fullpath of the plugin directory.

get_tag(xaf, name, not_found_value=None, counter_str_value='latest', force_process_name=None, force_plugin_name=None, **kwargs)

Read a tag on a file with good prefixes.

Parameters

• xaf (XattrFile) – file to read.

• name (string) – name of the tag (without prefixes).

• not_found_value – returned value if the tag is not found.

• counter_str_value (string) – counter string value.

• force_process_name – tagger name (if None, current process_name is taken)

• force_plugin_name – tagger plugin name (if None, current plugin name is taken)

• kwargs – for compatibility with force_step_name

get_tmp_filepath()

Get a full temporary filepath (including unique filename).

Returns

(string) full temporary filepath (including unique filename).

info(msg, *args, **kwargs)

Log an info message.

init()

Init what you want.

Called after CLI parsing but before processing any files.

move_to_plugin_step(xaf, plugin_name, step_name, keep_original_basename=False)

Move a XattrFile to another plugin/step.

Parameters

• xaf (XattrFile) – XattrFile to move.

• plugin_name (string) – plugin name.

• step_name (string) – step name.

• keep_original_basename (boolean) – if True, we keep the original basename of xaf.

Returns

True if ok.

Return type

boolean

ping()

Do something every second if possible.

The call can be blocked by a long running process() call.

process(xaf)

Process one file.

Process one XattrFile. You have to override this method (unless your class inherits from BatchStep, see batch_process() in that case).

The file is moved into a temporary directory before the call with an unique filename. Extended attributes are copied to it. So you can do what you want with it.

If the method returns True:

• we considerer that the processing is ok

• the file is delete if necessary

If the method doesn’t return True:

• we considerer that the processing is not ok (a warning message is

  logged).

• the file is given to the choosen failure policy.

Parameters

xaf – XattrFile object.

Returns

processing status (True: ok, False: not ok)

Return type

boolean

set_tag(xaf, name, value, add_latest=True)

Set a tag on a file with good prefixes.

Parameters

• xaf (XattrFile) – file to add/set tag on.

• name (string) – name of the tag (without prefixes)

• value (string) – value of the tag

• add_latest (boolean) – add latest prefix

warning(msg, *args, **kwargs)

Log a warning message.

18.2. AcquisitionBatchStep class

class acquisition.AcquisitionBatchStep

Abstract class to describe a batch acquisition step.

You have to override this class.

Constructor.

__init__()

Constructor.

add_extra_arguments(parser)

Add some extra argument to commande line parsing.

If you have to add some, you have to override this method.

Parameters

parser – an ArgumentParser object (with default options added).

after(status)

Method called after the process execution.

It’s called event in case of exceptions during process.

batch_process(xafs)

Process a batch of files.

Process several XattrFile (between 1 and max_batch_size files in one call). You have to override this method.

File are moved into a temporary directory before the call with unique filenames. Extended attributes are copied to them. So you can do what you want with them.

If the method returns True: - we considerer that the processing is ok - all files are deleted if necessary

If the method returns False: - we considerer that the result is False for each file - so each file follow the failure policy

If the method returns an array of booleans (of the same size than the xafs array), we consider that the processing status for each file.

Parameters

xafs (list) – XattrFile objects (list of files to process).

Returns

Processing status (True, False or array of booleans).

property batch_process_max_size

Return the max size of a batch in batch process mode.

If not overriden, the default value is 100.

property batch_process_max_wait

Max wait (in seconds) to fill the batch in batch process mode.

If not overriden, the default value is 10 (so 10 seconds).

copy_to_plugin_step(xaf, plugin_name, step_name, keep_original_basename=False)

Copy a XattrFile (with tags) to another plugin/step.

Parameters

• xaf (XattrFile) – XattrFile to move.

• plugin_name (string) – plugin name.

• step_name (string) – step name.

• keep_original_basename (boolean) – if True, we keep the original basename of xaf.

Returns

True if ok

Return type

boolean

critical(msg, *args, **kwargs)

Log a critical message.

debug(msg, *args, **kwargs)

Log a debug message.

destroy()

Destroy what you want just before exiting.

No file will be processed after calling this method.

error(msg, *args, **kwargs)

Log an error message.

error_and_die(msg, *args, **kwargs)

Log an error message and exit immediatly.

exception(msg, *args, **kwargs)

Log an error message with current exception stacktrace.

This method should only be called from an exception handler.

get_original_basename(xaf)

Return the original basename of the file.

The original basename is the not modified basename before step0 execution. “unknown” is returned if we can’t find the basename.

Parameters

xaf (XattrFile) – file to consider.

Returns

original basename.

Return type

string

get_original_dirname(xaf)

Return the original dirname of the file.

The original dirname is the not modified basename before step0 execution. “unknown” is returned if we can’t find the dirname.

Parameters

xaf (XattrFile) – file to consider.

Returns

original dirname.

Return type

string

get_original_uid(xaf)

Return the original uid of the file.

The original uid is the first unique id given before step0 execution. “unknown” is returned if we can’t find the original uid.

Parameters

xaf (XattrFile) – file to consider.

Returns

original uid.

Return type

string

get_plugin_directory_path()

Return the plugin directory (fullpath).

Returns

(string) the fullpath of the plugin directory.

get_tag(xaf, name, not_found_value=None, counter_str_value='latest', force_process_name=None, force_plugin_name=None, **kwargs)

Read a tag on a file with good prefixes.

Parameters

• xaf (XattrFile) – file to read.

• name (string) – name of the tag (without prefixes).

• not_found_value – returned value if the tag is not found.

• counter_str_value (string) – counter string value.

• force_process_name – tagger name (if None, current process_name is taken)

• force_plugin_name – tagger plugin name (if None, current plugin name is taken)

• kwargs – for compatibility with force_step_name

get_tmp_filepath()

Get a full temporary filepath (including unique filename).

Returns

(string) full temporary filepath (including unique filename).

info(msg, *args, **kwargs)

Log an info message.

init()

Init what you want.

Called after CLI parsing but before processing any files.

move_to_plugin_step(xaf, plugin_name, step_name, keep_original_basename=False)

Move a XattrFile to another plugin/step.

Parameters

• xaf (XattrFile) – XattrFile to move.

• plugin_name (string) – plugin name.

• step_name (string) – step name.

• keep_original_basename (boolean) – if True, we keep the original basename of xaf.

Returns

True if ok.

Return type

boolean

ping()

Do something every second if possible.

The call can be blocked by a long running process() call.

process(xaf)

Process one file.

Process one XattrFile. You have to override this method (unless your class inherits from BatchStep, see batch_process() in that case).

The file is moved into a temporary directory before the call with an unique filename. Extended attributes are copied to it. So you can do what you want with it.

If the method returns True:

• we considerer that the processing is ok

• the file is delete if necessary

If the method doesn’t return True:

• we considerer that the processing is not ok (a warning message is

  logged).

• the file is given to the choosen failure policy.

Parameters

xaf – XattrFile object.

Returns

processing status (True: ok, False: not ok)

Return type

boolean

set_tag(xaf, name, value, add_latest=True)

Set a tag on a file with good prefixes.

Parameters

• xaf (XattrFile) – file to add/set tag on.

• name (string) – name of the tag (without prefixes)

• value (string) – value of the tag

• add_latest (boolean) – add latest prefix

warning(msg, *args, **kwargs)

Log a warning message.

18.3. AcquisitionListener class

18.4. Acquisition decorators

acquisition.decorators.try_unbzip2(func)

Decorate the process() or batch_process() method to unbzip2 (on the fly).

Example

from acquisition import AcquisitionStep
from acquisition.decorators import try_unbzip2

# [...]

class MyStep(AcquisitionStep):

    # [...]

    @try_unbzip2
    def process(self, xaf):
        self.info("xaf is maybe unbzipped")
        # [...]
        return True

If we fail to unbzip the incoming file, nothing is logged and the decorated function is called as usual.

See also unbzip decorator.

acquisition.decorators.try_ungzip(func)

Decorate the process() or batch_process() method to ungzip (on the fly).

Example

from acquisition import AcquisitionStep
from acquisition.decorators import try_ungzip

# [...]

class MyStep(AcquisitionStep):

    # [...]

    @try_ungzip
    def process(self, xaf):
        self.info("xaf is maybe ungzipped")
        # [...]
        return True

If we fail to ungzip the incoming file, nothing is logged and the decorated function is called as usual.

See also ungzip decorator.

acquisition.decorators.unbzip2(func)

Decorate the process() or batch_process() method to unbzip2 (on the fly).

Example

from acquisition import AcquisitionStep
from acquisition.decorators import unbzip2

# [...]

class MyStep(AcquisitionStep):

    # [...]

    @unbzip2
    def process(self, xaf):
        self.info("xaf is unbzipped")
        # [...]
        return True

If we fail to unbzip2 the incoming file, a warning is logged and the decorated function is not called at all.

See also try_unbzip2 decorator.

acquisition.decorators.ungzip(func)

Decorate the process() or batch_process() method to ungzip (on the fly).

Example

from acquisition import AcquisitionStep
from acquisition.decorators import ungzip

# [...]

class MyStep(AcquisitionStep):

    # [...]

    @ungzip
    def process(self, xaf):
        self.info("xaf is ungzipped")
        # [...]
        return True

If we fail to ungzip the incoming file, a warning is logged and the decorated function is not called at all.

See also try_ungzip decorator.