13. ACQUISITION API¶

class acquisition.AcquisitionStep[source]¶

Bases: object

Abstract class to describe an acquisition step.

You have to override this class.

Variables:

debug_mode_allowed (boolean) – if True, the debug mode is allowed.
stop_flag (boolean) – if True, stop the daemon as soon as possible.
unit_tests (boolean) – if True, we are in unit tests mode.
failure_policy (string) – failure policy.
args (Namespace) – argparser Namespace object with parsed cli args.
__logger (Logger) – Logger object.
__last_ping (Datetime) – Datetime object of the last ping() call.

__init__()[source]¶: Constructor.

add_extra_arguments(parser)[source]¶

Add some extra argument to commande line parsing.

If you have to add some, you have to override this method.

Parameters:	parser – an ArgumentParser object (with default options added).

after(status)[source]¶

Method called after the process execution.

It’s called event in case of exceptions during process.

copy_to_plugin_step(xaf, plugin_name, step_name)[source]¶

Copy a XattrFile (with tags) to another plugin/step.

Parameters:	xaf (XattrFile) – XattrFile to move plugin_name (string) – plugin name step_name (string) – step name
Returns:	True if ok
Return type:	boolean

critical(msg, *args, **kwargs)[source]¶: Log a critical message.

debug(msg, *args, **kwargs)[source]¶: Log a debug message.

destroy()[source]¶

Destroy what you want just before exiting.

No file will be processed after calling this method.

error(msg, *args, **kwargs)[source]¶: Log an error message.

error_and_die(msg, *args, **kwargs)[source]¶: Log an error message and exit immediatly.

exception(msg, *args, **kwargs)[source]¶

Log an error message with current exception stacktrace.

This method should only be called from an exception handler.

get_original_basename(xaf)[source]¶

Return the original basename of the file.

The original basename is the not modified basename before step0 execution. “unknown” is returned if we can’t find the basename.

Parameters:	xaf (XattrFile) – file to consider.
Returns:	original basename.
Return type:	string

get_original_dirname(xaf)[source]¶

Return the original dirname of the file.

The original dirname is the not modified basename before step0 execution. “unknown” is returned if we can’t find the dirname.

Parameters:	xaf (XattrFile) – file to consider.
Returns:	original dirname.
Return type:	string

get_original_uid(xaf)[source]¶

Return the original uid of the file.

The original uid is the first unique id given before step0 execution. “unknown” is returned if we can’t find the original uid.

Parameters:	xaf (XattrFile) – file to consider.
Returns:	original uid.
Return type:	string

get_plugin_directory_path()[source]¶

Return the plugin directory (fullpath).

Returns:	(string) the fullpath of the plugin directory.

get_tag(xaf, name, not_found_value=None, counter_str_value='latest', force_step_name=None, force_plugin_name=None)[source]¶

Read a tag on a file with good prefixes.

Parameters:	xaf (XattrFile) – file to read. name (string) – name of the tag (without prefixes). not_found_value – returned value if the tag is not found. counter_str_value (string) – counter string value. force_step_name – tagger step name (if None, current step name is taken) force_plugin_name – tagger plugin name (if None, current plugin name is taken)

get_tmp_filepath()[source]¶

Get a full temporary filepath (including unique filename).

Returns:	(string) full temporary filepath (including unique filename).

info(msg, *args, **kwargs)[source]¶: Log an info message.

init()[source]¶: Method called after CLI parsing but before processing any files.

move_to_plugin_step(xaf, plugin_name, step_name)[source]¶

Move a XattrFile to another plugin/step.

Parameters:	xaf (XattrFile) – XattrFile to move plugin_name (string) – plugin name step_name (string) – step name
Returns:	True if ok
Return type:	boolean

ping()[source]¶

Do something every second if possible.

The call can be blocked by a long running process() call.

plugin_name¶

Return the name of the plugin.

This method is called if there is no “step_name” property defined. This said property MUST be defined. The name must match with ^[A-Za-z0-9_]+$ regexp.

Returns:	(string) the name of the plugin.

process(xaf)[source]¶

Process one file.

Process one XattrFile. You have to override this method (unless your class inherits from BatchStep, see batch_process() in that case).

The file is moved into a temporary directory before the call with an unique filename. Extended attributes are copied to it. So you can do what you want with it.

If the method returns True:

we considerer that the processing is ok
the file is delete if necessary

If the method doesn’t return True:

we considerer that the processing is not ok (a warning message is

logged).
the file is given to the choosen failure policy.

Parameters:	xaf – XattrFile object.
Returns:	processing status (True: ok, False: not ok)
Return type:	boolean

set_tag(xaf, name, value, add_latest=True)[source]¶

Set a tag on a file with good prefixes.

Parameters:	xaf (XattrFile) – file to add/set tag on. name (string) – name of the tag (without prefixes) value (string) – value of the tag add_latest (boolean) – add latest prefix

step_name¶

Get the name of the step.

This method is called if there is no “step_name” property defined. This said property SHOULD be defined. The name must match with ^[A-Za-z0-9_]+$ regexp.

Returns:	(string) the name of the step.

warning(msg, *args, **kwargs)[source]¶: Log a warning message.

class acquisition.AcquisitionBatchStep[source]¶

Bases: acquisition.step.AcquisitionStep

Abstract class to describe a batch acquisition step.

You have to override this class.

Variables:	_batch (AcquisitionBatch) – AcquisitionBatch object.

__init__()[source]¶: Constructor.

add_extra_arguments(parser)¶

Add some extra argument to commande line parsing.

If you have to add some, you have to override this method.

Parameters:	parser – an ArgumentParser object (with default options added).

after(status)¶

Method called after the process execution.

It’s called event in case of exceptions during process.

batch_process(xafs)[source]¶

Process a batch of files.

Process several XattrFile (between 1 and max_batch_size files in one call). You have to override this method.

File are moved into a temporary directory before the call with unique filenames. Extended attributes are copied to them. So you can do what you want with them.

If the method returns True: - we considerer that the processing is ok - all files are deleted if necessary

If the method returns False: - we considerer that the result is False for each file - so each file follow the failure policy

If the method returns an array of booleans (of the same size than the xafs array), we consider that the processing status for each file.

Parameters:	xafs (list) – XattrFile objects (list of files to process).
Returns:	Processing status (True, False or array of booleans).

batch_process_max_size¶

Return the max size of a batch in batch process mode.

If not overriden, the default value is 100.

batch_process_max_wait¶

Max wait (in seconds) to fill the batch in batch process mode.

If not overriden, the default value is 10 (so 10 seconds).

copy_to_plugin_step(xaf, plugin_name, step_name)¶

Copy a XattrFile (with tags) to another plugin/step.

Parameters:	xaf (XattrFile) – XattrFile to move plugin_name (string) – plugin name step_name (string) – step name
Returns:	True if ok
Return type:	boolean

critical(msg, *args, **kwargs)¶: Log a critical message.

debug(msg, *args, **kwargs)¶: Log a debug message.

destroy()¶

Destroy what you want just before exiting.

No file will be processed after calling this method.

error(msg, *args, **kwargs)¶: Log an error message.

error_and_die(msg, *args, **kwargs)¶: Log an error message and exit immediatly.

exception(msg, *args, **kwargs)¶

Log an error message with current exception stacktrace.

This method should only be called from an exception handler.

get_original_basename(xaf)¶

Return the original basename of the file.

The original basename is the not modified basename before step0 execution. “unknown” is returned if we can’t find the basename.

Parameters:	xaf (XattrFile) – file to consider.
Returns:	original basename.
Return type:	string

get_original_dirname(xaf)¶

Return the original dirname of the file.

The original dirname is the not modified basename before step0 execution. “unknown” is returned if we can’t find the dirname.

Parameters:	xaf (XattrFile) – file to consider.
Returns:	original dirname.
Return type:	string

get_original_uid(xaf)¶

Return the original uid of the file.

The original uid is the first unique id given before step0 execution. “unknown” is returned if we can’t find the original uid.

Parameters:	xaf (XattrFile) – file to consider.
Returns:	original uid.
Return type:	string

get_plugin_directory_path()¶

Return the plugin directory (fullpath).

Returns:	(string) the fullpath of the plugin directory.

get_tag(xaf, name, not_found_value=None, counter_str_value='latest', force_step_name=None, force_plugin_name=None)¶

Read a tag on a file with good prefixes.

Parameters:	xaf (XattrFile) – file to read. name (string) – name of the tag (without prefixes). not_found_value – returned value if the tag is not found. counter_str_value (string) – counter string value. force_step_name – tagger step name (if None, current step name is taken) force_plugin_name – tagger plugin name (if None, current plugin name is taken)

get_tmp_filepath()¶

Get a full temporary filepath (including unique filename).

Returns:	(string) full temporary filepath (including unique filename).

info(msg, *args, **kwargs)¶: Log an info message.

init()¶: Method called after CLI parsing but before processing any files.

move_to_plugin_step(xaf, plugin_name, step_name)¶

Move a XattrFile to another plugin/step.

Parameters:	xaf (XattrFile) – XattrFile to move plugin_name (string) – plugin name step_name (string) – step name
Returns:	True if ok
Return type:	boolean

ping()¶

Do something every second if possible.

The call can be blocked by a long running process() call.

plugin_name¶

Return the name of the plugin.

This method is called if there is no “step_name” property defined. This said property MUST be defined. The name must match with ^[A-Za-z0-9_]+$ regexp.

Returns:	(string) the name of the plugin.

process(xaf)[source]¶

Process one file.

Process one XattrFile. You have to override this method (unless your class inherits from BatchStep, see batch_process() in that case).

The file is moved into a temporary directory before the call with an unique filename. Extended attributes are copied to it. So you can do what you want with it.

If the method returns True:

we considerer that the processing is ok
the file is delete if necessary

If the method doesn’t return True:

we considerer that the processing is not ok (a warning message is

logged).
the file is given to the choosen failure policy.

Parameters:	xaf – XattrFile object.
Returns:	processing status (True: ok, False: not ok)
Return type:	boolean

set_tag(xaf, name, value, add_latest=True)¶

Set a tag on a file with good prefixes.

Parameters:	xaf (XattrFile) – file to add/set tag on. name (string) – name of the tag (without prefixes) value (string) – value of the tag add_latest (boolean) – add latest prefix

step_name¶

Get the name of the step.

This method is called if there is no “step_name” property defined. This said property SHOULD be defined. The name must match with ^[A-Za-z0-9_]+$ regexp.

Returns:	(string) the name of the step.

warning(msg, *args, **kwargs)¶: Log a warning message.

class acquisition.AcquisitionDeleteStep[source]¶

Bases: acquisition.step.AcquisitionStep

__init__()¶: Constructor.

add_extra_arguments(parser)¶

Add some extra argument to commande line parsing.

If you have to add some, you have to override this method.

Parameters:	parser – an ArgumentParser object (with default options added).

after(status)¶

Method called after the process execution.

It’s called event in case of exceptions during process.

copy_to_plugin_step(xaf, plugin_name, step_name)¶

Copy a XattrFile (with tags) to another plugin/step.

Parameters:	xaf (XattrFile) – XattrFile to move plugin_name (string) – plugin name step_name (string) – step name
Returns:	True if ok
Return type:	boolean

critical(msg, *args, **kwargs)¶: Log a critical message.

debug(msg, *args, **kwargs)¶: Log a debug message.

destroy()¶

Destroy what you want just before exiting.

No file will be processed after calling this method.

error(msg, *args, **kwargs)¶: Log an error message.

error_and_die(msg, *args, **kwargs)¶: Log an error message and exit immediatly.

exception(msg, *args, **kwargs)¶

Log an error message with current exception stacktrace.

This method should only be called from an exception handler.

get_original_basename(xaf)¶

Return the original basename of the file.

The original basename is the not modified basename before step0 execution. “unknown” is returned if we can’t find the basename.

Parameters:	xaf (XattrFile) – file to consider.
Returns:	original basename.
Return type:	string

get_original_dirname(xaf)¶

Return the original dirname of the file.

The original dirname is the not modified basename before step0 execution. “unknown” is returned if we can’t find the dirname.

Parameters:	xaf (XattrFile) – file to consider.
Returns:	original dirname.
Return type:	string

get_original_uid(xaf)¶

Return the original uid of the file.

The original uid is the first unique id given before step0 execution. “unknown” is returned if we can’t find the original uid.

Parameters:	xaf (XattrFile) – file to consider.
Returns:	original uid.
Return type:	string

get_plugin_directory_path()¶

Return the plugin directory (fullpath).

Returns:	(string) the fullpath of the plugin directory.

get_tag(xaf, name, not_found_value=None, counter_str_value='latest', force_step_name=None, force_plugin_name=None)¶

Read a tag on a file with good prefixes.

Parameters:	xaf (XattrFile) – file to read. name (string) – name of the tag (without prefixes). not_found_value – returned value if the tag is not found. counter_str_value (string) – counter string value. force_step_name – tagger step name (if None, current step name is taken) force_plugin_name – tagger plugin name (if None, current plugin name is taken)

get_tmp_filepath()¶

Get a full temporary filepath (including unique filename).

Returns:	(string) full temporary filepath (including unique filename).

info(msg, *args, **kwargs)¶: Log an info message.

init()[source]¶: Method called after CLI parsing but before processing any files.

move_to_plugin_step(xaf, plugin_name, step_name)¶

Move a XattrFile to another plugin/step.

Parameters:	xaf (XattrFile) – XattrFile to move plugin_name (string) – plugin name step_name (string) – step name
Returns:	True if ok
Return type:	boolean

ping()¶

Do something every second if possible.

The call can be blocked by a long running process() call.

plugin_name¶

Return the name of the plugin.

This method is called if there is no “step_name” property defined. This said property MUST be defined. The name must match with ^[A-Za-z0-9_]+$ regexp.

Returns:	(string) the name of the plugin.

process(xaf)[source]¶

Process one file.

Process one XattrFile. You have to override this method (unless your class inherits from BatchStep, see batch_process() in that case).

The file is moved into a temporary directory before the call with an unique filename. Extended attributes are copied to it. So you can do what you want with it.

If the method returns True:

we considerer that the processing is ok
the file is delete if necessary

If the method doesn’t return True:

we considerer that the processing is not ok (a warning message is

logged).
the file is given to the choosen failure policy.

Parameters:	xaf – XattrFile object.
Returns:	processing status (True: ok, False: not ok)
Return type:	boolean

set_tag(xaf, name, value, add_latest=True)¶

Set a tag on a file with good prefixes.

Parameters:	xaf (XattrFile) – file to add/set tag on. name (string) – name of the tag (without prefixes) value (string) – value of the tag add_latest (boolean) – add latest prefix

step_name¶

Get the name of the step.

This method is called if there is no “step_name” property defined. This said property SHOULD be defined. The name must match with ^[A-Za-z0-9_]+$ regexp.

Returns:	(string) the name of the step.

warning(msg, *args, **kwargs)¶: Log a warning message.

class acquisition.AcquisitionMoveStep[source]¶

Bases: acquisition.step.AcquisitionStep

__init__()¶: Constructor.

add_extra_arguments(parser)[source]¶

Add some extra argument to commande line parsing.

If you have to add some, you have to override this method.

Parameters:	parser – an ArgumentParser object (with default options added).

after(status)¶

Method called after the process execution.

It’s called event in case of exceptions during process.

copy_to_plugin_step(xaf, plugin_name, step_name)¶

Copy a XattrFile (with tags) to another plugin/step.

Parameters:	xaf (XattrFile) – XattrFile to move plugin_name (string) – plugin name step_name (string) – step name
Returns:	True if ok
Return type:	boolean

critical(msg, *args, **kwargs)¶: Log a critical message.

debug(msg, *args, **kwargs)¶: Log a debug message.

destroy()¶

Destroy what you want just before exiting.

No file will be processed after calling this method.

error(msg, *args, **kwargs)¶: Log an error message.

error_and_die(msg, *args, **kwargs)¶: Log an error message and exit immediatly.

exception(msg, *args, **kwargs)¶

Log an error message with current exception stacktrace.

This method should only be called from an exception handler.

get_original_basename(xaf)¶

Return the original basename of the file.

The original basename is the not modified basename before step0 execution. “unknown” is returned if we can’t find the basename.

Parameters:	xaf (XattrFile) – file to consider.
Returns:	original basename.
Return type:	string

get_original_dirname(xaf)¶

Return the original dirname of the file.

The original dirname is the not modified basename before step0 execution. “unknown” is returned if we can’t find the dirname.

Parameters:	xaf (XattrFile) – file to consider.
Returns:	original dirname.
Return type:	string

get_original_uid(xaf)¶

Return the original uid of the file.

The original uid is the first unique id given before step0 execution. “unknown” is returned if we can’t find the original uid.

Parameters:	xaf (XattrFile) – file to consider.
Returns:	original uid.
Return type:	string

get_plugin_directory_path()¶

Return the plugin directory (fullpath).

Returns:	(string) the fullpath of the plugin directory.

get_tag(xaf, name, not_found_value=None, counter_str_value='latest', force_step_name=None, force_plugin_name=None)¶

Read a tag on a file with good prefixes.

Parameters:	xaf (XattrFile) – file to read. name (string) – name of the tag (without prefixes). not_found_value – returned value if the tag is not found. counter_str_value (string) – counter string value. force_step_name – tagger step name (if None, current step name is taken) force_plugin_name – tagger plugin name (if None, current plugin name is taken)

get_tmp_filepath()¶

Get a full temporary filepath (including unique filename).

Returns:	(string) full temporary filepath (including unique filename).

info(msg, *args, **kwargs)¶: Log an info message.

init()[source]¶: Method called after CLI parsing but before processing any files.

move_to_plugin_step(xaf, plugin_name, step_name)¶

Move a XattrFile to another plugin/step.

Parameters:	xaf (XattrFile) – XattrFile to move plugin_name (string) – plugin name step_name (string) – step name
Returns:	True if ok
Return type:	boolean

ping()¶

Do something every second if possible.

The call can be blocked by a long running process() call.

plugin_name¶

Return the name of the plugin.

This method is called if there is no “step_name” property defined. This said property MUST be defined. The name must match with ^[A-Za-z0-9_]+$ regexp.

Returns:	(string) the name of the plugin.

process(xaf)[source]¶

Process one file.

Process one XattrFile. You have to override this method (unless your class inherits from BatchStep, see batch_process() in that case).

The file is moved into a temporary directory before the call with an unique filename. Extended attributes are copied to it. So you can do what you want with it.

If the method returns True:

we considerer that the processing is ok
the file is delete if necessary

If the method doesn’t return True:

we considerer that the processing is not ok (a warning message is

logged).
the file is given to the choosen failure policy.

Parameters:	xaf – XattrFile object.
Returns:	processing status (True: ok, False: not ok)
Return type:	boolean

set_tag(xaf, name, value, add_latest=True)¶

Set a tag on a file with good prefixes.

Parameters:	xaf (XattrFile) – file to add/set tag on. name (string) – name of the tag (without prefixes) value (string) – value of the tag add_latest (boolean) – add latest prefix

step_name¶

Get the name of the step.

This method is called if there is no “step_name” property defined. This said property SHOULD be defined. The name must match with ^[A-Za-z0-9_]+$ regexp.

Returns:	(string) the name of the step.

warning(msg, *args, **kwargs)¶: Log a warning message.