cctools
batch_wrapper.h File Reference

Wrapper operators for transforming batch jobs. More...

#include "batch_job.h"

Go to the source code of this file.

Functions

struct batch_wrapper * batch_wrapper_create (void)
 Create a builder for a batch wrapper.
 
void batch_wrapper_delete (struct batch_wrapper *w)
 Free a batch_wrapper.
 
void batch_wrapper_pre (struct batch_wrapper *w, const char *cmd)
 Add a shell command to the batch wrapper.
 
void batch_wrapper_argv (struct batch_wrapper *w, char *const argv[])
 Specify a command line to execute in the wrapper.
 
void batch_wrapper_args (struct batch_wrapper *w, char *const args[])
 Specify a command line to execute with shell interpretation.
 
void batch_wrapper_cmd (struct batch_wrapper *w, const char *cmd)
 Specify a shell command to execute.
 
void batch_wrapper_post (struct batch_wrapper *w, const char *cmd)
 Specify cleanup commands.
 
void batch_wrapper_prefix (struct batch_wrapper *w, const char *prefix)
 Set the name prefix to use for the wrapper script.
 
char * batch_wrapper_write (struct batch_wrapper *w, struct batch_job *t)
 Write out the batch_wrapper as a shell script.
 
char * batch_wrapper_expand (struct batch_job *t, struct jx *spec)
 Generate one or more wrapper scripts from a JX command spec.
 

Detailed Description

Wrapper operators for transforming batch jobs.

A wrapper object is a tool for performing a consistent transformation on a sequence of batch jobs. This is typically used for things like wrapping all commands in a container execution environment. A wrapper can modify a job at several points in its lifetime: batch_wrapper_pre gives commands to run before the job, batch_wrapper_post gives commands to run after the job, and batch_wrapper_argv defines that command to be run in the wrapper. Finally, batch_wrapper_write applies the wrapper to a specific batch job, producing a script with the desired results. This module is used primarily by Makeflow to perform consistent transformations.

See: Nick Hazekamp, "An Algebra for Robust Workflow Transformations", eScience 2018. https://ccl.cse.nd.edu/research/papers/workflow-transformation-escience-2018.pdf

Function Documentation

◆ batch_wrapper_create()

struct batch_wrapper * batch_wrapper_create ( void  )

Create a builder for a batch wrapper.

Use batch_wrapper_pre, batch_wrapper_cmd, etc. to add commands to the wrapper. These exist only in memory until calling batch_wrapper_write. Each command must exit successfully to continue executing the wrapper. A wrapper may only have a single main command (args, argv, or cmd).

◆ batch_wrapper_delete()

void batch_wrapper_delete ( struct batch_wrapper *  w)

Free a batch_wrapper.

Any scripts written out will continue to work after calling this function.

◆ batch_wrapper_pre()

void batch_wrapper_pre ( struct batch_wrapper *  w,
const char *  cmd 
)

Add a shell command to the batch wrapper.

Can be called multiple times to append multiple commands. These commands run before the main wrapper command. Each command must be a self-contained shell statement.

Parameters
cmdThe shell command to add.

◆ batch_wrapper_argv()

void batch_wrapper_argv ( struct batch_wrapper *  w,
char *const  argv[] 
)

Specify a command line to execute in the wrapper.

The arguments in argv are executed as-is, with no shell interpretation. This command executes after any pre commands. It is undefined behavior to add another command after calling this.

Parameters
argvThe command line to run.

◆ batch_wrapper_args()

void batch_wrapper_args ( struct batch_wrapper *  w,
char *const  args[] 
)

Specify a command line to execute with shell interpretation.

Same as batch_wrapper_argv, but each arg is individually interpreted by the shell for variable substitution and such.

Parameters
argsThe command line to run.

◆ batch_wrapper_cmd()

void batch_wrapper_cmd ( struct batch_wrapper *  w,
const char *  cmd 
)

Specify a shell command to execute.

Same as batch_wrapper_argv, but takes a shell command.

Parameters
cmdThe command line to run.

◆ batch_wrapper_post()

void batch_wrapper_post ( struct batch_wrapper *  w,
const char *  cmd 
)

Specify cleanup commands.

The shell statement specified will be executed before exiting the wrapper, even if previous commands failed. This is a good place for cleanup actions. Can be called multiple times.

Parameters
cmdThe shell command to add.

◆ batch_wrapper_prefix()

void batch_wrapper_prefix ( struct batch_wrapper *  w,
const char *  prefix 
)

Set the name prefix to use for the wrapper script.

The actual filename will consist of the prefix, an underscore, and some random characters to ensure that the name is unique. Defaults to "./wrapper"

Parameters
prefixThe filename prefix to use.

◆ batch_wrapper_write()

char * batch_wrapper_write ( struct batch_wrapper *  w,
struct batch_job t 
)

Write out the batch_wrapper as a shell script.

Does not consume the batch_wrapper.

Parameters
prefixThe prefix to use to generate a unique name for the wrapper.
Returns
The name of the generated wrapper, which the caller must free().
NULL on failure, and sets errno.

◆ batch_wrapper_expand()

char * batch_wrapper_expand ( struct batch_job t,
struct jx spec 
)

Generate one or more wrapper scripts from a JX command spec.

All generated scripts will be added as inputs to the given batch task.

Returns
The name of the outermost wrapper script.
NULL on failure, and sets errno.