Changing remote and output folders with pattern strings

When using the batch or the sched sub-commands, outputs are stored in a batch sub-folder named after a unique identifier. This structure is not strict, and can be changed, giving you interesting options to sort your results on the fly. A first option to modify this structure, is to use the --output-folders option, which allows you to specify the structure directly as a command line argument. To do so, we use a pattern string, i.e. a string that contains RUNAWAY-prefixed environment variables. For instance, the default pattern string is batch/$RUNAWAY_UUID. When the folder must be created, the variables gets interpolated by Runaway, giving the final path used to store the execution results. For instance, the $RUNAWAY_UUID is a variable that stores the unique identifier of the execution. Those variables are extracted from the environment the execution is performed in, a few of which are set automatically by Runaway (see the execution model for more on that). Among others we find:

For instance, you can sort execution results based on their exit code by using the batch/$RUNAWAY_ECODE/$RUNAWAY_UUID pattern string. Note that you can also capture your own variables at run-time as explained in the execution model page.

Moreover, the remote folder, i.e. the folder in which the code gets executed on the remote end, can also be specified using pattern strings. This can be done using the --remote-folders option. Using this option can be interesting if you plan to keep your data on the remote for backup purpose, and you want to store them with a particular structure.

Specifying batch with files

If you are using the batch subcommand and the product strings does not suit your need, you can use a file to specify your parameters. This option can be used through the --parameters-file argument, which takes a file path as input. This file must be filled with newline-separated product strings that will all get expanded to a set of parameters string. For instance you can have a following file:

# Classic strings are valid product strings, that expands to themselves:
--param1=A --flag1 --flag2
--param1=B --flag3
# The following will expand to 4 parameters string.
{`--param2=`; `--param3=`} + {` --flag1`; `--flag2`}

This file would be expanded to 6 parameter strings: 2 for the first two lines, and 4 for the last product string.

Moreover, if you also prefer to use files to specify remote and output folders, the same options are available for them: the --remotes-file and the --outputs-file options. The only thing you have to take care about is that the number of generated parameters must match!

Changing post-processing

When using the batch or the sched subcommand, you can notice that the standard output, standard error, and exit code are respectively stored in the stdout, stderr and ecode files. Again, this post-processing can be parameterized to suit your needs, and provide extended post-processing capabilities.

The first way to do so is to tune the --post-command, which gets executed once the data were brought back to the output folder. By default this post command is set to:

cd $RUNAWAY_OUTPUT_FOLDER && echo "$RUNAWAY_ECODE" > ecode && echo "$RUNAWAY_STDOUT" > stdout && echo "$RUNAWAY_STDERR" > stderr

As with pattern strings, you can use variables extracted from the execution environment directly in the post command (see execution model for a list of all the available variables).

If your post-processing logic is too big to fit in the --post-command, you can use --post-script to point to a bash script that gets executed as well.