Welcome to recast-workflow’s documentation!

Recast workflow is a library that allows you to quickly generate Yadage computational workflows to run simple truth level reinterpretations without spending a significant amount of time or resources.

inventory.py

Inventory is storage for generated workflows.

recast_workflow.inventory.add(path: str, name='', raw_text='')

Add workflow at path to inventory

recast_workflow.inventory.get_dir(name: str, output_path: str, reana: str = '')

Get directory with run script, inputs folder, and workflow

recast_workflow.inventory.get_inv_wf_path(name: str) → str

Return path to workflow saved in inventory. Returns None if not found.

recast_workflow.inventory.get_inv_wf_yml(name: str, output_file='', text=False) → Dict[KT, VT]

Return yaml text of given workflow name

recast_workflow.inventory.get_wf_path(path: str) → str

Get path to workflow. Returns None if not found.

recast_workflow.inventory.list_inv() → List[str]

Returns list all workflow names in inventory

recast_workflow.inventory.remove(name: str)

Remove workflow from inventory


catalogue.py

recast_workflow.catalogue.get_all_combinations()

Returns all possible subworkflow combinations.

recast_workflow.catalogue.get_environment_settings(step: str, subworkflow_name: str, include_default=False) → Union[List[str], Dict[str, str]]

Returns the possible environment settings for the given subworkflow.

recast_workflow.catalogue.get_invalid_inputs(step: str, subworkflow_name: str, inputs: Dict[str, str]) → Dict[str, str]

Returns a list of the elements from the given inputs that are invalid for the given subworkflow.

recast_workflow.catalogue.get_missing_inputs(step: str, subworkflow_name: str, inputs: Dict[str, str], include_descriptions=False, include_optional=False) → Union[List[str], Dict[str, str]]

Returns the inputs for the given subworkflow that are missing from the given inputs.

recast_workflow.catalogue.get_valid_combinations(common_inputs: Dict[str, str]) → List[OrderedDict[str, str]]

Given values for common inputs, returns all combinations of subworkflows that support those values. Each common input should be defined in recast_workflow/common_inputs.yml, in which it is associated with a set of steps. Each subworkflow for those steps is assumed to implement a common_inputs.py file with an is_valid() -> bool function. This function is called for each subworkflow to determine whether it should be included. Once the set of allowed subworkflows for these steps is determined, the set of all possible combinations is generated using the interfaces listed in description.yml for each subworkflow. :param common_inputs: A dict where each key-value pair is of the form (input name, input value).

Returns:A list of ordered dictionaries, where each ordered dictionary has key-value pairs of the form (step, subworkflow) which describe one possible subworkflow combination that is valid for the given common inputs.

workflow.py

recast_workflow.workflow.build_subworkflow(step: str, name: str, environment_settings: dict)

Builds the images corresponding to the given subworkflow and environment settings. The subworkflow’s base image directory is searched recursively for dockerfiles. Each directory with a dockerfile is assumed to correspond to an image, where the name of the directory is equivalent to the name of the image. If a build.py exists in an image directory, it is run and assumed to handle building the image. Otherwise, the ‘build_tags’ section of the description.yml for the subworkflow is checked for an appropriate tag. It is assumed that this tag is also the build arg for the image creation, and a corresponding docker build and push are run.

recast_workflow.workflow.expand_workflow(workflow_path: pathlib.Path, toplevel_path: pathlib.Path, versions: Dict[str, str])

Returns the workflow with all ref replaced by .yml files in the local folder. Version number will be specified in this process. :param versions: the dict containing all version numbers :param workflow_path: the workflow file :param toplevel_path: the local folder :return: the workflow with all ref replaced by .yml files in the local folder

recast_workflow.workflow.get_inputs(workflow, path='') → List[str]

Find all inputs to a yadage workflow with no encapsulated workflows. :param dictionary that represents workflow: :param Or path to workflow .yml file:

Returns:List of names of all parameters that come from the ‘init’ stage in yadage
recast_workflow.workflow.make_name(wf_dict)

Creates name for workflow by combining names of all steps :param dictionary that represents workflow:

Returns:Underscore-seperated list of names of steps
recast_workflow.workflow.make_subworkflow(step: str, subworkflow_name: str, environment_settings: Dict[str, str]) → Dict[KT, VT]

Creates a yadage workflow for the given subworkflow with the given environment settings. If make.py exists in the corresponding subworkflow directory, it is run with the assumption that it will handle making the subworkflow. Otherwise, workflow.py in the subworkflow directory is expanded using jsonref and any environment variables enclosed in braces are replaced with the corresponding value in environment_settings. :returns: A dict representing the contents of a yaml file that specifies a yadage workflow.

recast_workflow.workflow.make_workflow(steps: List[str], names: List[str], environment_settings: List[Dict[str, str]]) → Dict[KT, VT]

Creates a yadage workflow using the given subworkflow specifications. :param steps: A list of the subworkflow steps for each subworkflow. :param names: A list of the subworkflow names for each subworkflow. :param environment_settings: A list of dicts mapping environment setting names to values for each subworkflow.

Returns:A dict representing the contents of a yaml file that specifies a yadage workflow.

scan.py

recast_workflow.scan.build_multi(single_wf: dict, multi_params: List[str], name='') → dict

Convert single stage workflow to multistage workflow given scan parameters.

recast_workflow.scan.fill_inputs(inputs_path, field, folder_path)

Fills field in inputs.yml with list fo files in given folder_path. Returns list of files in given folder_path.

recast_workflow.scan.get_multi_params(tmpl_path: str)

Get multiparameters from template file.

recast_workflow.scan.make_inputs(tmpl_path: str, output_dir_path: str, multi_params: Dict[str, List[T]], prefix='')

Generate input files from template formatted with proper multi param values. multi_params is a dict that maps multi_params name to list of values.

recast_workflow.scan.make_reana(wf_path, output_path)

Make reana.yml for workflow at wf_path with outputs at output path.


Indices and tables