Data / Create job executions

Task in alpha phase

This Data Factory task is in phase. You can contact the Product-Live team at contact@product-live.com if you want more details and get an early access.

This task allows to create job executions. Creating a job execution triggers the execution of a job, running all its tasks in sequence.

Optionally, with the wait input, the task waits until the created job executions reach a terminal status and returns the terminal executions (including their output) in place of the creation responses. To wait for job executions created elsewhere, see the Await job executions task.

Task name:

Examples

JSON input

{
    "name": "data-job-execution-create",
    "taskReferenceName": "create_job_execution",
    "description": "Create a job execution",
    "type": "SUB_WORKFLOW",
    "optional": false,
    "inputParameters": {
        "request": {
            "method": "INLINE",
            "contentType": "application/json",
            "model": "default",
            "json": {
                "operations": [
                    {
                        "key": "a",
                        "element": {
                            "jobId": "65819ba43ad5b963658ab35d",
                            "info": {
                                "title": "Manual run"
                            },
                            "input": {}
                        }
                    }
                ]
            }
        }
    }
}

XML input

{
    "name": "data-job-execution-create",
    "taskReferenceName": "create_job_execution",
    "description": "Create a job execution",
    "type": "SUB_WORKFLOW",
    "optional": false,
    "inputParameters": {
        "request": {
            "method": "ATTACHMENT",
            "contentType": "application/xml",
            "model": "default",
            "file": "@my-request.xml"
        }
    }
}

Definition

Property	Type	Required	Description
			The name (or type) of the task
			The unique name of the task in your job. It is used to reference this task in the workflow.
			The functional description of this task in your job.
			The type of the task. It must be SUB_WORKFLOW.
	:		true: the job continues if there is an error on this task. false: the job fails.
			Input parameters of the task. See below

Inputs Parameters

Property	Type	Required	Description
			The Product-Live API key. If not provided, an automatically generated API key will be used to perform the request on behalf of the current user. If provided, the given API key will be used to perform the request.
			See below
			Today, application/json and application/xml are supported.
			INLINE: The request body is provided inline in the request object. ATTACHMENT: The request body is provided as an attachment in the request object.
			The request body as a JSON object.
			See below
			A unique key to identify the operation. This key will be associated to the element in the output.
			The element to create or update. See examples below
			Some element may require additional context to be created or updated.
			The request body as a file.
			The outputs of the request. Default: [{"mediaType": "application/json", "outputMode": "INLINE", "outputKey": "json"}].
			The media type of the output. Today, application/json and application/xml are supported.
			INLINE: The output is provided inline in the output object. ATTACHMENT: The output is provided as an attachment in the output object. INLINE is only supported for mediaType=application/json
			The key of the output. Either json or xml
			Default: default. The model/format used in the output. Today, only default is supported.
			Default: false. If true, the task fails when at least one operation is rejected by the API. If false, rejected operations are reported in the output with their error body.
			Default: false. If true, the task waits until every created job execution reaches a terminal status (COMPLETED, COMPLETED_WITH_ERRORS, FAILED, REMOVED, TERMINATED, TIMED_OUT, USER_ERROR) and returns the terminal executions in place of the creation responses. See Output with wait.
			Options of the wait. Only used when wait is true.
			Default: 15, minimum: 5. Delay between two status checks.
			Default: 3600, maximum: 86400. Overall wait budget: the task fails when it is exceeded. timeoutSeconds / pollingIntervalSeconds must not exceed 720.
			Default: true. If true, the task fails when an awaited job execution ends on a terminal status other than COMPLETED. If false, the task completes and the returned executions carry their terminal status.

contentType=application/json

• The expected elements are the entities described in the Product-Live API or the Data Factory API depending on the task.
• See examples below for more details.

One element

{
    "operations": [
        {
            "key": "a",
            "element": {
                "...": "..."
            }
        }
    ]
}

Multiple elements

{
    "operations": [
        {
            "key": "a",
            "element": {
                  "...": "..."
            }
        }, {
            "key": "b",
            "element": {
              "...": "..."
            }
        }
    ]
}

Job execution element properties

Property	Type	Required	Description
			The ID of the job to execute
			Additional information for the job execution
			A title for the job execution
			Input parameters to pass to the job

contentType=application/xml

• This XML representation follows the one used by the JSON query language of the Product-Live API.
• The expected elements are the entities described in the Product-Live API or the Data Factory API depending on the task. This task performs a full replacement of the target entity. All properties must be provided in each operation; any field omitted will be reset to its default value.
• See examples below for more details.

One element

<?xml version="1.0" encoding="UTF-8"?>
<map xmlns="http://www.w3.org/2005/xpath-functions">
  <array key="operations">
    <map>
      <string key="key">a</string>
      <map key="context"></map>
      <map key="element">
        <string key="...">...</string>
      </map>
    </map>
  </array>
</map>

Outputs

Property	Type	Description
		An object containing the result of the task. Each output key is the key of the output defined in the outputs[].outputKey input parameter.

Example

If your input parameters is as follows:

{
    "...": "...",
    "outputs": [
        {
            "mediaType": "application/json", 
            "outputMode": "INLINE", 
            "outputKey": "json_inline"
        },        
        {
            "mediaType": "application/xml", 
            "outputMode": "ATTACHMENT", 
            "outputKey": "xml_attachment"
        }
    ]
}

The output will be as follows:

{
    "result": {
        "json_inline": {
            "...": "...",
            "...": "..."
        },
        "xml_attachment": {
            "url": "..."
        }
    }
}

Json output - outputs[].outputMode=INLINE and outputs[].mediaType=application/json

• result.json is an array of objects as defined in the job executions API.
• json is the default output key but may be changed by setting the outputKey property.

XML output - outputs[].outputMode=ATTACHMENT and outputs[].mediaType=application/xml

Output with wait=true

The result output keeps the same contract, but each operation's response is the terminal job execution entity (final status, output filled by the job) instead of the just-created PENDING one:

• With the default INLINE json output, result.json is still an array of { key, response } objects.
• With outputs[].outputMode=ATTACHMENT (json or xml), outputs shapes the awaited result: result.<outputKey> is a file holding the terminal executions as returned by the Find job executions task (the { key, response } envelope is not rebuilt inside a file).
• Operations rejected at creation (with failOnError=false) keep their original error response and are not awaited. If no operation returned a job execution id, the task fails.
• A job produces files by exposing them in its own output: map the produced file in the job's outputParameters (or, if the job ends with a TERMINATE task, in that task's outputParameters, which win). The file then appears in the returned execution's output as a { "url": "..." } object.

Limits and additional notes

• When the outputMode=INLINE is selected, the maximum number of elements returned by this task is limited to 1000. If more elements are to be returned, the task ends with an error. To export up to the maximum of elements, use the limit input parameter to limit the number of elements return.

Additional examples

Create a single job execution - XML input & output

job.json

{
    "name": "data-job-execution-create",
    "taskReferenceName": "create_job_execution",
    "description": "Create a single job execution",
    "type": "SUB_WORKFLOW",
    "optional": false,
    "inputParameters": {
        "request": {
            "method": "ATTACHMENT",
            "contentType": "application/xml",
            "model": "default",
            "file": "@my-request.xml"
        },
        "outputs": [
            {
                "mediaType": "application/xml",
                "outputMode": "ATTACHMENT",
                "outputKey": "xml"
            }
        ]
    }
}

my-request.xml

<?xml version="1.1" encoding="UTF-8"?>
<map xmlns="http://www.w3.org/2005/xpath-functions">
    <array key="operations">
        <map>
            <string key="key">a</string>
            <map key="context"></map>
            <map key="element">
                <string key="jobId">65819ba43ad5b963658ab35d</string>
                <map key="info">
                    <string key="title">Manual run</string>
                </map>
                <map key="input"></map>
            </map>
        </map>
    </array>
</map>

Create many job executions - XML input & output

job.json

{
    "name": "data-job-execution-create",
    "taskReferenceName": "create_job_executions",
    "description": "Create many job executions",
    "type": "SUB_WORKFLOW",
    "optional": false,
    "inputParameters": {
        "request": {
            "method": "ATTACHMENT",
            "contentType": "application/xml",
            "model": "default",
            "file": "@my-request.xml"
        },
        "outputs": [
            {
                "mediaType": "application/xml",
                "outputMode": "ATTACHMENT",
                "outputKey": "xml"
            }
        ]
    }
}

my-request.xml

<?xml version="1.1" encoding="UTF-8"?>
<map xmlns="http://www.w3.org/2005/xpath-functions">
    <array key="operations">
        <map>
            <string key="key">a</string>
            <map key="context"></map>
            <map key="element">
                <string key="jobId">65819ba43ad5b963658ab35d</string>
                <map key="info">
                    <string key="title">Manual run 1</string>
                </map>
                <map key="input"></map>
            </map>
        </map>
        <map>
            <string key="key">b</string>
            <map key="context"></map>
            <map key="element">
                <string key="jobId">65819ba43ad5b963658ab35e</string>
                <map key="info">
                    <string key="title">Manual run 2</string>
                </map>
                <map key="input"></map>
            </map>
        </map>
    </array>
</map>

Create a single job execution - JSON input & output

job.json

{
    "name": "data-job-execution-create",
    "taskReferenceName": "create_job_execution",
    "description": "Create a single job execution",
    "type": "SUB_WORKFLOW",
    "optional": false,
    "inputParameters": {
        "request": {
            "method": "ATTACHMENT",
            "contentType": "application/json",
            "model": "default",
            "file": "@my-request.json"
        },
        "outputs": [
            {
                "mediaType": "application/json",
                "outputMode": "ATTACHMENT",
                "outputKey": "json"
            }
        ]
    }
}

my-request.json

{
    "operations": [
        {
            "key": "a",
            "context": {},
            "element": {
                "jobId": "65819ba43ad5b963658ab35d",
                "info": {
                    "title": "Manual run"
                },
                "input": {}
            }
        }
    ]
}

Create many job executions - JSON input & output

job.json

{
    "name": "data-job-execution-create",
    "taskReferenceName": "create_job_executions",
    "description": "Create many job executions",
    "type": "SUB_WORKFLOW",
    "optional": false,
    "inputParameters": {
        "request": {
            "method": "ATTACHMENT",
            "contentType": "application/json",
            "model": "default",
            "file": "@my-request.json"
        },
        "outputs": [
            {
                "mediaType": "application/json",
                "outputMode": "ATTACHMENT",
                "outputKey": "json"
            }
        ]
    }
}

my-request.json

{
    "operations": [
        {
            "key": "a",
            "context": {},
            "element": {
                "jobId": "65819ba43ad5b963658ab35d",
                "info": {
                    "title": "Manual run 1"
                },
                "input": {}
            }
        },
        {
            "key": "b",
            "context": {},
            "element": {
                "jobId": "65819ba43ad5b963658ab35e",
                "info": {
                    "title": "Manual run 2"
                },
                "input": {}
            }
        }
    ]
}

Create job execution using job ID with input parameters

job.json

{
    "name": "data-job-execution-create",
    "taskReferenceName": "create_job_execution_with_input",
    "description": "Create a job execution with input parameters",
    "type": "SUB_WORKFLOW",
    "optional": false,
    "inputParameters": {
        "failOnError": true,
        "request": {
            "method": "INLINE",
            "contentType": "application/json",
            "model": "default",
            "json": {
                "operations": [
                    {
                        "key": "a",
                        "element": {
                            "jobId": "12345",
                            "info": {
                                "title": "Automated job execution"
                            },
                            "input": {
                                "customParameter1": "value1",
                                "customParameter2": "value2"
                            }
                        }
                    }
                ]
            }
        }
    }
}

Create a job execution and wait for its completion

The task completes once the job execution reaches a terminal status; result.json[0].response is the terminal execution, whose output carries what the job exposed (e.g. a produced file).

job.json

{
    "name": "data-job-execution-create",
    "taskReferenceName": "run_export_job",
    "description": "Run the export job and wait for its result",
    "type": "SUB_WORKFLOW",
    "optional": false,
    "inputParameters": {
        "request": {
            "method": "INLINE",
            "contentType": "application/json",
            "model": "default",
            "json": {
                "operations": [
                    {
                        "key": "a",
                        "element": {
                            "jobId": "65819ba43ad5b963658ab35d",
                            "info": {
                                "title": "Export triggered by the parent job"
                            },
                            "input": {}
                        }
                    }
                ]
            }
        },
        "wait": true,
        "waitOptions": {
            "pollingIntervalSeconds": 15,
            "timeoutSeconds": 1800
        }
    }
}

A later task can then use the file produced by the export job, for example:

{
    "file": "${run_export_job.output.result.json[0].response.output.export_file}"
}

Trigger a job on another account and wait for its result

Combined with an API key of the other account (typically stored in a secret variable of a shared job), the job execution is created, runs and is polled on that account. Note that the API key grants every action on its account: store it in a secret variable.

job.json

{
    "name": "data-job-execution-create",
    "taskReferenceName": "run_partner_job",
    "description": "Run a job on the partner account and wait for its result",
    "type": "SUB_WORKFLOW",
    "optional": false,
    "inputParameters": {
        "apiKey": "${workflow.globalVariables.partner_api_key}",
        "request": {
            "method": "INLINE",
            "contentType": "application/json",
            "model": "default",
            "json": {
                "operations": [
                    {
                        "key": "a",
                        "element": {
                            "jobId": "65819ba43ad5b963658ab35d",
                            "info": {
                                "title": "Export triggered by the partner"
                            },
                            "input": {}
                        }
                    }
                ]
            }
        },
        "wait": true,
        "waitOptions": {
            "timeoutSeconds": 1800
        }
    }
}

One job per pipeline

A pipeline runs a single job at a time. The triggered job must run on a different pipeline than the job that waits for it, otherwise the waiting job blocks its own trigger until the wait times out.

Create job execution using dynamic job ID from previous task

job.json

{
    "name": "data-job-execution-create",
    "taskReferenceName": "trigger_dependent_job",
    "description": "Trigger a dependent job based on previous task output",
    "type": "SUB_WORKFLOW",
    "optional": false,
    "inputParameters": {
        "failOnError": true,
        "request": {
            "method": "INLINE",
            "contentType": "application/json",
            "model": "default",
            "json": {
                "operations": [
                    {
                        "key": "a",
                        "element": {
                            "jobId": "${workflow.input.targetJobId}",
                            "info": {
                                "title": "Job triggered from parent workflow"
                            },
                            "input": {
                                "sourceJobId": "${workflow.workflowId}",
                                "processedItems": "${previous_task.output.itemCount}"
                            }
                        }
                    }
                ]
            }
        }
    }
}