Data Factory Use Cases

Trigger a job from another job and use its result

Quick links

The job that triggers is available for download here, and its JSON representation here.

The triggered job is available for download here, and its JSON representation here.

A Data Factory job can run another job and then continue with what that job produced. This is done with a single task, data-job-execution-create, used with wait: true: the task creates the job execution, polls it until it reaches a terminal status, and returns that terminal execution - including its output - so the following tasks of the calling job can consume it. When the two jobs belong to two different accounts, the very same task carries an API key of the target account.

This use case models the following scenario. An account (the owner) shares one of its tables, and a job, with its partner accounts. When a partner runs the shared job, the job triggers a private export job on the owner account, waits for the file it produces, then imports that file on the partner's side of the same shared table. It intends to illustrate the use of tasks such as data-job-find, data-job-execution-create, table-export-items and table-import-items.

TIP

data-job-execution-create and data-job-execution-await are in alpha phase. You can contact the Product-Live team at contact@product-live.com if you want more details and get an early access.

The contract between the two jobs

Three things make the chaining work, and they are the only ones to remember.

1. The triggered job exposes its result in its own outputParameters. Here the export job publishes the file it produced under the key items_file. Nothing else of an execution is readable by its caller, so anything the calling job needs must be mapped there - see the User outputs tutorial. If the triggered job ends with a TERMINATE task, that task's outputParameters win.
2. The calling job passes the inputs and waits. Inputs are provided in the operation's element.input, and the triggered job reads them with ${workflow.input.<key>} - the export job has no userInputs of its own, it only reads eansCsv. wait: true turns the trigger into a blocking step; waitOptions sets how long and how often it polls.
3. The calling job reads the terminal execution. The exported file is handed to the request input of the import task with a single expression:

${run_export_on_owner_account.output.result.json[0].response.output.items_file}

In it, run_export_on_owner_account is the taskReferenceName of the trigger task, result.json is the array of { key, response } objects returned with the default inline JSON output, [0] is the position of the operation in request.json.operations, response is the terminal execution (it would be the freshly created, still pending one without wait: true), and output.items_file is the key declared by the triggered job.

Which account each task runs on

The two jobs run on two different accounts, and so do the tasks of the calling job:

• Tasks with no apiKey run in the context of the account running the job, here the partner. This is the case of the JavaScript task and of the import, which is why the imported items land on the partner's side of the shared table - the same shared job therefore works for every partner without modification.
• Tasks carrying an apiKey run in the context of the key's account, here the owner. This is the case of the job lookup and of the trigger, which is why the export job - private to the owner - can be found and executed at all.
• accountKey on table-import-items designates the shared-table context, not the executing identity.
• The exported file is designated by a URL that is sufficient on its own to download it. That is what lets the import task read, from the partner's context, a file produced on the owner account - and also why such URLs must be treated as confidential.

Sequence diagram

sequenceDiagram
    participant b as Partner account
    participant df as Data Factory
    participant a as Owner account
    b ->> df: Runs the shared job
    df ->> a: Resolves the export job from its key (owner API key)
    df ->> a: Creates the export job execution (owner API key)
    loop Every 10 seconds, for up to 30 minutes
        df ->> a: Checks the status of the export job execution
        a -->> df: Status
    end
    a -->> df: Terminal execution, carrying output.items_file
    df ->> b: Imports the exported file on the shared table

Prerequisites

To execute this use case, you need:

• A Product-Live account with access to the Data Factory platform
• A second account, with a table shared between the two, and the calling job shared by the owner account with it
• Two global variables on the owner account, read by the calling job as ${workflow.globalVariables.api_key} and ${workflow.globalVariables.account_key}: an API key of the owner account, and the owner's account key (Settings > Network > Partners). Variables of a shared job are resolved on the job owner's account, so a partner running the job gets the owner's values. Store the API key in a protected variable: an API key grants every action on its account. See Variables and the Variables and secrets tutorial
• Two Data Factory pipelines, one for each job

Setup

1. On the owner account, import the triggered job. Adapt its table key (PRODUCTS) and identifier key (EAN_13) to the table holding your items
2. On the owner account, import the calling job and share it with the partner accounts. The job resolves the triggered job from its key with a data-job-find task; the same resolution can be declared inline with a lookup, which removes that task altogether
3. On the owner account, create the two global variables api_key and account_key used by the calling job
4. Attach the two jobs to two different pipelines
5. From the partner account, run the calling job and enter the EANs to import, one per line

Variants

• Create now, wait later. data-job-execution-await is the standalone counterpart of the wait option. Use it when the wait must happen further down the job, or to wait for executions created elsewhere: it takes the execution ids, for example ${create_job_execution.output.result.json[0].response.id}
• Trigger several jobs at once. Add one entry per execution in request.json.operations; each one is awaited, and returned at its own index in result.json
• Trigger a job from outside Data Factory. The API counterpart is POST /v1/data_factory/job_executions

Things to watch out for

One job per pipeline

A pipeline runs a single job at a time. The triggered job must run on a different pipeline than the job that waits for it, otherwise the waiting job blocks its own trigger until the wait times out. See Pipelines.

• waitOptions.failOnError defaults to true: any terminal status of the triggered job other than COMPLETED fails the calling job. Set it to false to handle the outcome yourself - the returned executions then carry their own terminal status
• waitOptions.timeoutSeconds is the whole wait budget and the task fails when it is exceeded. This example waits up to 30 minutes, checking every 10 seconds
• That budget is consumed inside the calling job, whose own timeoutSeconds still applies: a job that waits too long is marked as timed out. See Handling failures
• A job execution is only visible to the account that created it. data-job-execution-create creates and polls within a single task, so its apiKey covers both; data-job-execution-await must be given the same key
• When the wait fails, the calling job only sees the statuses and a message. The report of the triggered job is on the account that ran it, where its execution can be opened
• Reading the result with result.json[0].response assumes the default inline JSON output. With outputs[].outputMode set to ATTACHMENT, the awaited executions are returned as a file instead

Data Factory Job details

TIP

The jobs presented here are intended to illustrate the mechanism, and can be enhanced and enriched to suit your needs. The table key, the identifier key and the default EAN value are specific to the demonstration table and must be adapted. For more information, please refer to the Data Factory platform documentation.

The job that triggers, shared by the owner account and run by the partner account:

flowchart TD
	_start(("Start"))
	_end(("End"))
	parse_eans["<b><a href='/data-factory/references/tasks/code-execute-javascript/'>code-execute-javascript</a></b><br />Parse the EAN list into an array and a comma-separated string"]
	find_export_job["<b><a href='/data-factory/references/tasks/data-job-find/'>data-job-find</a></b><br />Find the export job on the owner account, by job key"]
	run_export_on_owner_account["<b><a href='/data-factory/references/tasks/data-job-execution-create/'>data-job-execution-create</a></b><br />Run the export job on the owner account and wait for its result"]
	import_items["<b><a href='/data-factory/references/tasks/table-import-items/'>table-import-items</a></b><br />Import the exported items on the shared table"]
	_start --> parse_eans
	parse_eans --> find_export_job
	find_export_job --> run_export_on_owner_account
	run_export_on_owner_account --> import_items
	import_items --> _end

The triggered job, private to the owner account:

flowchart TD
	_start(("Start"))
	_end(("End"))
	build_export_request["<b><a href='/data-factory/references/tasks/file-transformation-xslt/'>file-transformation-xslt</a></b><br />Build the items export request, filtered on the requested EANs"]
	export_items["<b><a href='/data-factory/references/tasks/table-export-items/'>table-export-items</a></b><br />Export the items matching the requested EANs"]
	_start --> build_export_request
	build_export_request --> export_items
	export_items --> _end