Learning Center

Data Factory variables ​

Overview ​

Data Factory enables you to define variables that can store and reuse values across various tasks and jobs. These variables can be set at the account level, overridden at the job level, or even created dynamically during job execution.

Types of Variables ​

Global Variables ​

Global variables are defined at the account level and are accessible across all jobs and tasks within that account. They are ideal for storing shared values like API keys, database connection strings, or other configuration settings. You can define global variables in the Data Factory settings. If a global variable is redefined within a job, the job-specific value will take precedence. To override a global variable, edit the target job and navigate to the "Inputs and variables" section.

Global variables can be used in jobs as shown below:

json
{
    "schema": "1.0",
    "key": "example",
    "title": "Example",
    "tasks": [
        {
        "name": "protocol-http-perform",
        "taskReferenceName": "protocol_http_perform",
        "description": "",
        "optional": false,
        "type": "SUB_WORKFLOW",
        "inputParameters": {
            "domain": "my-api.example.com",
            "method": "GET",
            "headers": {
                "Authorization": "Bearer ${workflow.globalVariables.my_variable}",
                "Content-Type": "application/json"
            }
        }
    }
    ]
}

Here, ${workflow.globalVariables.my_variable} is replaced with the value of my_variable defined in the Data Factory settings.

Limitations ​

  • A global variable can only store a single value; it cannot hold arrays or objects. To work around this, you can use a JSON string and parse it with a JQ task.
  • Global variables can be used in templates, where they are replaced with their values before the template is processed (e.g., Bearer ${workflow.globalVariables.my_variable}). However, a single template cannot contain multiple global variables. For example, ${workflow.globalVariables.my_variable_prefix} ${workflow.globalVariables.my_variable} will not work.

Local Variables ​

Local variables are created during job execution and are useful for storing values specific to a particular job run, such as task results or script outputs. They can be defined using the SET_VARIABLE task and accessed with the ${workflow.variables.*} syntax.

json
{
    "schema": "1.0",
    "key": "example",
    "title": "Example",
    "tasks": [
        {
                "name": "Set domain",
                "taskReferenceName": "set_domain",
                "type": "SET_VARIABLE",
                "inputParameters": {
                    "domain": "my-api.example.com"
                }
            }, {
            "name": "protocol-http-perform",
            "taskReferenceName": "protocol_http_perform",
            "description": "",
            "optional": false,
            "type": "SUB_WORKFLOW",
            "inputParameters": {
                "domain": "${workflow.variables.domain}",
                "method": "GET",
                "headers": {
                    "Authorization": "Bearer ${workflow.globalVariables.my_variable}",
                    "Content-Type": "application/json"
                }
            }
        }
    ]
}

TIP

  • Unlike global variables, local variables can store complex data structures like JSON objects. For example:
json
{
    "schema": "1.0",
    "key": "example",
    "title": "Example",
    "tasks": [
        {
            "name": "Set domain",
            "taskReferenceName": "set_domain",
            "type": "SET_VARIABLE",
            "inputParameters": {
                "configuration": {
                    "domain": "my-api.example.com"
                }
            }
        }, 
        {
            "name": "protocol-http-perform",
            "taskReferenceName": "protocol_http_perform",
            "description": "",
            "optional": false,
            "type": "SUB_WORKFLOW",
            "inputParameters": {
                "domain": "${workflow.variables.configuration.domain}",
                "method": "GET",
                "headers": {
                    "Authorization": "Bearer ${workflow.globalVariables.my_variable}",
                    "Content-Type": "application/json"
                }
            }
        }
    ]
}
  • If a local variable is accessed but does not exist, it will return null.

Limitations ​

  • Local variables are only available within the current job execution and cannot be reused elsewhere.
  • The size of a local variable is limited to 256 KB. Exceeding this limit will result in a task error.