Data Factory Jobs API
Introduction
A job is the definition of a Data Factory "workflow". It defines the sequence of tasks within it, has a name and a description and is attached to an account.
A job is distinguished from its execution. We talk about job when it comes to designating the definition of a Data Factory workflow. We will rather speak of job instance when it comes to designating the "run" of a job.
The Data Factory Jobs API Swagger definition is available on our Product-Live API portal here.
General information
- If a task inside the job definition takes an input a file, the file must be uploaded to the server before the job can be created using the Files api
- If a file is used inside a job, it is from now on considered as a permanent file (i.e the file uploaded with the API
/v1/data_factory/filesare considered ephemeral and are deleted after 90 days) - Rate limit: 50 requests per minute
Create a Data Factory Job
Request URL
Path parameters
None
Request parameters
None
Request Headers
| Header | Type | Description |
|---|---|---|
| accept | string | */* |
| Content-Type | string | application/json |
| X-Api-Key | string | Your account API key |
Request Body
json
{
"pipelineId": "633bfeb86bddbde9262ad1bf",
"projectId": "633ec1f0829f993dedc288eb",
"periodicity": [],
"key": "table_import",
"title": "Import table",
"status": "DRAFT",
"tasks": [
{
"name": "table-import-schema",
"taskReferenceName": "Import table",
"type": "SUB_WORKFLOW",
"input": {
"request": {
"url": "http://api.product-live.com/v1/data_factory/files/70923562084497906fe165aa6ff2629f2d1e61d39d3e017709f7ac7722ddd2be/content",
"filename": "table-import-schema.xml"
},
"mode": "EXHAUSTIVE"
}
}
]
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
Request Example
bash
curl -X 'POST' \
'https://api.product-live.com/v1/data_factory/jobs' \
-H 'accept: */*' \
-H 'X-Api-Key: <REDACTED>' \
-H 'Content-Type: application/json' \
-d '{
"key": "table_import",
"title": "Import table",
"status": "DRAFT",
"periodicity": [],
"pipelineId": "633bfeb86bddbde9262ad1bf",
"projectId": "633ec1f0829f993dedc288eb",
"tasks": [
{
"name": "table-import-schema",
"taskReferenceName": "Import table",
"type": "SUB_WORKFLOW",
"input": {
"request": {
"url": "http://api.product-live.com/v1/data_factory/files/70923562084497906fe165aa6ff2629f2d1e61d39d3e017709f7ac7722ddd2be/content",
"filename": "table-import-schema.xml"
},
"mode": "EXHAUSTIVE"
}
}
]
}'1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
ts
1
Response Example
json
{
"object": "job",
"id": "65819ba43ad5b963658ab35d",
"createdAt": "2023-12-19T13:33:24.809Z",
"projectId": "633ec1f0829f993dedc288eb",
"tasks": [
{
"name": "table-import-schema",
"taskReferenceName": "Import table",
"description": "",
"type": "SUB_WORKFLOW",
"optional": false,
"input": {
"request": {
"url": "http://api.product-live.com/v1/data_factory/files/70923562084497906fe165aa6ff2629f2d1e61d39d3e017709f7ac7722ddd2be/content",
"filename": "table-import-schema.xml"
},
"mode": "EXHAUSTIVE",
"eventCorrelationId": "IC6TmGUY:Import table:0",
"context": "${workflow.input.input.context}"
}
}
],
"pipelineId": "633bfeb86bddbde9262ad1bf",
"key": "table_import",
"periodicity": [],
"status": "DRAFT",
"title": "Import table"
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
Get a Data Factory Job by Id
Request URL
Path parameters
| Parameter | Type | Description |
|---|---|---|
| jobId | string | A job id |
Request parameters
None
Request Headers
| Header | Type | Description |
|---|---|---|
| accept | string | */* |
| X-Api-Key | string | Your account API key |
Request Body
None
Request Example
bash
curl -X 'GET' \
'https://api.product-live.com/v1/data_factory/jobs/65819ba43ad5b963658ab35d' \
-H 'accept: */*' \
-H 'X-Api-Key: <REDACTED>'1
2
3
4
2
3
4
ts
1
Response Example
json
{
"object": "job",
"id": "65819ba43ad5b963658ab35d",
"jobAccountId": "2492",
"createdAt": "2023-12-19T13:33:24.809Z",
"projectId": "633ec1f0829f993dedc288eb",
"tasks": [
{
"name": "table-import-schema",
"taskReferenceName": "Import table",
"description": "",
"type": "SUB_WORKFLOW",
"optional": false,
"input": {
"request": {
"url": "http://api.product-live.com/v1/data_factory/files/70923562084497906fe165aa6ff2629f2d1e61d39d3e017709f7ac7722ddd2be/content",
"filename": "table-import-schema.xml"
},
"mode": "EXHAUSTIVE",
"eventCorrelationId": "IC6TmGUY:Import table:0",
"context": "${workflow.input.input.context}"
}
}
],
"pipelineId": "633bfeb86bddbde9262ad1bf",
"key": "table_import",
"periodicity": [],
"status": "DRAFT",
"title": "Import table"
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
List Data Factory Jobs
Request URL
Path parameters
None
Request parameters
| Parameter | Type | Description |
|---|---|---|
| page | integer | Optional The page number to retrieve. Default value if missing : 0 |
| size | integer | Optional The number of items per page. Default value if missing : 10 |
Request Headers
| Header | Type | Description |
|---|---|---|
| accept | string | */* |
| X-Api-Key | string | Your account API key |
Request Body
None
Request Example
bash
curl -X 'GET' \
'https://api.product-live.com/v1/data_factory/jobs?size=1&page=0' \
-H 'accept: application/json' \
-H 'X-Api-Key: <REDACTED>'1
2
3
4
2
3
4
ts
1
Response Example
json
{
"object": "list",
"data": [
{
"object": "job",
"id": "65819ba43ad5b963658ab35d",
"jobAccountId": "2492",
"createdAt": "2023-12-19T13:33:24.809Z",
"projectId": "633ec1f0829f993dedc288eb",
"tasks": [
{
"name": "table-import-schema",
"taskReferenceName": "Import table",
"description": "",
"type": "SUB_WORKFLOW",
"optional": false,
"input": {
"request": {
"url": "http://api.product-live.com/v1/data_factory/files/70923562084497906fe165aa6ff2629f2d1e61d39d3e017709f7ac7722ddd2be/content",
"filename": "table-import-schema.xml"
},
"mode": "EXHAUSTIVE",
"eventCorrelationId": "IC6TmGUY:Import table:0",
"context": "${workflow.input.input.context}"
}
}
],
"pipelineId": "633bfeb86bddbde9262ad1bf",
"key": "table_import",
"periodicity": [],
"status": "DRAFT",
"title": "Import table"
}
],
"totalElements": 1
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
Update a Data Factory Job
Request URL
Path parameters
| Paramter | Type | Description |
|---|---|---|
| jobId | string | The job id |
Request parameters
None
Request Headers
| Header | Type | Value |
|---|---|---|
| accept | string | */* |
| Content-Type | string | application/json |
| X-Api-Key | string | Your account API key |
Request Body
json
{
"pipelineId": "633bfeb86bddbde9262ad1bf",
"projectId": "633ec1f0829f993dedc288eb",
"periodicity": [],
"id": "65819ba43ad5b963658ab35d",
"key": "table_import",
"title": "Import table",
"status": "PRODUCTION",
"tasks": [
{
"name": "table-import-schema",
"taskReferenceName": "Import table",
"type": "SUB_WORKFLOW",
"input": {
"request": {
"url": "http://api.product-live.com/v1/data_factory/files/70923562084497906fe165aa6ff2629f2d1e61d39d3e017709f7ac7722ddd2be/content",
"filename": "table-import-schema.xml"
},
"mode": "EXHAUSTIVE"
}
}
]
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Request Example
bash
curl -X 'PUT' \
'https://api.product-live.com/v1/data_factory/jobs/65819ba43ad5b963658ab35d' \
-H 'accept: application/json' \
-H 'X-Api-Key: <REDACTED>' \
-H 'Content-Type: application/json' \
-d '{
"pipelineId": "633bfeb86bddbde9262ad1bf",
"projectId": "633ec1f0829f993dedc288eb",
"periodicity": [],
"id": "65819ba43ad5b963658ab35d",
"key": "table_import",
"title": "Import table",
"status": "PRODUCTION",
"tasks": [
{
"name": "table-import-schema",
"taskReferenceName": "Import table",
"type": "SUB_WORKFLOW",
"input": {
"request": {
"url": "http://api.product-live.com/v1/data_factory/files/70923562084497906fe165aa6ff2629f2d1e61d39d3e017709f7ac7722ddd2be/content",
"filename": "table-import-schema.xml"
},
"mode": "EXHAUSTIVE"
}
}
]
}'1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Response Example
json
{
"object": "job",
"id": "65819ba43ad5b963658ab35d",
"createdAt": "2023-12-19T13:33:24.809Z",
"projectId": "633ec1f0829f993dedc288eb",
"tasks": [
{
"name": "table-import-schema",
"taskReferenceName": "Import table",
"description": "",
"type": "SUB_WORKFLOW",
"optional": false,
"input": {
"request": {
"url": "http://api.product-live.com/v1/data_factory/files/70923562084497906fe165aa6ff2629f2d1e61d39d3e017709f7ac7722ddd2be/content",
"filename": "table-import-schema.xml"
},
"mode": "EXHAUSTIVE",
"eventCorrelationId": "IC6TmGUY:Import table:0",
"context": "${workflow.input.input.context}"
}
}
],
"pipelineId": "633bfeb86bddbde9262ad1bf",
"key": "table_import",
"periodicity": [],
"status": "PRODUCTION",
"title": "Import table"
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
Upload a zip to update a Data Factory Job
Request URL
Path parameters
| Parameter | Type | Description |
|---|---|---|
| jobId | string | A job id |
Request parameters
None
Request Headers
| Header | Type | Description |
|---|---|---|
| Content-Type | string | multipart/form-data |
| accept | string | */* |
| X-Api-Key | string | Your account API key |
Request Body
| Body | Type | Description |
|---|---|---|
| file | file | A Data Factory job zip file (more details here) |
TIP
The Data Factory job zip file must contain the same files as in this example : demo_products_table.zip: a job.json job definition and all assets files in an assets folder.
Request Example
bash
curl -X 'PUT' \
'https://api.product-live.com/v1/data_factory/jobs/65819ba43ad5b963658ab35d' \
-H 'accept: application/json' \
-H 'X-Api-Key: <REDACTED>' \
-H 'Content-Type: multipart/form-data' \
-F 'file=Import_table.zip'1
2
3
4
5
6
2
3
4
5
6
Response Example
json
{
"object": "job",
"id": "65819ba43ad5b963658ab35d",
"createdAt": "2023-12-19T13:33:24.809Z",
"updatedAt": "2023-12-19T13:47:38.061Z",
"projectId": "633ec1f0829f993dedc288eb",
"tasks": [
{
"name": "table-import-schema",
"taskReferenceName": "Import table",
"description": "",
"type": "SUB_WORKFLOW",
"optional": false,
"input": {
"request": {
"url": "http://api.product-live.com/v1/data_factory/files/70923562084497906fe165aa6ff2629f2d1e61d39d3e017709f7ac7722ddd2be/content",
"filename": "table-import-schema.xml"
},
"mode": "EXHAUSTIVE",
"eventCorrelationId": "78SHec7y:Import table:0",
"context": "${workflow.input.input.context}"
}
}
],
"pipelineId": "633bfeb86bddbde9262ad1bf",
"key": "table_import",
"periodicity": [],
"status": "PRODUCTION",
"title": "Import table"
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30