Job
Last updated
Last updated
To run a pipeline, you can create a job that describes how it should be executed. Job configuration can include:
Target environment: Choose between running the pipeline in development environment for testing, or applying it in production environment after thorough validation.
Schedule time: Select a scheduling strategy for the pipeline, or trigger it manually.
Variable settings: Configure values for variables referenced in the models
Notifications: Set up notifications for job failures via emails or third-party integrations.
During a job run, every node is wrapped inside a task - the basic unit of work that represents the operation to be performed on the asset. Tasks are executed in a specific order based on the asset's dependencies, which are retrieved the upstream and downstream relationship in the DAG.
Depending on the pipeline type, the functionality of the task may differ.
For modeling pipeline, each task performs two main steps:
Materialize the asset: execute the transformation query specified in the asset.
Run data tests: run the associated data tests of the asset.
Advanced pipeline can involves various types of operators and transformations, such as the TransferOperator for extracting and loading data, or SQLOperator for performing ad-hoc data transformation.
After a task has finished running, you can inspect the execution logs in .
By design, when a task fails, the entire job run is also marked as failed and all downstream tasks won't be executed.
When you create a job from a pipeline, the job registers only the most recently published versions of the pipeline and associated models. To ensure that all changes are included, we recommend publishing the pipeline and models before creating the job.
Follow these steps to create a job:
From the Pipelines tab, select a pipeline.
Switch to the Jobs tab and click + Create a job.
Provide the configuration for the job.
Provide the name, select the target environment, and customize the associated model variables.
Click Create.
The new job will be displayed in the Jobs section.
You can visit the Pipeline Health dashboard to monitor the execution status of the job or, as a shortcut, click on the job name to navigate to Job details.
