5. Set up a pipeline

After you've done creating and polishing the models, you can finally wrap the workflow in a pipeline and create a job to schedule the execution of the workflow. This ensures your transformations are reproducible and reliable every time they run.

Publish the models

Before creating a pipeline from the models, you need to publish them. Publishing models or any assets before using them in pipelines ensures reproducibility, makes dependencies clear, and enables parallel development across teams.

Open each model in the editor and follow these steps:

1. In the top-right corner, click Publish.

   

2. Provide a version number and description.

   For the first publish, we'll leave it as default.

   

3. Click Publish.

The models are now ready for use in a pipeline.

Create a pipeline

Follow these steps:

1. Switch to the Pipelines tab.

2. Click on the + icon and select Creating modeling pipeline.

   
3. Provide the name for your pipeline.

4. Select the models that the pipeline should contain.

   Here we add the three models set up in the previous step.

   
5. Click Confirm.

   The new modeling pipeline is then added under the Models folder and presented in the DAG view.

   
6. Click Publish.

Create a job

Now that the pipeline is created and published, you can create a job to schedule execution time. Follow these steps:

1. Open the pipeline that you've created.

2. Switch to the Jobs tab and click + Create a job.

3. Provide the configuration for the job.

1. Detail

Provide the name, select the target environment, and customize the associated model variables.

Since we've executed and verified the models using the console. We now can select the Production environment to run the transform against the production database.

2. Schedule

Provide the time schedule for the job and the trigger type:

• Standard setup: trigger the run at a specific time between certain intervals.

• Advanced setup: specify the schedule in crontab format, for those familiar with the cron scheduler. See: Cron.

• Manually trigger: no scheduling and can be manually triggered in the Pipeline Health dashboard. See: Pipeline Health.

3. Notification

Here you can configure notifications on job failure. By default, notifications are sent by email.

1. Click Create.

The new job will be displayed in the Jobs section. Recurve will execute the job at the scheduled time.

You can the execution status and progress of the job in the Pipeline health dashboard.

Congratulations! You have completed building a data workflow with Recurve.

For the next step, you can explore in detail each feature and module:

Connections

Learn about the available connections

Data modeling

Learn about the core assets in data modeling

Health tracking

Explore the health tracking dashboard