Sources

A source is a reference to raw tables from database that you can use in your models.

In traditional SQL transformation, raw tables are typically referenced directly by their names. This leads to a few challenges, such as code readability and dependency management between tables. In contrast, sources introduce a more structured and modular way to reference raw tables, providing several benefits:

• Clear data lineage: Sources help you explicitly identify raw tables as external data sources. This helps establish clear data lineage, making it easier to trace data from raw tables to the transformed models.

• Data testing: Declaring raw tables as sources allows you to implement data tests. You can validate assumptions about the data quality, ensuring transformations applied later are based on trusted data.

• Improved communication: In source metadata, you can add descriptions, owners, and other information to clearly express your use cases. This creates transparency and improves communication across teams.

Create a source

To create a source, follow these steps:

1. In the Models tab, click on the + icon and select Add source.

2. In the opened modal:

   1. Select the connection type. Currently, Recurve supports referencing from a warehouse.

   2. Select the target connection. If you haven't set up the project connection, refer to: Project connections.

3. Click Next. Recurve then displays all the tables available from the target connection.

Select tables from a connection

1. Select the desired raw tables or models.

2. Click Add source.

The selected tables will then be added to the sources folder and grouped by connection name.

You can view the schema of a source by opening it in the editor. A schema contains field name (or column name), data type, and comment.

Select from a source

With raw tables organized by sources, you can reference them in a model using the {{ source() }} Jinja function. For more details, see: Jinja templating.

The function requires two arguments:

• source_name: the folder name that contains the source.

• table_name: the name of the table.

For example, raw_orders table is added under the jaffle_shop folder, we can reference as below.

select 
    id as order_id,
    customer as customer_id,
    ordered_at as order_date

from {{ source("jaffle_shop", "raw_orders") }}