Incremental Ingestion

Learn how to create an automation with step-by-step instructions and videos in two parts.

Last updated on

January 31, 2024

Overview

The Incremental Ingestion Import feature allows you to extract and load new and modified data from your source to your destination. This method is efficient because it only updates new information, rather than re-importing all data.

Incremental imports rely on the source table/API possessing an appropriate replication key. While certain applications offer a cursor field like 'updated_at' to serve as the replication key, there are instances where only a primary key, such as 'task_id' can be identified for this purpose.

The presence and nature of a replication key are determined by the application itself and these factors play a pivotal role in the functionality of incremental imports. To view Incremental Ingestion details for specific connectors such as ACC, Procore, and P6, visit our Connectors page and click on the connector you want to explore.

Incremental Import

When setting up an Import automation in Toric, some connectors offer the Incremental Import capability (such as ACC, Procore and P6 EPPM).

If you toggle this feature ON, the automation will only import new or updated data every time it runs. For instance, if your automation runs every night and this feature is on, it will only import data that is new or updated since the last time the automation ran (which would be the day before).

By using this feature, you can highly improve efficiency and reduce data import costs.

However, some connections, such as Procore, have limitations when using this feature. Please refer to the table below to determine any limitations your connector may have:

Connector	Limitation
Procore	Our incremental ingestion feature saves on data import costs by only looking at one key column for every endpoint, such as “updated_at” for endpoints that may be updated multiple times (such as RFIs), or “{endpoint}_id” for endpoints that are usually created once and never updated again (such as Meetings). However, Procore doesn’t always update the “updated_at” field for certain endpoints, even if some data you would pull for the endpoint has changed. For example, if you were to update the “Question” in the RFIs page in Procore, “updated_at” would not be updated, and therefore, the incremental importer would not detect any changes since it only checks for changes in the “updated_at” field. We find that these are unique edge cases, and they only apply to fields that Procore doesn’t deem important enough to even update their “updated_at” field.

Connector

Limitation

Procore

Our incremental ingestion feature saves on data import costs by only looking at one key column for every endpoint, such as “updated_at” for endpoints that may be updated multiple times (such as RFIs), or “{endpoint}_id” for endpoints that are usually created once and never updated again (such as Meetings). However, Procore doesn’t always update the “updated_at” field for certain endpoints, even if some data you would pull for the endpoint has changed. For example, if you were to update the “Question” in the RFIs page in Procore, “updated_at” would not be updated, and therefore, the incremental importer would not detect any changes since it only checks for changes in the “updated_at” field. We find that these are unique edge cases, and they only apply to fields that Procore doesn’t deem important enough to even update their “updated_at” field.

Due to the limitations of certain APIs, you may want to reconcile your data in case the Incremental Importer missed any data (see Limitations table above).

We are building an Automatic Reconciliation feature at the moment. But while you wait, you can always manually reconcile your data for any data points the Increment Importer missed by following these steps:

Manually turn the Incremental Importer OFF,
Click Update Automation,
Click Run Once,
Manually turn the Incremental Importer back ON,
Click Update Automation

Repeat the above steps once per month (or at the pace you find necessary) to reconcile your data on a monthly basis until our Automatic Reconciliation feature is live.

Or, if you’d prefer, you can simply wait until our Automatic Reconciliation feature is live, which will pick up on any data points the Incremental Importer missed.

Enable Your Automation

Make sure your automation is Disabled while you make changes. Once your changes are complete, click the toggle on the top right to Enable the automation again and let it work.

Run Your First Automation

You could use the Manual option to test an automation action pulls. Once you confirm that your Automation works you could change the type of Trigger to one of the four options in Step 4.

For all Automations other than the manual option, the Automation will run as soon as your toggle to Enable.

Automation History

Check Automation runs from the Automation History tab. Click on an automation, then click on the History tab. Scroll down to find each automation run.

Connectors with Incremental Ingestion

References

Automations

‍

Table of Contents

Product