Data
What is the difference between Data Tables and Data Warehouse?
These two features are functionally similar but have a very different usage pattern. Both can be used as a way to write data from unstructured format (APIs) to a table structure. The key difference is in the access patterns.
For thorough information on Data Tables and Warehouses, please access the links outlined below.
Data Tables
Data tables are shared tables used to process data in flight. For example when building replication or re-conciliation workflows it’s sometimes useful to write IDs on a table to cross reference data, and determine fields that need to be re-written to our final destination. Data tables can be used from within flows for read-write operations, select, distinct, and summarize nodes.
Data Warehouses
Data Warehouses are designed as a final storage location. A Data Warehouse is a physical instance of a database deployed and managed by Toric. Data Warehouses are intended to be used to store final clean data thats used for reporting, or synching.
Toric Data Warehouses allow users to control the region for deployment, backup region including US, APAC and EMEA regions. With Data Warehouses user create tables with custom schemas supporting Primary keys, Dates, Currencies, Numbers Units of measure, Lists, Boolean even BIM models. Like Data Tables, Data Warehouses can be read/write from flows using the Write to Warehouse, Select, Distinct and Summarize nodes.
Externally managed data warehouses
Toric also supports externally managed Data Warehouses that can be connected to the platform. The same functionality applies however Toric cannot create tables in external warehouses, and the types of data that can be written are limited by that warehouse technology.
What data format types does Toric support for exporting?
Toric supports a variety of data types including date-time, durations, units of length, numbers, text, long text, blobs, links, lists, multi-lists, embedded arrays in cells, and many more.
Files can be exported as parquet, .xlsx, .csv and .json files.
What are Procore quotas? How do they affect my Toric automations?
The Toric Procore connector is limited by a fixed number of API requests (or calls) to Procore over a specified period of time. This “rate limit” or “quota” is defined by Procore. Please see Procore documentation here for details on these rate limits.
Can we see data lineage? Can we perhaps use your inbuilt viz tool to create data quality monitoring dashboards?
Yes, you can build data monitoring dashboards by pushing the data to a time-based warehouse table. You can setup this in Toric with a time dimension on the table.
Data lineage is kept in two ways:
- Per snapshot, so every automation stacks up the data on a version history or the endpoint.
- In the staging area or days pipelines: you can see each sync and navigate back in time using different catching criteria’s like “group by” “sync” or “group by date”. Each lake item is presented raw and can be processed at any point in time.
