How Suffolk Construction leverages Procore data in real-time with Toric

Overview

Suffolk is prioritizing data and analytics at the forefront of our growth strategy. As our CEO, John Fish says, “if you can’t measure it, you can’t manage it.” As we continue to grow and expand our data strategy, we are always looking for the top technology and partners to level us up. We quickly identified Toric as a perfect fit for us to help with our raw data ingestion. Before Toric, we could only retrieve Procore data at eight-hour intervals, which greatly delayed our data reporting. With Toric, we can extract Procore data and even compare the differences between data sets in minutes to effectively get real-time updates anytime there is a project change.

“Having real-time data from Toric is allowing both our internal and external stakeholders to see their data updating live, which they are extremely excited about. In addition, the Toric tool has a great user experience and is an excellent addition to my engineering workflow.”
Amanda Moulaison Data Engineer, Suffolk Construction

Challenge

Getting Procore project data using our existing solution was slow and nowhere near real-time. Additionally, the dynamic nature of the Procore API made our solution very fragile - it broke frequently and these breaks were difficult and time-consuming to debug. In addition to giving us real-time data, Toric is also reducing the workload on our engineering team.

With our original Procore data ingestion pipeline, we frequently spent several hours per week debugging issues and trying to keep up with the changing Procore API. This forced us to be reactive instead of proactive, which made it difficult to be agile with this data.

Because our original ingestion solution was such a heavy lift on our engineering team, we had to prioritize and select which Procore endpoints to ingest. As a result of this, we weren’t able to pull in any tables that weren’t absolutely required due to time constraints. Now, because Toric has eased our engineering burden, we’re able to ingest and process all available Procore data tables. This in turn allows us to present more data points in our downstream dashboards and analytics, creating richer insights for our stakeholders.

Lastly, my team and I love Toric because it’s data ingestion pipeline fits seamlessly into our existing data engineering workflow. Our pipeline leverages AWS S3 and Databricks to store and process our raw data before it lands in an AWS Redshift database to be used by our analysts and data scientists.

It was simple to set up our Toric pipelines to drop off our Procore data in S3 - just like our other existing data flows.

Solution

Switching to Toric has also not only helped our engineering team - our data analysts, data scientists, and even our stakeholders see and appreciate its benefits. We use Toric to pull data from our Procore Projects to gain real-time project data using a solution that integrates with our existing data stack and automates data extraction.

In our Toric data ingestion pipeline, we were able to implement the Diff Node feature. This functionality instantly showcases the differences in a data set across time intervals by marking each record as “new”, “updated”, or “deleted.” This makes it much easier for our analysts to track and report on the ongoing progress of any project.

Our stakeholders are enjoying seeing real-time data updates in their Tableau and Power BI dashboards and other analytics tools. In fact, the seamless and instantaneous nature of Toric data ingestion pipelines has led our team to investigate introducing new data tools to our company, such as installing interactive touch-screen displays throughout our office to display analytics and trending data.

Result

Our team’s original Procore data ingestion solution met our minimal needs by processing and delivering our highest-priority data at daily intervals. 

Since switching to Toric, we can’t imagine going back. Toric has significantly eased the burden on our engineering team, which has increased our bandwidth and allowed us to focus on new projects and integrations. This process has saved our data team valuable time, and now our organization can make better-informed decisions faster, thanks to Toric. 

Since completing our Procore data ingestion pipeline buildout, we have begun exploring the opportunity to leverage Toric for other data integrations as well, including Oracle Primavera and Revit. To learn more, visit toric.com.

                                                                     {View PDF}

                    To learn more, register for the 2/18 data leader panel feat. Jit Kee Chin

                               Chief Data & Innovation Officer at Suffolk Construction

*Authors note

Prior to joining the construction industry, I worked for a few years as a data scientist at several different pharmaceutical companies. In pharma, the data is typically very clean - highly regulated, cleanly formatted, clearly defined “acceptable ranges” - not a decimal point out of place. Truly any data scientist’s dream.

You can imagine the shock I experienced when I joined the construction industry as a data engineer and discovered the deep complexity of construction data! Endless parameters, free text fields, and a seemingly infinite trickle of data points were coming in from so many sources.