The Challenge
A global car manufacturer produces a new car every minute. This is the result of a complex collaboration between man and machine. Sometimes, defects occur during this high-paced manufacturing process and cars should be reviewed before they leave the factory floor en route to their buyer.
The Approach
Data ingestion
With several internal teams, we have aligned on a blueprint to extract near-realtime data from several manufacturing execution systems. These systems instruct both robots and in-line workers and hold information about instructions and measurements. As most of this information resides in Oracle databases, we have been able to execute data replication into Amazon S3 using Amazon Web Services DMS.
Since we commit on time-sensitive SLA’s, data has to be ingested as fast as possible. Therefore, we use a notification-based system to trigger microbatch ingestion in Snowflake as soon as new data becomes available on S3.
Data Processing
As a data platform, we opted to work with Snowflake. This allows us to integrate and report on near-real-time data streams sustainably. Data transformations – and related change processes – have been developed in dbt. A Kubernetes cluster ensures the execution of the data transformation tasks on a periodical basis?
Self-service
Given the global scale of this project, the team decided to decentralize the efforts to build tailor-made data products. To define transformations, each local site uses dbt. On the front end, we align on company standards PowerBI and Tableau.
The Results
After building the platform, we are able to track factory performance in terms of quality within a timeframe of minutes. Moreover, by deploying the same blueprint all over again for all factories within the group, we are able to get a near-realtime overview of quality on every car produced globally. This enables our client to execute complex analytics on reliability of parts and vendors, one that was too complex and too expensive to run before the cloud era.
