Overview
A leading global car manufacturer needed to maintain high-quality standards while producing a new vehicle every minute. This rapid production rate, achieved through a blend of human labor and robotic automation, sometimes led to defects. To ensure each car met stringent quality standards before leaving the factory, the company required a robust, near-real-time review system.
Challenge
The manufacturer needed a solution to monitor and manage defects in a fast-paced, automated production environment. The solution had to ingest, process, and report on data from multiple sources and locations in near-real-time, allowing the company to track each car’s quality and quickly address any issues.
Approach
Data Ingestion
The project team worked with internal teams to design a blueprint for extracting near-real-time data from the manufacturing execution systems (MES). These systems, which direct both robotic operations and human workers, contain vital instructions and measurements for car production. The majority of this data was stored in Oracle databases. The team used Amazon Web Services (AWS) Database Migration Service (DMS) to replicate the data into Amazon S3, providing a reliable and scalable storage solution.
Given the strict service level agreements (SLAs), it was essential to ingest data quickly. The team implemented a notification-based system to trigger micro-batch ingestion into Snowflake, a cloud-based data warehouse, as soon as new data became available in S3. This approach minimized latency and enabled prompt data processing.
Data Processing
Snowflake was chosen for its ability to handle near-real-time data streams effectively. Data transformations were managed using dbt (data build tool), which allowed for scalable and maintainable processes. A Kubernetes cluster managed and executed these transformations periodically, ensuring efficiency and scalability.
Self-Service
Given the global scope of the project, decentralizing data product development was key. Each manufacturing site defined its own data transformations using dbt, tailored to its specific needs. For reporting and visualization, the team standardized on PowerBI and Tableau, ensuring consistent data presentation across locations while allowing flexibility in local reports.
Results
The platform enabled the manufacturer to track factory performance in minutes, a significant improvement. By deploying the same blueprint across all factories globally, the company gained a near-real-time overview of quality for every car produced. The system not only improved defect detection but also enabled complex analytics on parts and vendor reliability, which were previously too complex and costly. The result was a more reliable production process, reducing defects and enhancing vehicle quality.
Conclusion
This case study highlights the power of advanced data processing technologies in a fast-paced manufacturing environment. By leveraging cloud solutions like AWS and Snowflake, combined with scalable tools like dbt, the manufacturer overcame the challenges of real-time quality monitoring on a global scale. This initiative demonstrates the value of real-time data integration in maintaining high product quality in modern manufacturing.
Lessons Learned
- Scalability is Key: Replicating and deploying a standardized blueprint across global sites ensured consistency and efficiency.
- Real-Time Data Matters: Near-real-time data ingestion and processing were crucial in maintaining high-quality standards and enabling rapid responses.
- Decentralization Empowered Local Teams: Allowing local sites to define their data products enabled tailored solutions while adhering to company standards.
