The Challenge

BMW – a German manufacturer of luxury automobiles, wanted to localize their approach to aftermarket sales of spare parts. Originally initiated as a European initiative, local sales regions got access to global order management systems to plan and track the sales of spares. While global systems excel at standardizing business processes, they typically don’t cater to local management needs to understand the data within a local context.

Therefore, data coming from global processes needs to be integrated with regional insights.

The Approach

Data Governance

Tasked for one of the European regions, our team was focused on tapping into the global data lake through a structured data access and governance platform, allowing our team to get access to order data with minor delays. Since the entire platform is built on AWS technology, we opted to tap into the AWS Glue Data Catalog as a metadata catalog for the data lake.

Localizing global data

Bound by strict governance controls, we used a spoke AWS account to store and process local coming from several unique source systems, and integrate these with the global order data. We tapped deeper into the AWS ecosystem and used AWS Glue for the data transformation and replication process. Glue Jobs took care of the daily orchestration of the 50+ pipelines involved. All data were reliably stored on AWS S3.

All of the data transformation code was authored in Python, and the lifecycle was managed using Terraform.

The approach generates a local data lake, built and governed using the same principles as the global one, but on a spoke account.

Fast reporting

Since data lakes excel at the fast storage of new data, at the cost of slow set-based operations, we introduced one final component in the ecosystem to speed up reporting.

Amazon Athena – a SQL engine based on the open-source Apache Presto – was used to query the data lake files. The widely used business intelligence tool of choice was added as a client application, ensuring that business analysts had localized insights with minor latency at their disposal.

The architecture

The Results

After a 16-week project, analysts had local insights at their disposal, including a solid and auditable change management system to track changes in definitions and transformations of data.