The basics of managing the release cycles of your data models using CI/CD.
Once you get more serious about Snowflake, you may want to optimize how you organize yourself and your team. No one likes to do things twice, don’t they, which includes repeating tasks you face when managing the lifecycle of projects. Mainly, if one would want to deploy models to UAT and production environments, you may meet some new concepts in Snowflake for the first time.
SDLC = software development life cycle, aka the practice of building, testing, and releasing software projects in separate environments.
This is what the process used to be.
We saw the same pattern in any other kind of data technology we used to work with before.
- The BI team wants to get a data refresh of their dev environment;
- The sysops team restores a backup of their production database;
- The sysops team runs some scripts to remove sensitive data from production;
- The BI team starts developing;
And the same pattern repeats — usually with some minor variations — for the user acceptance environment.
It’s 2021 — we can do better.
One big challenge with the — let’s call it traditional — scenario above is that it is hard to govern and inefficient in terms of what teams do with their time. After all, copying data over between servers isn’t adding much business value at all.
Snowflake — the data cloud — offers a new perspective on this practice. As we are working with fully cloud-native technology, we’re not blocked by the constraints — such as expensive storage and physically constrained technology capabilities — that made our SDLC process less efficient in the early days.
Zero-copy-cloning builds our SLDC on the fly.
In their underlying data model, Snowflake doesn’t store data as complete sets but rather as assemblies of changes that occurred over time. This is meaningful to us for multiple reasons, of which SDLC management is just one.
Timetravel — as this timeline-based approach is branded — allows us to create a virtual copy of a database, schema, or table without making a physical copy of the data set.
Neat. With this feature, we can create development and UAT environments using a simple SQL statement that replaces the whole backup/restore workflow outlined in the first paragraph.
create database my_dev_db clone my_prd_db;
GitOPS: scaling the zero-copy-cloning feature in CI/CD.
Now the ample opportunity is in automating the clone creation process when things happen during project development. Examples of such moments are :
- An engineer picks up a new chunk of work (an agile “story”) and wants to develop the functionality separately from anything anyone else is doing;
- The team prepares for a demo, and wants to merge all of their functionality into a “release candidate”;
- The product owner or client approves the work that has been delivered, and wants to promote the changes to a production environment;
All of these moments can be defined as events in a code versioning system, such as Github, Gitlab, or BitBucket.
More specifically, every level of code isolation (“my sandbox, our sandbox, my organization’s Snowflake tenant”) can be their own branch — or separately stored and managed code version — in Git’s terminology.
And here’s where everything merges. Our CI (continuous integration) product will define what script to execute once a branch is created, deleted, or some code change is submitted to one. And you don’t need to pick a fancy, expensive CI product as Github, Gitlab, and Bitbucket include their own.
By using the Git platform in a structural way, and setting clear standards for a team’s workflow, we’re able to manage the full release management process for even large Snowflake projects right out of our Git platform.
Depending on the branch type, we will use the zero-copy-clone feature to create an appropriate database object in Snowflake:
- Creating a new feature database, only accessible to the engineer who picked up a task and secured from anyone else’s development activities. Unit testing — the action of validating the quality of a single unit of work — can happen here;
- Creating a new release candidate database, where all of the individual contributions land altogether and can be validated as if it were a release to production
- Releasing tested and validated changes to production, even without any downtime at all for your data application. Even if you’re dealing with petabytes of data and complex transformations during release.
And this is just the starting point. In a real-life scenario, one would maybe link Jira to Snowflake so that every time an engineer starts working on a new agile story,
So how to take it from here?
This article is just a short introduction to how a proper release strategy, team setup, Snowflake, and just the right amount of automation at the right time can supercharge the velocity and output quality of an analytics engineering team.
A real-life scenario would start on this but add — potentially many — layers of quality control and data validation on top.
One of our most essential learnings in this domain is to keep things simple and structured, which might not be easy if you just start wandering through the forests of a modern data stack. Just get in touch if you’d want to share some thoughts on the concepts.
Stay tuned for parts 2 and 3, where we’ll dig deeper into the dynamics of automating tedious but critical jobs in our release cycles.