Data Versioning for Reproducibility, Auditability and Testability

The audit story we want

Our platform executes automated workflows on behalf of our clients, and every one of those runs needs to be explainable after the fact. So I want to make sure that we create an audit story consistent with our idea of being able to track every change, audit which workflow was executed, and, for each workflow that we run, capture all the configurable parameters.

To do this correctly, we need to version the workflow definition, we need to version the config parameters that go in each task and we want to version the task itself. The task is versioned for us in the git repository — so we will table that for this discussion and focus on workflow versioning and config versioning and how they interplay with each other.

Why data versioning matters

Before going further, we want to establish why data versioning is important and what it gives us. Data versioning gives us the ability to know what inputs went into which process and to reproduce any error that we have seen. Sometimes, we also want to know what our agents (or, for that matter, any business process) did — this is needed for compliance purposes so that we can give this information to auditors. Typically these needs arise in financial domains. Lastly, we can use auditability to improve testability: we can do parameter isolation (freeze all but one input), change that one, and see how our system behaves.

Learnings from AWS

While working at AWS — we had very strict controls on versioning our data, code and workflow definition i.e. the process itself. I built more than one system to handle complete data versioning and auditability for large-scale data processing along with namespacing for testing — but that's a story for another day.

Here, we apply some of those learnings and build a capable but lightweight system that can achieve similar goals for a fraction of the cost.

Now that we have established that data versioning is crucial for auditability, testability and reproducibility — let's walk through how to build a system that gives us all three.

Drafts: create, test, publish

At the user level we give the user the ability to create a draft and publish the draft when they are ready or discard it when they don't need it anymore. Note that we also want the user to be able to test a draft before publishing.

A file-based versioning system

We have built a simple file-based data versioning system and we store the file pointers in a database. This is very basic and it lets us version each file in namespaced folders (or S3 path definitions that are specific to workflows) and simply refer to the latest file in a database table. In this table we create a column for version number and then every time the file changes we increment the version number.

Atomic publishes

To make sure everything is atomic, we create a draft copy first, and the user can save this draft as many times as they want. When the user presses publish — the draft is moved to a permanent location (move is an atomic disk operation — so it guarantees consistency when working on the local disk). And then we update the version table to point to the new file location. Here the file name needs to be unique — this can be a SHA hash if integrity is important or a UUID.

After the publish is successful, we can see the new version of the file with an incrementally updated version number.

What this leaves out

This is a bare-minimum system that then needs to be enhanced with:

1. Data deletion — where all the data needs to be deleted,
2. Data compaction — or garbage collection of old versions that are no longer needed

— we will skip over these concerns for now. If you are using S3 some of these come for free: S3 file versioning if you are dealing with single files, and S3 lifecycle policies for versioned data.

Testing drafts together

This covers us for having versions. Now, let's talk about testing — to cover this, we use a simple approach where we track all config and workflow drafts in a table for drafts in progress. These drafts can be viewed in their respective pages, but also can be viewed collectively so that we know what drafts (configs and workflows) are in progress at any given time. A draft workflow can refer to draft configs and they can all be tested together if desired.

We have a neat test button that allows text-mode testing and soon we will also allow simulations.