How to Build a Data Platform

An Introduction.

Apr 23, 2023

Introduction

Like every other introduction to a data white paper will tell you, data in organisations continues to increase in volume, complexity and value. As such, not only do you have to build a data solution that meets the needs of today, but one that can also be efficiently adapted for the future.

Not a easy challenge to meet, especially when data platforms have many other objectives, including:

Provide a Return on Investment (ROI).
Be within budget (for build and maintenance)
Be easy to extend and enhance.
Outputting high quality data, reliably.
Have data that is easy to discover and share (within its security scope).
Have appropriate security controls
Should have a software development process (or processes) that suits the Data Platform requirements and team.
Should be well documented and have well defined processes.
There should be a great user experience for consumers, developers and support.

While we’d expect that most people would agree with the above, there are a number of compromises that need to be made across these statements. For example, you don’t want sensitive data to be easily shared with the entire organisation nor do you want to spend so much money on data quality that you blow your budget.

So instead of discussing the ideal, perfect data platform, we’ll discuss later in this guide your options and tradeoffs so you can make a more informed decision.

The guide is designed to be as tech and process agnostic as possible, although we are biased towards using the “Big 3” cloud providers (AWS, Azure, GCP) and Agile delivery.