How to Build a Data Platform: Cloud vs On-Premise
Or buying servers vs renting
Usually the first architectural decision you’ll make is where your data platform will be built. While I’ve personally worked in nothing but the cloud for the last 5 years, I previously worked on-premise and still see benefits in using on-premise in some circumstances, so it’s still worth doing a comparison.
The major difference between on-premise and public cloud (or any hosting provider) is you have to buy your server up front for on-premise and wait weeks or months for it to be configured and installed, whereas you can rent servers in the cloud, which will be ready in seconds. Another major difference with a large public cloud you’ll have hundreds of up-to-date managed services instantly available for use and thousands of thrid-party services that often require little configuration to use in your public cloud.
On-premise server costs will likely be cheaper (sometimes much cheaper) in the long term, especially at large scale. Though your personnel costs will likely be higher using on-premise as you’ll need a specialist team to manage the data centre. This not include paying for other costs with on-premise such as renting the building, insurance etc.
Data security is a big concern in the public cloud, as engineers that work for the public cloud can have access to your data (public clouds mitigate against this in various ways and now regulated industries like banks feel comfortable storing their data there). Hybrid clouds are common where sensitive data is kept on-premise and then anonymized and/or aggregated before being sent to the cloud.
When deciding which public cloud to build a Data Platform on, typically many choose the cloud they are already using or mostly closely align in culture (heavily Microsoft dependent companies choosing Azure, for example), as many services offered by public clouds are somewhat interchangeable in terms of features or organisations just heavily use cloud neutral services, such that public clouds just act as networking glue between services.
Large organisations have also started to adopt multi-cloud, where you operate on multiple public clouds as organisations want to spread their risk and be able to back up operations if one cloud goes down with the tradeoff being that you now have two or more clouds to maintain.
Sponsored by The Oakland Group, a full service data consultancy.
Cover Photo by Taylor Vick on Unsplash