All three are places to keep a copy of your company's data so it can be analysed without disturbing the systems that run the business. They differ in how much order they impose and when.
01Data warehouse
A warehouse stores data that has been cleaned and arranged for reporting. Structure is decided before data is loaded. It is fast, consistent and well suited to dashboards and finance, and less suited to documents, images or data whose use is not yet known.
02Data lake
A lake stores everything in its original form at low cost: tables, files, logs, documents, sensor readings. Structure is applied when the data is read. It keeps your options open, and without discipline it turns into a store nobody can find anything in.
03Lakehouse
A lakehouse keeps data in open formats on low-cost storage, as a lake does, and adds the reliability and query performance of a warehouse on top. One platform then serves reporting, data science and AI. Most new platforms are now designed this way.
04ETL and ELT
Both move data from source systems into the platform. ETL transforms data before loading it. ELT loads the raw data first and transforms it inside the platform, which keeps the original available and is simpler to change later. ELT has become the usual choice.
05How to choose
- Structured data from business systems, and the goal is reporting: a warehouse.
- Large volumes of varied data, with machine learning in mind: a lake or a lakehouse.
- Both needs, and no wish to run two platforms: a lakehouse.
- A small company with a few systems: a modest warehouse is plenty.
The platform is the easy decision. Agreeing what a customer, an order and a margin mean is the hard one, and it decides whether anyone trusts the result.
In short
- A warehouse is ordered and fast; a lake is open and cheap; a lakehouse aims to be both.
- ELT keeps raw data and transforms it later, and is now the common approach.
- Choose by the data you hold and the questions you ask.
- Shared definitions and clean master data matter more than the engine.
Questions
Does a data platform have to be in the public cloud?
No. It can run on your own servers or a private cloud. Open table formats mean the data remains readable by other tools wherever it sits.
What is master data management?
Keeping one agreed record for each customer, supplier, product and material across all systems. Without it, a data platform faithfully combines the disagreements of its sources.
Is a data platform needed before using AI?
For AI that is meant to know your business, yes in some form. A model can only use data it can reach, in a state it can trust.
Sounds like your problem?
Tell us about it. We'll say honestly whether the swarm can help, and what it would take.
Read next
- CRM, ERP, HRMS, CDP, PIM, WMS and the rest: thirty business systems explained in plain wordsBusiness systems
- Custom software or off-the-shelf: how to decide whether to build, buy or extendBuild or buy
- Custom CRM or off-the-shelf CRM: which does your sales team need?CRM
- When does a company need a custom ERP?ERP
- CDP vs CRM vs DMP: what each one stores, and which you needCustomer data
- CMS, DAM and PIM: what each one manages, and why a catalogue needs all threeContent and product data