Downsides of the Modern Data Stack

Integrating, storing and analyzing data in the cloud in a flexible, scalable and cost-efficient manner using a modular, integrated architecture - this is the ideology behind the "Modern Data Stack" (MDS). The MDS uses components from several manufacturers that data teams use to build and maintain their cloud data warehouse: ELT, workflow management, CI/CD, version control, data modeling, and sometimes specific automation requirements for data modeling approaches such as "Data Vault 2.0."

The best-of-breed approach is based on absolute specialization for every niche. However, all components must be integrated, made compatible with each other and then maintained. This sometimes costs so much time and resources that it often counteracts the advantages of a modern data stack.

There are currently no statistics on whether best-of-breed approaches in IT, especially in cloud data management with a modern data stack, are now falling behind all-in-one solutions. However, there are increasing voices saying that the complexity and long-term maintenance of an MDS outweighs its advantages. You get stuck in the administration of tools instead of achieving meaningful business results.

Against this background, all-in-one platforms that combine different functions promise simplicity and easier management. As always, the choice between these approaches depends heavily on the context, i.e. the specific requirements and resources of the company.

The specific MDS criticisms

1. Acquisition costs: With a modern data stack, you acquire several specialized tools, each with a specific benefit and its own licensing fees. As a result, costs add up quickly. Different vendors have different pricing schemes, making it difficult to evaluate and control costs over time.
2. Managing vendor relationships: Using multiple tool vendors means juggling contracts, negotiations, and support requests, and ensuring all vendors are aligned with the company's evolving needs. Managing relationships is time-consuming and labor-intensive, especially when vendors change their terms, update pricing models, or introduce new features.
3. Long learning curves: Each tool has its own learning curve and the cost of training programs, certifications and workshops is high to familiarize employees with all the tools. If these are complex - as is often the case - it takes time to master them. This leads to a delay in productivity. New employees have to learn an increasingly complicated web of technologies.
4. Integration and maintenance effort: Integrating the tools is not a one-time effort. Every time a single component is updated or upgraded, improvements must be made to keep the overall system functioning. Fixing integration issues and maintaining complex environments distracts from the core task: delivering data products.
5. Difficult documentation: As an MDS grows, the complexity of its documentation also increases. It is not always up to date, gaps grow over time, making it difficult to maintain the system.
   
6. Diminishing returns: As the complexity of the stack and the overhead increase, the value of the tools gradually diminishes. If you constantly have to deal with tool updates, vendor changes and integration problems, you risk losing sight of the core goal.

All-in-one platforms do not have these disadvantages. They reduce dependencies and minimize complexity. You are ready to start immediately, do not have to spend a long time integrating/installing and reduce training effort. Project runtimes are shortened, you have better cost control and more time for the development of data products.

This post is adapted from an article our Area VP Christoph Papenfuss wrote for the German-language publication Datacenter Insider in January 2025.