Should data development and operations be kept separate?

Do you keep your development and operations separate? 

“We have too many highly qualified data engineers!” This is a sentence you'll probably never hear in the current business environment. Virtually every company that we we speak to faces a serious challenge with attracting and keeping data engineering skills. 

The traditional practice of maintaining a strict separation between development and operations teams is partially responsible for this resource pressure. But it doesn’t stop there. The approach of building separate teams creates a plethora of additional problems. Leading data organizations no longer uphold this tradition. They merge development and operations in one collaborative team. Why would they do this? Is this only about resources? No, it isn’t. This blog post explores the prevalent issues with sticking to the old way. 

Not all traditions are beneficial 

Silos are never good unless they are used for storage. Maintaining two separate teams can be problematic. Let’s explore the common issues: 

• Delayed Issue Resolution: Operations teams are responsible for maintaining the system, but without direct involvement in the development process, they may lack the context needed to quickly resolve issues. This leads to longer wait times.
• Bottlenecks: Development teams might need to wait for operations teams to deploy changes, creating bottlenecks and slowing down the entire workflow.

• Redundant Efforts: Separate teams may duplicate efforts, especially in testing and monitoring. This redundancy wastes resources and time. 
• Hand-over: Once development is done, there is a hand-over period. This requires effort from both sides without providing any immediate business benefit. 
  

• Inadequate Testing: Separate teams may not fully integrate their testing processes, resulting in inadequate testing coverage. This can lead to undetected bugs and lower overall software quality. 
• Reactive Maintenance: Operations teams might focus on fixing issues as they arise rather than proactively improving the system, leading to a reactive rather than proactive approach to maintenance. 
  
  

• Increased Overhead: Maintaining separate teams with distinct processes and tools can increase administrative and operational overhead, leading to higher costs. 
• Inefficient Resource Utilization: Resources may not be used as efficiently as possible, with some team members being underutilized while others are overburdened. 
  

• Different Priorities: Development teams typically focus on innovation and rapid delivery of new features, while operations teams prioritize stability and reliability. This can create a cultural divide where each team views the other as an obstacle to their goals. 
• Blame Game: When problems arise, separate teams might engage in finger-pointing rather than working together to find solutions. This can create a toxic work environment and lower morale. This can also lead to a lack of collaboration and shared knowledge which hinders problem-solving and innovation.

Time to make a merge? 

If you are running your data warehouse team with the traditional model, you might want to think about making a change. As discussed in this post, the separation between development and operations teams can create significant challenges that impede efficiency, slow down response times, and increase costs. Are there arguments for keeping them separate? There certainly are. For example, some organizations feel that putting the most senior resources on development while asking juniors to take care of operations can lead to better resource utilization. Our experience, however, shows that the problems typically outweigh any potential benefits. 

 To overcome these issues, leading data organizations frequently adopt methodologies like DataOps, which promotes the integration of these teams. Our team at Agile Data Engine has witnessed the benefits of the later approach. Our data warehouse automation platform provides organizations to harness the power of DataOps. We will explore the power of DataOps in another article.