Have you ever planned a party with friends and ended up with five bags of chips but no drinks, or two people buying the same item from different brands? This kind of overlap can also happen in development practices, where teams might work on the same feature in different branches, leading to conflicts when merging their work.
Collaborating effectively from the start, like shopping as a group, makes communication smoother and more efficient, and can even be enjoyable. Both Trunk-Based and GitFlow approaches provide ways to manage collaboration and prevent these issues. Let’s explore how these approaches compare, especially in the context of data projects.
Data development can be complex. Compared to software development, there is a plethora of unique challenges. It involves managing not only the code itself but also data structures, workflow dependencies, and data quality. Data lake houses must account for changing schemas and the flow of data through complex pipelines, making version control, collaboration, and validation more intricate. Large-scale changes, such as schema modifications and updates to data load logic, must be carefully managed to maintain data integrity, performance, and avoid disruptions to existing ELT processes potentially causing costly data reloads.
There are two common approaches to managing the development process.
Popular in software development, GitFlow is a traditional branching model where data engineers work on multiple isolated feature branches to develop new data pipelines, models, or transformations. These branches are merged back into the main data codebase only after thorough testing and review. Code review and merge sessions can be intense when major changes in various branches are introduced in parallel. GitFlow, designed for use with Git version control, provides a structured approach to managing the development and release of data assets, ensuring a controlled flow from development to production. This approach is particularly suited for teams with scheduled releases, where multiple features or data changes are bundled into a single deployment.
Trunk-based development is a version control strategy that promotes a streamlined workflow where all engineers commit their changes directly to a single, shared branch, known as the trunk. It integrates frequent changes into a single main branch, reducing complexity, fostering collaboration, and keeping the codebase continuously deployable.
Both approaches are popular and valid. However, before selecting a particular approach, one should carefully consider the associated effort required to keep the main branch alive and kicking. Merge conflicts in data development can be especially challenging, as they often involve complex schema changes or workflow dependencies. If not resolved carefully, these conflicts can result in data corruption, broken pipelines, or significant rework.
The GitFlow method presents challenges in the context of data development. The process of isolating work in long-lived feature branches can delay feedback and slow down the incorporation of new data-driven insights. Additionally, merging changes from various branches can introduce conflicts in data models or schemas, especially if parallel development on related data sets occurs. This can lead to delays in release cycles, reduced agility, and potential quality issues if not managed carefully.
Trunk-based development, on the other hand, helps minimize merge conflicts by encouraging small, frequent updates to a single branch. Frequent integration of changes reduces the risk of large, conflicting updates building up over time, and when conflicts do occur, they are typically smaller and easier to resolve, preventing major disruptions. But this obviously requires more coordination between data engineers. Working in isolation for long stretches of time is difficult.
You could obviously adopt trunk-based practices within a GitFlow setup by committing directly to the main trunk, thus minimizing or avoiding the use of branches and enforcing frequent merges.
A single trunk branch fosters a culture of communication, collaboration and shared responsibility, where all team members are aware of ongoing changes and can provide feedback early in the development process, resulting in continuous knowledge sharing and frequent code reviews. Visibility into others' work prevents silos, improves decision-making, and cultivates a more agile environment, enabling quicker responses to evolving needs.
In contrast, GitFlow structures work into multiple long-lived branches, which can make communication more challenging as teams work in isolation for extended periods. While GitFlow can offer flexibility for managing complex releases, it can also lead to integration difficulties and less frequent feedback, potentially slowing down the development process.
The decision between trunk-based development and GitFlow is a crucial architectural choice that profoundly impacts team workflows and how code is integrated. Trunk-based development encourages frequent, small changes to a single branch, promoting collaboration, faster integration, and alignment with DataOps principles. GitFlow, on the other hand, allows for multiple simultaneous branches, providing flexibility, but in data projects, this can lead to challenges during merging, reduced visibility, delayed feedback, and increased complexity in managing data dependencies.