Data in retail is ever-changing

In retail, data is sourced from diverse channels including online transactions, in-store interactions, and social media engagements, making it a daunting task to develop, manage, and maintain effective data p

Moreover, retail data development is tighlly linked to and dependent on various business applications—from e-commerce platforms to customer relationship management systems—introducing an additional layer of complexity.

By adopting a DataOps approach, retailers can overcome these challenges, achieving not only greater efficiency and reliability in their data operations but also unlocking new opportunities for growth and innovation.

Retail ❤️ Data (a lot)

Unlike traditional industries, retail faces unique challenges, such as

• The need to rapidly adapt to seasonal trends.
• Manage complex supply chains
• Personalize customer experiences at scale.

The evolution of data management

The role of data governance and architecture in DataOps

Approaches to managing complexity in the architecture

Understanding DataOps: data development and operations

DataOps is not about data and operations, but DevOps adopted to data development. DataOps is a development methodology and a set of processes and tools that aim to continuously provide quality data pipelines, products, and analytics in time and with predictable costs.

To understand how DataOps differs from DevOps, we need to understand how data development differs from software development.

In a way, you could oversimplify it like this: Business applications are about automating and helping with business processes and functions.

Data development is about deriving insight from the data created by the said business processes and applications.

Navigating the challenges of upstream data sets

One of the major differences is that data and analytics development is always downstream development and dependent on the underlying business applications, while software development is most commonly focused on the functionalities of a stand-alone business application.

Getting the data out and combining it with data extracted from other business applications creates an extra layer of complexity to data development.

You also have to keep in mind that data is constantly changing. In most cases, the upstream business applications do not have integrated development or test data sets, even if the production usually does, to some extent. This creates different challenges for developing the data products as they typically focus on the data content rather than the functionalities.

Understanding the required skill set

Another major difference is that working with data emphasizes a slightly different skill set than software development. As a modern data engineer, it helps to have experience and a decent understanding of programming.

However, you should have a firm grasp of SQL and relational databases and some mathematical domains such as set logic and statistics. On top of that, having the required skills and an understanding of data modeling is a huge benefit because data modeling is all about communication and semantics, which is necessary when integrating data from several sources.

That said, the required skill set of data engineers and how you put together your DataOps team heavily depends on the specific needs of your organization.

First of all, you should put some up-front effort into the data platform architecture and consider and analyze at least the following questions:

• How your data platform architecture fits your enterprise architecture?
• What kind of possible DevOps/DataOps tools your organization possibly already has in use?
• What kind of expertise is available in the market?

Individuals and interactions over processes and tools

When we state that individuals and interactions over processes and tools, we would be really careful not to take that as an instruction not to have the needed processes and tools in place.

In DataOps, tools and processes are one of the main contributors to the development.

However, a well-performing team only performs well with proper means of internal and external communications.

Even the best processes and tools won't help you if the product does not match what was expected.

There is an extra layer of communication in data development as you have to have well-established communication channels to the technical side of the business applications, but also with the subject matter experts and other relevant stakeholders.

Well-performing DataOps team should have all the processes and tools in place.

As a matter of fact, we've never seen so disciplined and process-oriented teams than well-organized and performing DataOps teams.

Working software over comprehensive documentation

Development should always focus on producing working software over comprehensive documentation, but this does not mean that the documentation is non-existent.

It means that the documentation should be on the correct level. You should remember that in a correctly organized DataOps team, there is also the 'Ops' part.

This means that whoever – within the team – is responsible for the production support or operations of the data platform should be able to sort out the possible issues or at least be able to delegate the problems to the correct address.

When you organize your Ops with circulating production support within the team, it will become in their best interest to develop good enough data products that maintaining and running the platform does not mean constantly extinguishing fires.

It also means that the documentation has to be on the correct level, or the issues bounce back to the actual developer who built it. We can – and should – expect a certain level of professionalism from the data developers and engineers.

It doesn’t mean that the documentation has to be on a level that a random IT guy can sort out the problem based on the documentation.

But a data engineer familiar with the technologies used should be.

Customer collaboration over contract negotiation & Responding to change over following a plan

Customer collaboration over contract negotiation and responding to change over following a plan go hand in hand.

The basic idea is that the team should provide the output as a service.

When required, we should be able to correct the course in the right direction without heavy contract negotiations or hanging on a plan that everyone understands is obsolete. That said, we still should have a plan and contracts and not just start doing random stuff!

Achieving agile success: time, experience, and support from the organization

It takes time and experience to get an agile team to perform well.

It also requires new thinking and actions from the surrounding organization.

If the relevant stakeholders around your product do not understand or are not able to support agile delivery, it is hard to succeed.

It usually pays off to hire an agile coach for at least a couple of months to get the team on the right track. And when you do so, don't break up the team without a good reason: it always takes time to get a new team to perform well, even when the individuals are experienced.

One tool stack does not fit all

Tools used in DevOps are commonly found in DataOps as well. We have a stack of tools in DataOps that include version control, CI/CD pipelines, automated infrastructure configuration, ETL/ELT tools, orchestration, monitoring, data lineage, and data modeling tools.

Choosing the correct tools for your DataOps team should be based on the needs of your organization and level of experience of the people, and what is expected from them. There is no singular all-encompassing answer to the tooling.

Some of the complexity of DataOps comes from the tooling. There is quite a variety of different kinds of technological solutions to organize the stack to help your DataOps team to succeed.

There is a huge amount of open-source software you can leverage to most parts of your DataOps tool stack. Your team could even develop some of the needed tools with a reasonable effort, depending on their experience and skills.

Neatly packaged SaaS/PaaS tools and platforms are also available that will combine most of the required tool stack.

The real question behind the tooling is: Buy versus build? In order to answer the question, we need to dive a bit deeper into the differences in the development and operations of software and data.

Leveraging the right mix of software and data warehouse development skills for optimal performance

Tool stack has some variation between DevOps and DataOps, but the bigger difference comes from the skill set differences and experience between software developers and data warehouse developers.

Software developers usually have much broader experience working with and developing the tool stack. It takes time to set up different environments, CI/CD pipelines, etc.

The time to set up these naturally vary based on experience working with the chosen tools. It also takes time to master all these tools and to get the development process performing.

You should keep in mind that the tools are only one part of the DataOps methodology.

The more tools you have, the more complexity it creates, and the more time it takes the team to master these.

People with data management or warehousing backgrounds tend to have less experience with these tools.

Usually, you need both software engineering and data warehouse development skills in your team, and there is demand for a solution that enables a smooth development process no matter which background the developer comes from.

Choosing and maintaining the right tools for DataOps

We can't forget the operations, either. When a business application fails, there might be various reasons behind it, like malfunctioning hardware, incorrect software configuration, insufficient resources, human error, bad design, or problems with security.

When data pipelines – or whatever kind of data products you are building – fail, the reason might be any of the above.

In addition, this can also happen due to

• Failures or changes in the business applications, as the data platform is downstream and dependent on them.
• Maintenance needs. In case you built your DataOps tool stack yourself, you must maintain it yourself in addition to the data platform.
• If you leverage SaaS/PaaS services on your stack, you outsource the responsibility to the service provider.

This, of course, comes with a (subscription) price, but then again, is the engineering work free?

If you have multiple DevOps and DataOps teams, then it might pay off to have a centralized team responsible for the tools, but as we are writing this (beginning of 2023), it seems that the demand for people on the market with expertise in DevOps tooling is much higher than the availability.

Either way, there is no definitive answer to this question. It all depends on the people, the complexity of your environment, budget, scalability, and the team’s focus, and of course, this all should be aligned with your enterprise architecture.

Combining quality management and lean

Total Quality Management is the often-referred methodology to arrange quality management in DevOps or DataOps. The idea of Total Quality Management is to involve everyone in the quality improvement process and make it a continuous part of the process in the product's life cycle. 

We will not go through the details of Total Quality management in this post.

Still, we want to point out a couple of key principles that should be applied in data development and operation quality management as well as the actual data product quality management:

• As in Lean, there should be continuous improvement and elimination of waste of the products and processes
• The optimization should be done on the whole process from design to delivery rather than just individual parts of the process.
• The decisions on the optimization should be based on data and metrics (how can you even expect to optimize something without understanding it and without proper facts?) 

These principles overlap with Lean principles, and from the Lean point of view, we should focus on value by prioritizing the backlog. We claim that quality management and lean go tightly hand in hand. In DataOps, we can analyze quality management and Lean from two perspectives:

1. What is the quality and how lean is the development process of the data products?
2. How lean are the data pipelines and what is the quality of the product?

Streamlining the development

From the development perspective, the idea is to continuously measure and analyze the process and make improvements and adjustments accordingly.

When talking about the data development process, it is rather ironic if the development processes are not actually analyzed and optimized based on data and metrics.

However, the weight on the optimization should not only be on the throughput of the team. As said, on a higher level, there is the quality present as well.

It is not the easiest task to optimize the throughput and quality in a scalable cloud environment where the resources scale and the costs follow.

As the engineering work also has a price tag, it is a delicate balance between time to market, performance, and scalability of the data pipelines and products – not forgetting about the Ops part with the maintenance costs.

In the end, what we would like to see is a productive DataOps team and a well-working data platform delivering value with predictable run and development costs.

The more standardized development patterns you have in place, the more predictable development and running costs you will have on your platform!

One solution for streamlining or speeding the development is to automate as many parts of the development process as possible. Pattern-based development methodologies and architecture make automation usually more effective.

However, you should always pay attention to the effort-gain ratio when starting to automate tasks; if you save one day of work by using ten days to automate the task, is it worth it?

Balancing with data quality

From the pipeline or product perspective, the focus should be architecturally consistent, robust, and trustworthy data pipelines and products that perform within the requirements.

And then there is data quality. Data quality can, of course, be measured and monitored as part of the DataOps, but data quality issues rarely have anything to do with data development. Mostly the quality issues of the business applications became visible on reports or analytics built on top of the data platform.

Some of the data quality issues can be addressed on the data platform, but you should always keep in mind that the correct place to fix the data quality issues is where the data is produced.

The further downstream you go, the higher cost and the less impact the fix has. From the operations perspective, these and all other types of fixes should be done in a way that repairs the whole issue, not just the symptom.