Worried about vendor lock-in? Think twice

There are times in life when it doesn’t feel comfortable to be locked into a specific relationship. Say you agreed to a four-year car lease and you end up not liking or needing the car during the first year. But you are stuck with it. Doesn’t feel good. Either being stuck or feeling dependent on somebody or something never feels good. This happens in personal life and of course also in business.

Vendor lock-in in cloud data warehousing

Lock-in is a frequently debated topic when it comes to IT. You might encounter lock-in with your hardware vendor, your cloud provider or your general software tools vendor. But then there is also another kind of lock-in that is frequently ignored: resource lock-in. I would argue that this sort of dependency might be just as if not more critical. Let’s face it - when it comes to designing and operating a cloud data warehouse, you need great resources. It’s not by chance that job roles are called ‘Data Engineers’. Working in this space requires specialized skills and smarts. Especially across Europe, we have been facing a shortage of these sought-after skills.

Defining resource lock-in

Most corporations are getting more & more complex. Business processes need to be adjusted faster than you can react, systems get tweaked, compromises are made and before you know it, multiple processes are held together by a handful of people who know how things work. It’s not unusual for organizations to operate systems or complex spreadsheets that are named after their owners. And it’s also not unusual that these owners are sitting on a huge pile of knowledge that only they can access. It doesn’t matter if these resources are employees or consultants. What matters is that they in effect become business critical. I once met a guy who was called back from retirement three times because nobody could do what he was able to do. And let me tell you, it wasn’t cheap for his employer.

Resource lock-in in cloud data warehousing

Resource lock-in is a potential risk for your cloud data warehouse project that you need to have on the radar screen. Here are a few tips that you can consider to avoid running into a difficult resource lock-

1. Minimize custom code: Make sure to reduce the reliance on custom coded elements in your cloud data warehouse environment. Use standards and low code approaches where possible. Debugging pages of custom code written by another person can be a harrowing experience. The more custom code you have, the higher your dependency on the people who wrote it.
2. Foster collaborative development practices: Provide your team members with a platform to jointly work on problems. Not only can this increase overall project quality but also help distribute critical knowledge early on. Encourage peer reviews and incentivize collaboration.
3. Drive project documentation: As boring as it might be to create & update documentation, it is vital to have it in place for knowledge sharing & onboarding of new team members. This includes diagrams, code documentation, deployment processes and workflows.
4. Enforce standards: Use standards and conventions where possible and do it as early as possible. It’s like your set of laws. These standards provide guidance, they increase speed and they make it easier later on to decipher what has been done. For example, naming conventions can dramatically simplify the amount of effort to understand a data model.
5. Provide the right tools: The more complex something is, and the more training resources require, the higher the risk of becoming dependent on certain people. Make sure that you provide your team members with the right set of tools. Try to keep the number of tools to a minimum. Every additional tool increases complexity. It should be possible to train new employees within days not weeks.
6. Increase visibility: Allow people to see and share the work they are doing (within the boundaries of data protection & confidentiality requirements). Avoid silos where people run off to do their own thing. Critical elements of your work should always be accessible and visible to relevant team members. Visibility breaks down barriers and creates knowledge. And it also speeds up development time by unearthing potential conflicts or problems early on.

What is the goal?

In an ideal world no person should be irreplaceable. But let’s face it – this is a pipe dream. We will always have a few key people that are able to just do a bit more than other. But that should not deter us from making the effort to reduce resource lock-in. It’s good for everybody. Most people love to work in a collaborative environment. It’s motivating to work with people who openly share knowledge and where things just work well. The same is true for well-functioning processes. Experience has shown that the higher the complexity in a job, the lower the job satisfaction.

Enter Agile Data Engine

Agile Data Engine aims to help you with this challenge. Our platform is easy to learn, it provides guardrails & standards, it helps minimize custom coding and documents your good work along the way amongst other things. Take a look at the video below – it is an example of how easy it is to make data model changes. No coding required. Everything is fully documented.