Most companies will directly compare doing the work themselves versus automating the work. Naturally, the engineers performing the work could feel intimidated by this.
You’ve seen the term before: hyper automation. This refers to the automation of as many businesses, and IT processes as possible. Although many businesses find value in hyper automation, some may feel intimidated by the term. I would be remiss if I did not refer to ChatGPT in this automation era. For some, it represents the next wave in productivity, while others fear for their jobs. I have to admit generative AI seems a bit daunting at times, but the opportunity of how much we can do when used the right way, is also very promising.
Most businesses feel that automation is good and will advance business goals while creating the ability to increase output (without increasing staff). Yet, when you dig a little deeper, the conversation can get more convoluted, especially in the world of Kubernetes. There is growing awareness of what is required for a (multi-cloud) production Kubernetes set up. Not just for day 1 but also for day 2 operations. Maintaining compliance, increasing the security posture, and preventing vulnerabilities or, worse, exploits are extremely complex. In addition, there are more recent studies that show manual work leads to misconfigurations and potential vulnerabilities. This is understandable in an environment where so many configurations can be applied, and so many variables are at the hands of engineers. So far, it’s easy to understand the need for automation.
Yet, what we encounter very often is that the needs of the company are not necessarily aligned with the needs of the individual. Engineers are often looking to solve complex problems because that’s what they do, and they’re good at it. This goes for software engineers as well as platform and infrastructure engineers. The difference here is that infrastructure does not differentiate a business. Software does. Because of this, companies will be looking for ways to simplify and automate infrastructure. Engineers are fundamentally looking to do the same. However, they see themselves as responsible for building the systems to automate. So, what if there are solutions on the market to help simplify and automate?
Assessing the impact of automation
Most companies will directly compare doing the work themselves versus automating the work. Naturally, the engineers performing the work could feel intimidated by this. In these situations, the company could have an opposing interest towards the individual engineer. Because ultimately, the interest of the company prevails, while the engineer then has to combat the acquisition of a certain technology with arguments that would act as justification to continue doing the work. We often hear arguments such as; “we can’t use that technology in our specific set up” or “our current architecture and/or way of working would not align with the technology”. The challenge for managers is that the engineers often have in-depth knowledge about the environment and requirements, where these leaders don’t have that level of domain knowledge. This implies that it’s a challenge for managers and IT leaders to properly challenge the pushback toward automation and standardization.
This is especially the case within the Public Cloud and Kubernetes Ecosystem. Many companies are using these technologies to drive the same outcomes, speeding up product delivery by speeding up the software development life cycle. Anecdotally, I had a conversation with a freelance engineer a few weeks back working for a large bank setting up a system atop a public cloud. They were now almost ready to get the first app in production. He operates in a team of 5-7 people and was really proud of what they built. The work seemed tremendous, and I definitely believe he should be proud of the achievement because it appeared to be a very complex system.
However, the goal of that project was not to build cloud infrastructure. The goal was to get an app, or apps rather, in production. The downside was that this project had now been going on for three years due to its complexity and manual work. When we talked about automation and standardization with technologies available, the answer was exactly that: “The solution would not be a fit because our application has specific requirements.”
If we do the math on this project now, even before the first app is live, I doubt whether the same decision would have been made at the start. I know, the cloud world is evolving at tremendous speed, so I know that some of the current technologies were not available three years ago when this specific project started. Also, it’s hard for people to change course once they have come such a long way in what they set out to achieve.
See also: 8 Steps to Achieving Effective Hyperautomation
What fascinates me is the fact that companies that need to start the journey today are in the exact same situation, and some might make the exact same decision to build technology stacks when there are viable products out there to evaluate. Managers and leaders are then faced with the dilemma: how do I satisfy the needs of my very capable engineers that I want to keep happy and motivated, yet automate and standardize as much as possible? To drive value for the company, the question should always be what to engineer and what not to engineer.
There is not a single answer to this question. In general, I believe that most of a business’ investments and capacity should be geared toward what differentiates the business from its competition. Or, in the case of the government, does it drive value for the user? Building a complex cloud system with an even more impressive Kubernetes architecture can be an achievement, but how does that provide value for a business or governmental institution? If viable solutions exist on the market to solve a problem that does not differentiate you, consider adopting those solutions.
In the Kubernetes ecosystem, many companies are doing the same work of building the system to support automation in the software delivery process to drive the same outcomes. It almost seems normal for every company to engineer a platform or system specifically designed for that company. When specifically designed for one company, the risk is that it’s very hard to enforce standards, and by that, scale the system and optimize stability. There is also the risk of technical debt and reliance on a few engineers. Often, we see the need for a platform that automates and integrates technologies to allow for a smooth route to production for applications. All the different components to choose from in the ecosystem make this even more complex and, thus, that much more appealing to the good engineer to embark on the journey themselves, sometimes losing sight of whether the time spent on some of that heavy engineering actually delivers value for the business.
In my view, the evolution of DevOps towards Platform Engineering is a very welcome one because it implies there is an acknowledgment with businesses to centralize some of the security, compliance, automation, and operational activities that companies need to control and enforce. Secondly, it offloads some of the operational burden and cognitive load from developers. For many development teams, the activities are similar, so synergies will be gained from centralizing, allowing for development teams to spend more time on coding and driving business value.
But I have still not addressed how to address this dilemma as we see the same thing happening in these Platform Engineering teams. The goal is to deliver a secure, compliant, automated developer platform in order for developers to be more productive. In achieving that goal, the question should still be what to engineer and what not to engineer. If a company intends to automate pieces of that platform, that could be less appealing and intellectually challenging for the engineers.
The first thing could be to make sure there is a clear connection between the goal for the platform team and the goal for the company. Too often, we see engineers getting an assignment of “we need to have a platform for developers; can you build one?” The pitfall here is that platforms often get engineered inside out. So, engineers start with the foundation as opposed to the actual users. If there is a clear connection between company goals, developer goals, and thus platform goals, it becomes more apparent to engineers what is valuable for the company and thus allows them to prioritize according to the business needs. This makes the choice clearer for engineers what to take on by themselves and what readily available solutions to implement.
The second thing is to clearly document the end goal for the platform, with regards to capabilities, to get to the best developer experience. Note: in-depth conversations with developers (users of the platform) are required to understand what their needs are. By thoroughly understanding these needs, you can then truly understand the components required for success (whether this be technology or manpower).
While technologies used for mass adoption are standardized, which in fact, is useful, they still require integration and configuration to accurately match the needs of a company. Development, security, and compliance processes differ from business to business, and the technology used must reflect this. Requirements from software engineers will differ, and the software stack will differ. This implies that the tech you ‘buy’ will always be a semi-conduct and will have to be integrated into the processes of the company. Elements will have to be added to get to the required self-service capabilities for developers. Aligning company goals and talking through the value-add engineers have in achieving those goals with the end product, not the semi-conduct, can relieve the intimidation and competitive pressure from externally sourced solutions.
