I am an extremely opinionated developer.

I'm sure I'm not unique in this regard, but I am endlessly passionate about certain topics such as the management of "infrastructure", implementing data driven development and so on. The problem is, I am not unique in this regard. I work with a team of software developers all of whom take pride in their work, and accordingly have their own opinions about how best to craft the sites that we build. This can get frustrating; I would so love to simply shake some people sometimes and simply have them learn the lessons that I have so painfully learned over the years. But, alas, it's not to be.

The dynamics of power

It's tempting to think "oh, if I was in a position of power I could fix that! I would simply say to the people 'hey, implement my idea and you will see it's superiority!' and people will understand and all will be well". However, there are some flaws with this idea.

I like to think of this in terms of trying to shape a river. When you're trying to shape a river you can take one of two approaches:

  • Get a bulldozer and simply create a new river, or
  • Gradually cut away at the existing river banks until you redirect some of the river.

I am particularly passionate about infrastructure engineering. It's the most fun sort of project I work on at work (at the moment), and I read on it constantly. One of the projects I'm trying to get more people interested in is an infrastructure management project called "Kubernetes". So, I'm going to talk about a couple of ways we can introduce this stack.

Bulldozing stuff

Managers can bulldoze stuff. We've all seen this - legislative managers who decree "hey! This is how things are now". This sometimes works, and sometimes doesn't. It always comes with unexpected side effects, and will usually piss someone or other off.

Let's pretend for a minute I'm a manager. "Let's use Kubernetes!" I'd say! After all, I've used it -- what can go wrong? Well, I think it would go something like this:

  • I declare that we use Kubernetes
  • No one really knows how to use it. People struggle to get things happening quickly, and build up systems that are fragile and complex -- the things Kubernetes is supposed to help reduce.
  • Everything becomes a horrible fragile mess that no one really wants to take care of
  • Idea is dead
  • Everyone is pissed off

Cutting away at the river

As you might guess, I much prefer an approach of cutting away at the river slowly. Broadly speaking, the approach is pretty simple:

  1. Find direction you wish the river went in
  2. Cut a little bit against the bank in which you want to go, redirecting some of the stream
  3. Wait a while. Erosion will take care of the rest, and you can cut away more later if you're unhappy with the pace.

This approach is nice, because anyone can do it. In the case of cutting away a bank, it could be as simple as talking to someone -- finding out more about what their idea about these things is, and seeing if you can find a place for the thing you have passion for in their life, to make their job easier. I don't have to be a manager to have conversations! I can simply talk to colleagues. Indeed, this is how I get what I want from managers!

So, let's take a look at how we can chip away at the river boudaries by taking a look at the Kubernetes technology. This is a super complex technology, and requires significant technical investment.

Choosing your weapons

Trust

First and foremost, the most important thing that I try to maintain is that the people I work with can trust that I am carefully balancing their needs against the needs of others, including myself. It is not enough simply to have an idea and attempt to sell it to your team; we are sold things all the time, and do not buy into them. It must be that you have your teams best interests at heart, and are working towards their betterment.

I don't think there's any shortcut to this. Indeed, I'm not even sure that my team 100% trusts me! But I try to be open with the reasons I do things and genuine in the things I talk about. Maybe this is enough? I am unsure.

In the case of Kubernetes, I love critical questions of it. Questions like:

  • "Isn't this far more complicated than our current model?"
  • "How does ${X} fit into Kubernetes?"
  • "How do we debug this magic service if it breaks?"

Embracing the painful questions that shutdown my proponency has been the spawn of many fun conversations, and I think perhaps the reason that some are now looking into the technology themselves.

Listening

It's funny, now that I think about it - nothing changes because it's a problem I have. It changes because I'm able to connect to others, talk to them about how they do things and why, perhaps teach them some of the things I do and how I've been able to solve the same challenges they face, and then leaving them to stew.

I think it's easier for people to connect things to their world view, rather than try and understand yours.

In the case of Kubernetes, I love chatting to people about their sysadmin related enquiries. I have been quite fortunate to find excellent mentors in this area, of which I think Matty G was the first. Because of this, I'm able to help with a wide number of bespoke solutions that invariably turn up, and where people are interested, point them to the reading material that has been helpful to me.

This material invariably winds up in the same place I have (after all, it's material that I've read) - Kubernetes can provide a framework around which to solve some of these problems. But people must discover this in their own time, for their own problems.

History

We receive a bizarre amount of advise every day. Much of it is nonsense, and all of it is spoken authoratively. Indeed, this blog post might be nonsense! It's on you to decide how much is applicable to you -- these are just lessons that have worked for me.

But implementing some technology in an area, and then being able to talk about it, answering questions and discussing limitations has been super key to helping people to discover some of the same solutions I have. In the case of Kubernetes, there are two places in which I hope I can provide some guidance on this project:

  • We have several internal services all running on Kubernetes, and
  • I have built and delivered CI/CD successfully in the "old" VM based mechanism, and am still a proponent of this technologies.

Honestly, I don't know how much of an impact this has made. I find it useful to relate the stories I tell to concrete things that they can see, though.

Managing expectations

It takes a very long time

The problem with cutting away at a river is that it takes a really, really, reaaaaaallly long time. In the case of Kubernetes, I have been playing with it sice the 1.0 version - July 21st, 2015. In the two years I've been playing with it, it's never been used for a customer system.

However, it's getting there. Lots of the ideas it espouses are coming to fruition:

  • Infrastructure as code
  • Continuous integration / and delivery
  • Immutable infrastructure

As well, it's used now for several internal systems such as our wiki, Sitewards website and monitoring based on Prometheus. My colleagues (<3) now have some experience setting it up and using it, though it would be a few more hours before they were proficient. I am confident that it is a good fit for some of our clients, and I am working to make it a better fit than our existing tooling for the rest.

It rarely happens how you expect it to

If I had envisioned that when we started discussing these things a year ago that I'd have specialised in building environments from Ansible, I think I would have been quite annoyed. However, now that I am here, I am happy. The conversations that we need to have to adopt this technology are not the ones that I thought we did, and the infrastructure we have is a result of the ones that we've had.

I hope to continue the journey, and I hope it ends with Kubernetes. But as of now, our infrastructure is excellent - significantly better than it has been in years past, and continually improving. Coming back to our previous metaphor, I dug into the river and it went off wildly into a direction I didn't anticipate - but still one that I like.

In conclusion

People sometimes think they need to be in a position of power to make change. However, this is not usually the case. More often, making change is about having many small conversations, being generous with your time and honest and upstanding with your answers.

Or at least, this has worked for me.