Why Automation Matters : The cloud may not be right for you, but you still have to automate!

A few days ago I tweeted this link to an article about some workloads being better suited to on-prem infrastructure.

Jared Still sent me this link.

The executive summary in both cases is, if you have defined workloads that don’t require elastic resource allocation, and you are not making use of cloud-only platforms, you might find it significantly cheaper to run your systems on-prem compared to running it in the cloud.

With reference to the first article Freek D’Hooge responded with this.

“I agree that cloud is not always the best or most cost effective choice, but I find the article lacking in what it really takes to run on-prem equipment.”

I responded to Freek D’Hooge with this.

“Yes. On-prem works well if you have Infrastructure as Code and have automated all the crap, making it feel more like self-service.

For many people, that concept of automation only starts after they move to the cloud though, so they never realise how well on-prem can work…”

I’m assuming these folks who are moving back to on-prem are doing the whole high availability (HA) and disaster recovery (DR) thing properly.

There are many counter arguments, and I don’t want to start a religious war about cloud vs on-prem, but there is one aspect of this discussion that doesn’t seem to be covered here, and that is automation.

But you still have to automate!

Deciding not to go to the cloud, or moving back from the cloud to on-prem, is not an excuse to go back to the bad old days. We have to make sure we are using infrastructure as code, and automating the hell out of everything. I’ve mentioned this before.

Of course, servers in racks are a physical task, but for most things after that we are probably using virtual machines and/or containers, so once we have the physical kit in place we should be able to automate everything else.

Take a look at your stack and you will probably find there are Terraform providers and Ansible modules that work for your on-prem infrastructure, the same as you would expect for your cloud infrastructure. There is no reason not to use infrastructure as code on-prem.

For many people the “step change” nature of moving to the cloud is the thing that allows them to take a step back and learn automation. That’s a pity because they have never seen how well on-prem can work with automation.

Even as I write this I am still in the same situation. I’m currently building Azure Integration Services (AIS) kit in the cloud using Terraform. I have a landing zone where I, as part of the development team, can just build the stuff we need using infrastructure as code. That’s great, but if I want an on-prem VM, I have to raise a request and wait. I’ve automated many aspects of my DBA job, but basic provisioning of kit on-prem is still part of the old world, with all the associated lost time in hand-offs. For those seeking to remain on-prem, this type of thing can’t be allowed to continue.

In summary

It doesn’t matter if you go to the cloud or not, you have to use infrastructure as code and automate things to make everything feel like self-service. I’m not suggesting you need the perfect private cloud solution, but you need to provide developers with self-service solutions and let them get on with doing their job, rather waiting for you.

Check out the rest of the series here.

Cheers

Tim…

Why Automation Matters : Your automation is your documentation

How many times have you been following a process defined in a knowledge base note, only to find something has been omitted, or is unclear? This may be because of empire building, laziness or more often oversight, but the result is the same. Unless your processes are well documented, you always run the risk of progress drawing to a halt when “the right person” is not present.

One of the great things about automation is, by definition, every step of the process must be defined. If person X is on holiday, you can be 100% sure all the steps to complete the automation are present.

Of course, this doesn’t stop people writing stupid, ugly and hard to understand code, but your development process should have some control over that. Even if it doesn’t, you know the answer is there. It must be there because the process works.

Does that mean you don’t need to document automations?

No. The automations should be self documenting. I don’t mean that in the sense that “my code is so good it’s self-documenting”, which is the calling card of the lazy developer. I mean that automation code in your source control system should be documented. Markdown is a quick and easy tool that allows us to easily document our code, and the good thing about it is it remains close to the code. It’s right next to it in the repository. When we change our code, we should revise our documentation where necessary. The documentation becomes a living document, rather than some 1000 page word document that nobody ever reads, and nobody updates.

But documentation sucks!

Documentation gets a really bad rap because most people are doing it wrong. They fall into one of these traps.

  • They produce too little, which means people are unlikely to find what they are looking for.
  • They produce too much, which makes it daunting to look at, so nobody bothers.
  • It’s overly formal, which is dry and boring.
  • It’s hidden, or at least separate to the code, so people might not even know it exists.

Basic pointers and how-to examples are good enough for 90% of the cases, so make these the focus of your documentation. You can always give links to more detailed documentation for those people that need a little more. The context is slightly different, but this post on Structuring Content should give you some clues about how to structure your documentation. After all, documentation is content. 🙂

Conclusion

For some companies an automation or infrastructure as code project may well be the first time in their company history that they have got everything about a process documented. That has to be a positive result for the company!

Check out the rest of the series here.

Cheers

Tim…

Why Automation Matters : Taking a step backward to move forward!

There was a small thread on Twitter today about unit testing, which I’m going to extrapolate to automation generally.

There can be a certain reluctance towards writing unit tests. I guess the thinking goes, I’m here to write code and solve problems, not waste time writing test code. Unit test quality and code coverage varies, but it’s not unusual to hear people say their unit tests have more lines of code than the code they are testing. I guess that adds to the reluctance. What some people fail to see is once the unit tests are written, they may never have to manually test that code again. If the code changes over time, the unit test may only need a few small tweaks to bring them up to date. Over the lifetime of the project, that initial investment can represent a massive saving…

This is true of many aspects of automation. Yes, you can create a new database in a few minutes by clicking some buttons on a GUI, and that’s fine when you get one request a week. What happens when developers want a new database for every test they run? Your button presses don’t scale. If instead you automated the process, you would never have to manually create a database again, and developers could build and burn databases to their heart’s content.

You could be the gatekeeper who runs scripts in the production environment, but what happens when you’re on holiday? What happens when the rate of production deployments increase? You become the bottleneck doing meaningless work. If you had helped build a deployment pipeline, those production deployments could happen automatically, with the correct governance of course.

I’ve said it before in this series, but I’ll say it again. Working in the tech industry is like swimming upriver. You can’t just stop swimming, because that means you are moving backwards. If more water is added to the river, the flow rate increases and you are overwhelmed. You have to keep trying to improve your efficiency to protect yourself against what is coming round the next bend…

I know it can be hard when you have a pressing deadline, but you really are taking a step backward to move many steps forward!

Check out the rest of the series here.

Cheers

Tim…

Why Automation Matters : It’s Not New and Scary!

It’s easy to think of automation as new and scary. Sorry for stating the obvious, but automation may be new to you, or new to your company, but plenty of people have been doing this stuff for a long time. I’m going to illustrate this with some stories from my past…

Automated Deployments

In 2003 I worked for a parcel delivery company that were replacing all their old systems with a Java application running against an Oracle back end. Their build process was automated using Ant scripts, which were initiated by a tool called Ant Hill. Once developers committed their code to version control (I think we used CVS at the time) it was available to be included in the nightly builds, which were deployed automatically by Ant Hill. Now I’m not going to make out this was a full CI/CD pipeline implementation, but this was 19 years ago, and how many companies are still struggling to do automated builds now?

Automated Installations

Back at my first Oracle OpenWorld in 2006 I went to a session by Dell, who were able to deploy a 16 node Oracle RAC by just plugging in the physical kit. They used PXE network installations, which included their own custom RPM that performed the Oracle RAC installation and config silently. The guy talking about the technical stuff was Werner Puschitz, who was a legend in the Oracle on Linux space back in the day. I wrote about this session here. This was 16 years ago and they were doing things that many companies still can’t do today.

I can’t remember when the Oracle Universal Installer (OUI) first allowed silent installations, but I’m pretty sure I used them for the first time in Oracle 9i, so that’s somewhere around the 2001 period. I have an article about this functionality here. I think Oracle 9.2 in 2002 was the first time the Database Configuration Assistant (DBCA) allowed silent installations, but before the DBCA we always used to create databases manually using scripts anyway, so silent database creations in one form or another have been possible for well over 20 years. You can read about DBCA silent mode here. Build scripts for Oracle are as old as the hills, so there is nothing new to say here. The funny thing is, back in the day Oracle was often criticised for not having enough GUI tools, and nowadays nobody wants GUI tools. 🙂

Sorry, but if you are building stuff manually with GUIs, it kind-of means you’re a noob. If consultants are building things manually for you, they are wasting your time and need to be called out on it. At minimum you need build scripts, even if you can’t fully automate the whole process. A deliverable on any project should be the build scripts, not a 100 page word document with screen shots.

Random – Off Topic

While writing this post I thought of a recent conversation with a friend. He was showing me videos of his automated warehouse. It had automated guided vehicles (AGVs) zipping around the warehouse picking up products to ship. It was all new and exciting to him. We were laughing because in 1996 I was renting a room in his house, and my job at the time was writing software for automated warehouses using Oracle on the back end. It wasn’t even a new thing 26 years ago. One of the projects I worked on was upgrading an existing automated warehouse that had already been in operation for about 10 years, with AGVs and automated cranes.

New is a matter of perception.

Final Thoughts

I’m not saying all this stuff in an attempt to make out I’m some kind of automation or DevOps thought leader. If you read my blog, you know all about me. I’m just trying to show that many of us have a long history in automation, even if we can’t check all the boxes for the latest buzzwords. Automation is not new and scary. It’s been part of the day-to-day job for a long time. In some cases we are using newer tools to tidy up things that were either already automated, or at least semi-automated. If someone is presenting this stuff like it’s some brave new world bullshit, they are really trying to pull the wool over your eyes. It should be an evolution of what you were already trying to do…

I wrote a series of posts about automation here.

Cheers

Tim…

Why Automation Matters : Why You Will Fail!

The biggest problem you are likely to encounter with any type of change is people!

People don’t want to change, even if they say they do. You would think an industry that is based on constant innovation would be filled with people who are desperate to move forward, but that’s not true. Most people like the steady state. They want to come to work today and do exactly what they did yesterday.

Automation itself is not that difficult. The difficult part is the culture change required. There is a reason why new startup companies can innovate so rapidly. They are staffed by a small number of highly motivated people, who are all excited by the thought of doing something new and different. The larger and more established a company becomes, the harder it is to innovate. There are too many people who are happy to make do. Too many layers of management who, despite what they say in meetings, ultimately don’t want the disruption caused by change. Too many people who want to be part of the process, but spend most of their time focussing on “why not” and (sometimes unknowingly) sabotaging things, rather than getting stuck in. Too many people who suck the life out of you.

It’s exhausting, and that’s one of the worst things about this. It’s easy to take someone who is highly motivated and grind them down to the point where there is no more fight left in them, and they become a new recruit to the stationary crowd.

I’ve been around long enough to know this is a repeating cycle. When I started working in tech I encountered people telling me why relational databases were rubbish. Why virtualization was rubbish. Why NoSQL is rubbish. More recently why Agile is rubbish. Why containers are rubbish. Why cloud is rubbish. Why CI/CD is rubbish. Why DevOps is rubbish. The list goes on…

I’m not saying everything “new” is good and everything old is trash. I’m just saying you have to give things a proper go before you make these judgements. Decide what is the right tool for the job in question. Something might genuinely not be right for you, but that doesn’t mean it is crap for everyone. It also doesn’t mean it might not be right for you in the next project. And be honest! If you don’t want to do something, say you don’t want to do it. Don’t position yourself as an advocate, then piss on everyone’s parade!

I’m convinced companies that don’t focus on automation will die. If you have people trying to move your company forward, please support them, or at least get out of their way. They don’t need another hurdle to jump over!

I wrote a series of posts about automation here.

Cheers

Tim…

Why Automation Matters : Dealing With Vulnerabilities

The recent Log4j issues have highlighted another win for automation, from a couple of different angles.

Which Servers Are vulnerable?

There are a couple of ways to determine this. I guess the most obvious is to scan the servers and see which ones ping for the vulnerability, but depending on your server real estate, this could take a long time.

An alternative is to manage your software centrally and track which servers have downloaded and installed vulnerable software. This was mentioned by a colleague in a meeting recently…

My team uses Artifactory as a central store for a lot of our base software, like:

  • Oracle Database and patches.
  • WebLogic and patches.
  • SQLcl
  • ORDS
  • Java
  • Tomcat

In addition the developers use Artifactory to store their build artifacts. Once the problem software is identified, you could use a tool like Artifactory to determine which servers contained vulnerable software. That would be kind-of handy…

This isn’t directly related to automation, as you could use a similar centralised software library for manual work, but if you are doing manual builds there’s more of a tendency to do one-off things that don’t follow the normal procedure, so you are more likely to get false negatives. If builds are automated, there is less chance you will “acquire” software from somewhere other than the central library.

Fixing Vulnerable Software

If you use CI/CD, it’s a much simpler job to swap in a new version of a library or package, retest your software and deploy it. If your automated testing has good coverage, it may be as simple as commit to your source control. The quick succession of Log4j releases we’ve seen recently would have very little impact on your teams.

If you are working with containers, the deployment process would involve a build of a new image, then replacing all containers with new ones based on the new image. Pretty simple stuff.

If you are working in a more traditional virtual machine or physical setup, then having automated patching and deployments would give you similar benefits, even though it may feel more clunky…

Conclusion

Whichever way you play it, the adoption of automation is going to improve your reaction time when things like this happen again in the future, and make no mistake they will happen again!

I wrote a series of posts about automation here.

Cheers

Tim…

Why Automation Matters : Reducing the Cost of Failure

Recently I watched a video called The Future of Faster Enterprises by AWS Enterprise Strategist, Miriam McLemore. I think its a really good video, even if you don’t care about AWS or cloud in general. There is a wider message there.

One of the points Miriam raised was “Reducing the cost of failure”, which sparked a conversation between myself and a colleague. When you’re trying to improve the way you work, you are going to have to try new things. Not all of those things are going to work out. The important point is you try them, see if they work for you. If they do great. If they don’t, you throw them away and move on. Reducing the cost of failure is a really important part of encouraging the culture of experimentation needed for continuous improvement.

Recently I wrote a post called you have to keep working just to stand still. Now add to that the work required to move your company forward and I think you’ll see why any barrier to progress is a problem.

So what factors affect the cost of failure? Here are a few.

  • Lack of automation. If humans are involved in providing infrastructure, it’s going to increase the time it takes to set things up (see lost time), and they will get disgruntled when you ask them to throw it away 2 hours after you’ve got it. You need to be able to build and burn kit rapidly to have any hope of experimenting. This is why the focus on the automation part of flow in DevOps is so important, for both business as usual and experimentation.
  • Bloated waterfall process. If your company expects a detailed plan of action before you so much as fart, you are going to fail. You have to be agile. I’m not using the term agile in the, “I’m too lazy to plan”, sense. I mean proper agile.
  • Time. Your company has to value progress and be willing to allocate time to it. You can’t rely on the fact Beryl and Bert go home every night and no-life their way through learning something new, so the business can reap the benefit of it for free. Yes that happens, but companies that rely on it will fail.
  • Be accepting of failure. I’m not talking about being happy to be rubbish. I’m not talking about being accepting of failure in well defined business as usual (BAU) work. I’m talking about being accepting of failure during experimentation. Not everything will work. Not everything will be the right solution for you or your company. You have to be willing to try and fail or you will fall at the first hurdle.

Check out the rest of the series here.

Cheers

Tim…

Why Automation Matters : The Series

A few months ago I decided to write a post about the lost time associated with the hand-offs between teams. It was relevant to a conversation I wanted to have, and I wanted to order my thoughts before I went into that conversation. That post accidentally became a series of posts, which I’ve listed below.

I’m not an expert at automation and I’m far from being an expert at DevOps. Theses were just a useful exercise for me, so I thought they might be of interest to other people.

I’m not sure if I’ll write any more, but if I do, I’ll add them to this page.

I’ve added an Automation category to the blog, which I’ve been using to categorise these posts, and other things like my posts about Docker and Vagrant.

Cheers

Tim…