Automation : Increasing pressure on an existing constraint

Yesterday I tweeted that I was reminded of this post.

I was reminded of it because of something that is happening to me at work, so I thought I would talk about it here.

Production lines

If you’ve read anything about DevOps you will know it came from manufacturing. If you didn’t know that, check out The Goal, which was the basis for The Phoenix Project.

Manufacturing typically uses production lines made up of multiple stations, where each station performs a specific task, and the product moves forward through the stations until it is complete. If one station is slower than the others, it will become a blocker. Product will start to queue up behind it, and downstream stations will become starved. So production lines only work well if they are planned to enable a consistent flow.

What’s more, you can only sell the product when it is completed, so we could describe the product as having no value until it is finished and with the customer.

It’s not just about manufacturing

The processes born out of manufacturing also work really well for other industries. I would suggest most things can be described like a production line, and as soon as you do that, a similar approach can be adopted to identify and fix the constraints.

Tech is an obvious one, and variations on DevOps have grown in popularity because of that. My sister in law works in a medical practice, and we’ve discussed the processes used in the administration side of it. As a result of our discussions she’s started to use Kanban boards to visualise the flow of work.

Back to my problem

So with the concept of production lines and constraints in mind, we jump back to my problem…

We are in the process of replacing loads of Oracle Linux 7 servers with either Oracle Linux 8 or Oracle Linux 9, depending on vendor support. The first three links in that production line are as follows.

  • VMs are provisioned.
  • Operating system customisations are run.
  • Database or app server is installed and configured.

We are not perfect, but we’ve got pretty good at this part of the production line. When we are finished, the systems have to be tested, and go through various processes to get them live and used by the business. Those parts of the production line that follow us are slow due to a number of factors. So our improvements in the production line have just made things harder for those steps that follow us. A simplified view of the Kanban board looks something like this.

The obvious thing to do here is focus on the constraints and start working on downstream links in the chain, to improve the overall flow. That’s where we hit organisation and culture barriers, so we are pretty much stumped…

Thoughts

I’m pretty happy with what we’ve done over the last few years. We’ve definitely improved several aspects of our systems because of automation, but at the same time I can’t help thinking we’ve achieved nothing because ultimately the work is not getting completed as fast as it’s started.

I wrote here about reframing the goal, and I have to do that as copium. Unfortunately copium only goes so far… 🙂

Cheers

Tim…

Answering some questions about Vagrant

Someone on YouTube asked me some general questions about my experience of Vagrant, so I thought I would write them down as a blog post.

Could you share the story of when and how you first encountered Vagrant, and how did you feel about it at the time?

I was quite late to the party. In 2017 I was at a VMware workshop in Cork, Ireland. I was sitting in the hotel and Frits Hoogland was showing me his Vagrant build for a test Oracle database. Like most things when they are unfamiliar, it seemed a little complex. He gave me access to his Vagrant repository, but I hardly looked at it. It was on my list of things to do, but there is always so much on my to-do list. When Frits talks you should listen, but unfortunately I failed that mission. 🙂

About a year later a colleague at work asked me what Vagrant was, and I struggled to give a reasonable answer. That evening I Googled it, and tried a couple of really simple builds. As someone with lots of VMs at home it totally blew my mind. From that point on I was hooked. I wrote Vagrant builds for all my test databases, so I could rebuild them whenever I wanted to. I went from never using it, to never shutting up about it overnight.

Now my PC is a lot less bloated. I don’t have to keep loads of VMs for different database versions, RAC and Data Guard etc. If I need something to do a test I just build it from scratch.

Another benefit is it makes live demos feel a lot less stressful. I remember being in my hotel in India, and a few minutes before I was due to start presenting I was having some issues with my demo VM. I just typed “vagrant destroy -f” followed by “vagrant up” and my demo system was rebuilt and I was good to go. Nightmare averted, and no need for loads of backups and snapshots.

How long does it usually takes for you to create the builds that are used to create the database?

The first couple of real builds took some time as there was a learning curve with Vagrant. Fortunately I have a lot of Oracle skills and some basic system administration skills, so it wasn’t too bad. It would have been a lot harder if I was trying to pick up several new skills at the same time.

All my Vagrant builds are based on articles, so I already know what to do. I’m basically tweaking the instructions from my articles to form the Vagrant builds.

The more vagrant builds you do, the quicker you get at doing them, because you have a repository of previous builds to pull ideas from. I’m now at the point where most new builds are slight variations of a previous build, so they are really quick to write.

How are you using Vagrant at work? I assume that some companies will require us to install the database with its options in a bare metal server, and not with VMs.

I use Vagrant at home. All my writing is based on VMs I’m running at home. Those VMs are built using Vagrant.

Automation at work is a little tricky because there is a separation of duties between virtualization, system administration and DBA teams. Configuring a complete automation is quite time consuming and political. Terraform and Ansible are more commonly used at work, but we are still on that journey. We are less DevOps and more DevHopeful. 🙂

Our cloud automations all use Terraform. Terraform is similar to Vagrant, as both are produced by Hashicorp.

The tools you use for automation are not as important as the attitude. Once you get into automation you can switch between tools a lot more easily, because you understand the approach you need to take.

How does using Vagrant help you in serving your customers?

Using Vagrant at home makes it quick to set up new environments, which allows me to learn new stuff faster. I don’t like doing anything at work unless I’ve already tried it at home. I guess me knowing more helps me do my job better, which ultimately benefits the people depending on me at work, so I guess there is an indirect relationship. 🙂

How do you learn Vagrant?

If you want to know more about Vagrant you can start here.

I find the easiest way to learn about Vagrant is to build things. There are loads of builds on the internet to use for inspiration. You can find mine here.

Cheers

Tim…

Why Automation Matters : The cloud may not be right for you, but you still have to automate!

A few days ago I tweeted this link to an article about some workloads being better suited to on-prem infrastructure.

Jared Still sent me this link.

The executive summary in both cases is, if you have defined workloads that don’t require elastic resource allocation, and you are not making use of cloud-only platforms, you might find it significantly cheaper to run your systems on-prem compared to running it in the cloud.

With reference to the first article Freek D’Hooge responded with this.

“I agree that cloud is not always the best or most cost effective choice, but I find the article lacking in what it really takes to run on-prem equipment.”

I responded to Freek D’Hooge with this.

“Yes. On-prem works well if you have Infrastructure as Code and have automated all the crap, making it feel more like self-service.

For many people, that concept of automation only starts after they move to the cloud though, so they never realise how well on-prem can work…”

I’m assuming these folks who are moving back to on-prem are doing the whole high availability (HA) and disaster recovery (DR) thing properly.

There are many counter arguments, and I don’t want to start a religious war about cloud vs on-prem, but there is one aspect of this discussion that doesn’t seem to be covered here, and that is automation.

But you still have to automate!

Deciding not to go to the cloud, or moving back from the cloud to on-prem, is not an excuse to go back to the bad old days. We have to make sure we are using infrastructure as code, and automating the hell out of everything. I’ve mentioned this before.

Of course, servers in racks are a physical task, but for most things after that we are probably using virtual machines and/or containers, so once we have the physical kit in place we should be able to automate everything else.

Take a look at your stack and you will probably find there are Terraform providers and Ansible modules that work for your on-prem infrastructure, the same as you would expect for your cloud infrastructure. There is no reason not to use infrastructure as code on-prem.

For many people the “step change” nature of moving to the cloud is the thing that allows them to take a step back and learn automation. That’s a pity because they have never seen how well on-prem can work with automation.

Even as I write this I am still in the same situation. I’m currently building Azure Integration Services (AIS) kit in the cloud using Terraform. I have a landing zone where I, as part of the development team, can just build the stuff we need using infrastructure as code. That’s great, but if I want an on-prem VM, I have to raise a request and wait. I’ve automated many aspects of my DBA job, but basic provisioning of kit on-prem is still part of the old world, with all the associated lost time in hand-offs. For those seeking to remain on-prem, this type of thing can’t be allowed to continue.

In summary

It doesn’t matter if you go to the cloud or not, you have to use infrastructure as code and automate things to make everything feel like self-service. I’m not suggesting you need the perfect private cloud solution, but you need to provide developers with self-service solutions and let them get on with doing their job, rather waiting for you.

Check out the rest of the series here.

Cheers

Tim…

Why Automation Matters : It doesn’t get much simpler than this!

Just a little story about something that happened recently…

The July Oracle quarterly security patches dropped, along with the downstream OpenJDK releases a little later. As soon as the OpenJDK release dropped I updated all our non-production Oracle REST Data Services (ORDS) middle tier to the latest and greatest. Production will follow shortly. I figured that was the end of it…

On Friday evening I noticed a post by Jeff Smith saying version 22.2.1 of SQLcl and ORDS had dropped. Now it was Friday evening, and I typically don’t work on Fridays, but I really didn’t want to have this hanging over me all weekend, so this is what I did…

  • Download the software.
  • Upload the software to Artifactory.
  • Change a couple of files in Git, specifying the new software versions.
  • Press a button in TeamCity to rebuild the ORDS Docker image and push the new image to JFrog Platform.
  • Press a button in TeamCity to redeploy all the non-production ORDS containers.

The whole process took a few minutes, and I was confident it would work because it’s all automated and works every time. I sent this message to Jeff.

I {expletive} love automation. I just noticed your post about 22.2.1 of SQLcl and ORDS. I logged on to work. Updated a couple of files in git and pressed a button in TeamCity. New ORDS Docker image built and pushed to all non-prod systems. Boom…

This is why automation matters. The combination of picking the right platform, in this case containers, and investing a little time in the automation means we can react really quickly to new releases. It doesn’t get much simpler than this!

Check out the rest of the series here.

Cheers

Tim…

PS. This post explains the application topology we use for ORDS and APEX.

Video : Ansible Playbooks : Vault

In today’s video we demonstrate how to use Ansible Vault with Ansible Playbooks.

The video is based on this article.

You might find some useful stuff here.

The star of today’s video is Ilmar Kerm, who is very vocal about automation in the Oracle space.

Cheers

Tim…

Video : Ansible Playbooks : Roles

In today’s video we demonstrate how to use role with Ansible Playbooks.

The video is based on the following article.

You might find some useful stuff here.

The star of today’s video is Anton Els, who I first met when he worked for DBvisit.

Cheers

Tim…

Video : Ansible Playbooks : Users and Groups

In today’s video we demonstrate how to manage users and groups with Ansible Playbooks.

The video is based on the following article.

You might find some useful stuff here.

The star of today’s video is Connor McDonald of AskTom fame. You can check out his YouTube channel here.

Cheers

Tim…

Video : Ansible Playbooks : Tags


In today’s video we demonstrate how to use handlers in Ansible Playbooks.

The video is based on the following article.

You might find some useful stuff here.

The star of today’s video is Tanel Poder, who thinks he’s at the bow of a Viking ship by the look of it. 🙂

Cheers

Tim…

Video : Ansible Playbooks : Files and Templates

In today’s video we demonstrate how to use handlers in Ansible Playbooks.

The video is based on the following article.

You might find some useful stuff here.

The star of today’s video is Dan Norris, who is looking to the heavens for inspiration. 🙂

Cheers

Tim…

Video : Ansible Playbooks : Handlers

In today’s video we demonstrate how to use handlers in Ansible Playbooks.

The video is based on the following article.

You might find some useful stuff here.

The star of today’s video is my dad, Graham Wood, who spend most of his time galivanting and spending my inheritance, which I’ve been waiting patiently for!

Cheers

Tim…