Automation : Increasing pressure on an existing constraint

Yesterday I tweeted that I was reminded of this post.

I was reminded of it because of something that is happening to me at work, so I thought I would talk about it here.

Production lines

If you’ve read anything about DevOps you will know it came from manufacturing. If you didn’t know that, check out The Goal, which was the basis for The Phoenix Project.

Manufacturing typically uses production lines made up of multiple stations, where each station performs a specific task, and the product moves forward through the stations until it is complete. If one station is slower than the others, it will become a blocker. Product will start to queue up behind it, and downstream stations will become starved. So production lines only work well if they are planned to enable a consistent flow.

What’s more, you can only sell the product when it is completed, so we could describe the product as having no value until it is finished and with the customer.

It’s not just about manufacturing

The processes born out of manufacturing also work really well for other industries. I would suggest most things can be described like a production line, and as soon as you do that, a similar approach can be adopted to identify and fix the constraints.

Tech is an obvious one, and variations on DevOps have grown in popularity because of that. My sister in law works in a medical practice, and we’ve discussed the processes used in the administration side of it. As a result of our discussions she’s started to use Kanban boards to visualise the flow of work.

Back to my problem

So with the concept of production lines and constraints in mind, we jump back to my problem…

We are in the process of replacing loads of Oracle Linux 7 servers with either Oracle Linux 8 or Oracle Linux 9, depending on vendor support. The first three links in that production line are as follows.

  • VMs are provisioned.
  • Operating system customisations are run.
  • Database or app server is installed and configured.

We are not perfect, but we’ve got pretty good at this part of the production line. When we are finished, the systems have to be tested, and go through various processes to get them live and used by the business. Those parts of the production line that follow us are slow due to a number of factors. So our improvements in the production line have just made things harder for those steps that follow us…

The obvious thing to do here is focus on the constraints and start working on downstream links in the chain, to improve the overall flow. That’s where we hit organisation and culture barriers, so we are pretty much stumped…

Thoughts

I’m pretty happy with what we’ve done over the last few years. We’ve definitely improved several aspects of our systems because of automation, but at the same time I can’t help thinking we’ve achieved nothing because ultimately the work is not getting completed as fast as it’s started.

I wrote here about reframing the goal, and I have to do that as copium. Unfortunately copium only goes so far… 🙂

Cheers

Tim…

Business As Usual (BAU) vs Project Work

I’ve had this conversation so many times over the years, and I’m sure I’ve written about elements of it several times, but I’m not sure I’ve written about it specifically before, so here goes…

In every organisation there are conflicting demands from project work and business as usual (BAU) tasks. In case you’ve not heard the term BAU, here’s a definition.

Business as usual (BAU), the normal execution of standard functional operations within an organisation, forms a possible contrast to projects or programmes which might introduce change. BAU may also stand in contradistinction to external events which may have the effect of unsettling or distracting those inside an organisation.

Wikipedia

Swimming to stay still

I’ve written about this before in a rather angry post here. Working in tech is like swimming upstream. As soon as you stop swimming, you’re moving backwards. Let’s say today you have fully patched, secure and supported systems. How long can I do nothing before that is no longer the case?

  • Patches: For the operating system we may be talking days. For the database or an application server it may be months.
  • Support: Depending on where we are in the product support cycle, this is likely to be years, but we are fast approaching some important deadlines for Oracle Linux 7 and Oracle Database 19c, so we can’t wait much longer before we have a lot of work on our hands.

Standing still takes effort. If you are not putting in that effort, you are moving backwards, even if you don’t realise it.

BAU is invisible. Project work is shiny!

The problem with BAU is it is invisible to the users. Often you patch or upgrade a system and what they get after all that work is “exactly” what they had before. Of course, it’s not exactly the same, but from their perception it is. That can seem like a lot of time and effort for no perceivable gain, especially if you are asking for their resources to test things.

In comparison, project work often gives them something new and shiny to play with. It has perceivable value.

Faced with allocating resources between the two, you know there is going to be a lot of pressure to deliver new and shiny stuff over keeping the lights on…

Automation doesn’t solve all BAU tasks

Automation can certainly help with a lot of BAU work, but not everything. Even if you could magically upgrade a system without any downtime, somebody still needs to test the systems against it. Automation also brings with itself some additional BAU. Here are some examples I’ve seen recently.

Terraform: Providers change on a regular basis, which means you might be provisioning your kit using an old version of a provider. Over time you will start to see deprecation warnings, and have to update your provider. In some cases this will break your builds and you will have to do some refactoring. You need to revisit your Terraform builds on a regular basis, or put your automation at risk. Even the updates of the Terraform executable can introduce issues. One upgrade desupported a backend provider, which broke our project.

TeamCity: We use TeamCity for some on-prem automations. There are regular updates to this tool, usually because of security issues in some of the components such as Java or Tomcat. We have similar issues with Jenkins.

GitHub Actions: Have you seen that list of warnings and deprecation notices for those actions that are currently working fine? You are going to have to revisit those, or your lovely automations will break!

Cloud Platforms: If done well, cloud platforms can alleviate a lot of the operational BAU work, but they are not immune to issues around upgrades and deprecations. Many of us have lived through the desupport of previous generations of cloud architectures, and upgrades of underlying tech still have to happen, and require your systems to be tested when they are.

This is not meant to be an exhaustive list. Just examples.

BAU as internal projects

Just process your BAU as internal projects, and then they can be scheduled like any other project, right? That sounds fine, but someone still has to prioritise the projects, and BAU is not shiny! It’s still going to come in second place.

Education is the key

The only answer to this is education. The business has to understand that BAU is not negotiable. You have to be strong enough to push back on unrealistic demands, to make sure that systems remain up to date and safe. This can only be successful if you educate everyone on the importance of this boring and often invisible stuff…

A word about Oracle

It would seem wrong to finish this post without a mention of Oracle.

Most of the database upgrades I’ve done in my life have only happened because of the pressure of needing to stay in support. They have not been because people want the shiny features that are offered by the new release. That’s not to say they won’t end up being used down the line, but that is not the driving force a lot of the time. Stable and bug free beats new features every time!

In Oracle, just like any other company, there are competing pressures. I think most of us customer need to have 23c available so we can start the upgrade process to stay in support long term. There are no doubt a small number of important customers demanding features that will delay the release, and probably introduce bugs that affect all of us. There are also features that would sound cool for the sales teams and in keynote presentations. Who wins? Probably not me as I’m not working for an important customer, and I’m not in sales and marketing. 🙂

Conclusion

BAU is boring and often invisible, but it has to be done!

Cheers

Tim…

APEX : Keeping up to date is so easy…

Over the years I’ve extolled the virtues of Oracle Application Express (APEX) because of the ease of development. I think low code tools are a massive boon to productivity. Of course there are some tasks that need alternative tools, but for many scenarios low code tools are awesome.

Something else I find really appealing about APEX is the ease of upgrades. I’m not talking about how easy it is to apply the upgrade itself, because updating Java and Tomcat versions on a server is really easy too. I mean how simple it is from a wider perspective.

I was the first person in my company to use APEX. I used it to write some utility type applications, when it was still “forbidden”. Some of these applications were written over a decade ago, and they are still working fine. In that time we’ve had regular APEX upgrades, and they’ve just kept going. No refactoring. No drama.

Of course, they aren’t using all of the new features that were added in subsequent releases, but the important thing is all that development investment was not impacted by staying on the latest APEX release and patch set. In comparison, updating some of our other platforms and frameworks is a nightmare, requiring substantial development effort and testing.

So it’s not just about improving productivity during the development phase. It’s also about the reduction in the total cost of ownership (from a development perspective) over the lifespan of the application.

Just thought I would share that thought, as I upgrade & patch some production systems… 🙂

Cheers

Tim…

Continuous Improvement : It’s easy to get lazy…

I like to think that continuous improvement is always part of my process, but occasionally things crop up and remind me that I am just as lazy and set in my ways as everyone else. Here’s a little story to illustrate that…

The Issue

The main thing I’m supposed to be focussed on at work is server rebuilds. We are moving a bunch of stuff from Oracle Linux 7 (OL7) to Oracle Linux 9 (OL9), or OL8 if the vendor doesn’t support OL9 yet.

Anyone who follows my Vagrant builds knows I’ve got loads of scripts that build things, but because of our company organisation structure, there is only so much I can do without pinging requests to other teams, so full automation on-prem is not currently possible for me at work.

As a result of this limitation, there is a bunch of things I can do in Vagrant that I can’t do at work. The fact that work is semi-automated for builds has indirectly made me lazy in respect to continuously improving my build processes. In my head I guess I’ve been saying, “What’s the point if I can’t finish it?”

Making Progress

Due to the large number of rebuilds we are working through now, I’ve taken a step back and started again. I created a new build repo at work, and populated it with all the relevant stuff from our existing build processes, and some bits from my Vagrant builds. It’s still not fully automated, due to the company organisation issues mentioned before, but every step in the process is up to date with the best we can do, and the process is documented in the README.txt of the repo.

Along the way I noticed a bunch of things that were pretty crap in my original approach. It worked, but it wasn’t something to be proud of. 🙂

So What?

This just goes to illustrate the point I made in the title of this post. It’s easy to get lazy and let things start to slip, even when you know better. This is especially true if you have a “completionist” mentality like me. The thought of being able to finish something is quite compelling to me. Unfortunately, when I know I will not be able to complete something, I find it really hard to make the effort, because it all feels kind-of pointless. 🙂

Reframe The Goal

I guess this is really about reframing the goal. Rather than thinking that completion is when I have a fully automated process, it should be that I’ve got everything up to date and as far down the automation path as I can get at the moment.

The funny thing is I wrote about this in a previous post, but it seems I’m not always capable of taking my own advice. 🙂

Cheers

Tim…

Oracle VirtualBox 7.0.14

VirtualBox 7.0.14 has been released.

The downloads and changelog are in the usual places.

I’ve installed it on my Windows 10 and 11 machines with no drama.

Vagrant 2.4.0

Vagrant 2.4.0 is still the current release, so no change there.

If you are new to Vagrant and want to learn, you might find this useful.

Once you understand that, I found the best way of learning more was to look at builds done by other people. You can see all my Vagrant builds here.

I’ve updated my Vagrant builds to include the latest Oracle patches, along with GraalVM, ORDS, SQLcl and Tomcat.

Packer 1.10.0

I use Packer to rebuild my Vagrant boxes (Oracle Linux 7, 8 and 9) so they have the latest guest additions. The current set of boxes were built using Packer 1.10.0. The new version of the boxes can be seen here.

If you are interested in creating your own Packer builds, you might take inspiration from mine, available here.

Packer has been warning about using built in plugins for some time. This version doesn’t use built in plugins at all, so I had to do the following (in bold) to load the plugins, so I could perform a build.

set PATH=%USERPROFILE%\u01\software\hashicorp\packer_1.10.0_windows_amd64;%PATH%
packer plugins install github.com/hashicorp/vagrant
packer plugins install github.com/hashicorp/virtualbox

cd \git\oraclebase\vagrant\packer\ol9

packer build -only virtualbox-iso ol9.json

How did it all go?

The new versions of VirtualBox and Packer worked fine. 🙂

What about the VirtualBox GUI?

Just a quick warning. I do everything using Vagrant, so I rarely look at the VirtualBox GUI. Remember, when I say everything worked fine, I mean for what I use.

Cheers

Tim…

Life Update : A new member of the family…

I’ve been conspicuous by my absence of late, so I thought I would tell you a little story…

The Trigger

I’m a cat person. I don’t dislike dogs, but I prefer cats. I’m a sucker for a Rottweiler or an American Akita, but really cats are my thing.

I’ve been toying with the idea of getting a cat for ages, but it never seemed fair. I used to travel a lot, and worked long hours in the office. I’m now working from home and only work 4 days a week, so things have changed…

With the change in working conditions, I thought now was the time, but that meant decluttering my house. For the last couple of months I’ve been clearing all the crap from the house that I’ve accumulated over the last 20+ years. I gave myself until the end of the year to complete it, but things escalated pretty quickly…

The Cat

I mentioned to a friend I was going to adopt a senior cat. A couple of days later he rang me and asked if I wanted a kitten his friend was trying to rehome. I panicked and said yes. Yesterday he arrived.

This image has an empty alt attribute; its file name is klaus.jpg

I’m still trying to figure out a name, but I’m thinking it’s going to be Klaus.

He’s four months old. I’m told you can see Bengal markings on him in the right light, but he looks jet black to me. He’s really confident and will happily occupy himself. When I’m in the office he pops in, says hello, then goes out and starts playing with his toys.

I now have some furniture for the first time, but it’s cat furniture… 🙂

I’m not sure how many photos/videos I’ll post in future, because it’s hard to take a photo that doesn’t show you my house, and you will definitely judge me. 🙂

Cheers

Tim…

Update: I registered him with the vet, so he is definitely called Klaus now…

Fedora 39 and Oracle

Fedora 39 was released recently. Here comes the standard warning.

Here are the usual suspects.

I like messing about with this stuff, as explained in the first link.

I pushed Vagrant builds to my GitHub.

If you want to try these you will need to build a Fedora 39 box. You can do that using Packer. There is an example of that here.

What’s New?

So what’s new with Fedora 39? You can read about it here.

Cheers

Tim…

Oracle Database 19c on Oracle Linux 9 (OL9): Installation Articles and Vagrant Builds

Earlier this year I wrote a rant about the lack of product certifications on Oracle Linux 9 (OL9).

One of the points I made was we are having to replace OL7 servers, but were forced to go to OL8 because Oracle 19c was not certified on OL9, and Oracle 23c on-prem is not available.

This blog post by Mike DieTrich changed all that because now 19c is certified on OL9, provided you are on patch 19.19 or above, and are on the correct version of UEK or the RHEL kernel. See Mike’s post for details.

Installation Articles

Of course, this triggered some installation articles.

Vagrant Builds

There are database, RAC and Data Guard vagrant builds here.

Odd Occurrence

I noticed something a little odd when doing these builds using the 19.21 RU patches.

For the database and Data Guard builds I used the DB RU and OJVM combo patch, and I was still forced to fake the distribution using the CV_ASSUME_DISTID environment variable. For the RAC build I used the GI RU and OJVM combo patch, and I didn’t need to fake the distribution.

I went back to the DB build, and instead used the GI RU and OJVM combo patch, and I no longer needed to fake the distribution. So it looks like there is something different about the database patches between these two types of RUs that slightly affect the installation process. It’s no big deal, but it might catch you out.

Oracle 19c is old. Why do you care?

We are in the process of replacing a load of VMs that are currently running OL7, and we want to go to OL9. Prior to this announcement were were going to have to do one of two things.

  • Migrate to 19c on OL8, which would be OK for 23c when it drops, but not ideal as building an OL8 box now seems like a fail.
  • Wait for 23c on-prem to drop and move to 23c on OL9. The problem here is we could run out of time waiting for 23c to come.

This announcement gives us a new option.

  • Migrate to 19c on OL9, then upgrade to 23c when the on-prem version drops.

This third option is way better for us!

Remember

There are a couple of things to remember.

  • You need to be on 19c to upgrade to 23c, so getting your 19c database on an OS that is supported for 23c is important. We’ve had confirmation that 23c will be available for OL8 and OL9 on release.
  • The extended support waiver for 19c was increased from 1 year to 2 years. Mike also wrote about this here. That means you get free extended support for 19c until April 30, 2026.

Conclusion

This is massive for us. I’m very happy!

Cheers

Tim…

Oracle VirtualBox 7.0.12, Vagrant 2.4.0 and Packer 1.9.4

Oracle VirtualBox 7.0.12

VirtualBox 7.0.12 has been released.

The downloads and changelog are in the usual places.

I’ve installed it on my Windows 10 and 11 machines with no drama.

Vagrant 2.4.0

Releases of VirtualBox prompt me to check for new versions of Vagrant. The current version is Vagrant 2.4.0. All my test systems are built with Vagrant, so I installed it as well.

If you are new to Vagrant and want to learn, you might find this useful.

Once you understand that, I found the best way of learning more was to look at builds done by other people. You can see all my Vagrant builds here.

I’ll be doing some updates to my Oracle builds over the coming days, so this will get a lot of testing.

Packer 1.9.4

I use Packer to rebuild my Vagrant boxes (Oracle Linux 7, 8 and 9) so they have the latest guest additions. The current set of boxes were built using Packer 1.9.4. The new version of the boxes can be seen here.

If you are interested in creating your own Packer builds, you might take inspiration from mine, available here.

How did it all go?

The new versions of VirtualBox, Packer and Vagrant all did their jobs fine. 🙂

What about the VirtualBox GUI?

Just a quick warning. I do everything using Vagrant, so I rarely look at the VirtualBox GUI. Remember, when I say everything worked fine, I mean for what I use.

Cheers

Tim…

Joel Kallman Day 2023 : It’s a Wrap! (#JoelKallmanDay)

Yesterday was an Oracle community day called the Joel Kallman Day (#JoelKallmanDay) 2023.

I would like to say a big thank you to everyone who took the time to join in. Here is the list of posts I saw in chronological order. If I missed you out, give me a shout and I’ll add you. 

Thanks to everyone who got involved with blog posts, tweets and retweets. It all helps add to the buzz. We are not just an English speaking community, so it’s always good to see posts in different languages. If you want to read the non-English posts, Google Translate usually does a reasonable job.

Thanks again, and remember you’ve got a year to get ready for the 2024 event! 🙂

Cheers

Tim…