Why are RHEL/Oracle Linux upgrades still so unreliable?

The RHEL distribution is still really popular in the enterprise Linux market. The stats sometimes look worse because the users are split across both RHEL and all of its many clones, but it still represents a massive chunk of the enterprise market. With that in mind, why are RHEL upgrades still so unreliable?

My history of OS upgrades

Over the years I’ve done loads of operating system upgrades. In “recent” times I’ve done Windows upgrades (8 -> 8.1 -> 10 -> 11) and loads of Intel macOS upgrades with minimal drama. That’s not the case with Linux.

About a decade ago I was using Fedup to upgrade between versions of Fedora. That was later replace by the DNF upgrade process. More recently I’ve tried my hand using Leapp to upgrade Oracle Linux. I’m no expert, but these upgrades always feel really janky. You never really know what you are going to get until the upgrade is complete. The milage varies depending on configuration of the server, and what software has been installed on it. In some cases you have a system that is running fine. In some cases you have a running server, but a bunch of the configuration has to be redone to make your application work. In some cases you have bricked your server. Not exactly confidence building.

So what is the alternative?

My overall feel has always been don’t upgrade! Build new kit, migrate across to it, then ditch the old kit.

In the past we would typically make that process fit with the lifespan of the physical kit, but now when we use virtual machines we could potentially have a VM that outlives the physical kit many times over, so the desire to migrate is not so pressing, and a reliable in-place upgrade process is more desirable.

Migrating to a new server comes with a hole bunch of overheads. It’s a lot more work than an in-place OS upgrade.

Infrastructure as Code

If you have true infrastructure as code, you can build new systems really quickly, including all the VMs, networking and firewalls, which just leaves you with the data migration to worry about. If you are not in that position, it’s a nightmare of service tickets and endless waiting.

Questions

Is it too much to ask for such a dominant operating system to have reliable upgrades?

I would love to know your experiences of OS upgrades for RHEL and its clones. Have I just been really unlucky with Leapp, or are upgrade really as bad as I think?

Cheers

Tim…

Update 1: I ran a poll on Twitter. There weren’t many responses, but at least I can see I’m not completely alone. 🙂

Update 2: There are some interesting responses in the comments. Well worth you giving them a look.

Oracle product certifications on Oracle Linux 9. What is going on?

See update at the end.

Warning. Rant mode enabled…

It’s over a year since Oracle Linux 9 was released and we still haven’t seen any certifications of Oracle products on it. What is going on?

Check out the really important update at the end of the post.

Setting the scene

Here are the current support dates for Oracle Linux.

ReleaseGA DatePremier Support EndsExtended Support Ends
OL6Feb 2011Mar 2021Jul 2024
OL7Jul 2014Jul 2024Jun 2026
OL8Jul 2019Jul 2029Jul 2031
OL9Jun 2022Jun 2032Jun 2034

So unless you want to start paying for extended support, you need to be getting rid of OL7 soon. The problem is, you can only move to OL8, because OL9 is not supported for Oracle 19c yet.

So if you are trying to do the right thing by migrating off OL7, you are being forced on to an old version of the OS, which in turn reduces the amount of time you can run before the next OS move.

The next long term release of the Oracle database is Oracle 23c. It’s still in beta, so we don’t know what it will be supported on yet, but 23c Free is only available for RHEL8/OL8 at this time.

Your choices?

So your choices are:

  • Migrate on to OL8, which solves your OS support issue, and will allow you to upgrade to 23c when it is released.
  • Wait for 23c to be released and hope it is supported on OL9. If it is, do a big bang migration to OL9 and 23c, hoping you can get it all done before the OL7 support deadline.

I don’t like either of these choices.

What do I want?

Apart from unlimited wishes, I want:

  • A definitive statement about if Oracle 19c will ever be supported on OL9. If it will be, when is that likely to happen? My guess is this will never happen, but I want an official statement ASAP.
  • A definitive statement about support for Oracle 23c on OL9. Will it be supported when 23c goes GA?
  • I guess it would also be nice if RHEL/OL upgrades actually worked reliably, but that is the subject for another rant!

Conclusion

I think like most people, I just want some clarity. It’s hard to plan when you don’t have any of the vital information. The thought of having to move to OL8 rather than OL9 is kind-of depressing… 😞

Cheers

Tim…

Update from Oracle: “Oracle Database product management has confirmed that when Oracle Database 23c ships, it will be certified for both OL8 and OL9. Also, Oracle Database 19c will be certified on OL9 before end of 2023.”

Update: Oracle have now certified Oracle 19c on Oracle Linux 9, as discussed here.

Book : The Method R Guide to Tracing Oracle

In a game of word association, when someone says Oracle Trace you probably respond with Carry Millsap. 🙂 Well, Cary and Jeff Holt have a new book out called “The Method R Guide to Tracing Oracle”

You can get hold of it from a couple of locations:

Cary sent me a copy of it a few days ago and while I was reading it I tweeted the following.

“I’m so used to clicking “Like” buttons on social media, that when I’m reading a PDF (supplied by @CaryMillsap) and I see a statement I like, I start looking for the “Like” button. Technology has messed me up…”

Who is it for?

It’s a really easy read, and a good introduction to tracing Oracle, which is perfect for people who haven’t got much experience of doing it, or don’t understand why it is useful.

Even for those who have more experience it is a good reminder of the values of tracing, and you will certainly pick up some new tips. As you saw in my tweet I enjoyed it, and this isn’t my first rodeo.

What if I’m not the target audience?

If you are already experienced with Oracle Trace, you might prefer their existing book The Method R Guide to Mastering Oracle Trace Data, which goes into much more detail.

I was first introduced to this book when I attended Cary’s “Mastering Oracle Trace Data” workshop in 2011. I wrote about that here. I later bought the second edition of the book, and Cary has subsequently sent me the third edition.

So wherever you are on your Oracle Tracing journey, there is something for you!

Cheers

Tim…

Writing Tips : Don’t get blocked by a post you don’t want to write

I’ve written previously about writing and motivation, but I wanted to say something about self-inflicted demotivation, which is something I’ve been guilty of a number of times.

As you will probably know, I write a lot of posts about Oracle technology, but I’m not equally interested in every feature of the Oracle database. Sometimes this is because I just don’t see the point of a feature, and sometimes it is because I simply don’t get to use a feature very much, even if I think it is cool. It can be hard to be motivated to write about something that isn’t jumping out and screaming at you…

Baiting the trap with stupid goals

I often set myself little goals, and sometimes these are my undoing. I might make a list of topics to write about, and the “completionist” part of my brain adds things into the list that I don’t really care about.

For example, I might decide I want to write about all of the SQL new features in version X of the database, but there is something that I’m not interested in, so writing that post is a grind. I want to complete the list, but that one item on the list is not inspiring me…

Now the sensible thing to do it to just miss that post out, and move on, but my stupid head gets locked into finishing the list, and I kind-of cripple my progress by forcing myself to do something I don’t really want to.

A quick example

Something very similar happened to me this week. I had recently written four new posts, which I can’t publish until 23c is out of beta, and I had one more post to complete my list. The subject itself was OK, but there was a lot of setup involved, for very little payback. It felt like hours of work to prove a single sentence. Needless to say I was not highly motivated.

I kept telling myself to move on and do something different, but in the back of my mind I kept thinking about that final tick on the list…

So what did I do? I wasted the week playing Raft on peaceful mode. Just cruising round the sea picking up junk and gathering resources from reefs. I’ve completed the game about 15 times on harder difficulties, but I wanted something mindless to do, rather than face writing that post.

The solution

This is a case of “do as I says, not as I do”, but you really need to avoid situations that you know will block you. Each of us will have different blockers, and different displacement activities we use to distract us, but I bet most of us can spot a pattern that triggers us…

If you find yourself working on a post that is killing you, just walk away. You can always come back to it later…

Check out the rest of the series here.

Cheers

Tim…

Fedora 38 and Oracle

Fedora 38 was released recently. Here comes the standard warning.

Here are the usual things I do when a new version of Fedora comes out.

This is not a recommendation. I just like messing about with this stuff, as explained in the first link.

I pushed Vagrant builds to my GitHub.

If you want to try these you will need to build a Fedora 38 box. You can do that using Packer. There is an example of that here.

What’s New?

So what’s new with Fedora 38? You can read about it here.

Cheers

Tim…

VirtualBox 7.0.8

VirtualBox 7.0.8 maintenance release has arrived.

The downloads and changelog are in the usual places.

I’ve installed it on Windows 11 machine with no drama. I’ll be running my Packer builds of all my Vagrant boxes later today, which you can find on Vagrant Cloud here.

I’ll be doing lots of testing over the next few days as I add the latest patches to my Vagrant builds. I’ll update here if I notice anything unusual.

Cheers

Tim…

Update: On my Windows 10 box I had to uninstall VirtualBox 7.0.6, then install VirtualBox 7.0.8 for it to work correctly with Vagrant.

Oracle 23c Free : Why start learning now?

It’s been a couple of weeks since Oracle 23c Free was released. I’ve already put out a getting started post to help people start their journey, but I wanted to say something about why, in my opinion, you should start learning 23c now, rather than waiting.

Learning stuff takes time

I’ve written some tips about learning new things, and one of the posts was about time and repetition. It’s going to take a long time to get used to using 23c. If the latest version you have used is 19c, you’ve got the 21c and 23c new features to catch up on. If you’re using something older than 19c, then you’ve got even more to do. Sure, you can use Oracle 23c like it’s Oracle 7, and for the most part things will be fine, but really you should try and get the most out of the products you are paying for.

The sooner you start, the easier it will be, and the less rushed you will feel when you eventually have to start.

I’ve already put out a lot of 23c posts, but remember I’m on the 23c beta program, so I’ve been writing these posts for over 5 months now. They didn’t happen overnight.

Feeling comfortable with a new release takes me years. There is a difference between writing a post about something, and feeling comfortable using it in my day-to-day life. It will take some time before some of this stuff is a natural part of my toolbox. I want that to happen as quickly as possible, so that means getting stuck in now, so when the full blown 23c goes GA I can hit the ground running.

It might make you change the way you do things now

Sometimes, knowing what is coming around the corner will alter the way you approach a problem today. You may use the way Oracle has implemented a new feature as inspiration for how to solve your current problem in an older release. If possible, you may code it such that you can do a straight swap when you upgrade to 23c, throwing away your code and replacing it with built in functionality.

You don’t know what you don’t know

My initial reaction to new releases is often, “there’s nothing there I care about”. Then I start working through the new features and things jump out at me. The more I work with it, the more interesting it gets. I already feel like I can’t live without the IF [NOT] EXISTS DDL clause. On the surface it seems so trivial, but it makes a massive difference. 🙂

The Oracle 23c Free release contains mostly developer focused new features. We won’t see some of the administration features publicly until we get the full release, but there is still a lot to work through.

23c is new to everyone

I say this with each new release, but remember there are no Oracle 23c experts. Even Oracle staff and people in the beta program haven’t got something I would consider as significant production experience. Just remember this when you are reading posts about the 23c new features. We are all noobs! That doesn’t mean there is no value in these posts, but just remember what they represent.

It’s your choice

You are going to have to learn this stuff eventually, so you have two choices.

  • You can wait until the last minute, try to cram, and inevitably make mistakes when you are doing something real in your job.
  • You can start now and coast into the new release, so when you have to do something real with 23c, it’s not a problem.

It’s up to you, but you know what I think you should do. 🙂

Cheers

Tim…

Oracle Database 23c Free Developer-Release : Getting Started…

You may have noticed the flurry of posts about the new Oracle Database 23c Free Developer-Release.

In summary Oracle 23c Free is the replacement for what would have been Oracle 23c XE, but it is a developer release, so it’s not the final form of Oracle 23c Free. We should get an updated version of 23c Free once the main Oracle Database 23c release becomes GA.

Where do I get it?

If you want to install it from the RPM you can download it from here.

There is a VirtualBox appliance and a Docker image available from Oracle, so you don’t actually have to install it if you don’t want to.

How do I install it?

You have a few options here.

  • Install documentation here.
  • My installation article here.
  • My Vagrant build here.
  • VirtualBox appliance from Oracle here.
  • Docker image from Oracle here.

What about your articles?

I have a 23c page on the website ready to post Oracle 23c articles here.

I’ve written a bunch of articles against the 23c beta 1 release, but I’ve not published any of them yet because of the beta program NDA. I’m going to work through them against the 23c Free developer-release, and anything that I’m allowed to publish I will. Some of the articles will have to be held back until the GA release of 23c, as they are not covered by this release.

Basically, if it is documented in Oracle Database 23c Free, I can write about it. If not, I’m still under NDA, so I will release those articles later.

Documentation

The documentation is available here.

It is not the full 23c documentation set, as this is not the full release of the product.

You will notice it focusses on the application development side of things. There is no RAC and Data Guard stuff, so those subjects are off limits for now.

Bug Reports

There is a community forum for reporting bugs here.

What next?

Have fun! Remember, this is not the final 23c release, so this is meant as a way to get used to some of the new development features in 23c, while we wait for the full release.

Remember, if you see any problems, please shout out about them! You can report bugs here.

Cheers

Tim…

The Oracle ACE Program : My 17 Year Anniversary

It’s April 1st, which means it’s my 17th year anniversary of being an Oracle ACE.

As usual I’ll mention some of the other anniversaries that will happen throughout this year.

  • 28 years working with Oracle technology in August. (August 1995)
  • 23 years doing my website in July. (Original name: 03 July 2000 or current name: 31 August 2001)
  • 18 years blogging in June. (15 June 2005)
  • 17 years on the Oracle ACE Program. (01 April 2006)
  • 8 years doing videos on my YouTube channel, with some breaks of course.

Fingers crossed for next year…

Cheers

Tim…

Update Oracle Database Time Zone Files (Poll Results Discussed)

In case you didn’t know, countries occasionally change their time zones, or alter the way they handle daylight saving time (DST). To let the database know about these changes we have to apply a new database time zone file. The updated files have been shipped with upgrades and patches since 11gR2, but applying them to the database has always been a manual operation.

With the recent switch over to daylight savings time in the UK I decided to post this question on Twitter yesterday.

How often do you update your Oracle database time zone files?

We get less than 6% of people updating their time zone files on a regular schedule. Nearly 45% who only do the updates after a database upgrade, and nearly 50% of people who never do it at all.

I can’t say I’m surprised by the results. In terms of the reasoning for these responses, I’ll reference some of the comments on Twitter.

Regular Schedule

“Every ru patch, also thanks to 19.18 it is included now and with out of place upgrade and autoupgrade, i dont do it anymore 🙂 all automatic.”

Mustafa KALAYCI

If you are using AutoUpgrade to patch to a new Oracle Home, then applying updated time zone files is really easy. Before 19.18 it’s just a single entry “timezone_upg=yes” in the AutoUpgrade config file. From 19.18 onward the update of the time zone file is the default action (see here).

So interestingly, there may be some people who don’t know they are applying an update of their time zone file, who actually are now…

After Upgrades

This feels like the natural time to do it for me, and it seems many other people feel the same.

As mentioned previously, AutoUpgrade makes it simple. From 21c onward AutoUpgrade is the main upgrade approach, even for those that have resisted using it for previous versions, so this question goes away from an upgrade perspective.

We can specifically tell it not to perform the action using “timezone_upg=no”, but I’m guessing most people will just go with the default action.

Never

“NEVER. As an American-only company with very little need for time-specific data, quite unnecessary. Horrible design with no rollbacks and headaches w/data pump. Just not worth it if possible to avoid”

Taylor

I totally understand this response. Many of us work with systems that are limited to our own country. Assuming our country doesn’t alter its own daylight savings time rules, then using an old time zone file is unlikely to cause an issue.

When you consider the number of people that run *very old* versions of Oracle, you can see that using old versions of the time zone file doesn’t present a major issue in these circumstances.

With reference to the data pump issue, I’ve experienced this, and it was also picked up in the comments.

“My hypothesis: Most do it when datapump tells they need to do it to get the import file they just received to load”

Connor McDonald

Offline/Online Operation

The point about this being an offline operation was raised.

“Well it is an offline operation, so pretty exceptional thing to do. Only in a rare case where some feature requires the upgrade – like DataPump failing or query over dblink failing.”

Ilmar Kerm

Downtime is never welcome, but it was also pointed out it can be an online operation in 21c.

“Offline will be a thing of the past…

https://docs.oracle.com/en/database/oracle/oracle-database/21/refrn/TIMEZONE_VERSION_UPGRADE_ONLINE.html”

Connor McDonald

Conclusion

It seems like the time zone file version is not high on the list of priorities for most people, providing it is not causing a data pump issue. I totally understand this, and I myself only consider it during database upgrades.

I always like reading these poll results. I know the sample size is small, but it gives you a good idea of how your beliefs compare to the wider audience.

If you are interested to know how to manually upgrade your time zone file, you can read about it here.

Cheers

Tim…