Followers of the blog know I’m a big advocate for writing things down. The main reason I do this is because I want a record of everything I do.
I rarely type a command directly into the command line. I nearly always type it in a scratchpad first. Currently I have 67,250 lines in my work scratchpad and 12,309 lines in my personal scratchpad.
When I say scratchpad, I just mean a text file, which I edit using a text editor. Nothing fancy.
Why do I do this?
Inspiration
Most of my articles and blog posts start life as notes in my personal scratchpad. At work some of my scratchpad notes become more formal documentation, like knowledge base notes and how-to files in Git etc.
I know if I don’t make the notes as I go along, I will forget what I did, and struggle to write the documentation later.
If something makes it as far as being written up, it gets removed from my scratchpads, so what’s in there at the moment are notes that have not made the cut, so to speak. š
One of the reasons I’ve been able to produce content for so many years is there is a constant stream of stuff added to my scratchpads. Of course, some of it is junk, but some of it is not.
If you are struggling with documentation or inspiration, I think taking this approach will really help.
Reflection
One of the things that I find really useful about taking notes is it allows me to look back and reflect on what I did to complete something. For example I might search through my scratchpad to see what happened over the lifetime of a server. I can see all tickets that were raised and what firewall rules and configuration changes were required. When I get a similar request this allows me to estimate the amount of work that needs to be done, and I can see what teams will be involved in the process.
I could search though our ticketing system for much of this information, but I find it a lot easier to keep a record of my actions in a scratchpad, then drill into the tickets if I need more info, which I rarely do.
Rewrites
Much like my articles, if I read back through some notes and they aren’t 100% clear, I often rewrite them. Maybe adding some more text, or a clearer example. This process may result in something graduating into being a separate document, but sometimes it just stays in the scratchpad forever.
Give it a go
If you don’t already do this, give it a go and see how you feel about it. Especially you content creators.
We recently had an issue, which I suspect was caused by overlapping CRON jobs. By that I mean a CRON job had not completed its run by the time it was scheduled to run again.
CRON
If you’ve used UNIX/Linux you’ve probably scheduled a task using CRON. We’ve got loads of CRON jobs on some of our systems. The problem with CRON is it doesn’t care about overlapping jobs. If you schedule something to run every 10 minutes, but the task takes 30 minutes to complete, you will get overlapping runs. In some situations this can degrade performance to the point where each run gets progressively longer, meaning there are more and more overlaps. Eventually things can go bang!
Fortunately there is a really easy solution to this. Just use “flock”.
Let’s say we have a job that runs every 10 minutes.
The Oracle Scheduler (DBMS_SCHEDULER) doesn’t suffer from overlapping jobs. The previous run must be complete before the next run can happen. If we have a really slow bit of code that takes 30 minutes to run, it is safe to schedule it to run every 10 minutes, even though it may seem a little stupid.
The Oracle Scheduler also has a bunch of other features that CRON doesn’t have. See here.
Conclusion
I’m not a massive fan of CRON. For many database tasks I think the Oracle Scheduler is far superior. If you are going to use CRON, please use it safely. š
Upgrading a database is not about the technical side of things. It’s about the planning that is required. I can upgrade a database in a few minutes, but the project to upgrade all the environments for a specific application can take months to complete. In this post I want to discuss some of the issues we are discussing at the moment regarding our future Oracle 23c upgrades.
What support dates are relevant?
Here are the support dates for the Oracle database.
To upgrade directly to 23c we must be running Oracle 19c or 21c.
All our databases are on 19c, so this puts us in a good position. It took a lot of pain and effort to get us to 19c, but it was worth it!
PDB or non-CDB architecture?
The non-CDB architecture was deprecated in 12.1.0.2, but it has remained supported up to, and including, 19c. So Oracle 23c will be the first long term release where the non-CDB architecture is not an option. If you’ve not got up to speed on pluggable databases, you better get started soon! (Multitenant Articles)
With one exception, we have PDBs across the board, so there is nothing new for us here. It sometimes felt like I was swimming against the tide by pushing PDBs so hard over the years, but it all seems worth it now.
What OS are you running on?
I’m going to conveniently forget that anything other than RHEL/OL exist, because other operating systems don’t exist for me in the context of running Oracle databases.
It took us a long time to migrate from OL6 to OL7. The majority of our Oracle databases are currently still running on OL7, which is fast approaching end of life. Since Oracle 23c will not be supported on OL7, we are going to need to migrate to a newer operating system. I wrote about my scepticism around in-place RHEL/OL upgrades (here), so that leaves us two choices.
Move our existing databases to a new OS now, then upgrade to 23c later.
Wait for the 23c upgrade, and do a big-bang OS migration and database upgrade.
What’s stopping us from doing the first option now? Nothing really. We could migrate our 19c databases to OL8 servers. It would be nicer to migrate them to OL9, but it is not supported for 19c yet. I recently wrote a rant about product certifications on Oracle Linux 9, which resulted in this response from Oracle.
“Oracle Database product management has confirmed that when Oracle Database 23c ships, it will be certified for both OL8 and OL9. Also, Oracle Database 19c will be certified on OL9 before end of 2023.”
That’s really good news, as it gives us more options when we move forward.
Will Oracle Linux exist in the future? Yes!
Just as I thought I had got my head around the sequence of events, RHEL dropped the bombshell about how they would distribute their source in future (see here). This raised concerns about if RHEL clones such as Oracle Linux, Rocky Linux and AlmaLinux could even exist in future.
Without knowing the future of our main operating system, we were questioning what to deploy on new servers. Do we continue with OL or switch to RHEL? Rocky and Alma released some “don’t panic” messages, but Oracle were very quiet. I wasn’t surprised by that, because Oracle don’t say anything until it has passed through legal, but as a customer it was a very nervy time.
A couple of days ago we got a statement from Oracle (here), with a firm commitment to the future of Oracle Linux. I immediately spoke to our system admins and said OL8/OL9 is back on the table. Phew!
I have my own opinions on the RHEL vs clones situation, but as a customer it’s not about politics. I just need to know the OS we are using is still going to exist, and will be supported for our databases. In that respect the statement from Oracle was very welcome!
Do you need a hardware refresh?
If you are running on physical kit, you need to check your maintenance agreements. You may need a hardware fresh to keep everything up to date.
We run everything on virtual machines (VMs), so the hardware changes to the clusters have no impact on our VMs. We’ve had at least one hardware refresh during the lifespan of some of our database VMs.
We use a lot of third party applications, and some of the vendors are really slow to certify their applications on new versions of the database and operating systems.
Ultimately we will make a choice on destination versions and timings based on application vendor support.
Manual or AutoUpgrade?
In Oracle 23c manual upgrades are deprecated (but still supported). I was late to the party with AutoUpgrade, but now I’ve used it I will never do manual upgrades again. We will definitely be using AutoUpgrade for our 23c upgrades!
If you are new to AutoUpgrade I have some examples of using it when I was doing 21c upgrades (see here). That should help you get started.
What are you going to test?
Testing is always a big stumbling block for us. We are not very far down the path of automated testing, which means we need bodies to complete testing. The availability of testing resource is always an issue. There are times of the year when it is extremely unlikely people will be made available, so planning this resource is really important.
So what’s the plan?
It’s always a balancing act around support for the OS, database and application vendors. Ultimately each project will have to be dealt with on a case by case basis, as the allocation of testing resources and potential disruption to the business have to be factored in. Everything is open to change, but…
Our default stance is we will upgrade to Oracle 23c on OL9. We will build new OL9 servers and install 23c on them, then use AutoUpgrade to migrate and upgrade the databases. For some of our internal developments I feel this could happen relatively quickly (kiss of death).
Application vendor support is often a sticking point for us, and timing will have to factor in the OL7 end of life. If support for 19c on OL9 comes in time, we may migrate our 19c databases to OL9, while we wait for a vendor to support Oracle 23c. Alternatively we could pay for extended support for OL7, and do the OS and database in one go once the application vendor is happy.
I realise this has been a bit of a ramble, but I just wanted to write it down to get things straight in my own head. š
Cheers
Tim…
PS. I have some technical posts on upgrading to 23c that will be released once the on-prem version of 23c goes GA.
The RHEL distribution is still really popular in the enterprise Linux market. The stats sometimes look worse because the users are split across both RHEL and all of its many clones, but it still represents a massive chunk of the enterprise market. With that in mind, why are RHEL upgrades still so unreliable?
My history of OS upgrades
Over the years I’ve done loads of operating system upgrades. In “recent” times I’ve done Windows upgrades (8 -> 8.1 -> 10 -> 11) and loads of Intel macOS upgrades with minimal drama. That’s not the case with Linux.
About a decade ago I was using Fedup to upgrade between versions of Fedora. That was later replace by the DNF upgrade process. More recently I’ve tried my hand using Leapp to upgrade Oracle Linux. I’m no expert, but these upgrades always feel really janky. You never really know what you are going to get until the upgrade is complete. The milage varies depending on configuration of the server, and what software has been installed on it. In some cases you have a system that is running fine. In some cases you have a running server, but a bunch of the configuration has to be redone to make your application work. In some cases you have bricked your server. Not exactly confidence building.
So what is the alternative?
My overall feel has always been don’t upgrade! Build new kit, migrate across to it, then ditch the old kit.
In the past we would typically make that process fit with the lifespan of the physical kit, but now when we use virtual machines we could potentially have a VM that outlives the physical kit many times over, so the desire to migrate is not so pressing, and a reliable in-place upgrade process is more desirable.
Migrating to a new server comes with a hole bunch of overheads. It’s a lot more work than an in-place OS upgrade.
Infrastructure as Code
If you have true infrastructure as code, you can build new systems really quickly, including all the VMs, networking and firewalls, which just leaves you with the data migration to worry about. If you are not in that position, it’s a nightmare of service tickets and endless waiting.
Questions
Is it too much to ask for such a dominant operating system to have reliable upgrades?
I would love to know your experiences of OS upgrades for RHEL and its clones. Have I just been really unlucky with Leapp, or are upgrade really as bad as I think?
Cheers
Tim…
Update 1: I ran a poll on Twitter. There weren’t many responses, but at least I can see I’m not completely alone. š
Update 2: There are some interesting responses in the comments. Well worth you giving them a look.
It’s over a year since Oracle Linux 9 was released and we still haven’t seen any certifications of Oracle products on it. What is going on?
Check out the really important update at the end of the post.
Setting the scene
Here are the current support dates for Oracle Linux.
Release
GA Date
Premier Support Ends
Extended Support Ends
OL6
Feb 2011
Mar 2021
Jul 2024
OL7
Jul 2014
Jul 2024
Jun 2026
OL8
Jul 2019
Jul 2029
Jul 2031
OL9
Jun 2022
Jun 2032
Jun 2034
So unless you want to start paying for extended support, you need to be getting rid of OL7 soon. The problem is, you can only move to OL8, because OL9 is not supported for Oracle 19c yet.
So if you are trying to do the right thing by migrating off OL7, you are being forced on to an old version of the OS, which in turn reduces the amount of time you can run before the next OS move.
The next long term release of the Oracle database is Oracle 23c. It’s still in beta, so we don’t know what it will be supported on yet, but 23c Free is only available for RHEL8/OL8 at this time.
Your choices?
So your choices are:
Migrate on to OL8, which solves your OS support issue, and will allow you to upgrade to 23c when it is released.
Wait for 23c to be released and hope it is supported on OL9. If it is, do a big bang migration to OL9 and 23c, hoping you can get it all done before the OL7 support deadline.
I don’t like either of these choices.
What do I want?
Apart from unlimited wishes, I want:
A definitive statement about if Oracle 19c will ever be supported on OL9. If it will be, when is that likely to happen? My guess is this will never happen, but I want an official statement ASAP.
A definitive statement about support for Oracle 23c on OL9. Will it be supported when 23c goes GA?
I guess it would also be nice if RHEL/OL upgrades actually worked reliably, but that is the subject for another rant!
Conclusion
I think like most people, I just want some clarity. It’s hard to plan when you don’t have any of the vital information. The thought of having to move to OL8 rather than OL9 is kind-of depressing… š
Cheers
Tim…
Update from Oracle: “Oracle DatabaseĀ productĀ management has confirmed that when Oracle Database 23c ships, it will be certified for both OL8 and OL9. Also,Ā Oracle Database 19c will be certified on OL9 before end of 2023.”
Why do I do this? As mentioned in the first link, Fedora is a proving ground for future versions of RHEL, and therefore Oracle Linux. I like to see what is coming around the corner. Doing this has no āreal worldā value, but Iām a geek, and this is what geeks do.
Why do I do this? As mentioned in the first link, Fedora is a proving ground for future versions of RHEL, and therefore Oracle Linux. I like to see what is coming around the corner. Doing this has no “real world” value, but I’m a geek, and this is what geeks do. š
As an aside, when Fedora 35 was released I was having a lot of trouble getting 19c and 21c installed on it. I tried a number of times over the course of a few weeks and failed each time. When I tried those same installations on Fedora 36 they just worked, so I went back and tried on Fedora 35 again, and they worked there too. Clearly there have been some changes to underlying Fedora 35 packages that have fixed whatever the problem was with the Oracle installations. As a result, I also produced these.
So now you know how to do it, please donāt! š
What’s New?
So what’s new with Fedora 36? It’s a bleeding edge distribution, so as you might expect, loads of package version updates, bringing most things to the latest and greatest versions. The things that stand out for me are Ansible 5 and Podman 4.0. If you want a more complete perspective on this, you might want to look here.
