Oracle 21c : Being a newbie at upgrades again…

I was having a play with upgrades to 21c over the weekend. It all started when Stephen Butterworth asked this in a comment.

“… do you know if 21c would create a non-PDB if you plied it with beer, a Melton Mowbray and asked nicely?”

I answered with this.

“They claim non-CDB is dead in 21c. It’s desupported, so that means most companies will not consider it. I have no idea if it is actually possible to run it, and to be honest I don’t care. The desupport notice is the killer for me.”

I subsequently found it throws an error if you try to open a non-CDB database in Oracle 21c.

That got me thinking about upgrade scenarios. I’ve done loads of upgrades to 19c PDBs recently and in most cases they followed this format.

  1. Upgrade the existing non-CDB database to 19c.
  2. Test the impact of the 19c upgrade on the non-CDB.
  3. Create a new 19c CDB.
  4. Convert the upgraded 19c non-CDB database to a PDB in the new 19c CDB.
  5. Test the impact of the conversion of non-CDB to PDB.

So how do you do the equivalent of that when you can’t run a non-CDB instance in Oracle 21c? Short answer, you don’t! At least not the same way…

The preupgrade.jar is dead, long live autoupgrade.jar

The first thing I noticed when I started playing with 21c upgrades is the “preupgrade.jar” has been superseded by the “autoupgrade.jar”. If you try to use the “preupgrade.jar”, it tells you to use AutoUpgrade. AutoUpgrade is available for 19c, but I’ve not used it for all my 19c upgrades. During the process of moving to 19c I’ve done a lot of work to clean up some of our crap, and make the resulting databases as “generic” as possible. I think from here on in, using a tool like AutoUpgrade will be fine, but I was nervous about using it before now. Maybe it’s just me being overly cautious. 🙂

Anyway, AutoUpgrade is the main thing from now, and it’s really useful. At first I was using it as if it were the “preupgrade.jar” and I couldn’t see the big deal. Then I realised a bunch of the stuff I was doing manually, like copying config files and managing the instance state were all done for me when I did a deploy. It certainly makes things simpler, which is always a good thing.

Auto-Upgrade of Adopted PDBs

As I mentioned before, one of the things that was freaking me out was I couldn’t follow my “normal” approach.

Expanding on what I said previously, in previous versions I might convert a non-CDB database to a PDB as part of an upgrade like this.

  • Upgrade the non-CDB database to the correct version.
  • Test the impact of the upgrade.
  • Create a new CDB if a suitable one didn’t already exist.
  • Describe the non-CDB database using the DBMS_PDB package, which creates a file containing the description.
  • Create a new PDB in the CDB using the description file.
  • Run the “noncdb_to_pdb.sql” script to convert the non-CDB database to a real PDB.
  • Open the new PDB.
  • Test the impact of the PDB conversion.

In 21c we can’t do the upgrade of the non-CDB database, so that’s not an option. Instead we have to adopt the PDB directly. In the past we would have done this.

  • Create a new CDB if a suitable one didn’t already exist.
  • Describe the non-CDB database using the DBMS_PDB package, which creates a file containing the description.
  • Create a new PDB in the CDB using the description file.
  • Upgrade the PDB.
  • Run the “noncdb_to_pdb.sql” script to convert the non-CDB database to a real PDB.
  • Open the new PDB.
  • Test the impact of both the upgrade and conversion to a PDB.

In 21c this approach is a bit different.

  • Create a new 21c CDB if a suitable one didn’t already exist.
  • Describe the non-CDB database using the DBMS_PDB package, which creates a file containing the description.
  • Create a new PDB in the CDB using the description file.
  • Open the new PDB in read-write mode.
  • Test the impact of both the upgrade and conversion to a PDB.

In Oracle 21c opening the PDB in read-write mode automatically triggers the upgrade of the PDB so it matches the CDB version, and converts the non-CDB database into a PDB. Bingo!

You can choose to do this conversion manually, or let AutoUpgrade do it all for you.

Database Versions for Direct Upgrade

Oracle 21c only supports direct upgrades from Oracle 12.2, 18c and 19c. If you are on an older version, you will have to do one or two upgrades prior to the 21c upgrade.

I suspect this will be similar for Oracle 23c, so that might be an incentive to push your upgrades to 19c now…

But why bother with 21c upgrades?

I’ve asked you not to use Oracle 21c here, so why bother playing with the upgrades? This is the way things are going to work in 23c too, and I will be wanting to upgrade to 23c, so it’s good to get up to date now, rather than start my first upgrade in a couple of years and realise I don’t know what I’m doing.

Articles

Along the way I produced these articles. The first one repeats a lot of what I said here, but without the chatter. The others are upgrade scenarios I wanted to test.

Conclusion

Having spent so much of the last year doing database upgrades to 19c I was thinking about changing my job title to “Upgrade Guru”. With the advent of Oracle 21c I’m back to being a newbie again…

Cheers

Tim…

Oracle Enterprise Manager Cloud Control 13.5

The release of Oracle Enterprise Manager Cloud Control 13.5 was officially announced last week, but I think it’s been available for download for a little longer. The first I noticed was the official announcement though. 🙂

That prompted the usual flurry of articles.

As well as the articles, you can find the Vagrant builds of Cloud Control 13.5 on Ol7 and OL8 in my Vagrant GitHub repo here.

Our current installation is running on an Oracle Linux 6 VM, so I was just about to start a new build on a new OS, which would have the added bonus of allowing us to use Oracle 19c under the hood. Now I can go straight to 13.5, which is nice.

I’ve mentioned before, we use EMCLI to add targets and jobs into Cloud Control, so rather than upgrading as I normally do, this time I’ll probably do a clean build on the new kit and gradually migrate services across to it.

Happy upgrading! 🙂

Cheers

Tim…

APEX 19.1 Upgrades : My Experience

With the exception of one application, which is taking a little longer to test, we’ve upgraded all our APEX installations to APEX 19.1.

I thought it would be good to share my experience of this, in case anyone is a little on the cautious side. 🙂

The vast majority of upgrades went fine. We just did the standard upgrade (install), without trying to minimise the downtime at all.

I should spend some time playing with the method for reducing downtime, but it’s hard to justify when the downtime is already short and acceptable. 🙂

One group of databases did prove problematic. It has done for previous upgrades also. The upgrades failed and I had to uninstall APEX, do a little bit of cleanup (dropping users), install it again, recreate the workspaces and redeploy the applications. It was a pain in the ass, but not difficult or time consuming. I have no idea why they failed, but as I said, it’s not the first time that has happened for this group of instances. If I had more time to play with it I’m sure I could find out, but as I know a reinstall fixes it, and time is an issue, you know what I’m going to do. Just run the scripts and move on…

As far as the applications themselves, there were no dramas. We try to keep things simple and most things are pretty vanilla, so it’s not like we are pushing any boundaries, which will expose anything. 🙂

I’ve seen some people have had problems with APEX 19.1 when it’s fronted by Apache and Nginx reverse proxies. We’ve got a mix of ORDS on Tomcat, and some things still using mod_plsql (don’t judge me 🙂 ) on OHS. This is all fronted by F5 Big IP load balancers as the reverse proxy, and we’ve not had any of those issues. 🙂

So all in all, pretty easy and what I was expecting.

Cheers

Tim…

PS. As always, I feel the need to point out we are a relatively small user of APEX, but it’s growing. Having said that, it’s installed on almost all of our Oracle databases. Added to that, I’m really aggressive about APEX upgrades, and will be even more so when the last of our systems move over to ORDS. Others may want to exercise a little more caution than me. 🙂

Oracle Database 19c : Installations, RAC, Data Guard and Upgrades

I’ve been playing around with Oracle Database 19c on LiveSQL since it was upgraded, and I pretty much thought that would be what I was stuck with until the on-prem release, as I don’t have an Exadata and it’s not on Oracle Cloud DBCS yet. Having seen a bunch people doing stuff on VMs, I got a bit frustrated and looked on eDelivery and low and behold the 19c software is available for download, even if you don’t have a Exadata CSI. I’m sure 18c was restricted during this period…

I’m pretty sure you wouldn’t be supported to use this for anything real (that wasn’t Exadata of course) until the on-prem drop, which will probably be 19.3 if they repeat what happened for 18c, but it does allow you to have a play.

Having a bunch of Vagrant environments for 18c already, meant it was pretty easy to test a whole bunch of 19c stuff within a few minutes, as most of the basics are very similar. Just minor changes to package recommendations. As a result I’ve pushed out the following stuff in the last couple of evenings.

Along the way I’ve committed a whole bunch of stuff to GitHub.

  • Vagrant build of 19c on OL7 with APEX and ORDS (here).
  • Vagrant build of 19c on Fedora 29 (here).
  • Vagrant hands-off build of 19c RAC on OL7 (here).
  • Vagrant hands-off build of 19c Data Guard on OL7 (here).
  • Docker 19c on OL7 build (here).
  • Docker compose (here) and swarm (here) stacks.

It should be obvious, but remember this is literally the first time I’ve done this stuff with 19c, so things will change over time. I just wanted to try some stuff out to see what happened, and have some test environments to play with while I’m checking out the new features. Once the real on-prem drop happens I’ll bring these up to date.

If nothing else, this is once again proof of how awesome automation is. A few minor tweaks and boom, there’s a new set of test environments. 🙂

Now I can get back to doing what I was meant to be doing… 🙂

Cheers

Tim…

Enterprise Linux Upgrade : It’s a Sorry State!

It’s 2017 and it’s still not possible to reliably upgrade an Enterprise Linux distribution between major versions!

At this point you are scrolling down to the comments to “educate me” about the redhat-upgrade-tool, because you read about it somewhere and you once heard someone successfully upgraded an installation on a lab machine. Unless you have an ultra vanilla starting point, you are going to end up with a mess that probably won’t boot. By the time you come to upgrade a “real server”, there have been years of changes and it is unlikely to resemble some pristine minimal installation.

I know your next comment is going to be something about the architectural changes brought in by project X and version Y in RHEL7, which is why it is all so hard. Stop now! You are boring me already! Is it an enterprise distribution or isn’t it? If it is, you should be able to upgrade it reliably!

Next up comes, “But you have to reinstall when you get new hardware!” I present to you virtual machines. Physical hardware upgrades with no need to reinstall the OS on the VM.

I can’t believe I’ve been using Linux for about 19 years and this is where we are at.

Cheers

Tim…

Oracle Upgrade/Migration : What method are you using?

I tweeted a couple of days ago about an important upgrade/migration I was doing yesterday. I was moving a smallish, but high profile, database from Oracle 11g on HP-UX Itanium to Oracle 12c on Oracle Linux inside a VMware VM. Almost immediately Joey D’Antoni came back with the question, “What method are you using?” I thought it might be worth writing a little something about the decision process.

When you are deciding how to upgrade and/or migrate a database there are a few things to consider:

  • Platform : Prior to Oracle 10g, if you wanted to change platforms, you didn’t have much of a choice, it was exp/imp only. As well as introducing data pump, Oracle 10g introduced the ability to convert datafiles and image copy backups using the RMAN CONVERT command. This meant transportable tablespaces were a valid option for platform migration, with or without upgrades.
  • Size : Above a certain size, waiting for an expdp/impdp is not practical. You are going to want to minimise downtime.
  • Allowable Downtime : Despite what the marketing people would have us believe, not all databases have to be 24×7. You should always try to minimise downtime, but it is not always necessary to eliminate it completely. If downtime is an issue there are options, but they may involve additional effort and cost.
  • Money : You can do almost anything if you are willing to throw time and money at it.
  • Junk : When you take a look at a database that has gone through successive upgrades, you can often see the lingering signs of old crap and bad decisions that will haunt you forever. It is really nice if you can start fresh and correct some of the mistakes of the past.

Not everyone has the luxury of large amounts of downtime, so what can you do to minimise downtime? Here are a few options, each having their own pros and cons, as well as restrictions depending on if your focus is migration, upgrade or both. It’s not an exhaustive list. 🙂

  • Transportable tablespaces, or transportable database, reduce migration time to about the time it takes to copy the files between servers. Depending on your storage tech, this could be a very short time indeed. The nice thing about this is it can be used across platforms and between versions, so an “upgrade” using TTS can be really quick. I used this for a Solaris to Oracle Linux move about a year ago.
  • You can use backups to restore a database to a new location, then keep recovering it using archived redo logs until the changeover time. Then ship the final archived redo logs, recovery the new database and you’re done. You could do the same thing with incrementally updated image copy backups. This is more about migration than upgrade.
  • When you are reading the previous point, you are probably thinking that sounds similar to Data Guard, and you could use this to switch machines and/or perform a rolling upgrade. If you are using standard edition, you could use Dbvisit Standby. Either way, the migration could be as quick as a switchover.
  • You can do an RMAN DUPLICATE to switch servers. It’s not going to upgrade your database, but you can use this to move it, then upgrade later.
  • You can replicate between the old and new system until your changeover point. You might use something like Golden Gate or Dbvisit Replicate to do this.
  • You can use the good old expdp/impdp. It’s probably going to be slow, and require a lot of downtime, but it’s logically simple.

There are lots of variations on a theme. The important point is you pick what’s right for you.

So what did I pick for this upgrade/migration? It was good old expdp/impdp. Why?

  • The database was relatively small.
  • The downtime involved was acceptable. It’s mostly accessed during the day and I started the process stupidly early to minimise the effect on the users.
  • It allowed me to clean up a lot of crap in the process. The database was about 18 years old and had been through previous upgrades and server migrations. It was bearing the scars of those and I wanted to clean it all up.
  • It allowed me to both upgrade and migrate in a single step, so it was logically very simple.

As always, there is not a *best* solution. You have to pick what is right for you and the constraints you are working with.

How did it go? Fine. We had already gone through the process in a Dev and Test environment, and I had done a run through on the new production kit also, so I knew it would work and I also had accurate timings, which made it easier to sell it to the decision makers.

Happy days!

Cheers

Tim…

PS. We will inevitably have some firefighting to do, as people always forget about the odd interface or application that connects to the system once in a blue moon. The old instance is turned off, so we should find out if anything has been forgotten pretty quick! 🙂

APEX 5.1 Installations and Upgrades

APEX 5.1 was released for download a few days ago. I tried doing an upgrade against an installation on a VM at home and it worked fine, which was hardly surprising. 🙂

Officially I’m on holiday, but I figured I would upgrade all our Dev/Test installations while everything is quiet. Major version upgrades, changes in either of the first two numbers, require a full installation. There was no major difference between this and what I was doing for the 5.0 installations, so I just edited the existing article and altered the title.

Since all the apps at work use AD authentication, I tested that out against 5.1 too and it worked fine.

So it was really smooth sailing.

As I’ve mentioned previously, we’re a small scale user of APEX, so it’s relatively easy for me to upgrade and test our systems.

We’ll kick the tyres some more in the new year, then upgrade the live systems pretty soon.

Cheers

Tim…

Enterprise Manager Cloud Control 13c Upgrade

em-12cA couple of weeks ago I posted about doing a fresh installation of Enterprise Manager Cloud Control 13c (article, blog post). I’ve finally got around to doing an upgrade test from EM CC 12cR5 to 13cR1. You can see the result of that here.

upgrade-meme

Gokhan Atil did a post about this upgrade pretty much as soon as it was released, so I’m a little late to the party compared to him. 🙂

As you’ll see from the article, the upgrade process was similar to the patches that came before it. There are of course some extra prerequisites which you can read about in either my post, Gokhan’s or the docs…

Even though the upgrade tests were fine, after discussion with our system administrators, we are probably going to go for a clean installation and migrate the monitored hosts one at a time.

Why the slash and burn approach? I’ve made some mistakes with our installations in the past and they persist with every subsequent upgrade. It would be nice to take a step back and fix stuff. We are doing a similar thing with our WebLogic installations. I was learning new stuff all the time while I was installing our WebLogic 11g infrastructure. Rather than upgrading to WebLogic 12cR2, we are going to build a new infrastructure, migrate to it and throw the old one away.

This is relatively easy for us for a few reasons.

  1. We use virtualization for everything. We will provision the new VMs, set everything up. Start migrating stuff. When the migration is complete we will throw away the old VMs. No major hardware overhead.
  2. We are a pretty small operation. If we had a massive amount of infrastructure, a slash and burn approach would be very time consuming and as such, very costly.
  3. I am really anal about some things and I am willing to go the extra mile to get things right. I did the best I could at the time, but I’m happy to admit I made mistakes and I want to sort them out. This is not because I’m a company boy. It’s because those mistakes eat away at me and I want them eradicated so they will only haunt me in my memories, not in my day to day life.

If we had been going for the upgrade approach, I probably would have done it in the next couple of weeks. With clean slate approach, we’ll probably take a few more weeks to get ready for it. No point rushing in and making more mistakes. I would rather let the idea brew for a while before we start. 🙂

Cheers

Tim…

Cloud Control 12.1.0.5 : It’s production upgrade day…

cloudI mentioned a couple of months ago I was planning to upgrade our production Enterprise Manager Cloud Control installation from 12.1.0.4 to 12.1.0.5. Well, today was the day. I held back a while because I knew I would be out of the country for a while on the Latin America tour and I didn’t want to make a big change before I ran away. 🙂

So today I pretty much did exactly what was in my upgrade article and everything went well. I upgraded the OMS and the local agent and I’ll run like that for a couple of days before I start pushing out the agent updates to the monitored hosts.

Happy days!

If you are interested, you can see some of my Cloud Control articles here.

Cheers

Tim…

Upgrade Cloud Control 12cR4 to 12cR5

em-12cA couple of weeks ago I wrote a post about doing a Cloud Control 12cR5 installation and said I would be testing the upgrade from 12cR4. I’ve now done that.

The upgrade documentation is quite extensive and the prerequisites are going to be different depending on the database and cloud control versions you are starting with, so this is no way a “recommended” way to do the upgrade. Each one will need to be approached on a case-by-case basis. It’s just meant to give a flavour of what you have to do.

Suffice to say, it worked fine for me. 🙂

Cheers

Tim…