Enterprise Linux Upgrade : It’s a Sorry State!

It’s 2017 and it’s still not possible to reliably upgrade an Enterprise Linux distribution between major versions!

At this point you are scrolling down to the comments to “educate me” about the redhat-upgrade-tool, because you read about it somewhere and you once heard someone successfully upgraded an installation on a lab machine. Unless you have an ultra vanilla starting point, you are going to end up with a mess that probably won’t boot. By the time you come to upgrade a “real server”, there have been years of changes and it is unlikely to resemble some pristine minimal installation.

I know your next comment is going to be something about the architectural changes brought in by project X and version Y in RHEL7, which is why it is all so hard. Stop now! You are boring me already! Is it an enterprise distribution or isn’t it? If it is, you should be able to upgrade it reliably!

Next up comes, “But you have to reinstall when you get new hardware!” I present to you virtual machines. Physical hardware upgrades with no need to reinstall the OS on the VM.

I can’t believe I’ve been using Linux for about 19 years and this is where we are at.

Cheers

Tim…

Oracle Upgrade/Migration : What method are you using?

I tweeted a couple of days ago about an important upgrade/migration I was doing yesterday. I was moving a smallish, but high profile, database from Oracle 11g on HP-UX Itanium to Oracle 12c on Oracle Linux inside a VMware VM. Almost immediately Joey D’Antoni came back with the question, “What method are you using?” I thought it might be worth writing a little something about the decision process.

When you are deciding how to upgrade and/or migrate a database there are a few things to consider:

  • Platform : Prior to Oracle 10g, if you wanted to change platforms, you didn’t have much of a choice, it was exp/imp only. As well as introducing data pump, Oracle 10g introduced the ability to convert datafiles and image copy backups using the RMAN CONVERT command. This meant transportable tablespaces were a valid option for platform migration, with or without upgrades.
  • Size : Above a certain size, waiting for an expdp/impdp is not practical. You are going to want to minimise downtime.
  • Allowable Downtime : Despite what the marketing people would have us believe, not all databases have to be 24×7. You should always try to minimise downtime, but it is not always necessary to eliminate it completely. If downtime is an issue there are options, but they may involve additional effort and cost.
  • Money : You can do almost anything if you are willing to throw time and money at it.
  • Junk : When you take a look at a database that has gone through successive upgrades, you can often see the lingering signs of old crap and bad decisions that will haunt you forever. It is really nice if you can start fresh and correct some of the mistakes of the past.

Not everyone has the luxury of large amounts of downtime, so what can you do to minimise downtime? Here are a few options, each having their own pros and cons, as well as restrictions depending on if your focus is migration, upgrade or both. It’s not an exhaustive list. 🙂

  • Transportable tablespaces, or transportable database, reduce migration time to about the time it takes to copy the files between servers. Depending on your storage tech, this could be a very short time indeed. The nice thing about this is it can be used across platforms and between versions, so an “upgrade” using TTS can be really quick. I used this for a Solaris to Oracle Linux move about a year ago.
  • You can use backups to restore a database to a new location, then keep recovering it using archived redo logs until the changeover time. Then ship the final archived redo logs, recovery the new database and you’re done. You could do the same thing with incrementally updated image copy backups. This is more about migration than upgrade.
  • When you are reading the previous point, you are probably thinking that sounds similar to Data Guard, and you could use this to switch machines and/or perform a rolling upgrade. If you are using standard edition, you could use Dbvisit Standby. Either way, the migration could be as quick as a switchover.
  • You can do an RMAN DUPLICATE to switch servers. It’s not going to upgrade your database, but you can use this to move it, then upgrade later.
  • You can replicate between the old and new system until your changeover point. You might use something like Golden Gate or Dbvisit Replicate to do this.
  • You can use the good old expdp/impdp. It’s probably going to be slow, and require a lot of downtime, but it’s logically simple.

There are lots of variations on a theme. The important point is you pick what’s right for you.

So what did I pick for this upgrade/migration? It was good old expdp/impdp. Why?

  • The database was relatively small.
  • The downtime involved was acceptable. It’s mostly accessed during the day and I started the process stupidly early to minimise the effect on the users.
  • It allowed me to clean up a lot of crap in the process. The database was about 18 years old and had been through previous upgrades and server migrations. It was bearing the scars of those and I wanted to clean it all up.
  • It allowed me to both upgrade and migrate in a single step, so it was logically very simple.

As always, there is not a *best* solution. You have to pick what is right for you and the constraints you are working with.

How did it go? Fine. We had already gone through the process in a Dev and Test environment, and I had done a run through on the new production kit also, so I knew it would work and I also had accurate timings, which made it easier to sell it to the decision makers.

Happy days!

Cheers

Tim…

PS. We will inevitably have some firefighting to do, as people always forget about the odd interface or application that connects to the system once in a blue moon. The old instance is turned off, so we should find out if anything has been forgotten pretty quick! 🙂

APEX 5.1 Installations and Upgrades

APEX 5.1 was released for download a few days ago. I tried doing an upgrade against an installation on a VM at home and it worked fine, which was hardly surprising. 🙂

Officially I’m on holiday, but I figured I would upgrade all our Dev/Test installations while everything is quiet. Major version upgrades, changes in either of the first two numbers, require a full installation. There was no major difference between this and what I was doing for the 5.0 installations, so I just edited the existing article and altered the title.

Since all the apps at work use AD authentication, I tested that out against 5.1 too and it worked fine.

So it was really smooth sailing.

As I’ve mentioned previously, we’re a small scale user of APEX, so it’s relatively easy for me to upgrade and test our systems.

We’ll kick the tyres some more in the new year, then upgrade the live systems pretty soon.

Cheers

Tim…

Enterprise Manager Cloud Control 13c Upgrade

em-12cA couple of weeks ago I posted about doing a fresh installation of Enterprise Manager Cloud Control 13c (article, blog post). I’ve finally got around to doing an upgrade test from EM CC 12cR5 to 13cR1. You can see the result of that here.

upgrade-meme

Gokhan Atil did a post about this upgrade pretty much as soon as it was released, so I’m a little late to the party compared to him. 🙂

As you’ll see from the article, the upgrade process was similar to the patches that came before it. There are of course some extra prerequisites which you can read about in either my post, Gokhan’s or the docs…

Even though the upgrade tests were fine, after discussion with our system administrators, we are probably going to go for a clean installation and migrate the monitored hosts one at a time.

Why the slash and burn approach? I’ve made some mistakes with our installations in the past and they persist with every subsequent upgrade. It would be nice to take a step back and fix stuff. We are doing a similar thing with our WebLogic installations. I was learning new stuff all the time while I was installing our WebLogic 11g infrastructure. Rather than upgrading to WebLogic 12cR2, we are going to build a new infrastructure, migrate to it and throw the old one away.

This is relatively easy for us for a few reasons.

  1. We use virtualization for everything. We will provision the new VMs, set everything up. Start migrating stuff. When the migration is complete we will throw away the old VMs. No major hardware overhead.
  2. We are a pretty small operation. If we had a massive amount of infrastructure, a slash and burn approach would be very time consuming and as such, very costly.
  3. I am really anal about some things and I am willing to go the extra mile to get things right. I did the best I could at the time, but I’m happy to admit I made mistakes and I want to sort them out. This is not because I’m a company boy. It’s because those mistakes eat away at me and I want them eradicated so they will only haunt me in my memories, not in my day to day life.

If we had been going for the upgrade approach, I probably would have done it in the next couple of weeks. With clean slate approach, we’ll probably take a few more weeks to get ready for it. No point rushing in and making more mistakes. I would rather let the idea brew for a while before we start. 🙂

Cheers

Tim…

Cloud Control 12.1.0.5 : It’s production upgrade day…

cloudI mentioned a couple of months ago I was planning to upgrade our production Enterprise Manager Cloud Control installation from 12.1.0.4 to 12.1.0.5. Well, today was the day. I held back a while because I knew I would be out of the country for a while on the Latin America tour and I didn’t want to make a big change before I ran away. 🙂

So today I pretty much did exactly what was in my upgrade article and everything went well. I upgraded the OMS and the local agent and I’ll run like that for a couple of days before I start pushing out the agent updates to the monitored hosts.

Happy days!

If you are interested, you can see some of my Cloud Control articles here.

Cheers

Tim…

Upgrade Cloud Control 12cR4 to 12cR5

em-12cA couple of weeks ago I wrote a post about doing a Cloud Control 12cR5 installation and said I would be testing the upgrade from 12cR4. I’ve now done that.

The upgrade documentation is quite extensive and the prerequisites are going to be different depending on the database and cloud control versions you are starting with, so this is no way a “recommended” way to do the upgrade. Each one will need to be approached on a case-by-case basis. It’s just meant to give a flavour of what you have to do.

Suffice to say, it worked fine for me. 🙂

Cheers

Tim…

Fedora 21 : Upgrading from Fedora 20

I just did an upgrade of my old desktop from Fedora 20 to Fedora 21. The process was similar to this old blog post, but there were some variations, so I’ll list the procedure here.

  • Update your current Fedora 20 system by issuing the “yum update -y” command and restart once it is complete.
  • Install the latest “fedup” package using “sudo yum –enablerepo=updates-testing install fedup”
  • Run the “sudo fedup-cli –network 21 –product=nonproduct” command.
  • If you are using Dropbox, disable the repository using the “yum-config-manager –disable Dropbox” command. Re-enable it once the Fedora 21 repository is available.
  • Run the following clean up commands.
    sudo rpm --rebuilddb
    sudo yum distro-sync --setopt=deltarpm=0
    
    sudo yum install rpmconf
    sudo rpmconf -a
  • If you are using Chrome, uninstall and reinstall Chrome.

It seemed to go fine!

Cheers

Tim…

Enterprise Manager Cloud Control 12cR4 Production Upgrade

I’ve already written about the 12cR3 to 12cR4 upgrade here. I did a few run through’s at home to practice it and it all seemed good.

Setting The Scene

Just to set the scene, for our production environment we run Cloud Control in a VMware virtual machine, using Oracle Linux 6.5 as the guest OS. With that setup, we can use a simple installation (DB and OMS on the same VM) and use VMware to provide our failover, rather than having to worry about multiple OMS installations and any DB failover technology etc. If there’s one thing I’ve learned about Cloud Control, it’s Keep It Simple Stupid (KISS)! As far as our managed servers go, most of our databases and all our middle tier stuff runs on VMware and Oracle Linux too. We have a handful of things still hanging around on HP-UX and Solaris, which will hopefully be migrated soon…

Upgrade Attempt 1 : Non-Starter

Yesterday I started the upgrade of our production system. Pretty much straight out of the blocks I hit a road block. It didn’t like the agents running on our HP-UX servers. The upgrades of the HP-UX agents are so painful. Every time so far I’ve had to reinstall them. As a result, I didn’t bother to upgrade them last time and kept running with the previous version of the agents. The upgrade wouldn’t have anything to do with that, so I forgot about the Cloud Control upgrade and I spent yesterday attempting to upgrade the HP-UX agents to 12cR3, before I could attempt the 12cR4 Cloud Control upgrade.

As usual, the upgrade of the agents on HP-UX involved me uninstalling, removing all the targets, installing, discovering all the targets and setting up the backups etc. Not all of it is scripted yet, so it is an annoying and painful process. I’m not sure if other HP-UX users suffer this, but it seems pretty consistently bad for us. The sooner we get rid of these straggling HP-UX servers the better!

So this wasn’t so much a failure of the upgrade. It was really down to me being lazy and not bothering to upgrade some agents.

Fast forward to this morning and I was actually ready to start the upgrade. 🙂

Upgrade Attempt 2 : Success

With the 12cR3 agents in place on HP-UX, the upgrade ran past that step with no problems and on to the main body of the installation. The install and upgrade were textbook.

I’ve upgraded the agent on the cloud control server, but I’m not going to upgrade any of the other agents until I know things are working fine.

Happy days!

Cheers

Tim…

Fedora 20 : Upgrade from Fedora 19

It’s a little over a month since Fedora 20 was released, but during a terrible bout of insomnia last night I decided to upgrade my desktop PC.

The upgrades using “fedup” worked fine for the previous releases (Fedora 18, Fedora 19). Unfortunately, it failed abysmally for the upgrade to Fedora 20. I tried a few times, but I was not able to troubleshoot it, so I gave up and did a reinstall.

I’ve got an SSD for the system drive, but keep almost everything of importance on a second drive (and a backup drive). I tend to do most things in VMs, so I ended up doing the following:

  • Backup.
  • Copy a few config files to the second drive (smb.conf, hosts, fstab etc).
  • Clean installation on the SSD, not touching the second drive.
  • Put the mount information back into the “/etc/fstab” and mount the second drive.
  • Put the “/etc/hosts” file back in place and install dnsmasq.
  • Put the “smb.conf” file back in place and start samba.
  • Do a “yum update -y” and reboot.
  • Install some utilities, like UltraEdit, VirtualBox, DropBox and Chrome etc.
  • Open up the existing VMs on the second drive using the newly installed VirtualBox.
  • Backup.

That was pretty much it really. I’m, back up and running with a clean OS installation and I guess it took less than an hour from start to finish. I think in future I’ll avoid upgrades. There’s something nice about a sparkly new installation, without any of the old crap left hanging around.

During the installation, I picked MATE as my desktop. I’ve tried the others and this is the one that feels the most natural to me.

Cheers

Tim…

Captain Support and the Windows 8.1 Upgrades

Being the adventurous type of guy he is, Captain Support decided to launch into Windows 8.1 upgrades on his Mom’s and sister-in-law’s laptops. They were identical machines, both running Windows 8 and configured the same. One was local and the other connected to over LogMeIn…

The first thing he noticed about the upgrade is the size of it, approximately 3G. The download and initial install can be done while you’re still using the machine, then comes the inevitable reboot where the real work is done…

The second standout point was the update forced him to him to switch from a local user to a Microsoft Live login. Both Captain Support’s Mom and sister-in-law both use Hotmail/Outlook.com, so this did not present an immediate issue, but it was annoying. Perhaps there is a way to avoid this, but it was not immediately appareent to Captain Support… You can still create local users after the update of course…

The third annoyance was that of the two machines, one upgraded fine and kept all it’s customizations. The other upgraded OK, but seemed to lose some of it’s customizations, including Classic Shell. He was not sure if this related to the LogMeIn access or not. Fortunately, there was not much repair needed.

So after a bit of messing about, Captain Support had two Windows 8.1 laptops, that looked and felt just like Windows 8.0. 🙂

Whilst using the beta version of Windows 8.1, Captain Support noticed that his Citrix login for work was broken. Once the laptop upgrades were complete Captain Support noticed a problem at work and tried out the Citrix client on the production version of Windows 8.1 and it worked fine, so he was able to log in and save the day, or at least the backups of a dev system…

In the airport today I noticed that IE 11 on Windows 8.1 is reported to break some websites, including a number of Google services. Good job nobody uses IE these days. When I get a chance to contact Captain Support I will tell him what I read, in case the has to fly in and rescue his Mom and sister-in-law…

If any drama ensues I’m sure Captain Support will tell me about it, so I can pass it on…

Cheers

Captain Support (reported by Tim…)

PS. I wish I could fly like Captain Support. Aeroplane travel sucks…