APEX 19.1 Upgrades : My Experience

With the exception of one application, which is taking a little longer to test, we’ve upgraded all our APEX installations to APEX 19.1.

I thought it would be good to share my experience of this, in case anyone is a little on the cautious side. 🙂

The vast majority of upgrades went fine. We just did the standard upgrade (install), without trying to minimise the downtime at all.

I should spend some time playing with the method for reducing downtime, but it’s hard to justify when the downtime is already short and acceptable. 🙂

One group of databases did prove problematic. It has done for previous upgrades also. The upgrades failed and I had to uninstall APEX, do a little bit of cleanup (dropping users), install it again, recreate the workspaces and redeploy the applications. It was a pain in the ass, but not difficult or time consuming. I have no idea why they failed, but as I said, it’s not the first time that has happened for this group of instances. If I had more time to play with it I’m sure I could find out, but as I know a reinstall fixes it, and time is an issue, you know what I’m going to do. Just run the scripts and move on…

As far as the applications themselves, there were no dramas. We try to keep things simple and most things are pretty vanilla, so it’s not like we are pushing any boundaries, which will expose anything. 🙂

I’ve seen some people have had problems with APEX 19.1 when it’s fronted by Apache and Nginx reverse proxies. We’ve got a mix of ORDS on Tomcat, and some things still using mod_plsql (don’t judge me 🙂 ) on OHS. This is all fronted by F5 Big IP load balancers as the reverse proxy, and we’ve not had any of those issues. 🙂

So all in all, pretty easy and what I was expecting.

Cheers

Tim…

PS. As always, I feel the need to point out we are a relatively small user of APEX, but it’s growing. Having said that, it’s installed on almost all of our Oracle databases. Added to that, I’m really aggressive about APEX upgrades, and will be even more so when the last of our systems move over to ORDS. Others may want to exercise a little more caution than me. 🙂

Oracle Database 19c : Installations, RAC, Data Guard and Upgrades

I’ve been playing around with Oracle Database 19c on LiveSQL since it was upgraded, and I pretty much thought that would be what I was stuck with until the on-prem release, as I don’t have an Exadata and it’s not on Oracle Cloud DBCS yet. Having seen a bunch people doing stuff on VMs, I got a bit frustrated and looked on eDelivery and low and behold the 19c software is available for download, even if you don’t have a Exadata CSI. I’m sure 18c was restricted during this period…

I’m pretty sure you wouldn’t be supported to use this for anything real (that wasn’t Exadata of course) until the on-prem drop, which will probably be 19.3 if they repeat what happened for 18c, but it does allow you to have a play.

Having a bunch of Vagrant environments for 18c already, meant it was pretty easy to test a whole bunch of 19c stuff within a few minutes, as most of the basics are very similar. Just minor changes to package recommendations. As a result I’ve pushed out the following stuff in the last couple of evenings.

Along the way I’ve committed a whole bunch of stuff to GitHub.

  • Vagrant build of 19c on OL7 with APEX and ORDS (here).
  • Vagrant build of 19c on Fedora 29 (here).
  • Vagrant hands-off build of 19c RAC on OL7 (here).
  • Vagrant hands-off build of 19c Data Guard on OL7 (here).
  • Docker 19c on OL7 build (here).
  • Docker compose (here) and swarm (here) stacks.

It should be obvious, but remember this is literally the first time I’ve done this stuff with 19c, so things will change over time. I just wanted to try some stuff out to see what happened, and have some test environments to play with while I’m checking out the new features. Once the real on-prem drop happens I’ll bring these up to date.

If nothing else, this is once again proof of how awesome automation is. A few minor tweaks and boom, there’s a new set of test environments. 🙂

Now I can get back to doing what I was meant to be doing… 🙂

Cheers

Tim…

Enterprise Linux Upgrade : It’s a Sorry State!

It’s 2017 and it’s still not possible to reliably upgrade an Enterprise Linux distribution between major versions!

At this point you are scrolling down to the comments to “educate me” about the redhat-upgrade-tool, because you read about it somewhere and you once heard someone successfully upgraded an installation on a lab machine. Unless you have an ultra vanilla starting point, you are going to end up with a mess that probably won’t boot. By the time you come to upgrade a “real server”, there have been years of changes and it is unlikely to resemble some pristine minimal installation.

I know your next comment is going to be something about the architectural changes brought in by project X and version Y in RHEL7, which is why it is all so hard. Stop now! You are boring me already! Is it an enterprise distribution or isn’t it? If it is, you should be able to upgrade it reliably!

Next up comes, “But you have to reinstall when you get new hardware!” I present to you virtual machines. Physical hardware upgrades with no need to reinstall the OS on the VM.

I can’t believe I’ve been using Linux for about 19 years and this is where we are at.

Cheers

Tim…

Oracle Upgrade/Migration : What method are you using?

I tweeted a couple of days ago about an important upgrade/migration I was doing yesterday. I was moving a smallish, but high profile, database from Oracle 11g on HP-UX Itanium to Oracle 12c on Oracle Linux inside a VMware VM. Almost immediately Joey D’Antoni came back with the question, “What method are you using?” I thought it might be worth writing a little something about the decision process.

When you are deciding how to upgrade and/or migrate a database there are a few things to consider:

  • Platform : Prior to Oracle 10g, if you wanted to change platforms, you didn’t have much of a choice, it was exp/imp only. As well as introducing data pump, Oracle 10g introduced the ability to convert datafiles and image copy backups using the RMAN CONVERT command. This meant transportable tablespaces were a valid option for platform migration, with or without upgrades.
  • Size : Above a certain size, waiting for an expdp/impdp is not practical. You are going to want to minimise downtime.
  • Allowable Downtime : Despite what the marketing people would have us believe, not all databases have to be 24×7. You should always try to minimise downtime, but it is not always necessary to eliminate it completely. If downtime is an issue there are options, but they may involve additional effort and cost.
  • Money : You can do almost anything if you are willing to throw time and money at it.
  • Junk : When you take a look at a database that has gone through successive upgrades, you can often see the lingering signs of old crap and bad decisions that will haunt you forever. It is really nice if you can start fresh and correct some of the mistakes of the past.

Not everyone has the luxury of large amounts of downtime, so what can you do to minimise downtime? Here are a few options, each having their own pros and cons, as well as restrictions depending on if your focus is migration, upgrade or both. It’s not an exhaustive list. 🙂

  • Transportable tablespaces, or transportable database, reduce migration time to about the time it takes to copy the files between servers. Depending on your storage tech, this could be a very short time indeed. The nice thing about this is it can be used across platforms and between versions, so an “upgrade” using TTS can be really quick. I used this for a Solaris to Oracle Linux move about a year ago.
  • You can use backups to restore a database to a new location, then keep recovering it using archived redo logs until the changeover time. Then ship the final archived redo logs, recovery the new database and you’re done. You could do the same thing with incrementally updated image copy backups. This is more about migration than upgrade.
  • When you are reading the previous point, you are probably thinking that sounds similar to Data Guard, and you could use this to switch machines and/or perform a rolling upgrade. If you are using standard edition, you could use Dbvisit Standby. Either way, the migration could be as quick as a switchover.
  • You can do an RMAN DUPLICATE to switch servers. It’s not going to upgrade your database, but you can use this to move it, then upgrade later.
  • You can replicate between the old and new system until your changeover point. You might use something like Golden Gate or Dbvisit Replicate to do this.
  • You can use the good old expdp/impdp. It’s probably going to be slow, and require a lot of downtime, but it’s logically simple.

There are lots of variations on a theme. The important point is you pick what’s right for you.

So what did I pick for this upgrade/migration? It was good old expdp/impdp. Why?

  • The database was relatively small.
  • The downtime involved was acceptable. It’s mostly accessed during the day and I started the process stupidly early to minimise the effect on the users.
  • It allowed me to clean up a lot of crap in the process. The database was about 18 years old and had been through previous upgrades and server migrations. It was bearing the scars of those and I wanted to clean it all up.
  • It allowed me to both upgrade and migrate in a single step, so it was logically very simple.

As always, there is not a *best* solution. You have to pick what is right for you and the constraints you are working with.

How did it go? Fine. We had already gone through the process in a Dev and Test environment, and I had done a run through on the new production kit also, so I knew it would work and I also had accurate timings, which made it easier to sell it to the decision makers.

Happy days!

Cheers

Tim…

PS. We will inevitably have some firefighting to do, as people always forget about the odd interface or application that connects to the system once in a blue moon. The old instance is turned off, so we should find out if anything has been forgotten pretty quick! 🙂

APEX 5.1 Installations and Upgrades

APEX 5.1 was released for download a few days ago. I tried doing an upgrade against an installation on a VM at home and it worked fine, which was hardly surprising. 🙂

Officially I’m on holiday, but I figured I would upgrade all our Dev/Test installations while everything is quiet. Major version upgrades, changes in either of the first two numbers, require a full installation. There was no major difference between this and what I was doing for the 5.0 installations, so I just edited the existing article and altered the title.

Since all the apps at work use AD authentication, I tested that out against 5.1 too and it worked fine.

So it was really smooth sailing.

As I’ve mentioned previously, we’re a small scale user of APEX, so it’s relatively easy for me to upgrade and test our systems.

We’ll kick the tyres some more in the new year, then upgrade the live systems pretty soon.

Cheers

Tim…

Enterprise Manager Cloud Control 13c Upgrade

em-12cA couple of weeks ago I posted about doing a fresh installation of Enterprise Manager Cloud Control 13c (article, blog post). I’ve finally got around to doing an upgrade test from EM CC 12cR5 to 13cR1. You can see the result of that here.

upgrade-meme

Gokhan Atil did a post about this upgrade pretty much as soon as it was released, so I’m a little late to the party compared to him. 🙂

As you’ll see from the article, the upgrade process was similar to the patches that came before it. There are of course some extra prerequisites which you can read about in either my post, Gokhan’s or the docs…

Even though the upgrade tests were fine, after discussion with our system administrators, we are probably going to go for a clean installation and migrate the monitored hosts one at a time.

Why the slash and burn approach? I’ve made some mistakes with our installations in the past and they persist with every subsequent upgrade. It would be nice to take a step back and fix stuff. We are doing a similar thing with our WebLogic installations. I was learning new stuff all the time while I was installing our WebLogic 11g infrastructure. Rather than upgrading to WebLogic 12cR2, we are going to build a new infrastructure, migrate to it and throw the old one away.

This is relatively easy for us for a few reasons.

  1. We use virtualization for everything. We will provision the new VMs, set everything up. Start migrating stuff. When the migration is complete we will throw away the old VMs. No major hardware overhead.
  2. We are a pretty small operation. If we had a massive amount of infrastructure, a slash and burn approach would be very time consuming and as such, very costly.
  3. I am really anal about some things and I am willing to go the extra mile to get things right. I did the best I could at the time, but I’m happy to admit I made mistakes and I want to sort them out. This is not because I’m a company boy. It’s because those mistakes eat away at me and I want them eradicated so they will only haunt me in my memories, not in my day to day life.

If we had been going for the upgrade approach, I probably would have done it in the next couple of weeks. With clean slate approach, we’ll probably take a few more weeks to get ready for it. No point rushing in and making more mistakes. I would rather let the idea brew for a while before we start. 🙂

Cheers

Tim…

Cloud Control 12.1.0.5 : It’s production upgrade day…

cloudI mentioned a couple of months ago I was planning to upgrade our production Enterprise Manager Cloud Control installation from 12.1.0.4 to 12.1.0.5. Well, today was the day. I held back a while because I knew I would be out of the country for a while on the Latin America tour and I didn’t want to make a big change before I ran away. 🙂

So today I pretty much did exactly what was in my upgrade article and everything went well. I upgraded the OMS and the local agent and I’ll run like that for a couple of days before I start pushing out the agent updates to the monitored hosts.

Happy days!

If you are interested, you can see some of my Cloud Control articles here.

Cheers

Tim…

Upgrade Cloud Control 12cR4 to 12cR5

em-12cA couple of weeks ago I wrote a post about doing a Cloud Control 12cR5 installation and said I would be testing the upgrade from 12cR4. I’ve now done that.

The upgrade documentation is quite extensive and the prerequisites are going to be different depending on the database and cloud control versions you are starting with, so this is no way a “recommended” way to do the upgrade. Each one will need to be approached on a case-by-case basis. It’s just meant to give a flavour of what you have to do.

Suffice to say, it worked fine for me. 🙂

Cheers

Tim…

Fedora 21 : Upgrading from Fedora 20

I just did an upgrade of my old desktop from Fedora 20 to Fedora 21. The process was similar to this old blog post, but there were some variations, so I’ll list the procedure here.

  • Update your current Fedora 20 system by issuing the “yum update -y” command and restart once it is complete.
  • Install the latest “fedup” package using “sudo yum –enablerepo=updates-testing install fedup”
  • Run the “sudo fedup-cli –network 21 –product=nonproduct” command.
  • If you are using Dropbox, disable the repository using the “yum-config-manager –disable Dropbox” command. Re-enable it once the Fedora 21 repository is available.
  • Run the following clean up commands.
    sudo rpm --rebuilddb
    sudo yum distro-sync --setopt=deltarpm=0
    
    sudo yum install rpmconf
    sudo rpmconf -a
  • If you are using Chrome, uninstall and reinstall Chrome.

It seemed to go fine!

Cheers

Tim…

Enterprise Manager Cloud Control 12cR4 Production Upgrade

I’ve already written about the 12cR3 to 12cR4 upgrade here. I did a few run through’s at home to practice it and it all seemed good.

Setting The Scene

Just to set the scene, for our production environment we run Cloud Control in a VMware virtual machine, using Oracle Linux 6.5 as the guest OS. With that setup, we can use a simple installation (DB and OMS on the same VM) and use VMware to provide our failover, rather than having to worry about multiple OMS installations and any DB failover technology etc. If there’s one thing I’ve learned about Cloud Control, it’s Keep It Simple Stupid (KISS)! As far as our managed servers go, most of our databases and all our middle tier stuff runs on VMware and Oracle Linux too. We have a handful of things still hanging around on HP-UX and Solaris, which will hopefully be migrated soon…

Upgrade Attempt 1 : Non-Starter

Yesterday I started the upgrade of our production system. Pretty much straight out of the blocks I hit a road block. It didn’t like the agents running on our HP-UX servers. The upgrades of the HP-UX agents are so painful. Every time so far I’ve had to reinstall them. As a result, I didn’t bother to upgrade them last time and kept running with the previous version of the agents. The upgrade wouldn’t have anything to do with that, so I forgot about the Cloud Control upgrade and I spent yesterday attempting to upgrade the HP-UX agents to 12cR3, before I could attempt the 12cR4 Cloud Control upgrade.

As usual, the upgrade of the agents on HP-UX involved me uninstalling, removing all the targets, installing, discovering all the targets and setting up the backups etc. Not all of it is scripted yet, so it is an annoying and painful process. I’m not sure if other HP-UX users suffer this, but it seems pretty consistently bad for us. The sooner we get rid of these straggling HP-UX servers the better!

So this wasn’t so much a failure of the upgrade. It was really down to me being lazy and not bothering to upgrade some agents.

Fast forward to this morning and I was actually ready to start the upgrade. 🙂

Upgrade Attempt 2 : Success

With the 12cR3 agents in place on HP-UX, the upgrade ran past that step with no problems and on to the main body of the installation. The install and upgrade were textbook.

I’ve upgraded the agent on the cloud control server, but I’m not going to upgrade any of the other agents until I know things are working fine.

Happy days!

Cheers

Tim…