Oracle Database 18.7 Patch : Summary

I wrote a bunch of angry tweets yesterday about the 18.7 patch, which probably weren’t very coherent. I just thought I would summarise them here.

Before we start, I think it’s worth saying I try to keep things as vanilla as possible, so we rarely have problems with patches. I guess this is why I launched into these so quickly, and was surprised by how much hassle I had. So far I’ve only been doing the 18.7 patches on servers running OL6 (hence not upgraded to 19c yet). I have no idea at this point if patches for 19c and the lingering 12.1 instances will have problems or not (see updates below).

Issue 1

A few days ago I had upgraded a development database from 12.1.0.2 to 18.6. The upgrade went fine. No worries. Yesterday I patched that same instance from 18.6 to 18.7. The datapatch run complained about invalid objects, specifically some of our packages (not Oracle packages) that were invalid, because database links they referenced were not working.

The schema in question contains some archived tables and code, and the servers these DB links reference no longer exist. In an ideal world all this would be cleaned up, but that would take development time that we don’t have. Welcome to the real world.

At the end of the datapatch run the PDB was left in restricted mode, so I did the following.

  • Granted restricted session to the user.
  • Proxied in.
  • Dropped the invalid objects in question.
  • Revoked restricted session from the user.
  • Ran datapatch again.

At the end of that I had a functioning instance.

That sounds like it was relatively smooth and controlled, but of there was lots of checking log files, Googling for bugs and all that stuff before I eventually just dropped the objects and moved on.

Issue 2 (Issue 1b)

The second issue was a kind-of rerun of the first. This time it was an upgrade from 12.1.0.2 to 18.7 directly. This instance was similar to the previous one, as they were both clones of the same production instance. Unfortunately, I was doing this at the same time as the patch, so I couldn’t learn the lesson from the previous one.

The upgrade went fine until datapatch ran, then boom. Same problem has before. Also, same resolution, so I got through this pretty quick.

If I had completed the patch before starting this upgrade, I might have got though this with the preemptive cleanup, but maybe not. See Issue 3.

Issue 3

Having got through two development databases and learned the lessons, I did a preemptive cleanup of the offending objects from the UAT instance. It also was based on a clone of the same production database as the previous two.

Having done the clean-up, I started the upgrade from 12.1.0.2 to 18.7 feeling pretty confident.

The upgrade went fine, but when it ran datapatch it failed with an issue that looked very much like that described in this MOS note Doc ID 2054286.1.

I say “looked very much like”, rather than “was this”, because that issue relates to patching multiple instances on the same machine at the same time. This was datapatch running against a single instance on this machine. The symptom looked the same…

The resolution for this was to run datapatch again. It just ran through fine. No drama.

Conclusion

I’m not looking forward to upgrading the production instance to 18.7. All these databases were clones of it, so I suspect I’m going to have some drama. Luckily the project can tolerate some downtime.

I’m also curious how the 12.1.0.2 and 19c patches are going to play out… (see updates below).

So if you were following my various rants on Twitter yesterday and wondered why I was so angry, this is why. Keep in mind also, these were “background tasks” I threw in on top of the rest of the stuff I was meant to be doing yesterday.

I hope this was just a problem with my instances or me, not indicative of the quality of the patch. It is interesting that 18.6 was happy with it, but 18.7 was a nightmare…

Good luck everyone, and remember, don’t give me a job. I can’t be trusted… 🙂

Cheers

Tim…

Update 1: Sven-Olaf Hilmer wrote on Twitter.

“I have a problem with 12.1.0.2.190716DBBP. RMAN crashes on all patched DBs. Rollback to 190416 solves the problem. SR filed. 11.2, 12.2.0.1, 18, 19 are not affected.”

Update 2: Anil Vejendla wrote on Twitter.

“We are facing restricted mode issue for 12.2 after applying July patch and even after rollback also issue still persists.”

APEX 5.0.1 : We’re all patched up!

apexAPEX 5.0.1 was released about a week ago. I started to patch some stuff straight away. We were already on APEX 5.0 across the board, so we didn’t need to do any full installations, just patches.

During the patching I noticed we were getting some issues with supposed misconfiguration of static files. After clearing my browser cache, the message went away, so I tweeted this.

“Regarding APEX 5.0.1 patch. Clear your browser cache, or you may get that static files message, even though config is correct. 🙂 #orclapex”

Practically before I hit return, Patrick Wolf came back with this.

“@oraclebase Thanks for letting us know. I have filed bug# 21463521 and fixed it for 5.0.2”

We’re a week on now and all our APEX installations are happily running 5.0.1. We had no issues with the upgrades and no problems with the apps.

We are small fry where APEX is concerned, so don’t take this as the green light to upgrade everything yourself. I’m just saying we had no problems with it, which is pretty cool. If APEX is a strategic environment for you, you will probably need to do a lot more testing than us. 🙂

Cheers

Tim…

To patch or not to patch?

Mary Ann Davidson wrote a great piece on her security blog today, which basically talked about focusing on the important vulnerabilities, not necessarily the ones that get the most press. Added to that, the risk associated with a vulnerability may well be different for you compared to everyone else, depending on how your system is used. I agree with what she is saying, but I’m going to take a slightly different angle on the subject.

Over the years I’ve come across lots of different attitudes to database patching from management and DBAs. As more DBAs are now involved in looking after middle tier products like WebLogic, some of those attitudes to patching have moved into that world too. It seems to break down into three camps.

  1. We don’t need no stinkin’ patches. If this is the way you roll you are a dumb-ass!
  2. We only patch stuff that represents a vulnerability for us. This sounds kind-of sensible, but life can get very difficult trying to decide what constitutes a threat, especially when you have to consider all layers of the technology stack.
  3. We always apply all patches. This is logically simple, but you are going to apply a lot more patches, a lot more frequently, which is going to require a lot more testing.

Patching is not just about security.

  • Support of some products is tied into the patch version. We see this with Oracle products all the time. There are some important deadlines coming up soon! 🙂
  • The rest of the world is moving on around you. You might be happy with your unpatched product, but things might break because of external factors. If someone turns off SSLv3 on their application server, you aren’t doing HTTPS callouts from your database to it unless you patch up to 11.2.0.3 or later. Your applications will probably get browser compatibility issues on newer browsers and mobile devices unless you keep on top of patching your development frameworks.
  • Patches like the database PSUs come with extra functionality, including backports of features from newer releases (redaction – 11.2.0.4). They can also bring with them features that make future upgrades easier (transport database – 11.2.0.3 onward).

Choosing not to patch is not really an option these days. Your company has to understand this and allocate the correct amount of resource to getting it done. That might mean more staff resources allocated to patching and subsequent testing (rather than doing “productive” work), outsource the work where you can or moving to cloud services where regular patching is part of the deal.

Cheers

Tim…

Oracle Database 12c Release 1 (12.1.0.2) Patch Released

I just read this post by Dirk Nachbar, saying that 12.1.0.2 is now available from edelivery.oracle.com. I’ve downloaded it, so the rest of the world is now allowed to start their downloads. 🙂

I assume it will be available from MOS also at some point.

Cheers

Tim…

PS. It will allegedly be made available on OTN at some point in the future.