Stupid Stuff I’ve Done : The Clone Wars

stormtrooper-1343772_640Following on from the theme of yesterday’s confession about my screw-ups with pluggable databases comes one about cloning.

For some systems we use cloning (RMAN DUPLICATE) as a method of refreshing Dev and Test environments from Live. The process of doing a backup-based or active duplicate is not that complicated and typically you do it once, script it and you don’t have to worry too much about it again.

Yesterday we had to refresh a test environment from live, so we used our trusty clone script, but it failed with the following message.

RMAN-06054: media recovery requesting unknown archived log for thread 1 with sequence ???? and starting SCN of ????

Strange! The message is pretty clear. It is expecting an archived redo log that doesn’t exist. If this were a backup-based clone I would suspect one of the archived redo logs had not been copied across, but this was an active duplicate, so it automatically transports everything from the target to the auxiliary instance. How could it lose something?

I knew I had seen this before, but no matter how much I searched my website, private work notes, MOS or using Uncle Google I couldn’t find the solution. Everything pointed to a missing archived redo log, but that couldn’t be happening on an active duplicate right? Wrong!

After much denial, wailing and gnashing of teeth I remembered the archived redo log backups! During the clone operation an archived redo log backup had kicked in on the target database and deleted all the logs that had been backed up. By the time the active duplicate tried to grab them they were gone and hence the error…

Notes to self:

  • When you see something like this happen, write a note about it! Don’t think you will remember it next time. You won’t, or you will waste loads of time before you do remember it! How many times do you have to tell yourself, “If it’s not written down it doesn’t exist!”
  • The error message is trying to tell you the answer. Don’t try and intellectualise your way round it. It’s probably not a bug that is causing some spurious misreporting of the “real problem”. You probably don’t know better than the error trapping. Think less about why the error message might be wrong and think more about what could have caused the situation that resulted in the error, as it is reported.
  • Don’t write a blog post confessing how rubbish you are at your job or people will find out and… Doh!



Multitenant : It’s easy to screw up!

multitenantSome things happened to me yesterday which I thought I would pass on, in case it helps others. This is not necessarily a criticism of the Multitenant architecture itself, but more about my misuse of it. 🙂

I’ve complained numerous times about the inclusion of the shared APEX installation used with the Multitenant option. I’m sure it is great for some companies that provide APEX as a service, but it’s not the right option for me. I would typically recommend removing it before you create a PDB, as described here. So what did I do yesterday? I banged out a script as a variant of a non-CDB build and totally forgot about the shared installation. When everything was finished I patted myself on the back, then noticed my mistake. 🙁 I had to remove the PDBs from 3 instances, remove the shared APEX installations, the recreate the PDBs. Luckily it was early on in the process, so it wasn’t a great drama. It would have been significantly more annoying if I hadn’t noticed for a few days!

Next up I nearly cloned a PDB, which would have left me with 2 PDBs in the instance, and potentially a big bill for the Multitenant option. I was literally about to hit the return key when I realised what I was doing. 🙁 Needless to say I immediately added a trigger to all my instances to prevent any future stupidity on my part. You can see what I did here. As I say in the article, I’m not sure if there is some hidden long term tracking of the maximum number of PDBs in a CDB, but I’m a paranoid type… 🙂

As an aside, I also noticed that when I add a new PDB to a CDB it takes Cloud Control 13c a long time to notice the PDB is there. I did try to give it a nudge by doing the following, but it didn’t seem to notice.

  • Highlight the CDB in the database target list.
  • Click the “Configure” button.
  • Scroll down and click the “Sync Pluggable Databases” button.

It didn’t recognise the change, even though it was clear in the V$PDBS and CDB_PDBS views. Not sure if this is normal or if I just got unlucky. Everything looks OK now… 🙂