DG PDB : Oracle Data Guard for Pluggable Databases in 21c, and why you shouldn’t use it!

Last month you may have noticed the announcement of DG PDB. It’s Data Guard for PDBs, rather than CDBs, introduced in the Oracle 21.7 release update.

How do you use it?

I’ve had a play around with it, which resulted in this article.

I also did a Vagrant build, which includes the build of the servers, the database software installations, database creations and the perquisites, so you can jump straight to the DG PDB configuration section in the article. You can find that build here.

So that’s the basic how-to covered, and I really do mean “basic”. There is a lot more people might want to do with it, but it’s beyond the scope of my little Vagrant build.

What do I think about it?

Well I guess you know how this is going to go, based on the title of this post. I don’t like it (yet), but I’m going to try and be a bit more constructive than that.

  • It is buggy! : I know 21c is an innovation release, but this is a HA/DR solution, so it needs to be bullet proof and it’s not. There are a number of issues when you come to use it, which will most likely be fixed in a future release update, or database version, but for now this is a production release and I don’t feel like it is safe pair of hands for real PDBs. That is a *very* bad look for a product of this type.
  • Is it Data Guard? Really? : Once again, I know this is the first release of this functionality, but there are so many restrictions associated with it that I wonder if it is even deserving of the Data Guard name. I feel like it should have been a little further along the development cycle before it got associated with the name Data Guard. The first time someone has a problem with DG PDB, and they definitely will, they are going to say some choice words about Data Guard. I know this because I was throwing around some expletives when I was having issues with it. That’s not a feeling you want to associated with one of your HA/DR products…
  • Is this even scriptable? : The “add pluggable database” step in the DGMGRL utility prompts for a password. Maybe I’ve missed something, but I didn’t see a way to supply this silently. If it needs human interaction it is not finished. If someone can explain to me what I’ve missed, that would be good. If I’m correct and this can’t be done silently, it needs some new arguments. It doesn’t help that it consistently fails the first time you call it, but works the second time. Ouch!
  • Is the standby PDB created or not? : When you run the “add pluggable database” command (and it eventually works) it creates the standby PDB, but there are no datafiles associated with it. You have to copy those across yourself. The default action should be to copy the files across. Oracle could do it quite easily with the DBMS_FILE_TRANSFER package, or some variant of a hot clone. There should still be an option to not do the datafile copy, as some people might want to move the files manually, and that is fine, but to not have a way to include the file copy seems a bit crappy.
  • Ease of use : Oracle 21c introduced the PREPARE FOR DATA GUARD command, which automates a whole bunch of prerequisites for Data Guard setup, which is a really nice touch. Of course DG PDB has many of the same prerequisites, so you can use PREPARE FOR DATA GUARD to get yourself in a good place to start, but I still feel like there are too many moving parts to get going. I really want it to be a single command that takes me from zero to hero. I could say this about many other Oracle features too, but that’s the subject of another blog post.
  • Overall : A few times I got myself into such a mess the only thing I could do was rebuild the whole environment. That’s not a good look for a HA/DR product!

Conclusion

I’m sorry if I’ve pissed off any of the folks that worked on this feature. It wasn’t my intention. I just don’t think this is ready to be included in a production release yet. I’m hoping I can sing its praises of a future release of this functionality!

Cheers

Tim…

PS. I’m reminded of this post about The Definition of Done.

The Efficiency Paradox : Same Term, Different Meanings?

I’ve recently come across the term “Efficiency Paradox” being used by different people, in different contexts, and giving it different meanings. I thought I would share them…

The Efficiency Paradox in Economics

In 1865 William Stanley Jevons postulated, the more efficient a process gets in terms of resource usage, the higher demand you will see for that resource. This seems counter intuitive, as you might think the more efficient a process is, the less resources it requires, and therefore total resource usage would go down. Instead as a process becomes more efficient, costs drop and that drives demand, which eventually can result in more of the resource being needed. This is the heart of the Jevons Paradox, which is also referred to as the Efficiency Paradox by some sources.

Cost is always an important factor. We are currently going through a cost of living crisis in the UK. One of the factors affecting this is the cost of power. People are looking at ways to save money by reducing their power usage. When power was cheaper many people didn’t pay any attention to saving power. Now it is expensive, every little bit matters.

The Efficiency Paradox in Gaming

I watched a video by Josh Strife Hays, where he discussed the impact of guides and wikis on the enjoyment of playing video games. The term “grinding” refers to highly repetitive tasks that you must do to achieve a goal. Grinding can be exhausting, but when you achieve your goal there is a sense of satisfaction. Some games require a certain amount of detective work, where you try to figure out how to progress. Once again, the effort of trying to figure out how to progress can be exhausting, but the satisfaction on completing the task is high.

With the advent of the internet, there are loads of videos, wikis and websites dedicated to helping you play games in the most efficient manner possible. They might tell you how to minimise grinding, or flat out give you the answer to puzzles. These guides reduce the amount of time it takes to complete a task in a game, making you more efficient, but because you never have to deal with the adversity, you never get the same satisfaction when you complete a task.

So the efficiency paradox in gaming is, the more efficient you make the game play in an attempt to help the player, the less satisfying the game may become. Of course, if it is too difficult, they might leave before completing the task. There is a balance…

The Efficiency Paradox in Lean/DevOps

The previous versions of the efficiency paradox are interesting to me, but it’s this version that is really the subject of this post. In Lean and DevOps people often use the term efficiency paradox in subtly different ways, but invariably they are talking about resource efficiency vs. flow efficiency. Specifically, a focus on maximising resource efficiency resulting in less overall efficiency.

Lost Time : I’ve written about lost time before here. Lost time is about work waiting in queues while passing between siloed teams. Each team believe they are working efficiently because they have maximised their resource usage. All their staff are busy, but the flow of work through the chain of teams is really slow, making the flow efficiency low, and reducing the quality of work.

To counter this, some companies reorganise into self-sufficient teams that can progress a piece of work from conception to delivery, thereby reducing the hand-offs between teams. Some may retain the silos, but use automation to deliver self-service tools and APIs that others can pick up and run with. Regardless of the approach taken, they are attempting to reduce the constraints on the flow of work to improve flow efficiency.

Work in Process (WIP) : I’ve written about WIP before here. Most people can’t multitask well. Some think they can, but they just end up doing multiple things badly. Problem solving requires concentration, and it’s really hard to concentrate when you are being distracted by multiple projects competing for your attention. In an ideal world your WIP would be 1. You would work on a single task to completion, then move to another task. This can be tricky if you are constantly being blocked by other people and teams/silos, but it’s also complicated when a company wants to see staff being “busy” all the time.

In an effort to maximise resource (staff) usage, they increase the WIP, so there is always something for people to do. On the surface this increased resource usage looks like it is increasing efficiency, but often the work degenerates to the point where people are spinning plates, without actually achieving much. Also, the reduced attention on a specific task results in a lower quality of work. You should always try to keep WIP low, even if that means some people have idle time. If the idle time is excessive, it probably means there is a problem somewhere else in the organisation that needs to be fixed. Deal with the root cause, not the symptom!

Ultimately we have to forget about the resource efficiency and focus on flow efficiency. We can often see this in our normal working lives. We have some processes we know are going to take weeks to complete. Then there is a “Priority 1” incident that means we need to complete something ASAP. The P1 instantly aligns every team giving them the same priorities, and we race through and complete the work in a few hours. Once the P1 is over, every person goes back to their silo, with their differing priorities, and the process returns to taking weeks to complete again. We have proved it can be done in hours, but because of politics and the internal company organization, fast never becomes the norm.

Conclusion

I thought it was interesting that the term efficiency paradox came up in three different contexts in the space of a few days, so I thought I would write about it. The important point is that in all three cases people are often making incorrect assumptions about efficiency. People are doing things that they think will improving efficiency, but it is not having the desired result.

Cheers

Tim…

Why Automation Matters : It doesn’t get much simpler than this!

Just a little story about something that happened recently…

The July Oracle quarterly security patches dropped, along with the downstream OpenJDK releases a little later. As soon as the OpenJDK release dropped I updated all our non-production Oracle REST Data Services (ORDS) middle tier to the latest and greatest. Production will follow shortly. I figured that was the end of it…

On Friday evening I noticed a post by Jeff Smith saying version 22.2.1 of SQLcl and ORDS had dropped. Now it was Friday evening, and I typically don’t work on Fridays, but I really didn’t want to have this hanging over me all weekend, so this is what I did…

  • Download the software.
  • Upload the software to Artifactory.
  • Change a couple of files in Git, specifying the new software versions.
  • Press a button in TeamCity to rebuild the ORDS Docker image and push the new image to JFrog Platform.
  • Press a button in TeamCity to redeploy all the non-production ORDS containers.

The whole process took a few minutes, and I was confident it would work because it’s all automated and works every time. I sent this message to Jeff.

I {expletive} love automation. I just noticed your post about 22.2.1 of SQLcl and ORDS. I logged on to work. Updated a couple of files in git and pressed a button in TeamCity. New ORDS Docker image built and pushed to all non-prod systems. Boom…

This is why automation matters. The combination of picking the right platform, in this case containers, and investing a little time in the automation means we can react really quickly to new releases. It doesn’t get much simpler than this!

Check out the rest of the series here.

Cheers

Tim…

PS. This post explains the application topology we use for ORDS and APEX.

Oracle Support : Please give feedback on your Service Requests

I mentioned in yesterday’s post I was having a call today with the Senior Director of Global Support Services from Oracle. I’ve just come off that call…

I’m not going to go into lots of detail, because ultimately none of it matters unless I see things improve. I was honest in the call. My previous rants were over 2.5 years ago, and it’s hasn’t improved, so why would I assume it will this time?

One big factor that I think is worth mentioning is about feedback on SRs. I was told Oracle Support are driven by customer satisfaction reports, based on the feedback surveys they ask you to fill in when you close a SR. We discussed my current employer, and we have completed almost no feedback surveys, so support can’t tell if we as a company are happy with support or not. Most of our SRs relate to Oracle Cloud Apps, not the core tech stack, so I’ve messaged some folks involved in that side of things asking them to encourage people to fill in the feedback surveys.

I’m lucky in that I have direct access to many people in Oracle, so I don’t have to raise many technical SRs, but from now on I will make it a point to fill in the feedback on every SR. I suspect my rating for the SR that sparked these posts will be something like this.

Stars: The minimum I can give.
Comments: There was no suitable problem type for my SR, so I got forced into an automation that sent loads of irrelevant requests for information. The automation is “too chatty”, sending many messages when a single one would suffice. I’m giving this a low rating mostly because the process was very painful!

I would advise you to give feedback on every call. Be positive when it is positive, but be critical when it sucks.

Update: I got the survey and filled it in as I said I would above. Interestingly, the survey messages are picked up as spam by our company spam filter, and are converted to plain text, so the links don’t work. You have to manually paste them into a browser. I suspect this is one of the reasons why Oracle are getting very little feedback from our company…

Cheers

Tim…

PS. The guy on the call was nice. This is not about individuals. It’s about the service!

What a “simple” service request looks like on My Oracle Support (MOS)

I had a strop on Twitter again about My Oracle Support, and I just want to document why.

Someone on Twitter pointed out to me that the “active” real-time SQL monitoring reports weren’t working. It’s really easy to demonstrate, so I ran through it, and sure enough it was broken. When you opened the report in a browser, some of the dependent files (Javascript and a Flash movie) were missing, giving a 404 error.

I opened a service request (SR) on My Oracle Support (MOS). The first hurdle was the “Problem Type”. It’s mandatory, and the list of options is crap. Keep in mind that what you pick launches you into a load of automations, so picking the wrong thing is painful, but you typically have to because there is no “none of the above” option to pick… I decided on a performance related option, since the issue was a to do with the real-time SQL monitoring feature, even though I know this was not about the performance of my database…

I gave a link to a working example, and uploaded a generated report and the output from the Chrome developer console, showing all the 404 errors. It was a pretty self contained thing, so I was sure it would be understood as soon as a human saw it. This is what happened next…

An automated response.

Another automated response.

Another automated response.

Another automated response.

By now you can imagine I’m getting a little annoyed, so this is my response.

This was followed by a repost of my original post. Word for word. Seemed rather strange, but ok…

This was followed by three automated messages saying the same thing.

I was about to go supernova at this point, but I managed not to lose my shit.

Finally we have someone claiming to be a human!

I get asked to look at something that wasn’t directly related to the point of my SR.

I can feel the rage building.

The following day the person actually runs my report, which was uploaded in my initial SR opener, and it works for them. They update the SR. Once I see the update I try and sure enough it works for me now too. At some point between me opening the SR and this last interaction the “missing” files on http://download.oracle.com/otn_software/ have become available again. I tell them to close the SR.

Thoughts

Remember, this is about as simple an SR as you can get. I posted a test case to demonstrate it. It was really self-contained and simple. I don’t want to think what would happen with a “difficult” issue.

The “Problem Type” selection during SR creation is a problem. I understand MOS want to use it to help automate the gathering of information, but it needs a “none of the above” option, or you are *forced* to pick something you know is wrong.

The automated messages are terrible. I ranted about this before, which resulted in this post.

MOS Auto Responses : What’s my problem with them?

After that post I got a message from Oracle wanting to talk about it. That resulted in this post.

My Oracle Support (MOS) : Where do we go from here?

The messages that came out of that meeting were really positive, but it’s over 2.5 years later and if anything it’s got worse…

Someone from Oracle Support has reached out to me, so I have a call with them tomorrow. I’m not sure this call will result in any positive change. It feels a bit like groundhog day…

Overall, My Oracle Support (MOS) is a terrible user experience, and Oracle should not be charging this much money for this terrible service.

Cheers

Tim…

PS. It’s 2022. Why can’t people call me “Tim” rather than “Timothy”?

PPS. Please Oracle, stop breaking URLs…

PPPS. Please read the follow-up post here.

DBA and PL/SQL Development Tools (Poll Results Discussed)

I’ve been thinking about my DBA and PL/SQL tool choices recently, so I thought I would go out to Twitter and ask the masses what they are using.

As always, the sample size is small and my followers have an Oracle bias, so you can decide how representative you think these number are…

Here was the first question.

What tool do you do *most* of your Oracle DBA work with?

I expected SQL*Plus and SQLcl to be the winner here, and I was right. A lot of DBAs are still “old school” where administration is concerned. It may be tough for a beginner to use these command line tools, but over time you build up a list of scripts that mean it is much quicker than using GUI tools for most jobs.

SQL Developer had a pretty good showing at nearly 28%. I’m glad people are finding value in the DBA side of SQL Developer. TOAD/other were not doing so well. I know there are a lot of companies out there trying to make money with DBA tools, but maybe this is a tough market for them. Of course there are cross platform tools that may do well with other engines, even though they don’t register so well with the Oracle crowd.

I guess the real surprise was less than 8% using EM Cloud Control. Having said that, I’m considering ditching it myself. I like the performance pages and we use it as a centralized scheduler for backups, but I’m not sure our usage justifies the crazy bloat that is Cloud Control. It would be nice to remove all those agents and clean up! This figure of less than 8% is all the more surprising when you consider it is free (no cost option). Of course total cost of ownership is not just about the price tag…

This was the next question.

What tool do you do most of your PL/SQL development with?

I was expecting SQL Developer to do well here, but I was surprised by how low TOAD was in the list. I’ve worked at a few companies over the years where TOAD was a staple. I guess the consistent improvements to SQL Developer and a price tag of “free” have broken the TOAD strangle hold.

There were a few comments about Allround Aautomations PL/SQL Developer, which I used in one company many years ago. If I could have added an extra line in the poll, I would have put that as an option, because I know it is still popular. There were also mentions of DataGrip and a number of people using VS code with assorted extensions, including Oracle Developer Tools for VS Code.

Sadly, but understandably, SQL*Plus and SQLcl were low on this list. I’m an old timer, so I’ve had jobs where this was the only option. At one job I wrote my own editor in Visual Basic, then rewrote it in Java. Once SQL Developer (known as Raptor at the time) was released I stopped working on my editor…

When you’re doing “proper” PL/SQL development, it’s hard not to use an IDE. They just come with so much cool stuff to make you more productive. These days I tend to mostly write little utilities, or support other coders, so I find myself writing scripts in UltraEdit and compiling them in SQLcl. If I went back to hard core PL/SQL development, I would use an IDE though…

For fun I ended with this question.

SQL Developer and TOAD have a fight to the death. Who wins?

SQL Developer won, but it came out with a detached retina and some broken ribs!

Remember, you are most productive using the tools that suit your working style, but you should always keep your eyes open for better ways of working. Choice is a wonderful thing!

Cheers

Tim…

Packer, Vagrant and Docker : Latest Changes

The quarterly Oracle security patches trigger a whole bunch of build changes for me. This post just gives you a run through of what happened over the weekend.

Packer

The release of VirtualBox 6.1.36 means all my Vagrant boxes on Vagrant Cloud have the wrong guest additions, so I rebuilt all of them using Packer. The Oracle Linux 7, 8 and 9 boxes are now up to date. You can see my Packer builds under my Vagrant repository on GitHub here.

https://github.com/oraclebase/vagrant/tree/master/packer

Note. During the packer builds I noticed the VirtualBox 6.1.36 guest additions require some extra packages during the installation (libX11, libXt, libXext, libXmu).

Vagrant

I had recently updated all relevant Vagrant builds with the latest versions of Tomcat, SQLcl and ORDS updates, but I was still waiting on the OpenJDK 11.0.16 release. On Friday morning I noticed Adoptium released it, so I made the necessary changes to the builds to include it. I usually don’t include Oracle patches in my database builds, but some installations, like Oracle 19c on OL8, require them. I’ve updated them to include the 19.16 patches where necessary. You can find my Vagrant builds on GitHub here.

https://github.com/oraclebase/vagrant

Docker

Similar to the Vagrant section above, the relevant Docker/Podman builds were updated to use OpenJDK 11.0.16, and the Oracle 19c on OL8 build had it’s patch script modified for the 19.16 patches. You can find my container stuff on GitHub here.

https://github.com/oraclebase/dockerfiles

Happy building!

Cheers

Tim…

VirtualBox 6.1.36

VirtualBox 6.1.36 has been released.

The downloads and changelog are in the usual places.

I’ve installed it on Windows 11 and macOS Big Sur hosts with no dramas. Some time in the next 24 hours I’ll upload updated versions of my Oracle Linux 7, 8 and 9 Vagrant boxes.

Once that’s done I will be running through all my Vagrant builds using the latest Oracle patches.

Cheers

Tim…

Book : Data Science by John D Kelleher & Brendan Tierney

I was chatting with Brendan Tierney and the subject of books came up. He mentioned his Data Science book and I figured I would give it a go. I wasn’t sure I would be able to cope with it, as it is not my area of expertise, but he assured me it was more of an introduction to the subject, so one Audible credit later and I was off.

When judging books on Audible there are two criteria. There is the content, and then there is the voice narration.

Having just gone through the Dune series again, and recently doing loads of Larry Niven books, including the Ringworld series and the Fleet of Worlds series, I was a bit spoilt by the voice narration of fiction books.

I struggled at first with the voice narration on this book. It was clear, but the guys accent was grating on me. At one point I stopped listening, then realised I was being a dick and went back to it. I’ve now finished it, and I’ve actually started listening to it again. I’m sure I missed some stuff on my first pass through…

I’m not going to lie, there were some bits that definitely went over my head, but for the most part it was pretty easy to follow. It talks more about the general approaches to data science, rather than focusing on specific bits of technology. This was a good move by the authors because it means the book stays relevant, even as the “what’s cool today?” technologies keep changing…

The sections related to data protection and the ethical issues with data science were cool. I found myself talking over some of these points with myself while I was driving… 🙂

If you are new to Data Science, or just have a passing interest in it like me, I think this book is worth a shot.

Cheers

Tim…