On Sunday 17th May I started the process of putting together a Vagrant build of Oracle 19c RAC on Oracle Linux 8 (OL8.2 + EUK). I figured it would take me about 20 minutes to amend my existing OL7 build, but it took the whole of that Sunday, every evening for the following week, and the whole of the following Saturday and Sunday to complete it. There were some late nights, so from an hours perspective it well over 5 days of work. Most of that time would have been completely unnecessary if I wasn’t an idiot.
First things first. The result of that effort was this build on GitHub, with an associated article on my website describing the build in more detail.
- https://github.com/oraclebase/vagrant/tree/master/rac/ol8_19
- Oracle Database 19c RAC On Oracle Linux 8 Using VirtualBox and Vagrant
For the remainder of this post I want to describe the comedy of errors that went into this creation. These problems were not indicative of issues with the software. These problems were totally down to me being an idiot.
Changes when using OL8
The vast majority of the build remains the same, but there was one change that was necessary when moving to from OL7 to OL8. This became evident pretty quickly. When configuring shared disks and UDEV, I switched from using partprobe to partx. It did look like partprobe was working, but it chucked out loads of errors, and partx didn’t. I found out about this from Uncle Google.
There was also a slight difference in getting the UUID of a regular VirtualBox disk in OL8, but I had already noticed that on single instance builds, so that wasn’t a problem.
Where it all started to go wrong
So with those minor changes in place, all the prerequisites built fine and I was ready to start the Grid Infrastructure (GI) installation. I try to do these builds using the stock releases, so people without Oracle Support contracts can try them. In this case that meant using the 19.3 software. This really marked the point where it all started to go wrong. I knew this build was only certified on 19.7, but I continued anyway…
None of the 19.3 installers recognised OL8, so I had to fake the Linux distribution using the following environment variable.
export CV_ASSUME_DISTID=OEL7.6
The 19.3 GI software refused to install, saying there was a problem with passwordless SSH connectivity. I tested it and it all looked good to me. I searched for solutions to this on Google and MOS, checking out everything I could find that seemed relevant. None of the solutions helped.
For the hell of it I tried on older releases of OL8. I had the same issue with OL8.1, but OL8.0 worked fine. I figured this was something to do OpenSSL or the SSH config, so I searched for more MOS notes and tried everything I could find. After a very long time I reached out to Simon Coter, who came back with an unpublished note (Doc ID 2555697.1), which included a workaround. That solved my problem for the 19.3 GI installation on OL8.2.
The GI configuration step and the DB software-only installation went fine, as they had done on OL8.0 also. Unfortunately the DBCA was failing to create a database, producing errors about passwordless SSH problems. The previous fix wasn’t helping, and I tried every variation I could find, short of totally downgrading OpenSSL.
At this point I pinged Markus Michalewicz a message, hoping he would be my salvation. In short he confirmed the build did work fine on OL8.2 if I used the 19.7 software, so the writing was on the wall. Then I stumbled on a MOS note (Doc ID 29529394.8) that explained the DBCA problem on OL8. There was no workaround, and it said it was fixed in 19.7. There is always a workaround, even if it means hacking the Linux distribution to death, but the more you do that the less realistic your build is, so at that point I conceded that it was not sensible to continue with the 19.3.
When I write it down like this it doesn’t seem like a lot, but all this took a long time. I was trying different Vagrant boxes, and eventually built some of my own for OL8.0 and OL8.2, to make sure I knew exactly what was on them. Added to that, many of the tests required full rebuilds, so I was waiting sometimes in excess of an hour to to see a success/fail message on the next test. It was soul destroying, and I nearly gave up a few times during the week.
A New Hope
Once I decided to go with the 19.7 patch things moved pretty quickly. I had the 19.3 GI installed and configured and the 19.3 DB software installed, so I added in a script to patch the lot to 19.7, and the database creation worked fine. Job done.
I cleaned things up a bit, pushed it to GitHub and put together the longer description of the build in the form of an article (see links above).
Pretty soon after I put all this live I got a comment from Abdellatif AG asking why I didn’t just use the “-applyRU” parameter in the Grid and DB software installations to apply the patches as part of the installation. At this point I felt a mix of emotions. I was kind of frustrated with myself for wasting so much time trying to get 19.3 working in the first place, then annoyed at myself for being so blinkered by the existing build I hadn’t seen the obvious solution regarding the patching. Why build it all then patch it, when you can do it right first time?
The conversion of the build to use the “-applyRU” parameter with the runInstaller and gridSetup.sh calls was really quick, but the testing took a long time, because these builds take in excess of 90 minutes each try. Things pretty much worked first time.
Now that I was effectively using 19.7 software out of the gate I figured many of the tweaks I had put in place when using the 19.3 software were no longer needed. I started to remove these tweaks and everything was good. By the end of that process I was pretty much left with the OL7-type build I started with. The vast majority of my time over the last week has been unnecessary…
Conclusion
The 19c (19.7) RAC build on OL8.2+UEK6 is pretty straight forward and works without any drama.
This process has shown me how stubborn and blinkered I can be at times. Taking a step back and getting a fresh perspective would have saved me a lot of time in the long run.
Thanks to Simon Coter, Markus Michalewicz and Abdellatif AG who all witnessed my descent into madness.
Cheers
Tim…
PS. There are some extra notes at the end of the article, which include some of the MOS notes I tried along the way. They are unnecessary for the build, but I felt like I should record them.
PPS. The image is how I feel at the end of this process.
Thank you for all the work Tim!
People who never stepped into the unknown and/or the scarcely documented don’t know and don’t understand how small things that don’t work, but often too small things that still work but annoy you can take up a lot of time, especially since when you finally find out how to work around it, it’s seems like an insignificant detail.