The calm after the storm…

Things are just starting to return to normal… I hope 🙂

The disruption associated with the production hardware reconfiguration seems to have calmed down now. Of course, not all of the problems were associated directly with the hardware changes, but the time you spend on one problem distracts you from other jobs you should be doing, which in turn creates new issues.

We have finally moved all the database backups over to RMAN against HP Data Protector, rather than the disk-based RMAN backups we were doing before. The software seems to work fine, but the physical processes are taking some time to get used to. Rather than having a tape labeled “Monday night application X backup”, we now have a bank of tapes in a media pool that can hold portions of several backups. It’s not rocket science, but it can be a bit of a culture shock when you start changing processes that have been in place for years. It takes a while for people to get used to the idea that you don’t know which tapes must be removed and taken offsite until the software tells you.

Fun, fun, fun 🙂



SQLdetective Review

I was recently asked to review a product called SQLdetective, so here it is.

Q. What’s it for?
A. It’s for developers and DBAs and anyone else who does anything with an Oracle database.

Q. What features does it have?
A. It has everything you’ve come to expect from a general purpose DBA/Developer IDE. I’m not even going to attempt to list them because it would take forever. Suffice to say, if you’ve seen it in another product, it’s probably here.

Q. What’s it look like?
A. You can look at a bunch of screen shots here. Like most current IDEs it’s a bit busy. Not as plain as PL/SQL Developer, but not as busy as TOAD. The icons are “different”, but I’m not sure if that’s a good or bad thing.

Q. What’s it feel like?
A. I seems quite quick and responsive, but it does some odd things, like when you highlight a table in the navigator the default operation is to show you a page of data. I doubt this would be the first thing I would want to see.

Q. What’s it cost?
A. The basic price is US$99, but if you want all the add-on modules it’s going to cost a total of US$198. So the cost depends on what you need. It’s difficult to compare prices because you rarely have to pay list prices and each IDE has different optional modules. It’s more expensive than the free version of TOAD, and now JDeveloper is free you can do all your PL/SQL development and debugging using that if you’ve got the hardware to run it 🙂

Q. Tim, would you buy it?
A. If I was forced to use an IDE I would use TOAD or JDeveloper for free. I have both loaded on my PC and I never use them. If I was forced to buy a license for an IDE I would consider this product. My company recently bought licenses for PL/SQL Developer, but I still use a text editor and SQL*Plus 🙂


If you really want to pay money for an IDE give this one a try. It’s got a lot of stuff in it. Alternatively, write good intrumented code and save yourself the cash.

I guess I could just copy and paste this review about almost any of the current IDEs, but at least I’ve kept my promise…



Firefox 1.0.6 and RAC update.

Looks like Firefox 1.0.6 is out, but the update checker doesn’t know it… Again.

Got through a night without a server crash on the production RAC. Feels like the old days again 🙂



PS. Posted two more chapters to the publisher today. That leaves me with one new one to write and one to tighten up then I’m finished. Must crack on and get it sorted before I have a nervous break down cause by a complete lack of sleep 🙂

2 become 0, occasionally…

It’s been an “interesting” couple of days.

Since we moved to the new configuration we’ve have had some intermittent server crashes, including one last night where both nodes crashed simultaneously. We’ve applied an OS patch that is meant to solve the problem, but we’re still in the “fingers crossed” stage at the moment. It’s meant some busy nights and annoyed users, but we’re moving forward.

You’ve gotta love it…. Not!



3 become 2, RAC nodes that is…

This weekend our production system was switched from a 3-node to a 2-node RAC.

We were originally using a 2-node RAC (2 CPUs per node) and we added a third node because the system was struggling to cope with the workload. The third node helped us out in some ways, but it caused a lot of trouble in others. Ever since it’s inclusion it became impossible to take one node out of the RAC without bringing the lot crashing down, so much for high availability. In addition, a substantial proportion (about 30%) of the wait states on the system were due to inter-node communication. Now I expected with more nodes there would be more inter-node communication, but it seems a bit excessive. Heaven only knows what would happen in a 4-node cluster…

After a lot of banter with Oracle and HP we’ve finally decided to try a 2-node RAC again, but this time with 3 CPUs per node. OK, it’s actually 4 CPUs per node, but one CPU in each node is permanently offlined, so as not to affect our current Oracle licensing.

All the hardware modifications are complete and all tests indicate that the system is up and running normally. Of course the true test will happen tomorrow morning when the users log in and start to break things 🙂

The best news of all is that the move back to a 2-node cluster means that we can once again shut down one node at a time if we need to do maintenence. This is a big plus.

If everything goes quiet over the next few days it means that I’m fire-fighting and the switchover didn’t go well.

I’d be curious to see how many people out there are using RAC on more than 2 nodes. I’ve only done this on Tru64 with 1og Release 1, but I can say without a shadow of a doubt that it doesn’t work properly. I’m curious if this is Tru64 specific problem or if there is a fundamental flaw in RAC for clusters with more than 2 nodes.



The ever changing Oracle pricing model…

You’ve got to take your hat off to Oracle. Just when you thought they couldn’t make their pricing model any more ridiculous they come up with this gem. I guess we should be grateful that they’ve conceded this much. Prior to this announcement a single dual core processor was charged the same as two individual processors, even though it didn’t have the same performance.

Of course nobody pays list price for Oracle, but it is used to calculate the support and updates costs, so it’s in their interest to keep it artificially high.

I always find the TPC pricings a laugh. Oracle sounds real cheap until you realize they’ve only included a 3 year license. I’ve worked with Oracle products for over 10 years and I’ve yet to work for a company that has bought Oracle this way. They have all bought perpetual licenses for some serious wonga! Maybe we’re living in the dark ages in the UK 🙂

We’ve got some 3rd party applications running on mySQL and they work really well. Nice and cheap too. Well actually it’s free. We’ve got a few 3rd party applications running against SQL Server and they do the job nicely too. We’re looking to switch another project from Oracle to SQL Server part way through the implementation after some confusion over the Oracle licensing costs. It doesn’t take many incidents like this within a company before some momentum builds up and people start opting for cheaper alternatives.

Whenever anyone mentions licensing costs to me I hide under the desk. I’m a grunt, not an accountant!



Life is funny!

My osteopath just told me two funny stories today.

He treated a guy this week who had been drafted in as part of a team to organize a large firework display in Birmingham. The fireworks were all stored safely, but the guys took their homemade detonators back to the hotel with them. A cleaner walked in, saw all the gear, flipped and called the police. Next thing you know the police storm in arrest the guys and evacuate the center of Birmingham. The papers claim local businesses (mostly pubs) lost more than 1 million pounds of business as a result of the evacutation. If this sounds familiar see Welcome to my world!

You can imagine the scene down at the police station.

Firework guy: We’re not linked to Al Qaeda! We’re just preparing for Fireworks Fantasia!
Police: Now listen here sonny Jim, we weren’t born yesterday…

I know it’s not a funny subject, but it made me laugh.

One of his patients took their 20 year old Down’s Syndrome son to a local zoo & theme park recently. Whilst there he repeatedly asked to go on the log flume ride, which they refused because they didn’t want him to get wet. At one point he went off on his own and when he returned he was soaked through. Naturally they assumed he had gone on the log flume ride, which he denied. When they got home the boy went to his room leaving his mother to unpack the bags. When she opened her son’s rucksack she found a penguin staring at her!

She rang the zoo, who said, “We’ll check if it’s one of ours!”, to which she replied, “Where else would he get a penguin from in Birmingham?”. Anyway, they fed the penguin pilchards until a man from the zoo came to pick it up.

I can only assume that the guy has seriously quick reflexes or it was one tame penguin!

I cannot guarantee that either of these stories are true, but they had me in stitches between blood-curdling crunches of my spine!



PS. My back feels good now! I reckon I’ll be OK to go to Karate tomorrow 🙂

Oracle Support Sucks…. Again.

Once again Oracle provides a less than perfect service on the support front. Let’s take a look at my latest encounter.

20-JUN-2005 – I raised an iTAR because the CC and BCC lists of the UTL_MAIL.SEND procedure don’t work. Emails are sent properly for people listed in the RECIPIENTS list, but mail to CC and BCC lists never get sent. I also sent this example code:

UTL_MAIL.send(sender     => '',
recipients => '',
cc         => '',
bcc        => '',
subject    => 'UTL_MAIL Test',
message    => 'If you get this message it worked!');

Assuming these were real email addresses person1 would receive a mail, while person2 and person3 would not. I was able to repeat this issue on Tru64 and Windows.

Within a couple of hours support requested an OWC (Oracle Web Conferencing) session to investigate further. Unfortunately I never received the email of the iTAR update so I didn’t reply.

13-JUL-2005 – The iTAR gets updated asking me for an OWC session again. This time I get an email so I respond saying I don’t think an OWC session is necessary as there is nothing to show. The sample code says it all. In this case the OWC session seemed like complete waste of both our time.

14-JUL-2005 – The iTAR is updated requesting an OWC session again. I say OK and connect. During the 10 minute conference (accompanied by a phone call) my only input was to show the sample code, which was already in the iTAR. First I’m told the issue can’t be progressed as I’m not on the latest patch, to which I reply that is the latest patch for Tru64. After that the support guy searches and finds a generic bug listed as being fixed in If this bug had been public I would have found it and not raised the iTAR in the first place.

The iTAR is now closed.

Now I realize that the majority of the time wasted here is down to me waiting for an email that never came, rather than checking the iTAR status directly. Obviously, if this had been an important issue for me I would not have let it drag on so long, but the whole process took nearly 4 weeks to inform me that my problem was an existing bug. I think that’s pretty shocking, especially since the bug was found using the information from the original iTAR, not the subsequent OWC session.

Conclusion – I assume the support people work to quotas. By replying to ask a followup question or request an OWC session they can tick a box to say they’ve responded. I’m sure the statistics relating to response times at Oracle support make very impressive reading, but I believe the truth is very different.

I don’t have a problem with the support people themselves. Some are great and some are not. I just think the support process sucks! We pay a ridiculous amount of money for what I can only describe as a crappy service. These days I raise iTARs in an attempt to improve the product/documentation, not because I expect to get an answer. I’m more likely to do that by visiting a free forum or searching on Google.

I suppose I should be grateful. At least you get an answer to DB support requests. That’s more than can be said for AS10g support requests! We close those out of boredom 🙂



New Article – Partitioning an Existing Table

In a recent forum thread someone asked me to outline a method for Partitioning an Existing Table using the DBMS_REDEFINITION package. I figured this might be useful to other people so I wrote it up as an article.

I always approach partitioning with caution. Both the article and the forum thread warn against partitioning for the sake of it.



Oracle 10g Application Server, what’s the deal?

I’m begining to dislike Oracle 10g Application Server. That’s my polite and understated way of saying I loath, detest and hate it!

Before I move on I want to make it clear I’m a major fan of Oracle databases. I think Oracle consistently hit the nail on the head with respect to new database releases. Yes, they have a habit of adding chaff and bloat, but the core functionality is on the money every time.

I’ve been using Oracle’s application servers for a little over 2 years. My first experience was with 9iAS and if I ever see an installation of that again I will probably go on a killing spree. It’s like Oracle took a bunch of cool software, cobbled it together and made it totally unusable. If people ask me what 9iAS is like my immediate response is, “It’s an abortion!”.

When AS10g was released we moved to it right away. We had no choice, 9iAS didn’t work. For some months I basked in the glow of it’s brilliance, but little did I know the horrors that were waiting round the corner. Rather than list whats wrong with AS10g let’s look at it from another angle, let’s list what we want from an application server:

  • Reliability.
  • Speedy deployment of new applications.
  • Easy configuration.
  • High availability.
  • Simple problem diagnostics.
  • Simple performance monitoring.

The problem is AS10g gives me none of these. Let’s take these points one by one.

Reliability – We have logged untold amounts of bugs against AS10g, most of which have never been fixed to our satisfaction.

Speedy deployment of new applications – Our applications are pretty small and not exactly rocket science, but deployments to our 5 node application server cluster can take hours. You think I’m joking don’t you. I’m not! It’s not unheard of for us to lose our entire production system for a couple of hours during a deployment. Invariable a couple of nodes don’t deploy properly. By the time we’ve undeployed and redeployed the application, along with a few reboots, the user have packed up and gone home.

Easy configuration – Ok, it’s not the worst thing in the world, but there are so many products and layers to deal with that it becomes a nightmare if you want to do anything but the simplest application. I’ve just checked with one of my production app servers and it has 296 distinct log files. When someone asks me, “Are there any errors in the logs?”, it always brings a smile to my face.

High availability – I’ve already told you what happens when we deploy new applications! We have a 5 node cluster to make our application more resilient and maintain availability. Pitty we have to reboot before and after every application deployment. Until recently we were rebooting each app server once a day, but we’ve managed to get that down to once a week, provided we’re not deploying new versions of the application.

Simple problem diagnostics – Too many log files. Too many layers. We were hoping that grid control would come to our rescue, but it doesn’t work properly. I don’t even want to go there. You can read my earlier posts about that crap.

Simple performance monitoring – See previous answer. We’ve ended up writing some of our own tools. Sad I know!

I’m starting to depress myself so I’m going to knock this post on the head soon, but suffice to say, if I had my way we would ditch the lot and use Apache and PHP. No overcomplicated application servers and no J2EE. Simple, reliable and free!

I guess I can dream…



PS. For those of you that are assuming we’re just using it wrong, the consultants we’ve had in from HP and Oracle can’t make it work any better, so I guess we’re in good company 🙂