Test Cases Are Important : Again…

Over the weekend I was reminded of the importance of test cases again. I’ve written about this before, with probably the most consistent post here.

If you want my opinions of test cases, go and read that. In this post I want to tell a little story to demonstrate why I think test cases are important. I’m going to keep things a bit vague, because I don’t want to openly criticise the person in question, because they actually did an OK job of expressing their issue, but it did highlight some things.

The issue

I got a question that suggested a recent upgrade on Autonomous Database had altered the behaviour of something. Every time you patch or upgrade software there is a possibility of change, whether it is an intentional behaviour change, or a bug. The person had provided some evidence that did seem to suggest there was an issue, so my interest was piqued. Unfortunately there wasn’t a test case, but I have an article that includes a test case that was similar, so I was able to knock something together pretty quickly.

The test case

The first thing I did was try out my test case in an on-prem installation. Yes, I know the potential issue related to autonomous database, but I wanted to see the test case working, and prove to myself that what I believed to be true actually was true. Think of this as an experimental control. The test case ran as expected on-prem, which was good.

I then moved to trying to replicate this issue on autonomous database. Most cloud databases come with some restrictions on what you can do, so my test case setup was not ideal for running on autonomous database. I had to revise the setup a little. APEX to the rescue. Before you ask, yes, I did rerun the on-prem test with the new setup to make sure the control was still valid. 🙂 Having set up the base data, I was able to run the code for my test case, and it ran just the same in autonomous database as it did on-prem.

Test case vs in situ

In the original question, the issue was directed specifically at one feature, but my test case seemed to prove that feature was working as expected. When you are doing scientific experiments you try to reduce the number of variables. Too many variables and you have no idea what caused the result, so you can’t come to any reasonable conclusion. I was trying to prove a feature works as expected, so I reduced the possible variables to the point where I was specifically testing that feature, and it seems to work as expected.

So that’s the end of it right? Well not really. I’ll use an example from biology to explain. Biology is complicated because living things are complicated. When you are doing chemistry, it’s possible to isolate specific compounds and put them together in a controlled manner to observe an interaction. Kind-of like my test case, this is a very controlled approach. Living things have loads of working parts, and you can’t isolate things without killing the organism, so you have to deal with the fact you are working in the middle of a whole bunch of interactions. You still try to minimise your variables, but you have to accept that you can’t always do that to the extent you would like. You may define experimental controls that discount the other possible reasons for the result. This distinction between running things in isolation and in situ is really important. What has this got to do with my test case?

My test case is run in isolation. The original poster clearly has an issue in their system. Perhaps there is something in their system that affects the way this feature works, so although the feature works in isolation, maybe there is an issue in situ. My test case hasn’t resolved the issue. It has just ticked one thing off the list of possible causes.

What next?

Having ticked the base functionality off the list of possible causes, we then have to move up one step higher and incorporate more elements of the system, to see how that affect things. That could be as simple as we are using different session parameters, or it could be something more fundamental with the design of their system. Hell, it might be their data is corrupt for all I know (I really hope not).

It’s also possible when looking at the “next layer”, we notice something that shows the original test case is invalid. That sort of things happens.

What’s the point of this post?

I often get the impression some people think problem solving is some kind of witchcraft. In reality it is painstaking meticulous work. I look at all the people I think are good and they have one thing in common. They put in the the work and grind through this stuff. Yes, you get quicker the more experienced you get, but you still have to put in the effort. People are often looking for the “magic button” to solve their problem, but there isn’t one. If it were that simple, it would already be built into every piece of software you use. 🙂

You need a test case, even if all it does is prove your initial conclusion was wrong, and allows you to focus your attention elsewhere.

Once again, the question that promoted this post was not bad. The person did an OK job of expressing themselves. This is just a post that was triggered by that interaction. If we get to the bottom of their issue, and it proves to be interesting, I will probably write up something more specific about it. 🙂

You might find it useful to read these, as they are relevant to this post.

Cheers

Tim…

Update: This looks like it is a data/understanding issue. It’s starting to sound like the data isn’t stored in the format the original poster expected, so they are trying to do something with it that is impossible. If this is the case, it’s nothing to do with the upgrade.

Learning New Things : We don’t need no freakin’ test case!

When people ask a question about IT the first response will often be a request to provide a simple test case to demonstrate the issue. Typically these requests go unheard by the original poster, because it is seen as too much effort and/or of no relevance to their “real world” problem. Here are some of my thoughts about test cases.

  • A simple test case makes it easy for the person answering your question to jump straight in and help. They should not have to spend their time creating scripts to setup a test when you could do it. Don’t waste other people’s time. It gives the impression you think your time is worth more than theirs, which is just rude!
  • A test case is not a DESC of a table and the output of a “SELECT * FROM …”. It’s going to take time for someone to turn that into a table and insert statements. That’s your job! Don’t waste other people’s time.
  • I know you believe you have asked a really succinct and precise question, but you haven’t. Your question sucks. Your question brings with it a bunch of assumptions I can’t possibly know about. Often, the first time I truly understand the question being asked is when I see the test case.
  • A test case should be simplified to the point where the issue is still reproducible, but all the extra guff has been removed. It’s better if it doesn’t use your real database objects (tables, views, packages etc.), since you can post it without breaking data protection laws or your company IP policies. Also, it’s likely most of the objects and data in your system are nothing to do with the issue, so don’t include them in your test case.
  • Make the test case re-runnable. If objects need to be created, include the statements to drop them in the correct order too. This is important for a couple of reasons. It allows you to do a complete re-run for each test, so you don’t introduce some form of cumulative impact. It makes life simpler for the person who is trying to help you. Don’t waste other people’s time.
  • A test case acts like a regression test for your understanding of a system. You can rerun them against different database versions and patches and see that what you believe to be true still is.
  • Quite often, the process of defining the test case allows you to answer your own question. The removal of the unnecessary stuff allows you to see the real problem. I’m sure you’ve had the experience of trying to explain the problem to someone, only to see the solution for yourself. The test case is that process, without needing the other person. 🙂
  • A test case allows you to demonstrate the issue and eventually the resolution, which makes it far easier to get permission to apply the fix in production.

I think you get the picture!

Check out the rest of this series here.

Cheers

Tim…