Start with the simple things and work up…

 

We had a problem with one of our application servers on Friday. There was a long trail of breadcrumbs, which I followed all the way back to a blindingly obvious problem. We had run out of disk space. Doh!

Why did this happen?  There were two contributing factors:

  1. Unknown to me, marketing had requested we stop cleaning up the Apache access logs about two years ago, so they had been commented out of the automated cleanup script. Over that period, they had built up to over 7.5G.
  2. The tool for monitoring disk space usage was configured to send warning messages out via the wrong mail server.

Well, you have to chalk it down to experience, but it has served to remind me once again that often it is the most simple and obvious things that cause problems. Sometimes, the more you know, the easier it is to lose sight of the obvious.

Cheers

Tim…

Author: Tim...

DBA, Developer, Author, Trainer.

6 thoughts on “Start with the simple things and work up…”

  1. lol…really funny…once it happened with us also. Some procedure was there and it used to write some files using UTL_FILE. Suddenly there came an issue that it was creating empty files and not writing anything to them. Re-ran it n times, deleteing the files creating new directory and all but in vain 🙁 finally just by accident somebody hit “df -k” (we were on Solaris) and the space had gone upto 100% 😀

    Sidhu

  2. > it is the most simple and obvious things that cause problems.

    Are you trying to say I’m simple Tim…? 😉

    I take it you backed up all of the useful marketing info and forwarded it on to them…!!

    Cheers,

    Rob

    P.S. The new theme doesn’t always display properly…

  3. Rob. Course I did. 🙂

    Regarding the theme, it doesn’t work well on small screens. I’m waiting for a fix, or to find a new theme I like.

    Cheers

    Tim…

  4. Hi Tim,

    I have to ask… why would marketing be in control of apache logs? – if they want to keep them for analysis the fact that they are held on the servers still hints at marketing having server access. Or is it more likely that they are held and never used? – sounds like a security issue as well to me.

    cheers

    Pete

  5. Pete: They don’t have access to the servers.In fact, I don’t think they ever requested a copy of the data. I think they just wanted to keep the information “in case”!

    Unfortunately for them, I took the unilateral decision to delete them and reinstate that bit of the cleanup script. 🙂

    Cheers

    Tim…

  6. “Blindingly obvious” only when you notice it. We had the same issue with the Microsoft IIS logs filling up the disk space on the web server. As a result, our Intranet went down for quite sometime before we discovered the problem. It’s one of those things that once you know it, you’ll definitely add it to you list of “things to check” when things go wrong.

Comments are closed.