Category: Technology

This is a follow on from my server problems post from yesterday…

Regarding the general issue, misiaq came up with a great suggestion, which was to use watchdog. It’s not going to “fix” anything, but if I get a reboot when the general issue happens, that would be much better than having the server sit idle for 5 hours until I  wake up. 🙂 Let’s see how that works out…

Praveen asked if I use any tools like Webmin. The answer is yes and no. Just like my use of any tool (Cloud Control, SQL Developer etc.) I use a combination of command line and tools. I usually find command line more useful as I can script and reuse, but I always have tools available to fill in the gaps and provide inspiration. I don’t always invest enough time in learning the tools well, which is why useful bits of them pass me by on occasion, but I also don’t like to become dependent on tools. In the case of Webmin, it is installed on the server, but it is not exposed to the outside world. I have to tunnel in to use it, so during a problem, when I can’t SSH to the server, Webmin is not available. 🙂

Back to the specific issue from yesterday…

During my normal checks I noticed my RAID1 setup looked like this.

# cat /proc/mdstat
Personalities : [raid1]
md3 : active raid1 sdb3[1]
      970470016 blocks [2/1] [_U]

md1 : active raid1 sdb1[1]
      4194240 blocks [2/1] [_U]

unused devices: <none>
#

Last time it looked like this, one of the hard drives had died, so I contacted the hosting company to get it sorted. After a couple of false starts, they eventually took the machine offline, tested it and said the hard drives were fine. 🙁

I added the partitions from the “/dev/sda” disk back into the RAID config. I guess I should have tried that first. 🙂

# mdadm /dev/md1 -a /dev/sda1
mdadm: added /dev/sda1
# mdadm /dev/md3 -a /dev/sda3
mdadm: added /dev/sda3
#

After it had finished rebuilding it looked like this.

# cat /proc/mdstat
Personalities : [raid1]
md3 : active raid1 sda3[0] sdb3[1]
 970470016 blocks [2/2] [UU]

md1 : active raid1 sda1[0] sdb1[1]
 4194240 blocks [2/2] [UU]

unused devices: <none>
#

So it looks like the drive just dropped out of the RAID config for no reason… Does that happen?

As I said before, I’m not a system administrator. I just know enough to be dangerous. 🙂 Thanks for the comments from yesterday. They have been very helpful…

Cheers

Tim…

I’m pretty sure last night’s problem was caused by a disk failure in the RAID array. The system is working now, but it might go down sometime today to get the disk replaced. Hopefully they won’t do what they did last time and wipe the bloody lot! 🙂

The subject of this post is not about this specific incident, but more generally about what I’ve experienced over recent weeks/months.

Background

As many of you will know I’ve been having some intermittent issues with my server for a while now.

It’s not predictable. Sometimes it doesn’t happen for ages. Sometimes it happens several times in one day. Most of the time I notice it pretty quickly, reboot it and all is fine for a while. Sometimes, like last night, it happens while I’m asleep and it is down for a few hours before I notice it.

The last time it was giving me a lot of trouble I planned to spend a weekend rebuilding the site on AWS on two nodes with a load balancer between them to give me a bit of extra availability, but by the time the weekend came it was behaving again and I decided maybe the extra cost was not a great idea. 🙂

Without knowing what the problem is, I’m worried I will go to the effort of moving everything, only to find I get the same problems in the new location, especially if it is something stupid I’ve done. 🙂

What do I see? Nothing!

If I don’t spot it before, I get a message from Uptime Robot saying the site is down. It only checks ever 15 minutes, so it’s usually some time earlier that the actual failure occurs. When it happens I usually check the following.

• The last entries in the webserver access and error logs. So far they never show me anything interesting, just that the webserver stops delivering pages when the issue happens.
• The output from sar shows nothing out of the ordinary from a load perspective (CPU, memory, disk, network) in the lead up to the failure. The server is massively overpowered for what I need, so everything looks pretty much idle most of the time.
• There is nothing relevant in the “/var/log/messages” file. The entries just stop, then start again when the server is rebooted. There is no pattern to the last message in the log and the failure. I don’t get a lot of messages logged here, so typically the last message is several hours before the failure.
• Nothing in the MySQL logs.
• Nothing in the cron log. No jobs firing near the time of failure, that could have caused a problem.
• Nothing in any of the other logs available in the “/var/log” directory, or subdirectories.
• Previously I have done memory tests, hard disk scans and checks on the RAID config, which don’t reveal any problems. Obviously, this time there is a RAID issue, but this is not typically the case when I have these issues.

Typically, when it happens I can’t do anything on the server. Not even SSH to it. I have to force a restart using the admin tool on the hosting company admin console. I once managed to get a serial connection to the server, just as it was happening, but I was not able to run any commands, so it didn’t help.

It’s a dedicated server running fully patched CentOS 6. It has MySQL 5.7 and PHP7, but the issues pre-date those. Previously it was running MySQL 5.6 and whatever PHP version came with the CentOS 6 yum repository. 🙂

Question

Anyone got any ideas what I can check next time it happens? I’m at a bit of a loss.

Like I said, I don’t want to up-sticks and move to a new server or VM on AWS if the problem is of my own making. 🙂

Cheers

Tim…

PS. Remember, I’m not a system administrator. I just know enough to be dangerous. 🙂