http://www.sabresjunkie.com/forum/

Maintenance complete.
http://www.sabresjunkie.com/forum/viewtopic.php?f=8&t=2970		 Page 1 of 2
Author:  Crosscheck [ Tue Apr 13, 2010 10:25 pm ]
Post subject:  Maintenance complete.
Sorry folks we had a not so minor hardware issue again.
Everything looks ok at this point but please let me know if you see errors anywhere (especially ones referring to the database or "mysql")

========Tech crap below...only read if you're interested========
Basically the software RAID mirror broke...which has happened in the past, and remounted the filesystem read only.
I've got a sketchy disk in this machine which has failed on media errors before (sdb).
However, this time the partition from sda was the one that failed in the mirror.
After wiping the partition and adding it back to the array (as is my usual practice) the array rebuilt to 99.9%, threw media errors on sdb (the "good" disk mind you) and restarted the 40 minute long sync process.

At this point I pooped my pants (not really, I have backups, I'm just allergic to extra days of recovery work).

Luckily the second time through the sync it actually completed.
Bounce
fsck
Bounce
and here we are...no db corruption as far as I can tell so far :D

I've got one of my boys up north in San Jose running a replacement disk over to the datacenter tomorrow afternoon.
They're hot swappable, but you never know ;)
Author:  PuckSniperPensel [ Tue Apr 13, 2010 10:27 pm ]
Post subject:  Re: RFO (Nerd speak for why the damn site was down for 3 hours)
Wooooooo new junkie parts!
Author:  SABRESAllTheWay [ Tue Apr 13, 2010 10:36 pm ]
Post subject:  Re: RFO (Nerd speak for why the damn site was down for 3 hours)
PuckSniperPensel wrote:
Wooooooo new junkie parts!

-added to sig
Author:  Crosscheck [ Tue Apr 13, 2010 10:41 pm ]
Post subject:  Re: RFO (Nerd speak for why the damn site was down for 3 hours)
Well here's one of the most useless emails I've received.
Since this server has some of my email go through it, I was without email for some accounts while the server was down.
Here's one of the messages waiting for me after the server was fixed:

Quote:
This is an automatically generated mail message from mdadm
running on deathstar

A Fail event had been detected on md device /dev/md2.

It could be related to component device /dev/sda3.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid1]
md0 : active raid1 sdb1[1] sda1[0]
104320 blocks [2/2] [UU]

md1 : active raid1 sdb2[1] sda2[0]
6289344 blocks [2/2] [UU]

md2 : active raid1 sdb3[1] sda3[2](F)
286639680 blocks [2/1] [_U]

unused devices: <none>


I'm going to go ahead and disable the "after-the-fact" monitoring feature ;)
Author:  Squanto [ Wed Apr 14, 2010 9:46 am ]
Post subject:  Re: RFO (Nerd speak for why the damn site was down for 3 hours)
mdadm .... God you're a brave man.

Adaptec / 3ware cards or bust!! (Although they suck just as hard when they shit.)

I've just never had good luck with software based RAID. Maybe that's why I'm a network guy and not a systems admin. :p
Author:  PuckSniperPensel [ Wed Apr 14, 2010 10:13 am ]
Post subject:  Re: RFO (Nerd speak for why the damn site was down for 3 hours)
ETC?n

Deathstar :lol:
Author:  PuckSniperPensel [ Wed Apr 14, 2010 10:13 am ]
Post subject:  Re: RFO (Nerd speak for why the damn site was down for 3 hours)
SABRESAllTheWay wrote:
PuckSniperPensel wrote:
Wooooooo new junkie parts!

-added to sig


Win. :lol:
Author:  NYIntensity [ Wed Apr 14, 2010 11:17 am ]
Post subject:  Re: RFO (Nerd speak for why the damn site was down for 3 hours)
Squanto wrote:
mdadm .... God you're a brave man.

Adaptec / 3ware cards or bust!! (Although they suck just as hard when they shit.)

I've just never had good luck with software based RAID. Maybe that's why I'm a network guy and not a systems admin. :p


Software RAID? :pray:
Author:  Crosscheck [ Wed Apr 14, 2010 11:34 am ]
Post subject:  Re: RFO (Nerd speak for why the damn site was down for 3 hours)
Squanto wrote:
mdadm .... God you're a brave man.


The box was given to me and slapped in a rack...I've never seen it and I had no input into how it was built.
So bravery doesn't really enter into it ;)
Author:  jvaccaro6 [ Wed Apr 14, 2010 11:53 am ]
Post subject:  Re: RFO (Nerd speak for why the damn site was down for 3 hours)
Crosscheck wrote:
Squanto wrote:
mdadm .... God you're a brave man.


The box was given to me and slapped in a rack...I've never seen it and I had no input into how it was built.
So bravery doesn't really enter into it ;)


PSSH! Some people call it ignorance, others call it bravery.

I personally am one of the "bravest" people I know :lol:
Author:  Squanto [ Wed Apr 14, 2010 11:53 am ]
Post subject:  Re: RFO (Nerd speak for why the damn site was down for 3 hours)
Crosscheck wrote:
Squanto wrote:
mdadm .... God you're a brave man.


The box was given to me and slapped in a rack...I've never seen it and I had no input into how it was built.
So bravery doesn't really enter into it ;)


Guess you can't really look a gift horse in the mouth then... :p
Author:  jvaccaro6 [ Wed Apr 14, 2010 11:58 am ]
Post subject:  Re: RFO (Nerd speak for why the damn site was down for 3 hours)
Squanto wrote:
Crosscheck wrote:
Squanto wrote:
mdadm .... God you're a brave man.


The box was given to me and slapped in a rack...I've never seen it and I had no input into how it was built.
So bravery doesn't really enter into it ;)


Guess you can't really look a gift horse in the mouth then... :p


Unless you're a horse dentist... :dance:

Image
Author:  Crosscheck [ Wed Apr 14, 2010 2:30 pm ]
Post subject:  Re: RFO (Nerd speak for why the damn site was down for 3 hours)
FYI in 30 minutes or so I'm going to be taking this machine down to swap out the bad disk...don't panic ;)
Author:  Crosscheck [ Wed Apr 14, 2010 2:55 pm ]
Post subject:  Re: RFO (Nerd speak for why the damn site was down for 3 hours)
PuckSniperPensel wrote:
Wooooooo new junkie parts!


Junkie got its new parts...
...and there was much rejoicing.
Author:  Squanto [ Wed Apr 14, 2010 4:54 pm ]
Post subject:  Re: Maintenance complete.
Ed, you'd appropriate this.

Today, one of our sysadmins was working on a system in the back, a Debian box. He was trying to backup \home , so he copied it to \tmp\home . After rebooting the machine, he wondered where \tmp\home was........
Author:  Crosscheck [ Wed Apr 14, 2010 5:02 pm ]
Post subject:  Re: Maintenance complete.
...in related news Tom's company is in the market for a competent unix administrator.

:lol:
Oh man that's bad.
Author:  Squanto [ Wed Apr 14, 2010 5:26 pm ]
Post subject:  Re: Maintenance complete.
Yeah, this guy is ... interesting.

He can do good work sometimes, but other times he's a complete fucking tool. I'm glad I'm not his boss, otherwise I'd hang myself. He's the master of overreaction and overstatement. For one thing, he actually thinks that botnets actively monitor our inbound email system and test us within 30 seconds of him making any kind of change.

It's quite amusing. Sad thing is he's better than almost all of our previous systems folks.
Author:  Crosscheck [ Wed Apr 14, 2010 5:46 pm ]
Post subject:  Re: Maintenance complete.
yikes.
I've been around some "weird" systems guys but usually the weird factor increases with intelligence.
Author:  NYIntensity [ Wed Apr 14, 2010 7:50 pm ]
Post subject:  Re: Maintenance complete.
I'll work on your ESX or Windows boxes, I just haven't gotten used to nix yet :(
Author:  Timbo Slice [ Wed Apr 14, 2010 9:02 pm ]
Post subject:  Re: Maintenance complete.
Should I feel stupid that I have absolutely NO idea what that tech crap means?
Page 1 of 2 All times are UTC - 5 hours [ DST ]
Powered by phpBB® Forum Software © phpBB Group
http://www.phpbb.com/