It is currently Sat May 30, 2026 1:47 am

View unanswered posts | View active topics


Board index » Buffalo Sabres Hockey » The Lounge

All times are UTC - 5 hours [ DST ]

Forum rules


Please click here to view the forum rules



Post new topic Reply to topic  Page 1 of 2
  [ 21 posts ]  Go to page 1, 2  Next
  Print view Previous topic | Next topic 
Author Message
Crosscheck
 Post subject: Maintenance complete.
PostPosted: Tue Apr 13, 2010 10:25 pm 
Offline
Sober enough to run a server
User avatar

Joined: Wed Sep 09, 2009 3:10 am
Posts: 7477
Location: 2,568 miles from the F'n arena
Sorry folks we had a not so minor hardware issue again.
Everything looks ok at this point but please let me know if you see errors anywhere (especially ones referring to the database or "mysql")

========Tech crap below...only read if you're interested========
Basically the software RAID mirror broke...which has happened in the past, and remounted the filesystem read only.
I've got a sketchy disk in this machine which has failed on media errors before (sdb).
However, this time the partition from sda was the one that failed in the mirror.
After wiping the partition and adding it back to the array (as is my usual practice) the array rebuilt to 99.9%, threw media errors on sdb (the "good" disk mind you) and restarted the 40 minute long sync process.

At this point I pooped my pants (not really, I have backups, I'm just allergic to extra days of recovery work).

Luckily the second time through the sync it actually completed.
Bounce
fsck
Bounce
and here we are...no db corruption as far as I can tell so far :D

I've got one of my boys up north in San Jose running a replacement disk over to the datacenter tomorrow afternoon.
They're hot swappable, but you never know ;)

_________________
Hold my beer and watch this...
Top
 Profile  
 
PuckSniperPensel
PostPosted: Tue Apr 13, 2010 10:27 pm 
Offline
Page Side
User avatar

Joined: Wed Sep 09, 2009 7:22 pm
Posts: 6537
Wooooooo new junkie parts!

_________________
Wheelhouse
Top
 Profile  
 
SABRESAllTheWay
PostPosted: Tue Apr 13, 2010 10:36 pm 
Offline
MegaDouche
User avatar

Joined: Wed Sep 09, 2009 9:41 am
Posts: 2752
Location: Fairfax, VA
PuckSniperPensel wrote:
Wooooooo new junkie parts!

-added to sig

_________________
Image
Top
 Profile  
 
Crosscheck
PostPosted: Tue Apr 13, 2010 10:41 pm 
Offline
Sober enough to run a server
User avatar

Joined: Wed Sep 09, 2009 3:10 am
Posts: 7477
Location: 2,568 miles from the F'n arena
Well here's one of the most useless emails I've received.
Since this server has some of my email go through it, I was without email for some accounts while the server was down.
Here's one of the messages waiting for me after the server was fixed:

Quote:
This is an automatically generated mail message from mdadm
running on deathstar

A Fail event had been detected on md device /dev/md2.

It could be related to component device /dev/sda3.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid1]
md0 : active raid1 sdb1[1] sda1[0]
104320 blocks [2/2] [UU]

md1 : active raid1 sdb2[1] sda2[0]
6289344 blocks [2/2] [UU]

md2 : active raid1 sdb3[1] sda3[2](F)
286639680 blocks [2/1] [_U]

unused devices: <none>


I'm going to go ahead and disable the "after-the-fact" monitoring feature ;)

_________________
Hold my beer and watch this...
Top
 Profile  
 
Squanto
PostPosted: Wed Apr 14, 2010 9:46 am 
Offline
Carlos Spicy-Wiener
User avatar

Joined: Wed Sep 09, 2009 10:31 am
Posts: 9240
Location: FAP TURBO
mdadm .... God you're a brave man.

Adaptec / 3ware cards or bust!! (Although they suck just as hard when they shit.)

I've just never had good luck with software based RAID. Maybe that's why I'm a network guy and not a systems admin. :p
Top
 Profile  
 
PuckSniperPensel
PostPosted: Wed Apr 14, 2010 10:13 am 
Offline
Page Side
User avatar

Joined: Wed Sep 09, 2009 7:22 pm
Posts: 6537
ETC?n

Deathstar :lol:

_________________
Wheelhouse
Top
 Profile  
 
PuckSniperPensel
PostPosted: Wed Apr 14, 2010 10:13 am 
Offline
Page Side
User avatar

Joined: Wed Sep 09, 2009 7:22 pm
Posts: 6537
SABRESAllTheWay wrote:
PuckSniperPensel wrote:
Wooooooo new junkie parts!

-added to sig


Win. :lol:

_________________
Wheelhouse
Top
 Profile  
 
NYIntensity
PostPosted: Wed Apr 14, 2010 11:17 am 
Offline
Superstar Goalie
User avatar

Joined: Sun Sep 13, 2009 2:11 pm
Posts: 4463
Squanto wrote:
mdadm .... God you're a brave man.

Adaptec / 3ware cards or bust!! (Although they suck just as hard when they shit.)

I've just never had good luck with software based RAID. Maybe that's why I'm a network guy and not a systems admin. :p


Software RAID? :pray:

_________________
ksquier89 wrote:
Holy fucking fuck...Boyes couldn't suck a dick if it landed in his mouth.
Top
 Profile  
 
Crosscheck
PostPosted: Wed Apr 14, 2010 11:34 am 
Offline
Sober enough to run a server
User avatar

Joined: Wed Sep 09, 2009 3:10 am
Posts: 7477
Location: 2,568 miles from the F'n arena
Squanto wrote:
mdadm .... God you're a brave man.


The box was given to me and slapped in a rack...I've never seen it and I had no input into how it was built.
So bravery doesn't really enter into it ;)

_________________
Hold my beer and watch this...
Top
 Profile  
 
jvaccaro6
PostPosted: Wed Apr 14, 2010 11:53 am 
Offline
Certifiably Hotter than ETC
User avatar

Joined: Wed Sep 09, 2009 1:19 pm
Posts: 3689
Crosscheck wrote:
Squanto wrote:
mdadm .... God you're a brave man.


The box was given to me and slapped in a rack...I've never seen it and I had no input into how it was built.
So bravery doesn't really enter into it ;)


PSSH! Some people call it ignorance, others call it bravery.

I personally am one of the "bravest" people I know :lol:

_________________
"She's like plan C after an 18 pack"

Squanto 7/27/2010
Top
 Profile  
 
Squanto
PostPosted: Wed Apr 14, 2010 11:53 am 
Offline
Carlos Spicy-Wiener
User avatar

Joined: Wed Sep 09, 2009 10:31 am
Posts: 9240
Location: FAP TURBO
Crosscheck wrote:
Squanto wrote:
mdadm .... God you're a brave man.


The box was given to me and slapped in a rack...I've never seen it and I had no input into how it was built.
So bravery doesn't really enter into it ;)


Guess you can't really look a gift horse in the mouth then... :p
Top
 Profile  
 
jvaccaro6
PostPosted: Wed Apr 14, 2010 11:58 am 
Offline
Certifiably Hotter than ETC
User avatar

Joined: Wed Sep 09, 2009 1:19 pm
Posts: 3689
Squanto wrote:
Crosscheck wrote:
Squanto wrote:
mdadm .... God you're a brave man.


The box was given to me and slapped in a rack...I've never seen it and I had no input into how it was built.
So bravery doesn't really enter into it ;)


Guess you can't really look a gift horse in the mouth then... :p


Unless you're a horse dentist... :dance:

Image

_________________
"She's like plan C after an 18 pack"

Squanto 7/27/2010
Top
 Profile  
 
Crosscheck
PostPosted: Wed Apr 14, 2010 2:30 pm 
Offline
Sober enough to run a server
User avatar

Joined: Wed Sep 09, 2009 3:10 am
Posts: 7477
Location: 2,568 miles from the F'n arena
FYI in 30 minutes or so I'm going to be taking this machine down to swap out the bad disk...don't panic ;)

_________________
Hold my beer and watch this...
Top
 Profile  
 
Crosscheck
PostPosted: Wed Apr 14, 2010 2:55 pm 
Offline
Sober enough to run a server
User avatar

Joined: Wed Sep 09, 2009 3:10 am
Posts: 7477
Location: 2,568 miles from the F'n arena
PuckSniperPensel wrote:
Wooooooo new junkie parts!


Junkie got its new parts...
...and there was much rejoicing.

_________________
Hold my beer and watch this...
Top
 Profile  
 
Squanto
PostPosted: Wed Apr 14, 2010 4:54 pm 
Offline
Carlos Spicy-Wiener
User avatar

Joined: Wed Sep 09, 2009 10:31 am
Posts: 9240
Location: FAP TURBO
Ed, you'd appropriate this.

Today, one of our sysadmins was working on a system in the back, a Debian box. He was trying to backup \home , so he copied it to \tmp\home . After rebooting the machine, he wondered where \tmp\home was........
Top
 Profile  
 
Crosscheck
PostPosted: Wed Apr 14, 2010 5:02 pm 
Offline
Sober enough to run a server
User avatar

Joined: Wed Sep 09, 2009 3:10 am
Posts: 7477
Location: 2,568 miles from the F'n arena
...in related news Tom's company is in the market for a competent unix administrator.

:lol:
Oh man that's bad.

_________________
Hold my beer and watch this...
Top
 Profile  
 
Squanto
PostPosted: Wed Apr 14, 2010 5:26 pm 
Offline
Carlos Spicy-Wiener
User avatar

Joined: Wed Sep 09, 2009 10:31 am
Posts: 9240
Location: FAP TURBO
Yeah, this guy is ... interesting.

He can do good work sometimes, but other times he's a complete fucking tool. I'm glad I'm not his boss, otherwise I'd hang myself. He's the master of overreaction and overstatement. For one thing, he actually thinks that botnets actively monitor our inbound email system and test us within 30 seconds of him making any kind of change.

It's quite amusing. Sad thing is he's better than almost all of our previous systems folks.
Top
 Profile  
 
Crosscheck
PostPosted: Wed Apr 14, 2010 5:46 pm 
Offline
Sober enough to run a server
User avatar

Joined: Wed Sep 09, 2009 3:10 am
Posts: 7477
Location: 2,568 miles from the F'n arena
yikes.
I've been around some "weird" systems guys but usually the weird factor increases with intelligence.

_________________
Hold my beer and watch this...
Top
 Profile  
 
NYIntensity
PostPosted: Wed Apr 14, 2010 7:50 pm 
Offline
Superstar Goalie
User avatar

Joined: Sun Sep 13, 2009 2:11 pm
Posts: 4463
I'll work on your ESX or Windows boxes, I just haven't gotten used to nix yet :(

_________________
ksquier89 wrote:
Holy fucking fuck...Boyes couldn't suck a dick if it landed in his mouth.
Top
 Profile  
 
Timbo Slice
PostPosted: Wed Apr 14, 2010 9:02 pm 
Offline
Franchise Defenseman
User avatar

Joined: Sun Oct 04, 2009 6:14 pm
Posts: 2708
Should I feel stupid that I have absolutely NO idea what that tech crap means?
Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  Page 1 of 2
  [ 21 posts ]  Go to page 1, 2  Next

Board index » Buffalo Sabres Hockey » The Lounge

All times are UTC - 5 hours [ DST ]

Who is online

Users browsing this forum: No registered users and 5 guests

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to: