Tsky Blog

Just when you thought it was safe to Xen

2010-02-08 (23:05:39)
So, yeah... The processing nodes and SAN decided to detonate today quite spectacularly, taking out the entire network with them. That was nice.
Not entirely sure why the processing pool has decided to cease talking to the SAN cluster but it decided that it would rather twiddle thumbs than communicate to the rest of the planet. Ho hum.
Yet another flaw in our outdated network - moving a backup copy from one location to another (3Tb over a 1Gbps link) takes an awwwwwwwwwwful long time. If the reconstruction doesn't go well, we have to move it back down the same link.. Uuuuuuurggggh.
This is why you need a decent infrastructure people.. don't EVER scrimp on the infrastructure - lesson in life there.