What a nightmare week its been!
Head boffin was rudely awakened early last Saturday with the news that one of our ads was making the site unusable. Simply put the problem was a symptom of an ad house's servers not being able to cope with the traffic it was trying to serve.
Now you would of thought the company in question would notice and put things right, these people aren't short of a couple of quid. Despite our best efforts we've had no luck on that front and reports from you guys hint that this problem is occurring more and more often.
The result is freeloader.com suffers the site is just not as responsive as we'd like, things look bad and its not our fault. Rather than ignore the issue or write a temporary hack, head boffin took the decision last Saturday morning to rework the way we serve ads - fun weekend, but that's why he's paid the bug bucks i guess ;)
I got to see what he did late last Sunday evening after he'd finished up, its all cool stuff. For a kick off freeloader's main menu no longer waits for a page to load before it becomes active - It makes the site feel so much snappier, the fading sections on the features bar to the right of a page are a nice touch too.
The main benefit is that any advert displayed by a page isn't loaded until the reset of the freeloader page has fully loaded and become active - meaning you no longer have to wait for all the blasted ads to load before you can use a page's features!
Hopefully all this will mean you enjoy a much more responsive freeloader.
So there we were on Monday happily showing off the sites improvements and going about our business when the support guys noticed a common thread to the emails they we're receiving, along the lines of; 'I've paid for a subscription but can't login, what gives, you guys suck, I'm gonna report you!', you get the idea.
If there was a big red button in the office someone would of pushed it! Turns out our payment system was happily charging users for subscriptions it wasn't setting up correctly - hence users weren't receiving their login details.
Late Wednesday the bug was found and patched, (a recent upgrade had broken some old legacy code), users refunded, humble pie eaten and massive apologies given to all... Calm in the freeloader office was once again restored... For about 3-4 hours.
At 1:40AM on Thursday morning our Database decided enough was enough and proceeded to die, taking with it a days worth of data - new accounts, logs etc, etc. Faulty hard-drives were the cause, (a bust raid rank controller for those that care). Bleary eyed engineers we called out at 2AM and fixed the problem in the small hours. We'd lost a good mornings worth of data but backups were used to restore as much info as we could, apologies to all affected users!
It could of been a whole lot worst, we're all just one database crash away from disaster!
All in all a fun packed week, NOT, ow did I mention that we're also trying to pack-up and move into a new office whist all this is going on? Thank God It's Friday, lets play some games!
