We’re doing some house cleaning on our web servers. Some of our websites have been, and will be, moving around over the next couple weeks. If you happen to get an error about the server being down, this would be the reason. For our more important websites, we will be taking great care to minimize the downtime.
Archive for the ‘web server’ Category
Expect some Downtime on a few Perceptus Websites
Sunday, November 9th, 2014Reducing Memory Used by Milter-Greylist
Monday, July 21st, 2008Our VPS was running low on free memory the last few weeks. After a bit of research, we realized that our email greylisting software, Milter-Greylist was using the most memory of everything installed on our server. More than our database engine, web server, email server, and everything else (not combined)!
For those who don’t know, Grey Listing delays emails in an attempt to foil spammers which don’t typically follow standards for retrying email messages. Milter-greylist is a package that works with sendmail, our SMTP server. Milter-greylist is great, however, it keeps it’s working history in memory, which was OK for the two years that we have run it. However, the amount of spam attempts continues to rise… why don’t home users notice that their computers have become SPAM zombies anyway?
So, the milter-greylist was storing tens of thousands of records in memory. It had to be reduced. Rather than switch to a database driven greylisting package, we decided to start blocking some SPAM attempts earlier in the process.
We enabled the outright blocking of inbound email attempts by any IP address listed on Spamhaus.org’s SBL+XBL list. SBL+XBL are lists of computers (built by crazy wizardry) that one can use to blacklist email attempts. I’m uncomfortable using blacklists like this, but, what can you do? The odds of good mail being lost are very small, and hopefully, anyone who happens to get bounced unintentionally can phone us.
So, following the simple instructions here:
http://www.joeldare.com/papers/spamhaus.pdf
We were able to reduce the traffic to Milter-Greylist and it’s memory usage by 2/3 to 3/4! Uptime and performance of our VPS and therefore everything hosted on it should be slightly better.
Spammers suck.
A new blog we’re hosting: “You’re not sushi – you’re Chicken!”
Monday, July 21st, 2008Perceptus hosts a few sites for friends, family, and other non-commercial purposes.
We added a new one a few days ago for Leonard, his sister, and a couple of their cousins. This crazy subset of the Chan Clan is up to no good again.*
They are building a giant roast chicken on wheels and it’ll be screaming down the first Red Bull Soapbox Race in Canada, this September 7, 2008 in Vancouver.
So, check out the You’re not sushi – you’re Chicken! in Red Bull Soapbox Vancouver 2008 blog.
* They were part of the Red Bull Flugtag Vancouver 2006, with the amazingly large and beautiful craft, The Wild Sushi – this blog is also hosted by Perceptus Solutions Inc.
Apologies for the downtime, or, how our VPS provider ruined my weekend.
Monday, December 10th, 2007To our valued users of our websites, especially those at PapayaPolls.com and Print-Bingo.com, we’re sorry.
Due to circumstances that were generally outside of our control, Perceptus’ virtual private server was down for 3-4 days (there were a couple multi-hour stints of uptime here and there).
The long story, short:
Our VPS provider (a VPS is a super-fancy variant of web hosting), with whom we’ve hosted for the last three years or so, had a border router go down badly on Friday.
Now, when we were choosing a VPS provider, we specifically looked for one that had fully redundant power, networks connections, reasonably intelligent sounding support, etc. This one did and still does advertise as such; however, as we’ve now discovered, this supposedly fully-redundant network of our VPS provider turned out to be mostly redundant with at least one single point of failure exception. When this border-router went down, there was no backup link, nor was there a convenient replacement unit for a quick swap. So their network went dead to the world for hours.
Eventually, our VPS provider fixed their network. However, the Perceptus’ VPS remained down. Several email tickets and live chats with support later, they figured out what was wrong, and the Perceptus VPS is finally up again on Monday. It’s been several hours now, so our fingers are crossed that this might actually be over.
I won’t name our provider, but if you poke around enough, you’ll figure out who our VPS provider is. Suffice it to say that for the time being, don’t ask me for a VPS recommendation… I don’t have anyone to recommend. I don’t really blame them for the first few hours of downtime, overall I’ve been quite happy with them. But when a few hours stretched to a few days, they lost a lot of goodwill with me.
Lessons learned:
- When things go down and you’ve got no control over the fix, start implementing a fall-back plan right away. Even if someone who claims to be in a position of knowledge says it will be fixed in a few hours, start the work on the fallback plan anyways. Nine times out of ten, things will get fixed before you have to go live with Plan B, but your time isn’t wasted. Consider it a test-run of your backup plans for the one time that you will be very happy that you did start on Plan B ASAP.
- When possible, avoid single-points-of-failure, this includes your web host. Ironically, our VPS provider did a survey about a month ago asking about our interest in “high availability VPS'”… guess what would have happened to one of these last week? Yep, it would have been down anyway because the problem was at a choke point higher up the chain than the server.
- When choosing a web host, ask if they actually have staff in the same city as their data centre. If they are just relaying tickets to the data centre staff, they don’t really have control over when anything is done either. I don’t know if our VPS provider had such a setup when we first signed up, but I have every indication that they weren’t physically in the data centre this weekend.
The Future for Perceptus’ Web Server:
We’ll be looking at setting up a fail-over server on a totally separate network, completely unrelated to our current setup. The only question is how will we do it relatively cheaply and with relatively low maintenance. We’ll post something when we figure it out.