Posts Tagged ‘vps’

This blog moved. No moving trucks required.

Friday, June 24th, 2011

The Perceptus Back Story blog was down for an hour or so yesterday while we migrated it from one virtual private server to another.

The reason for the move? It’s about time we install a few WordPress plugins that we’ve had our eye on.  We know that commenting on this blog is awkward – you have to register.  We’ll be playing around with 3rd party logins for commenting.  It might even happen today.

Plus, the old server is destined to be rebuilt sometime in the hazy future – the software on it is nearing it’s end of life.

Reducing Memory Used by Milter-Greylist

Monday, July 21st, 2008

Our VPS was running low on free memory the last few weeks.  After a bit of research, we realized that our email greylisting software, Milter-Greylist was using the most memory of everything installed on our server.  More than our database engine, web server, email server, and everything else (not combined)!

For those who don’t know, Grey Listing delays emails in an attempt to foil spammers which don’t typically follow standards for retrying email messages. Milter-greylist is a package that works with sendmail, our SMTP server. Milter-greylist is great, however, it keeps it’s working history in memory, which was OK for the two years that we have run it.  However, the amount of spam attempts continues to rise… why don’t home users notice that their computers have become SPAM zombies anyway?

So, the milter-greylist was storing tens of thousands of records in memory.  It had to be reduced.  Rather than switch to a database driven greylisting package, we decided to start blocking some SPAM attempts earlier in the process.

We enabled the outright blocking of inbound email attempts by any IP address listed on Spamhaus.org’s SBL+XBL list.  SBL+XBL are lists of computers (built by crazy wizardry) that one can use to blacklist email attempts.  I’m uncomfortable using blacklists like this, but, what can you do?  The odds of good mail being lost are very small, and hopefully, anyone who happens to get bounced unintentionally can phone us.

So, following the simple instructions here:

http://www.joeldare.com/papers/spamhaus.pdf

We were able to reduce the traffic to Milter-Greylist and it’s memory usage by 2/3 to 3/4!  Uptime and performance of our VPS and therefore everything hosted on it should be slightly better.

Spammers suck.

A bit of downtime the over the last couple days.

Thursday, April 17th, 2008

Quick note to all who use the various Perceptus web sites, particularly our bingo card site and our hosted survey site:

Our virtual private server hardware was upgraded.  A couple hours of downtime occurred.  We’re sorry for the inconvenience.  Unfortunately, our provider did not give us much of a warning. Things seem to be up and running, but a bit slow – I think that will fix itself soon.

Apologies for the downtime, or, how our VPS provider ruined my weekend.

Monday, December 10th, 2007

To our valued users of our websites, especially those at PapayaPolls.com and Print-Bingo.com, we’re sorry.

Due to circumstances that were generally outside of our control, Perceptus’ virtual private server was down for 3-4 days (there were a couple multi-hour stints of uptime here and there).

The long story, short:

Our VPS provider (a VPS is a super-fancy variant of web hosting), with whom we’ve hosted for the last three years or so, had a border router go down badly on Friday.

Now, when we were choosing a VPS provider, we specifically looked for one that had fully redundant power, networks connections, reasonably intelligent sounding support, etc. This one did and still does advertise as such; however, as we’ve now discovered, this supposedly fully-redundant network of our VPS provider turned out to be mostly redundant with at least one single point of failure exception. When this border-router went down, there was no backup link, nor was there a convenient replacement unit for a quick swap. So their network went dead to the world for hours.

Eventually, our VPS provider fixed their network. However, the Perceptus’ VPS remained down. Several email tickets and live chats with support later, they figured out what was wrong, and the Perceptus VPS is finally up again on Monday.  It’s been several hours now, so our fingers are crossed that this might actually be over.

I won’t name our provider, but if you poke around enough, you’ll figure out who our VPS provider is. Suffice it to say that for the time being, don’t ask me for a VPS recommendation… I don’t have anyone to recommend. I don’t really blame them for the first few hours of downtime, overall I’ve been quite happy with them. But when a few hours stretched to a few days, they lost a lot of goodwill with me.

Lessons learned:

  1. When things go down and you’ve got no control over the fix, start implementing a fall-back plan right away. Even if someone who claims to be in a position of knowledge says it will be fixed in a few hours, start the work on the fallback plan anyways. Nine times out of ten, things will get fixed before you have to go live with Plan B, but your time isn’t wasted. Consider it a test-run of your backup plans for the one time that you will be very happy that you did start on Plan B ASAP.
  2. When possible, avoid single-points-of-failure, this includes your web host. Ironically, our VPS provider did a survey about a month ago asking about our interest in “high availability VPS'”… guess what would have happened to one of these last week? Yep, it would have been down anyway because the problem was at a choke point higher up the chain than the server.
  3. When choosing a web host, ask if they actually have staff in the same city as their data centre. If they are just relaying tickets to the data centre staff, they don’t really have control over when anything is done either. I don’t know if our VPS provider had such a setup when we first signed up, but I have every indication that they weren’t physically in the data centre this weekend.

The Future for Perceptus’ Web Server:

We’ll be looking at setting up a fail-over server on a totally separate network, completely unrelated to our current setup. The only question is how will we do it relatively cheaply and with relatively low maintenance. We’ll post something when we figure it out.