503 errors and PHPBB

Discussion in 'General troubleshooting' started by MediaThrall, Jul 1, 2011.

  1. I thought I'd go ahead and address the issue of frequent 503 errors in this thread, as it's something I've been struggling to deal with for the past month, and others may find my experience (and discovery) helpful.

    The Problem:
    A while back, my site's users began reporting that they were encountering 503 errors every 5 minutes or so, which would make the website inaccessible for several minutes each time. My site hosts a game, so my users are frequently on for several hours at a time, and these errors were crippling their experience, so I began investigating them.

    Contacting customer support, I was told that my site was consuming too much CPU on the server, causing it to reset the application pool because this was violating an IIS rule in place to regulate server performance. I was told my site was too expensive, and that I needed to optimize it to fix the errors.

    This seemed strange to me, as I hadn't made any significant changes to the site in several months. Still, I set forth optimizing, using RedGate's ANTS performance profiler, and spent over 30 hours improving the site's performance. To my dismay, this didn't fix the issue or improve conditions at all.

    It seemed odd to me that despite massive optimizations and very good performance profiles, the application was still hogging almost the entire CPU. So I rolled back the entire website to a backup I had made several months ago, which did not have the 503 issues - and they still occurred!

    At this point, there seemed like there was nothing I could do to fix it, and nothing customer service could do to help me. The site was unusable, it had been this way for a month, so I abandoned it and decided to shut it down. But before I did, I took one last glance at the page view statistics, and found something intriguing.

    The Discovery!
    The user control panel for my phpbb forum was getting HAMMERED with page views. Its number of views exceeded the total number of page views for all the rest of the site!

    As it turns out, my forum was suffering under an extended spam attack. Forum spam bots had identified the URL of my forum and had been continuously trying to gain access and post to it. Over time, the volume of page requests had begun to cripple the server's CPU.

    I disabled the forum to test my hypothesis and the errors immediately vanished. I can only assume that a web crawler found and indexed by forum for google, and that's how it became so "popular."

    I had chosen to use PHPBB as my forum solution because it's an extremely popular, works-out-of-the-box solution. Unfortunately, that's also what made it the most vulnerable - its popularity makes it a prime target for such attacks.

    For now, I've come up with no solution except to remove the forum. The configuration the forum was in was the most secure available, but it didn't prevent the spam bots from trying to get in anyway. I'm surprised Winhost didn't notice / couldn't help me detect this attack.

    Hopefully this post will help other Winhost users that deploy phpbb forums on their sites who may be encountering this same issue.
     
    Last edited by a moderator: Oct 14, 2015
  2. A number of spam bots trying to access parts of your site can be a big enough problem to consume all of the resources allocated to the individual site, but not enough to create even a blip in overall network traffic. I suspect that's what happened.

    Our network traffic is sizable, so I'm afraid we would not likely notice a few hundred new connections to your site. In order for any red flags to be raised on our end there has to be a considerable amount of unusual traffic. By "considerable" I mean hundreds of thousands (or millions) of new connections that weren't there the day before.

    There are also intrusion detectors in place, but since the bots were trying to do a legitimate action on your forum (and not a SQL injection, etc.), they wouldn't be blocked.
     
    Last edited: Oct 14, 2015

Share This Page