Killing phpBB softly

My Polish forum is powered by phpBB. Undoubtedly, it’s the most popular bulletin board package. It’s free (as in freedom), easy to install and it’s easy to use. Virtually every Internet user had some exposure to it. When starting a new forum, it’s a safe choice.

As the years were passing by and my forum was growing bigger, I started being somewhat dissatisfied with it. Smaller and bigger annoyances were biting me every now and then. I’d like to point out some of them.

  • Search. Its user interface is unnecessarily complicated. It yields unsatisfactory results. As a result, people don’t want to use it and tend to ask the same questions over and over again. A good forum engine needs a decent search. Look at Vanilla’s search, it’s so simple and functional! Although it doesn’t mean I wouldn’t like to simplify it even a little more
  • Uncomfortable add-ons installation. So-called mods are distributed as instructions on how to modify the code. You have to open files and edit them by hand. One missed dot, BANG! Your forum is down. Want to upgrade your modified phpBB? It’s very likely that you will have to install it from scratch and install all the mods again. That’s why my moderators still don’t have the “merge topics” mod back. (sorry! I’ll try to install it some time!)
  • Crufty URLs. Compare “/viewtopic.php?t=1234” with “/topics/1234/i-like-clean-urls/”
  • Google won’t index it. It’s a mystery. Perhaps Google recognizes phpBB and avoids it. phpBB has a nasty habit of “enriching” its URLs with things that are different each time, generating infinite number of addresses. Google can never know if it has got all the topics from the forum. No wonder it gets discouraged. This causes a major problem: if the forum is not indexed, it doesn’t come up in search results and there ain’t no people coming! I consider it the biggest problem with phpBB.

I could also complain about lack of several features, including tags, ranking, finding similar topics, etc. Many of them are available… as mods of course. Theoretically, I could fix three of above problems, but I once phpBB would require an upgrade, I’d have to edit all the source code again, by hand. It’s the main reason why I wasn’t adding much things to the forum.

I tried installing Vanilla. It’s brilliant, but once I launched a test installation, users who visited it, complained about everything they could. I tried to fix things they were mentioning, but there was one major and inevitable problem: Vanilla ain’t look like phpBB. For example, buttons are in different places. Users are so tied to the existing interface that they can’t stand a button moved from right to left. I gave up with Vanilla.

I considered writing my own forum engine, then started having doubts and finally gave up. It’s too much hassle. Loads of work, data migration, user complaints… I would have probably rewritten the whole thing if I were younger. I would work furiously for many weeks, then force users into the new version, take flame-war attacks on my chest… No, I don’t want to do that any more.

However, I’m still too young to just sit around. Having just a few hours time, I started playing around with Django, writing a model on top of the phpBB database. I was soon able to fiddle with forums, topics and posts using Django’s ORM. I created a read-only forum archive with clean URLs, an RSS feed and a Sitemap for Google. The forum sitemap consists of about three thousands URLs, where each URL is a starting point of a topic. Each topic can have several pages.

My models work directly on phpBB database tables without modifying them. phpBB itself doesn’t even “know” that someone else is reading its dear tables.

My forum users didn’t notice anything. They’re happily using the old phpBB forum. In the meantime, Googlebot is crawling the Django-powered forum archive with dogged persistence. I think it will soon include the archive in its index and start directing traffic to it.

I’ll keep on developing the Django-powered forum. I can do it slowly and on-line. I will add a nice search engine, posts ranking and all other stuff that will come to my mind. Thing is, I won’t be touching the original phpBB tables. If I ever need to extend some models, I’ll just use Django OneToOne mapping. Current phpBB users will be able to use their forum just as they were before. However, all the cool features will be appearing on the new, Django-powered forum. They might find it more useful and start using it instead of the PHP version. It doesn’t need to happen any time soon. I can take my time developing the features as I want them. If they don’t like it, they can always go back to the PHP version.

It will be all soft. There will be no data migration. No forced user interface change. I’m going to slowly attract phpBB users to the new, Django-powered forum interface.

I’ll put all the phpBB-related code in a separate package and once it’s mature enough, publish it. It won’t be necessarily a forum implementation. It will be a Django-phpBB integration layer that will allow Django programmers to develop their own ideas for their phpBB-powered forums.

I’ll be killing phpBB softly.

Django on Dreamhost: incomplete headers

I’ve recently bought a hosting in Dreamhost. There were two reasons:

  1. It’s possible to run Django on it
  2. It’s cheap

It’s a shared hosting, where many sites are served from a single physical machine. Each machine probably serves as much sites as possible, where the hardware capacity is the limit. The Dreamhost servers are pretty busy. My server for instance:

[shasta]$ uptime
16:03:42 up 31 days, 13:59, 6 users, load average: 10.48, 9.74, 9.24

It’s not the processing power that is the bottleneck here, at least on “my” server. There’s usually about 40% of idle processor time. However, when I ran tail command, it would get killed every now and then. Strange. Perhaps there’s a “garbage process collector” running on the site, terminating non essential jobs here and there.

Well, I bought the hosting and moved my main site to Dreamhost. PHP software ― PhpBB and MediaWiki ― is working great. I decided to try running Django, so I developed a small application and installed it. I followed the instructions, voila, it worked. I was happy.

At least until the Django app would eventually stop responding. I clicked a link and the browser just waited for data. The data never came. I looked into the logs.

[Thu Nov 30 14:56:16 2006] [error] [client 83.xx.xxx.xx] FastCGI: comm with (dynamic) server “/home/automatthias/atopowe.pl/django.fcgi” aborted: (first read) idle timeout (120 sec)
[Thu Nov 30 14:56:16 2006] [error] [client 83.xx.xxx.xxx] FastCGI: incomplete headers (0 bytes) received from server “/home/automatthias/atopowe.pl/django.fcgi”

Something was wrong. Django isn’t officially supported on Dreamhost, so I couldn’t submit a complaint to the support. I searched the Web and found out that some guys had similar problems. Some other guys hadn’t. Dreamhost has many servers, and I figure it’s got something do to with the load of each server and… perhaps killing “non-essential” processes.

After some more research, I have found out that it’s not only Django users who’ve been experiencing that. There were also Rails users! You need to know, that Rails are officially supported on Dreamhost. I got interested and read on. Dreamhost support responded to the affected Rails user:

Check with our support team and ask if our process monitor has been killing your processes. If you have a lot of processes hanging around that may be the case. We recently updated our process monitor to specifically handle dispatch.fcgi processes specially so that is probably not the problem but it’s worth asking.

“specifically handle dispatch.fcgi processes”? Aha!

Inspecting my processes with “ps -ef” revealed that there were several “django.fcgi” processes running. Some of them were zombie (defunct). If the “dispatch.fcgi” processes are specifically handled, why don’t I pretend to run them? So I’ve changed my setup a little bit: I renamed my “django.fcgi” file to “dispatch.fcgi” and altered two lines in the “.htaccess” file, so they would refer to the new name:

RewriteRule ^(dispatch\.fcgi/.*)$ – [L]
RewriteRule ^(.*)$ dispatch.fcgi/$1 [L]

Guess what?

No timeouts, no 500s, no incomplete headers. It works like a charm.

I’ve successfully moved my sites to Dreamhost

It was pretty easy. Access to the shell is very convenient, I could avoid downloading and uploading large files via my slow DSL; I moved them directly between servers instead. Database migration was also pretty easy, shell access was helpful again.

I’ve changed domain delegations yesterday’s night. Changes were effective this morning. The downtime was quite short and it was only the night hours.

I’m happy about the migration:

  • my sites are now independent from the company servers
  • I can run Django applications there
  • The new hosting is fast and has a lot of space
  • I’m always happy having new toys that work

Dreamhost has approved my account

I’ve sent them a fax with my other credit card number and they approved my account. I’m not sure about the payment. The money seems gone. The account seems working. I thought they would return the money and I would pay again with the credit card. However, they have approved my account even though the credit card numbers didn’t match.

The billing tab shows $0 account balance, all functions seem to work. If nothing happens, I’m just going to enjoy the new hosting service.

Dreamhost’s vicious circle

I wanted to buy a two year hosting on Dreamhost. I’ve heard very good opinions and it’s possible to run Django on it (although it’s not officially supported). Parameters are excellent, the price is low, I decided to jump in.

The beginning was easy: a registration form, secure on-line payment with Visa card (an eCard in my case) and I could instantly sign in to my brand new Dreamhost account.

Suddenly… an e-mail came.

We’d like to thank you for signing up for our service! Your account information has been submitted and is currently pending approval of our payment information.

If you are paying by credit card, please print out the authorization form at:
http://www.dreamhost.com/fax/

…and fax it to us at (a phone number). The amount to fill in on the form is (the price). We regret the inconvenience this causes, but due to he amount of credit-card related fraud we incur we are forced to equire fax-back authentication.

I visited the fax page which said that I should rub the pencil over the credit card, so the embossed imprints are “copied” to the paper. I examined my eCard. No embossed imprints. The last sentence of this e-mail said: “If you have any questions, do not hesitate to ask”, so I wrote to Dreamhost.

OK, so here I go asking. I’ve paid with an eCard, which is a version f Visa card. I can pay with it — I’ve successfully done it the past. The thing is, my card does NOT have embossed number, name or expiration date. Well, it’s a little embossed, but I’ve tried rubbing it with a pencil and it’s unreadable.

My credit card has been charged and that’s OK. I just want to use my account. I can fax the form, but you won’t be able to read anything from the fax.

What should I do?

Their on billing was quick to answer.

Unfortunately since this account has been flagged, our credit card processor requires the fax rubbing. If your card is flat or an e-card,
then you’ll need to use a different card. I’m terribly sorry for the inconvenience but it’s a requirement we must meet.

I can understand that. I do want to use a different card. However…

Impossible payment on Dreamhost

I wrote them about the problem.

I don’t see any way of making the payment again.

I cannot do it in the panel because it says: “You may not make a payment to this account at this time. You may only make payments on approved accounts.” My account isn’t approved, right?

I cannot re-do the registration because it says: “domain “atopowe-zapalenie.pl” is already in our system!”

What should I do then?

It’s hilarious. They respond quickly (usually on the same day), but the answers are like this:

It looks like your account requires approval by our billing department before we can activate it.

Please go to this web page and print out this form…

http://dreamhost.com/fax

Fill it out, imprint your credit card and fax it to us and we can process it and enable your account.

Sorry for the inconvenience.

I think I’ve found an error in the Matrix. A hole in the system. Something that was never expected. They want to help me. When I ask a question, they answer. However, the answer always regards just one side of the vicious circle, and it doesn’t help me out.

My last email (so far) to them says:

I would gladly do that, if only I didn’t pay with an eCard, which doesn’t have embossed imprints, which makes it useless for your faxing procedure.

I want to cancel the first payment and pay with a credit card which has embossed imprints.

This is a vicious circle:

The original sin:
I payed with an eCard -> I can’t send the fax;

The circle: {
I can’t send the fax -> my account isn’t approved;
account isn’t approved -> I can’t pay with the credit card;
I can’t pay with the credit card -> I can’t send the fax;
}

Is there any way out of the vicious circle?

I hope there is. Otherwise, I’ll have to die, be born again as somebody else, sign up for Dreamhost with a different identity and pay with a credit card.

And furiously rub the pencil over it.

Something opposite

A long time ago, computers were not connected in any way. They were autistic and boring, but they felt lonely, so people worked hard and implemented network connections.

Hooray! It was possible to send a message from one computer and receive it on another!

Unfortunately, it soon turned out, that the network connections could be maliciously used, so people started implementing something opposite to the network connections: firewalls. Personal firewalls for Windows 98, in-build firewall in XP, everybody built a fortress around his computer.

Computers were autistic again, but a little less boring because this time it was possible to browse some web.

Somebody thought that maybe computers could be used for something more than web browsing. A new generation of software was born. The software, that makes something opposite to what firewalls do! The software, that pierces them! It can actually send and receive data! Wow!

Wait, didn’t we have this already in the beginning of the ’90s? Eh, never mind. We’re happy now.

Given that everybody installs now software that circumvents firewalls and allows to control their computer remotely, we are more and more concerned about people’s security. I’d like to suggest doing something opposite to what Skype does: second generation of firewalls. Let’s build a laserwall, let’s block Skype, remote assistance tools, p2p programs. Let’s block them all! It will not only improve people’s security, but also stimulate the market to produce next generation of software, that pierces our laserwall.

It’s natural: invent the network to exchange data, fight the network with firewalls to improve security, fight the firewalls to order exchange data, then fight the ones who fight the firewalls to get the security back.

What’s next?