Django development server on Dreamhost

My Django application on Dreamhost is running fine since I did one simple trick. There is one problematic moment, though. Namely, new Python code to upload and application restart.

I usually develop my applications locally and submit the new code to a repository. To make the changes live, I check out the new code from the repository to my Dreamhost account and issue a command:

touch dispatch.fcgi

If my application still doesn’t restart, I push a little harder.

killall /usr/bin/python2.4

If I do just one restart, it goes fine. The problems start, when I do it few times in a row. In such a case, the old version of the application is already killed (killall’ed) but the new application isn’t alive yet and I get some 500 and incomplete headers error for a while. The application eventually starts up and works fine.

When the code is ready, one check-out and one restart is enough, but it’s sometimes necessary to make small changes on-site and frequently restart the application.

Frequent restarts are solved very nicely by the development server. It monitors all the Python files and restarts itself on every change detected. It would be perfect to run an development server on Dreamhost to make the necessary changes and then restart the application just once.

It’s easy to run the development server itself.

./manage.py runserver

The problem is, how to display what it serves, in a local browser?

EDIT: As Ryan Berg suggested, it’s enough to tell the development server to listen on an external interface:

./manage.py runserver http://www.mydomain.com:8001

The Dreamhost servers aren’t firewalled, which means that you can open your development site by pointing your web browser to “http://www.mydomain.com:8001”.

Please note: if you host more domains on Dreamhost, your development site will be available under all the domains you’re hosting, for example firstdomain.com:8001, seconddomain.com:8001, and so forth.

You can make frequent small changes and numerous development server restarts, and once You’re satisfied, do just one restart of the application served via FCGI.

Advertisements

Django on Dreamhost: incomplete headers

I’ve recently bought a hosting in Dreamhost. There were two reasons:

  1. It’s possible to run Django on it
  2. It’s cheap

It’s a shared hosting, where many sites are served from a single physical machine. Each machine probably serves as much sites as possible, where the hardware capacity is the limit. The Dreamhost servers are pretty busy. My server for instance:

[shasta]$ uptime
16:03:42 up 31 days, 13:59, 6 users, load average: 10.48, 9.74, 9.24

It’s not the processing power that is the bottleneck here, at least on “my” server. There’s usually about 40% of idle processor time. However, when I ran tail command, it would get killed every now and then. Strange. Perhaps there’s a “garbage process collector” running on the site, terminating non essential jobs here and there.

Well, I bought the hosting and moved my main site to Dreamhost. PHP software ― PhpBB and MediaWiki ― is working great. I decided to try running Django, so I developed a small application and installed it. I followed the instructions, voila, it worked. I was happy.

At least until the Django app would eventually stop responding. I clicked a link and the browser just waited for data. The data never came. I looked into the logs.

[Thu Nov 30 14:56:16 2006] [error] [client 83.xx.xxx.xx] FastCGI: comm with (dynamic) server “/home/automatthias/atopowe.pl/django.fcgi” aborted: (first read) idle timeout (120 sec)
[Thu Nov 30 14:56:16 2006] [error] [client 83.xx.xxx.xxx] FastCGI: incomplete headers (0 bytes) received from server “/home/automatthias/atopowe.pl/django.fcgi”

Something was wrong. Django isn’t officially supported on Dreamhost, so I couldn’t submit a complaint to the support. I searched the Web and found out that some guys had similar problems. Some other guys hadn’t. Dreamhost has many servers, and I figure it’s got something do to with the load of each server and… perhaps killing “non-essential” processes.

After some more research, I have found out that it’s not only Django users who’ve been experiencing that. There were also Rails users! You need to know, that Rails are officially supported on Dreamhost. I got interested and read on. Dreamhost support responded to the affected Rails user:

Check with our support team and ask if our process monitor has been killing your processes. If you have a lot of processes hanging around that may be the case. We recently updated our process monitor to specifically handle dispatch.fcgi processes specially so that is probably not the problem but it’s worth asking.

“specifically handle dispatch.fcgi processes”? Aha!

Inspecting my processes with “ps -ef” revealed that there were several “django.fcgi” processes running. Some of them were zombie (defunct). If the “dispatch.fcgi” processes are specifically handled, why don’t I pretend to run them? So I’ve changed my setup a little bit: I renamed my “django.fcgi” file to “dispatch.fcgi” and altered two lines in the “.htaccess” file, so they would refer to the new name:

RewriteRule ^(dispatch\.fcgi/.*)$ – [L]
RewriteRule ^(.*)$ dispatch.fcgi/$1 [L]

Guess what?

No timeouts, no 500s, no incomplete headers. It works like a charm.