The great web technology shootout - Round 3: Better, Faster, and Shinier

A lot of the information below is out of date. Please see the new framework shootout page for the latest benchmarks.

This post is the continuation of a series. Please read Round 1 and Round 2 first if you are just now joining.

As I mentioned briefly in Round 1, this whole thing came about as an experiment to satisfy my own curiosity. Unfortunately, I wasn't expecting these posts to draw the amount of attention they have been getting, and several people informed me of a few "issues" with the first round. Since my initial approach to this topic was somewhat casual, I didn't really take the time to perform each test in a "proper scientific fashion." Although this was clearly stated in the introduction to round one, it unfortunately resulted in performance estimations that were somewhat less than accurate.

After input from various people much smarter than myself, I quickly went to work tweaking my test environment and building "proper" test apps. In the midst of this, a conversation about PHP accelerators prompted me to put PHP under the spotlight, which brought about Round 2 as an interim round. This gave me a chance to demonstrate the necessity of PHP acceleration, and only continued to solidify my opinion of PHP as an inferior web development language (remember, I just said my opinion).

Which brings us to Round 3. A lot of work has gone into "doing it right" this time, so I am fairly confident that these results are a much more accurate representation of each test subject's performance estimations. Remember, benchmark test code typically has no real-world value, so "performance estimations" are about all I can promise here. Your mileage *will* vary. As a wise person once said:

"All this benchmarking is doing is proving what we already know: More code takes longer to execute." - Ben Bangert (dev lead of Pylons)

What you should know about Round 3:

The hardware/software platform is the same as in Round 1.
Python v2.5.4 was used for all the Python tests. Ruby v1.8.6 was used for the Rails test.
Apache v2.2.3 was used, with mod_wsgi v2.5 for the Python tests and Phusion Passenger v2.2.5 for the Rails test.
Note that while the mod_wsgi tests in Round 1 were using "daemon mode", the mod_wsgi tests here are using "embedded mode" which is often faster (although not necessarily the recommended configuration).
The Python component versions used here were: SQLAlchemy v0.5.6, Genshi v0.5.1, Mako v0.2.5, and Jinja 2.2.1.
ApacheBench was run with -n 10000 -c 10.
Each test was run several times to make sure that there were no anomalies, with the "optimum average" chosen as the numbers represented here.
In an attempt to try to make these tests more scientific, I am now including the source code for each test.

Plain WSGI – wsgi_test_src.zip

Let's start with a basic "hello world" WSGI test to set the baseline:

The 'hello world' test:

WSGIServer/0.1
Document Path:          /
Document Length:        12 bytes
Requests per second:    1532.76 [#/sec] (mean)


Apache/2.2.3 (mod_wsgi)
Document Path:          /
Document Length:        12 bytes
Requests per second:    5549.73 [#/sec] (mean)

As you can see, mod_wsgi allows Apache to put some serious muscle into Python's WSGI. It's no wonder that mod_wsgi is quickly becoming the de-facto standard for Apache Python deployment.

TurboGears v2.0.3 – tg2_test_src.zip

As I mentioned in Round 1, TurboGears has quickly become my web framework of choice. Building a "best-of-breed" component stack on top of Pylons may not be easy, but when it works I believe it pays off immensely.

The 'hello world' test:

Document Path:          /
Document Length:        12 bytes
Requests per second:    935.13 [#/sec] (mean)


Genshi template test:

Document Path:          /genshi_hello/
Document Length:        923 bytes
Requests per second:    601.07 [#/sec] (mean)


Mako template test:

Document Path:          /mako_hello/
Document Length:        932 bytes
Requests per second:    723.18 [#/sec] (mean)


Jinja2 template test:

Document Path:          /jinja_hello/
Document Length:        937 bytes
Requests per second:    764.35 [#/sec] (mean)


Database query tests:

Document Path:          /raw_sql/
Document Length:        1270 bytes
Requests per second:    569.71 [#/sec] (mean)

Document Path:          /genshi_sql/
Document Length:        2409 bytes
Requests per second:    388.96 [#/sec] (mean)

Document Path:          /mako_sql/
Document Length:        2418 bytes
Requests per second:    515.07 [#/sec] (mean)

Document Path:          /jinja_sql/
Document Length:        2472 bytes
Requests per second:    535.06 [#/sec] (mean)

TurboGears v2.1a1 – tg21_test_src.zip

The TurboGears community is alive and well, and just before finishing this post the release of v2.1 Alpha 1 was announced. Significant improvements were made to TG's object dispatch, so I was excited to see what the new numbers would look like.

The 'hello world' test:

Document Path:          /
Document Length:        11 bytes
Requests per second:    1118.92 [#/sec] (mean)


Genshi template test:

Document Path:          /genshi_hello/
Document Length:        923 bytes
Requests per second:    713.97 [#/sec] (mean)


Mako template test:

Document Path:          /mako_hello/
Document Length:        932 bytes
Requests per second:    812.64 [#/sec] (mean)


Jinja2 template test:

Document Path:          /jinja_hello/
Document Length:        937 bytes
Requests per second:    1000.98 [#/sec] (mean)


Database query tests:

Document Path:          /raw_sql/
Document Length:        1270 bytes
Requests per second:    673.04 [#/sec] (mean)

Document Path:          /genshi_sql/
Document Length:        2409 bytes
Requests per second:    433.45 [#/sec] (mean)

Document Path:          /mako_sql/
Document Length:        2418 bytes
Requests per second:    601.89 [#/sec] (mean)

Document Path:          /jinja_sql/
Document Length:        2472 bytes
Requests per second:    630.66 [#/sec] (mean)

As you can see, the future of TurboGears looks very bright, and I only expect these numbers to get better as the final 2.1 release approaches (expected in Q1 2010).

Pylons v0.9.7 – pylons_test_src.zip

Pylons is the foundation which TurboGears is built upon, and for those who don't need all "the extras" it is an excellent choice.

The 'hello world' test:

Document Path:          /hello/index
Document Length:        12 bytes
Requests per second:    2593.47 [#/sec] (mean)


Mako template test:

Document Path:          /hello/hello
Document Length:        932 bytes
Requests per second:    1737.03 [#/sec] (mean)


Database query tests:

Document Path:          /hello/raw_sql
Document Length:        1270 bytes
Requests per second:    964.07 [#/sec] (mean)

Document Path:          /hello/hellodb
Document Length:        2418 bytes
Requests per second:    831.26 [#/sec] (mean)

Django v1.1 – django_test_src.zip

While Django is probably Python's most popular web framework, I have never really clicked with it. Its way of doing things is significantly different than TG/Pylons, but if you don't mind its "MTV" architectural pattern and can survive without SQLAlchemy (although there are hacks), it is a very capable framework.

The 'hello world' test:

Document Path:          /hello
Document Length:        12 bytes
Requests per second:    3376.48 [#/sec] (mean)


Django template test:

Document Path:          /hellos
Document Length:        936 bytes
Requests per second:    1781.43 [#/sec] (mean)


Database query test:

Document Path:          /hellodb
Document Length:        2476 bytes
Requests per second:    972.11 [#/sec] (mean)

Bottle 0.5.8 – bottle_test_src.zip

Bottle caught my attention a few months ago while surveying the WSGI landscape for an alternative to PHP when building websites that only require a few pieces of dynamic functionality. As a "micro-framework", it does an excellent job at bridging the gap between "pure WSGI" and its "feature-rich" big brothers.

The 'hello world' test:

Document Path:          /
Document Length:        12 bytes
Requests per second:    5545.96 [#/sec] (mean)


Mako template test:

Document Path:          /hello
Document Length:        923 bytes
Requests per second:    3588.04 [#/sec] (mean)


Database query test (SQLAlchemy):

Document Path:          /hellodb
Document Length:        2413 bytes
Requests per second:    1479.54 [#/sec] (mean)

As you can see, there was practically no difference in speed between Bottle and pure WSGI in a basic "hello world" test. Even with the addition of Mako and SQLAlchemy, Bottle performed significantly faster than a bare Pylons or Django setup.

(Note: Mako template inheritance was not used in the Bottle Mako test.)

Ruby on Rails v2.3.3 – rails_test_src.zip

What web technology shootout would be complete without Rails? Although I am not a Rails fan (mostly because I can't stand Ruby), I believe that in many ways we owe much of the popularity and success of the web framework movement to Rails.

The 'hello world' test:

Document Path:          /hello/hello
Document Length:        12 bytes
Requests per second:    1048.18 [#/sec] (mean)


Rails template test:

Document Path:          /hello/hellos
Document Length:        937 bytes
Requests per second:    949.54 [#/sec] (mean)


Database query test:

Document Path:          /hello/hellodb
Document Length:        2482 bytes
Requests per second:    734.18 [#/sec] (mean)

(Note: I think my version of Ruby was using a newer version of the SQLite3 library than my version of Python. This may or may not have given Rails an advantage on the database query test.)

For those of you who like charts, here's a comparison of the "return hello world" test results for each framework:

And here's a comparison of the template test:

Finally, here's the template test with the database query added:

Closing thoughts

Gone are the days of Rails as "the one framework to rule them all." C'mon guys, it's not 2005 anymore.
Python has gotten a lot of flak from other language camps for its abundance of web frameworks. While some Pythonists have been embarrassed by this, I only see it as proof of Python's success as a language. My retort would be: "If it was as easy to write a web framework in your language as it is in Python, you'd have the same problem!"