[Update 10/6: Slight updates to the Pylons test after some input from Ben. Added a couple new charts for you visual folks.]

As I briefly mentioned in Round 1, this whole thing came about as an experiment to satisfy my own curiosity. Unfortunately, I wasn’t expecting these posts to draw the amount of attention they have been getting, and several people informed me of a few “issues” with the first round. Since my initial approach to this topic was somewhat casual, I didn’t really take the time to perform each test in a “proper scientific fashion.” Although this was clearly stated in the introduction to round one, it unfortunately resulted in performance estimations that were somewhat less than accurate.

After input from various people much smarter than myself, I quickly went to work tweaking my test environment and building “proper” test apps. In the midst of this, a conversation about PHP accelerators prompted me to put PHP under the spotlight, which brought about Round 2 as an interim round. This gave me a chance to demonstrate the necessity of PHP acceleration, and only continued to solidify my opinion of PHP as an inferior web development language (remember, I just said my opinion).

Which brings us to Round 3. A lot of work has gone into “doing it right” this time, so I am fairly confident that these results are a much more accurate representation of each test subject’s performance estimations. Remember, benchmark test code typically has no real-world value, so “performance estimations” are about all I can promise here. Your mileage will vary. As a wise person once said:

“All this benchmarking is doing is proving what we already know: More code takes longer to execute.” – Ben Bangert (dev lead of Pylons)

What you should know about Round 3:

• The hardware/software platform is the same as in Round 1.
• Python v2.5.4 was used for all the Python tests. Ruby v1.8.6 was used for the Rails test.
• Apache v2.2.3 was used, with mod_wsgi v2.5 for the Python tests and Phusion Passenger v2.2.5 for the Rails test.
• Note that while the mod_wsgi tests in Round 1 were using “daemon mode”, the mod_wsgi tests here are using “embedded mode” which is often faster (although not necessarily the recommended configuration).
• The Python component versions used here were: SQLAlchemy v0.5.6, Genshi v0.5.1, Mako v0.2.5, and Jinja 2.2.1.
• ApacheBench was run with -n 10000 -c 10.
• Each test was run several times to make sure that there were no anomalies, with the “optimum average” chosen as the numbers represented here.
• In an attempt to try to make these tests more scientific, I am now including the source code for each test.

Plain WSGI – wsgi_test_src.zip

Let’s start with a basic “hello world” WSGI test to set the baseline:

The 'hello world' test:
 
WSGIServer/0.1
Document Path:          /
Document Length:        12 bytes
Requests per second:    1532.76 [#/sec] (mean)
 
 
Apache/2.2.3 (mod_wsgi)
Document Path:          /
Document Length:        12 bytes
Requests per second:    5549.73 [#/sec] (mean)

As you can see, mod_wsgi allows Apache to put some serious muscle into Python’s WSGI. It’s no wonder that mod_wsgi is quickly becoming the de-facto standard for Apache Python deployment.

TurboGears v2.0.3 – tg2_test_src.zip

As I mentioned in Round 1, TurboGears has quickly become my web framework of choice. Building a “best-of-breed” component stack on top of Pylons may not be easy, but when it works I believe it pays off immensely (e.g. Ubuntu).

The 'hello world' test:
 
Document Path:          /
Document Length:        12 bytes
Requests per second:    935.13 [#/sec] (mean)
 
 
Genshi template test:
 
Document Path:          /genshi_hello/
Document Length:        923 bytes
Requests per second:    601.07 [#/sec] (mean)
 
 
Mako template test:
 
Document Path:          /mako_hello/
Document Length:        932 bytes
Requests per second:    723.18 [#/sec] (mean)
 
 
Jinja2 template test:
 
Document Path:          /jinja_hello/
Document Length:        937 bytes
Requests per second:    764.35 [#/sec] (mean)
 
 
Database query tests:
 
Document Path:          /raw_sql/
Document Length:        1270 bytes
Requests per second:    569.71 [#/sec] (mean)
 
Document Path:          /genshi_sql/
Document Length:        2409 bytes
Requests per second:    388.96 [#/sec] (mean)
 
Document Path:          /mako_sql/
Document Length:        2418 bytes
Requests per second:    515.07 [#/sec] (mean)
 
Document Path:          /jinja_sql/
Document Length:        2472 bytes
Requests per second:    535.06 [#/sec] (mean)

TurboGears v2.1a1 – tg21_test_src.zip

The TurboGears community is alive and well, and just before finishing this post the release of v2.1 Alpha 1 was announced. Significant improvements were made to TG’s object dispatch, so I was excited to see what the new numbers would look like.

The 'hello world' test:
 
Document Path:          /
Document Length:        11 bytes
Requests per second:    1118.92 [#/sec] (mean)
 
 
Genshi template test:
 
Document Path:          /genshi_hello/
Document Length:        923 bytes
Requests per second:    713.97 [#/sec] (mean)
 
 
Mako template test:
 
Document Path:          /mako_hello/
Document Length:        932 bytes
Requests per second:    812.64 [#/sec] (mean)
 
 
Jinja2 template test:
 
Document Path:          /jinja_hello/
Document Length:        937 bytes
Requests per second:    1000.98 [#/sec] (mean)
 
 
Database query tests:
 
Document Path:          /raw_sql/
Document Length:        1270 bytes
Requests per second:    673.04 [#/sec] (mean)
 
Document Path:          /genshi_sql/
Document Length:        2409 bytes
Requests per second:    433.45 [#/sec] (mean)
 
Document Path:          /mako_sql/
Document Length:        2418 bytes
Requests per second:    601.89 [#/sec] (mean)
 
Document Path:          /jinja_sql/
Document Length:        2472 bytes
Requests per second:    630.66 [#/sec] (mean)

As you can see, the future of TurboGears looks very bright, and I only expect these numbers to get better as the final 2.1 release approaches (expected in Q1 2010).

Pylons v0.9.7 – pylons_test_src.zip

Pylons is the foundation which TurboGears is built upon, and for those who don’t need all “the extras” it is an excellent choice.

The 'hello world' test:
 
Document Path:          /hello/index
Document Length:        12 bytes
Requests per second:    2593.47 [#/sec] (mean)
 
 
Mako template test:
 
Document Path:          /hello/hello
Document Length:        932 bytes
Requests per second:    1737.03 [#/sec] (mean)
 
 
Database query tests:
 
Document Path:          /hello/raw_sql
Document Length:        1270 bytes
Requests per second:    964.07 [#/sec] (mean)
 
Document Path:          /hello/hellodb
Document Length:        2418 bytes
Requests per second:    831.26 [#/sec] (mean)

Django v1.1 – django_test_src.zip

While Django is probably Python’s most popular web framework, I have never really clicked with it. Its way of doing things is significantly different than TG/Pylons, but if you don’t mind its “MTV” architectural pattern and can survive without SQLAlchemy (although there are hacks), it is a very capable framework.

The 'hello world' test:
 
Document Path:          /hello
Document Length:        12 bytes
Requests per second:    3376.48 [#/sec] (mean)
 
 
Django template test:
 
Document Path:          /hellos
Document Length:        936 bytes
Requests per second:    1781.43 [#/sec] (mean)
 
 
Database query test:
 
Document Path:          /hellodb
Document Length:        2476 bytes
Requests per second:    972.11 [#/sec] (mean)

Bottle 0.5.8 – bottle_test_src.zip

Bottle caught my attention a few months ago while surveying the WSGI landscape for an alternative to PHP when building websites that only require a few pieces of dynamic functionality. As a “micro-framework”, it does an excellent job at bridging the gap between “pure WSGI” and its “feature-rich” big brothers.

The 'hello world' test:
 
Document Path:          /
Document Length:        12 bytes
Requests per second:    5545.96 [#/sec] (mean)
 
 
Mako template test:
 
Document Path:          /hello
Document Length:        923 bytes
Requests per second:    3588.04 [#/sec] (mean)
 
 
Database query test (SQLAlchemy):
 
Document Path:          /hellodb
Document Length:        2413 bytes
Requests per second:    1479.54 [#/sec] (mean)

As you can see, there was practically no difference in speed between Bottle and pure WSGI in a basic “hello world” test. Even with the addition of Mako and SQLAlchemy, Bottle performed significantly faster than a bare Pylons or Django setup.

(Note: Mako template inheritance was not used in the Bottle Mako test.)

Ruby on Rails v2.3.3 – rails_test_src.zip

What web technology shootout would be complete without Rails? Although I am not a Rails fan (mostly because I can’t stand Ruby), I believe that in many ways we owe much of the popularity and success of the web framework movement to Rails.

The 'hello world' test:
 
Document Path:          /hello/hello
Document Length:        12 bytes
Requests per second:    1048.18 [#/sec] (mean)
 
 
Rails template test:
 
Document Path:          /hello/hellos
Document Length:        937 bytes
Requests per second:    949.54 [#/sec] (mean)
 
 
Database query test:
 
Document Path:          /hello/hellodb
Document Length:        2482 bytes
Requests per second:    734.18 [#/sec] (mean)

(Note: I think my version of Ruby was using a newer version of the SQLite3 library than my version of Python. This may or may not have given Rails an advantage on the database query test.)

For those of you who like charts, here’s a comparison of the “return hello world” test results for each framework:

The return hello world test chart

And here’s a comparison of the template test:

The template test chart

Finally, here’s the template test with the database query added:

Template test with database query

Closing thoughts

• Gone are the days of Rails as “the one framework to rule them all.” C’mon guys, it’s not 2005 anymore.
• Python has gotten a lot of flak from other language camps for its abundance of web frameworks. While some Pythonists have been embarrassed by this, I only see it as proof of Python’s success as a language. My retort would be: “If it was as easy to write a web framework in your language as it is in Python, you’d have the same problem!”

Coming soon: “Round 4: Micro-frameworks — Quick & Dirty, and leaving PHP in the dust!”

(Note: Please don’t leave benchmarking requests as comments. If you’d like me to take a look at your favorite framework, direct message me on twitter. Thanks!)

In Round 1, PHP was looking like quite the tortoise of the group. However, if you’re familiar with some of the core differences between Python & PHP, you’ll know that Python has been “cheating” slightly.

Let me explain: By default, Python compiles each script into bytecode on its first execution, allowing this bottleneck to be skipped on subsequent runs. PHP, however does not perform this type of optimization by default (in the 5.x line at least), so the PHP interpreter must re-compile each file every time it is run. As you can imagine, this can give PHP (without an accelerator) a huge disadvantage when compared to languages such as Python.

With this in mind, I have decided to take Round 2 to focus solely PHP. This will hopefully provide a clear picture of the benefits of PHP bytecode caching (at least when it comes to page-views — the memory benefits are a whole other story), and give you an idea of PHP’s performance with the help of an accelerator.

There are many PHP accelerators available, but I have chosen APC for use here (mostly due to its inclusion in the upcoming PHP 6 core).

What you should know about Round 2:

1. The hardware/software platform is the same as in Round 1.
2. PHP has been upgraded to v5.2.9.
3. The APC tests are using APC v3.0.19.
4. ApacheBench was run with -n # -c 10, with -n usually being 5000 or 10000, depending on the test subject’s speed.
5. Each test was run several times to make sure that there were no anomalies, with the “optimum average” chosen as the numbers represented here.
6. In an attempt to try to make these tests more scientific, I am now including the source code for each test (minus the CMS tests, which were performed on a standard install).

My apc.ini settings:

extension = apc.so
apc.enabled = 1
apc.shm_size = 96
apc.include_once_override = 1
apc.mmap_file_mask = /tmp/apc.XXXXXX

Raw PHP – php_test_src.zip

To investigate the speed of “raw PHP”, I created a couple of minimal PHP scripts that printed out a simple “hello world”, and read a few lines of text from a database. Take a look at the test source code for reference.

APC Disabled

The 'hello world' test:
 
Document Path:          /hello.php
Document Length:        12 bytes
Requests per second:    4199.43 [#/sec] (mean)
 
 
The simple database query test:
 
Document Path:          /hello_db.php
Document Length:        2524 bytes
Requests per second:    2797.54 [#/sec] (mean)

APC Enabled

The 'hello world' test:
 
Document Path:          /hello.php
Document Length:        12 bytes
Requests per second:    6618.42 [#/sec] (mean)
 
 
Simple database query test:
 
Document Path:          /hello_db.php
Document Length:        2524 bytes
Requests per second:    3713.99 [#/sec] (mean)

Right out of the gate, we can see that APC gives PHP a significant boost.

Of course, a script that simply does a “echo ‘hello world’” isn’t really practical, so let’s take a look at a couple of popular PHP frameworks to see what a full stack looks like.

CakePHP v1.2.5 – cakephp_test_src.zip

CakePHP was the slowest test subject by a large margin in Round 1, but if you take a look at the caching section of its core configuration file you can see that the developers have written it with acceleration in mind. It includes special configuration directives for not just APC, but also Xcache and Memcache.

Still though, without acceleration CakePHP is a sad story indeed:

APC Disabled – File Caching Engine (default)

The return 'hello world' test:
 
Document Path:          /hellos/
Document Length:        13 bytes
Requests per second:    45.25 [#/sec] (mean)
 
 
The template hello world test:
 
Document Path:          /hellos/hello
Document Length:        931 bytes
Requests per second:    39.90 [#/sec] (mean)
 
 
The simple database query test (using the ORM):
 
Document Path:          /hellos/hello_db
Document Length:        2469 bytes
Requests per second:    38.80 [#/sec] (mean)

However, with the help of APC things finally begin to look reasonable:

APC Enabled – APC Caching Engine (in core.php)

The return 'hello world' test:
 
Document Path:          /hellos/
Document Length:        13 bytes
Requests per second:    140.98 [#/sec] (mean)
 
 
The template hello world test:
 
Document Path:          /hellos/hello
Document Length:        931 bytes
Requests per second:    120.04 [#/sec] (mean)
 
 
The simple database query test (using the ORM):
 
Document Path:          /hellos/hello_db
Document Length:        2469 bytes
Requests per second:    117.56 [#/sec] (mean)

As you can see, APC increases CakePHP’s speed about threefold in these examples. These results may or may not be acceptable to you (remember that we are running these tests with a concurrency of 10), but I felt much better about CakePHP after pairing it with APC.

Kohana v3.0 – kohana3_test_src.zip

Kohana (a divergent of CodeIgnitor) is a much more lightweight PHP framework than CakePHP, but still gives you a pretty full featureset. Since v3.0 was released shortly after Round 1, I decided to use the new version for the tests here in Round 2.

It should be noted that although file caching is enabled in a default CakePHP install, Kohana’s caching is disabled by default. I have included a couple of tests with caching disabled here to exhibit the performance of a default Kohana install.

APC Disabled – Caching Disabled (default)

The return 'hello world' test:
 
Document Path:          /hello/index
Document Length:        12 bytes
Requests per second:    86.73 [#/sec] (mean)

APC Disabled – Caching Enabled (bootstrap.php)

The return 'hello world' test:
 
Document Path:          /hello/index
Document Length:        12 bytes
Requests per second:    180.50 [#/sec] (mean)
 
 
The template hello world test:
 
Document Path:          /hellos/index
Document Length:        930 bytes
Requests per second:    162.27 [#/sec] (mean)
 
 
The simple database query test (using the Database module):
 
Document Path:          /hellodb/index
Document Length:        2525 bytes
Requests per second:    113.92 [#/sec] (mean)
 
 
The simple database query test (using the ORM module):
 
Document Path:          /helloorm/index
Document Length:        2525 bytes
Requests per second:    99.73 [#/sec] (mean)

As you can see, even without APC Kohana is quite a bit faster than CakePHP.

APC Enabled – Caching Disabled

The return 'hello world' test:
 
Document Path:          /hello/index
Document Length:        12 bytes
Requests per second:    125.93 [#/sec] (mean)

APC Enabled – Caching Enabled (bootstrap.php)

The return 'hello world' test:
 
Document Path:          /hello/index
Document Length:        12 bytes
Requests per second:    280.67 [#/sec] (mean)
 
 
 
The template hello world test:
 
Document Path:          /hellos/index
Document Length:        930 bytes
Requests per second:    253.50 [#/sec] (mean)
 
 
The simple database query test (using the Database module):
 
Document Path:          /hellodb/index
Document Length:        2525 bytes
Requests per second:    173.28 [#/sec] (mean)
 
 
The simple database query test (using the ORM module):
 
Document Path:          /helloorm/index
Document Length:        2525 bytes
Requests per second:    153.90 [#/sec] (mean)

Although not a threefold gain as with CakePHP, Kohana with APC+caching definitely makes a good attempt at shortening the gap seen in Round 1 between the PHP frameworks and the offerings from other languages. Of course, if raw speed is what you’re looking for, there are faster PHP frameworks available (note that the benchmarks on that page were run on a completely different setup and should not be compared with any of the numbers here).

Next, let’s take a look at how APC enhances the performance of a few of PHP’s heavyweights.

WordPress v2.8.4 (default install)

Although I’ve heard that WordPress has a built-in caching setting, a quick glance at wp-settings.php did not reveal anything. So, there are simply two WordPress tests—with APC, and without:

APC Disabled

Document Path:          /
Document Length:        5519 bytes
Requests per second:    24.80 [#/sec] (mean)

APC Enabled

Document Path:          /
Document Length:        5519 bytes
Requests per second:    74.00 [#/sec] (mean)

Joomla v1.5.14 (default install, without sample data)

The results of these tests were quite disappointing to me, and made me wonder if I was doing something wrong. Sadly, it looks as though Joomla is quite the slow one of the bunch.

APC Disabled

Joomla Caching Disabled (default):
 
Document Path:          /
Document Length:        4006 bytes
Requests per second:    27.90 [#/sec] (mean)
 
 
Joomla Caching Enabled (File):
 
Document Path:          /
Document Length:        4006 bytes
Requests per second:    31.46 [#/sec] (mean)

APC Enabled

Joomla Caching Disabled:
 
Document Path:          /
Document Length:        4006 bytes
Requests per second:    41.27 [#/sec] (mean)
 
 
Joomla Caching Enabled (APC):
 
Document Path:          /
Document Length:        4006 bytes
Requests per second:    45.01 [#/sec] (mean)

Drupal v6.14 (default install, clean URLs disabled)

Drupal has several caching settings, so I decided to see what each one would look like and how much of a difference “aggressive” caching made.

APC Disabled

Drupal Caching Disabled (default):
 
Document Path:          /
Document Length:        5548 bytes
Requests per second:    36.72 [#/sec] (mean)
 
 
"Normal" Drupal Caching:
 
Document Path:          /
Document Length:        5548 bytes
Requests per second:    289.69 [#/sec] (mean)
 
 
"Aggressive" Drupal Caching:
 
Document Path:          /
Document Length:        5548 bytes
Requests per second:    335.68 [#/sec] (mean)

APC Enabled

Drupal Caching Disabled:
 
Document Path:          /
Document Length:        5548 bytes
Requests per second:    83.93 [#/sec] (mean)
 
 
"Normal" Drupal Caching:
 
Document Path:          /
Document Length:        5548 bytes
Requests per second:    498.37 [#/sec] (mean)
 
 
"Aggressive" Drupal Caching:
 
Document Path:          /
Document Length:        5548 bytes
Requests per second:    564.48 [#/sec] (mean)

As you can see, Drupal’s caching (especially when combined with APC) changes things drastically. These results seem to indicate that Drupal’s caching does a great job of allowing the CMS to get out of the way.

Closing thoughts

• The benefits of a PHP accelerator are really too good to pass up. Even without written-in caching, an accelerator can give your code a significant boost.
• While these tests have only really explored the impact on website page-load times, a PHP accelerator typically also cuts down on the amount of memory used by your PHP scripts.
• PHP 6 has promised to fix a lot of things, including the lack of built-in bytecode caching. It will be interesting to revisit these tests in a few years (?) to see how much optimization the new version brings to the table.

Can’t get enough of this madness? Just wait until you read Round 3!

(Note: Please don’t leave benchmarking requests as comments. If you’d like me to take a look at your favorite framework, direct message me on twitter. Thanks!)

Recently I went on a benchmarking spree and decided to throw ApacheBench at a bunch of the different web development technology platforms I interact with on a day-to-day basis. The results were interesting enough to me that I decided I’d take a post to share them here.

Disclaimer: The following test results should be taken with a massive grain of salt. If you know anything about benchmarking, you will know that the slightest adjustments have the potential to change things drastically. While I have tried to perform each test as fairly and accurately as possible, it would be foolish to consider these results as scientific in any way. It should also be noted that my goal here was not to see how fast each technology performs at its most optimized configuration, but rather what a minimal out-of-the-box experience looks like.

Test platform info:

• The hardware was an Intel Core2Quad Q9300, 2.5Ghz, 6MB Cache, 1333FSB, 2GB DDR RAM.
• The OS was CentOS v5.3 32-bit with a standard Apache Webserver setup.
• ApacheBench was used with only the -n and -c flags (1000 requests for the PHP frameworks, 5000 requests for everything else).
• Each ApacheBench test was run 5-10 times, with the “optimum average” chosen as the numbers represented here.
• The PHP tests were done using the standard Apache PHP module.
• The mod_wsgi tests were done in daemon mode set to 2 processes/15 threads.
• The SQL tests were done with mysqli ($mysql->query()) on PHP, and SQLAlchemy (conn.execute()) on Python fetching and printing 5 rows of data from a sample database.

Apache v2.2.3

We will start with the raw Apache benchmark.

For this test, Apache loaded a simple HTML file with random text:

Document Length:        6537 bytes
Requests per second:    8356.23 [#/sec] (mean)

As expected, Apache is lightning fast.

Ok, so now that we’ve set the high water mark, let’s take a look at some popular web technology platforms…

PHP v5.2.8

For this test, I created a PHP file that simply printed the $_SERVER array and some random text:

Document Length:        6135 bytes
Requests per second:    2812.22 [#/sec] (mean)

Next, I decided to add some mysqli() code to connect to a database and print out a few rows of data:

Document Length:        283 bytes
Requests per second:    2135.46 [#/sec] (mean)

Not at all bad, but who really wants to build a webapp using plain old PHP anymore?

Next I decided to test the two PHP frameworks that I am most familiar with.

CakePHP v1.2.5

For this test, I did a basic CakePHP install, and added a route that printed “hello world” from a template:

Document Length:        944 bytes
Requests per second:    42.00 [#/sec] (mean)

As you can see, frameworks can be a blessing and a curse. On a side note, CakePHP was the only test that gave me failed requests.

Kohana v2.3.4

For this test, I did a basic install and loaded Kohana’s default welcome/index setup (which seemed lightweight enough to me to qualify as the test focal point):

Document Length:        1987 bytes
Requests per second:    100.11 [#/sec] (mean)

Kohana fared much better than CakePHP, but it appears as though PHP’s speed breaks down quickly once you start building a framework around it.

Python v2.5.4

Unlike PHP, Python wasn’t specifically created for the web. However, several years ago the WSGI spec was introduced which makes building web applications with Python a breeze. There are many different ways to depoly a WSGI app, but my tests were limited to Python’s built-in WSGIServer, and Apache’s mod_wsgi.

WSGIServer

For this test, I used WSGI.org’s test.wsgi:

Document Length:        4123 bytes
Requests per second:    1658.35 [#/sec] (mean)

These results were quite impressive to me, considering that this is pure Python at work.

mod_wsgi

mod_wsgi is Apache’s module to process WSGI scripts. This test used the exact same script as the previous WSGIServer example:

Document Length:        9552 bytes
Requests per second:    3994.30 [#/sec] (mean)

Don’t look now, but those results are better than the vanilla PHP test!

Next, I added SQLAlchemy imports to the test file and did a conn.execute(), in an attempt to try to mimic the PHP mysqli() test:

Document Length:        10212 bytes
Requests per second:    3379.87 [#/sec] (mean)

I expected to lose quite a bit of speed with the introduction of an ORM, but it looks like SQLAlchemy is quite fast. These results were still over 1,000reqs/s more than the PHP example.

TornadoServer

There’s been a lot of buzz lately about the new TornadoServer, so I thought I’d throw it in the mix. This test uses their basic “hello world” example:

Document Length:        12 bytes
Requests per second:    3330.00 [#/sec] (mean)

Nothing shocking here at first glance, but the Tornado is built to handle extreme loads. So, I increased the concurrency to 1,000 and threw 10,000 requests at it:

Document Length:        12 bytes
Concurrency Level:      1000
COMPLETE requests:      10000
Requests per second:    3288.15 [#/sec] (mean)

Almost exactly the same results! On a side note, none of the other tests subjects were able to handle a concurrency of 1,000.

TurboGears v2.0.3 (mod_wsgi)

TurboGears has quickly become my framework of choice, so naturally it was the first Python framework I tested.

For this test I setup a default quickstart project, and then added a hello() controller method which simply returned a “hello world” string:

Document Length:        11 bytes
Requests per second:    470.18 [#/sec] (mean)

Next, I added a simple Genshi template with 1 level of inheritance:

Document Length:        1351 bytes
Requests per second:    143.26 [#/sec] (mean)

While those results are not terrible, they should encourage you to check out the other templating options.

Next, I performed the same test with Mako:

Document Length:        1534 bytes
Requests per second:    394.41 [#/sec] (mean)

As you can see, Mako is significantly faster than Genshi.

TurboGears also supports Jinja2, so I thought I’d also see how it held up:

Document Length:        1020 bytes
Requests per second:    401.04 [#/sec] (mean)

I was not expecting Jinja2 to be faster than Mako, so I am not entirely sure about these results. However, the difference is only slight, and might have something to do with the fact that the dotted template names finder is disabled for Jinja templates.

Pylons v0.9.7 (mod_wsgi)

Mark Ramm (the current development lead of TurboGears) has been quoted as saying “TurboGears is to Pylons as Ubuntu is to Debian.” With that in mind, I naturally wanted to see how much speed I was losing with TurboGears’ “additions”.

For this test I tried to mimic the first TurboGears test (basic layout, returning a string):

Document Length:        11 bytes
Requests per second:    1961.00 [#/sec] (mean)

While I was expecting a gain from TurboGears’ omissions, I wasn’t quite expecting over four times the increase. I can only assume that the bare Pylons setup is pretty minimal, while TurboGears bootstraps a lot of stuff on for you by default.

Duplicating the Mako test in Pylons was also quite fast:

Document Length:        9743 bytes
Requests per second:    1183.39 [#/sec] (mean)

Bottle v0.5.7 (mod_wsgi)

Bottle caught my attention recently as a minimalistic Python web framework, so I thought I’d put it through the grind as well.

For this test, I used their simple “hello world” example:

Document Length:        12 bytes
Requests per second:    3912.43 [#/sec] (mean)

As you can see, Bottle is very close to bare WSGI as far as speed goes. On a side note, adding a sample template using Bottle’s default templating package didn’t seem to change these numbers at all.

Next I tested Bottle with its templating system switched to Mako:

Document Length:        41 bytes
Requests per second:    3134.61 [#/sec] (mean)

Then I added a SQLAlchemy and performed a query:

Document Length:        9374 bytes
Requests per second:    2503.27 [#/sec] (mean)

What was exciting to me about this test was that I found myself looking at a minimal Python framework equipped with a full templating system and ORM which all performed faster than the PHP+mysqli() test.

Django v1.1 (mod_wsgi)

I don’t use Django, but I thought it would be worth including in this report. This is a basic Django setup printing out a simple hello world template:

Document Length:        1471 bytes
Requests per second:    564.41 [#/sec] (mean)

As expected, a minimal Django setup is quite fast. (Note: This test included a SQLite database query.)

Ruby On Rails v2.2.3 (mod_passenger)

I also don’t use Rails, but I figured this report would be incomplete without Rails on the list. So, I took MediaTemple’s Rails test app and loaded up a test page:

Document Length:        1291 bytes
Requests per second:    433.57 [#/sec] (mean)

Note that the test app used here is doing a little bit more than just returning “Hello World”.

My Conclusions

As I said in the disclaimer, to arrive at any sort of decisive conclusions based on these numbers alone would be foolish. However, here is what all of this left me thinking:

• All of these technologies performed adequately for use with most projects.
• CakePHP was the only test that produced less than 100 requests per second. However, there are many articles available on optimizing CakePHP.
• These results made me wish I had moved away from PHP much earlier in my web development carrier. There are just better options these days when it comes to choosing a web development language (imho). If you must use PHP for a large project, I strongly suggest that you look into an accelerator.
• The Bottle+Mako+SQLAlchemy results struck me as an extremely minimal drop-in replacement for simple PHP sites.
• mod_wsgi is a huge winner in deploying Python webapps. Way to go Graham!
• Do your own bencharking, because your mileage will vary!

Like what you’ve seen here? Then why not check out Round 2?

(Note: Please don’t leave benchmarking requests as comments. If you’d like me to take a look at your favorite framework, direct message me on twitter. Thanks!)

In an effort to help you understand where I’m coming from, let me (briefly) rehash some of my programming history: I spent much of the 90’s doing dynamic web development using Perl (weren’t those the days!). I eventually migrated to PHP which usually made things much easier on the web; and subsequently replaced most of my console scripting with BASH [shell scripting]. However, I’m kind of a hack and love languages so I have occasionally been known to write something in C; and although I’m not a complete stranger to Java and Ruby, I never really felt like I “clicked” with either of those languages.

Ok, now that I’ve hopefully convinced you that I’m not just a fly-by-night programmer, let me show you some Python code. Brace yourself, as this article is bound to get lengthy…

Reason #1: “Whitespace done right” is actually a good thing

The first thing that people either absolutely love or adamantly hate about Python is the fact that code blocks are heavily tied to proper usage of whitespace. At first glance, this causes many curious onlookers from other languages to shy away from Python and continue in their bracket-encapsulated bondage. I’ll admit, at first I wasn’t too wild about these new restrictions either (I’m a programmer—I should be able to do my code how I want!); but the first time I had to go back to doing something in PHP for a client, I quickly realized I didn’t ever want to put anything inside brackets again:

# A sample function that's free from brackets, parenthesis, and semicolons!
def check_number(x):
    if type(x) != type(0):
        return "That isn't a number!"
    elif x % 2:
        return "This number is odd"
    else:
        return "This number is even"

Not only does this make Python source code incredibly readable, but you also end up typing less!

Reason #2: I almost always know what type of object I’m working with

One thing that always annoyed me about Perl was that you set an array or hash using the ‘@’ or ‘%’ sigils, but typically accessed them using the ‘$’ sigil. PHP simplified this a bit, but “$array = array();” could be a simple list or an associative array (known as a dictionary in Python). With Python on the other hand, the enclosure defines the type (for most built-in types at least), and since everything is an object there’s no need for silly sigils:

# the type of enclosure makes a distinction between a string , a list, and a dictionary
string = 'This is a string'
print string # prints 'This is a string'
 
list = ['This', 'is', 'a', 'list']
print list[0] # prints 'This'
 
dict = {'first_key': 'This', 'second_key': 'is', 'third_key': 'a', 'fourth_key': 'dictionary'}
print dict['fourth_key'] # prints 'dictionary'

On a somewhat related note, you can use these objects on-the-fly in your scripts without even having to assign them to a name. This isn’t usually practical of course (and may not be recommended), but it is possible:

print ['This', 'is', 'a', 'list'][1] # prints 'is'
print {'first_key': 'This', 'second_key': 'is', 'third_key': 'a', 'fourth_key': 'dictionary'}['first_key'] # prints 'This'

Reason #3: I can throw practically any type of object almost anywhere I want

Ok, so that might be a slight exaggeration; but the truth is that since Python treats everything as an object, you’re free to pack things away in the strangest places. A great way of demonstrating the power and flexibility of Python is in an example showing a few ways different types of objects can be stored and accessed:

# create a function
def hello_function():
    return 'Hello world!'
 
# create a file handle
hello_file = open('~/hello.txt', 'r')
 
# create a list with the function, the file handle, and a string
list = [hello_function, hello_file, 'Hello world!']
print list[0]() # prints 'Hello world!'
print list[1].read() # prints 'Hello world!\n'
print list[2] # prints 'Hello world!'
 
# or how about a dictionary?
dict = {'function': hello_function, 'file_object': hello_file, 'string': 'Hello world!'}
print dict['function']() # prints 'Hello world!'
print dict['file_object'].read() # prints 'Hello world!\n'
print dict['string'] # prints 'Hello world!'
 
# print types
print type(list[0]) # prints <type 'function'>
print type(dict['file_object']) # prints <type 'file'>
print type(list[2]) # prints <type 'str'> (str for string)

The shelve and pickle modules are also work a look if you like what you’ve just seen here.

Reason #4: The “in” operator simplifies sequences and makes membership testing a breeze

To explain the power of “in”, it’s probably easier to just show you a few examples:

'p' in "spam" # == True
'item1' in ['item1', 'item2'] # == True
'key' in {"key": 'value'} # == True
 
# prints anchor tags for a navigation
for title, link in {'Home': 'index.html', 'About': 'about.html', 'Contact': 'contact.html'}.iteritems():
    print '<a href="%s">%s</a>' % (link, title)

Reason #5: Slicing, stepping, and striding are a walk in the park

Probably my most-hated function in PHP is the substr() function. It should just be easier to access a parts of a string. Thankfully, in Python it’s a piece of cake:

#string slices and stepping/striding
string = 'This is a string'
 
print string[3] # prints 's'
print string[5:9] # prints 'is a'
print string[::2] # prints 'Ti sasrn' (a stride of 2 means print every other item)
 
# works on lists and dictionaries as well
list = ['This', 'is', 'a', 'list']
 
print list[::2] # prints ['This', 'a']

Reason #6-XX: Python makes programming fun again!

I could go on and on, but this post would be endless. As you can see, Python is a flexible, powerful, and fun programming language. For more Python concepts (I didn’t have time here to go into list comprehensions, generators, decorators, etc), I’d suggest you take a gander at Wikipedia’s page on Python syntax and semantics.

Finally, I’ll leave you with a few relevant links that are worth taking the time to check out:

• Bruce Eckel’s slides: Why I Love Python
• Python’s official tutorial (by Guido—Python’s creator)
• Dive Into Python – Free online book
• xkcd’s famous Python comic

Let me start by first of all saying: Yes, it’s true. Distributed Version Control Systems really are everything that Centralized Version Control Systems (read: SVN/CVS) fall short on. However, having been a long-time Subversion user myself, as well as the fact that I hardly ever work on a team with more than a couple of other developers, it took a lot of noise to make me even care about the differences between DVCS and CVCS. So what is the difference you might ask? Almost everything; and that’s a good thing! (ok, that’s a slight exaggeration, but the benefits are so profound it almost feels that way)

Don’t worry, I’ll spare you the repetition of what has already been said elsewhere. My story is this: I spent about a month looking at the various DVCS’ and eventually narrowed it down to Git vs Mercurial (why anyone would use Bazaar over Mercurial is beyond me). Since I’m a Linux user, and since Github impressed me, I finally decided to go with Git on the next project that came my way. Boy oh boy do I wish I had run across this article beforehand…

Git: MacGyver

As the referenced article implies, Git does anything and everything you’d ever need it to do. Now, as a Linux guy I typically like the options I’m given with the “everything plus the kitchen sink” approach. However, I quickly found myself using 2 or 3 commands to do something I swear everyone must do and really should be only one command. Then there’s all the command options you have to throw in to make the command do exactly what you want it to do. This quickly became tiresome and once that project was over I decided to give Mercurial a try, hoping it would be much kinder to me.

Mercurial: Bond

The first thing I realized when using Mercurial on a “real” project was that it seemed to think the same way I did (an interesting trend I’ve noticed with Python programs). Whereas with Git I found myself typing “git help <command>” over and over again, I hardly ever needed to look up any special options with the Mercurial commands. Another thing I soon realized was that a lot of the negative press that I’d read on Mercurial was actually written up about the 1.0 version (or earlier). I was pleased to find that much of these “issues” no longer existed in my 1.2.x version of the tool.

I now run almost all of my new projects on Mercurial. I say “almost all” because there are a few cases where the project is small and a tool I’m using in the project already has a Git repository so it makes sense to keep everything streamlined. However, with new tools like hg-git, even that may become a thing of the past.

In conclusion, I strongly encourage you to check out Mercurial if you’re jumping in to the DVCS game. Unless you’re a huge project like the Linux kernel, you probably won’t ever need the “advanced functionality” that Git puts at your fingertips (which is probably in most cases also available in Mercurial, just not right in your face). If you’re still using SVN (or even, *gasp*, CVS), then why not get with the times? Mercurial has some amazing conversion options that will retain your history and make the switch a piece of cake. Sure, there are a few new concepts to learn (pull/push and the distributed nature), but it’s only a minor learning curve that will pay off immensely.

Links of interest:

• Mercurial
• Bitbucket – Mercurial’s version of Github
• WikiPedia’s article on DVCS
• Intro to Distributed Versioning Control – An excellent read
• “While git has many fans, it also provokes strong antipathies.” – Guido van Rossum
• “Subversion is the most pointless project ever started.” – Linus Torvalds

I’ve been playing around with Mozilla’s latest release, and I’ve got to say: Firefox is where it’s at! If anyone out there is still using Internet Explorer, you deserve anything that comes to you by way of malware, adware, spam, viruses, etc. Seriously, get with the times and install a real browser! And to you Apple/Mac users out there: Yeah, Safari is nice, but don’t let pride keep you from branching out of familiar territory.

One of the great thing about Firefox is its add-ons, and I’ve decided it’s time to dedicate a post to a few of my personal favorites:

Adblock Plus

To me, Adblock Plus is THE most incredible thing about Firefox (and its #1 selling point). This add-on actually strips about 99% of “web spam” and annoying banner ads from websites *before* they load into Firefox.

Note: Some web sites are financed soley by revenue from visitors loading ads on their sites. Keep in mind that stripping out ads may in some cases also affect the income generated by your favorite blog/social site.

Flashblock

Similar to Adblock, Flashblock turns flash elements into “play buttons” instead of auto-playing all those crazy flash items on pages.

Firebug

To me, the 2nd most important thing about Firefox. As a web developer I’m always having to dissect a page’s DOM, or do extra debugging on a JavaScript function. Firebug gives me complete access, control and information about a web page.

Tamper Data

A more technical add-on, Tamper Data allows you to view and modify request headers and form submits (as well as any other communication between the client & server). Very geeky, but very powerful.

FireFTP

FireFTP turns Firefox into a full-featured FTP client. A must, especially if you’re a Windows user (mounting FTP sites in Windows Explorer is still weird)

Honorable mention: SQLite Manager, ChatZilla, FoxyProxy, Morning Coffee, YSlow, User Agent Switcher

Give these a whirl… you’ll be glad you did!

- Seth