Curia

The great web technology shootout – Round 1: A quick glance at the landscape

by on Sep.12, 2009, under F/OSS, Web Development

[A lot of the information below is out of date. Please see the new framework shootout page for the latest benchmarks.]

Recently I went on a benchmarking spree and decided to throw ApacheBench at a bunch of the different web development technology platforms I interact with on a day-to-day basis. The results were interesting enough to me that I decided I’d take a post to share them here.

Disclaimer: The following test results should be taken with a massive grain of salt. If you know anything about benchmarking, you will know that the slightest adjustments have the potential to change things drastically. While I have tried to perform each test as fairly and accurately as possible, it would be foolish to consider these results as scientific in any way. It should also be noted that my goal here was not to see how fast each technology performs at its most optimized configuration, but rather what a minimal out-of-the-box experience looks like.

Test platform info:

  • The hardware was an Intel Core2Quad Q9300, 2.5Ghz, 6MB Cache, 1333FSB, 2GB DDR RAM.
  • The OS was CentOS v5.3 32-bit with a standard Apache Webserver setup.
  • ApacheBench was used with only the -n and -c flags (1000 requests for the PHP frameworks, 5000 requests for everything else).
  • Each ApacheBench test was run 5-10 times, with the “optimum average” chosen as the numbers represented here.
  • The PHP tests were done using the standard Apache PHP module.
  • The mod_wsgi tests were done in daemon mode set to 2 processes/15 threads.
  • The SQL tests were done with mysqli ($mysql->query()) on PHP, and SQLAlchemy (conn.execute()) on Python fetching and printing 5 rows of data from a sample database.

 

Apache v2.2.3

We will start with the raw Apache benchmark.

For this test, Apache loaded a simple HTML file with random text:

Document LENGTH:        6537 bytes
Requests per second:    8356.23 [#/sec] (mean)

As expected, Apache is lightning fast.

Ok, so now that we’ve set the high water mark, let’s take a look at some popular web technology platforms…

 

PHP v5.2.8

For this test, I created a PHP file that simply printed the $_SERVER array and some random text:

Document LENGTH:        6135 bytes
Requests per second:    2812.22 [#/sec] (mean)

Next, I decided to add some mysqli() code to connect to a database and print out a few rows of data:

Document LENGTH:        283 bytes
Requests per second:    2135.46 [#/sec] (mean)

Not at all bad, but who really wants to build a webapp using plain old PHP anymore?

Next I decided to test the two PHP frameworks that I am most familiar with.

 

CakePHP v1.2.5

For this test, I did a basic CakePHP install, and added a route that printed “hello world” from a template:

Document LENGTH:        944 bytes
Requests per second:    42.00 [#/sec] (mean)

As you can see, frameworks can be a blessing and a curse. On a side note, CakePHP was the only test that gave me failed requests.

Kohana v2.3.4

For this test, I did a basic install and loaded Kohana’s default welcome/index setup (which seemed lightweight enough to me to qualify as the test focal point):

Document LENGTH:        1987 bytes
Requests per second:    100.11 [#/sec] (mean)

Kohana fared much better than CakePHP, but it appears as though PHP’s speed breaks down quickly once you start building a framework around it.

 

Python v2.5.4

Unlike PHP, Python wasn’t specifically created for the web. However, several years ago the WSGI spec was introduced which makes building web applications with Python a breeze. There are many different ways to depoly a WSGI app, but my tests were limited to Python’s built-in WSGIServer, and Apache’s mod_wsgi.

WSGIServer

For this test, I used WSGI.org’s test.wsgi:

Document LENGTH:        4123 bytes
Requests per second:    1658.35 [#/sec] (mean)

These results were quite impressive to me, considering that this is pure Python at work.

mod_wsgi

mod_wsgi is Apache’s module to process WSGI scripts. This test used the exact same script as the previous WSGIServer example:

Document LENGTH:        9552 bytes
Requests per second:    3994.30 [#/sec] (mean)

Don’t look now, but those results are better than the vanilla PHP test!

Next, I added SQLAlchemy imports to the test file and did a conn.execute(), in an attempt to try to mimic the PHP mysqli() test:

Document LENGTH:        10212 bytes
Requests per second:    3379.87 [#/sec] (mean)

I expected to lose quite a bit of speed with the introduction of an ORM, but it looks like SQLAlchemy is quite fast. These results were still over 1,000reqs/s more than the PHP example.

TornadoServer

There’s been a lot of buzz lately about the new TornadoServer, so I thought I’d throw it in the mix. This test uses their basic “hello world” example:

Document LENGTH:        12 bytes
Requests per second:    3330.00 [#/sec] (mean)

Nothing shocking here at first glance, but the Tornado is built to handle extreme loads. So, I increased the concurrency to 1,000 and threw 10,000 requests at it:

Document LENGTH:        12 bytes
Concurrency Level:      1000
COMPLETE requests:      10000
Requests per second:    3288.15 [#/sec] (mean)

Almost exactly the same results! On a side note, none of the other tests subjects were able to handle a concurrency of 1,000.

 

TurboGears v2.0.3 (mod_wsgi)

TurboGears has quickly become my framework of choice, so naturally it was the first Python framework I tested.

For this test I setup a default quickstart project, and then added a hello() controller method which simply returned a “hello world” string:

Document LENGTH:        11 bytes
Requests per second:    470.18 [#/sec] (mean)

Next, I added a simple Genshi template with 1 level of inheritance:

Document LENGTH:        1351 bytes
Requests per second:    143.26 [#/sec] (mean)

While those results are not terrible, they should encourage you to check out the other templating options.

Next, I performed the same test with Mako:

Document LENGTH:        1534 bytes
Requests per second:    394.41 [#/sec] (mean)

As you can see, Mako is significantly faster than Genshi.

TurboGears also supports Jinja2, so I thought I’d also see how it held up:

Document LENGTH:        1020 bytes
Requests per second:    401.04 [#/sec] (mean)

I was not expecting Jinja2 to be faster than Mako, so I am not entirely sure about these results. However, the difference is only slight, and might have something to do with the fact that the dotted template names finder is disabled for Jinja templates.

Pylons v0.9.7 (mod_wsgi)

Mark Ramm (the current development lead of TurboGears) has been quoted as saying “TurboGears is to Pylons as Ubuntu is to Debian.” With that in mind, I naturally wanted to see how much speed I was losing with TurboGears’ “additions”.

For this test I tried to mimic the first TurboGears test (basic layout, returning a string):

Document LENGTH:        11 bytes
Requests per second:    1961.00 [#/sec] (mean)

While I was expecting a gain from TurboGears’ omissions, I wasn’t quite expecting over four times the increase. I can only assume that the bare Pylons setup is pretty minimal, while TurboGears bootstraps a lot of stuff on for you by default.

Duplicating the Mako test in Pylons was also quite fast:

Document LENGTH:        9743 bytes
Requests per second:    1183.39 [#/sec] (mean)

Bottle v0.5.7 (mod_wsgi)

Bottle caught my attention recently as a minimalistic Python web framework, so I thought I’d put it through the grind as well.

For this test, I used their simple “hello world” example:

Document LENGTH:        12 bytes
Requests per second:    3912.43 [#/sec] (mean)

As you can see, Bottle is very close to bare WSGI as far as speed goes. On a side note, adding a sample template using Bottle’s default templating package didn’t seem to change these numbers at all.

Next I tested Bottle with its templating system switched to Mako:

Document LENGTH:        41 bytes
Requests per second:    3134.61 [#/sec] (mean)

Then I added a SQLAlchemy and performed a query:

Document LENGTH:        9374 bytes
Requests per second:    2503.27 [#/sec] (mean)

What was exciting to me about this test was that I found myself looking at a minimal Python framework equipped with a full templating system and ORM which all performed faster than the PHP+mysqli() test.

Django v1.1 (mod_wsgi)

I don’t use Django, but I thought it would be worth including in this report. This is a basic Django setup printing out a simple hello world template:

Document LENGTH:        1471 bytes
Requests per second:    564.41 [#/sec] (mean)

As expected, a minimal Django setup is quite fast. (Note: This test included a SQLite database query.)

 

Ruby On Rails v2.2.3 (mod_passenger)

I also don’t use Rails, but I figured this report would be incomplete without Rails on the list. So, I took MediaTemple’s Rails test app and loaded up a test page:

Document LENGTH:        1291 bytes
Requests per second:    433.57 [#/sec] (mean)

Note that the test app used here is doing a little bit more than just returning “Hello World”.

 

My Conclusions

As I said in the disclaimer, to arrive at any sort of decisive conclusions based on these numbers alone would be foolish. However, here is what all of this left me thinking:

  • All of these technologies performed adequately for use with most projects.
  • CakePHP was the only test that produced less than 100 requests per second. However, there are many articles available on optimizing CakePHP.
  • These results made me wish I had moved away from PHP much earlier in my web development carrier. There are just better options these days when it comes to choosing a web development language (imho). If you must use PHP for a large project, I strongly suggest that you look into an accelerator.
  • The Bottle+Mako+SQLAlchemy results struck me as an extremely minimal drop-in replacement for simple PHP sites.
  • mod_wsgi is a huge winner in deploying Python webapps. Way to go Graham!
  • Do your own bencharking, because your mileage will vary!

 

Like what you’ve seen here? Then why not check out Round 2?

(Note: Please don’t leave benchmarking requests as comments. If you’d like me to take a look at your favorite framework, direct message me on twitter. Thanks!)

:, , , , ,

  • http://www.chrisgratigny.com Chris Gratigny

    ouch.

    at least Kohana is much faster than Cake :)

  • http://blog.dscpl.com.au Graham Dumpleton

    Hooray, someone who has actually tried to provide a realistic comparison of Tornado rather than their biased comparison of a Tornado hello world application with heavy weight Django. Using a WSGI hello world application is a much more realistic comparison for what they were running on Tornado. You didn’t even use the fastest configuration for mod_wsgi and it still came out equivalent and nothing like their claim of Tornado being 4 times better.

    As far as scaling up of Tornado, it still remains to be seen how well that will work when you start using a proper blocking WSGI interface on top of Tornado and with longer request times. Sure you can write purpose built event driven web applications for Tornado which may perform better, but that doesn’t mean existing WSGI applications somehow benefit from Tornado being event driven in its core. Multiprocess event driven servers arguably may also be a really bad idea for WSGI applications due to the possible blocking effects.

  • Seth

    Graham,

    Thanks for your input. I’m planning on re-doing this in mod_wsgi embedded mode soon. My initial testing seems to indicate an increase of several hundred req/sec.

    You have really given the community a killer module. Thank you!

  • http://bottle.paws.de/ Marcel Hellkamp

    fapws3 is another asynchronous HTTP/WSGI web server, similar to Tornado. It uses libev instead of epoll and is definitely worth a look. http://github.com/william-os4y/fapws3

    Oh, and thanks for this great article. I hope you don’t mind if I quote you on Bottles start page ;)

  • Seth

    Marcel,

    Thanks for the info. I’ll have to take a look at it.

    Feel free to quote, although you may want to wait for “part 3″ (which I should have done next week). I’ll be doing a bit more “scientific” tests for each Python framework, and revisiting Bottle in a little bit more detail (with the test source code).

    Stay tuned ;)

  • Pingback: The great web technology shootout – Round 2: PHP deserves a helping hand - Curia

  • Pingback: The great web technology shootout – Round 4: Pyramid vs Django vs TG vs Rails 2 & 3 - Curia

  • sunkang

    php is very popular,that is true.python needs keep on

Looking for something?

Use the form below to search the site:

Still not finding what you're looking for? Drop a comment on a post or contact us so we can take care of it!

Visit our friends!

A few highly recommended friends...