The Great Web Framework Shootout

Why a web framework shootout?

In a landscape filled with a vast array of web frameworks whose version numbers tend to increase rapidly, I'm often curious about how the different choices and versions compare to each other when submitted to a few simple performance benchmarks.

In September of 2009, I decided to share some of my findings in a blog post which eventually drew enough attention to cause me to decide to turn it into a series of rounds. After posting Round 4, it became apparent to me that it was probably time to combine the various benchmark results into their own page in order to give this topic the dedication (and ease of updating) that it deserves.

Do these results have any real world value?

Probably not. When it comes to code, the slightest adjustments have the potential to change things drastically. While I have tried to perform each test as fairly and accurately as possible, it would be foolish to consider these results as scientific in any way. It should also be noted that my goal here was not necessarily to figure out how fast each framework could perform at its most optimized configuration (although built-in caching and other performance tweaks were usually enabled if the default configuration permitted it), but rather to see what a minimal "out-of-the-box" experience would look like.

Additionally, nothing here is intended to make one web technology appear "better" than another. When it comes to using the right tool for the job, "faster" does not necessarily mean "better" (very few real world projects are going to depend solely on page request speeds).

Results Index

• Full stack(ish): Pyramid, Django, TurboGears, and Rails
• Microframeworks: Bottle, Flask, Sinatra, and web.go
• The PHP frameworks: CakePHP, Kohana, CodeIgniter, Symfony, and Yii
• All frameworks side-by-side.
• Test setup information and test app source code.

Full stack(ish): Pyramid, Django, TurboGears, and Rails

Note: The Pyramid/TG test apps used SQLAlchemy as the ORM and Jinja2 as the templating system.

Microframeworks: Bottle, Flask, Sinatra, and web.go

Note: The web.go tests were run using its own built-in server. Also, my C is extremely rusty and I'm only casually familiar with Go, so if somebody could add a SQLite test to the web.go test app I'd be extremely grateful.

The PHP frameworks: CakePHP, Kohana, CodeIgniter, Symfony, and Yii

All frameworks side-by-side

Remember: Comparing all of these framework tests side-by-side isn't really "fair" because they are all so different. Compiled languages (e.g. Go) are expected to be faster than scripted languages. Tests using an ORM (e.g. Rails, Django, Pyramid, etc.) are expected to be slower than tests using only a plain database library (e.g. Bottle, Flask, Sinatra, etc).

What kind of test setup are you using?

I find Amazon's EC2 fairly easy to work with and decided to go with a vanilla Ubuntu Server setup:

• The EC2 instance used was: ami-fbbf7892 m1.large ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20110719.manifest.xml
• As a "Large" instance, Amazon describes the resources as: 7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB of local instance storage, 64-bit platform.
• Apache 2.2.14 was used. (Yes, I know there are other options, but with Apache's market share I figured it would be a good baseline.)
• Python 2.6.5 and mod_wsgi 2.8 (embedded mode) were used for the Python based tests.
• Ruby 1.9.2p290 and Phusion Passenger 3.0.9 were used for the Ruby based tests (Ruby 1.8.x is slow).
• PHP 5.3.2 (with APC enabled) was used for the PHP based tests.
• ApacheBench was run locally with -n 10000 and -c 10 about 5-10 times each, and the "best guess average" was chosen.

The three basic tests that I set each framework up to run are:

1. The "Hello World" test: This test simply spits out a string response. There's no template or DB calls involved, so the level of processing should be minimal.
2. The template test: This test prints out Lorem Ipsum via a template (thus engaging the framework's templating systems).
3. The template/db test: This test loads 5 rows of Lorem Ipsum from a SQLite DB (via the default ORM or a sqlite3 driver) and then prints them out through a template (thus engaging both the framework's ORM/DB driver and the templating system).

Will you please add XYZ to the results?

Maybe, if you can convince me that enough people would be interested in having it displayed next to heavyweights like Rails and Django. Fork the repository on GitHub and submit a pull request under the "dev" branch with a test app in the same format as the other tests, and make sure you include your best sales pitch. Otherwise, I'd suggest you boot up the EC2 AMI and do your own benchmarking.

Closing thoughts and future updates

I will do my best to keep this page updated as new framework releases become available, but I will probably wait until a few different frameworks have major releases rather than updating for every single release. Follow me on Google+ or watch the GitHub repository (which might see updates more quickly than this page) to stay informed.

Sorry guys, but I decided to disable comments on this post. Apparently people are just looking at the graphs and not reading the disclaimers and other info. Contact me on Google+ if you have any questions or comments that still aren't resolved after reading the full content above.