The great web technology shootout – Round 3: Better, Faster, and Shinier
by Seth on Oct.05, 2009, under F/OSS, Web Development
[Note: This post is the continuation of a series. Please read Round 1 and Round 2 first if you are just now joining us.]
[Update 10/6: Slight updates to the Pylons test after some input from Ben. Added a couple new charts for you visual folks.]
As I briefly mentioned in Round 1, this whole thing came about as an experiment to satisfy my own curiosity. Unfortunately, I wasn’t expecting these posts to draw the amount of attention they have been getting, and several people informed me of a few “issues” with the first round. Since my initial approach to this topic was somewhat casual, I didn’t really take the time to perform each test in a “proper scientific fashion.” Although this was clearly stated in the introduction to round one, it unfortunately resulted in performance estimations that were somewhat less than accurate.
After input from various people much smarter than myself, I quickly went to work tweaking my test environment and building “proper” test apps. In the midst of this, a conversation about PHP accelerators prompted me to put PHP under the spotlight, which brought about Round 2 as an interim round. This gave me a chance to demonstrate the necessity of PHP acceleration, and only continued to solidify my opinion of PHP as an inferior web development language (remember, I just said my opinion).
Which brings us to Round 3. A lot of work has gone into “doing it right” this time, so I am fairly confident that these results are a much more accurate representation of each test subject’s performance estimations. Remember, benchmark test code typically has no real-world value, so “performance estimations” are about all I can promise here. Your mileage will vary. As a wise person once said:
“All this benchmarking is doing is proving what we already know: More code takes longer to execute.” – Ben Bangert (dev lead of Pylons)
What you should know about Round 3:
- The hardware/software platform is the same as in Round 1.
- Python v2.5.4 was used for all the Python tests. Ruby v1.8.6 was used for the Rails test.
- Apache v2.2.3 was used, with mod_wsgi v2.5 for the Python tests and Phusion Passenger v2.2.5 for the Rails test.
- Note that while the mod_wsgi tests in Round 1 were using “daemon mode”, the mod_wsgi tests here are using “embedded mode” which is often faster (although not necessarily the recommended configuration).
- The Python component versions used here were: SQLAlchemy v0.5.6, Genshi v0.5.1, Mako v0.2.5, and Jinja 2.2.1.
- ApacheBench was run with -n 10000 -c 10.
- Each test was run several times to make sure that there were no anomalies, with the “optimum average” chosen as the numbers represented here.
- In an attempt to try to make these tests more scientific, I am now including the source code for each test.
Plain WSGI – wsgi_test_src.zip
Let’s start with a basic “hello world” WSGI test to set the baseline:
The 'hello world' test: WSGIServer/0.1 Document Path: / Document Length: 12 bytes Requests per second: 1532.76 [#/sec] (mean) Apache/2.2.3 (mod_wsgi) Document Path: / Document Length: 12 bytes Requests per second: 5549.73 [#/sec] (mean)
As you can see, mod_wsgi allows Apache to put some serious muscle into Python’s WSGI. It’s no wonder that mod_wsgi is quickly becoming the de-facto standard for Apache Python deployment.
TurboGears v2.0.3 – tg2_test_src.zip
As I mentioned in Round 1, TurboGears has quickly become my web framework of choice. Building a “best-of-breed” component stack on top of Pylons may not be easy, but when it works I believe it pays off immensely (e.g. Ubuntu).
The 'hello world' test: Document Path: / Document Length: 12 bytes Requests per second: 935.13 [#/sec] (mean) Genshi template test: Document Path: /genshi_hello/ Document Length: 923 bytes Requests per second: 601.07 [#/sec] (mean) Mako template test: Document Path: /mako_hello/ Document Length: 932 bytes Requests per second: 723.18 [#/sec] (mean) Jinja2 template test: Document Path: /jinja_hello/ Document Length: 937 bytes Requests per second: 764.35 [#/sec] (mean) Database query tests: Document Path: /raw_sql/ Document Length: 1270 bytes Requests per second: 569.71 [#/sec] (mean) Document Path: /genshi_sql/ Document Length: 2409 bytes Requests per second: 388.96 [#/sec] (mean) Document Path: /mako_sql/ Document Length: 2418 bytes Requests per second: 515.07 [#/sec] (mean) Document Path: /jinja_sql/ Document Length: 2472 bytes Requests per second: 535.06 [#/sec] (mean)
TurboGears v2.1a1 – tg21_test_src.zip
The TurboGears community is alive and well, and just before finishing this post the release of v2.1 Alpha 1 was announced. Significant improvements were made to TG’s object dispatch, so I was excited to see what the new numbers would look like.
The 'hello world' test: Document Path: / Document Length: 11 bytes Requests per second: 1118.92 [#/sec] (mean) Genshi template test: Document Path: /genshi_hello/ Document Length: 923 bytes Requests per second: 713.97 [#/sec] (mean) Mako template test: Document Path: /mako_hello/ Document Length: 932 bytes Requests per second: 812.64 [#/sec] (mean) Jinja2 template test: Document Path: /jinja_hello/ Document Length: 937 bytes Requests per second: 1000.98 [#/sec] (mean) Database query tests: Document Path: /raw_sql/ Document Length: 1270 bytes Requests per second: 673.04 [#/sec] (mean) Document Path: /genshi_sql/ Document Length: 2409 bytes Requests per second: 433.45 [#/sec] (mean) Document Path: /mako_sql/ Document Length: 2418 bytes Requests per second: 601.89 [#/sec] (mean) Document Path: /jinja_sql/ Document Length: 2472 bytes Requests per second: 630.66 [#/sec] (mean)
As you can see, the future of TurboGears looks very bright, and I only expect these numbers to get better as the final 2.1 release approaches (expected in Q1 2010).
Pylons v0.9.7 – pylons_test_src.zip
Pylons is the foundation which TurboGears is built upon, and for those who don’t need all “the extras” it is an excellent choice.
The 'hello world' test: Document Path: /hello/index Document Length: 12 bytes Requests per second: 2593.47 [#/sec] (mean) Mako template test: Document Path: /hello/hello Document Length: 932 bytes Requests per second: 1737.03 [#/sec] (mean) Database query tests: Document Path: /hello/raw_sql Document Length: 1270 bytes Requests per second: 964.07 [#/sec] (mean) Document Path: /hello/hellodb Document Length: 2418 bytes Requests per second: 831.26 [#/sec] (mean)
Django v1.1 – django_test_src.zip
While Django is probably Python’s most popular web framework, I have never really clicked with it. Its way of doing things is significantly different than TG/Pylons, but if you don’t mind its “MTV” architectural pattern and can survive without SQLAlchemy (although there are hacks), it is a very capable framework.
The 'hello world' test: Document Path: /hello Document Length: 12 bytes Requests per second: 3376.48 [#/sec] (mean) Django template test: Document Path: /hellos Document Length: 936 bytes Requests per second: 1781.43 [#/sec] (mean) Database query test: Document Path: /hellodb Document Length: 2476 bytes Requests per second: 972.11 [#/sec] (mean)
Bottle 0.5.8 – bottle_test_src.zip
Bottle caught my attention a few months ago while surveying the WSGI landscape for an alternative to PHP when building websites that only require a few pieces of dynamic functionality. As a “micro-framework”, it does an excellent job at bridging the gap between “pure WSGI” and its “feature-rich” big brothers.
The 'hello world' test: Document Path: / Document Length: 12 bytes Requests per second: 5545.96 [#/sec] (mean) Mako template test: Document Path: /hello Document Length: 923 bytes Requests per second: 3588.04 [#/sec] (mean) Database query test (SQLAlchemy): Document Path: /hellodb Document Length: 2413 bytes Requests per second: 1479.54 [#/sec] (mean)
As you can see, there was practically no difference in speed between Bottle and pure WSGI in a basic “hello world” test. Even with the addition of Mako and SQLAlchemy, Bottle performed significantly faster than a bare Pylons or Django setup.
(Note: Mako template inheritance was not used in the Bottle Mako test.)
Ruby on Rails v2.3.3 – rails_test_src.zip
What web technology shootout would be complete without Rails? Although I am not a Rails fan (mostly because I can’t stand Ruby), I believe that in many ways we owe much of the popularity and success of the web framework movement to Rails.
The 'hello world' test: Document Path: /hello/hello Document Length: 12 bytes Requests per second: 1048.18 [#/sec] (mean) Rails template test: Document Path: /hello/hellos Document Length: 937 bytes Requests per second: 949.54 [#/sec] (mean) Database query test: Document Path: /hello/hellodb Document Length: 2482 bytes Requests per second: 734.18 [#/sec] (mean)
(Note: I think my version of Ruby was using a newer version of the SQLite3 library than my version of Python. This may or may not have given Rails an advantage on the database query test.)
For those of you who like charts, here’s a comparison of the “return hello world” test results for each framework:
The return hello world test chart
And here’s a comparison of the template test:
The template test chart
Finally, here’s the template test with the database query added:
Template test with database query
Closing thoughts
- Gone are the days of Rails as “the one framework to rule them all.” C’mon guys, it’s not 2005 anymore.
- Python has gotten a lot of flak from other language camps for its abundance of web frameworks. While some Pythonists have been embarrassed by this, I only see it as proof of Python’s success as a language. My retort would be: “If it was as easy to write a web framework in your language as it is in Python, you’d have the same problem!”
Coming soon: “Round 4: Micro-frameworks — Quick & Dirty, and leaving PHP in the dust!”
(Note: Please don’t leave benchmarking requests as comments. If you’d like me to take a look at your favorite framework, direct message me on twitter. Thanks!)
October 6th, 2009 on 5:10 am
Very interesting posts, if not entertaining to read on the road home. Keep digging, I’m sure there’s more insights to be excavated. More charts would be good for visual junkies like me!
October 6th, 2009 on 6:00 pm
I don’t use twitter, so sorry, leaving comment here.
One thing I have always wanted to look at is how the URL routing mechanism in specific frameworks affects performance.
Specifically, how does performance slow, if it does, when any URL mapping table has more and more entries in it.
For example, is request throughput significantly affected if URL matches 15th of 20th or so rules in ordered Django urls.py file, compared to if there was only a couple of entries in the urls.py file. Similar for whatever URL routing mechanism is used for other web frameworks.
Do some web frameworks blindly just do a linear match against all patterns, or do they try to optimise it by constructing some elaborate search tree that operates across all possible URL patterns.
If URL routing isn’t optimised in some way, can easily see how web applications could get slower the more URL patterns you add to them.
October 6th, 2009 on 9:33 pm
Bottle differs between static routes (stored as keys in a dict) and dynamic routes (precompiled regular expressions stored in a list). A static route is found with a single key lookup. The dynamic routes have to be tested one by one, but they are optimized (reordered) automatically, so frequently used routes are tested first. To speed things up even more, each HTTP method (POST/GET/…) has it’s own set of routes.
Here are some numbers.
(#/sec for “Hello World” using “ab -n10000 -c10″ and fapws3)
3599.40 – 10 static routes
3511.58 – 1000 static routes
3469.29 – 10 dynamic routes
3346.54 – 1000 dynamic routes (first match)
3135.88 – 1000 dynamic routes (last match, with optimization)
2425.36 – 1000 dynamic routes (last match, no optimization)
October 6th, 2009 on 10:59 pm
Marcel, if you reorder dynamic routes, how do you handle precedence, ie., where one pattern may actually effectively overlay the URL namespace of another?
Imagine in Apache where you can say:
AliasMatch ^/suburl/.*$ /some/script/suburl.cgi
AliasMatch ^/.*$ /some/script/root.cgi
In Apache you would always be guaranteed that sub URL gets precedence because of it being listed first in the file.
October 7th, 2009 on 5:18 am
Graham, I don’t. Route optimization is disabled by default. If you use it, you have to make sure that your routes don’t overlap each other. The default placeholder in dynamic routes matches anything but a slash, so they are save most of the time. I most cases, your scenario would rather look like this:
AliasMatch ^/suburl/[^/]*$ /some/script/suburl.cgi
AliasMatch ^/[^/]*$ /some/script/root.cgi
You can however add a ‘catchall’ handler (@bottle.default()) that will never be shifted in front of other routes.
October 7th, 2009 on 8:02 am
These new charts are interesting. Django seems to have problems with the templates-only test, almost loosing his second place to Pylons. Rails seems to handle databases very well, leaving TG2.1 behind in the database-query test.
October 7th, 2009 on 4:52 pm
Interesting comparison. I have ported the test code for TurboGears2 to TurboGears 1.1. On my hardware (a Mac mini 1.87 Ghz Core Duo with 2GB RAM), the results for TG 1.1 vs TG 2 are:
TG1 (using CherryPy built-in server):
Document Path: /
Requests per second: 284.03 [#/sec] (mean)
Document Path: /genshi_hello
Requests per second: 144.52 [#/sec] (mean)
Document Path: /mako_hello
Requests per second: 198.85 [#/sec] (mean)
Document Path: /jinja_hello
Requests per second: 208.91 [#/sec] (mean)
Document Path: /raw_sql
Requests per second: 121.98 [#/sec] (mean)
Document Path: /genshi_sql
Requests per second: 119.51 [#/sec] (mean)
Document Path: /mako_sql
Requests per second: 120.14 [#/sec] (mean)
Document Path: /jinja_sql
Requests per second: 120.77 [#/sec] (mean)
TG 2 (using paste built-in server:
Document Path: /
Requests per second: 199.22 [#/sec] (mean)
Document Path: /genshi_hello
Requests per second: 109.45 [#/sec] (mean)
Document Path: /mako_hello
Requests per second: 142.79 [#/sec] (mean)
Document Path: /jinja_hello
Requests per second: 153.79 [#/sec] (mean)
Document Path: /raw_sql
Requests per second: 90.98 [#/sec] (mean)
Document Path: /genshi_sql
Requests per second: 62.28 [#/sec] (mean)
Document Path: /mako_sql
Requests per second: 91.30 [#/sec] (mean)
Document Path: /jinja_sql
Requests per second: 95.47 [#/sec] (mean)
I’ll upload the code to http://chrisarndt.de/projects/wf-shootout/ shortly.
October 7th, 2009 on 11:34 pm
IMHO a Benchmark make only sense if you test is near a real world example instead of a “Hello World”.
With a minimal Test app, you will only test the Overhead of this small part of the Framework.
October 8th, 2009 on 5:12 pm
Jens, a very basic hello world test may not say too much, but hello world tests can still be informative for developers of frameworks if they go to the extent of breaking down how much time is spent in each major component within their system. That is, routing, database, data processing and templating. And then varying the complexity of configured routing and templates to gauge how the growing complexity affects the throughput ability. To do this in a meaningful way you can’t just use any application as you need to control that complexity in a step wise fashion to get meaningful results.
This is why I was pushing for some benchmarks for increases in routing complexity as I have a feeling this is an area which is often neglected and where users may not even appreciate the implications on performance by adding more and more URLs within an application using whatever routing mechanism configuration has to be used.
October 8th, 2009 on 9:13 pm
Graham, your question is intriguing and perhaps in the coming weeks I will find the time to put such a test together.
Chris, as I believe CherryPy is expected to be faster than Paster, I put TG 1.1 through my mod_wsgi test and came up with the following results: http://bit.ly/1czNm3 . I was not able to get Mako to work for some reason.
October 8th, 2009 on 10:00 pm
@Graham
That is a very good point. And I think that should be a really nice benchmark. And In fact I believe TG will come up really nice on that, mainly because TG is not affected by the “more routes more lookups”, in case you are not familiar TG builds a graph of objects ie:
class BlogController:
def index(self):
pass
class RootController:
blog = BlogController
def index(self):
pass
so /index and /blog/index are the urls. Since this is pure python traversal having as many methods/instances will not slow TG down. However it has one drawback in deep nested structures, that said I believe if you go beyond 5-6 elements in your PATH_INFO you are probably doing something else wrong.
October 8th, 2009 on 10:01 pm
@Chris Arndt
is that TG2 or TG2.1? as you can see from the tests 2.1 has been getting some performance improvements.
October 9th, 2009 on 1:51 am
Nice tests, thank you :)
October 9th, 2009 on 4:02 pm
@Jorge I ran the TG2 tests with the 2.0.3 release.