Eventually I discovered boto and settled in to using a hacked together (but stable) Python/boto solution for these type of tasks—all the while wondering why nobody took the time to write a “real” rsync-like client for S3.

Well, this last time around I finally decided to stop whining and take matters into my own hands. After a couple of late nights fleshing out my original boto solution, I’m happy to announce what I’m calling “boto rsync”—an rsync like wrapper for boto’s cloud storage interfaces (both S3 and Google Storage).

Please take a look at the project on github and let me know what you think: http://github.com/seedifferently/boto_rsync

Not only have most of the frameworks seen new releases since the last revision, but I finally decided to move all of the tests over to Amazon’s “release” version of the Ubuntu LTS AMI.

Below is a quick summary of what’s new in this revision:

• All tests were performed on the updated Ubuntu LTS AMI (ami-fbbf7892 ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20110719.manifest.xml)
• The updated AMI was configured with Python 2.6.5, PHP 5.3.2, Ruby 1.9.2p290, Apache 2.2.14 (default config), mod_wsgi 2.8 (embedded mode), and mod_passenger 3.0.9
• Rails 2.x and 3.0 were dropped from the “full stack(ish)” tests in favor of Rails 3.1.
• CakePHP 1.2 was dropped from the PHP tests in favor of 1.3, but Symfony and Yii were added as they seem to have considerable market share.
• CakePHP’s caching engine was incorrectly configured during the last round of tests, and this has been corrected.

Circle me on Google+ to keep track of further updates, and feel free to contact me there with any questions or comments.

In response to this, I thought I’d share a few of my own thoughts on G+, and why I believe it is a valuable and needed addition the online social ecosystem:

1. It’s not Facebook, and it’s not Twitter—it’s a bit of both, and the key is Circles.

What Twitter does best is giving busy professionals, celebrities, and business entities an outlet to interact publicly with their audience on a pseudo-personal level in 140 characters or less. What Facebook does best is giving people an outlet to interact somewhat privately with people they trust on as personal of a level as they like. For many, the separation between Twitter vs. Facebook mirrors their own separation of business vs. personal. For example: A picture of your 2-year old squeezing the cat is more likely to wind up posted to a limited audience on Facebook than publicly shared on Twitter, while a quick blurb updating anyone interested on a recent professional achievement often needs a more public forum such as Twitter to gain the visibility you want it to have. Even in each of their post boxes you can see the difference in what’s expected to be shared: Twitter asks “What’s happening?” expecting you to want the whole world to know, while Facebook asks “What’s on your mind?” which is quite a bit more personal. 

What Google+ gives you is both (and much more), through the power of a brilliant feature called Circles. No longer is it a question of public vs. private, but rather the much more honest and straightforward ”Who do you want to share this with?” Through Circles, you now have complete control over who will and who wont see the content you are about to share. You may only want “close friends” to see how crazy that party was last night, while “friends” can be given access to pictures of your recent vacation to Hawaii, and the whole world can have access to that presentation you did last week that received a standing ovation.

(Note: Yes I am aware that Facebook offers something similar to Circles, but its integration is clumsy and inconvenient in comparison.)

2. For now, it may only be relevant to those who manage multiple online identities.

Sure, there are plenty of people who use Twitter and Facebook as practically the same outlet, and if you’re one of those then Google+ is probably going to be little more than a redundant nuisance—one more social network to have to keep track of.

Or perhaps you’re the Facebook-only or Twitter-only type and simply don’t see the point in jumping ship or picking up another service. I applaud you for your loyalty, but in this day and age resistance to “the big G” is futile. Which leads me to my final thought…

3. If you can look to the horizon, it might just be bigger than anything we’ve seen yet!

The real thing to keep in mind in these early days is that G+ is just the beginning of Google’s plan to bring a social element to many of its products and services. In the coming months, you are probably going to see the google.com ecosystem revolve more and more tightly around G+ to a point where involvement in the + will be unavoidable. They have already somewhat merged Picasa and there are whispers of a full-on integration with Gmail. What else could be coming? Well, just think about all of the services that Google currently offers and simply add a “+” to them: Music+, Calendar+, Maps+, Docs+, Checkout+, etc.—and I’m not even going to take the time here to mention Android and the possible mobile implications!

But what about speculative projects? Well, Google recently sank loads of cash into a little company called Zynga and not too long ago partnered with Rovio to bring Angry Birds to Chrome. Can you imagine playing “Words With Friends+” or “Angry Birds+” online with all your G+ friends?

The future of Google is here folks, and it’s as subtle as a +.

[This post is the continuation of a series. Please read Round 1, Round 2, and Round 3 first if you are just now joining us.]

While I had originally intended for round 4 to showcase how microframeworks are changing the way we do “quick and dirty” web development (and how they make using PHP as “an extension to HTML” old hat), my current programming habits have kept me involved in the more “full-stack” framework solutions. So, rather than spitting out various benchmarks of frameworks that I have little or no interaction with, and since enough time has passed since the last “shootout” that the landscape has changed a bit (with the introduction of Pyramid and the release of Rails 3), I have instead decided to showcase the most recent data on the frameworks that I personally find myself in contact with on a regular basis.

Warning: Everything is different this time around!

These benchmarks were all run on a fresh Amazon EC2 instance in order to (hopefully) achieve a more isolated environment. Obviously, since these benchmarks have all been run on a completely different box than any of the previous rounds, no previous data should be compared with these numbers.

What you should know about Round 4:

• The EC2 instance used was: ami-da0cf8b3 m1.large ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20101020.manifest.xml
• As a “Large” instance, Amazon describes the resources as: 7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB of local instance storage, 64-bit platform.
• The various system software used was whatever was current as of November 18, 2010 on Ubuntu 10.04′s repositories.
• Apache 2.2.14 was used for all tests.
• Python 2.6.5 and mod_wsgi 3.3 (embedded mode) were used for the Python tests.
• Ruby 1.8.7 (except for the last Rails 3 test) and Phusion Passenger 3.0 were used for the Rails tests.
• ApacheBench was run with -n 10000 and -c 10 about 5-10 times each, and the “best guess average” was chosen (yeah, I’m lazy).
• The Pyramid/TG test apps used SQLAlchemy as the ORM and Jinja2 as the templating system (Jinja2 was consistently around 25-50r/s faster than Mako for me).

Remember, nothing here is really all that scientific and your mileage WILL vary. Now on to the results…

The “Hello World” string test

The “Hello World” string test simply spits out a string response. There’s no template or DB calls involved, so the level of processing should be minimal.

The return hello world test chart

The template test

The template test simply spits out Lorem Ipsum via a template (thus engaging the framework’s templating systems).

The template test chart

The template + DB query test

The template/db test simply loads 5 rows of Lorem Ipsum from a SQLite DB and spits it out via a template (thus engaging both the framework’s ORM and templating system).

Template test with database query

Closing thoughts

• The Pyramid Alpha results excite me quite a bit with all the talk of the various frameworks looking to merge with it. Could this really become the “one Python framework to rule them all”?
• Django is still going strong, and continues to be an exceptional choice if you’re in to its way of doing things.
• The Rails results left me a little confused are interesting because Rails 2 apparently outperforms Rails 3 on Ruby 1.8. Of course, once you move to Ruby 1.9, Rails 3 runs quite nicely.
• Don’t believe these numbers? Feel free to boot up your own EC2 instance and run one of the test apps:
  • Pyramid 1.0a3 test app code
  • Django 1.2 test app code
  • TurboGears 2.1 test app code
  • Rails 3 test app code
  • Rails 2 test app code

(Please don’t leave benchmarking requests as comments. If you’d like me to take a look at your favorite framework, direct message me on twitter. Thanks!)

Well, I am the black sheep in my culture because I have an Android phone. This sometimes causes a bit of a stir with co-workers and friends (all in good fun though, right?), but every now and then I come across someone who is thinking about upgrading phones and wants to know why I’ve chosen Android over the iPhone. The answer is actually quite simple in my mind, and basically comes down to what I would consider the three “killjoy” facts about the iPhone:

Fact #1: The iPhone is part of a “closed ecosystem”

Non-techies will probably see the phrase “closed ecosystem” and dismiss it as tech jargon and quickly move on to the next point; but in my mind this is probably the most important issue of them all, and it actually has a much larger affect on the end-user than most people realize. (I won’t even start about how this affects the developers. That could be a whole blog post of its own!)

So, what exactly do I mean by a “closed ecosystem” and how does it affect the end-user? In my opinion, this is best described in an over-simplified and somewhat silly little story:

Bob decides one day to go and buy a ~$500 ultra-mobile computing device from Steve. Bob loves how sleek and sexy his new device is, but whenever Bob wants to install a new program on his new device, Bob has to check-in with Steve and make sure Steve is okay with Bob installing that program. Bob isn’t given the option to install whatever he wants whenever he wants—he always has to go through whatever process has been set up to gain Steve’s approval in order to install anything.

That sounds kind of silly doesn’t it? We typically expect to have the freedom to do anything we like with something we’ve bought for ourselves; and yet, this is exactly how the iPhone’s integration with the Apple App Store works. While most people might laugh at the idea of this being much of an issue (the App Store certainly doesn’t feel very restrictive to the uninformed), it actually is a pretty big deal to some people, and it’s why the word “jailbreak” has become a household word for many Apple extremists.

So, how does this play out in the real world for the end-user? Well, to be fair, let me first highlight the fact that there can be some “pluses” (depending on how you look at it) to a more restrictive system. For one, Steve can make sure Bob never installs a “bad” program on his device (of course, Steve decides what “bad” means). Also, by acting as the sole party in charge of “authorizing” applications (and therefore the gateway-*cough*-bottleneck between the app developers and their customers), Steve can create all kinds of developer requirements based on what’s important to him (aesthetics, functionality, etc), thus raising the bar of what kind of apps are created for his device. (Of course, this also causes headache after headache for the developers; but I promised I wouldn’t get into that here…)

That sounds nice right—but what happens if your good ol’ buddy Steve decides he doesn’t like a certain program anymore, even if it is widely used and supported by 99% of the market? I’m sure you iPhone users reading this have encountered this fun little image before. Yeah, Steve decided that Flash was “not cool” and has therefore decreed that it will forever be banned from iOS devices. Or how about something a little more offbeat? Wouldn’t you love to have this awesome little text-input app on your iPhone? Well, sorry guys, applications that ask for that much core integration with the OS aren’t cool either according to Steve, so tough luck! In fact, Steve won’t even give you tethering! Now where’s the fun in that (especially if you travel as much as I do)? Sure, farts are cool. And birds hitting pigs? Yeah, that’s cool too. But at the end of the day, Steve has the final say. Not you.

Remember: You paid to use your iPhone how Steve wants you to use it, not necessarily how you want to use it.

Fact #2: The iPhones (yes, even the 3GS!) really do have reception issues

While I agree with the Apple fanboys that “antennagate” was blown way out of proportion, I would still argue that as a phone, the iPhone sucks. Most people have been too busy blaming AT&T (see my next point) to accept the truth, but the fact of the matter is that studies have shown that the iPhone has poorer reception than other leading AT&T phones. Yes, you read that right—AT&T might not have as much to do with your dropped calls as you may think.

And how about that “revolutionary new antenna design” in the iPhone 4 that Mr. Jobs kept going on about? Well, do you remember this slide from the epic “antennagate” emergency press meeting? According to “call drop data from AT&T” the iPhone 4 actually drops more calls than the 3GS. Oops!

Fact #3: AT&T has some serious issues with its network in many areas

It’s no secret that AT&T’s network has issues. Over the past 10 years I’ve had contracts with all four major carriers and AT&T was definitely the worst in terms of call reliability. After switching back to a carrier other than AT&T, I think I’ve only had one dropped call so far this entire year (as opposed to what sometimes felt like every other call on AT&T). What a novel idea: having a complete conversation without ever having to hit redial!

But I don’t even live in a major metropolitan area where AT&T’s network is over-saturated. Did you know that the average dropped call rate for NYC is 30% (according to a September 2009 report)? I would find that absolutely unacceptable.

Oh, and don’t keep holding your breath, the iPhone still doesn’t seem to be coming to Verizon anytime soon.

But, isn’t Android susceptible to these issues as well?

Without going into too much detail, the answer is both “no” and “maybe.”

The biggest difference is the ecosystem. Since Android is deeply rooted in open source, both the developers and the end-users are given an incredible amount of freedom to do whatever they want with their devices. This freedom has spawned a thriving “aftermarket” community that has gifted us with such marvels as custom ROMs, 3rd party app stores, and all combinations of tweaks, tricks and enhancements. But have no fear—if you’re someone who doesn’t like to get his or her feet wet pushing the bounds of your freedom, Google still has you covered with its own app store for an “official” application install channel.

Of course, any phone can have antenna issues (something Mr. Jobs has been clear to point out), so it would be nice to see data on dropped calls for a comparable Android phone on AT&T’s network. However, the nice thing about the fact that Android is an operating system and not a phone (remember: the Droid is a phone and Android is the operating system that it and many other Android phones run) is that there are Android devices available on pretty much *every* mobile carrier of any size in the US and all over the world.

But the iPhone 4 still blows Android out of the water feature and spec wise, right?

Actually, no. There are Android phones out there with more memory and processing power, higher resolution cameras (on the front and back), better battery life, and more vibrant displays (although “Retina” vs SuperAMOLED is probably still a personal preference thing).

Okay, I know what you’re thinking now: “What about FaceTime?” Well, the truth is that Android was doing face-to-face mobile video calls before the iPhone 4 was even announced. Crazy right? Well, here’s something else Apple probably doesn’t want you to know: Not only can you use an Android device to have FaceTime-like video chats with both iPhone 4 users and other Android users, but you also aren’t restricted to WiFi-only usage!

Oh, and remember: “successfully placing calls” should probably be pretty high up on your feature-list for a smartphone.

“I Don’t Care! I Want An iPhone 4!”

At the end of the day though, most people approach choosing a phone in much the same way as if they were choosing a watch or a new pair of sunglasses—it’s all about what looks good to them. The iPhone has an aura about it that is beautiful and sexy; and evidentially many people either do all of their phone talking within excellent AT&T coverage areas (somewhere out there I’m sure such a thing exists), or they really don’t mind having calls drop on them. After all, you can always just hit redial, right? The overwhelming response by most iPhone fans to the issues I’ve raised above is either denial, or an emphatic ”I Don’t Care” (warning: lots of NSFW language in that hilarious youtube video).

And why should they care? After all, they have an iPhone!

Now, this guy hasn’t exactly been living under a rock for the past 10 years—his resume was quite impressive and included projects in a number of different programming languages; But as you can imagine, I was rather surprised by his response, and it made me wonder: Has the Python community really been that bad at promoting the strengths of Python for web development? Or, does the nirvana experienced by switching from a language like PHP to Python just make us so at peace with the world that we forget the hordes of developers still stuck with C-style syntax? Either way, it got me thinking about a few of the reasons why I decided to switch from PHP to Python; and why I not only see Python as an excellent PHP replacement, but am surprised it is such a “best-kept secret” for web development.

There are already plenty of pages out there discussing Python vs. PHP as a language, so I probably won’t get too technical here. I also want to try avoid turning this into a “Python is better than PHP because…” rant (for more on that, please see the end of this post), so I will simply share with you a few of the main reasons why I decided to replace PHP with Python as my primary language for web development:

» Python is more flexible than PHP

Python was engineered from the beginning to be a general purpose programming language. PHP on the other hand was written to overcome many of the “limitations” of the cgi-bin. This should say a lot about the “soul” of each language, but the conclusion I have come to is:

• PHP is meant to do web things really well
• Python is meant to do most things really well

Sure, these days you could write a desktop GUI in PHP, but it would most likely not be the right tool for the job (and you’d probably get laughed at). On a somewhat related note: many PHP developers I know find themselves switching between PHP for their website tasks, and a variety of other languages for other tasks (Shell Scripting for command-line scripts, Perl for text processing, etc.). As a Python programmer, I can typically stick to one programming language no matter what the task at hand requires (especially when armed with iPython).

Now, the obvious argument for PHP at this point is: “But wait, since PHP was written specifically for the web, shouldn’t that actually give it more credibility as the right tool for the web job?” Well, 5 or 10 years ago I might have agreed with you, but web development has changed a lot in the past few years (not to mention the introduction of Python’s WSGI spec).

The internet as a platform has grown and mutated drastically and now requires much more than “a server-side extension to HTML.” We are now in an age of full-blown internet applications, which require a true “application-level” language. In my opinion, PHP has only recently begun to move in that direction with the v5.3 release (as well as updates expected in PHP 6). That alone should give you pause when pondering PHP’s ability to “keep up” with the growth of the web.

In short, the internet continues to transform into an arena that is much less of a website platform, and much more of an application platform. To me, this makes a general purpose language such as Python the obvious choice for “future-proof” internet development.

» Python is typically faster than PHP, even especially on the web

You might find PHP beating out Python in a simple “hello world”-type test, but “real-world” applications will typically run much faster in Python. In a Web 2.0 world where everyone has broadband, speed is now an important factor in keeping your visitors happy.

In addition to “raw, under-the-hood” speed, Python’s language philosophy means it will typically scale better than PHP (although this can be argued a million different ways, since scalability usually involves much more than the language).

I’ve already done benchmarks comparing a few popular web frameworks, but if you’re still curious I’d suggest you dig around with Google for a while.

» Python is easier to read and maintain than PHP

Any veteran programmer will tell you that code readability plays a huge factor in code maintainability. With that in mind, Python was designed with some pretty strict syntax rules in regards to code-blocks and whitespace. While this may initially frustrate programmers who are used to a looser “bracket-encapsulated” language, it actually forces you to write more readable (and therefore, more maintainable) code.

Other than readability, Python’s clean syntax also means:

• You can write more code with less errors (because they’re so easy to spot). This often comes as a pleasant surprise to programmers migrating from C-style languages (take Eric Raymond for example).
• You end up typing less — No more semicolons or brackets, and fewer parentheses.
• Your eyes don’t have to move around as much to figure out what’s going on.
• Your first guess about what’s happening in a code-block is usually correct.

When you have time to kill, I’d recommend a stroll through Rosetta Code to compare how different languages “look” when tackling the same problem. Unless you just love brackets, I think you’ll find the non C-style syntaxes much easier to read.

» Python is more consistent than PHP

Continuing somewhat in the vein of readability is the topic of consistency. Since PHP’s lack of consistency has already been studied exhaustively by the Perl folks, I will spare you my rant and suggest you read theirs.

As a side note on this topic: I often hear PHP programmers talk about how awesome PHP’s documentation is in that you can open a web browser and just type “php.net/some_function” to get help. Well, documentation recall usually isn’t that big of a deal for me (because I have a somewhat photographic memory), but I will say that after switching to Python I found myself turning to its documentation significantly less than I had with PHP, simply because I didn’t have to keep reminding myself whether that infrequently used function was named some_function or somefunction. With Python, the function/method I need is usually named exactly what I expect it to be named, and returns exactly what I expect it to return.

Oh, and don’t even get me started on PHP’s recent namespace decision.

» However, PHP still excels in a few areas «

Having said all of that, there are a few areas where I believe PHP may still shine brighter than Python:

» PHP hosting support is may be more prevalent than Python

It’s practically unheard of to find yourself on a web host that doesn’t support PHP out of the box. However, this is becoming less and less of an issue since most Python web applications can now be deployed using tools such as FastCGI or mod_proxy (as opposed to the popular mod_wsgi setup, which often requires root access to configure). These days any “reputable” web host is probably going to support some sort of strategy for deploying Python web applications. Of course, this only really applies to shared hosting environments without root access. If you’ve got a VPS or dedicated server you’re fine.

» PHP is arguably “more seasoned” on the web than Python

The communities behind projects like Drupal and WordPress really stand out to me as examples of the maturity of PHP on the web. Although similar Python web projects have been slowly emerging (and Plone has been around for years, if that’s your cup of tea), they don’t really compete yet in my opinion.

Of course, I am only talking about specific web projects here. Language wise, Python has been around longer than PHP. Web 2.0 framework wise, Python’s offerings have been around about as long as PHP’s (pretty much every language had its own answer to Ruby on Rails in 2005).

» PHP seems easier than Python at first (and might be for simple deployments)

There’s something magical about throwing together a few quick index.php/about.php/contact.php files and uploading them to the httpdocs directory of your webserver and it “just working.” A lot of simple websites have no need for acronyms like MVC, ORM, and DRY, and in those cases PHP is an excellent “server-side extension” to HTML. However, if and when you begin to scale, you’ll either find yourself switching to a full-blown framework, or throwing one together to help facilitate your growth. At that point, you’ll probably begin discovering the “dark corners” of PHP, and may find yourself wishing you had “done it right” the first time.

» So, which is better? «

Remember, the point of this post is to solidify the fact that Python can be an excellent PHP replacement. When it comes to programming languages, “better” is often more a case of personal preference than anything else. However, to me the points discussed above make Python the better choice when it comes to what I like in a web development language.

So, is Python a better web development language than PHP?
Well, does Nike make a better shoe than Adidas?

The most honest answer is probably: “It depends.” Python definitely has some significant advantages over PHP in certain circumstances, but the same could probably be said of PHP in other circumstances (although limited to the web I imagine).

I hope this post has encouraged you PHP programmers out there to take another look at Python. It may take some getting used to (Python’s way of doing things is inherently quite different than PHP’s), but you will probably find yourself thanking me later.

A lot of work has been going into the TG docs in preparation for the upcoming v2.1 launch, and one little gem that was recently added was the Pagination Quickstart (thanks Lukas). While this sent me down the right path towards pagination perfection, there were a few things that came up that I felt like deserved further explanation:

1. The @paginate() decorator is an extremely simple wrapper for the paginate webhelper module.
2. When paginating SQLAlchemy queries, paginate runs a duplicate count() query to figure out its current position in the data collection. While the magic of this is nice, the possible performance issues with duplicating complex SQL queries on a high-traffic website could be undesirable.
3. It’s common to have controller methods dynamically switch to a “feed template” when doing AJAX pagination. Don’t worry, paginate + override_template makes this a piece of cake!

The @paginate() decorator makes pagination a breeze

The @paginate decorator is a wrapper for the paginate module that can be added to your controller methods. Simply specify your data collection’s name as the first argument of the decorator, and pagination is “automagically” setup for you.

An example controller method with pagination would look like this:

# dead-simple pagination in TurboGears 2
 
@expose('myproject.templates.posts')
@paginate("posts")
def posts(self):
    posts = DBSession.query(Posts)
    return dict(posts=posts)

Really? That was easy!

The documentation on this decorator seems quite clear to me, so instead of repeating its content here I will simply give you a link for further reading.

SQLAlchemy pagination performance wizardry

The main issue here is that the paginate module has to figure out its current position in the data collection in order to be able to spit out the right “chunk” of data (as well as to provide you with fancy stuff like “page 3 of 6″). So, by default when you give it an SQLAlchemy query object, it runs a count() query to figure this out. This results in two practically identical SQL queries (one to get the total count, and the other to return your actual collection of data). While this may not be an issue on simple queries, duplicating complex queries on a high-traffic website could be disastrous.

Remember, magic can be unpredictable, so before we dive into this topic there are a few things you should take into consideration:

• This technique is not compatible with the @paginate() decorator due to the fact that the current page arg isn’t passed into the controller method when using the decorator, so be ready to get your hands dirty with the full-blown paginate module.
• The SQLAlchemy all() query method results in immediate query execution. This may have undesirable side-effects.
• I’m sure there is a cleaner/sexier way to do this, but for the sake of time and in an effort to be verbose this is what I have come up with so far. Please leave a helpful comment below if your magic is cleaner than mine.

Ok, so now that we are familiar with the problem, what is a good solution? Well, a rough hack to work around this issue is to use SQLAlchemy’s limit() and offset() query methods combined with the all() method to get SQLAlchemy to return a list instead of a query object. This puts paginate to work on the list instead of on the database, avoiding the extra count() query. However, as tricky as this may be, there are still problems:

1. In order for paginate to calculate a correct current position based on the “page” it thinks it’s on, you’ve got to pad the beginning of the list you send to the paginate.Page() method with enough items to accurately reflect the “current” size of the data collection.
2. In order for paginate to know there are more records waiting to be loaded, you’ve got to pad the end of the list you send to the paginate.Page() method with at least one more item than each page’s specified item limit.

This may seem confusing at first, but an example controller method should help clear things up for you:

# import the paginate webhelper
from webhelpers import paginate
# specify the number of items you want per page
items_per_page = 20
 
@expose('myproject.templates.posts')
def posts(self, page=1):
    # specify the limit, plus 1 so the Page() method can tell if extra data is available
    limit = items_per_page + 1
    # specify the offset to start with
    offset = (int(page) - 1) * items_per_page
 
    # if the offset is greater than 0, we have to pad the beginning of the list
    if offset: posts = [x for x in xrange(offset)]
    # otherwise we're starting at the beginning (with an empty list)
    else: posts = list()
 
    # load the DB data into the list
    posts.extend(DBSession.query(Posts).limit(limit).offset(offset).all())
 
    # setup the pagination object
    pagination = paginate.Page(posts, page, items_per_page=items_per_page)
    # either use tmpl_context or return the object in the return dict
    tmpl_context.pagination = pagination
 
    return dict(posts=pagination.items)

Voilà! Now that you’re dealing with a list, pagination should work as expected without the extra “count()” query.

Use override_template() for dynamic template rendering

AJAX pagination is a common practice and often provides a richer user experience than “static” pagination. Fortunately, AJAX pagination is almost as easy to setup as normal pagination, and only requires a few extra tweaks to an already simple configuration.

The only “tricky” thing here is if you’re accessing the same controller method for partial data as well as the full page. In such a case, you’ll need to have the controller method render two separate templates depending on if you’re asking for the “full” page, or just the listing of data. Don’t worry though, paginate automatically provides you with a “partial” arg that you can check against, and override_template() allows you to override whatever template you have defined in the @expose() decorator.

Modifying the previous example’s controller method for this type of functionality looks like this:

# add the override_template import
from tg import override_template
 
# add the "partial" arg to the controller method
def posts(self, page=1, partial=0):
    # ...snip...
 
    if partial:
        override_template(self.posts, 'mako:myproject.templates.post_feed')
 
    return dict(posts=pagination.items)

More information on override_template() can be found in the docs.

I hope this tutorial has been helpful. I will be submitting a TG docs pull request soon, so please leave comments if you have any suggestions or feedback.

Take the following set of data for example:

[[1, 'John Smith', 'admin'],
 [2, 'Jane Doe', 'superuser'],
 [3, 'Sam Jones', 'user']]

What we have here is a few rows of user data. In this example, the data is in a Python list (which in PHP would be an array).

In PHP, this would look something like (using print_r):

Array (
    [0] => Array (
        [0] => 1
        [1] => John Smith
        [2] => admin
    )
    [1] => Array (
        [0] => 2
        [1] => Jane Doe
# (...etc...)

So, what if I found myself writing some code that needed to be able to access each record by its first value, which in this case would be the user_id?

Well, in PHP the easiest way to make this happen would be a good-old-fashioned foreach loop:

foreach ($row as $val) {
    $out[$val[0]] = $val;
}

Not very “sexy” for something I find myself having to more often than I would like, but it works nonetheless.

Now, here’s where the beauty of Python kicks in. Python has a built-in feature called “comprehension” which allows you to modify and/or convert sets of data quickly and easily.

Remember what our original data set looks like? Here’s a refresher:

>>> print row
[[1, 'John Smith', 'admin'],
 [2, 'Jane Doe', 'superuser'],
 [3, 'Sam Jones', 'user']]

Now, in Python all we need to do is use the dictionary comprehension:

>>> out = {val[0]: val for val in row} # Python 3.X
# (or, for Python 2.X)
>>> out = dict((val[0], val) for val in row) # Python 2.4+
 
>>> print out
{1: [1, 'John Smith', 'admin'],
 2: [2, 'Jane Doe', 'superuser'],
 3: [3, 'Sam Jones', 'user']}

See how easy that was?

And what if I had a dictionary in the above format, and needed to turn it into the type of list I had previously? Well, I could simply use the list comprehension to re-generate this set of data as a list:

>>> print [val for val in out.values()]
[[1, 'John Smith', 'admin'],
 [2, 'Jane Doe', 'superuser'],
 [3, 'Sam Jones', 'user']]

This is an extremely simple comprehension example. For a deeper look, check out Wikipedia’s Python syntax page or the official Python documentation.

[Note: This post is the continuation of a series. Please read Round 1 and Round 2 first if you are just now joining us.]

As I briefly mentioned in Round 1, this whole thing came about as an experiment to satisfy my own curiosity. Unfortunately, I wasn’t expecting these posts to draw the amount of attention they have been getting, and several people informed me of a few “issues” with the first round. Since my initial approach to this topic was somewhat casual, I didn’t really take the time to perform each test in a “proper scientific fashion.” Although this was clearly stated in the introduction to round one, it unfortunately resulted in performance estimations that were somewhat less than accurate.

After input from various people much smarter than myself, I quickly went to work tweaking my test environment and building “proper” test apps. In the midst of this, a conversation about PHP accelerators prompted me to put PHP under the spotlight, which brought about Round 2 as an interim round. This gave me a chance to demonstrate the necessity of PHP acceleration, and only continued to solidify my opinion of PHP as an inferior web development language (remember, I just said my opinion).

Which brings us to Round 3. A lot of work has gone into “doing it right” this time, so I am fairly confident that these results are a much more accurate representation of each test subject’s performance estimations. Remember, benchmark test code typically has no real-world value, so “performance estimations” are about all I can promise here. Your mileage will vary. As a wise person once said:

“All this benchmarking is doing is proving what we already know: More code takes longer to execute.” – Ben Bangert (dev lead of Pylons)

What you should know about Round 3:

• The hardware/software platform is the same as in Round 1.
• Python v2.5.4 was used for all the Python tests. Ruby v1.8.6 was used for the Rails test.
• Apache v2.2.3 was used, with mod_wsgi v2.5 for the Python tests and Phusion Passenger v2.2.5 for the Rails test.
• Note that while the mod_wsgi tests in Round 1 were using “daemon mode”, the mod_wsgi tests here are using “embedded mode” which is often faster (although not necessarily the recommended configuration).
• The Python component versions used here were: SQLAlchemy v0.5.6, Genshi v0.5.1, Mako v0.2.5, and Jinja 2.2.1.
• ApacheBench was run with -n 10000 -c 10.
• Each test was run several times to make sure that there were no anomalies, with the “optimum average” chosen as the numbers represented here.
• In an attempt to try to make these tests more scientific, I am now including the source code for each test.

Plain WSGI – wsgi_test_src.zip

Let’s start with a basic “hello world” WSGI test to set the baseline:

The 'hello world' test:
 
WSGIServer/0.1
Document Path:          /
Document LENGTH:        12 bytes
Requests per second:    1532.76 [#/sec] (mean)
 
 
Apache/2.2.3 (mod_wsgi)
Document Path:          /
Document LENGTH:        12 bytes
Requests per second:    5549.73 [#/sec] (mean)

As you can see, mod_wsgi allows Apache to put some serious muscle into Python’s WSGI. It’s no wonder that mod_wsgi is quickly becoming the de-facto standard for Apache Python deployment.

TurboGears v2.0.3 – tg2_test_src.zip

As I mentioned in Round 1, TurboGears has quickly become my web framework of choice. Building a “best-of-breed” component stack on top of Pylons may not be easy, but when it works I believe it pays off immensely (e.g. Ubuntu).

The 'hello world' test:
 
Document Path:          /
Document LENGTH:        12 bytes
Requests per second:    935.13 [#/sec] (mean)
 
 
Genshi template test:
 
Document Path:          /genshi_hello/
Document LENGTH:        923 bytes
Requests per second:    601.07 [#/sec] (mean)
 
 
Mako template test:
 
Document Path:          /mako_hello/
Document LENGTH:        932 bytes
Requests per second:    723.18 [#/sec] (mean)
 
 
Jinja2 template test:
 
Document Path:          /jinja_hello/
Document LENGTH:        937 bytes
Requests per second:    764.35 [#/sec] (mean)
 
 
Database bold;">QUERY tests:
 
Document Path:          /raw_sql/
Document LENGTH:        1270 bytes
Requests per second:    569.71 [#/sec] (mean)
 
Document Path:          /genshi_sql/
Document LENGTH:        2409 bytes
Requests per second:    388.96 [#/sec] (mean)
 
Document Path:          /mako_sql/
Document LENGTH:        2418 bytes
Requests per second:    515.07 [#/sec] (mean)
 
Document Path:          /jinja_sql/
Document LENGTH:        2472 bytes
Requests per second:    535.06 [#/sec] (mean)

TurboGears v2.1a1 – tg21_test_src.zip

The TurboGears community is alive and well, and just before finishing this post the release of v2.1 Alpha 1 was announced. Significant improvements were made to TG’s object dispatch, so I was excited to see what the new numbers would look like.

The 'hello world' test:
 
Document Path:          /
Document LENGTH:        11 bytes
Requests per second:    1118.92 [#/sec] (mean)
 
 
Genshi template test:
 
Document Path:          /genshi_hello/
Document LENGTH:        923 bytes
Requests per second:    713.97 [#/sec] (mean)
 
 
Mako template test:
 
Document Path:          /mako_hello/
Document LENGTH:        932 bytes
Requests per second:    812.64 [#/sec] (mean)
 
 
Jinja2 template test:
 
Document Path:          /jinja_hello/
Document LENGTH:        937 bytes
Requests per second:    1000.98 [#/sec] (mean)
 
 
Database bold;">QUERY tests:
 
Document Path:          /raw_sql/
Document LENGTH:        1270 bytes
Requests per second:    673.04 [#/sec] (mean)
 
Document Path:          /genshi_sql/
Document LENGTH:        2409 bytes
Requests per second:    433.45 [#/sec] (mean)
 
Document Path:          /mako_sql/
Document LENGTH:        2418 bytes
Requests per second:    601.89 [#/sec] (mean)
 
Document Path:          /jinja_sql/
Document LENGTH:        2472 bytes
Requests per second:    630.66 [#/sec] (mean)

As you can see, the future of TurboGears looks very bright, and I only expect these numbers to get better as the final 2.1 release approaches (expected in Q1 2010).

Pylons v0.9.7 – pylons_test_src.zip

Pylons is the foundation which TurboGears is built upon, and for those who don’t need all “the extras” it is an excellent choice.

The 'hello world' test:
 
Document Path:          /hello/INDEX
Document LENGTH:        12 bytes
Requests per second:    2593.47 [#/sec] (mean)
 
 
Mako template test:
 
Document Path:          /hello/hello
Document LENGTH:        932 bytes
Requests per second:    1737.03 [#/sec] (mean)
 
 
Database bold;">QUERY tests:
 
Document Path:          /hello/raw_sql
Document LENGTH:        1270 bytes
Requests per second:    964.07 [#/sec] (mean)
 
Document Path:          /hello/hellodb
Document LENGTH:        2418 bytes
Requests per second:    831.26 [#/sec] (mean)

Django v1.1 – django_test_src.zip

While Django is probably Python’s most popular web framework, I have never really clicked with it. Its way of doing things is significantly different than TG/Pylons, but if you don’t mind its “MTV” architectural pattern and can survive without SQLAlchemy (although there are hacks), it is a very capable framework.

The 'hello world' test:
 
Document Path:          /hello
Document LENGTH:        12 bytes
Requests per second:    3376.48 [#/sec] (mean)
 
 
Django template test:
 
Document Path:          /hellos
Document LENGTH:        936 bytes
Requests per second:    1781.43 [#/sec] (mean)
 
 
Database bold;">QUERY test:
 
Document Path:          /hellodb
Document LENGTH:        2476 bytes
Requests per second:    972.11 [#/sec] (mean)

Bottle 0.5.8 – bottle_test_src.zip

Bottle caught my attention a few months ago while surveying the WSGI landscape for an alternative to PHP when building websites that only require a few pieces of dynamic functionality. As a “micro-framework”, it does an excellent job at bridging the gap between “pure WSGI” and its “feature-rich” big brothers.

The 'hello world' test:
 
Document Path:          /
Document LENGTH:        12 bytes
Requests per second:    5545.96 [#/sec] (mean)
 
 
Mako template test:
 
Document Path:          /hello
Document LENGTH:        923 bytes
Requests per second:    3588.04 [#/sec] (mean)
 
 
Database bold;">QUERY test (SQLAlchemy):
 
Document Path:          /hellodb
Document LENGTH:        2413 bytes
Requests per second:    1479.54 [#/sec] (mean)

As you can see, there was practically no difference in speed between Bottle and pure WSGI in a basic “hello world” test. Even with the addition of Mako and SQLAlchemy, Bottle performed significantly faster than a bare Pylons or Django setup.

(Note: Mako template inheritance was not used in the Bottle Mako test.)

Ruby on Rails v2.3.3 – rails_test_src.zip

What web technology shootout would be complete without Rails? Although I am not a Rails fan (mostly because I can’t stand Ruby), I believe that in many ways we owe much of the popularity and success of the web framework movement to Rails.

The 'hello world' test:
 
Document Path:          /hello/hello
Document LENGTH:        12 bytes
Requests per second:    1048.18 [#/sec] (mean)
 
 
Rails template test:
 
Document Path:          /hello/hellos
Document LENGTH:        937 bytes
Requests per second:    949.54 [#/sec] (mean)
 
 
Database bold;">QUERY test:
 
Document Path:          /hello/hellodb
Document LENGTH:        2482 bytes
Requests per second:    734.18 [#/sec] (mean)

(Note: I think my version of Ruby was using a newer version of the SQLite3 library than my version of Python. This may or may not have given Rails an advantage on the database query test.)

For those of you who like charts, here’s a comparison of the “return hello world” test results for each framework:

The return hello world test chart

And here’s a comparison of the template test:

The template test chart

Finally, here’s the template test with the database query added:

Template test with database query

Closing thoughts

• Gone are the days of Rails as “the one framework to rule them all.” C’mon guys, it’s not 2005 anymore.
• Python has gotten a lot of flak from other language camps for its abundance of web frameworks. While some Pythonists have been embarrassed by this, I only see it as proof of Python’s success as a language. My retort would be: “If it was as easy to write a web framework in your language as it is in Python, you’d have the same problem!”

Looking for updated results? Check out Round 4…

(Note: Please don’t leave benchmarking requests as comments. If you’d like me to take a look at your favorite framework, direct message me on twitter. Thanks!)

[This post is the continuation of a series. Please read Round 1 first if you are just now joining us.]

In Round 1, PHP was looking like quite the tortoise of the group. However, if you’re familiar with some of the core differences between Python & PHP, you’ll know that Python has been “cheating” slightly.

Let me explain: By default, Python compiles each script into bytecode on its first execution, allowing this bottleneck to be skipped on subsequent runs. PHP, however does not perform this type of optimization by default (in the 5.x line at least), so the PHP interpreter must re-compile each file every time it is run. As you can imagine, this can give PHP (without an accelerator) a huge disadvantage when compared to languages such as Python.

With this in mind, I have decided to take Round 2 to focus solely PHP. This will hopefully provide a clear picture of the benefits of PHP bytecode caching (at least when it comes to page-views — the memory benefits are a whole other story), and give you an idea of PHP’s performance with the help of an accelerator.

There are many PHP accelerators available, but I have chosen APC for use here (mostly due to its inclusion in the upcoming PHP 6 core).

What you should know about Round 2:

1. The hardware/software platform is the same as in Round 1.
2. PHP has been upgraded to v5.2.9.
3. The APC tests are using APC v3.0.19.
4. ApacheBench was run with -n # -c 10, with -n usually being 5000 or 10000, depending on the test subject’s speed.
5. Each test was run several times to make sure that there were no anomalies, with the “optimum average” chosen as the numbers represented here.
6. In an attempt to try to make these tests more scientific, I am now including the source code for each test (minus the CMS tests, which were performed on a standard install).

My apc.ini settings:

extension = apc.so
apc.enabled = 1
apc.shm_size = 96
apc.include_once_override = 1
apc.mmap_file_mask = /tmp/apc.XXXXXX

Raw PHP – php_test_src.zip

To investigate the speed of “raw PHP”, I created a couple of minimal PHP scripts that printed out a simple “hello world”, and read a few lines of text from a database. Take a look at the test source code for reference.

APC Disabled

The 'hello world' test:
 
Document Path:          /hello.php
Document LENGTH:        12 bytes
Requests per second:    4199.43 [#/sec] (mean)
 
The simple database bold;">QUERY test:
 
Document Path:          /hello_db.php
Document LENGTH:        2524 bytes
Requests per second:    2797.54 [#/sec] (mean)

APC Enabled

The 'hello world' test:
 
Document Path:          /hello.php
Document LENGTH:        12 bytes
Requests per second:    6618.42 [#/sec] (mean)
 
Simple database bold;">QUERY test:
 
Document Path:          /hello_db.php
Document LENGTH:        2524 bytes
Requests per second:    3713.99 [#/sec] (mean)

Right out of the gate, we can see that APC gives PHP a significant boost.

Of course, a script that simply does a “echo ‘hello world’” isn’t really practical, so let’s take a look at a couple of popular PHP frameworks to see what a full stack looks like.

CakePHP v1.2.5 – cakephp_test_src.zip

CakePHP was the slowest test subject by a large margin in Round 1, but if you take a look at the caching section of its core configuration file you can see that the developers have written it with acceleration in mind. It includes special configuration directives for not just APC, but also Xcache and Memcache.

Still though, without acceleration CakePHP is a sad story indeed:

APC Disabled – File Caching Engine (default)

The RETURN 'hello world' test:
 
Document Path:          /hellos/
Document LENGTH:        13 bytes
Requests per second:    45.25 [#/sec] (mean)
 
The template hello world test:
 
Document Path:          /hellos/hello
Document LENGTH:        931 bytes
Requests per second:    39.90 [#/sec] (mean)
 
The simple database bold;">QUERY test (bold;">USING the ORM):
 
Document Path:          /hellos/hello_db
Document LENGTH:        2469 bytes
Requests per second:    38.80 [#/sec] (mean)

However, with the help of APC things finally begin to look reasonable:

APC Enabled – APC Caching Engine (in core.php)

The RETURN 'hello world' test:
 
Document Path:          /hellos/
Document LENGTH:        13 bytes
Requests per second:    140.98 [#/sec] (mean)
 
The template hello world test:
 
Document Path:          /hellos/hello
Document LENGTH:        931 bytes
Requests per second:    120.04 [#/sec] (mean)
 
The simple database bold;">QUERY test (bold;">USING the ORM):
 
Document Path:          /hellos/hello_db
Document LENGTH:        2469 bytes
Requests per second:    117.56 [#/sec] (mean)

As you can see, APC increases CakePHP’s speed about threefold in these examples. These results may or may not be acceptable to you (remember that we are running these tests with a concurrency of 10), but I felt much better about CakePHP after pairing it with APC.

Kohana v3.0 – kohana3_test_src.zip

Kohana (a divergent of CodeIgnitor) is a much more lightweight PHP framework than CakePHP, but still gives you a pretty full featureset. Since v3.0 was released shortly after Round 1, I decided to use the new version for the tests here in Round 2.

It should be noted that although file caching is enabled in a default CakePHP install, Kohana’s caching is disabled by default. I have included a couple of tests with caching disabled here to exhibit the performance of a default Kohana install.

APC Disabled – Caching Disabled (default)

The RETURN 'hello world' test:
 
Document Path:          /hello/INDEX
Document LENGTH:        12 bytes
Requests per second:    86.73 [#/sec] (mean)

APC Disabled – Caching Enabled (bootstrap.php)

The RETURN 'hello world' test:
 
Document Path:          /hello/INDEX
Document LENGTH:        12 bytes
Requests per second:    180.50 [#/sec] (mean)
 
The template hello world test:
 
Document Path:          /hellos/INDEX
Document LENGTH:        930 bytes
Requests per second:    162.27 [#/sec] (mean)
 
The simple database bold;">QUERY test (bold;">USING the Database module):
 
Document Path:          /hellodb/INDEX
Document LENGTH:        2525 bytes
Requests per second:    113.92 [#/sec] (mean)
 
The simple database bold;">QUERY test (bold;">USING the ORM module):
 
Document Path:          /helloorm/INDEX
Document LENGTH:        2525 bytes
Requests per second:    99.73 [#/sec] (mean)

As you can see, even without APC Kohana is quite a bit faster than CakePHP.

APC Enabled – Caching Disabled

The RETURN 'hello world' test:
 
Document Path:          /hello/INDEX
Document LENGTH:        12 bytes
Requests per second:    125.93 [#/sec] (mean)

APC Enabled – Caching Enabled (bootstrap.php)

The RETURN 'hello world' test:
 
Document Path:          /hello/INDEX
Document LENGTH:        12 bytes
Requests per second:    280.67 [#/sec] (mean)
 
The template hello world test:
 
Document Path:          /hellos/INDEX
Document LENGTH:        930 bytes
Requests per second:    253.50 [#/sec] (mean)
 
The simple database bold;">QUERY test (bold;">USING the Database module):
 
Document Path:          /hellodb/INDEX
Document LENGTH:        2525 bytes
Requests per second:    173.28 [#/sec] (mean)
 
The simple database bold;">QUERY test (bold;">USING the ORM module):
 
Document Path:          /helloorm/INDEX
Document LENGTH:        2525 bytes
Requests per second:    153.90 [#/sec] (mean)

Although not a threefold gain as with CakePHP, Kohana with APC+caching definitely makes a good attempt at shortening the gap seen in Round 1 between the PHP frameworks and the offerings from other languages. Of course, if raw speed is what you’re looking for, there are faster PHP frameworks available (note that the benchmarks on that page were run on a completely different setup and should not be compared with any of the numbers here).

Next, let’s take a look at how APC enhances the performance of a few of PHP’s heavyweights.

WordPress v2.8.4 (default install)

Although I’ve heard that WordPress has a built-in caching setting, a quick glance at wp-settings.php did not reveal anything. So, there are simply two WordPress tests—with APC, and without:

APC Disabled

Document Path:          /
Document LENGTH:        5519 bytes
Requests per second:    24.80 [#/sec] (mean)

APC Enabled

Document Path:          /
Document LENGTH:        5519 bytes
Requests per second:    74.00 [#/sec] (mean)

Joomla v1.5.14 (default install, without sample data)

The results of these tests were quite disappointing to me, and made me wonder if I was doing something wrong. Sadly, it looks as though Joomla is quite the slow one of the bunch.

APC Disabled

Joomla Caching Disabled (DEFAULT):
 
Document Path:          /
Document LENGTH:        4006 bytes
Requests per second:    27.90 [#/sec] (mean)
 
Joomla Caching Enabled (File):
 
Document Path:          /
Document LENGTH:        4006 bytes
Requests per second:    31.46 [#/sec] (mean)

APC Enabled

Joomla Caching Disabled:
 
Document Path:          /
Document LENGTH:        4006 bytes
Requests per second:    41.27 [#/sec] (mean)
 
Joomla Caching Enabled (APC):
 
Document Path:          /
Document LENGTH:        4006 bytes
Requests per second:    45.01 [#/sec] (mean)

Drupal v6.14 (default install, clean URLs disabled)

Drupal has several caching settings, so I decided to see what each one would look like and how much of a difference “aggressive” caching made.

APC Disabled

Drupal Caching Disabled (DEFAULT):
 
Document Path:          /
Document LENGTH:        5548 bytes
Requests per second:    36.72 [#/sec] (mean)
 
"Normal" Drupal Caching:
 
Document Path:          /
Document LENGTH:        5548 bytes
Requests per second:    289.69 [#/sec] (mean)
 
"Aggressive" Drupal Caching:
 
Document Path:          /
Document LENGTH:        5548 bytes
Requests per second:    335.68 [#/sec] (mean)

APC Enabled

Drupal Caching Disabled:
 
Document Path:          /
Document LENGTH:        5548 bytes
Requests per second:    83.93 [#/sec] (mean)
 
"Normal" Drupal Caching:
 
Document Path:          /
Document LENGTH:        5548 bytes
Requests per second:    498.37 [#/sec] (mean)
 
"Aggressive" Drupal Caching:
 
Document Path:          /
Document LENGTH:        5548 bytes
Requests per second:    564.48 [#/sec] (mean)

As you can see, Drupal’s caching (especially when combined with APC) changes things drastically. These results seem to indicate that Drupal’s caching does a great job of allowing the CMS to get out of the way.

Closing thoughts

• The benefits of a PHP accelerator are really too good to pass up. Even without written-in caching, an accelerator can give your code a significant boost.
• While these tests have only really explored the impact on website page-load times, a PHP accelerator typically also cuts down on the amount of memory used by your PHP scripts.
• PHP 6 has promised to fix a lot of things, including the lack of built-in bytecode caching. It will be interesting to revisit these tests in a few years (?) to see how much optimization the new version brings to the table.

Can’t get enough of this madness? Just wait until you read Round 3!

(Note: Please don’t leave benchmarking requests as comments. If you’d like me to take a look at your favorite framework, direct message me on twitter. Thanks!)