A lot of work has been going into the TG docs in preparation for the upcoming v2.1 launch, and one little gem that was recently added was the Pagination Quickstart (thanks Lukas). While this sent me down the right path towards pagination perfection, there were a few things that came up that I felt like deserved further explanation:

1. The @paginate() decorator is an extremely simple wrapper for the paginate webhelper module.
2. When paginating SQLAlchemy queries, paginate runs a duplicate count() query to figure out its current position in the data collection. While the magic of this is nice, the possible performance issues with duplicating complex SQL queries on a high-traffic website could be undesirable.
3. It’s common to have controller methods dynamically switch to a “feed template” when doing AJAX pagination. Don’t worry, paginate + override_template makes this a piece of cake!

The @paginate() decorator makes pagination a breeze

The @paginate decorator is a wrapper for the paginate module that can be added to your controller methods. Simply specify your data collection’s name as the first argument of the decorator, and pagination is “automagically” setup for you.

An example controller method with pagination would look like this:

# dead-simple pagination in TurboGears 2
 
@expose('myproject.templates.posts')
@paginate("posts")
def posts(self):
    posts = DBSession.query(Posts)
    return dict(posts=posts)

Really? That was easy!

The documentation on this decorator seems quite clear to me, so instead of repeating its content here I will simply give you a link for further reading.

SQLAlchemy pagination performance wizardry

The main issue here is that the paginate module has to figure out its current position in the data collection in order to be able to spit out the right “chunk” of data (as well as to provide you with fancy stuff like “page 3 of 6″). So, by default when you give it an SQLAlchemy query object, it runs a count() query to figure this out. This results in two practically identical SQL queries (one to get the total count, and the other to return your actual collection of data). While this may not be an issue on simple queries, duplicating complex queries on a high-traffic website could be disastrous.

Remember, magic can be unpredictable, so before we dive into this topic there are a few things you should take into consideration:

• This technique is not compatible with the @paginate() decorator due to the fact that the current page arg isn’t passed into the controller method when using the decorator, so be ready to get your hands dirty with the full-blown paginate module.
• The SQLAlchemy all() query method results in immediate query execution. This may have undesirable side-effects.
• I’m sure there is a cleaner/sexier way to do this, but for the sake of time and in an effort to be verbose this is what I have come up with so far. Please leave a helpful comment below if your magic is cleaner than mine.

Ok, so now that we are familiar with the problem, what is a good solution? Well, a rough hack to work around this issue is to use SQLAlchemy’s limit() and offset() query methods combined with the all() method to get SQLAlchemy to return a list instead of a query object. This puts paginate to work on the list instead of on the database, avoiding the extra count() query. However, as tricky as this may be, there are still problems:

1. In order for paginate to calculate a correct current position based on the “page” it thinks it’s on, you’ve got to pad the beginning of the list you send to the paginate.Page() method with enough items to accurately reflect the “current” size of the data collection.
2. In order for paginate to know there are more records waiting to be loaded, you’ve got to pad the end of the list you send to the paginate.Page() method with at least one more item than each page’s specified item limit.

This may seem confusing at first, but an example controller method should help clear things up for you:

# import the paginate webhelper
from webhelpers import paginate
# specify the number of items you want per page
items_per_page = 20
 
@expose('myproject.templates.posts')
def posts(self, page=1):
    # specify the limit, plus 1 so the Page() method can tell if extra data is available
    limit = items_per_page + 1
    # specify the offset to start with
    offset = (int(page) - 1) * items_per_page
 
    # if the offset is greater than 0, we have to pad the beginning of the list
    if offset: posts = [x for x in xrange(offset)]
    # otherwise we're starting at the beginning (with an empty list)
    else: posts = list()
 
    # load the DB data into the list
    posts.extend(DBSession.query(Posts).limit(limit).offset(offset).all())
 
    # setup the pagination object
    pagination = paginate.Page(posts, page, items_per_page=items_per_page)
    # either use tmpl_context or return the object in the return dict
    tmpl_context.pagination = pagination
 
    return dict(posts=pagination.items)

Voilà! Now that you’re dealing with a list, pagination should work as expected without the extra “count()” query.

Use override_template() for dynamic template rendering

AJAX pagination is a common practice and often provides a richer user experience than “static” pagination. Fortunately, AJAX pagination is almost as easy to setup as normal pagination, and only requires a few extra tweaks to an already simple configuration.

The only “tricky” thing here is if you’re accessing the same controller method for partial data as well as the full page. In such a case, you’ll need to have the controller method render two separate templates depending on if you’re asking for the “full” page, or just the listing of data. Don’t worry though, paginate automatically provides you with a “partial” arg that you can check against, and override_template() allows you to override whatever template you have defined in the @expose() decorator.

Modifying the previous example’s controller method for this type of functionality looks like this:

# add the override_template import
from tg import override_template
 
# add the "partial" arg to the controller method
def posts(self, page=1, partial=0):
    # ...snip...
 
    if partial:
        override_template(self.posts, 'mako:myproject.templates.post_feed')
 
    return dict(posts=pagination.items)

More information on override_template() can be found in the docs.

I hope this tutorial has been helpful. I will be submitting a TG docs pull request soon, so please leave comments if you have any suggestions or feedback.

[Update 10/6: Slight updates to the Pylons test after some input from Ben. Added a couple new charts for you visual folks.]

As I briefly mentioned in Round 1, this whole thing came about as an experiment to satisfy my own curiosity. Unfortunately, I wasn’t expecting these posts to draw the amount of attention they have been getting, and several people informed me of a few “issues” with the first round. Since my initial approach to this topic was somewhat casual, I didn’t really take the time to perform each test in a “proper scientific fashion.” Although this was clearly stated in the introduction to round one, it unfortunately resulted in performance estimations that were somewhat less than accurate.

After input from various people much smarter than myself, I quickly went to work tweaking my test environment and building “proper” test apps. In the midst of this, a conversation about PHP accelerators prompted me to put PHP under the spotlight, which brought about Round 2 as an interim round. This gave me a chance to demonstrate the necessity of PHP acceleration, and only continued to solidify my opinion of PHP as an inferior web development language (remember, I just said my opinion).

Which brings us to Round 3. A lot of work has gone into “doing it right” this time, so I am fairly confident that these results are a much more accurate representation of each test subject’s performance estimations. Remember, benchmark test code typically has no real-world value, so “performance estimations” are about all I can promise here. Your mileage will vary. As a wise person once said:

“All this benchmarking is doing is proving what we already know: More code takes longer to execute.” – Ben Bangert (dev lead of Pylons)

What you should know about Round 3:

• The hardware/software platform is the same as in Round 1.
• Python v2.5.4 was used for all the Python tests. Ruby v1.8.6 was used for the Rails test.
• Apache v2.2.3 was used, with mod_wsgi v2.5 for the Python tests and Phusion Passenger v2.2.5 for the Rails test.
• Note that while the mod_wsgi tests in Round 1 were using “daemon mode”, the mod_wsgi tests here are using “embedded mode” which is often faster (although not necessarily the recommended configuration).
• The Python component versions used here were: SQLAlchemy v0.5.6, Genshi v0.5.1, Mako v0.2.5, and Jinja 2.2.1.
• ApacheBench was run with -n 10000 -c 10.
• Each test was run several times to make sure that there were no anomalies, with the “optimum average” chosen as the numbers represented here.
• In an attempt to try to make these tests more scientific, I am now including the source code for each test.

Plain WSGI – wsgi_test_src.zip

Let’s start with a basic “hello world” WSGI test to set the baseline:

The 'hello world' test:
 
WSGIServer/0.1
Document Path:          /
Document Length:        12 bytes
Requests per second:    1532.76 [#/sec] (mean)
 
 
Apache/2.2.3 (mod_wsgi)
Document Path:          /
Document Length:        12 bytes
Requests per second:    5549.73 [#/sec] (mean)

As you can see, mod_wsgi allows Apache to put some serious muscle into Python’s WSGI. It’s no wonder that mod_wsgi is quickly becoming the de-facto standard for Apache Python deployment.

TurboGears v2.0.3 – tg2_test_src.zip

As I mentioned in Round 1, TurboGears has quickly become my web framework of choice. Building a “best-of-breed” component stack on top of Pylons may not be easy, but when it works I believe it pays off immensely (e.g. Ubuntu).

The 'hello world' test:
 
Document Path:          /
Document Length:        12 bytes
Requests per second:    935.13 [#/sec] (mean)
 
 
Genshi template test:
 
Document Path:          /genshi_hello/
Document Length:        923 bytes
Requests per second:    601.07 [#/sec] (mean)
 
 
Mako template test:
 
Document Path:          /mako_hello/
Document Length:        932 bytes
Requests per second:    723.18 [#/sec] (mean)
 
 
Jinja2 template test:
 
Document Path:          /jinja_hello/
Document Length:        937 bytes
Requests per second:    764.35 [#/sec] (mean)
 
 
Database query tests:
 
Document Path:          /raw_sql/
Document Length:        1270 bytes
Requests per second:    569.71 [#/sec] (mean)
 
Document Path:          /genshi_sql/
Document Length:        2409 bytes
Requests per second:    388.96 [#/sec] (mean)
 
Document Path:          /mako_sql/
Document Length:        2418 bytes
Requests per second:    515.07 [#/sec] (mean)
 
Document Path:          /jinja_sql/
Document Length:        2472 bytes
Requests per second:    535.06 [#/sec] (mean)

TurboGears v2.1a1 – tg21_test_src.zip

The TurboGears community is alive and well, and just before finishing this post the release of v2.1 Alpha 1 was announced. Significant improvements were made to TG’s object dispatch, so I was excited to see what the new numbers would look like.

The 'hello world' test:
 
Document Path:          /
Document Length:        11 bytes
Requests per second:    1118.92 [#/sec] (mean)
 
 
Genshi template test:
 
Document Path:          /genshi_hello/
Document Length:        923 bytes
Requests per second:    713.97 [#/sec] (mean)
 
 
Mako template test:
 
Document Path:          /mako_hello/
Document Length:        932 bytes
Requests per second:    812.64 [#/sec] (mean)
 
 
Jinja2 template test:
 
Document Path:          /jinja_hello/
Document Length:        937 bytes
Requests per second:    1000.98 [#/sec] (mean)
 
 
Database query tests:
 
Document Path:          /raw_sql/
Document Length:        1270 bytes
Requests per second:    673.04 [#/sec] (mean)
 
Document Path:          /genshi_sql/
Document Length:        2409 bytes
Requests per second:    433.45 [#/sec] (mean)
 
Document Path:          /mako_sql/
Document Length:        2418 bytes
Requests per second:    601.89 [#/sec] (mean)
 
Document Path:          /jinja_sql/
Document Length:        2472 bytes
Requests per second:    630.66 [#/sec] (mean)

As you can see, the future of TurboGears looks very bright, and I only expect these numbers to get better as the final 2.1 release approaches (expected in Q1 2010).

Pylons v0.9.7 – pylons_test_src.zip

Pylons is the foundation which TurboGears is built upon, and for those who don’t need all “the extras” it is an excellent choice.

The 'hello world' test:
 
Document Path:          /hello/index
Document Length:        12 bytes
Requests per second:    2593.47 [#/sec] (mean)
 
 
Mako template test:
 
Document Path:          /hello/hello
Document Length:        932 bytes
Requests per second:    1737.03 [#/sec] (mean)
 
 
Database query tests:
 
Document Path:          /hello/raw_sql
Document Length:        1270 bytes
Requests per second:    964.07 [#/sec] (mean)
 
Document Path:          /hello/hellodb
Document Length:        2418 bytes
Requests per second:    831.26 [#/sec] (mean)

Django v1.1 – django_test_src.zip

While Django is probably Python’s most popular web framework, I have never really clicked with it. Its way of doing things is significantly different than TG/Pylons, but if you don’t mind its “MTV” architectural pattern and can survive without SQLAlchemy (although there are hacks), it is a very capable framework.

The 'hello world' test:
 
Document Path:          /hello
Document Length:        12 bytes
Requests per second:    3376.48 [#/sec] (mean)
 
 
Django template test:
 
Document Path:          /hellos
Document Length:        936 bytes
Requests per second:    1781.43 [#/sec] (mean)
 
 
Database query test:
 
Document Path:          /hellodb
Document Length:        2476 bytes
Requests per second:    972.11 [#/sec] (mean)

Bottle 0.5.8 – bottle_test_src.zip

Bottle caught my attention a few months ago while surveying the WSGI landscape for an alternative to PHP when building websites that only require a few pieces of dynamic functionality. As a “micro-framework”, it does an excellent job at bridging the gap between “pure WSGI” and its “feature-rich” big brothers.

The 'hello world' test:
 
Document Path:          /
Document Length:        12 bytes
Requests per second:    5545.96 [#/sec] (mean)
 
 
Mako template test:
 
Document Path:          /hello
Document Length:        923 bytes
Requests per second:    3588.04 [#/sec] (mean)
 
 
Database query test (SQLAlchemy):
 
Document Path:          /hellodb
Document Length:        2413 bytes
Requests per second:    1479.54 [#/sec] (mean)

As you can see, there was practically no difference in speed between Bottle and pure WSGI in a basic “hello world” test. Even with the addition of Mako and SQLAlchemy, Bottle performed significantly faster than a bare Pylons or Django setup.

(Note: Mako template inheritance was not used in the Bottle Mako test.)

Ruby on Rails v2.3.3 – rails_test_src.zip

What web technology shootout would be complete without Rails? Although I am not a Rails fan (mostly because I can’t stand Ruby), I believe that in many ways we owe much of the popularity and success of the web framework movement to Rails.

The 'hello world' test:
 
Document Path:          /hello/hello
Document Length:        12 bytes
Requests per second:    1048.18 [#/sec] (mean)
 
 
Rails template test:
 
Document Path:          /hello/hellos
Document Length:        937 bytes
Requests per second:    949.54 [#/sec] (mean)
 
 
Database query test:
 
Document Path:          /hello/hellodb
Document Length:        2482 bytes
Requests per second:    734.18 [#/sec] (mean)

(Note: I think my version of Ruby was using a newer version of the SQLite3 library than my version of Python. This may or may not have given Rails an advantage on the database query test.)

For those of you who like charts, here’s a comparison of the “return hello world” test results for each framework:

The return hello world test chart

And here’s a comparison of the template test:

The template test chart

Finally, here’s the template test with the database query added:

Template test with database query

Closing thoughts

• Gone are the days of Rails as “the one framework to rule them all.” C’mon guys, it’s not 2005 anymore.
• Python has gotten a lot of flak from other language camps for its abundance of web frameworks. While some Pythonists have been embarrassed by this, I only see it as proof of Python’s success as a language. My retort would be: “If it was as easy to write a web framework in your language as it is in Python, you’d have the same problem!”

Coming soon: “Round 4: Micro-frameworks — Quick & Dirty, and leaving PHP in the dust!”

(Note: Please don’t leave benchmarking requests as comments. If you’d like me to take a look at your favorite framework, direct message me on twitter. Thanks!)

Recently I went on a benchmarking spree and decided to throw ApacheBench at a bunch of the different web development technology platforms I interact with on a day-to-day basis. The results were interesting enough to me that I decided I’d take a post to share them here.

Disclaimer: The following test results should be taken with a massive grain of salt. If you know anything about benchmarking, you will know that the slightest adjustments have the potential to change things drastically. While I have tried to perform each test as fairly and accurately as possible, it would be foolish to consider these results as scientific in any way. It should also be noted that my goal here was not to see how fast each technology performs at its most optimized configuration, but rather what a minimal out-of-the-box experience looks like.

Test platform info:

• The hardware was an Intel Core2Quad Q9300, 2.5Ghz, 6MB Cache, 1333FSB, 2GB DDR RAM.
• The OS was CentOS v5.3 32-bit with a standard Apache Webserver setup.
• ApacheBench was used with only the -n and -c flags (1000 requests for the PHP frameworks, 5000 requests for everything else).
• Each ApacheBench test was run 5-10 times, with the “optimum average” chosen as the numbers represented here.
• The PHP tests were done using the standard Apache PHP module.
• The mod_wsgi tests were done in daemon mode set to 2 processes/15 threads.
• The SQL tests were done with mysqli ($mysql->query()) on PHP, and SQLAlchemy (conn.execute()) on Python fetching and printing 5 rows of data from a sample database.

Apache v2.2.3

We will start with the raw Apache benchmark.

For this test, Apache loaded a simple HTML file with random text:

Document Length:        6537 bytes
Requests per second:    8356.23 [#/sec] (mean)

As expected, Apache is lightning fast.

Ok, so now that we’ve set the high water mark, let’s take a look at some popular web technology platforms…

PHP v5.2.8

For this test, I created a PHP file that simply printed the $_SERVER array and some random text:

Document Length:        6135 bytes
Requests per second:    2812.22 [#/sec] (mean)

Next, I decided to add some mysqli() code to connect to a database and print out a few rows of data:

Document Length:        283 bytes
Requests per second:    2135.46 [#/sec] (mean)

Not at all bad, but who really wants to build a webapp using plain old PHP anymore?

Next I decided to test the two PHP frameworks that I am most familiar with.

CakePHP v1.2.5

For this test, I did a basic CakePHP install, and added a route that printed “hello world” from a template:

Document Length:        944 bytes
Requests per second:    42.00 [#/sec] (mean)

As you can see, frameworks can be a blessing and a curse. On a side note, CakePHP was the only test that gave me failed requests.

Kohana v2.3.4

For this test, I did a basic install and loaded Kohana’s default welcome/index setup (which seemed lightweight enough to me to qualify as the test focal point):

Document Length:        1987 bytes
Requests per second:    100.11 [#/sec] (mean)

Kohana fared much better than CakePHP, but it appears as though PHP’s speed breaks down quickly once you start building a framework around it.

Python v2.5.4

Unlike PHP, Python wasn’t specifically created for the web. However, several years ago the WSGI spec was introduced which makes building web applications with Python a breeze. There are many different ways to depoly a WSGI app, but my tests were limited to Python’s built-in WSGIServer, and Apache’s mod_wsgi.

WSGIServer

For this test, I used WSGI.org’s test.wsgi:

Document Length:        4123 bytes
Requests per second:    1658.35 [#/sec] (mean)

These results were quite impressive to me, considering that this is pure Python at work.

mod_wsgi

mod_wsgi is Apache’s module to process WSGI scripts. This test used the exact same script as the previous WSGIServer example:

Document Length:        9552 bytes
Requests per second:    3994.30 [#/sec] (mean)

Don’t look now, but those results are better than the vanilla PHP test!

Next, I added SQLAlchemy imports to the test file and did a conn.execute(), in an attempt to try to mimic the PHP mysqli() test:

Document Length:        10212 bytes
Requests per second:    3379.87 [#/sec] (mean)

I expected to lose quite a bit of speed with the introduction of an ORM, but it looks like SQLAlchemy is quite fast. These results were still over 1,000reqs/s more than the PHP example.

TornadoServer

There’s been a lot of buzz lately about the new TornadoServer, so I thought I’d throw it in the mix. This test uses their basic “hello world” example:

Document Length:        12 bytes
Requests per second:    3330.00 [#/sec] (mean)

Nothing shocking here at first glance, but the Tornado is built to handle extreme loads. So, I increased the concurrency to 1,000 and threw 10,000 requests at it:

Document Length:        12 bytes
Concurrency Level:      1000
COMPLETE requests:      10000
Requests per second:    3288.15 [#/sec] (mean)

Almost exactly the same results! On a side note, none of the other tests subjects were able to handle a concurrency of 1,000.

TurboGears v2.0.3 (mod_wsgi)

TurboGears has quickly become my framework of choice, so naturally it was the first Python framework I tested.

For this test I setup a default quickstart project, and then added a hello() controller method which simply returned a “hello world” string:

Document Length:        11 bytes
Requests per second:    470.18 [#/sec] (mean)

Next, I added a simple Genshi template with 1 level of inheritance:

Document Length:        1351 bytes
Requests per second:    143.26 [#/sec] (mean)

While those results are not terrible, they should encourage you to check out the other templating options.

Next, I performed the same test with Mako:

Document Length:        1534 bytes
Requests per second:    394.41 [#/sec] (mean)

As you can see, Mako is significantly faster than Genshi.

TurboGears also supports Jinja2, so I thought I’d also see how it held up:

Document Length:        1020 bytes
Requests per second:    401.04 [#/sec] (mean)

I was not expecting Jinja2 to be faster than Mako, so I am not entirely sure about these results. However, the difference is only slight, and might have something to do with the fact that the dotted template names finder is disabled for Jinja templates.

Pylons v0.9.7 (mod_wsgi)

Mark Ramm (the current development lead of TurboGears) has been quoted as saying “TurboGears is to Pylons as Ubuntu is to Debian.” With that in mind, I naturally wanted to see how much speed I was losing with TurboGears’ “additions”.

For this test I tried to mimic the first TurboGears test (basic layout, returning a string):

Document Length:        11 bytes
Requests per second:    1961.00 [#/sec] (mean)

While I was expecting a gain from TurboGears’ omissions, I wasn’t quite expecting over four times the increase. I can only assume that the bare Pylons setup is pretty minimal, while TurboGears bootstraps a lot of stuff on for you by default.

Duplicating the Mako test in Pylons was also quite fast:

Document Length:        9743 bytes
Requests per second:    1183.39 [#/sec] (mean)

Bottle v0.5.7 (mod_wsgi)

Bottle caught my attention recently as a minimalistic Python web framework, so I thought I’d put it through the grind as well.

For this test, I used their simple “hello world” example:

Document Length:        12 bytes
Requests per second:    3912.43 [#/sec] (mean)

As you can see, Bottle is very close to bare WSGI as far as speed goes. On a side note, adding a sample template using Bottle’s default templating package didn’t seem to change these numbers at all.

Next I tested Bottle with its templating system switched to Mako:

Document Length:        41 bytes
Requests per second:    3134.61 [#/sec] (mean)

Then I added a SQLAlchemy and performed a query:

Document Length:        9374 bytes
Requests per second:    2503.27 [#/sec] (mean)

What was exciting to me about this test was that I found myself looking at a minimal Python framework equipped with a full templating system and ORM which all performed faster than the PHP+mysqli() test.

Django v1.1 (mod_wsgi)

I don’t use Django, but I thought it would be worth including in this report. This is a basic Django setup printing out a simple hello world template:

Document Length:        1471 bytes
Requests per second:    564.41 [#/sec] (mean)

As expected, a minimal Django setup is quite fast. (Note: This test included a SQLite database query.)

Ruby On Rails v2.2.3 (mod_passenger)

I also don’t use Rails, but I figured this report would be incomplete without Rails on the list. So, I took MediaTemple’s Rails test app and loaded up a test page:

Document Length:        1291 bytes
Requests per second:    433.57 [#/sec] (mean)

Note that the test app used here is doing a little bit more than just returning “Hello World”.

My Conclusions

As I said in the disclaimer, to arrive at any sort of decisive conclusions based on these numbers alone would be foolish. However, here is what all of this left me thinking:

• All of these technologies performed adequately for use with most projects.
• CakePHP was the only test that produced less than 100 requests per second. However, there are many articles available on optimizing CakePHP.
• These results made me wish I had moved away from PHP much earlier in my web development carrier. There are just better options these days when it comes to choosing a web development language (imho). If you must use PHP for a large project, I strongly suggest that you look into an accelerator.
• The Bottle+Mako+SQLAlchemy results struck me as an extremely minimal drop-in replacement for simple PHP sites.
• mod_wsgi is a huge winner in deploying Python webapps. Way to go Graham!
• Do your own bencharking, because your mileage will vary!

Like what you’ve seen here? Then why not check out Round 2?

(Note: Please don’t leave benchmarking requests as comments. If you’d like me to take a look at your favorite framework, direct message me on twitter. Thanks!)

Well, about halfway through this process I decided I wanted to have the collected information dump into its own SQLite DB, keeping it safely away from the rest of my client’s data. I had heard that setting up multiple databases in TG2 was supposed to be easy, and with some help from Google I soon ran across this thread on the ML. In it, Chris supplies some very helpful example code which Mike subsequently posted on his blog as a nice tutorial. However, neither of these resources were exhaustive enough to achieve what I was looking for without a bit of “stumbling around”, so in an effort to be overly verbose (and perhaps unnecessarily repetitive) I’ve decided to post what I hope will be a more comprehensive run-down of how to accomplish this task.

Disclaimer: I am in no way a TurboGears or SQLAlchemy expert. There’s probably an easier/better way to do this, but since there’s no “official” TurboGears tutorial on this topic yet I’m afraid this is the best method I’ve found so far. If anyone reading this knows a better way to implement this kind of thing, please leave a comment and I will update this post as the suggestions come in.

Step 1: Define your database urls in the [app:main] section of your .ini file(s)

This is where the magic begins. Instead of one simple sqlalchemy.url = assignment, you’ve got to create assignments for each of the databases you want to use:

# in myapp/development.ini (or production.ini, or whatever.ini you are using)
 
#sqlalchemy.url = sqlite:///%(here)s/devdata.db
sqlalchemy.first.url = sqlite:///%(here)s/database_1.db
sqlalchemy.second.url = sqlite:///%(here)s/database_2.db

Step 2: Change the way your app loads the databases

Now we need to instruct the app to load the multiple databases correctly. This requires telling base_config (in app_cfg.py) to load our own custom AppConfig with the proper multi-db assignments and a call to the model’s init_model method (more on that in Step 3):

# in myapp/config/app_cfg.py
 
# make sure these imports are added to the top
from tg.configuration import AppConfig, config
from pylons import config as pylons_config
from myapp.model import init_model
 
# add this before base_config =
class MultiDBAppConfig(AppConfig):
    def setup_sqlalchemy(self):
        """Setup SQLAlchemy database engine(s)"""
        from sqlalchemy import engine_from_config
        engine1 = engine_from_config(pylons_config, 'sqlalchemy.first.')
        engine2 = engine_from_config(pylons_config, 'sqlalchemy.second.')
        # engine1 should be assigned to sa_engine as well as your first engine's name
        config['pylons.app_globals'].sa_engine = engine1
        config['pylons.app_globals'].sa_engine_first = engine1
        config['pylons.app_globals'].sa_engine_second = engine2
        # Pass the engine to init_model, to be able to introspect tables
        init_model(engine1, engine2)
 
#base_config = AppConfig()
base_config = MultiDBAppConfig()

Step 3: Update your model’s __init__ to handle multiple sessions and metadata

Switching the model’s init from a single-db config to a multi-db simply means we have to duplicate our DBSession and metata assignments, and then update the init_model method to assign/configure each engine correctly:

# in myapp/model/__init__.py
 
# after the first maker/DBSession assignment, add a 2nd one
maker2 = sessionmaker(autoflush=True, autocommit=False,
                     extension=ZopeTransactionExtension())
DBSession2 = scoped_session(maker2)
 
# after the first DeclarativeBase assignment, add a 2nd one
DeclarativeBase2 = declarative_base()
 
# uncomment the metadata2 line and assign it to DeclarativeBase2.metadata
metadata2 = DeclarativeBase2.metadata
 
# finally, modify the init_model method to allow both engines to be passed (see step 2) and
# assign the sessions and metadata to each engine
def init_model(engine1, engine2):
    """Call me before using any of the tables or classes in the model."""
 
#    DBSession.configure(bind=engine)
    DBSession.configure(bind=engine1)
    DBSession2.configure(bind=engine2)
 
    metadata.bind = engine1
    metadata2.bind = engine2

Step 4: Have your model classes inherit from either DeclarativeBase or DeclarativeBase2, and use DBSession or DBSession2 as your database session handler

Now that the configuration has all been taken care of, you can instruct your models to use either the first or second DeclarativeBase and DBSession based on what DB engine you want it to access:

# myapp/model/spam.py (uses engine1)
from sqlalchemy import Table, ForeignKey, Column
from sqlalchemy.types import Integer, Unicode, Boolean
from myapp.model import DeclarativeBase
 
class Spam(DeclarativeBase):
    __tablename__ = 'spam'
 
        self.id = id
        self.variety = variety
 
    id = Column(Integer, autoincrement=True, primary_key=True)
    variety = Column(Unicode(50), nullable=False)
 
#---------------------------------------------------------------------#
 
# myapp/model/eggs.py (uses engine2)
from sqlalchemy import Table, ForeignKey, Column
from sqlalchemy.types import Integer, Unicode, Boolean
from myapp.model import DeclarativeBase2
 
class Eggs(DeclarativeBase2):
    __tablename__ = 'eggs'
 
    def __init__(self, id, pkg_qty):
        self.id = id
        self.pkg_qty = pkg_qty
 
    id = Column(Integer, autoincrement=True, primary_key=True)
    pkg_qty = Column(Integer, default=12)

Optional: Create and populate each database in websetup.py

If you want your setup_app method to populate each database with data, simply use the appropriate metadata/DBSession objects as you would in a single-db setup:

# in myapp/websetup.py
def setup_app(command, conf, vars):
    """Place any commands to setup myapp here"""
    load_environment(conf.global_conf, conf.local_conf)
    # Load the models
    from myapp import model
    print "Creating tables for engine1"
    model.metadata.create_all()
    print "Creating tables for engine2"
    model.metadata2.create_all()
 
    # populate spam table
    spam = [
        model.Spam(1, u'Classic'),
        model.Spam(2, u'Golden Honey Grail'),
    ]
    # DBSession is bound to the spam table
    model.DBSession.add_all(spam)
 
    # populate eggs table
    eggs = [
        model.Eggs(1, 12),
        model.Eggs(2, 6),
    ]
    # DBSession2 is bound to the eggs table
    model.DBSession2.add_all(eggs)
 
    model.DBSession.flush()
    model.DBSession2.flush()
    transaction.commit()
    print "Successfully setup"

Hope this helps! If you have any questions or suggestions please leave a comment below.

The goal was to get the form to go through two layers of validation before passing:

1. Use the validation packages provided by TG (tw.forms and formencode)
2. If the first layer of validation passes, try to run the authorize.net charge. If this returns a response code of 1 (approved) then all validation has passed. Otherwise, invalidate the form and flash the authorize.net error.

tw.forms and formencode are awesome packages, but at first glance there didn’t seem to be “one obvious way” to do things. I found the documentation and tutorials to be scant at best, and eventually went to the TurboGears mailing list for help. In spite of the fact that I’d probably give TurboGears a “C” at best for its documentation (not to mention the availability of “verbose” tutorials), the folks on the mailing list and IRC channel (when it’s actually active) seem to be quite helpful.

Eventually (and with additional help from Google’s source code search) I was able to hack together something that worked as planned, and was overjoyed to observe chained validators in action. Below is some example code provided for your hacking pleasure. To download the full example file, see the links at the bottom of the post.

# FancyValidator to process the Credit Card using the authorize package
class ProcessCard(validators.FancyValidator):
    def _to_python(self, value, state):
        from authorize import aim as aim_api
 
        # Setup the aim Api object.
        aim = aim_api.Api(AUTHNET_LOGIN, AUTHNET_KEY, is_test=False)
 
        # Create a transaction against a credit card
        result_dict = aim.transaction(
            amount=u"16.00",
            card_num=unicode(value['card_number']),
            exp_date=unicode(value['card_expiry']),
            # ...and others...
            )
 
        if result_dict['code'] == '1':
            # success
            return value
        else:
            # failure
            raise validators.Invalid(result_dict['reason_text'], value, state)
 
class AuthnetForm(TableForm):
    submit_text='Process Card'
 
    validator = validators.Schema(
        chained_validators = [
            validators.CreditCardValidator('card_type','card_number'),
            validators.CreditCardSecurityCode('card_type','card_cvv'),
            # you could also add an expiry validator, but authnet will handle this for you
            ProcessCard()
        ]
    )
 
    class fields(WidgetsList):
        name = TextField(validator=validators.String(not_empty=True))
        # ...and others like address, city, state, zip...
        spacer = Spacer(suppress_label=True)
        card_type = SingleSelectField(options=[('visa', 'Visa'), ('mastercard', 'Master Card'), ('discover', 'Discover'), ('amex', 'American Express')], validator=validators.NotEmpty)
        card_expiry = CalendarDatePicker(date_format="%m/%Y", validator=validators.NotEmpty)
        card_number = TextField(label_text='Card #', validator=validators.NotEmpty)
        card_cvv = TextField(label_text='CVV Code', validator=validators.NotEmpty)
 
authnet_form = AuthnetForm('authnet_form', action=url('/authnet/process/'))

Links of interest:

• Download the authnet.py file
• This post’s original Google group thread
• The Authorize package

Ok, let me back up before I get ahead of myself…

I’ve been a web developer for over 10 years now. When it comes to web programming languages and platforms, I’ve been around the block a bit: Back in the 90s I was building shopping carts from scratch using Perl (ouch!). From Perl, I eventually migrated to PHP which usually made things much easier. From time to time I would play around with Rails; but I never could stomach Ruby’s syntax (I think it reminded me too much of Perl). Even though I never felt like I really “clicked” with PHP, I eventually settled into a few PHP frameworks I liked and finally called myself a PHP developer.

Then one day I stumbled across Python, and it was love at first sight. I quickly vowed that I would soon be doing 99% of my web development in my newfound favorite language, but was disappointed to discover that (at that time) the available web frameworks were either ugly or poorly documented/supported. Reluctantly I returned to PHP, unable to justify moving away from the “quick & simple” nature of what I was used to with PHP and the frameworks I was used to.

I’m not one to give up easily though, and on a recent “re-visit” of the Python web framework scene I was excited to see that things had changed dramatically: Django’s community had exploded and its documentation looked rock-solid. Pylons had cropped up as a serious WSGI-based contender (and modeling quite a bit of it’s functionality after Rails). But what really caught my attention was the (at the time) upcoming TurboGears v2.0 release.

What made TG2 stand out to me was its attempt to build a “best-of-breed” module-set on top of Pylons. Mark Ramm (the current leader of TG development) has been quoted as comparing TG2 to Ubuntu in that  “TG 2 is to Pylons as Ubuntu is to Debian.” The sound of that intrigued me, and I started playing around with the TG 2 Betas (and liking what I was seeing!).

Recently one of my clients asked me to whip up a medium-sized project with various amounts of CRUD and Ajax. Ironically, that very same day TurboGears’ v2.0 release went gold. Seeing this as my opportunity to finally begin transitioning into the Python world with a stable “next-generation” Python web-development framework, I have decided to implement my latest project using TurboGears 2. Could this be the beginning of the rest of my life?

I will keep you posted on my progress and what I encounter along the way (good, bad, and ugly). Keep it here for what will likely be an interesting adventure.

Related links of interest:

• TurboGears
• Introducing Three Python Web Frameworks