Curia

How to use multiple databases in TurboGears 2.0

by Seth on Jul.30, 2009, under Python, Web Development

I recently had to setup a special marketing web-portal for a client of mine that would collect some basic information and throw it into a database to be retrieved later. Since I’ve already got most of the client’s web-stuff on TG2 (and in an effort to keep things DRY), I decided I’d just add a controller for the new pages to TG and use some mod_proxy kung-fu to make it look like it all lived autonomously. Easy enough, right?

Well, about halfway through this process I decided I wanted to have the collected information dump into its own SQLite DB, keeping it safely away from the rest of my client’s data. I had heard that setting up multiple databases in TG2 was supposed to be easy, and with some help from Google I soon ran across this thread on the ML. In it, Chris supplies some very helpful example code which Mike subsequently posted on his blog as a nice tutorial. However, neither of these resources were exhaustive enough to achieve what I was looking for without a bit of “stumbling around”, so in an effort to be overly verbose (and perhaps unnecessarily repetitive) I’ve decided to post what I hope will be a more comprehensive run-down of how to accomplish this task.

Disclaimer: I am in no way a TurboGears or SQLAlchemy expert. There’s probably an easier/better way to do this, but since there’s no “official” TurboGears tutorial on this topic yet I’m afraid this is the best method I’ve found so far. If anyone reading this knows a better way to implement this kind of thing, please leave a comment and I will update this post as the suggestions come in.

Step 1: Define your database urls in the [app:main] section of your .ini file(s)

This is where the magic begins. Instead of one simple sqlalchemy.url = assignment, you’ve got to create assignments for each of the databases you want to use:

# in myapp/development.ini (or production.ini, or whatever.ini you are using)
 
#sqlalchemy.url = sqlite:///%(here)s/devdata.db
sqlalchemy.first.url = sqlite:///%(here)s/database_1.db
sqlalchemy.second.url = sqlite:///%(here)s/database_2.db

Step 2: Change the way your app loads the databases

Now we need to instruct the app to load the multiple databases correctly. This requires telling base_config (in app_cfg.py) to load our own custom AppConfig with the proper multi-db assignments and a call to the model’s init_model method (more on that in Step 3):

# in myapp/config/app_cfg.py
 
# make sure these imports are added to the top
from tg.configuration import AppConfig, config
from pylons import config as pylons_config
from myapp.model import init_model
 
# add this before base_config =
class MultiDBAppConfig(AppConfig):
    def setup_sqlalchemy(self):
        """Setup SQLAlchemy database engine(s)"""
        from sqlalchemy import engine_from_config
        engine1 = engine_from_config(pylons_config, 'sqlalchemy.first.')
        engine2 = engine_from_config(pylons_config, 'sqlalchemy.second.')
        # engine1 should be assigned to sa_engine as well as your first engine's name
        config['pylons.app_globals'].sa_engine = engine1
        config['pylons.app_globals'].sa_engine_first = engine1
        config['pylons.app_globals'].sa_engine_second = engine2
        # Pass the engine to init_model, to be able to introspect tables
        init_model(engine1, engine2)
 
#base_config = AppConfig()
base_config = MultiDBAppConfig()

Step 3: Update your model’s __init__ to handle multiple sessions and metadata

Switching the model’s init from a single-db config to a multi-db simply means we have to duplicate our DBSession and metata assignments, and then update the init_model method to assign/configure each engine correctly:

# in myapp/model/__init__.py
 
# after the first maker/DBSession assignment, add a 2nd one
maker2 = sessionmaker(autoflush=True, autocommit=False,
                     extension=ZopeTransactionExtension())
DBSession2 = scoped_session(maker2)
 
# after the first DeclarativeBase assignment, add a 2nd one
DeclarativeBase2 = declarative_base()
 
# uncomment the metadata2 line and assign it to DeclarativeBase2.metadata
metadata2 = DeclarativeBase2.metadata
 
# finally, modify the init_model method to allow both engines to be passed (see step 2) and
# assign the sessions and metadata to each engine
def init_model(engine1, engine2):
    """Call me before using any of the tables or classes in the model."""
 
#    DBSession.configure(bind=engine)
    DBSession.configure(bind=engine1)
    DBSession2.configure(bind=engine2)
 
    metadata.bind = engine1
    metadata2.bind = engine2

Step 4: Have your model classes inherit from either DeclarativeBase or DeclarativeBase2, and use DBSession or DBSession2 as your database session handler

Now that the configuration has all been taken care of, you can instruct your models to use either the first or second DeclarativeBase and DBSession based on what DB engine you want it to access:

# myapp/model/spam.py (uses engine1)
from sqlalchemy import Table, ForeignKey, Column
from sqlalchemy.types import Integer, Unicode, Boolean
from myapp.model import DeclarativeBase
 
class Spam(DeclarativeBase):
    __tablename__ = 'spam'
 
        self.id = id
        self.variety = variety
 
    id = Column(Integer, autoincrement=True, primary_key=True)
    variety = Column(Unicode(50), nullable=False)
 
#---------------------------------------------------------------------#
 
# myapp/model/eggs.py (uses engine2)
from sqlalchemy import Table, ForeignKey, Column
from sqlalchemy.types import Integer, Unicode, Boolean
from myapp.model import DeclarativeBase2
 
class Eggs(DeclarativeBase2):
    __tablename__ = 'eggs'
 
    def __init__(self, id, pkg_qty):
        self.id = id
        self.pkg_qty = pkg_qty
 
    id = Column(Integer, autoincrement=True, primary_key=True)
    pkg_qty = Column(Integer, default=12)

Optional: Create and populate each database in websetup.py

If you want your setup_app method to populate each database with data, simply use the appropriate metadata/DBSession objects as you would in a single-db setup:

# in myapp/websetup.py
def setup_app(command, conf, vars):
    """Place any commands to setup myapp here"""
    load_environment(conf.global_conf, conf.local_conf)
    # Load the models
    from myapp import model
    print "Creating tables for engine1"
    model.metadata.create_all()
    print "Creating tables for engine2"
    model.metadata2.create_all()
 
    # populate spam table
    spam = [
        model.Spam(1, u'Classic'),
        model.Spam(2, u'Golden Honey Grail'),
    ]
    # DBSession is bound to the spam table
    model.DBSession.add_all(spam)
 
    # populate eggs table
    eggs = [
        model.Eggs(1, 12),
        model.Eggs(2, 6),
    ]
    # DBSession2 is bound to the eggs table
    model.DBSession2.add_all(eggs)
 
    model.DBSession.flush()
    model.DBSession2.flush()
    transaction.commit()
    print "Successfully setup"

Hope this helps! If you have any questions or suggestions please leave a comment below.

:

1 Comment for this entry

  • Ants Aasma

    You don’t necessarily need separate sessions for the different databases. If the session isn’t explicitly bound to an engine it will try to find the engine via the mapper, by looking at bind of the metadata of the mapped table. By default declarative_base() will create its own metadata. This means that just binding the two metadatas to the engines and leaving the session unbound will do what you want, objects and queries will get routed to the correct database and connections acquired on each database as needed. Lazy loading of relations will also work across databases, but joins won’t – each query needs to go to one database. It will even manage coordinating two-phase transactions across the two databases for you if you manage the after crash cleanup of prepared-but-not-commited transactions on your own. Only gotcha might be that you’ll need to provide a mapper or a class to use for engine resolution if you execute plain-text SQL.

Leave a Reply

Looking for something?

Use the form below to search the site:

Still not finding what you're looking for? Drop a comment on a post or contact us so we can take care of it!

Visit our friends!

A few highly recommended friends...

Archives

All entries, chronologically...