# SQLObject and Database Programming in Python

### Ian Bicking

#### Imaginary Landscape

# The Ongoing Example

## Address book:

- `person`:
  - `first_name`
  - `last_name`
  - `email` (many emails for one person)
- `email`:
  - `type`
  - `address`

# The DB API

Database programming in Python is based on the DB-API. Some supported databases:

- MySQL
- PostgreSQL (multiple drivers)
- SQLite
- Firebird (Interbase)
- MAXDB (SAP DB)
- Oracle, Sybase, MS SQL Server

# DB API Example

Example:

    import MySQLdb
    conn = MySQLdb.connect(
        db='test', username='ianb')
    cur = conn.cursor()
    cur.execute(
        "SELECT id, first_name, last_name FROM person")
    result = cur.fetchall()
    for id, first_name, last_name in result:
        print "%2i %s, %s" % (id, last_name, first_name)

# DB API

- Import the database module (`MySQLdb`, `psycopg`, `sqlite`, etc)
- Use `module.connect(...)` to create a connection.
- Use `connection.cursor()` to get a cursor. Cursors do all the work.
- Use `cursor.execute(sql_query)` to run something.
- Use `cursor.fetchall()` to get results.

# DB API problems

1. Connections and cursors are tedious.
2. The databases don't all work the same; multiple SQL dialects.
3. Lots of time spent writing SQL.
4. Database updates can effect all the SQL you've written.
5. Unless you make abstractions...

# Abstractions

- Every significant database application has a database abstraction layer.
- You write one whether you mean to or not.

# Object-Relational Mappers

The table:

person
id | first_name | last_name
--- | --- | ---
1 | John | Doe
2 | Tom | Jones
...

# ORMs: Classes

The class:

person
id | first_name | last_name
--- | --- | ---
1 | John | Doe
2 | Tom | Jones
...
    class Person(SQLObject):
        # id is implicit
        first_name = StringCol()
        last_name = StringCol()

# ORMs: Instances

An instance:

person
id | first_name | last_name
--- | --- | ---
1 | John | Doe
2 | Tom | Jones
...
    >>> john = Person.get(1)
    >>> john.first_name
    'John'

# ORM: summary

- Every table is a class
- Every row is an instance
- Columns become attributes

# Usage

    __connection__ = 'postgres://pgsql@localhost/test'
    class Person(SQLObject):
        first_name = StringCol()
        last_name = StringCol()
        emails = MultipleJoin('Email')

# Usage (classes)

    class Email(SQLObject):
        person = ForeignKey('Person')
        type = EnumCol(['home', 'work'])
        address = StringCol()

# Usage (creating tables)

    def createTables():
        for table in (Person, Email):
            table.createTable(ifNotExists=True)

# Usage (instances)

Using your classes:

    >>> john = Person(first_name='John', last_name='Doe')
    >>> email = Email(person=john, type='home',
    ...     address='john@work.com')
    >>> john.emails
    []

# Usage (instances)

    >>> tom = Person(first_name='Tom', last_name='Jones')
    >>> tom is Person.get(tom.id)
    True
    >>> list(Person.selectBy(first_name='John'))
    []

# Defining Classes

- The class name matches the table name
- The attributes match the column names
- (kind of...)

# Defining Classes...

You can add extra methods:

    class Person(SQLObject):
        ...
        def _get_name(self):
            return self.first_name + ' ' + self.last_name

    >>> tom.name
    'Tom Jones'

# Automatic properties

- `_get_attr()` methods are called whenever the `obj.attr` attribute is called
- `_set_attr()` methods are called whenever the `obj.attr = value` statement is run
- `_set_attr()` is optional

# Defining classes...

You can override columns:

    class Person(SQLObject):
        ...
        last_name_searchable = StringCol()
        def _set_last_name(self, value):
            self._SO_set_last_name(self, value)
            self.last_name_lower = re.sub(r'[^a-zA-Z]', '', value).lower()

# Defining classes...

You can fiddle with the naming:

    class Person(SQLObject):
        _table = "people"
        first_name = StringCol(dbName="fname")
        ...

# Connecting classes

Foreign Keys:

    class Email(SQLObject):
        person = ForeignKey('Person')
        ...

Note we use a string for `'Person'`.

# Backreferences

The other side of one-to-many:

    class Person(SQLObject):
        ...
        emails = MultipleJoin('Email')

# Many-to-many

Many to many relationships imply a "hidden" table:

    class Address(SQLObject):
        people = RelatedJoin('Person')
        ...
    class Person(SQLObject):
        addresses = RelatedJoin('Address')

# Many-to-many...

The intermediate table created:

    CREATE TABLE address_person (
      address_id INT NOT NULL,
      person_id INT NOT NULL
    );

# Lazy classes

SQLObject can use reflection to figure out what columns your table has:

    class Person(SQLObject):
        _fromDatabase = True

You can start with `_fromDatabase = True` and add in explicit columns, overriding defaults.

# Instances

- Use `Person.get(id)` to retrieve a row
- Use `Person(first_name=...)` to insert a row (the new object is returned)
- Use `aPerson.destroySelf()` to delete a row

# Updating

- `aPerson.first_name = "new_name"`
  `UPDATE` is executed immediately
- Update multiple columns at once:
  `aPerson.set(first_name="new_fname", last_name="new_lname")`

# Selecting

You can use Python expressions (kind of):

    query = Person.q.first_name == "Joe"
    Person.select(query)

# Selecting...

- `Class.q.column_name` creates a magical object that creates queries.
- `sqlrepr()` turns these query objects into SQL (mostly hidden)
- `sqlrepr(Person.q.first_name == "Joe", 'mysql')` creates the SQL `person.first_name = 'Joe'`

# Selecting...

Complicated joins are possible:

    Person.select((Person.q.id == Email.q.personID)
                  & (Email.q.address.startswith('joe')))

Becomes:

    SELECT person.id, person.first_name, person.last_name
    FROM person, email
    WHERE person.id = email.person_id
          AND email.address LIKE 'joe%'

# SelectResult

Select results are more than just lists:

- You can slice them, and `LIMIT` and `OFFSET` statements are added to your query
- You can iterate through them, and avoid loading all objects into memory
- You can do things like `select_result.count()` to run aggregate functions
- To get a real list, you must do `list(select_result)`

# Caching

- SQLObject caches everything
- Maybe too much (bad if you have multiple updaters)
- Needs to because there's no joins
- Has potential to be faster than ad hoc queries
- Thread safety or thread confusion?

# Performance

- Not good if you are dealing with lots and lots of rows
- Inserts and selects substantially slower
- But for many applications you'll pay that price sooner or later anyway

# The problem with ORMs

ORMs live in the world between Object Oriented programmers (and programs) and Relational programmers (and programs). Neither will be happy.

# ORM Mismatch

- Python classes aren't tables
- Python instances aren't rows

# ORM Mismatch: Classes

- Databases don't have inheritance
- Even when they do, no one uses it
- Some tables are too boring (e.g., intermediate tables for many-to-many relationships), or don't warrant the overhead

# ORM Mismatch: Instances

- Instances get garbage collected, rows are forever
- Rows exist even when instances don't
- Instances may still exist when the row is gone
- In the relational calculus, rows don't have real identity

# ORM Mismatch: Relations

- Things like joins don't make sense for objects, but are central to relations
- Views and stored procedures also don't make sense
- Aggregate functions and set operations aren't natural in Python (the `for` loop is natural)

# Solving the question by avoidance

- SQLObject doesn't let you forget SQL
- You still have to think relationally
- But many relational concepts don't work either
- Advantage: it's easy

# Solving the Mismatch

SQLObject's answer: don't try too hard.

- There exist database restrictions; not every database schema maps well. (But most are fine)
- Sometimes you have to do things more slowly or more imperatively, compared to a purely relational technique
- There exist object restrictions; not every object-oriented concept maps well; e.g., no inheritance
- Inheritance isn't planned; this isn't transparent persistence

# Some other alternatives

Several Python ORMs exist; contrasts:

- SQLObject doesn't try to be too OO, so it works well with normal database schemas
- SQLObject doesn't have a compile step
- SQLObject isn't just a SQL wrapper
- SQLObject is reasonably complete

# Other alternatives...

See the Python Wiki at: python.org/moin
The page is HigherLevelDatabaseProgramming.