Showing posts with label python. Show all posts
Showing posts with label python. Show all posts

Sunday, May 07, 2006

Using MySql 5.0 stored procedures with MySQLdb

Using MySql 5.0 stored procedures with MySQLdb

When I wanted to try out MySQL 5.0 stored procedures I didn't find too much on the web. This was a few months ago and there may be some better tutorials out there now but I figured I would share some of my tricks.

Why use stored procedures in the first place? While stored procedures give a you a performance boost (I have not benchmarked so I can't say how much) the argument I usually use when making the case for using stored procedures is abstraction. One can make changes to a stored procedure and not have to touch the calling code just as long as you don't break the interface. There have been many instances where I have avoided a code change - QA cycle - rollout just by adding an additional field to a select statement in a stored procedure.

Step 0: Install and configure MySql 5.0
MySql is at version 5.0.21 at the time of this writing. Installing MySql can be tricky, and going into all the detail of installing MySql is beyond the scope of this article, but here are some tips...

I usually install from source. The key is to follow the directions provided in the install notes and make sure that the directory the MySql installs to is owned by "mysql:mysql". Once the permissions are correct and the database has been installed, mysql usually fires right up.

Step 1: Installing MySQLdb.
Get the latest version of MySQLdb (1.2.1_p2 when I wrote this) and install...
$> tar zxfv MySQL-python-1.2.1_p2.tar.gz && cd MySQL-python-1.2.1_p2/
#> python setup.py install

Step 2: Create some tables and insert some data

mysql> use test;
mysql> create table products (
productid int unsigned not null primary key auto_increment,
categoryid int unsigned not null,
productname varchar(80) not null,
description varchar(255) not null,
statuscd tinyint(1) unsigned not null,
createdt datetime not null,
index cat_prod_idx
(categoryid, productid, productname, description, statuscd)
) type=innodb;

mysql> create table categories (
categoryid int unsigned not null primary key auto_increment,
category varchar(80) not null,
description varchar(255) not null,
statuscd tinyint(1) unsigned not null,
createdt datetime not null
) type=innodb;

mysql> insert into categories (category, description, statuscd, createdt)
values ('cool category', 'these are all cool products', 1, now());

mysql> insert into categories (category, description, statuscd, createdt)
values ('lame category', 'these are the not so cool products', 1, now());

mysql> insert into products (categoryid, productname, description, statuscd, createdt)
values (1, 'slackware linux', 'this is my distro of choice.', 1, now());

mysql> insert into products (categoryid, productname, description, statuscd, createdt)
values (1, 'MicroSoft Windows', 'see category for description.', 2, now());



Step 3: Build the 4 basic types of queries (select, insert, update and delete)

Select stored procedures are pretty simple the things to note are:
1. It is good to only bring back fields that you need 'select *'s' are evil
2. I prefixed the input parameter with an underscore so that it is easily identified by MySql and the next person that has to read this sp.


mysql> delimiter //
mysql> create procedure usp_get_products_by_category
(
_categoryid int unsigned
)
begin
select
p.productid,
p.productname,
p.description,
c.category,
c.description
from
products p
inner join categories c on p.categoryid = c.categoryid
where
p.categoryid = _categoryid
and p.statuscd = 1
and c.statuscd = 1;
end;
//
mysql> delimiter ;



When writing an insert sp you always want to return the id of the record you just created hence the "select LAST_INSERT_ID()". This means that you will get a one record result set back after you call the insert sp and the first and only field in the the record will be the id.


mysql> delimiter //
mysql> create procedure usp_ins_product
(
_categoryid int unsigned,
_productname varchar(80),
_description varchar(255),
_statuscd tinyint(1) unsigned
)
begin
insert into products
(categoryid, productname, description, statuscd, createdt)
values
(_categoryid, _productname, _description, _statuscd, now());
select LAST_INSERT_ID();
end;
//
mysql> delimiter ;



When I write my update and delete stored procedures I usually let MySql do it's thing. I always check the number of rows updated through the MySQLdb cursor object:
"count = curs.execute(stmt)"


mysql> delimiter //
mysql> create procedure usp_upd_productstatus
(
_productid int unsigned,
_statuscd tinyint(1) unsigned
)
begin
update products
set statuscd = _statuscd
where
productid = _productid;
end;
//
mysql> delimiter ;

mysql> delimiter //
mysql> create procedure usp_del_product
(
_productid int unsigned
)
begin
delete from products where productid = _productid;
end;
//
mysql> delimiter ;



Step 4: Testing it all out!

Fire up a python interpreter and ...

>>> import MySQLdb
>>> from MySQLdb.constants import CLIENT
>>> cnn = MySQLdb.connect(host='localhost', user='USER', passwd='PASS', db='test', client_flag=CLIENT.MULTI_STATEMENTS)
>>> cnn.autocommit(1)
>>> curs = cnn.cursor()
>>> count = curs.execute('call usp_upd_productstatus(%d, %d)' % (1,0)) # update
>>> count = curs.execute('call usp_get_products_by_category(%d)' % (2)) # select
>>> curs.fetchall()
>>> curs.execute("call usp_ins_product(1, 'FreeBSD', 'Another great OS', 1)")
>>> curs.fetchall() # this returns the id of the new 'FreeBSD' record
>>> count = curs.execute('call usp_del_product(%d)' % (2)) # delete
>>> curs.close()
>>> cnn.close()


The INNODB storage engine supports row level locking which means writes won't tie up the entire table. Because I created the tables using the 'type=innodb' I had to set the autocommit flag to true in the connection otherwise my queries won't get commited.

Friday, February 17, 2006

The Technology behind JobJitsu.com

JobJitsu Technology Overview:
So now a little bit about the technology behind the job board...

JobJitsu is more of a home grown web app. we have some general purpose modules that handle most of the common web application facilities:

  • cache -- a wrapper around the memcache client provided by tummy.com
  • conf -- where we keep common configuration settings
  • db -- a data access module that manages connections and uses Greg Stein's DTuple.py
  • dispatch -- this guy handles the url to resource mapping, sessions and a few other things.
  • log -- wrapper around python logging module
  • mail -- mail wrapper
  • re -- common precompiled regex's (email, url and etc.)
  • sesh -- our session module
  • util -- utility stuffs like (string manipulation, url parsing and etc.)

The presentation is handled by Cheetah. Cheetah has nice syntax (no-xml), flexibility, and performance (precompiled templates). One other feature that I find hard to live without is the template inheritance. Cheetah allows you to define a base page which other pages can extend (good for header + sidebar + footer).

We are using MySQL 5.0 (which supports stored procedures) on top of the innodb storage engine for persistence.

Finally what modern web app would be complete without some cool AJAX? Matt has some neat JSON stuffs that he is working on.

Next post will be an in depth look at how we manage our data including how we defined our data model, a quick discussion on stored procedures, indexing strategies and how we actually pass data back and forth between Python and the MySQL

Monday, January 09, 2006

Creating your own mod_python request dispatcher

Creating your own mod_python request dispatcher that maps url's to python request handlers.

Conceptually I have something that looks like this:

+-------------+
| Clients |
| Web Browser |
+-------------+
| [http://somecooldomain.com/coorequest]
| ^
| | [Cool Response]
V |
+--------+ +------------+ +---------------+ +---------+
| Apache | --> | Mod_Python | --> | dispatcher.py | --> | cool.py | --+
| (A) | <-- | | <-- | (B) | | (D) | |
+--------+ +------------+ +---------------+ +---------+ |
| ^ |
| | |
| +------------------------+
V
+-------------+
| urlmap.conf |
| (C) |
+-------------+


A) Apache Configuration:
On my server I have mod_python set up to handle any request that comes in without a file extension. That way I can have those cool REST style URI's. I use the apache "FilesMatch" directive in my httpd.conf file to look for any request that does not contain a '.' before the '?' in the query string:


<FilesMatch "(^[^\.]*$|^[^\?]*[\?]+[^$]+$)">
SetHandler python-program
PythonHandler common.dispatch.dispatcher
PythonDebug On
</FilesMatch>


B) dispatcher.py
The source for dispatcher.py can be found here.

C) Example urlmap.conf


[somecooldomain.com]
/=cooldomain.handlers.home.handler
/signup=cooldomain.handlers.member.signup
/login=cooldomain.handlers.member.login
/logout=cooldomain.handlers.member.logout
/cool=cooldomain.handlers.cool.handler



D) cool.py
This is the handler that generates your content. In my content handlers I usually do things like query the database, process business logic, select a cheetah template and return the result as html.

In summary I use this approch for a couple of reasons, first it allows me to decouple my url's from my python code that way I don't have python code + html sitting around in the same directory. Secondly I get one piece of code that handles every request. (good for sessions and things like that)

Friday, January 06, 2006

urlencoder/decoder for python

EDIT (I so stand corrected.):

I could have just used:
urllib.quote
urllib.unquote

---------------------------------------------------

Every once and a while I run into the need for a urlencoder/decoder function that just accepts a string and returns a string.

The one that comes w/ urllib(2) takes a dictionary and returns a "name=val" formatted string.
My only problem is that it does 2 things at once:
1) url encodes.
2) constructs a query or post string.

So I wrote a quick utility to do just #1:


_keys = [
"$", "&", "+", ",", "/", ":", ";", "=", "?", "@", " ", '"',
"<", ">", "#", "%", "{", "}", "|", "\\", "^", "~", "[", "]", "`"]

_vals = [
'%24', '%26', '%2B', '%2C', '%2F', '%3A', '%3B', '%3D', '%3F',
'%40', '%20', '%22', '%3C', '%3E', '%23', '%25', '%7B', '%7D',
'%7C', '%5C', '%5E', '%7E', '%5B', '%5D', '%60']

def encode(str=""):
""" URL Encodes a string with out side effects
"""
return "".join([_swap(x) for x in str])

def decode(str=""):
""" Takes a URL encoded string and decodes it with out side effects
"""
if not str: return None
for v in _vals:
if v in str: str = str.replace(v, _keys[_vals.index(v)])
return str

def _swap(x):
""" Helper function for encode.
"""
if x in _keys: return _vals[_keys.index(x)]
return x

### units ###
if __name__ == "__main__":
assert("".join(_keys) == decode(encode("".join(_keys))))
assert("".join(_vals) == encode(decode("".join(_vals))))
print "passed all unit tests."

Monday, December 26, 2005

DTuple for O/R mapping

Why use ActiveRecord or SQLObject when there is DTuple?

Both of these systems attempt to hide sql through an object oriented abstraction. (I thought that this was something that only the J2EE guys tried to do.) Long story short it won't perform nearly as well as hand coded SQL and it may be trying to solve one problem by introducing another. Why learn another DSL when SQL works so well?

It's not the mapping that I have issue with it's the fact that sql is code generated behind the scenes as if it were not meant for human consumption. Don't get me wrong I am a huge fan of O/R mapping and I am a huge fan of abstraction, however I think that attempting to talk to a database while speaking objects actually makes it harder to express what you actually want the database to do. The other disadvantage that this approach has is that it may make your code hard to debug.

What am I suggesting? Well for Ruby I am not quite sure but for python there is DTuple -- written by Greg Stein. I am not sure why DTuple gets no ink but it should. First off it has been around 4-ever since 1996 I think, and secondly it roks! It examines the result of your query and returns a collection of typed objects. This is done dynamically and preforms quite nicely. You basically get O/R Mapping for free and don't have to learn any object query syntax.

Example Data Access object:
(caution this code is almost 2 years old and may need a little tweaking with the latest version of MySQLdb)

Pay special attention to the get_records where DTuple is used...

import dtuple
import MySQLdb
from common.util import config

def _connect():
""" Login to the database return the connection
"""
conf = config.get_config()
return MySQLdb.connect(
host=conf.get('db', 'host'),
user=conf.get('db', 'user'),
passwd=conf.get('db', 'passwd'),
db=conf.get('db','database'))

def _get_conn(relogin = 0):
""" Returns a persistent connection to the database
attempts to reconnect if the connection is lost.
"""
global CONN

if relogin:
CONN = _connect()
return CONN
else:
try:
conn = CONN
return conn
except NameError:
CONN = _connect()
return CONN

def _exec_stmt(stmt):
""" Runs SQL statement and returns (count, curs)
"""
curs = None
try:
curs = _get_conn().cursor()
count = curs.execute(stmt)
except MySQLdb.OperationalError, e:
curs = _get_conn(relogin=1).cursor()
count = curs.execute(stmt)
return count, curs

def do_test(stmt):
count, curs = _exec_stmt(stmt)
try: return curs.fetchall()
finally: curs.close()

def get_first_record(stmt=None):
""" Runs select statement and returns 1st record
"""
rec = get_records(stmt, 0, 1)
if rec[0] == 1: return rec[1][0]
return None

def get_records(stmt=None, start=0, max=500):
""" Runs select statement and returns a list of the form:
(record count, result set)
"""
if stmt == None:
return None
limit_stmt = "%s limit %d, %d" % (stmt, start, max)
count, curs = _exec_stmt(limit_stmt)
try:
return count, map(
lambda x:
dtuple.DatabaseTuple(
dtuple.TupleDescriptor(curs.description),
x),
curs.fetchall())
finally: curs.close()

def do_insert(stmt=None):
""" Runs insert statement and returns the id of the newly
inserted record or -1 on error.
"""
if stmt == None:
return -1
count, curs = _exec_stmt(stmt)
if curs == None:
return -1
try: return curs.insert_id()
finally: curs.close()

def do_update(stmt=None):
""" Runs update statement and returns the number of records
updated or -1 on failure.
"""
if stmt == None:
return -1
count, curs = _exec_stmt(stmt)
curs.close()
return count

def do_delete(stmt=None):
""" Runs delete statement and returns the number of records
deleted or -1 on failure.
"""
return do_update(stmt) # same impl as update

def escape_string(str=""):
return MySQLdb.escape_string(str)