Easy deployment with Python and Fabric

Posted August 26th, 2010 in Python by Florentin


This article is about writing tools in Python to easily deploy your web projects.
A well know deployment library in Python is Fabric. While it’s not really mature (version 0.9.1), it is easy to understand and work with it. The best place to start is the Fabric tutorial.

The Challenge

You develop and maintain many projects. You backup and restore databases several times a day during the development phase or you need to upload or download files from the hosting server often. How would you improve these processes instead using Ftp, mysql/mysqldump or phpmyadmin ?

Quick Fabric Introduction

Fabric gives you instruments to run commands on different hosts and copy/download files to/from remote hosts. In order to do that, you have to prepare a file of available commands named fabfile.py which will be placed in the project’s directory. Having some methods in that file allows you to run the commands like that:

fab hello

where hello is one function defined in fabfile.py
Fabric installation process.
sudo easy_install fabric
sudo pip install fabric

The Solution

Having a lot of projects to maintain and deploy makes difficult to write one fabfile.py for each of them.
Also using remote hosts or writing database queries require you to put sensitive information in the fabfile.py (or pass them as parameters but that’s a different talk)
My thought is to have one location available where I place a fabfile.py and configuration files of all the sensitive information required for deployments, i.e. remote host addresses and maybe passwords, database connection information, etc. If you care more about security than speed of development, don’t store sensitive information there.
I have created a directory /work/fabric with the following structure:
├── fabfile.py
├── projects
│   ├── __init__.py
│   ├── ajaxdaddy.py
│ ├── default.py

fabfile.py contains the deployment code based on Fabric
local.py store configuration details of the localhost database
projects/* store configuration details of specific projects
The project name is the same with the file name used for the configuration.

Instead of placing a fabfile.py inside the project’s directory, we will instruct Fabric where to find the fabfile.
Create a file ~/.fabricrc and write the following line:
fabfile = /path/to/your/fabfile.py
I have mine placed in /work/fabric/fabfile.py

There are more details on fabfile discovery here.

Now we can run “fab” in any location. Here are some examples:
fab c:localhost c:localdb e:db_name=test q:clean
fab e:db_name=test_elgg e:db_file=elgg.dump.sql q:export

Commands Supported

The fabfile.py available for download in the end of this article gives you the power to run several commands useful in deploying projects or databases. Many commands require some parameters to exist, for example a database “clean” command which deletes all the tables from one database would require some connection details i.e. host,user,pass,db_name

Several commands may use the same parameters. To make these parameters available they are loaded from configuration files or passed to the “fab” command. All parameters are stored inside fabfile.py in a dictionary called “env” (environment). The environment is a big dictionary like variable (associative array for the Php users) which may store connection parameters for remote hosts, database connection properties or other information.

Most of the time you will need several params available for one command, e.g. a database command cannot function with only host name and no user provided. To make things easy, configuration files can be written where you define groups of parameters. Such a configuration file is /work/fabric/projects/default.py which is loaded by default. Configurations are simple Python dictionaries defined in Python files under the “projects” directory. It’s up to you to choose the names and the purpose of those settings, i.e., some might refer to database settings and other to remote hosting details. If you want to enable a set of parameters from a config file, first you need to load the file then enable the params.

Let’s take for example:
fab c:localhost c:localdb e:db_name=test q:clean

q:clean is the command and it means: delete all tables from the database define before
e:db_name defines the database
c:localdb loads the localhost database configuration, i.e. host, user, password, etc
c:localhost loads the localhost host details, i.e. host name, port, etc

Example of /work/fabric/projects/default.py file:
config = {
‘localdb': { # configuration name
‘db_user': ‘root’,
‘db_pass': ‘betterprogramming rules’,
‘db_host': ‘localhost’,
‘db_name': ‘test’, # default db_name if none is givven
‘db_file': ‘dump.sql’, # default dump file

“l” loads a configuration file located under “projects” directory (see the tree structure above)

fab l:ajaxdaddy

# not needed because default is loaded
fab l:default

“c” enables a configuration

# “localhost” is hardcoded into the fabfile.py because it’s mandatory
fab c:localhost

# enables a database configuration
fab c:localdb
fab l:default c:localdb

It’s possible to enable one configuration for a command then enable a different config for the next command.

# the second configuration “secondlocaldb” would overwrite the first one, “localdb”
fab l:default c:localdb c:secondlocalddb

After you enable custom configuration, it is possible to change specific parameters by using the “e” command.

# after i load and enable ajaxdaddy’s parameters, i.e. db_user, db_pass, db_host, db_name i can change the previously stored db_name value by using “e”
fab l:ajaxdaddy c:ajaxdaddydb e:db_name=db2_ajaxdaddy e:db_user=ajaxuser

In conclusion, there are 3 ways to set the parameters which will be used by your commands:
– by writing configuration files and place them under “projects” directory, load the config file (e.g. “l:ajaxdaddy”) and enable a specific group (e.g. “c:ajaxdaddydb”)
– by passing single parameters to the “fab” command (e.g. “e:db_user=gigi”)
– by passing parameters to the command itself (you will see how it works later)

Datase Commands

Database commands starts with “q” and are followed by an action.
The available actions are:
clean: delete all the tables inside the “db_name” database
export: export the database “db_name” to the file “db_file” (db_file may be provided or created programmatically by the fabfile.py from db_name)
import: import “db_file” into “db_name”
show: list all the databases where the “db_user” has access
create: create a new database called “db_name”
grant: grant all on “db_name_new” to “db_user_new” identified by “db_pass_new”

You may define your own actions in the configuration files. You just have to create a class named “DbAction”+actionname , e.g. class DbActionCreate(DbAction) or class DbActionClean(DbAction)
When loading a configuration file, your action class will also load.

# enables localhost (it’s enabled by default anyway), enables localdb so that “q” knows where to operate, sets the environment variable “db_name” to “test” then execute command
fab c:localhost c:localdb e:db_name=test q:clean
same with
fab c:localdb e:db_name=test q:clean

# selects the “test” db and export that db into “dump.sql” file
fab c:localhost c:localdb e:db_name=test e:db_file=dump.sql q:export

# create a new database “test_2″ and grant access to the user defined in “localdbnew”
fab c:localhost c:localdb c:localdbnew e:db_name=test_2 q:create q:grant

# delete all tables from “test” db and import the “test_elgg.sql” file
fab c:localdb e:db_name=test e:db_file=test_elgg.sql q:clean q:import

The list of all parameters used by the database commands:

File commands

# zip the temp directory, temp must be located under the current directory, it will create “temp.zip”
fab zp:temp

# zip the “temp” directory and create the archive named “mytemp.zip”
fab e:file=temp,dest=mytemp.zip zp

# zip “temp” directory’s content, i.e. do not include the “temp” parent directory in the archive
fab zpi:temp

# unzip “filename.zip” into the “dest” directory
fab uzp:filename.zip,dest
# same with
fab e:file=filename uzp:,dest
# same with
fab e:file=filename,dest=dest uzp

# load the “temp” config file and enable dreamhost parameters (“c:dreamhost”) then upload fabric.localhost.zip into the “stuff” directory located on the remote host inside the base dir. The base directory is defined in “base_dir” parameter, if base_dir=/var/www then the uploaded file will be located in /var/www/stuff/fabric.localhost.zip
fab l:temp c:dreamhost up:fabric.localhost.zip,stuff

# download /var/www/remotefile.zip to the current directory if base_dir=/var/www
fab l:temp c:dreamhost dl:remotefile.zip

List of file commands:
zp: zip a “file” (file or directory) to “dest”
zpi: zip a directory’s contents, exclude the directory itself from the archive
uzp: unzip a “file” to “dest”
up: upload a “source” (local host) to “dest” (remote host)
dl: download a “source” (remote host) to “dest” (local host)
cl: delete a “file”

Complex Commands

# exports “test_elgg” database into “dump.sql” and upload it to “dreamhost”. “up:e.db_file” means that the value of “db_file” located in the environment should be uploaded.
fab c:localhost c:localdb e:db_name=test_elgg e:db_file=dump.sql q:export l:dreamhost c:dreamhost up:e.db_file
# same with
fab c:localdb e:db_name=test_elgg e:db_file=dump.sql q:export l:dreamhost c:dreamhost up:dump.sql
# similar but different dump name “test_elgg.sql” created by default from “db_name” + “.sql”
fab c:localdb e:db_name=test_elgg q:export l:dreamhost c:dreamhost up:test_elgg.sql

# same as above, but upload a zip file which will be called by default “dump.sql.localhost.zip”
fab c:localdb e:db_name=test_elgg e:db_file=dump.sql q:export zp:dump.sql l:dreamhost c:dreamhost up

# same but unzip on destination
fab c:localdb e:db_name=test_elgg e:db_file=dump.sql q:export zp:dump.sql l:dreamhost c:dreamhost up uzp

# same but delete “cl” the dump.sql from localhost and delete “cl” dump.sql from the remote host
fab c:localdb e:db_name=test_elgg e:db_file=dump.sql q:export zp:dump.sql cl:dump.sql l:dreamhost c:dreamhost up uzp cl:dump.sql

Download fabfile.py and a few examples

Synonyms find and replace tool

Posted March 15th, 2010 in Portfolio, Uncategorized by Florentin

Synonyms find and replace tool: based on FCKeditor, this tool was used to find and replace synonyms in a text. I have extracted all terms from Wordnet dictionary.

Domain tools

Posted March 15th, 2010 in Portfolio by Florentin

Description: develop domain business related tools
– domain name scripts: I’ve written several Php scripts to find niche domains. One of these tool can
identify the ‘article’ part from a page and then extracts keywords from it. Then it looks for available
domains composed from those keywords. Another tool acts on RSS feeds. It tries to extract relevant
keywords then search for available domain names. Several other tools were developed for this
purpose, to find good domain names. I have managed to find some good .info domains like exfoliators.info, uninstallers.info, thermoses.info using information from the EBay site (the category
– domain reporting tools: a Php tool for a client to get statistics about dropped domains. Some of these
statistics are: Google relevance (keywords, back links, number of Google ads), overture results for
several countries, Alexa ranking, Google pr, similar domains with different extensions, etc) The system
had a cron job which automatically processed domains to get the statistics. The client had an
administration interface to view / filter / order domains and statistics.
Tools: Php, Curl
Skills: web services, Php admin interfaces, batch operations
Date: 2007-2008

Social bookmarking service

Posted March 15th, 2010 in Portfolio by Florentin

Description: Building a social bookmarking service. Submit articles/site descriptions to several social sites.
Role: developer
– developed some Javascript tools to help me submit one website to 100+ social services like del.icio.us or digg.com.
This tool loads in Firefox as a Greasemonkey script and allows me to ‘learn’ the forms from one page and later to automatically fill in that form using database information about the site I wish to advertise. I also made some VBS tools to use imacros software in order to fully automate the process of submission.
– worked with Selenium RC in order to automate some form submissions though ruby and Php interfaces.
Tools: Php, Ruby, Selenium, Imacros
Skills: Build advanced interfaces, working with curl, multi threaded Php, using web services
Date: 2008

Ruby web scraper

Posted March 15th, 2010 in Portfolio by Florentin

Description: develop a software platform and techniques to extract information from websites.
Role: concepts, specifications, development
– screen scrapers and web crawlers in ruby – I had a ruby framework developed for
crawling and scraping websites based on predefined rules, then export, normalize and merge
data from different sources. Recently I’ve studied a lot of semantic web technologies like RDF in order
to create an advance semantic scraper.
Tools: Ruby
Skills: in depth ruby knowledge, ruby as DSL, using specific ruby libraries like
Hpricot, Nokogiri, active record
Date: 2008-2009

News aggregator

Posted March 15th, 2010 in Portfolio by Florentin

Description: Aggregate, categorize and display news on specific topics.
Role: analysis, concepts and development
– I’ve created a custom plugin for WordPress MU which allows the admin to control various settings of all the blogs installed with wpmu.
– Article recognition software – it’s a ruby script which identifies a large section of text inside a html page (that is the main article) and then extracts the relevant paragraphs based on keywords. It was used by the client to create over 60 niche blogs using the same platform, WordPress MU.
Tools: PHP/Mysql, WordPress MU, Ruby
Skills: Working with WordPress plugins, handle specific ruby packages like nokogiri.
Date: 2009