I won’t cover a complete Ruby and Rails/Ramaze/Sinatra installation. This post assumes you have that much down pat, so we’ll pick up the ball with installing FreeTDS, unixODBC, Ruby ODBC

If you’re using Ubuntu 8.10, you can get away with installing everything from packages with the following:

apt-get install  unixodbc freetds-common tdsodbc \
             libodbc-ruby1.8 \
             libdbi-ruby1.8 libdbd-odbc-ruby \
             autoconf make

NOTE: I wrote this article and right before publishing, attempted to replicate these results on both mac OS X and CentOS 5. Both failed to work with the technique I thought I had all nice and worked out. I spent several hours over many months trying to find out exactly what’s different between the packages, but never got very far with my investigations. For a long time, it looked to me like both the ODBC packages and the FreeTDS packages need to be tweaked to properly handle [SERVER]\[DB_INSTANCE] construct that MSDE uses vs. just [SERVER] that the full-fledged SQL Server uses. This, after all, is the convention of the Windows ODBC driver and DSN definitions. However, there’s a really simple solution that I hit upon on Actual Technologies’ website that led me to a true universal solution and that’s to learn the port the INSTANCE runs on and set up the DSN specifically to that port!

But beware, the above doesn’t get you all of the freetds tools, most importantly the tsql command which can help you with debugging issues before getting into the ODBC game (that is, direct TDS connections to your SQL Server). So, to cover the most distros possible, I’ll document the “from source” approach for *nix systems and macports for Macs so you’ll find two sets of installation instructions for every step.

Preparing your Environment

I prefer to pull down all source packages into my local user directory under a “~/src” folder. The instructions that follow assume you’re in this folder. If you’re not, then please make the appropriate substitutions as you follow along. Additionally, I configure and compile most things as a regular user then use sudo to install system-wide as a “super user.”

Install unixODBC

unixODBC provides the ODBC connectivity for the TDS driver.

wget http://www.unixodbc.org/unixODBC-2.2.14.tar.gz
tar zxfv unixODBC-2.2.14.tar.gz
cd unixODBC-2.2.14
./configure  --enable-gui=no
make
sudo make install

On Macs

sudo port install unixODBC

On CentOS

sudo yum install unixODBC unixODBC-devel

Install FreeTDS

FreeTDS provides the open source implementation of the protocol implemented by both Sybase and Microsoft SQL Server.

wget ftp://ftp.ibiblio.org/pub/Linux/ALPHA/freetds/stable/freetds-stable.tgz
tar zvxf freetds-stable.tgz
cd freetds-0.82/
sudo ./configure  --with-unixodbc=/usr/local
sudo make
sudo make install

On Macs

sudo port install freetds

On CentOS

sudo yum install freetds

Install ruby-odbc

The ruby-odbc package provides the ruby bindings for ODBC connectivity.

wget http://www.ch-werner.de/rubyodbc/ruby-odbc-0.9995.tar.gz
tar zvxf ruby-odbc-0.9995.tar.gz
cd ruby-odbc-0.9995/
ruby extconf.rb
make
sudo make install

On Macs

sudo port install rb-odbc

Install ruby DBI

wget http://rubyforge.org/frs/download.php/655/ruby-dbi-all-0.0.23.tar.gz
tar zvxf ruby-dbi-all-0.0.23.tar.gz
cd ruby-dbi-all
ruby setup.rb config --with=dbi,dbd_odbc
ruby setup.rb setup
sudo ruby setup.rb install

Configure FreeTDS and ODBC

If you’re connecting to either SQL Server 2000 or SQL Server 2005 Standard, Enterprise, etc., then the typical approach is to add an entry for the server to /etc/freetds/freetds.conf and then reference that entry in the /etc/odbc.ini. Sometimes, these files get installed in /usr/local/etc. If so, then I will usually symlink the files into /etc for convenience, so that’s what’s referenced below.

My server is called “apvdbs01″ (the WINS name) with an IP Address of 10.0.2.2 and the database I’m connecting to is called “test_development” while “username” and “passwd” should be substituted with an valid user account and credentials for successfully connection to the database server.

Add a server entry for your server (which I typically give same name as Windows Server) to /etc/freetds.conf, so that host points to the SQL Server address:

[apvdbs01]
host = 10.0.2.2
port = 1433
tds version = 7.0

TDS version 7.0 is for SQL Server 2000. If you have SQL Server 2005, then tds version is 8.0.

tsql –S apvdbs01 –U username –P passwd

Once you can connect directly with the TDS drivers, move on to adding the ODBC driver entries.

Add the FreeTDS ODBC driver by editing “/etc/odbcinst.ini” (make sure “/usr/local/lib/libtdsodbc.so” exists first):

[FreeTDS]
Description     = TDS driver (Sybase/MS SQL)
Driver          = /usr/lib/odbc/libtdsodbc.so
Setup           = /usr/lib/odbc/libtdsS.so

Add a new DSN to “/etc/odbc.ini”:

[test_development]
Driver          = FreeTDS
Description     = ODBC connection via FreeTDS
Trace           = No
ServerName      = apvdbs01
Database        = test_development

So, “FreeTDS” is the driver that we gave for the lib TDS ODBC connector in the odbcinst.ini file (this allows you to install and maintain multiple versions of the libtdsodbc.so file). We reference this driver in the odbc.ini file with the “Driver” entry and we reference the actual server data in the freetds.conf file via the “ServerName” entry.

Now that you have ODBC configured for FreeTDS, test it with:

	isql test_development username passwd

But that doesn’t work for MSDE!

The above will get you connected to a standard or enterprise SQL Server 2000/2005 installation, but does not work for an Embedded SQL Server 2000/2005 installation. After trying many permutations, I finally came up with a configuration that actually works. One of the key things to know about MSDE is that the MSDE server instance is in the form of “servername\instance” rather than straight up “servername” as with standard/enterprise installations. This is known as a “named instance.” The trick to connecting? Find out the port the named instance is listening on and connect to that port rather than the 1433 port!

You can determine the port used by the instance by using the Server Network Utility in the Microsoft SQL Server program group on the Windows server where your database resides. Select the instance name from the drop down list and select TCP/IP from the list of enabled protocols. This will show you the port number for that instance.

[test_msde2000]
Driver         = FreeTDS
Description    = ODBC connection via FreeTDS
Server         = 10.0.2.2
Database       = my_test_db
Port           = 1348
TDS_Version    = 7.0

I could then connect to the MSDE instance like so:

isql test_msde2000 username passwd

Test ODBC connectivity from Ruby

If both tsql and isql commands work, then you’re well on your way to connecting to SQL Server via Ruby. To test Ruby with the above ODBC configurations:

irb
require "dbi"
dbh = DBI.connect("dbi:ODBC:test_development", "username", "passwd")

Connecting with Sequel

One of my favorite tools for connecting to SQL DBMS’ is Sequel, which is maintained by Jeremy Evans. To connect using Sequel, install the gem:

sudo gem install sequel

And then give this a shot:

irb
require 'rubygems'
require 'sequel'

DB = Sequel.odbc('test_development', :db_type => 'mssql', :user => 'username', :password => 'passwd')

References

http://www.freetds.org/userguide/confirminstall.htm

http://www.unixodbc.org/odbcinst.html

http://www.actualtech.com

Introducing Ramaze

One I found is Ramaze. I have to say that Ramaze is a very minimalist approach to a framework that is actually rather enlightening. For one coming from a Rails’ world, it can feel a bit sparse at first. But lately, I’m attributing my lack of comfort to being put on my toes to do some actual real Ruby coding. Its both a little scary and liberating at the same time. I have found that I am enjoying Ruby a lot more as a result of my exposure to Ramaze and my thought process is actually radically changing, almost like a Visual Basic programmer going to Borland Delphi might feel.

So I decided to take a hard look at performance metrics between Rails and Ramaze’s. At the same time, I wanted to see how MRI 1.8.7 stacked up against Ruby Enterprise Ed. 1.8.6, so I’m satisfying a dual curiosity herein. Being quite happy with Phusion Passenger, I decided to set things up with Apache 2.2.9 + Passenger + Rails/Ramaze. Since I’m quite comfortable with ERB and ActiveRecord, I tweaked Ramaze to be a bit more like the familiar Rails world I am familiar with.

Un-DRYing the Results

There’s no point in generating a lot of data and reporting on it if others cannot repeat yourself (CRY!) in replicating the results, so with that in mind, I attempt to refrain from further bad turn of words by busying myself in pushing everything I utilized to github.com and documenting the basic bootstrapping of Ubuntu Intrepid. For these tests in particular, I used two scripts, bootstrap-passenger-std.sh and bootstrap-passenter-ent.sh to set up four distinct configurations under the Ruby1.8 interpreter. Ruby 1.9.1. was a bit difficult to set up and there is a bootstrap-passenger-191.sh script that I assembled as I installed things, but I haven’t run it to ensure it works end-to-end.

I utilized this simple call to Apache Bench to run through the four test scenarios:
All data collected was based on 10,000 requests at 100 concurrent users after an initial “warmup” of the passenger threads with start of 100 requests at 10 concurrent users. The following command shows the basic set of parameters issued to Apache Bench:

ab -n 10000 -c 100 http://demo.u64rails01.local/posts

What was tested?

In order to compare, as close as possible, apples to apples, I opted to use Active Record and ERubis in Ramaze to set up the same MVC classes in both the Rails and Ramaze project. ActiveRecord is probably the heaviest weighted ORM that can be utilized with Ramaze. I tried briefly to get Erubis working with Rails so that the two frameworks’ projects would be more closely aligned than the out-of-box ERB rendering Rails ships with would have afforded. I did attempt Erubis with Rails, however, I was either doing something very wrong or Erubis under Rails takes a sizable performance hit and I stuck with out-of-box ERB for these tests given its better performance. As such, Rails uses out of box ERB rendering in these tests.

The application itself is ridiculously simple, offering only one model and performing only a read against the database, but hopefully, an effective start towards building this benchmark framework towards more substantive benchmark tests to come. For now, lets take a look at how Rails and Ramaze stack up to each other and what affect the underlying interpreter can have on performance. Standard packaged Main Ruby Interpreter (MRI) 1.8.7 consisted of simple package install provided by the Ubuntu Intrepid repositories whle Ruby Enterprise Edition (REE) 1.8.6, which hails from the same folks bringing you Phusion Passenger, is compiled from source.

As a bonus, I also compiled and clocked metrics for Ramaze on 1.9.1. This was a tricky environment to get set up and working. And my initial results made me think I did something horribly wrong with compiling and configuring Ruby 1.9.1. However, I fully replicated Antonio Cangiano’s benchmarks in my test environment (see The Great Ruby Shootout (December 2008)) and MRI 1.9.1. did indeed crush the other Ruby interpreters in Fibonacci number crunching (amongst others). The theory tossed around in the Ramaze IRC channel was that 1.9.1. change in implementation of the String class (and it being unicode native now) slows things down a good bit with string processing.

The Results

The following table tabulates the results gathered from Apache Bench as well as passenger-memory-stat. I ran a short run to warm things up (100 requests, 10 concurrently). And then ran the big bench at 10,000 requests, 100 concurrently, and then ran again and recorded the results. At around 5,000 requests, I ran passenger-memory-stat to collect the data during mid-processing and then again at the end. These tests were run on a Ubuntu 64-bit Intrepid 8.10 virtual server that was allocated 1 CPU core and 512MB of RAM. Swap space was never triggered (0k utilization), and CPU utilization was roughly 85% within the Virtual Machine during the test runs. The host machine, which is four core CPU, showed one core busy at near 100% with the other three cores pretty much idle.

Throughput

Everybody likes to talk about throughput, so lets start there. As you can see, MRI and Rails was the worst performer while REE and Ramaze clocked in with the highest throughput in terms of requests per second. Be forewarned that throughput doesn’t necessarily equate to user response times. This metric simply tells you the load at which your system is capable of delivering content at.

Time per Request

Time per request helps you see about how long an average user request takes for a given load. As you can see here, with times in the milliseconds, we’re talking about numbers that will easily be swallowed in the noise of network latency. Even so, its not hard to be impressed by what the REE folks managed to pull out of their hat with regards to performance on the main ruby interpreter.

Memory Consumption

This final graph shows the sizeable dent REE makes in memory consumption for running a Rails application. Ramaze is already pretty efficient and lean and REE made practically no dent in memory consumption here. However, REE’s use of the faster memory allocation routines does have a good effect on boosting Ramaze’s performance (as seen in above graphs), so there’s still lots to be gained by running Ramaze with REE.

Conclusion

For me, I have two very, very viable solutions before me. Rails on Enterprise Ruby or Ramaze on Ruby 1.8.7. The final choice comes down to how much I can bet on my own Ruby prowess. Ramaze is a very lean framework that gets you going, but doesn’t get you a nine inch pillowed mattress topping for your bed like Rails does. Because Ramaze is so lean, I realize I will have to really take my Ruby skills to the next level and truly understand the ruby code that other developers produce. Unlike Rails, where there’s gobs and gobs of plugins and blogs on the topic that make pretty much any need simple to satisfy and get working, I have to know my Ruby to get a solution implemented on Ramaze. Not a bad thing, but certainly a more challenging and potentially rewarding path to take.

Aside from my personal concerns about making the jump, looking at these results, it is good to see that I have options and how easy it is to get a good deployment up and working compared to 3 or 4 years ago when I was banging my head against the wall for days on end to keep lighttpd + fcgi processes up and running around the clock. In these tests, it was very, very satisfying that not one single crash or dropped request was recorded in all the benchmarking I performed. Mongrel came along and made a big impact, but you still needed something like monit to watch Mongrel and restart it on ocvasion. Now Passenger has come along and made yet another big impact in the ease of deploying Rails, Ramaze, or any of the other ruby-based frameworks. We’ve come a long ways, indeed. I, for one, certainly look forward to what the Ruby sphere will bring us over the next couple of years.

My System

2.6.18-92.1.22.el5 #1 SMP Tue Dec 16 11:57:43 EST 2008 x86_64 x86_64 x86_64 GNU/Linux

The Problem

If you simply execute:

sudo gem install mysql

You’ll get the following errors on a 64-bit system:

$ sudo gem install mysql
Building native extensions.  This could take a while...
ERROR:  Error installing mysql:
        ERROR: Failed to build gem native extension.

/usr/local/bin/ruby extconf.rb install mysql
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lm... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lz... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lsocket... no
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lnsl... yes
checking for mysql_query() in -lmysqlclient... no
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

Provided configuration options:
        --with-opt-dir
        --without-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=/usr/local/bin/ruby
        --with-mysql-config
        --without-mysql-config
        --with-mysql-dir
        --without-mysql-dir
        --with-mysql-include
        --without-mysql-include=${mysql-dir}/include
        --with-mysql-lib
        --without-mysql-lib=${mysql-dir}/lib
        --with-mysqlclientlib
        --without-mysqlclientlib
        --with-mlib
        --without-mlib
        --with-mysqlclientlib
        --without-mysqlclientlib
        --with-zlib
        --without-zlib
        --with-mysqlclientlib
        --without-mysqlclientlib
        --with-socketlib
        --without-socketlib
        --with-mysqlclientlib
        --without-mysqlclientlib
        --with-nsllib
        --without-nsllib
        --with-mysqlclientlib
        --without-mysqlclientlib

Gem files will remain installed in /usr/local/lib/ruby/gems/1.8/gems/mysql-2.7 for inspection.
Results logged to /usr/local/lib/ruby/gems/1.8/gems/mysql-2.7/gem_make.out

After digging into the gem options and logs a bit and looking at where mysql actually got installed, I discovered that it was hitting all the locations the 32-bit version would normally install to, but not the 64-bit version, which ends up in /usr/lib64/mysql.

Installing the Ruby gems

To properly install the Ruby gems for mysql, you’ll have to explicitly tell the gem where the mysql config is and it will then handle all the rest.

gem install mysql -- --with-mysql-config=/usr/lib64/mysql/mysql_config

With that, you should see the following:

Building native extensions.  This could take a while...
Successfully installed mysql-2.7
1 gem installed

Check it out with:

irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> require 'mysql'
=> true
irb(main):003:0> quit

Life is good.

To get started, I needed to know how to get the data into Trac. The following links gave me an API I could utilize to create trac pages and content:

Trac Data Models
The Data model Python code

Great! Except there’s one small problem. I don’t really know python all that well, even though I did do a bit of Zope development, oh, five years ago. These days, my language of choice is Ruby, so I needed to figure out how to parse the Oddmuse syntax and turn it into Trac syntax. The strategy: Parse with Ruby, load cleaned page files with Python.

This script will take a filename at command line and convert it and save back to file, but different extension (I chose *.om for “oddmuse”). In the script below, I made the command-line optional as I discovered I didn’t handle every case as I worked on developing this script, so I simply changed the line that pulls the command-line argument to point to a specific file and output such to terminal and this allowed me to debug and fix as I went:

#!/opt/local/bin/ruby
# $FS  = "\xb3";      # The FS character is the RECORD SEPARATOR control char in ASCII
# $FS0 = "\xb3";      # The old FS character is a superscript "3" in Latin-1
# $FS1 = $FS . '1';   # The FS values are used to separate fields
# $FS2 = $FS . '2';   # in stored hashtables and other data structures.
# $FS3 = $FS . '3';   # The FS character is not allowed in user data.

FS = 179.chr

SITE_NAME = "corporate_wiki"
ATTACHMENT_URL_BASE = "https://intranet.example.com/#{SITE_NAME}/attachment/wiki/ConvertedAttachments/"

require 'find'
require 'ftools'

def safe_page_name(pagename)
  result = pagename.gsub("'","")
  result = result.slice(/(\w+)/).first
end

def get_section(page, section)
  page.each_with_index {|e,i| return page[i+1] if e == section}
  return ''
end

def asterick_prefixed(text)
  t = text.match(/(\*+)([^\b]*)/)
  return text if t.nil?

  tokens = t.to_a
  return ('  ' * tokens[1].size) + '* ' + tokens[2]
end

def colon_prefixed(text)
  return "  #{colon_prefixed(text.slice(1,text.size))}" if text[0] == ":"[0]
  return text
end

def pound_prefixed(text)
  return "  1. #{text.slice(1,text.size)}" if text[0] == "#"[0]
  return text
end

def fix_headers(text)
  t = text.match(/(=+)([^=]+)(=+)/)
  return text if t.nil?

  tokens = t.to_a
  return "#{tokens[1]} #{tokens[2].strip} #{tokens[1]}"
end

def replace_a_url(url, text)
  text.gsub!("http://#{url}", ATTACHMENT_URL_BASE)
  text.gsub!("https://#{url}", ATTACHMENT_URL_BASE)
  text.gsub!("http:/#{url}", ATTACHMENT_URL_BASE)
  text.gsub!("https:/#{url}", ATTACHMENT_URL_BASE)
  return text
end

def replace_wiki_urls(text)
  matches = text.match(/\[http(.)*\]/)
  result = text
  if matches
    result = replace_a_url("wiki.example.com/#{SITE_NAME}/wikifiles/", result)
    result = replace_a_url("pdfreviewfiles/", result)
    result = replace_a_url("cgi-bin/", result)
  end
  result
end

def convert_formats(text)
  result = text.gsub(/\[\[(\w+)\]\]/,"[wiki:" + '\1' + "]")
  result = asterick_prefixed(result)
  result = colon_prefixed(result)
  result = pound_prefixed(result)
  result = replace_wiki_urls(result)
  result = fix_headers(result)
  result
end

def oddmuse_to_trac(text)
  result = ''
  in_pre = false
  text.split("\n").each do |line|

    # Detect indented lines as these are rendered as PRE html blocks
    trimmed = line.lstrip
    is_prefixed = ((trimmed.size < line.size) and (trimmed[0] != '*'[0]))
    result << "{{{\n" if (is_prefixed and !in_pre)

    result << "}}}\n" if (!is_prefixed and in_pre)
    in_pre = is_prefixed

    # Don't process special characters when in PRE block
    if in_pre
      result << line << "\n"
    else
      result << convert_formats(line) << "\n"
    end
  end
  result += "}}}\n" if in_pre
  result
end

path = ARGV[0] || "page/U/UsingThePhones.db"
raw_page = File.new(path).read
page2 = raw_page.split(FS + '2')
page3 = raw_page.split(FS + '3')

text = get_section(page3, 'text')

pagename = File.basename(path).split('.').first
author = get_section(page2, 'username')
address = get_section(page2, 'ip')
body = oddmuse_to_trac(text)

puts "writing: #{pagename}.om"
outfile = File.new(path.gsub('.db','.om'),'w')
outfile.puts 'pagename:' + pagename
outfile.puts 'author:' + author
outfile.puts 'address:' + address
outfile.puts body
outfile.close

puts body if ARGV[0].nil?

Below, comes the Python side of the equation. Once I did the heavy lifting in Ruby, I needed to generate all the pages. As you can see, there's some parsing happening, but its very simple parsing and I was able to apply what simple Python knowledge I still retained pretty effectively:

import sys
from trac.env import Environment
from trac.wiki.model import WikiPage

print sys.argv[1]
file = open(sys.argv[1], 'r')
pagename = file.readline().split(':')[1].split('\n')[0]
author_name = file.readline().split(':')[1].split('\n')[0]
address = file.readline().split(':')[1].split('\n')[0]

text = file.read()

env = Environment('/srv/www/trac/corporate_wiki')

# Read an existing or new WikiPage:
page = WikiPage(env, pagename)
if page.text != text:
        page.text = text
        page.save(author=author_name, comment='imported from oddmuse wiki', remote_addr=address)

As you can see, this file simply takes the command line argument, open the file for reading, pull the various variables out with the remainder of the file going to the page content.

During the conversion, I noticed some pages were failing to convert at all, like "Michael'sTodoList" The culprit, the tick in the page name and thus the filename. The scripts themselves weren't the problem, it was the xargs parameter passing that was the issue (getting to that below). So I ran the below script to rename all the troublesome filenames and then added similar tick to underscore substitution in the main Ruby script above.

require 'find'
require 'ftools'

total_size = 0

Find.find('./page') do |path|
  if FileTest.directory?(path)
    if File.basename(path)[0] == ?.
      Find.prune       # Don't look any further into this directory.
    else
      next
    end
  else
    if path.match(/[\']/)
      safe_path = path.gsub("'", "_")
      puts path + ' to ' + safe_path
      File.move(path, safe_path)
    end
    end
  end

A simple up-front execution of the script was all that was needed to now be able to pick up the missing pages:

ruby fix_unsafe_files.rb

Finally, to bring it all together, I utilized find to locate all the *.db Oddmuse files which I had located into the various wiki subfolders where these scripts sat, passing the filename to the script via the xargs utility:

find . -name *.db -print0 | xargs -0 -L 1 ruby rdpage.rb

I then essentially repeated the above, but for the generated *.om files, passing to the Python mkpage.py script:

find . -name *.om -print0 | xargs -0 -L 1 python mkpage.py

One last thing, before we're done. Each wiki had attachments. So I had to think how to get those attachments referenced and linked to the Trac wiki. A little investigation revealed that each attachment got a database entry in the attachment table. My task was a bit complicated by the fact that the users over the years in the Oddmuse environment scattered their attachments everywhere and also used varying syntax to refer to them, hence, if you review the main Ruby parsing script, you'll see the "replace_a_url" method above that cleans all those funky URLs up. So the script I used below simply scans through all the directories where we had files stored and attached them to one "ConvertedAttachments" page. Probably not the ideal solution, but again, got the job done quickly.

#  type |          id          |  filename  |  size  |    time    | description | author |      ipnr
# ------+----------------------+------------+--------+------------+-------------+--------+----------------
#  wiki | ConvertedAttachments | gotapi.png | 200272 | 1218381947 |             | michael | 192.168.16.100

path = ARGV[0] || 'generate_attachments.rb'
filename = File.basename(path)

puts "insert into attachment values ('wiki', 'ConvertedAttachments', '#{filename}', #{FileTest.size(path)}, null, null, 'oddmuse','192.168.1.246');"

So, running the above needs to be piped to a SQL fisle and then executed against your Trac database. In my case, Postgresql. Since I had many directories to snatch up, I ran it for each folder as appropriate and simply kept appending until I had all the attachment directories represented.

ls | xargs -0 -L 1 >> attachments.sql
psql trac_db < attachments.sql

All in all, not too bad of a conversion for a few hour's work. Its about 95% correct. Some of the page names didn't translate to Trac, and could probably do with some regexp search and replace on the page names themselves. The biggest gain is that we have preserved the information in the old systems by porting it to the new systems we all know and use on a daily basis, so WikiGardening is more likely to keep the content fresher and its one less system for our end-users to figure out how to use.

I can now retire the server running Oddmuse, stop doing backups and keeping up with documentation and training new system administrators on how to manage Oddmuse AND Trac.

A bit of hunting around turned up this list of zip code databases. The first link to CivicSpace database appears to be defunct (I got a Go Daddy “this page is parked” page). So, the next free one, offered by Popular Data turned out to be exactly what I was looking for. It may be a “little old” as the review site says, but its good enough for me to get started with associating my data with geocoded locations and explore the wonderful world of mapping offered by Google’s Maps API.

To load it up in MySQL and PostgreSQL, I used the following Ruby script to generate a bunch of insert statements:

require 'rubygems'
require 'fastercsv'
source_file = 'ZIP_CODES.txt'
outfile = File.new('us_zip_codes.sql','w')

# generate the truncate statement if we want to empty the table
outfile.puts "truncate us_zip_codes;"

# generate the SQL Insert statements
FasterCSV.foreach(source_file, :headers => false) do |cols|
  row = "INSERT INTO us_zip_codes VALUES ("
  row << as_string(cols[0]) << ","
  row << as_number(cols[1]) << ","
  row << as_number(cols[2]) << ","
  row << as_string(cols[3]) << ","
  row << as_string(cols[4]) << ","
  row << as_string(cols[5]) << ","
  row << as_string(cols[6]) << ");"
  outfile.puts row
end
outfile.close
puts "copied #{i} records."

Create the zip code table with the following SQL:

create table us_zip_codes (
  zip_code char(5),
  latitude float(10,6),
  longitude float(10,6),
  city varchar(255),
  state char(2),
  county varchar(255),
  zip_class varchar(255)
);

And finally, with the table in place and the SQL insert statements generated, I loaded the data into MySQL with the following from command line:

mysql target_db_name < us_zip_codes.sql

The problem: I wanted to introduce a property editor metaphor for the website I’m working on where properties could expose edit forms for just about any situation, much like wordpress’ widget interface does.

That is, there’s an area of the current page that shows basic information about the property in a read-only (and preferably compact) form, and user could click an “edit” button and exposed an editable version of the properties. The “edit” link becomes a “cancel” link, which, when clicked, simply discards the edit form and return the initial compact view state.

The problem is, Rails doesn’t really have any good helpers for reusing a design concept in your views. I found that even with partials, the code was really starting to bog down and become both unmaintainable and unreadable. I needed a simple mechanism to construct my property editors. I started out with helper methods. But that wasn’t much better as the code just started ending as ruby here-docs and became a maintenance issue of another sort! Take a look at this embarrassment:

result = "<table><tr>"
result += "<th colspan=2>"
result += button_image_action('Group selected together', 'link.png')
result += button_image_action('Copy selected to this group', 'link_add.png')
result += button_image_action('Move selected to this group', 'link_go.png')
result += button_image_action('copy group', 'page_copy.png')
result += button_image_action('cut group', 'cut.png')
result += button_image_action('paste group', 'page_paste.png')
result += button_image_action('delete group', 'delete.png')
result += "</th></tr>"
result += "</table>"

Ok, I admit it, I chopped out more than half again as many lines as you’re looking at there, and perhaps worse, this isn’t actually the original HTML code that led me down this path, but lets take a look at this approach anyway, as I learned something valuable in the process and it was my first attempt at DRYing my views. First thing, first, get the above HTML back into the erb file:

			<%= button_image_action('Group selected together', 'link.png') %>
			<%= button_image_action('Copy selected to this group', 'link_add.png') %>
			<%= button_image_action('Move selected to this group', 'link_go.png') %>
			<%= button_image_action('copy group', 'page_copy.png') %>
			<%= button_image_action('cut group', 'cut.png') %>
			<%= button_image_action('paste group', 'page_paste.png') %>
			<%= button_image_action('delete group', 'delete.png') %>
		

Which promptly let me to trying something like this:

def button_image_action(title, image_filename, button_text = '', url = '')
    form_for @cube url do |f|
  	  <<-TEXT
  	    #{button_text}
  	    
  	    
  	  TEXT
     end
  end

I thought it was shaping up nicely, but I got completely blindsided when it came to rendering this bit. You see, the Rails Form Helpers are unavailable in a regular helper! They require ERB (being inside <% %> and <%= %>) to actually properly render their blocks.

I fiddled around for a bit to try to jump this hurdle, but it just simply ain’t gonna work without some serious hacking. I’m happy to report that I got a little wiser and stepped back from the problem and reevaluated my approach in general. I realized, I wasn’t getting away from doing HTML code in helpers! Bottom line: Helpers are great, but you’re doing something clearly against “The Rails Way” when you start rendering large chunks of HTML code from helpers. It just isn’t a great approach and if you have designers on your team, they can get quite lost hunting around for all those HTML snippets.

So, how could I get all of my HTML back into *.erb files where they belonged, yet develop some helpers that would make mincemeat of the coding effort to build these fancy toolbars and forms I had coming? Fellow colleague, Steven Beales said, “you need to use Ruby blocks.” To which, my first reaction was, “what are Ruby blocks?” To my great benefit, he explained by pointing me straight at Block Helpers and DRY Views in Rails.

Truth be told, I mostly understood this article, but not entirely. Even so, I dutifully set off to solve my problem with this approach. This led me to a bit of copying and pasting (and crediting)…

  # From:  http://errtheblog.com/posts/11-block-to-partial
  #
  # captures the output of our passed block using Rails’ capture method and adds
  # it to our options hash under the key of :body. Next it renders our passed
  # partial, sending in our options hash as :locals
  def block_to_partial(partial_name, options = {}, &block)
    options.merge!(:body => capture(&block))
    concat(render(:partial => partial_name, :locals => options), block.binding)
  end

Look at that carefully and, if I may be so bold to suggest, go read up on ERR the Blog to better understand as this method is key and I daresay was appropriately called “tomfoolery” by the wise authors therein. This bit of code really forced me to finally begin to learn and understand what the heck block passing was all about. In a nutshell, this little gem lets you pass options that become local variables to the partial and the block is the contents to be rendered within the partial when you yield to the passed block.

  # Calls given partial, injecting the options hash into the partial's local hash
  def variables_to_partial(partial_name, options = {})
    render(:partial => partial_name, :locals => options)
  end

The block_to_partial is a great start, but I wanted to simplify my calls and also enforce a bit of use-pattern on the developer (i.e. me) so that all property editors had essentially the same behavior pattern (and components). Thus, we introduce the property editor container which in turn can have one or more sections associated with it:

  # Generates a division around the given block that is styled with "property-editor"
  # Property editors are intended as a view on data that has both a condensed display and
  # an expanded display that lets you edit the information contained in the property editor.
  def property_editor(id, title, options = {}, &block)
    @id = id
    block_to_partial 'layouts/property_editor', options.merge(:id => id, :title => title), &block
  end

  # Renders content of a section (provided by block) inside a property editor division
  # The stylesheet contains styling for the following section_id's:
  #   => "show" - the section initially visible
  #   => "edit" - the section made visible when the edit button is clicked
  def property_editor_section(section_id, options = {}, &block)
    block_to_partial('layouts/property_editor_section',
      options.merge(:section_class => "property-editor-#{section_id}",
        :section_id => "#{@id}-#{section_id}"),
        &block)
  end

The property editor partial:

<div <%= %[id="#{id}"] %> class="property-editor">
	<div class="property-editor-title">
		<div style="float:right">
			<a href="javascript: void(0)" onclick="toggle_div('<%= id %>-show');toggle_div('<%= id %>-edit');">edit</a>
		</div>
  		<%= title %>
	</div>
	<%= body %>
</div>

and the property editor sections are rendered with this partial:

<div <%= %[id="#{section_id}"] %> <%= %[class="#{section_class}"] %> <%= %[style="display:none"] if section_id =~ /(.+)?\-edit/  %>>
	<%= body %>
</div>

With these helpers and partials, property editors as a concept is very easy to pull together in my view files. As you can see, if my section is an “edit” section, its hidden at the outset. The button to toggle between show and edit is in the property editor partial itself in which an onclick javascript calls toggle_div to make this happen. The Javascript is very straightforward:

function toggle_div(div_name) {
	var div = $(div_name);
	div.getStyle('display') == 'none' ? div.setStyle('display', 'block') :	div.setStyle('display', 'none');
}

Here’s just one such property editor operating on my “cube model”:

<% property_editor 'properties', 'Properties' do %>
	<% property_editor_section 'show' do %>
		<%= @cube.name %><br/>
		<%= render :partial => '/cubes/cube_stats' %>
	<% end %>

	<% property_editor_section 'edit' do %>
		<%= render :partial => 'form' %>
	<% end %>
<% end %>

What’s more, I had a complete set of CSS selectors with which I could control my property editor and its associated sections:

.property-editor {
	margin-top: 5px;
	border: 3px solid #FDEFE6;
}

.property-editor-title {
	padding: 2px;
	padding-left: 5px;
	padding-right: 5px;
	background-color: #EEE;
	color: #2F6569;
}

.property-editor-show, .property-editor-edit {
	padding: 5px;
}

.property-editor p {
	margin: 0px;
	padding: 0px;
}

Well, all in all, that’s a good bit of rambling, but I hope others can benefit from my experiences with Ruby, blocks, partials, and DRY up those views.