I won’t cover a complete Ruby and Rails/Ramaze/Sinatra installation. This post assumes you have that much down pat, so we’ll pick up the ball with installing FreeTDS, unixODBC, Ruby ODBC

If you’re using Ubuntu 8.10, you can get away with installing everything from packages with the following:

apt-get install  unixodbc freetds-common tdsodbc \
             libodbc-ruby1.8 \
             libdbi-ruby1.8 libdbd-odbc-ruby \
             autoconf make

NOTE: I wrote this article and right before publishing, attempted to replicate these results on both mac OS X and CentOS 5. Both failed to work with the technique I thought I had all nice and worked out. I spent several hours over many months trying to find out exactly what’s different between the packages, but never got very far with my investigations. For a long time, it looked to me like both the ODBC packages and the FreeTDS packages need to be tweaked to properly handle [SERVER]\[DB_INSTANCE] construct that MSDE uses vs. just [SERVER] that the full-fledged SQL Server uses. This, after all, is the convention of the Windows ODBC driver and DSN definitions. However, there’s a really simple solution that I hit upon on Actual Technologies’ website that led me to a true universal solution and that’s to learn the port the INSTANCE runs on and set up the DSN specifically to that port!

But beware, the above doesn’t get you all of the freetds tools, most importantly the tsql command which can help you with debugging issues before getting into the ODBC game (that is, direct TDS connections to your SQL Server). So, to cover the most distros possible, I’ll document the “from source” approach for *nix systems and macports for Macs so you’ll find two sets of installation instructions for every step.

Preparing your Environment

I prefer to pull down all source packages into my local user directory under a “~/src” folder. The instructions that follow assume you’re in this folder. If you’re not, then please make the appropriate substitutions as you follow along. Additionally, I configure and compile most things as a regular user then use sudo to install system-wide as a “super user.”

Install unixODBC

unixODBC provides the ODBC connectivity for the TDS driver.

wget http://www.unixodbc.org/unixODBC-2.2.14.tar.gz
tar zxfv unixODBC-2.2.14.tar.gz
cd unixODBC-2.2.14
./configure  --enable-gui=no
make
sudo make install

On Macs

sudo port install unixODBC

On CentOS

sudo yum install unixODBC unixODBC-devel

Install FreeTDS

FreeTDS provides the open source implementation of the protocol implemented by both Sybase and Microsoft SQL Server.

wget ftp://ftp.ibiblio.org/pub/Linux/ALPHA/freetds/stable/freetds-stable.tgz
tar zvxf freetds-stable.tgz
cd freetds-0.82/
sudo ./configure  --with-unixodbc=/usr/local
sudo make
sudo make install

On Macs

sudo port install freetds

On CentOS

sudo yum install freetds

Install ruby-odbc

The ruby-odbc package provides the ruby bindings for ODBC connectivity.

wget http://www.ch-werner.de/rubyodbc/ruby-odbc-0.9995.tar.gz
tar zvxf ruby-odbc-0.9995.tar.gz
cd ruby-odbc-0.9995/
ruby extconf.rb
make
sudo make install

On Macs

sudo port install rb-odbc

Install ruby DBI

wget http://rubyforge.org/frs/download.php/655/ruby-dbi-all-0.0.23.tar.gz
tar zvxf ruby-dbi-all-0.0.23.tar.gz
cd ruby-dbi-all
ruby setup.rb config --with=dbi,dbd_odbc
ruby setup.rb setup
sudo ruby setup.rb install

Configure FreeTDS and ODBC

If you’re connecting to either SQL Server 2000 or SQL Server 2005 Standard, Enterprise, etc., then the typical approach is to add an entry for the server to /etc/freetds/freetds.conf and then reference that entry in the /etc/odbc.ini. Sometimes, these files get installed in /usr/local/etc. If so, then I will usually symlink the files into /etc for convenience, so that’s what’s referenced below.

My server is called “apvdbs01″ (the WINS name) with an IP Address of 10.0.2.2 and the database I’m connecting to is called “test_development” while “username” and “passwd” should be substituted with an valid user account and credentials for successfully connection to the database server.

Add a server entry for your server (which I typically give same name as Windows Server) to /etc/freetds.conf, so that host points to the SQL Server address:

[apvdbs01]
host = 10.0.2.2
port = 1433
tds version = 7.0

TDS version 7.0 is for SQL Server 2000. If you have SQL Server 2005, then tds version is 8.0.

tsql –S apvdbs01 –U username –P passwd

Once you can connect directly with the TDS drivers, move on to adding the ODBC driver entries.

Add the FreeTDS ODBC driver by editing “/etc/odbcinst.ini” (make sure “/usr/local/lib/libtdsodbc.so” exists first):

[FreeTDS]
Description     = TDS driver (Sybase/MS SQL)
Driver          = /usr/lib/odbc/libtdsodbc.so
Setup           = /usr/lib/odbc/libtdsS.so

Add a new DSN to “/etc/odbc.ini”:

[test_development]
Driver          = FreeTDS
Description     = ODBC connection via FreeTDS
Trace           = No
ServerName      = apvdbs01
Database        = test_development

So, “FreeTDS” is the driver that we gave for the lib TDS ODBC connector in the odbcinst.ini file (this allows you to install and maintain multiple versions of the libtdsodbc.so file). We reference this driver in the odbc.ini file with the “Driver” entry and we reference the actual server data in the freetds.conf file via the “ServerName” entry.

Now that you have ODBC configured for FreeTDS, test it with:

	isql test_development username passwd

But that doesn’t work for MSDE!

The above will get you connected to a standard or enterprise SQL Server 2000/2005 installation, but does not work for an Embedded SQL Server 2000/2005 installation. After trying many permutations, I finally came up with a configuration that actually works. One of the key things to know about MSDE is that the MSDE server instance is in the form of “servername\instance” rather than straight up “servername” as with standard/enterprise installations. This is known as a “named instance.” The trick to connecting? Find out the port the named instance is listening on and connect to that port rather than the 1433 port!

You can determine the port used by the instance by using the Server Network Utility in the Microsoft SQL Server program group on the Windows server where your database resides. Select the instance name from the drop down list and select TCP/IP from the list of enabled protocols. This will show you the port number for that instance.

[test_msde2000]
Driver         = FreeTDS
Description    = ODBC connection via FreeTDS
Server         = 10.0.2.2
Database       = my_test_db
Port           = 1348
TDS_Version    = 7.0

I could then connect to the MSDE instance like so:

isql test_msde2000 username passwd

Test ODBC connectivity from Ruby

If both tsql and isql commands work, then you’re well on your way to connecting to SQL Server via Ruby. To test Ruby with the above ODBC configurations:

irb
require "dbi"
dbh = DBI.connect("dbi:ODBC:test_development", "username", "passwd")

Connecting with Sequel

One of my favorite tools for connecting to SQL DBMS’ is Sequel, which is maintained by Jeremy Evans. To connect using Sequel, install the gem:

sudo gem install sequel

And then give this a shot:

irb
require 'rubygems'
require 'sequel'

DB = Sequel.odbc('test_development', :db_type => 'mssql', :user => 'username', :password => 'passwd')

References

http://www.freetds.org/userguide/confirminstall.htm

http://www.unixodbc.org/odbcinst.html

http://www.actualtech.com

Preamble (a.k.a. Ramblings)

I can hear you exclaiming, “Two hours!? Are you nuts? You really don’t want to do that on a web application,” and you would be correct; I did not. But alas, my users wanted it and they waved money in front of my face to get it done. What’s a poor sap like me to do but to comply?

So, what is this crazy task that needs to run so urgently it couldn’t wait until a midnight cronjob task to kick it off? Well, the task is basically exporting data from the live production database to a reporting database and the users sometimes have to generate those reports as soon as humanly possible against the then current production data. The problem is, the export process takes anywhere from 15 minutes to almost two hours, depending on how much data has changed since the last midnight run.

Such a long-running process is clearly outside the normal bounds of a website’s usual request/respond lifecycle and, if you’re like me and fronting your web app with Phusion’s Passenger, you’re quite at risk of Passenger’s process recycling action killing the task in mid-stream and losing your work in progress.

I didn’t quite know what I was going after when I started solving this problem and the main thing I could think to google for was “background jobs ruby,” which led to a rash of articles about backgrounDRB and many many rubyists complaining about its many shortcomings and creating their own spin on what, at core is known as a message queue. Many of the articles I read just did not cover the whole scope of the problem I faced and usually had simple solutions that lived well within the boundaries of their Rails (or Ramaze) framework’s inner workings.

Since I had a process that was capable of consuming resources upwards of two hours, I knew I had to completely escape the bounds of the framework and do the heavy lifting without fear that the parent process’ lifespan would risk the job getting terminated mid-stream. I also didn’t need a Rails plugin because, well, I’m building a Ramaze application and those fancy plugins just aren’t much of a help to me there.

One of the things I’m loving about Ramaze is that its forcing me to learn the Ruby language and its also forcing me to work harder to understand the components I’m pulling together in a solution. With Rails, I just pulled a plugin out of thin air, plugged it in and moved along. Which is great for rapid development, but more often than not, I have found myself later in a bind after the site’s live finding out that wonderful plugin is running way too slow and not knowing a darn thing about how to fix the problem without tearing up the whole plugin with a complete rewrite (in many cases).

With Ramaze, I feel much closer to the language, and I’m also finding that there’s just a lot of fluff I flat out don’t need that the Rails framework brings along, or rather, encourages developers to toss into the soup. On the other hand, Ramaze’s philosophical approach of squeezing every ounce of functionality out of every line of code has really pushed me towards being a better Ruby programmer and yet the same old time crunch every nearly developer faces actually forces me to get the job done with less code and with utilizing only the critical components actually necessary to get the job done. Which amounts to a one-two knockout punch for me because the solutions I deploy are consistently rock solid and I fully understand what I just put out there end-to-end.

Getting Started

The full project source can be found on github.com and you will also find my database enhancements to ProgressBar on github.com as well.

This project utilizes the following:

• Ruby Language
• Ramaze
• beanstalkd
• JQuery
• A modified version of ProgressBar (included in the demo source)
• These Gems
  • daemons
  • json
  • beanstalk-client

The Players

With the ramblings out of the way, its time to get to the heart of the matter at hand. There are three players in this game:

1. The trigger – This is the process that starts the whole chain reaction. In this case, the web app where the user can click a link to start the export. In my case, that web app was built with Ramaze.
2. The message queue – This is the middleman in the game. He sits around and accepts the message from trigger and passes the buck to the worker. We call him, “beanstalkd.”
3. The worker – This is the process doing all the work. Amazingly, Ruby was up for the task and the daemons gem is the ticket to making this work.

So lets cover these in more detail…

The Worker (sort of…)

The worker has got to always be around to do the job whenever that job comes in, so this suggested to me that I needed a daemonized process to watch the message queue for incoming jobs. Many solutions that I saw called for simple cronjobs running every minute or 5 minutes or whatever. But I couldn’t fathom why I’d want to resort to such hackery. How are you going to scale that? What happens if two cronjob instances fire and grab the same job request? What about if the job crashes or the server dies in mid-job? How is it reliably restarted? How do I ensure long-running jobs don’t pile up and consume all memory and CPU resources (e.g. every minute start processing the next big job and next thing you know, ten of ‘em are running when you only have the capacity to run one or two)? Simply put: cronjob workers is brute force mentality and rife with potential pitfalls.

Point being, I rapidly came to understand that I needed a message queue to serialize the jobs until the workers could take the jobs and run them and guarantee the jobs run before flushing them away. What was surprising, was the number of message queue implementations out there in the wild! Luckily, beanstalkd existed and had a very elegant implementation and API well-suited for Ruby. I easily installed from source to a Ubuntu server and also installed via macports on my Macbook Pro laptop AND installed the ruby client AND tested the small demos in irb all in about 30 minutes. Sadly I can’t say the same for RabbitMQ, Starling/Workling, ApacheMQ, and others I tried in vain to get up and going with. So, beanstalkd won out by default. Go to the beanstalkd website and follow their instructions. Its that simple.

How are we gonna keep up with the jobs?

Although the message queue has a way of peeking into it to learn about number of jobs and what’s running and the status of said jobs, the background processes I was going to be doing was going to run long enough that I wanted to pass along exactly what stage in the process the job was crunching and this meant building periodic update logging into the actual background job itself. The problem then is, how do we pass the information cleanly between worker and web app?

After some thinking on the matter, I chose to pass along progress updates via the database since all the headaches of I/O contention, persistence and deadlocking had already been solved there. I wasn’t quite sure how I was going to format my status updates and post the info to the database until I thought about the great little ProgressBar gem that I was already using for the task when I kicked it off at console. After a few minutes of looking into the internals of ProgressBar, I knew I could easily extend it to report its status to the database rather than emitting to the console. An hour later, I had my DbProgressBar and another hour later I had my unit tests written and passing.

Oh, yeah, the Worker…

I digressed from talking about the worker to covering the message queue and passing messages around, so lets get back to that worker script. As I pointed out earlier, I needed a worker script to do the heavy lifting for the job at hand and I wanted that puppy daemonized. One thing was perfectly clear in my mind: I didn’t want cronjobs nor Threads that depended on the parent processes staying alive (remember how Passenger aggressively scrubs ruby processes from memory!) nor any other means that depended on the parent process within the Ramaze framework kicking off the process directly.

The “daemons” rubygem fits the bill perfectly! I was really surprised by this little gem of a gem. Daemons makes it extremely easy to wrap ordinary Ruby scripts in the usual start|stop|restart|status world you find inside the typcial /etc/init.d/ Linux folder. What blew my mind was that I didn’t have to look at the usual long-winded case statement for all those daemonizing commands. Just focus on writing the main task for the worker, wrap that task in an infinite loop, install it rc.d folders as appropriate and you’re pretty much done. The following shows an example worker script:

require 'rubygems'
require 'daemons'
require 'beanstalk-client'
require 'sequel'

ROOT_DIR = File.dirname(File.expand_path(__FILE__))
$LOAD_PATH.unshift(File.join(ROOT_DIR, 'lib'))

require 'dbprogressbar'
require 'sequel_output'

Daemons.run_proc('process_astroid_jobs', :log_output => true) do
  DB = Sequel.sqlite(File.join(ROOT_DIR, 'demo.db'))
  SequelOutput.prepare_database(DB)
  beanstalk = Beanstalk::Pool.new(["0.0.0.0:11300"])

  loop do
    job = beanstalk.reserve
    jh = job.ybody
    puts "processing #{jh[:job_id]}"

    total = 15
    pbar = DbProgressBar.new("job ##{jh[:job_id]}", total,
      SequelOutput.new(jh[:job_id], DB))

    total.times do
      pbar.inc
      sleep(1)
    end

    pbar.finish
    puts "finished #{jh[:job_id]}"
    job.delete
  end
end

There are a few key things to note about the above script. Firstly, wrapping it in the Daemons.run_proc tells the daemons gem we want to run this like we’d run a typical server. SequelOutput is the extension I wrote for ProgressBar to get it to outout to the database and at the start of the script, we ensure the database is ready (namely that the “progress_bars” table is created) before we go into our infinite loop to process jobs.

Next up, the loop itself. Its not a runaway infinite loop that’ll end up consuming all resources because we are calling the beanstalk queue with “reserve” which is a blocking call, thus we sit and wait until a job comes through. As soon as we get a new job request, we create the DbProgressBar and get to work processing….twiddling our thumbs for 15 seconds in this case! When the job’s completed, we remove it form the message queue and begin again.

The Trigger

Finally, with the back-end done, we could focus on enabling the user to trigger the job on the front-end. Of course, that was as simple as firing up a connection to the beanstalk server and pushing a job onto the queue in a controller’s action. But how do we watch the job and report to the user what’s going on without overwhelming our server? The answer came along with a “Comet with Ramaze” example by Pistos. What a lovely idea: Let the server intentionally hold onto the client’s request until it had something to report back! I knew it’d never hold onto the request for more than 2 or 3 seconds, so this was a perfectly workable model for keeping traffic levels sane and timely.

This was also my first time playing around with JSON in AJAX calls and so I got to spend some time flexing my JQuery muscles a bit and also discovered the Ruby json gem that makes it dirt simple to turn a hash into a JSON response. This is just a small excerpt from the controller/main.rb check_job_progress action:

# report status as finished when the job is completed
if progress[:job_finish]
  return {:progress_text => "finished.", :progress_status => "finished"}.to_json

# otherwise report the percent done as the job status 
else
  return {:progress_text => progress[:progress_text].gsub(' ', ' '),
      :progress_status => progress[:progress_percent]}.to_json
end

Notice the call to the to_json to convert that hash to JSON response and how clean the Ruby code is even though we’re passing a rich data object back to our browser client. I substituted blanks with the   character to keep the progress bar from collapsing all those empty blanks between the two vertical bars. I decided to pass on a progress status so that I could check when things were done and thus stop polling the server for job updates. This is just a demo, so try not to snicker too much at my use of plain English to convey critical job statuses. At least the intent is clear!

The JQuery script is also correspondingly simple.

function check_progress() {
	job_progress_element = $('#job-progress');
	$.getJSON('/check_job_progress', {job_id: job_progress_element.attr('value')},
	  function(data){
	    job_progress_element.html(data.progress_text);
      	if (data.progress_status != "finished") check_progress();
	});
}

$(document).ready(function() {
	if($('#job-progress').length) check_progress();
});

As you can see, it simply asks the server for an update and when it receives one, updates itself and asks again! It will check the job status after each update to see if the job has completed or not, thus stopping its persistent nagging once we’ve reached the Kingdom of Far, Far Away.

References

Here are a few other references, not mentioned above, that were invaluable in helping me pull this solution together.

Ruby Beanstalkd Distributed Worker Basics
How to Create a Daemon Process Using Ruby and the Daemons RubyGem
Err the Blog – About this blog Beanstalk Messaging Queue”>

Its not that I don’t think these tools are worthy contenders. Most of them, I have used fairly extensively myself over the years. The suggested contenders are solid and mature projects, but they feel decidedly “old hat” to me much as apache, postfix, vim, bind, and the linux kernel probably feels to the system administrator. In other words, they’re ubiquitous, so why signal them out with some sort of top 100 reward?

These days, I have adopted Ruby as my language of choice, so I have moved away from Python and PHP. While this has largely meant I develop Ruby on Rails applications. However, as my Ruby skills grow, this also means going far and wide in many other ruby-based tools (such as rake, sake, and rack for system-level or middleware tools). I am now using ruby for quick and dirty systems administration tasks (such as the short script I wrote to find all currently assigned IP addresses on our office network), automating deployments (capistrano, puppet, chef), marshaling data from one database system to another, and even with extensive image-processing (resizing, conversion of formats, compressing, etc.).

So what, in my view, warrants a top 100 award recognition? I like to see new and exciting projects on these lists. If a project’s on the list, I like to know that they are extremely active and rapidly gaining mind-share and penetration into the community. I want to know a little bit about how they put a new spin on an old problem and the impact they have been able to make, not only in the project’s direct vertical niche, but across the board. A few years ago, Ruby on Rails arrived on the scene as a game-changer, not only for putting Ruby on the map as a viable language, but as a bona fide development framework for delivering web-based services. Ruby on Rails turned the Java world on its head and forced Java developers to drastically improve their productivity tools then in common practice; they evangelized convention over configuration and brought many fantastic ideals such as Agile development and RESTful APIs to the forefront of many, many developers’ minds.

To my way of thinking, though, Ruby on Rails is now old news. Newcomers will stumble upon Ruby and Rails through word of mouth or simply googling for a solution to some problem they’re trying to solve. That’s the problem I see with top-100 lists paying tribute year after year to the same old mature projects. Whether its Ruby on Rails or Word Press or Joomla or Django, all of these are easily discovered and well-written about by anybody with decent querying skills. So what’s out there today that’s refreshing and hard to find? Yet, is making a huge impact on the community? Here are a few that I think are worthy contenders (and yes, they are a little heavy on the Ruby side)…

Rack

Rack: a Ruby Webserver Interface. What I find phenomenal about Rack is its adoption by so many frameworks that are delivering web-based services. Micro-frameworks like Ramaze and Camping and Sinatra were early adopters of Rack and such adoption led to far smaller code-bases in each as well as significant performance boosts and, as if that wasn’t enough, enabled much easier transition from Ruby 1.8 to 1.9.

Inkscape and ImageMagick

Speaking of image processing, I am not sure if many have encountered ImageMagick and the Ruby bindings offered via RMagick, but definitely worth a look-see. As of this year, I do almost all of my creative graphics work in Inkscape and The Gimp and use ImageMagick to automate some pretty impressive transformations of those images outside the GUI. Before Inkscape became mature enough to use on daily basis (which, by my reckoning is only in the last 18 months or so), I was heavily reliant on Adobe’s Illustrator and companion Photoshop apps. I never would’ve imagined two years ago that I could completely do away with these packages, let alone begin to automate much of the repetitive tasks that I put those resulting images through (resizing, cropping, web-optimizing, etc.) with a collection of well-written OSS tools.

Ramaze and Sequel

One last pair of tools I think is worthy of consideration, especially for web-based ruby developers is Sequel and Ramaze. Sequel is an object-relational-mapping library that makes working with SQL databases a real pleasure for Ruby developers. The maintainers seem to have taken the best of the best and rolled those ideas into one coherent and extremely fast library for accessing just about any database out there. I have fooled around with DataMapper and have worked extensively with ActiveRecord and I must say Sequel really is rung or two above these libraries and I look forward to continuing to adopt Sequel.

I ran into Sequel by way of the Ramaze project, which is a micro-framework for building web-based applications. Unlike Ruby on Rails, which tends to generate tons of scaffold/skeletal code for you to begin hacking away on to build your projects, Ramaze takes the opposite tack in that it provides little to no scaffolding or auto-generated code. Instead, the mindset is squeezing the maximum amount of functionality out of every line of code you do write. For a Ruby on Rails developer coming to Ramaze, the feeling is very much like the first time I took off on a bicycle without the training wheels attached. You feel wildly uncontrolled and like you may fall at any moment, but a little encouragement from the warm friendly community gets you going straight and true and the amount of code you’re NOT writing, reading, reviewing, nor specking tests for really begins to jump out at you!

The Problem

I am working on a content management system (CMS) where I want the pages to have a hierarchical structure that turns into a menu with sub-menus. The content manager needs to also be able to order these pages so that the menu structure renders in the desired order.

The Solution

Two plugins jumped to mind almost immediately: acts_as_tree and acts_as_list. The tree plugin will manage the hierarchy, hinging off the parent_id field of the model whilist the list plugin uses the position column to manage the order. What’s unique here is that I have never used both on one model, but doing so was surprisingly easy:

   1  class Page < ActiveRecord::Base
   2    acts_as_tree
   3    acts_as_list
   4  end

Constructing a Hierarchical Menu

Before going too far, if you haven’t seen the Suckerfish menus, yet, please do check out the article as it will help you quickly see how I approached the menu rendering. Secondly, to install the two plugins, its a simple pair of command line calls as follows:

   1  script/plugin install acts_as_tree
   2  script/plugin install acts_as_list

With the CSS handling all the styling, all I needed was to render the nested unordered lists. I began by grabbing all of the pages at the top level (where parent_id is null) with this bit of code:

   1  class Page < ActiveRecord::Base
   2    acts_as_tree :order => :position
   3    acts_as_list
   4 
   5    def self.top_level_pages
   6      find(:all, :conditions => ["parent_id IS NULL"], :order => :position)
   7    end
   8  end

If you noticed the “:order => :position” clause and thought, “but acts_as_list handles that for you,” then you have spotted the first “gotcha” I encountered with using both tree and list on a model. The tree plugin loses the position ordering that the list plugin mixes in and adding these order clauses in preserves the order of the records. With the query to get the top-level menus in place, I set the @pages variable by calling Page.top_level_pages in the controller and then rendered with this call in my view:

   1  <div id="site-navigation">
   2  	<ul>
   3  		<%= render(:partial => "/layouts/site_navigation", :collection => @pages) %>
   4  	</ul>
   5  </div>

To make it all nice and nested, I simply recursively called the same site_navigation partial on the children like so:

   1  <li><%= link_to(site_navigation.name, url_for(site_navigation))  %></li>
   2  <ul><%= render(:partial => "/layouts/site_navigation", :collection => site_navigation.children) %></ul>

To give the user ability to move the menu items around, I added “move_up” and “move_down” actions to the Page Controller in which I called the “move_higher” and “move_lower” methods that are mixed in by the acts_as_list. I realized rather quickly that the position index was getting out of sync and added a reindex method to the Page model to clean up the data along with a scope declaration to the model. Along the way, I also realized that if I moved a page from one parent node to another parent node, I potentially opened a gap in the position index in the collection of Pages and this again breaks the position index sequencing. The acts_as_list mix-in expects the position index to always go [0, 1, 2, 3, ...N]. Whenever this is not the case, the move_higher and move_lower methods stop working and the user interface no longer responds correctly. So this is why we care so much about the position index sequence. So, to handle scoping of the position index sequence correctly and to handle Pages being moved to another parent, we arrive at this final version of the Page model:

   1  class Page < ActiveRecord::Base
   2    acts_as_tree :order => :position
   3    acts_as_list :scope => :parent_id
   4 
   5    before_save :keep_position_sane
   6 
   7    def self.top_level_pages
   8      find(:all, :conditions => ["parent_id IS NULL"], :order => :position)
   9    end
  10 
  11    def self.reindex_top_level_pages(recurse = true, departing_child = nil)
  12      reindex_pages(self.top_level_pages, recurse, departing_child)
  13    end
  14 
  15    def reindex_children(recurse = true, departing_child = nil)
  16      Page.reindex_pages(children, recurse, departing_child)
  17    end
  18 
  19    private
  20 
  21    # takes a given array of pages and recursively (or not) reindexes
  22    # if departing_child is supplied, it is removed from the array so 
  23    # that former siblings are reindexed as though it was already 
  24    # removed from the collection.
  25    def self.reindex_pages(pages, recurse, departing_child)
  26      pages.select{|r| r != departing_child}.each_with_index do |page, index|
  27        page.reindex_children(true) if recurse
  28        page.update_attributes(:position => index + 1)
  29      end
  30      true
  31    end
  32 
  33    # When the parent id of a node changes, the acts_as_list gets lost, so 
  34    # we need to reindex the affected nodes to keep things sane
  35    def keep_position_sane
  36      return unless self.parent_id_changed?
  37 
  38      # reindex the group this page is being removed from
  39      if self.parent_id_was.nil? then
  40        Page.reindex_top_level_pages(false, self)
  41      else
  42        Page.find(self.parent_id_was).reindex_children(false, self)
  43      end
  44 
  45      # make this page the last sibling of the new parent group of pages
  46      last_page = (self.parent_id.nil? ? Page.top_level_pages.last : Page.find(self.parent_id).children.last)
  47      self.position = (last_page.nil? ? 1 : last_page.position + 1)
  48      true
  49    end
  50  end

Check out the “keep_position_sane” callback. You’ll see a nifty application of the new Dirty records feature of ActiveRecord (which I believe was released with Rails 2.2) and was covered by Ryan Diagle in his post,
What’s New in Edge Rails: Dirty Objects. In order to detect that the parent node was indeed changing and which parent’s collection the Page belonged to, I checked the “parent_id_was” value. In this case, I had to handle top level pages being moved into another Page’s collection as well as A sub-page being promoted to top-level.

A Handy Rake Task

If you’re wondering why the recursive parameter and otherwise unnecessarily complex procedures, its because I also added a rake task to recursively reindex all pages in the menu hierarchy:

   1    namespace :pages do
   2      desc "re-index page positions"
   3      task :reindex => :environment do
   4        Page.reindex_top_level_pages
   5      end
   6    end

Conclusion

Rails has really come a long ways since its early days, but its still not without its little “gotchas” and sometimes its tough to uncover silent failures (like with the position order clause not being rendered from the acts_as_list plugin) when mix-ins from different plug-ins are utilized. Hopefully, this article shows you a few new tricks and how to utilize somewhat competing plugins safely within one model. The new Dirty attributes feature of ActiveRecord definitely made the chore of implementing this functionality much easier and sane than it would’ve been back in the Rails 1.x days.

Environment

This post will guide you through setting up the following environment on Ubuntu Intrepid (8.10):

• Apache 2.2.9
• Passenger 2.0.6
• ActiveRecord 2.2.2
• Ramaze 2009.01
• Erubis 2.6.2
• MySql 5.0.67 (assumed installed)

To get started:

sudo apt-get install apache2 passenger
sudo passenger-install-apache2-module

Install the Rack, Ramaze, Activerecord, and Erubis gems

sudo gem install rack, ramaze, activerecord, erubis

Establishing Project

Now create your first ramaze project with:

ramaze --create demo

Then change to the demo directory. If you see a start.ru file, rename it to config.ru or Passenger won’t find it. Create a file called passenger.rb with the following:

require 'rubygems'
require 'ramaze'

# Add directory start.rb is in to the load path, so you can run the app from
# any other working path
$LOAD_PATH.unshift(__DIR__)

# Initialize controllers and models
require 'controller/init'
require 'model/init'

Ramaze::Global.sourcereload = false
Ramaze::Global.sessions = true
Ramaze::Log.ignored_tags = [:debug, :info]

Important: Do not include “Ramaze.start …” in this file or this will surely mess things up for Passenger.

Now edit the config.ru file (formerly start.ru):

/usr/bin/env rackup
# start.ru for ramaze apps
# use thin>=0.6.3
# thin start -r start.ru
#
# rackup is a useful tool for running Rack applications, which uses the
# Rack::Builder DSL to configure middleware and build up applications easily.
#
# rackup automatically figures out the environment it is run in, and runs your
# application as FastCGI, CGI, or standalone with Mongrel or WEBrick�..all from
# the same configuration.

require 'app'
Ramaze.trait[:essentials].delete Ramaze::Adapter
Ramaze.start
run Ramaze::Adapter::Base

Add the following virtual host file to /etc/apache2/sites-enabled/001-ramaze-demo

<VirtualHost *:80>
    ServerName ramaze.demo.local
    RackBaseURI /
    RailsAutoDetect off
    RackAutoDetect on
    DocumentRoot /var/www/ramaze/demo/public
</VirtualHost>

And edit your /etc/hosts file to include “ramaze.demo.local” so it resolves to localhost.

Database

Next, create a database called ramaze_demo (or your choice) and add a posts table and
some data via SQL or migrate scripts (this being a minimal demonstration and all that).

create table posts (
    id        int(11) not null auto_increment
  , title     varchar(255) not null
  , body      varchar(1024) not null
  , primary key (id)
);

insert posts values (1, "First Post", "Thank hrnt for getting passenger talking to ramaze.");
insert posts values (2, "Another Post", "Ask Pistos what SelfMarks is, because he is bored with diigo");
insert posts values (3, "IRC, you say?", "Visit #ramaze on Freenode");

Model

Edit the model.init.rb file with the following:

require 'activerecord'
require 'cleanup_connection_patch'

  ActiveRecord::Base.establish_connection(
    :adapter  => "mysql",
    :database => "ramaze_demo",
    :host     => "localhost",
    :username => "root",
    :password => "secret"
  )

class Post < ActiveRecord::Base
end

Important: cleanup_connection_patch comes from github.com and makes the ActiveRecord behave correctly in a threaded environment. Please see Monkey patching ActiveRecord to automatically release connections « coderrr for more information.

Controller

With the model squared away, time to turn our attention to the controller. Edit controller/init.rb:

class Controller < Ramaze::Controller
  layout '/page'
  helper  html
  engine :Erubis
end

class PostsController < Controller
  def index
    @posts = Post.find(:all)
  end
end

View

Create a folder under view for posts and add the following Erubis template view/posts.rhtml:

<h1>Listing Posts</h1>

<ul>
  <% for post in @posts %>
      <li><%= A(post.title, :href => "/post/#{post.id}", :title => post.body) %></li>
  <% end %>
</ul>

Note: that while Rails uses filename.html.erb, the Erubis engine expects *.rhtml extension in Ramaze.

Running

Test out the above with webrick, mongrel, or thin with:

ruby start.rb

If all goes well, you should see the three postings above listed.

Launch Apache2, if you haven't already and point your localhost's browser at http://ramaze.demo.local/posts

You should now have a listing of posts rendered through Apache via Passenger.

Conclusion

I am finding much to like about Ramaze, especially after developing with Rails for a few years now and watching it grow at an alarming rate in complexity. Ramaze is a refreshing return to basics. I am finding that Ramaze just may be able to give me most of what I need while giving me a sizeable performance boost in the process. Here's a rails bench on the equivalent of the above:

Concurrency Level:      50
Time taken for tests:   4.277 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      10218492 bytes
HTML transferred:       9595000 bytes
Requests per second:    233.83 [#/sec] (mean)
Time per request:       213.828 [ms] (mean)
Time per request:       4.277 [ms] (mean, across all concurrent requests)
Transfer rate:          2333.42 [Kbytes/sec] received

And this is Ramaze:

Concurrency Level:      50
Time taken for tests:   2.248 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      9815000 bytes
HTML transferred:       9555000 bytes
Requests per second:    444.91 [#/sec] (mean)
Time per request:       112.383 [ms] (mean)
Time per request:       2.248 [ms] (mean, across all concurrent requests)
Transfer rate:          4264.42 [Kbytes/sec] received

Elemental provides enumerated collection of elements that allow you to associate ruby symbols to arbitrary “display” values, thus allowing your code to think symbolically and unambiguously while giving you the means to easily display what end-users need to see. Additionally, symbols are associated with ordinal values, allowing easy storage/retrieval to persistent stores (databases, files, marshalling, etc) by saving the  Element#value as appropriate (String by default, Fixnum if you “persist_ordinally”).

The primary aim of Elemental is to collect and abstract literals away from your code logic. There’s an old programmer’s wisdom that you should not encode your logic using literal values, especially those the end-user is exposed to.

Complete details provided in README.txt

http://github.com/mwlang/elemental/tree/master

http://rubyforge.org/projects/elemental/

From Gem:

sudo gem install elemental

From Source:

gem install bones, rake, test-unit
git clone git://github.com/mwlang/elemental.git
cd elemental
rake gem:package
cd pkg
sudo gem install elemental

Example of Using:
Here’s an example of how Elemental might be used with a Rails application. In your ~/config/initializers/elementals.rb file:

require 'elemental'

class PublicStatus
  extend Elemental

  member :unpublished,      :display => "Not Published", :default => true
  member :editor_approval,  :display => "Editorial Approval Needed"
  member :published,        :display => "Published"
  member :archived,         :display => "Archived"
end

class CommentType
  extend Elemental
  persist_ordinally

  member :all,        :display => "Anyone can post"
  member :moderated,  :display => "Moderated", :default => true
  member :closed,     :display => "Closed to new posts"
end

Create a “posts” table with this migration:

class CreatePosts < ActiveRecord::Migration
  def self.up
    create_table :posts do |t|
      t.string :title
      t.text :body
      t.string :public_status
      t.integer :comment_type

      t.timestamps
    end
  end

  def self.down
    drop_table :posts
  end
end

In your ~/app/views/posts/edit.html.erb:

<h1>Editing post</h1>

<% form_for(@post) do |f| %>
  <%= f.error_messages %>

  <%= render :partial => 'form' %>

  <p>
    <%= f.submit "Update" %>
  </p>
<% end %>

<%= link_to 'Show', @post %> |
<%= link_to 'Back', posts_path %>

Similarly for ~/app/views/posts/new.html.erb

<h1>New post</h1>

<% form_for(@post) do |f| %>
  <%= f.error_messages %>

	<%= render :partial => 'form' %>

  <p>
    <%= f.submit "Create" %>
  </p>
<% end %>

<%= link_to 'Back', posts_path %>

An form partial that utilizes the Elemental classes above to populate a pair of drop down lists:

<p>
	<label>Title</label><br />
	<%= text_field :post, :title, :value => @post.title %>
</p>
<p>
	<label>Body</label><br />
	<%= text_area :post, :body, :value => @post.body %>
</p>

<p><label>Public Status</label><br />
  <%= select "post", "public_status",
    PublicStatus.sort.map{|a| [a.display, a.value]},
      { :include_blank => false, :selected => PublicStatus.defaults.first.value }
  %></select>
</p>

<p><label>Comment Type</label><br />
  <%= select "post", "comment_type",
    CommentType.sort.map{|a| [a.display, a.value]},
      { :include_blank => false, :selected => CommentType.defaults.first.value }
  %></select>
</p>

An example ~/app/views/posts/index.html.erb file:

<h1>Listing posts</h1>

<table border=1 cellpadding=5px>
  <tr>
    <th>Post</th>
    <th>Public status</th>
    <th>Comment type</th>
    <th colspan=3>Actions</th>
  </tr>

<% for post in @posts %>
  <tr>
    <td><%= link_to post.title, post, :title => post.body %></td>
    <td><%=h PublicStatus[post.public_status].display %></td>
    <td><%=h CommentType[post.comment_type].display %></td>
    <td><%= link_to 'Show', post %></td>
    <td><%= link_to 'Edit', edit_post_path(post) %></td>
    <td><%= link_to 'Destroy', post, :confirm => 'Are you sure?', :method => :delete %></td>
  </tr>
<% end %>
</table>

<br />

<%= link_to 'New post', new_post_path %>

The following Browser Edit form is what we’re going for. That is, having a list of Operating Systems to check off while editing a Browser object:

class CreateBrowsers < ActiveRecord::Migration
  def self.up
    create_table :browsers do |t|
      t.string :name
      t.string :version
      t.string :short_name
      t.timestamps
    end
  end

  def self.down
    drop_table :browsers
  end
end

class CreateOses < ActiveRecord::Migration
  def self.up
    create_table :oses do |t|
      t.string :vendor
      t.string :name
      t.string :version
      t.string :short_name
      t.timestamps
    end
  end

  def self.down
    drop_table :oses
  end
end

The Browser and Os table are very straightforward migration scripts. However, pay attention to the next script as we do a couple of interesting things. A) we tell the migration not to create the default ID column and B) we use the t.references declaration in defining our columns.

class BrowsersOses < ActiveRecord::Migration
  def self.up
      create_table :browsers_oses, :id => false do |t|
        t.references :browser
        t.references :os
        t.timestamps
      end
    end

    def self.down
      drop_table :browsers_oses
  end
end

Note that I used the singular version of the model names. I originally used the plural form and got browsers_id and oses_id instead of the expected browser_id and os_id column names.

Which brings me to another issue I encountered. “Os” didn’t pluralize properly, so I needed to add a custom inflector to the project’s config/initializers/inflections.rb file:

ActiveSupport::Inflector.inflections do |inflect|
  inflect.irregular 'os', 'oses'
end

For simplicity and clarity, I chose the irregular form since I didn’t figure on needing to match regular expressions against more complex model names.

Since there are many articles dedicated to the model side of things where relationships are concerned, I will skimp on discussing model implementations. Please see many-to-many-dance-off by Josh Susser or the has_and_belongs_to_many page on the Ruby on Rails wiki site for a couple of classics on the topic.

The models are straightforward, as shown below. Note that the browsers_oses table does not get its own model. Rails simply expects the table to be a concatenation of the two models pluralized and the model names in alphabetical order.

class Os < ActiveRecord::Base
  has_and_belongs_to_many :browsers
end

class Browser < ActiveRecord::Base
  has_and_belongs_to_many :oses
end

I’m not a big fan of the default views the rails scaffolding produces, but I do tend to run script/generate scaffold “model_name” to get the ball rolling quickly, anyway. So fire up a couple of views with:

script/generate scaffold browser
script/generate scaffold os

Note: I removed all previously generated files and regenerated the Os model and views after adding the irregular plural inflection rule described above.

Once I get my models and views, the first thing I do is create a form partial for each:

app/views/oses/_form.html.erb

<fieldset>
	<legend>OS</legend>
	<p>
		<label for="vendor">Vendor</label>
		<%= text_field :os, :vendor, :value => @os.vendor %>
	</p>

	<p>
		<label for="name">Name</label>
		<%= text_field :os, :name, :value => @os.name %>
	</p>

	<p>
		<label for="version">Version</label>
		<%= text_field :os, :version, :value => @os.version %>
	</p>

	<p>
		<label for="short_name">Short Name</label>
		<%= text_field :os, :short_name, :value => @os.short_name %>
	</p>
</fieldset>

There’s nothing fancy about the Os form as there’s no references to select drop-downs or the likes. With the above wrapped in a fieldset with legend of Os. I then tweak app/views/oses/edit.html.erb like so:

<h1>Editing OS</h1>

<% form_for(@os) do |f| %>
  	<%= f.error_messages %>
	<%= render :partial => 'form' %>

	<fieldset>
    	<%= f.submit "Update" %>
		<%= link_to 'Show', @os %> |
		<%= link_to 'Back', oses_path %>
  	</fieldset>
<% end %>

And similarly for app/views/oses/new.html.erb:

<h1>New OS</h1>

<% form_for(@os) do |f| %>
  	<%= f.error_messages %>
	<%= render :partial => 'form' %>

  	<fieldset>
    	<%= f.submit "Create" %>
		<%= link_to 'Back', oses_path %>
  	</fieldset>
<% end %>

app/views/oses/show.html.erb is nearly a cut-n-paste of the _form.html.erb file with the input tags stripped out:

<fieldset>
	<legend>OS</legend>
	<p>
		<label for="vendor">Vendor</label>
		<%= @os.vendor %>
	</p>

	<p>
		<label for="name">Name</label>
		<%= @os.name %>
	</p>

	<p>
		<label for="version">Version</label>
		<%= @os.version %>
	</p>

	<p>
		<label for="short_name">Short Name</label>
		<%= @os.short_name %>
	</p>
</fieldset>

<fieldset>
	<%= link_to 'Edit', edit_os_path(@os) %> |
	<%= link_to 'Back', oses_path %>
</fieldset>

It would be nice if the tag fields were action aware and intelligently generated inputs or text rendering according to whether new, edit, or show was the action, wouldn’t it??

The OsesController model is unchanged from the generated scaffolding code, so I won’t list its code here. Lets hop over to Browsers model and view as this is the model I do the most interesting things with. I will be rendering the Browser inputs so that one of the input is a checkbox list of all available OSes. To do this, I am effectively performing a “find all” on the Os model, so I DRY my code by adding the following method to the BrowsersController:

def get_all_oses
  @oses = Os.find(:all, :order => 'vendor, version')
end

Declaring small, single purpose methods is a good practice to be in the habit of. As it happened, I decided at some point to order the checklist by vendor and version. Instead of having to change code in four or five methods, I only had to update the one get_all_oses method. A tad bit of extra work up front pays off!

Similarly, compound names get their own method in the model class so I have ability to easily change without hunting down references everywhere in the view. So the Os model presented above becomes:

class Os < ActiveRecord::Base
  has_and_belongs_to_many :browsers

  def display_name
    "#{vendor} #{name} #{version}"
  end
end

And then I reference like so in the Browser view’s form partial:

<fieldset>
	<legend>Browser</legend>
	<p>
		<label for="name">Name</label>
		<%= text_field :browser, :name, :value => @browser.name %>
	</p>

	<p>
		<label for="version">Version</label>
		<%= text_field :browser, :version, :value => @browser.version %>
	</p>

	<p>
		<label for="oses">Operating Systems</label>
		<% @oses.each do |os| %>
			<%= check_box_tag(
                            "browser[os_list][#{os.id}]", 
                            "1", 
                            @browser.oses.detect{|d| d == os}) %>
			<%= "#{os.display_name}"%><br />
		<% end %>
	</p>

	<p>
		<label for="short_name">Short Name</label>
		<%= text_field :browser, :short_name, :value => @browser.short_name %>
	</p>
</fieldset>

A few important things to note here about the Operating Systems checklist:

• the @oses variable is a list of all Oses as per the get_all_oses method.
• I gave the checkbox name “browser[os_list][#{os.id}]” which causes the params to collect all checkboxes into an “os_list” hash that is contained in the browser keyword on the params hash. Keep this in mind because we’re going to leverage this in a moment.
• I don’t do anything with the values of the checkbox, so the value assigned (“1″ in this case) is of little concern.
• The @browsers.oses.detect{|d| d == os} line activates the checkbox if the browser already has the Os associated with it. (quite important for edit actions).

So the Operating Systems checklist is simply an iteration over all OSes, emitting a checkbox input control for each and setting the checked flag accordingly. So how do we handle this in the controller? We don’t! That’s right. Not in the controller, but in the model. This is “The Rails Way” as described in the Skinny Controller, Fat Model approaches championed by Jamis Buck and The Rails Way crew. My approach is to add a attribute accessor called “os_list” to the Browser model and a callback to the after_save event. So the Browser model now becomes:

class Browser < ActiveRecord::Base
  has_and_belongs_to_many :oses

  attr_accessor :os_list
  after_save :update_oses

  private 

  def update_oses
    oses.delete_all
    selected_oses = os_list.nil? ? [] : os_list.keys.collect{|id| Os.find_by_id(id)}
    selected_oses.each {|os| self.oses << os}
  end
end

So there’s a bit of magic! The update_attributes and save methods will assign the os_list hash from the params to our accessor “os_list” property automatically and thus provide us convenient access to the hash data in our callback handler. This is Rails at its best. Paul Barry gets the credit for being the first to put it together (at least for me).

What this method does is first clear out all the currently associated Oses. Then, if there are no selected Oses (in which case, items will be []), nothing happens with the loop, otherwise, each item is pushed onto the browser.oses list. When I first started writing Ruby on Rails apps, I was a stickler for not hitting the database unnecessarily and I would write lots of conditionals to avoid doing so.

However, the number of bugs I found in such logic eventually led me down the path of aggressively reducing lines of code in situations where it simply does not matter. Here, we’re talking about, at most, about 10 operating systems defined in the app I’m writing and I am very infrequently going to be adding new browsers to the list (about as often as a major version comes out as a rule of thumb). Is it really worth the extra code and the extra specs (you are writing test cases, aren’t you?) to gain so little? Rails is smart. It’ll issue a delete statement only if there are actually Oses in the list and it’ll only issue one DELETE statement, too! Check this session history out (for PostgreSQL):

  SQL (0.000153)   BEGIN
  SQL (0.000337)   DELETE FROM "browsers_oses" WHERE browser_id = 14 AND os_id IN (8,9)
  SQL (0.009701)   COMMIT
  SQL (0.000131)   BEGIN
  SQL (0.000194)   INSERT INTO "browsers_oses" ("browser_id", "updated_at", "os_id", "created_at") VALUES (14, '2008-10-24 19:33:43.107494', 8, '2008-10-24 19:33:43.107494')
  SQL (0.000649)   COMMIT
  SQL (0.000094)   BEGIN
  SQL (0.000191)   INSERT INTO "browsers_oses" ("browser_id", "updated_at", "os_id", "created_at") VALUES (14, '2008-10-24 19:34:01.088552', 9, '2008-10-24 19:34:01.088552')
  SQL (0.000570)   COMMIT
  SQL (0.000086)   BEGIN
  SQL (0.000199)   INSERT INTO "browsers_oses" ("browser_id", "updated_at", "os_id", "created_at") VALUES (14, '2008-10-23 20:24:59.809242', 2, '2008-10-23 20:24:59.809242')
  SQL (0.000657)   COMMIT
  SQL (0.000076)   BEGIN
  SQL (0.000110)   COMMIT
Redirected to http://localhost:3000/browsers/14
Completed in 0.08188 (12 reqs/sec) | DB: 0.01585 (19%) | 302 Found [http://localhost/browsers/14]

Now, if you’ve got a situation where 100’s of users are performing a similar action and the small doses truly add up to something significant in your daily logs, then definitely optimize. Just be cognizant of your use-case and profile your application for performance before getting fancy with the conditional updates, otherwise you’re in the trap of premature optimization and not getting something else done you might have wish you were!

Ok, having digressed a bit, lets get back on course. So what does the Browser Controller now look like? Well, the only thing I have to add is a call to “get_all_oses” just in case of a failure to save and the redirect back to the same page occurs. So this is what we have:

def get_all_oses
  @oses = Os.find(:all, :order => 'vendor, version')
end  

# POST /browsers
# POST /browsers.xml
def create
  @browser = Browser.new(params[:browser])
  get_all_oses

  respond_to do |format|
    if @browser.save
      flash[:notice] = 'Browser was successfully created.'
      format.html { redirect_to(@browser) }
      format.xml  { render :xml => @browser, :status => :created, :location => @browser }
    else
      format.html { render :action => "new" }
      format.xml  { render :xml => @browser.errors, :status => :unprocessable_entity }
    end
  end
end

# PUT /browsers/1
# PUT /browsers/1.xml
def update
  params[:oses] ||= {}
  @browser = Browser.find(params[:id])
  get_all_oses

  respond_to do |format|
    if @browser.update_attributes(params[:browser])
      flash[:notice] = 'Browser was successfully updated.'
      format.html { redirect_to(@browser) }
      format.xml  { head :ok }
    else
      format.html { render :action => "edit" }
      format.xml  { render :xml => @browser.errors, :status => :unprocessable_entity }
    end
  end
end

One last trick for the eagle eyes: The form image I posted positions the labels in bold above the inputs and with a trailing colon, but nary a <BR> element (or colons) to be seen in the views. I accomplished this with simple CSS styling by switching the label element to a block element (rather than inline as is the default) and then providing the colon as additional content:

label {
	display: block;
	font-weight: bold;
}
label:after {
	content: ':';
}

So there you have it. Leveraging the Rails framework appropriately, you can easily provide a checklist generated from one model on another’s edit form and have those records maintained in a many-to-many relationship without bloating your views or controllers.

To get started, I needed to know how to get the data into Trac. The following links gave me an API I could utilize to create trac pages and content:

Trac Data Models
The Data model Python code

Great! Except there’s one small problem. I don’t really know python all that well, even though I did do a bit of Zope development, oh, five years ago. These days, my language of choice is Ruby, so I needed to figure out how to parse the Oddmuse syntax and turn it into Trac syntax. The strategy: Parse with Ruby, load cleaned page files with Python.

This script will take a filename at command line and convert it and save back to file, but different extension (I chose *.om for “oddmuse”). In the script below, I made the command-line optional as I discovered I didn’t handle every case as I worked on developing this script, so I simply changed the line that pulls the command-line argument to point to a specific file and output such to terminal and this allowed me to debug and fix as I went:

#!/opt/local/bin/ruby
# $FS  = "\xb3";      # The FS character is the RECORD SEPARATOR control char in ASCII
# $FS0 = "\xb3";      # The old FS character is a superscript "3" in Latin-1
# $FS1 = $FS . '1';   # The FS values are used to separate fields
# $FS2 = $FS . '2';   # in stored hashtables and other data structures.
# $FS3 = $FS . '3';   # The FS character is not allowed in user data.

FS = 179.chr

SITE_NAME = "corporate_wiki"
ATTACHMENT_URL_BASE = "https://intranet.example.com/#{SITE_NAME}/attachment/wiki/ConvertedAttachments/"

require 'find'
require 'ftools'

def safe_page_name(pagename)
  result = pagename.gsub("'","")
  result = result.slice(/(\w+)/).first
end

def get_section(page, section)
  page.each_with_index {|e,i| return page[i+1] if e == section}
  return ''
end

def asterick_prefixed(text)
  t = text.match(/(\*+)([^\b]*)/)
  return text if t.nil?

  tokens = t.to_a
  return ('  ' * tokens[1].size) + '* ' + tokens[2]
end

def colon_prefixed(text)
  return "  #{colon_prefixed(text.slice(1,text.size))}" if text[0] == ":"[0]
  return text
end

def pound_prefixed(text)
  return "  1. #{text.slice(1,text.size)}" if text[0] == "#"[0]
  return text
end

def fix_headers(text)
  t = text.match(/(=+)([^=]+)(=+)/)
  return text if t.nil?

  tokens = t.to_a
  return "#{tokens[1]} #{tokens[2].strip} #{tokens[1]}"
end

def replace_a_url(url, text)
  text.gsub!("http://#{url}", ATTACHMENT_URL_BASE)
  text.gsub!("https://#{url}", ATTACHMENT_URL_BASE)
  text.gsub!("http:/#{url}", ATTACHMENT_URL_BASE)
  text.gsub!("https:/#{url}", ATTACHMENT_URL_BASE)
  return text
end

def replace_wiki_urls(text)
  matches = text.match(/\[http(.)*\]/)
  result = text
  if matches
    result = replace_a_url("wiki.example.com/#{SITE_NAME}/wikifiles/", result)
    result = replace_a_url("pdfreviewfiles/", result)
    result = replace_a_url("cgi-bin/", result)
  end
  result
end

def convert_formats(text)
  result = text.gsub(/\[\[(\w+)\]\]/,"[wiki:" + '\1' + "]")
  result = asterick_prefixed(result)
  result = colon_prefixed(result)
  result = pound_prefixed(result)
  result = replace_wiki_urls(result)
  result = fix_headers(result)
  result
end

def oddmuse_to_trac(text)
  result = ''
  in_pre = false
  text.split("\n").each do |line|

    # Detect indented lines as these are rendered as PRE html blocks
    trimmed = line.lstrip
    is_prefixed = ((trimmed.size < line.size) and (trimmed[0] != '*'[0]))
    result << "{{{\n" if (is_prefixed and !in_pre)

    result << "}}}\n" if (!is_prefixed and in_pre)
    in_pre = is_prefixed

    # Don't process special characters when in PRE block
    if in_pre
      result << line << "\n"
    else
      result << convert_formats(line) << "\n"
    end
  end
  result += "}}}\n" if in_pre
  result
end

path = ARGV[0] || "page/U/UsingThePhones.db"
raw_page = File.new(path).read
page2 = raw_page.split(FS + '2')
page3 = raw_page.split(FS + '3')

text = get_section(page3, 'text')

pagename = File.basename(path).split('.').first
author = get_section(page2, 'username')
address = get_section(page2, 'ip')
body = oddmuse_to_trac(text)

puts "writing: #{pagename}.om"
outfile = File.new(path.gsub('.db','.om'),'w')
outfile.puts 'pagename:' + pagename
outfile.puts 'author:' + author
outfile.puts 'address:' + address
outfile.puts body
outfile.close

puts body if ARGV[0].nil?

Below, comes the Python side of the equation. Once I did the heavy lifting in Ruby, I needed to generate all the pages. As you can see, there's some parsing happening, but its very simple parsing and I was able to apply what simple Python knowledge I still retained pretty effectively:

import sys
from trac.env import Environment
from trac.wiki.model import WikiPage

print sys.argv[1]
file = open(sys.argv[1], 'r')
pagename = file.readline().split(':')[1].split('\n')[0]
author_name = file.readline().split(':')[1].split('\n')[0]
address = file.readline().split(':')[1].split('\n')[0]

text = file.read()

env = Environment('/srv/www/trac/corporate_wiki')

# Read an existing or new WikiPage:
page = WikiPage(env, pagename)
if page.text != text:
        page.text = text
        page.save(author=author_name, comment='imported from oddmuse wiki', remote_addr=address)

As you can see, this file simply takes the command line argument, open the file for reading, pull the various variables out with the remainder of the file going to the page content.

During the conversion, I noticed some pages were failing to convert at all, like "Michael'sTodoList" The culprit, the tick in the page name and thus the filename. The scripts themselves weren't the problem, it was the xargs parameter passing that was the issue (getting to that below). So I ran the below script to rename all the troublesome filenames and then added similar tick to underscore substitution in the main Ruby script above.

require 'find'
require 'ftools'

total_size = 0

Find.find('./page') do |path|
  if FileTest.directory?(path)
    if File.basename(path)[0] == ?.
      Find.prune       # Don't look any further into this directory.
    else
      next
    end
  else
    if path.match(/[\']/)
      safe_path = path.gsub("'", "_")
      puts path + ' to ' + safe_path
      File.move(path, safe_path)
    end
    end
  end

A simple up-front execution of the script was all that was needed to now be able to pick up the missing pages:

ruby fix_unsafe_files.rb

Finally, to bring it all together, I utilized find to locate all the *.db Oddmuse files which I had located into the various wiki subfolders where these scripts sat, passing the filename to the script via the xargs utility:

find . -name *.db -print0 | xargs -0 -L 1 ruby rdpage.rb

I then essentially repeated the above, but for the generated *.om files, passing to the Python mkpage.py script:

find . -name *.om -print0 | xargs -0 -L 1 python mkpage.py

One last thing, before we're done. Each wiki had attachments. So I had to think how to get those attachments referenced and linked to the Trac wiki. A little investigation revealed that each attachment got a database entry in the attachment table. My task was a bit complicated by the fact that the users over the years in the Oddmuse environment scattered their attachments everywhere and also used varying syntax to refer to them, hence, if you review the main Ruby parsing script, you'll see the "replace_a_url" method above that cleans all those funky URLs up. So the script I used below simply scans through all the directories where we had files stored and attached them to one "ConvertedAttachments" page. Probably not the ideal solution, but again, got the job done quickly.

#  type |          id          |  filename  |  size  |    time    | description | author |      ipnr
# ------+----------------------+------------+--------+------------+-------------+--------+----------------
#  wiki | ConvertedAttachments | gotapi.png | 200272 | 1218381947 |             | michael | 192.168.16.100

path = ARGV[0] || 'generate_attachments.rb'
filename = File.basename(path)

puts "insert into attachment values ('wiki', 'ConvertedAttachments', '#{filename}', #{FileTest.size(path)}, null, null, 'oddmuse','192.168.1.246');"

So, running the above needs to be piped to a SQL fisle and then executed against your Trac database. In my case, Postgresql. Since I had many directories to snatch up, I ran it for each folder as appropriate and simply kept appending until I had all the attachment directories represented.

ls | xargs -0 -L 1 >> attachments.sql
psql trac_db < attachments.sql

All in all, not too bad of a conversion for a few hour's work. Its about 95% correct. Some of the page names didn't translate to Trac, and could probably do with some regexp search and replace on the page names themselves. The biggest gain is that we have preserved the information in the old systems by porting it to the new systems we all know and use on a daily basis, so WikiGardening is more likely to keep the content fresher and its one less system for our end-users to figure out how to use.

I can now retire the server running Oddmuse, stop doing backups and keeping up with documentation and training new system administrators on how to manage Oddmuse AND Trac.

A bit of hunting around turned up this list of zip code databases. The first link to CivicSpace database appears to be defunct (I got a Go Daddy “this page is parked” page). So, the next free one, offered by Popular Data turned out to be exactly what I was looking for. It may be a “little old” as the review site says, but its good enough for me to get started with associating my data with geocoded locations and explore the wonderful world of mapping offered by Google’s Maps API.

To load it up in MySQL and PostgreSQL, I used the following Ruby script to generate a bunch of insert statements:

require 'rubygems'
require 'fastercsv'
source_file = 'ZIP_CODES.txt'
outfile = File.new('us_zip_codes.sql','w')

# generate the truncate statement if we want to empty the table
outfile.puts "truncate us_zip_codes;"

# generate the SQL Insert statements
FasterCSV.foreach(source_file, :headers => false) do |cols|
  row = "INSERT INTO us_zip_codes VALUES ("
  row << as_string(cols[0]) << ","
  row << as_number(cols[1]) << ","
  row << as_number(cols[2]) << ","
  row << as_string(cols[3]) << ","
  row << as_string(cols[4]) << ","
  row << as_string(cols[5]) << ","
  row << as_string(cols[6]) << ");"
  outfile.puts row
end
outfile.close
puts "copied #{i} records."

Create the zip code table with the following SQL:

create table us_zip_codes (
  zip_code char(5),
  latitude float(10,6),
  longitude float(10,6),
  city varchar(255),
  state char(2),
  county varchar(255),
  zip_class varchar(255)
);

And finally, with the table in place and the SQL insert statements generated, I loaded the data into MySQL with the following from command line:

mysql target_db_name < us_zip_codes.sql

The problem: I wanted to introduce a property editor metaphor for the website I’m working on where properties could expose edit forms for just about any situation, much like wordpress’ widget interface does.

That is, there’s an area of the current page that shows basic information about the property in a read-only (and preferably compact) form, and user could click an “edit” button and exposed an editable version of the properties. The “edit” link becomes a “cancel” link, which, when clicked, simply discards the edit form and return the initial compact view state.

The problem is, Rails doesn’t really have any good helpers for reusing a design concept in your views. I found that even with partials, the code was really starting to bog down and become both unmaintainable and unreadable. I needed a simple mechanism to construct my property editors. I started out with helper methods. But that wasn’t much better as the code just started ending as ruby here-docs and became a maintenance issue of another sort! Take a look at this embarrassment:

result = "<table><tr>"
result += "<th colspan=2>"
result += button_image_action('Group selected together', 'link.png')
result += button_image_action('Copy selected to this group', 'link_add.png')
result += button_image_action('Move selected to this group', 'link_go.png')
result += button_image_action('copy group', 'page_copy.png')
result += button_image_action('cut group', 'cut.png')
result += button_image_action('paste group', 'page_paste.png')
result += button_image_action('delete group', 'delete.png')
result += "</th></tr>"
result += "</table>"

Ok, I admit it, I chopped out more than half again as many lines as you’re looking at there, and perhaps worse, this isn’t actually the original HTML code that led me down this path, but lets take a look at this approach anyway, as I learned something valuable in the process and it was my first attempt at DRYing my views. First thing, first, get the above HTML back into the erb file:

			<%= button_image_action('Group selected together', 'link.png') %>
			<%= button_image_action('Copy selected to this group', 'link_add.png') %>
			<%= button_image_action('Move selected to this group', 'link_go.png') %>
			<%= button_image_action('copy group', 'page_copy.png') %>
			<%= button_image_action('cut group', 'cut.png') %>
			<%= button_image_action('paste group', 'page_paste.png') %>
			<%= button_image_action('delete group', 'delete.png') %>
		

Which promptly let me to trying something like this:

def button_image_action(title, image_filename, button_text = '', url = '')
    form_for @cube url do |f|
  	  <<-TEXT
  	    #{button_text}
  	    
  	    
  	  TEXT
     end
  end

I thought it was shaping up nicely, but I got completely blindsided when it came to rendering this bit. You see, the Rails Form Helpers are unavailable in a regular helper! They require ERB (being inside <% %> and <%= %>) to actually properly render their blocks.

I fiddled around for a bit to try to jump this hurdle, but it just simply ain’t gonna work without some serious hacking. I’m happy to report that I got a little wiser and stepped back from the problem and reevaluated my approach in general. I realized, I wasn’t getting away from doing HTML code in helpers! Bottom line: Helpers are great, but you’re doing something clearly against “The Rails Way” when you start rendering large chunks of HTML code from helpers. It just isn’t a great approach and if you have designers on your team, they can get quite lost hunting around for all those HTML snippets.

So, how could I get all of my HTML back into *.erb files where they belonged, yet develop some helpers that would make mincemeat of the coding effort to build these fancy toolbars and forms I had coming? Fellow colleague, Steven Beales said, “you need to use Ruby blocks.” To which, my first reaction was, “what are Ruby blocks?” To my great benefit, he explained by pointing me straight at Block Helpers and DRY Views in Rails.

Truth be told, I mostly understood this article, but not entirely. Even so, I dutifully set off to solve my problem with this approach. This led me to a bit of copying and pasting (and crediting)…

  # From:  http://errtheblog.com/posts/11-block-to-partial
  #
  # captures the output of our passed block using Rails’ capture method and adds
  # it to our options hash under the key of :body. Next it renders our passed
  # partial, sending in our options hash as :locals
  def block_to_partial(partial_name, options = {}, &block)
    options.merge!(:body => capture(&block))
    concat(render(:partial => partial_name, :locals => options), block.binding)
  end

Look at that carefully and, if I may be so bold to suggest, go read up on ERR the Blog to better understand as this method is key and I daresay was appropriately called “tomfoolery” by the wise authors therein. This bit of code really forced me to finally begin to learn and understand what the heck block passing was all about. In a nutshell, this little gem lets you pass options that become local variables to the partial and the block is the contents to be rendered within the partial when you yield to the passed block.

  # Calls given partial, injecting the options hash into the partial's local hash
  def variables_to_partial(partial_name, options = {})
    render(:partial => partial_name, :locals => options)
  end

The block_to_partial is a great start, but I wanted to simplify my calls and also enforce a bit of use-pattern on the developer (i.e. me) so that all property editors had essentially the same behavior pattern (and components). Thus, we introduce the property editor container which in turn can have one or more sections associated with it:

  # Generates a division around the given block that is styled with "property-editor"
  # Property editors are intended as a view on data that has both a condensed display and
  # an expanded display that lets you edit the information contained in the property editor.
  def property_editor(id, title, options = {}, &block)
    @id = id
    block_to_partial 'layouts/property_editor', options.merge(:id => id, :title => title), &block
  end

  # Renders content of a section (provided by block) inside a property editor division
  # The stylesheet contains styling for the following section_id's:
  #   => "show" - the section initially visible
  #   => "edit" - the section made visible when the edit button is clicked
  def property_editor_section(section_id, options = {}, &block)
    block_to_partial('layouts/property_editor_section',
      options.merge(:section_class => "property-editor-#{section_id}",
        :section_id => "#{@id}-#{section_id}"),
        &block)
  end

The property editor partial:

<div <%= %[id="#{id}"] %> class="property-editor">
	<div class="property-editor-title">
		<div style="float:right">
			<a href="javascript: void(0)" onclick="toggle_div('<%= id %>-show');toggle_div('<%= id %>-edit');">edit</a>
		</div>
  		<%= title %>
	</div>
	<%= body %>
</div>

and the property editor sections are rendered with this partial:

<div <%= %[id="#{section_id}"] %> <%= %[class="#{section_class}"] %> <%= %[style="display:none"] if section_id =~ /(.+)?\-edit/  %>>
	<%= body %>
</div>

With these helpers and partials, property editors as a concept is very easy to pull together in my view files. As you can see, if my section is an “edit” section, its hidden at the outset. The button to toggle between show and edit is in the property editor partial itself in which an onclick javascript calls toggle_div to make this happen. The Javascript is very straightforward:

function toggle_div(div_name) {
	var div = $(div_name);
	div.getStyle('display') == 'none' ? div.setStyle('display', 'block') :	div.setStyle('display', 'none');
}

Here’s just one such property editor operating on my “cube model”:

<% property_editor 'properties', 'Properties' do %>
	<% property_editor_section 'show' do %>
		<%= @cube.name %><br/>
		<%= render :partial => '/cubes/cube_stats' %>
	<% end %>

	<% property_editor_section 'edit' do %>
		<%= render :partial => 'form' %>
	<% end %>
<% end %>

What’s more, I had a complete set of CSS selectors with which I could control my property editor and its associated sections:

.property-editor {
	margin-top: 5px;
	border: 3px solid #FDEFE6;
}

.property-editor-title {
	padding: 2px;
	padding-left: 5px;
	padding-right: 5px;
	background-color: #EEE;
	color: #2F6569;
}

.property-editor-show, .property-editor-edit {
	padding: 5px;
}

.property-editor p {
	margin: 0px;
	padding: 0px;
}

Well, all in all, that’s a good bit of rambling, but I hope others can benefit from my experiences with Ruby, blocks, partials, and DRY up those views.