The Stage

I am setting up a Ubuntu server with Apache 2.2 and authenticating users against a Windows 2003 Server’s Active Directory. This is standard fare when all you need is a require valid-user directive. But when it came to getting require ldap-group directive going, I really stumbled. And on multiple fronts, too. I am not a Windows guru by any measure, nor am I an LDAP guru, so there was a huge cultural and vocabulary barrier for me to cross on both the LDAP vernacular and the somewhat different parlance surrounding Microsoft’s documentation on Active Directory. Its like a Southern Redneck talking to a Scottish Highlander about home-brewing beer.

So here’s what we’re going to do:

• Apache 2.2
• Microsoft Active Directory (Windows 2003 Server)
• authnz_ldap module

For this effort, I am authenticating users that reside in the MyBusiness -> Users folders and I’m restricting access to the Auditors Security Group, also residing under MyBusiness. The domain I use is “sample.local” herein. See the screen shot of the Active Directory structure and details of the Auditors group below:

There are plenty of resources out there about getting the authnz_ldap module working with Apache, so I won’t be covering this part of the setup. Instead, I will focus on the gestalt of the actual configuration.

Will the Real AuthLdapUrl Please Stand Up?

There seems to be no real standard way to set the AuthLdapURL configuration string. I think I visited ten or fifteen web sites on the topic and each one was a little bit different! And none really explained fully the chosen syntax and LDAP selectors being used. Fortunately, I came across a site dedicated to explaining CSVDE, which turned out to be a bit of a rosetta stone for me.

CSVDE is a tool I had never heard of, but yet is apparently installed on every Windows 2003 Server. After about five minutes of reading and 20 minutes of perusing the CSV export I generated, I knew exactly how to construct the AuthLdapURL configuration string. Well, with one caveat, that is. I still am not sure what “(objectClass=*)” does for me, but several examples had it and I kept it.

Know your Users

All of my users are maintained under group policies and they reside under the domain’s root node in “MyBusiness” and “Users” under that. Standard Windows accounts, especially system administrator accounts reside in a “Users” folder directly off the domain’s root node. This was one of the first points of contention that I had to understand when looking at the vast majority of other people’s solution. Simply put, know which “Users” you wish to work with in Apache.

Knowing I wanted to authenticate users under the MyBusiness node, my AuthLdapUrl becomes:

AuthLDAPURL ldap://ad.sample.local:389/OU=MyBusiness,DC=sample,DC=local?sAMAccountName?sub?(objectClass=*)

Bind a User so Apache can Query

You do have to bind to Active Directory so that Apache can actually query. To do this, I created a User under the standard “Users” folder (where administrator, and other such accounts traditionally reside). I don’t know if I applied correct rights or not, but the account is no more than a “Domain User” member. I called the user “AD_VIEWER” with description “Account to bind Apache to Active Directory for authentication.” This leads to the following Apache directives:

AuthLDAPBindDN "AD_VIEWER@sample.local"
AuthLDAPBindPassword bottom_secret_password

I tested the above getting the usual “require valid-user” directive working with Apache. Once I was sure I could do this much, it was time to turn my attention to restricting access to users that are members of a specific group.

Where’s my Group?

I struggled for quite a while with the groups before I finally got wise and set the debug level to get more details into the apache log files. The debug logging level can be set with:

LogLevel debug

And this leads to quite detailed information about what’s going on with Apache and Authentication. For example, before I got my ldap-group configuration correct, I was seeing this:

[debug] mod_authnz_ldap.c(721): [2934] auth_ldap authorise: require group: testing for uniquemember: CN=Code Connoisseur,OU=MyBusiness,OU=Users,DC=sample,DC=local (CN=Auditors,OU=MyBusiness,DC=sample,DC=local)
[debug] mod_authnz_ldap.c(737): [2934] auth_ldap authorise: require group "CN=Auditors,OU=MyBusiness,DC=sample,DC=local": authorisation failed [Comparison complete][No such object]
[debug] mod_authnz_ldap.c(852): [2934] auth_ldap authorise: authorisation denied

Note the “No such object” as this was telling me the group itself was not found. I tried many permutations without success until I found the CSVDE program and was able to dump the Active Directory to Excel for further inspection. I realized with the CSVDE output that I was doing both the
AuthLDAPURL (the above AuthLDAPURL is correct as I have spared you from my mistakes). The correct ldap-group config string for accessing a Security Group defined under MyBusiness is:

require ldap-group CN=Auditors,OU=Security Groups,OU=MyBusiness,DC=sample,DC=local

In short, you have to exactly declare the selector for your Active Directory Group or Apache will fail to find it. After getting the group setting right, I now see “attribute member” and “Comparison true” in the logs, which indicates that the group was found and that my user is a member of that group.

[debug] mod_authnz_ldap.c(377): [3134] auth_ldap authenticate: using URL ldap://10.0.1.5:389/OU=MyBusiness,DC=sample,DC=local?sAMAccountName?sub?(objectClass=*)
[debug] mod_authnz_ldap.c(474): [3134] auth_ldap authenticate: accepting mwlang
[debug] mod_authnz_ldap.c(715): [3134] auth_ldap authorise: require group: testing for group membership in "CN=Auditors,OU=Security Groups,OU=MyBusiness,DC=sample,DC=local"
[debug] mod_authnz_ldap.c(721): [3134] auth_ldap authorise: require group: testing for member: CN=Michael Lang,OU=Employees,OU=Users,OU=Users,OU=MyBusiness,DC=sample,DC=local (CN=Auditors,OU=Security Groups,OU=MyBusiness,DC=sample,DC=local)
[debug] mod_authnz_ldap.c(730): [3134] auth_ldap authorise: require group: authorisation successful (attribute member) [Comparison true (adding to cache)][Compare True]

The complete Apache Config

Bringing all of the items discussed above together, we arrive at this Apache configuration. I am using Virtual Hosts to uniquely identify the service. Also of note, I am using Ramaze to provide the functionality of the actual site, so please do take this into account if you’re copying and pasting. This config is a complete virtual host entry:

<VirtualHost *:80>
    ServerName audit.sample.local
    RackBaseURI /
    RackAutoDetect on
    DocumentRoot /var/www/balancer/public

    LogLevel debug
    <Location "/data_entry">
        AuthBasicProvider ldap
        AuthType Basic
        AuthName "Sample Realm"
        AuthLDAPURL ldap://10.0.1.5:389/OU=MyBusiness,DC=sample,DC=local?sAMAccountName?sub?(objectClass=*)
        AuthzLDAPAuthoritative on

        AuthLDAPBindDN "AD_VIEWER@sample.local"
        AuthLDAPBindPassword bottom_secret_password

        AuthLDAPGroupAttributeIsDN on
        require ldap-group CN=Auditors,OU=Security Groups,OU=MyBusiness,DC=sample,DC=local
    </Location>
</VirtualHost>

I was having a good bit of trouble getting ports to install ImageMagick and have had trouble many other times getting ports to install things. As such, I often fell back on fink or compiling and installing from source rather than figuring out the root causes of my macports problems.

Macports was surprisingly easy to fix up and get fully operational again, and this was on a fairly old machine that I am pretty certain I started with “Darwin Ports” roughly 3.5 years ago.

First Thing’s First

First, know that the official macports website is: trac.macports.org. Be wary of anything “darwinports” as name is officially retired in favor of “macports.”

Trouble a-brewing

It all started innocently enough. I wanted to install ImageMagick on my mac. I’ve been through installing ImageMagick on Linux and knew it was not going to be a cakewalk to install from source. ImageMagick simply has too many dependencies to relish the thought of an install from source route. So I googled and found instructions for ports and began:

sudo port install tiff -macosx imagemagick +q8 +gs +wmf
Warning: Group file could not be located.
--->  Activating tiff 3.8.2_3
Error: Target org.macports.activate returned: Image error: Another version of this port (tiff @3.8.2_2+darwin_9+macosx) is already active.

Oh, ok, lets remove the old one…

sudo port uninstall tiff -macosx imagemagick +q8 +gs +wmf
--->  The following versions of tiff are currently installed:
--->  	tiff @3.8.2_2+darwin_9+macosx (active)
--->  	tiff @3.8.2_3
Error: port uninstall failed: Registry error: Please specify the full version as recorded in the port registry.
[mwlang@macdoze shared] sudo port uninstall tiff @3.8.2_2+darwin_9+macosx
--->  Unable to uninstall tiff 3.8.2_2+darwin_9+macosx, the following ports depend on it:
--->  	gtk2
--->  	lcms
--->  	sane-backends
--->  	xsane
Error: port uninstall failed: Please uninstall the ports that depend on tiff first.
[mwlang@macdoze shared] sudo port uninstall tiff @3.8.2_2+darwin_9+macosx gtk2 lcms sane-backends xsane
--->  Unable to uninstall tiff 3.8.2_2+darwin_9+macosx, the following ports depend on it:
--->  	gtk2
--->  	lcms
--->  	sane-backends
--->  	xsane
Error: port uninstall failed: Please uninstall the ports that depend on tiff first.
[mwlang@macdoze shared] sudo port uninstall gtk2 lcms sane-backends xsane
--->  Unable to uninstall gtk2 2.12.9_0+x11, the following ports depend on it:
--->  	gconf
--->  	gnome-keyring
--->  	libglade2
--->  	gail
--->  	libgnomecanvas
--->  	libbonoboui
--->  	poppler
--->  	py25-gtk
--->  	gtk-nodoka-engine
--->  	xsane
Error: port uninstall failed: Please uninstall the ports that depend on gtk2 first.

Not a very auspicious start, is it? Time to step back and think about this one and dig into the macports facility to learn how to properly manage things. New strategy: Get an education.

Discovering Deactivate/Activate

So, looking at the above problem, I realized that there was almost certainly an easier route out of this dependency nightmare and I needed to learn how to upgrade those old, stale packages. Going through the docs reveals activate and deactivate commands. Sounds perfect, lets try:

sudo port activate tiff @3.8.2_3
--->  Activating tiff @3.8.2_3
Error: port activate failed: Image error: Another version of this port (tiff @3.8.2_2+darwin_9+macosx) is already active.
[mwlang@macdoze shared] sudo port deactivate tiff @3.8.2_2+darwin_9+macosx
--->  Deactivating tiff @3.8.2_2+darwin_9+macosx
[mwlang@macdoze shared] sudo port activate tiff @3.8.2_3
--->  Activating tiff @3.8.2_3

So there’s the kicker. First, deactivate the old version, then activate the new version.

So Macports itself is out of date, you say?

sudo port install tiff -macosx imagemagick +q8 +gs +wmf
Error: Unable to execute port: invalid command name "use_autoreconf"

I had no clue what this was. Googling around was turning up precious little. So I hopped over into the IRC #macports channel. Right off the bat, Toby tells me to selfupdate. I had made the mistake of thinking that “sudo port sync” was keeping macports packages all up to date in one fell swoop. But the two are two distinct activities. So remember to keep ports itself updated with this:

sudo port selfupdate

With that, port was upgraded from a very old version to a bright and shiny new version. Time to try to install ImageMagick again. And I’m happy to report that I got much further along this time. But still hit a roadblock…

Mysterious failure demystified

A little down the chain of installing various xorg packages, I hit upon this one with the xorg-libx11 package:

checking for XPROTO... configure: error: Package requirements (xproto >= 7.0.13) were not met.
Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively you may set the XPROTO_CFLAGS and XPROTO_LIBS environment variables
to avoid the need to call pkg-config.  See the pkg-config man page for
more details.

Error: The following dependencies failed to build: ghostscript xorg-libXext xorg-libX11 xorg-libXt xorg-libsm xorg-libice
Error: Status 1 encountered during processing.
[mwlang@macdoze shared] sudo port install ImageMagick
Error: Requested variants do not match original selection.
Please perform 'port clean ImageMagick' or specify the force option.
Error: Status 1 encountered during processing.

Strange, what’s that xproto >= 7.0.13? How odd that a dependency would fail! I tried a sudo port install xproto, but that was a no-go (so don’t bother trying, it’ll fail for you, too). but then I noticed the dependency list (which oddly is missing xorg-xproto) and most of the packages are prefixed with “xorg-”, and tried sudo port install xorg-xproto and thus, was on my way again.

I lie! I lie! I had no clue. I googled “xproto port install” and learned the truth for the real package name that way. However, as it happens, I found the answer on darwinports dot com in a dependency list for xorg-libx11. I casually mentioned this site in the IRC channel and the #macports tenants were very quick to point out was an imposter site, so be wary of anything you read and learn there. Information is liable to be stale while the site owner attempts to lighten your wallet a bit for no good cause. Little did I know!

Final tidbit

Once I got the ports system updated and the xproto package installed, it was a fair breeze to finish with the ImageMagick installation. One last command I learned from the good folks in #macports is to run the following on a frequent basis to keep your macports installation in fair good shape:

sudo port selfupdate && sudo port upgrade outdated

Conclusion

For years, I struggled with ports, and for no good reason. Once I took the time to learn the package management’s commands (along with the help of those in #macports, for which I’m very grateful to them for saving me a few hours of head-scratching), I am finding I’m able to install packages painlessly on the mac and have even tried a couple of others I had problems with in the past and they’re all installing without a hitch. The lesson for me: take the time to learn your tools. It may be slow-going at first, but the rewards always seem to pay back big dividends down the road.

Introducing Ramaze

One I found is Ramaze. I have to say that Ramaze is a very minimalist approach to a framework that is actually rather enlightening. For one coming from a Rails’ world, it can feel a bit sparse at first. But lately, I’m attributing my lack of comfort to being put on my toes to do some actual real Ruby coding. Its both a little scary and liberating at the same time. I have found that I am enjoying Ruby a lot more as a result of my exposure to Ramaze and my thought process is actually radically changing, almost like a Visual Basic programmer going to Borland Delphi might feel.

So I decided to take a hard look at performance metrics between Rails and Ramaze’s. At the same time, I wanted to see how MRI 1.8.7 stacked up against Ruby Enterprise Ed. 1.8.6, so I’m satisfying a dual curiosity herein. Being quite happy with Phusion Passenger, I decided to set things up with Apache 2.2.9 + Passenger + Rails/Ramaze. Since I’m quite comfortable with ERB and ActiveRecord, I tweaked Ramaze to be a bit more like the familiar Rails world I am familiar with.

Un-DRYing the Results

There’s no point in generating a lot of data and reporting on it if others cannot repeat yourself (CRY!) in replicating the results, so with that in mind, I attempt to refrain from further bad turn of words by busying myself in pushing everything I utilized to github.com and documenting the basic bootstrapping of Ubuntu Intrepid. For these tests in particular, I used two scripts, bootstrap-passenger-std.sh and bootstrap-passenter-ent.sh to set up four distinct configurations under the Ruby1.8 interpreter. Ruby 1.9.1. was a bit difficult to set up and there is a bootstrap-passenger-191.sh script that I assembled as I installed things, but I haven’t run it to ensure it works end-to-end.

I utilized this simple call to Apache Bench to run through the four test scenarios:
All data collected was based on 10,000 requests at 100 concurrent users after an initial “warmup” of the passenger threads with start of 100 requests at 10 concurrent users. The following command shows the basic set of parameters issued to Apache Bench:

ab -n 10000 -c 100 http://demo.u64rails01.local/posts

What was tested?

In order to compare, as close as possible, apples to apples, I opted to use Active Record and ERubis in Ramaze to set up the same MVC classes in both the Rails and Ramaze project. ActiveRecord is probably the heaviest weighted ORM that can be utilized with Ramaze. I tried briefly to get Erubis working with Rails so that the two frameworks’ projects would be more closely aligned than the out-of-box ERB rendering Rails ships with would have afforded. I did attempt Erubis with Rails, however, I was either doing something very wrong or Erubis under Rails takes a sizable performance hit and I stuck with out-of-box ERB for these tests given its better performance. As such, Rails uses out of box ERB rendering in these tests.

The application itself is ridiculously simple, offering only one model and performing only a read against the database, but hopefully, an effective start towards building this benchmark framework towards more substantive benchmark tests to come. For now, lets take a look at how Rails and Ramaze stack up to each other and what affect the underlying interpreter can have on performance. Standard packaged Main Ruby Interpreter (MRI) 1.8.7 consisted of simple package install provided by the Ubuntu Intrepid repositories whle Ruby Enterprise Edition (REE) 1.8.6, which hails from the same folks bringing you Phusion Passenger, is compiled from source.

As a bonus, I also compiled and clocked metrics for Ramaze on 1.9.1. This was a tricky environment to get set up and working. And my initial results made me think I did something horribly wrong with compiling and configuring Ruby 1.9.1. However, I fully replicated Antonio Cangiano’s benchmarks in my test environment (see The Great Ruby Shootout (December 2008)) and MRI 1.9.1. did indeed crush the other Ruby interpreters in Fibonacci number crunching (amongst others). The theory tossed around in the Ramaze IRC channel was that 1.9.1. change in implementation of the String class (and it being unicode native now) slows things down a good bit with string processing.

The Results

The following table tabulates the results gathered from Apache Bench as well as passenger-memory-stat. I ran a short run to warm things up (100 requests, 10 concurrently). And then ran the big bench at 10,000 requests, 100 concurrently, and then ran again and recorded the results. At around 5,000 requests, I ran passenger-memory-stat to collect the data during mid-processing and then again at the end. These tests were run on a Ubuntu 64-bit Intrepid 8.10 virtual server that was allocated 1 CPU core and 512MB of RAM. Swap space was never triggered (0k utilization), and CPU utilization was roughly 85% within the Virtual Machine during the test runs. The host machine, which is four core CPU, showed one core busy at near 100% with the other three cores pretty much idle.

Throughput

Everybody likes to talk about throughput, so lets start there. As you can see, MRI and Rails was the worst performer while REE and Ramaze clocked in with the highest throughput in terms of requests per second. Be forewarned that throughput doesn’t necessarily equate to user response times. This metric simply tells you the load at which your system is capable of delivering content at.

Time per Request

Time per request helps you see about how long an average user request takes for a given load. As you can see here, with times in the milliseconds, we’re talking about numbers that will easily be swallowed in the noise of network latency. Even so, its not hard to be impressed by what the REE folks managed to pull out of their hat with regards to performance on the main ruby interpreter.

Memory Consumption

This final graph shows the sizeable dent REE makes in memory consumption for running a Rails application. Ramaze is already pretty efficient and lean and REE made practically no dent in memory consumption here. However, REE’s use of the faster memory allocation routines does have a good effect on boosting Ramaze’s performance (as seen in above graphs), so there’s still lots to be gained by running Ramaze with REE.

Conclusion

For me, I have two very, very viable solutions before me. Rails on Enterprise Ruby or Ramaze on Ruby 1.8.7. The final choice comes down to how much I can bet on my own Ruby prowess. Ramaze is a very lean framework that gets you going, but doesn’t get you a nine inch pillowed mattress topping for your bed like Rails does. Because Ramaze is so lean, I realize I will have to really take my Ruby skills to the next level and truly understand the ruby code that other developers produce. Unlike Rails, where there’s gobs and gobs of plugins and blogs on the topic that make pretty much any need simple to satisfy and get working, I have to know my Ruby to get a solution implemented on Ramaze. Not a bad thing, but certainly a more challenging and potentially rewarding path to take.

Aside from my personal concerns about making the jump, looking at these results, it is good to see that I have options and how easy it is to get a good deployment up and working compared to 3 or 4 years ago when I was banging my head against the wall for days on end to keep lighttpd + fcgi processes up and running around the clock. In these tests, it was very, very satisfying that not one single crash or dropped request was recorded in all the benchmarking I performed. Mongrel came along and made a big impact, but you still needed something like monit to watch Mongrel and restart it on ocvasion. Now Passenger has come along and made yet another big impact in the ease of deploying Rails, Ramaze, or any of the other ruby-based frameworks. We’ve come a long ways, indeed. I, for one, certainly look forward to what the Ruby sphere will bring us over the next couple of years.

My System

2.6.18-92.1.22.el5 #1 SMP Tue Dec 16 11:57:43 EST 2008 x86_64 x86_64 x86_64 GNU/Linux

The Problem

If you simply execute:

sudo gem install mysql

You’ll get the following errors on a 64-bit system:

$ sudo gem install mysql
Building native extensions.  This could take a while...
ERROR:  Error installing mysql:
        ERROR: Failed to build gem native extension.

/usr/local/bin/ruby extconf.rb install mysql
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lm... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lz... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lsocket... no
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lnsl... yes
checking for mysql_query() in -lmysqlclient... no
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

Provided configuration options:
        --with-opt-dir
        --without-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=/usr/local/bin/ruby
        --with-mysql-config
        --without-mysql-config
        --with-mysql-dir
        --without-mysql-dir
        --with-mysql-include
        --without-mysql-include=${mysql-dir}/include
        --with-mysql-lib
        --without-mysql-lib=${mysql-dir}/lib
        --with-mysqlclientlib
        --without-mysqlclientlib
        --with-mlib
        --without-mlib
        --with-mysqlclientlib
        --without-mysqlclientlib
        --with-zlib
        --without-zlib
        --with-mysqlclientlib
        --without-mysqlclientlib
        --with-socketlib
        --without-socketlib
        --with-mysqlclientlib
        --without-mysqlclientlib
        --with-nsllib
        --without-nsllib
        --with-mysqlclientlib
        --without-mysqlclientlib

Gem files will remain installed in /usr/local/lib/ruby/gems/1.8/gems/mysql-2.7 for inspection.
Results logged to /usr/local/lib/ruby/gems/1.8/gems/mysql-2.7/gem_make.out

After digging into the gem options and logs a bit and looking at where mysql actually got installed, I discovered that it was hitting all the locations the 32-bit version would normally install to, but not the 64-bit version, which ends up in /usr/lib64/mysql.

Installing the Ruby gems

To properly install the Ruby gems for mysql, you’ll have to explicitly tell the gem where the mysql config is and it will then handle all the rest.

gem install mysql -- --with-mysql-config=/usr/lib64/mysql/mysql_config

With that, you should see the following:

Building native extensions.  This could take a while...
Successfully installed mysql-2.7
1 gem installed

Check it out with:

irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> require 'mysql'
=> true
irb(main):003:0> quit

Life is good.

The Main Players

VMWare is the most mature offering of the popular offerings. It was (one of) the first to fully virtualize the Windows Operating system and has been steadily fine-tuning and optimizing its implementation while expanding further into the Enterprise setting than all others. With it far more mature VMMotion for delivering high-availability features, VMWare currently (Q4 2008) has the leg up in terms of product offering. VMWare has a product for nearly every niche, from virtualization products for the Windows, Mac, and Linux desktops to a complete Enterprise Infrastructure offering comprehensive business continuity solutions along with the tools to migrate and convert your physical and virtual images up the platform chain.

Xen is the leading Open Source virtualization solution available. XenSource was the leading firm providing sponsorship and stewardship of Xen and was recently acquired by Citrix, Inc., long a player in Thin-Application (terminal server) delivery of the Windows platform. Xen is variously offered and supported by Citrix, Oracle, IBM, SuSE, Sun, Red Hat, and numerous others.

Microsoft Hyper-V (formerly Viridian) is, of course, Microsoft’s entry into the virtualization platform and will become much more prevalent after Windows 2008 Server editions launch. Hyper-V is still in beta at the time of this writing and expected to emerge from beta within six month of Windows 2008 General Availability (GA).

Virtualization Technology

There are two terms to get familiar with when you start assessing virtualization solutions. Those terms are “fully-virtualized” and “para-virtualized” (which Microsoft uses “enlighted Windows” in their Hyper-V documentation).

A fully-virtualized OS image is one in which all hardware access is fully protected and/or emulated to the guest Operating System. This means an unmodified OS and kernel is running in the guest instance and the guest OS typically has no awareness that it is operating as a virtualized instance.

A para-virtualized OS image is one in which a special kernel is deployed for the guest OS and this typically gives the guest OS direct access to many of the hardware components and drivers that the Host OS itself utilizes. These para-virtualized guests are “aware” of their virtualization state and can communicate and coordinate with the other running guest OS’s on the Host in order to safely share hardware resources.

CPU Technology

Both Intel and AMD have gotten into the virtualization game by offering new instruction sets on their respective CPUs that provide hardware assisted virtualization support. These hardware optimized instruction sets allow for very efficient hypervisor implementations that allow virtualization technologies to offload the oft-complex resource scheduling and sharing logic to the processors themselves. Those two technologies are called, respectively, Intel-VT (Virtualization Technology) and AMD-V (Virtualization).

Because of VMWare’s approach to its hypervisor technology, VMWare traditionally makes the least use of these new instruction sets as it already claims to have the most highly optimized “trap and translate” and mature hardware management, allowing it to run fully-virtualized (unmodified) versions of guest OS’s.

Xen and Hyper-V, on the other hand, are heavily reliant on these new instruction sets and consequentially have a much “thinner” hypervisor since neither has to get into the business of trapping those unsafe instructions and rewriting them (as VMWare does) to offer a safe and reliable side-by-side hosting of OS’s.

The bottom line: VMWare does not require the special instruction sets while Xen and Hyper-V do. The one caveat to the above is that, because of some design decisions Intel made on memory management on their 64-bit instruction set, VMWare does require Intel-VT to host 64-bit guests, but is using only a small subset of the VT instruction set as it concerns memory, not specifically for managing safe access to hardware resources (so yes, it does get confusing!). Please see the table on slide #5 in the above presentation for a requirements matrix regarding these CPU instruction sets.

The benefits of Virtualizing

I believe most people tend to get interested in virtualization technology when they hear the pitches for the high-availability (HA) solutions that the technology makes almost trivially simple to implement. High-availability is a catch-all for describing a class of services that allow a guest OS to remain up and running with no apparent downtime should a physical host server go down. VMWare implements this with VMMotion while Citrix XenServer calls their solution “Live Migration.”

However, these HA solutions require a considerable investment in licensing to deploy and may not be fully justifiable for all IT shops. There are a number of other benefits that a data center can gain from implementing a virtualization platform that can be considerably more valuable than High-availability:

Increase Hardware Utilization

Probably the chief advantage the most basic virtualization deployment can offer is simply increasing your CPU and RAM utilization. Virtualization chiefly takes advantage of the fact that most services on a server are not running at full CPU and RAM capacity 100% of of the time so its very possible to get a perceived 100% of bare-metal performance even in the face of 3 or 4 (or more) guest OS’s running on one physical machine.

One of the most astounding discoveries I found in utilizing virtualization technologies is in implementing EnterpriseDB SQLGrid in a virtual environment. Grid SQL works pretty much as the name implies in that a SQL query is farmed out to all participating nodes in a grid and serviced in parallel with the results being compiled into final result by the master node on the grid. I found that the DBMS simply wasn’t taking full advantage of all the CPU cores and RAM when running on bare metal and decided to try running two or three slave nodes within guest OS’s on the same physical server. It turned out I could get nearly linear improvement in query result performance and much higher utilization of all CPU cores and available RAM.

Separate Service from Hardware

Another very beneficial avantage to virtualizing your operating systems, especially with Windows systems is that you can separate your OS install from the hardware. Our typical approach these days is to install Linux to the bare-metal and then install Windows into a virtual guest instance on our older hardware. By doing so, we’re getting near bare-metal performance out of our Windows systems on hardware beyond its reliable lifespan and when the hardware fails, we’re able to quickly bring up the entire OS instance on another server and continue along our merry way without having to get into re-installing, re-activating, and re-installing all the latest services patches (which, altogether now takes about 4 to 8 hours to get through all those mouse clicks and reboots).

This approach does take you down an entirely new way of thinking about backups and restores. Instead of deploying expensive backup solutions such as Veritas and Backup-Exec and installing and maintaining agents and directory exception lists, we now use very simple tools such as rsync and gzip to backup full disk images with the server offline and then only backup essential files (such as the SQL Server database backups) on the live running.

When we do service patches or change software settings, we again down the server and snapshot the disk image for backup (the live snapshots can probably do the same, but we haven’t truly explored this in our environment since its a relatively new feature to come along since establishing our standard operating procedures). We’re not a 24×7 critical shop and are able to establish service windows for maintenance, so this works very, very well for us in practice.

Business Continuity

The phenomenal thing we realized about server virtualization technologies and the simple backup and restore procedures discussed above is that we can carry our entire data center of over 60 servers around on one 500gb USB drive (or an offsite storage provider such as Amazon’s). Albeit scary from a security standpoint (you will definitely need some good security policies and checkpoints (such as backups being encrypted at rest)), but very liberating from a business continuity standpoint.

One no longer has to fully invest in a second, continuously running data center and keeping near identical hardware in sync in both locations, not to mention paying for being “online” and maintaining a broadband connection for continuously transferring the files to the second data center. If you have a solid working relationship with your hardware vendor or even a fully-hosted ISP that offers very quick server provisioning in their distance data center, you can literally walk in with your Install CDs for the virtualization host servers, install the hosts on the server equipment, transfer the server images off the USB drives, fire up the guest OS’s, restore the database backups and other user files (such as the office’s common drive), change your public DNS entries and you’re back online for your entire data center in an entirely new data center location.

Try New Things Cost Effectively

The final big benefit gained from virtualization technologies is being able to quickly try out new things with new servers without actually having to have a bunch of spare machines sitting around. New servers can be provisioned in under an hour. New services can be tried, demoed and evaluated and then the provisioned servers disposed of (or further tuned, licensed, and promoted to “production”).

Additionally, you can clone your production servers and painlessly perform “dry-run” upgrades and service patches on the instances and run full regression tests on the services all before ever actually performing the work on your live, production systems. In some cases, you can perform the upgrades on the cloned system in a sandbox environment, then move the new image over to production, down the live instance and up the new instance and be back online in a matter of minutes rather than having to take your services down for extending lengths of time.

Hardware Consideration

With the power in today’s CPUs and server hardware, its often difficult to determine where to invest to get the best bang for the buck. The following sub-components are listed in order of best ROI to least ROI:

1. Sufficient RAM – to run many OS guests
2. Fast Front-Side Bus (FSB) – to facilitate pushing the memory <--> CPU traffic around
3. Fast Hard Drives and Controllers – to boost disk I/O to optimal levels
4. Multi-core CPU’s – to boost number of guests on a single physical server
5. CPU speed – To get tasks done in a reasonable response time
6. Network bandwidth – To keep all guests responsive even when one is pushing a heavy network load

In addition to the above basic hardware consideration, high-availability options require the following:

• Dedicated iSCSI VLAN w/dedicated NICs
• Line-speed, non-blocking switches
• Host Bus Adaptors with TCP/IP offload Engines (TOE)
• Wide-striped LUN’s
• Load-balancing Storage Processors on your SAN
• Clustered Filesystems such as Oracle Cluster File System (OCFS2) or Global File System (GFS)

Getting Started

Many have asked me how to best get started in exploring and deploying virtualized solutions. My recommendation is to simply start small and take small steps all along the way. You do not need to invest heavily and take a great leap of faith to begin reaping the benefits of virtualization. Building a performant virtualization environment requires experience and knowledge of what you’re working with and an extension of your sleuthing skills for finding performance bottlenecks and resolving those issues. Instead of simply solving one physical machine’s problem, you’re now faced with the task of resolving any one guest OS’s performance problems in light of any other guest OS that may be running on that box. So you will have to become adept at looking at both the big picture for the host server and simultaneously looking at the individual guests to see where the issues are coming from.

Also, I would not recommend starting out virtualizing core infrastructure components (such as email, DNS, firewalls, etc.) or critical services, and especially I/O intensive services such as database servers until you have become adept at running a very stable and performant virtualization infrastructure.

Instead, try to cut your virtual teeth on currently unstable services or servers that are verily on their last legs hardware-wise. After all, if you migrate an old Pentium IV 500mhz server into very poorly performing virtual host environment that is running 2ghz multi-core processors, you are still hard-pressed to be taking a step backwards performance-wise.

Conclusion

Virtualization technologies are changing the landscape of server and data management in the data centers. More services can be run on significantly less hardware and those services begin to become much more portable and deployable. Virtualization addresses many issues from business continuity and high-availability management to quickly provisioning and simplifying server configurations while significantly boosting utilization levels of the hardware itself.

Background

In preparing this post, I ran across a very good resources that give an in-depth background look at Xen and virtualization technologies in general. These are provided here:

• Virtualization Terminology
• A moment of Xen: Virtualize Linux to test your apps
• x86 virtualization
• An Introduction to Virtualization

Turns out the mouse wheel would stop working after a switcharoo.  It took me a little while to track down the solution on the Ubuntu forums.  There was a lot of back and forth in the comments, but this is the dead-simple solution:

Ensure the psmouse module is loading

Edit and add the following line to /etc/modules:

psmouse

You’ll need to use sudo (i.e. sudo vi /etc/modules).

Set the imps option for the module

As sudo user, edit /etc/modprobe.d/options and append the following:

 # Make my mouse work with KVM
 options psmouse proto=imps

Reload the mouse module

Once you have the above, issue the following commands:

 # sudo modprobe -r psmouse
 # sudo modprobe -a psmouse

Once you do this, you should have a functioning mouse wheel. Test it out before and after switching machines with your KVM hot keys.

Thanks to the contributors of the Ubuntu Forums for this.

A little background…

Our shop has been using virtual technologies for just over two years. At the time we started taking it seriously, the open source offerings were just on the cusp of being production quality; Citrix had not yet bought into Xen; and VMWare had just rolled out their free VMWare Server option.  We had a mixture of Linux and Windows servers, all physical, and were really feeling some of the budget pince in our rapid growth. We decided it was time to rethink how we were growing our data-center and that led us to try out VMWare and Xen side-by-side, eventually siding with Xen since the bulk of our servers turned out to be Linux hosts while our predominant application, SQL Server 2005 was not all that conducive to being virtualized without some heavy, heavy investment in hardware purchases.

The gameplan, get some beefy servers, run Xen on them, with images residing on SAN and migrate all our physical Linux hosts to virtualized environment. Once converted, we’d simply reclaim all the excess hardware to fuel the Windows SQL Server hunger.

So, off we went. We purchased a three of Dell’s first dual quad-core servers at 1.66ghz and loaded them up with 16gb of RAM and a Dell AX150i iSCSI dual storage processor SAN to run our wonderfully envisioned HA cluster off of.

Xen had just turned 3.0 at the time and we really went through a lot of permutations in our stress testing and daily use before we finally came up with a stable blending of several offerings from the open source community. As wild as it may seem, we settled on Oracle VM’s first release as the Xen server, CentOS 5 repositories to manage packages on the host, Red Hat’s kickstart project to build the VM’s and ran a mixed guest environment of CentOS 4, CentOS 5, and Fedora Core 8. The guest environment’s heterogeneity had a purpose for existing. You see, our clients often dictated what version of Red Hat they wanted us to support when we deployed our systems in their data center, so we needed these three items (all relatively free of compulsory licensing fees) to give us the ability to effectively emulate any of our clients’ data center in-house for Q/A, stress/load, and deployment “dry-run” testing purposes.

After trying for many moons to effect a HA clustering solution that was rock-solid and backed by the SAN, we sadly had to regress from that vision (for now). VM images that were once all homed on the SAN and connected via iSCSI to the Xen hosts simply overwhelmed our SAN with I/O dropping low enough to force the filesystems of the guests into read-only state. We gave up the HA vision (for now) and purchased a bunch of very fast SCSI drives and RAID 5′d them, filling up each of the Xen Host’s drive bays and relocated the VM images to each Xen host’s local volumes.

This infrastructure has been our “norm” for the better part of the year now and we have a rock-solid deployment running about 80 virtualized Linux hosts on just four Dell servers. The ease with which we can set up a new server or tear one down and dismiss it has had quite an effect on our ability to test widely varying deployment scenarios and automate the deployments themselves.

Fast forward October 2008

The newest version of Xen from Red Hat (and hence CentOS for us) is looking rather sharp. Virtually all the issues we were encountering with just standalone Xen hosts appear to be addressed (in our limited testing). One of the more critical one, where our Dom0 environment would become unstable and we’d lose ability to manage it has been fixed (our lone-remaining headache in the above setup). We have not, however, ventured to test whether our HA woes have been resolved as well. One other thing we did notice, though was that VM guests shutdown and launch a whole lot faster than they do on our production servers, so that could hint to some very good performance improvements lurking in the newest releases of Xen.,

The downside: Six, yes count ‘em, six CDs must be downloaded to install the CentOS Xen version. Oracle VM, by comparison clocks in at roughly ONE 500mb CD image plus two companion CDs that give you their VM Manager and some other handy tools that go hand in hand with managing a Virtual Server environment. Oracle wisely dispenses with all the developer tools and Gnome desktop and office productivity tools that bloat up the RHEL/CentOS distros. Good security practices recommend only running what you need to run on your server and nothing else!

What follows below is how to get your first Virtualized server up and running by downloading just the one Oracle VM Xen Server CD. As far as I know, this is about as simple as it gets. The graphical environment is not used and none of the iSCSI or LVM (logical volume management) tools are employed. This is just a simple exercise that gets your hands dirty and playing with virtual technologies.

Step 1 – Install the Xen Server

Download the Oracle VM Server 32-bit Edition CD from Oracle and burn the ISO to CD. Then plop it into your CD drive on the machine you want and boot up. Follow the installation instructions on the screen (or the Online documentation). Once installed, boot up and log in as root.

Note: At the time of this writing, we’re installing Oracle VM 2.1.2

You should now be able to issue:

# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0   543     1     r-----   3036.7

And see an output similar to the above. Congratulations, you have a Xen Server up and running.

Step 2 – Initial File Preparation for the Guest

Initialize diskspace for your first Virtual Machine by performing the following commands (as root):

# cd /OVS
# mkdir -p local/numberone && cd local/numberone
# mkdir numberone
# cd numberone
# dd if=/dev/zero of=numberone_hda1.img bs=1024k count=6000
# dd if=/dev/zero of=numberone_hda2.img bs=1024k count=1500

This will set up a “local/numberone” folder on the Oracle Clustered File System partition, “/OVS” with first file being 6gb and 2nd file being 1.5gb. These will then become root partition and swap space of the guest VM, which we’re calling “numberone” here (you can name it any valid hostname you desire). Once these files are established, we create the filesystems for them with:

# mkfs.ext3 numberone_hda1.img
# mkswap numberone_hda2.img

Step 3 – Establish the Root partition of the Guest

Its time to push some files onto the root partition so that we have a valid system that can boot. Begin by issuing the following statements:

# cd /OVS/local/numberone
# mkdir hda1
# mount -o loop numberone_hda1.img hda1
# rsync  -avH /boot hda1
# rsync  -avH /root hda1
# rsync  -avH /dev hda1
# rsync  -avH /var hda1
# rsync  -avH /etc hda1
# rsync  -avH /usr hda1
# rsync  -avH /bin hda1
# rsync  -avH /sbin hda1
# rsync  -avH /lib hda1
# rsync -avH /selinux hda1
# mkdir hda1/{proc,sys,home,tmp}
# chmod 777 hda1/tmp
# unmount hda1
# rmdir -rf hda1

Step 4 – Configure the Guest

You really don’t want the guest OS coming up with the same configuration as your Xen Host, so lets get into the guest’s /etc folder and do some tweaking:

Edit /OVS/local/numberone/hda1/etc/fstab and set the root and swap partitions

/dev/hda1               /                       ext3    defaults        1 1
/dev/hda2               swap                    swap    defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
sysfs                   /sys                    sysfs   defaults        0 0

Edit /OVS/local/numberone/hda1/etc/sysconfig/network and change the hostname

NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=numberone
GATEWAY=192.168.0.5

Note: Gateway is the firewall on your network providing Internet connectivity. Mine happens to be 192.168.0.5.

Edit /OVS/local/numberone/hda1/etc/hosts and change the hostname

127.0.0.1               numberone localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6

Rename /OVS/local/numberone/hda1/etc/sysconfig/network-scripts/ifcfg-eth0 to /OVS/local/numberone/hda1/etc/sysconfig/network-scripts/ifcfg-eth0.bak so its not used when the guest first boots up.

Similarly, rename /OVS/local/numberone/hda1/lib/tls to /OVS/local/numberone/hda1/lib/tls.disabled

Then unmount the guest’s root partition:

# cd /OVS/local/numberone
# umount hda1

Create a domain configuration file for your new guest in /etc/xen/numberone on the Xen Server (not the guest’s root partition):

#  -*- mode: python; -*-
#============================================================================
# Python configuration setup for 'xm create'.
# This script sets the parameters used when a domain is created using 'xm create'.
# You use a separate script for each domain you want to create, or
# you can set the parameters for the domain on the xm command line.
#============================================================================

#----------------------------------------------------------------------------
kernel = "/boot/vmlinuz-2.6.18-8.1.15.1.16.el5xen"
ramdisk = "/boot/initrd-2.6.18-8.1.15.1.16.el5xen.img"
memory = 512
name = "numberone"

vif = [ 'bridge=xenbr0' ]

disk = ['file:/OVS/local/gox/gox_hda1.img,hda1,w',
        'file:/OVS/local/gox/gox_hda2.img,hda2,w' ]

root = "/dev/hda1 ro"

# Sets runlevel 4.
extra = "4"

The mac address was left out of the vif definition. We’ll boot up the guest to console and see what “reasonable value” the Xen host establishes for the server then come back and hard code the value so its not changing all the time.

Step 5 – Booting for the first time

To boot the guest, we’ll use the command-line tools, specifically “xm” to create (boot) the guest and shutdown rather than the Oracle VM Manager. So lets have a look:

# xm create -c numberone

The “-c” flag boots you with console view of the guest. If all went well, you’ll see a typical Linux boot up process that leads all the way to a login prompt similar to this:

Oracle VM server release 2.1.2
Hypervisor running in UNDETERMINED bit mode (WARNING: XEND IS PROBABLY NOT RUNNING).

Network :
Management Interface  :
If : eth1(Down)  Mac :  IP address : 

Configured Networks and Bridges :
If : eth0	 Mac : 00:16:3E:22:F3:4F

CPU :
cpu family	: 15
model		: 31
model name	: AMD Athlon(tm) 64 Processor 3200+

numberone login:

Copy that Mac address to clipboard (or write it down) because you’re going to need it for the next steps. Log in as root, same password as the Xen Host (since we copied everything off the host to begin with!).

Edit /etc/sysconfig/network-scripts/ifcfg-eth0:

DEVICE=eth0
BOOTPROTO=static
HWADDR=00:16:3E:22:F3:4F
ONBOOT=yes
IPADDR=192.168.0.150
NETMASK=255.255.255.0

Once you’ve done this, you should be able to start your network and ping your gateway:

# ifup eth0
# ping 192.168.0.5
PING 192.168.0.5 (192.168.0.5) 56(84) bytes of data.
64 bytes from 192.168.0.5: icmp_seq=1 ttl=64 time=3.33 ms
64 bytes from 192.168.0.5: icmp_seq=2 ttl=64 time=1.04 ms

If not, check for typos. Otherwise, you’re done with configuring the guest. Continue by logging out of your guest then break from console with CTRL-] (control-key plus left-bracket key). Edit /etc/xen/numberone and change the vif line to read:

vif = [ 'mac=00:16:3E:22:F3:4F, bridge=xenbr0' ]

This gels your mac address for the guest VM so the network will be reliably started on reboots for eth0 device. Now issue the following:

# xm shutdown numberone
# xm top

And watch your guest disappear from the listing. Once its gone, issue the following to bring it back up:

# xm create numberone

Finally, if everything’s gone well, you will be able to ssh to the guest VM with:

# ssh 192.168.0.150 -l root

(Hey, I didn’t say it was uber secure or that we’re following best practices regarding root accounts!)

If you can’t ssh into the guest from the xen server, you’ll have to get back into console and review logs and figure out what went wrong. You may need to fix typos in your configs or try another xen bridge such as xenbr1. If you are able to issue “ifup eth0″ within the guest and can ping your firewall, its not network connectivity, but a typo in your /etc/sysconfig/* files somewhere. Probably the best place to double-check things is the mac address.

# xm console numberone

One last tip. If you want to install packages that didn’t come on the Oracle VM install CD, you can add the CentOS5 repository to the VM guest and install various packages using yum thereafter. To do this, log into your Guest VM, and edit the following new file: /etc/yum.repos.d

[CentOS5 base]
name=CentOS-5-Base
mirrorlist=http://mirrorlist.centos.org/?release=5&arch=$basearch&repo=os
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
gpgcheck=0
enabled=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

[CentOS5 updates]
name=CentOS-5-Updates
mirrorlist=http://mirrorlist.centos.org/?release=5&arch=$basearch&repo=updates
gpgcheck=0
enabled=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

[CentOS5plus]
name=CentOS-5-Plus
mirrorlist=http://mirrorlist.centos.org/?release=5&arch=$basearch&repo=centosplus
gpgcheck=0
enabled=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

So there you have it. One of the simplest approaches to getting your first Xen guest Virtual Machine up and running and well on your way to experimenting with virtual technologies.

Special Note: Much credit must go to my colleague, Manyuan Jin, who spent many, many tireless hours researching early instability issues and treading through the jungle of open source solutions to assemble a working solution for our shop.

To get started, I needed to know how to get the data into Trac. The following links gave me an API I could utilize to create trac pages and content:

Trac Data Models
The Data model Python code

Great! Except there’s one small problem. I don’t really know python all that well, even though I did do a bit of Zope development, oh, five years ago. These days, my language of choice is Ruby, so I needed to figure out how to parse the Oddmuse syntax and turn it into Trac syntax. The strategy: Parse with Ruby, load cleaned page files with Python.

This script will take a filename at command line and convert it and save back to file, but different extension (I chose *.om for “oddmuse”). In the script below, I made the command-line optional as I discovered I didn’t handle every case as I worked on developing this script, so I simply changed the line that pulls the command-line argument to point to a specific file and output such to terminal and this allowed me to debug and fix as I went:

#!/opt/local/bin/ruby
# $FS  = "\xb3";      # The FS character is the RECORD SEPARATOR control char in ASCII
# $FS0 = "\xb3";      # The old FS character is a superscript "3" in Latin-1
# $FS1 = $FS . '1';   # The FS values are used to separate fields
# $FS2 = $FS . '2';   # in stored hashtables and other data structures.
# $FS3 = $FS . '3';   # The FS character is not allowed in user data.

FS = 179.chr

SITE_NAME = "corporate_wiki"
ATTACHMENT_URL_BASE = "https://intranet.example.com/#{SITE_NAME}/attachment/wiki/ConvertedAttachments/"

require 'find'
require 'ftools'

def safe_page_name(pagename)
  result = pagename.gsub("'","")
  result = result.slice(/(\w+)/).first
end

def get_section(page, section)
  page.each_with_index {|e,i| return page[i+1] if e == section}
  return ''
end

def asterick_prefixed(text)
  t = text.match(/(\*+)([^\b]*)/)
  return text if t.nil?

  tokens = t.to_a
  return ('  ' * tokens[1].size) + '* ' + tokens[2]
end

def colon_prefixed(text)
  return "  #{colon_prefixed(text.slice(1,text.size))}" if text[0] == ":"[0]
  return text
end

def pound_prefixed(text)
  return "  1. #{text.slice(1,text.size)}" if text[0] == "#"[0]
  return text
end

def fix_headers(text)
  t = text.match(/(=+)([^=]+)(=+)/)
  return text if t.nil?

  tokens = t.to_a
  return "#{tokens[1]} #{tokens[2].strip} #{tokens[1]}"
end

def replace_a_url(url, text)
  text.gsub!("http://#{url}", ATTACHMENT_URL_BASE)
  text.gsub!("https://#{url}", ATTACHMENT_URL_BASE)
  text.gsub!("http:/#{url}", ATTACHMENT_URL_BASE)
  text.gsub!("https:/#{url}", ATTACHMENT_URL_BASE)
  return text
end

def replace_wiki_urls(text)
  matches = text.match(/\[http(.)*\]/)
  result = text
  if matches
    result = replace_a_url("wiki.example.com/#{SITE_NAME}/wikifiles/", result)
    result = replace_a_url("pdfreviewfiles/", result)
    result = replace_a_url("cgi-bin/", result)
  end
  result
end

def convert_formats(text)
  result = text.gsub(/\[\[(\w+)\]\]/,"[wiki:" + '\1' + "]")
  result = asterick_prefixed(result)
  result = colon_prefixed(result)
  result = pound_prefixed(result)
  result = replace_wiki_urls(result)
  result = fix_headers(result)
  result
end

def oddmuse_to_trac(text)
  result = ''
  in_pre = false
  text.split("\n").each do |line|

    # Detect indented lines as these are rendered as PRE html blocks
    trimmed = line.lstrip
    is_prefixed = ((trimmed.size < line.size) and (trimmed[0] != '*'[0]))
    result << "{{{\n" if (is_prefixed and !in_pre)

    result << "}}}\n" if (!is_prefixed and in_pre)
    in_pre = is_prefixed

    # Don't process special characters when in PRE block
    if in_pre
      result << line << "\n"
    else
      result << convert_formats(line) << "\n"
    end
  end
  result += "}}}\n" if in_pre
  result
end

path = ARGV[0] || "page/U/UsingThePhones.db"
raw_page = File.new(path).read
page2 = raw_page.split(FS + '2')
page3 = raw_page.split(FS + '3')

text = get_section(page3, 'text')

pagename = File.basename(path).split('.').first
author = get_section(page2, 'username')
address = get_section(page2, 'ip')
body = oddmuse_to_trac(text)

puts "writing: #{pagename}.om"
outfile = File.new(path.gsub('.db','.om'),'w')
outfile.puts 'pagename:' + pagename
outfile.puts 'author:' + author
outfile.puts 'address:' + address
outfile.puts body
outfile.close

puts body if ARGV[0].nil?

Below, comes the Python side of the equation. Once I did the heavy lifting in Ruby, I needed to generate all the pages. As you can see, there's some parsing happening, but its very simple parsing and I was able to apply what simple Python knowledge I still retained pretty effectively:

import sys
from trac.env import Environment
from trac.wiki.model import WikiPage

print sys.argv[1]
file = open(sys.argv[1], 'r')
pagename = file.readline().split(':')[1].split('\n')[0]
author_name = file.readline().split(':')[1].split('\n')[0]
address = file.readline().split(':')[1].split('\n')[0]

text = file.read()

env = Environment('/srv/www/trac/corporate_wiki')

# Read an existing or new WikiPage:
page = WikiPage(env, pagename)
if page.text != text:
        page.text = text
        page.save(author=author_name, comment='imported from oddmuse wiki', remote_addr=address)

As you can see, this file simply takes the command line argument, open the file for reading, pull the various variables out with the remainder of the file going to the page content.

During the conversion, I noticed some pages were failing to convert at all, like "Michael'sTodoList" The culprit, the tick in the page name and thus the filename. The scripts themselves weren't the problem, it was the xargs parameter passing that was the issue (getting to that below). So I ran the below script to rename all the troublesome filenames and then added similar tick to underscore substitution in the main Ruby script above.

require 'find'
require 'ftools'

total_size = 0

Find.find('./page') do |path|
  if FileTest.directory?(path)
    if File.basename(path)[0] == ?.
      Find.prune       # Don't look any further into this directory.
    else
      next
    end
  else
    if path.match(/[\']/)
      safe_path = path.gsub("'", "_")
      puts path + ' to ' + safe_path
      File.move(path, safe_path)
    end
    end
  end

A simple up-front execution of the script was all that was needed to now be able to pick up the missing pages:

ruby fix_unsafe_files.rb

Finally, to bring it all together, I utilized find to locate all the *.db Oddmuse files which I had located into the various wiki subfolders where these scripts sat, passing the filename to the script via the xargs utility:

find . -name *.db -print0 | xargs -0 -L 1 ruby rdpage.rb

I then essentially repeated the above, but for the generated *.om files, passing to the Python mkpage.py script:

find . -name *.om -print0 | xargs -0 -L 1 python mkpage.py

One last thing, before we're done. Each wiki had attachments. So I had to think how to get those attachments referenced and linked to the Trac wiki. A little investigation revealed that each attachment got a database entry in the attachment table. My task was a bit complicated by the fact that the users over the years in the Oddmuse environment scattered their attachments everywhere and also used varying syntax to refer to them, hence, if you review the main Ruby parsing script, you'll see the "replace_a_url" method above that cleans all those funky URLs up. So the script I used below simply scans through all the directories where we had files stored and attached them to one "ConvertedAttachments" page. Probably not the ideal solution, but again, got the job done quickly.

#  type |          id          |  filename  |  size  |    time    | description | author |      ipnr
# ------+----------------------+------------+--------+------------+-------------+--------+----------------
#  wiki | ConvertedAttachments | gotapi.png | 200272 | 1218381947 |             | michael | 192.168.16.100

path = ARGV[0] || 'generate_attachments.rb'
filename = File.basename(path)

puts "insert into attachment values ('wiki', 'ConvertedAttachments', '#{filename}', #{FileTest.size(path)}, null, null, 'oddmuse','192.168.1.246');"

So, running the above needs to be piped to a SQL fisle and then executed against your Trac database. In my case, Postgresql. Since I had many directories to snatch up, I ran it for each folder as appropriate and simply kept appending until I had all the attachment directories represented.

ls | xargs -0 -L 1 >> attachments.sql
psql trac_db < attachments.sql

All in all, not too bad of a conversion for a few hour's work. Its about 95% correct. Some of the page names didn't translate to Trac, and could probably do with some regexp search and replace on the page names themselves. The biggest gain is that we have preserved the information in the old systems by porting it to the new systems we all know and use on a daily basis, so WikiGardening is more likely to keep the content fresher and its one less system for our end-users to figure out how to use.

I can now retire the server running Oddmuse, stop doing backups and keeping up with documentation and training new system administrators on how to manage Oddmuse AND Trac.

I am using Mac OS X Leopard 10.5.4 at the time of this writing. These are the steps I took to fix that most hideous of all colors, dark blue on black background:

1. Set your terminal theme colors to your liking
2. Set up ls to colorize directory listings by default. There are two ways to do this: Either set the CLICOLOR Environment variable or add an alias for ls to include the -G flag. I chose the CLICOLOR flag:

   export CLICOLOR=1
   

3. Next up, install the ANSI color SIMBL plugin found here: http://niw.at: works of Yoshimasa Niwa. Make sure Terminal.app is closed while you install the plugin then relaunch Terminal to find under its main menu, “Color Preferences…”
4. Finally, edit that dark blue away!

If you’re interested in tailoring the output of the colors even further, you can also export the LSCOLORS variable:

export LSCOLORS="exfxcxdxbxegedabagacad"

And then change according to the manual entry for ls (“man ls”):

One final trick to get my bash shell exactly like I like it was to set the prompt. This can likewise be done by editing ~/.profile:

PS1="[\u@\h \W] "

I combed the documentation for days trying to figure out exactly how to set up a server to hand out time to all my other servers. Normally, my Google-fu can pierce just about any need, but “time” “server” “local” and so on are all too common terms in both client-and-server and client-only configurations and by far and away, the bulk of the documentation guides you through the latter. The documentation that guides you through the former is dense with references to peers, stratum, broadcasting, multi-casting, and so on.

So, here’s how to set up an NTP service as both client and local time server on CentOS 4:

yum install ntp

Then edit the /etc/ntp.conf file and set up the public ntp server pools and uncomment the broadcast line and change IP mask to match your network’s:

server 0.rhel.pool.ntp.org
server 1.rhel.pool.ntp.org
server 2.rhel.pool.ntp.org
broadcast 192.168.1.255

Modify your /etc/sysconfig/iptables to allow other servers to connect:

-A RH-Firewall-1-INPUT -p udp -m udp --dport 123 -j ACCEPT

Finally, go to the “client server” and install ntp on it just as you did with the ntp server, except, this time, you edit your /etc/ntp.conf file to point to your new local time server. In my case, I set up two servers to give out times and I added ntp.1 and ntp.2 to my local DNS servers, so my config file for the “client servers” uses:

server ntp.1.example.com
server ntp.2.example.com

Its really that simple! Its probably not the most secure way to do the job as there’s plenty of information overload in the ntp documentation on configuring securely. However, with all of my servers sitting in a trusted VLAN and the DNS entries are only served to these servers and NTP port is blocked between this VLAN and others, I reckon it would be fairly hard to hijack the local ntp services. In other words, the simple approach is good enough, for now.