I was having a good bit of trouble getting ports to install ImageMagick and have had trouble many other times getting ports to install things. As such, I often fell back on fink or compiling and installing from source rather than figuring out the root causes of my macports problems.

Macports was surprisingly easy to fix up and get fully operational again, and this was on a fairly old machine that I am pretty certain I started with “Darwin Ports” roughly 3.5 years ago.

First Thing’s First

First, know that the official macports website is: trac.macports.org. Be wary of anything “darwinports” as name is officially retired in favor of “macports.”

Trouble a-brewing

It all started innocently enough. I wanted to install ImageMagick on my mac. I’ve been through installing ImageMagick on Linux and knew it was not going to be a cakewalk to install from source. ImageMagick simply has too many dependencies to relish the thought of an install from source route. So I googled and found instructions for ports and began:

sudo port install tiff -macosx imagemagick +q8 +gs +wmf
Warning: Group file could not be located.
--->  Activating tiff 3.8.2_3
Error: Target org.macports.activate returned: Image error: Another version of this port (tiff @3.8.2_2+darwin_9+macosx) is already active.

Oh, ok, lets remove the old one…

sudo port uninstall tiff -macosx imagemagick +q8 +gs +wmf
--->  The following versions of tiff are currently installed:
--->  	tiff @3.8.2_2+darwin_9+macosx (active)
--->  	tiff @3.8.2_3
Error: port uninstall failed: Registry error: Please specify the full version as recorded in the port registry.
[mwlang@macdoze shared] sudo port uninstall tiff @3.8.2_2+darwin_9+macosx
--->  Unable to uninstall tiff 3.8.2_2+darwin_9+macosx, the following ports depend on it:
--->  	gtk2
--->  	lcms
--->  	sane-backends
--->  	xsane
Error: port uninstall failed: Please uninstall the ports that depend on tiff first.
[mwlang@macdoze shared] sudo port uninstall tiff @3.8.2_2+darwin_9+macosx gtk2 lcms sane-backends xsane
--->  Unable to uninstall tiff 3.8.2_2+darwin_9+macosx, the following ports depend on it:
--->  	gtk2
--->  	lcms
--->  	sane-backends
--->  	xsane
Error: port uninstall failed: Please uninstall the ports that depend on tiff first.
[mwlang@macdoze shared] sudo port uninstall gtk2 lcms sane-backends xsane
--->  Unable to uninstall gtk2 2.12.9_0+x11, the following ports depend on it:
--->  	gconf
--->  	gnome-keyring
--->  	libglade2
--->  	gail
--->  	libgnomecanvas
--->  	libbonoboui
--->  	poppler
--->  	py25-gtk
--->  	gtk-nodoka-engine
--->  	xsane
Error: port uninstall failed: Please uninstall the ports that depend on gtk2 first.

Not a very auspicious start, is it? Time to step back and think about this one and dig into the macports facility to learn how to properly manage things. New strategy: Get an education.

Discovering Deactivate/Activate

So, looking at the above problem, I realized that there was almost certainly an easier route out of this dependency nightmare and I needed to learn how to upgrade those old, stale packages. Going through the docs reveals activate and deactivate commands. Sounds perfect, lets try:

sudo port activate tiff @3.8.2_3
--->  Activating tiff @3.8.2_3
Error: port activate failed: Image error: Another version of this port (tiff @3.8.2_2+darwin_9+macosx) is already active.
[mwlang@macdoze shared] sudo port deactivate tiff @3.8.2_2+darwin_9+macosx
--->  Deactivating tiff @3.8.2_2+darwin_9+macosx
[mwlang@macdoze shared] sudo port activate tiff @3.8.2_3
--->  Activating tiff @3.8.2_3

So there’s the kicker. First, deactivate the old version, then activate the new version.

So Macports itself is out of date, you say?

sudo port install tiff -macosx imagemagick +q8 +gs +wmf
Error: Unable to execute port: invalid command name "use_autoreconf"

I had no clue what this was. Googling around was turning up precious little. So I hopped over into the IRC #macports channel. Right off the bat, Toby tells me to selfupdate. I had made the mistake of thinking that “sudo port sync” was keeping macports packages all up to date in one fell swoop. But the two are two distinct activities. So remember to keep ports itself updated with this:

sudo port selfupdate

With that, port was upgraded from a very old version to a bright and shiny new version. Time to try to install ImageMagick again. And I’m happy to report that I got much further along this time. But still hit a roadblock…

Mysterious failure demystified

A little down the chain of installing various xorg packages, I hit upon this one with the xorg-libx11 package:

checking for XPROTO... configure: error: Package requirements (xproto >= 7.0.13) were not met.
Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively you may set the XPROTO_CFLAGS and XPROTO_LIBS environment variables
to avoid the need to call pkg-config.  See the pkg-config man page for
more details.

Error: The following dependencies failed to build: ghostscript xorg-libXext xorg-libX11 xorg-libXt xorg-libsm xorg-libice
Error: Status 1 encountered during processing.
[mwlang@macdoze shared] sudo port install ImageMagick
Error: Requested variants do not match original selection.
Please perform 'port clean ImageMagick' or specify the force option.
Error: Status 1 encountered during processing.

Strange, what’s that xproto >= 7.0.13? How odd that a dependency would fail! I tried a sudo port install xproto, but that was a no-go (so don’t bother trying, it’ll fail for you, too). but then I noticed the dependency list (which oddly is missing xorg-xproto) and most of the packages are prefixed with “xorg-”, and tried sudo port install xorg-xproto and thus, was on my way again.

I lie! I lie! I had no clue. I googled “xproto port install” and learned the truth for the real package name that way. However, as it happens, I found the answer on darwinports dot com in a dependency list for xorg-libx11. I casually mentioned this site in the IRC channel and the #macports tenants were very quick to point out was an imposter site, so be wary of anything you read and learn there. Information is liable to be stale while the site owner attempts to lighten your wallet a bit for no good cause. Little did I know!

Final tidbit

Once I got the ports system updated and the xproto package installed, it was a fair breeze to finish with the ImageMagick installation. One last command I learned from the good folks in #macports is to run the following on a frequent basis to keep your macports installation in fair good shape:

sudo port selfupdate && sudo port upgrade outdated

Conclusion

For years, I struggled with ports, and for no good reason. Once I took the time to learn the package management’s commands (along with the help of those in #macports, for which I’m very grateful to them for saving me a few hours of head-scratching), I am finding I’m able to install packages painlessly on the mac and have even tried a couple of others I had problems with in the past and they’re all installing without a hitch. The lesson for me: take the time to learn your tools. It may be slow-going at first, but the rewards always seem to pay back big dividends down the road.

To get started, I needed to know how to get the data into Trac. The following links gave me an API I could utilize to create trac pages and content:

Trac Data Models
The Data model Python code

Great! Except there’s one small problem. I don’t really know python all that well, even though I did do a bit of Zope development, oh, five years ago. These days, my language of choice is Ruby, so I needed to figure out how to parse the Oddmuse syntax and turn it into Trac syntax. The strategy: Parse with Ruby, load cleaned page files with Python.

This script will take a filename at command line and convert it and save back to file, but different extension (I chose *.om for “oddmuse”). In the script below, I made the command-line optional as I discovered I didn’t handle every case as I worked on developing this script, so I simply changed the line that pulls the command-line argument to point to a specific file and output such to terminal and this allowed me to debug and fix as I went:

#!/opt/local/bin/ruby
# $FS  = "\xb3";      # The FS character is the RECORD SEPARATOR control char in ASCII
# $FS0 = "\xb3";      # The old FS character is a superscript "3" in Latin-1
# $FS1 = $FS . '1';   # The FS values are used to separate fields
# $FS2 = $FS . '2';   # in stored hashtables and other data structures.
# $FS3 = $FS . '3';   # The FS character is not allowed in user data.

FS = 179.chr

SITE_NAME = "corporate_wiki"
ATTACHMENT_URL_BASE = "https://intranet.example.com/#{SITE_NAME}/attachment/wiki/ConvertedAttachments/"

require 'find'
require 'ftools'

def safe_page_name(pagename)
  result = pagename.gsub("'","")
  result = result.slice(/(\w+)/).first
end

def get_section(page, section)
  page.each_with_index {|e,i| return page[i+1] if e == section}
  return ''
end

def asterick_prefixed(text)
  t = text.match(/(\*+)([^\b]*)/)
  return text if t.nil?

  tokens = t.to_a
  return ('  ' * tokens[1].size) + '* ' + tokens[2]
end

def colon_prefixed(text)
  return "  #{colon_prefixed(text.slice(1,text.size))}" if text[0] == ":"[0]
  return text
end

def pound_prefixed(text)
  return "  1. #{text.slice(1,text.size)}" if text[0] == "#"[0]
  return text
end

def fix_headers(text)
  t = text.match(/(=+)([^=]+)(=+)/)
  return text if t.nil?

  tokens = t.to_a
  return "#{tokens[1]} #{tokens[2].strip} #{tokens[1]}"
end

def replace_a_url(url, text)
  text.gsub!("http://#{url}", ATTACHMENT_URL_BASE)
  text.gsub!("https://#{url}", ATTACHMENT_URL_BASE)
  text.gsub!("http:/#{url}", ATTACHMENT_URL_BASE)
  text.gsub!("https:/#{url}", ATTACHMENT_URL_BASE)
  return text
end

def replace_wiki_urls(text)
  matches = text.match(/\[http(.)*\]/)
  result = text
  if matches
    result = replace_a_url("wiki.example.com/#{SITE_NAME}/wikifiles/", result)
    result = replace_a_url("pdfreviewfiles/", result)
    result = replace_a_url("cgi-bin/", result)
  end
  result
end

def convert_formats(text)
  result = text.gsub(/\[\[(\w+)\]\]/,"[wiki:" + '\1' + "]")
  result = asterick_prefixed(result)
  result = colon_prefixed(result)
  result = pound_prefixed(result)
  result = replace_wiki_urls(result)
  result = fix_headers(result)
  result
end

def oddmuse_to_trac(text)
  result = ''
  in_pre = false
  text.split("\n").each do |line|

    # Detect indented lines as these are rendered as PRE html blocks
    trimmed = line.lstrip
    is_prefixed = ((trimmed.size < line.size) and (trimmed[0] != '*'[0]))
    result << "{{{\n" if (is_prefixed and !in_pre)

    result << "}}}\n" if (!is_prefixed and in_pre)
    in_pre = is_prefixed

    # Don't process special characters when in PRE block
    if in_pre
      result << line << "\n"
    else
      result << convert_formats(line) << "\n"
    end
  end
  result += "}}}\n" if in_pre
  result
end

path = ARGV[0] || "page/U/UsingThePhones.db"
raw_page = File.new(path).read
page2 = raw_page.split(FS + '2')
page3 = raw_page.split(FS + '3')

text = get_section(page3, 'text')

pagename = File.basename(path).split('.').first
author = get_section(page2, 'username')
address = get_section(page2, 'ip')
body = oddmuse_to_trac(text)

puts "writing: #{pagename}.om"
outfile = File.new(path.gsub('.db','.om'),'w')
outfile.puts 'pagename:' + pagename
outfile.puts 'author:' + author
outfile.puts 'address:' + address
outfile.puts body
outfile.close

puts body if ARGV[0].nil?

Below, comes the Python side of the equation. Once I did the heavy lifting in Ruby, I needed to generate all the pages. As you can see, there's some parsing happening, but its very simple parsing and I was able to apply what simple Python knowledge I still retained pretty effectively:

import sys
from trac.env import Environment
from trac.wiki.model import WikiPage

print sys.argv[1]
file = open(sys.argv[1], 'r')
pagename = file.readline().split(':')[1].split('\n')[0]
author_name = file.readline().split(':')[1].split('\n')[0]
address = file.readline().split(':')[1].split('\n')[0]

text = file.read()

env = Environment('/srv/www/trac/corporate_wiki')

# Read an existing or new WikiPage:
page = WikiPage(env, pagename)
if page.text != text:
        page.text = text
        page.save(author=author_name, comment='imported from oddmuse wiki', remote_addr=address)

As you can see, this file simply takes the command line argument, open the file for reading, pull the various variables out with the remainder of the file going to the page content.

During the conversion, I noticed some pages were failing to convert at all, like "Michael'sTodoList" The culprit, the tick in the page name and thus the filename. The scripts themselves weren't the problem, it was the xargs parameter passing that was the issue (getting to that below). So I ran the below script to rename all the troublesome filenames and then added similar tick to underscore substitution in the main Ruby script above.

require 'find'
require 'ftools'

total_size = 0

Find.find('./page') do |path|
  if FileTest.directory?(path)
    if File.basename(path)[0] == ?.
      Find.prune       # Don't look any further into this directory.
    else
      next
    end
  else
    if path.match(/[\']/)
      safe_path = path.gsub("'", "_")
      puts path + ' to ' + safe_path
      File.move(path, safe_path)
    end
    end
  end

A simple up-front execution of the script was all that was needed to now be able to pick up the missing pages:

ruby fix_unsafe_files.rb

Finally, to bring it all together, I utilized find to locate all the *.db Oddmuse files which I had located into the various wiki subfolders where these scripts sat, passing the filename to the script via the xargs utility:

find . -name *.db -print0 | xargs -0 -L 1 ruby rdpage.rb

I then essentially repeated the above, but for the generated *.om files, passing to the Python mkpage.py script:

find . -name *.om -print0 | xargs -0 -L 1 python mkpage.py

One last thing, before we're done. Each wiki had attachments. So I had to think how to get those attachments referenced and linked to the Trac wiki. A little investigation revealed that each attachment got a database entry in the attachment table. My task was a bit complicated by the fact that the users over the years in the Oddmuse environment scattered their attachments everywhere and also used varying syntax to refer to them, hence, if you review the main Ruby parsing script, you'll see the "replace_a_url" method above that cleans all those funky URLs up. So the script I used below simply scans through all the directories where we had files stored and attached them to one "ConvertedAttachments" page. Probably not the ideal solution, but again, got the job done quickly.

#  type |          id          |  filename  |  size  |    time    | description | author |      ipnr
# ------+----------------------+------------+--------+------------+-------------+--------+----------------
#  wiki | ConvertedAttachments | gotapi.png | 200272 | 1218381947 |             | michael | 192.168.16.100

path = ARGV[0] || 'generate_attachments.rb'
filename = File.basename(path)

puts "insert into attachment values ('wiki', 'ConvertedAttachments', '#{filename}', #{FileTest.size(path)}, null, null, 'oddmuse','192.168.1.246');"

So, running the above needs to be piped to a SQL fisle and then executed against your Trac database. In my case, Postgresql. Since I had many directories to snatch up, I ran it for each folder as appropriate and simply kept appending until I had all the attachment directories represented.

ls | xargs -0 -L 1 >> attachments.sql
psql trac_db < attachments.sql

All in all, not too bad of a conversion for a few hour's work. Its about 95% correct. Some of the page names didn't translate to Trac, and could probably do with some regexp search and replace on the page names themselves. The biggest gain is that we have preserved the information in the old systems by porting it to the new systems we all know and use on a daily basis, so WikiGardening is more likely to keep the content fresher and its one less system for our end-users to figure out how to use.

I can now retire the server running Oddmuse, stop doing backups and keeping up with documentation and training new system administrators on how to manage Oddmuse AND Trac.

It all started as our quest to find a reliable laptop that would be powerful enough to do development work and not spend half its time in the shop. We had everything from Dell Laptops, Sony Viaos, to Asus and Acer, not to mention the behemoth Xtreme “desktop replacement” laptops that were weighing in at some 14lbs.

As the company’s network manager, the first thing I noticed about the Macs was that their users rarely said another word to me after that first day receiving them. Silent workers aren’t usually attention getters, but when all you’re doing is constantly repairing or shipping off hardware that has failed, the absence of doing so, even for a small group of users, does tend to grab your attention.

A few months later with still no chatter from the “mac guys,” I decided to check up on how the macs were doing. The developers loved them! Ok, so now I have to get one for myself and find out firsthand. So we bought another wave of Macs and passed out to more developers and I collected one of the older slower ones to give a test drive of my own.

Well, to be perfectly honest, I stumbled greatly in the beginning trying to understand the Mac interface and essentially brute forced my way to productivity by living in Windows running on Parallels and the console when on the Mac side. Little by little, I began to learn “the mac way” and utilize their silly one button mouse and Desktop GUI. I’m sorry, but in this day and age, you need two buttons!

As I left Windows behind, I began to notice something. I wasn’t constantly updating and rebooting the system. Nearly daily service patching. Gone! Anti-virus protection. Gone! Dealing with Firewall messages. Gone! Insane “are you sure you want to do that?” followed by “are you really sure?” prompts. Gone! System slowing down to a crawl with each new software installed. Gone! Cleaning the registry to get rid of all those auto-start components after new installs. Gone! Answering “No!” to “you have unused icons on your desktop…” Gone! Genuine Advantage Assurances? Yup, gone!

Good gosh. Was I really that numb to how much maintaining a working Windows systems had encroached on my life? Not only as a user, but as a systems support person for our entire company. No wonder those Mac adopters were so quiet.

All these little things make a huge difference. Since I’ve started using a mac, my own productivity is a lot higher. I’m able to do Ruby development in the best supported environment (Mac OS X) for development…which is a bonus of being UNIX based, which suits me well. I’m able to run Windows XP in Parallels Virtual Machine in order to run SQL Server 2005. I’m also able to VPN to all the various clients using the same Parallels setup, which is great because VPN’ing usually quarantines the machine altogether (meaning I can’t reach local LAN resources (printer, common drive, etc.) nor check emails and use IM).

I constantly see new PC based laptops that look promising and at a very affordable price, but I worry about getting into yet another laptop hardware platform (we already have Xtreme, Sony Viaos, Macs, Acers, Asus, and recently IBM Thinkpad) and figuring out where that machine’s gonna get shipped to and what kind of service we’re going to get when we inevitably have to send it off for repairs.

In the ensuing two and half years as I count how many Macs have been sent back for repairs and warranty work, thus putting developers out of commission, I come up with a grand total of 3! And that’s for nearly 50 laptops in active duty, of which approximately 30 are macs. To my knowledge, virtually every PC based laptop but two or three have been returned for repair work. That is HUGE drain on our productivity. Macs are faster to set up and get working. They are easy to back up and fully restore (and clone, too). I changed out my Mac when I upgraded in under 3 hours total. When I was done, there was no difference other than a boost in speed. You can’t touch that with a PC based system.

Most of the developers coming on board these days want a Linux system, so we hand them macs now. Instead of spending untold hours configuring The display and wireless (the two trickiest items left with Linux installs), and involving me and numerous other developers in their effort to get the hardware to fully function with the OS, the new developer quickly adapts to Mac’s BSD-derived OS and are usually done with major configuring within first couple of days.

Macs are also not so prone to virus attacks nor are they loaded down with a whole bunch of crapware. What’s more, when you install applications, they get added to the Application folder and you can easily add them to the dock. Its rare application indeed that also ties itself into the auto-launch to fire up along with the system’s reboot. So what does this mean? Your system doesn’t constantly degrade in performance just by merely installing software!

While Macs are costing a little more upfront, they are showing very solid ROI over the long-term. As a systems administrator, I spend very little time servicing these machines. I’m not a Mac zealot. There’s plenty about Macs that can be improved, but right now, its got the best of breed status all around and definitely worth having in your company’s hardware portfolio.