• Log in

• Home
• About Me

Ramaze vs. Padrino Benchmarks

Posted by Michael on May 1, 2010 in General

I have been developing all of my web-based projects in Ramaze since January 2009 when I switched from Rails to Ramaze. At that time, I did some benchmark comparisons. The other day, I heard about Padrino, the micro-framework for Sinatra, which is comparable to the Ramaze micro-framework for Innate. What caught my eye was the benchmark tests the Padrino folks had run that put Ramaze at the bottom of the stack. I thought to myself, “no way!” and set out to run my own benchmarks comparing Padrino to Ramaze as I know at the very least, Ramaze quite outshines Rails’ performance. Something ain’t quite right.

What was tested?

There are two things I have grown to love since switching from Rails to Ramaze and that’s Sequel as the object relational mapper (ORM) and Erubis as the templating engine. So these are exactly the two components you will find in both of these tests. Fortunately, both frameworks make it exceedingly easy to utilize these components whereas I had to jump through a few loops to successfully test ActiveRecord in Ramaze against Rails then-only choice as an ORM.

The main goal in this benchmarking effort is to compare apples to apples and I tried to control for that at every level in the stack from deployment method and worker threads to as close as possible code and computations in rendering each page. Although the applications remain ridiculously simple, they do test a variety of things most developers would do in a typical app in terms of configuring a sane rendering engine and layout that pulls content from an action’s template. Some miscellaneous computations were done for each page such as globally set titles and searching for “posts” to render.

All project sources and the data collected and reported herein is available in the 2010 folder on github.com. I am not entirely sure what the Padrino folks did to control for variables in their benchmarking tests, but here’s what I did:

1. Same hardware: Macbook Pro 2.33 GHz Intel Core 2 Duo with 4GB RAM, OS X 10.5.8.
2. Deployed via Phusion Passenger 2.2.11 on Apache 2.2.14 with six workers running both in live/production mode (using each framework’s default for these environments).
3. Same Ruby MRI: ruby 1.8.7 (2009-06-12 patchlevel 174) [i686-darwin9] for both.
4. Sequel as the ORM, Erubis as the rendering engine.
5. Nearly identical page and layout templates (varying only by the name “Ramaze” and “Padrino” in the title tags).
6. A representive set of identical computations in each project (current time, globally set title, helpers that generate links, etc.
7. Identical sqlite3 data file with identical model declaration and querying through Sequel.

I utilized this simple call to Apache Bench to run through the two test scenarios:
All data collected was based on 10,000 requests at 100 concurrent users after an initial “warmup” of the passenger threads with start of 100 requests at 10 concurrent users. The following command shows the basic set of parameters issued to Apache Bench:

ab -n 10000 -c 100 http://padrino.macdoze.local/

The Results

The following table tabulates the results gathered from Apache Bench as well as passenger-memory-stat. I ran a short run to warm things up (100 requests, 10 concurrently). And then ran the big bench at 10,000 requests, 100 concurrently, and then ran again and recorded the results. At around 5,000 requests, I ran passenger-memory-stat to collect the data during mid-processing. Swap space was never triggered (0k utilization) during these tests, and CPU utilization was roughly 75% during the test runs.

Throughput

Everybody likes to talk about throughput, so lets start there. As you can see, Padrino processed the requests 1.34 times faster than Ramaze. In other words, Padrino is pushing bytes at about 25% faster than Ramaze is. Be forewarned that throughput doesn’t necessarily equate to user response times. This metric simply tells you the load at which your system is capable of delivering content at.

Time per Request

Time per request helps you see about how long an average user request takes for a given load. As you can see here, with times in the milliseconds, we’re talking about numbers that will easily be swallowed in the noise of network latency. Even so, we are looking at a similar performance ratio as with the throughputs above.

Memory Consumption

This final graph shows a surprising difference in memory consumption between the two micro-frameworks. Here, Ramaze trumps Padrino’s memory footprint, utilizing only 63% of the memory that Padrino requires. This means that you can run more workers per server using Ramaze than you can with Padrino before you consume all the RAM on the server.

Conclusion

Padrino has certainly set the new standard for raw throughput performance in Ruby-based web frameworks, but don’t be deceived into thinking its blowing its competition away by leaps and bounds. If you’re one of those shops that is fortunate to have an infinite supply of hardware to throw at the problem, then Padrino makes a good choice for maximizing throughput per server. I know I will certainly strongly consider utilizing Padrino in a couple of up-coming projects.

I have found that one area Padrino is shining where Ramaze is not quite there yet, is an excellent website with just the right information to get started developing new projects. I was really impressed with Sinatra’s approach to named routes with the RESTful inspired get/post/put/delete DSL that is further enhanced by Padrino’s helpers. The recent Padrino enhancements to their router helpers for “mounting” more than one application is a bit more intuitive to me, especially with the ample documentation on just how to configure for all the most common routing scenarios than Ramaze’s. I still have not quite implemented mulitiple mini-apps under Ramaze despite multiple attempts.

Ramaze has been around long enough that it has a sizable body of legacy/obsolete documentation floating around the ‘net that you need to wade through before finding the right documentation that’s relevant to the post Ramaze/Innate split/re-architecture. The documentation’s certainly there for Ramaze, but if you don’t know exactly where to look, or your Google-fu is suffering during your time of need, the only place to find it is by resorting to reading the source or hoping someone on the #Ramaze IRC readily finds it on your behalf (which they’re pretty darn good at doing).

Padrino also differs itself from Ramaze and following the Rails camp a little more by introducing an opinionated project structure and set of helpers that is well-documented and easy to find. Ramaze, on the other hand, strives hard not to push opinions onto the developers on how they choose to structure their projects. Because of Padrino’s choices, they’re also able to provide a nice set of well-rounded generators and rake tasks to give you a much nicer “out of box experience” especially suitable for the Rubyist that is still learning his way around the Ruby language. I would love to see the Ramaze community come together to generate an equally strong set of documentation and best-practices rolled into a few good generators to help the aspiring developer get off the ground with Ramaze a lot faster from the get-go.

Both Ramaze and Padrino have strong technical merits, and as you can see from the benchmarks above, both are quite performant platforms in their own right with Padrino having the edge in speed and Ramaze having the edge in memory consumption. For developers like me churning out many different small websites for various customers and running all of those websites off one physical box, Ramaze continues fill my needs in that corner with its smaller memory consumption footprint. 1.3 microseconds for a collection of customers registering 25 to 100 page views a day each isn’t enough to of a difference to concern myself with a wholesale switch in platforms. However, I definitely see a bright future for Padrino in my toolbox as I endeavor to build collections of reusable mini-apps that can be mounted in many easily configurable combinations for each deployment.

Copyright 2008 Michael Lang. All rights reserved.

• Home
• About Me
• Entries RSS
• Comments RSS