Handling the Twitter Stream API with Ruby

I've been doing a lot of things with the Twitter API lately. More on that another time though. I started messing around with their stream API last night. It's pretty fun to play with and I wanted to put the code up somewhere so I can use it later. Right now it works, but it clearly isn't very robust. It needs retry logic and better error handling. Right now, it's just a toy :)

You'll need the json gem installed. Once you have that you can run the script like...
./twitter_stream.rb <username> <password>

Windy City Rails 2009

Yesterday I attended the Windy City Rails 2009 conference. This is the first conference I've attended in a few years. More notably it was my first Ruby related conference. I have to say, I loved the format most of all. The conference was only $99 to attend, there were no tracks and therefor no session overlap. It was a quick one day in and out affair with great speakers and really good material. There were also AM and PM tutorials that did overlap with the conference itself. The tutorials were a deep dive into one particular subject each. They were great for those that wanted to go that route, and though I'm not an expert in the respective subjects, I decided to stick to the conference itself. Also, the tutorials cost extra and I'm a huge cheap ass, so $99 it is :P. There was also a "Coding Dojo". This was an interesting gathering of folks pairing up to work on various challenges. I didn't visit, but it sounded like a neat idea.

Better Ruby through Functional Programming - Dean Wampler

This was the first session of the morning to kick things off. Functional programming is something I battle with. One part of me wants to do more with it, the other part of me gets angry when the examples always devolve into convoluted math examples *yawn*. Dean avoided that (mostly) and covered more realistic topics. Overall the talk was very good, and showed how it is possible to follow a functional style with Ruby. Although, it seemed to me, that attempting to follow functional patterns leads to somewhat noisy and obtrusive Ruby code. Having to freeze things and boilerplate immutable classes seemed to add white noise.

Super-easy PDF Generation with Prawn and Prawn-to - John McCaffrey

Honestly I figured I wouldn't find this talk very interesting, but I was wrong. John was entertaining to listen to. He kept things moving at a fast pace and talked even faster; occasionally trailing off into these mumbles that we couldn't understand. He reminded me of Jim Gaffigan, I expected him to go into a "Hot Pockets" routine. Getting an overview of the different PDF generation tools and then taking a deep dive into Prawn was what kept me listening. Seeing the effect of combining the Google Charts API with Prawn PDF generation was very cool. It turns out my company currently does that and I didn't even know it :)

"Comics" is Hard: On Domains and Databases - Ben Scofield

This was a long road to an interesting topic. Ben broke into biology off the bat and then went into comic books. At first I had no idea where he was going with this. He was entertaining to listen to though, so I kept listening. He kept breaking down the subject to show that those subjects are far more complicated than they first seem. Also this lead into his real point. That these subjects are so complex, they're hard to map to the simple relational model simple SQL databases rely on. He then started into how other persistence mechanisms may hold the key. For example, key/value store systems like Tokyo Tyrant, Voldemort, and Redis. As well as more exotic graph databases like Neo4J.

LUNCH!

I'm making mention of lunch because it was awesome. Marinated steak, shrimp pasta, some kinda chicken rolled up in a noodle. It was all so good!

UI Fundamentals for Programmers - Ryan Singer

This talk had a bit of a buzz at the start. Ryan is, of course, a 37 Signals guy, that makes him a celebrity to the geeks (like me) in attendance. His talk did not disappoint. His initial example of a UI done by a programmer is exactly the kind of UIs I've always built for web applications. All function, no humanity. He talked about how interfaces need text, friendly text, that clues people into what it is they're doing. Fields and labels for every field on your model just doesn't cut it. He also talked a lot about how the eye and brain work together to scan a page and take in information. Providing the right levels of contrast can really make a huge difference.

How to Test Absolutely Anything - Noel Rappin

Noel had a tough job here. He got stuck following Ryan Singer. Noel covered a lot of interesting points on how to test some of the more difficult parts of a rails application. Things like views, email, and timestamps can be difficult to get some good tests around. Interesting material and it was covered well. Unfortunately his slides looked like a website from 1995. This effect was increased by his unfortunate position following Ryan as I mentioned. I think the corny background images and "comets" used for text transitions distracted a lot from the actual words on the screen. Sorry Noel :)

Optimizing Perceived Performance - David Eisinger

One word, hysterical. David's delivery reminded me of Steven Wright. His talk was focused changing the way your interface behaves to give the user the appearance of improved performance. He showed off the power of JQuery to give this effect. He demonstrated several techniques and even uploaded it all to Github. Great talk, I really enjoyed it.

Dojo Retrospectives - Jake Scruggs & Dave Hoover

Like I said at the start, I didn't visit the dojo at all. If I had, I might have gotten more out of this. It seems the organizers knew this would be tough to make it interesting for those that never stuck their head in and kept it to 10 minutes.

Rails 3 Update - Yehuda Katz

This really seemed to be less of a Rails 3 update and more of a Rails retrospective with a bit of a rally cry built in. Yehuda got into Rails 3 at the end of his talk, but by then he was rushing due to time constraints. Don't get me wrong though, what he had to say about Ruby and Rails and the community was incredibly interesting.

Awesomeness Achieved

The Wisdom Group and Chicago Ruby folks really did a great job putting this together. It was well organized, simple, and had a great price. I will definitely attend next year. Oh, and I won a free copy of "The Ruby Way", so I got to get some free exercise lugging home a big ass book :P

Using JRuby and ActiveRecord Migrations to Manage Database Change

Managing change to your application databases is a very common requirement for any development team. The usual approach seems to consist of piling scripts into some "db/changes" folder in your project. This is probably fine for the most part, but when you have multiple databases in multiple environments, managing that across multiple developers can be a problem. If have the usual minimum of three environments (dev, test, prod) you have to remember to apply the right scripts in the right order in each environment. If you are doing iterative development you may be making small database tweaks here and there over a period of a week or two. The difficulty comes in trying to remember what script you applied last and which script you need to apply today.

Following our first instinct of storing sql scripts in source control, let's add another complication into the mix. What if you have to support multiple database engines, say MySQL and Postgres. There will be subtle differences in statements, and database specific properties that need to be set. Managing two sets of scripts becomes a huge pain pretty quickly. Now, as as Java developer, we get to use Hibernate to maintain database portability. Hibernate is a fantastic ORM library, and it comes with tools to generate the DDL needed to create your database from scratch. Unfortunately, it does not include any tools for managing incremental updates to that schema.

The Ruby on Rails folks came up with a solution to the issues above; ActiveRecord Migrations. Migrations are small, incrementally versioned Ruby classes that are designed to bring your database up to date in a controlled fashion. The basic concept is very simple. Store a "version" in the database and maintain a set of Ruby classes that can bring your database up to the latest version. The Ruby classes are numbered using the version they represent and include an "up" and "down" operation. So, let's say we have a current version in the database of "6" and Ruby classes numbered up through "10". When we execute the "migrate" command ActiveRecord will see the current version and see we have scripts that go beyond that and begin to execute their "up" methods in order 7.up, 8.up, 9.up, 10.up. If we wanted to bring our database down to version 8 ActiveRecord would in-turn run the "down" methods. Executing 10.down, 9.down; leaving us at version 8. ActiveRecord Migrations also address our second issue of portability. Your individual migration classes are written in a database agnostic "Domain Specific Language", which allows for the SQL to be generated at runtime for your specific database. Note that you can also execute arbitrary SQL if needed as well.

I intend to show how to do this from the perspective of a Java developer, deploying these changes into a Java deployment environment. This is where JRuby comes in. At my company we already have Java on all of our servers and development machines, but not Ruby (except maybe on dev boxes). Using JRuby allows us to run Ruby on top of the installed JVM without having to worry about installing Ruby. So, let's get crackin' here and do something.

Install JRuby

Download Jruby (1.1.4 as I write this) and extract it somewhere. For me I've dumped it in "~/dev/jruby-1.1.4". Set an environment variable called JRUBY_HOME that points to this location; EX: in .bash_profile: export JRUBY_HOME="~/dev/jruby-1.1.4". Add JRUBY_HOME/bin to your path and you're good to go; EX: in .bash_profile: export PATH=$PATH:$JRUBY_HOME/bin

Install Required Gems

Ruby has a library distribution mechanism called ruby-gems. A gem is a library, every gem has a version and some dependencies. JRuby comes with ruby-gems pre-installed, all you need to do is tell it to install the other libraries you want. We'll need a few of them to get started. Note that I'm going to use mysql via jdbc in this example. You can use the native Ruby drivers for your database of choice, or one of the supported jdbc adapters.

• jruby -S gem install jruby-openssl (we won't use this, but JRuby will complain when installing gems without it)
  
• jruby -S gem install activerecord
  
• jruby -S gem install rake
  
• jruby -S gem install activerecord-jdbcmysql-adapter
  

Since we're using the jdbcmysql adapter we'll need to throw the mysql jdbc driver into our JRUBY_HOME/lib directory.

Configure the Project

Let's start by creating a directory structure to work with. Let's create a folder called "activerecord" with a subfolder named "db" and a subfolder inside "db" named "migrate". Executing "mkdir -p activerecord/db/migrate" will do the trick. In our "activerecord" folder create a file named "database.yml". The .yml file extension means this is a yaml file, Yaml stands for "YAML Ain't Markup Language"; cute right? We're not going to get into yaml much, suffice to say our use of it here is like a properties file. We'll stick to name/value pairs. Our "database.yaml" file will have the following contents...

adapter: jdbc
driver: com.mysql.jdbc.Driver
url: jdbc:mysql://localhost/test
username: root
password: whatever

Note that I am assuming mysql running on the localhost with a database called test. Adjust this file accordingly for whatever your database settings are.

Prepare Your Rakefile

Again, in our "activerecord" folder we're going to create a file named "Rakefile.rb". If you are familiar with Ant then you're familiar with the concept of Rake. Rake is the defacto build tool for Ruby and Rails applications. The Rakefile.rb file is equivalent to the Ant build.xml file, except that it's written in Ruby not XML. As a straight up copy and paste operation, your "Rakefile.rb" will have the following contents...

require "active_record"
require "yaml"

task :default => :usage

task :usage do
  print "Executing \"rake migrate\" will bring the databse up to date\n"
  print "To upgrade/downgrade to a specific version use: \"rake migrate VERSION=X\"\n"
end

task :migrate => :configure do
  ActiveRecord::Migrator.migrate("db/migrate", ENV["VERSION"] ? ENV["VERSION"].to_i : nil )
end

task :configure do
  ActiveRecord::Base.establish_connection(YAML::load(File.open("database.yml")))
  ActiveRecord::Base.logger = Logger.new(File.open('migrate.log', 'a'))
end

If you look at it this file you can see how it is similar to an Ant build.xml file. The "task" blocks (the block is denoted by the do/end) are like "targets" in Ant. Here we've got a task named "migrate" that depends on the "environment" task. The "environment" task sets up our database connection using our Yaml file, and configures a log file to write to. Our migrate task simply tells the ActiveRecord Migrator where the classes are and optionally a specific VERSION to move to. Specifying a version is commonly used for rolling back to some point in the past.

Write Your First Migration

I'll put together the migration classes necessary to build a "user" table, and a "messages" table, with a cross-reference table joining users to messages in a one-to-many relationship. I'll do this in small steps so we can see it as a serious of migration steps. Let's start right off with the User table. In the "db/migrate" folder create a file named "001_create_user_table.rb". The name of this file is important and significant. The first part represents our version, in this case "1". Everything after that is actually ruby standard class file naming. We're defining a class named "CreateUserTable", which therefore means it should be in a file named "create_user_table". Why the swich from camel to case to underscores? I haven't a clue...

class CreateUserTable < ActiveRecord::Migration
def self.up
  create_table :user do |t|
  t.string :username, :null => false, :limit => 15
  t.string :fullname, :null => false, :limit => 30
end
end

def self.down
  drop_table :user
end
end

The migration above is a Class that extends ActiveRecord Migration, it defines two instances methods named "up" and "down". The "up" method calls a method (on Migration) called create_table and passes it a block. When create_table executes the block it will pass it a "TableDefnition" instance, which we named "T". We then call the "string" method on the TableDefinition and pass it "symbols" stating the name and a map (the comma-delimited name/value pairs) containing the options for the column. The "down" method just calls the "drop_table" method. If I have to explain what that does, you're in way over your head :)

OK, let's try it! Bring open a terminal to your "db" folder and run "jruby -S rake migrate". The "-S" I keep using tells Jruby to execute one of command from JRUBY_HOME/bin. If all goes well you should see the following output:

(in C:/source/activerecord)
== 1 CreateUserTable: migrating ===============================================
-- create_table(:user)
-> 0.0031s
-> 0 rows
== 1 CreateUserTable: migrated (0.0036s) ======================================

Once we've executed this migration we should be able to see our new table in mysql:

mysql> show tables;
+-------------------+
| Tables_in_test    |
+-------------------+
| schema_migrations |
| user              |
+-------------------+

The user table is our new table and the schema_migrations table was created by ActiveRecord to track the current version essentially. ActiveRecord actually keeps track of all the migrations applied to it as of ActiveRecord 2.0. If you see a schema_versions, table then you weren't following along very well as you have a really old version of ActiveRecord :)

Migrating the Rest of the Way

W'e're now going to push through our last migration; that message table I mentioned. First create file in "db/migrate" named "002_create_message_table.rb". That file will have the following:

class CreateMessageTable < ActiveRecord::Migration
def self.up
  create_table :message do |t|
  t.references :user
  t.text :message
end
end

def self.down
  drop_table :message
end
end

The new feature we're seeing here in the message table migration is the use of "t.references". Our create_table operation will always create an "id" column for us to represent the surrogate primary key for our table. In doing so ActiveRecord can follow some conventions here. Using "t.references" tells ActiveRecord to setup a foreign key to our user table id column. After that we tell ActiveRecord to create a text (clob) column named "message".

We're ready to execute our next migration. Again, ActiveRecord will look at the versions on our scripts we have provided and compare that with the schema_migrations table to decide what needs executing. Once that decision is made it will walk through them in order...

Executing "jruby -S rake migrate" should give the following output...

(in C:/source/dbupdate/activerecord)
== 2 CreateMessageTable: migrating ============================================
-- create_table(:message)
-> 0.0061s
-> 0 rows
== 2 CreateMessageTable: migrated (0.0085s) ===================================

Now let's have a look at what's in mysql...

mysql> show tables;
+-------------------+
| Tables_in_test    |
+-------------------+
| message           |
| schema_migrations |
| user              |
+-------------------+

We should also see that our schema has been migrated up to version 2 by looking at our schema_migrations table...

mysql> select * from schema_migrations;
+---------+
| version |
+---------+
| 1       |
| 2       |
+---------+
2 rows in set (0.00 sec)

Downgrading

Our Rakefile.rb was looking for a VERSION parameter to read from to force our migration to go to a specific version. Generally speaking you use that to downgrade to a prior version for whatever reason. Let's try that now, execute the "jruby -S rake migrate VERSION=1" command to tell ActiveRecord to step us back to version 1. You should get the following...

== 2 CreateMessageTable: reverting ============================================
-- drop_table(:message)
-> 0.0044s
-> 0 rows
== 2 CreateMessageTable: reverted (0.0067s) ===================================

As you might have guessed ActiveRecord executed our CreateMessageTable.down method, at which point we executed the drop_table method to undo that version. And, again, if we look at the "schema_migrations" table we'll see that it too has been stepped down to version 1.

mysql> select * from schema_migrations;
+---------+
| version |
+---------+
| 1       |
+---------+
1 row in set (0.00 sec)

The Right Tool for the Job

By trade, I am a Java guy. I dabble in Groovy and Ruby and fun stuff like that, but still, I'm paid to be a Java guy. Sometimes you have to know when to use the right tool for the job. Even if you're not familiar with Ruby you really don't need to think of it that way. Ruby is used to provide you with the domain specific language used to handle portable database migrations. You don't need to concern yourself with learning the full extent of the Ruby libraries, or all of the fancy dynamic language features. You just need to pay attention to what values you're passing to these methods and in some cases (maps/hashes) their funky syntax. Honestly I've been looking at doing an article like this for a long time. I'm glad I finally sat down and right it. Hopefully this helps some folks see the power in stepping out from behind our Java comfort zone.

References

• http://rails.aizatto.com/2007/05/27/activerecord-migrations-without-rails/
  
• http://api.rubyonrails.org/classes/ActiveRecord/Migration.html
  
• http://www.scribd.com/doc/1083725/Migrations-in-Rails-20

If you'd like to see what a large sampling of ActiveRecord migrations take a look at the ones in Trisano.

Ruby on Rails Stuff

I've found a few interesting resources for Ruby on Rails stuff and I figure I'll link 'em up here.

First up, Railscasts.com; a collection of video screencasts on various small Rails tricks.
Railscasts.com

Next, a presentation from Dave Thomas at QCon about meta-programming in Ruby.
MetaProgramming - Extending Ruby for Fun and Profit

Rails 2.0 just came out, here's a post from DHH about some of the new features.
Rails 2.0: Preview Release

Also, if you don't own a Mac and therefore don't have that sexy Textmate editor, you might consider NetBeans 6.0. Netbeans used to be a joke of an IDE for Java but it's really come along nicely for Java and has really excelled in it's support for Ruby and Ruby on rails.
NetBeans: Ruby Developer's New Best Friend (Part 1)
NetBeans: Ruby Developer's New Best Friend (Part 2)

One App, Multiple Architectures

I've got my eye on a few different technologies these days. I figure it might be fun to write one application in each of them and see how it goes. I intend to write a basic wine cellar application (gotta scratch my itch) using a few different technology stacks.

First, I'll go with my usual set. Java with Spring 2.0, Spring MVC, Spring Webflow, and Hibernate. I may not finish this as it will probably bore me, but we'll see. Maybe I'll do this as a 100% Webflow app; who knows.

Second, I'll play with something I've been dying to try for a bit now. This round will be Java with Wicket, Spring 2.0 and Hibernate. Spring and Hibernate are like peanut butter and jelly to me, they are just meant for eachother. Writing a java app without them these days is just masochism. I'm dying to try out Wicket for the MVC layer, I love the vision they've come up with.

The third iteration will be Ruby on Rails of course. This will probably take me like 2 hours to do what the previous two iterations will take me 2 days to do. Really, it will take me longer as I'm not as fluent in Ruby as I am in Java. I love the database conventions set forth in RoR and will be following those throughout this little exercise.

I may or may not make a pass at Seam. I'm on the fence as to whether or not I like it. I can't pass judgment yet as I haven't spent enough time looking into it.

I figure this will be an interesting thing to do. It gives me something to do with these technologies. I've always said that programming is like painting. You can't just paint something, you need inspiration. Same with programming, you can't just "write an app". You have to have a purpose. Hell, I may never get past the first app with all the stuff I got going on right now anyway (still gotta work on that OpenId support for Acegi).

We'll see how it goes, and I'll post little tidbits of what I find along the way. Maybe I'll throw all the code up at googlecode when I'm done, who knows.

Buildr

Nope, that's not a typo in the title...

Buildr is a new java application build tool. The fun part is that it's written in Ruby and based on Rake. Rake is an extremely powerful build tool for Ruby applications. There have been a few attempts at adapting it to building Java applications. Most of those attempts were based on the idea of pretending that Java app is really a Ruby app. That principal packs the tool full of workarounds and hacks. Buildr isn't like that at all.

Buildr ties together a few components to build java applications; knowing full well they are java applications. It embraces all of the good ideas behind Maven, without the main bad part: Maven itself. Maven can be very powerful, but that is overridden by Maven being amazingly difficult and unpredictable. The good parts of Maven are the conventions and guidelines around the project directory structures and dependency management via repositories.

I originally discovered buildr through a post at Tim Bray's Ongoing blog. In that post he links to an article by Assaf Arkin. Assaf is one of the folks behind Buildr. Assaf describes his pain with Maven and how they solved it with buildr.

Buildr is still very limited in scope (that's a good thing right now). It doesn't appear to have means of running cobertura for unit test coverage yet. Buildr also lacks an idea task for building Intellij IDEA module files, though it does have an eclipse task. You can see were Assaf's loyalties lie on that one! I'm toying around with writing the idea task myself, but I've got a lot of learning to do in the Rake department. Buildr also doesn't do any of the site generation stuff that Maven can do. This can easily be fixed with a few rhtml templates and ERB.

I toyed with Buildr this weekend for a little bit and was able to get the code for Fizzle (more on that later) to build and publish very easily. You can build multi-component projects with only a single RakeFile at the head of the project. Maven requires a pom.xml in every module. Each repeating various inter-module dependency names and versions all over (bleh). Multi-component stuff is much imporved in Maven2 vs. Maven1, but still hard to approach.

Right off the bat I fell in love with Buildr over Maven for one very simple trick.
The following tests failed:
com.googlecode.fizzle.command.DefaultCommandProviderManagerTest
com.googlecode.fizzle.command.engine.DefaultCommandExecutionEngineTest

Come on Maven, how hard is that? I really don't want to scroll back through 1000+ lines of unit (in Acegi's case) test output to see which one says FAILED. That's ridiculous. I was so inspired by this I stopped everything I was doing to mention it to the buildr folks.

As I said above, Buildr is very light right now. I think overall it has a chance to really be great if it gets enough love. I wonder if we can move Acegi to it, that might be fun :P

Check out Buildr, and discuss it on their google-group.

IDEs for Ruby

A nice comparison sheet of the various Ruby posted over at O'Reilly Ruby.
http://www.oreillynet.com/ruby/blog/2007/03/ides_for_ruby.html

This sheet is actually from another blog: "The Nameless One"

It is funny, to me at least, is that the O'Reilly post is written by Curt Hibbs. Curt is one of the principle developers on the FreeRIDE Ruby IDE; he mentions that in his post. The interesting part is that the comparison did not include FreeRIDE. This is for a good reason. FreeRIDE is a joke. FreeRIDE really stands to make Ruby look bad to people being introduced to it for the first time. FreeRIDE is unstable at best, bordering on unusable.

A first timer on a Windows box is probably going to download the one-click installer to install Ruby. This installer comes with FreeRIDE and will put shortcuts out for it. The afore mentioned first-timer will launch it and try to play with it. It will be slow, clunky and featureless; then it will crash. With an immediate "wow, well screw that" from the newbie.

It sort of reminds me of much of the worlds view of Java due to all the shitty Applets and Swing/Awt apps out there.

If you're looking for a good 'free' rails IDE check out RadRails, if you're a Java guy with IntelliJ installed, definitely check out the Rails plugin.