Category: Ruby on Rails

iTerm crashed on my this morning taking down my development Rails process. Now, I’m busy tailing cullect.com‘s production.log and listening to some podcasts so I didn’t want to restart¹

A little Googling found John Nunemaker’s year-old, “Oops I did it again” post.

His tips worked. An I’m rewriting the following for Future Reference²:

$ ps aux | grep script/server


To get the process id you need to kill

$ kill [PID #]

1. My standard way of restarting Rails in this situation.
2. If memory serves, this was the name of the first inhouse blog I started .

This is an update to earlier post on Performance and MySQL Indexes.

While 10-12 seconds per feed is an improvement, it’s completely unacceptable when we’re talking 2,000+ feeds (full parse would take 5.5 hours) and slow down the system to unusable.

In an attempt to find the bottlenecks, I loaded up the query_analyzer plugin on development and opened up a terminal window to watch the processes in production via prstat -Z¹.

Both showed me the database – not the Rails app itself – was the slow one².

I was able to get the average per feed parse time under 5 seconds by doing two things:

1. re-writing the straight find_by_sql queries to use JOIN ... ON ... rather than the clumsy ...WHERE a.id = b.a_id AND b.id = c.b_id AND etc.... Why not use Rails’ find helpers? I wasn’t able to track down how to :include multiple tables/models
2. Compare the feed’s last-modified date against the last-checked date in the database prior to running it through the parser.

Both of them are ‘duuuuh’ improvements, easily cutting the CPU burden of my database process in a third, often down to barely noticeable.

UPDATE 2 Dec 2007
Continued refactoring now has the average feed parse time around 3 seconds. Yes, that means in some instances I’m seeing 5+ feeds / second.

1. This monitoring process is now my biggest burden. 🙂
2. I’m sure any reasonably experienced Rails developer says this throughout the day.

If you’re a Ruby on Rails developer wanting to keep an eye on what’s happening with your production apps, you already know Capistrano will do it for you:
cap shell
tail -f web/shared/log/production.log¹

Thanks to John Nunemaker‘s comment at anodyne.ca

1. Your path may vary.

I’m at the point in the development cycle where performance is the worth working on, for two reasons; everything else that’s going in is in, it’s too slow.

After one too many out-of-memory errors and far too many times wondering if I should have purchased a larger server to begin with, I sent an SOS message to McClain.

I’ll paraphrase his response, “You don’t have any indexes in your database. Get some.”

I took McClain’s advice and drew up a migration to add indexes for any table with a *_id in it, and a couple heavily-used non _id columns.
add_index :publishings, %w(item_id feed_id)
remove_index :publishings, %w(item_id feed_id)

The results in my feed parsing engine are stunning.
Before indexes: 50-60 seconds / feed
After indexes: 10-12 seconds / feed

This week, I file my first major complaint with Rails.

I was innocently running tests and tweakings some views – when I discovered function that should work longer did.

A little background, my app is filled with routes the make the URLs more contextually appropriate and memorable than the standard REST URL construction. If you have any Rails experience – you can see the problem with using the link_to helper

Rails likes to turn this:
< %= link_to 'Edit', :controller => 'items', :action => 'edit', :id => '1' %>
into this:
/items/1/edit

Now my standard routes are more like this:
/name_of_item_1/edit

If I do something like this, consistent with how my routes are structured:
< %= link_to 'Edit', :controller => 'items', :action => 'edit', :item_name => 'name_of_item_1' %>
link_to doesn’t pick up the item_name. If I throw other variables into the mix, say to pre-pop some attributes, link_to grabs it’s 3 perferred variables and forgets the rest.

Makes me feel like link_to is a lot like scaffolding: great for getting things up and running quickly, but something you want to clear out once you get serious.

So, you’ve set up your Rails app to present different views to authenticated people verses non. Now, how do you test functionality for the authenticated people?

Seems silly to have your tests sign in before checking whatever it is you want them to check.

Well, pass the session data in the test as a hash. Turns out the action tests are formatted like this:
method :action, {action variables hash}, {session variables hash}
Pretty straight-forward, except the session variables need to be passed as ‘strings’, not :symbols.
This works:
post :add_member, {:email => 'test@test.com', :group => '1'}, {'person_id' => '1'}

This doesn’t
post :add_member, {:email => 'test@test.com', :group => '1'}, {:person_id => '1'}

A less than intuitive distinction that I’ve bumped into before with cookies

Also a big thanks to Jon @ Ruby Nuggets for the tip passing referers info in the tests:
@request.env['HTTP_REFERER'] = 'http://foo'
not the @request.referer = ‘http://foo’ that I assumed.

I’ve ignored Rails Tests for too long. In fact, I’ve skipped over Test chapter in Agile Web Development with Rails in both the first and second editions. After fighting with the same bug over and over again, I knew now was the time for tests.

On the surface, tests seemed very un-DRY¹ – like I’m writing the code twice – once to make it work and again to make sure it works. Rails is so good at minimizing dupliation and using scripts to generate things that might be duplicated – shouldn’t the generation of tests be auto-generated². Plus, if I’m continually refactoring code and I have a test for each method – feels like I’d be continually rewriting the test. Argh. I’m too lazy for that.

This morning as Ben helped me fix a previously hidden bugs via a single functional test³ he framed tests in a way that finally clicked: Tests are requirements.

Also known as the Test Driven Development approach and it’s attractive for a number of reasons:

1. Eliminiates separate, external, no fun, and less DRY, documentation
2. Encourages tests to be written early and in English
3. Puts the focus on refactoring the code, not keeping the tests synchronized.

1. Don’t Repeat Yourself, a core programming principle from early Ruby advocates.
2. I know they are when using the scaffolding. I haven’t found scaffolding to be useful for anything other than Wow-ing people in screencasts.
3. We didn’t even need to run the test to identify and fix the bug. But we did. Passed.


def reverse_tinyurl(tinyurl)
  require 'open-uri'
  require 'net/http'

  url = Net::HTTP.get_response(URI.parse(tinyurl)).to_hash['location'].to_s
end

This should also work for any url that is a known redirect whether it be a tinyurl.com, urltea.com, rubyurl.com, or what have you.