Thursday, 15 November 2007

Rails Cheap, MySQL Expensive

This is an update to earlier post on Performance and MySQL Indexes.

While 10-12 seconds per feed is an improvement, it’s completely unacceptable when we’re talking 2,000+ feeds (full parse would take 5.5 hours) and slow down the system to unusable.

In an attempt to find the bottlenecks, I loaded up the query_analyzer plugin on development and opened up a terminal window to watch the processes in production via prstat -Z1.

Both showed me the database – not the Rails app itself – was the slow one2.

I was able to get the average per feed parse time under 5 seconds by doing two things:

  1. re-writing the straight find_by_sql queries to use JOIN ... ON ... rather than the clumsy ...WHERE a.id = b.a_id AND b.id = c.b_id AND etc.... Why not use Rails’ find helpers? I wasn’t able to track down how to :include multiple tables/models
  2. Compare the feed’s last-modified date against the last-checked date in the database prior to running it through the parser.

Both of them are ‘duuuuh’ improvements, easily cutting the CPU burden of my database process in a third, often down to barely noticeable.

UPDATE 2 Dec 2007
Continued refactoring now has the average feed parse time around 3 seconds. Yes, that means in some instances I’m seeing 5+ feeds / second.

1. This monitoring process is now my biggest burden. 🙂
2. I’m sure any reasonably experienced Rails developer says this throughout the day.

Thursday, 8 November 2007

Tuesday, 6 November 2007

The Performance Power of MySQL Index

I’m at the point in the development cycle where performance is the worth working on, for two reasons; everything else that’s going in is in, it’s too slow.

After one too many out-of-memory errors and far too many times wondering if I should have purchased a larger server to begin with, I sent an SOS message to McClain.

I’ll paraphrase his response, “You don’t have any indexes in your database. Get some.”

I took McClain’s advice and drew up a migration to add indexes for any table with a *_id in it, and a couple heavily-used non _id columns.
add_index :publishings, %w(item_id feed_id)
remove_index :publishings, %w(item_id feed_id)

The results in my feed parsing engine are stunning.
Before indexes: 50-60 seconds / feed
After indexes: 10-12 seconds / feed

Wednesday, 24 October 2007

REST-less with link_to

This week, I file my first major complaint with Rails.

I was innocently running tests and tweakings some views – when I discovered function that should work longer did.

A little background, my app is filled with routes the make the URLs more contextually appropriate and memorable than the standard REST URL construction. If you have any Rails experience – you can see the problem with using the link_to helper

Rails likes to turn this:
< %= link_to 'Edit', :controller => 'items', :action => 'edit', :id => '1' %>
into this:
/items/1/edit

Now my standard routes are more like this:
/name_of_item_1/edit

If I do something like this, consistent with how my routes are structured:
< %= link_to 'Edit', :controller => 'items', :action => 'edit', :item_name => 'name_of_item_1' %>
link_to doesn’t pick up the item_name. If I throw other variables into the mix, say to pre-pop some attributes, link_to grabs it’s 3 perferred variables and forgets the rest.

Makes me feel like link_to is a lot like scaffolding: great for getting things up and running quickly, but something you want to clear out once you get serious.

Friday, 19 October 2007

Passing Sessions and Referers in Rails Functional Tests

So, you’ve set up your Rails app to present different views to authenticated people verses non. Now, how do you test functionality for the authenticated people?

Seems silly to have your tests sign in before checking whatever it is you want them to check.

Well, pass the session data in the test as a hash. Turns out the action tests are formatted like this:
method :action, {action variables hash}, {session variables hash}
Pretty straight-forward, except the session variables need to be passed as ‘strings’, not :symbols.
This works:
post :add_member, {:email => 'test@test.com', :group => '1'}, {'person_id' => '1'}

This doesn’t
post :add_member, {:email => 'test@test.com', :group => '1'}, {:person_id => '1'}

A less than intuitive distinction that I’ve bumped into before with cookies

Also a big thanks to Jon @ Ruby Nuggets for the tip passing referers info in the tests:
@request.env['HTTP_REFERER'] = 'http://foo'
not the @request.referer = ‘http://foo’ that I assumed.

Thursday, 18 October 2007

MySQL on OS X Reinstall Reminders

First off, if you can avoid reinstalling MySQL on OS X – by all means do. Whatever the reason is that you want to do a reinstall, it’s not a good one1. Just buck up and work around whatever issue you have. It’s not as bad as what follows.

Let’s say you decide to reinstall anyway.

Since there’s no easy way to uninstall, you go on a crazy rabbit hunt – including showing all the hidden files2 – through your system to eradicate all instances of MySQL and any dbs (bu-bye data). Then, you downloaded and installed a fresh binary from dev.mysql.com. Now, there’s no reason to do 3 or 4 system restarts, but you haven’t listened to me this far.

Weeee.

Now, I’m pretty sure that if you go back into your Rails app and run rake db:migrate you’ll get the following:
dyld: Library not loaded: /usr/local/mysql/lib/mysql/libmysqlclient.15.dylib

Thank Peter Morris for reminding you to run:
sudo install_name_tool -change /usr/local/mysql/lib/mysql/libmysqlclient.15.dylib /usr/local/mysql/lib/libmysqlclient.15.dylib /usr/local/lib/ruby/gems/1.8/gems/mysql-2.7/lib/mysql.bundle

Hey, not so fast, make sure you set up the same root pass you had before:
/usr/local/mysql/bin/mysqladmin -u root password new_password_here

1. My reason – I was intermittently getting "Lost connection to MySQL server during query" when running rake test. It seems better now. There had to be
2. defaults write com.apple.finder AppleShowAllFiles ON

Saturday, 13 October 2007

Framing Rails Tests: Tests are Requirements

I’ve ignored Rails Tests for too long. In fact, I’ve skipped over Test chapter in Agile Web Development with Rails in both the first and second editions. After fighting with the same bug over and over again, I knew now was the time for tests.

On the surface, tests seemed very un-DRY1 – like I’m writing the code twice – once to make it work and again to make sure it works. Rails is so good at minimizing dupliation and using scripts to generate things that might be duplicated – shouldn’t the generation of tests be auto-generated2. Plus, if I’m continually refactoring code and I have a test for each method – feels like I’d be continually rewriting the test. Argh. I’m too lazy for that.

This morning as Ben helped me fix a previously hidden bugs via a single functional test3 he framed tests in a way that finally clicked: Tests are requirements.

Also known as the Test Driven Development approach and it’s attractive for a number of reasons:

  1. Eliminiates separate, external, no fun, and less DRY, documentation
  2. Encourages tests to be written early and in English
  3. Puts the focus on refactoring the code, not keeping the tests synchronized.

1. Don’t Repeat Yourself, a core programming principle from early Ruby advocates.
2. I know they are when using the scaffolding. I haven’t found scaffolding to be useful for anything other than Wow-ing people in screencasts.
3. We didn’t even need to run the test to identify and fix the bug. But we did. Passed.

Sunday, 16 September 2007

The Power of Vendor/Gems

Right now, I’m heavily reliant on an unsupported (if not completely abandoned) ruby gem library. Historically, I’ve just gem install bizarro-gem and moved on.

A couple of issues have changed my perspective:

  • My host, textdrive, doesn’t allow installing bizarro gems on their shared servers and I’ve had difficulty freezing them, so I worked around the desired functionality.
  • I needed to make some modification to the library. Not easy to do if it’s installed system wide.

Thankfully, I found Chris Wanstrath’s Vendor Everything , his post makes it trivial to freeze gems in a reliable, testable, hackable way.

The core of Chris’s technique is added this line to your Rails::Initializer.run block:
config.load_paths += Dir["#{RAILS_ROOT}/vendor/gems/**"].map do |dir|
File.directory?(lib = "#{dir}/lib") ? lib : dir
end

And unpack the required gems into vendor/gems

With that, I was able to make the necessary modifications and make a more portable app.

Thanks Chris.

Wednesday, 22 August 2007

Regrets: Not Getting a CS Degree

“My advice to young people is to get a computer science degree, if for no other reason than you can avoid those odd jobs and get right to the programming.” – Brent Simmons

I’m continually bumping up against my ignorace of good programming practices that one would get in Comp. Sci. 101. Not that it stops me, just makes the time from idea to reality much longer than I’d prefer – especially for tiny things.

On the flip side, I’m suspect if I got a CS degree instead of a Graphic Design degree, I’d probably be fighting with my ignorance of visual composition, color theory, typography, and interface design.

Seem to me, taking a couple Comp Sci classes in college not only helps you “find out if you were, in fact, born a programmer”, it makes you less of a user.

More of a doer.

Elsewhere:
Phil Crissman talks about the differences between software development and Computer Science degrees

Saturday, 18 August 2007

How to Decode TinyURLs with Ruby


def reverse_tinyurl(tinyurl)
  require 'open-uri'
  require 'net/http'


  url = Net::HTTP.get_response(URI.parse(tinyurl)).to_hash['location'].to_s
end

This should also work for any url that is a known redirect whether it be a tinyurl.com, urltea.com, rubyurl.com, or what have you.