Category: cullect

In part 1 and part 2, I laid out my initial approaches on caching and performance in Cullect.

While both of them pointed in the right direction, I realized I was caching the wrong stuff in the wrong way.

Since then, I tried replicating the database – one db for writes, one for reads. Unfortunately, they got terribly out of sync just making things worse. I turned off the 2nd database and replaced it with another pack of mongrels (a much more satisfying use of the server anyway).

Cullect has 2 very database intensive processes: grabbing the items within a given reading list (of which calculating ‘importance’ is the most intensive) and parsing the feeds.

Both cause problems for the opposite reasons – the former is read and sort intensive while the latter is write intensive. Both can grind the website itself to a halt.

Over the last couple weeks, I moved all the intensive tasks to a queue processed by Delayed_Job and I’m caching the reading lists’ items in database table – rather than memcache.

Yes, this means every request is pulled from the cache, and a ‘update reading list’ job is put into the queue.

So far, this ‘stale while update’ approach is working far better than the previous approaches.

BTW, as this practice confirms, the easiest way to increase the performance of your Ruby-based app is to give your database more resources.

After a couple very rough weeks – I’m happy with where Cullect and it’s caching strategy is. It’s slightly different from where I talked about last. I’ve also added a slave DB to the mix since my last write up. Overall, it feels more solid, and is performing at or better than before

In my part 1, I laid out my initial approach on caching in Cullect.

It had some obvious deficiencies;

1. This approach really only sped up the latest 20 (or so items). Fine if those items don’t change frequently (i.e. /important vs /latest) or you only want the first 20 items (not the second 20),
2. The hardest, most expensive database queries weren’t being cached effectively. (um, yes that’s kinda the purpose).

I just launched a second approach. It dramatically simplified, cache key (6 attributes down from 10) and rather than caching entire the items in that key, I just stored the pointer object to them.

Unfortunately, even the collection of pointer objects was too big to store in the cache, so I tried a combination of putting a LIMIT on the database query and trying to store 20 items a time in a different cache object.
This second approach had the additional problem of continually presenting hidden/removed items ( there’s 2 layers of caching that need to be updated).

Neither was a satisfactory performance improvement.

I’ve just launched a solution I’m pretty happy with and seems to be working (the cache store is updating as I write this).

Each Cullect.com reading list has 4 primary caches – important, latest, recommended, hidden – with a variants for filters in keyword searches. Each of these primary caches is a string containing the IDs of all the items in that view. Not the items themselves, or any derivative objects – both of those take up too much space.

When items are hidden, they’re removed from the appropriate cache strings as well.

Murphy willing, there won’t be a part 3. 😉

1. Storing the items in the cache as objects