There is much buzz about redis lately, and most of it with just cause, redis is a blazing fast single threaded data store, that is simple to shard when dealing with large datasets, and can be easily replicated when reads are a problem.
Much of the more advanced code I’ve seen, including some of my own makes good use of the redis set data-type and the intersection and union operators to facilitate filtering.
Recently I had the chance to experiment with trying to port a huge dataset to redis, which resulted in a possible edge case where redis would have to calculate the intersection of three ridiculously large sets (700K items), this caused a freeze of about four seconds and during that time no other operations were possible, in a production environment, locking of this nature could cause catastrophic results and has pushed back any plans of an all redis backend.
This is a problem that cant really be solved by the existing scaling mechanisms redis provides, and it is problem that could affect any code that relies on automatically adding values to a set, especially many of the auto completion and search solutions I’ve seen presented.
Recently I was fortunate enough to be able to speak to Rob and Adam from playnice.ly at the recent Redis London meetup, and thrash out a few ideas on the problem.
The solutions above are a good starting point, but far from fully formed, and this is an area which deserves far more attention, I’m hoping that by putting the word out there, that the redis community can start to move forward towards concrete solutions and that the code we are producing doesn’t come back to haunt us in the future.
© 2010-2012 Daniel Knell