Introducing migr8 a Concurrent Redis Migration Utility Written in Go
Here at Reverb, we’ve got quite a few places that we like to store our data. One of those places is Redis. We use Redis in quite a few ways including our job queues for Sidekiq and our analytics tracking for our internal service called Bump.
As a scrappy startup we thought to ourselves “oh one redis instance should be just fine forever and ever”…until it wasn’t. Earlier this year we started looking at our rate of growth in our redis keyspace and noticed we were quickly running out of memory. We knew something had to be done.
We came up with a plan: split Bump out into its own Redis instance. With this plan in mind we started looking to see if anybody else has solved this problem before us. We stumbled upon this script which was the initial inspiration for our tool Migr8. One of the first problems we noted about this script is its use of “keys *”.
Running keys * is a pretty bad idea if you’ve got a decent sized data set in Redis. This command is fine to run in development or staging but please heed our warnings(we’ve made the mistake) do not run “keys *” in production. Your Redis instance will lock while trying to process the command and will likely fail in the process. Redis locks on “keys *” because it is O(n) with the number of keys. If you have a significant number of keys, lock city awaits your arrival.
Luckily we ran the command on a slave so we just had to resync the slave with the master. You’ve been warned :)
So after a lot of toying around with different implementations in Ruby, we decided to give it a shot writing a tool in Golang. Our initial Ruby implementations were processing keys at a rate of 100 keys per second. Ruby is slow because of the GIL (Global Interpreter Lock). This makes Ruby a poor choice if we want fast / concurrent code. Go has native concurrency built into the language. The Go implementation made our network card the bottleneck at 20k keys per second. Yeah, wow. Go is pretty fast.
At the time we had to move around 40 million keys from our main Redis instance to this new bump instance. If we were to stick with the Ruby implementation, it would have taken us 100 hours to migrate keys from one instance to the other. This is way too long.
Using the migr8 utility, we were able to complete the migration in 30~ minutes. Now that’s a much more acceptable number in terms of downtime.
Here’s some quick examples of how to use the Migr8 utility:
Using Go for this tool was a huge win over Ruby. So now we’d like to share the tool with you in hopes that it helps you move some Redis.
Til next time,
