Brian Aker had mentioned to me at one time that TokyoTyrant was way better than memcachedb and we should run it instead. I took notice and it turns out he’s right! It does basically the same thing, applying the memcache protocol to an on disk key/value store. However, the code is incredibly clean, well maintained, and runs extremely fast. There’s also a lot more flexibility, with the ability to choose between in-memory or on disk storage, hash tables, B+Tree’s, etc.

The availability of log based asynchronous master/master replication (somewhat similar to MySQL’s replication in concept) was probably one of the biggest wins, allowing much simpler failover (just move the IP, or DNS, or whatever) when compared to MemcacheDB’s adherence to BerkeleyDB’s replication setup, which is a single-master system implementing an election algorithm.

Somewhere during migration, we missed one tiny detail though. Sometimes, the devil is in the details. This is really the only evidence in the documentation that tokyo tyrant has support for the memcache protocol. It is very clear:

Memcached Compatible Protocol

As for the memcached (ASCII) compatible protocol, the server implements the following commands; “set”, “add”, “replace”, “get”, “delete”, “incr”, “decr”, “stats”, “flush_all”, “version”, and “quit”. “noreply” options of update commands are also supported. However, “flags”, “exptime”, and “cas unique” parameters are ignored.

Now, as I said, there’s nothing ambiguous about this. That would have helped, if anyone on my team had ever read it. We installed TokyoTyrant, pointed our basic test code at it, and it worked. This is really a process problem, not so much a technical one. The process must be to assume it won’t work, and test all the different use cases to make sure it works.

Now, why is that bit of the manual important? Well we use PHP. Specifically, we use the PECL “Memcache” module to access memcache protocol storage. Now, the Memcache module is mostly oriented toward caching in the memory based original memcached. It works great for memcachedb too, which simply ignores the exptime parameter. However, memcacheDB *does not* ignore “flags”.

And therein lies the problem. Users of the PECL Memcache module may not know this, but the flags are *important*. There are two bits in that flags field that the Memcache module may set. Bit 0 is used to indicate whether or not the content has been serialized, and, therefore, on read, must be unserialized. Bit 1 is used to indicate whether or not the content has been gzipped.

So, while all of the strings that were stored in MemcacheDB and subsequently copied to TokyoTyrant worked great, the serialized objects, arrays, and gzipped values, were completely inoperative, as they were coming back to the code as strings and binary compressed data. The gzipped data was easy (turn off automatic gzip compression). The serialized data took some quick tap dancing to remedy, with code something like this:


class Memcache_BrokenFlags extends Memcache
{
public function get($key, &$flags)
{
$v = parent::get($key, $flags);
$uv = @unserialize($v);
return $uv === false ? $v : $uv;
}
}


Luckily our code all uses one Factory method to spawn all “MemcacheDB” connections, so it was easy to substitute this in.

Eventually we can just change the code by segregating into things that always serialize, and things that don’t, and just do the serialization ourselves. This should eventually allow us to use the new tokyo_tyrant module in PECL, which only reliably stores scalars (I noticed recent versions have added a call to the internal PHP function convert_to_string().. this is, I think, a mistake, but one that still leaves it up the programmer to explicitly serialize when serialization is desired).

This was a pretty big gotchya, and one that illustrates that even though sometimes us cowboy coders and sysadmins get annoyed when those pesky business people ask us for plans, schedules, expected impact, etc., and we keep assuring them we know whats up, its still important to actually know whats up, and make sure to RTFMC .. C as in, CAREFULLY.

As we navigated from the Santa Clara conference center to Mountain View’s quaint downtown, Brian, Patrick, and I were discussing memcached stuff. I mentioned my idea, and subsequent implementation of the Mogile+Memcached method for storing data more reliably in memcached. I knew in my head why we had chosen to read from all of the replica servers, not just the first one that worked, but I forgot (The reason, btw, is that if one of the servers had missed a write for some reason, you might get out-of-date data). I guess I was a little overwhelmed by Brian’s mountain of experience w/ memcached.

Anyway, the next thing I mentioned was that we had also tried MemcacheDB with some success. Brian wasn’t exactly impressed with MemcacheDB, and immediately suggested that we should be using Tokyo Tyrant instead. I had heard of Tokyo Cabinet, the new hotness in local key/value storage and retrieval, but what is this Tyrant you speak of?

I’ve been playing with Tokyo Tyrant ever since, and advocating for its usage at Adicio. Its pretty impressive. In addition to speaking memcached protocol, it apparently speaks HTTP/WEBDAV too. The ability to select hash, btree, and a host of other options is nice, though I’m sure some of these are available as obscure options to berkeleydb as well.

Anyway, I was curious what performance was like, so I did some tests on my little Xen instance, and came up with pretty graphs.

I used the excellent Brutis tool to run these benchmarks using the most interesting platform for me at the moment.. which would be, php with the pecl Memcache module.

These numbers were specifically focused on usage that is typical to MemcacheDB. A wide range of keys (in this case, 10000 is “wide” since the testing system is very small), not-small items (2k or so), and lower write:read ratio (1:50). I had the tests restart each daemon after each run, and these numbers are the results of the average of 3 runs each test.

I also tried these from another xen instance on the same LAN, and things got a lot slower. Not really sure why as latency is in the sub-millisecond range.. but maybe Xen’s networking just isn’t very fast. Either way, the numbers for each combination didn’t change much.

What I find interesting is that memachedb in no-sync mode actually went faster than memached. Of course, in nosync mode, memcachedb is just throwing data at the disk. It doesn’t have to maintain LRU or slabs or anything.

Tokyo Tyrant was very consistent, and used *very* little RAM in all instances. I do recall reading that it compresses data. Maybe thats a default? Anyway, tokyo tyrant also was the most CPU hungry of the bunch, so I have to assume having more cores might have resulted in much better results.

I’d like to get together a set of 3 or 4 machines to test multiple client threads, and replication as well. Will post that as part 2 when I pull it together. For now, it looks like.

In case anybody wants to repeat these tests, I’ve included the results, and the scripts used to generate them in this tarball.

– Additional info, 6/4/2009
Another graph that some might find interesting, is this one detailing CPU usage. During all the tests, brutis used about 60% of the CPU available on the machine, so 40% is really 100%:

This tells me that the CPU was the limiting factor for Tokyo Tyrant, and with a multi-core machine, we should see huge speed improvements. Stay tuned for those tests!

Its not entirely clear from the documentation how to setup memcachedb for fault tolerance. Here’s the procedures I’ve found useful.

• Set up replication right. With all due respect to Steve Chu, The docs aren’t really clear on how to setup replication. Its much simpler than it looks. Just run MemcacheDB as you would if it were standalone, but then add a combination of these 3 options:
  • You must have a -R line if you want to participate in replication. This is your hostname and port that listens for connections from other machines for replication. It is the same value that should be listed in every other machine’s -O.
  • a -O for *every* other machine that may want to replicate to/from this machine. I am sure there are situations where you won’t need these, but it makes re-syncing and elections more predictable. You won’t be able to re-sync “live” after a failure without -O options.
  • -M/-S are not required. If you start n machines without -M or -S, but with appropriate -R and -O lines, they will arbitrarily elect a master. If you run them with -M and -S, then the -M box will just be pushy and always elect itself the master, and the -S boxes will, likewise, always try to defer to slave status.
  • Lets say we wanted to listen for memcache protocol on port 45000 on host ‘node1′ and replicate to ‘node2′
  • Standalone: memcachedb -p 45000 -H /home/memdb/data -u memdb -N
  • Replication w/ elected master: memcachedb -p 45000 -H /home/memdb/data -u memdb -N -R node1:46000 -O node2:46000
  • Replication Master: memcachedb -p 45000 -H /home/memdb/data -u memdb -N -R node1:46000 -O node2:46000 -M
• Only the current master can accept writes. You can see which machine is the master with the ‘stats rep’ command. In v1.2.1 its shown as an environment id. Below st_env_id and st_master are the same, so this is the master:
  
stats rep
STAT st_bulk_fills 0
STAT st_bulk_overflows 0
STAT st_bulk_records 11
STAT st_bulk_transfers 3
STAT st_client_rerequests 0
STAT st_client_svc_miss 0
STAT st_client_svc_req 0
STAT st_dupmasters 0
STAT st_egen 3
STAT st_election_cur_winner 2147483647
STAT st_election_gen 0
STAT st_election_lsn 1/28
STAT st_election_nsites 0
STAT st_election_nvotes 1
STAT st_election_priority 100
STAT st_election_sec 5
STAT st_election_status 0
STAT st_election_tiebreaker 3676766282
STAT st_election_usec 69747
STAT st_election_votes 0
STAT st_elections 1
STAT st_elections_won 1
STAT st_env_id 2147483647
STAT st_env_priority 100
STAT st_gen 2
STAT st_log_duplicated 0
STAT st_log_queued 0
STAT st_log_queued_max 0
STAT st_log_queued_total 0
STAT st_log_records 0
STAT st_log_requested 0
STAT st_master 2147483647
STAT st_master_changes 0
STAT st_max_lease_sec 0
STAT st_max_lease_usec 0
STAT st_max_perm_lsn 0/0
STAT st_msgs_badgen 0
STAT st_msgs_processed 5
STAT st_msgs_recover 0
STAT st_msgs_send_failures 2
STAT st_msgs_sent 10
STAT st_newsites 0
STAT st_next_lsn 1/8916
STAT st_next_pg 0
STAT st_nsites 2
STAT st_nthrottles 0
STAT st_outdated 0
STAT st_pg_duplicated 0
STAT st_pg_records 0
STAT st_pg_requested 0
STAT st_startsync_delayed 0
STAT st_startup_complete 0
STAT st_status 2
STAT st_txns_applied 0
STAT st_waiting_lsn 0/0
STAT st_waiting_pg 0
END

  However, its much simpler, I think, to just try and store a value on an instance. If you get “STORED” back, then this is the master. If you get NOT_STORED back, this is a slave. If it blocks (timeouts are hard in simple scripts, I know.. perldoc -f alarm), you are in a “DOWN” state. The danger here is one of split brain where both nodes thing they’re the master.. but.. if they’re not talking, you have bigger problems!
• Out of sync slaves can’t READ either! This one bit us just the other day. Something ocurred where our slave wasn’t able to retrieve the latest log entries from the master. Because of this, it was reporting errors in replication. During this time, *all* commands blocked. We were relying on basic round-robin DNS for failover, thinking that memcachedb was simple enough, it was either “up” or “down”. Unfortunately, it was stalled on one box, so everything that hit that box blocked and timed out until we firewalled the port so connections wouldn’t succeed. We eventually had to stop the instance, copy a db_hotbackup from the master, then start it again. This still had to catch up from the point at which the db_hotbackup copies logs were checkpointed, which was (because we’re on v1.0.3) many hours before. While it was catching up, all commands (even stats commands.. which is disappointing..) blocked.
• Use a load balancer, not round robin. With that said, a load balancer is a far better solution then round robin. In this case, because the box was “up”, but failing to respond, we were at the mercy of the pecl memcache module’s definition of what was “up” or “down” for reads. A load balancer separates this logic out into monitors so the code can just connect to a virtual IP, or use some list of servers it is given.
• Even better.. just use a floating IP. MemcacheDB seems to scale to ridiculous levels with reads. Like, 400:1 read:write performance. Do you really need lots of slaves? Just having an IP that follows the master will give you fault tolerance. Its easy to determine if a box is the master. You can even do a ‘rep_set_priority 500′ to make sure a box stays the master as long as it has the IP. If you’re running on Linux, Good old Heartbeat is perfect for this. If you need to scale past the write capabilities of one box, then partitioning by using a stable hash algorithm on the keys is a far better solution than master/slave replication, and is already built in to pretty much every memcache client.
• Be careful with db_archive/db_checkpoint. This is mostly regarding v1.0.3, as I don’t know the impact of these commands on v1.1 or 1.2. However, it would seem that even with a replication policy of “ACK_ONE”, its still possible to purge logs that the slave needs. This may or may not be true (something else could have gone wrong) but it seems that running db_checkpoint/db_archive too aggressively seems to have broken our replication. There’s no reason to purge logs too often, so be wary when doing so.

Hopefully this will help other users who are starting to setup MemcacheDB and need fault tolerance.

For caching, this “failover” methodology works fine. If I go to set a value in memcached, and the server fails over to the second one, thats ok. The next get to the primary will fail, and get set properly, and the old entry on the secondary will *eventually* get pushed out of the cache.

However, for storing data reliably, this becomes a problem. Lets say there is a scenario where a network cable is bad on one of the memcached servers. 1 in 100 requests fails. With caching, failover will go a little nuts, but its entirely possible nobody will even notice, as results will be cached, data won’t get stale.. no big deal.

With storage though, this could happen..

- session is created on memache1

- session tries to read from memcache1, and fails.. so new session is created on memache2

- session is then read from memache1

- session is updated on memcache1 with new information

- session fails to read from memcache1, and old session data is read from memacache2, then the set succeeds on memcache1, and the old data is lost.

The point isn’t really this scenario’s details, but that this hashing algorithm is vulnerable, even designed to lose data that was written to it. That is the caching paradigm.

As I discussed this with some colleagues, my mind immediately jumped to MemcacheDB. Maybe that would work for session storage. It has replication, so we could use the traditional active/passive paradigm for it. However, this limits our scale to whatever a single instance of MemcacheDB can handle. Honestly thats probably fine for most sites, as MemcacheDB can probably handle tens of thousands of small writes per second.

However, there are multiple problems. The biggest problem with MemcacheDB is there’s no easy way (yet, they’re working on it) to pull keys out of it to do garbage collection. Likewise, session data really doesn’t need to live for a long time. We just need to be reasonably certain that the data we’re getting is reasonably new.

If we store the data in *all* of the servers, and if we store a highly accurate (meaning if it takes you milliseconds to complete a request, this timestamp needs to be down to microseconds) timestamp of when the data was given to us (meaning we use the same timestamp for each server) along side it, we can then just read it from all of the servers, and pick the newest one. Ew, that means we are still limited to the scale of one instance of memcached.

Then I had a flash back to the way MogileFS works. It stores data on a number of replica servers. Of course, it also keeps track of where it stored them. But I figured, for sessions, thats a lot of overhead. There’s an easier way. We can use the consistent hashing algorithm that the PHP Memcache module uses to pick servers, and just read and write the data from nReplicas servers. If a server fails, we’ll move on to the next one, and there’s a reasonable degree of certainty that it will remain the same. If we write stale data to a server and then fail back to it later, we’re protected by the timestamp rules. The higher nReplicas, the higher the reliability that a server failure won’t cause issues. I even found a PHP implementation of consistent hashing falled FlexiHash.

There’s one last issue that bugs me about using memcached for sessioning, and the timestamp helps us solve. We recently found that there was a problem where a request would take, say, 45 seconds to complete. At 20 seconds, the user would hit the back button out of frustration. This would put other stuff in the session, then the 45 second request would complete, and write the version of the session it thinks is right to the session store, losing the user’s new activity.

There are two ways to solve this. One is to introduce locking. This actually isn’t hard to do with Memcached, it is described in the memcached faq. However, this introduces something to block or fail on in the read. I think its simpler than that. You simply read the record before you write it, and if it has changed since you read it the first time, you don’t write it. You just throw the session write away. Obviously the user has moved on, so there’s no reason to make your update.  If you used locking, the user would still be waiting on the old thread to finish.

Of course, this all hinges on you caring that your session data is accurate, and that you care that users don’t lose their sessions when one server goes down. If neither of those apply to you, then you can just use sessions like cache.

One of my favorite parts of memcached is how dead simple it is. The protocol is very easy to read, making debugging issues and writing new clients very easy. It uses the “least recently used” algorithm to move things out of the cache when it starts to fill up, so its extremely easy to understand how the whole thing works.

The cleverest part of using memcached has nothing to do with the service itself, but the API. The smart guys who developed it figured out that they could hash the key, and pick the same server for reads/writes every time as long as the number of servers doesn’t change. This allows it to scale out to a ridiculous size and retains its simplicity and performance

Two problems arise when a site uses any caching, be it memcached or aggressive HTTP headers.

First, the site starts to rely on caching too heavily for performance. As an example, I had a situtation where the entire corpus of settings for each client site (hundreds of clients, hundreds of settings) was kept in memcached as one massive 200kB+ serialized PHP object. Every page view that needed to access any settings would grab this object at the beginning of the code, and use the object throughout.

This worked really great in some instances, as most of the biggest pages needed to access 30 – 50 settings each time. However, the trouble would come when there was a page that would get a high degree of concurrency, such as an iframe that gets displayed on every page of a major website, or on a page that gets slashdotted. It would be blazing fast, generating almost no load at all for a while, but whenever a setting would be changed (the settings application would clear the cache of settings for whichever client was edited), or the cache object would expire, the database would spike out of control.

The reason was this object took about 1-3 seconds to fetch from the database. Well with 1000 requests per second, thats 3000 requests that get a negative hit on the cache, and so, ask the database for the information. The solution was to cache each setting individually, and use a random skew on the expire time. This prevented the storm of requests whenever there was an expire, and it allowed items looked up in rapid succession to not expire all at once.

This brings us to the second problem with caching, and specifically memcached. The cache is sometimes mistaken for a data store. In the above example, by clearing out entries from memcached, the caching was essentially neutered. Any time during the day somebody might come along and blow out the cache. Thats fine with MySQL’s query cache, for instance, because that just makes queries come back faster. The connection is already made, one of the most painful parts has already happened. With memcached however, the cache can scale to many thousands of connections very cheaply, whereas doing this with most databases is expensive, if not impossible.

So to combat this, what is really needed is a persistent place to keep your data up to date when it is needed in an extremely high reads to write ratio. Thats where memcachedb is so attractive. Instead of keeping everything in RAM, memcachedb stores anything you put into it in a berkeleydb database. To boot, it can replicate this data to another machine, adding to its reliability and availability. This means that writes will be slower, and it won’t scale out nearly as cheaply, but thats ok for situations like this.

With memcachedb, we can change the setting management program to save the data into the database and memcachedb, confident in the fact that it will be there later. Then we don’t have write-back caching code in our application, we just remove the part that connects to the database for that data at all.

This has a huge benefit beyond just performance. With this scheme, we can write simple applications that won’t rely on the read/write database server ever being up. It also means that we don’t have to have a giant database server, or a huge replication fanout to get this data available in realtime.

There is of course the danger that memcachedb gets out of sync with the main db. Thats why in addition to writing to memcachedb whenever you write to the database server, you can also run a refresh script periodically that grabs all of the data from the database and walks through, writing items to memcachedb. Care must be taken here to make sure one doesn’t write stale data to memcachedb. The safest way is to include a timestamp with each record that can easily be compared. Another way to go is to just have this script alert you to items that are out of sync, requiring manually re-saving these records.

Memcachedb is, unfortunately, still a little raw. The replication setup is rather complex. It took me a little while to get it working the way I wanted with just two boxes. It definitely could use command line options to set replication options, so that slaves don’t accidentally promote themselves to masters. Right now one can only do that through the protocol, so I have a nagios plugin that checks it and changes it if it is wrong.

I think its important to note just how cool it is that 90% of memcachedb was written before it was conceived of. BerkeleyDB is one of the great open source success stories, having a successful business model built on free code, and eventually attracting enough attention from Oracle to get purchased. Then to merge that with memcached, which is one of those projects that makes you wish you had written it first, well, I think thats a stroke of genius. Good job Mr. Chu.