If you came here between April 28 and about an hour ago, you got a “couldn’t connect to database” error. Oops! Seems my limited memory EC2 instance got a little overwhelmed by php processes and decided the db server, drizzled, should die to make more room for PHP. Ooops! Time to drop pm.max_children.

I don’t have any monitoring setup for the site, so I just now figured it out. Until I get proper monitoring, I’ve installed this fancy bit of duct-tape upstart magic:

start on stopping
task
script
env | mail -s "$JOB is stopping!" me@myemail.com
end script


What does this do? Well is emails me whenever upstart gives up respawning something, or I manually stop a service.

Its not monitoring. I need monitoring. But this is a nice little hack to prevent a regression while I figure that out.

I’m pretty excited today, as I’m preparing to go live with the first real, high load application of Gearman that I’ve written. What is it you say? Well it is a simple trigger based replicator from mysql to SOLR.

I should say (because I know some of my colleagues read this blog) that I don’t actually believe in this design. Replication using triggers seems fraught with danger. It totally makes sense if you have a giant application and can’t track down everywhere that a table is changed. However, if your app is simple and properly abstracted, hopefully you know the 1 or 2 places that write to the table.

I should also say that I really can’t reveal all of the details. The general idea is pretty simple. Basically we have a trigger that dumps a primary key into gearman via the gearman MySQL UDFs. The idea is just to tell a gearman worker “look at this record in that table”.

Once the worker picks it up, it applies some logic to the record.. “should this be searchable or not”. If the answer is yes it should be searchable, the worker pushes the record into SOLR. If not, the worker will make sure it is not in solr.

This at least is pretty simple. The end result is a system where we can rebuild the search index in parallel using multiple CPU’s (thank you to solr/lucene for being able to update indexes concurrently and efficiently btw). This is done by pushing all of the records in the table into the queue at once.

Anyway, gearmand is performing like a champ, libgearman and the gearman pecl module are doing great. I’m just really happy to see gearman rolled out in production, as I really do think it has that nice mix of simplicity and performance. I love the commandline client which makes it easy to write scripts to inject things into queues, or query workers. This allows me to access a worker like this:

$ gearman -h gearmanbox -f all_workers -s
Known Workers: 11

boxname_RealTimeUpdate_Queue_TriggerWorker_1 jobs=627366,restarts=0,memory_MB=4.27,lastcheckin=Tue, 23 Mar 2010 22:37:59 -0700
boxname_RealTimeUpdate_Queue_Subject_13311 jobs=304134,restarts=0,memory_MB=7.03,lastcheckin=Tue, 23 Mar 2010 22:37:58 -0700
boxname_RealTimeUpdate_Queue_Subject_13306 jobs=606126,restarts=0,memory_MB=7.03,lastcheckin=Tue, 23 Mar 2010 22:37:59 -0700
boxname_RealTimeUpdate_Queue_Subject_13314 jobs=576714,restarts=0,memory_MB=7.03,lastcheckin=Tue, 23 Mar 2010 22:37:59 -0700
boxname_RealTimeUpdate_Queue_Subject_13342 jobs=294846,restarts=0,memory_MB=7.03,lastcheckin=Tue, 23 Mar 2010 22:37:59 -0700
boxname_RealTimeUpdate_Queue_Subject_13347 jobs=376998,restarts=0,memory_MB=7.03,lastcheckin=Tue, 23 Mar 2010 22:37:59 -0700
boxname_RealTimeUpdate_Queue_Subject_13359 jobs=470508,restarts=0,memory_MB=7.03,lastcheckin=Tue, 23 Mar 2010 22:37:58 -0700
boxname_RealTimeUpdate_Queue_Subject_13364 jobs=403182,restarts=0,memory_MB=7.03,lastcheckin=Tue, 23 Mar 2010 22:37:58 -0700
boxname_RealTimeUpdate_Property_SolrPublish_ jobs=219630,restarts=0,memory_MB=6.19,lastcheckin=Tue, 23 Mar 2010 22:37:59 -0700
boxname_RealTimeUpdate_Queue_TriggerWorker_2 jobs=393642,restarts=0,memory_MB=4.27,lastcheckin=Tue, 23 Mar 2010 22:37:59 -0700
boxname_RealTimeUpdate_Property_SolrBatchPub jobs=6,restarts=0,memory_MB=6.23,lastcheckin=Tue, 23 Mar 2010 22:37:28 -0700

Brilliant.. no need for html or HTTP.. just a nice simple commandline interface.

I think gearman still has a ways to go. I’d really like to see some more administration added to it. Deleting empty queues and quickly flushing all queues without restarting gearmand would be nice to haves. We’ll see what happens going forward, but for not, thanks so much to the gearman team (especially Eric Day who showed me gearman, and Brian Aker for pushing hard to release v0.12).

w00t!

I think everybody at one point looked at MySQL and tought.. “that would work fine as a queue system”. For low volume stuff, it *is* fine. But then somebody grabs your little transactional, relational, reliable queue system and plugs 5 million messages per hour through it, and somewhere, a man name Heikki cries.

So then you start to look around.. and for those of us who have meager budgets and tend to use open source, there aren’t a lot of choices. The guys at Second Life did some research for all of us…. Once you get through that though, you realize that the needs of second life, a MMORPG, are quite a bit different from your average web app.

So, without further ado, my “queue” system round up.

• ActiveMQ – This shining star of the queueing world seems to come up quickly in conversation. At Adicio, we actually gave it a good try. The main problem was, we’re a PHP shop. The PHP accessibility comes not through the normal Java Messing Service connector, but “STOMP”.

  Honestly, I’m not a big fan of these giant Apache sponsored java projects. SOLR has changed my mind a bit, as it seems to work well and doesn’t really crash. Then again, I’m not carrying a pager anymore, so maybe it does suck and I’m just not seeing it.

  Anyway, at first, ActiveMQ was winning me over. It was pretty quick.. had a pretty simple setup curve (just start up the latest version, and you have a working persistent queue system), and despite having mountains of documentation that reads like the text spammers shove into their emails randomly to pass bayesian filters, it made sense.

  However, its fall was pretty quick, as the first problem we hit was its Producer Throttling. This probably works fine when you’re using the JMS connector. However, with Stomp, when ActiveMQ decides your queue is too full, and it needs you to stop, it just stops acking your packets. Your stomp client blocks (or spins, in non-block mode) and you wait. This is made worse by the fairly naive php stomp driver, which doesn’t really check to see why its write failed, or even try to see if it can.

  Things got better when that was disabled, but the stomp driver was still haphazard. After figuring out that the Master/Slave protocol requires one to shut down the slave whenever failing back to a downed master, I had had enough. Sionara ActiveMQ.

• RabbitMQ – This one seems to be a favorite of many. My experience is limited, and I really haven’t tried it that much. Its written in erlang, which I guess automatically makes something “telco reliable”. Cool.
• QPID – Wow, this one is supposedly INCREDIBLE. “500,000 messages per second per LUN.” . WOW. It also has RedHat’s backing, which is a big win for me.
  In fact, as I write this, I’m doing my best to build and install the latest qpid on CentOS 5.4.

   gcc -DHAVE_CONFIG_H -I. -I. -I./src/config -I./include/ -I/usr/src/redhat/BUILD/xerces-c-src_2_8_0/src -I./src/lexer/ -D_GNU_SOURCE -D_REENTRANT -O2 -g -m64 -mtune=generic -MT mapm_add.lo -MD -MP -MF .deps/mapm_add.Tpo -c src/mapm/mapm_add.c  -fPIC -DPIC -o .libs/mapm_add.o
  ...
  

  In case you’re familiar, I’m there. Oops, thats not qpid. Thats xerces-c. Which I have to build.. and I also have to build xqilla after that. Luckily, 40 other packages required to build qpid were available in the standard CentOS yum repository.

  Another unfortunate reality is that there is no qpid connectivity available for PHP. Unless the php-amqp module works. Its really not clear yet.
  Anyway, this looks like a promising messaging technology. However, this much software leaves a lot of room for things to break.. so, while I will probably complete the build, as I want to find out how it stacks up to the others in terms of simplicity and performance, I think this one is dead.

• Gearman – Ok I’m going to say it up front. I like this one. Its really not a “queue” system per sé. The name is an anagram of ‘manager’ (say that 5 times fast!). Its one of those great things that came out of the Danga group, the same people who created MogileFS and Memcached.

  Call me stupid, but I like to be able to read things. QPID is in C++, and is so big, I don’t even know where to start. Java gives me the shivers, and I don’t even know what erlang looks like. But damn, who doesn’t like poring over well written C? Thats pretty much what the new C port of gearmand is.

  I’m especially fond of the ease with which one can write a persistence layer. I recently submitted code to make the tokyocabinet queue store better. Its a simple B+Tree store that everybody’s going crazy about these days. Its also written in really nice C.

  The built in ability for gearman clients/workers (producers/consumers) to have a 2 way conversation is especially appealing. Its not like they can just freely pass messages back and forth. But clients can choose to wait for the job they submitted to complete. They can also check on the status of the job fairly easily. Workers can send back two integers (numerator and denominator), which is particularly useful for sending back a count of things done over the count of things to do.

  Combine all this cool stuff with the dead simple ‘gearman’ command line client, and you have a happy Clint. I wrote a little PHP worker that just sits around collecting data sent to it by the other workers running. When it receives a “show_all_workers” message (function in gearman-ese), it just spits back a text report of what it knows. This can be triggered by just saying:

  $ gearman -s -f show_all_workers
  
  Known Workers: 5
  
  dev3.adicio.com_Adicio_App_Reverse_Worker_29336 jobs=26508,restarts=0,memory_MB=1.47,lastcheckin=Thu, 21 Jan 2010 15:33:32 -0800
  dev3.adicio.com_Adicio_App_Reverse_Worker_29333 jobs=19194,restarts=0,memory_MB=1.47,lastcheckin=Thu, 21 Jan 2010 15:33:32 -0800
  dev3.adicio.com_Adicio_App_Reverse_Worker_29356 jobs=29208,restarts=0,memory_MB=1.47,lastcheckin=Thu, 21 Jan 2010 15:33:32 -0800
  dev3.adicio.com_Adicio_App_Reverse_Worker_29370 jobs=27638,restarts=0,memory_MB=1.47,lastcheckin=Thu, 21 Jan 2010 15:33:32 -0800
  dev3.adicio.com_Adicio_App_Reverse_Worker_29332 jobs=10636,restarts=0,memory_MB=1.47,lastcheckin=Thu, 21 Jan 2010 15:33:32 -0800
  
  $
  

  This is pretty damn cool. Now double the fun with MySQL UDF’s, and you have a workable solution for queueing via MySQL trigger.
  

  So, I can’t help but give this one the nod for simplicity of design. There are no massive books written to explain what gearman does. Just a nice easy C library, and perhaps one of the most important things, a really useful PHP extension.

For web apps, testing actually isn’t *that* hard. The client program (the browser) is readily available on every platform known to man, and they generally don’t do much more than store and retrieve data in clever ways. So, its not like we have to fire up a Large Hadron Collider to observe the effects of our web app.

Therein lies the problem though, as clicking around on web forms and entering the same email address, password, address, phone number, etc. etc., 100 times, is BORING.

Enter Selenium. This amazing little tool has been on the scene for a little while now, but its just now getting some momentum. Click through to the website and watch “the magic” as they put it, but basically here’s how it goes:

• open their firefox plugin and click ‘record
• do something
• click ‘record’ again.

Then just save this little test case to a file, and the next time you change anything that might relate to the series of clicks and data entries you just made, run this test again. There are all kinds of assertions you can make while you’re doing something. Like ‘Make sure the title is X’ or ‘make sure a link to Y exists’.

But wait, I could have done that with something like Test::More,  PHPUnit, or lime. Where’s the real benefit?

Well because Selenium remotely controls your browser, all those gotchya’s regarding javascript CSS incompatibilities can come into play here. Because Selenium can control Internet Explorer, Firefox, *and* Safari. In fact it can also control Opera, and according to their website, any browser that properly supports javascript fully.

This is really a nice evolutionary step for web shops, as tools like this generally are OS specific and cost a lot of money. Once again open source software appears where a need becomes somewhat ubiquitous.

You can even take it a step further. The next thing that generally happens in a web dev shop when they get bigger than 20 or 30 people is they hire people who actually like testing. Well not really, but they dislike it *less* than software engineers. These are QA engineers. And they DO like things to be orderly and efficient.

Bromine is the answer for that. Its still pretty rough around the edges, but it gets the job done.

Again check out their website and watch the screencast, but basically it goes like this:

• Write selenium tests as specified above
• Upload tests to Bromine server
• Attach tests to requirements
• Run selenium remote control on all required OS/browser version combinations (can you say virtualbox?)
• Run tests

Another nice thing about using bromine is now you are running your tests in a server side language, not just the Selenium IDE, which is limited to the IDE’s generated “Selenese” XML commands for tests. The IDE exports your basic test into PHP or Java, and then on the bromine server you can do interesting things, like check an IMAP box for an email, run a backend process, or send an SMS.

At first it may not seem like much, but eventually you end up with a multitude of useful tests for your web app that can be run all the time against development branches before release, and catch many problems. Quality means happier users, which hopefully means loyal users that keep coming back.

Brian Aker had mentioned to me at one time that TokyoTyrant was way better than memcachedb and we should run it instead. I took notice and it turns out he’s right! It does basically the same thing, applying the memcache protocol to an on disk key/value store. However, the code is incredibly clean, well maintained, and runs extremely fast. There’s also a lot more flexibility, with the ability to choose between in-memory or on disk storage, hash tables, B+Tree’s, etc.

The availability of log based asynchronous master/master replication (somewhat similar to MySQL’s replication in concept) was probably one of the biggest wins, allowing much simpler failover (just move the IP, or DNS, or whatever) when compared to MemcacheDB’s adherence to BerkeleyDB’s replication setup, which is a single-master system implementing an election algorithm.

Somewhere during migration, we missed one tiny detail though. Sometimes, the devil is in the details. This is really the only evidence in the documentation that tokyo tyrant has support for the memcache protocol. It is very clear:

Memcached Compatible Protocol

As for the memcached (ASCII) compatible protocol, the server implements the following commands; “set”, “add”, “replace”, “get”, “delete”, “incr”, “decr”, “stats”, “flush_all”, “version”, and “quit”. “noreply” options of update commands are also supported. However, “flags”, “exptime”, and “cas unique” parameters are ignored.

Now, as I said, there’s nothing ambiguous about this. That would have helped, if anyone on my team had ever read it. We installed TokyoTyrant, pointed our basic test code at it, and it worked. This is really a process problem, not so much a technical one. The process must be to assume it won’t work, and test all the different use cases to make sure it works.

Now, why is that bit of the manual important? Well we use PHP. Specifically, we use the PECL “Memcache” module to access memcache protocol storage. Now, the Memcache module is mostly oriented toward caching in the memory based original memcached. It works great for memcachedb too, which simply ignores the exptime parameter. However, memcacheDB *does not* ignore “flags”.

And therein lies the problem. Users of the PECL Memcache module may not know this, but the flags are *important*. There are two bits in that flags field that the Memcache module may set. Bit 0 is used to indicate whether or not the content has been serialized, and, therefore, on read, must be unserialized. Bit 1 is used to indicate whether or not the content has been gzipped.

So, while all of the strings that were stored in MemcacheDB and subsequently copied to TokyoTyrant worked great, the serialized objects, arrays, and gzipped values, were completely inoperative, as they were coming back to the code as strings and binary compressed data. The gzipped data was easy (turn off automatic gzip compression). The serialized data took some quick tap dancing to remedy, with code something like this:


class Memcache_BrokenFlags extends Memcache
{
public function get($key, &$flags)
{
$v = parent::get($key, $flags);
$uv = @unserialize($v);
return $uv === false ? $v : $uv;
}
}


Luckily our code all uses one Factory method to spawn all “MemcacheDB” connections, so it was easy to substitute this in.

Eventually we can just change the code by segregating into things that always serialize, and things that don’t, and just do the serialization ourselves. This should eventually allow us to use the new tokyo_tyrant module in PECL, which only reliably stores scalars (I noticed recent versions have added a call to the internal PHP function convert_to_string().. this is, I think, a mistake, but one that still leaves it up the programmer to explicitly serialize when serialization is desired).

This was a pretty big gotchya, and one that illustrates that even though sometimes us cowboy coders and sysadmins get annoyed when those pesky business people ask us for plans, schedules, expected impact, etc., and we keep assuring them we know whats up, its still important to actually know whats up, and make sure to RTFMC .. C as in, CAREFULLY.

Its not entirely clear from the documentation how to setup memcachedb for fault tolerance. Here’s the procedures I’ve found useful.

• Set up replication right. With all due respect to Steve Chu, The docs aren’t really clear on how to setup replication. Its much simpler than it looks. Just run MemcacheDB as you would if it were standalone, but then add a combination of these 3 options:
  • You must have a -R line if you want to participate in replication. This is your hostname and port that listens for connections from other machines for replication. It is the same value that should be listed in every other machine’s -O.
  • a -O for *every* other machine that may want to replicate to/from this machine. I am sure there are situations where you won’t need these, but it makes re-syncing and elections more predictable. You won’t be able to re-sync “live” after a failure without -O options.
  • -M/-S are not required. If you start n machines without -M or -S, but with appropriate -R and -O lines, they will arbitrarily elect a master. If you run them with -M and -S, then the -M box will just be pushy and always elect itself the master, and the -S boxes will, likewise, always try to defer to slave status.
  • Lets say we wanted to listen for memcache protocol on port 45000 on host ‘node1′ and replicate to ‘node2′
  • Standalone: memcachedb -p 45000 -H /home/memdb/data -u memdb -N
  • Replication w/ elected master: memcachedb -p 45000 -H /home/memdb/data -u memdb -N -R node1:46000 -O node2:46000
  • Replication Master: memcachedb -p 45000 -H /home/memdb/data -u memdb -N -R node1:46000 -O node2:46000 -M
• Only the current master can accept writes. You can see which machine is the master with the ‘stats rep’ command. In v1.2.1 its shown as an environment id. Below st_env_id and st_master are the same, so this is the master:
  
stats rep
STAT st_bulk_fills 0
STAT st_bulk_overflows 0
STAT st_bulk_records 11
STAT st_bulk_transfers 3
STAT st_client_rerequests 0
STAT st_client_svc_miss 0
STAT st_client_svc_req 0
STAT st_dupmasters 0
STAT st_egen 3
STAT st_election_cur_winner 2147483647
STAT st_election_gen 0
STAT st_election_lsn 1/28
STAT st_election_nsites 0
STAT st_election_nvotes 1
STAT st_election_priority 100
STAT st_election_sec 5
STAT st_election_status 0
STAT st_election_tiebreaker 3676766282
STAT st_election_usec 69747
STAT st_election_votes 0
STAT st_elections 1
STAT st_elections_won 1
STAT st_env_id 2147483647
STAT st_env_priority 100
STAT st_gen 2
STAT st_log_duplicated 0
STAT st_log_queued 0
STAT st_log_queued_max 0
STAT st_log_queued_total 0
STAT st_log_records 0
STAT st_log_requested 0
STAT st_master 2147483647
STAT st_master_changes 0
STAT st_max_lease_sec 0
STAT st_max_lease_usec 0
STAT st_max_perm_lsn 0/0
STAT st_msgs_badgen 0
STAT st_msgs_processed 5
STAT st_msgs_recover 0
STAT st_msgs_send_failures 2
STAT st_msgs_sent 10
STAT st_newsites 0
STAT st_next_lsn 1/8916
STAT st_next_pg 0
STAT st_nsites 2
STAT st_nthrottles 0
STAT st_outdated 0
STAT st_pg_duplicated 0
STAT st_pg_records 0
STAT st_pg_requested 0
STAT st_startsync_delayed 0
STAT st_startup_complete 0
STAT st_status 2
STAT st_txns_applied 0
STAT st_waiting_lsn 0/0
STAT st_waiting_pg 0
END

  However, its much simpler, I think, to just try and store a value on an instance. If you get “STORED” back, then this is the master. If you get NOT_STORED back, this is a slave. If it blocks (timeouts are hard in simple scripts, I know.. perldoc -f alarm), you are in a “DOWN” state. The danger here is one of split brain where both nodes thing they’re the master.. but.. if they’re not talking, you have bigger problems!
• Out of sync slaves can’t READ either! This one bit us just the other day. Something ocurred where our slave wasn’t able to retrieve the latest log entries from the master. Because of this, it was reporting errors in replication. During this time, *all* commands blocked. We were relying on basic round-robin DNS for failover, thinking that memcachedb was simple enough, it was either “up” or “down”. Unfortunately, it was stalled on one box, so everything that hit that box blocked and timed out until we firewalled the port so connections wouldn’t succeed. We eventually had to stop the instance, copy a db_hotbackup from the master, then start it again. This still had to catch up from the point at which the db_hotbackup copies logs were checkpointed, which was (because we’re on v1.0.3) many hours before. While it was catching up, all commands (even stats commands.. which is disappointing..) blocked.
• Use a load balancer, not round robin. With that said, a load balancer is a far better solution then round robin. In this case, because the box was “up”, but failing to respond, we were at the mercy of the pecl memcache module’s definition of what was “up” or “down” for reads. A load balancer separates this logic out into monitors so the code can just connect to a virtual IP, or use some list of servers it is given.
• Even better.. just use a floating IP. MemcacheDB seems to scale to ridiculous levels with reads. Like, 400:1 read:write performance. Do you really need lots of slaves? Just having an IP that follows the master will give you fault tolerance. Its easy to determine if a box is the master. You can even do a ‘rep_set_priority 500′ to make sure a box stays the master as long as it has the IP. If you’re running on Linux, Good old Heartbeat is perfect for this. If you need to scale past the write capabilities of one box, then partitioning by using a stable hash algorithm on the keys is a far better solution than master/slave replication, and is already built in to pretty much every memcache client.
• Be careful with db_archive/db_checkpoint. This is mostly regarding v1.0.3, as I don’t know the impact of these commands on v1.1 or 1.2. However, it would seem that even with a replication policy of “ACK_ONE”, its still possible to purge logs that the slave needs. This may or may not be true (something else could have gone wrong) but it seems that running db_checkpoint/db_archive too aggressively seems to have broken our replication. There’s no reason to purge logs too often, so be wary when doing so.

Hopefully this will help other users who are starting to setup MemcacheDB and need fault tolerance.

checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ANSI C... none needed
checking dependency style of gcc... gcc3
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking dependency style of g++... gcc3
checking for ranlib... ranlib
checking for bash... /bin/sh
checking for perl... /usr/bin/perl
checking for python... /usr/bin/python
checking for ar... /usr/bin/ar
checking for ant... no
checking Checking EXTERNAL_PATH set to... /usr/local/src/scribetest/thrift/contrib/fb303
checking whether to enable optimized build... yes
checking whether to enable static mode... yes
checking Checking thrift_home set to... /usr/local
checking for boostlib >= 1.33.1... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: creating cpp/Makefile
config.status: creating py/Makefile
config.status: executing depfiles commands
EXTERNAL_PATH /usr/local/src/scribetest/thrift/contrib/fb303
[root@wolverine fb303]# make
make all-recursive
make[1]: Entering directory `/usr/local/src/scribetest/thrift/contrib/fb303'
Making all in .
make[2]: Entering directory `/usr/local/src/scribetest/thrift/contrib/fb303'
make[2]: Nothing to be done for `all-am'.
make[2]: Leaving directory `/usr/local/src/scribetest/thrift/contrib/fb303'
Making all in cpp
make[2]: Entering directory `/usr/local/src/scribetest/thrift/contrib/fb303/cpp'
make all-am
make[3]: Entering directory `/usr/local/src/scribetest/thrift/contrib/fb303/cpp'
if g++ -DPACKAGE_NAME=\"libfb303\" -DPACKAGE_TARNAME=\"libfb303\" -DPACKAGE_VERSION=\"20080209\" -DPACKAGE_STRING=\"libfb303\ 20080209\" -DPACKAGE_BUGREPORT=\"\" -DHAVE_BOOST= -I. -I. -I.. -Igen-cpp -I/usr/local/include/thrift -I/usr/local/include/boost-1_37 -Wall -O3 -MT FacebookService.o -MD -MP -MF ".deps/FacebookService.Tpo" -c -o FacebookService.o `test -f 'gen-cpp/FacebookService.cpp' || echo './'`gen-cpp/FacebookService.cpp; \
then mv -f ".deps/FacebookService.Tpo" ".deps/FacebookService.Po"; else rm -f ".deps/FacebookService.Tpo"; exit 1; fi
In file included from gen-cpp/FacebookService.cpp:6:
gen-cpp/FacebookService.h:28: error: `facebook::thrift' has not been declared
gen-cpp/FacebookService.h:28: error: `Service' has not been declared
gen-cpp/FacebookService.h:28: error: ISO C++ forbids declaration of `_return' with no type
gen-cpp/FacebookService.h:72: error: `facebook::thrift' has not been declared
gen-cpp/FacebookService.h:72: error: `Service' has not been declared
gen-cpp/FacebookService.h:72: error: ISO C++ forbids declaration of `parameter' with no type
gen-cpp/FacebookService.h:1077: error: `facebook::thrift' has not been declared
gen-cpp/FacebookService.h:1077: error: ISO C++ forbids declaration of `Service' with no type
gen-cpp/FacebookService.h:1077: error: expected `;' before "success"
gen-cpp/FacebookService.h: In member function `bool facebook::fb303::FacebookService_getLimitedReflection_result::operator==(const facebook::fb303::FacebookService_getLimitedReflection_result&) const':
gen-cpp/FacebookService.h:1086: error: `success' was not declared in this scope
gen-cpp/FacebookService.h:1086: error: 'const class facebook::fb303::FacebookService_getLimitedReflection_result' has no member named 'success'
gen-cpp/FacebookService.h:1086: warning: unused variable 'success'
gen-cpp/FacebookService.h: At global scope:
gen-cpp/FacebookService.h:1107: error: `facebook::thrift' has not been declared
gen-cpp/FacebookService.h:1107: error: ISO C++ forbids declaration of `Service' with no type
gen-cpp/FacebookService.h:1107: error: expected `;' before '*' token
gen-cpp/FacebookService.h:1241: error: `facebook::thrift' has not been declared
gen-cpp/FacebookService.h:1241: error: `Service' has not been declared
gen-cpp/FacebookService.h:1241: error: ISO C++ forbids declaration of `_return' with no type
gen-cpp/FacebookService.h:1243: error: `facebook::thrift' has not been declared
gen-cpp/FacebookService.h:1243: error: `Service' has not been declared
gen-cpp/FacebookService.h:1243: error: ISO C++ forbids declaration of `_return' with no type
gen-cpp/FacebookService.h:1434: error: `facebook::thrift' has not been declared
gen-cpp/FacebookService.h:1434: error: `Service' has not been declared
gen-cpp/FacebookService.h:1434: error: ISO C++ forbids declaration of `_return' with no type
gen-cpp/FacebookService.cpp: In member function `uint32_t facebook::fb303::FacebookService_getLimitedReflection_result::read(apache::thrift::protocol::TProtocol*)':
gen-cpp/FacebookService.cpp:1795: error: 'class facebook::fb303::FacebookService_getLimitedReflection_result' has no member named 'success'
gen-cpp/FacebookService.cpp: In member function `uint32_t facebook::fb303::FacebookService_getLimitedReflection_result::write(apache::thrift::protocol::TProtocol*) const':
gen-cpp/FacebookService.cpp:1821: error: 'const class facebook::fb303::FacebookService_getLimitedReflection_result' has no member named 'success'
gen-cpp/FacebookService.cpp: In member function `uint32_t facebook::fb303::FacebookService_getLimitedReflection_presult::read(apache::thrift::protocol::TProtocol*)':
gen-cpp/FacebookService.cpp:1851: error: 'class facebook::fb303::FacebookService_getLimitedReflection_presult' has no member named 'success'
gen-cpp/FacebookService.cpp: At global scope:
gen-cpp/FacebookService.cpp:2614: error: `facebook::thrift' has not been declared
gen-cpp/FacebookService.cpp:2614: error: variable or field `getLimitedReflection' declared void
gen-cpp/FacebookService.cpp:2614: error: `int facebook::fb303::FacebookServiceClient::getLimitedReflection' is not a static member of `class facebook::fb303::FacebookServiceClient'
gen-cpp/FacebookService.cpp:2614: error: `Service' was not declared in this scope
gen-cpp/FacebookService.cpp:2614: error: `_return' was not declared in this scope
gen-cpp/FacebookService.cpp:2615: error: expected `,' or `;' before '{' token
gen-cpp/FacebookService.cpp:2633: error: `facebook::thrift' has not been declared
gen-cpp/FacebookService.cpp:2633: error: variable or field `recv_getLimitedReflection' declared void
gen-cpp/FacebookService.cpp:2633: error: `int facebook::fb303::FacebookServiceClient::recv_getLimitedReflection' is not a static member of `class facebook::fb303::FacebookServiceClient'
gen-cpp/FacebookService.cpp:2633: error: `Service' was not declared in this scope
gen-cpp/FacebookService.cpp:2633: error: `_return' was not declared in this scope
gen-cpp/FacebookService.cpp:2634: error: expected `,' or `;' before '{' token
gen-cpp/FacebookService.cpp: In member function `void facebook::fb303::FacebookServiceProcessor::process_getLimitedReflection(int32_t, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*)':
gen-cpp/FacebookService.cpp:3070: error: 'class facebook::fb303::FacebookService_getLimitedReflection_result' has no member named 'success'
make[3]: *** [FacebookService.o] Error 1
make[3]: Leaving directory `/usr/local/src/scribetest/thrift/contrib/fb303/cpp'
make[2]: *** [all] Error 2
make[2]: Leaving directory `/usr/local/src/scribetest/thrift/contrib/fb303/cpp'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/usr/local/src/scribetest/thrift/contrib/fb303'
make: *** [all] Error 2


I googled around but found nothing. I tried logging into #thrift on Freenode, and “SteveC_” informed me that he had the same problem, and pointed me at this thrift bug report that explains it a little better. It would seem that Thrift is no longer in the ‘facebook’ namespace, but rather ‘apache’. This makes perfect sense. However, fb303 still expects it to be facebook. So, on Steve’s advice of replacing the word ‘facebook’ with ‘apache’ in the fb303 tree, I did this in fb303:

find . -type f | xargs perl -p -i -e ‘s/facebook/apache/g’

And the make continued a little further until I hit this:


make[2]: Leaving directory `/usr/local/src/scribetest/thrift/contrib/fb303/cpp'
Making all in py
make[2]: Entering directory `/usr/local/src/scribetest/thrift/contrib/fb303/py'
/usr/bin/python setup.py build
running build
running build_py
error: package directory 'fb303' does not exist
make[2]: *** [all-local] Error 1
make[2]: Leaving directory `/usr/local/src/scribetest/thrift/contrib/fb303/py'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/usr/local/src/scribetest/thrift/contrib/fb303'
make: *** [all] Error 2


However, this error seemed to be irrelevant, as ‘make install’ put fb303 where it was supposed to go.

Building scribe, the same ‘find’ command from above was required. I ran it after running into the first ‘no such thing as facebook’ error, and the make continued, and scribe seemed to work just fine.

Hopefully this will find its way into google’s indexes and help somebody.

For caching, this “failover” methodology works fine. If I go to set a value in memcached, and the server fails over to the second one, thats ok. The next get to the primary will fail, and get set properly, and the old entry on the secondary will *eventually* get pushed out of the cache.

However, for storing data reliably, this becomes a problem. Lets say there is a scenario where a network cable is bad on one of the memcached servers. 1 in 100 requests fails. With caching, failover will go a little nuts, but its entirely possible nobody will even notice, as results will be cached, data won’t get stale.. no big deal.

With storage though, this could happen..

- session is created on memache1

- session tries to read from memcache1, and fails.. so new session is created on memache2

- session is then read from memache1

- session is updated on memcache1 with new information

- session fails to read from memcache1, and old session data is read from memacache2, then the set succeeds on memcache1, and the old data is lost.

The point isn’t really this scenario’s details, but that this hashing algorithm is vulnerable, even designed to lose data that was written to it. That is the caching paradigm.

As I discussed this with some colleagues, my mind immediately jumped to MemcacheDB. Maybe that would work for session storage. It has replication, so we could use the traditional active/passive paradigm for it. However, this limits our scale to whatever a single instance of MemcacheDB can handle. Honestly thats probably fine for most sites, as MemcacheDB can probably handle tens of thousands of small writes per second.

However, there are multiple problems. The biggest problem with MemcacheDB is there’s no easy way (yet, they’re working on it) to pull keys out of it to do garbage collection. Likewise, session data really doesn’t need to live for a long time. We just need to be reasonably certain that the data we’re getting is reasonably new.

If we store the data in *all* of the servers, and if we store a highly accurate (meaning if it takes you milliseconds to complete a request, this timestamp needs to be down to microseconds) timestamp of when the data was given to us (meaning we use the same timestamp for each server) along side it, we can then just read it from all of the servers, and pick the newest one. Ew, that means we are still limited to the scale of one instance of memcached.

Then I had a flash back to the way MogileFS works. It stores data on a number of replica servers. Of course, it also keeps track of where it stored them. But I figured, for sessions, thats a lot of overhead. There’s an easier way. We can use the consistent hashing algorithm that the PHP Memcache module uses to pick servers, and just read and write the data from nReplicas servers. If a server fails, we’ll move on to the next one, and there’s a reasonable degree of certainty that it will remain the same. If we write stale data to a server and then fail back to it later, we’re protected by the timestamp rules. The higher nReplicas, the higher the reliability that a server failure won’t cause issues. I even found a PHP implementation of consistent hashing falled FlexiHash.

There’s one last issue that bugs me about using memcached for sessioning, and the timestamp helps us solve. We recently found that there was a problem where a request would take, say, 45 seconds to complete. At 20 seconds, the user would hit the back button out of frustration. This would put other stuff in the session, then the 45 second request would complete, and write the version of the session it thinks is right to the session store, losing the user’s new activity.

There are two ways to solve this. One is to introduce locking. This actually isn’t hard to do with Memcached, it is described in the memcached faq. However, this introduces something to block or fail on in the read. I think its simpler than that. You simply read the record before you write it, and if it has changed since you read it the first time, you don’t write it. You just throw the session write away. Obviously the user has moved on, so there’s no reason to make your update.  If you used locking, the user would still be waiting on the old thread to finish.

Of course, this all hinges on you caring that your session data is accurate, and that you care that users don’t lose their sessions when one server goes down. If neither of those apply to you, then you can just use sessions like cache.