Ubuntu 10.04.1 LTS released | The Fridge.

This is pretty cool. I know as a system administrator, I never wanted to run .0 anything. So 10.04 is really like 10.04.0, and means “let somebodye lse find the bugs.”.

Well 10.04.1 means that the more conservative administrators can at least have a reasonable expectation that it will be even more stable than it was on release day in April.

If you’re already running Ubuntu servers, btw, check this out:

http://maps.ubuntu.com

Hit it and be counted as a server user. Pretty amazing how many little orange circles there are all over the world.

For those of you who’ve been telling me that my blog posts sound like “gleep ork boog florg”, a quick primer:

Ubuntu is an operating system, like Mac OS X or Windows (except more awesomer).

10.04 was their April, 2010 release (10 == 2010 04 == april).

LTS means Long Term Support. This means that the people who maintain Ubuntu will support this release for 3 – 5 years (depending on the context.. 3 for desktops, 5 for servers).

10.04.1 is a fixed up release, mainly marking the release of updated CD images for installing. If you install 10.04 and choose automatic updates, you’re already on 10.04.1 before the release.

Today I realized that a quest I set out on long ago was achieved, and I don’t know how happy I am about it. As I look around my house, I see but one laptop running. The wife has a little Netbook, and you might count the Wii or evne the AT&T U-Verse cable boxes as computers, but when it comes down to it, the only thing I need is my 15″ laptop.

Back when I got into computers, I was cobbling together every little piece of funky hardware I could to build a pseudo-production network inside my house. I had an old AMD 5×86-133 based box that served as my firewall and router. I had a little AMD 900 with redundant cheap IDE disks that was my server. I ran Debian GNU/Linux on them because thats what all the smartest people I knew recommended, and it was incredible because it let me do everything I wanted to do my way, without making me do anything to get it working. Even though I had just a 56kbit modem connection, I used squid and heavy tuning to make it the best web browsing experience possible.. for.. me.

Oh sure, at work I had servers to play with and I definitely enjoyed work. But the autonomy of doing this my way, and learning new things, was what really made it a passion.

One of the most amazing things that came out of that exercise was the realization that while I had the most amazing modem based home network ever, I wasn’t always aware of what was going on. Sometimes I’d wonder, what the heck is going on?

Then I moved the little 5 port switch from behind the desk, to on top of the desk. Instantly, I felt better. I enjoyed watching the two little ports that my server and workstation were on blink themselves silly, almost solid, when I was copying a file over the network. And the constant little twinkles just made me feel good that my network was busy, useful, and active.

But what about the internet connection? What was it doing? I found the answer to this one in one of those great hacks that just makes you smile. tleds. This little program simply hooked into Linux’s networking stack and made the TX and RX functions on my modem force the keyboard LED’s on my server to blink. Now, my server actually didn’t have a keyboard before this, but I grabbed an old one from the closet, plugged it in, and concealed all but the keyboard LED’s so that it just blinked.

Why do these blinking lights make us feel better? I don’t know. But thats one thing lost in the cloud. No blinking lights. No feedback that its doing something.

Maybe somebody should make a ‘cloudleds’ command that blinks your keybaord when your cloud instances send and receive.

While attending OSCON 2009 w/ my faithful sidekick fluffy, we constantly kept finding instances of a common theme. The leading companies and projects seem to share one attribute that might shock you.

They all have at least *some* crappy code. At some point, all of them have set aside their principles and thrown in a hack to get things working. This is reinforced by those projects that have their dignity, but no market share. FreeBSD users are famous for saying that Linux is coded by 10,000 monkeys. FreeBSD is an awesome project, that has powered some huge websites. However, the primary Free OS is Linux. Even further along that line is Windows, which is pretty much a hack on a hack on a hack, but somehow, everybody ends up running it.

This isn’t to say that all of the code in popular projects sucks. Just that some of it does. I’m still waiting for the example of an organization that has produced pure, beautiful code with no compromises, and then gone on to garner a large market share and/or massive profits.

The site TheDailyWTF exists primarily because of this fact. I hit that site at least twice a week to have a good laugh. Many times it causes me to reminisce about some of the things I saw early in my career. Just as often, I’m reminded of something more recent. The trend doesn’t seem to stop, despite advances in computing and human understanding, it goes back decades. I imagine Ogg, the first guy who designed a wheel, snarked about how Thag’s wheels weren’t perfectly round. But ultimately, Thag was able to produce wheels that weren’t perfectly round, but rolled pretty well. He probably got them out in half the time, and ended up trading more wheels for Mammoth pelts than Ogg by a factor of five. No doubt Thag was able to attract more mates with his Mammoth Pelt fortune, so maybe its just in our nature.

Really though, this flies in the face of code purity, which we all want. Code sucking == profit? Hacks == market share? This doesn’t sit well with those of us who pride ourselves on brace placement discipline, and knowing at least 5 design patterns without looking them up in a book. But there it is, that pile of dung you knocked out at 3am the day before release to QA… 3 years ago. Still powering the site despite being closer to Alpaca bile than beautiful code.

This doesn’t mean projects fail without hacks. What it means though, is that projects that obsess over doing things “the right way” tend to languish, and rarely achieve success on a massive scale. For some that is ok, they’re happy to have produced something great that a few people like and that works right for them. In fact, this is largely the (healthy) attitude I see from the PostgreSQL project.

The PostgreSQL developers and users tend to feel strongly that their database is far superior to the likes of say, MySQL. They’ll tell you that they have always had full ACID compliance, that their bug counts are low, and performance continues to rise with every release.

I know a lot of people are successfully running PostgreSQL, but really, by contrast, seems like everybody’s running MySQL. MySQL is not bad code either. It just has hacks. Ok, having dug into it a bit now, it has a lot of hacks. But, why is MySQL the leader, and PostgreSQL the follower.

I think the answer is right there in that last sentence. As Cesar Milan will tell you, “choo gotta be da pack leader”. PostgreSQL probably would have continued on as a fine, but obscure, database engine had MySQL not revolutionized data storage in the same way Apache revolutionized web serving. MySQL has managed to carve out a huge market with Free software, while PostgreSQL’s market is only now beginning to grow. Really PostgreSQL has refused to follow in MySQL’s footsteps for a long time, and because of that, they’ve avoided many of the pitfalls MySQL has fallen in to as their scope creeps larger and larger like an amoeba slowly devouring the edges of the enterprise market that used to seem so far from its original targets.

However, even the Postgres guys know that hacks may be necessary. As of May, 2008 they have given in and will produce a general purpose master/slave replication system. The message to the “pgsql-hackers” list has an air of reluctance to it..

Users who might consider
PostgreSQL are choosing other database systems because our existing
replication options are too complex to install and use for simple cases.
In practice, simple asynchronous single-master-multiple-slave
replication covers a respectable fraction of use cases, so we have
concluded that we should allow such a feature to be included in the core
project.

Its like they’re finally saying “ok we want more users, so we’ll include this thing that goes against our principles.” Personally I think this is great, as PostgreSQL is a nice RDBMS, and to be able to use it for small-medium scaleout just like MySQL is really quite exciting.

So, the moral of the story is, if you want your project to be successful, throw in some crap code. Otherwise your developers will be up on their high horses too long, and not down in the trenches getting things done.

It semeed so simple, just setup two memcachedb instances and point them at eachother. Instant fault tolerance, Right? If only it were so simple!

Its not entirely clear from the documentation how to setup memcachedb for fault tolerance. Here’s the procedures I’ve found useful.

• Set up replication right. With all due respect to Steve Chu, The docs aren’t really clear on how to setup replication. Its much simpler than it looks. Just run MemcacheDB as you would if it were standalone, but then add a combination of these 3 options:
  • You must have a -R line if you want to participate in replication. This is your hostname and port that listens for connections from other machines for replication. It is the same value that should be listed in every other machine’s -O.
  • a -O for *every* other machine that may want to replicate to/from this machine. I am sure there are situations where you won’t need these, but it makes re-syncing and elections more predictable. You won’t be able to re-sync “live” after a failure without -O options.
  • -M/-S are not required. If you start n machines without -M or -S, but with appropriate -R and -O lines, they will arbitrarily elect a master. If you run them with -M and -S, then the -M box will just be pushy and always elect itself the master, and the -S boxes will, likewise, always try to defer to slave status.
  • Lets say we wanted to listen for memcache protocol on port 45000 on host ‘node1′ and replicate to ‘node2′
  • Standalone: memcachedb -p 45000 -H /home/memdb/data -u memdb -N
  • Replication w/ elected master: memcachedb -p 45000 -H /home/memdb/data -u memdb -N -R node1:46000 -O node2:46000
  • Replication Master: memcachedb -p 45000 -H /home/memdb/data -u memdb -N -R node1:46000 -O node2:46000 -M
• Only the current master can accept writes. You can see which machine is the master with the ‘stats rep’ command. In v1.2.1 its shown as an environment id. Below st_env_id and st_master are the same, so this is the master:
  
stats rep
STAT st_bulk_fills 0
STAT st_bulk_overflows 0
STAT st_bulk_records 11
STAT st_bulk_transfers 3
STAT st_client_rerequests 0
STAT st_client_svc_miss 0
STAT st_client_svc_req 0
STAT st_dupmasters 0
STAT st_egen 3
STAT st_election_cur_winner 2147483647
STAT st_election_gen 0
STAT st_election_lsn 1/28
STAT st_election_nsites 0
STAT st_election_nvotes 1
STAT st_election_priority 100
STAT st_election_sec 5
STAT st_election_status 0
STAT st_election_tiebreaker 3676766282
STAT st_election_usec 69747
STAT st_election_votes 0
STAT st_elections 1
STAT st_elections_won 1
STAT st_env_id 2147483647
STAT st_env_priority 100
STAT st_gen 2
STAT st_log_duplicated 0
STAT st_log_queued 0
STAT st_log_queued_max 0
STAT st_log_queued_total 0
STAT st_log_records 0
STAT st_log_requested 0
STAT st_master 2147483647
STAT st_master_changes 0
STAT st_max_lease_sec 0
STAT st_max_lease_usec 0
STAT st_max_perm_lsn 0/0
STAT st_msgs_badgen 0
STAT st_msgs_processed 5
STAT st_msgs_recover 0
STAT st_msgs_send_failures 2
STAT st_msgs_sent 10
STAT st_newsites 0
STAT st_next_lsn 1/8916
STAT st_next_pg 0
STAT st_nsites 2
STAT st_nthrottles 0
STAT st_outdated 0
STAT st_pg_duplicated 0
STAT st_pg_records 0
STAT st_pg_requested 0
STAT st_startsync_delayed 0
STAT st_startup_complete 0
STAT st_status 2
STAT st_txns_applied 0
STAT st_waiting_lsn 0/0
STAT st_waiting_pg 0
END

  However, its much simpler, I think, to just try and store a value on an instance. If you get “STORED” back, then this is the master. If you get NOT_STORED back, this is a slave. If it blocks (timeouts are hard in simple scripts, I know.. perldoc -f alarm), you are in a “DOWN” state. The danger here is one of split brain where both nodes thing they’re the master.. but.. if they’re not talking, you have bigger problems!
• Out of sync slaves can’t READ either! This one bit us just the other day. Something ocurred where our slave wasn’t able to retrieve the latest log entries from the master. Because of this, it was reporting errors in replication. During this time, *all* commands blocked. We were relying on basic round-robin DNS for failover, thinking that memcachedb was simple enough, it was either “up” or “down”. Unfortunately, it was stalled on one box, so everything that hit that box blocked and timed out until we firewalled the port so connections wouldn’t succeed. We eventually had to stop the instance, copy a db_hotbackup from the master, then start it again. This still had to catch up from the point at which the db_hotbackup copies logs were checkpointed, which was (because we’re on v1.0.3) many hours before. While it was catching up, all commands (even stats commands.. which is disappointing..) blocked.
• Use a load balancer, not round robin. With that said, a load balancer is a far better solution then round robin. In this case, because the box was “up”, but failing to respond, we were at the mercy of the pecl memcache module’s definition of what was “up” or “down” for reads. A load balancer separates this logic out into monitors so the code can just connect to a virtual IP, or use some list of servers it is given.
• Even better.. just use a floating IP. MemcacheDB seems to scale to ridiculous levels with reads. Like, 400:1 read:write performance. Do you really need lots of slaves? Just having an IP that follows the master will give you fault tolerance. Its easy to determine if a box is the master. You can even do a ‘rep_set_priority 500′ to make sure a box stays the master as long as it has the IP. If you’re running on Linux, Good old Heartbeat is perfect for this. If you need to scale past the write capabilities of one box, then partitioning by using a stable hash algorithm on the keys is a far better solution than master/slave replication, and is already built in to pretty much every memcache client.
• Be careful with db_archive/db_checkpoint. This is mostly regarding v1.0.3, as I don’t know the impact of these commands on v1.1 or 1.2. However, it would seem that even with a replication policy of “ACK_ONE”, its still possible to purge logs that the slave needs. This may or may not be true (something else could have gone wrong) but it seems that running db_checkpoint/db_archive too aggressively seems to have broken our replication. There’s no reason to purge logs too often, so be wary when doing so.

Hopefully this will help other users who are starting to setup MemcacheDB and need fault tolerance.

So a few days ago, my big mean MySQL server started having problems that were very hard to explain. It was slowing down, taking a minute to run queries that usually take a few seconds, and Linux load averages were in the teens, despite having quiet disks (less than 0.1% cpu IO wait time) and plenty of RAM (128G for about 200G of data total…).

The developers were stumped. The other systems guys were stumped. So was I. But it still seemed ok. We found all sorts of things to point fingers at, but nothing made sense.

Then this Monday, everything came to a screeching halt. 3 second queries were taking 15 minutes. 30 second queries were never completing. The CPU’s were only a little busy. What gives?! This box has 8 CPU cores and 128G of RAM.. nothing can take it down, right?!

We threw our hands in the air and failed over to the active standby (the other side of our master<->master replication pair). Suddenly all was well. But something smelled wrong. We blamed some kind of bug in MySQL.

I spent all day trying to make Memcached more efficient, and trying to explain why suddenly this beast was felled by such tiny arrows as instantaneous queries that should have been cached anyway.

Oh wait, did somebody say cached? As in the MySQL query cache? I mentioned this in the #mysql channel on Freenode, and Mr. Eric Bergen (ebergen) from Proven Scaling immediately said something like “well duh, turn off the cache, moron”. I was dumbfounded. Shouldn’t it be helping us with all those tiny queries?

Well apparently not. This recent thread on the MySQL internals list talks about mutex contention in the query cache while it is *searched*, not just while it is updated. This is disasterous for an environment where thousands and thousands of tiny queries are being run constantly. Even with query_cache_type set to 2, or “cache on demand” mode, every query in the system must run through this mutex.

So, this morning when the standby box again cried for mercy, hitting max_connections and spinning all queries around in circles, I ran ‘SET GLOBAL query_cache_type=2′. Instantly the server became more healthy. I half expected to trade one problem for another.. with the server being consumed by tiny queries. But instead, these tiny queries did as expected, and took very little time to complete. And large queries against tables that change every second or 2 didn’t have to contend for the query cache, they just ran through like nothing.

So, it would appear that for any sort of multi-core installations of MySQL, the query cache is not only a waste, but a hazard!

Thanks again to Mr. Bergen. I would not have thought about that until he said it.

Anybody who is in IT in America, has probably experienced that sinking feeling when somebody somehow introduces the latest version of Microsoft Office into their organization. It usually comes in like some corporate ninja while you’re not looking. Whether its an application that your accounting department writes with the new version of Access, or that Outlook plugin that somebody locked in to, you have to deal with it.

The most frustrating part of this for me is never that people are going to use Office. Its not a bad product. Whats frustrating, is that every 3 or 4 years, Microsoft somehow gets people to pay $300-$400 per user. As somebody who has used OpenOffice since it was called “StarOffice”, this is perplexing. There’s even a high quality mac version called NeoOffice, in case anybody still thinks you have to have X11 installed to run OpenOffice on Mac. The file format problem isn’t even an issue anymore. Microsoft has has priced office so high, people stay on very old versions as long as they can, ensuring that even office to office incompatibilities are common.

So why not use it? OpenOffice is totally free, and has all the features that most users care to use in Office.

Oh wait.. except one. The email client. OpenOffice has no Outlook competitor. Calc is like Excel. Writer is like Word. There’s a powerpoint equivilent too. But no Outlook. StarOffice was written before email was really on the radar. But they’ve had enough time by now, why haven’t they solved this?

I’m sure some OpenOffice users are happy with Thunderbird. I’m not. It just doesn’t work very well. I’ve been using Linux with Gnome/Ximian/SuSE/Novell’s Evolution for years, and it just keeps getting better. Even for users of MS Exchange, IMAP access works, and Evolution actually implements the shared calendaring and address book of Exchange. On my mac, I use Apple’s excellent mail application. But on Windows, people are kind of stuck.

I recently had a friend who tried 4 or 5 email clients on Windows, trying to get away from constantly dealing with his “pst” files crashing or just going slow. He gave up in defeat. There’s just nothing. Another friend has all his email forwarded to his Gmail account. I’m not a big fan, but he is. I think this is just crazy that people woudl choose a webmail-only client for email.

Maybe I’m just being too hard on Thunderbird. I got all excited when I saw a blog entry that said OpenOffice 3.0 would compete with Outlook. But its just Thunderbird, with Lightning. At first I was deflated by this. But with Lightning, maybe Thunderbird will work well. I still think Outlook is lightyears ahead of it in terms of usability.

Just a few weeks ago, I found out that thread_concurrency’s purported magical effects at correcting MySQL’s concurrency limitations (especially in 4.1) were something of a myth. It was a post on mysql’s lists that alerted me to this. Apparently it only works on Solaris, Linux’s threading library ignores this parameter completely. This is not to be confused with innodb_thread_concurrency, which is quite useful in controlling the flow of transactions through InnoDB. I think the problem really lies in the fact that the default my.cnf example configs tell us to set thread_concurrency to the number of CPU’s*2. They fail to mention that this only matters on Solaris, though the manual is quite clear.

I think I set this parameter to 1, 4, and 8 trying to see if it would affect things positively or negatively on quite a few 4.1 boxes. I always just sort of assumed it was going to help prevent any sort of snowballing of server load if it ever got hit hard.

Yet another example where its important to RTFM!