FewBar.com - Make It Good This is the personal website of Clint Byrum. Any words on here are expressly Clint's, and not those of whatever employer he has at the moment. http://fewbar.com/ Fri, 16 Dec 2016 17:39:56 +0000 Fri, 16 Dec 2016 17:39:56 +0000 Jekyll v3.3.1 OpenStack's nova-compute's border is porous - We need to build a wall <p>In the beginning there was Nova. It included volumes, networking, hypervisors, and scheduling. Since then, Nova components have either been replaced (nova-network with Neutron) or forklifted out and enhanced (Cinder). In so doing, interfaces were defined for how Nova would continue to make use of these now-external services, but nova-compute, the place where the proverbial rubber meets the road, was left inside Nova. This meant that agents for Cinder and Neutron had to interact with nova-compute through the high level message bus, despite being right on the same physical machine in many (but not all) cases. Likewise, some cases take advantage of that, and require operator cooperation in configuring for certain drivers.</p> <p>This has led to implementation details leaking all over the API’s that these services use to interact. Neutron and Nova do a sort of haphazard dance to plug ports in, and Cinder has drivers which require locking files on the local filesystem a certain way. These implementation details are leaking into public API’s because it turns out nova-compute is actually a shared service that should not belong to any of the three services, and which should define a more clear API which Nova, Cinder, and Neutron, should be able to use to access the physical resources of machines from an equal footing.</p> <p><a href="https://review.openstack.org/#/c/411527/">We’re starting a discussion in the OpenStack Architecture Working Group</a> around whether this is creating real problems, and how we can address it.</p> <p>What I think we need to do is build a wall around nova-compute, so we can accurately define what goes in or out, and what belongs specifically in nova-compute’s code base. That way we can accept the things that should live and work permanently inside its borders vs. what should come in through an API port of entry and declare its intentions there.</p> <p>But before we can build that wall, we need nova-compute to declare its independence from Nova. That may be as much a social challenge as a technical one. However, I think once we complete some analysis, and provide a path toward a more sustainable compute service, we’ll end up with a more efficient, less error-prone, more optimizable OpenStack.</p> <p>If you’re interested in this, I recommend you come to the next IRC meeting for the <a href="https://wiki.openstack.org/wiki/Meetings/Arch-WG">Architecture WG</a> , on January 12, 2017.</p> Fri, 16 Dec 2016 00:00:00 +0000 http://fewbar.com/2016/12/mr-nova-build-that-wall/ http://fewbar.com/2016/12/mr-nova-build-that-wall/ openstack architecture nova neutron cinder microservices OpenStack OpenStack needs an Architecture WG - Because we all can't be Gaudi <p><a href="https://en.wikipedia.org/wiki/Antoni_Gaud%C3%AD">Antoni Gaudi</a> designed one of the most beautiful things I have ever seen in my life, the <a href="https://en.wikipedia.org/wiki/Casa_Batll%C3%B3">Casa Batlló</a> in Barcelona. <img src="/images/Casa-Batllo-Barcelona.jpg" alt="Casa Batlló" /></p> <p>While touring the site, the audio tour guide explained multiple times that this entire site, from the basement to the roof, had no written detailed plans. Gaudi had to supervise every aspect of the construction, so that it was exactly the perfect masterpiece it is today. Gaudi is truly a legendary human being, and one of the greatest architectural minds in history. This means that he produced masterpieces, but it also means they can never be duplicated, are nearly impossible to improve, and are even quite difficult to maintain.</p> <p>And it is now, while I’m stuffed into “The giant metal tube” (Thanks <a href="https://twitter.com/robynbergeron">Robyn</a>), that I’m enjoying a rare moment of very clear thought after an <a href="https://www.openstack.org/summit/barcelona-2016/">OpenStack Summit</a> related to “God’s Architect”.</p> <p>The most important session that I attended, and led, was <a href="https://etherpad.openstack.org/p/BCN-architecture-wg">a fishbowl to discuss the Architecture Working Group</a>. Even with <a href="http://lists.openstack.org/pipermail/openstack-dev/2016-June/097657.html">mailing list threads</a>, <a href="https://review.openstack.org/#/c/335141/">review cycles</a>, <a href="https://etherpad.openstack.org/p/architecture-working-group">etherpads</a>, and <a href="https://wiki.openstack.org/wiki/Meetings/Arch-WG">meetings</a> behind us, it was clear from the discussion that there was some broad misunderstanding of what we were doing and why we want it to be a part of the community. We definitely used the time well and I think scraped away some of the boilerplate “we’re a team here we are” boring stuff and dug down to what it is we want to do.</p> <p>In a nutshell, we’re here to look at the Nova, Neutron, Oslo, et. al masterpieces, and write down the plans that were never created before. While there are change specs, and manuals for much of it, quite a bit has no binding theory of operation. As a result, there is quite a strong cargo cult inside OpenStack, leading to forward progress without understanding. This creates an OpenStack where there are more people writing code than can understand code, which complicates every aspect of developing and even operating it.</p> <p>We want to make sure that OpenStack can continue moving forward, and so, we need to write down how things work now, record the current theory of operation, and then collaborate on improvements to those theories and the actual implementations.</p> <p>Without the Architecture Working Group, I’m sure OpenStack will remain valuable and even be maintainable. However, I’d like to see it continue to evolve and thrive even faster, and I’m proud to be working with people to try and provide a safe place to make that happen.</p> Sun, 30 Oct 2016 00:00:00 +0000 http://fewbar.com/2016/10/openstack-architecture-wg-because-we-all-arent-gaudi/ http://fewbar.com/2016/10/openstack-architecture-wg-because-we-all-arent-gaudi/ communication openstack summit barcelona architecture Life OpenStack Open Source People Communicate <p>As I sit here preparing to cross the Atlantic, I am pondering on what we’ll do <a href="https://www.openstack.org/summit/barcelona-2016/">in barcelona</a>.</p> <p>This will be my… (stopping to count on fingers.. running out of fingers…) 11th Summit. Back in the Essex days, I was communicating about <a href="https://jujucharms.com/">Juju</a> whilst working for <a href="http://www.canonical.com">Canonical</a>. It was a fantastic experience to see some of the same communication methods we had used at <a href="http://uds.ubuntu.com/">Ubuntu Developer Summits</a>, and new ones, coming together to form this massive community.</p> <p>This will be the last summit where we ask <em>every</em> technical contributor to join the fray. An evolution of the process is under way, which has been called the <a href="http://www.openstack.org/ptg">Project Teams Gathering</a>. So this may be the last time we do it the way it has always been done, with technical contributors mingling with business folks at the OpenStack Conference. There are some <a href="http://lists.openstack.org/pipermail/openstack-dev/2016-October/105524.html">concerns about this</a>, some even <a href="http://lists.openstack.org/pipermail/openstack-dev/2016-October/105260.html">expressed by me</a>. But I trust those who have been formulating this plan to be dilligent at iterating on it to improve our throughput.</p> <p>And the reason I trust them is that I have seen one constant throughout successful Open Source contributors. We all communicate. I take it for granted how well we actually communicate, given how distributed we are, and how few shared objectives we have. But I think what separates a pet project on github from an Ansible or OpenStack sized project is contributors who communicate early, and communicate often.</p> <p>So, I am very much looking forward to this upcoming summit. I expect that we will all do our best to communicate by listening, recording, reflecting, and adding our voics. But I am also quite excited to see how the new format works out not only at the PTG in February, but also the <a href="https://www.openstack.org/summit/boston-2017/">next Summit in Boston</a>.</p> Sat, 22 Oct 2016 00:00:00 +0000 http://fewbar.com/2016/10/opensource-people-communicate/ http://fewbar.com/2016/10/opensource-people-communicate/ communication openstack summit barcelona Life OpenStack Mitaka, you taka, we all taka in Tokyo <p>OpenStack is weird.</p> <p>In past professional lives, when I've taken on projects bigger than myself, I've always had to learn to make peace where there was conflict, and drive past all of the fun houses and arcades to get to an even better, happier, cleaner place at the end of that long dark road. But OpenStack makes that weird because sometimes you literally end up in arcades and fun houses.<a id="more"></a><a id="more-657"></a></p> <p>Take Robot Restaurant. Why wouldn't you put a tourist-aimed show highlighting robots, manga, and taiko drums in a basement in the middle of the red light district? Same question, why not <a href="http://lists.openstack.org/pipermail/openstack-dev/2015-October/076918.html">post to a thread </a>about experimental scheduler rewrites? Like the show, these threads are just a never ending sequence of retorts, with the tanks defeating the panda, and the giant snake defeating the tanks, and the giant laser robot dragon defeating the snake, and then robot ballet, and eventually you forget that the point of this was to save nature and instead you just want <span style="text-decoration: underline;"><a href="http://lists.openstack.org/pipermail/openstack-dev/2015-October/076825.html"><strong><span style="color: #ff0000; text-decoration: underline;">MORE DRAGONS</span></strong></a></span>. <a href="http://www.shinjuku-robot.com/pc/"><img class="alignnone" title="MOAR DRAGONS" src="https://lh3.googleusercontent.com/_6g9nEzO2oQRy3i_cD2yKawnzJYJdrTFAQ2-3DWJCk628u7f6gi_WyiXhVWPSOiecxefC2PCjymgpcNmAcQdutkC_UZ_YxJHfIB60vBuQFkqeuvMFMXOz2SvXKMl9FsE9QLN_GJ68MVB7pvpS-QqCxjsb0kqEzbDexWjNPgFiJ0VMYQjfB-QzyePUctioj5O27tB3Aq3e97f7NkltMK6cXxZ2OUeAwBegt0utLgVzAjVuRX3l9EKUSY8BZepdcgFyLp4xiNav3_5mu6z_K-BmnbgxIX81P87lE5dvo-4WnrJHuiO8ONGUJ1oVb84Uabc0Ow_Q6NuPBJNwikAoAHWJ09DpnxOTcUyb0BPWKe4j_g1DTMNngLpNnTkHSd9_kZJsnTpabUWYFJevCJDOfAabc8hvz3D79RzwYqwDARo-Qn6nFnw5kKP-5q_xRiGbRZyc-mXmfnojAo8rmjzx4V5u-i3ealeQPLoGtDuJXbQFjV39_GyudCJHSocTRLsQIIpcDJF2IUnZnwYVo7d1PiyfugsWTJFNf0d7gAKUDihlGoa=w1920-h1032-no" alt="" width="700" /></a>Whatever the point of this, once you escape it, you realize that while that idea may have been on the road to the end goal of scaling OpenStack up and out, it probably wasn't the next step. And beside that, we haven't even gotten to the summit yet!</p> <p>OpenStack is weird, because a lot of the time you read the code, you think you know what it means, and then you have a conversation whilst walking by a Koi pond, and it shatters what you thought you knew.</p> <p>For instance, I had always assumed that cells were in fact just sharding the control plane for the sake of scaling out the control plane. But whilst strolling past Shinto shrines in the hotel garden in Tokyo, John Garbutt explained to me that cells are so much more. They also are about scaling at the rate that current cells users like to scale their business. The whole control plane that goes along with that scaling unit is just a handy layer around new servers which has some nice aspects like being able to test this new gaggle of servers a bit before throwing the wild and crazy users onto it. This helps me understand why single cells were never pushed beyond a few hundred servers worth of scale now. It also makes me want to double-down my efforts to scale cells up quite a bit, as some of us would like to scale our business in smaller and larger batches. Inner peace achieved.</p> <p>Also, sometimes you think "well certainly people will have just accepted the JVM as their RAM's eventual keeper", and then you find out that there are entire ops departments that flatly refuse to support Java in any way, and if you asked them to run <a href="https://zookeeper.apache.org/">Zookeeper</a> (a java based distributed lock manager++) to help coordinate your python based OpenStack services, they'd probably start looking for a new way to run a cloud. Point one for <a href="http://consul.io">consul.io</a> and abstractions like tooz that will let us experiment and run other things written in the new crop of languages which I like to call "notjava". This isn't a fun house on the way to scaling, this is the arcade. This arcade is similar to the one we wandered into in Shinjuku, where there are many, many old boring looking games that we've all played before, are still fun, and actually work great, and a few new shiny games that look fantastic, but are supremely crappy and mostly just take your money. My advice: either play street fighter (Zookeeper), or rock that new Taiko drum thing (Consul).<a href="http://consul.io"><img class="alignnone" title="Taiko Drums as a Service" src="https://lh3.googleusercontent.com/uBj0FEomq76TpdneR5iqsoVREJhXSbyX2p_hWbIoXPz9VexkmokxW7UvKPFlX493zOSF5DcedUhNVvh90L_twlntqH9CZOTZn5XxKM1KUMqrGV5wbz_jUA0TyYST5pRnHTqIrJ0axDLAKwHaQij2p7WGDg-CCgDS7m9mlIBj6bsXjefAGTaoqYOBkPiquhEXEbN87nLL47ynoNAiJY6ee_m1lGKfBiGR8-D20n4I71q7gjLlOMsqA3QaSc50SEMoOYKS4p2eg1SSQxzw-jWeb6ZUNPP2rJeDU9-RyCFXmEzKQiTo3jretYIqUyApWZqsV_SkZnO7NA7l0vgvY3woZN26rhQLkYIa31kS6KnjsnhzquPgXmz7A55hM1i2MUK8vpM_iH2Ti3XHYiyd5vYHntumno_SpTiuLseCJS7dhHSrxBQsSAe5DU2duE4lQC-A5TRsiPqp0VNTN3nqHTbC4y_bTzpVhNCwz2EEUzT7qjzTWw1TBd8FKyv_ODYM2JTf1oouGBAp14qEBvQnztgqjx5F7wTtUuSKn7cJ8pS3HitZ=w829-h1105-no" alt="" width="700" /></a>Whatever you do, <strong>make sure you're doing it because it gets you out of that arcade as soon as possible</strong>. You're not making a distributed lock thing, you're making an OpenStack!</p> <p>And after wandering around all of these fascinating places, and sampling all of these grand ideas, one realizes that all you really want is something fresh and new that hasn't been sitting around getting older. So instead of chatting into the wee hours over Yoichi Whiskey and so many bottles of sake, you turn in early, wake up at 0600, and wander alone into the fish market for sushi breakfast. And there, you realize that you do not need to wait for others in OpenStack. Certainly, tell them where you're going, and why you're going to bed early and doing things that might confuse them. And if they ask to come, be polite and offer them a way to join you. But ultimately, get your sushi. Enjoy your <a href="https://review.openstack.org/#/c/230183/">fresh ideas</a>, and enjoy implementing them. Sure, you might wander into the wrong subway station, take 3 stops in the wrong direction, even drop a receipt, shaming your entire nationality when you force a thoughtful Japanese person to pick it up and inform you of your transgression politely. But ultimately, you will find your sushi, and as a result, your belly will be full, and commerce will continue, whether you participate or not.</p> Tue, 03 Nov 2015 18:52:34 +0000 http://fewbar.com/2015/11/mitaka-you-taka-we-all-taka-in-tokyo/ http://fewbar.com/2015/11/mitaka-you-taka-we-all-taka-in-tokyo/ Scalability openstack mitaka tokyo zookeeper consul cells sushi Life Scalability OpenStack One small step for an open source developer, one giant leap for OpenStack kind <p>The great thing about working in Open Source is that often one can drive a single mission through multiple organizations to an end goal that makes the most sense for the community. I hope that some find the changes I made to OpenStack while at HP useful both inside HP, and outside. They were a fantastic employer and I'm eager to see where HP Enterprise goes from here.</p> <p>That said, I'm extremely excited to be joining IBM to work on OpenStack from a different angle. We'll be doing some really cool things, and I just can't wait to get the ball rolling! In fact I won't have to wait as today is my first day, and I'm here in North Carolina at the Research Triangle Park to begin my assimilation into the borg.. er.. to be oriented for employment at IBM.</p> Mon, 21 Sep 2015 11:00:24 +0000 http://fewbar.com/2015/09/one-small-step-for-an-open-source-developer-one-giant-leap-for-openstack-kind/ http://fewbar.com/2015/09/one-small-step-for-an-open-source-developer-one-giant-leap-for-openstack-kind/ cloud openstack ibm hp Life OpenStack The Rocket Ship to Havana - OpenStack Summit Spring 2013 <p>I started this blog post as "The Road to Havana", but immediately it struck me that the term "road" just doesn't do this summit justice.</p> <p>Day 1 was full of <a href="http://wiki.openstack.org/Heat">Heat</a> for me. As a recent addition to the Heat core reviewer team, it was quite helpful and a pleasure to meet most of the other developers in person. This happened about 10 minutes before our first session together. It never ceases to impress me how easy it is to meet somebody in real life whom you've been corresponding with over only IRC and email. In this case, I felt face to face contact just added warmth and depth to already warm and friendly professional relationships.</p> <p>Heat is on a path toward being a really great solution for managing large application deployments in OpenStack clouds. <a id="more"></a><a id="more-608"></a>Six months ago I was focused on <a href="http://juju.ubuntu.com/">Juju</a> as a part of that discussion, and Heat was just this little incubating <a href="http://aws.amazon.com/cloudformation/aws-cloudformation-templates/">AWS CloudFormation</a> compatibility engine. Juju has been focusing on a rewrite, which I think is a mistake the project will likely be regretting for a long time. Meanwhile, Heat has turned into a project to gather effort around making orchestration and high order control services built in features of OpenStack.</p> <p>We had a few discussions about scaling and performance, including <a href="https://blueprints.launchpad.net/heat/+spec/concurrent-resource-scheduling">concurrent launching of resources</a> and <a href="https://blueprints.launchpad.net/heat/+spec/multiple-engines">scaling out the Heat engine</a>. These were pretty low level discussions involving mostly the developers already involved, and like any good summit, we had contributions of ideas from many attendees, and solutions seem clear.</p> <p>One thing that was clear to me before the summit was that a storm was brewing with regards to how Heat users would express their application deployments.</p> <p>From some of the larger "enterprise" focused vendors, my own employer (HP) included, comes <a href="https://blueprints.launchpad.net/heat/+spec/tosca-support">a recommendation to support TOSCA</a>. This is a really large, <a href="https://www.oasis-open.org/committees/tosca/">wide ranging standard</a> that only partially even applies to Heat's current scope. However, it seems like a natural fit for TOSCA users to use something built into the cloud to deploy their applications.</p> <p><a href="https://wiki.openstack.org/wiki/Heat/DSL">Rackspace contributed a spec</a> and will likely contribute some existing code for a native format that Heat users can use. This format is more narrow in focus than TOSCA, and I think has real potential. It was in need of a good name, so I dubbed it "Heat Orchestration Template" during a session. HOT seems to have stuck for at least a while, though "Heat DSL" may also end up being its name.</p> <p>There were also some interesting discussions around auto scaling going into Heat. I think it is understood by everyone that this is a different interface to similar, but not identical, control services. Because having competing control services can be problematic, it makes sense to have them all live in Heat for now. Rackspace has committed some developers to getting the problem solved and code published during this cycle, so we are all excited to see their work.</p> <p>On a larger scale, the "<a href="http://github.com/tripleo">TripleO</a> (OpenStack on OpenStack)" program that our team at HP has been driving under the tutelage of Robert Collins got extreme amounts of exposure. With Bare-Metal nova landing in OpenStack for grizzly, and Heat fully integrated, nearly all of our components are blessed and thus prepped for the usual community contribution fire hose that OpenStack brings. The other pieces, <a href="http://github.com/stackforge/diskimage-builder">diskimage-builder</a>, <a href="http://github.com/stackforge/os-config-applier">os-config-applier</a>, and <a href="http://github.com/stackforge/os-refresh-config">os-refresh-config</a>, are all in StackForge and will be absorbed into OpenStack as we flesh out the TripleO effort.</p> <p>After the day 1 "Heat firehose" of sessions, I spent a lot of time just communicating to various interested parties about what it all means and where Heat and TripleO fit in with the OpenStack ecosystem. My talk about using Heat to manage OpenStack was well attended and there were some great questions. <a href="https://region-a.geo-1.objects.hpcloudsvc.com:443/v1/42841271810042/Docs/OpenStackSummit_HeatOpenStackManagement.pdf">Slides are available here</a>. <em>Update 2013-04-26: videos have been posted!</em></p> <p><iframe width="560" height="315" src="http://www.youtube.com/embed/O-SdNaFq2CQ" frameborder="0" allowfullscreen></iframe></p> <p>One big surprise was to see the tool chain of TripleO mentioned in the keynote by Mark Shuttleworth. The concept we have subscribed to is to copy the Unix tool method. One needs to support incremental adoption of tools and thus one tool should do one job well. To demonstrate this, I have used this image a few times:</p> <p><img class="alignnone" src="http://spamaps.org/tripleo-gospel.svg" alt="" width="1052" height="744" /></p> <p>This is not a controversial pattern, and has proven successful for solving large problems in computing before. So, imagine my surprise when I saw that Mark Shuttleworth was arguing against it in a slide, showing Juju trying to do all of these jobs, and suggesting this was better than our (unfinished) effort to break the problem up and write a single tool for each distinct task. I spoke with Mark afterwards, and I think we will just agree to disagree on the approach. Having been quite involved with Juju since very early in its existence, I am still rooting for Juju to accomplish what it has set out to do. However, I am troubled by the lack of focus and unclear integration path.</p> <p>I want to end on a happy note. This was my first OpenStack summit as an employee of <a href="http://hpcloud.com">HP Cloud Services</a>. I want to thank HP for the opportunities provided to me, and also for sponsoring OpenStack. There are so many talented and focused people at HP, I think we're going to do some really amazing work together.</p> <p>Overall, this was a great OpenStack Summit. The Havana cycle will see OpenStack growing more features and improving the deployment story, which is good for everyone. I am particularly excited about the proposition of gathering for "OpenStack 'I'", which will be in Hong Kong! So, rock on Stackers, can't wait to see you all again in the fall!</p> Mon, 22 Apr 2013 21:00:58 +0000 http://fewbar.com/2013/04/the-rocket-ship-to-havana-openstack-summit-spring-2013/ http://fewbar.com/2013/04/the-rocket-ship-to-havana-openstack-summit-spring-2013/ ubuntu cloud juju openstack tripleo heat Cloud OpenStack