Heroku

Heroku has hit Hacker News‘ top 10 multiple times in the last few days. RapGenius has been unhappy about the performance of their Rails application and posted an in-depth analysis of performance issues on Heroku’s Bamboo and Cedar stacks.

RapGenius unearthed issues in Heroku’s stacks that made it scale badly. Heroku posted a post-mortem a few days later and promised some fixes.

There are some interesting lessons to learn from this tale.

Monitoring is the new debugging and the new profiling. If you’re watching the wrong things, you’re blind to what happens in your systems.

Heroku choose badly what performance indicators to track. Instead of focusing on holistic end-to-end measurements, they assumed that their stack behave as they thought it would and they measured only specific indicators in their routing mesh. They were the wrong things to focus on and it made them blind.

“Over the past couple of years Heroku customers have occasionally reported unexplained latency on Heroku. There are many causes of latency—some of them have nothing to do with Heroku—but until this week, we failed to see a common thread among these reports.”

Heroku designed their original stack to solve a specific problem. Rails was single-threaded and Bamboo promise was to provide horizontal scalability given that constraint.

The Bamboo stack does not support concurrency. On Bamboo, a single process can serve only one request at a time. To support this architecture, Heroku’s HTTP router was designed to queue requests at the router level. This enabled it to efficiently distribute requests to all available dynos.

The Bamboo router never used a global per-application request queue. The router is a clustered service where each node in the cluster maintains its own per-application request queue. This is less efficient than routing with a global request queue, but it is a reasonable compromise as long as the cluster is small.

At scale, things get qualitatively different. Things that happen once in a zillion are rare when you’re small, but happen every second if you serve a zillion of requests per second. When you’re debugging, you’re once again doing things in small scale. Thus, debugging and profiling without live traffic doesn’t really help at scale.

The inefficiency in request routing gets worse as the number of routers increases. This is essentially what’s been happening with Rails apps running on the Bamboo stack. Our routing cluster remained small for most of Bamboo’s history, which masked this inefficiency. However, as the platform grew, it was only a matter of time before we had to scale out and address the associated challenges.

Once you scale out Bamboo’s original design, their “intelligent routing mesh” effectively turned into a “random routing mesh” plus additional routing costs.

The lesson here is that as you scale out, things change not just quantitatively but qualitatively. If you assume your system behave in a certain matter and fail to measure it properly, you’ll have quite some surprise when it changes under your feet.

I think Heroku will recover from this quagmire. If they play their cards right, the company will come out strengthened. They handled the issue in a professional way from a PR point of view, although Hacker News acceptance of their explanation has been a bit lukewarm as people wait for a technical fix. It’s going to be interesting to see how they proceed.

On one side, developing better routing algorithms is certainly possible. On the other side, once you have a more performant platform, they will have a more appealing product.

The key issue is that developers cannot really be oblivious of the differences between running a few servers and running a few hundreds. Heroku business proposition is that their customers can develop applications, while remaining oblivious of their scale. It didn’t work out.

Designing applications with bounded latency is a good idea. If you have an application with many tasks that take microseconds and a bunch that take seconds, you’re in trouble at scale no matter the routing algorithm that you’re using. It’s much better to offload long-running tasks to batch workers and check the status in subsequent calls, or use push-based technologies like WebSockets for the same purpose.

The Tail at Scale

Jeff Dean and Luiz André Barroso, two of Google’s most talented engineers, published an interesting article in the Communications of the ACM, February 2013: The Tail at Scale. Its subtitle is “Software techniques that tolerate latency variability are vital to building responsive large-scale Web services“.

The article’s key insights are prophetic of what happened at Heroku and RapGenius, and how to solve them:

• Even rare performance hiccups affect a significant fraction of all requests in large-scale distributed systems.
• Eliminating all sources of latency variability in large-scale systems is impractical, especially in shared environments.
• Using an approach analogous to fault-tolerant computing, tail-tolerant software techniques form a predictable whole out of less-predictable parts.

The biggest hurdle Heroku faces isn’t solving the technical problems of its platform, it is educating its users about the correct techniques to evolve small-scale systems into large-scale software.

Antifragile

Aside from computing, I have heard good things about Nassim Talebs’ latest book, Antifragile. When a friend’s birthday came up, I picked up two. One as a present for him, one as a present for me.

It’s latest work is all about how to build things that thrive in instability and disorder, instead of fail. The contrary of frail is not robust, but what Talebs called “antifragile”. It’s a bit of a thick book, perfect for a long haul flight over the Atlantic.

Related articles

• High Scalability Blog: Google: Taming The Long Latency Tail – When More Machines Equals Worse Results
• Microstorm in a teacup: Are you suffering from the long tail effect?

I want to challenge this discriminating notion about singles. Demonstrating, with a little math, that it’s not a problematic male trait but just a consequence of the situation.

By the way, it’s not just males that are weird when they live alone:

Let’s consider a couple of order-challenged singles, whatever is their gender or sexual orientation. Let’s assume that, when they are dwelling alone, these poor souls cannot really control what’s going on at home so it turns out into total chaos (until a mother storms in for seasonal cleaning, the deus ex machina you can count on in the comedy that is called life). In math-speak, they create chaos at rate λ and they clean up at rate μ, with λ > μ.

When they are paired together, they share some chores: there is only a single kitchen to clean, a single bathroom to wash, etc. Let’s say that it adds up to a difference δ of their cleaning activity when they lived alone. Provided that they aren’t completely hopeless (δ/2 > λ-μ), here magic happens. Their new shared troll cave turns into a passable human dwelling:

Chaos is cumulative. A minor twist in the parameters can result in huge differences.

Mathematica code for the graph above:

With[
 {\[Lambda] = 11, \[Mu] = 10, \[Delta] = 2.5, n = 100},
 ListLinePlot[
 {
 Legended[
 RandomFunction[QueueingProcess[\[Lambda], \[Mu]], {0, n}],
 "Single"],
 Legended[
 RandomFunction[
 QueueingProcess[2*\[Lambda] - \[Delta], 2*\[Mu]], {0, n}],
 "Couple"]
 },
 Filling -> Axis,
 PlotLegends -> SwatchLegend["Expressions"],
 ImageSize -> Large]]

Do you think it’s just a funny story? Replace “people, stuff and shared chores” with “servers, queries and caches”.

When you grow up professionally as a lone player, blogging becomes less of a mean of self-expression and more of an exercise of personal branding.

The Hero

I spent most of my early career working part-time for a trusted, small company while doubling as a contractor in the other half of the time. It was a good bargain for both of us: the part-time job guaranteeing a steady income, while the contracts giving me the thrill of applying new knowledge to interesting projects. One side built on the other, and viceversa.

When you rely on a steady stream of contracts to grow professionally, you quickly realize the benefits and the truths of personal branding. Presenting your best side becomes a necessity, living up to the image you project becomes a matter of trust and a requirement to keep going on.

“I am Winston Wolfe. I Solve Problems“, this was my motto for a long time. Lupi, my lastname, actually means wolves in Italian.

Ruthless, precise, professional… at least on the outside. It was fun being Winston Wolfe. I remember a long time ago, it must have been around 2002 or 2003, when a colleague and I went to Milan to install our CMS in a big branch of a multinational bank. For us at the time, it was a big deal. We had a month from start to finish, from negotiating price to delivering the final code. It wasn’t easy for a two developers shop. It had to talk in real-time, during a web request, with a database on big iron mainframe and we have never seen one before. I spent the last week doing all nighters to make it work reliably.

It was a make or break situation. I was so scared about issues that I brought with me my HD backups, to be sure that even if my machine died I would be able to deliver. We did it and it worked without a glitch for years, but that day I was falling asleep in the clients’ office… my colleague kicked me and sent me to for a quick nap in a stall of their bathroom. I don’t know if they ever noticed!

When we were going back home, we stopped at a restaurant and fuel station on the highway. We ate a couple of sandwiches and were away from the car for no more than 5 minutes, but when we came back we found it open and everything was gone: the laptops and my precious backups. No customer data was on those machines but we lost all copies of the source. We had to ask for a copy of the installed software, with an excuse, to recover it. Luckily, it was written mostly in Python.

Playing the hero programmer is fun, but tiring. It’s not the best way toward a better future.

The worse aspect is that you grow the stupid instinct that you’re a great programmer because they have to call you to accomplish stuff that all those people in the big companies can’t do. You judge them without understanding what are their operational constraints and why they are valuable for their organizations. It’s a poisonous idea. If you’re smart enough, you realize how fallacious that thought is and you grow up. If not, you condemned yourself to be an hired gun forever and live a precarious life.

The Virtuoso

I moved on, but not yet completely. If you asked me what was my job a handful of years ago, I’d have answered: “I write poems for machines“. At that time, I had been writing code for startups for several years. After dealing on and on with the same tools and techniques for a while, you don’t just want to write good code, you want to write elegant and astounding code. You start to draw arabesques with your class diagrams.

Being a Virtuoso is a bad habit because it makes yourself irreplaceable. If you’re in a small startup, it means it cannot grow fast enough to survive. If you’re in a bigger company, it means you can’t be promoted and you become a liability, a potential bus target.

In addition, you’re still as stupid as before. Just in a different way.

Shu Ha Ri

You eventually grow up from it. It’s much like the martial artist concept of Shu Ha Ri:

• Shu (守, protect, obey), you first learn the fundamentals and the techniques. Your style is mechanical and methodic, filled with predefined forms.
• Ha (破, detatch, digress), then you break with the tradition and experiment. Your style becomes an arabesque, a canvas to try interesting ideas.
• Ri (離, leave, separate), at last you find your way and things become natural. Strangely, more often than not, your style outwardly resembles your first steps, but now there is no clinging to sketched forms, all the roughness is gone. All the superstructures you experimented with are reduced to their minimalist forms. Tradition is trascended, yet it acquires new meanings and provides new insights.

There are good reasons why our coding style guides are so strict and constrictive, or why Go is designed the way it is. Both are optimized for coding in a large corporation, where individual programmers come and go, contributing to a code base so huge that it requires its own new class of tools. They are key enablers that make it possible to work in humongous projects, whose design no single person can completely keep in their head.

The Big Corporation

So, why am I struggling? It’s not a new experience. I have worked in these settings before. I enjoyed and thrived in similar places!

The first novel problem working for an Internet biggie is that now you don’t have just to care about your personal brand, you have to think about the corporate one. You can’t talk about the future of your industry, even if it doesn’t touch your company at all and you’re just a random new hire, because someone could misinterpret your words for new directions of your corporation. You have to think not just about the effect of your words, but what they would mean after that new product goes live, the one that the company will release next quarter and that you don’t even know about. It’s taxing and many people shy off it and shut up once they get hired.

The real cruelty, at least for me, is just how much open my current employer is on the inside, and how fiercely secretive it has to be on the outside. Bondage is not just an appropriate metaphor for our coding style guides, but also for our NDAs and confidentiality clauses.

When you join a company like Google, you finally feel matched by peers if not superior developers instead of the old feeing of being first of class. Far from being off putting, it’s refreshing and provides extraordinary motivation: “New stuff to learn! Amazing!”. It’s almost barbarous that we can’t talk about all these shiny things, but there are good reasons.

Chiefly: there are those who talk and those who do. Which one do you want to be?

The conscious bit clearly agrees, but the old habits of an inflated ego still ramble and grumble in the last shadows of the past.

Once you develop the instinct of an individualist and a closet extrovert, working for such a Big Corporation feels a bit like a Faustian bargain. You obtain access to great knowledge and resources, but you can’t share it and contribute to the world immediately. You can build on it, but it’s a collaborative effort. Everything needs to be vetted, remixed, corrected by others and it must always contribute to the well being of the organization that gives you these copious supplies in the first place. It’s clearly a requirement to preserve this virtuous process of creation, but the individualist struggles with these corporate needs. It is only the fact that this arduous process turns the rough stones of your contributions into the shiny and honed gems of great products, that you’d never be able to do alone, that makes it bearable. Only at the end of this process, if you’re lucky and you’ve been an important cog in the machine, you’ll have your five minutes of fame.

A wise guy once told me that he switched careers, from developer to management consultant, when he realized that building good software is much more about people than it is about computer science. Broadening subjects is not only useful, to keep this blog going on, but also beneficial for its very main topic. I hope you’ll enjoy the new course.

It’s a pity because I would have so much to tell you. When your other computer is a data center, things become interesting.

It’s not that I won’t touch software development ever again, but I’ll have to tiptoe around the most interesting bits.

1. The Deification of Hugo Chávez, by Xavier Marquez (Abandoned Footnotes), a Venezuelan who teaches political theory and political science at Victoria University of Wellington.
2. A long essay on interaction rituals, by Randall Collins, professor of sociology at University of Pennsylvania:
   

   IR theory is an explanation of what people will think, as well as what they will do. At any particular moment, people are speaking certain words or thinking certain thoughts; the thoughts that go through one’s head are internalized from previous talk with other people; more innovative thoughts are assembled out of the ingredients of verbal ideas already internalized. The world is a network of conversations, and what people think at any point in it is a product of what has circulated in previous conversations.

   It has interesting applications for entrepreneurs, companies and startups. In my personal experience, the most successful companies I worked for where the ones who had the best organizations and cultures. They also were the most innovative and the less competition-shy ones, but that was as much an effect as it was a cause of their success.

3. My friend Ilaria Mauric reports her experience attending a UX course at Cooper U (in Italian): intro, day 1, day 2, day 3, day 4.

Interesting talk.

One of the best thing of working at Google is all the TechTalks that you can attend or watch. Just a little bit trickles outside.

A lecture by Alexander Gallaway on Deleuze’s Postscript on Societies of Control. Recommended if you’re interested in philosophy and critical theory.

Deleuze poses the basis of a critique of contemporary societies, the kind of stuff that Jaron Lanier loves to talk about.

I think hearing critics is the only way for a discipline to progress in a sane way and prosper. IT has way too few intelligent critics, and a plethora of stupids both among its cheerleaders and luddites.