It is not necessarily so. Don’t get me wrong, I don’t pretend to be able to circumvent it in any way or outsmart the math giants of the past. The naive way to put together count-min sketches and particle filters is bound to hit this wall, but if we are a little bit smart we can use the central limit theorem itself to guide us around these perilous shores.

What is the problem? The core concept of count-min sketches is to use a set of hash functions to distribute hits on various counters, relying on the fact that good hash functions will have few collisions. The counter with the lowest number of hits will be an upper-bound approximation of the true frequency for a given item.

Count-Mean-Min Sketches are a bit smarter and incorporate the number of expected collisions in the estimation itself.

When we replace counters with statistical models, like particle filters, we are replacing integers with random variables.

What’s the deal with adding together independent random variables? Under very mild conditions, the result is the normal distribution (if the variance of the independent variables is finite) or one of the other well known and well studied distributions that you can find in literature (if the variance is infinite, such as in power-laws).

The key here is the word independent. That’s surely the case if we choose a set of random hash functions in the same way we would do with normal count-min sketches. We can be smarter than that. If our items are not opaque labels for the object they stand for, but instead have an internal structure, we can use that internal structure to drive the selection of which particle filter to update.

In simpler terms, we can use the attributes of our items to cluster them into specific, non-random categories. Let these attributes drive our hash functions: if they are effective properties around which we can cluster our items, we’ll see non uniform distributions arise among various buckets. If not, we’ll end up with very similar distributions along a single line.

Let’s see a concrete example. A tweet is not just the text that the user sees, it includes a lot of metadata: the user location and timezone, the user bio, the number of followers the user has and the number of users he follows, how many times it has been retweeted, the kind of tweet (reply, retweet, plain text, with a link, with hashtags, with user mentions).

We can maintain separate set of buckets for each of these dimensions, using hash functions that map closely related items in the domain to the same slots in the co-domain.

If the processes underlying the phenomenon we are tracking do not change, we can expect these distributions not to change. In which case, we can discard these dimensions as not interesting.

If the processes do evolve, we can use a key strength of particle filters to still keep the computing load manageable: we can dynamically reduce the number of particles used to track these uninteresting dimensions, allocating resources to more interesting ones. Over time, what is interesting and uninteresting may change and we’ll just have to focus (i.e have more particles) where the real action is at the moment. That’s one reason I like particle filters over other statistical methods. The other reason is that they make it possible to estimate any kind of phenomenon, you don’t have to commit to a specific statistical model up front.

Squirro is the personal digital research app. Broader than feeds and more specific than search, Squirro scans multiple sources from Internet channels and social media, private databases and even internal systems such as Salesforce.com and SAP to find the most relevant information on your topic of interest, then updates it continuously and automatically. The result is a living collection of curated content you can save, synthesize and share with friends and colleagues in your own private workspace.

Squirro gives you timely, relevant information to navigate fast changing business relationships.

When I joined Nektoon in January, Squirro was still diagrams on a drawing board and some minimal proof of concept code. Now it’s a powerful, responsive web application that runs on your phone, tablet and computer. It supports thousands of users and can track hundreds of thousand sources.

It’s amazing how far and how fast we progressed. A tribute to the talent of my colleagues, I am humbled every day by how good you are, friends, and energized by how it fells to work with you. Great vibes!

The key idea is to combine particle filters and sketches in a single powerful tool. That’s why I call them Particle Sketches.

I devised them as a tool to track the posting behavior of a very large set of Twitter users, in order to decide who to follow using Twitter streaming API and who to follow using the Rest API based on their expected tweeting frequency and distribution across the day and week. The data structure is however totally generic and suitable to all kind of large-scale data estimation problem: I can surely see applications in high-frequency trading and website traffic forecast.

Let’s look at the building blocks first.

Particle Filters —otherwise known as Sequential Monte Carlo method— are model estimation techniques based on simulations. They are used to estimate Bayesian models in which the latent variables are connected in a Markov chain, typically when the state space of the latent variables is continuous and not sufficiently restricted to make exact interference tractable.

In practice, they work by maintaining a set of differently-weighted samples —called particles— that represent the expected distribution of the latent variable. You can then “filter”, which in this context mean determine the distribution at a specific time, by estimating how the particle swarm should have evolved under the observed variable (e.g. time).

On update, a new set of weighted particles is generated influenced by new observations. You can check Wikipedia’s page on particle filters for details or the multitude of pages about them available on the Internet.

The key fact here is that particle filters are composed by a set of discrete, importance-weighted particles and their distribution is effectively the sum of these particles.

We can thus split up a particle filter into multiple smaller filters by splitting up the particles into them. If we pick the particles at random, we should end up with essentially the same distribution among all of them.

If we add noise to a few of these sub-filters, the cumulative distribution of the sum of all of them will still be fairly close to the original estimated distribution. It will be even more true if we can account for the noise and remove its effect.

Count-Min (CM) Sketches are probabilistic, memory efficient data structures that allow one to estimate frequency-related properties of a data set (estimate frequencies of particular items, find top-K frequent elements, performance range frequency queries —i.e. find the sum of the frequencies of elements within a given range— such as estimate percentiles).

They are conceptually quite simple, they maintain a two-dimensional array [w, d] of integer counters. A set of d random hash functions with co-domain between 0 and w is used to choose which counters to update whenever we a new sample arrives.

Given an item, we can estimate its frequency by looking at all buckets in the two-dimensional array and choosing the one with the lowest value.

CM sketches are extremely memory efficient: for example, 48 KB are sufficient to calculate the top-100 most frequent elements in a 40 MB dataset with 4% error.

CM sketches tend to perform better with highly-skewed distributions, such as those generated by power laws or Zipf-like distributions.

In case of low or moderately skewed data, a more careful estimation can be done by accounting for the noise due to hash collisions: for example, by estimating and removing the noise for each hash function (calculated as the average value of all remaining counters) and choosing the median instead of the minimum as the estimated frequency. This variation is called Count-Mean-Min Sketch.

There are many other interesting properties of CM Sketches, such as various algebraic operations that they support or a way to measure how different two sketches are — calculating the cosine distance between them.

For a good explanation of CM Sketches with sample code, check this nice post on highlyscalable. For deeper knowledge, check this site.

Particle Sketches replace the counters with particle filters. On each new observation, a set of filters is selected using hash functions and, for each of them, only a random subset of the particles is updated.

To obtain a distribution for a given item, we put together all the particle filters that are associated with that item. For better performance, we can estimate and remove the expected noise due to collisions in a way analogous to Count-Mean-Min sketches.

The nice thing is that many cool properties of CM Sketches are preserved. For example, by putting all particle filters together we have an estimation of the cumulative distribution of the whole population. Using techniques similar to those used to estimate the top-K elements in a set, we can calculate the cumulative particle filters of a sub-population and so on.

It’s still a work in progress, I am still exploring the characteristics of these models and the best way to merge together multiple particle filters discarding the peculiar noise generated by (pardon the repetition) structure of this data structure.

Possible improvements: stacking together particle sketches at different resolutions, it should be possible to decompose these models in a way analogous to wavelet decomposition. It should make it possible to use even a more compact representation, thus dually to have —if we use the same bits of information— a more faithful model with less noise.

NEXT: Particle Sketches / 2 — working around the central limit theorem

Image via CrunchBase

I decided at the end of last year that I was working in the wrong place. The company was nice, the people were great, but the development practices and project management felt on the wrong track. I quit.

I looked all over Europe for a new interesting adventure. London, Berlin, Amsterdam — I was all over the place meeting new people, screening companies and interviewing with the few interesting choices.

In the process, I clarified my goals. I want:

• to remain sane, so I need a company that understands and applies agile software development practices
• to learn cloud computing, so I need a company that has expertise in that area and has at least a successful deployment or product in the area of large scale computing
• to improve my data mining skills, so I need a company that tackles interesting problems
• to spend most of my time developing software, so I will avoid mixed half-management, half-development positions, particularly in large corporations.

You can imagine how delighted I was when I found a company that meets these goals here in Zurich. I couldn’t believe that to be possible. I thought that so improbable that I already gave up my rented apartment here and was about to move somewhere else. Now I am stuck in a rented room in a shared flat, while I search for a new place.

The company I joined is Nektoon AG. A startup built by the core team of local.ch, a Switzerland whitepages site, that has already shipped a successful product, Memonic, a Evernote-like tool geared toward teams. Memonic predates Evernote, but when you have 100 times the money it’s easy to beat the competition.

Memonic is built entirely on Amazon Web Services, is written in my old friend Python and it’s quite an interesting tool. You can find some details on its architecture on our Memonic blog.

Squirro

I, however, did not join Nektoon to work on Memonic. I am working on a shiny new product called Squirro. Memonic and Squirro attack the same problem, they help you pick valuable needles from the Internet haystack, but they do it in quite a different way.

Memonic sits at your side while you browser and help you save interesting leads, shortlist relevant information and curate them for your peers. Searching for the right stuff, scoring sources and keeping up to date with new content is still a manual process.

Squirro does that and more for you: give our little squirrel a topic and he will find the best acorn trees and harvest the tastier nuts for you. He will keep coming with better and better fruits, because it learns your taste over time.

He can do much more, he’s a sucker for serendipity: do you work at a Big Pharma company and have you got a meeting with Dr. Bob next Thursday about that new promising molecule for Alzheimer? Squirro will come up with relevant studies and papers that you haven’t yet read, will tell about unexpected weather conditions/road blocks/strikes at the place where you’re going to meet and will give you the latest spicy gossip about FCZ Zurich… because he knows Bob is a big fan of that soccer team.

Search is the past, Google’s business is telling you about what you are already thinking about. Social is the present, Facebook and Twitter tell you what others around you are thinking right now. Squirro is the future, we’re in the business of telling you what you are not yet thinking about.

When things screw up, you end up with multiple records relating to the same entity or relation. How do you clean things up?

Let’s take the simple case of a many-to-many relation, implemented as a linking table:

CREATE TABLE First (
	First_ID INT IDENTITY(1,1),
	-- more fields
	PRIMARY KEY (First_ID)
);

CREATE TABLE Second (
	Second_ID INT IDENTITY(1,1),
	-- more fields
	PRIMARY KEY (Second_ID)
);

CREATE TABLE First_Second(
	First_Second_ID INT IDENTITY(1,1),
	First_ID INT FOREIGN KEY REFERENCES First,
	Second_ID INT FOREIGN KEY REFERENCES Second
);

If we end up with duplicate First_Second records, we can get rid of them using this code:

WITH records(First_Second_ID, RecNo) AS
  SELECT
    First_Second_ID,
    ROW_NUMBER() OVER( PARTITION BY First, Second ORDER BY First, Second) AS RecNo
  FROM First_Second

DELETE FROM records WHERE RecNo > 1;

Doing the same without CTE is possible, but less clean and straightforward. It involves temporary tables, cursors or nested subqueries.

I learned this trick with Microsoft SQL Server. Postgresql, DB2, Oracle and a bunch of other databases support CTE as well.

Image via Wikipedia

Once upon a time, I worked for a small company that had a tiny office. It was just a big room where all the developers, designers and creatives assembled, plus a couple of small ones where the owner and his secretary spent most of the time on their phone pitching to new clients or trying to get them pay overdue bills.

Italy is full of old buildings with tall roofs, hard to heat and with terrible acoustics. This was not different. The “Production” room was small and noisy. It was much like cubicle-land, but worse. Being a bunch of nerds, we devised a nerdy solution.

Whenever one deemed the room too noisy, he would cast a Cone of Silence — embodied in a physical paper cone placed in the middle of the central desk. It magically turned the noisy office room into a silent library one. Meetings, chit chat, even working together was banned for a few hours. Like any respectable spell, you couldn’t cast it in rapid succession. You could cast it once per week, then you had to wait and recharge to prevent abuse and let others finish their work.

All in all, Cone of Silence worked really well.

I don’t want to debate the pro and cons of the remainder operator in ANSI C and CPython: the two standards adopt one the divisor, the other the dividend’s sign for the result of the remainder operator. While the original K&R C left it to be platform dependent, if I remember correctly. I am not much of  a systems hacker and I am on the wrong side of the Atlantic to submit my candidature, unless they’d consider sponsoring people from Europe — which I doubt.

I find however interesting their additional challenge, the dog command: it’s really a sort of glorified, asynchronous version of tee.

I am preparing for an interview in Berlin tomorrow, so I don’t have the time to write a full blown dog command today, but I did some preliminary research.

Writing non-blocking, Asynchronous I/O is still somewhat platform dependant:

• On Windows, your best bet is overlapped I/O with FILE_FLAG_OVERLAPPED.
• On POSIX, you can use aio.h.
• If you are specifically on Linux, you can go even faster avoiding copying in and out of user space by using tee(2), splice(2) and vmsplice(2).

Given that they wrote a man page for dog, I’d go with the POSIX or Linux solution.

While searching for a job, I met a proprietary trading company that uses functional languages for their own systems. Jane Street is a private equity firm, specializing in statistical arbitrage. They use algorithms to take advantage of the small inefficiencies in (or between) financial markets.

Their main needs are correctness, agility and performance. Their tool of choice is OCaml.

They trade 1 to 2 millions shares a day, 2 to 4 billion dollars flow through their system. There is no faster way to run yourself out of business than making lots of bad decisions automatically in a tight loop. Algorithmic trading is an arms race, people compete on milliseconds in placing orders to get an edge but the best way to win is to outsmart adversaries.

Jane Street was a pretty conservative shop a decade ago. They were smaller, Excel and VBA satisfied their needs. As they outgrew them, they looked at the usual tools in the enterprise market and found them lacking.

This firm deeply cares about the quality of their software: they are betting their own money, billions of dollars, on it. It’s no wonder that senior partners committed to review each and every line of code that goes into production for critical components.

Jane Street considered rewriting their systems in C#, but abandoned it in favor of OCaml. They cite verbosity and lack of clarity as important reasons in this choice.

Verbosity is kind of obvious for languages like C# or Java, although in the recent years there were mild improvements. An interesting point that Yaron Minsky (Jane Street’s managing director) made is that these languages —when applied to large systems— tend to make people cut and paste, reimplementing the wheel over and over, because they are not expressive enough to capture high level variability and invariants properly. You can’t pay enough people to code review boilerplate, dull code. The tenth time they see the same-looking code block, they skim over it and miss important bugs.

I do agree: although C# and Java greatly improved in the last years, they are nowhere close to the expressiveness of OCaml polymorphic variants, pattern matching and module functors.

I expect F# and Scala —the former a OCaml cousin running on the .NET platform, the latter a functional/object oriented language running on the Java VM— to slowly make inroads in the enterprise in the next few years.

It is both a matter of technology — functional languages needs are different from object oriented ones and platforms need time to adapt — and people — we need a new generation of coders before enterprises will truly grasp functional programming. It’s a kind of catch-22 situation that slows down adoption.

Lack of clarity is due to the “spooky action at distance” things in OOP called inheritance. Senior partners at Jane Street, bright people but not professional programmers, found it hard to understand what was going on and what code was actually being executed. They are right.

As the OOP community has finally figured out, inheritance is often a bad way to get polymorphism, but I don’t fully buy into Minsky’s argument: in order to write concise programs in functional languages, you have to introduce the same kind of indirection through the use of high-order functions and module functors. It’s only slightly less confusing.

It is true that type systems, when used properly, will help you write highly readable, high-order code better in functional languages than in object oriented ones — but that is only because the functional languages are younger and come from an academic background, they benefit from more recent research.

There are languages —Eiffel, for an example that is not so cutting edge— that let you reason about the correctness of your programs even when a high degree of polymorphism is involved.

Agility is the ability to adapt code to new requirements. Being able to refactor code fast and without errors is key to this goal.

Minsky cites algebraic data types and the semantic of the match statement in OCaml as important tools in this regard. The same is true for object oriented code, using double dispatch or —if you have a sane enough language— multi-method dispatch.

A great gem from Mr. Minsky is: “you have to think about the type system as a tool of its own”, you have to encode the invariants of your domain in your types. This is key to using static languages properly, no matter if they are functional or object oriented.

Performance is the last big theme that nudged Jane Street to choose OCaml.

They think that ML languages sit in a sweet spot between expressiveness and performance, a kind of local equilibrium where you get a lot on both axes. If you move either way, you have to give on the other one.

That is true: OCaml, in particular, has quite a good native code compiler. It produces fast code without requiring esoteric optimization techniques. It’s easy to reason about its output and it’s relatively easy to fine-tune its garbage collector, facts that help a lot in the kind of soft realtime arena where Jane Street plays.

Minsky cites UIs, Concurrency, External libraries and Programming in the Large as weak points in OCaml. Because of the particular needs of his firm, these are less of a problem than in other situations.

In my opinion, these are actually the key strengths that make Scala or F# better functional languages for common situations than niche players like OCaml.

Related articles

• Caml Trading talk at CMU (ocaml.janestreet.com)

Being an effective project manager is not about doing the work yourself, it is about making sure the right resource is applied to the right problem.

(from The gorilla is named Hogarth)

A depressing theme with many of today’s software shops is the need to only make two kinds of hires. The first is a developer. After all, a developer codes, and that makes money! The second hire is an MBA-style manager. This manager is an HR-type who handles budgets, spreadsheets, and politics.

Is Your Software Project in the Trenches? (Image via Wikipedia)

Then someone like me comes along. I’ve got a development background and I’ve managed as well, but today I don’t do either. Instead I work with a team to see where the problems are. I sit with them and look for the areas that have become blind spots, and then find ways to solve those problems. I’ve saved large organizations substantial amounts of money by improving how their teams work. But I’ve usually done this by hiring in as a developer or manager. It’s a rare company that hires someone to improve their process. They’d much rather sit in the trenches and inspire their soldiers to leap out in the face of concertina wire and machine guns, sure that with the right mix of courage and moral fiber, this time they’ll finally ship that product!

As we look around, this attitude seems so… stupid. They seem to think that by trying harder they’ll succeed. How many of our favorite sports teams don’t have coaches, but ask their professional athletes to try harder?

From “Is Your Software Project in the Trenches?” by Jared Richardson