When things screw up, you end up with multiple records relating to the same entity or relation. How do you clean things up?

Let’s take the simple case of a many-to-many relation, implemented as a linking table:

CREATE TABLE First (
	First_ID INT IDENTITY(1,1),
	-- more fields
	PRIMARY KEY (First_ID)
);

CREATE TABLE Second (
	Second_ID INT IDENTITY(1,1),
	-- more fields
	PRIMARY KEY (Second_ID)
);

CREATE TABLE First_Second(
	First_Second_ID INT IDENTITY(1,1),
	First_ID INT FOREIGN KEY REFERENCES First,
	Second_ID INT FOREIGN KEY REFERENCES Second
);

If we end up with duplicate First_Second records, we can get rid of them using this code:

WITH records(First_Second_ID, RecNo) AS
  SELECT
    First_Second_ID,
    ROW_NUMBER() OVER( PARTITION BY First, Second ORDER BY First, Second) AS RecNo
  FROM First_Second

DELETE FROM records WHERE RecNo > 1;

Doing the same without CTE is possible, but less clean and straightforward. It involves temporary tables, cursors or nested subqueries.

I learned this trick with Microsoft SQL Server. Postgresql, DB2, Oracle and a bunch of other databases support CTE as well.

Image via Wikipedia

Once upon a time, I worked for a small company that had a tiny office. It was just a big room where all the developers, designers and creatives assembled, plus a couple of small ones where the owner and his secretary spent most of the time on their phone pitching to new clients or trying to get them pay overdue bills.

Italy is full of old buildings with tall roofs, hard to heat and with terrible acoustics. This was not different. The “Production” room was small and noisy. It was much like cubicle-land, but worse. Being a bunch of nerds, we devised a nerdy solution.

Whenever one deemed the room too noisy, he would cast a Cone of Silence — embodied in a physical paper cone placed in the middle of the central desk. It magically turned the noisy office room into a silent library one. Meetings, chit chat, even working together was banned for a few hours. Like any respectable spell, you couldn’t cast it in rapid succession. You could cast it once per week, then you had to wait and recharge to prevent abuse and let others finish their work.

All in all, Cone of Silence worked really well.

I don’t want to debate the pro and cons of the remainder operator in ANSI C and CPython: the two standards adopt one the divisor, the other the dividend’s sign for the result of the remainder operator. While the original K&R C left it to be platform dependent, if I remember correctly. I am not much of  a systems hacker and I am on the wrong side of the Atlantic to submit my candidature, unless they’d consider sponsoring people from Europe — which I doubt.

I find however interesting their additional challenge, the dog command: it’s really a sort of glorified, asynchronous version of tee.

I am preparing for an interview in Berlin tomorrow, so I don’t have the time to write a full blown dog command today, but I did some preliminary research.

Writing non-blocking, Asynchronous I/O is still somewhat platform dependant:

• On Windows, your best bet is overlapped I/O with FILE_FLAG_OVERLAPPED.
• On POSIX, you can use aio.h.
• If you are specifically on Linux, you can go even faster avoiding copying in and out of user space by using tee(2), splice(2) and vmsplice(2).

Given that they wrote a man page for dog, I’d go with the POSIX or Linux solution.

While searching for a job, I met a proprietary trading company that uses functional languages for their own systems. Jane Street is a private equity firm, specializing in statistical arbitrage. They use algorithms to take advantage of the small inefficiencies in (or between) financial markets.

Their main needs are correctness, agility and performance. Their tool of choice is OCaml.

They trade 1 to 2 millions shares a day, 2 to 4 billion dollars flow through their system. There is no faster way to run yourself out of business than making lots of bad decisions automatically in a tight loop. Algorithmic trading is an arms race, people compete on milliseconds in placing orders to get an edge but the best way to win is to outsmart adversaries.

Jane Street was a pretty conservative shop a decade ago. They were smaller, Excel and VBA satisfied their needs. As they outgrew them, they looked at the usual tools in the enterprise market and found them lacking.

This firm deeply cares about the quality of their software: they are betting their own money, billions of dollars, on it. It’s no wonder that senior partners committed to review each and every line of code that goes into production for critical components.

Jane Street considered rewriting their systems in C#, but abandoned it in favor of OCaml. They cite verbosity and lack of clarity as important reasons in this choice.

Verbosity is kind of obvious for languages like C# or Java, although in the recent years there were mild improvements. An interesting point that Yaron Minsky (Jane Street’s managing director) made is that these languages —when applied to large systems— tend to make people cut and paste, reimplementing the wheel over and over, because they are not expressive enough to capture high level variability and invariants properly. You can’t pay enough people to code review boilerplate, dull code. The tenth time they see the same-looking code block, they skim over it and miss important bugs.

I do agree: although C# and Java greatly improved in the last years, they are nowhere close to the expressiveness of OCaml polymorphic variants, pattern matching and module functors.

I expect F# and Scala —the former a OCaml cousin running on the .NET platform, the latter a functional/object oriented language running on the Java VM— to slowly make inroads in the enterprise in the next few years.

It is both a matter of technology — functional languages needs are different from object oriented ones and platforms need time to adapt — and people — we need a new generation of coders before enterprises will truly grasp functional programming. It’s a kind of catch-22 situation that slows down adoption.

Lack of clarity is due to the “spooky action at distance” things in OOP called inheritance. Senior partners at Jane Street, bright people but not professional programmers, found it hard to understand what was going on and what code was actually being executed. They are right.

As the OOP community has finally figured out, inheritance is often a bad way to get polymorphism, but I don’t fully buy into Minsky’s argument: in order to write concise programs in functional languages, you have to introduce the same kind of indirection through the use of high-order functions and module functors. It’s only slightly less confusing.

It is true that type systems, when used properly, will help you write highly readable, high-order code better in functional languages than in object oriented ones — but that is only because the functional languages are younger and come from an academic background, they benefit from more recent research.

There are languages —Eiffel, for an example that is not so cutting edge— that let you reason about the correctness of your programs even when a high degree of polymorphism is involved.

Agility is the ability to adapt code to new requirements. Being able to refactor code fast and without errors is key to this goal.

Minsky cites algebraic data types and the semantic of the match statement in OCaml as important tools in this regard. The same is true for object oriented code, using double dispatch or —if you have a sane enough language— multi-method dispatch.

A great gem from Mr. Minsky is: “you have to think about the type system as a tool of its own”, you have to encode the invariants of your domain in your types. This is key to using static languages properly, no matter if they are functional or object oriented.

Performance is the last big theme that nudged Jane Street to choose OCaml.

They think that ML languages sit in a sweet spot between expressiveness and performance, a kind of local equilibrium where you get a lot on both axes. If you move either way, you have to give on the other one.

That is true: OCaml, in particular, has quite a good native code compiler. It produces fast code without requiring esoteric optimization techniques. It’s easy to reason about its output and it’s relatively easy to fine-tune its garbage collector, facts that help a lot in the kind of soft realtime arena where Jane Street plays.

Minsky cites UIs, Concurrency, External libraries and Programming in the Large as weak points in OCaml. Because of the particular needs of his firm, these are less of a problem than in other situations.

In my opinion, these are actually the key strengths that make Scala or F# better functional languages for common situations than niche players like OCaml.

Related articles

• Caml Trading talk at CMU (ocaml.janestreet.com)

Being an effective project manager is not about doing the work yourself, it is about making sure the right resource is applied to the right problem.

(from The gorilla is named Hogarth)

A depressing theme with many of today’s software shops is the need to only make two kinds of hires. The first is a developer. After all, a developer codes, and that makes money! The second hire is an MBA-style manager. This manager is an HR-type who handles budgets, spreadsheets, and politics.

Is Your Software Project in the Trenches? (Image via Wikipedia)

Then someone like me comes along. I’ve got a development background and I’ve managed as well, but today I don’t do either. Instead I work with a team to see where the problems are. I sit with them and look for the areas that have become blind spots, and then find ways to solve those problems. I’ve saved large organizations substantial amounts of money by improving how their teams work. But I’ve usually done this by hiring in as a developer or manager. It’s a rare company that hires someone to improve their process. They’d much rather sit in the trenches and inspire their soldiers to leap out in the face of concertina wire and machine guns, sure that with the right mix of courage and moral fiber, this time they’ll finally ship that product!

As we look around, this attitude seems so… stupid. They seem to think that by trying harder they’ll succeed. How many of our favorite sports teams don’t have coaches, but ask their professional athletes to try harder?

From “Is Your Software Project in the Trenches?” by Jared Richardson

What happens when you’re put in a position when you have to do both development and project management? If you’re wicked smart or know from the start, you refuse the job. You know it’s bound to fail – no need to waste your time. If you’re not so smart or you succeeded in either or both positions in the past, you accept the challenge and, when you realize how pointless it is, you quit.

You’re bound to find yourself sooner than later in a situation when the capacity of your team is below the expected output. That’s when the problems start.

If your “team” is new, that often happen because you don’t actually have a team in the first place: you have a a group of people. They are just an imperfect approximation what a team should be: people who fit very well together, knows each other strengths and weaknesses and how to compensate each others.

Your team needs to deliver more. You need to grow the technical talent in your developers or build more automation in your tasks. If you do the first, you’re thinking as a manager. If you do the latter, you’re thinking as a developer. Pick one and deliver. If you can do both, you’re doing a great job.

What happens when you can’t grow technical talent or you can’t build more automation fast enough? The worst thing you can do is to think that you can compensate by taking over more development tasks. That’s your basic instinct if you have a history as a developer, fight it! If you go down that route, you’ll do a big disservice to your team mates – they are the first to feel your lack of focus on management and they are not becoming better developers – and you fail your company – you’re trying to cope with contingent issues, but you’re not improving the team’s output on the long run.

You may even go that route in good faith. You may even have explained to your own managers that you boost your team output temporarily working overtime, since you’re working at 60% of your potential, or 8 hours per day, and you could put in 10/12 for a few weeks. DON’T DO THAT! They will only hear “60%” and consider you a slacker and chances are they haven’t read Slack.

The fact is, as a technical lead compensating by doing more development work is the wrong decision. You’re failing to apply the very lesson explained in that book: if something unexpected goes work, if something that you relied upon fails in unexpected ways, you have no more capacity to fix it.

What would someone with the instinct of a manager do? They would refuse to take over more tasks than his team can deliver, asking for the proper resources or the proper time to build up capacity. Maybe you did, but then you let your developer instincts take over. That’s a really bad idea.

XML is like violence – if it doesn’t solve your problems, you are not using enough of it.

nokogiri.org

• When you create a new F# console solution, it won’t compile cleanly out of the box. You need to add mscorlib as an explicit reference in the F# project. I had to switch to the .NET 4.0 toolset in mono to get the plugin to compile.
• Code competition seems to be broken.

Related articles

• MonoDevelop 2.6 brings new features (h-online.com)
• MonoDevelop on Lion (tirania.org)
• MonoDevelop 2.6 is out (tirania.org)