I feel like today may have been the day that I decided I'm switching careers.

I have a reasonably good job, and a good life, all-in-all.  But there are definitely frustrations, and limits with what I'm able to do in my current capacity.

If you don't follow my professional life regularly, well, I sell my time for a living.  That's what I do -- I'm a very well-paid temp, essentially.  I do high-tech work.  All of it is related to software development.  Custom software development -- I'm not writing a general-purpose tool like Excel or Acrobat Reader.  I'm writing specific tools to solve specific problems for specific customers.  A company needs software, they don't have the people they need inside their company to build it ... they come to me and I help.  Many times, what they really do is go to temp agencies (they are called consulting firms, but that's just because of the high-prestige occupation I am in) and ask for people who meet a set of criteria.  Many temp agencies either know about me or can search me out on the web.  Years of experience with particular flavors of software tools are one of the main criteria in play.  As I get older, I meet more and more of these criteria, because I have more years of experience with the tools I use.  These criteria make little sense, but this is how the market for software temps (or "consultants") works.

There are some interesting things about my line of work, though.  And they are things that cause the market to make very bad decisions.  This is tough for me -- despite my basically leftist nature, I usually place great trust in the market.  When a market is inefficient, firms can exploit the inefficiency in order to make lots of money.  Other firms swiftly notice the innovation the exploiting firm has made and imitate it.  Suddenly, the market has changed to reflect reality.

So I believe in an efficient market -- and someday, I hope this will help me.  Because the market for custom software development is highly distorted.

I think the main reason that it is highly distorted is that many of the managers who manage software developers are not software developers themselves -- and many of the others are not good software developers.  So they don't understand some fundamental things -- things that are well-known amongst people who have studied the field, and are known instinctively by people who excel in the field, but are counterintuitive.

Let's start with an example.

Given two software development teams:

• Team A consists of one highly productive software developer and ten average software developers,
• Team B consists of one highly productive software developer and no one else.

Team B is more productive, and can get more done in a given amount of time.

To managers, this seems impossible, but it flows relatively naturally from two mathematical relationships:

• Good software developers are tens or hundreds as times as productive as average ones,
• The overhead required to manage interfaces and communication between team members is proportional to the square of the size of the team.

So basically, Team A has 55 lines of communication between team members (hope I did that math right), and Team B has zero.  The highly skilled developer on Team A has ten separate relationships to manage -- relationships that take time away from doing the work of developing software at the expense of partitioning tasks so that the relatively unproductive developers can do them.  And given that the highly productive person on Team A can do things 10 or 20 or 100 times faster than everyone else, the costs of communicating with the other 10 people to split up the work exceed the costs of doing all of that work him/herself.  That's why Team B is more productive.

Managers can't deal with this for several reasons:

• They can't believe it -- they can't believe that one person can be 10 or 20 or 100 times as productive as another person, because it is not true for most jobs,
• They don't understand the costs associated with communications, or the extra costs imposed on a task by splitting it into parts that can be done independently,
  
• Even if they did believe the productivity differential (and they don't), they can't tell which people are productive, so it is safer for them to invest in quantity because it increases the chances of finding a good person,
• More insidiously, in Manager World, the more people you supervise, the more powerful you are.  (A related problem is that bigger budgets are better -- saving money is not a priority.)  Their incentives are all wrong,
• Even more insidiously, acknowledging these massive differences in the ability level of software developers would mean acknowledging that success is determined more by who the software developers are and less by ... well, management.  Understandably, they don't want to believe this.  Even if they were to believe it, they have powerful professional incentives to persuade (or allow) others not to believe it -- for example, managers want salary structures to reflect that management is the most important job in software development.
  

So ... I know this will shock you, but I am a highly productive software developer.  I assert it.  You may not believe it (but doesn't everyone say they're highly productive?, you ask!) but I am the author of this essay and I make the rules.  So you will have to assume it as true in order to read on.  There.

Remember Team A and Team B above?  They mean that I should work alone.  Or should I?  One can easily imagine a Team C.  Team C consists of 11 highly productive software developers.  Sure, Developer One (me) still has 10 lines of communication to manage, but now perhaps the additional contributions of the other 10 software developers make up for the costs of those 10 lines of communication.  And they do, quite easily.  Team C is much more productive than Team B.  So the truth is, I can work on a team and have that be a productive activity -- as long as the other members of the team are good.  And I've done this in the past, a couple of times.  But tellingly, my examples of Team Cs are smaller than 11, and a few people repeat across several of them -- because by definition, most people in the field are not "highly productive," if that means significantly more productive than average.

What's the problem with that model?  What's the problem with Team C?

The problem is, teams that consist of highly productive software developers do not hire temps.  They don't need help as they hum along, kicking butt.  They may hire highly productive people for permanent jobs (Google, which I assume to be a set of highly productive teams, inquired about hiring me rather recently, based upon some work I had done in the open-source world) but they don't bring in temporary specialists to augment their team.  So almost by definition, in my line of work, I am not going to brought on to assist a highly productive team -- I'm going to be brought on to assist a team that is not highly productive.  That's why they are looking for help.  And then, the problem is, the fastest and best way to get things done is to do all the work myself.

There's another related problem -- the work I do is very simple, and thus requires a good software developer to expand upon or modify.  I know, this also seems counterintuitive.  Stay with me for a minute.

Basically, the dynamic is the following.  The enemy of software development is complexity.  So good developers use sophisticated techniques to battle that complexity -- doing "smart work" to keep the system as simple as possible as it grows, so that it is easier to fix, modify, maintain.  However, less skilled (or less diligent) software developers do "dumb work."  They just add new things, and allow the software to grow more and more complex, because they don't do anything to keep it organized -- often because they don't understand the basic problem, sometimes because they don't see the solution, occasionally because they're too lazy to care.  The software sprawls all over the place and eventually grows to the point where -- some time in the future -- someone will demand that it be rewritten from scratch because it is no longer possible to maintain it. 

Good software developers prefer to work with things that are simple, but less skilled software developers prefer to work with things that are straightforward ("dumb work"); they can't understand simple software that contains sophisticated techniques ("smart work") to help keep it simple.

Believe it or not, this means my customers prefer dumb work.  They fear that if you do not do dumb work, they will have difficulty replacing you.

Now, the funny thing is, their strategy is horrendously inefficient.  The highly productive software developer can do 10 or 20 or 100 times the work of the average one.  So it is much more effective to hire people like that to replace me and pay the premium -- because, guess what?  The highly productive software developer does not get paid 10 or 20 or 100 times as much as the average one.  (I gave up on being annoyed by this years ago.)  So a smart company hires the good ones and gladly pays the extra 10% or 20% or 50%, because that's all it is.  My customers could hire a good person -- even someone less productive than me, but more productive than average -- to do maintenance of my work going forward and that person could replace several of the unproductive developers they have today at a much lower cost. 

And this assumes they don't have any good people now -- but they probably do.  Those good people are probably just stuck trying to deal with the dumb work they've been handed, and would love the opportunity to sink their teeth into some smart work for a change.  Now those good people would be more productive because they'd be working with something simpler and could do more useful work faster.

But, instead, my customers believe that it is dangerous to rely on the assumption that you will have anyone smart around in the future -- that reliance on good people is risky.

I have struggled with this problem for years, as have the other highly productive software developers with whom I have worked (and you know who you are -- the people who keep repeating on my Team Cs).  How can it be that we do really good -- really simple -- work and then have our customers complain about it being "too complicated?"

The funny thing is that I really used to think that the problem was my lack of people skills.  I lacked diplomacy, I lacked the ability to get along with people.  And over the past 6 years or so (primarily due to my involvement in politics, but also because of my involvement with indiglo11 and all the changes that entailed), I've improved that situation.  I now have a hugely different ability to charm, persuade, and cajole.

But now, I find myself at Cabinet -- and despite all of those weaknesses-turned-strengths, I am not doing the job as they see it.  They want me to be replaceable, to be just another cog in their machine -- because they are wedded to the fantasy that the quality of managerial interaction with their project is more important than the quality of the people doing the work.  So it turns out that me being a little rough around the edges wasn't the problem at all; it's a structural problem.

It's not clear what to do.  But what is clear is that the way I am doing my work is not working for me or for my customers.  I do my best -- which I assert to you is very, very good -- and they are less pleased with it than I think they ought to be.  In 1998-9, I was considered by my customers to be very skilled, but unlikable and difficult to work with.  Now it's 2008 -- and I am considered to be very skilled, likable -- and difficult to work with.  I thought the "difficult" and "unlikable" parts were related, but it turns out "unlikable" was only a minor factor in "difficult to work with."

The reason I am difficult to work with is because I am good at my job, and my customers don't effectively use people who are good at this job.  I am added to teams that include people who are not highly productive -- and thus the additional members of the team actually harm the team's productivity.  This presents a conflict, because my customers are expecting me to produce a high-quality product that meets their needs as quickly as possible.  No wonder other team members (those who fall below a certain threshold of productivity) think I'm difficult to work with -- I don't want to let them do anything! 

You can imagine the caricatures of me that result -- ranging from the benign ("nutty professor"), to the gently demeaning ("perfectionist"), to the unflattering ("control freak") to the ... even less flattering ("prima donna").  You can also imagine why I hate this -- because I am trying to do my job (get the thing done as well and fast as possible), and so all of these labels feel very unfair to me.  The customer gives me two conflicting goals ("get it done quickly and do it right" and "use our staff to do some of the work") and then refuses to resolve the conflict in favor of one or the other because they refuse to believe the goals conflict in the first place.

So that's why it's so difficult for me to succeed in the roles that I am playing now, and thus on the career path I am on.  Granted, I'm making a good living, so in a sense I am "succeeding."  But I'm not happy -- and even when I do good work, my customers aren't always that happy either.

But my faith is that the market has, or can develop, a way of valuing what I do -- of valuing someone who is actually very good at the job.  I have a few theories on how to do that.  One is finding a way to market Team B -- a one-stop shop that has proven its/his ability to do large amounts of work alone.  Another is building my own Team C of highly productive people and bidding on jobs that way.  Yet another is building my own product (probably as a Team B) where I answer to no one except people who buy it.  Or maybe I should be using my "soft" skills in teaching, persuading, and cajoling (developed in the crucible of political campaign management, of all things) to find my niche in my field as an opinion leader.  (I've recently taken some steps in this direction, and they've been going well.)  And then obviously there are various combinations of these alternatives.

I'm not sure what I'll pick.  But I reiterate as I write this sentence, that I think today is the day that I decided I needed to choose a new direction.  Why today?  There were a confluence of things today.  I think it became clear that my current customer isn't appreciating the work I'm doing, all-in-all.  That I could never succeed at Cabinet, and that the number of annoyances created at Cabinet per month place me under undue stress.  The constant adrenaline isn't helping my long-term health.  There must be a better way to do my work.  I just need to find a way from here to there.