Thursday, November 26, 2009

Paradox of the Poor (Money No Enough) Programmers

Recently, I got involved in a some discussions over the future of Computer Science in Singapore. The truth of the matter is that Singapore has done very well with regards to attracting lots of world-class Infocomm companies to set up shop in Singapore. REGARDLESS of whether the current economic recovery is going to be U-shaped, W-shaped, O-shaped, whatever-shaped, the jobs are coming over the next couple of years. There are going to be a large number of them and these are going to require good programmers (and not just run-of-the-mill graduates).

But there's apparently also a shortage of good programmers. For a CS prof who is concerned about the state of the local IT industry, this is clearly something that I am deeply troubled about.

After discussing this issue with some folks over the last few days, I came upon this apparent paradox: it is well-known that a good programmer is 10 times more productive than a mediocre one. In this light, suppose a mediocre programmer makes $30K a year, the good programmers should be expected to make $300K a year. However, we are not seeing this in the market today. WHY?

While thinking about this problem, I suddenly recalled this conversation I had over lunch a couple of weeks ago with my ex-classmate who is the co-founder of a successful startup. Over lunch, we were discussing the issues of manpower and pay. He made this one remark, "I have this guy who is worth 10 times what I'm paying him - but I'm not paying him that much. The reason is that he doesn't KNOW he's worth that much, and I have no incentive to tell him."

This indicates that there's information asymmetry. The classic example of this is what's known as the Market for Lemons. The idea here is that in the used-car market, there are two kinds of cars, lemons and peaches. Lemons are cars that will break down frequently and peaches are good and reliable cars. The catch is that only the seller knows whether his car is a lemon or a peach. A buyer cannot tell. Hence the information asymmetry. The question is this: suppose lemons are worth $20,000 and peaches are worth $60,000, what is the price of a used car?

Clearly, the price must be at least $20,000 and at most $60,000. A market price of $60,000 is impossible because the buyers risk getting a lemon and suffering a huge loss. Suppose the clearing price for used cars is $20,000+x, where 0<x<40,000. The lemon sellers will gladly sell their car 'cos they will enjoy surplus x>0. The peach sellers however will not sell 'cos they will lose $40,000-x>0. This means that at this price, the market is full of lemons. If the buyers were rational, they would never pay $20,000+x since they know that they must be looking at a lemon. The stable equilibrium is therefore a market clearing price of $20,000 and where only lemons get sold. This is a market failure because peaches cannot be sold.

So how is this issue of information asymmetry and lemons related to programmers?

Well, it seems to me that there's some weirdness in the market. If there's a so-called shortage of good programmers, then logically, the salaries of good programmers must shoot through the roof (in Singapore, not talking about Google, Mountain View here), but that's not happening. Hold that thought.

Over the past couple of years that I have been teaching at NUS, I have come to realize one curious thing. The class of Honours is NOT an entirely good indicator of whether a student is a good programmer. There is some correlation, i.e. First Class Hons typically better programmer than Third Class Hons, but there are very good programmers who are only 2nd Upper/Lower students (and quite a number of them!).

I have had students who are elite programmers who fail MA1505 (calculus) and get their CAPs thrashed. There are also First Class Honours students that I would never touch with a ten-foot pole. Software companies try to do their due diligence and set tests and exams, etc. However, the laupok First Class Honours will still probably get through.

All in all, I believe that there's an information gap. Companies look at the prospects and they are not entirely sure. Good programmers who just graduate fresh from school don't really understand their own market value and are willing to accept somewhat low salaries from the bigger firms. Bigger firms don't pay more because the programmers are docile enough to accept and they also have no interest to pay more. Cheap is good what. Smaller firms don't have the financial resources to compete and also take the cue from the large firms.

I had an idea: suppose we have an oracle that will magically reveal the value of each programmer. We take a programmer and stick him into the oracle and out comes a dollar value, i.e. say $40,000, or $60,000 and both the market and the programmer believe that the oracle is accurate. How will this change the situation? :-)

I have a theory - and I plan to build the oracle to test if my theory is correct. Wish me luck.



Afternote:
Here's another post on the same issue (Thanks Zi!). The funny thing about productivity in programming is that we actually want programmers who write less code, not more. Also, the majority of the time is spent not so much on writing code, but in debugging. Good programmers generate fewer bugs. :-)

9 comments:

  1. There's also the likelihood that good programmers will just go their own way and start their own business?

    With the advent of the App Store and clones, it seems easier nowadays for the lone programmer to just set up shop and start coding without having to worry about overheads.

    ReplyDelete
  2. Further on the topic of information asymmetry: there's been lots of econs literature that theorise about the signalling effects of higher education, whether it signals actual quality, or just ability to take tests.

    Further the route programmers can take to proficiency nowadays is not so clear cut. While many come from university backgrounds, there are also many open source hackers who are self taught and gain reputation through their code contributions to open source projects. The even more impressive thing about open source projects is how you can learn from other people's code, and indeed if you're lucky you might even learn directly from interactions with the veteran coders.

    And now with the App Store, you hear stories of that 9 year old kid who learned ObjC on his own and then making apps. Given that education materials are so easily accessible online nowadays, if you add in a dash of aptitude plus ample amounts of perseverance and determination, the question is, is a uni degree still a good signal of quality?

    Would it not be better to just force yourself to try making code contributions to open source projects and/or make your own apps, even if it is crap at first, and then iteratively improve along the way?

    ReplyDelete
  3. "I have had students who are elite programmers who fail MA1505 (calculus) and get their CAPs thrashed."

    This year onwards, 1521 has the honor of doing that.

    I think they need to put techies in HR as well. Cos only a pro will know a pro.

    Good luck with your oracle =).
    Wait, you can be THE oracle =P

    ReplyDelete
  4. Cannot agree more.

    Due to the poor pay and perception of "lower" prestige, many good programmers in Singapore switch to non-programming jobs (e.g. especially sales-related jobs) or get promoted to managerial positions (which require non programming skills).

    I think there are two reasons for the lack of good, locally trained programmers (in addition, to a lousy pay package).

    (1) IT education in non-tertiary schools are outdated. They used to teach basic Pascal and Visual Basic 6!

    (2) After university/polytechnic life, programming is hard work. Programmers require passion, a sense of adventure (to try many languages, projects, work with great mentors and colleagues). There are few local programmers in this league.

    ReplyDelete
  5. In theory, a programmer (good or bad) can source for any job from any paymaster from anywhere in the world. The internet has flattened the world (to borrow Friedman's phrase), so that you do not have to work for any fixed "boss" anymore. You simply auction your programming skills to the highest bidder.

    Over time, your reputation as a good (bad) programmer will be revealed, and you will command a pay closer to what your skills are worth. For many people though, this route is too risky to take, since there is no guarantee of a stable income.

    ReplyDelete
  6. I just have one question. Isn't the time of your evaluation of a programmer also a factor? Isn't it possible (maybe not probable) that an OK-cert programmer can eventually grow into a good one through experience and self-study?

    I am really looking forward to your theory, also hoping it will consider the programming a student does outside school also! (out of interest)

    ReplyDelete
  7. Thanks everyone for your comments.

    To respond to Amar's question, the point here is to ensure that our graduating students get pay commensurate with their ability. Perhaps a student might over time get better - then his/her pay should get better pay WHEN he/she is actually better.

    No reason why industry should be paying for what they cannot see. :-P

    The key challenge lies in building this oracle.

    ReplyDelete
  8. Dr Ben,

    IMO, it is all about perceptions. If everyone (else the programmers) think that programmers are like "labour workers", they get paid like one. In this world, I guess it is never really about how good you are, but how good people think you are.

    You even mentioned in your previous post saying "First, why do we use grades for university admission. Answer is very simple: because there is a limited amount of resources" and "Again, there's limited resources, either scholarship places or jobs and there's a need to shortlist the candidates efficiently. Interviewing EVERYONE is NOT an option by a long shot."

    Basically I think it applies to this situation too, how employers judge programmers is by their school results (the most efficient way).

    P.S. I used to have a friend during my poly days who were forced to learn and even excelled at 5-6 programmming languages and was paid a paltry sum of $800 (internship)

    ReplyDelete
  9. I.H.I.S.,

    how employers judge programmers is by their school results (the most efficient way).

    You have to give the industry some credit. It's not quite as bad as you described. They actually give the applicants programming exams, interviews, etc.

    It's just not very easy for them to judge.

    I used to have a friend during my poly days who were forced to learn and even excelled at 5-6 programmming languages and was paid a paltry sum of $800 (internship)

    I think internships pay very little for several reasons: (i) many interns are actually no good; (ii) even if an intern is good, it's hard for the employers to figure that out; (iii) employers can get away with paying very little ('cos students NEED to do the internship) and so they pay very little.

    ReplyDelete