Productivity is one of my pet topics, because it's always dogged me a bit, especially early in my career.  I'd pull long days and nights and then realize I only actually worked (as in, typing in code, debugging stuff, and thinking about problems and their solutions) maybe 20% of the time.  Upon talking to coworkers, this seemed to be normal, a part of the expected friction costs incurred working in an office environment.  Meetings, shooting the shit with coworkers, lunch, email, and, er, stuff.
This is what I mean by 'stuff' in 1996

Eventually working around high-productivity professionals like John Carmack made me realize that if you want to excel, then you have to work hard and focus the whole time.

John Carmack Productivity Measurement Tool ca. 1998
I remember Carmack talking about productivity measurement.  While working he would play a CD, and if he was not being productive, he'd pause the CD player.  This meant any time someone came into his office to ask him a question or he checked email he'd pause the CD player.  He'd then measure his output for the day by how many times he played the CD (or something like that -- maybe it was how far he got down into his CD stack).  I distinctly remember him saying "So if I get up to go to the bathroom, I pause the player".

You know what's pretty hardcore?  Thinking that  going to the bathroom is essentially the same as fucking off.

(Sidenote: my memory is hazy on this since it was long ago -- CD PLAYERS yo).

That level of work ethic and focus is rare, particularly with my generation of programmers who were raised with the vile "work smart, not hard" mantra, coupled with the sense that we were somehow significantly brighter than most of our peers.  Combine those two notions and add the Scotty Principle and you get....me.

If my coworkers were grinding through stuff that took them 20 hours, I'd be all "I work smart, not hard" with accompanying snarky eye roll, and I'd assume I could bust through an equivalent work load in four hours and still look like a goddamn superhero.

And sometimes I could, so, hey, validation!  

And you can skate by like this (God knows I did) for a long time before a couple things eventually rear their heads and bite you in your entitled face.

Productivity Deficit: Your Attitude Writing Checks Your Work Ethic Can't Cash

An overinflated sense of your own abilities creates a constant state of production deficit, because you assume that you can make it up with a burst of brilliance and/or crunch.  

But there is no countering surplus to offset the deficit.  The only way surpluses show up is when you finish a (presumably) hard task much faster than you anticipated.  But instead of banking the surplus (i.e. moving on immediately to your next task), you spend it relaxing and screwing off because, whew, you just earned a small vacation by busting shit out in an hour that you thought would take all day.  Hey, what's on Facebook and Twitter right now?  Do I have any new mail?  No?  I wonder if anyone has tweeted something recently since the last time I checked...what about Reddit?  Oh, an inspirational/funny/cool YouTube video, it's only eight minutes long, let me watch it now!  (Eight minutes later) Sweet...what's up Twitter?  Oh man, I have to make a snarky response, but first I have to Google for that XKCD comic that totally makes my point for me...

And before you know it, the day is done, and you're still feeling good because you finished in one hour what should have taken all day, so all good, right?

Trap of the Easy Task

And yeah, it's often all good, but when operating at a slight deficit things can go pear shaped quickly when you accidentally spring the trap of the easy task.  Tasks so trivial that they barely register as work.  Update an SDK?  Hour, tops.  Implement this feature that I've already done on every other platform?  Hour, tops.  This is all mindless grunt work that anyone could do, it's just clicking through dialog boxes, maybe changing a few lines of source, rebuilding, whatever.

In other words, an easy task like this is so easy that it's a constant time cost for everyone irrespective of ability, so there's no opportunity nor need for crazy overestimation since what could possibly go wrong?

Well, as with so many things, it's the unexpected things that screw you the hardest.  The things that should be trivial end up taking days because, shit, that SDK install required a new compiler install, and the new compiler install failed and I can't uninstall it, and now I have to fucking WIPE MY WHOLE GODDAMN HARD DRIVE TO RECOVER.  Or you need to implement something that was trivial on nine other platforms, and it should be 30 minutes on platform number ten, except it lacks a base feature that is present on everything else, the compiler is dying with an "Internal compiler failure 2033", the API call you need to use hangs the dev console immediately and, oh, the dev support site you use to get release notes is totally down for the foreseeable future.

So what should have taken less than an hour took a week.

It's like having a perfect monetary budget that assumes no crazy "one time" only bills, except life is full of crazy one time only bills and the only way you can keep those under control is by giving yourself a budgetary capacitor to dampen the fluctuations.

And now you're defensive about losing a week to something stupid because you budgeted an hour for it and waited until the last second to do it and now the schedule has gone to hell, but it's not your fault, because it could have happened to anyone!  But if you had banked your surplus hours before and/or worked at closer to your theoretical peak effectiveness then this type of thing would get absorbed in the wash.

And now you live in a state of mild panic because you're way smarter than all this but you're never actually getting things done on time.

Identity Recalibration Crisis

Which leads to a potential identity recalibration crisis upon landing at a company with high performers that work hard and smart.  And that will happen if you're good.  Now you're no longer at the top of the curve.  In fact, shit, you're in the middle or bottom of the curve, a situation your brain probably never considered as an option.

This dude was my wake up call with respect to my
over inflated sense of skill
I went through this when I went to id software.  I had rationalized that John Carmack's success was just a factor of timing and luck since, hell, he was my age, and I was pretty smart, so what else could it be?  Upon my arrival I had a crash course in humility, because he was way smarter than me and way more productive as well.

This took a while to sink in, because until then I was used to believing I was one of the better programmers at a company.  And then working with Carmack I realized he was not just a little better, he was orders of magnitude better.  And I couldn't dismiss it with a "But I write a lot of code" hand wave, because he also wrote like 80% of the Quake code base.

Sometimes it takes a while for this to sink in.  Sometimes it doesn't sink in at all, and you're let go because you're not very productive compared to your new team of high performers.  Sometimes it sinks in, and you panic and get depressed because your self-image has always been that of being the strongest swimmer in the school, and right now you're just trying not to drown, much less keep up with everyone else.

But ideally you shrug off the old counter productive mentality and habits and emerge as another one of the high functioning team members, but that can take a lot of work, particularly if you have to get over years of giving it twenty percent.

Killing the Underachiever

If my pithy advice were little more than "Be more like John Carmack" then I can imagine a lot of readers throwing up their hands and saying "Well, fuck it, I'm a lost cause, because that's not going to happen."  But what I can do is relate some of the things that helped me kill off some of my underachieving habits.  The point isn't to become a superstar, it's to become better, since that's always the first step.

I don't believe in mechanical solutions ("Turn off the internet", "Listen to music", and stuff like that) because I don't think they address the core issues, which are psychological.  Instead I found that I had to do the following.

  1. Develop self-awareness.  It took working with John Carmack and other high productivity programmers and admitting that they were way more productive than me before I really understood how much more productive I could be.  In other words, if you don't admit it's something worth improving, then obviously you're not going to search for a solution, and if that's the case, you should go here or here.
  2. Give a shit.  Originally I called this "develop a sense of urgency", but really it's just about caring about getting your work done.  It doesn't even matter what you specifically care about!  It can be your professional image, your product, your team, your customers, whatever.  You just have to care about something that will drive you to getting things done, because if you don't, apathy will occupy that void.
  3. Minimize uncertainty.  In another blog article, Productivity vs. Uncertainty and Apathy, I talk about how poorly defined goals can lead to poor productivity.  If it's unclear what you need to get done today, then there's a reasonable chance you won't actually do anything today.
  4. Commit to getting something done every day.  When you show up in the morning have a well defined set of things to finish that day.  Stay as late as you have to in order to finish.  By committing to finishing that task other distractions will naturally fall by the wayside.  For example, I have jiu-jitsu training at 7pm.  If I screw off too much during the day, I don't get to train.  Also, by committing to a task, you avoid "being busy" instead of "getting work done", they're not the same thing.  
  5. Never say "I'll finish it up tomorrow" or "I'll make up for it by coming in early/staying late/working the weekend".  This is an easy trap to get into, where you keep incurring time debt until at some point you realize you're now three weeks behind on a task that should have taken two days.  This is like racking up credit card bills assuming you can pay them off later.  Which is fine, until "later" arrives and you've only accumulated more debt.
  6. Do not overpromise to make up for poor productivity.  There's a tendency when we're falling behind to try to overcompensate with future promises.  "When I'm done, it'll be AWESOME" or "I know I'm late, but I'm positive I'll be done by Monday".  By doing those things we just build more debt we can't pay off, and that will eventually lead to a catastrophic melt down when the super final absolutely last deadline date shows up.  Just get shit done, don't talk about how you're going to get shit done.
  7. Have an objective productivity metric.  This is a mechanical thing, but it acts as a reasonable backstop.  If you have changelogs you can reference, then it's easy to sit down on Friday and ask yourself "What have I done this week?"  And if you know that it's possible for others to check on you, then it makes you take each day a lot more seriously.  If you judge your value solely on output, not subjective things like "being smart", you will be more productive.
  8. Accept that "the grind" is part of the job.  A friend of mine's father has a great quote: "Son, i don't wake up every day and go to a place called fun. I wake up and go to a place called work"  You can't get irate or frustrated that the bulk of the day is typing in boring code and dealing with bugs and other people.  
I still screw off during the day.  I am not a code grinding automaton.  I read Facebook, chat with others, Tweet, shop on Amazon, get coffee, read forums, write blog posts like this one and I'm totally fine with that.  

Because I know I'm committed to getting things done as well.  So even though I'm writing  this right now at 3pm, if I have to work until 8pm to get my shit done and checked in, I'll do that, so that tomorrow is a fresh start, not a day of "Oh goddamit, let's try again to finish that task I started a week ago."

Once I developed that sense of daily urgency my productivity went up considerably.  And if I'm really productive in the morning and afternoon and feel like I can go home at 4pm with a clean conscience, I do that too, so I don't burn out.  

It's a push/pull between banking surplus and generating a deficit, just make sure you don't do just one or the other.
Posted by Unknown
34

View comments

  1. Good post!

    I'm dealing with this recently. Working with a bunch of really smart people is pretty awesome though.

    ReplyDelete

  2. Lol.. love that last image.

    I find having a totem to help focus (e.g. Carmack's CD player) works well for me. I got a 90 minute sand timer a while back that I can physically reach for and flip over when I feel the urge to check facebook, then promise not to check until it's out. I usually look up and find it's been empty for a while because I was too focused on work to have noticed.

    ReplyDelete

  3. Great article! Good you recognized JC's self-discipline which may sound simple but that doesn't necessarily make it easy. I've written a post about building that discipline and having fun at the same time, it's at http://www.blog.namar0x0309.com/2013/02/the-gamification-of-time/.

    Thanks!

    ReplyDelete

  4. This is such a brilliant take on a syndrome that is often subjected to denial. But in the part where you mention about ways to mend it, I'd also recommend — try and get yourself in a work environment where others are not just as smart or smarter than you are , but also productivity wise on a different plane.

    ReplyDelete
    Replies

    1. Surrounding yourself with high performers can help tremendously, but it can also lead to the identity crisis I mention in the blog post (the identity recalibration).

      Delete

    2. The identity recalibration is really the best thing that can happen to you though, IMO. I work with high performers compared to me and that has pushed me to work faster, make better decisions, focus and generally level up. I think a wide swath of the programming world actively seeks to avoid a situation where the identity recalibration is even possible. The others go through that again and again making them better and beter.

      Delete

    3. It's definitely an ideal situation, but a fraction of people have their identities so threatened by it that they shut down. It's a hard pill to swallow.

      Delete

  5. I used to have that same problem, still do to an extent I guess. Ended up fixing it by using the pomodoro technique ... which I've gotten a bit sloppy with lately.

    ReplyDelete

  6. Seriously, its like you wrote this post after watching me work. I'd like to commit to being better, but I'm not sure where to start. My main problem, I guess, is that I currently work from home, and while I'm strict about not goofing off I find things can some times take a lot longer to complete that they used to. Maybe I'm just getting old. But I find it frustrating that I'm not as productive as I know I can be.

    ReplyDelete

  7. A take on a old subject, now with John Carmack! I particularly like the "Give a shit" part and I'd like to see that bit expanded. How do you exactly give a damn sometimes, specially when you admit that sometimes you just have to grind through it? There will be at points you really don't feel like caring about what you're doing and how it's going to affect you, but you should so what are the mentality tricks that help us care?

    ReplyDelete
    Replies

    1. This is a very hard question, because in the absence of passion you will have to rely on discipline and work ethic, and relying on those will lead to extreme burnout (and dissatisfaction). So ideally you find something to care about, which means maybe your product, your team, your professional image, your company, pride, intellectual curiousity, etc. It's different for everyone.

      For example, working on a game's resource loader. That's not fun for a lot of people, but some people might like the technical challenge, others grind through it because it's key to a game's engine, others work through it because they want to ship, or they don't want to lead the content guys down, etc. Everyone has to find their own reasons to push ahead.

      If you can't, that's a problem, and really I'm not sure there's a great solution.

      Delete

    2. This is not a permanent solution, but if you have one thing or a set of things in particular that you are having trouble finding motivation for, you should consider http://www.stickk.com/. I discovered it via Tim Ferriss's 4-hour Chef. Basically, you choose a non-trivial amount of money and put it in escrow, and if you don't achieve your goal it gets sent to an organization you hate (pro-life/pro-choice groups, politicians you don't like, etc.)

      Delete

    3. Passion is key, John Carmack certainly didn't work so much without passion. When this passion is fading, maybe it's just time for a break, maybe it's time to work on other things. Passion is so strong that it can be dangerous, it can make you ignore fatigue, make you work all night, but nothing can beat the productivity of a passionate person.

      Delete

  8. I tried this method of "committing to do a specific task in the day".
    This could make me stay at work quite long, such as 22h30pm or more.
    But that's not the only issue : as a logical consequence, i found my productivity of the following day to be way way below normal. Which made me wonder : is that a net benefit in the end ?

    ReplyDelete
    Replies

    1. The question isn't "is there a net benefit", the question should be "Why is it taking me until 22h30pm to complete this task". It could be time estimation being off, distractions, or just poor productivity with no reason to go home. For me there's a strong impetus to get things done if i say "Once I finish this, I get to do X", where X might be play a video game I've wanted to play, watch a show I'm eager to watch, go do some hobby, etc. If you don't have something better to do than work, then that's an issue in and of itself.

      Delete

  9. Awesome post! I've been working from home for the past 4 years and I'm finding my productivity has been waning over the past 6 months. Next month I begin my lease in shared office space hoping that working among other professionals will get me back up to mark.

    ReplyDelete

  10. This is the right post at the right time for me. I'm glad I came across it, and it's comforting to know that I am not alone.

    ReplyDelete

  11. I remember all of the activity on your .plan file at id (which I absolutely loved reading and often wrote email responses back and forth with you on various topics) and I'd think, "Does that guy ever work?!" Of course it was douchey of me to think that as I did most of the reading/correspondence work from work ;p

    This blog is extremely helpful. I've spent most of my career being the defacto subject matter expert at my office and being laid back all the time. I recently joined my own version of idSoftware and I work with a team of Carmacks, and so far, it has scared the shit out of me. But that's been a great thing, because I don't think I've pushed my self-development this hard in my entire life. I'll definitely be reading through this a few times and utilizing the tools.

    Glad to see you've still got a square head on your shoulders.

    ReplyDelete

  13. This was great. It's something I've already come to learn about myself over the last 15 years or so... However, since it's largely driven by introspection, whenever I try to relate it with others, they often see it as a self-esteem issue and feel the need to encourage me by refuting my explanation (good intentions, but frustrating, to say the least).

    As has been said, it's validating to know that I'm not alone. This post is also an encouraging light for moving forward. Thanks!

    ReplyDelete

  14. I've been using a fairly simple time-boxing approach recently. I work without any distraction for 25 minutes. Then take five minutes break. Then plan the next 'box' and deal with 'shit', attend meetings etc. The important this is, as you say, is have a list of specific things you want to work on, with specific objectives you want to achieve. Spending a few minutes in the morning or night before figuring out what you want to achieve is a great way to start improving your productivity.

    ReplyDelete

  15. This is so awesome, and very true as well. I figured out long ago that we all tend to over-inflate how much work we actually do in a day, usually without even realizing it. I never realized Carmack did this, but I gauge my work the exact same way---I have a 27-hour (approximately) playlist of video game music that I listen to every day while doing level design work. If I go to FB, check e-mail, eat, go to the bathroom, or chat in IRC for more than a minute, I pause it. At the end of the day, I measure how far I progressed through the playlist, and I have a pretty exact idea of how much work I did.

    ReplyDelete

  16. Great post - early in my career I would procrastinate and then pull heroics of fat the last minute and 9/10 it went flawlessly. Developing self discipline though and being able to focus and work hard whenever I work made a huge difference.

    Love the tips!

    ReplyDelete

  17. I came to your blog from the ProfHacker article by Jason B. Jones. Your blog was insightful and contained several points that gave me pause. However, your writing would be so much richer if you could clean up your language. I am a 57-year-old graphic designer/part-time professor that gained much from your insights, but at times it was difficult to stomach the casual use of inappropriate language. I encourage you to rethink your blogging language as you continue to reach a broader audience. Thanks for a thought-provoking article.

    ReplyDelete

  18. Seriously great post. I was ready to throw my hands in the air exactly as you described if this was just going to boil down to "Be John Carmack", but the advice you gave is incredibly practical. I often find tasks where the solution is obvious but there is just grind it away work to be done can take 10x longer than difficult but interesting issues. I have definitely accrued promised overtime debt as well.

    Self Awareness, Giving a Shit are absolutely the two most important factors in becoming better at your craft and producing better work.

    I might also add that you can be more effective by bouncing ideas off fellow workers to see if you are walking into a trap before you get there. Especially working around people you do very smart and productive, get their feedback and talk to them about your approach to solving a problem and concerns you have. They can often bring up issues you would run into later that would require redoing or changing work you already did. Simply not being afraid of looking like you don't already have all the answers has done wonders for me. You are doing no one a favor by burning time on a problem someone 20ft from you might already know the answer to.

    ReplyDelete

  19. Great post, resonates with a lot of information I personally soaked recently about "Critical Chain" project management described in a book bearing the same name from Eliyahu M. Goldratt. Biggest problem thinking about a specific task and getting it done much faster than it should have been, is that in the overall project that lost time is ... lost. Making the developers look a the whole project and its completion, but still keeping it broken down to small tasks instead of one huge monolithic impossible task - this makes it easier to focus on the big picture. You can google explanations about "Fever Chart" to see what I mean, the buffer is project-wide and engineers involved in making it work worry about the whole project and not their particular task for the day - making it much more apparent what need to be done next.

    ReplyDelete

  20. Great article, and one I relate to as well. I've often wondered how much more in life I could accomplish if I could stay in the 'zone' more. I've found that avoiding distractions and bad mental habits helps a lot. I'm going to print out your list and put it next to my desk (so I can read it and think about productivity instead of being productive :)

    By the way it was great to meet you in person at I/ITSEC last year!

    ReplyDelete

  21. Great post, but I found one big thing that's missing. Why do you think your brain wants to slack off, to read Facebook or Twitter, and you have to force it to be self-aware and take shit? There are numerous serious scientific researches about brain performance, loosing concentration and regaining it. And they all consider tiredness.
    Some researchers even compare brain with a muscle. If you over-push it, it will become tired. If you train it, it will stay fresh longer. And it's all explainable by neuro-chemistry: available nutrients, hormones, etc. None of the researchers say that you could train your brain to become as productive Euler, all agree that's a combination of your training and your abilities.
    So why don't you speak about burn-outs? I've gone through them multiple times and had to accept that my slacks off for a reason. I find it's very easy to stay at office till 22:30 if the task is interesting, but it's hard to force myself go home and forget about the interesting task, because I know by experience that if I do that for a week, I'd get burn-out which would burn not only my productivity, but also serotonin and some of my general health.
    I could do heavy brain work for tens of hours without burn-outs until somewhere around 21. Now I have to accept I'm not Euler, and try to do best of what I can with what I have (e.g. doing exercises helps, good sleep schedule and all the common things that are scientifically proven). I'm very glad you don't experience tiredness and stress, and heartily wish you'll never will.

    ReplyDelete

  22. Thank you very much for this article. I'm dealing with this as well, partly because I returned to university, which has considerably more "grind" which makes me procrastinate considerably more on stuff that I don't (instinctively) want to work on. I definitely feel this feeling of falling behind and not being up to 100% of what I could be, and looking at people like Carmack makes me feel like people have figured it out so I can figure it out. Thanks for this.

    ReplyDelete

  23. Fantastic. I've been a consultant for 20 years, charging a way-high hourly rate but cranking out solutions so well that they keep paying it, due to many of the standards you have put down. Hanging out with the corporate employee coders shows a profound lack of everything you posted about.

    ReplyDelete

  25. Great blog. Inspirational to me, very resonant.

    ReplyDelete

  26. great post, it's great to mention non-mechanical solutions.

    ReplyDelete

  27. This comment has been removed by a blog administrator.

    ReplyDelete

  1. I was originally going to write this blog about a different topic, one of the endless online discussions about "How to get better at this thing you suck at".  Then I realized it was pointless.

    Many "How to make your life/code/job/hobby better" blogs are very well written, insightful, and even inspirational.  They provide awareness into our failings and weaknesses and hint at ways in which we can improve.  They give us strategies and high level conceptual ideas on how to approach problems.

    The issue is that it's all mostly pointless.  Finding the best plan doesn't mean we'll execute it, because very often the problem is that we suck at execution. The very things we're trying to fix are often the very things that prevent us from fixing them.

    Or, put another way, it's a rare situation where someone doesn't know what to do, is told what to do, then goes off and just does it.

    Have a nice day.

    Posted by Unknown
    4

    View comments

  2. Productivity is one of my pet topics, because it's always dogged me a bit, especially early in my career.  I'd pull long days and nights and then realize I only actually worked (as in, typing in code, debugging stuff, and thinking about problems and their solutions) maybe 20% of the time.  Upon talking to coworkers, this seemed to be normal, a part of the expected friction costs incurred working in an office environment.  Meetings, shooting the shit with coworkers, lunch, email, and, er, stuff.
    This is what I mean by 'stuff' in 1996

    Eventually working around high-productivity professionals like John Carmack made me realize that if you want to excel, then you have to work hard and focus the whole time.

    John Carmack Productivity Measurement Tool ca. 1998
    I remember Carmack talking about productivity measurement.  While working he would play a CD, and if he was not being productive, he'd pause the CD player.  This meant any time someone came into his office to ask him a question or he checked email he'd pause the CD player.  He'd then measure his output for the day by how many times he played the CD (or something like that -- maybe it was how far he got down into his CD stack).  I distinctly remember him saying "So if I get up to go to the bathroom, I pause the player".

    You know what's pretty hardcore?  Thinking that  going to the bathroom is essentially the same as fucking off.

    (Sidenote: my memory is hazy on this since it was long ago -- CD PLAYERS yo).

    That level of work ethic and focus is rare, particularly with my generation of programmers who were raised with the vile "work smart, not hard" mantra, coupled with the sense that we were somehow significantly brighter than most of our peers.  Combine those two notions and add the Scotty Principle and you get....me.

    If my coworkers were grinding through stuff that took them 20 hours, I'd be all "I work smart, not hard" with accompanying snarky eye roll, and I'd assume I could bust through an equivalent work load in four hours and still look like a goddamn superhero.

    And sometimes I could, so, hey, validation!  

    And you can skate by like this (God knows I did) for a long time before a couple things eventually rear their heads and bite you in your entitled face.

    Productivity Deficit: Your Attitude Writing Checks Your Work Ethic Can't Cash

    An overinflated sense of your own abilities creates a constant state of production deficit, because you assume that you can make it up with a burst of brilliance and/or crunch.  

    But there is no countering surplus to offset the deficit.  The only way surpluses show up is when you finish a (presumably) hard task much faster than you anticipated.  But instead of banking the surplus (i.e. moving on immediately to your next task), you spend it relaxing and screwing off because, whew, you just earned a small vacation by busting shit out in an hour that you thought would take all day.  Hey, what's on Facebook and Twitter right now?  Do I have any new mail?  No?  I wonder if anyone has tweeted something recently since the last time I checked...what about Reddit?  Oh, an inspirational/funny/cool YouTube video, it's only eight minutes long, let me watch it now!  (Eight minutes later) Sweet...what's up Twitter?  Oh man, I have to make a snarky response, but first I have to Google for that XKCD comic that totally makes my point for me...

    And before you know it, the day is done, and you're still feeling good because you finished in one hour what should have taken all day, so all good, right?

    Trap of the Easy Task

    And yeah, it's often all good, but when operating at a slight deficit things can go pear shaped quickly when you accidentally spring the trap of the easy task.  Tasks so trivial that they barely register as work.  Update an SDK?  Hour, tops.  Implement this feature that I've already done on every other platform?  Hour, tops.  This is all mindless grunt work that anyone could do, it's just clicking through dialog boxes, maybe changing a few lines of source, rebuilding, whatever.

    In other words, an easy task like this is so easy that it's a constant time cost for everyone irrespective of ability, so there's no opportunity nor need for crazy overestimation since what could possibly go wrong?

    Well, as with so many things, it's the unexpected things that screw you the hardest.  The things that should be trivial end up taking days because, shit, that SDK install required a new compiler install, and the new compiler install failed and I can't uninstall it, and now I have to fucking WIPE MY WHOLE GODDAMN HARD DRIVE TO RECOVER.  Or you need to implement something that was trivial on nine other platforms, and it should be 30 minutes on platform number ten, except it lacks a base feature that is present on everything else, the compiler is dying with an "Internal compiler failure 2033", the API call you need to use hangs the dev console immediately and, oh, the dev support site you use to get release notes is totally down for the foreseeable future.

    So what should have taken less than an hour took a week.

    It's like having a perfect monetary budget that assumes no crazy "one time" only bills, except life is full of crazy one time only bills and the only way you can keep those under control is by giving yourself a budgetary capacitor to dampen the fluctuations.

    And now you're defensive about losing a week to something stupid because you budgeted an hour for it and waited until the last second to do it and now the schedule has gone to hell, but it's not your fault, because it could have happened to anyone!  But if you had banked your surplus hours before and/or worked at closer to your theoretical peak effectiveness then this type of thing would get absorbed in the wash.

    And now you live in a state of mild panic because you're way smarter than all this but you're never actually getting things done on time.

    Identity Recalibration Crisis

    Which leads to a potential identity recalibration crisis upon landing at a company with high performers that work hard and smart.  And that will happen if you're good.  Now you're no longer at the top of the curve.  In fact, shit, you're in the middle or bottom of the curve, a situation your brain probably never considered as an option.

    This dude was my wake up call with respect to my
    over inflated sense of skill
    I went through this when I went to id software.  I had rationalized that John Carmack's success was just a factor of timing and luck since, hell, he was my age, and I was pretty smart, so what else could it be?  Upon my arrival I had a crash course in humility, because he was way smarter than me and way more productive as well.

    This took a while to sink in, because until then I was used to believing I was one of the better programmers at a company.  And then working with Carmack I realized he was not just a little better, he was orders of magnitude better.  And I couldn't dismiss it with a "But I write a lot of code" hand wave, because he also wrote like 80% of the Quake code base.

    Sometimes it takes a while for this to sink in.  Sometimes it doesn't sink in at all, and you're let go because you're not very productive compared to your new team of high performers.  Sometimes it sinks in, and you panic and get depressed because your self-image has always been that of being the strongest swimmer in the school, and right now you're just trying not to drown, much less keep up with everyone else.

    But ideally you shrug off the old counter productive mentality and habits and emerge as another one of the high functioning team members, but that can take a lot of work, particularly if you have to get over years of giving it twenty percent.

    Killing the Underachiever

    If my pithy advice were little more than "Be more like John Carmack" then I can imagine a lot of readers throwing up their hands and saying "Well, fuck it, I'm a lost cause, because that's not going to happen."  But what I can do is relate some of the things that helped me kill off some of my underachieving habits.  The point isn't to become a superstar, it's to become better, since that's always the first step.

    I don't believe in mechanical solutions ("Turn off the internet", "Listen to music", and stuff like that) because I don't think they address the core issues, which are psychological.  Instead I found that I had to do the following.

    1. Develop self-awareness.  It took working with John Carmack and other high productivity programmers and admitting that they were way more productive than me before I really understood how much more productive I could be.  In other words, if you don't admit it's something worth improving, then obviously you're not going to search for a solution, and if that's the case, you should go here or here.
    2. Give a shit.  Originally I called this "develop a sense of urgency", but really it's just about caring about getting your work done.  It doesn't even matter what you specifically care about!  It can be your professional image, your product, your team, your customers, whatever.  You just have to care about something that will drive you to getting things done, because if you don't, apathy will occupy that void.
    3. Minimize uncertainty.  In another blog article, Productivity vs. Uncertainty and Apathy, I talk about how poorly defined goals can lead to poor productivity.  If it's unclear what you need to get done today, then there's a reasonable chance you won't actually do anything today.
    4. Commit to getting something done every day.  When you show up in the morning have a well defined set of things to finish that day.  Stay as late as you have to in order to finish.  By committing to finishing that task other distractions will naturally fall by the wayside.  For example, I have jiu-jitsu training at 7pm.  If I screw off too much during the day, I don't get to train.  Also, by committing to a task, you avoid "being busy" instead of "getting work done", they're not the same thing.  
    5. Never say "I'll finish it up tomorrow" or "I'll make up for it by coming in early/staying late/working the weekend".  This is an easy trap to get into, where you keep incurring time debt until at some point you realize you're now three weeks behind on a task that should have taken two days.  This is like racking up credit card bills assuming you can pay them off later.  Which is fine, until "later" arrives and you've only accumulated more debt.
    6. Do not overpromise to make up for poor productivity.  There's a tendency when we're falling behind to try to overcompensate with future promises.  "When I'm done, it'll be AWESOME" or "I know I'm late, but I'm positive I'll be done by Monday".  By doing those things we just build more debt we can't pay off, and that will eventually lead to a catastrophic melt down when the super final absolutely last deadline date shows up.  Just get shit done, don't talk about how you're going to get shit done.
    7. Have an objective productivity metric.  This is a mechanical thing, but it acts as a reasonable backstop.  If you have changelogs you can reference, then it's easy to sit down on Friday and ask yourself "What have I done this week?"  And if you know that it's possible for others to check on you, then it makes you take each day a lot more seriously.  If you judge your value solely on output, not subjective things like "being smart", you will be more productive.
    8. Accept that "the grind" is part of the job.  A friend of mine's father has a great quote: "Son, i don't wake up every day and go to a place called fun. I wake up and go to a place called work"  You can't get irate or frustrated that the bulk of the day is typing in boring code and dealing with bugs and other people.  
    I still screw off during the day.  I am not a code grinding automaton.  I read Facebook, chat with others, Tweet, shop on Amazon, get coffee, read forums, write blog posts like this one and I'm totally fine with that.  

    Because I know I'm committed to getting things done as well.  So even though I'm writing  this right now at 3pm, if I have to work until 8pm to get my shit done and checked in, I'll do that, so that tomorrow is a fresh start, not a day of "Oh goddamit, let's try again to finish that task I started a week ago."

    Once I developed that sense of daily urgency my productivity went up considerably.  And if I'm really productive in the morning and afternoon and feel like I can go home at 4pm with a clean conscience, I do that too, so I don't burn out.  

    It's a push/pull between banking surplus and generating a deficit, just make sure you don't do just one or the other.
    Posted by Unknown
    34

    View comments

    1. Good post!

      I'm dealing with this recently. Working with a bunch of really smart people is pretty awesome though.

      ReplyDelete

    2. Lol.. love that last image.

      I find having a totem to help focus (e.g. Carmack's CD player) works well for me. I got a 90 minute sand timer a while back that I can physically reach for and flip over when I feel the urge to check facebook, then promise not to check until it's out. I usually look up and find it's been empty for a while because I was too focused on work to have noticed.

      ReplyDelete

    3. Great article! Good you recognized JC's self-discipline which may sound simple but that doesn't necessarily make it easy. I've written a post about building that discipline and having fun at the same time, it's at http://www.blog.namar0x0309.com/2013/02/the-gamification-of-time/.

      Thanks!

      ReplyDelete

    4. This is such a brilliant take on a syndrome that is often subjected to denial. But in the part where you mention about ways to mend it, I'd also recommend — try and get yourself in a work environment where others are not just as smart or smarter than you are , but also productivity wise on a different plane.

      ReplyDelete
      Replies

      1. Surrounding yourself with high performers can help tremendously, but it can also lead to the identity crisis I mention in the blog post (the identity recalibration).

        Delete

      2. The identity recalibration is really the best thing that can happen to you though, IMO. I work with high performers compared to me and that has pushed me to work faster, make better decisions, focus and generally level up. I think a wide swath of the programming world actively seeks to avoid a situation where the identity recalibration is even possible. The others go through that again and again making them better and beter.

        Delete

      3. It's definitely an ideal situation, but a fraction of people have their identities so threatened by it that they shut down. It's a hard pill to swallow.

        Delete

    5. I used to have that same problem, still do to an extent I guess. Ended up fixing it by using the pomodoro technique ... which I've gotten a bit sloppy with lately.

      ReplyDelete

    6. Seriously, its like you wrote this post after watching me work. I'd like to commit to being better, but I'm not sure where to start. My main problem, I guess, is that I currently work from home, and while I'm strict about not goofing off I find things can some times take a lot longer to complete that they used to. Maybe I'm just getting old. But I find it frustrating that I'm not as productive as I know I can be.

      ReplyDelete

    7. A take on a old subject, now with John Carmack! I particularly like the "Give a shit" part and I'd like to see that bit expanded. How do you exactly give a damn sometimes, specially when you admit that sometimes you just have to grind through it? There will be at points you really don't feel like caring about what you're doing and how it's going to affect you, but you should so what are the mentality tricks that help us care?

      ReplyDelete
      Replies

      1. This is a very hard question, because in the absence of passion you will have to rely on discipline and work ethic, and relying on those will lead to extreme burnout (and dissatisfaction). So ideally you find something to care about, which means maybe your product, your team, your professional image, your company, pride, intellectual curiousity, etc. It's different for everyone.

        For example, working on a game's resource loader. That's not fun for a lot of people, but some people might like the technical challenge, others grind through it because it's key to a game's engine, others work through it because they want to ship, or they don't want to lead the content guys down, etc. Everyone has to find their own reasons to push ahead.

        If you can't, that's a problem, and really I'm not sure there's a great solution.

        Delete

      2. This is not a permanent solution, but if you have one thing or a set of things in particular that you are having trouble finding motivation for, you should consider http://www.stickk.com/. I discovered it via Tim Ferriss's 4-hour Chef. Basically, you choose a non-trivial amount of money and put it in escrow, and if you don't achieve your goal it gets sent to an organization you hate (pro-life/pro-choice groups, politicians you don't like, etc.)

        Delete

      3. Passion is key, John Carmack certainly didn't work so much without passion. When this passion is fading, maybe it's just time for a break, maybe it's time to work on other things. Passion is so strong that it can be dangerous, it can make you ignore fatigue, make you work all night, but nothing can beat the productivity of a passionate person.

        Delete

    8. I tried this method of "committing to do a specific task in the day".
      This could make me stay at work quite long, such as 22h30pm or more.
      But that's not the only issue : as a logical consequence, i found my productivity of the following day to be way way below normal. Which made me wonder : is that a net benefit in the end ?

      ReplyDelete
      Replies

      1. The question isn't "is there a net benefit", the question should be "Why is it taking me until 22h30pm to complete this task". It could be time estimation being off, distractions, or just poor productivity with no reason to go home. For me there's a strong impetus to get things done if i say "Once I finish this, I get to do X", where X might be play a video game I've wanted to play, watch a show I'm eager to watch, go do some hobby, etc. If you don't have something better to do than work, then that's an issue in and of itself.

        Delete

    9. Awesome post! I've been working from home for the past 4 years and I'm finding my productivity has been waning over the past 6 months. Next month I begin my lease in shared office space hoping that working among other professionals will get me back up to mark.

      ReplyDelete

    10. This is the right post at the right time for me. I'm glad I came across it, and it's comforting to know that I am not alone.

      ReplyDelete

    11. I remember all of the activity on your .plan file at id (which I absolutely loved reading and often wrote email responses back and forth with you on various topics) and I'd think, "Does that guy ever work?!" Of course it was douchey of me to think that as I did most of the reading/correspondence work from work ;p

      This blog is extremely helpful. I've spent most of my career being the defacto subject matter expert at my office and being laid back all the time. I recently joined my own version of idSoftware and I work with a team of Carmacks, and so far, it has scared the shit out of me. But that's been a great thing, because I don't think I've pushed my self-development this hard in my entire life. I'll definitely be reading through this a few times and utilizing the tools.

      Glad to see you've still got a square head on your shoulders.

      ReplyDelete

    13. This was great. It's something I've already come to learn about myself over the last 15 years or so... However, since it's largely driven by introspection, whenever I try to relate it with others, they often see it as a self-esteem issue and feel the need to encourage me by refuting my explanation (good intentions, but frustrating, to say the least).

      As has been said, it's validating to know that I'm not alone. This post is also an encouraging light for moving forward. Thanks!

      ReplyDelete

    14. I've been using a fairly simple time-boxing approach recently. I work without any distraction for 25 minutes. Then take five minutes break. Then plan the next 'box' and deal with 'shit', attend meetings etc. The important this is, as you say, is have a list of specific things you want to work on, with specific objectives you want to achieve. Spending a few minutes in the morning or night before figuring out what you want to achieve is a great way to start improving your productivity.

      ReplyDelete

    15. This is so awesome, and very true as well. I figured out long ago that we all tend to over-inflate how much work we actually do in a day, usually without even realizing it. I never realized Carmack did this, but I gauge my work the exact same way---I have a 27-hour (approximately) playlist of video game music that I listen to every day while doing level design work. If I go to FB, check e-mail, eat, go to the bathroom, or chat in IRC for more than a minute, I pause it. At the end of the day, I measure how far I progressed through the playlist, and I have a pretty exact idea of how much work I did.

      ReplyDelete

    16. Great post - early in my career I would procrastinate and then pull heroics of fat the last minute and 9/10 it went flawlessly. Developing self discipline though and being able to focus and work hard whenever I work made a huge difference.

      Love the tips!

      ReplyDelete

    17. I came to your blog from the ProfHacker article by Jason B. Jones. Your blog was insightful and contained several points that gave me pause. However, your writing would be so much richer if you could clean up your language. I am a 57-year-old graphic designer/part-time professor that gained much from your insights, but at times it was difficult to stomach the casual use of inappropriate language. I encourage you to rethink your blogging language as you continue to reach a broader audience. Thanks for a thought-provoking article.

      ReplyDelete

    18. Seriously great post. I was ready to throw my hands in the air exactly as you described if this was just going to boil down to "Be John Carmack", but the advice you gave is incredibly practical. I often find tasks where the solution is obvious but there is just grind it away work to be done can take 10x longer than difficult but interesting issues. I have definitely accrued promised overtime debt as well.

      Self Awareness, Giving a Shit are absolutely the two most important factors in becoming better at your craft and producing better work.

      I might also add that you can be more effective by bouncing ideas off fellow workers to see if you are walking into a trap before you get there. Especially working around people you do very smart and productive, get their feedback and talk to them about your approach to solving a problem and concerns you have. They can often bring up issues you would run into later that would require redoing or changing work you already did. Simply not being afraid of looking like you don't already have all the answers has done wonders for me. You are doing no one a favor by burning time on a problem someone 20ft from you might already know the answer to.

      ReplyDelete

    19. Great post, resonates with a lot of information I personally soaked recently about "Critical Chain" project management described in a book bearing the same name from Eliyahu M. Goldratt. Biggest problem thinking about a specific task and getting it done much faster than it should have been, is that in the overall project that lost time is ... lost. Making the developers look a the whole project and its completion, but still keeping it broken down to small tasks instead of one huge monolithic impossible task - this makes it easier to focus on the big picture. You can google explanations about "Fever Chart" to see what I mean, the buffer is project-wide and engineers involved in making it work worry about the whole project and not their particular task for the day - making it much more apparent what need to be done next.

      ReplyDelete

    20. Great article, and one I relate to as well. I've often wondered how much more in life I could accomplish if I could stay in the 'zone' more. I've found that avoiding distractions and bad mental habits helps a lot. I'm going to print out your list and put it next to my desk (so I can read it and think about productivity instead of being productive :)

      By the way it was great to meet you in person at I/ITSEC last year!

      ReplyDelete

    21. Great post, but I found one big thing that's missing. Why do you think your brain wants to slack off, to read Facebook or Twitter, and you have to force it to be self-aware and take shit? There are numerous serious scientific researches about brain performance, loosing concentration and regaining it. And they all consider tiredness.
      Some researchers even compare brain with a muscle. If you over-push it, it will become tired. If you train it, it will stay fresh longer. And it's all explainable by neuro-chemistry: available nutrients, hormones, etc. None of the researchers say that you could train your brain to become as productive Euler, all agree that's a combination of your training and your abilities.
      So why don't you speak about burn-outs? I've gone through them multiple times and had to accept that my slacks off for a reason. I find it's very easy to stay at office till 22:30 if the task is interesting, but it's hard to force myself go home and forget about the interesting task, because I know by experience that if I do that for a week, I'd get burn-out which would burn not only my productivity, but also serotonin and some of my general health.
      I could do heavy brain work for tens of hours without burn-outs until somewhere around 21. Now I have to accept I'm not Euler, and try to do best of what I can with what I have (e.g. doing exercises helps, good sleep schedule and all the common things that are scientifically proven). I'm very glad you don't experience tiredness and stress, and heartily wish you'll never will.

      ReplyDelete

    22. Thank you very much for this article. I'm dealing with this as well, partly because I returned to university, which has considerably more "grind" which makes me procrastinate considerably more on stuff that I don't (instinctively) want to work on. I definitely feel this feeling of falling behind and not being up to 100% of what I could be, and looking at people like Carmack makes me feel like people have figured it out so I can figure it out. Thanks for this.

      ReplyDelete

    23. Fantastic. I've been a consultant for 20 years, charging a way-high hourly rate but cranking out solutions so well that they keep paying it, due to many of the standards you have put down. Hanging out with the corporate employee coders shows a profound lack of everything you posted about.

      ReplyDelete

    25. Great blog. Inspirational to me, very resonant.

      ReplyDelete

    26. great post, it's great to mention non-mechanical solutions.

      ReplyDelete

    27. This comment has been removed by a blog administrator.

      ReplyDelete

  3. Telemetry was the first product I worked on where I interacted directly with customers.  This interaction creates an interesting real time tension between working on big features with long development periods versus working on smaller features that can immediately benefit a small number of customers.

    Typically it starts with a reasonable request -- let's call it Feature X -- so we don't want to just say 'no'.  But of course, we have to deal with bug reports, customer support, sales assistance, new platforms and previously promised or planned features, and all within the constraint of monthly releases.  

    As a result the choice boils down to "Do you want some of Feature X now, or all of Feature X much later?"  In a perfect world we just do Feature X right and everyone wins, but in the real world of constrained bandwidth we have to be deliberate about scheduling work.

    Very often Feature X can be implemented in a limited form and the customer is happy because, hey, what wasn't possible is now possible.  And if we focus on just that customer's response then this is a net win, because that partial implementation may have taken only an hour whereas a proper, fully fleshed out implementation might have taken several days due to cascading side effects.

    The problem arises when you look at this new feature in context of other customers.  Now they see "Whoa, Feature X, cool!" or maybe just "Huh, didn't know about Feature X, I'll give it a try".  The former camp has learned to work around the absence of Feature X, the latter camp didn't even know it existed and didn't care, but now that they're aware of it they want to use it.

    And this is when the partial implementation absolutely turns out to be the wrong call.  Because for that one customer who thinks "Those guys are awesome, they were super responsive to a feature request!" you now have multiple (potential) customers thinking "This feature is janky, what the hell?"

    And they don't care about the reasons for it.  They don't care that it wasn't even going to get implemented until a customer requested it, or the reasons that a full implementation isn't practical or maybe even possible, they're just kind of annoyed that Feature X is incomplete and/or buggy.

    The lesson is this: if you're not going to do it right, then you have to really think hard about whether it's worth doing.  Saying no to one customer is a lot better than explaining to all the others why Feature X isn't as good as they expect. And it's a tough thing to say, because it's easy to rationalize that a limited Feature X is better than no Feature X, but in my experience this is almost never the case.

    Posted by Unknown
    0

    Add a comment

  4. In software development it's easy to lose sight of the fact that we're supposed to be shipping software.  We can show up to work, be legitimately busy for eight or nine hours, go home, and then realize that we didn't get any closer to shipping.

    Many people are fine with this -- "Well, I 'did stuff', so hey, all good, right?" -- despite not getting any closer to shipping.  Their day was consumed with necessary-but-not-shipping tasks.  Things like installing software, replacing a hard drive, helping a coworker, setting up a dev kit, or debugging a feature you thought you were done with a year ago.  This hidden but large frictional load can easily consume so much of our time that over a given period of time, we never actually get closer to ship.

    This is particularly true with mature products, where you now spend a lot of time on non-development tasks  like support, sales, meetings, and training, and the 'being busy' part can become so dominant that you never get around to shipping a new version.  This overhead is rarely factored into any schedule since coders are notoriously shitty at accounting for non-coding things.  We think in best case terms like "That should take like a day" with the assumption "day" means "calendar day" but we never get a calendar day to do stuff, so "day" means "eight hours" which takes a week to get done.

    Assuming that crunching non-stop is off the table as an option, I've found that the biggest productivity boost I've had comes from asking myself "What do I need to get done today?" instead of "What will I work on today?".  By creating a sense of urgency to finish a well defined task I now have to be cognizant of all the little interruptions that interfere with that goal through the day.

    So get shit done and stop just being busy.  Or to quote David Wong (or Jesus, take your pick), a tree is judged by its fruit.


    Posted by Unknown
    0

    Add a comment

  5. If you're not getting shit done, it may not be because you suck.  Traditionally unproductive workers get nailed with labels like undisciplined, lack of focus, or poor work ethic, but in my experience these are not nearly as influential as I'd thought.

    Over the past two decades of coding, including stints in management/as a lead/and being self-directed with no immediate manager, I've come to the conclusion that the main issues that impact productivity are uncertainty and apathy.

    When uncertainty kicks me in the nuts, it's usually in one of several flavors: high-level uncertainty, strategic uncertainty, and low-level uncertainty.

    "I feel like I should be doing something, but I have no idea where the fuck I am"
    High-level uncertainty, aka "Where am I and what an I trying to do?", is when you have a general goal that comes with a set of implicit assumptions.  Those assumptions are a minefield you have to traverse as you try to get to your finished product.  You end up paralyzed because you're not sure which direction is safe, especially since none of the mines are marked.  For example, you might have the task of "In the next month improve our network performance by 50%".  It seems like a concrete goal, but there are still a ton of hidden assumptions in there.


    Solution: Constrain your problem so that what was once vague is now extremely well specified.  Constraints tend to dictate solutions.  Okay, I have to improve network performance by fifty percent.  Can I use more memory to do this?  Can I use more CPU?  How much more CPU?  Can I use a third party library?  Is our performance poor sending or receiving or both?  Under what conditions?

    Or, using our lost in the woods analog, get a goddamn map.
    "So we have a lot of bugs, and apparently orcs and dragons"
    Once questions like those are answered the scope of the problem and available solutions becomes a lot clearer, and we can move on to tackling the next problem.

    Strategic uncertainty occurs when you have a well defined goal but you're unsure how to go about it.  Using our previous example you've determined that the problem is total network bandwidth, on both sides, and that you can sacrifice a lot of CPU to do it.  It seems to happen all the time under all conditions, and unfortunately most of the data going across the network comes from a bunch of different systems so you can't just pare down the data at one location.

    Now you have to come up with a technical solution.  You know you need to compress and decompress the data, but how?  Maybe you have no experience with compression algorithms, so now you're kind of stuck.  Do you just use zlib?  Or miniz?  Or do you license something?  Is there a hardware solution?  If there's no clear solution at hand ("drop in zlib") then even very good programmers will spin their wheels, unsure of how to tackle it.  They just hope for a brilliant insight.

    Solution: Grind it out by talking to someone.  Having an office mate helps tremendously so you can just say "I have to do this thing, I'm not even sure how to tackle it".  People like to sound smart and give their opinions, so give someone else the opportunity to solve your problem for you!

    In this case the simplest solution is to use a package like zlib to compress and decompress your data.  Strategy defined!

    This leaves the day to day tasks of getting it done, and here's where low-level uncertainty can creep in.  Now you know what you're supposed to do at a fairly specific level ("Implement zlib in our codebase so that the network layer is much faster") but at any given moment you're just staring at your editor, unsure what to do next.

    Solution: Make a list.  It's that simple.  Instead of writing code, start by getting yourself organized and enumerate all your fine grained tasks, no matter how small.  "Find compression and decompression entry points".  "Get compression code into build system".  "Implement dummy implementation to verify integration works." "Write a test harness".  etc.  Once you have a list, sort in terms of priority.  And if you have any tasks that would take longer than a day, go ahead and break them out into smaller tasks.

    You now have a clear road map from A to B, broken down into actionable items at a very fine grained level.  There's no more uncertainty.  If you're driving, this is the part where your GPS has figured out how to get you from  your current location to the nearest Five Guys without having to ask for directions or pull over.

    So what do you do if you're still not getting shit done?!  The glib answer is something along the lines of...

    ...but this doesn't really help, now does it?
    It's likely that you're just not interested in the work.  If you're passionate and interested in something, then the work is fun and interesting and you want to do it.  Unfortunately a lot of the work we do in software -- in fact, the bulk of work in software development -- is excruciatingly dull.  Finding obscure bugs.  Dealing with build system issues.  Hunting down compiler switches.  Typing in the same code we've typed in hundreds of times before.  Dealing with poor or missing API documentation.

    For this stuff, you just have to grind.  This is where work ethic and discipline can come into play, but I prefer to cheat and just try to stick to things I find interesting.  Unfortunately the amount of interesting things to work on in a typical software project is dwarfed by the sheer amount of bullshit tasks.


    Posted by Unknown
    0

    Add a comment

  6. Related to my principle of "suffer no jankiness", I've realized that almost all bugs can be distilled into three main categories: I Have No Idea What I'm Doing; Oops; and Oh Fuck You.

    I Have No Idea What's Going On bugs are the worst, because they're due to a lack of comprehension of what's supposed to be occurring in the software (i.e. jankiness).  We attempt to "fix" these bugs by near random transformations until things "work", but of course it's just masked, at best, until it crops up somewhere else.

    Examples of this include:

    v.z = -v.z; // WTF?  No idea why this works.

    And:

    char buffer[ sizeof( Foo ) + 128 ]; // Add extra just in case...

    "In case" of what?

    LaunchThreads();
    Sleep(1000); // give it some time to startup

    "Some time"?  Where did the magical 1000ms value come from?

    And:

    if ( !system.Initialized() )
       system.Initialize(); // Just in case it wasn't initialized already

    Any time you see "just in case" in a comment, that's bad.

    I Have No Idea What's Going On bugs are a symptom of flailing about trying to make something work that would probably work if you just took a step back and started over from first principles.  But you're in a hurry, you're bored, you're frustrated, and you hope that one more transposition or sign flip will suddenly make everything right.

    These bugs are also really, really bad because they breed.  They create more bugs like themselves and they also create a ton of Oh Fuck You bugs for other programmers who have to figure out why the hell you decided that you needed to reset the graphics hardware state every time your function is called "just in case" it wasn't in a known good state.

    Oops bugs are the forgivable ones.  They don't show a lack of comprehension of the software, they were just dumb oversights, typos, copy and paste bugs, off-by-1, etc.  These are also generally the easiest bugs to hunt down because you're starting with a concrete mental model of how things are supposed to work, and then figuring out why it's not.

    double l = sqrt( v.x*v.x + v.x*v.x + v.x*vx );

    Those happen to everyone.  Yeah, you can avoid them with better testing and what have you, but by and large they're not symptoms of incomprehension.  In my experience each programmer has a fixed rate at which they create these bugs, which is a reflection of their habits.

    Oh Fuck You bugs are the ones where, once you realize what's happened, you say "Oh, fuck you [entity name here]" because someone else has inadvertently sent you on an hours or days long bug hunt that is not of your doing.

    Compilers generate incorrect code, tools corrupt data, SDKs are incorrectly documented, third party libraries do the wrong thing, drivers are broken, someone overclocked their CPU which led to weird memory errors, etc.  These bugs tend to occur at a fixed rate irrespective of the individual programmer since they're out of control of the individual programmer.

    In a perfect world you only ever encounter Oh Fuck You bugs, but if you can keep the frequency of Oh Fuck You and Oops bugs higher than those of I Have No Idea What's Going On then you're probably doing pretty well.

    Posted by Unknown
    3

    View comments

  7. I'm fairly old (40+) for a programmer, and you'd think most skill jumps would have occurred by now, and I think that in terms of raw ability that's true. Most people learn and apply everything in their first 10 years or whatever of doing things -- their experience will broaden as they get older, which provides a larger set of experiences to pattern match against, but by and large I think their basic 'programming smarts' is set at that point.

    BUT, philosophical outlooks and habits can still grow, and these can materialize as skill jumps. Lately I've been trying to be more intolerant of perceived jankiness in my code.

    A quick definition: code jankiness occurs when the mental model of your software is no longer accurate or complete.  Hacks are not janky, for example, as long as you know how and why they work.  This, on the other hand, is classic jankiness:

    char buffer[ sizeof( foo ) + 128 ]; // a little extra, just in case...

    Every time you see "This should never happen, but just in case", that's janky.  In fact, any time you see "Just in case" in a comment -- probably janky.  Defensive coding?  Total jankiness.  Checking against NULL even when there is no reason why that pointer should ever be NULL?  Janky.

    "Creeping jankiness" are all the little things that make you go "Huh, that's weird" but you ignore them because, hey, it didn't crash and nothing obviously wrong occurred. The absolute best programmers I've ever known, without exception, are radically intolerant of their software acting in any way that does not match their mental models. If something happens -- no matter how small -- that they do not understand, they stop and go "What just happened?"

    Even if it takes them all day and even if it means that they fall behind in other areas.  It seems like a net productivity loss, but it's actually a net productivity win in the long term because you maintain a rock solid understanding of how your software works.

    I can think of at least three great examples of this.  The first was when I was at 3Dfx and the Voodoo chip was being developed.  Gary Tarolli, 3Dfx's CTO and the principal architect of Voodoo (then known as SST-1), was running some simulations that took a long time (a single frame of rendering would take well over a minute).  We'd generate the frames (sometimes over night), then play them back as a movie for demos or whatever.

    Obligatory reminder of that horrible 3Dfx logo
    We did this with one of the demos, and during rendering there was a pixel that flickered for exactly one frame.  That was it.  Hardly noticeable, but there it was.  As a software guy I remember kinda shrugging and thinking "Screw it, it looks right most of the time and it didn't crash, SHIP IT!"  But Tarolli would not have any of that.

    After spending what seemed like a huge amount of time (I want to say it was at least a day), he found the bug and discovered that it wasn't a bug, but an unintended side effect of a feature that was totally legitimate.

    The second example was when I was at id software.  John Carmack was absolutely intolerant of rendering errors -- especially cracks, slivers and T-junctions in his BSP renderer.  He was devoutly religious about it, and this is one of the reasons that the Quake games had such an immensely heavy, solid feel to them.

    So of course one day we're testing things, and we see the occasional flash of a crack or a sliver.  This was pretty much an all-stop situation for him.  He spent a long time (I want to say at least a day -- and one Carmack Day is like Five Regular Dude Days) and finally narrowed it down to an issue specific to an ATI driver or card.  Now, most of us would be content to be all "Whew, not my problem, let's move on!" but of course Carmack had to know how the ATI driver was broken.

    It turned out to be their guard band clipper.  Guard band clipping is a graphics driver optimization, but it can introduce cracks when done wrong.  If you're going to use guard band clipping, you have to clip against the guard band planes all the time.  You can't clip against them for some triangles then clip against the viewport for others -- that can introduce slivers between triangle edges.

    I just needed a Quake 2 screenshot for symmetry...and to remember how goddamn dark that game was
    I don't remember if that was a hardware bug or just a driver bug, but the point is that Carmack knew he didn't have those types of rendering errors, and when confronted with them he had to track down what was causing them.

    A more recent example of suffering no jankiness is Casey Muratori's dissection of a five second Windows stall.  It's obvious that once the problem was fixed he could have moved on, but he was compelled to figure out what the hell was going on under the hood so he could have a better understanding of the software's interactions.

    That diligence reaps huge benefits, because very often these small problems are the first cracks in the codebase. By chasing after these now you stop larger problems from erupting later. In addition, by ensuring that only the things that you expect to happen actually happen, you're forced to keep a mental model of exactly what you think is happening in your head. If you become flippant about jankiness it's easy to eject global understanding ("This is how the program works") to focus on localized concerns ("Why is this function crashing?"). Once that mental model of your software is corrupt or vague, you're in major trouble because you no longer know what to expect, and it becomes extremely hard to even detect non-crash bugs because you can't tell the difference between a bug and expected behaviour any more!

    I used to blow off this attitude as overengineering or intellectual wanking -- if it worked, that's what mattered, and I can be getting in a new feature while you're sitting there obsessing over code that works (for the most part...) -- but since then I've decided I've been doing it all wrong, and I need to have a solid understanding of all the code I write, if for no other reason than it makes things easier on me in the long run.

    Posted by Unknown
    0

    Add a comment

Blog Archive
Blog Archive
Loading
Dynamic Views theme. Powered by Blogger.