This is the script of a lightning talk I developed for the Boston Software Craftsmanship group. It sparked some great discussion around craftsmanship, good programmers vs. bad programmers, and computer science education. There were too many ideas to put to paper, really, but I'll reproduce the original talk here and let you draw your own conclusions.
So, I think I just had a revelation. Most software developers—myself included—are basically just labor. Skilled labor, yes, but basically similar to construction workers. We always talk about crafting things. Often those things aren’t very difficult to craft, logically or mathematically. “When someone checks this box, disable this field; when the database is full, show a warning message,” and so on. We often talk about “plumbing” or “wiring things through” as if we’re plumbers and electricians. There’s just a lot of parts of programming that aren’t hard, they’re just tedious and they take time.
Now, there are some programmers who are engineers, who do hard math to develop the core algorithms. But the rest of us just polish up those algorithms and build a harness for them, build an interface for people to interact with them. Often times the “houses” these algorithms live in are the builder's own design, so they are often as skilled as an architect, but the majority of their time is spent doing things I would more closely associate with carpentry or woodworking (to borrow an analogy from The Pragmatic Programmer). And anything really complex can only be handled by the engineer.
That's how it is at my company; my team builds the interface for the visualizations created by our statisticians (sometimes we create the whole visualization, and they only write the algorithm that spits out the p-values, but the analogy holds). Maybe at your company it works a bit differently. Maybe you work for a startup, where you wear both hats. But I still think that most developers work in a labor capacity. And this is interesting, because we often like to think of ourselves as pretty intelligent people; we went to school alongside all the other engineering students; we see ourselves as problem solvers; we see ourselves as nerds. But as for myself, at my current stage I would rate myself a laborer — maybe an architect (on a good day).
So there seem to be two distinct domains here. Now how might that change the way we think of ourselves?
I’ll give you an example from an algorithms class I recently took. We had to write a program to process a graph with 875,000 nodes: this involved two very different, very distinct subproblems.
One was coming up with the algorithm itself, to efficiently process the graph and solve the problem. A purely mathematical problem, minimizing time complexity while proving correctness. This part was given to us by the professor.
The other part—my part of the problem, since I had chosen to do all my homework in Python—was how to get Python to handle the damn data. As it turned out, this was a harder problem than I had expected, since Python isn’t particularly performant. It took minutes to run, so you had to come up with some really good test data instead. Not only that, but once you ran it on the full dataset, it would overflow the stack immediately, so you had to either learn how to increase Python’s stack size and recursion limit, or you had to turn it into an iterative algorithm (which in essence just means that you maintain the stack yourself instead of hijacking the program stack).
So this was a completely distinct problem, and is more of an practical problem than a math problem... somehow it’s just a bit “softer.” It doesn’t necessarily need to be solved with equations (maybe we can throw more hardware at it, or switch to C instead; you know, come up with some practical workaround). Whereas the other problem definitely does require equations and proofs and deep analysis.
An aside: I’ve met a lot of programmers who shrug off mathematical problem-solving as being outside their realm, or impractical, or overkill, or premature optimization. They don’t seem to appreciate the fact that the only reason our programs can run with any kind of reasonable limit is because of all the hard work already done for us in the languages that we use, the compilers that build them, the libraries that support them. You can’t just parrot “optimization is evil” and pretend it doesn’t exist! But I digress.
Point is, both these kinds of computer science were necessary to solve the problem of the 875,000-node graph. And both were a big pain in the butt, believe you me. But the algorithm was already done for me. And I bet it took a lot more work than the afternoon I spent on turning it into an iterative implementation. But then again, maybe some of these practical engineering problems really do take just as long to solve well.
Take all the recent obsession over “big data” and scalability for instance. There are some seriously hard problems there. Many of them are operational problems, the realm of sysadmins. Many of Google’s problems involve machine learning, heuristic-based searches, etc. Many of Facebook’s problems involve very efficient graph traversal and storage. But are these problems that are solved by the same programmers who create the user interface for Gmail? I honestly don’t think so. I suspect that these companies have a separation between their scientists on the one hand, and their construction workers on the other.
You might take our meetup for example. What do we discuss here? We discuss best practices. We discuss design, and aesthetics, and agile methodology. We discuss soft things, subjective things. How many programming meetups do you know of that go over novel data structures; or where the attendees solve math problems, or discuss algorithmic complexity. I’ve yet to find a meetup like that. A meetup where I can actually learn hard things.
It begs a lot of questions. Are we really as skilled as we think we are, or is it just that not many other people enjoy doing what we do? Are we in demand because we’re skilled, or because the “construction” business is booming? Should universities be separating these two types of students, and teaching them different things? Should they have bothered to teach me all that linear algebra, if I was just gonna forget it anyway?
I think this also ties into recent efforts to get kids and others interested in programming: the FIRST Lego League, Google App Inventor, Code Academy, and the like. I think I’m not the first person to realize how easy programming really is, and how much demand there is for it, and that maybe if we can just get people to try it they’ll discover how easy it is, too. It might be the first time that a labor job was so plentiful and yet so understaffed, because people associate it with higher-order thought.
And speaking of higher-order thought, I still have more questions. At what point does a problem become complex enough to enter into this “mathematical” category? Are there really two distinct categories of programming problem? Or is it perhaps a linear progression, a gradient running from easy to hard to NP-hard? … I dunno.
What kind of computer scientist are you? Are you the mathematician? Or are you the carpenter? Or perhaps a bit of both? Is one more “elite” than the other? I dunno... but I think at least the mathematician in all of us deserves a little bit more respect.