Lucid Mesh

As programmers, as engineers, we solve problems all the time. It's a significantly large part of what we do. How do we solve problems efficiently and effectively? It requires a balanced mix of creativity and hard work. Both are hard to achieve, and a good combination of the two is even harder. Creativity requires getting yourself in a mood where you can let your thoughts run free and play with ideas, and hard work requires a mood where you can knuckle down and produce results. These moods are in constant conflict, and you can be at your most productive when you manage the two moods well.

I was going to make this post about how creativity is not what most people think. Instead of being driven by flashes of insight, I was going to argue that it was driven mainly by hard work. While I still think that's true in a sense, and I'll get into that more later, after looking at my notes and links to references on the subject, I believe creativity is a combination of insight and hard work in the form of intense thinking. Both things require time and space to accomplish, and both are required to creatively solve problems.

How to be Creative

I recently found this excellent talk by John Cleese on how to be creative. Much of his talk is based on the work of Dr. Donald W MacKinnon, and although I haven't read the book, I thoroughly enjoyed the talk by John Cleese. The gist of the talk is that there's no defined way to be creative, as that's somewhat of an oxymoron, but there are ways to enable creativity, and that is what Cleese focuses his talk on.

He starts off with an explanation of what creativity is, and it's not a personality trait, but a way of operating:

...the most creative had simply acquired a facility for getting themselves into a particular mood - a way of operating - which allowed their natural creativity to function. 

That mood is playfulness. They would play with ideas without any immediate practical purpose, but purely for the enjoyment of doing so. Cleese goes on to explain the two modes of operation that he labels the open and closed modes. Creativity requires an open mode of operation, but completion of the idea requires a closed mode of operation. To be most efficient, we must be able to readily switch between the open and closed modes.

The ability to get into the open mode requires five elements:

1. Space free from distraction
2. Time also free from distraction
3. Time playing with uncertainty before deciding
4. Confidence that silly thoughts may still be fruitful
5. Humor to keep the mind playful and exploratory

Once you achieve this setup, Cleese recommends about an hour and a half to play (with ideas)—a half hour to let your mind calm down and an hour of creative work. I realized that this time frame mirrors my own flow for blog posts pretty closely. Every post is a problem that I need to solve, and when I sit down at night to work on a post, I'll spend about a half an hour collecting my thoughts, looking through my notes, and considering what I want to write and how to write about it. Sometimes I have to wrangle my thoughts into submission because my mind wanders around, not wanting to focus on the task at hand. Then I'll have an hour or so of good productive time where I can write down thoughts and play with ideas in a pretty good state of flow. I'll reach a point sometime after the two hour mark where I start to tire out and my productivity slows down. I thought it had more to do with getting tired before going to bed, but it might be as much a factor of having worked creatively for an hour or two and I'm simply tired of being creative.

You probably noticed there was not one, but two time elements in the list. The second time element is surprising, and Cleese had a great justification for it:

One of my Monty Python colleagues who seemed to be more talented than I was never produced scripts as original as mine. And I watched for some time and then I began to see why. If he was faced with a problem and saw a solution he was inclined to take it even if he knew it was not very original.  Whereas if I was in the same situation, while I was sorely tempted to take the easy way out, and finish by 5 o’clock, I just couldn’t. I’d sit there with the problem for another hour and a quarter and by sticking at it, would in the end, almost always come up with something more original. It was that simple.

My work was more creative than his simply because I was prepared to stick with the problem longer. So imagine my excitement when I found this was exactly what MacKinnon found in his research. He discovered the most creative professionals always played with the problem for much longer before they tried to resolve it. Because they were prepared to tolerate that slight discomfort, as anxiety, that we all experience when we haven’t solved it.

I find this type of thing happening frequently when I'm trying to solve program design problems. I'll come to a workable solution fairly quickly most of the time, but if I pocket that solution for later and think about the problem a while longer, I can usually come up with a more elegant solution.

Humor may seem out of place in solving problems, but Cleese strongly held that creativity is closely related to humor - it is a way of connecting two separate ideas that results in something new and interesting. In the case of humor, it's funny, and in the case of creativity, it's useful for solving a problem. He certainly opposed the idea that humor and serious business should be separated:

There is a confusion between serious and solemn. Solemnity…I don’t know what it’s for. What is the point of it? The two most beautiful memorial services I’ve ever attended both had a lot of humor. And it somehow freed us all and made the services inspiring and cathartic. But solemnity serves pomposity. And the self important always know at some level of their consciousness that their egotism is going to be punctured by humor, and that’s why they see it as a threat. And so dishonestly pretend that their deficiencies makes their views more substantial, when it only makes them feel bigger…ptttttth.

Humor is an essential part of spontaneity, an essential part of playfulness, an essential part of the creativity we need to solve problems no matter how serious they may be.

Solemnity should have no place in the creative process. It kills our confidence in being able to play with ideas freely and distracts us from thinking fully about the problem by limiting thought to the safest, most acceptable paths. Humor is definitely more productive. My coworkers and I joke around a lot at work, and we get a tremendous amount of creative work done. On the other hand, I've never heard someone claim that their workplace is serious and they do especially creative work. More likely seriousness is accompanied with words like deliberate, careful, and protocol. Oh, and if a company tells you that they work hard, but they know how to have fun once in a while, that's a big warning sign.

Hard Work is Still Necessary

Creativity alone is not enough, and while creativity does involve hard work, coming up with a great idea to solve a problem is not the end of the task. Now comes the extra hard work of implementing the idea, resolving all of the details that you hadn't thought of before, and actually finishing solving the problem. Scott Berkun has written extensively on this topic, and it's some of his better writing. In his post on The Secrets of Innovation Secrets, he reminds us of how much work is involved in innovation:

If there’s any secret to be derived from Steve Jobs, Jeff Bezos, or any of the dozens of people who often have the name innovator next to their names, is the diversity of talents they had to posses, or acquire, to overcome the wide range of challenges in converting their ideas into successful businesses.

These people didn't succeed because of flashes of insight that just happened to them, and they didn't succeed by coming up with a great idea and selling it. The ideas themselves would have been nothing if it wasn't for their superb execution. (The importance of execution is one reason why patent trolls are so destructive to innovation—they haven't done any of the hard work of implementing an idea, but they want to get paid royalties simply for patenting an idea with no intention of doing the hard work themselves.)

Even the flash of insight that leads to an idea is the result of a lot of hard work. If it wasn't for all of the studying, researching, and experimenting that came before the actual moment of clarity, the idea may not have materialized at all. Scott Berkun again has some clear reasoning on why epiphanies are more of a process than an event:

One way to think about the experience of epiphany is that it’s the moment when all of the pieces fall into place. But this does not require that the last piece has any particular significance (the last piece might be the hardest, but it doesn’t have to be). Whichever piece of the puzzle is sorted out last becomes the epiphany piece and brings the satisfying epiphany experience. However the last piece isn’t necessarily more magical than the others, and has no magic without its connection to the other pieces.

I have this experience all the time when I'm trying to solve problems (which is also pretty much all of the time). I'll be mulling over a problem for hours, days, or weeks, struggling to find a workable solution, and then the answer hits me all of a sudden, usually while I'm laying down to go to sleep. I call it Bedtime Debugging because that's when it normally happens to me, but other people have it happen when they're taking a shower or brushing their teeth. It feels like a sudden eureka moment, but it would never have happened if I hadn't spent all of that time thinking about the problem, researching options, and studying the code I'm working on. The final connection that made everything make sense may have happened in a moment, but bringing all of the other pieces of the puzzle together so that that moment could happen took much more time.

Preparing for Epiphanies

I spend huge amounts of time reading and learning. I never know when I'll need to use a particular piece of knowledge, and the more of it that I have at my disposal, the better. Constantly exercising my mind and learning new things also keeps my thinking process flexible so that I can connect ideas from different fields and think about problems in new and different ways.

Our culture tends to romanticize the eureka moment while ignoring all of the hard work that's involved in the process because the eureka moment is so much more exciting than the work that came before it and must follow it. For one of innumerable examples, Cal Newport, an assistant professor of computer science at Georgetown University, contrasts the theatrical impression of Stephen Hawking's discovery of Hawking Radiation in The Theory of Everything with the reality:

In a pivotal scene in the Stephen Hawking biopic, The Theory of Everything, the physicist is staring into the embers of a dying fire when he has an epiphany: black holes emit heat!

The next scene shows Hawking triumphantly announcing his result to a stunned audience — and just like that, his insight vaults him into the ranks of scientific stardom.…

In reality, Hawking had encountered a theory by two Russian physicists that argued rotating black holes should emit energy until they slowed to a stationary configuration.

Hawking, who at the time was a promising young scientist who had not yet made his mark, was intrigued, but also skeptical.

So he decided to look deeper.

In the (many) months that followed, Hawking trained his preternatural analytical skill to investigate the validity of the Russians’ claims. This task required any number of minor breakthroughs, all orbiting the need to somehow reconcile (in a targeted way) both quantum theory and relativity.

The reality of Hawking's discovery is a clear example of the hard work involved in solving big problems. He needed to have a lot of knowledge about multiple complex fields of physics and develop new advances in those fields over a long stretch of time to make progress towards his goal of answering a question he had about a theory he had come across. Creativity was absolutely involved in this process, but the eureka moment is a vanishingly small part of the story. These instances of major breakthroughs and discoveries through hard work are the norm, rather than the exception. After all, if these moments were as simple and easy as they are portrayed, they would be much more common than they actually are. We can't all just sit staring into a campfire or reading under a tree and expect all of the answers to hit us in the head.

In a way, creativity is only a small part of succeeding in solving a big problem, such as producing a great product, creating a market around it, and building a business from it. The idea of what product to make is a drop in the bucket compared to the colossal amount of work involved in this process. Yet creativity is also an integral part of the entire process. Every step of the way there are little problems to be solved and details to be resolved, and doing this creatively is incredibly important. Creativity and hard work are intimately woven together in the process of solving big problems.