Why Habit Formation is Hard

Recently, I moved from Las Vegas to Seattle. In the process I realized that activities like moving belongings and getting a new driver’s license are not the hardest part. The difficulty of moving habits is much higher. About 80% of the cost of a move, I suspect, is the cost of moving habits. We lose months of time in the run-up to a move and after.

An example is your gym routine. It’s possibly the most important habit in your life. But it is surprisingly hard to “move” from one context to another.

In my case, I signed up for a gym very similar to the one I used to go to in Vegas. It has similar facilities and a similar range of equipment, trainers and programs. Like my old Vegas gym, my Seattle gym is about a mile and a half from home. The membership cost is about the same.

Yet, it’s been more than a month and I still haven’t found my rhythm. By contrast, when I joined the Vegas gym, it took me less than a week to settle into a great routine.

Why is this?

The Structure of Habits

A habit is a stable, repeatable pattern of behavior that involves minimal meta-cognition, and achieves predictable results within a particular local range of conditions, defined as a combination of a cognitive context and a physical context.

In other words, it is a predictable behavior you can execute without thinking too much about it, so long as you are in a particular state of mind and in the right place/time for it.

We can borrow a term from mathematics and call the “local range of conditions” the region of attraction. Think of it like the gravitational region around a planet. Within a certain distance, and a certain range of velocities, small asteroids that enter the region will be captured into orbit.

So every instance of trying to execute a “gym workout” is like an asteroid flying by a planet. Getting captured into orbit is like successfully finishing a workout.

A large region of attraction makes for a stable and robust habit. A small one makes for a fragile, easily derailed habit.

From the definition, we can infer that every habit is actually two intertwined habits. There is a habit of thought and a coupled habit of action.

• A habit of thought is a set of coupled patterns of thought and a practiced ability to switch among them appropriately and effectively.
• A habit of action is a learned pattern of physical behavior involving sensory processing and physical movements.

Both are context-dependent. The former is dependent on your immediate state of mind, the latter is dependent on your immediate environment.

Some habits, like switching between brain-dump mode and edit-mode in writing, are almost entirely habits of mind. As you mature at the writing game, you become better at switching between the two modes at the right time. Inexperienced writers often get writers’ block primarily because they stay too long in one or the other mode and get frustrated. The action component is trivial (the physical behavior in both modes is typing or working with pen and paper; switching at most involves taking a short break or changing locations).

Other habits, like your bedtime ritual (end-of-day chores, brushing your teeth, changing into pajamas, and whatever else you do), are almost entirely habits of action.

Most complex habits are a mix of non-trivial thought and action components. Exercising is an example of a complex habit. For some of us, being outside of a narrow range of moods makes it impossible to work out, and expanding that range is hard work. For others, the mood is never a problem, but even a slight change in physical conditions derails the intention to work out.

Learning and Habits

Habits that have been successfully acquired involve very low meta-cognition, but getting there involves plenty of meta-cognition: learning the habit.

Almost all this learning has to do with increasing the size of the region of attraction where the habit “works.”  We can distinguish two phases of learning a habit.

1. Doing it right for the very first time (10%)
2. Expanding the region of attraction till it is large enough to be worthwhile (90%)

As we’ve discussed before, beginners typically learn (and get anxiously attached to) one way of doing something. For example, in a video game, you might learn exactly one fragile way to pass a level. In a level of a vertical shooter I used to play a few years ago, that “one way” involved starting in the bottom right and shooting the aliens in a very specific sequence.

This “one way” is obviously fragile, since if initial conditions are not exactly right, your behavior will be derailed. This is why you need the second phase: gradually expanding the region of attraction by introducing small contextual variations — both mental and physical.

In terms of the idea of a region of attraction, the one way of a beginner can be thought of as a single point. Any deviation from ideal conditions will derail the habit.

So if the behavior is “mounting a bicycle” and you can initially do it only when you are in a confident and energetic mood and are starting on a hill, headed down, you might expand the domain of your “bicycle mounting” habit by learning how to do it in a variety of moods and energy levels, and in all sorts of conditions.

Eventually, you hit diminishing returns. We all have different stopping points when we give up refining a habit. In mounting a bike, most of us give up once we can start on any reasonable grade, and learn the standing-astride and running mounts. Circus performers might try and learn how to mount easily from both sides, from behind, facing backwards and on extreme grades you’d never encounter on normal bike-paths.

This tells you that a habit is also an algorithm that has been gradually evolved to handle a sufficiently large range of conditions to make it worthwhile. Your investment in learning is wasted if the region of attraction does not grow sufficiently large.

The shape of the learning curve for a given habit (either new, or ported) in a given context depends on the complexity of cognitive and physical environments.

As a first-order approximation, you could say that things you learn about the physical context require very few repetitions. Once you’ve done laundry in a new apartment once, you’re pretty much done learning. The problem is that there are a lot of such things in a given physical environment. You might have to learn a hundred little things, each involving 1-3 repetitions.

Learning mental things usually involves more repetitions but fewer discrete elements. For example, when you go from high school to college, your study habits have to change because you are at a different level of the education game. It might take studying for several dozen quizzes or midterms before you settle into your new study habits.

Jokes aside, each kind of learning can go on indefinitely. People generally stop or declare a temporary detente once they start to derive a net positive “return on investment” for their learning efforts around one habit (or group of habits), and direct their limited learning energies to another front with higher returns.

As a result, our behavioral personality is always a set of habit-fronts at various stages of evolution. Some are stable, some are being learned (and bleeding red), some are being ported, some are atrophying. Some are in a state of diminishing returns.

Habits and Personality

Depending on whether they are primarily internally focused or externally focused, people tend to be better at either the mental or physical components of habit formation. It seems to be a zero-sum trade-off. I’ve never met anyone who was equally good at both. It’s like being right or left-handed.

I am pretty lousy at forming physical habits. I am much better at forming habits of thought. There are different sources of difficulty for the two kinds.

Habits of action are difficult to acquire because they require processing or memorizing a large amount of arbitrary information. To learn to get from point home to gym without too much thinking or a GPS, you need to learn various routes, turnings, traffic conditions, construction conditions, hacks, shortcuts, and so forth. Much of this habituation to the arbitrariness of a context is not portable. It is also not efficiently compressible, so it is energy intensive.

Responding to traffic signs is a kind of physical learning that ports within a country. But learning the quirks of a specific city (for example, here in Seattle, one rule of thumb is “avoid Mercer Avenue during rush hour”) is not very helpful when you move to a different city with different quirks.

Habits of thought are difficult to acquire for a different reason, though they might seem superficially easier to acquire. Your mind goes with you from context to context, so anything you learn about yourself, like whether you are more of a lark or owl goes with you everywhere. But learning about your mind is also fundamentally harder than learning something like “Maple Street is one-way east to west.” Since identity hangups, biases, demons and shadows all reside inside your head, every single useful true thing you learn about yourself comes at a 10x cost.

So you may not pay “porting costs” each time you move to a different physical or cognitive context (new city, new level of the mental game via a promotion or role change), but you pay more upfront, and the learning curve is much longer.

For those who like their finance metaphors, habits of action are op-ex heavy, habits of thought are cap-ex heavy.

Healthy habits are the ones which are delivering a return and have an appropriate region of attraction. When the region of attraction for a habit expands where it is potentially harmful, you have an addiction. Addictions can form once a habit is generating a predictable “profit” after the learning curve has been traversed.

Once you get used to the “profit” and come to expect it, the motivational structure for the learned behavior can change. You no longer exercise because you recognize the benefits. You exercise because you are addicted to (say) the “Runner’s High.” You might turn into an extreme, obsessive runner and ruin your knees.

Habits can also turn into aversions. This happens in two ways. Either the habit becomes a displacement behavior for another activity (for example, cleaning the apartment to avoid working on your thesis), or the habit itself can develop a convoluted region of attraction to avoid certain painful regimes.

So you might develop a habit of using humor to steer conversations away from uncomfortable topics and develop a (justified) reputation as a great conversationalist.

Many behaviors are ambiguous, in that they are healthy habits, addictions or aversions depending on the context. These are the behaviors around which developing a strong sense of narrative rationality is very important.

Porting a Habit

Porting a habit to a new context is hard because the learned elements that don’t port well cause a shrinkage of the size of the region of attraction. Sometimes the shrinkage is so dramatic that an expert behavior becomes completely useless in a new context, or at least “unprofitable,” to use our finance metaphor.

One way to think about porting a habit is to think of it as recompiling a program on a new computer. Depending on how big the context difference, you may need to do anything from tweak a few settings to rewrite the entire program to compile on the new computer.

For the gym example I started with, the basic context-dependent algorithm is the same in both cases:

1. Change into gym clothes
2. Go to gym
3. Workout
4. Shower and change
5. Move on to next activity

But this is deceptively simple. There is a whole lot of context dependency that is not captured here. For my Vegas to Seattle example, here are some key ones:

1. In Vegas, the weather is always suitable for any mode of transport, in Seattle, the weather varies in non-trivial ways that affect whether you can walk or ride a bike or drive.
2. In Seattle, driving the 1.5 miles is much harder because it is urban driving with a lot of pedestrians, bicycles, traffic lights and one-way streets. In Vegas, I had an easy suburban route with just one left turn and no pedestrians. I have to be 3x as alert to drive to the gym in Seattle. So I am sufficiently alert less often.
3. In Vegas, parking was not a concern. It was always available and always free. In Seattle, depending on the time of day, day of week and season of the year, my gym has different rules about parking and how much I pay. I have to get the parking ticket validated at the front desk. So the algorithm has a whole branch of logic having to do with parking decisions that didn’t exist in Vegas.
4. In Vegas, transitioning to/from the next activity was easy. I could make it up as I went along. In Seattle, if I plan to work at a coffee shop after the workout, I have to plan differently depending on whether I drove, walked or took the bus.

This is just a small subset of the differences. If I wrote out the pseudocode for “go to gym” for Vegas and Seattle, I suspect, the latter program would be at least 5x as long. Much of the added complexity is because my living situation in Seattle has a good deal more physical complexity in the environment, since I live close to the city center rather than in a suburb.

As an aside, the closer you live to the center of a city, the more computationally demanding all habits become. This is one reason where you live is such a useful variable for personality typing.

Acquisition versus Porting

You can quantify the difficulty of acquiring or porting a habit loosely by assessing the physical and cognitive context complexity. It is easier to assess the latter (porting) because you only have to assess a pair of differences between before/after contexts. As mathematicians know, the “differential” or “variational” version of any kind of problem is generally easier than the absolute version. There is even a well-developed theory of how to do it well (it’s called perturbation analysis).

Another reason porting is easier than acquiring a new behavior from scratch is that a lot of learning is illegible and unconscious. You have to build up a robust algorithm in your head that you don’t fully understand. Say a habit algorithm has a conscious component C that you could write down, and an unconscious component U that you are not really aware of. So moving the habit from one context to the other is about moving C to C1 and U to U1.

In my experience, the unconscious component generally ports far more easily. But when it breaks, it is also much harder to debug.

It’s like the algorithm for every habit has two components: a piece for which you have the source code, and a piece for which you only have the compiled binary code. The former nearly always needs some modification when you change contexts, while the latter usually ports without any effort. But when the latter fails to port, you’re in serious trouble.

Complex ports can be understood as ports where C1 and U1 are both functions of both C and U rather than C1 being a function of C and U1 of U.  In other words, you may have to consciously process previously unconscious components of behavior, and vice-versa. This takes a higher level of self-awareness.

In a way, context-switching and simply expanding the region of attraction in one context don’t differ much. So ports are simply disconnected expansions of the region of attraction. You are adding a planet, so you can capture asteroids around two centers.

Meta-Habits

A meta-habit is a habit you use to port or acquire other habits. For example, dive off the deep end, is a specific learning strategy (immersion) that can jumpstart any new habit. If you know how to handle the extreme disorientation, chaos and anxiety it induces, and if you know when it is safe to do so.

In the prototypical example of swimming, diving off the deep end is actually a terrible way to learn unless there is a teacher around. You will almost certainly drown if you aren’t close enough to the edge to thrash your way to something secure within a few seconds.

I don’t know if there has been much research into meta-habits, but here are some I am aware of, some of which I even practice.

1. Diving off the deep end
2. Confidence building with small wins
3. Not perturbing behaviors along too many dimensions at once (however, the naive experimentation idea that you should tweak only one variable at a time is misguided and inefficient when there are many dimensions).
4. Gradualism (push yourself only a tiny bit extra with each attempt to expand the region of attraction)
5. “Exercise to failure” (keep pushing yourself in one direction until you cannot handle a particular case, a matter of finding the boundary of the region of attraction)
6. Never make a decision when depressed, especially a quit/persist decision
7. Fail fast — not in the product development sense, but in the sense of quickly putting a scratch or dent on your pristine learning effort (remember the lowering of anxiety you felt when you first put a scratch on a new car?)
8. Shut up when learning a physical habit (verbalization slows down acquisition of tacit knowledge — if you have a teacher who talks too much during teaching of a physical behavior like swinging a tennis racket, find a new teacher: you need periods of silent repetition between being given instructions and suggestions)

Curiously, I am much better at incorporating some of these meta-habits into my teaching than into my learning.

Variety Reduction as Anti-habit Formation

You can come at habit formation from the other direction. Instead of trying to learn a habit to handle a sufficiently large region of attraction to make it worthwhile, you can change the environment to simplify the habit acquisition or porting problem.

You do this primarily by reducing variety and introducing homogeneity in both cognitive and physical contexts. This depends on your resources. If you are rich enough to hire a chauffeur you can forget about the complexities of driving. If you’re powerful enough that a lot of people want to work with you, you can pick and choose people who adapt to you, so you can keep your thinking the same.

“Simplifying” your life, far from being a monkish thing to do, is actually a behavior practiced most often by the rich. Monks (and sour-grapes foxes) simplify in a different way: by giving up desires, either mindfully or via sour-grapes rationalization.

For most of us, variety reduction is a limited option. We make up rueful aphorisms about the human condition that reflect this, like if you don’t get what you like, you are forced to like what you get. Shaw’s famous line,  “The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself” speaks to the relatively higher difficulty of changing the environment to reduce variety. It is an option that is dependent on your power.

But suppression of contextual variety to make habit formation cheaper comes at a price. It reduces your general ability to learn. This is perhaps the primary reason power leads to its own downfall. By making variety-reduction and homogenization an attractive alternative to the harder problems of habit formation and porting, power causes learning abilities to atrophy.

Organizational Habit Formation

Organizational habit formation is surprisingly similar to habit formation in individuals. We know it as economies of scale and scope.  The primary difference is that since organizations are typically far more powerful than individuals, the option of reducing variety and introducing homogeneity is more available to them.

Organizations become bad at learning and dealing with circumstances that cannot be homogenized away.

This is the reason industrial age organizations are associated with homogeneity and “seeing like a state.”

A major challenge in designing post-industrial organizations is to build into them the ability to accommodate variety and the discipline to not use the variety-reduction/homogenization option even when they have the power to do so. I’ve called this problem the “economies of variety” problem that I am still trying to figure out.

Thanks to Kartik Agaram and Jason Morton for useful discussions that informed this post.

Get new post updates by email

New post updates are sent out once a week

1. Tying in two related ideas:

2. Different forms of abundance (http://www.ribbonfarm.com/2012/10/04/the-abundances-of-ages/)

It worth pointing out how various forms of abundance reduce/increase the difficulty of habit formation, and how these interact with personality. Driving – in contrast to public transportation – is a big one I have noticed since moving to the bay area and giving up my car.

A car dramatically reduces the algorithmic complexity of almost all activities beyond traveling from point A to point B. On public transportation it is relatively easy to make short trips from A to B, but anyone who has ever relied on public transportation knows that travel time increases rapidly once transfers become necessary. That also means that multi-destination routes become much more difficult. A –> B and A –> C may both be easy trips, but if B C is difficult then each trip must be made individually.

The other point that is easy to overlook is the convenience of having a portable storage locker. Most errands are much simpler when you can pick something up, throw it in the trunk, and then move on with your business rather than carrying that item around with you all day. Without this convenience simple errands require far more planning and preparation.

Lack of parking (or complicated parking rules) is one of those caveats that undermines an otherwise simple activity. My friend Christy Pettit refers to these as “how will we get it up the stairs?” problems. Imagine you find some great piece of furniture, equipment or whatever…everyone is talking excitedly about all the things you can do with it, and then someone asks – But how will we get it up the stairs? – and immediately everyone’s enthusiasm is deflated.

Often this is particularly deflating because part of the enthusiasm is premised on an unconscious assumption of simplicity or the related desire for serendipity. Having to pay a few dollars for parking can dramatically skew our subjective cost/benefit calculation so we fight reality – either by avoiding the activity altogether or by driving around looking for free parking and becoming increasingly frustrated.

• Greg’s comment is sparking new ideas in my head. Has anyone tried to get organizations to resist variety-reduction by gradually increasing their lookahead? This could even be interesting for individuals. Going off on a tangent still more, is the difference between good and evil just a matter of lookahead horizon? Evil is simply the good you’re trying to do, the metric you’re trying to optimize, without looking ahead enough. Organizations fail to use variety-reduction well because we still don’t know how to incent their actors to operate on a longer Buxton Index (http://akkartik.name/blog/1240338). Does that explain why they are more sociopathic than individuals so far? We don’t yet know how to translate conscience to collectives. If corporations are our current best attempt at AIs, is creating Friendly AI just a matter of engineering it with a long-enough lookahead horizon? (Perhaps a too-long time-horizon would make humans incidental? But if it understands that plans are uncertain it will care about means and ends, about the importance of the trajectory in getting from here to there.)

• There is some sense to this. My first thought was that there has to be some hypothetical we could concoct wherein long-term thinking would still be entirely predatory. However, the further you look ahead the further your universe expands, i.e. the larger your sphere of potential interactions (interdependencies) becomes. So it might be reasonable to say that with a sufficiently long horizon pure self-interest would be indistinguishable from reciprocal altruism.

Though after pondering this further another objection occurs to me. At a certain point the calculating mindset itself starts to sound evil. Beyond a certain point rational calculation becomes impossible and “good behavior” comes to be perceived in terms of effectively timeless social norms. These social norms may not actually be timeless but we expect people to treat them as such. If a foreigner asked you why you say please and thank you, the correct answer would be – “because it’s polite”. You would just sound like a jerk if instead you said – “well it makes him feel good and there is a .x probability that I will need his help in the future…”

The latter example is not necessarily untrue. Obviously we can translate between the two modes of understanding. But still, the latter response begs the question – Why do other people expect you to say such things? The real answer underlying “because it’s polite” is that observance of social norms indicates that we are of the same culture, that in some sense we are on the same team. And that fact is an appreciative (static) reality, not an instrumental choice. Short time horizons aside, corporation seem evil (or simply unnatural) to the extent that they apply instrumental logic to domains that organic communities conceive of appreciatively.

• Josh W says

Or perhaps that communities consider as ends in themselves! (This may be another way to say the same thing)

There is a certain niceness in having an ordered and reliable community, in harmony in general. And excluding visible signs of global harmony or peace from your goals can either be the mystical seeking of peace beyond peace (like revolutionaries who seek to destroy the existing structures to produce better ones) or it can be just having no appreciation for the wellbeing of society as a cohesive construct.

I suspect that there are full ways to complete the circle, such that proper social cohesion instrumentally requires certain kinds of self-directing goals among it’s members, if it is to operate long term.

Of course it could be that that whole individual/society thing only works out as forming a loop for certain classes of societies and goals, or for certain goals within a society; the society needs me to continue following my goals, but both clash with that guy’s goals over there.

2. Alexander Boland says

Interestingly, there is always a need for constraints, and one could easily equate [i]power[/i] with the ability to reduce variety by means of relating them to [i]negentropy[/i]. The book that comes to mind is [i]Grammatical Man[/i] by Jeremy Campbell, which talks about some “sweet spot” between constraints and freedom via Shannon’s theorems. A bunch of other such books come to mind as well, but I digress.

The question is how to find this “sweet spot”. [i]Antifragile[/i] was one such attempt, and had the simple and effective (albeit slightly morbid) answer of letting the smaller parts crystalize and break for the benefit of a more elastic whole. That suggests to me that either organizations break, or they adapt by changing to the point of being continuous in name only.

This seems to lead to a very elegant theorem about narrative rationality: we learn in order to gain power, but that power contains the seeds of its own destruction. I notice that the most established intellects eventually become a bit too stubborn in their opinions–their OODA loops getting too static and not engaging in a dialectic with the outside world–and are destined to break in the same way that a great Fortune 500 company or sprawling empire will not be around for nearly as long as it thinks. It brings a whole new meaning to Hugh McLeod’s aphorism “Enjoy the obscurity while it lasts.”

3. Alexander Boland says

Sorry, messed up the tagging on the last post (some forums have square brackets instead of carrots):

Interestingly, there is always a need for constraints, and one could easily equate power with the ability to reduce variety by means of relating them to negentropy. The book that comes to mind is Grammatical Man by Jeremy Campbell, which talks about some “sweet spot” between constraints and freedom via Shannon’s theorems. A bunch of other such books come to mind as well, but I digress.

The question is how to find this “sweet spot”. Antifragile was one such attempt, and had the simple and effective (albeit slightly morbid) answer of letting the smaller parts crystalize and break for the benefit of a more elastic whole. That suggests to me that either organizations break, or they adapt by changing to the point of being continuous in name only.

This seems to lead to a very elegant theorem about narrative rationality: we learn in order to gain power, but that power contains the seeds of its own destruction. I notice that the most established intellects eventually become a bit too stubborn in their opinions–their OODA loops getting too static and not engaging in a dialectic with the outside world–and are destined to break in the same way that a great Fortune 500 company or sprawling empire will not be around for nearly as long as it thinks. It brings a whole new meaning to Hugh McLeod’s aphorism “Enjoy the obscurity while it lasts.”

4. I like your view on the added/altered “programming steps”. Related old post of mine re: habits and myelination:

Shut up and workout dude! Don’t intellectualize everything.

6. Josh W says

Public transport has a great relationship to urban complexity, when it works: Take the bus to the gym and back, make sure you have some basic lightweight clothing options in your bag for the walk to and from the bus on a cold or wet day.

Don’t take any extra stuff but a pad and a phone that is switched to airplane mode, both in a waterproof bag, and leave enough time to get there and back without having to do anything else. (Say an extra half an hour depending on if you have to catch the next bus.)

The problem with this is that places that are closer to bus routes are often on main roads and so more expensive, so it’s often better to have a short walk to places close to the bus route but not visible from it. It also means that people suggesting you do things after going to the gym or otherwise attached to this routine will seem very weird and unlikely to you. It’s a quite rigid and separated habit, based on extreme variety attenuation.

It also tends to include moments spacing out on the bus or waiting for it, pondering various things, but that’s what the pad is for, to make that useful pondering time.

7. Nice blog right here! Also your web site lots up very fast!
What web host are you using? Can I get your associate link
to your host? I wish my site loaded up as quickly as yours lol