Cartography

Maps and Layers

If you've ever played around with maps (paper or digital) you'll have noticed that they change dramatically with the scale of area covered. If you're looking at a map that details a single city or town the detail will be so high as to point out individual museums, alley ways, and the like. If on the other hand you look at a map of a whole state, all of that detail will have disappeared to be replaced by the major routes, towns, and features of the state. There is obvious usefulness to this change in the level of abstraction. If you're trying to understand how to move throughout the state, any detail beyond the major routes and towns is just going to overwhelm and confuse you. But if a high level of detail is not present when you're navigating your destination, you'll never get where you're going. This property of maps provides a nice perspective for what I think is a general rule of human thought and organization. There is only so much detail we can think about at any one time without getting overwhelmed and losing the value of the information. Therefore the only way to organize extremely large systems is to create multiple layers of organization that build one on top of the next.

The process for building these layers is relatively straightforward. Essentially you start by building a "map" at the highest level of resolution until you recognize that you've reached that "overwhelming limit". At this point you create a new layer of abstraction that allows you to break your full map into many individual maps with a layer on top to guide you through them. Then you keep building out your map with these layers until it becomes clear that your uppermost layer is once again too dense at which point you create a new layer on top of that and repeat.

This hierarchy of layers is what allows us to organize any amount of detail and scope without blitzing our brains. It should also be noted that building a hierarchy of layers this way doesn't only help us build the map in the first place, but also makes it much easier to think about and update in the future. Rather than having to update or think about the whole space all at once, we can focus on one piece at a time. In summary, these layers allow us to tame the space and think effectively.

Turning back to our catalysts we can already see that we've been doing this layering. Each of the questions we've outlined has represented a different "town" or "province" within the larger map of each catalyst. By building out a "map" for each of these we've been able to concentrate the full detail of each of our catalysts into manageable droplets while also providing detail on how they all connect.

Yet as we build more and more successful tools, opportunities to combine our catalysts and build off of them will arise. You can think of this like individual hamlets and villages becoming aware of one another. No longer do maps of each town suffice - we now need a map of the whole continent. And so, a new layer arises. This new layer represents the ways in which catalysts connect and form entities far greater than the sum of their parts and thus ends up being a mapping of the full problem space in which the catalysts exist. Obviously this is a very important layer, so how do we go about building it?

Drawing the Problem Space

The whole point of this new layer of abstraction is to understand how our individual catalysts can connect into components larger than the sum of their parts. But before we can connect them we must know what we are trying to connect. This gives us our first step in drawing up this new layer of our map:

Identify the catalysts and tools (not all will necessarily be built by you) that you're looking to connect and coordinate.

With that done it's time to start making connections! At first blush this seems like a pretty vague and undirected activity, but I find that starting from the goals and working backwards really gets the creativity flowing. We can begin by collecting what we already know:

Catalog the supported activities, intended catalyzations, expected extensions, and currently known use cases relevant to the tools and catalysts in question.

Having the known goals laid out gives us an opportunity to consolidate. Are there overlaps between these various goals? Are some of these goals actually multiple sides of the same coin? Are there themes throughout that can be discovered and elaborated upon? Through this exercise we'll end up building a map of the problem space that is disentangled from and therefore elevated above the specificities of any one tool or catalyst - we're starting to a get a view of the bigger picture.

Consolidate the problem space into a map disentangled from and thus elevated above the individualities of each tool/catalyst. Get a sense of the bigger picture.

This exercise tends to leave one with a horribly thick net of connections between catalysts and problems. Suddenly, one catalyst will be handling way too many problems, different catalysts will be solving the same problems, interfaces will be missing, etc. All of which means it's time for modularization!

Reconsider the modularization of your catalysts and tools based off of the newly consolidated and mapped out problem space. Fill in gaps as well.

Now that we've got these new catalysts, we can sit back and think about whether new opportunities exist for these individual catalysts. This brings us back to step two. Indeed, from this point forward we can just iterate through steps two to four until we reach convergence. Congratulations! You've got yourself a new layer.

Before moving on, I'd like to note something. In building out this layer we started with catalysts, used them to build out a new problem space, and used that new problem space to refine our set of catalysts. This tendency for updates in one layer of abstraction to change other layers of abstraction is absolutely standard. No layer represents the truth or has the final word. As each layer develops it will inform and update the other layers. This is why map building of this kind is an iterative and not reductive process.

Avoiding Madness

So far we've talked about what these maps are and how to build them. What I'd now to do is spend a moment getting a little closer to why they are so important. To do so let's imagine that you're a developer helping build out a particular catalyst. The tools you are building have become successful enough that many people are taking notice and you've started receiving a lot of feedback and ideas from stakeholders and developers alike. At first the general human reaction tends to be to take these bits of feedback and throw them into the mental equivalent of a bucket - just a big bin of largely unorganized ideas. This bucket represents the counterpoint to a clear, map-like organization. So let's understand just why the bucket can become such a problem.

The first problem we'll draw attention to is that of busy work. There is nothing saying that all the problems being thrown at you are coherent. Some ideas may end up sending you in one direction while another set may be directing you to go the opposite way. Some will lead to valuable long term gains, others are stop gaps for the problems felt most urgently today. Just pulling things randomly off the queue is therefore akin to sailing a rudderless ship. Without organizing all of the ideas being thrown at you, you'll just end up pulling things out of your bucket without really having the confidence that you're actually eating into the queue in a meaningful way.

The second issue of note is that of lost ideas. It's much easier and faster to create ideas than it is to actually do the work. Therefore, without a means for organizing and filtering out ideas, your bucket is just going to get fuller and fuller. This means that no matter how hard you work, ideas people are throwing into the mix are just never going to get seriously looked at and will end up lost. Not only does this mean we're losing valuable ideas, but it will also make your stakeholders feel as if they're not being listened to. And that feeling of being unheard contributes to our third problem - duplication.

Once people feel like their ideas are at risk of being lost they start sending you the same ideas repeatedly over and over again to try to impress some sense of urgency. This results in our bucket filling even faster and potentially overflowing even though there's no new content being added. Put another way, you as a developer are left with a feeling that there is a lot more work than there actually is - something that will quickly lead to feeling overwhelmed.

Finally (although this list is in no way complete) there is the issue of prioritization and ordering. Stakeholders will simultaneously push for work to be done on whatever they believe is the most urgent and at the same time not have consensus on what that is. So unless you have clear reasons for ordering things in a particular fashion each will spend a lot of time lobbying for their particular ordering. This incessant lobbying simply contributes to the sense of confusion and chaos.

In short, just throwing things into a bucket leads to enormous amounts of noise from stakeholders, rudderless sailing, and a growing queue of tasks made even larger by duplications - in a word, chaos. Obviously, being able to find flow in this kind of environment is next to impossible. Instead developers end up spending more of their time fielding requests from frustrated stakeholders and trying to make sense of the chaos all while feeling overwhelmed, frustrated, and stuck.

This is why building a map of the space is so incredibly important. Going back to the spatial map analogy, throwing everything in a bucket is like going out into the wilderness with no map. Before long everyone is shouting at everyone else, no one knows where they are or what's going on, and there's clearly no direction. On the other hand, with a well layered map, we can clearly identify where we are, where we want to go, and how we're going to get there. Indeed if at any point we're uncertain of our path, we can just look back at the map, clearly see the plan of action, and then go back to enjoying the hike and the landscape. By creating these layers of organization we carve out space that allows people to just relax, focus on the task at hand, and find flow, knowing that the route ahead is laid out and so long as they follow the path they'll eventually get to their destination. Within complex problem spaces maps aren't just handy, they're essential for people to find flow and have happy and productive work. Important indeed.

A Changing World

There is though, one issue with my trail map analogy. Whereas trail maps don't change during the course of your hike, maps of a problem space change with every new piece of information. With each new MLP, idea, and experiment the map changes and, as we discussed earlier, changes at any layer tend to have ripple effects into the layers around them. As a result, in order for folks to have confidence in the map, there needs to be a clear process around incorporating changes that everyone trusts and is timely enough to keep things up to date without causing outright thrashing. What creating this process looks like is a topic for another post, but what I wanted to point out here is that because the landscape is ever changing and because there is a process that has to be maintained, driving the cartography of a problem space is a job in and of itself. And it is not a job that can just be handed off to the group because:

1. There is a lot of work here. Map changes mean the whole map - from the individual hamlets, to the regions, to the whole country. From the finest details to the most overarching strategies there is an enormous amount of information to keep track of, maintain, and develop. This is far too much "overhead" to encumber a whole group with.

2. Distributed responsibility never works out well. There has to be someone, at the end of the day, who knows the responsibility falls on their head.

So, if you want to keep people productive and happily finding flow, you need not only maps but a dedicated cartographer to build and maintain them.

Sailing Through the Storm

Building out a fully layered map of your problem space is not just a matter of convenience but a necessity if you hope to be happy and productive. Such a map is only complete when you're able to reach from the details of the tasks, into the designs of the catalysts you're building, and all the way up to how those catalysts combine into a broader strategy. Given the fact that changes at each of these layers ripples out into the others and that change comes with each new piece of development, experiment, or idea, keeping this map current is a job of its own. So, if you want happy development teams producing exceptional work, make sure you find yourself a dedicated cartographer and give them the space and resources required to keep your maps up to date. For without them your team is but a rudderless ship in a storm of ideas.