Recently, my friend Steven Hicks published blog post on complex software. Despite his claims early on in the post, what he’s describing is exactly a documentation problem. Nobody actually takes time to document how to install and configure these services and as a result you never really know how to get these things to ever work together. While it’s comforting to know that he believes complex products are perfectly viable, the issue isn’t that people aren’t making complex products, it’s that the usage guides for the products out there suck.
The issue isn’t that people aren’t trying to make complex software, it’s that they’re trying to abstract out the complicated parts. Stuff like Mesos, Swarm, and Kubernetes are meant to help create and manage clusters of computing resources like it’s 1 giant server. In this set of cases, they’re abstracting away the details involved in managing a bunch of computing resources (be they servers or Docker containers) with the intent of letting you focus on building out your application and not worry about how to manage running it on a cluster of machines.
All any software is trying to do is help you solve a problem so you can focus on other things that are more central to your daily life. The software mentioned above helps you manage a resource pool so you can focus on whatever large computing project you’re using that pool for. AWS’s Lambda expressions are designed to let you run simple snippets of code without having to deal with spinning up and running any server(s) for that code. RabbitMQ is designed to help developers respond to events from somewhere else in the system. All of these products actually do a pretty good job of solving their respective problems once you get them working. But the issue isn’t that none of the examples in this post are complicated pieces of software – they’re components of (what are often) very complex systems. The issue is that nobody bothered to write down what types of things you can connect to this building block, and which pieces of said building block are the connecting bits. In other words, documentation of these pieces of software is bad and as a result there’s getting any of this to work together is a nightmare.
Generally speaking, most “getting started” examples give you a “Hello, World” style setup and then declare you ready to go off and do anything you would want to do, rather than building an incrementally more complicated project until you have a simple enough application that you can use it as a template for the initial version of whatever you’re trying to do.
All of this said, good software encapsulates complexity – if it’s not making our lives easier or more productive, why in the world would anyone use that application? Just because the software is managing a complex task for you doesn’t mean it needs to feel complex. If it did, you might as well be doing all of this stuff manually. It’s important to remember there’s an important distinction between “software that does something complex” and “complex software” – Steven seems all in favor of the former, but his blog post is lamenting the latter.
Any complexity software does expose ought to scale with the power level of the user. Someone just starting out should be able to follow a very hand-holding process with sane defaults and end up with something good enough for their simple needs, whereas someone with more familiarity and a more complicated problem should be able to dig deeper into settings, plugins, APIs, etc. and put together something better suited to their situation.
Now, for people just getting started with new software who would claim that their problem is already too complicated for largely going with defaults, I’d say that they’ve done a poor job breaking down their problem. Remember, you’re just trying this software out. You should be building a basic proof-of-concept first, then learning some more about it to gradually get it to fit all of your needs. It takes time to master new software and get it tuned to your exact problem. That’s not to say the time isn’t worth it, but it’s a cost of improvement that you need to be willing to pay.
Overall, I think software has done a good job of abstracting out complexity in solving technical problems. I think we’ve done a bad job at documenting our solutions, and that they’re not nearly as intuitive as we like to think they are. That’s not a problem with companies avoiding complex software, but rather a documentation failing. We’re not doing a good enough job of identifying our target userbase and putting ourselves in their shoes. In the end, the problem with getting complex software up and running is, as it always has been, a documentation problem. The sooner we can own that fact, the sooner we can start fixing the issue.