1 of the last things I worked on in my previous job was converting a cron-generated set of emails about server SLA stats into a web service. Early on into this process, I looked at the ELK stack, discussed it with my team lead, and the two of us ultimately came to the conclusion that going that route would be overkill, and that instead we should just write our own utility. That, boys and girls, was a stupid, stupid mistake. Instead of using an existing, free, open-source, project, we decided to re-invent a wheel that other people had already invented, debugged, and was working well for a lot of projects, including bigger ones than what we were intending to do.
There are plenty of companies bragging about the custom tools and libraries they built to solve a problem that was solved for smaller problems, but “not at their scale”, or some other excuse for why their problem is a special snowflake that required building their own special library around. This is as opposed to adding features to an existing open source project and releasing those changes back to community- the worst-case scenario is you still end up with your own custom project but you had a running start using well-known and established code base, the best-case scenario is that everyone gets a better version of the library they were using and you don’t have a whole new project to manage and maintain. By the way, this whole paragraph was about people creating new libraries because their project is oh-so-big. We wrote our own ELK-like utility because our project was oh-so-small. That should have been the first glaring red flag in this plan, but yeah, we pushed on anyways.
The biggest thing you’re going to get when you re-invent an existing wheel is bugs that somebody else has probably solved already. The secondary benefit, if you’re situation really is all that unique and different is that you’re going to get all-new bugs that you have to fix instead – yippee! I’m sure that’s just what everyone wants from their software engineering projects, time spent fixing bugs when you could be re-using perfectly good components and moving on with the things that actually make your company money. Remember folks – you should be prioritizing your time and attention on your secret sauce – not side utilities that duplicate stuff that’s already out there, especially free and open source projects.
Just because the ELK stack seemed like overkill at the time didn’t mean it was a bad fit for what we were doing. It not only covered 90+% of what we were trying to do, but it also added a lot more functionality as well. The one remaining piece of functionality could easily have been solved by me writing a simple plugin and installing it, rather than starting with custom software from scratch. Not to mention we wouldn’t have had to maintain this project, there would have been an entire company doing that for us, all we would have had to do is keep 1 little plugin working, and to be brutally honest, we probably could have had the whole thing up and running doing exactly what we wanted without writing the plugin. Even better, we could have starting doing a lot more with this utility faster, without a single bit of extra development effort. There was no way that wouldn’t have been a huge win for us in the long run.
Odds are, you’re not inventing something new with these libraries, so if you’re going to go ahead and write 1 of these things, then you have to have a very clear, concise, well-thought, and well-documented reason for why you’re doing this. Generally speaking, you can make whatever it is you’re doing work with 1 of the many existing libraries out there – especially if they allow developers to write their own custom plugins or other forms of customization. Going that route is generally the best (and fastest) way to get your project up and running. It also takes advantage of the existing community supporting an existing library that’s generally already stable and largely bug-free.
By the way, that aforementioned SLA reporting project – took way longer than originally anticipated, had a very minimal set of hard-coded functionality, was buggy, and sucked up months of time testing and ensuring data accuracy. In short, the project was a disaster, at best. All of this over something that already existed, and could have delivered more functionality immediately if I had just used the software that was already out there.
For a profession that’s allegedly against re-inventing the wheel, we seem all to eager to do just that with our utilities. It’s time we stopped doing that and instead focused on taking the tools that already exist and enabling them to be what we need them to be. Only when that’s just not possible should we start writing new tools from scratch. That’s especially true when the tools that are already out the are open source and can be incorporated for free. So instead of re-writing yet another tool from scratch, use what’s already out there. If it’s overkill, use it anyways and have something that will easily grow along with your needs. If it’s not enough, extend and enhance so that it’s even better, submit the new and improved version back out to the community so everyone can benefit. Either way, get your tool up and running and then get back to whatever it is that’s unique to your business – that’s where all the really important work is getting done.