• • • •

The Case for DIY Software

When does it make sense to write your own software?

January 10, 2021

The Case for DIY Software

When does it make sense to write your own software?

Published: January 10, 2021.

When does it make sense to write your own software?

Perils and Bounties

In this blog post, I will be discussing the unique advantage that programmers have in using their own software for day-to-day tasks. The ability to write your own software provides boundless possibility, but this does not come without caution. I will be discussing some of the things that anyone writing software for themselves should be wary of, and how to manage the perils and bounties of DIY software.

Programmers have, to varying degrees of proficiency, an established understanding of how the software they use on a day-to-day basis works. Similarly, oftentimes programmers can be unfamiliar with how the software they are using is working, which can be very frustrating if the software isn't meeting their needs. Worst-of-all, when the software is proprietary or closed-source, there is no way of knowing what problems lurk beneath, or any hope of ever fixing these problems or any other issue with the software yourself. For this reason, programmers will often find themselves unsatisfied with the software that they're using, and no doubt entertain the idea of developing their own solution to the problem.

This blog post is primarily targeted to those readers who either have some level of ability to program in any capacity, and those who would consider themselves expert programmers and use their own software on a daily basis. Parts of this post may also be applicable to those who are technically minded enough to get the computer to do what they need it to do, from automating their taxes in a spreadsheet to having their photos automatically back-up to a cloud service.

A Plumber's Taps Should Never Leak

Anyone who has at some point attempted a DIY project themselves (to which I would wager is most people), has undoubtedly encountered a familiar theme of setbacks. Maybe you're missing the correct tool (or misusing a similar tool), you complete steps out-of-sequence, the project took you five hours instead of only one hour, or the project just didn't turn out the way you expected. These setbacks are normal and a part of developing any skill, yet they often seem like a much greater deal than they are in reality. This theme of setbacks is absolutely applicable to the context of DIY software. However, in this context, programmers should be wary enough to avoid these setbacks, as the process of writing software is not entirely foreign given that writing software is a skill they practice often.

This does not mean that programmers are entirely immune to setbacks when writing software for themselves. The specific areas which programmers should proceed with caution are the ways in which their personal and professional programming habits and practices differ, such as the use of an unfamiliar programming language, paradigms, or concept. Additionally, it is important that programmers avoid carelessness when writing their own software, ensuring they apply good-practice to their own software projects as they would while programming professionally. The aformentioned, all-too-familiar setbacks, are absolutely avoidable with an appropriate amount of care and attention.

Being able to write software in any capacity is an incredibly useful and valuable skill, one that businesses are happy to compensate programmers very well for. Furthermroe, often a simple piece of software is necessary to offer its user great benefits. Despite the cautions mentioned above, the utility of personally tailored software is therefore often worth the effort.

The remainder of this blog post will describe some examples of DIY projects, many of which any programmer would find useful in their day-to-day work. I will also outline some of the common traps to avoid specifically when writing your own software. I will outline when it makes sense to develop your own software solution to a problem, and outline some tips for doing so. Finally, I'll leave the reader with some places to look for inspiration for their own DIY projects.

But Wait, I'm No Linus Torvalds!

The best part about writing your own software are that there are no rules! There are so many ways in which you can add value and utility to your digital life, and they certainly don't have to consume your summer.

The absolute best place to start would be with taking control of your dotfiles, because at some point you're going to have to set your environment up on a new machine, and being able to quickly git clone your dotfiles from your own repository is a luxury you didn't know you needed. The first dotfile you should start with is for your shell. Spend an hour actually putting usefull stuff in there, such as environment variables, adding some aliases for most frequently used commands, or appending your path to include useful executables. After this, explore your home directory with ls -la ~/ to look for other programs that you frequently use which also have dotfiles waiting to be customised.

From there, the next level of DIY software is probably scripting1. Scripting allows you to very quickly automate common tasks, reliably and repeatably! What's more, this is where you start getting into the real utility of DIY software. If you can't think of tasks you can automate immediately, you will absolutely encounter them while you're working.

The final-form of DIY software is writing your own application. This may be something that includes separate or interconnected parts, involve overcoming an obscure technical challenge, or maybe even have the capacity to provide something of value beyond yourself to the wider community. There are numerous examples of these projects, and while they may now have numerous contributors, many had humble beginnings as DIY projects. To name some of my favourite examples:

If You Won't Do It, It'll Never Get Done

When does it make sense to write your own software? I believe that DIY software can be categorised by: necessity, frustration, and/or passion. If your idea for your own DIY software falls under at least one of these categories, then it's probably a good indication that it's worth taking the first steps towards developing. However, before beginning with that blank text-file you should ask yourself the following questions to help you decide if your project is worthwhile.

This could refer to software skills, available hardware, and/or most importantly, available spare-time.

Have a good look to see if the problem has been solved previously. This practice can actually aid you in developing your own software by exposing you to new ideas and methods.

If so, this could make developing particular parts of your project much quicker and easier2.

At this stage, it is a good idea to re-evaluate some of the initial assumptions that you made. For example, maybe you can achieve the end-goal in a more familiar programming language as opposed to a foreign one, or perhaps you can re-purpose an existing server instead of having to set up a new one. The optimisations you make here have the potential to make your DIY software project go from a waste of an afternoon to something more successful.

The Only Free Cheese is in the Mouse Trap

Programmers are particularly susceptible to this because the line between useful automation and immensely complicated time-wasting tinkering is a fine one indeed. This can be common in programming where you can say, build up a Rube Goldberg collection of shell scripts and Emacs functions and manual edits to text because you wanted to avoid writing a SQL function (because it would take 20 minutes of consulting the SQL documentation to get it right); but by the time you're consulting the Bash FAQ or resetting IFS variables to deal with a problem half an hour later, it's good to wake up and ask 'am I yak-shaving?' - and then you might realize that the data or problem has turned out to be sufficiently painful (eg lots of special characters or oddity in data formatting) that you can't catch all the special cases and you would've been better off writing the SQL query in the first place.

This is the boring section where I tell you – the self-confident programmer – that you are going to encounter hurdles when attempting to write software for yourself. This much should be obvious, as everyone is prone to error, and programming presents many opportunities for error. In the case of DIY software, there are errors which arise from the nature of DIY software often being novel and different to the norm of professional programming work. Despite this, careful attention to well-established, best-practices when it comes to software development can help to manage these errors. The best advice on this front is to avoid complacency. This is easier said than done, as DIY software can often be a creative or intellectual outlet for programmers – a past time which can be practised away from the office, and without the rigour that is necessary in such circumstances. Despite this, the occurrence of errors in DIY software can affect a programmers motivation to overcome them once encountered. This is especially true when DIY projects are in their infancy, and can often result in these projects never reaching the point of providing value for the user. It is therefore a good idea to pay close attention to what I believe are the most critical practices in DIY software.

Planning Fallacy

With good project management, time-management is a practice that is very easily managed in a professional setting. To manage this in your own DIY projects, you must be the project manager. Estimate how long you expect the project to take, outline some of the potential encounters which could blow this estimate out, and then either pad your estimate or adjust the scope of the project to your liking. This will help manage your expectations and your motivation, both of which are a large part of what is going to determine whether or not your project is successful.

Sunken Cost Fallacy

Being the sole developer on a project can make it hard to realise when something just isn't working the way you expected it to. Regarding sunken cost, it is useful to remember that DIY projects often have a very steep learning curve, particularly so if you are experimenting with novel or unfamiliar technologies. Therefore, more so than in a professional setting, the option of starting-over is holds reasonable promise if you find yourself considering it. Signs that this might be a good option are when you are experiencing similar troubles to non-DIY software – software that wasn't written for you in mind. If this is the case, then you've already forfeited many of the benefits of DIY software, and unless the path to restoring them is clear and reasonable, it may just be best to start-over with the knowledge of what has and hasn't worked so far.

Tests

This one is obvious, but is often the first-thing to go when it comes to DIY software, as it's often far more satisfying to spend an hour implementing a new feature than to write tests for existing ones, especially if you're experimenting with a new language and aren't proficient with the testing tools available3. However, in addition to all the traditional benefits of software testing, adding tests to your DIY software can actually speed-up development. Well-written tests will absolutely save you time tracking down bugs, implementing new features, and instill in you more confidence in using your DIY software. Most importantly, it goes a long way in making up for the fact that you're the sole developer of the software – without colleagues to hold you accountable for your code, it's just down to you and your tests4!

Tips

Start With A Specific, Achievable Aim

Try and distil your idea into its simplest form, and specify exactly what is required to achieve this. While this does aid in helping you stay on-task and working efficiently towards a goal, once the aim has been achieved it will provide you motivation and a solid foundation to build upon.

Don't Take Short Cuts

This one is obvious, but like writing tests, it's important to make your code as generalisable as possible, especially when it is in its infancy – you're likely to be iterating on your code fast, and general, well-written code will facilitate painless code iteration and minimise the time you spend writing, bug-fixing, and re-writing boiler-place code. Another tip here is to use logging to your advantage to minimise the gap between what you intend for your software to be doing, and what it's actually doing.

Use Source Code Management

Your SCM-practices can be a bit more relaxed compared to what you're probably used to professionally, but it is important that you use some form of SCM. If you do need to revert to a previous state, then you'll be glad you did. The use of either Github or Gitlab at the very least is absolutely a no-brainer given the ease of integrating features such as Continuous Integration, Static Web Hosting, and more. Plus, it puts your project out-there for others to find and contribute to.

Inspiration

RefMan and mkws

The idea for writing this article came while developing two DIY projects of my own:

RefMan solves a problem I have had for a while, which is my general dissatisfaction with reference managers (such as Mendeley or Papis). I find these tools to be far too complicated for my simple needs, which are: Find and store the PDF for me (by whatever means necessary), and maintain a BibTeX file of reference data. In the future, I may want to add my own metadata to each reference – such as notes or tags – but for now RefMan meets my reference management needs.

Without having any background in web-development, I have wasted a lot of time over the past few years trying to develop my own website. I have attempted to use jekyll, Hugo, and ssg5, all with increasing but not adequate levels of satisfaction. However, using ssg5 showed me the bare minimum of what was necessary to satisfy my own web-development needs to the point where I realised I could re-implement what it achieves in posix-compliant sh very quickly with Python. This allowed me to, finally, have a static site generator that I understand from end-to-end – as opposed to the other tools which I am far less familiar with – allowing me to have full control over how my website is produced, and to implement my own new features in the future to meet potential change in needs.

Github

Github is the place where DIY projects can become collaborative projects. If your project is valuable to you, then in all-likelihood it is probably valuable to someone else also. It also provides a place to see what other people are building for themselves. As much as you can, have a look at other users personal repositories - you might discover a new package, language, or an entire community you didn't know existed. And from all of this, you may be inspired to develop a DIY project of your own.

YouTube & Twitch

Video sharing platforms such as YouTube and Twitch are becoming an increasingly popular medium for people to share their own software projects. For example:

Conclusion

Practising writing your own software is a skill that absolutely makes you a better programmer. The traps outlined in this blog post are, on balance, an inconvenience at worst, and good exercise for your habits and tendencies as a programmer at best. Taking actions to avoid these traps has the benefit of helping maintain efficiently when writing your own software and eventually start using it. The satisfaction experienced when using and benefiting from something you've built yourself is something that continues to pay dividends.

  1. If you modified your .bashrc as from the previous paragraph, then you've already written a bash script!↩︎

  2. Extra points if you can re-purpose the associated tests!↩︎

  3. testing is seldom one of the more interesting aspects of a new language to learn↩︎

  4. And I know which I'd rather trust!↩︎