Top down or bottom up hierarchical builds?

About a year ago, after much umming and ahing about which build system to use for my ongoing project, I went with SCons. I liked the idea that my build system could be written in the same language as a fair chunk of the rest of the project (Python). It was a slightly non-trivial build scenario that lent itself to the flexibility that having a full and proper programming language as the building script.

Now, the outcome is neatly summarised by a recent observation from a friend: “SCons is dangerous”. The point isn’t that SCons is not an exceptionally powerful and flexible tool, but that SCons is an exceptionally powerful and flexible tool. Its far too easy to invent a weird and wonderful elaborate build structure that will have programming geniuses weeping in confusion. I discovered this through my own work.

I think the issue ultimately came down to how to structure a hierarchical build system. Through SCons, is simple to impose a top down structure. By this, I mean that the root directory contains logic about how all the subdirectories should be built. Its then a (logically) simple extension to have cross subdirectory dependencies and targets. This naively seems like a wonderful idea when you begin: adding new parts to the build is a simple case of adding a directory to the hierarchy which contains SConscript files that obey the relevant rules. We can have the root SConscript file look into the hierarchy and see how the subdirectories fit into the overall build system.

So, for example, the root SConscript peers into the subdirectories and looks for those that contain SConscript files. The SConscript files get run with edicts about what is allowed to be built, and a command to return information about the targets that have been build back to the root SConscript file. The root file then uses its programmed intelligence to decide what to do with each target that has been built.

This is actually why its a terrible idea. As the (slightly non-trivial) build develops, the logic gets more complicated, to the point where adding a new section to the build system requires an elaborate set of SConscript files and directories to deal with the expectation of the “intelligent” root SConscript file. On top of this, understanding what an earth is going on is no mean feat.

For all these reasons I’ve rewritten my build system around a much simpler bottom up approach. In this scenario, each subdirectory is a standalone directory containing all the relevant logic and information about how to build the targets and where to put them after the build. The root level just delegates responsibility for a subsection to the SConscript file in a subdirectory, passing as little knowledge as possible up the hierarchy.

The point I’m trying to get at is that building crazy build systems is a crazy idea, and SCons makes that ludicrously simple. I think the take home message is something like:

  1. If you’re using SCons, think hard about keeping the build simple stupid.
  2. Don’t have logic distributed about your hierarchy – keep it as close to the target as possible.
Advertisements

About Henry Gomersall

I'm a engineer wanting to make things actually useful. I'm someone that wants to drive technology and ideas to be helpful for everyone. I'm someone that realises the disconnect between technology and utility and I hope to correct that...
This entry was posted in Programming. Bookmark the permalink.

2 Responses to Top down or bottom up hierarchical builds?

  1. david rufino says:

    Hi Henry!

    My 2c is that makefiles should be purely declarative and have the minimum typing possible. Then the logic should live at the top-level. Often it’s a bit tempting to hack the makefiles rather than change the framework, and using a fully blown programming language makes this temptation worse I guess.

    When I was investigating build systems I quite liked Omake (http://omake.metaprl.org/index.html), as it’s quite like unix make, but extensible enough to allow ‘non-recursive make’ to work. And it’s really fast.

    Cheers,
    David

    • So, in my situation, the rationale was that I wanted to be able to move subdirectories around without breaking everything, perhaps putting them into a new tree entirely. What’s the benefit of non-recursive make? I still don’t quite get it.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s