CIRR.COM Compile Time Building Portable Build Systems

Last Month, we talked about writing portable code, in the form of easily configured and extremely readable programming style. We discussed using feature test macros instead of OS matches for feature definition; coding standards; emulating/re-implementing less than common functions; data encapsulation; and other topics.

This month, we'll talk about the mechanisms needed to easily, and successfully build the programs we so carefully wrote to be portable after reading last months column. We'll spend most of the column discussing make(1) in its many variants, and how/why to choose a particular variant. We'll also briefly talk about other build systems, and how they might be set up.

What's a build system?

The obvious answer is make. However, a build system really consists of much more than make. In fact, it may not even use make at all.

A build system is all the components needed to turn your program source into an executable. That means compilers, libraries, include files, and supporting utilities. make is just a small part of it, A driver program, if you will.

Your coding standard (remember that?) is a good place to document all the dependencies your project has on other software. If you're distributing source, another good place would be the README file that seems to be in the top directory of all source released software packages. The README should also explain how to build the software.

It's entirely possible that your package is simple enough that the following command could build it all:

cc -o program *.c

However, most interesting software is more complex than that.

In reality, the build system has two different sets of users, with two different sets of goals.

The first set of users is the software developers, who would like to avoid rebuilding everything if only one small file changed.

The second set of users are the customers building and installing the software on their systems for production use. They are unlikely to care that they can avoid rebuilding everything each time something changes, because nothing is going to change for them. They're going to unpack the software source, configure it, build it, and install it (once.)

For the customers, a long, simple, shell script might well suffice as the build tool. Such a script could also verify that other needed build tools are already installed, and perhaps even verify the configuration.

The first set of users, the software developers, have a more complex set of requirements on the software build tool. They'd like to avoid any unnecessary software rebuilds, but to be sure that all the modules are correctly rebuilt when something they depend upon changes.

As software developers, they may not realize all of the dependencies in use. This can lead to serious trouble down the road, as the unintentional dependencies rear their ugly heads, and cause a great deal of confusion and problems either on other developers systems, or on end user systems. The list of expected dependencies on other projects and base tools should be included the project documentation. The coding standard may be a good place for this. The project README or INSTALL document, which is included with the release, is probably a better place.

The most common compile management tool used on UNIX(tm) like systems is make(1). make will create dependency trees, and only rebuild those portions of the project that are required by the changes made. From here one, we'll focus on make.

A good Makefile should be as readable as a good C program. It should be well commented, and well organized. It should list all the real dependencies between source modules. One of the guidelines we discussed last month, one executable per directory, helps to make Makefile>s more readable, and easier to understand. Ideally, the Makefile layout is defined in the projects coding standard.

What's in a make?

Currently, there are three dominant makes in wide use. They are GNU make, from the Free Software Foundation, and widely available on Linux; Berkeley make, derived from 4.4 BSD, and available on FreeBSD, NetBSD, and OpenBSD; and System V/SUS make, available on Solaris, HP-UX, and other System V derived systems. There are also several minor makes available, but none are as widely used.

GNU make and Berkeley make are both proper supersets of the SUS make. Thus, a Makefile written for the SUS make will be interpreted correctly on both GNU make and Berkeley make, as well as on the System V variants. Sadly, the same cannot be said for either Berkeley make or GNU make. Both have a large number of extensions that are incompatible with System V make, and with each other.

While all of the above make's include a mechanism for including other files into the currently running Makefile, they all implement it differently and incompatibly. This removes an extremely useful mechanism for sharing configuration information across multiple Makefiles. However, it is something that can be worked around, using makes own facilities.

To insure maximum portability of Makefiles, the definition from the Single Unix Specification (SUS) should be used. The SUS has worked extremely hard to create a proper subset of all the currently existing make formats.

If a project must use the features of one of the more extended makes, be sure to call out the dependency on the make variant in the projects documentation (README or INSTALL are common places for this declaration.)

On the other hand, if the project is using only features as defined by the SUS, adding the target .POSIX would be a good idea. Declaring the target .POSIX tells various vendor implementations that your project is using only features defined by the SUS, and to turn off features that might conflict with the SUS.

Dependencies

Dependencies would seem like such an obvious thing. However, they can be very painful to generate, and when improperly defined, can cause great pain during the project development.

From Webster's Dictionary (http://www.webster.com), a definition of dependent:

Main Entry: de-pen-dent
Pronunciation: di-'pen-d&nt
Function: adjective
Etymology: Middle English dependant, from Middle French, present
participle of dependre
Date: 14th century

1 : hanging down
2 a : determined or conditioned by another : CONTINGENT
b (1) : relying on another for support
(2) : affected with a drug dependence
c : subject to another's jurisdiction
d : SUBORDINATE 3a
3 a : not mathematically or statistically independent <a dependent set of vectors> <dependent events>
b : EQUIVALENT 6a <dependent equations>

For the purposes of software development, definition's 2a, 2b(1), 2c, and 3a all describe a dependency. Fundamentally, fileA is dependent upon fileB if changing fileB requires fileA to be rebuilt.

For the use of the software developer, getting the source/object/executable dependencies correct can be the hardest part of writing a Makefile. For large projects, it can be extremely hard to do manually, at least if they are added late in the game. If the developers keep dependency generation in mind, and add the dependencies between source files as they become obvious, then the task is much easier.

Fortunately, tools exist to help generate file dependencies. mkdep>, gcc and other tools exist for preprocessing sources, and emitting a list of files included by each source module. To use gcc to create dependencies, the -MM flag is used to generate a list of user included files in the source module. Other compilers support an -M flag that will list all files included, both system header files and user files. The list of included files is emitted to standard output, which should be redirected to append to the Makefile.

To use gcc to generate a list of dependencies for the source file foo.c, the following command can be used:

gcc -MM foo.c

The above command will emit the dependency list to standard output. It should be redirected to a file, perhaps the Makefile itself. Here's an example of doing that:

gcc -MM foo.c >>Makefile

To generate dependencies using the commercial compilers that support the -M flag, the following command should be used:

cc -M foo.c >>Makefile

mkdep does a similar task, creating the file .depend to store the dependency information. The resulting file (.depend) should be included in the Makefile.

The dependence generated by gcc or mkdep should be culled to remove any listings of system include files (they aren't likely to change during the course of development), and then included in the distributed Makefile or Makefile template.

A good Makefile should also include three standard targets. The first is all, as the first target in the Makefile. all should be dependent on the executables (and anything else to be installed) in the directory. Having all as the first target in the Makefile means it's the task chosen when no target is defined on the make command line.

The second standard target is install. The install target should be dependent on anything in the directory that needs to be installed for the resulting software to execute. It should copy the dependents into their final resting place on the system, optionally allowing a relocation of the installation root using the make variable DESTDIR.

The final standard target is clean. The clean target should remove all the detritus that's left behind as part of building the software. This would be any intermediate files, such as objects, and the final executable. The clean target should leave the directory similar to its initial checkout/un-archived state.

Your local coding standard may mandate other standard targets as well. Be sure to document them, and implement them.

Getting the dependencies correct greatly eases the development process, as all affected files are regenerated when a properly marked dependency is modified.

Build Anywhere?

We've talked about a couple of different ways to build your project, and looked at who the customers are. The end user could really care less how their software gets built, so most anything would work for them. On the other hand, you, the software developer, would like to avoid having the entire source tree rebuilt every time something changes.

To avoid rebuilding the world, make was introduced, and we discussed how to be sure your Makefile>s are readable, and highly portable. To recap:

• Avoid using vendor specific extensions as much as possible. Use the SUS make definition to aid in writing your Makefiles.

• Make your Makefiles as readable as your code.

• Document your system dependencies (compilers, libraries, etc.)

• Maintain the file dependencies in the Makefile>s. Use tools like gcc -MD> or mkdep to generate them if needed.

References

Single Unix Specification make(1) manual page: http://www.opengroup.org/onlinepubs/007904975/utilities/make.html

Single Unix Specification m4(1) manual page: http://www.opengroup.org/onlinepubs/007904975/utilities/m4.html

GNU Autoconfig, Automake, and Libtool
2001 New Riders; ISBN 1-57870-190-2
Gary V. Vaughan, Ben Elliston, Tom Tromey, and Ian Lance Taylor

Hey Good looking -- an explaintion of a readable Makefile structure http://www.cirr.com/compile-time/2002-10.html

Making make include a file

The Single UNIX Specifications edition of make doesn't document an include feature, leaving that as a vendor extension. It isn't included because historic implementations of make have evolved using different include mechanisms. And a there are still a few makes out there that don't support includes at all.

There are ways around this. GNU's Automake supports a mechanism for including files into it's Makefile.in output. We'll discuss Automake and Autoconf next month.

Another way is using the text preprocessors available on UNIX systems. The most commonly thought of preprocessor is cpp, the C preprocessor. The X Window system uses cpp at the heart of imake to generate Makefiles for projects using X. Unfortunately, cpp being a preprocessor designed for C has some problems. None the less imake, with suitable include files, can be an excellent mechanism for getting files included in Makefiles.

Another preprocessor, m4, is actually more powerful in doing text substitution and macro expansion. The most common visible use of m4 these days is for building sendmail configuration files. Automake and Autoconf also use m4 at their core. Here's how to use m4 directly to generate Makefile>s.

Assume the following directory layout: Makefile
common/rules.m4
common/sources.m4
progA/Makefile.m4
progB/Makefile.m4


Makefile looks like this:

#	@(#) ./Makefile -- master makefile for ProjectZ
# 
# default target and dependencies
COMMON_MK=	common/rules.m4 \
		common/sources.m4


all:	progA/Makefile progB/Makefile build


build: progA/a.out progB/a.out


progA/a.out: progA/Makefile
	cd progA; $(MAKE)


progB/a.out: progB/Makefile
	cd progB; $(MAKE)


progA/Makefile: progA/Makefile.m4 $(COMMON_MK)
	cd progA; m4 Makefile.m4 > Makefile


progB/Makefile: progB/Makefile.m4 $(COMMON_MK)
	cd progB; m4 Makefile.m4 > Makefile

progA/Makefile.m4 looks like this:

changecom()dnl	
#	@(#) progA/Makefile -- build progA for ProjectZ
#
include (`../common/rules.m4')dnl


PROGA_OBJS=srcA.o srcB.o srcC.o
PROGA_SRCS=$(PROGA_OBJS:.o=.c)


progA: $(PROGA_OBJS)
	cc -o a.out $(PROGA_OBJS)


include (`../common/sources.m4')dnl

The m4 directives in progA/Makefile.m4 are include(), changecom(), and dnl>.

include() does pretty much what you expect, suspending input from the current file, and reading from the new file until end of file is reached. Thus, our final Makefile contains the comments of the files ../common/rules.m4>, itself, and ../common/sources.m4>.

changecom() changes the comment delimiters used by m4>. By default, m4 uses the same comment delimiters as nearly every other interpreter on UNIX, the octothorpe (#) (commentary from the sister-in-law: Screw the phone company, this is a pound!) and new-line. Since we'd like to leave the comments in the resulting Makefile's, using changecom() allows us to set m4's idea of the comment character to something else, in this case, turning it off entirely.

The last m4 directive we use is dnl, which stands for "delete to new-line". It suppresses the blank lines emitted where the m4 directives are processed. It's not strictly necessary, but leaves the output looking a little prettier.

progB/Makefile.m4 looks similar to progA/Makefile.m4.

Using m4>, you've successfully emulated the include feature of the different make's without depending upon a particular one.