Devs on Acid

Modern autotools

9 Mar 2021 14:28 UTC

GNU autotools aka the GNU Build System is a build system designed to produce a portable source code package that can be compiled about everywhere.

The intentions are good: when properly used, a configure script is generated that runs everywhere a POSIX compatible shell is available, and a Makefile that can be used everywhere a make program is available. No further dependencies are required for the user, and the process to build source with ./configure, make and make install is well-established and understood.

From the developer's perspective though, things look a bit different. In order to create the mentioned configure script and Makefile, autotools uses the following 3 main components:

To use them, the developer needs perl and gnu m4 installed in addition to the tools themselves, as well as a basic understanding of m4, shell scripting, Makefiles, and the complex interaction between autoconf and automake.

He also needs a lot of time and patience, because each change to the input files requires execution of the slow autoreconf to rebuild the generated sources, and running ./configure and make for testing.

Libtool is a shell script wrapper around the compiler and the linker with >9000 lines, which makes every compiler invocation about a 100 times slower. It is notorious for breaking static linking of libraries and cross-compilation due to replacing e.g. "-lz" with "/usr/lib/libz.so" in the linker command. Apart from being buggy and full of wrong assumptions, it's basically unmaintained (last release was 6 years ago).

While there's reasonably complete documentation for autoconf and automake available, it is seriously lacking in code examples, and so many supposedly simple tasks become a continuous game of trial and error.

Due to all of the above and more, many developers are overwhelmed and frustrated and rightfully call autotools "autocrap" and switch to other solutions like CMake or meson.

But those replacements are even worse: they trade the complexity of autotools on the developer side against heavy dependencies on the user side.

meson requires a bleeding edge python install, and CMake is a huge C++ clusterfuck consisting of millions of LOC, which takes up >400 MB disk space when built with debug info. Additionally meson and cmake invented their own build procedure which is fundamentally different from the well-known configure/make/make install trinity, so the user has to learn how to deal with yet another build system.

Therefore, in my opinion, the best option is not to switch to another build system, but simply to only use the good parts of autotools.

Getting rid of libtool

It's kinda hard to figure out what libtool is actually good for, apart from breaking one's build and making everything 100x slower. The only legitimate usecase I can see is executing dynamically linked programs during the build, without having to fiddle around with LD_LIBRARY_PATH. That's probably useful to run testcases when doing a native build (as opposed to a cross-compile), but that can also be achieved by simply statically linking to the list of objects that need to be defined in the Makefile anyhow. Libtool being invoked for every source file is the main reason for GNU make's reputation of being slow. If GNU make is properly used, one would need to compile thousands of files for a noticeable difference in speed to the oh-so-fast ninja.

Getting rid of automake

Automake is a major pain in the ass.

Makefiles are generated by the configure script by doing a set of variable replacements on Makefile.in, which in turn is generated by automake from Makefile.am on the developer's end.

The only real advantage that automake offers over a handwritten Makefile is that conditionals can be used that work on any POSIX compatible make implementation, since those are still not standardized to this day, and that dependency information on headers is generated automatically. The latter can be implemented manually using the -M options to gcc, like -MMD.

The Good Part(s)

The only good part of autotools is the generated portable configure script, and the standard way of using the previously mentioned trinity to build. The configure script is valuable for many reasons:

Additionally to the above, autoconf-generated configure scripts have some useful features built in:

On the other hand generated configure scripts tend to be quite big, and, as they are executed serially on a single CPU core, rather slow.

Fortunately this can be fixed by removing the vast majority of checks, just assume C99 support as a given and move on. While you're at it, throw out that check for a 20 year old HP-UX bug, please.

The Solution

A modern project using autotools should only use autoconf to generate the configure script, and a single top-level Makefile that's handwritten. Build configuration can be passed to the Makefile using a single file that's included from the Makefile. The number of configure checks should be reduced to the bare minimum, there's little point in testing e.g. for the existence of stdio.h which is standardized since at least C89, especially if then later the preprocessor macro HAVE_STDIO_H isn't even used and stdio.h is included unconditionally. Used this way, the configure script will be quick in execution and small enough to be included in the VCS which allows the users to checkout and build any commit without having to do the autotools dance. A good guide for writing concise configure scripts is available here.

As for conditionals in make, I'm pretty much in favor of simply assuming GNU make as a given and using its way to do conditionals.

It's in widespread use (default make implementation on any Linux distro) and therefore available as gmake even on BSD installations that usually prefer their own make implementation. Apart from that it's lightweight (my statically linked make 3.82 binary has a mere 176 KB) and one of the most portable programs around.

The alternative is to target POSIX make and do the conditionals using automake-style text substitutions in the configuration file produced by the configure run.

The Ingredients

Our example project uses the following files: configure.ac, Makefile, config.mak.in, main.c, foo1.c and foo42.c with the following contents respectively.

configure.ac:

AC_INIT([my project], 1.0.0, maintainer@foomail.com, myproject)
AC_CONFIG_FILES([config.mak])
AC_PROG_CC
AC_LANG(C)

AC_ARG_WITH(foo,
    AS_HELP_STRING([--with-foo=1,42], [return 1 or 42 [1]]),
    [foo=$withval],
    [foo=1])

AC_SUBST(FOO_SOURCE, $foo)
AC_OUTPUT()

Makefile:

include config.mak

OBJS=main.o $(FOO).o
EXE=foo-app

all: $(EXE)

$(OBJS): config.mak

$(EXE): $(OBJS)
    $(CC) -o $@ $(OBJS) $(LDFLAGS)
clean:
    rm -f $(EXE) $(OBJS)

install:
    install -Dm 755 $(EXE) $(DESTDIR)$(BINDIR)/$(EXE)

.PHONY: all clean install

config.mak.in:

# whether to build foo1 or foo42
FOO=foo@FOO_SOURCE@

PREFIX=@prefix@
BINDIR=@bindir@

CFLAGS=@CFLAGS@
CPPFLAGS=@CPPFLAGS@
LDFLAGS=@LDFLAGS@

main.c:

#include <stdio.h>
extern int foo();
int main() {
    printf("%d\n", foo());
}

foo1.c:

int foo() { return 1; }

foo42.c:

int foo() { return 42; }

You can get these files here.

After having the files in place, run autoreconf -i to generate the configure script. You'll notice that it runs unusually quickly, about 1 second, as opposed to projects using automake where one often has to wait for a full minute.

The configure script provides the usual options like --prefix, --bindir, processes the CC, CFLAGS, etc variables exported in your shell or passed like

CFLAGS="-g3 -O0" ./configure

just as you'd expect it to, and provides the option --with-foo=[42,1] to let the user select whether he wants the foo42 or foo1 option.

How it works

configure.ac:

AC_CONFIG_FILES([config.mak])

Here we instruct autoconf that config.mak is to be generated from config.mak.in when it hits AC_OUTPUT() (which causes config.status to be executed). It will replace all values that we either specified with AC_SUBST(), or the built-in defaults like prefix and bindir (see config.status for the full range) with those specified by the user.

AC_ARG_WITH(...)

This implements our --with-foo multiple choice option. You can read about how it works in the usual autoconf documentation. Other autoconf macros that you will find handy include AC_CHECK_FUNCS, AC_CHECK_HEADERS, AC_CHECK_LIB, AC_COMPILE_IFELSE to implement the various checks that autoconf offers, as well as AC_ARG_ENABLE to implement the typical --enable/--disable switches.

AC_SUBST(FOO_SOURCE, $foo)

This replaces the string @FOO_SOURCE@ in config.mak.in with the value assigned by the user via the AC_ARG_WITH() statement, when config.mak is written.

The rest of the contents in configure.ac are the standard boilerplate for C programs.

Makefile:

include config.mak

This statement in Makefile includes the config.mak generated by configure. If it is missing, running make will fail as it should. config.mak will provide us with all the values in config.mak.in, where each occurence of @var@ is replaced with the results of the configure process.

OBJS=main.o $(FOO).o

This sets OBJS to either main.o foo1.o or main.o foo42.o, depending on the choice of the user via the --with-foo switch. We didn't even have to use conditionals for it.

$(OBJS): config.mak

We let $(OBJS) depend on config.mak, so they're scheduled for rebuild when the configuration was changed with another ./configure execution, as the user might have changed his CFLAGS or --with-foo setting to something else. For bonus points, you could put all build-relevant settings into e.g. config-build.mak, and directory-related stuff stuff into config-install.mak (unless you hardcode directory names into the binary) and make the dependency to config-build.mak only.

    install -Dm 755 $(EXE) $(DESTDIR)$(BINDIR)/$(EXE)

Two important things about this line:

The rest of the Makefile contents are pretty standard. You might notice the absence of a specific rule to build .o files from .c, we use the implicit rule of make for this purpose.

config.mak.in:

FOO=foo@FOO_SOURCE@

config.status will replace the string @FOO_SOURCE@ with either 1 or 42, depending on which --with-foo option was used (1 being the default), shortly before configure terminates and writes config.mak. The values for @CFLAGS@ and the other variables will be replaced with the settings the configure scripts defaults to or those set by the user.

foo1.c, foo42.c and main.c:

... should be self-explanatory.

Testing

You can run ./configure && make now and see that it works - foo-app is created, and make DESTDIR=/tmp/foobar install installs foo-app into /tmp/foobar/bin/foo-app. ./configure --with-foo=42 && make should cause the foo-app binary to print 42 instead of 1.

Further reading

If you want to learn more about the build process and especially how it works in regard to cross-compilation, you can check out my article Mastering and designing C/C++ build systems.