Free software developer. Huawei, ex-Canonical, ex-Linaro, ex-Samsung. I write in C, Go and Python. I usually do system software. Zyga aka Zygoon
6741 words
@zygoon

Unexpected control flow

Today I will talk about unexpected control flow in shell


I was doing code review of a piece of shell code. It looked good on paper, if some poor soul is printing shell on paper, and was even shellcheck-clean, but we decided to add set -e at the top of the script, to see if anything was failing silently.

It was, obviously.

After a moment of shock and disbelief, it struck us what was broken. The broken part was inside out heads. Shell itself did exactly what we told it to do. Our understanding of the semantics of all the shell features, combined, was the faulty element.

The code we looked at can be reduced to this snippet:

set -e
a=0
((a++))
# surprise, exit 1 here

If you don't immediately see the problem and are unwilling to look at the bash(1) manual page for clues, let me just jump to the conclusion.

((++a))

Should shellcheck warn about this issue? It might. I doubt any developer intentionally wrote code with the desire to always exit after the assignment, just because set -e and ((a++)) together require that.

Fixing VMware Workstation 16.1 after kernel upgrade on Ubuntu 20.04

Today I will talk about making VMware Workstation survive a kernel update


This is a continuation of https://listed.zygoon.pl/21339/installing-vmware-workstation-16-pro-on-ubuntu-20-04

When you get a new kernel you need to re-compile out-of-tree modules in order
for them to continue working, if possible. VMware currently uses two kernel
modules: vmmon and vmnet. Surprisingly after all those years VMware did not
adopt DKMS so you you, the customer, are responsible for doing the hard work
that ought to be just integrated and never seen.

Enough complaints. Let's make it work for us right now, when we updated and are
grumpy that it broke:

sudo vmware-modconfig --console --install-all
sudo /usr/src/linux-headers-$(uname -r)/scripts/sign-file sha256 /var/lib/shim-signed/mok/MOK.priv /var/lib/shim-signed/mok/MOK.der $(modinfo -n vmmon)
sudo /usr/src/linux-headers-$(uname -r)/scripts/sign-file sha256 /var/lib/shim-signed/mok/MOK.priv /var/lib/shim-signed/mok/MOK.der $(modinfo -n vmnet)
sudo systemctl restart vmware

If anyone at VMware ever reads this, I would love to know the backstory and
understand the technical problems with adopting DKMS.

Integrating PVS Studio and Coverity with make

Today I will talk about integration of PVS Studio and Coverity into a make-based build system.


As a part of my work on Open Harmony, I'm looking into static analysis for C and
C++ projects.

I had prior experience with PVS Studio and Coverity. I've used them in my
personal projects as well as in my work on snap-confine, a privileged part of
snapd responsible for creation of the execution environment for snap application
processes, where security is extremely important.

Let us briefly look at how static analysis tools integrate with the build
system. In general static analysis tools do not run the code but instead
read the code and deduce useful properties this way. Since C includes a
pre-processor, which handles #include and various #if statements, the ideal
input to a static analyzer is pre-processed code. This way the static analysis
tool can be fed standalone information that no longer relies on system headers
or third party libraries that are necessary to understand definitions in the
code. This also makes such pre-processed input convenient for SAAS-like service,
where the analysis tool is not running locally with access to the local
compiler.

In general all the static analysis tools I've tried behave this way. The main
difference from an integrator point of view if the pre-process step is done
implicitly or explicitly. The cheapest way to integrate with a tool if this can
be done implicitly, namely by following and observing an existing build process.
The analyzer support tool can trigger an otherwise standard build process,
observe how the compiler is executed, extract the relevant -I, -iquote and -D
flags, find the translation units and eventually pre-process the code internally.

Some tools offer the "build system observer", others do not, or in special case,
gcc doesn't need to as the analyzer is an integral part of the compiler itself.

Let's consider two examples, explicit pre-processing with PVS Studio and
implicit pre-processing with Coverity. Examples below use make(1) syntax.

First we need a way to pre-process arbitrary file. Interestingly this works for
both C and C++ as the pre-processor does not care about the actual source
language. Well, maybe except __cplusplus defines.

%.i: %:
    $(CPP) $(CPPFLAGS) $< -E -o $@

For those who don't read make, this will invoke the pre-processor, which by
convention is named by the $(CPP) variable, pass all the options defined by
another variable, $(CPPFLAGS), then the input file $<, then the -E option,
which asks the compiler to stop at the pre-processing, followed by -o $@ to
write the result to the output file, which is named on the left hand side of the
rule header %.i. The rule header shows how to make a file with the .i
extension out of any file % behaves like * in globs.

Now we can ask the static analysis tool to do its job. Let's write another rule:

%.PVS-Studio.log: %.i
    pvs-studio --cfg .pvs-studio.cfg --i-file $< --source-file $* --output-file $@

Some bits are omitted for clarity. The real rule has additional dependencies on
the PVS-Studio license file and some directories. Interestingly we need to both
provide the pre-processed file $< and the original source file $*. Here $<
is the first dependency and $* is the text that was matched against
%`.

The resulting *.PVS-Studio.log files are opaque. They represent arbitrary
knowledge extracted by the tool from an individual translation unit. This mode
of operation is beneficial for parallelism, as those tasks can be executed
concurrently. As with compilation, in the end we need to link the results
together to get the result of our analysis.

pvs-report: $(wildcard *.PVS-Studio.log)
    plog-converter --setings .pvs-studio.cfg \
        $(PLOG_CONVERTER_FLAGS) \
        --srcRoot . \
        --projectName foo \
        --projectVersion 1.0 \
        --renderTypes fullhtml \
        --output $@ \
        $^

Here we use another tool, plog-converter to merge the result of the analysis.
There are some additional options that influence the format and contents of the
generated report but those are self-explanatory. One interesting observation is
that this command does not fail in the make check style, by existing with an
error code. If you intend to block on static analysis results there are some
additional steps you need to take. For PVS Studio I've created the appropriate
rules inside zmk, so that the logs can be processed and displayed in the same
style that a compiler would otherwise produce, so that the output is useful to
editors like vim.

That's it for PVS Studio, now let's look at Coverity. Coverity offers a sizable
(715MB) archive with all kinds of tooling. From the point of view we need just
one tool, the cov-build wrapper. The wrapper invokes arbitrary build command,
in our case make and stores the analysis inside a directory. Coverity requires
that directory to be called cov-int so we will follow along for simplicity.

Here is our make rule:

cov-int: $(MAKEFILE_LIST) $(wildcard *.c *.h) # everything!
    cov-build --dir $@ $(MAKE) -B

The rule is simple and could be improved. Ideally, to avoid the -B (aka
--always-make argument) we would perform an out-of-tree build in a temporary
directory. What we are after is a condition where make invokes the compiler,
even for the files we may have built in our tree before, so that cov-build
gets to observe the relevant arguments, as was described before. The more
problematic part is the build-dependency, which technically depends on
everything that the input may need. Here I simplified the real dependency set.
For practical CI systems that's sufficient, for purists it requires some care to
properly describe the recursive dependencies if the implicit target (commonly
called all).

The result is a directory we need to tar and send to Coverity for analysis. That
part is not interesting and details are available inside the zmk library.

Coverity processing is both asynchronous and capped behind a quota. I would not
recommend using it to block builds in CI, except if you have a commercial local
instance that never rejects your uploads. Coverity has a REST API for accessing
analysis results so with some extra integration that API can be queried and
appropriate blocking rules can be used to "break the build". Personally I did
not attempt this, mainly due to quota.

As a last note, Coverity puts additional requirements on valid submissions. At least 80%
of compilation units must be "ready for analysis". This can be checked with ad-hoc rule
that uses some shell to look at the analysis log file. The following shell snippet is not
quoted for correct use inside Make, see zmk for the quoted original:

test "$(tail cov-int/build-log.txt -n 3 | \
        awk -e '/[[:digit:]] C\/C\+\+ compilation units \([[:digit:]]+%) are ready for analysis/ { gsub(/[()%]/, "", $6); print $6; }')" \
     -gt 80

Next time we will look at the perceived value of the various static analysis
tools I've tried.

Footnote: ZMK can be found at https://github.com/zyga/zmk/

Installing VMware® Workstation 16 Pro on Ubuntu 20.04

Today I will talk about installing VMware® Workstation 16 Pro on Ubuntu 20.04 x86_64


For the most part, installation of this program has been streamlined. Compared to earlier versions, you really don't have do do anything more than:

chmod +x VMware-Workstation-Full-16.1.0-17198959.x86_64.bundle
sudo ./VMware-Workstation-Full-16.1.0-17198959.x86_64.bundle

This will give you working application but won't let you run any virtual machines yet.

For a while, kernel lockdown is in effect, where the kernel will not load unsigned kernel modules, so that they cannot be used to circumvent secure boot. Details are not interesting for now. Let's focus on the installation.

The following instructions assume you have enabled support for proprietary drives during your installation process. If you don't remember setting "secure boot password" and seeing weirdly looking and weirdly named MOK prompt during first boot following the installation of Ubuntu, you probably did not do this and the following instructions will not work.

If you did you have all the bits necessary now:

sudo /usr/src/linux-headers-$(uname -r)/scripts/sign-file sha256 /var/lib/shim-signed/mok/MOK.priv /var/lib/shim-signed/mok/MOK.der $(modinfo -n vmmon)
sudo /usr/src/linux-headers-$(uname -r)/scripts/sign-file sha256 /var/lib/shim-signed/mok/MOK.priv /var/lib/shim-signed/mok/MOK.der $(modinfo -n vmmet)
sudo modprobe vmmon
sudo modprobe vmnet

What is going on here is that the kernel lockdown prevents usage of unsigned modules. You can find this message in your journal / syslog and follow the breadcrumbs to the relevant manual page.

kernel: Lockdown: modprobe: unsigned module loading is restricted; see man kernel_lockdown.7

You can read the manual page here: https://man7.org/linux/man-pages/man7/kernel_lockdown.7.html

Remember that this has to be done every time you change your kernel.

zmk 0.4.2 released

I've released zmk 0.4.2 with several small bug-fixes.


ZMK is a Make library for writing makefiles that behave like autotools without having the related baggage. It works out-of-the-box on POSIX systems, including MacOS.

You can find ZMK releases at https://github.com/zyga/zmk/releases

The changelog for zmk 0.4.2 is:

  • The PVS module no longer fails when running the pvs-report target.

  • The Header module no longer clobbers custom InstallDir.

  • The Library.DyLib template no longer creates symlink foo -> foo.dylib when
    the library is not versioned. In addition the amount of code shared between
    Library.So and Library.DyLib has increased.

Raspberry Pi 4B and 4K display at 60Hz

Today I will talk about using Raspberry Pi 4B as a Ubuntu 20.10 desktop, on a 4K TV.


If you are interested in using Ubuntu 20.10 on a Raspberry Pi 4B with 8GB of RAM and want to use a TV as a display I have a bit of advice that can help you out and save your time.

1) Do not upgrade from 20.04 - after upgrading the essential boot section changes won't happen, and you won't have hardware acceleration. Unless you know what to change, just install 20.10 from scratch.

2) You need to edit /boot/firmware/config.txt and add hdmi_enable_4kp60=1, preferably to the [pi4] section. This setting is applied on boot. Your Raspberry Pi should have proper cooling.

3) You need to use the micro-HDMI port next to the USB-C power connector.

4) Your TV needs to be set to "PC" mode. Details differ but at least with my TV I was only getting 30Hz in any other mode. It may be related to the color format, I don't know.

5) If you boot and see the login screen just fine but get a blank / no signal screen after logging in then look at your ~/.config/monitors.xml file. I had the 30Hz refresh rate selected there and (again) it was not accepted by my TV in PC mode.

Good luck!

Introduction to bashunit - unit testing for bash scripts

Today I will talk about bashunit - a unit testing library for bash scripts.


All the posts about bash were building up to this. I wanted to be able to test bash scripts, but having found nothing that makes that practical, I decided to roll my own.

Having created bashcov earlier, I needed to connect the dots between discovering tests, running them, reporting errors in a nice way and measuring coverage at the same time. I also wanted to avoid having to source bashunit from test scripts, to avoid complexity related to having two moving parts, the tests and the program running them. I settled on the following design.

bashunit is a standalone bash script. It can enumerate tests by looking for files ending with _test.sh. In isolation, it sources each one and discovers functions starting with test_. In further isolation, it calls each test function, having established tracing in order to compute coverage. All output from the test function is redirected. On success only test function names and test file names are printed. On failure the subset of the trace related to the test function is displayed, as is any output that was collected.

The ultimate success of failure is returned with the exit code, making bashunit suitable for embedding into a larger build process.

In addition, since coverage is enlightening, similar to bashcov the collected coverage data is used to create an annotated script with the extension .coverage, corresponding to each sourced program that was executed through testing. This data is entirely human readable and is meant to pinpoint gaps in testing or unexpected code flow due to the complexity in bash itself.

Let's look at a simple example. We will be working with a pair of files, one called example.sh, housing our production code, and another one called example_test.sh, with our unit tests.

Let's look at example.sh first

#!/bin/bash
hello_world() {
    echo "Hello, World"
}

if [ "${0##*/}" = example.sh ]; then
    hello_world
fi

When executed, the script prints Hello World. When sourced id defines the hello_world function and does nothing else. Simple enough. Let's look at our unit tests.

#!/bin/bash

. example.sh

test_hello_world() {
    hello_world | grep -qFx 'Hello World'
}

In the UNIX tradition, we use grep to match the output of the hello_world function. The arguments to grep are -q for --quiet, to avoid printing the matching output, -F for --fixed-strings to avoid using regular expressions and finally -x for --line-regexp to consider matches only matching entire lines (avoids matching a substring by accident).

Running bashunit in the same directory yields this the following output:

bashunit: sourcing example_test.sh
bashunit: calling test_hello_world
bashunit: calling test_hello_world resulted in exit code 1
bashunit: trace of execution of test_hello_world
bashunit:    ++ . example.sh
bashunit:    + hello_world
bashunit:    + grep -qFx 'Hello World'

What is that? Our tests have failed. Well, I made them to, if you look carefully the example code has a comma between Hello and World, while test the code does not.

Correcting that discrepancy produces the following output:

bashunit: sourcing example_test.sh
bashunit: calling test_hello_world

The exit status is zero, indicating that our tests have passed. In addition to this, we have coverage analysis in the file example.sh.coverage, it looks like this:

  -: #!/bin/bash
  -: hello_world() {
  1:    echo "Hello, World"
  -: }
  -: 
  1: if [ "${0##*/}" = example.sh ]; then
  -:     hello_world
  -: fi
  -: 

The two 1s indicate that the corresponding line was executed one time. If you add loops or write multiple tests for a single function you will see that number increment accordingly.

Interestingly, not only test functions are executed, the guard at the bottom, where we either execute hello_world or do nothing more is also executed. This is done when the example.sh script is sourced by bashunit.

Much more is possible, but this is the initial version of bashunit. It requires a bit more modern bash than what is available on MacOS, so if you want to use it, a good Linux distribution is your best bet.

You can find bashunit, bashcov and the example at https://github.com/zyga/bashcov

Broken composition or the tale of bash and set -e

Today I will talk about a surprising behavior in bash, that may cause issues by hiding bugs.


Bash has rather simple error checking support. There are, in general, two ways one can approach error checking. The first one is entirely unrealistic, the second one has rather poor user experience.

The first way to handle errors is to wrap, every single thing in and if-then-else statement, provide a tailored error message, perform cleanup and quit. Nobody is doing that. Shell scripts, it's sloppy more often than not.

The second way to handle errors is to let bash do it for you, by setting te errexit option, by using set -e. To illustrate this, look at the following shell program:

set -e

echo "I'm doing stuff"
false
echo "I'm done doing stuff"

As one may expect,false will be the last executed command.

Sadly, things are not always what one would expect. For good reasons set -e is ignored in certain contexts. Consider the following program:

set -e
if ! false; then
   echo "not false is true"
fi

Again, as one would expect, the program executes in its entirety. Execution does not stop immediately at false, as that would prevent anyone from using set -e in any non-trivial script.

This behavior is documented by the bash manual page:

Exit immediately if a pipeline (which may consist of a single simple command), a list, or a compound command (see SHELL GRAMMAR above), exits with a non-zero status. The shell does not exit if the command that fails is part of the command list immediately following a while or until keyword, part of the test following the if or elif reserved words, part of any command executed in a && or || list except the command following the final && or ||, any command in a pipeline but the last, or if the command's return value is being inverted with !. If a compound command other than a subshell returns a non-zero status because a command failed while -e was being ignored, the shell does not exit. A trap on ERR, if set, is executed before the shell exits. This option applies to the shell environment and each subshell environment separately (see COMMAND EXECUTION ENVIRONMENT above), and may cause subshells to exit before executing all the commands in the subshell.

The set of situations where set -e is ignored is much larger. It contains things like pipelines, commands involving && and ||, except for the final element. It is also ignored when the exit status is inverted with !, making something as innocent as ! true silently ignore the exit status.

What may arguably need more emphasizing, is this:

If a compound command or shell function executes in a context where -e is being ignored, none of the commands executed within the compound command or function body will be affected by the -e setting, even if -e is set and a command returns a failure status. If a compound command or shell function sets -e while executing in a context where -e is ignored, that setting will not have any effect until the compound command or the command containing the function call completes.

What this really says is that set -e is disabled for the enitre duration of a compound command. This is huge, as it means that code which looks perfectly fine and works perfectly fine in isolation, is automatically broken by other code, which also looks and behaves perfectly fine in isolation.

Consider this function:

foo() {
    set -e # for extra sanity
    echo "doing stuff"
    false
    echo "finished doing stuff"
}

This function looks correct and is correct in isolation. Lets carry on and look at another function:

bar() {
    set -e # for extra sanity
    if ! foo; then
        echo "foo has failed, alert!"
        return 1
    fi
}

This also looks correct. In fact, it is correct as well, as long as foo is an external program. If foo is a function, like what we defined above. The outcome is that foo executes all the way to the end, ignoring set -e's normal effect after the non-zero exit code from false. Unlike when invoked in isolation, foo prints the finished doing stuff message. What is worse, because echo succeeds, foo doesn't fail!

Bash breaks composition of correct code. Two correct functions stop being correct, if the correctness was based on the assumption, that set -e stops execution of a list of commands, after the first failing element of that list.

It's a documented feature, so it's hardly something one can report as a bug on bash. One can argue that shellcheck should warn about that. I will file a bug on shellcheck, but discovering this has strongly weakened my trust in using bash for anything other than an isolated, executed script. Using functions or sourcing other scripts is a huge risk, as the actual semantics is not what one would expect.

Poor man's introspection in bash

Today I wanted to talk about bashunit, a unit testing library for bash scripts. This turned out to be a bigger topic, so you will have to wait a bit longer for the complete story. Instead I will talk about how to do poor man's introspection in bash, so that writing tests is less cumbersome.


In some sense, if you have to know this sort of obscure bash feature, it may be a good indication to stop, take a step back and run away. Still if you are reading this, chances are you run towards things like that, not away.

While writing bashunit, the library for unit testing bash scripts, I wanted to not have to enumerate test functions by hand. Doing that would be annoying, error-prone and just silly. Bash being a kitchen sink, must have a way to discover that instead. Most programming languages, with the notable exception of C, have some sort of reflection or introspection capability, where the program can discover something about itself through a specialized API. Details differ widely but the closer a language is to graphical user interfaces or serialization and wire protocols, the more likely it is to grow such capability. Introspection has to have a cost, as there must be additional meta-data that describes the various types, classes and functions. On the upside, much of this data is required by the garbage collector anyway, so you might as well use it.

Bash is very far away from that world. Bash is rather crude in terms of language design. Still it has enough for us to accomplish this task. The key idea is to use the declare built-in, which is normally used to define variables with specific properties. When used with the -F switch, it can also be used to list function declarations, omitting their body text.

We can couple that with a loop that reads subsequent function declarations, filter out the declaration syntax and end up with just the list of names. From there on all we need is a simple match expression and we can find anything matching a simple pattern. Et voilà, our main function, which discovers all the test functions and runs them.

#!/bin/bash
bashunit_main() {
    local def
    local name
    declare -F | while IFS= read -r def; do
        name="${def##declare -f }"
        case "$name" in
            test_*)
                if ! "$name"; then
                    echo "bashunit: test $name failed"
                    return 1
                fi
                ;;
        esac
    done
}

Tomorrow we will build on this idea, to create a very simple test runner and coverage analyzer.

Measuring execution coverage of shell scripts

Today I will talk about measuring test coverage of shell scripts


Testing is being honest about our flawed brains that constantly make mistakes regardless of how much we try to avoid it. Modern programming languages make writing test code a first-class concept, with intrinsic support in the language syntax and in the first-party tooling. Next to memory safety, concurrency safety, excellent testing support allows us to craft ever larger applications with an acceptable failure rate.

Shell scripts are as old as UNIX, and are usually devoted to glue logic. Normally testing shell scripts is done the hard way, in production. For more critical scripts there's a tendency to test the end-to-end interaction but as far as I'm aware of, writing unit tests and measuring coverage is unexpected.

In a way that's sensible, as long as shell scripts are small, rarely changed and indeed are battle tested in production. On the other hand nothing is unchanged forever, environments change, code is subtly broken and programmers on the entire range of the experience spectrum, can easily come across a subtly misunderstood, or broken, feature of the shell.

In a way static analysis tools have outpaced the traditional hard way of testing shell programs. The utterly excellent shellcheck program should be a mandatory tool in the arsenal of anyone who routinely works with shell programs. Today we will not look at shellcheck, instead we will look at how we can measure test coverage of a shell program.

I must apologize, at all times when I wrote shell I really meant bash. Not because bash is the best or most featureful shell, merely because it happens to have the right intersection of having enough features and being commonly used enough to warrant an experiment. It's plausible or even likely that zsh or fish have similar capabilities that I have not explored yet.

What capabilities are those? Ability to implement an execution coverage program in bash itself. Much like in when using Python, C, Java or Go, we want to see if our test code at least executes a specific portion of the program code.

Bash has two features that make writing such a tool possible. The first one is most likely known to everyone, the set -x option, which enables tracing. Tracing prints the commands, just as they are executed, to standard error. This feels like almost what we want, if only we could easily map the command to a location in a source file, we could construct a crude, line-oriented analysis tool. The second feature is also standard, albeit perhaps less well-known. It is the PS4 variable, which defines the format of the trace output. If only we could put something as simple as $FILENAME:$LINENO there, right? Well, in bash we can, although the first variable has a bash-specific name $BASH_SOURCE. The second feature which makes this convenient, is the ability to redirect the trace to a different file descriptor. We can do that by setting $BASH_XTRACE_FD=... to a file descriptor of an open file.

With those two features combined we can easily run a test program, which sources a production program, exercises a specific function and quits. We can write unit tests. We a can also run integration tests and check if any of the production code is missing coverage that indicates important test is missing.

I pieced together a very simple program that uses this idea. It is available at https://github.com/zyga/bashcov and is written in bash itself.

Signal to noise ratio in build systems

Today I will argue why silent rules are a useful feature of good build systems.


Build systems build stuff, mainly by invoking other tools, like compilers, linkers, code generators and file system manipulation tools. Build tools were traditionally printing some indication of progress. Make displays the commands as they are executed. CMake displays a quasi progress bar, including the name of the compiled file and a counter.

Interestingly, it seems the more vertically oriented, the less output shows up by default. If you need to hand-craft a solution out of parts, like with make, debugging the parts is important to the program you are building. Compare the verbosity of a autotools build system with a go build ./... invocation, that can build many thousands of programs and libraries. The former prints walls of text, the latter prints, nothing, unless there's an error.

As an extreme case, this is taken from the build log of Firefox 79. This is the command used to compile a single file. Note that the command is not really verbatim, as the <<PKGBUILDDIR>> parts hide long directory names used internally in the real log (this part is coming from the Debian build system). Also note that despite the length, this is a single line.

/usr/bin/gcc -std=gnu99 -o mpi.o -c -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -DNDEBUG=1 -DTRIMMED=1 -DNSS_PKCS11_2_0_COMPAT -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -DSTATIC_EXPORTABLE_JS_API -I/<<PKGBUILDDIR>>/third_party/prio -I/<<PKGBUILDDIR>>/build-browser/third_party/prio -I/<<PKGBUILDDIR>>/security/nss/lib/freebl/mpi -I/<<PKGBUILDDIR>>/third_party/msgpack/include -I/<<PKGBUILDDIR>>/third_party/prio/include -I/<<PKGBUILDDIR>>/build-browser/dist/include -I/usr/include/nspr -I/usr/include/nss -I/usr/include/nspr -I/<<PKGBUILDDIR>>/build-browser/dist/include/nss -fPIC -include /<<PKGBUILDDIR>>/build-browser/mozilla-config.h -DMOZILLA_CLIENT -Wdate-time -D_FORTIFY_SOURCE=2 -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -fno-strict-aliasing -ffunction-sections -fdata-sections -fno-math-errno -pthread -pipe -g -freorder-blocks -O2 -fomit-frame-pointer -funwind-tables -Wall -Wempty-body -Wignored-qualifiers -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wduplicated-cond -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=coverage-mismatch -Wno-error=free-nonheap-object -Wno-multistatement-macros -Wno-error=class-memaccess -Wno-error=deprecated-copy -Wformat -Wformat-overflow=2 -MD -MP -MF .deps/mpi.o.pp /<<PKGBUILDDIR>>/security/nss/lib/freebl/mpi/mpi.c

This particular build log is 299738 lines long. That's about 40MB of text output, for a single build.

Obvously, not all builds are alike. There is value in an overly verbose log, like this one, because when something fails the log may be all you get. It is useful to be able to repeat the exact steps taken to see the failure in order to fix it.

On the other end of the spectrum you can look at incremental builds, performed locally while editing the source. There some things are notable:

  • The initial build is much like the one quoted above, except that the log file will not be looked at by hand. An IDE may parse it to pick up warnings or errors. Many developers don't use IDEs and just run the build and ignore the wall of text it produces, as long as it doesn't fail entirely.

  • As code is changed, the build system will re-compile the parts that became invalidated by the changes. This can be as little as one .c file or as many as all the .c files, that include a common header that was changed. Interestingly computing the number of files that need changes may take a while and it may be faster to fire start compiling even before the whole set is known. Having a precise progress bar may be detrimental to the performance.

  • The output of the compiler may be more important than the invocation of the compiler. After all, it's very easy to invoke the build system again. Reading a page-long argument list to gcc is less relevant than the printed error or warning.

That last point is what I want to focus on. The whole idea is to hide or simplify some information, in order to present other kind of information more prominently. We attenunate the build command to amplify the compiler output.

Compare those two make check output logs from my toy library. I'm working on a few new manual pages and I have a rule which uses man to verify syntax. Note that I specifically used markup that wraps long lines, as this is also something you'd see in a terminal window.

This is what you get out of the box:

zyga@x240 ~/D/libzt (feature/defer)> make check
/usr/bin/shellcheck configure
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/ZT_CMP_BOOL.3 2>&1 >/dev/null | sed -e 's@tbl:@man/ZT_CMP_BOOL.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/ZT_CMP_INT.3 2>&1 >/dev/null | sed -e 's@tbl:@man/ZT_CMP_INT.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/ZT_CMP_PTR.3 2>&1 >/dev/null | sed -e 's@tbl:@man/ZT_CMP_PTR.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/ZT_CMP_RUNE.3 2>&1 >/dev/null | sed -e 's@tbl:@man/ZT_CMP_RUNE.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/ZT_CMP_UINT.3 2>&1 >/dev/null | sed -e 's@tbl:@man/ZT_CMP_UINT.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/ZT_CURRENT_LOCATION.3 2>&1 >/dev/null | sed -e 's@tbl:@man/ZT_CURRENT_LOCATION.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/ZT_FALSE.3 2>&1 >/dev/null | sed -e 's@tbl:@man/ZT_FALSE.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/ZT_NOT_NULL.3 2>&1 >/dev/null | sed -e 's@tbl:@man/ZT_NOT_NULL.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/ZT_NULL.3 2>&1 >/dev/null | sed -e 's@tbl:@man/ZT_NULL.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/ZT_TRUE.3 2>&1 >/dev/null | sed -e 's@tbl:@man/ZT_TRUE.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/libzt-test.1 2>&1 >/dev/null | sed -e 's@tbl:@man/libzt-test.1@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/libzt.3 2>&1 >/dev/null | sed -e 's@tbl:@man/libzt.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_check.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_check.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_claim.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_claim.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_closure.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_closure.3@g'
mdoc warning: A .Bd directive has no matching .Ed (#20)
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_closure_func0.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_closure_func0.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_closure_func1.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_closure_func1.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_defer.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_defer.3@g'
Usage: .Fn function_name [function_arg] ... (#16)
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_location.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_location.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_location_at.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_location_at.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_main.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_main.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_pack_boolean.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_pack_boolean.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_pack_closure0.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_pack_closure0.3@g'
mdoc warning: A .Bd directive has no matching .Ed (#20)
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_pack_closure1.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_pack_closure1.3@g'
mdoc warning: A .Bd directive has no matching .Ed (#21)
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_pack_integer.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_pack_integer.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_pack_nothing.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_pack_nothing.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_pack_pointer.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_pack_pointer.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_pack_rune.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_pack_rune.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_pack_string.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_pack_string.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_pack_unsigned.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_pack_unsigned.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_test.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_test.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_test_case_func.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_test_case_func.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_test_suite_func.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_test_suite_func.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_value.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_value.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_visit_test_case.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_visit_test_case.3@g'
LC_ALL=C MANROFFSEQ= MANWIDTH=80 man --warnings=all --encoding=UTF-8 --troff-device=utf8 --ditroff --local-file man/zt_visitor.3 2>&1 >/dev/null | sed -e 's@tbl:@man/zt_visitor.3@g'
plog-converter --settings ./.pvs-studio.cfg -d V1042 --srcRoot . --renderTypes errorfile zt.c.PVS-Studio.log zt-test.c.PVS-Studio.log | srcdir=. abssrcdir=/home/zyga/Dokumenty/libzt awk -f /usr/local/include/zmk/pvs-filter.awk
./libzt-test
libzt self-test successful

This is what you get when you use ./configure --enable-silent-rules:

zyga@x240 ~/D/libzt (feature/defer)> make check
SHELLCHECK configure
MAN man/ZT_CMP_BOOL.3
MAN man/ZT_CMP_INT.3
MAN man/ZT_CMP_PTR.3
MAN man/ZT_CMP_RUNE.3
MAN man/ZT_CMP_UINT.3
MAN man/ZT_CURRENT_LOCATION.3
MAN man/ZT_FALSE.3
MAN man/ZT_NOT_NULL.3
MAN man/ZT_NULL.3
MAN man/ZT_TRUE.3
MAN man/libzt-test.1
MAN man/libzt.3
MAN man/zt_check.3
MAN man/zt_claim.3
MAN man/zt_closure.3
mdoc warning: A .Bd directive has no matching .Ed (#20)
MAN man/zt_closure_func0.3
MAN man/zt_closure_func1.3
MAN man/zt_defer.3
Usage: .Fn function_name [function_arg] ... (#16)
MAN man/zt_location.3
MAN man/zt_location_at.3
MAN man/zt_main.3
MAN man/zt_pack_boolean.3
MAN man/zt_pack_closure0.3
mdoc warning: A .Bd directive has no matching .Ed (#20)
MAN man/zt_pack_closure1.3
mdoc warning: A .Bd directive has no matching .Ed (#21)
MAN man/zt_pack_integer.3
MAN man/zt_pack_nothing.3
MAN man/zt_pack_pointer.3
MAN man/zt_pack_rune.3
MAN man/zt_pack_string.3
MAN man/zt_pack_unsigned.3
MAN man/zt_test.3
MAN man/zt_test_case_func.3
MAN man/zt_test_suite_func.3
MAN man/zt_value.3
MAN man/zt_visit_test_case.3
MAN man/zt_visitor.3
PLOG-CONVERTER zt.c.PVS-Studio.log zt-test.c.PVS-Studio.log static-check-pvs
EXEC libzt-test
libzt self-test successful

I will let you decide which output is more readable. Did you spot the mdoc warning lines on your first read? If you build system supports that, consider using silend rules and fix those warnings you now see.

Build system griefs - autotools

I always had a strong dislike of commonly used build systems. There's always something that would bug me about those I had to use or interact with.

Autoconf/Automake/Libtool are complex, slow and ugly. Custom, weird macro language? Check. Makefile lookalike with different features? Check. Super slow single threaded configuration phase. Check. Gigantic, generated scripts and makefiles, full of compatibility code, workarounds for dead platforms and general feeling of misery when something goes wrong. Check. Practical value for porting between Linux, MacOS and Windows. Well, sort of, if you want to endure the pain of setting up the dependency chain there. It feels very foreign away from GNU.

Autotools were the first build system I've encountered. Decades later, it's still remarkably popular, by both broad feature support and inertia. Decades later it still lights up exactly one core, on those multi-core workstations we call laptops. The documentation is complete but arguably cryptic and locked in weird info pages. Most projects I've seen cargo-cult and tweak their scripts and macros from one place to another.

Despite all the criticism autotools did get some things right, in my opinion. The build-time detection of features, as ugly, slow and abused for checking things that are available everywhere now, is still the killer feature. There would be no portable C software as we know it today without the ability to toggle those ifdefs and enable sections of the code depending on the availability and functionality of an API, dependency or platform feature.

The user-interaction via the configuration script, now commonly used to draw lines in the sand and show how one distribution archetype differs from the other, is ironically still one of the best user interfaces for building that does not involve a full blown menu system.

The theory where you don't need autotools to use a project built with it. The theoretical portability, albeit to mostly fringe systems, is also a noble goal. Though today I rarely see systems that don't rip out the generated build system and re-generate it from source, mainly to ensure nobody has snuck in anything nasty into that huge, un-auditable, generated shell monstrosity that's rivaling the size of modest projects.

Will autotools eventually be replaced? I don't think so. It seems like one of those things that gets phased out only when a maintainer retires. The benefits of rewriting the whole build system and move to something more modern must outweigh the pain and cost of doing so. In the end it would help to modernize autotools more than it would help to convince everyone to port their software over.

New Blog

Just testing the whole blogging via-notes-thing, thanks to https://standardnotes.org/.

The idea is that you can blog from a desktop or mobile client,
by creating a set of notes that appear as distinct posts. Not all notes are public, in fact, by default they are all encrypted and private.

The effort to set this up is remarkably low. The only downside is that, as all hosted products, there's a free tier that is not as nice as the paid subscription.

There's a snap or appimage for Linux. The way to get your blog listed on, wait for it, https://listed.to, (har har), is a bit cumbersome, but this is all thanks to privacy so it's not too bad.

I may keep this.

Oh and thanks to https://listed.to/@jasone for the idea.