Using macros to create “custom verbs”

Day-to-day interaction with Bazel happens primarily through a few commands: build, test, and run. At times, though, these can feel limited: you may want to push packages to a repository, publish documentation for end-users, or deploy an application with Kubernetes. But Bazel doesn’t have a publish or deploy command – where do these actions fit in?

The versatility of bazel run

Bazel’s focus on hermeticity, reproducibility, and incrementality means the build and test commands aren’t helpful for the above tasks. These actions may run in a sandbox, with limited network access, and aren’t guaranteed to be re-run with every bazel build.

Instead, we rely on bazel run: the workhorse for tasks we want to have side effects. Bazel users are accustomed to rules that create executables, and rule authors can follow a common set of patterns to extend this to “custom verbs”.

In the wild: rules_k8s

For an example, consider rules_k8s, the Kubernetes rules for Bazel. Suppose we have the following target:

# BUILD file in //application/k8s
k8s_object(
    name = "staging",
    kind = "deployment",
    cluster = "testing",
    template = "deployment.yaml",
)

The k8s_object rule builds a standard Kubernetes YAML file when bazel build is used on the staging target. However, the additional targets are also created by the k8s_object macro with names like staging.apply and :staging.delete. These build scripts to perform those actions, and when executed with bazel run staging.apply, these behave like our own bazel k8s-apply or bazel k8s-delete commands.

Another example: ts_api_guardian_test

This pattern can also be seen in the Angular project. The ts_api_guardian_test macro produces two targets. The first is a standard nodejs_test target which compares some generated output against a “golden” file (that is, a file containing the expected output). This can be built and run with a normal bazel test invocation. In angular-cli, we can run one such target with bazel test //etc/api:angular_devkit_core_api.

Over time, this golden file may need to be updated for legitimate reasons. Updating this manually is tedious and error-prone, so this macro also provides a nodejs_binary target that updates the golden file, instead of comparing against it. Effectively, the same test script can be written to run in “verify” or “accept” mode, based on how it’s invoked. This follows the same pattern we’ve learned already! There is no native bazel test-accept command, but the same effect can be achieved with bazel run //etc/api:angular_devkit_core_api.accept.

This pattern can be quite powerful, and turns out to be quite common once you learn to recognize it.

Adapting your own rules

Macros are the heart of this pattern. Macros are used like rules, but they can create several targets. Typically, they will create a target with the specified name which performs the primary build action: perhaps it builds a normal binary, a Docker image, or an archive of source code. In this pattern, additional targets are created to produce scripts performing side effects based on the output of the primary target, like publishing the resulting binary or updating the expected test output.

To illustrate this, we’ll wrap an imaginary rule that generates a website with Sphinx with a macro to create an additional target that allows the user to publish it when ready. Consider the following existing rule for generating a website with Sphinx:

_sphinx_site = rule(
     implementation = _sphinx_impl,
     attrs = {"srcs": attr.label_list(allow_files = [".rst"])},
)

Next, consider a rule like the following, which builds a script that, when run, publishes the generated pages:

_sphinx_publisher = rule(
    implementation = _publish_impl,
    attrs = {
        "site": attr.label(),
        "_publisher": attr.label(
            default = "//internal/sphinx:publisher",
            executable = True,
        ),
    },
    executable = True,
)

Finally, we define the following macro to create targets for both of the above rules together:

def sphinx_site(name, srcs = [], **kwargs):
    # This creates the primary target, producing the Sphinx-generated HTML.
    _sphinx_site(name = name, srcs = srcs, **kwargs)
    # This creates the secondary target, which produces a script for publishing
    # the site generated above.
    _sphinx_publisher(name = "%s.publish" % name, site = name, **kwargs)

In our BUILD files, we use the macro as though it just creates the primary target:

sphinx_site(
    name = "docs",
    srcs = ["index.html", "providers.html"],
)

In this example, a “docs” target is created, just as though the macro were a standard, single Bazel rule. When built, the rule generates some configuration and runs Sphinx to produce an HTML site, ready for manual inspection. However, an additional “docs.publish” target is also created, which builds a script for publishing the site. Once we check the output of the primary target, we can use bazel run :docs.publish to publish it for public consumption, just like an imaginary bazel publish command.

It’s not immediately obvious what the implementation of the _sphinx_publisher rule might look like. Often, actions like this write a launcher shell script. This method typically involves using ctx.actions.expand_template to write a very simple shell script, in this case invoking the publisher binary with a path to the output of the primary target. This way, the publisher implementation can remain generic, the _sphinx_site rule can just produce HTML, and this small script is all that’s necessary to combine the two together.

In rules_k8s, this is indeed what .apply does: expand_template writes a very simple Bash script, based on apply.sh.tpl, which runs kubectl with the output of the primary target. This script can then be build and run with bazel run :staging.apply, effectively providing a k8s-apply command for k8s_object targets.