One of the major point points of sbt has been the lack of first class support for file tracking. Many build tools, most notably make, provide a mechanism for declaring the file dependencies of a task. The tool can then skip evaluation of tasks whose dependencies have not changed. While sbt has long supported ad hoc incremental tasks using its caching apis, many sbt tasks are inefficient because they do not implement incremental evaluation. The situation has been improved with the release of sbt 1.3.0. Building on sbt 1.3.0, I introduce sbt-make, an sbt plugin that adds syntax to write make style incremental tasks directly in a build.sbt file.

sbt 1.3.0 adds new file tracking apis to write incremental tasks without explicitly having to track the input files. Tasks written using the new apis seamlessly interoperate with continuous execution (aka the ~ command) so that the file dependencies of a task are automatically monitored. Nevertheless, compared to tools like make, the new file tracking apis are relatively low level and not entirely user friendly. For example, a task that compiles c source files into object files might look something like:

import scala.sys.process._
val buildObjects =
    taskKey[Seq[Path]]("Compiles c source files into object files")
buildObjects / fileInputs := sourceDirectory.value.toGlob / "*.c" :: Nil
buildObjects := {
  val targetPath = target.value.toPath
  def objectPath(path: Path): Path =
    targetPath / path.getFileName.toString.replaceAll("\\.c", "\\.o")
  val changes = buildObjects.inputFileChanges
  (changes.created ++ changes.modified).foreach { file =>
    s"gcc -c $file -o ${objectPath(file)}".!!
  }
  buildObjects.inputFiles.map(f => objectPath(f))
}

While this approach gets the job done, there is a lot of ceremony to determine which files to rebuild and the names of the output files. The naive implementation is also serial by default even though targets may often safely be built from their sources in parallel.

To address these issues and more, I wrote the sbt-make plugin for sbt 1.3.0. It adds syntax for defining incremental tasks using a make inspired, declarative syntax. Using sbt-make, the example above could be written:

pat"$target/%.o" :- pat"$sourceDirectory/%.c" build
  sh(s"gcc -c ${`$<`} -o ${`$@`}")

This syntax should be familiar to anyone who has ever written a Makefile. At any rate, I think we can all agree that neither scala nor sbt has enough symbolic operators.

sbt-make in action

The sbt-make repository includes a few examples. To illustrate the features of sbt-make, I will demonstrate a few of the make commands in a local clone of the simple c project example.

Targets that are defined using the

$TARGET :- $SOURCE build
    $IMPL

syntax can be evaluated by running make $TARGET in the sbt shell. The make task supports tab completions as can be seen in this video:

Notice that no files were rebuilt in the call to make run.

Because the sbt 1.3.0 file tracking apis integrate with sbt’s continuous execution, ~, the dependencies of tasks defined by sbt-make are automatically monitored by the ~ command:

Another example in the sbt-make repository is for a simple static site generator. I enhanced the example with a task called refresh1 that rebuilds the site (using a number of sbt-make incremental tasks) and then refreshes any safari tabs that have one of the site file urls open:

These are just a few example of how sbt-make, and sbt itself, makes it possible to create responsive tasks that deliver users immediate feedback on their changes.

Parallelism

An sbt-make task that is specified with a pattern target, builds multiple targets for each of the files matching a corresponding source pattern. Patterns are defined using the custom string interpolator pat. The targets of a pattern task are evaluated in parallel by default. This can be seen by piping the output of sbt to ts "%H:%M:%.S". Running make target/objects/%.o, the output is:

17:29:26.756863 sbt:make-example> clean; debug; make target/objects/%.o
17:29:26.818471 [success] Total time: 0 s, completed Sep 10, 2019, 5:29:26 PM
17:29:26.838387 [debug] Checking for meta build source updates
17:29:26.855727 [debug] Running shell command: 'gcc -c /private/tmp/make-example/mylib/src/bar.c -I/private/tmp/make-example/mylib/src/include -o /private/tmp/make-example/target/objects/bar.o'
17:29:26.855851 [debug] Running shell command: 'gcc -c /private/tmp/make-example/mylib/src/foo.c -I/private/tmp/make-example/mylib/src/include -o /private/tmp/make-example/target/objects/foo.o'
17:29:26.913451 [debug] Checking for meta build source updates

Notice that the commands to compile bar.c and foo.c began within 100 microseconds of one another.

We can limit the parallelism to one task at a time by running make with -j1:

17:31:11.439628 sbt:make-example> clean; debug; make -j1 target/objects/%.o
17:31:11.494084 [success] Total time: 0 s, completed Sep 10, 2019, 5:31:11 PM
17:31:11.511232 [debug] Checking for meta build source updates
17:31:11.526338 [debug] Running shell command: 'gcc -c /private/tmp/make-example/mylib/src/bar.c -I/private/tmp/make-example/mylib/src/include -o /private/tmp/make-example/target/objects/bar.o'
17:31:11.592264 [debug] Running shell command: 'gcc -c /private/tmp/make-example/mylib/src/foo.c -I/private/tmp/make-example/mylib/src/include -o /private/tmp/make-example/target/objects/foo.o'
17:31:11.643097 [debug] Checking for meta build source updates

This time, foo.c wasn’t compiled until roughly 600ms after bar.c. Unsurprisingly, the total runtime was also about 600ms longer with -j1 set.

Why use make inspired syntax?

Make is a tool that has stood the test of time. In spite of quirks like mandatory tab characters, the syntax does an excellent job of communicating the dependencies of a task. If I had invented a new syntax for sbt-make, that would have just been yet another dsl for users to learn. Restrictions on valid syntax in scala make it impossible to exactly mimic Makefile syntax, but sbt-make gets close enough that it should be recognizable to anyone familiar with make. Though the translation isn’t perfect, there are many things that easier to express in scala/sbt. Thus an sbt-make build.sbt can often be written more compactly than an equivalent Makefile.

Recycling the syntax of make also guided the evaluation semantics. For the most part, sbt-make evaluates tasks in the way one would expect given the behavior of make.

Looking forward

sbt-make is a complement to the new file tracking apis introduced in sbt 1.3.0. In fact, the strength of sbt-make is how seamlessly it integrates with the rest of sbt. sbt-make defined tasks can be called from regular sbt tasks and vice versa. Some incremental tasks do not have a one to one relationship between input and output files (e.g. scala or java compilation) or the input files are output files (e.g. scalafmt). In those situations, the built in sbt apis afford build and plugin authors with the tools they need to write reliable incremental tasks. For the more straightforward cases, sbt-make can make writing incremental tasks dramatically simpler and often higher performance than a naive implementation.

In subsequent posts, I will go into more detail about the new sbt 1.3.0 file tracking apis. While sbt 1.3.0 does most of the heavy lifting for sbt-make, the implementation of sbt-make relies on some advanced sbt techniques that may be useful, or at least interesting, to other sbt plugin authors.

Appendix

1 The refresh task is implemented like so:

TaskKey[Unit]("refresh") := {
  val paths = install.value
  val cmd = s"""#!/usr/bin/osascript
set paths to {${paths.map(p => s""""file://$p"""").mkString(",")}}
tell application "Safari"
  repeat with aWindow in windows
      repeat with aTab in tabs of aWindow
        if paths contains (URL of aTab) then
          tell aTab to do JavaScript "location.reload();"
        end if
      end repeat
  end repeat
end tell"""
  val script = Files.createTempFile("script", "")
  val permissions = Files.getPosixFilePermissions(script);
  import java.nio.file.attribute.PosixFilePermission
  permissions.add(PosixFilePermission.OWNER_EXECUTE);
  Files.setPosixFilePermissions(script, permissions);
  Files.write(script, cmd.getBytes)
  import scala.sys.process._
  try streams.value.log.info(s"$script".!!)
  finally Files.deleteIfExists(script)
}

The install task returns all of the paths that are created by the static site generator. The applescript command will reload any tab that has one of those files open.