fury

Writing a build for shapeless

About this tutorial

This tutorial will walk through the migration of a Scala project to use Fury. We have chosen shapeless because its build is not too complicated. In particular, shapeless has no dependencies other than Scala.

These instructions are for Fury version 0.4.0.

Creating the layer

To get started, we need to create a new layer. You can think of a layer as a ”workspace”; a collection of projects we are working on together. We will see later that layers compose: we can import the projects from one layer into another. But first, let’s create an empty directory and initialize it.

mkdir shapeless
cd shapeless
fury layer init

This will create a new layer.fury file, and initialize the directory as a git repository so that we can version-control any changes we make to the build.

Creating projects and modules

We start with an empty layer, so we will need to define a shapeless project. Projects typically consist of several modules, with dependencies between them, each one representing an invocation of the compiler. So we can use modules to compile a project in several steps.

In the case of shapeless, it only needs to call the compiler once on all its source code, so we only need a single module. We create them like this:

fury project add -n shapeless
fury module add -n core

This will create a project called shapeless, and a module called core, or shapeless/core, when referring to it outside of the project. We can check that this worked by running,

fury project list

and

fury module list

or just fury project and fury module for short. These commands will each display a table showing details of the projects and modules, respectively.

You might notice that both of the fury module commands performed actions which were applied to the shapeless project. That’s because, after we created the shapeless project, it became the “current” or “focused” project, and subsequent actions which operate on a project don’t need to specify it every time.

Specifying a compiler

The list of modules should look like this:

   MODULE  TYPE     DEPENDENCIES  SRCS  BINS  COMPILER       PARAMS
>  core    library                            java/compiler

This shows us some details about the module. It is a library, has no dependencies, sources, binaries or parameters, the > to the left of its name indicates that it is the “current” module. And we can see that its compiler defaults to java/compiler. We should change this to Scala.

But, by default, an empty layer does not know about any compilers other than Java, so we either need to define our own, or use an existing definition. Thankfully, a layer has already been created containing some versions of the Scala compiler, at propensive/.scala on GitHub. Note the . in the repository name; this is to distinguish it from a repository called scala.

These compilers were defined as modules in projects in a layer and published as a git repository, which means that it is available to us to import into our layer. This is a two-step process: we first specify a link to the git repository, and then import its projects into our layer.

fury repo add -u gh:propensive/.scala
fury import add -i .scala:scala-2.12.8

These commands require some explanation. Firstly, we create a new reference to the repository gh:propensive/.scala. This is just shorthand for GitHub repositories, and will be expanded automatically by Fury. Running this command will cause the repository to be cloned (as a bare repository) inside a new directory called .fury.

This repository, along with everything else inside the .fury directory, is managed by Fury, and we should not modify it. Though if we delete it, Fury will simply recreate it as necessary.

Adding the repository gives Fury access to its contents, to depend on for source files to compile, or for other layer definitions, but we do need to explicitly import a layer. We can do this using the import add command.

When we import a layer, we must specify which schema or variant of that layer we wish to use. Here, we have imported the 2.12.8 schema, whose name was chosen to indicate the version of the compiler that it defines. A schema could be called anything, like latest, scala-js or scala-2.11, a bit like a git branch. You may not know the name of the schema you wish to import, so Fury will offer tab-completions on all schemas it’s able to import from any repositories you have defined.

After running these two commands, our layer now includes the scala project, which we can see by running,

fury layer projects

We can now specify that our module will use the Scala compiler, by running,

fury module update -c scala/compiler

Tab-completion should offer it as a suggestion, so you don’t need to type the full name. You will notice that the compiler is called scala/compiler, rather than scala/compiler-2.12 or scala/compiler-2.12.8. This is so that we can refer to scala/compiler many times in the build, but if we wanted to switch to a different compiler, we would only need to make one change to the schema we import. Additionally, it makes it harder to ever have two different compilers in the same build (but not impossible; it might be desirable)!

Adding sources

We have chosen a compiler, but we need to tell it what to compile. Our sources will come from the milessabin/shapeless repository, so we first need to include that in the Fury build.

fury repo add -u gh:milessabin/shapeless

However, we would like to build a specific tagged version, rather than just taking the latest master branch. We can add a -V or —version argument to the command. If we add this after the -r (or —repo) argument, tab-completion will get a list of the tags and branches available in the repository. It might take a couple of seconds, depending on the speed of your connection. Let’s just use master, though.

fury repo add -u gh:milessabin/shapeless -V master

Having specified this repository, Fury will clone it. You can see a list of repositories Fury is tracking in this layer by running,

fury repo list

or just fury repo.

We will now use the source add command to attach sources to our shapeless/core module. The easiest way to do this is to type,

fury source add -d

and to press the tab-key to have Fury suggest a list of completions. These are in the form of a repository id, followed by a colon (:), followed by a path to a directory within that repository, and Fury will suggest any directory it finds in any repository containing files ending in .scala or .java. Shapeless’s sources are organized into several different directories, some of which are only used for compiling certain major versions of Scala. We need to add three:

fury source add -d shapeless:core/src/main/scala
fury source add -d shapeless:core/src/main/scala_2.11+
fury source add -d shapeless:core/src/main/scala_2.13-

If we run fury module again, we should see the new compiler and sources shown in the table.

   MODULE  TYPE         DEPENDENCIES  SRCS                                           BINS  COMPILER        PARAMS

>  core    library                    shapeless:core/src/main/scala                        scala/compiler
                                      shapeless:core/src/main/scala_2.11+
                                      shapeless:core/src/main/scala_2.13-

We can now attempt to compile shapeless, by running,

fury build compile

or just,

fury

This will attempt to compile the shapeless sources. After about 30 seconds, the results will become clean: a lot of compile errors! This is because shapeless has some additional sources which are not included in the repository, but need to be generated automatically by the build.

Source code generation

The strategy we will use to generate and compile shapeless’s extra sources will be to create a new module which will be run to generate these sources into a ”shared” directory, and and then to add this as a source to the core module.

Let’s start by creating a new module called gen. This will be a module like the core module, but in addition to compiling it, we want to execute its main method, too. We do this by specifying its “type” to be application, rather than the default of library. We can specify the compiler at the same time.

fury module add -n gen -t application -c scala/compiler

The source code generation code is in the milessabin/shapeless repository, `Boilerplate.scala` but it’s not quite right for our purposes: it has a dependency on sbt which we don’t want to include, and it doesn’t have a main method, which is Fury’s entry-point to any application module.

There’s more than one way to work around this, but we will just take a copy of the file and store it in our build repository. It is, after all, part of the build, rather than part of shapeless.

mkdir src
curl -o src/boilerplate.scala https://raw.githubusercontent.com/milessabin/shapeless/master/project/Boilerplate.scala
git add src

We must also include this new src directory as a source for the gen module. As it’s not in an external repository, we don’t need to prefix it with a repository name, and can simply run,

fury source add -d src

We need to make some modifications, though. Let’s remove this sbt import on line 17,

import sbt._

and replace the gen method which looks like this,

  def gen(dir : File) = for(t <- templates) yield {
    val tgtFile = dir / "shapeless" / t.filename
    IO.write(tgtFile, t.body)
    tgtFile
  }  

with this more verbose main definition that doesn’t depend on anything except the Java standard library,

  def main(args: Array[String]): Unit = {
    import java.io._
    val dir = new File(new File(System.getenv("SHARED")), "shapeless")
    dir.mkdirs()
    for(t <- templates) {
      val file = new File(dir, t.filename)
      val writer = new BufferedWriter(new FileWriter(file))
      writer.write(t.body)
      writer.close()
    }
  }

Apart from using the Java standard library, this code will use the a path specified by the SHARED environment variable. This variable is set by Fury before any application is run, and points to a consistent directory inside the .fury folder, which will be accessible to all modules running or compiling within this layer.

Other projects which the shapeless build does not know about may also read or write to this directory, so it’s good practice to create a subdirectory named after the project you are defining. Project names within a layer are guaranteed to be unique, so if you follow this convention, there is no risk that your project’s build will clash with another.

Examining this file will show that the main method is defined inside an object called Boilerplate inside the default package, and we need to tell Fury this is the class it should invoke.

fury module update -M Boilerplate

We can now test that this works with,

fury

This will compile and run the gen module. When it has finished, you should see some new source files inside .fury/shared/shapeless/.

Note that the fury command will use the current module, which (since it was the last module we created) is gen. If we switch back to the core module again,

fury module select -m core

we can make some final changes to have it compile these source files we have just generated. We know that the shared directory is inside the .fury directory, but there is no guarantee that this will remain unchanged, so we use the special shared: prefix to refer to files inside this directory.

Add the sources using,

fury source add -d shared:shapeless

We have just run the gen module, so the generated sources happen to exist, but we need to make gen an explicit dependency of core. This is usually as simple as,

fury dependency add -l shapeless/gen

but this have a side effect that it would include the compiled codegen classfiles on the classpath for every downstream project. But we don’t want this; they should not be part of the shapeless distribution. So we add the —intrasitive/-I flag to indicate this.

fury dependency add -l shapeless/gen -I

Finally, running,

fury

should now do a full compilation of shapeless.

Supporting different compilers

Shapeless supports several different versions of Scala, but has slightly different builds for each major version. We can create variations of the build using a feature of Fury called schemas.

A schema is a complete copy of the entire layer, including all the details we specified for projects, modules, dependencies, sources, repositories and imports, so we can tweak any of these details.

Let’s create an additional schema for Scala 2.13.

When we initialized the layer, although we did not explicitly specify it, we were making changes in a schema called default. Let’s first rename this to scala-2.12 to make it more descriptive. If we later publish our shapeless layer, and other users want to import it, they will need to specify which schema to import, so the name should be as descriptive as possible.

fury schema update -n scala-2.12

And we can create a copy of this schema with,

fury schema add -n scala-2.13

Now we have more than one schema, any Fury commands we run will, by default, apply the changes to all the schemas at once, unless we explicitly specify one schema to change, using the —schema or -s argument to most commands.

This is true if, but only if, the element or value being changed is the same consistently across all the schemas. If differences already exist between the schemas for that value, Fury will not apply the change.

In order to support Scala 2.13, we need to make two changes: update the compiler, and change the source directories.

We can change the compiler definition by importing a different schema from the propensive/.scala repository. When we started, we imported the scala-2.12.8 schema, but we would like to use scala-2.13.0-m5 in our new scala-2.13 schema. Updating an existing import is not yet implemented, but we can just remove and re-add the import, like so,

fury import remove -i .scala:scala-2.12.8 -s scala-2.13
fury import add -i .scala:scala-2.13.0-m5 -s scala-2.13

Both the old and the new imported layers contain compiler modules called scala/compiler so these references still resolve.

There is one additional source directory to compile for Scala 2.13, which isn’t required for Scala 2.12, so we need to add that as well, like so:

fury source add -m core -d shapeless:core/src/main/scala_2.13+ -s scala-2.13

And that’s it. We can compile both schemas:

fury build compile -s scala-2.12
fury build compile -s scala-2.13

Were it not for this bug, we could even run both builds concurrently!

Publishing the layer

Having developed this build for shapeless, we would probably like to share it with other users so they can depend on the shapeless/core module in their builds. Publishing a build is about as easy as publishing a git repository, so we first need to create that new repository. This step depends on where you want to publish the layer, but if you have a GitHub account, and use the hub command-line tool, you can do this with,

hub create .shapeless

I have chosen to name the repository .shapeless (prefixed with a .) rather than shapeless as a very simple way of highlighting that this is a repository which corresponds to a project called shapeless, but is not the project itself. It will be quite common that users maintain or have a forked copy of a project and maintain a build for it, and it’s useful to have a naming convention. It is just a convention, though, and you may prefer to use shapeless-build or shapeless.fury, for example.

If you added the new repository without the hub command, you will need to use git to specify the location of that repository, for example, using,

git remote set-url origin git@github.com:propensive/.shapeless.git

Then, all you need to do to publish, is choose a tag, such as 2.3.4-r1 (the first build release corresponding to Shapeless 2.3.4), and run the command,

fury layer publish -t 2.3.4-r1

Later versions of Fury will take this opportunity to perform several validation checks on the layer to ensure that it is self-consistent and safe to publish, though the current version does none of these.

Nevertheless, it will be immediately available for other people to use. They can do that in their own layers with, for example,

fury repo add -u gh:propensive/.shapeless
fury import add -i .shapeless:scala-2.12

Summary

This tutorial provided a tour of some of the features of Fury which are implemented and working in the v0.3.0 private beta. It should be possible to follow all of the commands verbatim, and have the same experience, but Fury is not yet well tested, so issues certainly remain. Other features of Fury, not described here, may also be implemented.

Reports of any unexpected behavior are very welcome. Please reports bugs regarding Fury itself, or this documentation, to the issue tracker on GitHub.