[CI Caching] Downloading project dependencies without building the app?

[Janiczek]

Hello! After the recent package.elm-lang.org situation we’re trying to cache the ELM_HOME directory in Docker/Drone CI, to be more resilient to zipballs HTTP request outages and also to not do needless work in each CI build.

The problem we’re dealing with is: we don’t want to trigger downloading packages on every application code change (source_directories).

The way Dockerfiles work makes us do:

• ADD elm.json
• ADD all the various source_directories (we have a lot of them)
• RUN elm make some/File.elm --output /dev/null

which has the disadvantage mentioned above.

A workaround I can see is making temporary Elm files which the Dockerfile will then try to compile. Thus skipping the “ADD source directories” step.

But it seems to me something like elm make --install-only or elm-json download (cc @ilias) would be more correct solution.

Do you see a different solution?

[ilias]

Running elm make without arguments is basically an --install-only command.

Caveats:

• the source-directories must exists, however, they don’t have to have anything in them. So if you can get away with something like mkdir -p foo bar/baz, that would be an easy solution!
• it exits with 1 so it’s hard to distinguish from “actual” failure

[Janiczek]

At the moment, until a better solution arrives, we went with the hack:

# A hack to be able to download dependencies without compiling our source code
# ------
# We want to avoid `COPY ./client /pro/client` before the `elm make` above all costs.
# That would download the dependencies on every code change...
# ------
# Plan:
# * `mkdir -p` the source directories
# * create a dummy Elm module in one of them
# * compile that one

COPY elm.json /pro/elm.json

RUN mkdir -p \
    client/_factories \
    client/_share/src \
    client/app-monolithic/src \
    client/audience-builder \
    client/chart-builder/src \
    client/crosstab-builder/src \
    client/dashboards/src \
    client/elm-dashboards/src \
    client/fullscreen-search \
    client/products/src \
    client/settings/src \
    client/tv-study/src

RUN echo "module DummyMain exposing (..)\nx = 1" >client/app-monolithic/src/DummyMain.elm

RUN elm make ./client/app-monolithic/src/DummyMain.elm --output /dev/null
# ^ Now we have downloaded the dependencies in a layer that's before adding the source code!
# That means dependencies will only download when elm.json changes, not when the source directories change.

RUN rm -rf ./client/app-monolithic/src/DummyMain.elm

# Sometime later: COPY ./client /pro/client

[choonkeat]

Adapted from your code, I made a pair of Dockerfile and Makefile that is cache friendly (only injecting elm.json and Dockerfile itself for the build) and can be applied to individual projects. As long as it’s run against the same docker server, the layers should cache automatically.

make elm-stuff/.elm

then subsequently, CI commands just need to have env ELM_HOME=elm-stuff/.elm set

e.g.

export ELM_HOME=elm-stuff/.elm
elm-test
elm make src/Main.elm

[turboMaCk]

While I think work around like this can work for most of the people there are some additional things you might be interested in.

• ELM_HOME environment variable can be used to define location of elm cache

• Elm uses registry.dat binary which is a snapshot of packages.

• This is how you can download package data without elm

• This is how binary file is generated