# Private Package Tool Spec

**URL:** <https://discourse.elm-lang.org/t/private-package-tool-spec/6779>\
**Category:** Request Feedback\
**Created:** [January 19, 2021, 2:05pm UTC](https://discourse.elm-lang.org/t/private-package-tool-spec/6779 "2021-01-19T14:05:05Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![rupert](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/rupert/32/1775_2.png) [@rupert](https://discourse.elm-lang.org/u/rupert)\
**Post date:** [January 19, 2021, 2:05pm UTC](https://discourse.elm-lang.org/t/private-package-tool-spec/6779/1 "2021-01-19T14:05:05Z")

</div>

I have written up a spec for a tool that will allow private packages with Elm. Still to fill in some details, but there is enough here to explain what it is and what it will do:

> <https://gist.github.com/rupertlssmith/d0671157ca30d272970de00864512236>

Any thoughts on this? Will it work with the compiler, and how it interacts with packages?

---

<div class="post-metadata">

**Author:** ![malaire](https://avatars.discourse-cdn.com/v4/letter/m/b782af/32.png) [@malaire](https://discourse.elm-lang.org/u/malaire)\
**Post date:** [January 19, 2021, 3:08pm UTC](https://discourse.elm-lang.org/t/private-package-tool-spec/6779/2 "2021-01-19T15:08:25Z")

</div>

I’m currently playing around with [this minimal patch](https://gist.github.com/malaire/e879a1fbe15229b90e14050ee3d1de9c) to elm compiler which allows using any package server instead of `https://package.elm-lang.org`.

```
ELM_PACKAGE_SERVER=https://elm.local elm make
```

---

<div class="post-metadata">

**Author:** ![rupert](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/rupert/32/1775_2.png) [@rupert](https://discourse.elm-lang.org/u/rupert)\
**Post date:** [January 19, 2021, 3:10pm UTC](https://discourse.elm-lang.org/t/private-package-tool-spec/6779/3 "2021-01-19T15:10:22Z")

</div>

Ok, but I am hoping to be able to make this work without patching the compiler. Not sure yet, if it will.

---

<div class="post-metadata">

**Author:** ![malaire](https://avatars.discourse-cdn.com/v4/letter/m/b782af/32.png) [@malaire](https://discourse.elm-lang.org/u/malaire)\
**Post date:** [January 19, 2021, 3:28pm UTC](https://discourse.elm-lang.org/t/private-package-tool-spec/6779/4 "2021-01-19T15:28:29Z")

</div>

The problem with changing `~/.elm` and/or `elm-stuff` (in addition to figuring out correct format) is that the program must duplicate exactly the dependency-resolution algorithm used by Elm compiler to add appropriate versions of all dependencies. If even a single package is missed or has wrong version then `elm make` will attempt to fetch missing ones from `package.elm-lang.org`.

---

<div class="post-metadata">

**Author:** ![francescortiz](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/francescortiz/32/1721_2.png) [@francescortiz](https://discourse.elm-lang.org/u/francescortiz)\
**Post date:** [January 19, 2021, 3:51pm UTC](https://discourse.elm-lang.org/t/private-package-tool-spec/6779/5 "2021-01-19T15:51:48Z")

</div>

I love the idea. What language do you intend to use to write the server?

On the other hand, I also think that patching or extending the compiler is the way to go. What you propose of splitting compilation and packaging makes a lot of sense IMHO.

---

<div class="post-metadata">

**Author:** ![rupert](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/rupert/32/1775_2.png) [@rupert](https://discourse.elm-lang.org/u/rupert)\
**Post date:** [January 19, 2021, 4:12pm UTC](https://discourse.elm-lang.org/t/private-package-tool-spec/6779/6 "2021-01-19T16:12:28Z")

</div>

> [@francescortiz](#):
>
> I love the idea. What language do you intend to use to write the server?

Probably Elm using `the-sett/elm-serverless`. Will be able to run locally with a javascript wrapper, or in the cloud as a serverless process. Cloud version would use say S3, and standalone version the local filesystem.

> [@francescortiz](#):
>
> What you propose of splitting compilation and packaging makes a lot of sense IMHO.

It makes a lot of sense to me too. I think Elm is unusual in having the compiler and packaging system to tightly bound together. But still, in the first instance my aim is to do it without a special compiler version, or breaking any existing system.

===

I write a lot of packages and have packages that depend on others. Its frustrating developing and having to switch things in and out of the `src` list in `elm.json`, and of course adjust the transitive dependencies as you do it. Quite a few times, I’ve just published a package that is a work in progress, just so I can use it in another project I am working on it, as that can be the easiest way.

I’d be interested in being able to support versions \< 1.0.0 to help with that. But that seems unlikely to work with the compiler.

I can always publish a private package to `my-project-alpha 1.0.0`, and bump it as many times as needed during development, then publish as `my-project 1.0.0`, once its ready. – _Added this as an example in the Gist._

---

<div class="post-metadata">

**Author:** ![rupert](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/rupert/32/1775_2.png) [@rupert](https://discourse.elm-lang.org/u/rupert)\
**Post date:** [January 19, 2021, 4:28pm UTC](https://discourse.elm-lang.org/t/private-package-tool-spec/6779/7 "2021-01-19T16:28:10Z")

</div>

> [@malaire](#):
>
> If even a single package is missed or has wrong version then `elm make` will attempt to fetch missing ones from `package.elm-lang.org` .

That should be ok though. If I install a private package, it is there and `elm make` does not need to go looking for it. If its not installed, `elm make` fails with missing depenency. Correct behaviour either way.

The case I am not sure about, is when the package is there, say version `1.0.0`, does `elm make` go looking for updates on `package.elm-lang.org`? I guess that is ok too, it just doesn’t find a newer `1.x.y` package, so will build the `1.0.0` that it does have.

---

<div class="post-metadata">

**Author:** ![malaire](https://avatars.discourse-cdn.com/v4/letter/m/b782af/32.png) [@malaire](https://discourse.elm-lang.org/u/malaire)\
**Post date:** [January 19, 2021, 5:15pm UTC](https://discourse.elm-lang.org/t/private-package-tool-spec/6779/8 "2021-01-19T17:15:13Z")

</div>

The problem I’m speaking of is when your command like tool concludes that `foo 1.2.3` is one of the dependencies and install it, but `elm make` concludes that instead `foo 1.2.4` is one of the dependencies and goes looking for that from `package.elm-lang.org` because it wasn’t installed by the tool.

> This package management system can also act as a local cache of the [package.elm-lang.org](http://package.elm-lang.org) site, allowing work to continue if the central package site is down.

This goal isn’t fulfilled here because the tool didn’t install correct version.

---

<div class="post-metadata">

**Author:** ![mattpiz](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/mattpiz/32/860_2.png) [@mattpiz](https://discourse.elm-lang.org/u/mattpiz)\
**Post date:** [January 19, 2021, 5:17pm UTC](https://discourse.elm-lang.org/t/private-package-tool-spec/6779/9 "2021-01-19T17:17:19Z")

</div>

> [@rupert](#):
>
> The case I am not sure about, is when the package is there, say version `1.0.0` , does `elm make` go looking for updates on `package.elm-lang.org` ? I guess that is ok too, it just doesn’t find a newer `1.x.y` package, so will build the `1.0.0` that it does have.

From my experience, elm requires that the package and version you are building exist officially. Otherwise compilation will fail. That’s the reason I created this package: [elm-placeholder-pkg 1.0.0](https://package.elm-lang.org/packages/mpizenberg/elm-placeholder-pkg/latest/).

When I want to replace the content of a package, I use that one and replace everything within `~/.elm/0.19.1/packages/mpizenberg/elm-placeholder-pkg/1.0.0/` by whatever elm code I need to test.

Warning for the casual elm dev, in general you should never touch things inside `~/.elm/` or it might mess up your elm environment.

---

<div class="post-metadata">

**Author:** ![malaire](https://avatars.discourse-cdn.com/v4/letter/m/b782af/32.png) [@malaire](https://discourse.elm-lang.org/u/malaire)\
**Post date:** [January 19, 2021, 5:20pm UTC](https://discourse.elm-lang.org/t/private-package-tool-spec/6779/10 "2021-01-19T17:20:03Z")

</div>

> [@mattpiz](#):
>
> From my experience, elm requires that the package and version you are building exist officially.

But this tool changes what Elm compiler thinks “exist officially”.

---

<div class="post-metadata">

**Author:** ![mattpiz](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/mattpiz/32/860_2.png) [@mattpiz](https://discourse.elm-lang.org/u/mattpiz)\
**Post date:** [January 19, 2021, 5:20pm UTC](https://discourse.elm-lang.org/t/private-package-tool-spec/6779/11 "2021-01-19T17:20:21Z")

</div>

> [@malaire](#):
>
> This goal isn’t fulfilled here because the tool didn’t install correct version.

There is flexibility regarding the elm binary. There generally isn’t a unique solution to a given dependency system. And so however you solve the problem is not relevant. In the end, the elm binary just check that the solution you picked actually is a correct solution. But if it is you are fine.

---

<div class="post-metadata">

**Author:** ![rupert](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/rupert/32/1775_2.png) [@rupert](https://discourse.elm-lang.org/u/rupert)\
**Post date:** [January 19, 2021, 5:22pm UTC](https://discourse.elm-lang.org/t/private-package-tool-spec/6779/12 "2021-01-19T17:22:59Z")

</div>

> [@mattpiz](#):
>
> From my experience, elm requires that the package and version you are building exist officially. Otherwise compilation will fail. That’s the reason I created this package: [elm-placeholder-pkg 1.0.0](https://package.elm-lang.org/packages/mpizenberg/elm-placeholder-pkg/latest/).
> 
> When I want to replace the content of a package, I use that one and replace everything within `~/.elm/0.19.1/packages/mpizenberg/elm-placeholder-pkg/1.0.0/` by whatever elm code I need to test.

Interesting, thanks for that.

I wonder if messing with `~/.elm/0.19.1/packages/registry.dat` can get the compiler to play nicely?

---

<div class="post-metadata">

**Author:** ![malaire](https://avatars.discourse-cdn.com/v4/letter/m/b782af/32.png) [@malaire](https://discourse.elm-lang.org/u/malaire)\
**Post date:** [January 19, 2021, 5:24pm UTC](https://discourse.elm-lang.org/t/private-package-tool-spec/6779/13 "2021-01-19T17:24:24Z")

</div>

> [@mattpiz](#):
>
> And so however you solve the problem is not relevant. In the end, the elm binary just check that the solution you picked actually is a correct solution. But if it is you are fine.

It is relevant if you want to work offline (which is a major goal for me personally). If you solve problem via modifying `~/.elm` and/or `elm-stuff` and get it wrong, then Elm compiler will contact `package.elm-lang.org` and the goal of working offline fails.

If you solve the problem by modifying elm compiler so it contacts local server instead of `package.elm-lang.org` then elm compiler will never contact `package.elm-lang.org` and the goal of working offline succeeds.

---

<div class="post-metadata">

**Author:** ![malaire](https://avatars.discourse-cdn.com/v4/letter/m/b782af/32.png) [@malaire](https://discourse.elm-lang.org/u/malaire)\
**Post date:** [January 19, 2021, 5:55pm UTC](https://discourse.elm-lang.org/t/private-package-tool-spec/6779/14 "2021-01-19T17:55:05Z")

</div>

> [@rupert](#):
>
> I wonder if messing with `~/.elm/0.19.1/packages/registry.dat` can get the compiler to play nicely?

Yes, I believe that is the list which must contain all packages and their versions, any package/version not in that list will be rejected. (Alternative solution if modifying compiler would be to return modified `all-packages` response to compiler.)

---

<div class="post-metadata">

**Author:** ![rupert](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/rupert/32/1775_2.png) [@rupert](https://discourse.elm-lang.org/u/rupert)\
**Post date:** [January 19, 2021, 9:28pm UTC](https://discourse.elm-lang.org/t/private-package-tool-spec/6779/15 "2021-01-19T21:28:03Z")

</div>

> [@malaire](#):
>
> It is relevant if you want to work offline (which is a major goal for me personally). If you solve problem via modifying `~/.elm` and/or `elm-stuff` and get it wrong, then Elm compiler will contact `package.elm-lang.org` and the goal of working offline fails.

Offline working is also a goal of mine - I shall add an example use case for it to my spec write up. Offline builds, and also being able to build locally (on the company network) if the package site is down, which it is occasionally is.

Actually, I’m curious now as to when the Elm compiler will contact `package.elm-lang.org` - does it always do it on every build? I have heard that people running offline/repeatable CI builds, with a pre-populated `~/.elm’ folder are experiencing a delay in compilation - presumably as the compiler tries to contact the package site, and fails but waits for a connection timeout.

At some point, perhaps it was 0.18 or before, did the elm compiler have an `offline` flag that could be passed on the command line? Like `elm make --offline` or similar? I seem to remember this, and assumed it might still be there on 0.19, but `elm make --help` does not describe any such option. Did I just imagine it being there on some earlier version?

---

<div class="post-metadata">

**Author:** ![malaire](https://avatars.discourse-cdn.com/v4/letter/m/b782af/32.png) [@malaire](https://discourse.elm-lang.org/u/malaire)\
**Post date:** [January 19, 2021, 9:53pm UTC](https://discourse.elm-lang.org/t/private-package-tool-spec/6779/16 "2021-01-19T21:53:23Z")

</div>

> [@rupert](#):
>
> Actually, I’m curious now as to when the Elm compiler will contact `package.elm-lang.org` - does it always do it on every build? I have heard that people running offline/repeatable CI builds, with a pre-populated `~/.elm’ folder are experiencing a delay in compilation - presumably as the compiler tries to contact the package site, and fails but waits for a connection timeout.

Looks like `elm make` will fetch `/all-packages/since/XXX` every time `elm-stuff` doesn’t exist and needs to be created. Even if `~/.elm` contains everything needed and so `elm make` doesn’t actually need to download anything.

So yes, pre-populating `~/.elm` isn’t enough to have offline build.

---

<div class="post-metadata">

**Author:** ![malaire](https://avatars.discourse-cdn.com/v4/letter/m/b782af/32.png) [@malaire](https://discourse.elm-lang.org/u/malaire)\
**Post date:** [January 19, 2021, 10:46pm UTC](https://discourse.elm-lang.org/t/private-package-tool-spec/6779/17 "2021-01-19T22:46:39Z")

</div>

I’m currently running [proof-of-concept local cache for package.elm-lang.org](https://gist.github.com/malaire/fb94585e0e15fe0ba44d8d78220e1c27) (70 lines PHP) to observe and log network accesses of elm compiler, while I learn enough Rust to do this properly.

---

<div class="post-metadata">

**Author:** ![rupert](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/rupert/32/1775_2.png) [@rupert](https://discourse.elm-lang.org/u/rupert)\
**Post date:** [January 20, 2021, 8:53am UTC](https://discourse.elm-lang.org/t/private-package-tool-spec/6779/18 "2021-01-20T08:53:27Z")

</div>

> [@malaire](#):
>
> Looks like `elm make` will fetch `/all-packages/since/XXX` every time `elm-stuff` doesn’t exist and needs to be created. Even if `~/.elm` contains everything needed and so `elm make` doesn’t actually need to download anything.
> 
> So yes, pre-populating `~/.elm` isn’t enough to have offline build.

I was thinking my installer would populate `elm-stuff`.

---

<div class="post-metadata">

**Author:** ![malaire](https://avatars.discourse-cdn.com/v4/letter/m/b782af/32.png) [@malaire](https://discourse.elm-lang.org/u/malaire)\
**Post date:** [January 20, 2021, 10:11am UTC](https://discourse.elm-lang.org/t/private-package-tool-spec/6779/19 "2021-01-20T10:11:03Z")

</div>

> [@rupert](#):
>
> I was thinking my installer would populate `elm-stuff` .

Since that data is the result of compiling Elm, I don’t think it’s possible to create that without compiler?

~~EDIT: Actually, do you intend to pre-create `elmi`/`elmo` files with elm compiler and then just copy them into `elm-stuff` by installer? That might work, but still leaves files `d.dat`, `i.dat` and `o.dat` to be created by the installer. Presumably those are some kind of indexes?~~

`elm-stuff` contains compiler artifacts of the package being compiled - not its dependencies. So I don’t see any way to pre-create any of those files.

(Compiler artifacts of dependencies are in `artifacts.dat` files under `~/.elm`.)

---

<div class="post-metadata">

**Author:** ![rupert](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/rupert/32/1775_2.png) [@rupert](https://discourse.elm-lang.org/u/rupert)\
**Post date:** [January 20, 2021, 12:06pm UTC](https://discourse.elm-lang.org/t/private-package-tool-spec/6779/20 "2021-01-20T12:06:16Z")

</div>

> [@malaire](#):
>
> `elm-stuff` contains compiler artifacts of the package being compiled - not its dependencies. So I don’t see any way to pre-create any of those files.

You are right - for some reason I thought a copy of the dependencies source files was put in there too.

[Next page](https://discourse.elm-lang.org/t/private-package-tool-spec/6779.md?page=2)