I have written up a spec for a tool that will allow private packages with Elm. Still to fill in some details, but there is enough here to explain what it is and what it will do:
Any thoughts on this? Will it work with the compiler, and how it interacts with packages?
I’m currently playing around with this minimal patch to elm compiler which allows using any package server instead of https://package.elm-lang.org.
ELM_PACKAGE_SERVER=https://elm.local elm makeOk, but I am hoping to be able to make this work without patching the compiler. Not sure yet, if it will.
The problem with changing ~/.elm and/or elm-stuff (in addition to figuring out correct format) is that the program must duplicate exactly the dependency-resolution algorithm used by Elm compiler to add appropriate versions of all dependencies. If even a single package is missed or has wrong version then elm make will attempt to fetch missing ones from package.elm-lang.org.
I love the idea. What language do you intend to use to write the server?
On the other hand, I also think that patching or extending the compiler is the way to go. What you propose of splitting compilation and packaging makes a lot of sense IMHO.
Probably Elm using the-sett/elm-serverless. Will be able to run locally with a javascript wrapper, or in the cloud as a serverless process. Cloud version would use say S3, and standalone version the local filesystem.
It makes a lot of sense to me too. I think Elm is unusual in having the compiler and packaging system to tightly bound together. But still, in the first instance my aim is to do it without a special compiler version, or breaking any existing system.
===
I write a lot of packages and have packages that depend on others. Its frustrating developing and having to switch things in and out of the src list in elm.json, and of course adjust the transitive dependencies as you do it. Quite a few times, I’ve just published a package that is a work in progress, just so I can use it in another project I am working on it, as that can be the easiest way.
I’d be interested in being able to support versions < 1.0.0 to help with that. But that seems unlikely to work with the compiler.
I can always publish a private package to my-project-alpha 1.0.0, and bump it as many times as needed during development, then publish as my-project 1.0.0, once its ready. – Added this as an example in the Gist.
That should be ok though. If I install a private package, it is there and elm make does not need to go looking for it. If its not installed, elm make fails with missing depenency. Correct behaviour either way.
The case I am not sure about, is when the package is there, say version 1.0.0, does elm make go looking for updates on package.elm-lang.org? I guess that is ok too, it just doesn’t find a newer 1.x.y package, so will build the 1.0.0 that it does have.
The problem I’m speaking of is when your command like tool concludes that foo 1.2.3 is one of the dependencies and install it, but elm make concludes that instead foo 1.2.4 is one of the dependencies and goes looking for that from package.elm-lang.org because it wasn’t installed by the tool.
This package management system can also act as a local cache of the package.elm-lang.org site, allowing work to continue if the central package site is down.
This goal isn’t fulfilled here because the tool didn’t install correct version.
From my experience, elm requires that the package and version you are building exist officially. Otherwise compilation will fail. That’s the reason I created this package: elm-placeholder-pkg 1.0.0.
When I want to replace the content of a package, I use that one and replace everything within ~/.elm/0.19.1/packages/mpizenberg/elm-placeholder-pkg/1.0.0/ by whatever elm code I need to test.
Warning for the casual elm dev, in general you should never touch things inside ~/.elm/ or it might mess up your elm environment.
But this tool changes what Elm compiler thinks “exist officially”.
There is flexibility regarding the elm binary. There generally isn’t a unique solution to a given dependency system. And so however you solve the problem is not relevant. In the end, the elm binary just check that the solution you picked actually is a correct solution. But if it is you are fine.
Interesting, thanks for that.
I wonder if messing with ~/.elm/0.19.1/packages/registry.dat can get the compiler to play nicely?
It is relevant if you want to work offline (which is a major goal for me personally). If you solve problem via modifying ~/.elm and/or elm-stuff and get it wrong, then Elm compiler will contact package.elm-lang.org and the goal of working offline fails.
If you solve the problem by modifying elm compiler so it contacts local server instead of package.elm-lang.org then elm compiler will never contact package.elm-lang.org and the goal of working offline succeeds.
Yes, I believe that is the list which must contain all packages and their versions, any package/version not in that list will be rejected. (Alternative solution if modifying compiler would be to return modified all-packages response to compiler.)
Offline working is also a goal of mine - I shall add an example use case for it to my spec write up. Offline builds, and also being able to build locally (on the company network) if the package site is down, which it is occasionally is.
Actually, I’m curious now as to when the Elm compiler will contact package.elm-lang.org - does it always do it on every build? I have heard that people running offline/repeatable CI builds, with a pre-populated `~/.elm’ folder are experiencing a delay in compilation - presumably as the compiler tries to contact the package site, and fails but waits for a connection timeout.
At some point, perhaps it was 0.18 or before, did the elm compiler have an offline flag that could be passed on the command line? Like elm make --offline or similar? I seem to remember this, and assumed it might still be there on 0.19, but elm make --help does not describe any such option. Did I just imagine it being there on some earlier version?
Looks like elm make will fetch /all-packages/since/XXX every time elm-stuff doesn’t exist and needs to be created. Even if ~/.elm contains everything needed and so elm make doesn’t actually need to download anything.
So yes, pre-populating ~/.elm isn’t enough to have offline build.
I’m currently running proof-of-concept local cache for package.elm-lang.org (70 lines PHP) to observe and log network accesses of elm compiler, while I learn enough Rust to do this properly.
I was thinking my installer would populate elm-stuff.
Since that data is the result of compiling Elm, I don’t think it’s possible to create that without compiler?
EDIT: Actually, do you intend to pre-create elmi/elmo files with elm compiler and then just copy them into elm-stuff by installer? That might work, but still leaves files d.dat, i.dat and o.dat to be created by the installer. Presumably those are some kind of indexes?
elm-stuff contains compiler artifacts of the package being compiled - not its dependencies. So I don’t see any way to pre-create any of those files.
(Compiler artifacts of dependencies are in artifacts.dat files under ~/.elm.)
You are right - for some reason I thought a copy of the dependencies source files was put in there too.