Private Package Tool Spec

Request Feedback
21

I’ve come up with a name for this project too - Eco. It kind of stands for Elm Compiler Offline, or at least that is one way the name can be read.

I have an idea about tagging packages too, which I’ll explain in another post soonish.

So far discussion has mostly been around technical details of whether or not this can work with the Elm compiler unaltered.

I’m also interested to get some feedback on the bigger picture. Would this be useful to you or the organisation where you work? Do any of the use cases fit with things you find yourself wanting to do? Would it be a welcome addition to the overall Elm system?

===

I added some more example use cases around working offline, as definitely a motivation for this:

Working Offline

You grab your laptop and head out to the cabin in the woods, to get some peace and quiet and your work done. There is no internet there. Before leaving town you refresh the package server on your laptop, so that you have all the latest Elm packages - at that moment in time at least. As you work you are able to install more packages that your application comes to need as you work on it. They are cached on your laptop and you are able to browse their docs too through the package server UI.

Avoiding Productivity Outages

Back in town, a large development team at ACME corp is busy getting a new software product in shape for the pending launch date. You have a cache of all Elm packages running on the local network, and this is being updated frequently from the main public package site. You notice on Elm Slack that the main package site is down, users are getting anxious because the site admin is in the US Pacific Time Zone, and likely to be asleep. With a large team, ACME could be losing many man days of work in this scenario. The development manager at ACME is relaxed and happy today, her team are able to continue working from the local package server that she set up on the company network.

22

So here is the package tagging idea:

Package Tagging

Tags will be used to describe what the runtime requirements of packages are. Tags will be inherited from package dependencies, by forming the super-set of all tags on the dependencies.

The following packages will be root tagged as core : elm/core

The following packages will be root tagged as browser : elm/browser

The following packages will be root tagged as vdom : elm/virutal-dom

All other packages will inherit some combination of core , browser and vdom tags.

A potential alternate compiler for the Elm language, as opposed to The Elm Architecture, may support only a sub-set of these tags. This feature is not useful at the present, but is included now, so that it is part of the package server API from the start.

Motivation for the package domain tags:

The intention is to be able to have an Elm build system that only supports say core , for running Elm code outside of the browser, with no virtual DOM to render as a view .

A future version may cater for private tags. For example ACME Corp can tag all its packages as acme , making it obvious when code depends on that private package domain. The private package domain at the head of those packages will not allow them to be published upstream, unless the private tag is removed by using only packages with public tags.

A build can be set to fail, if it includes packaging domains that it is not configured to allow. For example, ACME Corp might publish a popular open source library that gives it kudos alongside its regular business. This would not include the acme domain, and if a developer accidentally starts including that domain, the build will fail.

2 Likes
23

private registry packages could prepend the registry url so that they are never fetched from packages.elm-lang.org.

24

I wasn’t talking about private packages.

25

That’s true, you can see that sscce which shows that a 5s delay is present when you remove your elm-stuff/ and the compiler then tries to update its list of packages.

It turns out there is a rather simple trick to get rid of that 5s delay. The library used for http in the elm compiler can pick up a proxy set in the https_proxy environment variable. So I found a simple config for tinyproxy that fail all requests and effectively removes the 5s delay observed. Here is the config and how I use it in a docker container to run tests: elm-test-runner/bin/run.sh at 51df0350f0e74bd6cc8e7ecaf49e8c0ae6f81212 · exercism/elm-test-runner · GitHub

1 Like
26

The compiler seems to honor the environment https_proxy, like env https_proxy=http://my.proxy.local:3128 elm init, this could be a way to go without compiler patching, maybe someone could use node.js or an other app-server as a proxy, but also TLS needs to be broken…
Sometimes I think, I have something like a “stockholme syndrome” regarding elm. :thinking:

2 Likes
27

What do you mean by that? You mean it checks it gets a valid cert because its over HTTPS, and a local package server will have to serve up a self-signed cert, so maybe the compiler complains at that point?

Its a good idea anyway, I’ll try it out and see what happens.

28

For what it’s worth, there are a couple of use cases where I would find a tool like this to be useful.

  • Alternate ecosystems (for example, packages intended for use with platforms that build on Elm such as elm-pages, lamdera, etc.)
  • Commercial (paid) packages

I think that what you’re describing about being able to “tag” packages as having certain runtime requirements could pair nicely with ecosystem use case. It would be nice to be have a way to avoid cluttering up the main Elm package repository with packages that depend on specific runtime requirements.

For the commercial use case, I know there are some people working on some commercial Elm packages, and right now there isn’t a good distribution channel. (I happen to be working on a commercial NPM package for Elm at the moment, but at some point I could imagine wanting to work on a paid Elm package, too). It could be a really great thing to have a tool that could help with distribution of commercial packages.

I hope those use cases are helpful, I’d be happy to discuss them more if you have any questions!

29

Yes, that is precisely my thinking too - that is why I put:

“If a hacked version of the compiler is needed, it must have the publish command removed, so as to ensure the package.elm-lang.org site is treated as read-only, and source of truth.”

It is also the intended use case for tags.

  • An alternate compiler with different runtime requirements should use its own set of tags to describe them. The idea being that it is very clear that these are not part of the Elm kernel.
  • It should also be very clear that having a different runtime carries a risk, if depending on Elms undocumented kernel interface.
  • It should also be very clear that there is absolutely no obligation on Elm to avoid breaking changes to that interface, or to port third party packages using it, to new versions of Elm.

Another possibility is that an Elm implementation could arise that compiles to a completely different back-end. A known example is Morphir, which compiles Elm to Scala. As it is focussed on business rules, I am guessing its runtime requirement is just core, and it does not need vdom or browser at all:

With the tags it would be easy to set up a mirror of the Elm package site, just for core, and get all packages that can compile on core alone - mostly data structures/algorithms/update helpers/etc. Anyone using Morphir would then have a clear view on what is available to them.

Of course, as Morphir compiles to Scala, it needs to completely re-implement the core kernel packages with Scala implementations, and needs to have identical APIs to them. The API comparison can be automated. Its private package server would have the core packages in it, maybe provided as .jar files (a JVM thing). It would implement the core tag, but not using the originals. Or maybe it might use the originals, and run them as javascript on the JVM (which is also a thing). Scala also compiles to JS doesn’t it?

Also worth noting that Morphir is not an identical implementation of Elm as the original - it has 64-bit integers for one thing. Some kind of formal language spec and test suite might eventually be needed for Elm. The beauty of Elm is that it is such a small language, so this is a less daunting proposition than for many languages.

elm-pages is an interesting example too. I originally assumed it would not depend on browser, but I see it does - well of course it does, because it runs client side too. I think it would be possible to split the client side out into a separate package, if you ever wanted too. But elm-pages was what got me thinking that vdom and browser should be separate tags - since you need vdom for server side rendering, but not browser. I can imagine a future where elm-pages has a different implementation when doing server side rendering, but runs with pure Elm on the client side.

Also note, in the spec I put:

  1. Confirm the package does not contain ports.
  2. Confirm the package does not contain kernel code.

No intention at the present time of overriding these restrictions. I just want to focus on doing private packages with minimal disruption to the current system. The reason for introducing the tags idea now, is that its going to be easier to get that in right from the start, even though it is not immediately useful.

30

Depending on how elm compiler does HTTPS, it might not accept self-signed certificate and might require proper certificate instead (meaning you need local CA so you can create proper spoofed certificate for any website).

I have such a setup on localhost, but I havn’t tried playing with HTTPS proxies before …

31

This does not make sense to me. You would need a fork of the compiler in order to have it use a different (private) domain for the packages. With this new domain for packages running, it would make sense to keep the publish functionality because all the publishing will be done ONLY to the private packages server. The packages server would have to also be forked and logic implemented that would treat differently the packages belonging to the private company.

If one needs to publish to the official package repository, they would use the official elm binary.

32

Correct.

But for publishing to a private repository, the idea is to use the new command line tool I am speccing out. Something along these lines:

> eco-install publish

Reading `eco.json`...
Reading `elm.json` ...
Publishing to `packages.acme.com`.
... Package uploaded.
... Git tag correct.
... Checksum calculated.
... API version is correct, 2.1.0.
... Compilation succeeded using 'elm 0.19.1'
... Publishing succesful.

Its not 100% clear yet if a fork is needed (if HTTP_PROXY works, for example).

The reason I put this in, is that I just want to make absolutely sure that it is not possible to publish a private package to package.elm-lang.org accidentally. For example, there is already an Elm compiler out there that removes native code restrictions, and someone has used it to publish a package with kernel code to package.elm-lang.org. I don’t think that is right, although it is not really causing any harm since the official compiler will not build that package. However, if it happened a lot, it might start getting hard to understand what packages you can use, and what needs some special compiler.

So my policy is - leave the existing Elm system alone, and make sure I don’t do anything that might break it. Treat it as an upstream system that is a read-only source of truth, so far as eco is concerned.

You are right though, in that a forked compiler could have the publish command, and publish to a private repository. Personally, I think tight coupling of compiler and package system is not a good design. I am thinking a system like Java and Maven is a better way - Java is the compiler, Maven is the build/packaging tool.

33

I agree with this policy too. A private fork of the compiler should work with a private repository and, if implemented properly, there should be zero impact on the current ecosystem.

1 Like
34

Quick test shows that using https proxy seems to work well here.

I have simple package server at https://box/elm and can use it with modified Elm compiler as

ELM_PACKAGE_SERVER=https://box/elm elm make

With unmodified Elm compiler and mitmproxy I can do same with

mitmproxy --ssl-insecure -M '|//package.elm-lang.org/|//box/elm/'
https_proxy=http://127.0.0.1:8080 elm make

ps. --ssl-insecure is needed because mitmproxy doesn’t know about my local CA. There is probably an option somewhere to fix that.

2 Likes
35

Nice, so it does HTTPS but does not validate the CA. - I kind of expected that would be the default.

Its not the prettiest way of running the compiler, but I think its a good start for a proof of concept. Thanks for figuring this out.

36

As I read the documentation of mitmproxy, a CA-certificate from mitmproxy must be installed on the client. This mitmproxy-CA-certificate is working system-wide. That’s what I mean by breaking TLS, as an example.
The Elm-compiler probably checks the certificate of package.elm-lang.org, created by mitmproxy in this case, using the system-certificates. It trusts the mitmproxy-CA, because the system trusts the mitmproxy-CA.
(O.T.: This is nearby a common way used in companies to find malware encrypted in HTTPS-traffic, causing often web-sockets not to work, as the web-browsers use the same “CONNECT”-method to the proxy for requesting a connection a server, independent of the used protocol, like HTTPS or WSS, while the used proxy can only handle HTTPS)
There are more advanced checks, but not used very often, like HPKP and now CAA.

37

yes, the reason why I need --ssl-insecure is that the certificate for box (my synonym for localhost) is signed by my personal CA, which is in system-certificates, but for some reason mitmproxy doesn’t seem to be using system certificates and it complains that certificate of box isn’t trusted.

39

@rupert I am willing to help you with this project is you need and want help.

40

@rupert I haven’t touched elm-pubgrub for a while since development switched to rust, so there are still bugs in there. But if you need a robust dependency solver in elm at some point let me know and I could take some time to update it.

1 Like
41

Yes, this could come in handy. I’ll get in touch if I need some help with it, thanks. :+1: