A process for core library fixes and improvements

Eco - Can stand for Elm Compiler Offline. That is, its the elm compiler with the built in packaging stuff removed into a separate tool (eco-install). Also about opening up the Elm ecosystem.

1 Like

I think we are generally on the same page on this.

One complication and part of the reason for my viewpoint, is that currently we cannot send PRs to the core repos because the BDFL is not engaging with this (and has given his reasons). So a company maintaining Elm would necessarily have to create its own package repo system. For example, supermario mentioned that Lamdera could probably already take on that role, since he pretty much had to solve that already.

I am not against business paying for fixes, or having employees or consultants with commiter rights on the open source projects they consume. That is very often the case with Apache projects for example - many AWS services are obviously bundled and branded Apache projects. Another example of a company that consumes many of them is RedHat. They definitely have contributors on the projects they are interested in, but the governance model of Apache does not allow them to exclusively capture a project.

I think if you work as an Elm consultant or a web-dev business, it can be a good selling point if you can tell your customers that you are involved in core maintenance, have their back, can get critical fixes in if the need arises. Selling that as a service might be a good source of occasional side revenue. Or it might get you onto a project where you are expected to play that role if the need arises.

But I still think we would need the neutral, democratic, collectively run ownership of the core being maintained sitting outside of any particular business operation.

1 Like

I think that if the community were to do be pro-active in solving the mentioned issues, this is likely the way to go, as this is the least diverging or obtrusive solution.

From looking at the Google sheet link with the list of issues, most issues are located in core libraries, not in the compiler. Other issues (like 2 ^ -1 maybe?) could be patched by altering the compiled code. A combination of the 2 would resolve most problems I think.

Forking the compiler does not seem that “necessary” to me, and it would by far be the most arduous task as well, and I don’t think it’s cost-effective to do so.


On a separate note, as a tooling author, I am quite worried about the directions this may go. I don’t think me and other tooling authors want to support both Elm and a fork (or several forks), as that could be a lot of work for us, and each individual author would need to be convinced individually.

If some decide to fork Elm, my advice if you want to continue using Elm tooling (IDEs, elm-review, …) is to not change anything about Elm’s workings. For instance (but not limited to):

  • Don’t change the way the Elm CLI works
  • Don’t create an alternative elm.json nor change its behaviour
  • Don’t add/change new APIs to core modules
  • etc.

Most tools out there already support passing an Elm binary as an argument, so an alternative compiler would be very easy to support, as long as it works almost exactly like the Elm compiler.

That is why I am thinking that applying patches to the packages and altering the compiled source code is probably the way to go. Users still get to benefit from Elm tooling, tooling authors don’t have to spend time supporting a fork. And also, we probably then don’t need to change the name of the fork because there wouldn’t be a fork, just an additional tool to point people to. elm-optimize-level-2 kind of already does this (note that this would not be the tool to do this), and it didn’t have to fork the project.

I would urge whoever that feels like taking up these task to change as little as possible to Elm in their quest to solve the issues mentioned in this thread.

15 Likes

Just to further the points in 3. a little, in addition to @pdamoc 's proof of concept with patching ELM_HOME linked above I’ve started dreaming up a format of how these “patched” snapshots could be described declaratively and applied.

I’m sure there is a bunch of other prior art in all of the elm-git-install and shelm-like tools that we could explore.

Here’s my noodling:

elm-patchwork.json

{
  "elmVersion": "0.19.1",
  "packages": {
    "elm/core": {
      "version": "1.0.5",
      "base": {
        "gitRepoUrl": "https://github.com/elm/core",
        "gitCommitHash": "84f38891468e8e153fc85a9b63bdafd81b24664e"
      },
      "patchPlan": [
        {
          "name": "List.concatMap performance improvements",
          "filePath": "./patches/elm-core-1.0.5-list-concatmap.patch"
        }
      ]
    }
  }
}

consumed by some tool into types like:

{-| The top-level type corresponding to the elm-patchwork.json file
-}
type alias Project =
    { elmVersion : String -- "0.19.1"
    , packages : List Package
    }


type alias Package =
    { name : String -- "elm/core"
    , version : String -- "1.0.5"
    , base : PackageBase
    , patches : List Patch
    }


type PackageBase
    = BaseFromGit
        { gitRepoUrl : String -- not strictly GitHub?
        , gitCommitHash : String -- this is how we'll download that package
        }
    | BaseFromFilesystem
        { folderPath : String -- "./patchwork/bases/elm/core/hacked-1.0.5"
        }


type alias Patch =
    { name : String -- "List.concat: performance improvement"
    , contents : PatchContents
    }


type PatchContents
    = PatchFromGit
        { gitRepoUrl : String -- not strictly GitHub?
        , gitCommitHash : String -- this is how we'll download that patch
        , filePath : String -- "patches/elm-core-1.0.5-list-concat.patch"
        }
    | PatchFromFilesystem
        { filePath : String -- "./patches/elm-core-1.0.5-list-concat.patch"
        }
3 Likes

I did encounter a show-stopper bug in elm/core in my app in Jan. 2019. It was fixed in Nov. 2019.

After waiting about a month for my PR to be merged, I began applying the fix in my nix build script. When the fix was eventually merged, I updated and stopped patching.

Should I encounter any other show-stoppers in elm/* code in the future, I plan to use a similar approach. The main downside is that the fix didn’t kick in during development, only in “real” builds. (In this case, that was fine, I could just avoid the known landmine of albums containing more than 100 photos during development.)

3 Likes

I think nobody wants to change Elm, but would like to use Elm in more cases. Supporting certain JavaScript browser APIs requires the ability to use JavaScript more directly. I.e. it would be good if you could write core libraries more easily to allow for experimentation.

Manipulating compiler-output with scripts works perfectly well - we could make a bunch of those in a Git-repo to support various browser APIs, fix issues, and make performance-improvements.

elm-optimize-level-2 sort of does this

I do this a lot instead of asking for work elsewhere :sunny:

2 Likes

By giving a direct access to JavaScript, you would lose the guarantees the Elm compiler offers (no runtime errors) and probably its ability to make a so aggressive tree shaking.

2 Likes

I’ve been a consumer of both Lua and SQLite. They both appear to follow these principles:

  1. The core is under strict control of the author or author(s). Maybe they’ll look at pull requests, but it is largely the work of a single individual or a small set of individuals.

  2. The core is “small”. For Lua, it’s just the language and the core libraries defined as part of the language standard. For SQLite, it’s the SQL implementation.

  3. Bugs in releases are fixed very quickly in double-dot-releases. They generally stabilize out these bugs quickly but there have been occasional patches to older releases. Versions before a certain point are not patched, but anything expected to be potentially in use does get patched.

It seems like Elm matches up on 1. It is over expansive in the efforts about control on 2 though there is a pretty clear boundary that could be drawn about what is inside the scope of control/responsibility and what is not. On 3, this is clearly not the behavior since many bug fixes if they ever come also tend to come with significant language revisions. I feel that 1 is a healthy choice because it provides a clarity that committee driven processes often lack. It is, however, enabled for the community by 3 which is further enabled by 2.

4 Likes

Besides tooling, another reason not to fork more aggresively is the compiler and language itself. There is still work going on there by Evan, and the further away a fork gets from the original, the harder it becomes to pick up what he builds in the next version of Elm. That is why I advocate a Release Branch approach, at least until such time as everything that can be fixed in a release branch has been fixed. It would give us the opportunity to at least fix some stuff and get things moving. Its not possible to say exactly how much of that stuff might be able to carry forward into a newer version of Elm, but it seems likely at least some patches would carry over cleanly.

I don’t see a hurry to fork, there is plenty time. I did Java for 20 years, and now its becoming possible to make a living doing Elm, maybe I’ll be here for 20 years too. So I am enjoying this thread and listening to peoples points of view and spending some time thinking about possible directions and what is best for all.

1 Like

Just tiny opinion from Korea!
It seems like the writer of this post doesn’t content with the current Elm process. But I have different idea. In my opinion, current way is a great way to maintain Elm. I don’t want Wikipedia like system(Many people edit, so amount is huge, but quality isn’t reliable), I want Britannica like system(Fewer people edit, so amount is smaller, but quality is reliable). Of course, this way isn’t perfect match for “open source” but I hardly believe there is such a truly open source model.
And, I think Elm is the greatest language for reliable frontend coding even if there isn’t further updates for 2 years from now. Elm is already good. So, imprudently changing library implementation/adding features/adding syntax sugar will harm elm-way-clean-coding.

6 Likes

Of course nobody wants broken libraries, unfortunately it’s the case right now in some places.

Given the efforts that went into fixing them by really smart people from the community, along with documentation, benchmarks and testing, I find it harsh that you use the word “imprudently”.

Just my 2¢.

1 Like

I believe in staying very close to elm in semantics and implementation, so I like the actual idea of a release branch. Unfortunately I think the elm core team will still require a change of name and logo. In my humble opinion, when the name and logo changes, it is a fork. You can’t call it a release branch anymore.

So my two cents is that we should fork now, but make very minimal changes on the fork (with exactly the same minimalist / conservative attitude that one would adopt in a release branch, which Rupert has described). Somehow I think the elm core team will not like the idea of a release branch with the same name and logo as elm, but I would be happy to be corrected if I’m wrong.

I think most of us are agreed that we don’t want huge changes. Just the ability to make essential minimal changes in a timely fashion.

2 Likes

I’m sorry if I made you feel bad. I used the word “imprudently” to express thought that “just small changes could cause big problems”.
But, Elm’s philosophy is “A delightful language for reliable web applications.”. If Fix fits this philosophy, I think it is reflected quickly and if Fix doesn’t fits Elm’s philosophy it won’t be reflected. That is the reason why I love this language.

1 Like

You could kind of say that all Linux distros are releases of the upstream Linux kernel - I know its stretching the analogy with Elm a bit, but …

Each Linux distro has its own name, logo and identity. They all feed from a huge number of upstream projects and try to bring them together into a stable curated system. They all create opportunities to feedback patches to those upstream projects. Some have their own private repos and kernel patches (but must make those available as per the GPL) - RedHat Enterprise Linux being the obvious example.

Would put a different name and logo on it, but be completely open about the fact that it is actually Elm with a few modifications.

1 Like

I’m a bit of an outsider, but definitely interested in and appreciative of Elm. I wish I had more time to contribute. Here’s my two-cents worth.

Out of interest I looked at @evancz’s profile page and found his most recent post is:

He also provides some personal updates relating to the “silicon valley mindset” and the benefits of “physically moving to a region with different cultural norms”.

Evan concludes by saying he really appreciates the people who have been supportive of his present working style and priorities. And he mentions by name Mario, Dillon, Jeroen, and Luca.

As an inexperienced outsider, I’m reluctant to contribute further to the present discussion.

3 Likes

I went on a bit of a twitter rant about this recently, but I look a the bugs in the core libraries and I’m honestly quite impressed by how few of them there are that are actually critical and how many have straightforward workarounds - often just implementing a different function that checks inputs and short circuits in some way or otherwise behaves differently.

If I was looking to make the situation around bugs in the core libraries better (I think it’s fine, honestly) and I had the time to put in (which I definitely don’t) I would do something like this:

Look at the issues @rupert has collected (It’s a nice and thorough list). Anything that can be fixed with a wrapper function, or can otherwise be improved in pure Elm, put it in a library. Call it “elm-sandpaper” or something because it’s smoothing the rough edges. Also drop some elm-review rules in there that can disallow usage of the core version and suggest the sandpaper version instead. When you’re happy with the implementation, leave a little comment on the relevant issues saying “Hey there are workarounds for this and several other issues that you can find here…”

For things that affect kernel/JS code, JS transformation seems to have been broadly successful, so provide some tools for that, maybe piggyback off of elm-optimize. I’d put big warning labels on it to point out that if you use it to run Elm in places it’s not designed for like phones, electron, etc, there’s no guarantee of continued support.

For any significant doc issues, write some nice tutorials that point out important caveats to the docs. Brand them under the same “elm-sandpaper” project. It’s not the primary source, but most developers start with tutorials before API docs, and if these things are common knowledge in the community, people will easily find fixes, particularly if there’s a well-known set of folks providing those answers.

All this is not to say that I think folks shouldn’t fork the compiler if they want to. If that seems like a nice solution to you, or an interesting space to work in, go for it. To me, it looks like there are much simpler solutions available with far less overhead.

9 Likes

I am generally in agreement with what you conclude from the issue analysis work. There are not many surprises in it. Part of reason for doing it is to have some evidence to show that the situation is not so terribly bad as it stands. Another reason is to help put together a roadmap describing what is possible to fix without forking or breaking backwards compatability, and that there is a decent amount of work that should be done before we could justify a fork.

I think patching through elm-optimize-level-2 is a pretty gruesome idea. Especially if you consider that the only reason for doing it is because Elm has no bug fixing process that the community can engage with to get results.

By patching the core modules directly, those exact patches can be fed upstream and applied to the official versions. Comparing patched versions to the official ones can also be achieved with git diff (and elm diff for the APIs if you modified it a bit). The mapping between patches applied through JS transformation and those to the core modules is much harder to understand.

There are also examples of bugs that need additions to core APIs, as they cannot be done as user packages (only recorded this one in my sheet so far (^) can return 0.5 typed as an Int · Issue #960 · elm/core · GitHub, but I think there will be more). Those ones could not be fixed through JS transformation post-compilation, as the compiler would yield errors before you got that far. Actually, I think maybe 960 could be a user package, and use elm-review to prevent use of ^, as you suggest. But I feel pretty sure there are issues people want to fix with new APIs that need to be kernel packages.

JS transformation feels like a way of not dealing with the underlying issue of a broken process. Also why I am much more in favour of patching, as it is at least taking a step towards fixing that for all of us.

===

Another example of a fix that adds to the API but must be in the kernel package:

6 Likes

Yes fully agree, the community needs to have a way to allow Elm to be used in many more places, and an ability to fix clear bugs quickly. As long as it’s clear to everybody that Evan determines direction and language feature, and this is just smoothing things over, and allowing Elm to be used in more places.

2 Likes

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.