A process for core library fixes and improvements

Request Feedback
1

Hello!

:warning: EDIT (2021-11-20) to add some summary to this thread (there’s no “Mark as answer” but I’d mark Terezka’s post as one).

The answer to my questions below is that there are no official plans to change the current Elm process of how open-source contributions / package maintenance is dealt with. The situation and concerns are noted; to quote Terezka, “the current strategy is a deliberate choice.”

In my mind it has been helpful to think of Elm as Evan’s personal project, research language, source-available, “no promises.” The fact that it’s been very usable and productive choice for companies and developers is a great success, but it’s almost “by accident,” and doesn’t force a responsibility on Evan to change anything about how he approaches the project’s future.

In that sense, the fork option, while more time-consuming, makes a lot of sense in that the community can say “This new language X is based on Elm 0.19, but it’s 1.0 in the semver sense; you can migrate easily and it will not introduce large language changes and deviations from Elm 0.19 but will receive maintenance care and bugfixes and the package API will be able to evolve; and we will handle open-source contributions in a transparent community-led process.”

Essentially, sending a signal to companies: “This is stable and maintained, you can depend on it,” and giving Evan back tthe space to take Elm where his explorations lead him.

The question is, while this is a cleaner solution than eg. patches applied by tooling on Elm libraries, do we have enough resources as a community to make the jump?

:warning: End of summary edit

I’d like to start a respectful discussion between @evancz and the community about the possible ways to improve the situation where elm/* libraries don’t get attention and care fast enough due to the batched maintenance model, and the “time-to-being-dealt-with” is realistically measured in years rather than days/weeks/months.

A few examples:
• elm/parser: deadEndsToString or the bug about int committing to chomp e
• elm/json: maybe should be discouraged
• elm/core: performance improvements without API or behavioural changes (also see elm-benchmarks)

I probably don’t need to describe the pain points these create (that horse has already been beaten to death), and I don’t want this thread to become a list of “me too!” reactions.

I’d like to instead :one: ask @evancz what criteria would a solution have to pass before it would be realistically considered, and :two: discuss/draft possible solutions that meet them.

For an example, let me draft a (flawed!) proposal of how we might make the overall maintenance situation better: keeping space for big changes at the beginning / as the muse strikes, but allowing a judicious way to handle fixes / improvements to a library.

PROPOSAL 1

  • new elm/* packages start in an HOT mode: Evan’s actively working on them / monitoring their usage, batching issues/PRs and possibly drafting a new API, etc…

  • after Evan indicates he’s moved on to something else or after some time period (eg. 6 months since last Evan’s version), the package switches to a COLD mode (forgive me the simulated annealing metaphor :grinning_face_with_smiling_eyes:) - the maintenance burden is shared by Evan with a set of maintainers picked (trusted) by Evan; the maintainers vote on PRs and make patch/minor semver commits, while major version bumps would need Evan’s approval.

  • Evan can move a package from COLD mode to HOT mode at will

  • Evan has a veto power for any PR

I have a feeling something like this might let non-controversial improvements through without Evan needing to spend time maintaining all the libs and react to all the PRs and issues. So, passing the ball to Evan’s side of the court:

Evan, what are you generally thinking about the “small fixes” maintenance need for elm/core etc. repositories?

Could you please share whether you considered some other-than-batching maintenance model (perhaps in later stages of a project) and what criteria would such a model have to pass? Also, any thoughts on delegation? Generally, how to move this discussion forwards?

36 Likes
2

+💯 I wrote this recently somewhere else:

What I miss sometimes is an Elm janitor. Someone who make 0 language and API changes, but fixes bugs and rough edges.

2 Likes
3

Maybe Evan will like this idea and get involved, who knows. But judging by previous statements from him, I get the impression he sets his own agenda.

I think you need a new Proposal that works completely independantly and asynchronously to Evan. That seems to be the reality and any wish for it to be otherwise just leads to a dead end.

5 Likes
4

Right. I think that agrees with my point of needing to hear the viewpoint of the current maintainer (Evan).

Until we know his thoughts on this, we can only guess whether he wants to maintain control / be a part of the process, or whether he’d like to pass the maintenance onto some team, or keep the status quo etc.

5

My assumption is that he wants to maintain control, but not be a part of any community process.

3 Likes
6

Hello, long time lurker here.

I think the hot/cold approach sounds nice, but also requires a lot of thought by Evan before it could be applied by him, and it would require him to continuously invest time adding/removing access to elm/* repos.

So I wanted to post this proposal that could “[work] completely independently and asynchronously to Evan”.

PROPOSAL 2: Fork core official packages.

The organization elm-explorations (description: “Packages that may one day be in @elm-lang”) is allowed publish Elm packages with kernel code written in JS. It has no public member list, but I assume that Evan already trusts the members there, so no new list of “maintainers” or “janitors” would need to be gathered from scratch - it could be extended though.

Use cases (as listed in the first post by @janiczek)

  1. If bug fixes do not (yet) land in an offical package, it could be forked and use the official version in its name.
    E.g. elm-explorations/parser-1.1.0-fixes or elm-community/elm-parser-1.1-fixes.
    Confusing might be that such a package would always start with a version number 1.0.0 unrelated to the official one (and also stay at this major version number). But the name of the package should make clear which official package it improves.
    Whenever the official package changes, existing forks must be marked as deprecated and the repository is archived.
    If the ofificial package changes in another way and introduces a new version number without the fixes, the fixes could be applied again to a new fork. This would only require work by the community, and not from Evan.
    If it is decided that the patched community fork should introduce a changed API, then I think the existing fork should be archived and the second use case should be followed.

  2. For explorations, a fork of the official package can be created.
    Something like the optimizations to elm/core as mentioned in the first post could be published as elm-explorations/elm-core-1.0.5-performance-dict-approach1 or under an unrelated name like stylish-elephants. The latter would decrease visiblity of the package and make it harder to find, but maybe that is also a good thing - at least it was a good idea for elm-ui.

In both cases, a maintainer could create a PR against the official repository and Evan could decide if he wants to merge it. And there would (hopefully) be no pressure to look into it soon and no constant nagging.
Also the code could already be used by community members and prove its usefulness, which might reduce the time Evan needs to decide if a change is a good idea for the official package, too.
If Evan should be no longer interested in continuing work on an official package, that one could be marked as deprecated and direct users towards the community/exploration fork instead.

The biggest threat that I see with this proposal is fragmentation and being confusing to new Elm users. But I think at least the risk of fragmentation could be reduced with documentation and discipline and quickly (and reliably) marking the community forks as deprecated.

Another thing: I think it would be great if Evan would allow the organization elm-community (description: “Unofficial group for shared work on Elm packages and documentation”) to also publish packages with JS kernel code.
Then bugfixes for official packages could be published there and elm-explorations could be used solely for cool new experiments.

The manifesto of elm-community already says
“It sometimes happens that packages which are widely used need a bit of maintenance […] and other work can be blocked until the maintenance is performed. […] elm-community can provide a home for a fork, and the necessary work to maintain the package.”

I assume it might be hard for Evan to give someone extensive rights inside the elm organization for a huge number of good reasons. But the entry barrier to elm-exploration and elm-community might be lower - especially as they are not advertized as being part of the official language or an officially-sanctioned package.

Edit: I used the word “core” misleadingly before, I now changed it to “official” to indicate that I don’t only mean the elm/core package, but elm/* in general. Also I now always use “package” instead of “library”.

3 Likes
7

In order to fork core packages, which is a must, the elm compiler needs to be forked first. And Evan wants that to happen. He isn’t against forking. He think it might improve the state of things.

8

Do you know if there is any elm/compiler community fork and what it offers?

9

And Evan wants that to happen. He isn’t against forking.

I’m sorry, but I could use some further explanation on this claim. I have been following the Elm scene for the last four years (checking out the forums, watching Evan’s talks, reading the few articles around, experimenting and using Elm for some small projects) and I can’t say I get that impression.

Beside one comment - that I’m too lazy to find and link - were Evan says that he wouldn’t be opposed to a fork as long as it used a different name (as stated by the BSD-3 license), there hasn’t been much “official” talk on the topic. On the other hand, it seems to me that actions and policy around the Elm community give the exact opposite message.

For one, the development process is completely closed to a restrict circle of developers. It’s hard to create a fork when no one is allowed to contribute to - and therefore has experience with - the original project.

The decision to lock some language features behind package DRM goes in this direction as well: powerful tools like native modules would help in experimenting with the limits of the language before resorting to a compiler fork. I find this issue (int parser fails on trailing decimal separator · Issue #14 · elm/parser · GitHub) for elm/parser exemplary of this: any community made number parsing package (created to fix a couple of ignored issues) would always be a second hand alternative due to custom operators being a privilege for inner Elm developers.

All of this raises the bar in the effort required to go through with it and dissuades potential maintainers.

Finally I can quote a single attempt (or rather, the serious discussion of) to fork the compiler that followed version 0.19: Support for elm 0.19 · Issue #62 · gdotdesign/elm-github-install · GitHub
It is a discussion of a possible fork that is abruptly shut down by a prominent figure in the Elm community, basically trying to bully these developers into desisting.
Now, there is no evidence that the interference actually hindered the effort and I don’t think the person in question is officially affiliated with Elm or Evan, but to the best of my knowledge the latter has remained silent about that whole exchange. I find hard to believe that he did not know about it and it would have been a great occasion to make explicit his favorable opinion on a fork.

I’m sorry if I sound harsh, but given my point of view I have an hard time believing Evan would look positively at a fork.

3 Likes
10

@Maldus512 I’m fairly certain Lamdera is a fork and a commercial product, which contradicts the idea that Evan is against forks.

Additionally, can we keep this focused on maintenance and not theories of what Evan may or may not think? This thread would benefit from staying on topic and not speculating about the intentions of others.

11 Likes
11

So in summary:

  1. Having a process for fixes and improvements in the core library would require some involvement from Evan → But we as a community should not depend on him
  2. Forking the core libraries would make us independent of Evan → But this requires forking the compiler
  3. Forking the compiler is hard to do, and past attempts were shut down.

So to continue the discussion:

I’d like to help forking the compiler. In my eyes there isn’t a lot that needs to be changed:

2 Likes
12

I’d still like this thread to wait for Evan’s response to:

Evan, what are you generally thinking about the “small fixes” maintenance need for elm/core etc. repositories?

Could you please share whether you considered some other-than-batching maintenance model (perhaps in later stages of a project) and what criteria would such a model have to pass? Also, any thoughts on delegation? Generally, how to move this discussion forwards?

2 Likes
13

Hi everyone!

[have] you considered some other-than-batching maintenance model

He most definitely has, and the current strategy is a deliberate choice. Regarding delegation, then he touches upon the topic in his most recent talk The Hard Parts of Open Source. I recommend watching it if you’d like to understand his position better!

P.S. I think a fork would be great! Given that it is not the first time it has been suggested, it seems at least a few more people would be interested in exploring a more distributed governance style :blush: As already mentioned, just remember to change the name and logo!

11 Likes
14

Indeed, this fork option has been discussed a huge amount of times (including by you @terezka if I remember well) but never actually shown up.

My interpretation of this is that maintaining a whole language/platform/community is a huge work, asking for a lot of investment (and I hope this previous sentence will be read as a recognition of the amazing Evan’s work quality). One have to considerably want improving/changing the language to see enough benefits counterweighting this investment (like Mario did with Lamdera).

I read Martin’s request as “How could we help fixing this little int bugs in elm/parser reported 3 years ago and fulfill this “TODO” in deadEndsToString?”. I hear the will to “batch pull requests” but here this really are bugs which really give a bad image (“What? You want using a language which has a stupid known bug in its std lib for 3 years and didn’t fix it?”). Forking the compiler to just fix those little bugs clearly doesn’t outweigh the investment needed.

12 Likes
15

I wrote a spec for this before starting work on it, and this still reflects pretty well what I am trying to achieve in the first version - see the use cases listed at the bottom: Elm Package Server · GitHub

In the interests of being as uncontroversial as possible for the first release I also wrote:

  • Confirm the package does not contain ports.
  • Confirm the package does not contain kernel code.

I do also think, if I get to V1, that I would be interested in being able to publish patch releases to core packages. I have actually patched and produced a couple of PRs for core packages partly as I am intersted in exploring how this might work in practice. My process is to fork the package at its latest published tag, create an issue describing what is being fixed, linking to related issues in the core packages, and tracking its release status and make a clean patch. That is, to keep things as transparent and maintainable as possible, and to always feed back improvements to the upstream projects - whether or not they are ever accepted there.

I think great care needs to be taken when deciding what to patch. Anything that causes a runtime exception (apart from Debug.todo obviously) is fair game. Issues like this one - int parser fails on trailing decimal separator · Issue #14 · elm/parser · GitHub - I am not so sure about. If the patch never makes it back into Elm, then this introduces divergent behaviour in the alternative system, which could significantly affect program behaviour. There is also a simple workaround, to write your own int parser. So maybe some guidelines on what is suitable for patching need to be written up.

I don’t really see this as a fork of Elm, I would instead call it a release branch, where the Elm Janitor tidying up work goes. I have no desire at all to fork the language, its really a great language and I could not do better.

Some other related thoughts:

  • At some point beyond V1 I can imagine having a package repo that allows kernel code, this might be called the ‘unsafe’ repo. The domain tags concept in my spec would be very useful here as the ‘unsafe’ domain could be tagged and excluded (would be excluded by default, you would have to opt-in).

  • I have never yet run into a show-stopper bug in Elms core or compiler. Probably the closest example was the recent Safari for ... in bug, which I note Evan did get involved in the discussion of. As it affects more than just Elm, it seems like Safari should really own that one. The quality is generally very high.

  • I am interested in using Elm outside of web development because I think it has real potential as a cloud language. Also motiviation for the domain tags concept, since the browser domain is not needed there. This is analogous to the platforms concept in Roc that a few people have mentioned recently.

  • I like the name Eco, but not so keen on the ‘pro’ bit, although it does reflect my intention that it is for supporting professional work. Eco was already taken, so I had to add something to it, better suggestions very welcome.

2 Likes
16

If Elm core packages are patched, can they be reliably installed and used with elm-git-install? I have never tried this with a core package, but if it works it provides a way of patching them without forking the compiler. Could be a path of least resistance to move this forward in the short term.

1 Like
17

Could be a path of least resistance to move this forward in the short term.

Yes and I really think it is unfortunate. One of elm’s strength is having super nice tooling and easy setup. If we start having those kind of instructions floating around (“oh just add this third party tool, setup this unofficial package”), elm will become way less friendly.

13 Likes
18

FWIW @terezka’s reply effectively answers the original post’s questions:

Looks like there are roughly three options:

  1. keep the status quo
  2. fork the language and use a different maintenance model there
  3. make tooling to easily apply patches to packages or use forked packages

To me the 2nd option looks very time-consuming; the 3rd option has a proof of concept done by @pdamoc: GitHub - pdamoc/elm-pkg-patch: Simple example on how to patch elm packages

7 Likes
19

I’m curious – are there community members who are familiar with other languages and how they solve the core maintenance problem? It seems like a hard problem in general. If anyone can offer a bit of compare and contrast, that might be useful as reference.

20

I think even before going to 3, it is worth assessing what bugs do we want to fix? Building a tool will help to automate this, but I would like to know, what is the demand for the automation and organizational effort? I ask, because as I say, I have not yet run into a showstopper bug with Elm, only minor annoyances. Of course, it is valuable to fix stuff that is obviously fixable, especially where someone has taken the time to make a good PR for it.

So I created a google sheet to collect information on what are the essential fixes that people want?

Just add the description and github link of the issues that are bothering you, or are obviously broken things you can fix, or things that are fixed and have good PRs that should really be merged.

I am happy to do some triaging on this list, if others help make it. The triaging will be to go through the list and look at all the github issues, and fill in some additional fields like - is the issue still current, is it a runtime exception fix, is it a behaviour change but not an API change, is it an API addition, is it an API change, is there a PR, and so on.

5 Likes