Afterthoughts around about Kernel code and custom operators complaints

Request Feedback
#54

Strongly agree.

The idea of it being called ‘explorations’ in my mind suggests that it would be perfectly ok for experimental, even bad, ideas to be tried out there. It should be a proving ground to shape up good APIs that could become part of the kernel and the barrier to entry should be knowable and achievable. Sad to see that Dan is keen, has the energy and know-how but somehow cannot get there.

===

Bad ideas in elm-explorations should be killed off. elm-explorations/markdown is very easy to break and make it generate runtime exceptions. It was useful to get markdown started, but now we have much better pure Elm solutions. I think it only hangs on because it is used for the package site.

On the other hand, has elm-explorations/webgl not worked out as an experiment? Should it not be going the other way and moving from explorations into the core?

9 Likes
#55

I totally agree, a clear route to getting a project in elm-explorations would go a long way to dealing with people’s uneasiness around privileged access to kernel code. Maybe there could be "expert panels” for particular themes (video, storage etc) that review proposed APIs and implementation plans for modules, and lightly filter obviously bad ideas, letting the rest through, maybe with some feedback/guidance. The most important thing is that the process of both selecting the panel and reviewing module proposals should be as transparent possible.

5 Likes
#56

That’s how Linus Torvalds operates, isn’t it? Appears to have worked well for the Linux kernel over the years.

A small hierarchy of trusted decision makers to filter, feedback, and help improve on ideas coming up from the community, with the core team making the final decisions of what makes it in or not.

4 Likes
#57

Maybe a different way of framing it would also be useful: Elm itself is designed for user success, or making it easy to do the right thing, and hard to do the wrong thing.

Is this a way of thinking that could be applied to working on Elm too? Because right now it seems that the process of trying to contribute to Elm actually sets people up for failure. elm-explorations is one place where we’ve seen this now, but I think it’s the same problem that occurs when someone creates a PR or an issue that doesn’t get a timely or useful response, like in this thread.

I realize things are this way for a reason, and I’m not suggesting any change here is easy or simple. But maybe just asking “How can we set the user up for success both when using Elm as a tool, but also when interacting with the process around Elm?” is helpful.

3 Likes
#58

I think the reason things are the way they are is likely to be lack of time on the part of our BDFL. It would take time and effort to set up a process for these Elm explorations to be coordinated, managed and evaluated.

Here is a suggestion of how I think we could set up our own Elm explorations that functions better and opens the door to contribution.

  1. Fork the compiler and rename it as requested above. I sort of like the name ‘Elm Pro’ - Elm but with a more professional attitude (not to imply that people here don’t have a profession attitude, I guess I mean an attitude that is better aligned to professional needs and adoption). Perhaps that is too close to the original name.

  2. Clone and modify the package repo to create a brand new empty one. This one will accept kernel code, so long as it is from the ‘elm-experiments’ or ‘elm-pro’ namespace (or some other names) on github.

  3. Modify compiler so it extends the moratorium on kernel code to ‘elm-experiments’ and ‘elm-pro’ namespaces. To use an ‘elm-experiment’ a flag must be passed to the compiler --experimental. The reason for this, is so that users must opt in and know that they are using experimental code - there is no come-back to complain if an experiment is abandoned or removed. Perhaps the version numbers on experiments could even be kept under 1.0.0.

  4. Modify compiler so it will also install packages from a new primary repo described in 2. If a package is not found in 2, it will fall-back to package.elm-lang.org. In fact, 2 will probably set itself up to mirror package.elm-lang.org, rather than redirect to it.

  5. The new package repo will not accept any non-kernel packages, unless they depend on kernel packages that already exist in it. This is so these pure Elm packages find their proper home in package.elm-lang.org.

  6. Create a written set of rules for a new project to be started in elm-experiements. Set expectations and a minimum barrier to entry but the aim would be to be quite open to new ideas.

  7. Decide and write down the rules for moving a project from elm-experiments to elm-pro once it has reached sufficient maturity and quality. The quality gate needs to be stronger here obviously than for creating an experiment. The process needs to be transparent and fair.

The forked compiler could find a new home somewhere like apache.org. I don’t really know if that is the right place, its just somewhere grown up that knows how to manage open source communities. You want somewhere that can bring some impartial supervision to a project - so there is somewhere to go if you need to complain, get advice on how to manage things, etc. Somewhere with some standards, like forcing a project to have a written set of rules and guidelines to contributing and so on. Anyway, maybe Apache is not possible because of its licence, does it clash with the existing Elm compiler licence?

The forked eco-system would have a stated aim that all new kernel based API functions must be pure functions. Exposing non-pure functions is grounds for a package to be removed, or rolled-back to its last pure version.

There is an opportunity for a forked compiler to get some patches applied to it to fix long-standing bugs. This is a slightly risky thing in a set up like this, as a package may depend on a compiler bug. I think some odd things around how arithmetic behaves in Elm should be left alone for example. But the recently referenced bug around the debugger overflowing the stack is an example of something that could be safely fixed (would need to fork and patch the elm/virtual-dom package to fix that one). You want to fix bugs but without changing the intended behaviour - its probably best to be fairly conservative about this, at least to begin with.

The closer the forked compiler remains to the original - the better. Could continue to track all new releases of it that way. The forked version would also build original version code, so no need to keep switching between them. You could even symlink it to elm on your system.

===

I don’t know about you, but since the coronavirus started, I’m down to about 2 days a weeks computer time due to home commitments. I also have done the due diligence on my current project and satisfied myself that the existing Elm eco-system can do everything I need it to. So I don’t have the time or energy to do something like this.

Also note, this is not meant as a 2 fingers to anybody or some kind of hostile attempt to undermine things. The idea popped into my head so I just want to put it out there and see what people think - in a grown up way please.

I also think the steps I outlined above are a minimum effort plan to getting a working elm-explorations and as a lazy engineer my brain always likes to throw up the MVP. It really is not going to do any good complaining about not having a working elm-explorations - its seems pretty clear that will not change anything.

7 Likes
#59

@rupert Thank you for writing this out!

If this can be pulled off in a generally “approved” manner, I really think a community-driven fork for experiments would benefit upstream as well, as long as the goal really is and stays to be a research project.

  • This fork, being generally advertised as non-stable, can test and review patches to the compiler or core packages much more easily, since the (production-readiness) guarantees are a lot weaker.

  • People without any other immediate option will have a clear path to unblock themselves, as long as they are aware that they are entering unstable, unsupported territory - by downloading a seperate binary, reconfiguring all the tools, passing some additional --unsafe-beware-dragons-here flags.

  • Feedback on experiments would not rely on the core team reviewing and approving things anymore. The Elm core team does an amazing job, but only has limited resources themselves. There are probably more people sharing similar experiences to @Dan_Abrams. I’d rather collect all such projects and ideas in a central place for the community to try and discuss them, instead of having them end in a single “Proposal”-style Discourse post, where the OP will get demoralized by technical and process-related anwers of why “things don’t work like this here.” This will also keep those folks motivated to work on these projects, and with Elm in general.

  • The core team does not have to look into everything anymore, but can focus on the much smaller output of that project - the community can iterate and test different designs independently and much more quickly, meaning the core Elm team only has to look at experiments that worked. Proposals and ideas will be essentially filtered by the community itself.

  • Discussions like these (+ the original medium post) would have been resolved by a posting link.

I’d really like to focus on how to handle this as a community though. I’m not certain that clear communication from the fork will be enough to ensure not fully splitting off from the main project. How will we deal with people just using the “Pro” version? This will definitely happen, especially if answers to the common questions people encounter while learning Elm (localStorage, WebSockets, that-jQuery-Fancybox-Plugin-they-have-to-use-at-work integration) become “you could do this the official way, by manually building all this elaborate wiring where there’s a chance you might not understand half the code and concepts if you just started learning, or btw this community fork has a package for it that might work, just need to pass that extra flag, don’t worry about the details”

Can we prevent the official channels (Slack, Discourse) from having to deal with the problems of “pro” users unrelated to “core” Elm? Would this be a problem for the Discourse? On Slack, making and referring to a separate channel sounds reasonable. A more clearly separated branding would certainly help, but it also increases the possibility of separating the community as well.

How could something like this be organized?

PS: It may be a good idea to split this discussion into a new thread.

3 Likes
#60

I like the idea of community based experiments that can feed back into core elm. I am concerned about it becoming a complete fork that’s just Elm + native access. Would it make sense to place restrictions on the forked compiler? For example, only allow the use of elm/* and elm-explorations/* packages. This should allow freedom to experiment while hindering the use of the fork for anything production.

If this is a fork for the purpose of experimenting/exploring, then I wouldn’t see this as something for “there’s this one bug and I want it fixed now” scenarios. Make it easy to test ideas and hard to build production apps.

2 Likes
#61

Why do you think core team would want to merge forked compiler experiments if that does not happen now?

3 Likes
#62

That is what I was proposing, but no good to limit to just those 2. It would obviously need to include some new community controlled namespace in order to be able to commit code to somewhere that the compiler can build.

Make it easy to test ideas.

Make it possible to build production applications with less risk of hitting technical blockers - in other words put the power to fix blockers into the hands of people building the application.

I was not thinking it would just be a research project. The ‘experiments’ part of it would be, and would exist as a way of letting kernel code in but with a fairly low barrier to entry - an inclusive way to experiment. The ‘pro’ part of it would not be just for research, its intention would be for production code and to provide a way for people that cannot run Elm in production due to some bug or perceived closedness of its development process that make it too risky for them to adopt.

It should be ok to just use the pro version - so long as its compiler does not diverge from the official one. On pure Elm code, it should yield the exact same result. Personally, I would probably take care to use the right compiler for the situation though.

The hacky, and bad, way of doing things might be to use an experiment (that has a version number less than 1.0.0. Those would be clearly labelled in the package repo as ‘Experimental - Use at your own risk’.

Anything that made it through to the pro repository would have passed some quality gate, and is ready for production use - there might be an active group of people or businesses supporting it. The kinds of things I imagine you would find in the pro repository would be Dan’s video package; so only people working with video would be interested.

Not really sure how that would work. Maybe it needs a completely different name, and its own comms mechanism to avoid confusion. I guess it depends on whoever runs those channels and what their views are.

I don’t think they would. But I forgot to mention in my first post on this - I think any compiler/core fixes should still get PRed back, it seems only courteous.

1 Like
#63

Just one note. I don’t think any fork should include elm in a name. I’m kind of fan of interpreting this Clause in BSD3 quite strictly:

3. Neither the name of the <organization> nor the
   names of its contributors may be used to endorse or promote products
   derived from this software without specific prior written permission.

I think it protects original authors from being tight to stuff they don’t approve of, what ever it might be. I would suggest picking up some name of the tree for name of fork will it happen. That way it would be still related but less obviously. It can be ash or oak for instance. Also if you want to be super creative there seems to be thing called Camperdownii (which would be arguably hard to type though).

Disclaimer: I personally not dissatisfied enough to switch to fork so take it as just personal opinion / suggestion.

4 Likes
Clients expressing doubt about choosing Elm
#64

I don’t think they would. But I forgot to mention in my first post on this - I think any compiler/core fixes should still get PRed back, it seems only courteous.

changes in official elm/compiler happens in infrequent large chunks/batches. Be prepared this won’t be an easy job.

#65

The way I read the situation, I think there are two distinct groups/use-cases that are dissatisfied with how things are1. I think it would be helpful to be clear in discussions which of these one is addressing, as they have quite different needs:

  1. People who are using Elm for work and are worrying about getting blocked. They
    • are building applications
    • are highly time sensitive (i.e. waiting six months for maybe a solution isn’t an option for them)
    • would rather have a good enough solution now than a perfect solution later
    • don’t particularly care about long term concerns like future portability
    • care about smooth tooling (it is not sufficient that just the compiler works, it needs to work with CI systems, JS bundlers, testing toolkits, code generators, etc – this makes low effort forks problematic for them)
  2. People who are trying to contribute to Elm in some way. They
    • are building packages or fixing the compiler or tooling
    • care about making a high quality contribution (even if this takes somewhat longer)
    • care about affecting others (i.e. they are not just trying to fix their own code, they want to enable others to achieve things)

For example the ability to compile Kernel code is only useful to group 1, whereas to group 2 it would only be useful if it was also possible to publish packages containing Kernel code2.

I believe that the main counter-concern is, that we all want to maintain the high quality of Elm and the benefits that brings.

  • For the first group, it has been repeatedly reported that after the ban of native, these people were forced to figure out ports and custom elements and they were often happier at how their code turned out3. That suggests that when the ability was there, it was fairly often used for things where it was not essential but rather as a path of least resistance.

  • For the second group, the issue is that packages, once published, are somewhat immortal. Once people start depending on them, it is difficult to remove them. We have heard people complaining about Websockets going away for years now (despite that being a use case remarkably well suited to ports). This creates a lot of pressure on getting these foundational packages right.

A wacky idea for group 1

I wonder if some of these things couldn’t be balanced with adjusting the incentives by adding artificial friction. Perhaps only allowing to compile Kernel code without --optimize like the Debug modules work now and adding a 5s sleep call for each non elm or elm-explorations Kernel module compiled would be sufficient that people would only use this after trying to solve their problem every other way. Other disincentives to consider:

  • Printing a humongous and nasty warning on every compile. (Elm Foundation warns: Using Kernel code may lead to lung cancer! :wink: )
  • Only allowing Kernel code compilation with a crypto key. One would need to print, sign and fax a long form to the Elm Foundation to acquire one.
  • Failing compilation randomly with a 15% probability.

A more reasonable idea for group 2

Personally I think the current guidelines on how to contribute to Elm are simply not good enough. Perhaps simply clearly stating something like:

Contributions to Elm are limited to the core team. The core team is based around personal relationships and we do not accept strangers from the internet. Reporting bugs is encouraged, but due to our process, please don’t expect a response.

Would be more fair in terms of expectation setting. There are many open source projects were code changes are limited to some group of people. But something like this would at least set people up with expectations that wouldn’t be frustrated. Any violations of the above would simply be happy exceptions to the rule. Whereas the current talk about how contribution works in Elm (talk, do research, coordinate, then code, …) makes people believe that as long as they follow that somewhat vague process, they are likely to succeed.

1 Of course this is a simplification of reality. Many application developers would also like to reuse/share solutions to common problems for example. But I am here focusing on concerns that become extreme enough that people feel like they need to exit the community (often with a bang).

2 With the small caveat that it would at least make testing Kernel code based patches somewhat easier and perhaps enable some hacky ways to get around the package system.

3 Of course there is massive survivorship bias here. Many were not and churned away - in some cases even building tooling to transform their code into a different language.

13 Likes
#66

Quick name idea to throw out there: Morel. Morel mushrooms are delicacies that grow on Elm wood.

I like the idea of a pathway for community experimentation to be able to open up like this without impeding the core team in their goals. Clearly there are some warts to work out on all this, but I think this is a good discussion to be having.

#67

I’d love a compiler which does all of these things :joy: I think those ideas would generate just enough WTF feelings for users to stay away from Kernel code :grin:

#68

I think the problem with those things is that they all smack of treating the user like they are stupid - and from reading comments on Reddit and Hacker News it would seem that people do feel that Elm treats them that way. None of these things are going to go down well with businesses considering adopting Elm - “So if we hit a technical blocker and need some kernel code, what are our options? Ohhh…”.

2 Likes
#69

@gampleman On a more serious note. I think group 2 is really important. I would still like to prevent making the use case for group 1 too easy.
Here’s my reasoning, I think that explorative activity that is community driven would be really helpful in terms of exploring APIs, fixes, browser features, etc., since that part of community experimentation is currently sorely missing.

I also like the idea of printing the warning you suggested about how “Contributions to Elm are limited to the core team.”

#70

Most of these are a joke.

But I think the idea itself isn’t completely hopeless. Limiting these away from --optimize mode and printing a warning explaining why Kernel code is a bad choice and what the alternatives are might just be good enough. Making it compile slowly (whether that’s a sleep call or just some expensive computation that is intentionally not optimised) could help as well. That might just strike the right balance of not being the path of least resistance, but being there when it is truly needed.

In fact I believe that Group 1 likely doesn’t need Kernel code in the majority of cases. It’s more the fear of needing it in the future and suddenly running out of options that scares them.

#71

These are technical solutions to an adaptive challenge. They are simply the wrong class of solutions.

The challenge with Elm’s private API is not technical. It is not a matter of safety or technical complexity of the code but a matter of values, a matter of beliefs and a matter of approach.

I would be less surprised by Evan giving up programming and switching careers to become a political analyst than by him implementing the kind of technical solutions that I’ve seen suggested when this kind of discussions come up.

I believe that for the foreseeable future, Elm will have a single author that will work on it on his own terms. Some people will accept this and stay. Some will not accept it, they will get progressively more frustrated and at one point they will leave.

1 Like
#72

While I agree with the general thrust of your post, I would not call these technical solutions, as they are chiefly psychological: they are designed to address a psychological problem (fear of future problems) by psychological means (changing the incentive balance by various nudges).

In other words, my hypothesis runs like this:

  1. A certain group of people (I call them group 1) are worried that they will unexpectedly1 need the full flexibility of Kernel code in the future and they will face unpleasant circumstances in their workplace because they will not be able to deliver.
  2. However, when this flexibility used to exist, despite being discouraged in the community, it was used in a number of situations where it was not necessary and better solutions existed. This is undesirable for everyone involved and part of the reason for the current restrictions. The main explanation is that it seemed easier than the alternatives.
  3. Suitably changing the incentive structure may allow both of these goals to be accommodated.

1 Unexpected, since if you are expecting upfront to need capabilities for which Elm is currently unsuitable, then according to the advice which is now even in the official guide, you should just pick a different technology for the project.

On a more personal note from running an Elm project in a commercial setting where the powers that be had a very begrudging approval of using Elm it had some emotional rollercoasters for me: The joy of how much easier most things were compared to the React stack we were using for other projects combined with the fear of being asked to do something for which there was no pre-existing library and suddenly I would spend 4 times as long working out the best way to hack something together. Generally these fears mostly didn’t materialise, but I can empathise with folks who feel this a lot stronger in less secure situations.

6 Likes
#73

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.