Afterthoughts around about Kernel code and custom operators complaints

HTML elements are not stateful, DOM is. Bindings provided by VirtualDom are not stateful even though they might result in producing stateful DOM (which I’ve described in post above precisely because I expected reaction like this). With any vdom implementation, a lot of effort goes into tuning diffing algorithm in a way that these states won’t mess up on user.

Example 1: Global State

Now have a look at this https://ellie-app.com/8DhgqBYnXvfa1 and try to increment and decrement the counter. This shows how purity can be broken with web component. Call to pure function Html.node "x-state" mutates global state every time diffing algorithm adds this node into the DOM.

Example 2: Node State Loss

Survival of local state of a component depends on diffing algorithm. This means that if diffing algorithm decides that it will remove the node and insert the new instance, the state of the component is lost.
This is demonstrated by this example https://ellie-app.com/8DhvpDkLdMKa1.
Even inputs and other things suffer from this issue. Anyway I don’t think it’s wise to make this issue in your app even worse by adding tons of additional state using web-components api.

2 Likes

Yes, I could see that working. I also think that tool would work with ports too - install an Elm package giving the port type, and an npm package giving the JS side of the port. Obviously the application developer still has to declare the actual port, just as they still have to use the web component.

Perhaps it could work by having an additional config file next to elm.json - say elm-ext.json. And define a package format for these Elm extensions that specified how Elm and JS code is to be layed out in a package, basically an intersection of the Elm and npm package formats. There could then be a new package site that holds meta-data for these extended packages, and the tool to install them.

I’m sorry, but some built-in HTML elements ARE stateful—that is they update their state without any user interaction. You don’t need to create a web component to have this issue, and a web component can solve it.

<audio src=“whatever.mp3” autoplay muted></audio> is a stateful and changing element.

In the virtual dom, this element would be created as a dom node, and without any intervention by either the user or the developer, properties on it would change regularly.

There are built-in elements in HTML5 that are designed explicitly to work with function calls to update state, particularly audio and video, but also canvas. How would one create a video conferencing tool or a YouTube clone without interacting with stateful elements in some way that involves either a web component or reaching into the dom via some sort of query?

Declarative is very nice. Not all Built-in html elements can be used appropriately in a purely declarative way. Web components are one way we can build a declarative api around these cases, and I would argue a better one than document.querySelector(), given the current state of the virtualDOM and the language.

UPDATE: @turboMaCk you talk about state flags like ::focus, my point above that there are other cases like focus in built-in elements that have not been wrapped by Elm natively.

I’m sorry, but some built-in HTML elements ARE stateful

I guess the part of problem is terminology. What I mean by html element is declarative HTML markup <audio src=...>. The stateful part is DOM node which is created after this HTML is parsed and translated to DOM. Anyway that’s just to clarify something I said above.

Yes indeed audio is one of the elements I didn’t even thought about but it’s really nice example. As I said above even input has this hidden state. For instance inputs loose focus and position of text cursor in situation demonstrated in example 2 above. But it’s a good point that audio is even more problematic. For input we at least have Dom.focus to at least restore the focus (we can’t restore position as far as I know though). I guess audio node would in example above also be broken.

You’re definitely right that these binding exposed by Html module are as problematic as web components are. Thanks for the example :+1:

1 Like

So I used to make my living working with audio and video tags, and to the small extent I’ve done frontend development in the past year it’s made heavy use of them.

It’s a daily reminder for me, and has been for four years, that these elements are broken in Elm.

If the answer is, well you shouldn’t use Elm, use Reason or PureScript instead, because these elements are broken, then, to answer the original question, this bothers me. It makes me feel excluded, it makes me feel unvalued by Elm’s design, and it’s frustrating and at times depressing.

When I’m also told don’t use one of the two, fairly limited outlets provided by the language because it violates purity, it frustrates me tremendously.

It’s great that Elm goes for this purity. It’s part of the magic of the language. But there are parts of the underlying platform where some impurity is required, like the dom, like focus and blur, like WebGL, where either the core team has come up with a solution or allowed third-party to via an elm-exploration.

Elm-explorations was supposed to be a solution to this very problem, but as far as I can tell, the only elm-explorations are the ones that were in the initial batch almost two years ago. And some of them are great: people are creating incredible abstractions on top of Elm-WebGL that make our whole community richer. That’s not realistically going to happen with ports or Web Component solutions.

Maybe this is becoming my own “Why I’m leaving Elm letter,” I don’t know. After four years of banging my head on this, trying to get Evan’s attention, trying to contribute a good design to help others, using the alternatives we have in my own work, I really am fed up. I love this community and my friends within it, and I appreciate the beauty of Elm and Evan’s vision for it, but it’s depressing, frustrating, annoying and painful, and it feels like some Browser APIs are just more important here than others, and ergo, some people’s work is more important than others.

Some of us are frustrated not because we think it should be impure, not because we want to wrap some third party library or implement jQuery, but because we have APIs we use, we care about, and we are expert in and which are going scant attention by Elm, while we watch other APIs, like file, bytes, webGL get implemented.

6 Likes

TLDR I’ve been in this community for four years and witnessed the same argument happen over and over.

I think the reason is more complex than there are bad actors muckraking. It stems from the good design decisions which necessary exclude some things.

Some of us have APIs we care about that don’t have native wrappers and we wish they did, so that abstractions could be built and distributed on top of them. Some of us have the APIs we care about wrapped already and therefore don’t fully understand the frustration of those who don’t.

It’s not a gap in expectations, it’s a gap in inclusion.

We don’t really have any communications outlet to resolve this gulf, and elm-explorations, which was supposed to provide an outlet for us to experiment with these APIs but the barrier to entry there has been enormous. Without some sort of outlet, some people get very upset.

Even if I disagree with the articles arguments and his need to write it, I sympathize with the pain he felt, because I’ve felt it too, and it does seem like that pain is not fully felt by everyone in the community, which is isolating.

6 Likes

that these elements are broken in Elm.

I agree. They’re broken in my opinion too.

shouldn’t use Elm, use Reason or PureScript instead

Well I think if you really need to work with similar things a lot I think you might be better off using one of these languages instead. At least I would use them myself. I don’t mean to exclude you though. In my view at least I’m trying to do quite opposite. I’m afraid to recommend Elm to anyone with use-case in which it might be painful to use. That leads to frustration a potential conflicts which will lead to feeling even more excluded and therefore frustrated (see the recursion). I understand some people need to deal with these problems and I want them to succeed.

Elm-explorations was supposed to be a solution to this very problem, but as far as I can tell, the only elm-explorations are the ones that were in the initial batch almost two years ago.

I myself agree with this part of criticism. I belive elm would be better off if cmmunity would be allowed to do a bit more to help. For instance one of the motivations for locking kernel code was possible future in which elm would compile to native on mobile. Meanwhile while there is nothing like that in elm, purescript (which is now basically controled by community) already has backends to golang and C++. I think that’s empirical prove that while it is risky to hand more control to community, great things can happen as a result of that.

Maybe this is becoming my own “Why I’m leaving Elm letter,”

Even if you decide to go with other language for these project it doesn’t need to be about leaving the language. I’m for instance always looking for opportunity to find a nice project to do in other language with different set of trade-offs. I did projects in both PureScript and Reason and even GHCjs and never thought about it as about leaving the Elm.

we care about, and we are expert in and which are going scant attention by Elm

I checked your github account and have seen elm-media. I think your design is good and aligns pretty well with what I feel (without being expert myself) correct solution should look like.

For what it’s worth I think your and mine arguments are the same. Web components (and broken stuff from elm/html) are not optimal solution and have own problems. I also think it would be better to make elm-exporations more open and community controlled. I just didn’t express that in first post because I wanted to focus solely on explaining why I don’t consider web-components to be the optimal solution. Anyway I don’t want to feed your or anyone’s frustration. I have no influence on openness of elm-explorations and it probably deserves it’s own discussion.

I feel I’ve said everything I wanted. I’m switching back to the role of silent observer.
image

6 Likes

Strongly agree.

The idea of it being called ‘explorations’ in my mind suggests that it would be perfectly ok for experimental, even bad, ideas to be tried out there. It should be a proving ground to shape up good APIs that could become part of the kernel and the barrier to entry should be knowable and achievable. Sad to see that Dan is keen, has the energy and know-how but somehow cannot get there.

===

Bad ideas in elm-explorations should be killed off. elm-explorations/markdown is very easy to break and make it generate runtime exceptions. It was useful to get markdown started, but now we have much better pure Elm solutions. I think it only hangs on because it is used for the package site.

On the other hand, has elm-explorations/webgl not worked out as an experiment? Should it not be going the other way and moving from explorations into the core?

9 Likes

I totally agree, a clear route to getting a project in elm-explorations would go a long way to dealing with people’s uneasiness around privileged access to kernel code. Maybe there could be "expert panels” for particular themes (video, storage etc) that review proposed APIs and implementation plans for modules, and lightly filter obviously bad ideas, letting the rest through, maybe with some feedback/guidance. The most important thing is that the process of both selecting the panel and reviewing module proposals should be as transparent possible.

5 Likes

That’s how Linus Torvalds operates, isn’t it? Appears to have worked well for the Linux kernel over the years.

A small hierarchy of trusted decision makers to filter, feedback, and help improve on ideas coming up from the community, with the core team making the final decisions of what makes it in or not.

4 Likes

Maybe a different way of framing it would also be useful: Elm itself is designed for user success, or making it easy to do the right thing, and hard to do the wrong thing.

Is this a way of thinking that could be applied to working on Elm too? Because right now it seems that the process of trying to contribute to Elm actually sets people up for failure. elm-explorations is one place where we’ve seen this now, but I think it’s the same problem that occurs when someone creates a PR or an issue that doesn’t get a timely or useful response, like in this thread.

I realize things are this way for a reason, and I’m not suggesting any change here is easy or simple. But maybe just asking “How can we set the user up for success both when using Elm as a tool, but also when interacting with the process around Elm?” is helpful.

3 Likes

I think the reason things are the way they are is likely to be lack of time on the part of our BDFL. It would take time and effort to set up a process for these Elm explorations to be coordinated, managed and evaluated.

Here is a suggestion of how I think we could set up our own Elm explorations that functions better and opens the door to contribution.

  1. Fork the compiler and rename it as requested above. I sort of like the name ‘Elm Pro’ - Elm but with a more professional attitude (not to imply that people here don’t have a profession attitude, I guess I mean an attitude that is better aligned to professional needs and adoption). Perhaps that is too close to the original name.

  2. Clone and modify the package repo to create a brand new empty one. This one will accept kernel code, so long as it is from the ‘elm-experiments’ or ‘elm-pro’ namespace (or some other names) on github.

  3. Modify compiler so it extends the moratorium on kernel code to ‘elm-experiments’ and ‘elm-pro’ namespaces. To use an ‘elm-experiment’ a flag must be passed to the compiler --experimental. The reason for this, is so that users must opt in and know that they are using experimental code - there is no come-back to complain if an experiment is abandoned or removed. Perhaps the version numbers on experiments could even be kept under 1.0.0.

  4. Modify compiler so it will also install packages from a new primary repo described in 2. If a package is not found in 2, it will fall-back to package.elm-lang.org. In fact, 2 will probably set itself up to mirror package.elm-lang.org, rather than redirect to it.

  5. The new package repo will not accept any non-kernel packages, unless they depend on kernel packages that already exist in it. This is so these pure Elm packages find their proper home in package.elm-lang.org.

  6. Create a written set of rules for a new project to be started in elm-experiements. Set expectations and a minimum barrier to entry but the aim would be to be quite open to new ideas.

  7. Decide and write down the rules for moving a project from elm-experiments to elm-pro once it has reached sufficient maturity and quality. The quality gate needs to be stronger here obviously than for creating an experiment. The process needs to be transparent and fair.

The forked compiler could find a new home somewhere like apache.org. I don’t really know if that is the right place, its just somewhere grown up that knows how to manage open source communities. You want somewhere that can bring some impartial supervision to a project - so there is somewhere to go if you need to complain, get advice on how to manage things, etc. Somewhere with some standards, like forcing a project to have a written set of rules and guidelines to contributing and so on. Anyway, maybe Apache is not possible because of its licence, does it clash with the existing Elm compiler licence?

The forked eco-system would have a stated aim that all new kernel based API functions must be pure functions. Exposing non-pure functions is grounds for a package to be removed, or rolled-back to its last pure version.

There is an opportunity for a forked compiler to get some patches applied to it to fix long-standing bugs. This is a slightly risky thing in a set up like this, as a package may depend on a compiler bug. I think some odd things around how arithmetic behaves in Elm should be left alone for example. But the recently referenced bug around the debugger overflowing the stack is an example of something that could be safely fixed (would need to fork and patch the elm/virtual-dom package to fix that one). You want to fix bugs but without changing the intended behaviour - its probably best to be fairly conservative about this, at least to begin with.

The closer the forked compiler remains to the original - the better. Could continue to track all new releases of it that way. The forked version would also build original version code, so no need to keep switching between them. You could even symlink it to elm on your system.

===

I don’t know about you, but since the coronavirus started, I’m down to about 2 days a weeks computer time due to home commitments. I also have done the due diligence on my current project and satisfied myself that the existing Elm eco-system can do everything I need it to. So I don’t have the time or energy to do something like this.

Also note, this is not meant as a 2 fingers to anybody or some kind of hostile attempt to undermine things. The idea popped into my head so I just want to put it out there and see what people think - in a grown up way please.

I also think the steps I outlined above are a minimum effort plan to getting a working elm-explorations and as a lazy engineer my brain always likes to throw up the MVP. It really is not going to do any good complaining about not having a working elm-explorations - its seems pretty clear that will not change anything.

7 Likes

@rupert Thank you for writing this out!

If this can be pulled off in a generally “approved” manner, I really think a community-driven fork for experiments would benefit upstream as well, as long as the goal really is and stays to be a research project.

  • This fork, being generally advertised as non-stable, can test and review patches to the compiler or core packages much more easily, since the (production-readiness) guarantees are a lot weaker.

  • People without any other immediate option will have a clear path to unblock themselves, as long as they are aware that they are entering unstable, unsupported territory - by downloading a seperate binary, reconfiguring all the tools, passing some additional --unsafe-beware-dragons-here flags.

  • Feedback on experiments would not rely on the core team reviewing and approving things anymore. The Elm core team does an amazing job, but only has limited resources themselves. There are probably more people sharing similar experiences to @Dan_Abrams. I’d rather collect all such projects and ideas in a central place for the community to try and discuss them, instead of having them end in a single “Proposal”-style Discourse post, where the OP will get demoralized by technical and process-related anwers of why “things don’t work like this here.” This will also keep those folks motivated to work on these projects, and with Elm in general.

  • The core team does not have to look into everything anymore, but can focus on the much smaller output of that project - the community can iterate and test different designs independently and much more quickly, meaning the core Elm team only has to look at experiments that worked. Proposals and ideas will be essentially filtered by the community itself.

  • Discussions like these (+ the original medium post) would have been resolved by a posting link.

I’d really like to focus on how to handle this as a community though. I’m not certain that clear communication from the fork will be enough to ensure not fully splitting off from the main project. How will we deal with people just using the “Pro” version? This will definitely happen, especially if answers to the common questions people encounter while learning Elm (localStorage, WebSockets, that-jQuery-Fancybox-Plugin-they-have-to-use-at-work integration) become “you could do this the official way, by manually building all this elaborate wiring where there’s a chance you might not understand half the code and concepts if you just started learning, or btw this community fork has a package for it that might work, just need to pass that extra flag, don’t worry about the details”

Can we prevent the official channels (Slack, Discourse) from having to deal with the problems of “pro” users unrelated to “core” Elm? Would this be a problem for the Discourse? On Slack, making and referring to a separate channel sounds reasonable. A more clearly separated branding would certainly help, but it also increases the possibility of separating the community as well.

How could something like this be organized?

PS: It may be a good idea to split this discussion into a new thread.

3 Likes

I like the idea of community based experiments that can feed back into core elm. I am concerned about it becoming a complete fork that’s just Elm + native access. Would it make sense to place restrictions on the forked compiler? For example, only allow the use of elm/* and elm-explorations/* packages. This should allow freedom to experiment while hindering the use of the fork for anything production.

If this is a fork for the purpose of experimenting/exploring, then I wouldn’t see this as something for “there’s this one bug and I want it fixed now” scenarios. Make it easy to test ideas and hard to build production apps.

2 Likes

Why do you think core team would want to merge forked compiler experiments if that does not happen now?

3 Likes

That is what I was proposing, but no good to limit to just those 2. It would obviously need to include some new community controlled namespace in order to be able to commit code to somewhere that the compiler can build.

Make it easy to test ideas.

Make it possible to build production applications with less risk of hitting technical blockers - in other words put the power to fix blockers into the hands of people building the application.

I was not thinking it would just be a research project. The ‘experiments’ part of it would be, and would exist as a way of letting kernel code in but with a fairly low barrier to entry - an inclusive way to experiment. The ‘pro’ part of it would not be just for research, its intention would be for production code and to provide a way for people that cannot run Elm in production due to some bug or perceived closedness of its development process that make it too risky for them to adopt.

It should be ok to just use the pro version - so long as its compiler does not diverge from the official one. On pure Elm code, it should yield the exact same result. Personally, I would probably take care to use the right compiler for the situation though.

The hacky, and bad, way of doing things might be to use an experiment (that has a version number less than 1.0.0. Those would be clearly labelled in the package repo as ‘Experimental - Use at your own risk’.

Anything that made it through to the pro repository would have passed some quality gate, and is ready for production use - there might be an active group of people or businesses supporting it. The kinds of things I imagine you would find in the pro repository would be Dan’s video package; so only people working with video would be interested.

Not really sure how that would work. Maybe it needs a completely different name, and its own comms mechanism to avoid confusion. I guess it depends on whoever runs those channels and what their views are.

I don’t think they would. But I forgot to mention in my first post on this - I think any compiler/core fixes should still get PRed back, it seems only courteous.

1 Like

Just one note. I don’t think any fork should include elm in a name. I’m kind of fan of interpreting this Clause in BSD3 quite strictly:

3. Neither the name of the <organization> nor the
   names of its contributors may be used to endorse or promote products
   derived from this software without specific prior written permission.

I think it protects original authors from being tight to stuff they don’t approve of, what ever it might be. I would suggest picking up some name of the tree for name of fork will it happen. That way it would be still related but less obviously. It can be ash or oak for instance. Also if you want to be super creative there seems to be thing called Camperdownii (which would be arguably hard to type though).

Disclaimer: I personally not dissatisfied enough to switch to fork so take it as just personal opinion / suggestion.

4 Likes

I don’t think they would. But I forgot to mention in my first post on this - I think any compiler/core fixes should still get PRed back, it seems only courteous.

changes in official elm/compiler happens in infrequent large chunks/batches. Be prepared this won’t be an easy job.

The way I read the situation, I think there are two distinct groups/use-cases that are dissatisfied with how things are1. I think it would be helpful to be clear in discussions which of these one is addressing, as they have quite different needs:

  1. People who are using Elm for work and are worrying about getting blocked. They
    • are building applications
    • are highly time sensitive (i.e. waiting six months for maybe a solution isn’t an option for them)
    • would rather have a good enough solution now than a perfect solution later
    • don’t particularly care about long term concerns like future portability
    • care about smooth tooling (it is not sufficient that just the compiler works, it needs to work with CI systems, JS bundlers, testing toolkits, code generators, etc – this makes low effort forks problematic for them)
  2. People who are trying to contribute to Elm in some way. They
    • are building packages or fixing the compiler or tooling
    • care about making a high quality contribution (even if this takes somewhat longer)
    • care about affecting others (i.e. they are not just trying to fix their own code, they want to enable others to achieve things)

For example the ability to compile Kernel code is only useful to group 1, whereas to group 2 it would only be useful if it was also possible to publish packages containing Kernel code2.

I believe that the main counter-concern is, that we all want to maintain the high quality of Elm and the benefits that brings.

  • For the first group, it has been repeatedly reported that after the ban of native, these people were forced to figure out ports and custom elements and they were often happier at how their code turned out3. That suggests that when the ability was there, it was fairly often used for things where it was not essential but rather as a path of least resistance.

  • For the second group, the issue is that packages, once published, are somewhat immortal. Once people start depending on them, it is difficult to remove them. We have heard people complaining about Websockets going away for years now (despite that being a use case remarkably well suited to ports). This creates a lot of pressure on getting these foundational packages right.

A wacky idea for group 1

I wonder if some of these things couldn’t be balanced with adjusting the incentives by adding artificial friction. Perhaps only allowing to compile Kernel code without --optimize like the Debug modules work now and adding a 5s sleep call for each non elm or elm-explorations Kernel module compiled would be sufficient that people would only use this after trying to solve their problem every other way. Other disincentives to consider:

  • Printing a humongous and nasty warning on every compile. (Elm Foundation warns: Using Kernel code may lead to lung cancer! :wink: )
  • Only allowing Kernel code compilation with a crypto key. One would need to print, sign and fax a long form to the Elm Foundation to acquire one.
  • Failing compilation randomly with a 15% probability.

A more reasonable idea for group 2

Personally I think the current guidelines on how to contribute to Elm are simply not good enough. Perhaps simply clearly stating something like:

Contributions to Elm are limited to the core team. The core team is based around personal relationships and we do not accept strangers from the internet. Reporting bugs is encouraged, but due to our process, please don’t expect a response.

Would be more fair in terms of expectation setting. There are many open source projects were code changes are limited to some group of people. But something like this would at least set people up with expectations that wouldn’t be frustrated. Any violations of the above would simply be happy exceptions to the rule. Whereas the current talk about how contribution works in Elm (talk, do research, coordinate, then code, …) makes people believe that as long as they follow that somewhat vague process, they are likely to succeed.


1 Of course this is a simplification of reality. Many application developers would also like to reuse/share solutions to common problems for example. But I am here focusing on concerns that become extreme enough that people feel like they need to exit the community (often with a bang).

2 With the small caveat that it would at least make testing Kernel code based patches somewhat easier and perhaps enable some hacky ways to get around the package system.

3 Of course there is massive survivorship bias here. Many were not and churned away - in some cases even building tooling to transform their code into a different language.

13 Likes

Quick name idea to throw out there: Morel. Morel mushrooms are delicacies that grow on Elm wood.

I like the idea of a pathway for community experimentation to be able to open up like this without impeding the core team in their goals. Clearly there are some warts to work out on all this, but I think this is a good discussion to be having.