Afterthoughts around about Kernel code and custom operators complaints

Request Feedback
1

The “Why I left Elm” post and the following discussions here on slack a couple of days ago got me thinking a little bit. And in my mind I came up with some questions about two of the main topics that were brought up: Restrictions around Kernel Code and custom operators?

  1. Why do people complain that only makers of the Elm language(Evan) can define operators when the same is true for JavaScript?
  2. Why do people not accept that Kernel code can only be written by Elm Makers(Evan) when in JS only makers can make calls to native© code(at least in Browser JS, excluding web assembly)?
  3. I wonder if kernel code wasn’t part of the package that uses it, but centrally defined in the compiler if there were less discussion about how people want it in their packages too? Then it would seem more like a central kernel and less like a privileged package.

Thoughts about that anyone?

5 Likes
2

I think the problem is that people want to do things with Elm it is not designed for (e.g. use it on the server). Also, ports are a bit annoying to use in some situations.

3

Why do people complain that only makers of the Elm language(Evan) can define operators when the same is true for JavaScript?

Elm used to have the capability, and it was removed. That aside, there are other languages that do allow custom operators, or operator overloading at least, so comparing strictly to JavaScript is a bit limited. I also think it would be less of a deal if the a package like parser didn’t get to make its own operators, since everyone can clearly see that it’s still possible, just not for them.

If you look at our brains, one theory that describes how we interact socially is the SCARF model, where the F stands for Fairness. The short version of it is that many of us are built to be triggered when we perceive unfairness, and this is a clear version of an “unfair” restriction, since the core team can utilize it, but the rest of us can’t.

I don’t personally care much about operators, but I think having a limited set of operators that everyone can overload hits the sweet spot. That way you don’t get operator bonanza (where users define arbitrary operators that make code harder to read, which was also the reason the capability was removed), but you still give users the ability to use some common (and useful) operators on their own types.

Why do people not accept that Kernel code can only be written by Elm Makers(Evan) when in JS only makers can make calls to native© code(at least in Browser JS, excluding web assembly)?

Because the reason people want to write kernel code is usually to access some browser API (where JavaScript already has full access), or for performance reasons (since JavaScript can be notably more performant than Elm). JavaScript is thus “good enough” to achieve what needs to be done.

I wonder if kernel code wasn’t part of the package that uses it, but centrally defined in the compiler if there were less discussion about how people want it in their packages too? Then it would seem more like a central kernel and less like a privileged package.

I think it’s important to separate a few concerns: I haven’t seen anyone advocate for having kernel code in packages they upload to the official Elm package repository for example. I do think the markdown package falls foul of the same “fairness” behavior I described above, since it does exactly what some people also want to do (wrap some perfectly good JavaScript code in an Elm interface), but they lack the privilege to do so (and a clear, reasonable, and fair way to attain that privilege).

It’s also interesting that many languages do have an escape hatch of some sort (Java, Rust, C#, and more), because it is generally something that someone will need at some point. This escape hatch is also present in Elm, but is again only available to the core team. This means there are both those who just consider it unfair to have it locked away, and those who actually need it to complete a task.

The reason the ability to write native code was removed was both because it was never actually intended to be a thing; it was an undocumented “feature” that people started using, but also because it could compromise some of the Elm guarantees, and make it harder for the compiler to transform the generated code. I do think it would have been worth exploring a well defined unsafe API though, that would still allow people to use JavaScript if they need to.

Note that I don’t have any personal desire to write native code or custom operators, but I think it’s a fascinating discussion, so I just like participating :smile:

9 Likes
4

Its sad to see a language we love being trashed online and some of the comments I (skim) read of peoples opinions on Evan were just ridiculous. I also read on Evan’s twitter that it made him feel like quiting, which makes me sad since he has built such an excellent thing that I enjoy using so much. I don’t know why people insist on engaging in an unpleasant way - but it is the internet!

A small but vocal group of people can create a big stink, in a small pond like Elm. But I am very sure there is a quiet majority that are mostly very happy with Elm and just getting along with it and creating good stuff - I know I am.

I acknowledge that things are not perfect (yet), but 0.19 has been an incredibly productive experience for me. It took me some time to “get it” too, but I really appreciate that kernel code is off limits to package and applications developers - I know I have written better code as a result; and I know I have not accidentally pulled in bad code from package repo too.

22 Likes
5

Thanks for the responses everyone.

Elm used to have the capability, and it was removed.

I get that. Having them exposed first and then taken is annoying for heavy users of operators. I get that people are not happy about that. I never thought about operator overloading. I just know from using Haskell, that most operators there are just plain weird and confusing.

Because the reason people want to write kernel code is usually to access some browser
API (where JavaScript already has full access), or for performance reasons (since JavaScript can be notably more performant than Elm). JavaScript is thus “good enough” to achieve what needs to be done.

I get the itch that people may have for it, but I don’t that justifies creating an escape hatch for it. Browser only offer such a wide range of API for camera, accelerometer, VR, etc now. Think about how it was 15 years ago pre HTML5. Had there been a way to extend JS to call API that didn’t exist back then, I think the development of browsers would have been way different. I think sometimes it just takes time till something is ready. Browsers took time and Elm does too.

1 Like
6

Sure, but the key here is that this is subjective. I don’t have any personal preference, but I think any reasonable person could decide to go in either direction; it’s just a matter of which trade-offs matter most to you.

So I think it’s fine that Elm has been set down its particular path, but I also don’t think it’s been handled very well in the community. You have people who could previously do a thing, and now they can’t, and they need to adapt. They don’t necessarily feel like they got much say in the decision, or their criticism has been met with a reasonable response. Now they can see the tool they used to use is still there, but just locked away.

Meanwhile, a common response they get when airing their frustrations is that “their goals just don’t align with Elm” or “they don’t have to upgrade their compiler” and so on. This is an incredibly frustrating response to get, because it is very antagonizing. Maybe they feel that their goals aligned perfectly with Elm in version 0.18, and now they only align 95%, but those last 5% are critical to their success. They may even have thousands of lines of Elm code that they have been happy to build and maintain so far. At the same time, version 0.19 also brought many desirable improvements that they would like to work with.

So even though it’s not my experience, I can empathize with anyone who feels they have been thrown to the side, and now feel they can’t use Elm, and wouldn’t recommend it to others, because of that experience.

1 Like
7

I had a discussion about this with someone on Slack recently - who seemed quite angry and kept projecting his feeling about “the Elm core team” onto me, accusing me of being some sort of shill for them, when I was merely giving an account of my opinions. The article about leaving Elm seems to center around a GitHub Issue about resuscitating the old elm-github-install project, to allow packages with kernel code to be built and shared through GitHub at least. The person I encountered on Slack seemed annoyed that the package site does not actually prevent these packages being uploaded there and using that to say that it means it is actually allowed and a forked compiler that does it should be acceptable.

I don’t have any issue with someone forking Elm and starting a new package repository that does allow kernel code; it is open source after all. I also note that none of the people complaining seem to be willing to put in the effort to do just that - perhaps they know it would not be the best direction to take. But I do have issue with someone doing it and attempting to undermine the package repository that we do rely on for use with the official compiler and its guarantees.

8

If I were writing an application (not a package) and I felt that I absolutely must have kernel code in an Elm application, what I would do is to create a private fork of the compiler and remove the restriction. That would obviously come at a cost of maintenance, but I think it is also true that it is likely that small change could be fairly easily rebased if say a 0.20 compiler comes out, so I don’t think this cost is prohibitively high. You would be able to consume all the existing packages and participate in the community - in fact no-one even need know you have done this.

The cost is high enough to put you off doing it on a whim, but its not a complete deal breaker. I was not totally convinced that disallowing kernel code in applications was a good idea, but I think it has a couple of other benefits. It makes you consider the alternatives first, and I have always found a way of doing what I need without kernel code that I think is actually better than if I had done something with it. It also means that anyone doing it is operating outside of the official compiler, so has no legitimate come back to complain when the FFI is changed between compiler versions; they had to take very deliberate steps to break out of the official narrative and understand the costs and implications of doing so.

2 Likes
9

For completeness I just wanted to throw in that it doesn’t seem possible to allow kernel code inside an application with just removing the elm and elm-explorations restriction, you’d have to use a package to use kernel code. If it’s possible it’s certainly not as easy as it sounds :slight_smile:

2 Likes
10

Also good luck getting your custom compiler to work properly with other tools like Parcel, webpack, various CI systems, etc.

Personally I think for the (hypothetical) case where you suddenly find yourself stuck on a deadline and need to make some dirty hacks to save yourself, this is a complete straw man in terms of practicality.

2 Likes
11

Practical solutions are nice of course, but I would advise everyone involved in this to think about emotional solutions too. The technology side is as it is, but I think a large problem right now is that so many people have been burnt in this discussion, and there hasn’t been a place to talk about healing. This is also why Luke ended up writing his blog post:

A bad process is fixable if there is meta-process, but there is none in Elm. This is in fact why I’m forced to write all of this in a blog post — the Elm core team have clearly communicated that there are no channels within Elm spaces for us to talk about these things.

I haven’t seen any public retrospectives or introspection on this, so people like Luke end up with pent-up frustrations that have nowhere to go, and we end up in the situation we are in now.

5 Likes
12

It is not possible to use Kernel modules inside applications, though. You would have to seriously mess with the compiler to get anything useful out of it.

13

I see. In that case, the simplest way I can think of doing it would be to remove restriction, write kernel code in a package, and then install that package with Skinney/elm-git-install.

Perhaps a future version of Elm could allow kernel code in applications? And if the FFI is not going to be publicly supported, a prominent warning added saying that it is not supported and anyone relying on it can expect it to break in future releases.

14

I see. In that case, the simplest way I can think of doing it would be to remove restriction, write kernel code in a package, and then install that package with Skinney/elm-git-install

That is also not possible, since all elm-git-install does is download the package from git and add its source to source-directories in elm.json, it doesn’t actually compile the package.

I am currently writing a proper production-ready fork of the compiler that “just works” and as part of my research I have gone through everything related to escape hatches in the elm toolchain. The community is greatly underestimating the amount of effort needed to use kernel/native modules and the “just fork the compiler and remove the restriction” argument is both absurd and technically incorrect.

4 Likes
15

Ah, Not that I was planning on hacking my way to use Kernel Module. But it’s also new to me that it very hard to achieve even when forking the compiler.
I do wonder though how Evan does that himself when developing a package that isn’t published yet.

16

Actually, it is not. One can get pass the restrictions in the compiler without modifying it. Simple scripts that copy files around are enough with the current version.

1 Like
17

@gigobyte If you do this and intend not to keep it private, please consider respecting the wishes from Evan to avoid more drama. The ones I remember are:

  1. Do not use the name “elm”.
  2. Do not use the packages server to allow publishing packages that cannot be published by the current compiler (so either fork the packages server and host it yourself or keep the current restrictions).

Also I don’t remember anything said about it, but using a non-confusing different logo is likely desirable too.

Thank you

9 Likes
18

As so often happens, this thought started snowballing a bit, and I came to the conclusion that ports are already (almost) perfect. The only thing they’re missing is the ability to be called synchronously. But I imagine an API ala:

sync port MyJsCall : Decoder Int -> Encode.Value -> Result JsError Int

callThePort : Result JsError Int
callThePort =
    encodeSomeValue |> MyJsCall Decoder.int

But otherwise functioning like regular ports, both in how they are hooked up on the JavaScript side, and that they can’t be used in packages. I also imagine the Elm runtime would have to do a try catch around it, to avoid any exceptions bringing everything down.

The question is of course if their mere presence would discourage users to discover good ways of doing things in Elm, but that’s a question for someone else to answer :slightly_smiling_face:

EDIT: Wait, that still allows side effects like writing to localStorage. That might be a deal breaker :thinking:

19

One thing that blog post changed for me is how I think about custom operators, while I agreed with the removal, I never though of it like @mzero put it

Of all the changes in 0.19, this is the one that most hurt my code: I have parser combinator library, and used just two custom operators, for the very reasons that Evan points out in at the top.

Now I learn that elm/parser can, and does, define two operators for parsing, for the same reasons my library had done so. There are indeed times when custom embedded languages with custom operators are worth the mental effort on the programming staff. Parsing is one of them, which Evan acknowledges, and indeed uses in elm/parser .

However, it is not realistic to assume that elm/parser will become the only parsing package we ever need. For one, it only works on String . Parsing over byte arrays is quite common, (and what mine did). Even if elm/parse had been parameterized on the stream type - there are still differing implementation and functionality tradeoffs in parsers (backtracking, error tracking, error recovery, etc…) that make different parser libraries useful even they support the same stream type.

20

Yes, that is the point of not allowing it as it makes side-effects possible. All IO has to got through the update loop, which keeps the functional program pure.

If side-effect were allowed, I bet one of the first things we would see is:

module EasyTime exposing (..)

{-| Grab the current timestamp without all the boiler-plate 
hassle of defining a Msg and adding a branch into the 
`update` function and so on.
-}
getTime : Posix

And it would be downhill from there…

3 Likes