History
I want to start by telling the history of “Native Code” as I remember it.
Drawing a Pentagon
Back in 2011 I needed a way to call JavaScript functions to render a pentagon on screen. Maybe my thesis readers would look at my online editor, and that was what I needed to make it work! It was just some code that I needed to write.
We need a package ecosystem!
I released elm-package in December 2014. That means there was no package ecosystem at all for quite a while, and based on studying other small programming communities, I felt that having shared resources like this is extremely valuable. I faced a question at this point. How can I (1) publish my libraries which need those original hacks and (2) not have people write jQuery bindings for Elm? (That was a more popular idea then!)
So I added this "native-modules" field to elm-package.json and only allowed it to be used by packages listed in this native-whitelist.json file. Keep in mind, at no point so far was the goal to have application authors write code like this. I just needed a way to release elm-package and this seemed alright. It was not meant to be a public API, so the "native-modules" field was just a little detail tucked away that I figured no one would ever look at. That turned out to be wrong!
Unintended Consequences
I was very surprised when people began reverse engineering it to write packages, and we did not really know how to handle that. “I guess some folks know about it now, and maybe they can be helpful!” So we had about ten people as a “review committee” to look over packages that people wanted to add to native-whitelist.json. As far as I know, all of the people involved in those reviews felt that it was a failure overall. We got a lot of “bindings to whatever.js” and our discussions kept converging towards it not feeling like a good call. Those experiences guided this post about wanting to focus “native code” on covering the Web Platform specifically.
As time passed, I felt that people were just using this thing more and more. I always wondered “How do I document that a-thing-that-does-not-exist does not exist?” Where does that documentation live? Who would want to use such an API? So eventually I felt that I had communicated the intent poorly and tried to address that here. I suggested that we rename “native code” to “kernel code” and encode the intended usage into the software itself.
Can you summarize those two posts? I will try. Important considerations include:
- Reliability. People observe that if it compiles, it works. This is a huge benefit in Elm. Having arbitrary JS in packages means giving this up. Code usually works. Packages often are fine. Wait, do you have the IE9 polyfill? I think that is a huge loss.
- Portability. Elm will likely compile to WebAssembly some day. It may target other domains, like servers where there is no JavaScript. It would be hugely valuable if the entire Elm ecosystem works across a boundary like that.
- Security. Right now, people cannot put tricks in their packages that mess with your computer or with your users computer. I do not know of big exploits like this, but it seems like a strange door to open casually.
- Context. Languages like Ruby, Python, and Haskell have a traditional FFI to C, and they are fine! Why not Elm? The thing is that there are not hundreds of C libraries that Haskell and Python people are already comfortable with. I cannot think of the equivalent of “I want jQuery (and d3 and react and moment.js) in Elm” for Ruby, Python, or Haskell. And disallowing these bindings is how we ended up with great Elm packages like
elm-plotandstyle-elementsandelm-cssandhtmland others. It is a slower path, but in the end we have packages written in Elm, with Elm in mind, and I think the ecosystem is stronger for it!- Performance / Code Size. The fact that generated code is not pinned to some specific format means that I can change it dramatically without the community paying a big upgrade cost. In fact, I have done that a few times already! This has allowed me to (1) make Elm code a bunch faster overall and (2) shrink the size compiled output quite significantly. The compiler is not that sophisticated about optimization right now, and if there was a real “public interface” for writing kernel code, that would mean that we are locked into how things are today. Knowing what I want to do with optimization, I think that would be a big loss.
I hoped that reframing this code as kernel code, like in operating systems, would help highlight other systems that face similar tradeoffs. And point out that “why doesn’t JS allow C calls?” is basically the same question. Having everyone have arbitrary access to everything is not free.
That said I really recommend reading those posts for better versions of these considerations! Okay, back to the history…
Discipline is Unreliable
Since then, I have come to believe that there is no amount of documentation or communication that will really be effective. If the path exists, people will walk it. And with WebAssembly on the horizon, this is a door that is going to close on us anyway. I can just ignore that and implicitly let more people find themselves in the same situation in a few years, or we can deal with it now. I think dealing with it now is the more responsible choice.
That brings us to the present.
Today
So in 0.19 only the elm-lang and elm-explorations organizations can compile and publish kernel code or effect managers. It will not be available in applications or other packages. I hope the history I wrote, and the posts here and here clarify the design trade-offs that went into this.
Now I am very aware that certain users think this is an extremely terrible choice. Some folks have expressed that feeling to me in very strong terms. My point is that I understand that this decision is not necessarily the easiest possible path for all existing Elm users. But there is a path forward in every case I have heard about so far.
How will I upgrade?!
We will have upgrade scripts and nice documentation to make the normal upgrade path as easy as possible (like always) but that cannot cover folks who have native code or effect modules.
Now, part of why I have not made the decision about this restriction public until now is that I wanted to see how it would work at NoRedInk. There were a couple “native code” uses, and if I believe everyone on the internet, it is impossible to get rid of them. Perhaps that was going to be true! We brainstormed other approaches, and we found ways for each case. My understanding is that removing the shortcuts ended up being neutral or improving things a bit.
My hope is that folks who are struggling with this will share their specific scenarios, and folks can help work through them. Maybe you can use ports even though you really did not want to. Maybe you can use web components. Maybe there is some JS monkey patch that is possible. Maybe it is not extraordinarily urgent to upgrade to 0.19 on the first day, and in the months following the release, I can focus on any web platform libraries that are show stoppers for people. There are many paths!
All that said, Elm 0.19 is not out today. The alpha period will last one or more months, and that has not started yet. We have a lot of time to figure out each case, and I think we can do it in a good way.
Goals for this Thread
I think responding to this post directly is not really going to go anywhere, so I am going to disable that. Many folks have weighed in on this before, and I do not expect we will learn anything new from theoretical arguments. I want folks with specific questions to start new threads. “How do I deal with this exact thing?” That data can be really valuable for me to prioritize my work, and in most cases, I suspect other community members can help find a good path forward.
For folks who are really anxious about this, I really encourage you to talk to folks about it in a friendly way. I have not yet heard a case with no answer, and we have lots of time to work through things in a good way.