Elm, JavaScript, and Web APIs

[chris_cadds]

There have been quite a few threads recently (won’t bother posting them here) about concerns that people have related to the ability to write code that covers Web APIs in a way that isn’t roundabout like ports. I have nothing against the ultimate end-goal that the elm core team is trying to achieve. The idea of keeping the Elm ecosystem tight and healthy is a noble one.

One problem that I - and it seems many others, from the threads - have can be summarized as the following:
Before solving the underlying symptoms of why people are using Native/embedded JS code in their elm apps, steps have been made to remove our ability to solve these problems for ourselves.

This ranges from mildly annoying to hugely problematic for some of us. There is quite a large gap between what is possible in the browser, and what a language - which is only usable within the browser - is able to do. I know people believe that ports solve everything, and I agree they do solve quite a bit, but they don’t nearly solve it all. Sometimes the port solution is quite ugly or just a big hack.

My question is why not solve the problems of missing Web APIs, and then remove the average user’s ability to write Native or embedded JS code in their Elm apps? This seems much less user hostile and still accomplishes the same stated goals of the core team for the ecosystem.

Hopefully this comes across as I intend. I have nothing but deep appreciation and admiration for the maintainers, contributors, and community of Elm and want to see it live a long and healthy life!

[Dan_Abrams]

As someone who’s working on proposing a wrapping of one of these API’s, I understand the concern, but also see the other side.

The decisions made by the core team are all about improving Elm’s quality and portability in the long term, and ensuring excellent API design. These are laudable goals, and important. I particularly agree that we should try at all costs to avoid simply wrapping existing JavaScript libraries if at all avoidable.

On the other hand, there are several major API’s that are missing, the three most obvious being local storage, media, and (I believe) canvas. I can see an argument for simply performing local storage through ports on a per-app basis, but I agree that the other two desperately need native wrappers for the sake of reusability (hence, I’m trying to make a proposal for the one I know well). All three, I believe, have design challenges associated with them, and all three probably require domain specific knowledge.

And obviously there’s a lot of discontent in the community about this decision and the lack of progress on the web API generally. But I suspect you could satisfy the concerns of something like 70% of Elm users by simply making an effort to include the above 3 APIs in 0.19, and calm another 20% by outlining an official process to make proposals for other APIs. I think it’s also important to make clear that such a process should be used for wrapping web API’s, not wrapping libraries that can be written in native Elm.

I think it’s important to remember that the goals the language team are trying to achieve are really difficult, and require some difficult decision making that will inevitably make some people unhappy at times, but are made to make the language better in the long term. Personally, I’d rather see more effort made to wrap the Web API than a more permissive FFI. I guess my gripe could be summarized as “Hurry up! I want to use this awesome thing already!”

[luke]

@chris_cadds

I think the framing of this post needs some reconciliation.

@chris_cadds

it seems many others, from the threads

@Dan_Abrams

And obviously there’s a lot of discontent in the community

People that are generally okay with how things are tend not to spend their time posting on the internet about how ports are pretty much good enough. There’s no conflict with the present situation, so there’s nothing to express. For this reason, conversation that is purely about how some people don’t like something, or think it can be ugly, or a hack, tends not to factor into decision making. Developing Elm involves weighing an incredible number of tradeoffs at every decision. If one of those tradeoffs was “I can do this thing that is good from every important angle, but some people will write negative posts about it,” nothing would ever get done. I don’t mean that community feedback isn’t important or legitimate. I mean that reacting to the mood on the forums in a given month isn’t an accurate or reasonable way to evaluate it.

@chris_cadds

My question is why not solve the problems of missing Web APIs, and then remove the average user’s ability to write Native or embedded JS code in their Elm apps?

First I want to address that you are contending in this post and with this question that sometimes using ports is not an acceptable way to use Web APIs. Personally, what is important to me about writing software is building the thing that I want to build. In that respect, using a port or taking advantage of some other means of interop that does not use Native is fine. I’ve also used Native in the past because it was expedient. We did the same at NoRedInk. Now it’s going away, and we changed stuff to use ports and custom elements and other DOM APis, and we found that it was also fine and that in some cases code quality actually improved.

So I’m thrilled to engage in discussions about scenarios where the thing you need to do is (a) critical to your software exactly as designed, and (b) actually impossible without assistance from the kernel. I don’t agree that ports aren’t good enough to cover just about everything, because my experience and my observations and preferences lead me to feel that way. For that reason, I don’t think conversations about how people feel about ports provide very much value because nothing is blocked in a technical way, and the advantages of ports over FFI have been widely discussed and are firmly embedded into Elm’s existence.

---

I can also try to address the question of timing of Web API development, absent the notion that ports aren’t acceptable:

The work on 0.19 needs to be finished first, and it’s not finished. I don’t want to communicate more about that because it is either (a) not known, or (b) up to Evan to decide how to talk about it. After it’s out, attention will turn to the web platform. Packages aren’t going to start being released over night. There’s a lot of design work that needs to be done, and sometimes we’ll want to identify and design for use cases rather than just building bindings to window.X. For example, we might learn that most people want to use localstorage for caching request data, in which case a higher-level caching API that is backed by the safer and more powerful indexedDB might be what ultimately gets made.

If you feel strongly about a particular API, have a lot of experience with it, and have an idea for what a really great Elm API could be, then you can start the conversation on here, or on slack, or via email, or however you want to communicate, and we’ll work with you to see how it fits in and whether the story that explains the API is compelling. That process is long and unique for every individual idea. It isn’t done by just submitting code for review, and it requires more and different interest in the particular API and the Elm community than just “I want to use it in my app.” You can use ports for that.

[chris_cadds]

@luke

I don’t mean that community feedback isn’t important or legitimate. I mean that reacting to the mood on the forums in a given month isn’t an accurate or reasonable way to evaluate it.

Hmm. I agree here, but I also think it’s important to keep in mind that for some people, what people have been discussing on the forums, or on reddit, or in slack (and it’s been more than a month) is pretty much the entirety of what they see in regards to Elm.

Not everyone has access to core team discussions, a local meetup, etc. It seems like the people who are happiest/least worried are either privy to core team discussions, have talked with the core team at a meetup, etc.

I agree that the discontent voices are always loudest, but it’s also important not to discount them because they are unhappy or just because they are posted on the forums. Put another way: assuming people really are concerned - where can they post, and how can they frame it in a way so that it will be taken seriously?

@luke

Packages aren’t going to start being released over night. There’s a lot of design work that needs to be done, and sometimes we’ll want to identify and design for use cases rather than just building bindings to window.X. For example, we might learn that most people want to use localstorage for caching request data, in which case a higher-level caching API that is backed by the safer and more powerful indexedDB might be what ultimately gets made.

Certainly no one should expect these to be done overnight. It’s more a question of ordering.

Also, I’m not sure when the idea of Elm changed from “it should be released as to be useable and we will get feedback, but maybe we will miss some functionality or there will be a bug on some platform” to this new notion of “it needs to be perfect” with a big design up front. In fact in Elm we often encourage not doing a big design up front, just doing the simplest thing and then improving incrementally.

Look at Http - http 1.0.0. It was released and only supported string parts. Obviously this didn’t cover the whole (even known) API. There are countless issues with bugs in the repos that prove that rough edges and browser compatibility etc weren’t fully handled.

It would be great if we could release a reasonable API, take feedback on what works and doesn’t, and any bugs, and improve that for the next version. Otherwise, we wait forever for these “perfectly well thought out/designed” Web API packages.

[rupert]

In Elm 0.18 you cannot do native code either - your package must exist outside of the official structure and can only really be used with elm-github-install. So I would say the situation with 0.19 will really be no different to that, in the sense that it will be mildly annoying to hack the compiler, but from there you can do unofficial native code - or perhaps elm-github-install will evolve some work-around.

It may not be hard for example, to fool the package installer into thinking it is taking code from elm-explorations. Think about how you would go about hacking the elm virtual dom today - you can’t modify the code on github as you don’t have write access. So you use elm-github-install to pull it in from a different repo, or your local disk. Perfect, now you are ready to hack some kernel code.

What I am saying is, this issue may be causing some consternation, but really the situation is not any different to where we currently are. We will still manage to hack kernel code because its open source. Oh the wonderful liberty of the open source revolution.

Also, if you check the old elm-dev mailing list, there are a couple of posts about how to go about writing a new native Web API in Elm. I expect that policy will also evolve and hopefully become clearer, but I also think that it is not a path that many have taken. So it kind of seems like people are grumbling about losing something they never took advantage of in the first place.

Let us hope though, that more people do want to follow some official guidelines on how to develop new native web APIs for Elm, and that a robust community process is brought to life to enable this.

So I think the situation with 0.19 is an improvement wrt kernel code because elm-explorations. The structure is being put in place to make developing the remaining web APIs possible and publishable, just in a more carefully controlled manner.

[Dan_Abrams]

@rupert

Let us hope though, that more people do want to follow some official guidelines on how to develop new native web APIs for Elm, and that a robust community process is brought to life to enable this.

I think this is what many people are hoping for. There’s a lot of ambiguity over the path forward, and I think having some sort of official process, or even unofficial guideline would be very helpful. It would also be helpful to identify people in the community who have experience in specific domains, who might be willing to volunteer their time and expertise to help on a specific API.

[rupert]

@Dan_Abrams

I think this is what many people are hoping for. There’s a lot of ambiguity over the path forward, and I think having some sort of official process, or even unofficial guideline would be very helpful. It would also be helpful to identify people in the community who have experience in specific domains, who might be willing to volunteer their time and expertise to help on a specific API.

I think though, that this process always did exist. For example:

https://groups.google.com/forum/#!msg/elm-dev/oZ3xW_nMPNo/0y8j-N8HCQAJ

and there were other posts to elm-dev explaining a little about how it would work. I cannot find them at the moment, but IIRC it was something along the lines of write a technical paper explaining what you are trying to achieve, possibly look at how it is done in other languages for comparison, contact elm devs on slack and discuss the idea, develop a really good API for it by working with real world examples and users who want that API, own the work and don’t expect to just be an ideas person and so on. I don’t think the process was every really spelled out so clearly, but some general guidance along these lines was given.

Would it even make sense to try and spell out a process in exact terms? That might just lead to design-by-committee (though the Java community process was very successful despite that).

So what will really be different in 0.19? I am guessing but would it not be similar? produce some technical docs, discuss with elm devs to win them over, get an elm-explorations repo opened, do the work and iterate until something really good comes out of it.

I’ll start a new thread on this topic, as I have something in mind that is worth sharing and discussing.

[Dan_Abrams]

Thanks. The old elm-dev post was something I hadn’t seen. I’ve looked extensively for info on asuch a process and had difficulty finding.

[norpan]

I think there is somewhat of a clash between thoughts here. Something along the line of cognitive dissonance (I don’t know if it’s me or others that are afflicted, that is the nature of cognitive dissonance).

Let me clarify by an example. Look at HTML component libraries, we have a plethora of those. We have bootstrap, style elements, elm-mdl, etc. We don’t say this is a bad thing because this is how we do things, we try things out, we change, we try to work towards the best api:s.

Now contrast this with the process suggested for implementing, say, an interface to FileReader. Why would it be bad to use the same process there? Is there something fundamentally different that makes it so that it’s bad to have multiple choices regarding that?

I understand full and well that once there is an official FileReader API, then people will see that as the definite API, but what is the way towards that?

Nobody would be happier than me if there was an official FileReader API that was superior to my “hackish” norpan/elm-file-reader so that it could be retired and migrated from.

I’m just putting it out there, that maybe the source of the irritation can be found in this observation.

[rupert]

@norpan

Now contrast this with the process suggested for implementing, say, an interface to FileReader. Why would it be bad to use the same process there? Is there something fundamentally different that makes it so that it’s bad to have multiple choices regarding that?

I understand full and well that once there is an official FileReader API, then people will see that as the definite API, but what is the way towards that?

Nobody would be happier than me if there was an official FileReader API that was superior to my “hackish” norpan/elm-file-reader so that it could be retired and migrated from.

So why do you not do the following.

1. Try to document use cases for FileReaders. Or really for all file handling in Elm. Write down what it is that people want to do with files. Post about it here, or do some kind of survey online.
2. Take a look around at how other languages do it. How does purescript handle files? Is it good, can we use it in Elm, or can Elm be better than that.
3. Sketch out an API that satisfies these requirements and is also a good API for Elm. Using your experience and intuition as an Elm developer to guide you.
4. Propose that we create an elm-exploration for this, providing 1, 2 and 3 to document your proposal.

[norpan]

@rupert

So why do you not do the following.

Try to document use cases for FileReaders. Or really for all file handling in Elm. Write down what it is that people want to do with files. Post about it here, or do some kind of survey online.
Take a look around at how other languages do it. How does purescript handle files? Is it good, can we use it in Elm, or can Elm be better than that.
Sketch out an API that satisfies these requirements and is also a good API for Elm. Using your experience and intuition as an Elm developer to guide you.
Propose that we create an elm-exploration for this, providing 1, 2 and 3 to document your proposal.

Of course I can do that, but that’s not really the point I’m trying to make here. Why do we think it’s a good thing when people publish packages not involving web api:s and let them evolve organically based on usage and input from users, but a bad thing when they involve web api:s?

[rtfeldman]

@norpan

Of course I can do that, but that’s not really the point I’m trying to make here. Why do we think it’s a good thing when people publish packages not involving web api:s and let them evolve organically based on usage and input from users, but a bad thing when they involve web api:s?

We don’t. Robin’s data structure work followed the path @rupert suggested you follow!

[norpan]

@rtfeldman

We don’t.

We don’t what? Think that it’s a good thing that people publish packages? I still don’t think you get the point I’m trying to make, but then again it may well be that I’m the one experiencing some cognitive disfunction.

[rupert]

@norpan

Of course I can do that, but that’s not really the point I’m trying to make here. Why do we think it’s a good thing when people publish packages not involving web api:s and let them evolve organically based on usage and input from users, but a bad thing when they involve web api:s?

I don’t necessarily think we do think it is a bad thing. For example your file reader package as it currently stands already provides a model for thinking about how an API for files could be built into the platform. There is nothing to stop it evolving organically from where it currently is, it already is a good thing, just not a perfect thing.

Also, I suspect you don’t do steps 1…4 because it is too time consuming, too much work and too much responsibility to try and own such a process. My point being that the process we currently have (to the best of my understanding) is too heavy weight; it is a blocker.

[rtfeldman]

@norpan

We don’t what?

Sorry, to clarify - as I see it, we don’t think the one type of publishing is good and the other is bad; rather, we think improvements to foundational building blocks like core data structures and Web APIs should begin with discussions, not code - discussing things like use cases, goals, design considerations, and research.

[norpan]

@rtfeldman

we think improvements to foundational building blocks like core data structures and Web APIs should begin with discussions, not code

Ok, then I get you. I think there are two different things here.

1. foundational buildings blocks. Completely with you there.
2. using web API:s. Currently people are already using web api:s via ports and javascript. There is not so much any foundational language work being done there. Having a package use a part of a web api is different than trying to encompass a whole api in a foundational way.

I think it’s number 2 that is the source of the irritation.

[DanAbrams]

@rtfeldman

begin with discussions, not code - discussing things like use cases, goals, design considerations, and research.

It certainly wasn’t clear to me when I was starting to work on this where this discussion would take place. I had an enormous advantage living in the northeast and having physical access to you and others, but even then, I wasn’t sure where to post it. A big part of that was that it fell right smack in the middle of the transition from Google Groups to here, but it was definitely not obvious to me.

I really think there needs to be a big red button somewhere obvious that says “If you want to propose a javascript api wrapper, start reading here how to do it.”

It’s scary to do as an outsider, as well, and although this is a very encouraging community, it’s scary to propose and API design, even if you know a lot about that API, if you’re not already expert in the language.