Chadtech/mail : Making ports act like http requests


#1

tl;dr
Tired of writing so many ports and subscriptions to request and retrieve values from JavaScript, I made a package that just automates the whole thing internally so you dont have to wire up any incoming or outgoing ports. Its called Chadtech/mail, and you can see an example project here. Below is the most basic example code:

mailLogin : Model -> Mail Msg
mailLogin model =
    [ ( "username", Encode.string model.username )
    , ( "password", Encode.string model.password )
    ]
        |> Encode.object
        |> Mail.letter "login"
        |> Mail.expectResponse loginDecoder LoginResult
        |> Mail.send

This code is saying "Pass along this Json.Encode.Value, to the javascript function named "login", and decode its response value with loginDecoder, which will be handled by the Msg LoginResult". Thats it. Theres no outgoing port that needs to get called, and you dont need to set up the subscription to listen for the result.

Background
@splodingsocks gave that great talk on ports at Elm conf, where he showed everyone a code technique of having only one incoming and outgoing port. Its a really great idea that excited a whole lot of us. At my job, we ended up implementing it in our Elm apps. @pdamoc made Elm Ports Driver, which are an Elm and Npm package for this technique too.

I ended up talking about it with enough people that I wrote this gist, summarizing how I do it. I ended up talking to James Hopkins, a new member of the Elm slack, about his project using ports to access firebase. Somehow an idea emerged between us in our conversation:

What if ports were like http requests, and you didnt have to wire up any subscriptions at all? Instead you could just pack up an address, a value, and an expected return value and send it off.

So thats what I set out to make, and thats what Chadtech/mail is. The whole one-incoming-and-outgoing-port thing is still there, but its entirely encapsulated inside this new Mail.Program type. Its all automated behind the scenes. On the JavaScript side you have functions with addresss, and you can send Letters to those addresses just by passing along a String with a Json.Encode.Value. The JS function also gets a call back, and if you mailed your Letter along with a Decoder a and a Result String a -> msg, Chadtech/mail decodes the return value and passes the Msg right into your update function.

The whole thing ended up being really intensive
I didnt realize this at the onset, but to make this work I had to kind of overhaul everything. You know how theres Html.program, but separately theres also Navigation.program? I had to make a Mail.program. The most fundamental change from Html.programs is that Mail.Program update functions return (Model, Mail Msg) instead of (Model, Cmd Msg). Its just an extension of Cmds, regular old Cmd Msgs can become Mail Msgs by means of Mail.cmd : Cmd Msg -> Mail Msg and they work just how they did before. Thats the most fundamental difference.


Thats my project. Im curious for any feedback, or if you think this might be useful. I was really just trying to get a proof of concept together but now I have some hope this could actually be practical. Let me know if you dont think so. Seeing as this was a lot more of a fundamental change in the Elm architecture than I thought it would be, it seems to me that the best implementation of this would be in the language itself, and not as a package. It really would just be making a Ports.send : String -> Letter msg -> Cmd msg function, along with a callback inside app.ports.f.subscribe functions.


#2

That looks really neat and convenient to use.

It is no more a fundamental change than say Navigation.program or any other specialized program types. I think as a package is fine.

Would it be possible instead to change the type of send to:

send : Letter msg -> Cmd msg

?

That way you would not need to wrap Cmd as Mail to use it, and your update function type could be the normal one.


#3

Im pretty sure its not possible to change the return type to Cmd msg. The way Mail works internally, is that it manages what ports have been sent out in the form of a Dict Int (Decoder msg). When ports go out, it assigns each one an Int id. When when ports come in, Mail.Program can read their id, and use the corresponding Decoder msg in its Dict Int (Decoder msg). But to store the Decoder msg at all, it needs to be able to extract what Decoder msg to store in the Dict Int (Decoder msg) at the point in which its sending the outgoing port, and that cant be done if outgoing ports are bundled into a Cmd msg because theres no way to get a Decoder msg from a Cmd msg.


#4

It reminds me of a lot of previous discussions about “task ports”. I wish I could find one with a more detailed response than this one.

The idea of ports is like asynchronous messages in Erlang. It is possible to create synchronous messages in Erlang, but that goes against the core design goals of having processes be independent of failure elsewhere. It would be odd to spend tons of effort to do this in Erlang.

I think these are all reiterations of wanting a traditional FFI in Elm. I understand that people want that, but as I say again and again, if we have a traditional FFI, we start getting direct ports of JS libraries. If that sounds nice, ReasonML and PureScript are both going to be nicer for that. You can go see if you like the results. Is their ecosystem better in your opinion? Maybe those languages are a better fit for your personal preferences or your particular needs?

So if you are not going for a way around kernel code stuff, I don’t really get why something more than Murphy’s advice is needed.


#5

I don’t know it you are aware of it, but elm-porter is another package that handles the synchronous pair of ports case:

http://package.elm-lang.org/packages/peterszerzo/elm-porter/latest


#6

You are correct that async messages are the basic unit of process communication in Erlang, but in actual practice, most processes utilize the gen_server OTP behavior. The gen_server behavior implements call/2,3 which provides request-response semantics for process communication without compromising any of the failure tolerance characteristics of the process model.

@chadtech’s approach here strikes me as a similar idea. I’m interested to give it a try.


#7

@xtian that is very interesting, thanks for sharing. I’ve always wondered how a good request/response API looks like with message passing.

Something that is mentioned on the task ports discussions is the question of what to do if the request fails or never answers. I think I read somewhere that the only option seemed to be having a timeout, which is what the OTP signature has:

call(ServerRef, Request, Timeout) -> Reply

…
Makes a synchronous call to the ServerRef of the gen_server process by sending a request and waiting until a reply arrives or a time-out occurs.
…
Timeout is an integer greater than zero that specifies how many milliseconds to wait for a reply, or the atom infinity to wait indefinitely. Defaults to 5000. If no reply is received within the specified time, the function call fails. If the caller catches the failure and continues running, and the server is just late with the reply, it can arrive at any time later into the message queue of the caller. The caller must in this case be prepared for this and discard any such garbage messages that are two element tuples with a reference as the first element.

Very interesting. Specially the part about the message being able to arrive any time later, and the caller having to be prepared to discard those messages.

I guess that means you still have to keep track of the requests that you’ve made and are waiting response for by some sort of id, to ignore any messages you receive that are not for those.


#8

Async messages almost always come with the problem that you may receive an answer to a question you no longer care about. For example, an HTTP request result can arrive after the data is no longer relevant to the user context. What I will give credit to Elm’s HTTP implementation for which evidently Erlang did not choose to do is managing the request and the timeout in such a way that you are guaranteed to receive at most one response.

On the other hand, the whole “what about tasks that never deliver results” thing is a bit of a red herring. Platform.sleep can take an arbitrarily long — though programmer chosen — time to complete. A JavaScript task that doesn’t return doesn’t introduce anything new as a “hazard”. For that matter, it’s worth noting that this proposal and task ports proposals before this did not introduce any corruption or crash risks that aren’t already present with ports. The code presented in this thread shows that since it uses ports to build an implementation.

So, I would take Evan at his word here and on some of the similar threads that the real concern is a desire not to make the interface to JavaScript too convenient lest that result in too many couplings to existing JavaScript libraries and functionality. Elm supports interoperation with JavaScript, but integration seems to be labeled as something to be resisted and avoided.

Mark


#9

Although it is true, 99.9% of the cases when doing a genserver:call/3 you expect some value to arrive and when it doesn’t you want to crash as fast as possible to let the supervision system to take over and restart anything affected.

So a scenario when “… the caller catches the failure and continues running” virtually almost never happens in everyday development of Erlang systems.


#10

This sounds like what I implemented in Chadtech/mail. If messages come in with an id that doesnt correspond to a pending request, then it just gets ignored.

I didnt implement a timeout thing, so I guess as it currently exists my project could just build up a lot of pending requests that never get resolved. I wonder what kind of problems that leads to.


Ports as Routes, sort-of
#11

Very nice @Chadtech, excited to try this out! Love how these interesting patterns are being discovered given the constraints of Elm.

This quote comes to mind. “Creativity comes from constraint.”

Any chance you’ll add wrappers for stuff like Mail.programWithFlags, Mail.navigationProgram, Mail.navigationProgramWithFlags?

Cheers,
Coury


#12

Yeah I would like to add navigationProgram and programWithFlags etc. Other things that need to be implemented are Mail.batch : List (Mail msg) -> Mail msg and the timeout stuff. The project just wont really be complete until then.

But I dont know when. This project isnt really at the top of my todo list.


#13

I do lots of crypto in my Elm app and thus a heavy user of ports. Especially the request/reply type of communication.

I like what I saw from briefly looking at Mail. However, I don’t believe in the mono-port idea. Mainly, because it loses type safety. elm-typescript-interop currently generates my typescript definition preventing me even from compiling something that does feed proper data through the ports. I would really like to maintain that property.


#14

Chadtech, very interesting! Great work!!