Yeah, elm-concurrent-task seems to be a fork of elm-pages scripts. As I say, the “port funnel” technique never appealed to me - why multiplex everything through a single pair of ports? I would prefer to define individual ports for each IO thing, and even group them into separate packages. Also I have existing ports code and its hassle to convert it work with that system, which is aimed at request/response patterns only - when mostly the ports I already have already work in an elm-procedure type fashion, and are a mix of request-reponse patterns and fire and forget commands, and standalone subscriptions.
That is really what I am trying to do - write new effects for Elm and provide a convenient way of assembling them into larger programs that sequence lots of io operations together - cli tools, cloud functions, back end logic and so on.
FWIW, I have something similar in my REPL worker. However, I use ports for synchronous communication between an Elm app and an Elm worker, so there’s no JavaScript peer. All of this is based on the IO type I used in my compiler port. From your description, my implementation doesn’t offer anything you don’t already have, though, but if you do want to look at the code and have questions, I’m happy to explain more.
Yes, it does look very similar. Main difference would be that I have added support for Sub so that Cmd+Sub can be paired to form task port.
Is your worker always running in the main javascript thread? From reading the javscript glue code it looks like it, I was wondering if you might have run it in a Web Workers thread.
Unfortunately I don’t see a way to share MVar between threads in javascript, so cannot see a way to parallelize the execution.
The other thing that elm-concurrent-task does that your library does not yet do is associate requests with responses. If you make multiple requests out the same port, your implementation relies on those responses coming back in order, but elm-concurrent-task associates metadata with each request so that the responses can arrive out of order. My intuition is that one of the reasons a single port pair implementation was chosen was to avoid the application author having to write the boilerplate for that metadata for each port pair
If you have a list of Tasks, convert them all to Cmd using Task.attempt, and then return them as a Cmd.batch, will they all run concurrently? I am assuming they would, but I have never tested it to check. This would not use Task.map.
mapN pretty much has to be sequential, since you give it a function like a -> b -> c -> d -> ... and it needs the parameters in-order to give to the function. Could it be that is all that is blocking Tasks from running concurrently, and its just a question of avoiding the Task.mapN?
It can be done out of order by elm-procedure and my implementation also, since the current implementation is copied from that. The way it works is like this:
It opens the channel and passes through the unique channel key, and then filters the return subscription by that same key value. If open were to be run many times concurrently, there would be as many Subs created as the number of open requests, and each of them would get every response - hence the need to filter just the one that you want, The open-connect-filter-acceptOne flow assembles a task port. This does allow out-of-order execution of the task ports concurrently, but it is obviously not a very scalable way of doing it.
I have not read the code of elm-concurrent-task in detail yet so I do not really know how it works, but would certainly like to know if anyone can explain it.
Ideally I would like the scalable concurrency solution, but also trying to combine that together with the API I already have, avoiding “port funnel”, and other design choices, so perhaps it might not work out.
I think the way elm-concurrent-task works with having just a single Cmd+Sub pair makes it simpler to do concurrency, since there only ever needs to be 1 subscription.
The problem elm-procedure implementation has is that a Channel does not just represent the ability to create a task port from a Cmd+Sub, it also represents a particular request to that Channel. Mutliple requests to the same task port require multiple Subs to the same subscription port, since the channel and request have the same lifecycle.
To fix this, this needs to shift so that each Channel represents a Cmd+Sub pair, but not any particular request to it. There must be a separate function to create a request with a fresh id. When a response comes into the Sub, the matching Proc based on request id must then be continued. So need a Dict Int Proc rather than a Dict Int Sub.
Channels would need some kind of unique identifier per Sub, which can be left up to the caller to provide and ensure uniqueness of. There also needs to be a way to scan all the active requests to find out which Subs are still active, in order to return them in subscriptions. Some data structure needed to efficiently track which Channels are currently active.
This will allow Subs to be used efficiently, with 1 Sub servicing many concurrent requests, and all the responses resolving correctly against the requests that created them by unique id.
Not sure how that is going to work out in code and types yet, but I think that describes the nagging conceptual fault line in elm-procedure that prevents it scaling concurrency.