One background worker, multiple views of the same state possible?

Hi all :wave:

Before I come to the actual question, first the important background:

I’m working on a browser extension in Elm (WebExtension / Chrome Extension).

I want my extension to be running all the time in the background, so I created a Platform.program and start it as a worker inside my background script. This works well so far.

Now, I want to have different views on the internal state of my worker, e.g. one for the popup of the extension, another simpler one to inject into the current tab.

I do have an idea how I could do this, but I don’t like it very much:

I could create a new elm Program for each view that looks kinda like this:

port module Popup exposing (main)

import Html exposing (Html)
import Json.Encode exposing (Value)
import Background    -- This is my background worker
import MainView      -- A view of the background model


type alias Model =
    Result String Background.Model


init : Value -> ( Model, Cmd Msg )
init model =
    ( Background.decodeModel model, Cmd.none )


type Msg
    = NewState Value
    | BackgroundMsg Background.Msg


update : Msg -> Model -> ( Model, Cmd msg )
update msg model =
    case msg of
        NewState bgModel ->
            ( Background.decodeModel bgModel, Cmd.none )

        BackgroundMsg bgMsg ->
            ( model, sendMsgToBackground (Background.encodeMsg bgMsg) )


port sendMsgToBackground : Value -> Cmd msg


port newState : (Value -> msg) -> Sub msg


subs : Model -> Sub Msg
subs model =
    [ Result.map Background.subs model
        |> Result.withDefault Sub.none
        |> Sub.map BackgroundMsg
    , newState NewState
    ]
        |> Sub.batch


view : Model -> Html Msg
view model =
    Result.map MainView.view model
        |> Result.withDefault (Html.text "")
        |> Html.map BackgroundMsg


main : Program Value Model Msg
main =
    Html.programWithFlags
        { init = init
        , update = update
        , view = view
        , subscriptions = subs
        }

Then, I could wire up those ports and let the background page communicate with those multiple frontends this way.

However I don’t like this solution.

Since my Model and Msg both contain union types, I would have to encode/decode the entire thing as a Json.Value, as the automatic conversation elm does can’t deal with union types.
Plus, this is a lot of unnecessary encoding/decoding. I don’t know if that will impact performance.
Plus, an identical version of the state now lives at multiple locations in memory.
Additionally, that’s a lot of boilerplate…

My question now is the following:

Is there a better way?
Can I somehow share the state of one elm program with another?

I’m also prepared to write native/kernel code if that would help reduce the boilerplate.

3 Likes

I now went ahead and just did it.

It works exactly as described above. It’s pretty hacky, but it does the job.

To save me from having to write an encoder/decoder for both Model and Msg, I took a very dirty shortcut.

I created a native module that implements these impossible functions:

toJs : a -> Value and fromJs : Value -> a.
(On the js side, the implementation is just identity)

This allows me to pass all possible elm values out of elm, to js and back into elm again, without any encoder/decoder overhead, but also without any type safety.

Does anyone have an idea how this could be solved more nicely?
My current approach feels pretty much like madness…

2 Likes

I’ve done something similar with Elm Lens (https://github.com/mbuscemi/elm-lens). In my case, it was not to get multiple views, but to allow one part of the application run as a worker process and another part of the application run as part of the Atom plugin directly.

There are two Elm applications, Main.elm and Worker.elm, which each share a bunch of data structures. I use the Report type to pass information between them. That’s ultimately the JSON structure being turned into a Value and passed through JS.

For encoders and decoders, in my case, you’ll notice that Report has a bunch of fields which are themselves complex types. Each of those modules has an encoder and decoder. Report just aggregates all those up. So, my values are type safe. I noticed that this strategy is prone to a given decoder not being properly aligned with a given encoder, so I create InteropTests.elm, which checks that all the pairs of those throughout my system work as intended.

I initially found decoding and encoding union types daunting, but these are actually pretty straight forward, and I found my way through with a few examples. I’m curious whether or not this helps make them easier to grok: https://medium.com/@matthew.buscemi/the-five-stages-of-json-decoding-in-elm-e695adb9162a.

3 Likes

And this is a short and sweet example of encoding/decoding a small union type: https://github.com/mbuscemi/elm-lens/blob/master/elm/src/Types/SpecialType.elm.

Thanks for your reply.
That’s interesting, good to see somebody else came up with a similar method.
Although it would have been more educational if you had come up with a completely different method :smile:.

Regarding encoder/decoders:
I don’t have a problem writing encoder/decoders for more complex types, but it’s a lot of work to maintain.
As you noted, decoder/encoder pairs tend to go out of sync easily. By using tests, you can catch this, however you now have to also keep your tests in sync with the model.
My solution does not require any changes when I update the model, but it could easily blow up in my face if I’d use those native functions in the wrong way :boom:

Writing encoders/decoders + tests is probably still the cleaner solution than my hacky native code, but I was too lazy to do it the “proper” way.

1 Like

Actually, InteropTests.elm is the least brittle set of tests in that system. Changing a data structure causes both its decoder and encoder to change implementation, but not signature. So, the test remains fairly stable even when there are changes to the implementation of the data type.

What I meant is that for a single change in your model, you have to change code at a bunch of different places.

Say you add a new field to your Report type. Now you also have to change the decoder, since the compiler will tell you to do so. When you try to run your tests, the compiler will also tell you to change them. Now your new test should fail because you forgot to encode the new field in the encoder. Now you change the encoder and your tests are green, so you’re good to go.

This is great at catching all errors, but it’s a pretty slow process if you have to change your Report type often.

I guess this works nicely for you since you probably don’t have to change that type very often.
But in my case it would slow down development quite a bit, since I want to share the whole Model, which does change all the time.

So, if I change Report to include a new field, I will be immediately forced to change Report.encoder and Report.decoder. The only part of InteropTests that will change is that I’ll have to update the representation of Report with some new, dummy data structure. Because the signature of Report.decoder and Report.encoder never change, the rest of the test remains stable.

This is super simple, and this remains the easiest test in the system to change. I suspect it has saved me a ton of time when compared to the alternative of tracking down JSON decode errors between supervisor and workers. Most new features in Elm Lens have involved making subtle modifications to Report or its child elements, so I wouldn’t say it’s been particularly stable either.

I’m not trying to argue against your approach, it apparently works very well for you and seems to be the best in your situation.

But I’ve got one little nitpick about your last reply: if you add a field to Report, your are not forced to change Report.encoder. The compiler is perfectly happy if you don’t use all the fields in the implementation of Report.encoder.
This has happened to me a few times, e.g. I changed the type, changed the decoder, but forgot to change the corresponding encoder.

But since you also have tests, it will be caught by the test, so it’s ok.

1 Like

Yes, you are correct. I recall that now. As long as the change is merely an addition, the encoder is happy to ignore the new values. But if I change the data types or subtract a field, both will break.

Hi, a question about this. Are the background script, the popup and the current tab all separate threads? That is to say, each must have its own Html.program or Platform.program entry point, and all run concurrently?

Or is it possible for a single Elm program running in one thread to provide the views for the popup and current tab?

The reason I ask is because the two scenarios are quite different. There is also nothing provided within Elm itself to help you with communication between Elm programs in different threads, you need to figure out to glue them together with some javascript yourself.

Did you have to write some kind of threaded producer/consumer queue in javascript to link the programs together?

Sorry I don’t have any answers for you, but I am interested in the idea of how to make multiple Elm programs communicate together at the moment, both within a single Elm application (by combining programs to for a single top-level program) and between multiple Elm applications running in 1 or more threads. In both cases the uni-directional message passing style of Elm ports “a → Cmd msg” and “(a → msg) → Sub msg” is a good abstraction as you do not really have to care whether it is implemented synchronously or asynchronously.

I don’t know if they actually run in a separate thread, but what I do know is that they run concurrently and in a different JavaScript context. The popup can access the window object of the background thread, but the content script is completely isolated from the background. They can communicate via a message passing API.

So what I do is create an elm program per context and then let those communicate via ports via the messaging API with each other.

No, the background program just has two ports, one for incoming messages, the other for outgoing state updates.
The frontend program also has two ports, one for incoming state, another for outgoing messages.
Then I just link those ports together. (The content script additionally has to go through the extension messaging API)

I hope that answered your questions.

Yes it did. So I think you found the only way to make this work which is to make the 2 programs communicate through ports.

Also, nothing wrong with this. Great way to get up and running fast while you are still thinking about the data model. Possibly you never need to write the encoder is toJS : a → Value is working nicely, I’d put my effort into the decoding side first.

For decoder there is strict from this package https://package.elm-lang.org/packages/zwilias/json-decode-exploration/latest/Json-Decode-Exploration#strict

or this approach https://ellie-app.com/74fZ5D5PLKba1

Maybe for encoders similar approach?