Elm Design Question: Do we need JSON decoders?

Hey folks!

I just started a discussion with a friend at work, and wanted to learn more about why Elm has JSON decoders.

Here’s some context to help understand what I’m curious about.

Some context:

Using Flags

Here’s a simple demo of using Elm flags, and an example on Ellie (thanks, Luke!):

<html>
  <head>
    <!-- ... head stuff ... -->
  </head>
  <body>
    <div id="app"></div>
    <script src="/compiled-elm-app.js"></script>
    <script>
      Elm.Main.init({
        node: document.getElementById('app'),
        flags: {
          person: { name: 'Ryan', age: 25 }
        }
      })
    </script>
  </body>
</html>
module Main exposing (main)


type alias Flags =
    { person : Person
    }


type alias Person =
    { name : String
    , age : Int
    }


type alias Model =
    Person


type Msg
    = NoOp

main =
    Browser.element
        { init = init
        , update = \_ model -> ( model, Cmd.none )
        , view = \person ->
            Html.text ("Hello " ++ person.name ++ "!")
        , subscriptions = always Sub.none
        }


init : Flags -> (Model, Cmd Msg)
init flags =
    ( flags.person
    , Cmd.none
    )

Here is a simple Elm element program, where we define our Flags as a type alias, and Elm is able to create a record for us from the JSON we pass in from JS.

Just like an API request, this JSON we give Elm might not match the Flags type, so Elm gives us an error like this:

Uncaught Error:

Problem with the flags given to your Elm program on initialization. 

Problem with the given value: null Expecting an OBJECT with a field named `person`

Using JSON Decoders

If you aren’t familiar with JSON decoders, they tell Elm how to intrepret JSON. The elm/json package allows us to create decoders, and call decodeValue, which turns JSON into a Result (a type that can represent a success or failure)

import Json.Decode as Decode exposing (Decoder)


type alias Person =
    { name : String
    , age : Int
    }


decoder : Decoder Person
decoder =
    Decode.map2 Person
        (Decode.field "name" Decode.string) 
        (Decode.field "age" Decode.int) 


viewJsonPerson : Decode.Value -> Html msg
viewJsonPerson value =
    case Decode.decodeValue decoder of
        Ok person ->
            text ("Hello, " ++ person.name ++ "!")

        Err reason ->
            text (Decode.errorToString reason)

Decoders can be tricky to explain to beginners coming from JS, but once you get the hang of them, the API is really nice.

You can compose them really well into bigger and better things.

decodeValue returns a Result Error Person, so it will give us the reason if we want to share that with the user or log it or something cool.

My big question

If Elm already understands how to turn JSON into a Person using flags, what are the tradeoffs to using that instead of defining decoders?

My oversimplified solution

I understand that flags do not support custom types or complex data structures, so we would just write functions to turn generic JSON into more complex data values.

Imagining a made-up Json module like this :mage:

module Json exposing (Value, parse)

parse : Value -> Result Error a
-- ... does the magical flag stuff
module Person exposing (Person, fromJson)


import Json


type alias JsonPerson =
    { name : String
    , age : Int
    , color : String
    }


type alias Person =
    { name : String
    , age : Int
    , color : Color
    }


type Color
    = Red
    | Green
    | Blue


fromJson : Json.Value -> Result String Person
fromJson value =
    Json.parse value -- Result Error JsonPerson
        |> Result.andThen toPerson  -- Result String Person


toPerson : JsonPerson -> Result String Person
toPerson person =
    colorFromString person.color
        |> Maybe.map
            (\color -> Person person.name person.age color)
        |> Maybe.withDefault
            (Err "The `color` value was no good, dood.")


colorFromString : String -> Maybe Color
colorFromString str =
    case str of
        "red" ->
            Just Red

        "green" ->
            Just Green

        "blue" ->
            Just Blue

        _ ->
            Nothing

In this alternate reality, we just write functions to convert from our JSON data type to the one we want to work with in the code.

We use types to describe JSON structure instead of types and decoders, and just like decodeValue, the JSON we receive might not match up with our types. (This is why Json.parse would still return a Result type)

That’s it!

I’m sure there’s an embarassing reason for why this suggestion doesn’t make sense.

Let me know what your thoughts are on this, I’d love to understand more about why decoders might still be the best choice for handling JSON values.

2 Likes

This is something that comes up often from beginners trying to understand what the value of Decoders are. They provide a way for you to convert and validate incoming untyped values coming from JS in to typed values that you can confidentially use in your Elm application.

Incoming values are often not in the format that you actually want to use them, they’re often in the format that made sense to the program that produced them but don’t make much sense in how you’d like to use them in your Elm app. eg. json can give you a list of items but you may want that as a Dict keyed on the ids of the items for faster lookup.

The automatic conversion from json to an Elm value could be done automatically for some types (like how it’s done for Ports and Flags) but unless Elm had a way for you to specify a default json decoder for custom types then you’d be in the position of writing out the full decoder just for the custom type (which dis-incentivises using custom types) or defining some type that is almost the same as the type you actually want, decoding to that and doing the conversions yourself, which is essentially the same amount of code as writing out the decoder would be anyway.

Looking at your example, using the actual decoder api works out to be about the same amount of code. Since defining the extra JsonPerson record is the same amount of code as defining the Person decoder itself.

type alias Person =
    { name : String
    , age : Int
    , color : Color
    }


type Color
    = Red
    | Green
    | Blue


fromJson : JD.Value -> Result JD.Error Person
fromJson value =
    JD.decodeValue personDecoder value


personDecoder : JD.Decoder Person
personDecoder =
    JD.map3 Person (JD.field "name" JD.string) (JD.field "age" JD.int) (JD.field "color" (JD.andThen colorFromString JD.string))


colorFromString : String -> JD.Decoder Color
colorFromString str =
    case str of
        "red" ->
            JD.succeed Red

        "green" ->
            JD.succeed Green

        "blue" ->
            JD.succeed Blue

        _ ->
            JD.fail "The `color` value was no good, dood."
2 Likes

One pretty common use case that isn’t covered by “decode to a data type that directly matches the JSON, then transform that into whatever custom type you want” is that you can have things in JSON like a list of values of different types, for example a list of shapes:

[
    { "type": "circle", "radius": 3 },
    { "type": "rectangle", "width": 4, "height": 5 }
]

I don’t see a way you could have that converted automatically into an Elm data type (List of what?), unless you declared something super ugly like

type alias ShapeJson =
    { type : String
    , radius : Maybe Float
    , width : Maybe Float
    , height : Maybe Float
    }

which wouldn’t work anyways because type is a keyword in Elm, so you can’t use it as the name of a record field.

3 Likes

Ah, that’s a really good use case to consider!

Thanks for pointing that out! :smile:

Edit: Wait a minute… doesn’t this mean that Flags don’t support the type keyword? :grimacing:

1 Like

Hey @jessta,

I recognize your profile picture from the Elm slack- you were one of the folks that helped me when I was first getting started with Elm! Thanks for all your help contributing to the community, I know it personally made an impact for me!

Custom Types

For custom types, I definitely see how my proposed solution makes custom types less ergonomic.

But using decoders, doesn’t actually fix that concern. The colorToString function is the real burden, and we need it for both cases…

Hopefully, the Elm compiler is incentive enough for folks to reach for a custom type instead of passing around strings.

Code size

Also, I loved your point about how it’s actually the same amount of code. That is pretty funny :sweat_smile:

1 Like

A clarification on the prompt / my dreams

The reason I’m really excited about learning more from this discussion, isn’t because I dislike using decoders. I know they are among the trickier concepts to explain to my friends at first, but over time I became comfortable with them.

I can also see folks coming from a functional programming background picking up decoders quickly. The idea of a JSON decoder is actually a beautiful abstraction in Elm.

I’m just curious if there’s an opportunity to simplify how we interact with JSON without giving up the benefits of Elm, or introducing new abstractions for beginners (or experienced folks maintaining an Elm codebase)

I wasn’t around before version 0.17, when we were using the Signal type. I imagine it was a pretty cool abstraction also. I’d be hyped if we had the opportunity to make Elm more accessible, by addressing another (potential) pain point of dealing with data.

This is a highly complex topic with extensive implications that are not obvious from the start.

Evan addressed this in his The Hard Parts of Open Source talk (I’ve linked to the relevant section but the entire talk is worth watching)

In short it is a large engineering effort and other things take priority.

This is worth reading: A vision for data interchange in Elm. This bit made a lot of sense to me:

" At this moment in the Elm ecosystem, folks can be quite upset that you have to write JSON decoders by hand. Rather than questioning the idea of using JSON at all, they wonder if Elm itself should be generating JSON decoders. But the history of building in support for a particular interchange format is pretty poor. For example, Java assumed that XML was the way to go. Scala followed in that tradition and actually has XML syntax as part of its parser, so XML is valid Scala. I do not think it is easy to argue that this direction was wise, and I know there are efforts to get this out of Scala."

So I think there is definitely a case for more and better tooling for generating decoders/encoders, but that this should be something that sits on top of Elm rather than gets built into it.

Tooling that generates decoder/encoders could also evolve to generate ones for encoding/decoding Bytes to Protocol Buffers for example - or some other binary format.

An area that currently interests me is turning Swagger specs into Elm code, or transforming between Elm and Json schema, and so on.

5 Likes

Thanks @rupert !

That article makes the most sense to me. It would be super strange if Elm had baked in support for JSON and not any other format.

I suppose trying to make JSON work nicely is more of a short term goal.

Might not make sense in the long term for the language!

Thanks again everyone for your comments and examples!

I came here because I wanted to learn something and I definitely did :sweat_smile:

2 Likes

If you want to help me build a CLI to generate JSON-encoders/decoders based on static-analysis, get in touch here :sunny:

1 Like

In terms of phrasing, I think it makes sense not to focus on why you have to use decoders, but on why you want to use decoders.

In my experience working on very similar apps both in React and in Elm, there is a huge difference in design – in React, people rarely design their own data structures. They just use whatever the backend gives them, potentially applying some lightweight ad-hoc transformations on top. In contrast, in Elm, we approach things by first thinking about what would be the ideal type to represent the data for the purposes of our application. Once we’ve got that, we take a look at what the backend actually gives us. This can often be pretty different. We then write decoders as a way to transform the two – and even better, this happens at the edges of our system, so we have strong internal guarantees from then on.

Second, we are protected against bugs due to the backend and frontend getting subtly out of sync. There are many stories out there when the backend suddenly decides to encode what was previously an int as a string. In JS, this might even keep working for a while - implicit type casting to the rescue. But it will eventually do something unexpected. Decoders allow us to deal with this issue in a place which is easy to debug.

Finally, decoders also give you confidence in dealing with messy data.

6 Likes

I can speak only for myself but if elm core team would for what ever reason decide to ditch decoders in favor of some derived only decoding logic I would most likely leave the elm community (& leave the job) and invest my time to other language.

Why? Json is pretty stupid format. It can’t talk about much but few basic types. Its specification is lacking on many fronts and is generally pain to deal with in my experience. I hate the way Golang handles JSON marshalling and unmarshalling and I’m simply not able to to tolerate derived by default option when it comes to dealing with this format.

Swift’s Codable offers a reasonable middle ground IMHO. It’s a combination of two protocols (roughly type classes), Encodable and Decodable, that capture the general idea of serializing and deserializing a value with no particular format in mind. The standard library then offers JSONEncoder and JSONDecoder classes that do the actual JSON encoding and decoding. And the compiler has special knowledge of Codable so it can generate the protocol conformances for simple types automatically. So we get type-safe JSON encoding and decoding for free, we can customize the compiler-generated code a bit and if we want anything special, we can always write the conformances by hand. Overall it’s not perfect, but it’s quite close.

Please look at the discussions here - Generic decoding of json to an elm record

I had the same question too…

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.