Decoding really "flexible" JSON schema?

[keisisqrl]

Hi all, I want to build a frontend for an API and it has a JSON schema where the properties aren’t fixed. It produces object structures something like this:

{
  "type": "bus",
  "color": "blue",
  "left": { "type": "wheel" },
  "right": { "type": "wheel" },
  "fred": { "type": "rider" },
  "wilma": {
    "type": "rider", 
    "buddy": { "type": "dog" },
    "franklin": { "type": "cat"}
  }
}

You get a handful of fixed properties on each object type which mean something distinct, then there can be any number of additional properties (with any name) which can be one or more types, depending on the object. You have to descend into these objects to determine what type they are.

I’m trying to figure out a good approach to this. This schema seems kind of antithetical to Elm’s strong typing.

What I’m thinking - having not tried it yet - is to double-decode. That is, get all the fixed parameters out of a Value, then re-decode it into a Dict Decode.Value, filter fixed parameters out of the Dict, and map the Dict through a Decoder that can distinguish legal child objects (which will double-decode etc) and produce… maybe {a | type = b} where b is either String or a type representing legal children. Does this make sense? Is there a better approach?

As far as the schema… yeah, I know. I’m going to quote the elm/json docs on andThen:

Why would someone generate JSON like this? Questions like this are not good for your health.

[rupert]

When you have variable fields, you can decode them into a Dict.

Are all the fields Strings? That makes it easier as your dict will be of Dict String String type in that case. If they are not all strings, it gets more complicated.

I have a package for generic JSON decoding: decode-generic 1.0.0

I would try to avoid going for full generic decoding unless there is no alternative. See the README for the package for suggestions as to where it is applicable.

@keisisqrl

What I’m thinking - having not tried it yet - is to double-decode. That is, get all the fixed parameters out of a Value, then re-decode it into a Dict Decode.Value , filter fixed parameters out of the Dict, and map the Dict through a Decoder that can distinguish legal child objects

Yes, I have taken a similar approach on occasion. You can do a 2 pass decoder using Decode.map2.

Suppose you have a decoder that picks out the fixed fields, and one that picks out the remaining fields and puts them in a Dict:

Decode.map2
    (\fixed dict -> { type_ = fixed.type_, ..., others = dict })
    fixedDecoder
    dictDecoder

dictDecoder = 
    let
        fixedFields = Set.fromList [ "type", ... ]

        filter = Dict.filter (\k _ -> Set.member k fixedFields)
    in
    Decoder.dict
        |> Decoder.map filter

[antoine]

Here is an example based on the json you provided: https://ellie-app.com/6CwZ9CYWdCra1

My strategy was to decode the fields into a Dict String IntermediaryType and then use the intermediary field types to compose the bigger record. There is no double decoding going on.

I like solving json decoder problems, but I must admit this one was not easy to figure out. I hope the code isn’t too cryptic as a result. I’m sure this code can and should still be improved if you decide to go with it.

[jacob]

If you could have any Elm type you wanted to represent the same concept that these JSON objects represent, what would you give yourself?

In my experience, when you’re thinking about problems like this it’s much more fruitful to work backwards from the Elm type you’d like and derive the decoder from that than it is to try to work forwards from the JSON you want to decode.

[joelq]

@jacob

In my experience, when you’re thinking about problems like this it’s much more fruitful to work backwards from the Elm type you’d like and derive the decoder from that than it is to try to work forwards from the JSON you want to decode

Strongly agree with @jacob . It’s much easier to figure out decoders once you’ve defined the problem as “I have this JSON and I want to decode it into this type”

In this situation, you’ll have to resolve a lot of the ambiguity just to design types that capture the information you need. Is everything going to be stored in some super-generic tree or graph structure? Perhaps you want to force everything into one of n predefined record shapes? Something else altogether?

designing these types is the hardest part. Once you have these it makes figuring the decoders much more straightforward. Generally some combination of andThen and oneOf (with a bit of keyValuePairs thrown in if you need the names of keys)