# How to write a generic JSON decoder?

**URL:** <https://discourse.elm-lang.org/t/how-to-write-a-generic-json-decoder/3179>\
**Category:** Learn\
**Created:** [February 20, 2019, 2:03pm UTC](https://discourse.elm-lang.org/t/how-to-write-a-generic-json-decoder/3179 "2019-02-20T14:03:08Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![rupert](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/rupert/32/1775_2.png) [@rupert](https://discourse.elm-lang.org/u/rupert)\
**Post date:** [February 20, 2019, 2:03pm UTC](https://discourse.elm-lang.org/t/how-to-write-a-generic-json-decoder/3179/1 "2019-02-20T14:03:08Z")

</div>

Some data formats are extensible, in addition to a fixed set of fields they may also allow additional, not known in advance fields to be included in the JSON.

The case I am looking at just now is the Open API spec, which allows extensions in the form of fields starting with `x-`. So if you had some record type definition, and for code generation purposes you could specify what type something is to be mapped to in the target language with an extra field like this: `x-elm-type: String`. There can be any number of these additional `x-` fields for any purpose as a general mechanism for making the spec extensible - they can also have values that are JSON objects or arrays.

So my question is, how do you write a Json decoder that can handle any valid Json, decoding it into a Dict?

My first attempt was to use the `Json.Decode.dict` function, and try and decode things as Strings. The example given in its docs is:

```auto
decodeString (dict int) "{ \"alice\": 42, \"bob\": 99 }"
  == Ok (Dict.fromList [("alice", 42), ("bob", 99)])

```

but if I change that to:

```auto
decodeString (dict string) "{ \"alice\": 42, \"bob\": 99 }"
  == Ok (Dict.fromList [("alice", 42), ("bob", 99)])

```

it fails, because `Json.Decode.string` won’t allow 42 as a `String`.

Perhaps I can try all the possibilities and combine them with `Json.Decode.oneOf`?

---

<div class="post-metadata">

**Author:** ![jacob](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/jacob/32/108_2.png) [@jacob](https://discourse.elm-lang.org/u/jacob)\
**Post date:** [February 20, 2019, 2:26pm UTC](https://discourse.elm-lang.org/t/how-to-write-a-generic-json-decoder/3179/2 "2019-02-20T14:26:25Z")

</div>

With decoders, I find that it’s impossible to give good advice without knowing both the format coming in and what you want to do with the result in Elm. Ultimately a decoder is about mapping a concept in JSON to a concept in your Elm program’s data structures, and it depends on both of those two ends equally.

For instance, if all you’re going to do with these extended fields is remember them and reencode them as JSON later, you can use `Json.Decode.value` to get the fields as unparsed JSON blobs. You can do that with

```auto
decodeDict : Json.Decode.Decoder (Dict String Json.Decode.Value)
decodeDict = Json.Decode.dict Json.Decode.value 

```

If you want to _use_ the values in Elm, though, this doesn’t help very much. You could try to decode them into some completely generic JSON elm data structure, but that’s unlikely to be of much help (I wrote [an article about that](https://dev.to/matthewsj/you-could-have-designed-the-jsondecode-library-2d8) a while back).

So I guess my question is: Suppose you could do anything you wanted with your decoder. What would its type be and how would you use it in Elm?

---

<div class="post-metadata">

**Author:** ![rupert](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/rupert/32/1775_2.png) [@rupert](https://discourse.elm-lang.org/u/rupert)\
**Post date:** [February 20, 2019, 2:42pm UTC](https://discourse.elm-lang.org/t/how-to-write-a-generic-json-decoder/3179/3 "2019-02-20T14:42:09Z")

</div>

Thanks - I need to specify the problem a bit more clearly.

Suppose this is the JSON I was trying to decode:

```
{
   name: "rupert",
   age: 42,
   x-superpower: "teleportation"
}

```

I want to decode the `{ name : String, age : Int }` part which will always be the same and known in advance. I also want to decode the `x-superpower` part, which is optional and can be any number of additional `x-` fields of any type.

Those `x-` fields are perhaps not so useful, since they do not map to static types in my Elm program, it is true. In the first instance all I may end up doing with them is displaying them in some UI - so for that reason at least I would like to capture them, what their names are and what their values are.

---

<div class="post-metadata">

**Author:** ![jacob](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/jacob/32/108_2.png) [@jacob](https://discourse.elm-lang.org/u/jacob)\
**Post date:** [February 20, 2019, 4:17pm UTC](https://discourse.elm-lang.org/t/how-to-write-a-generic-json-decoder/3179/4 "2019-02-20T16:17:48Z")

</div>

In that case I think I’d try using the `decodeDict` decoder I wrote above and filtering the result so that the only keys are the `x-*` fields. Out of the box Elm doesn’t have good support for formatting JSON values as strings, but it looks like you can use [https://package.elm-lang.org/packages/ThinkAlexandria/elm-pretty-print-json/latest/](https://package.elm-lang.org/packages/ThinkAlexandria/elm-pretty-print-json/latest/) for that.

---

<div class="post-metadata">

**Author:** ![rupert](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/rupert/32/1775_2.png) [@rupert](https://discourse.elm-lang.org/u/rupert)\
**Post date:** [February 20, 2019, 4:42pm UTC](https://discourse.elm-lang.org/t/how-to-write-a-generic-json-decoder/3179/5 "2019-02-20T16:42:46Z")

</div>

Thanks, that gives me an idea.

Use a generic `Dict` decoder that lifts out the `x-` fields, and also a specific `Decoder` for the fixed format fields that are expected and ignores any `x-` fields. I can then combine those together to get what I am after using `Decode.map2` - gives me a direction to get started with anyway.

---

<div class="post-metadata">

**Author:** ![pdamoc](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/pdamoc/32/36_2.png) [@pdamoc](https://discourse.elm-lang.org/u/pdamoc)\
**Post date:** [February 21, 2019, 8:44am UTC](https://discourse.elm-lang.org/t/how-to-write-a-generic-json-decoder/3179/6 "2019-02-21T08:44:13Z")

</div>

What you can do is lift all the fields to strings.

```auto
module Main exposing (main)
 
import Html exposing (text)
import Json.Decode as Json 
import Dict

fromBool val = 
    if val then "True" else "False"

asString = 
    Json.oneOf 
        [ Json.string 
        , Json.map String.fromInt Json.int
        , Json.map String.fromFloat Json.float
        , Json.map fromBool Json.bool
        , Json.null ""
        ]

myDecoder = 
    Json.map3 (\name age powers -> {name=name, age = age, powers = powers} )
        (Json.field "name" Json.string)
        (Json.field "age" Json.int)
        powersDecoder
        
 
powersDecoder = 
    Json.map (Dict.filter (\k _ -> not (List.member k ["name", "age"]))) (Json.dict asString)  
 
 
inputJson = """
{
   "name": "rupert",
   "age": 42,
   "x-superpower": "teleportation",
   "x-king": true,
   "x-op": 9001
}
"""

main =
     text <| Debug.toString <| Json.decodeString myDecoder inputJson 

```

[link to Ellie](https://ellie-app.com/4MXCqnttprRa1)

Of course, this would not work for collection fields (arrays, objects) but you can extend `asString` to cover those cases if you need to.

---

<div class="post-metadata">

**Author:** ![thought2](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/thought2/32/2478_2.png) [@thought2](https://discourse.elm-lang.org/u/thought2)\
**Post date:** [February 21, 2019, 9:56am UTC](https://discourse.elm-lang.org/t/how-to-write-a-generic-json-decoder/3179/7 "2019-02-21T09:56:48Z")

</div>

This looks good to me! I’d maybe use this filter function instead: `(\key _ -> String.startsWith "x-" key)`

---

<div class="post-metadata">

**Author:** ![pdamoc](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/pdamoc/32/36_2.png) [@pdamoc](https://discourse.elm-lang.org/u/pdamoc)\
**Post date:** [February 21, 2019, 10:31am UTC](https://discourse.elm-lang.org/t/how-to-write-a-generic-json-decoder/3179/8 "2019-02-21T10:31:46Z")

</div>

> [@thought2](#):
>
> I’d maybe use this filter function instead: `(\key _ -> String.startsWith "x-" key)`

This would work too if all the other fields have a predictable prefix. The solution I proposed has the advantage that it would capture all fields other that a known set (`name`, `age`).

---

<div class="post-metadata">

**Author:** ![rupert](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/rupert/32/1775_2.png) [@rupert](https://discourse.elm-lang.org/u/rupert)\
**Post date:** [February 21, 2019, 10:39am UTC](https://discourse.elm-lang.org/t/how-to-write-a-generic-json-decoder/3179/9 "2019-02-21T10:39:44Z")

</div>

Here is what I did so far to decode to the data structure from Jacob’s article:

```
import Dict exposing (Dict)
import Json.Decode as Decode exposing (Decoder)

type Json
    = JString String
    | JBool Bool
    | JInt Int
    | JFloat Float
    | JNull
    | JObj (Dict String Json)
    | JArr (List Json)

string : Decoder Json
string =
    Decode.map JString Decode.string

bool : Decoder Json
bool =
    Decode.map JBool Decode.bool

int : Decoder Json
int =
    Decode.map JInt Decode.int

float : Decoder Json
float =
    Decode.map JFloat Decode.float

null : Decoder Json
null =
    Decode.null JNull

array : Decoder Json
array =
    Decode.list generic
        |> Decode.map JArr

dict : Decoder Json
dict =
    Decode.dict generic
        |> Decode.map JObj

generic : Decoder Json
generic =
    Decode.oneOf [bool, int, float, null, string, Decode.lazy (\_ -> array), Decode.lazy (\_ -> dict)]

```

Its recursive so had to make use of `Decode.lazy`. Seems to work nicely - just need to add the `Dict` filter and so on.

For OpenAPI, the extra fields always start with `x-`, so I can use that filter pattern.

---

<div class="post-metadata">

**Author:** ![rupert](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/rupert/32/1775_2.png) [@rupert](https://discourse.elm-lang.org/u/rupert)\
**Post date:** [February 21, 2019, 11:27am UTC](https://discourse.elm-lang.org/t/how-to-write-a-generic-json-decoder/3179/10 "2019-02-21T11:27:06Z")

</div>

> [@jacob](#):
>
> Out of the box Elm doesn’t have good support for formatting JSON values as strings, but it looks like you can use [https://package.elm-lang.org/packages/ThinkAlexandria/elm-pretty-print-json/latest/](https://package.elm-lang.org/packages/ThinkAlexandria/elm-pretty-print-json/latest/) for that.

I was intrigued to take a look at the pretty printer, as it also must do generic JSON decoding. It bypasses having a generic model to describe the JSON, and goes straight to building the pretty printable document. However, the structure is similar to what I came up with:

```
decodeDoc : Int -> Decoder Doc
decodeDoc indent =
    Decode.map
        nullToDoc
        (Decode.maybe
            (Decode.oneOf
                [ Decode.map stringToDoc Decode.string
                , Decode.map numberToDoc Decode.float
                , Decode.map boolToDoc Decode.bool
                , Decode.map (listToDoc indent) (Decode.lazy (\_ -> Decode.list (decodeDoc indent)))
                , Decode.map (objectToDoc indent) (Decode.lazy (\_ -> Decode.keyValuePairs (decodeDoc indent)))
                ]
            )
        )

```

---

<div class="post-metadata">

**Author:** ![jacob](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/jacob/32/108_2.png) [@jacob](https://discourse.elm-lang.org/u/jacob)\
**Post date:** [February 21, 2019, 4:24pm UTC](https://discourse.elm-lang.org/t/how-to-write-a-generic-json-decoder/3179/11 "2019-02-21T16:24:58Z")

</div>

Interestingly, that article was a result of me doing this same exercise, and writing just about the same code as you did, while trying to figure out if I could make a better decoder API. Doing that work and then trying to use the resulting `Json` values really drove home for me the idea that a decoder isn’t simply bringing JSON into Elm, but bringing JSON into _your Elm program_. A converter of JSON into a generic `Json` data structure is pretty simple to write, but doesn’t buy you very much – you still need to turn `Json` into data structures your program understands, and that transformation is basically just as difficult as the original problem.

That’s why I now try to steer people away from generic decoding as a first step, and instead steer them towards deciding what they’d want the native Elm representation of the JSON to be if they could choose anything, and then writing a decoder directly from JSON to that.

---

<div class="post-metadata">

**Author:** ![ilias](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/ilias/32/6_2.png) [@ilias](https://discourse.elm-lang.org/u/ilias)\
**Post date:** [February 22, 2019, 8:11am UTC](https://discourse.elm-lang.org/t/how-to-write-a-generic-json-decoder/3179/12 "2019-02-22T08:11:12Z")

</div>

> [@jacob](#):
>
> Out of the box Elm doesn’t have good support for formatting JSON values as strings

Quick reminder to y’all that [`Json.Encode.encode : Int -> Json.Encode.Value -> String`](https://package.elm-lang.org/packages/elm/json/latest/Json-Encode#encode) exists 😃

---

<div class="post-metadata">

**Author:** ![rupert](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/rupert/32/1775_2.png) [@rupert](https://discourse.elm-lang.org/u/rupert)\
**Post date:** [February 22, 2019, 3:09pm UTC](https://discourse.elm-lang.org/t/how-to-write-a-generic-json-decoder/3179/13 "2019-02-22T15:09:57Z")

</div>

> [@jacob](#):
>
> Doing that work and then trying to use the resulting `Json` values really drove home for me the idea that a decoder isn’t simply bringing JSON into Elm, but bringing JSON into _your Elm program_ . A converter of JSON into a generic `Json` data structure is pretty simple to write, but doesn’t buy you very much – you still need to turn `Json` into data structures your program understands, and that transformation is basically just as difficult as the original problem.

I agree, and its a good observation.

There are a couple of situations where a generic decoder can be useful:

- If you wanted to validate an arbitrary json against an arbitrary json-schema - which of course has its own fixed schema, that you would decode and write a program around.
- If you wanted to write a UI to help a user understand and work with arbitrary JSON. Say to visualise it or search it.
- Programs that take JSON and try to infer its schema, or automatically map it to a data model. For example, [Main](https://noredink.github.io/json-to-elm/)

---

<div class="post-metadata">

**Author:** ![system](https://canada1.discourse-cdn.com/flex035/uploads/elm_lang/original/1X/50a05e53677a2c3b47776d7abd0f113eb50193a1.png) [@system](https://discourse.elm-lang.org/u/system)\
**Post date:** [March 4, 2019, 3:13pm UTC](https://discourse.elm-lang.org/t/how-to-write-a-generic-json-decoder/3179/14 "2019-03-04T15:13:56Z")

</div>

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.