Is it possible to capture type information about records?

I’m trying to automate some code generation for some rather complex data structures.

I was wondering if I can capture the type information about a record in such a way as to be able to use it to generate the derivate data structures.

All the derivate data structures are uniform over the type data that I’m trying to capture. (one-to-one correspondence).

In order to simplify the use case, imagine unifying encoding and decoding.

So, I would need some kind of Meta a data structure that I could then use like this:

encode : Meta a -> a -> Value 
encode meta = 
    ...

decoder : Meta a -> Decoder a 
decoder meta = 
    ...

Now, imagine a record like


type alias Record = 
    { foo : String 
    , bar : Int 
    }

Can I declare Meta Record somehow in Elm?

Please note that I know how to do this the hard way by declaring (in this case) Meta as two functions that would do the required functionality but this is what I’m trying to avoid. What I want is one data structure that could be used to derive the two functions needed in this case (one for producing the decoder and one for producing the encoder).

The requirements data structure I have looks like this:

type alias Record = 
    { complex : Complex 
    , lots : String 
    , of_ : Int 
    , fields : List Foo
    }



type Complex = 
    | First FirstRecord 
    | Second SecondRecord 
    ...

Where the FirstRecord and SecondRecord have distinct shapes and there are around 10 tags in Complex

I also need to capture at least 4 operations that are uniform over the type structure. In other languages I would have just derived the operations by iterating over the fields and using the type info for the field to produce the derived value.

2 Likes

Hi Peter,

I have also tried to unify encoders/decoders by working off of a meta-model of record types.

Take a look at this: https://github.com/rupertlssmith/json-schema-builder

Under the /test folder you can find some examples of using it. For example the code to derive an encoder and decoder for the same record type looks identical:

objectSimpleFieldsDecoder =
    object ObjectSimpleFields
        |> with (field "a" .a string)
        |> with (field "b" .b integer)
        |> with (field "c" .c number)
        |> with (field "d" .d boolean)
        |> build

objectSimpleFieldsEncoder =
    object ObjectSimpleFields
        |> with (field "a" .a string)
        |> with (field "b" .b integer)
        |> with (field "c" .c number)
        |> with (field "d" .d boolean)
        |> build

However, the types of the builder functions in each case do not unify, so I could not yet derive the encoder and decoder from a single meta-model.

It is a work in progress, and as to answering the question of whether or not it is possible to unify encoder/decoder in this way, I do not yet know the answer. I tried to do this, but found that as encoder and decoder are kind of ‘inside out’ relative to each other, the meta data structure needed to derive both is necessarily different - leading to the non-unifying types of the builder functions.

To try and fix that, I came up with the idea of doing it in 2 steps, so have some kind of higher level ‘builder’ that describes the meta-data model, then from that derive intermediate models suitable for encoders/decoders (or json schema, or decoders with extra checking against a schema, or XML encoder/decoders and so on).

I have a gut feeling that will work out. It gets complex, and writing this code was hard going, so I just parked the project for now. Hopefully, it can provide you with some inspiration.

Actually, it’s not that hard to create a Meta object describing encoding and decoding from a single structure.

Example:

module Main exposing (main)

import Html exposing (Html)
import Json.Decode as JD exposing (Decoder)
import Json.Encode as JE exposing (Value)


type alias ObjectSimpleFields =
    { a : String
    , b : Int
    , c : Float
    , d : Bool
    }


objectSimpleFieldsMeta : Meta ObjectSimpleFields
objectSimpleFieldsMeta =
    object ObjectSimpleFields
        |> field "a" .a string
        |> field "b" .b int
        |> field "c" .c float
        |> field "d" .d bool
        |> build


main : Html msg
main =
    let
        (Meta { encode, decode }) =
            objectSimpleFieldsMeta

        osf =
            { a = "a", b = 0, c = -1.3, d = False }

        encoded =
            encode osf

        decoded =
            JD.decodeValue decode encoded
    in
    Html.text <|
        String.join " "
            [ "Encoded:"
            , toString encoded
            , "decoded:"
            , toString decoded
            ]


build : ObjectMeta a a -> Meta a
build (ObjectMeta ometa) =
    Meta
        { encode = JE.object << ometa.encode
        , decode = ometa.decode
        }


string : Meta String
string =
    Meta
        { encode = JE.string
        , decode = JD.string
        }


int : Meta Int
int =
    Meta
        { encode = JE.int
        , decode = JD.int
        }


float : Meta Float
float =
    Meta
        { encode = JE.float
        , decode = JD.float
        }


bool : Meta Bool
bool =
    Meta
        { encode = JE.bool
        , decode = JD.bool
        }


field : String -> (a -> f) -> Meta f -> ObjectMeta a (f -> b) -> ObjectMeta a b
field name getter (Meta meta) (ObjectMeta ometa) =
    ObjectMeta
        { encode = \v -> ( name, meta.encode <| getter v ) :: ometa.encode v
        , decode = JD.map2 (\f x -> f x) ometa.decode (JD.field name meta.decode)
        }


type Meta a
    = Meta
        { encode : a -> Value
        , decode : Decoder a
        }


type ObjectMeta a b
    = ObjectMeta
        { encode : a -> List ( String, Value )
        , decode : Decoder b
        }


object : b -> ObjectMeta a b
object ctor =
    ObjectMeta
        { encode = always []
        , decode = JD.succeed ctor
        }
5 Likes

Nice effort @miniBill! I could not get my head around how to do that.

Can you extend this Meta object to cover not just Records, but also tagged union types, records with recursive structures, and Lists, Sets and Dicts?

Thank you for this example.

I will have to look at it in the morning with a fresh mind in order to understand it.

I should probably publish this as a library. The API is not perfect but it’s reasonable, and it’s quite a compact way to define encoding and decoding.

I would really love some feedback.

Ellie: https://ellie-app.com/8hS7BwxpJa1/0

module Main exposing (main)

import Dict exposing (Dict)
import Html exposing (Html)
import Json.Decode as JD exposing (Decoder)
import Json.Encode as JE exposing (Value)
import Set exposing (Set)


type alias ObjectSimpleFields =
    { a : String
    , b : Int
    , c : List Float
    , d : MyList Bool
    }


type MyList a
    = Empty
    | Cons a (MyList a)


objectSimpleFieldsMeta : Meta ObjectSimpleFields
objectSimpleFieldsMeta =
    object ObjectSimpleFields
        |> field "a" .a string
        |> field "b" .b int
        |> field "c" .c (list float)
        |> field "d" .d (mylist bool)
        |> build


mylist : Meta a -> Meta (MyList a)
mylist e =
    let
        matchEmpty e =
            case e of
                Empty ->
                    Just ()

                _ ->
                    Nothing

        matchCons e =
            case e of
                Cons x xs ->
                    Just ( x, xs )

                _ ->
                    Nothing
    in
    adt
        [ alternative0 matchEmpty Empty
        , alternative2 matchCons Cons e (lazy (\_ -> mylist e))
        ]


exampleObject : ObjectSimpleFields
exampleObject =
    { a = "a"
    , b = 0
    , c = [ -1.3 ]
    , d = Cons False Empty
    }


main : Html msg
main =
    let
        (Meta { encode, decode }) =
            objectSimpleFieldsMeta

        encoded =
            encode exampleObject

        decoded =
            JD.decodeValue decode encoded
    in
    Html.span []
        [ Html.text "Encoded: "
        , Html.text <| toString encoded
        , Html.text " decoded: "
        , case decoded of
            Ok v ->
                Html.text <| toString v

            Err e ->
                String.split "\n" e
                    |> List.map Html.text
                    |> List.intersperse (Html.br [] [])
                    |> Html.span []
        ]


type Meta a
    = Meta
        { encode : a -> Value
        , decode : Decoder a
        }


string : Meta String
string =
    Meta
        { encode = JE.string
        , decode = JD.string
        }


int : Meta Int
int =
    Meta
        { encode = JE.int
        , decode = JD.int
        }


float : Meta Float
float =
    Meta
        { encode = JE.float
        , decode = JD.float
        }


bool : Meta Bool
bool =
    Meta
        { encode = JE.bool
        , decode = JD.bool
        }


list : Meta a -> Meta (List a)
list =
    bimap
        (\e -> JE.list << List.map e)
        JD.list


dict : Meta a -> Meta (Dict String a)
dict =
    bimap
        (\e -> JE.object << Dict.toList << Dict.map (always e))
        JD.dict


set : Meta comparable -> Meta (Set comparable)
set =
    bimap
        (\e -> JE.list << List.map e << Set.toList)
        (JD.map Set.fromList << JD.list)


bimap : ((a -> Value) -> b -> Value) -> (Decoder a -> Decoder b) -> Meta a -> Meta b
bimap menc mdec (Meta meta) =
    Meta
        { encode = menc meta.encode
        , decode = mdec meta.decode
        }


type ObjectMeta a b
    = ObjectMeta
        { encode : a -> List ( String, Value )
        , decode : Decoder b
        }


object : b -> ObjectMeta a b
object ctor =
    ObjectMeta
        { encode = always []
        , decode = JD.succeed ctor
        }


field : String -> (a -> f) -> Meta f -> ObjectMeta a (f -> b) -> ObjectMeta a b
field name getter (Meta meta) (ObjectMeta ometa) =
    ObjectMeta
        { encode = \v -> ( name, meta.encode <| getter v ) :: ometa.encode v
        , decode = JD.map2 (\f x -> f x) ometa.decode (JD.field name meta.decode)
        }


build : ObjectMeta a a -> Meta a
build (ObjectMeta ometa) =
    Meta
        { encode = JE.object << ometa.encode
        , decode = ometa.decode
        }


type AlternativeMeta a
    = AlternativeMeta
        { encode : a -> Maybe (List Value)
        , decode : Decoder a
        }


apply : Decoder a -> Decoder (a -> b) -> Decoder b
apply =
    JD.map2 (|>)


alternative : (a -> Maybe b) -> (b -> List Value) -> Decoder a -> AlternativeMeta a
alternative match toList ctor =
    AlternativeMeta
        { encode = match >> Maybe.map toList
        , decode = ctor
        }


alternative0 : (a -> Maybe ()) -> a -> AlternativeMeta a
alternative0 match ctor =
    alternative match (always []) (JD.succeed ctor)


alternative1 : (a -> Maybe b) -> (b -> a) -> Meta b -> AlternativeMeta a
alternative1 match ctor (Meta p1) =
    alternative
        match
        (\v1 -> [ p1.encode v1 ])
        (JD.succeed ctor
            |> apply (JD.field "0" p1.decode)
        )


alternative2 : (a -> Maybe ( b, c )) -> (b -> c -> a) -> Meta b -> Meta c -> AlternativeMeta a
alternative2 match ctor (Meta p1) (Meta p2) =
    alternative
        match
        (\( v1, v2 ) -> [ p1.encode v1, p2.encode v2 ])
        (JD.succeed ctor
            |> apply (JD.field "0" p1.decode)
            |> apply (JD.field "1" p2.decode)
        )


adt : List (AlternativeMeta a) -> Meta a
adt alternatives =
    let
        maybeEncode : a -> Int -> AlternativeMeta a -> Maybe ( Int, List Value )
        maybeEncode e i (AlternativeMeta { encode }) =
            encode e |> Maybe.map (\v -> ( i, v ))

        aencode e =
            alternatives
                |> List.indexedMap (maybeEncode e)
                |> List.filterMap identity
                |> List.head
                |> Maybe.map
                    (\( i, v ) ->
                        JE.object
                            [ ( "tag", JE.int i )
                            , ( "values", JE.list v )
                            ]
                    )
                |> Maybe.withDefault JE.null

        adecode =
            JD.field "tag" JD.int
                |> JD.andThen
                    (\tag ->
                        case List.drop tag alternatives of
                            [] ->
                                JD.fail <| "Wrong tag, got " ++ toString tag

                            (AlternativeMeta { decode }) :: _ ->
                                JD.field "values" decode
                    )
    in
    Meta
        { encode = aencode
        , decode = adecode
        }


lazy : (() -> Meta a) -> Meta a
lazy f =
    Meta
        { encode =
            \v ->
                let
                    (Meta { encode }) =
                        f ()
                in
                encode v
        , decode =
            JD.lazy
                (\l ->
                    let
                        (Meta { decode }) =
                            f l
                    in
                    decode
                )
        }
2 Likes

This is progressing quite rapidly. I guess you could also write an example for a record containing a record.

I’m probably missing something, but:

type alias InceptionInner =
    { a : String
    , b : List Inception
    }


type Inception
    = Inception InceptionInner


inceptionMeta : Meta Inception
inceptionMeta =
    let
        unpack (Inception i) =
            i

        inner =
            object InceptionInner
                |> field "a" .a string
                |> field "b" .b (list <| lazy (\_ -> inceptionMeta))
                |> build
    in
    bimap
        (\e -> unpack >> e)
        (JD.map Inception)
        inner

not really pretty tbh

Ok, found a way to make it a tad better:

type Inception
    = Inception
        { a : String
        , b : List Inception
        }


inceptionMeta : Meta Inception
inceptionMeta =
    object (\a b -> { a = a, b = b })
        |> field "a" .a string
        |> field "b" .b (list <| lazy (\_ -> inceptionMeta))
        |> build
        |> newtype (\(Inception i) -> i) Inception

newtype : (a -> i) -> (i -> a) -> Meta i -> Meta a
newtype focus defocus =
    bimap (\e -> focus >> e) (JD.map defocus)

Is there interest in the community in having this as a library?

1 Like

Ok, I’ve improved the adt case a lot (for example it’s now much harder to leave out cases!)

type Adt l r
    = Left
    | Middle l
    | Right l r


adtCata : o -> (l -> o) -> (l -> r -> o) -> Adt l r -> o
adtCata lk mk rk v =
    case v of
        Left ->
            lk

        Middle l ->
            mk l

        Right l r ->
            rk l r


adtMeta : Meta l -> Meta r -> Meta (Adt l r)
adtMeta l r =
    adt adtCata
        |> alternative0 Left
        |> alternative1 Middle l
        |> alternative2 Right l r
        |> buildAdt

the magic sauce is:

adt : a -> AdtMeta a b
adt cata =
    AdtMeta
        { cata = cata
        , atag = 0
        , decode = always identity
        }


alternative :
    ((Value -> Value) -> a)
    -> Decoder v
    -> AdtMeta (a -> b) v
    -> AdtMeta b v
alternative x y (AdtMeta p) =
    AdtMeta
        { atag = p.atag + 1
        , cata =
            p.cata <|
                x <|
                    \v ->
                        JE.object
                            [ ( "tag", JE.int p.atag )
                            , ( "value", v )
                            ]
        , decode =
            \tag orElse ->
                if tag == p.atag then
                    y
                else
                    p.decode tag orElse
        }


alternative0 : v -> AdtMeta (Value -> a) v -> AdtMeta a v
alternative0 ctor =
    alternative
        (\c -> c <| JE.object [])
        (JD.succeed ctor)


alternative1 : (a -> y) -> Meta a -> AdtMeta ((a -> Value) -> b) y -> AdtMeta b y
alternative1 ctor (Meta m1) =
    alternative
        (\c v -> c <| JE.object [ ( "0", m1.encode v ) ])
        (JD.map ctor m1.decode)


alternative2 :
    (a -> b -> v)
    -> Meta a
    -> Meta b
    -> AdtMeta ((a -> b -> Value) -> c) v
    -> AdtMeta c v
alternative2 ctor (Meta m1) (Meta m2) =
    alternative
        (\c v1 v2 -> c <| JE.object [ ( "0", m1.encode v1 ), ( "1", m2.encode v2 ) ])
        (JD.map2 ctor (JD.field "0" m1.decode) (JD.field "1" m2.decode))


buildAdt : AdtMeta (a -> Value) a -> Meta a
buildAdt (AdtMeta { cata, decode }) =
    Meta
        { encode = cata
        , decode =
            JD.field "tag" JD.int
                |> JD.andThen
                    (\tag ->
                        let
                            error =
                                "tag " ++ toString tag ++ "did not match"
                        in
                        JD.field "value" <| decode tag <| JD.fail error
                    )
        }


type AdtMeta p v
    = AdtMeta
        { atag : Int
        , cata : p
        , decode : Int -> Decoder v -> Decoder v
        }

As @ilias pointed out in Slack you should also check out jamesmacaulay/elm-json-bidirectional and prozacchiwawa/elm-json-codec.

1 Like

As you mentioned on Slack, elm-json-bidirectional and elm-json-codec do not have an intermediate ADT. I think the ADT approach is potentially better in that it could be a lot more flexible.

For example, you can map the ADT to encoder + decoder - you could also map the ADT to a json schema, or a UI that renders an input form, or some code that builds a graphql query and so on.

Do you have all of the code for this in one place? Its a bit hard to piece together from the snippets given above, and I would really like to try it out.

Actually my code only builds the encoder/decoder pair, but it could be easily modified to also provide a representation of the data (dunno about recursive paths tho).

What I meant to say on Slack is that those two packages have no facility for encoding/decoding of union types (they can handle them just fine but you have to provide your own pairs)

Ellie with the complete code (also added tuples): https://ellie-app.com/8hS7BwxpJa1/2

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.