How to filter a JSON decoded list to remove "impossible" elements?

[francois]

Hi!

I use PostgREST on the backend, and I have a two-level relationship that I return from an RPC function:

party has many addresses

The structure I’d like to return is:

{"name":"Mac and sons",
 "kind":"customer",
 "slug":"mac-and-sons",
 "addresses":[
   {"name":"principal",
    "address":"500 some St"}
   ]
}

I created an RPC because not all parties have addresses, hence I need a LEFT JOIN, which PostgREST does not support in the URL query language.

The issue I have is when a party doesn’t have any addresses. In that case, the returned JSON looks like this:

{"name":"Mac and sons",
 "kind":"customer",
 "slug":"mac-and-sons",
 "addresses":[null]
}

Notice the stray null in the addresses. This is due to my use of an array_agg in the backend query in PostgreSQL. array_agg is defined to return an array with a single null element in it when there are no rows to aggregate.

I know that I can probably find a way around this in PostgreSQL itself, but because I’m a beginner Elm coder, I’d like to solve the problem in Elm-land. My types and decoders look like this:

type alias PartyAddress =
    { name : String
    , address : String
    }


type alias FullParty =
    { slug : Slug
    , name : String
    , kind : PartyKind
    , addresses : List PartyAddress
    }

partyKindDecoder =
    let
        mapper s =
            case s of
                "customer" ->
                    Customer

                "supplier" ->
                    Supplier

                "group" ->
                    Group

                "troop" ->
                    Troop

                _ ->
                    Customer
    in
    Decode.map mapper Decode.string


partyAddressDecoder =
    Decode.map2 PartyAddress
        (Decode.field "name" Decode.string)
        (Decode.field "address" Decode.string)


completePartyDecoder =
    Decode.map4 FullParty
        (Decode.field "slug" Decode.string)
        (Decode.field "name" Decode.string)
        (Decode.field "kind" partyKindDecoder)
        (Decode.field "addresses" (Decode.list partyAddressDecoder))


getParty : Maybe Token -> Slug -> Cmd Msg
getParty maybeToken slug =
    Http.request
        { method = "GET"
        , url = Builder.absolute [ "api", "rpc", "edit_party" ] [ Builder.string "slug" slug ]
        , body = Http.emptyBody
        , expect = Http.expectJson CompletePartyLoaded completePartyDecoder
        , headers = buildHeadersForOne maybeToken
        , timeout = Nothing
        , tracker = Nothing
        }

Elm correctly complains that it can’t parse the addresses field:

Problem with the value at json.addresses[0]:

    null

Expecting an OBJECT with a field named `name`

Json.Decode.list has signature Decoder a -> Decoder (List a). It looks like I need to write a function like Decoder (List a) -> Decoder (List a), but I don’t know how to do that. Looking at the signature of andThen, (a -> Decoder b) -> Decoder a -> Decoder b, it looks like I should be using it, but I don’t know how to assemble the pieces to make Elm do what I want.

Elm - How to decode a json list of objects into a Dict seems tantalizingly close, is a more complex example, but I don’t have the expertise to understand exactly how it works. I see how the pieces are assembled, but not how to build such pieces myself.

JSON decoding in Elm, explained step by step has an example (near the bottom) where the 1st parameter of map3 is replaced by a function, and work can be done within the function.

Given this article, I tried the following:

fullPartyAssembler : Slug -> String -> PartyKind -> List PartyAddress -> FullParty
fullPartyAssembler slug name kind addresses =
    FullParty
        { slug = slug
        , name = name
        , kind = kind
        , addresses = List.filter (\x -> x) addresses
        }


completePartyDecoder =
    Decode.map4 fullPartyAssembler
        (Decode.field "slug" Decode.string)
        (Decode.field "name" Decode.string)
        (Decode.field "kind" partyKindDecoder)
        (Decode.field "addresses" (Decode.list partyAddressDecoder))

but this gives me errors:

Something is off with the body of the `fullPartyAssembler` definition:

859|>    FullParty
860|>        { slug = slug
861|>        , name = name
862|>        , kind = kind
863|>        , addresses = List.filter (\x -> x) addresses
864|>        }

This `FullParty` call produces:

    String -> PartyKind -> List PartyAddress -> FullParty

But the type annotation on `fullPartyAssembler` says it should be:

    FullParty

What would be the correct way to implement such a filter?

[lydell]

Hi!

When you write type alias FullParty = { ... you not only get a shorter name for a record, FullParty, you also get a function that can create such a record for free! (This only happens when you make an alias for a record, not other types.)

That bonus function is also called FullParty. It lets your write FullParty someSlug "John" Customer [] instead of { slug = someSlug, name = "John", kind = Customer, addresses = [] } if you want to. They both mean the same thing.

So when you write FullParty { slug = slug , name = name , kind = kind , addresses = List.filter (\x -> x) addresses }, you give the bonus function FullParty a record as a single parameter, but it requires a Slug as the first parameter (and then 3 more parameters). You need to remove the word FullParty in front of that record, which already is a FullParty (the type alias).

Now let’s take a look at addresses = List.filter (\x -> x) addresses. In JavaScript, you can do someArray.map(x => x) to get rid of undefined and null (and empty strings, false, 0 and other falsy things), but not in Elm. List.filter requires that the function you give it looks like this: a -> Bool. In other words, you have to return a boolean. Since your function is \x -> x, that means that the addresses variable must be a list of booleans! But that’s not what we intended – it should be a list of addresses of course.

If you remove that non-functioning filter, we end up with:

fullPartyAssembler : Slug -> String -> PartyKind -> List PartyAddress -> FullParty
fullPartyAssembler slug name kind addresses =
    { slug = slug
    , name = name
    , kind = kind
    , addresses = addresses
    }

Now, fullPartyAssembler is a function identical to the bonus function FullParty! Which means that in completePartyDecoder you can change from Decode.map4 fullPartyAssembler to Decode.map4 FullParty. And now we’re back with your original example.

So, what can we do? Well, as far as I understand, the "addresses" field in the JSON is either a list of objects, or a list with a single null in it. We can use Decode.oneOf to try both of those possibilities!

Decode.oneOf
    [ Decode.list (Decode.null ()) |> Decode.map (\_ -> [])
    , Decode.list partyAddressDecoder
    ]

Decode.list (Decode.null ()) turns [null] in JSON into [()] in Elm (a list of “unit”). |> Decode.map (\_ -> []) throws that [()] away, and pretends that we just got an empty list instead.

Another way of doing it is to allow mixing null and objects in the array. That can look like this:

Decode.list (Decode.nullable partyAddressDecoder)
    |> Decode.map (List.filterMap identity)

Decode.nullable allows null. null is turned into Nothing. For the objects, you get Just PartyAddress. So now we just need to get rid of the Nothings! That’s what |> Decode.map (List.filterMap identity) is doing. List.filterMap (\x -> x) someList is basically someArray.filter(x => x) in JavaScript – at least that’s how I like to think about it sometimes.

Finally, as a side note: It’s more convential to have your decoders fail for unknown values rather than defaulting to something. You could write your partyKindDecoder like so:

partyKindDecoder : Decoder PartyKind
partyKindDecoder =
    Decode.string
        |> Decode.andThen
            (\kind ->
                case kind of
                    "customer" ->
                        Decode.succeed Customer

                    "supplier" ->
                        Decode.succeed Supplier

                    "group" ->
                        Decode.succeed Group

                    "troop" ->
                        Decode.succeed Troop

                    _ ->
                        Decode.fail ("Unknown PartyKind: " ++ kind)
            )

Hope this helps!

[francois]

Whoa, excellent writeup! Thank you very much, @lydell, for the detailed explanations.

I ended up using oneOf in my implementation. That works just fine for me. My difficulty is understanding how/when Decode.map can be used. It looks like I get a concrete value in Decode.map, and I can probably dig my way out of my next problem with it

Thanks also for the partyKindDecoder recommendation. That too is going in the code immediately!

Have a great day!
François

[lydell]

If you have a Decoder String, use |> Decode.map (\hereComesThatStringAsAParam -> 3.14) to access the string and return whatever you want (in this case, you’ll end up with a Decoder Float). If that conversion can fail, use Decode.andThen instead. |> Decode.andThen (\hereComesThatStringAsAParam -> Decode.succeed 3.14) means the same thing (but there’s no reason to use andThen if something can’t fail).

map and andThen work the same way for Decoder, Maybe, Result, etc.

• map: Let’s you swap the value “inside the box”.
• andThen: Same thing, but you have to return “a new box”. You can also think of andThen as “map, then flatten” or “flatMap”.

Neither map nor andThen do anything if your Decoder ends up failing, or your Maybe becomes a Nothing or your Result becomes an Err.