Optional Key Records - Syntax Proposal


#1

Background

One of the biggest causes of confusion for users of dillonkearns/elm-graphql is the syntax for optional arguments. In fact, there is a blog post dedicated to teaching people how to understand and use optional arguments in dillonkearns/elm-graphql.

I’ll use specific examples from dillonkearns/elm-graphql in this post since that’s the context where I have the most experience with this problem. Specifically, I’ll use a somewhat simplified version for querying search in the Github GraphQL API. Here’s an example using the search endpoint with raw GraphQL, to make this post more concrete (you can run the query live here):

query {
  search(type: REPOSITORY, query: "language: Elm") {
    repositoryCount
  }
}

The Problem

Because there is no way to do a record with some fields missing in Elm, there are two ways to do default record values:

  1. Have a record with default values populated in each field. Then use the record update syntax to change these values.
type alias SearchOptionalArguments =
    { first : Maybe Int
    , after : Maybe String
    , last : Maybe Int
    , before : Maybe String
    }

searchOptionalDefaults : SearchOptionalArguments
searchOptionalDefaults =
    { first = Nothing, after = Nothing, last = Nothing, before = Nothing }

In dillonkearns/elm-graphql, code is generated for these defaults, but for other use cases it would have to be hand coded.

The two pain points with this syntax are 1) you have to import the right type and reference it like { MyModule.defaults | episode = NewHope }. I think it would be confusing and cumbersome for users to have to know to look for a record that is provided that is of the right record type (there are a lot of record types like this in dillonkearns/elm-graphql generated code, one for each set of optional arguments in a GraphQL API) 2) you currently can’t have a qualified reference in record update syntax, so you actually have to do this workaround

let
    defaults = Github.Query.searchOptionalDefaults
in
    { defaults | episode = NewHope }

This update syntax limitation is being tracked in this issue under “Record Suggestions”.

  1. This is the approach the dillonkearns/elm-graphql currently uses. Build
    a function which passes the default value to the user, allowing them to
    use the record update syntax to add optional fields:
Github.Query.search (\optionals -> { optionals | first = Just 100 }) { query = "Elm" }

Or use identity to add no optional fields.

Github.Query.search identity { query = "Elm" }

This solves the decoupling problem above, but has its own complexity. This trips a lot of people up. It can be challenging to understand how to use it and read the type signatures. Understanding how and why to use identity when you don’t want any optional arguments is confusing. And this also requires the optional and required arguments to be passed in separately ({ query = "Elm" } is the required argument in this example).

When I present type-safe GraphQL code in Elm as a cool example of the power of Elm, this is a detail I go to lengths to try to hide to avoid confusion.

A Possible Improvement

I think that an Optional Key Record syntax could improve this quite a bit. The syntax for creating Optional Key Records could look similar to an extensible record annotation.

So this code (without Optional Key Records):

Github.Query.search (\optionals -> { optionals | first = Just 100 }) { query = "Elm" }

Would turn into this (with Optional Key Records):

Github.Query.search { ? | first = Just 100, query = "Elm" }

The { ? | ... } explicitly indicates that missing values are filled in with Nothing. Type annotations for such records could use a similar syntax if they wanted to omit some of the Maybe fields in the record from the annotation, even though they might be passed to a function that depends on that Maybe field.

For example,

addLanguageToQuery : { searchArguments? | query : String } -> { searchArguments? | query : String }
addLanguageToQuery searchArguments =
    { searchArguments? | query = searchArguments.query ++ " language: Elm"  }

For reference, the full searchArguments type without an Optional Extensible Record notation would look like this:

type alias SearchArguments =
    { first : Maybe Int
    , after : Maybe String
    , last : Maybe Int
    , before : Maybe String
    , query : String
    }

I think that users would find writing their GraphQL queries in Elm much easier with this change. It would also make the GraphQL query code much more consistent and predictable (instead of some functions taking an optional arguments function, some taking a required arguments record, or a permutation of the two).

I’m curious to hear if there are other use cases that could be improved by an Optional Key Record syntax. And I’d love to hear thoughts on the idea or alternatives to the syntax. Thanks!


#2

First of all, thank you for this detailed write up! I happen to have myself been frustrated by some limitations on the record syntax on several occasion (knowing very well that Elm records are among the most delightful to work with out of any programming language I ever encountered, so there’s that).

However, I am absolutely not in favour of the Optional Key Record syntax proposed here.

Please feel free to correct me if I’m wrong but the proposal, as stated here, would basically mean promoting the Maybe a type to have special compiler support and I can’t personally endorse that. This would open a whole class of problems:

  • What if someone would like some other type to have a default empty value (basically, the mempty part of the Monoid type class ala Haskell) ?
  • What should be a proper default for a Maybe a ? It could be either be Nothing but it could also be the Just default assuming the a type defines a default of its own.
  • Shouldn’t we also define a default for lists ? That should be easy enough, it’s []. What about booleans ? It depends, it could be True it could be False depending on what you may want to do.

At that point, it becomes more interesting to define some kind of ad-hoc polymorphism mechanics (be it type classes or not) to allow users to define default or empty. Notice that we’re already in the same kind of debate than with the number, comparable and appendable magic type variables in Elm. And I think adding a magical defaultable in the same spirit would be the absolute worst out of several already bad solutions.

Evan already stated that record syntax should stay minimal (cf. here and here).

To be clear, I would wholeheartedly support some additions to record syntax to make them a little easier to work with. But I don’t think the propositions stated thus far are any good, and I am truly sorry that I don’t have any other one to contribute myself at that time.

Really interesting discussion though.

As a side-note, I find it interesting that such a request comes more or less at the same time than this one whose “quick and easy” solution also seems to be some kind of type-class mechanic (Enum / Bounded) and I look forward to read some out of the box possible ideas on the subject (I am in no way promoting addition of type-classes to Elm, but I do feel that some form of ad-hoc polymorphism would be beneficial in the long run).


#3

Have you tried an API similar to how https://package.elm-lang.org/packages/BrianHicks/elm-particle/latest/ does it? Maybe you can do something like:

let
    query =
        Github.Query.searchQuery { query = "Elm" }
            |> withFirst 100
in
    Github.Query.search query

I think it’s quite nice looking and seems easy to use :slight_smile:


#4

@Shump I was literally just typing such a thing up! I think that could work quite well. We can’t normally do this since we don’t have a syntax for setting record fields, but since this code is generated it 100% could! I guess the internals would look like:

type alias SearchRequired = { query : String }

type alias SearchArguments =
    { query : String
    , first : Optional Int
      -- removing the other optional arguments for demo purposes
    }

searchArguments : SearchRequired -> SearchArguments
searchArguments { query } =
    { query = query, first = Absent }

withFirst : Int -> SearchArguments -> SearchArguments
withFirst first args =
    { args | first = Present first }

withoutFirst : SearchArguments -> SearchArguments
withoutFirst args =
    { args | first = Absent }

-- similar thing for `null` argument, and for the rest of the fields.

I think you could make this so that multiple queries could accept the same arguments, as long as they’re of the same type, by accepting an open record:

withFirst : Int -> { args | first : Int } -> { args | first : Int }

or by modifying their names:

withSearchFirst : Int -> SearchArguments -> SearchArguments

#5

Thanks everyone for taking the time to share your feedback!

@Punie I don’t see this as a slippery slope situation. The purpose of the Maybe a type is that it allows you to express an absence of a value, and in fact one of the core things that’s used for is Maybe.withDefault! So if you were building a function for an API that allowed some Maybe Int or Maybe (List String) values, you could then turn absent (Nothing) values into their defaults easily:

withDefaultValues : { limit : Maybe Int } -> { limit : Int }
withDefaultValues values =
    { limit = values.limit |> Maybe.withDefault 20 }

I think that if this syntax were adopted, it should only support Nothing as the absent value, and then users can leverage the Maybe module to do what they want to to transform Just a or Nothing values.


#6

@Shump @brian, since there could be arguments of the same name with different types at the same level, I would have to use extensible records with a type variable here, which is quite a cool idea!

withId : idType -> { arguments | id = Maybe idType } -> { arguments | id = Maybe idType }
withId idValue arguments =
    { arguments | id = Just idValue }

I like this idea, but now the problem is we’re back where we started with the initial 2 problems I stated. We either need to “inject” a record with all optional fields filled in as Nothing (see the problems I listed with approach 2, which dillonkearns/elm-graphql is currently using). Or you need to grab the record from another module (see the problems I listed with approach 1).

So I don’t think this addresses those particular pain points. The problem I’m really trying to find a way to improve is how to decouple giving some specific optional values from having to inject the default values manually somehow (either with approach 1, passing the values in through a function and using record update on that, or approach 2, grabbing a record from another module and doing record update on that).

Note that I do use this approach to tacking on optional values in the dillonkearns/elm-graphql API, see the Graphql.Http.withHeader, for example, and I think it’s super slick! But the problem here is that you still need to define a starting point which includes explicit absent values, which is the problem I’d be really interested to find a way to improve.


#7

@dillonkearns I totally get that, and you’re right, it would make sense for Nothing to be the default.

However, I still am very uneasy on the topic of giving Maybe special compiler support and I would be very pleased if, somehow, we could converge to a different solution that does not requires it. In my mind, there are no special relations between records and Maybe and thus, adding a special rule that brings them together somehow, hardcoded into the compiler feels like a big no no.

I firmly believe it is worth exploring though, because such a solution could also potentially address a lot of other seemingly orthogonal pain points (such as the ones regarding comparables and other magical compiler trickery) that have been raised over the past few years.

In the meantime, I would totally support allowing fully qualified names in record update syntax. To this day I even had no idea it wasn’t possible :sweat_smile:

Oh, and on a side note: thank you so much for dillonkearns/elm-graphql! This library is absolutely brilliant and I can’t even begin to think working with GraphQL endpoints using something else.


#8

I don’t think this has the same problems. Here was my thought process, in a couple steps:

Github.Query.search identity { query = "lamp" }

You’re right! This is confusing! What if we got rid of identity?

Github.Query.search { query = "lamp" }

Nice, but now we can’t have optional arguments. Not so nice! Well, let’s look at the signature. Seems like it’ll be something like this (with SelectionSet and Field arguments elided for brevity.)

search : 
    (SearchOptionalArguments -> SearchOptionalArguments) ->
    SearchRequiredArguments ->
    SelectionSet ... ->
    Field ...

Hmm, we’ve already got a record for required arguments. It looks like { query : String }. What if that was the starting point to a builder?

searchArguments : { query : String } -> SearchArguments

search : SearchArguments -> SelectionSet ... -> Field ...

Important note: you don’t have to do the awkward record update syntax dance this way since searchArguments can compose the required and optional arguments together into a new field with all arguments including Absent defaults.

Ok, so now the query above would look like this:

search (searchArguments { query = "toast" })

Nice. Then since it’s a separate data structure, we can make a builder thing (with*):

withFirst : Int -> SearchArguments -> SearchArguments

withLast : Int -> SearchArguments -> SearchArguments

(with the polymorphism we’ve kind of talked about already, of course. I’m leaving that out, again for brevity.)

That’s where I stopped the first time I thought about it. It means the API for composing a query now would look like this:

search (searchArguments { query = "toast" } |> withFirst 10)

EDIT: I posted the following, but now I’m having second thoughts… it’s way harder to teach since it separates required and optional arguments further than they are now. I’m leaving it in, though, since I think it’s useful to also talk about things that are not good ideas.


So I think that’s an improvement, but we can do better. Let’s keep going. First problem: it’s awkward that that has to be a spearate pipeline. @Shump I don’t know if you felt this was awkward explicitly, but putting it in a let block like you did at least intuitively shows it could improve.

What if the arguments modified the result of search instead?

search : SearchRequiredArguments -> SelectionSet ... -> Field ...

withSearchFirst : Int -> Field ... -> Field ...

Then the API could look like:

search { query = "billiards" } results |> withSearchFirst 10

(caveat: I think this could work with the types as-is, but I don’t know how awkward it would actually be in practice.)

From a user perspective, I think this might be much better! The only awkward bit remaining now is explaining why optional arguments are not colocated with the required ones. The first approach in this post might be better for that, albeit with slightly weird types… at least the compiler can give nice messages for record errors!


#9

OK, sorry for the double post but I want to address the pains with the current API point-by-point and talk about how I think this design improves them:

The type signatures are now very simple withFirst : Int -> SearchArguments -> SearchArguments and can be kept simple by providing longer names, like withSearchFirst, or by putting them in separate modules.

We could be fancy and use open records and polymorphism, but I think that would be much worse from the perspective of someone trying to understand how to use the library for the first time. It’s exciting and succinct, but since the code is being generated instead of hand-written, I think it’s worth it to use as explicit types as we can.

identity is gone in this design!

This is still the case, but in a way that I think will be more intuitively approachable. Compare:

items
    |> List.filter (\{ name } -> name == "elderberries")
    |> List.take 10

searchArguments { query = "elderberries" }
    |> withSearchFirst 10

I think it’s pretty reasonable to do things this way!

This last bit is really annoying, I’m sure! But check out what you get to show people as the default query now:

search (searchArguments { query = "phantasmagorical" })

This is not far off from your proposed syntax changes; it has only an extra function call. That’s easy to explain: “it means that we can add optional arguments later.”


#10

The other standard way optional arguments are solved in elm is that your function accepts a list of some Option type. You can write/generate functions to create values of that type

search [ query “foo”, results 5 ]

The advantage is this avoids Maybe everywhere, since missing in the list indicates the default value.

The only issue is that you cannot prevent the error of repeated arguments.


#11

@brian thanks for taking the time to respond! It’s great to explore different approaches and see what the code would look like. The approach you outline is neat in a lot of ways (not writing Present, optional and required arguments being in one place). As a user and a teacher of the library, though, I still prefer the current approach of function syntax (approach 2).

I find it easier to use and explain approach 2 because you don’t need to call a specific function from a specific module in order to get it to compile. You just pass in a function which gets the defaults injected. Once you have that function, you’re just adding fields onto a record, and you get all the nice compiler features for a field (there’s no field called searchTerm, did you mean query?). I think that the compiler with the alternative you proposed would not be able to suggest typos or possible values.

When I’m writing queries, or any Elm code, my first pass is always to get something compiling right away so I can keep it in a compiling state and get nice feedback. So for the first pass of writing a query (and Elm code in general), I pass in [], Nothing, or identity to get things compiling as quickly as possible. I think this is a good approach because you can validate that the basic layout of your code is sound, and then get the correct values later. Passing in identity as the argument in the first pass is quite easy, but going to reach for a specific function would feel cumbersome to me in this context. Just because it’s something you’d have to do dozens of times for a pretty simple query.

I appreciate the idea @gamebox! I’ve also used this approach in the library for certain things. I think that it has some advantages, but it shares the downsides that I describe above in this reply.

Ultimately, what I’d really like is some way to decouple the creation of the optional arguments. That is, a way that doesn’t need to inject the defaults at all from the user code, only the library code would need to know about the entire list of options. From what I’ve seen, this is currently the biggest stumbling block and most challenging part of teaching someone to use dillonkearns/elm-graphql, and for me its currently the most cumbersome part of using the library.


#12

Thanks again to everybody for your messages, I love discussing different approaches with this community! :heart:


#13

I believe that using a list of options transformers as suggested by @gampleman and a record for the required arguments provides some compelling advantages over the current solution (not vs the proposal), at least for newcomers:

search [] { query = "Elm" }
search [ first 10 ] { query = "Elm" }

or using the full module:

Github.Query.search [] { query = "Elm" }
Github.Query.search
    [ Github.Query.first 10
    ]
    { query = "Elm"
    }

Pros:

  • Default options do not require to import anything except the search function:
    Github.Query.search [] { query = "Elm" }
  • The options syntax looks simpler to me and does not require Just or lambdas:
    [] or [ first 10 ] vs identity or \optionals -> { optionals | first = Just 10 }
  • You still get the compiler help for the required arguments as it uses a record
  • The syntax is quite similar to the elm/html library except it uses a record for the required arguments, so it might help newcomers
  • There is not the issue with the two steps nested update

Downsides:

  • Compiler help is not as nice for the optional arguments as with a record
  • Optional argument functions must be used from the module
  • It’s a little awkward to have a list of functions then a record
  • Potential name conflicts if there are several queries in the same module? This one may be a problem :thinking:
  • No compilation check of duplicate optional arguments

I would say that it seems easier to understand and use for newcomers without optional arguments, and I’m not convinced that it is more complex even with those.

Example with two optional arguments: https://ellie-app.com/4472szXTHLYa1


#14

Another very different but interesting solution in my opinion is to use a pipeline API with phantom types to handle arguments:

https://ellie-app.com/448sFyBjVTZa1

type Arguments state
    = Arguments
        { first : Maybe Int
        , after : Maybe String
        , query : String
        }

type alias Missing =
    { first : Optional
    , after : Optional
    , query : Absent
    }


type alias Expected =
    { first : Optional
    , after : Optional
    , query : Required
    }

default : Arguments Missing
default =
    Arguments
        { first = Nothing
        , after = Nothing
        , query = ""
        }

query : String -> Arguments { r | query : Absent } -> Arguments { r | query : Required }
query queryString (Arguments args) =
    Arguments { args | query = queryString }


search : Arguments Expected -> String
search (Arguments args) =
    ...

then the search query:

default
    |> query "Elm"
    |> first 10
    |> search

or

Github.Query.default
    |> Github.Query.query "Elm"
    |> Github.Query.first 10
    |> Github.Query.search

If you forget the query function in the pipeline, the compiler will return the following error:

This function cannot handle the argument sent through the (|>) pipe:

89|         (default
90|             |> search
                   ^^^^^^
The argument is:

    Arguments Missing

But (|>) is piping it a function that expects:

    Arguments { after : Optional, first : Optional, query : Required }

with Required very visible in yellow. It also prevents to call query twice, but not optional ones.

If name conflicts are a problem, this might be solved by splitting the module or eventually prefixing the functions.
Also the default name is not that great, maybe this can be improved with something like:

search
    |> query "Elm"
    |> first 10
    |> run

Edit:
This can also handle duplicate optional arguments: https://ellie-app.com/448Hdq4bc8ca1

83|         (default
84|             |> first 10
85|             |> first 10
                   ^^^^^^^^
The argument is:

    Arguments { after : Optional, first : AlreadySet, query : Absent }

But (|>) is piping it a function that expects:

    Arguments { after : Optional, first : Optional, query : Absent }

#15

At last (sorry for the triple post, but damn, those Elm APIs are so obsessive :sweat_smile:), for your specific use case, as you generate the code, and may need several queries in the same Query module, you could try some pretty unorthodox solution that mixes most of the previous solutions and exports all the query functions in a record :open_mouth:

https://ellie-app.com/44nX8r22nMSa1

The query example:

import Github.Query exposing (search)

-- Default options
search.with { query = "Elm" }
     |> search.query

-- Optional first modifier
search.with { query = "Elm" }
    |> search.first 10
    |> search.query

or if you have several query modules:

import Github.Query as Github

Github.search.with { query = "Elm" }
    |> Github.search.query

that would use among other things the following generated code:

module Github.Query exposing (search)

type alias SearchRequired =
    { query : String
    }

type alias SearchOptions args =
    { args
        | first : Maybe Int
        , after : Maybe String
    }

type SearchArguments state
    = SearchArguments (SearchOptions SearchRequired)

{-| Explicit record argument is generated to improve compiler error messages.
-}
searchWith : { query : String } -> SearchArguments SearchDefault
searchWith args =
    SearchArguments
        { first = Nothing
        , after = Nothing
        , query = args.query
        }

searchFirst : Int -> SearchArguments { args | first : Unset } -> SearchArguments { args | first : Set }
searchFirst queryFirst (SearchArguments args) =
    SearchArguments { args | first = Just queryFirst }

search =
    { with = searchWith
    , query = searchQuery
    , first = searchFirst
    , after = searchAfter
    }

You get:

  • the required arguments record for very nice beginner-friendly compiler errors
  • the default syntax is always the same: name.with requiredArgs |> name.query
  • duplicate optional arguments check using phantom types
  • limited names conflicts as only the exposed record must have simple fields names (the actual functions and types can be prefixed by the unique query name)
  • limited module functions discovery/search with the single exposed record
  • no maybes nor lambdas
  • nice errors messages

Of course as you generate the code, you can also generate a very nice documentation at the beginning of the generated module to describe the exposed records and give some examples.

Errors examples:

1. Invalid required arguments

search.with { wrong = 42 }
    |> search.query

returns

The 1st argument to this function is not what I expect:

87|         (search.with { wrong = 42 }
                         ^^^^^^^^^^^^^^
This argument is a record of type:

    { wrong : number }

But this function needs the 1st argument to be:

    { query : String }

Hint: Seems like a record field typo. Maybe wrong should be query?

2. Duplicate optional arguments

search.with { query = "Elm" }
    |> search.first 10
    |> search.first 15
    |> search.query

returns

This function cannot handle the argument sent through the (|>) pipe:

88|         (search.with { query = "Elm" }
89|             |> search.first 10
90|             |> search.first 15            
                   ^^^^^^^^^^^^^^^
The argument is:

    SearchArguments { after : Unset, first : Set }

But (|>) is piping it a function that expects:

    SearchArguments { after : Unset, first : Unset }

#16

Hello @dmy, thank you for exploring these ideas, your code snippets here are super interesting!

I think that this design would work very well for a library with a single builder chain (something like Luke’s http-builder, or my CLI options parser library actually!). The pain point I see with this approach for this particular use case of dillonkearns/elm-graphql generated code is that it’s not just a single place where you have to build up these records. You’re doing it all over the place. So to me, having to pull in a specific constant or function to manually inject the default values as absent makes it more cumbersome than the current approach. Agin, these features are very nice! I’m impressed by the ability to give such nice errors here, and even prevent duplicate entires! But to make the pain point more concrete, here is a 150-line module that builds up a request using dillonkearns/elm-graphql. You would have to manually inject the defaults in 3 places in this code, and in real-world applications (with larger queries and more frequent optional arguments) it could easily be more.

If we took @gampleman’s List-based approach, but used functions that were decoupled from the specifics, and instead were based only on the argument name, then that could be promising! As you hinted at @dmy, there would be collisions otherwise (because you could have several arguments with the same name, and even different types, within the same module… a module represents a collection of endpoints, not just one endpoint).

This is just a sort of blue-skying exercise, but it could be feasible since it’s generated code. This approach would literally put fieldName : Invalid for everything but the allowed optional arguments, like so:

after :
    value
    -> OptionalArgument { otherThings | after : Present }
after value =
    OptionalArgument


invalid :
    value
    -> OptionalArgument { otherThings | invalid : Present }
invalid value =
    {- This would be an argument that is invalid for Query.search,
       but is valid for something else
    -}
    OptionalArgument

needsSearchOptionals :
    List
        (OptionalArgument
            { anythingNotStatedIsValid
                | invalid : Invalid
            }
        )
    -> String
needsSearchOptionals list =
    "This would actually build up the values from the list argument"


checkErrorMessageDemo =
    needsSearchOptionals
        [ first 10

        -- , invalid "If you comment this out it becomes invalid"
        ]

Here’s a live example of the above code https://ellie-app.com/44zmjzKFWJRa1

There’s probably a more elegant way to do this, this is just the first thing that came to mind. The error messages aren’t so good with the above code.

    List (OptionalArgument { a | first : Present, invalid : Present })

But `needsSearchOptionals` needs the 1st argument to be:

    List (OptionalArgument { a | first : Present, invalid : Invalid })

But still, it’s a start! It solves the problem of decoupling. Some advantages:

  • The default is just [], not identity (as @gampleman described, much easier for beginners)
  • Don’t need to use Justs (as mentioned in previous posts).
  • Don’t need to inject a default starting point
  • Don’t need to use the proper function from the proper module (and name/type collisions can’t happen)… the functions are all in one module (OptionalArgument), and the names are based only on the argument name.

I think this prototype could potentially be refined to include those nifty innovations you came up with @dmy! That is:

  • Warn about duplicates (not the most important thing, though)
  • Give clear error messages about the allowed values (not a full list of everything that is disallowed!)
  • Include both optional and required arguments in the same list

With this decoupling from injecting defaults, this is starting to seem like something where the tradeoffs could be preferable to those of the current approach (if this can be refined to something with nice error messages).

I do still wonder if this is pointing to a language feature that could provide a first-class way to handle cases like this, but regardless I really appreciate brainstorming with everybody! And I’m also a firm believer in the approach that Elm takes to using a slim set of language features, and expanding them based on concrete problems and real-world examples.


#17

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.