Open-generator-cli problems, or Swagger/OpenApi recommendations

I used to generate my json decoders using http://andreashultgren.se/swagger-elm/ and everything was fine.

However, since I upgrade the server to dotnet-core-3.0 and swagger was upgraded to openapi 3.0, it cannot parse the swagger.json file anymore. The author also told me he doesn’t recommend using it anymore.

So, I tried java -jar openapi-generator-cli-3.3.4.jar generate -g elm -i swagger.json

instead. The code is slightly different, for example, no extra type around the records.

However, I run into several minor problems:

  1. [main] WARN o.o.codegen.DefaultCodegen - Unknown type found in the schema: object during compilation
  2. Some encoders do not compile due to complaints about null
  3. Some decoders fail due to error 1 (but not all).

I do understand why elm needs encoders/decoders, I just do not want to write them myself.

Handcrafting my decoders is not an option since I want the server to be the master, just add more data on the server, regenerate the elm-decoders, and start using the data in the client.

This is not going to be of much practical use to you right now but…

This is an area I am interested in. I have a start of a package for Open API here: GitHub - the-sett/elm-swagger: Tools for building Swagger specifications in Elm, and for generating Elm code from them.. The name is a bit misleading its really for Open API 3.0.0, not Swagger 2.0.

And am currently writing an Elm code generator for AWS services, which have their own swagger-like service definition files: GitHub - rupertlssmith/elm-aws-codegen: Code generation of Elm stubs for AWS Services.

The AWS one is taking priority right now. At some point I hope to circle back and complete the Open API package and write a code generator for it too.

The code is slightly different, for example, no extra type around the records.

An extra type around records is needed where records are recursive, since an Elm type alias does not permit recursion. For example

type alias Nestable =
    { val : String
    , inner : List Nestable
    }

Yields the error message:

This type alias is recursive, forming an infinite type!

1758| type alias Nestable =
                 ^^^^^^^^
When I expand a recursive type alias, it just keeps getting bigger and bigger.
So dealiasing results in an infinitely large type! Try this instead:

    type Nestable =
        Nestable
            { val : String, inner : List Nestable }

This is the sort of thing I am aiming to handle nicely in my Elm code generation efforts. The main AWS service I am interested in using at the moment is DynamoDB which happens to have a recursive data model. I will be adding logic to the model analysis part of the code generator to detect cycles and wrap in types, but only where it is really needed.

Wish I had something more complete to offer - but if you get inspired to have a go at writing some code generators for Elm…

2 Likes

Hi @mattiasw, great to see you are using OpenAPI to generate your encoders/decoders. I am the creator of the Elm generator for the OpenAPI generator. There are still some known issues with the Elm generator, but most of the spec should be covered. Can you try generating your code with the latest version? There have been many improvements from v4.0.0 onwards.

Feel free to post a detailed error here or on GitHub if you need some additional help.

openapi generator 4.2.0 was much better than 3.3.4, but finding the latest jar-file wasn’t trivial. Thank you!

Since I use C# 8 and I am a functional programmer, of course I want to use nullable notation. However, that creates some strange OneOf thing in the swagger.json. I made a bug report. In the meantime, I changed my C# code.

Some more suggestions:

  1. A big problem is that it only covers 90% of my classes, since I use recursive datastructures like Comment in https://github.com/elm/compiler/blob/9d97114702bf6846cab622a2203f60c2d4ebedf2/hints/recursive-alias.md It would really be nice if this was supported. Maybe some extra information, where I have to list my recursive data.

  2. I do not really understand why the code also generates Main.elm. This means that I cannot just generate the files into my /src/ folder, instead, I need a script like:

    java -jar openapi-generator-cli-4.2.0.jar generate -g elm -i swagger.json -o src/Swagger/
    rm src/Swagger/src/Main.elm
    rm -rf src/Request
    rm -rf src/Data
    cp -rf src/Swagger/src/* src
    rm -rf src/Swagger

  3. Also, the code generated in the Request folder is almost useless, since basePath is hardcoded. I typically download the swagger.json from my dev-system, and then it says http://localhost:5000 in every file in the Request folder. It would actually be better if there was a HttpConfig.elm file where base path and the other options are defined, and these are used by the stuff in request. HttpConfig.elm would only be created unless it already exists, and then never be replaced.

Interesting - I wonder how hard support for recursive types would be to add to these code generators. It might even require that the Java code for the data model would need to be modified to to support a cycle detection algorithm and some additional meta-data to mark places in the model where there is recursion.

An easy workaround might be to assume that everything is always going to be recursive, so wrap all records in types and always use Decode.lazy whether it is really needed or not.

If we would just list them as input to openapi-generator, like saying that FormElement is recursive, then the difference is really small for the decoder. In my example, we need to add a type which I called FormElements.

@@ -36,9 +35,11 @@ type alias FormElement =
     , dropDownLabels : (Maybe (List String))
     , help : String
     , verbatim : Bool
-    , elements : (Maybe (List FormElement))
+    , elements : FormElements
     }

+type FormElements = FormElements (Maybe (List FormElement))
+

 decoder : Decoder FormElement
 decoder =
@@ -56,9 +57,7 @@ decoder =
         |> optional "dropDownLabels" (Decode.nullable (Decode.list Decode.string)) Nothing
         |> required "help" Decode.string
         |> required "verbatim" Decode.bool
-        |> optional "elements" (Decode.nullable (Decode.list FormElement.decoder)) Nothing
-
-
+        |> required "elements" Decode.map FormElements (Decode.nullable (Decode.list (Decode.lazy (\_ -> decoder))) Nothing)

This is directly taken from

Good to hear things work better with 4.2.0. There is also a npm package for it!

  1. Recursive structures are indeed not yet supported. Partly on my request a detection has been added to the OpenAPI generator project, but I haven’t checked how good it is yet. Before, I’ve used my own recursion detection in my Elm ProtoBuf generator, based on this. There I make a new type for the fields that loop back to itself. I’ll first have a look on the state of the existing detection, but I would like the behaviour to be the same;
  2. TBH I am not a big fan of the OpenAPI generator creating complete projects. But it does so for all languages, so for Elm it’s not different. I think it does make more sense to create it as a library though. The good news is you do not need to cp/rm files. I typically generate my code into a generated/src directory which I have added to my source-directories in elm.json;
  3. You can configure the generator to enable setting the base path for each request using a config option. This allows you to initialize your app with the base path which you then feed to all requests.

Hope this helps. I’ll have a look at the recursion later.

Appreciate that you look at the recursion again.

Regarding the duplicated Main.elm, you just ignore you have 2 Main.elm and hope you always choose the right one.

Adding an extra path to source-directories is a good idea.

What I am missing and didn’t find is some tips for elm-users on how to use the generator. We need a page somewhere.

Regarding npm, I use mixed environments, and npm sometimes screws things up, so I prefer to have just a plain .jar file which I can commit to git, and I have complete control.

Regarding the source-directories approach, are you sure this still works with 0.19.1? I had problems with some of my included sources also having a Main.elm with 0.19.1 so you might run into this too.

In case it’s helpful, you can see how I’ve implemented recursive GraphQL input object detection in elm-graphql (with some pretty nice unit tests because it’s all pure Elm code):

I’d be more than happy to discuss it on Slack or on a video chat some time, feel free to ping me.

1 Like

Hey! Here is how we use it in our Makefile at Niteo:

src/Api: .installed openapi-client
	@rm -rf src/Api/
	@cp -r openapi-client/src/Api/ src/Api/

openapi-client: .installed ../backend/openapi.yaml
	@rm -rf openapi-client/
	@npx openapi-generator generate \
		--input-spec ../backend/openapi.yaml \
		--enable-post-process-file \
		--generator-name elm \
		--config openapi-generator.yml \
		--output openapi-client/

	@npx elm-format --yes openapi-client/src/

and openapi-generator.yaml:

elmPrefixCustomTypeVariants: true

The caveat is that we are using a fork we made few months ago (at version 4.1.1 then). It’s a little bit messy, but we are using it in production ever since. The main difference is introduction of Request data structure and send function. This allows us to post-process generated requests before we send them. We are using the generated code as a library, not a project scaffold. Unfortunately I didn’t have time to prepare a PR. We urgently needed to scratch our own itch. If there is any interest I’ll be happy to work toward merging our changes.

1 Like

Sure, it’s on my wish-list for a long time already. I checked the current status: the generator only detects self referring types. I am going to try to find some time next week to add a PR for detecting cyclic references as well. Once this is place in we can update the Elm generator to deal with recursive types.

src/Main.elm is the main file for my app. I never import a Main so this should not be an issue.

Good point. I haven’t find the time for it, but it would be great if somebody would pick this up!

Haven’t used 0.19.1 yet, so I’ll give it a try as well!

Can you elaborate a bit on this? It would be great to receive feedback/a PR like this as I am always open for improvements. Currently, I am adding proper support for oneOf and discriminator. I also want to change the way allOf is handled. I was thinking of using composition as inheritance is not an option in Elm (currently all properties are just copied over). This will be a breaking change anyway, so it would a good moment for other revisions as well.

This indeed no longer works (and I guess that makes sense). Thanks for pointing this out.

1 Like

I’m working on a more detailed description of our changes with rationale and usage samples. I should have it today evening (Amsterdam time). For now you can see the diff here: Comparing d0d545b...elm-request-data-structure · tad-lispy/openapi-generator · GitHub

In my mind the most significant one is creation of modules/openapi-generator/src/main/resources/elm/Request.mustache that generates Request a type and making code in Api.Request work with this type instead of Cmd msg. This makes it possible for the application developers to modify the request before turning it to a command. This allows to think about generated code as a sort of a package. By using an Api prefix we can emit the code directly to src/ without messing with source-directories in elm.json.

Below I describe the changes made in our fork of the generator. I hope it can be an opener for a discussion about incorporating this changes into the upstream.

Changes made to OpenAPITools/open-api-generator

We have forked and branched off from revision d0d545b (Prepare 4.1.2 snapshot). The diff can be seen here: https://github.com/tad-lispy/openapi-generator/compare/d0d545b...tad-lispy:elm-request-data-structure

What changed

Conceptually following changed:

  • All generated modules are prefixed with Api

    e.g. Api.DateTime, Api.Request.Articles

    The Main module is an exception. It is still generated in src/, but we do not copy it into our application. See usage section below.

  • We now generate an Api.Request module

    This module exposes a Request and Method types:

    type alias Request a =
      { method : Method
      , path : List String
      , query : List QueryParameter
      , headers : List Http.Header
      , body : Maybe Json.Encode.Value
      , decoder : Json.Decode.Decoder a
      }
    
    type Method
      = GET
      | POST
    

    The type parameter a is a type of response body. This types are exposed by code generated in src/Api/Data/ directory, e.g. module Api.Request.Articles would expose a getArticle : String -> Request Api.Data.Article.Article. All generated modules work with this type. Since it’s a record, application code can modify it in any way it needs before sending the actual request. Sending is a responsibility of the application. See usage section below for more details.

  • All parameters are passed as individual arguments

    Not as records. The order is the following:

    • Path params
    • Query params
    • Header params*
    • Body param
  • Decoders for not required fields always have default value

    Even when they are nullable. Previous behaviour seems incorrect. Why would nullabel fields not have default value? I think it was generating broken code, but my memory may not be accurate here.

  • I fixed something about record field encoders

    Frankly I don’t remember how it was broken. It’s about these two changes:

    I seem to remember that the generated code would not compile without this change.

How we use it

We are using make to build our project, so it’s easiest to explain by showing relevant parts of the Makefile:

.PHONY: all
all: .installed dist lint test openapi-client

.PHONY: dist
dist: .installed
	@npx parcel build src/index.html

src/Api: .installed openapi-client
	@rm -rf src/Api/
	@cp -r openapi-client/src/Api/ src/Api/

openapi-client: .installed ../backend/openapi.yaml
	@rm -rf openapi-client/
	@npx openapi-generator generate \
		--input-spec ../backend/openapi.yaml \
		--enable-post-process-file \
		--generator-name elm \
		--config openapi-generator.yml \
		--output openapi-client/

	@npx elm-format --yes openapi-client/src/

As you see the src/Api target is not a dependency of all. We run it manually whenever we need to. The generated code is checked in to our version control. We also run this target in continuous integration and fail the build if it results in any changes (if the diff is not empty). This way we make sure that developers use latest API. Because we only copy files from openapi-client/src/Api/, the generated Main.elm is not imported into our application code.

In our application all requests require authentication. To send request with the auth header we have a following function:

sendAuthenticatedRequest :
    (Result Http.Error data -> msg)
    -> Credentials
    -> Maybe String
    -> Request data
    -> Cmd msg
sendAuthenticatedRequest callback (Credentials credentials) tracker request =
    let
        method =
            case request.method of
                Api.Request.POST ->
                    "POST"

                Api.Request.GET ->
                    "GET"

        url =
            Url.Builder.crossOrigin credentials.apiURL
                request.path
                request.query

        headers =
            case credentials.token of
                Nothing ->
                    request.headers

                Just token ->
                    Http.header "Authorization" ("Bearer " ++ token)
                        :: request.headers

        body =
            request.body
                |> Maybe.map Http.jsonBody
                |> Maybe.withDefault Http.emptyBody
    in
    Http.request
        { method = method
        , url = url
        , headers = headers
        , body = body
        , expect =
            Http.expectJson callback request.decoder
        , timeout = Just 30000
        , tracker = tracker
        }

As you can see we only support GET and POST - this is application specific. Also in our case the function is part of a Credentials module that exposes opaque Credentials type. The goal of this module is to make it very difficult to accidentally leak sensitive user information: e-mail address, password or authentication token. This is an implementation of the exit gatekeeper technique described here: https://incrementalelm.com/articles/exit-gatekeepers. I suppose if we are going to merge my changes, there should be an Api.Request.send function that abstracts part of the logic above.

In our Main module we make requests like this:

content
  |> Api.Request.Articles.saveArticle
  |> Credentials.sendAuthenticatedRequest
      ArticleSaved
      model.credentials
      Nothing

Generated code for saveArticle looks like this:

module Api.Request.Articles exposing (generateArticle, getArticle, getArticles, saveArticle)

import Api.Data.Article as Article exposing (Article)
import Api.Data.SaveArticle as SaveArticle exposing (SaveArticle)
import Api.Request exposing (Method(..), Request)
import Json.Decode as Decode
import Maybe.Extra as Maybe

{-| Save an article. Auth is required
-}
saveArticle :
    SaveArticle
    -> Request Article
saveArticle body =
    Request
        POST
        [ "articles" ]
        (Maybe.values
            []
        )
        [{- [] -}]
        (body |> SaveArticle.encode |> Just)
        Article.decoder

Known shortcomings

The fork was made primarily to cover our own use case, so some obvious shortcomings were accepted. If we want to merge the changes with upstream we should address them.

  • We only support Elm version 0.19 (I expect 0.19.1 to work, but we didn’t test it). The code generated for 0.18 code most likely will be completely broken.

  • Header params are not supported

  • Query params other than integers are not supported

2 Likes

Thanks for your extensive description @lazurski! I especially like the idea behind the Request type.

I am going to sum up some of the changes I would like to apply. Feel free to comment on them as I this list is open for debate:

Proposed changes

  • Drop support for Elm 0.18;
  • Drop Main.elm as this does not add anything and since Elm 0.19.1 this file can no longer exists when using multiple source-directories. I do intend to keep elm.json and README.md in. These can be added to the .openapi-generator-ignore file if you do not wish to use multiple source-directories;
  • Put all generated models and operations in a nested module (Api), i.e. Api.Data and Api.Request.*;
  • Put all generated models in a single module (Api.Data), which makes it
    • easier to import a generated type;
    • possible to support recursive types, composed types (oneOf, allOf, discriminator) as this could need cyclic imports otherwise;
    • nicer to support aliases like DateTime and Uuid as they do not required additional modules;
  • Add a Request a type for all operations. This type allows for smoother configurations (so the OpenAPI configurations can be dropped). This also allows for customization like in @lazurski’s Credentials example;
  • Revert back to inline parameters (no records) in the following order: ‘header’, ‘path’, ‘query’, ‘body’.

Some (pseudo) code examples:

Click to expand

AllOf

A:
  properties:
    a:
      type: string
B:
  allOf:
    - $ref: '#/components/schemas/A'
    - properties:
        b:
          type: string

would become

type alias A =
    { a : String
    }
    
type alias B =
    { a : String
    , b : String
    }

AllOf + discriminator

A:
  required:
    - type
  properties:
    a:
      type: string
    kind:
      type: string
  discriminator:
    propertyName: kind
    mapping:
      b: '#/components/schemas/B'
B:
  allOf:
    - $ref: '#/components/schemas/A'
    - properties:
        b:
          type: string

would become

type A
    = A BaseA
    | AB B

type alias BaseA =
    { a : String
    , kind : String 
    }
    
type alias B =
    { baseA : BaseA
    , b : String
    }

OneOf

A:
  properties:
    a:
      type: string
B:
  oneOf:
    - $ref: '#/components/schemas/A'

would become

type alias A =
    { a : String
    }
    
type B
    = BA A

If the discriminator is present, its value will be used for decoding. Otherwise Decode.oneOf is used.

Enum

Foo:
  type: string
  enum: [bar, baz]

would become

type Foo
    = FooBar
    | FooBaz

Recursive types

Comment:
  required:
    - message
    - responses
  properties:
    message:
      type: string
    responses:
      $ref: '#/components/schemas/Comment'

would become something like

type alias Comment =
    { message : String
    , responses : CommentList
    }
    
type CommentList
    = CommentList (List Comment)

For indirect recursive types the first idea to use an additional type for each circular referring property, i.e.:

Comment:
  required:
    - message
    - responses
  properties:
    message:
      type: string
    responses:
      type: array
      items:
        $ref: '#/components/schemas/Response'
Response:
  required:
    - comment
  properties:
    comment:
      $ref: '#/components/schemas/Comment'

would become something like

type alias Comment =
    { message : String
    , responses : ResponseList
    }
    
type alias Response =
    { comment : CommentType
    }
    
type ResponseList
    = ResponseList (List Response)
    
type CommentType
    = CommentType Comment

Variantions on this are also possible:

  • always wrap the base type only:
    • type CommentType = CommentType Comment
    • type ResponseType = ResponseType Response
  • make a wrapping type for each property:
    • type CommentResponses = CommentResponses (List Response)
    • type ResponseComment = ResponseComment Comment

I am not sure yet what gives the nicest result.

Request

The Request module would be something like

module Api.Request exposing (Request, request, with...)


type Request a =
    Request
        { method : String
        , headers : List Http.Header
        , basePath : String
        , pathParams : List String
        , queryParams : List Url.Builder.QueryParameter
        , body : Http.Body
        , createExpect : (Result Http.Error a -> msg) -> Http.Expect msg
        , timeout : Maybe Float
        , tracker : Maybe String
        }
        
        
request : String -> List (Maybe Http.Header) -> List String -> List (Maybe Url.Builder.QueryParameter) -> Http.Body -> ((Result Http.Error a -> msg) -> Http.Expect msg) -> Request a

send : Request a -> (Result Http.Error a -> msg) -> Cmd msg

withBasePath : String -> Request a -> Request a

withTimeout : Float -> Request a -> Request a

withTracker : String -> Request a -> Request a

withHeader : Http.Header -> Request a -> Request a

withHeaders : List Http.Header -> Request a -> Request a

The Request type is used for operations, e.g.

/users/{userId}:
  put:
    tags:
      - user
    parameters:
      - name: userId
        in: path
        required: true
        schema:
          type: string
          format: uuid
      - name: dry-run
        in: query
        required: false
        schema:
          type: boolean
    operationId: updateUser
    requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/User'
    responses:
        '200':
          description: Successful operation
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/User'

would become something like

module Api.Request.User exposing (updateUser)

updateUser : String -> Maybe Bool -> User -> Request User
updateUser = ...

Please let me know what you think. All suggestions are welcome!

Edit: added recursion
Edit: updated allOf’s (2x)

@eriktimmers Looks very good. When you say possible to support recursive types , how will that work?

Also, from a quality point of view, I think it is a good idea to generate a dummy file that loads all encoders, so that we can be sure it all compiles, not just the modules I happen to use.

Good point! Forget to add that. Please see the updated post.

What do you mean by this? Do you mean in the CI?

No, not only in the CI, but that is of course a good idea, to have a number of swagger.json files in a CI somewhere.

Since ELM only compiles referenced modules, which is rather uncommon for most programming languages, I didn’t detect all problem on the first generation. I detected it bit by bit. If I would have included a file all All.elm initially, just to test them all, I would have detected all problems upfront. I would of course not reference the All.elm file in the final code.

CircleCI is set to parse some test spec files. I plan on adding a version that covers more cases. I use it for the current refactoring as well.

The CI also runs elm make src/**/*.elm which compiles every module. I think that does what you intend here.