Strategies for representing persisted and un-persisted remote data

[wmakley]

Can you write out an example spec? Like “I have a form which asks the users for these pieces of information, I want to persist these parts of it to the server under these circumstances, I want to gracefully handle errors in these ways,” etc.

You mean like as an exercise in design discipline before my next attempt? Or would you like to see one for my latest train wreck?

I think the model I want to move toward is using a “Form” type as an intermediary that deals only with strings, and converting to an API record type when it’s time to try and persist it.

[rtfeldman]

Not code, just a description of what the code will need to accomplish! Maybe someone here can come up with a nice approach.

It’s easier to think about things to try in the context of a specific problem statement.

[wmakley]

Okay, here we go:

App’s purpose: Create printable document consisting of some header information about a job (contract), and some lines representing items in that contract. All user input will be persisted as soon as possible, so they don’t lose work. Ideally, it should not be possible to enter invalid data. We are using a REST JSON API for each record type.

1. Given that we have a contract, we want to start a new document about that contract and show some details about the contract.

2. We want to wait until the user makes a minimal change to save the document, to avoid database clutter.
   A minimal change will be “adding a line to the document” by clicking “Add Line”.

3. When the user clicks “Add Line”, we have to persist the document, and then use its ID to persist the line so that it can be modified. (I would like to avoid persisting the line immediately, but all the line edits involve adding other records that reference the line that we won’t go into, so we have to do that first.)

4. While the document is saving, the document needs to be “read-only”, to avoid complications.

5. We want to show the new line immediately, but disallow editing until it has been persisted, to make the page feel snappier.

6. Subsequent “Add Line” clicks will skip step 3 (save the document), but still need to wait until they are persisted for editing.

The main pain point I had is the app will always end up in a happy state eventually where we have all persisted records and the user can do anything they want, but I kept having to check if I was in that state everywhere I dealt with the line and document types.

[justinmimbs]

I’ll describe approaches I’ve used that I’ve been happy with.

For records that are persisted in a database and have an arbitrary key, instead of storing that key in the model of the record, I would keep a Dict of those records indexed by the key. So instead of storing something like

    records = List { id : Int, name : String }

I would instead store

    records = Dict Int { name : String }

where the Int is the arbitrary key.

This gives a consistent model for valid records, whether they’re persisted or not.

Let’s say the type for a valid record is Record; I would have a separate type for unvalidated forms, RecordForm. You can convert between the two as needed:

    recordToForm : Record -> RecordForm
    
    formToRecord : RecordForm -> Result RecordFormErrors Record

Like you said, the advantage is that the RecordForm only has to deal with types that relate to user inputs. It also makes it more natural to define an emptyForm value without resorting to definitions like id = -1.

This approach to validation has been mentioned before and there is at least one library that promotes the approach.

At the top-level model, I tend to store server data separately from any other state. This makes it simple to reload server data, as it can safely supersede existing server data without needing to resolve conflicts with user-entered data or any other state.

    type alias Model =
        { serverData :
            { records : Dict RecordId (Remote Record)
            , ..
            }
        , clientData :
            { editedRecords : Dict RecordId RecordForm
            , newRecord : Maybe RecordForm
            , ..
            }
        }

It sounds like you’re on the right track! I hope this helps.

[wmakley]

Thank you, that looks excellent! You just gave me a lot of brain waves. I was definitely heading toward separate form types and separate server data repositories slowly, but you just got me there a lot quicker, and I have already had some success with a repository type like that.

I do have a few questions about specific implementation details, but I will need some time to write it up.

[ericgj]

I’m not sure I understand your exact pain point - there are a number of different issues wrapped up together here. But I’m glad for the discussion because I think they are issues most of us face one way or the other.

If your document data is not too big, I think I would try to model every edit within a document at the Document level. In other words, “Add Line” triggers a backend call that returns data representing the entire changed document. Which you use to replace the whole document in your Elm model. And some edit within of an existing line likewise triggers a backend call that returns the entire document. For one thing, then you are only dealing with the Saving state of the Document, not individual Lines.

This may be completely obvious to you and more or less what you’re doing, so I don’t know if that helps at all.

The main pain point I had is the app will always end up in a happy state eventually where we have all persisted records and the user can do anything they want, but I kept having to check if I was in that state everywhere I dealt with the line and document types.

I wouldn’t think you would need to check this state (i.e. case on it) more than once in the view – where else did you find you needed to do it?

[wmakley]

Yeah, that is my whole problem: lots of issues! It’s difficult to describe the minimal case. Situation: My model has to transition between un-persisted and persisted states. Issue: How to I represent un-persisted vs persisted data so that both are easy to work with?

If it is un-persisted, it may be missing all kinds of strings and keys that it will have later, but I still need to let the user change it. Once it is persisted, it will always have those values, so it is annoying for them to be Maybe.

I wouldn’t think you would need to check this state (i.e. case on it) more than once in the view – where else did you find you needed to do it?

When updating the data from the server, via much the model you described. I need to case over the data every time the user makes a change, if I wrap it in a union type.

[z5h]

I’m having a lot of success with the following pattern (using Firebase).

1. Subscribe to data I’m interested in, as I become interested. The incoming data is the only source of truth.
2. I cannot change my model locally. I can only craft and send an update over a port (which gets applied to Firebase).
3. If/when the data gets updated successfully (by me or someone else), I’ll be notified because I’m subscribed to changes, and my model will get updated appropriately.

I have an Id type and sentinel value of empty, which is used to determine if something exists in the database, or has not been persisted.

PROs: simple, has always worked.
CONs: if it doesn’t work, the experience would be: I tried an update and nothing happened. No error, etc.

[Chadtech]

I think I am in the same position as you @wmakley , and I have been thinking about the same questions. At work and in my own projects at home I have been trying to manage persisted and unpersisted data and I think I have gone through both of your ideas of different record types for persisted and unpersisted data, and having a Maybe Id kind of thing.

Eventually a coworker of mine recommended something like what @justinmimbs suggested, and I would say it works pretty well. Its my favorite option right now. At work we’ve been calling the Dict Id Record thing Entities, and in my personal projects Ive been calling it a Db. I think the only draw back is if you are passing around ids and Db around, theres the logical possibility that your model is saying something like the selected record is such and such id, but theres no record with that Id in your Db, and accounting for that possibility is a real drag.

I have a library where I am trying to formalize good practices on this. Its at http://package.elm-lang.org/packages/Chadtech/id/3.2.0 . Right now its not documented very well and doesnt reflect best practices as I currently know them. PRs welcome! But whether it be in my existing package or somewhere else I would love to collaborate on a community-wide solution to this general problem.

[ericgj]

All user input will be persisted as soon as possible, so they don’t lose work. Ideally, it should not be possible to enter invalid data.

If it is un-persisted, it may be missing all kinds of strings and keys that it will have later, but I still need to let the user change it.

Between these statements I only have a fuzzy sense of what the intended UX is. On the one hand you are describing a situation where the aim is to save as soon as possible; on the other the aim is to save only when the input is valid. Yet it sounds like there is no explicit user ‘submit’ action, is that right? Or are you saying the “Add Line” is a submit action after the user enters the contents of the new (not yet persisted) Line?

And is it possible to be a little more concrete, for instance give an example or two of cases where there is invalid data and you don’t want to save it yet?

In general if you are dealing with wanting to validate input, and edit validated models, I agree you are best served by separate data structures and something like recordToForm and formToRecord (aka validate) functions as @justinmimbs outlined.

And how/when do you want to communicate to the user that their input is invalid, or is this not needed?

[wmakley]

Hey me too, but thanks for pointing it out. As usual, I was up against a deadline without a clear idea of how to proceed. This doesn’t work as well with Elm as it does for me with Rails CRUD apps.

Let me break it down to a VERY small almost incidental part of this app.

1. The user needs to select an item from a dropdown.
2. They need the ability to add option to the dropdown, if the desired option is not available.
3. The form to create a new option has two fields, and both are required. The “submit” button should not be enabled until both fields are filled in with valid data. (The user should never see an error, just a clear indication of what they need to enter, primarily because parsing and displaying errors from the API would be even more code I don’t have time for.)
4. When the user submits their new option, it will selected automatically.

Behind the scenes, each item can only be selected after it has been persisted, and has a database ID. So when the user submits the form, we need to send a “new” record to the API without an ID, and get a “persisted” back with an ID. Then we can add that record to the select box.

I coded my API functions to all work on the same type of record, so when I need to create a new one I give it a record with Nothing for an ID, and get back a record with Just Int. This little hitch is constantly annoying when I can usually gurantee that the ID will be one or the other given the context. I always persist everything as soon as possible so the users don’t lose work.

[Brian_Carroll]

I coded my API functions to all work on the same type of record

That’s the key design decision right there, I think. Once you decide to use the same record type everywhere, you fundamentally have to have case or map all over the place. No way round it.

Your other option could be to rewrite the function signatures to take the ID as a separate argument from the rest of the stuff.

type alias DatabaseId = Int
type alias OtherStuff = {field1: String, field2: String}
type ThingModel = ThingModel (Maybe DatabaseId) OtherStuff
viewUnsavedThing : OtherStuff -> Html Msg
viewSavedThing : DatabaseId -> OtherStuff -> Html Msg

You may find that you only need one or two case expressions, in the top-level functions of your view and update.

[ericgj]

So when the user submits the form, we need to send a “new” record to the API without an ID, and get a “persisted” back with an ID.

To add to what @Brian_Carroll said the stuff you send to the backend does not have to match what you get back from the backend – assuming that you control how the backend works. It’s a kind of dogma drilled into us that we are passing back and forth models (objects that we C/R/U/D), but it doesn’t have to be that way. In a lot of situations we are just modifying an aspect of a resource, for instance. Or there might be different ways of creating a resource, with more or less data attached.

I think of the bits of data we encode in messages to the backend less as models and more as Msgs – Msgs that are used by the backend to init and update persisted models. Just like Msgs in an Elm app, they might contain whole models; or they might contain just bits of data used to update models in various application-specific ways.

So for instance your dropdown example. I could be wrong, but it sounds like you are implementing something like a combobox, but with two fields instead of one for the “manual entry” option? Setting aside the validation, and assuming the two fields are String, it seems like you could have something like this (simplified):

type ComboBoxValue
    = Selected ID
    | Entered String String

type Msg
    = Submit ComboBoxValue
    | Submitted (Result Http.Error Model)

update : Msg -> Model -> (Model, Cmd Msg)
update msg model
    case msg of
        Submit value ->
            case value of
                Selected id ->
                    ( model, Task.attempt Submitted (submitSelected id model) )
                Entered field1 field2 ->
                    ( model, Task.attempt Submitted (submitEntered field1 field2 model) )
        -- ...

[dmy]

For what it’s worth, here is what I’m doing at the moment. This is a work in progress so I do not yet have enough hindsight but it seems to work well.

My objects are records like:

type alias Object =
    { id : ObjectId
    ...
    }

and their id are like:

type alias Id =
    Int


type ObjectId
    = ObjectId Id
    | LocalObjectId Id

with an “unboxer” like:

unObjectId : ObjectId -> Id
unObjectId objectId =
    case objectId of
        Objectd id ->
            id

        LocalObjectId id ->
            id

The objects are stored in an opaque Dict Id a that I initialize with the “unboxing” function so I can actually have several ones using different id types.

Also:

• objects from the database have a ObjectId Id with a positive integer as the id (the one from the db)
• objects generated locally have a LocalObjectId Id with a negative integer as the id that is unique locally (using Maybe.map ((+) -1) (minimum localIds) |> Maybe.withDefault -1 to generate one)

When objects are sent to the backend to be persisted, the backend generates a new unique db positive index and replace the negative local id by it, and sends back the object with the local id as a reference in the answer to allow the client to update the object.

This way:

• All objects, persisted or not, are stored in the same data structure, without using Maybe
• It is easy to filter reliably unpersisted ones to send them to the backend (using the record typed id field)
• Most functions using the objects do not care if they are local or not and I can use Keyed as every “unboxed” id is unique locally
• The typed ids are used to identify the objects in messages, so I cannot not inadvertently mix them up between different types of objects

The drawback is that I must be careful to avoid race conditions in messages that could use a local id after it has been changed to a database one, but this does not seem to be a problem specific to this solution. Also I store the id both as the dict key (unboxed) and in the object record (typed), so there is some slight duplication waste. I will see how it goes…

[wmakley]

@dmy I have considered a scheme like that! I have seen the negative ID thing done using SQL databases in a non-elm project, with satellite clients that update the master. It can definitely work.

You’re right to point out the race condition, something that I feel more Elm writings should touch on. I usually generate dumb “UID” values for my collections of GUI elements and wrap them up in a Tuple or record, and ignore the database ID’s entirely. I hate dealing with this implementation detail, and am trying to find ways to abstract it out. I would love an efficient ordered Dict type.

[joelq]

I read a great article a while ago that I can’t find now, frustratingly! It had some great recommendations on how to structure code with Maybe, Result and friends. Might have been Haskell examples.

The idea was to write your logic as if you always have the data and it’s always successful. Then use Maybe.map, Maybe.andThen, etc to deal with the Nothing cases. It’s a really nice approach and cleans things up a lot. You get rid of all the case expressions.

@Brian_Carroll perhaps it was https://robots.thoughtbot.com/problem-solving-with-maybe ?

[joelq]

@wmakley this is a fascinating data modeling problem

Stepping back a bit, it sounds like you have some core data (your record) and several dimensions of contextual metadata.

The core data is what your application is about and it is what ultimately needs to be sent to the server. Most functions that implement business logic can operate directly on this data.

The contextual metadata is used to model the uncertainty at the edges of the system such as user input or network requests. A lot of conditional UI logic is based on these such as displaying errors, showing spinners, or showing/hiding fields.

Based on earlier conversation, these contexts seem to be:

1. User input might be valid or not valid (or possibly not checked yet)
2. Data might only exist locally or might be persisted to the server (and possibly persisted but also changed locally)
3. HTTP Requests to the server may succeed or fail or be in flight

Does this accurately describe the situation?

[wmakley]

@joelq yes pretty much!

In the past, I have embedded the “core” data in the “contextual” data. So in a UI for managing a list of things, each list item will contain all the UI state, and an embedded “core” record with the latest canonical data from the API. I want to try creating a completely separate repository of “core” data on my next project, because the embedded crap is getting annoying. I guess I have been inadvertently using the “Form -> Data” pattern for a long time, but my forms have always had the core data nested in them.

Dealing with the record ID is just a very small part of the problem, but still an annoying implementation detail.

[Brian_Carroll]

perhaps it was https://robots.thoughtbot.com/problem-solving-with-maybe ?

Yes! Thanks! Bookmarking it this time!

[z5h]

After thinking about how my Firebase app works, and how I’d like it to work, I came up with a minimal example where the domain model “truth” is external to the application.

z5h/elm-external-domain-model

The idea is, I have a module that let’s me operate on my local representation of the data, but those operations only ever return an opaque Diff, which can only be converted to a Cmd (here via ports). This is the way to ask for the real external data to be updated. A subscription gets opaque Domain.Updates values which can be applied to the domain model to actually update it.

In my real app, Firebase is fast enough (and possibly does local optimistic writes) so that the app experience is that of working with local data. If you’re talking to something fast, or talking to a JS library that has optimistic writes, this should work well.

I’d appreciate any feedback.