Strategies for representing persisted and un-persisted remote data

[joelq]

@wmakley this is a fascinating data modeling problem

Stepping back a bit, it sounds like you have some core data (your record) and several dimensions of contextual metadata.

The core data is what your application is about and it is what ultimately needs to be sent to the server. Most functions that implement business logic can operate directly on this data.

The contextual metadata is used to model the uncertainty at the edges of the system such as user input or network requests. A lot of conditional UI logic is based on these such as displaying errors, showing spinners, or showing/hiding fields.

Based on earlier conversation, these contexts seem to be:

1. User input might be valid or not valid (or possibly not checked yet)
2. Data might only exist locally or might be persisted to the server (and possibly persisted but also changed locally)
3. HTTP Requests to the server may succeed or fail or be in flight

Does this accurately describe the situation?

[wmakley]

@joelq yes pretty much!

In the past, I have embedded the “core” data in the “contextual” data. So in a UI for managing a list of things, each list item will contain all the UI state, and an embedded “core” record with the latest canonical data from the API. I want to try creating a completely separate repository of “core” data on my next project, because the embedded crap is getting annoying. I guess I have been inadvertently using the “Form -> Data” pattern for a long time, but my forms have always had the core data nested in them.

Dealing with the record ID is just a very small part of the problem, but still an annoying implementation detail.

[Brian_Carroll]

@joelq

perhaps it was Problem Solving with Maybe ?

Yes! Thanks! Bookmarking it this time!

[z5h]

After thinking about how my Firebase app works, and how I’d like it to work, I came up with a minimal example where the domain model “truth” is external to the application.

z5h/elm-external-domain-model

The idea is, I have a module that let’s me operate on my local representation of the data, but those operations only ever return an opaque Diff, which can only be converted to a Cmd (here via ports). This is the way to ask for the real external data to be updated. A subscription gets opaque Domain.Updates values which can be applied to the domain model to actually update it.

In my real app, Firebase is fast enough (and possibly does local optimistic writes) so that the app experience is that of working with local data. If you’re talking to something fast, or talking to a JS library that has optimistic writes, this should work well.

I’d appreciate any feedback.

[popara]

I have noticed one problem that is shadowing all these thoughts:
a) Do we have a need to create type system that encodes a difference between Local and “Source of Truth” data, or the way we define complex UI elements, that often have their own states, hence we need to define some kind of structure for them, which is forcing us to encode that difference on per-problem basis.
b) Is the need to make a distinction between a Model and ViewModel, (in Firebase example Domain.Model and UIState) some kind of faulty thinking, because, then you need to manage to TWO states, ui details and data details.

I have a sense that b) is breaking TEA in some sense. I have just deployed one small solution to similar problem:

Lets say we want a small form from which you can create new DateMarkers (send it to server), edit them (by changing the date via datepicker):

type alias DateMarker = 
  { date : Date 
  , id : Uuid 
  }

type alias MarkerForm = 
  { markers : List DateMarker
  , editing : EditState
  , datepicker : DatePicker
  , seed : Random.Seed
  }

type EditState
  = NotEditing
  | NewMarker EditableMarker
  | Marker Uuid EditableMarker

type alias EditableMarker  = 
  { date: Maybe Date
  }

Ok, main point is, that EditableMarker which is basically model for actual form that we use in view, doesn’t care about peculiarities of DateMarker (except that shares part of the name, which is maybe ill practice). EditableMarker describes the data in the form, if the user cancels change, I just update the state to NotEditing, and that way you get “discard change” for free, because only on Saveis where you apply the change depending what the EditState is (sending PATCH or POST, and handling markers list of items, (that is the reason why I store Uuid in one of EditState. so we have local snapiness that Elm is giving us <3)

I was thinking of generalizing it, but first I want to get one more use case of similar form and then I will maybe try to think about it more abstractly 8)

I am aware that your use case is probably orders of magnitude more complicated that this piece right here, but still, I really think it is really important to have concrete requirements, because they drive it more than anything else. Elm is beautiful because it can express requirements so elegantly

[dmy]

@wmakley

I usually generate dumb “UID” values for my collections

I understand. I would like too to use UUIDs but collisions when they are generated by clients could be an issue (Only 32 bits of randomness, right? · Issue #10 · danyx23/elm-uuid · GitHub). What strategy do you use to handle them?

@wmakley

I would love an efficient ordered Dict type.

Ditto. For now I usually use an opaque type encapsulating a Dict and an Array (or eventually a List), with one of them used as an Index just pointing to a value in the other. elm-dictlist 2.1.2 might be a solution but I usually prefer to be able to work with core types.

Anyway, to come back to your types issue:

@wmakley

I thought I’d really hit on something here, but it is a royal pain to deal with this type. I have to write code to handle every case everywhere it appears, because no helper I have come up with seems useful.

Have you considered mixing:

• phantoms types (5 Things you didn’t know about Elm | by Ilias | Ilias Van Peer) to narrow record states on which functions can operate without having to pattern match everywhere
• changing types in records updates (5 Things you didn’t know about Elm | by Ilias | Ilias Van Peer) to avoid Maybe values that are actually always Nothing or Just value depending on the record state
• extensible records in signatures to have functions that can operate on a subset of these types

Something like https://ellie-app.com/9WbTKMN7na1/1 :

module Record exposing (..)

import Html exposing (Html, text)

type Record state record
    = Record record


type Unsaved
    = Unsaved


type Saving
    = Saving


type Persisted
    = Persisted


type Failure
    = Failure


type Error
    = NameAlreadyExists
    | ServerUnavailable


type alias NewRecord =
    { name : ()
    , id : ()
    , error : ()
    }   

type alias UnsavedRecord =
    { name : String
    , id : ()
    , error : ()
    }


type alias NamedRecord record =
    { record | name : String }


type alias SavedRecord =
    { name : String
    , id : Int
    , error : ()
    }


type alias InvalidRecord =
    { name : String
    , id : ()
    , error : Error
    }


new : Record Unsaved NewRecord
new =
    Record (NewRecord () () ())


name : String -> Record Unsaved NewRecord -> Record Unsaved UnsavedRecord
name recordName (Record record) =
    Record { record | name = recordName }

save : Record Unsaved UnsavedRecord -> Record Saving UnsavedRecord
save (Record record) =
    Record record

view : Record state (NamedRecord record) -> Html msg
view (Record record) =
    text record.name

addId : Int -> Record Saving UnsavedRecord -> Record Persisted SavedRecord
addId id (Record record) =
    Record { record | id = id }


handleError : Error -> Record Saving UnsavedRecord -> Record Failure InvalidRecord
handleError error (Record record) =
    Record { record | error = error }

The types require a little more work as there is currently some potential confusion between phantom types and record ones, but hopefully you see the idea.

Then you can do for example:

> new = Record.new
Record { name = (), id = (), error = () }
    : Record.Record Record.Unsaved Record.NewRecord
> named = Record.name "test" new
Record { name = "test", id = (), error = () }
    : Record.Record Record.Unsaved Record.UnsavedRecord
> saving = Record.save named
Record { name = "test", id = (), error = () }
    : Record.Record Record.Saving Record.UnsavedRecord
> saved = Record.addId 42 saving
Record { name = "test", id = 42, error = () }
    : Record.Record Record.Persisted Record.SavedRecord
> 
> Record.view named
{ type = "text", text = "test" } : Html.Html msg
> Record.view saving
{ type = "text", text = "test" } : Html.Html msg
> Record.view saved
{ type = "text", text = "test" } : Html.Html msg

You will however have to use an encapsulating union type if you want to have different record types in a useful single collection. Or you can use several collections and move records between them.

[wmakley]

So there is a lot to think about in this thread, but I thought I would respond to this at least:

@dmy

I understand. I would like too to use UUIDs but collisions when they are generated by clients could be an issue (Only 32 bits of randomness, right? · Issue #10 · danyx23/elm-uuid · GitHub ). What strategy do you use to handle them?

My client-side UUID’s are only used for routing messages to the correct UI element. For example, if I have a list of elements, I generate a UUID for each one on page load simply by incrementing an Int, and any time a new item gets added, it gets a UUID that never changes.

The list itself is probably sorted either by some kind of implicit order, or a database column, but this is getting very off-topic from representing database records. My UUID’s are a UI implementation detail.

[joelq]

I played with a bunch of different types here. I think there might actually be two concepts hiding here:

1. The state of the local data vs data on the server (is it local only? do we have diverging values?)
2. The state of an attempt at synchronizing with the server

Here’s what I came up with. It’s inspired by the approaches taken by stoeffel/elm-editable, ericgj/elm-validation, and krisajenkins/remotedata

type ClientServerData id local remote                                            
  = OnClientOnly local                                                           
  | Diverging id local remote                                                    
  | Synced id remote

and

type Synchronizable id local remote                                                
  = AtRest (ClientServerData id local remote)                                  
  | InFlight (ClientServerData id local remote)                                
  | FailedSync Http.Error (ClientServerData id local remote)

I’ve written a few utility functions around these. I wanted to write up a simple app to show off it would all work together but didn’t get that far.

I figured it would be helpful to share the types here anyway since the conversation heavily focused on modeling