# Strategies for representing persisted and un-persisted remote data
**URL:** https://discourse.elm-lang.org/t/strategies-for-representing-persisted-and-un-persisted-remote-data/1115
**Category:** Learn
**Created:** [April 12, 2018, 5:03pm UTC](https://discourse.elm-lang.org/t/strategies-for-representing-persisted-and-un-persisted-remote-data/1115 "2018-04-12T17:03:49Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@wmakley](https://discourse.elm-lang.org/u/wmakley)
#### Post date: [April 12, 2018, 5:03pm UTC](https://discourse.elm-lang.org/t/strategies-for-representing-persisted-and-un-persisted-remote-data/1115/1 "2018-04-12T17:03:49Z")
I’ve written something like a dozen small Elm apps that are in production in internal tools, and I’m still struggling to find a pattern for managing API data that I’m happy with. The types always spiral out of control when I try to model every use case perfectly.
I am not a fan of the various “RemoteData” packages I have seen, since they are mostly about loading data. Most of my remote data needs revolve around persisting form data in the following contexts:
1. Bound to a form as the form changes.
2. New “blank” records, and un-persisted data that may or may not be valid.
3. Data that we are in the process of attempting to save.
4. Valid data that we just got from the server.
**Here are some strategies I have tried in chronological order:**
**1. One naive record to rule them all.**
Define a record type matching the API exactly, for example:
```auto
type alias Record =
{ id : Int
, name : String
, optional_value : Maybe String
}
```
If I need a blank one, I just stuff anything in it:
```auto
blank : Record
blank =
{ id = -1
, name = ""
, optional_value = Nothing
}
```
I also use some simple helper functions like: `isPersisted { id } = id > 0` to check if the data has been persisted or not.
This felt un-Elmy, and I obviously had some problems with treating un-persisted records as persisted occasionally, or POST-ing a “-1” to the API accidentally. The “isPersisted” checks were all over the place and ugly.
**2. Use a union type to indicate “Blank” values.**
At first I used `Maybe`, but then I decided to get more explicit:
```auto
type PrimaryKey
= PrimaryKey Int
| Unsaved
type alias Record = { id : PrimaryKey, ... }
isPersisted : { a | id : PrimaryKey } -> Bool
isPersisted {id} =
case id of
PrimaryKey _ ->
True
Unsaved ->
False
```
I’m using this a lot now, and if anything it’s honestly messier. I have to unwrap the ID every time I want to use it, and I can never assume that the record has an ID, even though it will only ever be `Unsaved` once in its lifetime!
Calling `toString` on my record ID has actually been a large source of bugs as well, in instances where I replaced `Int` with `PrimaryKey`. The compiler didn’t catch it. Lesson learned: **toString is a bit unsafe.** Use your own String conversion helpers that expect a specific type.
**3. Separate types for “new” and “persisted” records and a union type to combine them.**
```auto
type alias NewRecord =
{ name : String }
type alias PersistedRecord =
{ id : Int, name : String }
type RemoteRecord e a b =
= NotAttempted a
| Saving a
| Persisted b
| Failure e
```
I thought I’d really hit on something here, but it is a royal pain to deal with this type. I have to write code to handle every case everywhere it appears, because no helper I have come up with seems useful. The “Saving” state is particularly annoying, and only occasionally meaningful. The fact that the encapsulated record might be different is an incredible pain. It actually makes me miss just putting a “-1” in the ID and calling it a day.
That said, I never reference an ID when I don’t have one now!
**But it really makes me ask,** just how many of my Elm bugs had to do with handling persisted and un-persisted data with one record type in the first place? I think my answer has to be: Not many, it just felt ugly and object-oriented.
What do you guys think? I think I am still missing something.
---
### Author:  [@Brian\_Carroll](https://discourse.elm-lang.org/u/Brian_Carroll)
#### Post date: [April 12, 2018, 5:59pm UTC](https://discourse.elm-lang.org/t/strategies-for-representing-persisted-and-un-persisted-remote-data/1115/2 "2018-04-12T17:59:45Z")
Really interesting to hear about your journey with this!
I read a great article a while ago that I can’t find now, frustratingly! It had some great recommendations on how to structure code with `Maybe`, `Result` and friends. Might have been Haskell examples.
The idea was to write your logic as if you always have the data and it’s always successful. Then use `Maybe.map`, `Maybe.andThen`, etc to deal with the `Nothing` cases. It’s a really nice approach and cleans things up a lot. You get rid of all the `case` expressions.
This would apply most directly to your strategy #2, with the database ID as a `Maybe Int`. I think this is something you tend to lose when you do a custom union type rather than a standard one. You could implement your own `map` and `andThen`, but experimentation is a lot easier if you can try out functions from the `Maybe` package, or even `Maybe.Extra`.
You might be able to create a custom `map` over your union type in strategy #3, I’m not sure how well it would work.
---
### Author:  [@rtfeldman](https://discourse.elm-lang.org/u/rtfeldman)
#### Post date: [April 12, 2018, 6:00pm UTC](https://discourse.elm-lang.org/t/strategies-for-representing-persisted-and-un-persisted-remote-data/1115/3 "2018-04-12T18:00:49Z")
Can you write out an example spec? Like “I have a form which asks the users for these pieces of information, I want to persist these parts of it to the server under these circumstances, I want to gracefully handle errors in these ways,” etc.
---
### Author:  [@wmakley](https://discourse.elm-lang.org/u/wmakley)
#### Post date: [April 12, 2018, 6:11pm UTC](https://discourse.elm-lang.org/t/strategies-for-representing-persisted-and-un-persisted-remote-data/1115/4 "2018-04-12T18:11:35Z")
> [@Brian\_Carroll](#):
>
> You might be able to create a custom map over your union type in strategy #3, I’m not sure how well it would work.
Haha not well! I tried it. The two record types are a real problem, and what I found is that the way I wanted a `map` to work changed in the context of where I was using the record. It’s very difficult to explain, but a big problem I’m having is that **context matters**. The `RemoteData` packages that are available publically do not match my context well, and my RemoteRecord really isn’t perfect either.
---
### Author:  [@wmakley](https://discourse.elm-lang.org/u/wmakley)
#### Post date: [April 12, 2018, 6:12pm UTC](https://discourse.elm-lang.org/t/strategies-for-representing-persisted-and-un-persisted-remote-data/1115/5 "2018-04-12T18:12:25Z")
> [@rtfeldman](#):
>
> Can you write out an example spec? Like “I have a form which asks the users for these pieces of information, I want to persist these parts of it to the server under these circumstances, I want to gracefully handle errors in these ways,” etc.
You mean like as an exercise in design discipline before my next attempt? Or would you like to see one for my latest train wreck? 🙂
I think the model I want to move toward is using a “Form” type as an intermediary that deals only with strings, and converting to an API record type when it’s time to try and persist it.
---
### Author:  [@rtfeldman](https://discourse.elm-lang.org/u/rtfeldman)
#### Post date: [April 12, 2018, 6:17pm UTC](https://discourse.elm-lang.org/t/strategies-for-representing-persisted-and-un-persisted-remote-data/1115/6 "2018-04-12T18:17:47Z")
Not code, just a description of what the code will need to accomplish! Maybe someone here can come up with a nice approach.
It’s easier to think about things to try in the context of a specific problem statement. 😃
---
### Author:  [@wmakley](https://discourse.elm-lang.org/u/wmakley)
#### Post date: [April 12, 2018, 6:39pm UTC](https://discourse.elm-lang.org/t/strategies-for-representing-persisted-and-un-persisted-remote-data/1115/7 "2018-04-12T18:39:29Z")
Okay, here we go:
**App’s purpose:** Create printable document consisting of some header information about a job (contract), and some lines representing items in that contract. All user input will be persisted as soon as possible, so they don’t lose work. Ideally, it should not be **possible** to enter invalid data. We are using a REST JSON API for each record type.
1. Given that we have a contract, we want to start a new document _about that contract_ and show some details about the contract.
2. We want to wait until the user makes a minimal change to save the document, to avoid database clutter.
A minimal change will be “adding a line to the document” by clicking “Add Line”.
3. When the user clicks “Add Line”, we have to persist the document, and then use its ID to persist the line so that it can be modified. (I would like to avoid persisting the line immediately, but all the line edits involve adding **other** records that reference the line that we won’t go into, so we have to do that first.)
4. While the document is saving, the document needs to be “read-only”, to avoid complications.
5. We want to show the new line immediately, but disallow editing until it has been persisted, to make the page feel snappier.
6. Subsequent “Add Line” clicks will skip step 3 (save the document), but still need to wait until they are persisted for editing.
The main pain point I had is the app will always end up in a happy state eventually where we have all persisted records and the user can do anything they want, but I kept having to check if I was in that state everywhere I dealt with the line and document types.
---
### Author:  [@justinmimbs](https://discourse.elm-lang.org/u/justinmimbs)
#### Post date: [April 12, 2018, 7:38pm UTC](https://discourse.elm-lang.org/t/strategies-for-representing-persisted-and-un-persisted-remote-data/1115/8 "2018-04-12T19:38:04Z")
I’ll describe approaches I’ve used that I’ve been happy with.
For records that are persisted in a database and have an arbitrary key, instead of storing that key in the model of the record, I would keep a `Dict` of those records indexed by the key. So instead of storing something like
```elm
records = List { id : Int, name : String }
```
I would instead store
```elm
records = Dict Int { name : String }
```
where the `Int` is the arbitrary key.
This gives a consistent model for valid records, whether they’re persisted or not.
Let’s say the type for a valid record is `Record`; I would have a separate type for unvalidated forms, `RecordForm`. You can convert between the two as needed:
```elm
recordToForm : Record -> RecordForm
formToRecord : RecordForm -> Result RecordFormErrors Record
```
Like you said, the advantage is that the `RecordForm` only has to deal with types that relate to user inputs. It also makes it more natural to define an `emptyForm` value without resorting to definitions like `id = -1`.
This approach to validation has been mentioned before and there is at least [one library that promotes the approach](http://package.elm-lang.org/packages/stoeffel/elm-verify/latest/).
At the top-level model, I tend to store server data separately from any other state. This makes it simple to reload server data, as it can safely supersede existing server data without needing to resolve conflicts with user-entered data or any other state.
```elm
type alias Model =
{ serverData :
{ records : Dict RecordId (Remote Record)
, ..
}
, clientData :
{ editedRecords : Dict RecordId RecordForm
, newRecord : Maybe RecordForm
, ..
}
}
```
It sounds like you’re on the right track! I hope this helps.
---
### Author:  [@wmakley](https://discourse.elm-lang.org/u/wmakley)
#### Post date: [April 12, 2018, 10:29pm UTC](https://discourse.elm-lang.org/t/strategies-for-representing-persisted-and-un-persisted-remote-data/1115/9 "2018-04-12T22:29:47Z")
Thank you, that looks excellent! You just gave me a lot of brain waves. I was definitely heading toward separate form types and separate server data repositories slowly, but you just got me there a lot quicker, and I have already had some success with a repository type like that.
I do have a few questions about specific implementation details, but I will need some time to write it up.
---
### Author:  [@ericgj](https://discourse.elm-lang.org/u/ericgj)
#### Post date: [April 12, 2018, 10:59pm UTC](https://discourse.elm-lang.org/t/strategies-for-representing-persisted-and-un-persisted-remote-data/1115/10 "2018-04-12T22:59:00Z")
I’m not sure I understand your exact pain point - there are a number of different issues wrapped up together here. But I’m glad for the discussion because I think they are issues most of us face one way or the other.
If your document data is not too big, I think I would try to model every edit within a document at the Document level. In other words, “Add Line” triggers a backend call that returns _data representing the entire changed document_. Which you use to replace the _whole document_ in your Elm model. And some edit within of an existing line likewise triggers a backend call that returns the _entire_ document. For one thing, then you are only dealing with the `Saving` state of the Document, not individual Lines.
This may be completely obvious to you and more or less what you’re doing, so I don’t know if that helps at all.
> The main pain point I had is the app will always end up in a happy state eventually where we have all persisted records and the user can do anything they want, but I kept having to check if I was in that state everywhere I dealt with the line and document types.
I wouldn’t think you would need to check this state (i.e. case on it) more than once in the view – where else did you find you needed to do it?
---
### Author:  [@wmakley](https://discourse.elm-lang.org/u/wmakley)
#### Post date: [April 13, 2018, 3:27pm UTC](https://discourse.elm-lang.org/t/strategies-for-representing-persisted-and-un-persisted-remote-data/1115/11 "2018-04-13T15:27:43Z")
Yeah, that is my whole problem: lots of issues! It’s difficult to describe the minimal case. **Situation:** My model has to transition between un-persisted and persisted states. **Issue:** How to I represent un-persisted vs persisted data so that both are easy to work with?
If it is un-persisted, it may be missing all kinds of strings and keys that it will have later, but I still need to let the user change it. Once it is persisted, it will _always_ have those values, so it is annoying for them to be `Maybe`.
> [@ericgj](#):
>
> I wouldn’t think you would need to check this state (i.e. case on it) more than once in the view – where else did you find you needed to do it?
When updating the data from the server, via much the model you described. I need to case over the data every time the user makes a change, if I wrap it in a union type.
---
### Author:  [@z5h](https://discourse.elm-lang.org/u/z5h)
#### Post date: [April 13, 2018, 6:26pm UTC](https://discourse.elm-lang.org/t/strategies-for-representing-persisted-and-un-persisted-remote-data/1115/12 "2018-04-13T18:26:26Z")
I’m having a lot of success with the following pattern (using Firebase).
1. Subscribe to data I’m interested in, as I become interested. The incoming data is the only source of truth.
2. I cannot change my model locally. I can only craft and send an update over a port (which gets applied to Firebase).
3. If/when the data gets updated successfully (by me or someone else), I’ll be notified because I’m subscribed to changes, and my model will get updated appropriately.
I have an Id type and sentinel value of empty, which is used to determine if something exists in the database, or has not been persisted.
PROs: simple, has always worked.
CONs: if it doesn’t work, the experience would be: I tried an update and nothing happened. No error, etc.
---
### Author:  [@Chadtech](https://discourse.elm-lang.org/u/Chadtech)
#### Post date: [April 13, 2018, 6:59pm UTC](https://discourse.elm-lang.org/t/strategies-for-representing-persisted-and-un-persisted-remote-data/1115/13 "2018-04-13T18:59:18Z")
I think I am in the same position as you @wmakley , and I have been thinking about the same questions. At work and in my own projects at home I have been trying to manage persisted and unpersisted data and I think I have gone through both of your ideas of different record types for persisted and unpersisted data, and having a `Maybe Id` kind of thing.
Eventually a coworker of mine recommended something like what @justinmimbs suggested, and I would say it works pretty well. Its my favorite option right now. At work we’ve been calling the `Dict Id Record` thing `Entities`, and in my personal projects Ive been calling it a `Db`. I think the only draw back is if you are passing around ids and `Db` around, theres the logical possibility that your model is saying something like the selected record is such and such id, but theres no record with that `Id` in your `Db`, and accounting for that possibility is a real drag.
I have a library where I am trying to formalize good practices on this. Its at [http://package.elm-lang.org/packages/Chadtech/id/3.2.0](http://package.elm-lang.org/packages/Chadtech/id/3.2.0) . Right now its not documented very well and doesnt reflect best practices as I currently know them. PRs welcome! But whether it be in my existing package or somewhere else I would love to collaborate on a community-wide solution to this general problem.
---
### Author:  [@ericgj](https://discourse.elm-lang.org/u/ericgj)
#### Post date: [April 14, 2018, 4:13am UTC](https://discourse.elm-lang.org/t/strategies-for-representing-persisted-and-un-persisted-remote-data/1115/14 "2018-04-14T04:13:53Z")
> All user input will be persisted as soon as possible, so they don’t lose work. Ideally, it should not be possible to enter invalid data.
> If it is un-persisted, it may be missing all kinds of strings and keys that it will have later, but I still need to let the user change it.
Between these statements I only have a fuzzy sense of what the intended UX is. On the one hand you are describing a situation where the aim is to save as soon as possible; on the other the aim is to save only when the input is valid. Yet it sounds like there is no explicit user ‘submit’ action, is that right? Or are you saying the “Add Line” is a submit action _after_ the user enters the contents of the new (not yet persisted) Line?
And is it possible to be a little more concrete, for instance give an example or two of cases where there is invalid data and you don’t want to save it yet?
In general if you are dealing with wanting to validate input, and edit validated models, I agree you are best served by separate data structures and something like `recordToForm` and `formToRecord` (aka `validate`) functions as @justinmimbs outlined.
And how/when do you want to communicate to the user that their input is invalid, or is this not needed?
---
### Author:  [@wmakley](https://discourse.elm-lang.org/u/wmakley)
#### Post date: [April 15, 2018, 4:25pm UTC](https://discourse.elm-lang.org/t/strategies-for-representing-persisted-and-un-persisted-remote-data/1115/15 "2018-04-15T16:25:48Z")
Hey me too, but thanks for pointing it out. As usual, I was up against a deadline without a clear idea of how to proceed. This doesn’t work as well with Elm as it does for me with Rails CRUD apps.
Let me break it down to a VERY small almost incidental part of this app.
1. The user needs to select an item from a dropdown.
2. They need the ability to add option to the dropdown, if the desired option is not available.
3. The form to create a new option has two fields, and both are required. The “submit” button should not be enabled until both fields are filled in with valid data. (The user should never see an error, just a clear indication of what they need to enter, primarily because parsing and displaying errors from the API would be even more code I don’t have time for.)
4. When the user submits their new option, it will selected automatically.
Behind the scenes, each item can only be selected after it has been persisted, and has a database ID. So when the user submits the form, we need to send a “new” record to the API without an ID, and get a “persisted” back with an ID. Then we can add that record to the select box.
I coded my API functions to all work on the same type of record, so when I need to create a new one I give it a record with `Nothing` for an ID, and get back a record with `Just Int`. This little hitch is constantly annoying when I can usually gurantee that the ID will be one or the other given the context. I always persist everything as soon as possible so the users don’t lose work.
---
### Author:  [@Brian\_Carroll](https://discourse.elm-lang.org/u/Brian_Carroll)
#### Post date: [April 16, 2018, 7:53pm UTC](https://discourse.elm-lang.org/t/strategies-for-representing-persisted-and-un-persisted-remote-data/1115/16 "2018-04-16T19:53:38Z")
> [@wmakley](#):
>
> I coded my API functions to all work on the same type of record
That’s the key design decision right there, I think. Once you decide to use the same record type everywhere, you fundamentally have to have `case` or `map` all over the place. No way round it.
Your other option could be to rewrite the function signatures to take the ID as a separate argument from the rest of the stuff.
```
type alias DatabaseId = Int
type alias OtherStuff = {field1: String, field2: String}
type ThingModel = ThingModel (Maybe DatabaseId) OtherStuff
viewUnsavedThing : OtherStuff -> Html Msg
viewSavedThing : DatabaseId -> OtherStuff -> Html Msg
```
You may find that you only need one or two `case` expressions, in the top-level functions of your view and update.
---
### Author:  [@ericgj](https://discourse.elm-lang.org/u/ericgj)
#### Post date: [April 17, 2018, 5:10am UTC](https://discourse.elm-lang.org/t/strategies-for-representing-persisted-and-un-persisted-remote-data/1115/17 "2018-04-17T05:10:48Z")
> So when the user submits the form, we need to send a “new” record to the API without an ID, and get a “persisted” back with an ID.
To add to what @Brian_Carroll said the stuff you send to the backend does not have to match what you get back from the backend – assuming that you control how the backend works. It’s a kind of dogma drilled into us that we are passing back and forth models (objects that we C/R/U/D), but it doesn’t have to be that way. In a lot of situations we are just modifying an aspect of a resource, for instance. Or there might be different ways of creating a resource, with more or less data attached.
I think of the bits of data we encode in messages to the backend less as _models_ and more as _Msgs_ – Msgs that are used by the backend to init and update persisted models. Just like Msgs in an Elm app, they might contain whole models; or they might contain just bits of data used to update models in various application-specific ways.
So for instance your dropdown example. I could be wrong, but it sounds like you are implementing something like a [combobox](https://en.wikipedia.org/wiki/Combo_box), but with two fields instead of one for the “manual entry” option? Setting aside the validation, and assuming the two fields are String, it seems like you could have something like this (simplified):
```elm
type ComboBoxValue
= Selected ID
| Entered String String
type Msg
= Submit ComboBoxValue
| Submitted (Result Http.Error Model)
update : Msg -> Model -> (Model, Cmd Msg)
update msg model
case msg of
Submit value ->
case value of
Selected id ->
( model, Task.attempt Submitted (submitSelected id model) )
Entered field1 field2 ->
( model, Task.attempt Submitted (submitEntered field1 field2 model) )
-- ...
```
---
### Author:  [@dmy](https://discourse.elm-lang.org/u/dmy)
#### Post date: [April 17, 2018, 9:34am UTC](https://discourse.elm-lang.org/t/strategies-for-representing-persisted-and-un-persisted-remote-data/1115/18 "2018-04-17T09:34:02Z")
For what it’s worth, here is what I’m doing at the moment. This is a work in progress so I do not yet have enough hindsight but it seems to work well.
My objects are records like:
```auto
type alias Object =
{ id : ObjectId
...
}
```
and their `id` are like:
```auto
type alias Id =
Int
type ObjectId
= ObjectId Id
| LocalObjectId Id
```
with an “unboxer” like:
```auto
unObjectId : ObjectId -> Id
unObjectId objectId =
case objectId of
Objectd id ->
id
LocalObjectId id ->
id
```
The objects are stored in an opaque `Dict Id a` that I initialize with the “unboxing” function so I can actually have several ones using different `id` types.
Also:
- objects from the database have a `ObjectId Id` with a **positive integer** as the `id` (the one from the db)
- objects generated locally have a `LocalObjectId Id` with a **negative integer** as the `id` that is unique locally (using `Maybe.map ((+) -1) (minimum localIds) |> Maybe.withDefault -1` to generate one)
When objects are sent to the backend to be persisted, the backend generates a new unique db positive index and replace the negative local `id` by it, and sends back the object with the local `id` as a reference in the answer to allow the client to update the object.
This way:
- All objects, persisted or not, are stored in the same data structure, without using `Maybe`
- It is easy to filter reliably unpersisted ones to send them to the backend (using the record typed `id` field)
- Most functions using the objects do not care if they are local or not and I can use `Keyed` as every “unboxed” `id` is unique locally
- The typed `ids` are used to identify the objects in messages, so I cannot not inadvertently mix them up between different types of objects
The drawback is that I must be careful to avoid race conditions in messages that could use a local `id` after it has been changed to a database one, but this does not seem to be a problem specific to this solution. Also I store the `id` both as the dict key (unboxed) and in the object record (typed), so there is some slight duplication waste. I will see how it goes…
---
### Author:  [@wmakley](https://discourse.elm-lang.org/u/wmakley)
#### Post date: [April 18, 2018, 5:28pm UTC](https://discourse.elm-lang.org/t/strategies-for-representing-persisted-and-un-persisted-remote-data/1115/19 "2018-04-18T17:28:25Z")
@dmy I have considered a scheme like that! I have seen the negative ID thing done using SQL databases in a non-elm project, with satellite clients that update the master. It can definitely work.
You’re right to point out the race condition, something that I feel more Elm writings should touch on. I usually generate dumb “UID” values for my collections of GUI elements and wrap them up in a Tuple or record, and ignore the database ID’s entirely. I hate dealing with this implementation detail, and am trying to find ways to abstract it out. I would love an efficient ordered Dict type.
---
### Author:  [@joelq](https://discourse.elm-lang.org/u/joelq)
#### Post date: [April 18, 2018, 5:48pm UTC](https://discourse.elm-lang.org/t/strategies-for-representing-persisted-and-un-persisted-remote-data/1115/20 "2018-04-18T17:48:41Z")
> [@Brian\_Carroll](#):
>
> I read a great article a while ago that I can’t find now, frustratingly! It had some great recommendations on how to structure code with Maybe, Result and friends. Might have been Haskell examples.
>
> The idea was to write your logic as if you always have the data and it’s always successful. Then use Maybe.map, Maybe.andThen, etc to deal with the Nothing cases. It’s a really nice approach and cleans things up a lot. You get rid of all the case expressions.
@Brian_Carroll perhaps it was [Problem Solving with Maybe](https://robots.thoughtbot.com/problem-solving-with-maybe) ?
[Next page](https://discourse.elm-lang.org/t/strategies-for-representing-persisted-and-un-persisted-remote-data/1115.md?page=2)