Strategies for fetching only the required data for a page in an SPA

Learn
1

I have a multi-page Elm app which needs various different bits of data for the different pages and I am wondering what the best strategy is for dealing with them. In particular, I have an ‘all locations’ page that needs to show all the locations in the system which is a very large amount of data in some situations but I also have pages for viewing a single location. And the same for other kinds of domain objects.

I feel like I would like to keep track of whether I’ve fetched all the locations already, or maybe just one or two locations (due to viewing one or two of them individually) or maybe none of them at all. It has lead me to experiment with a type like:

 type ActivityTemplateDict
    = AllDict (Dict Int ActivityTemplate)  -- Represents all the data having been fetched already
    | SomeDict (Dict Int ActivityTemplate) -- Represents the individual locations that have been fetched already
    | NotRequested

It is motivated by the volume of the location data as I do not wish to request it all unless I absolutely have to and I don’t want to end up re-requesting it when I already have it.

I don’t have a lot of experience with this kind of stuff. Is there an established best practice approach? Or any reading recommendations?

2 Likes
2

Is it a single-user application? Otherwise, wouldn’t you want to load the data fresh on each page init to avoid stale data?

You could let http mechanisms handle caching instead of Elm. I’ve done this before with some success (though it probably could be much better if I had the time to master http caching, tweak memcached etc).

3

I think I would use a dictionary as a cache. If I can get the data from the cache all is good, if not make a request for the data, and cache it upon return for future use. Something like this, roughly speaking, there may errors in here, but this does compile. I’m also eliding over some other implementation details, such as initialization, views etc.

import Dict exposing (Dict)
import Http exposing (Error)
import Task exposing (Task)


type alias Cache =
    Dict String String


type alias Model =
    { cache : Cache
    , key : String
    , locations : List String
    }


model : Model
model =
    { cache = Dict.empty
    , key = ""
    , locations =
        [ "foo", "bar", "baz" ]
        -- get these via an API call, or load them via flags, etc
    }


type Msg
    = GetItem String
    | GetAllItems (List String)
    | GetItemFromCache String (Result Http.Error String)


update : Msg -> Model -> ( Model, Cmd Msg )
update msg model =
    case msg of
        GetAllItems locations ->
            ( model
            , Cmd.batch
                (List.map
                    (\key ->
                        Task.attempt (GetItemFromCache key) (getItem key model.cache)
                    )
                    locations
                )
            )

        GetItem key ->
            ( model, Task.attempt (GetItemFromCache key) (getItem key model.cache) )

        GetItemFromCache key result ->
            case result of
                Ok data ->
                    ( { model | cache = Dict.insert key data model.cache }, Cmd.none )

                Err err ->
                    -- add error handling here
                    ( model, Cmd.none )


getItem : String -> Cache -> Task Http.Error String
getItem key cache =
    case (Dict.get key cache) of
        Just item ->
            Task.succeed item

        Nothing ->
            Http.getString "https://path-to-api?query=key"
                |> Http.toTask
4

Good point about it getting stale. We have another project here that for other reasons does a ‘sync’ request to the server that grabs anything that has changed since the last update so I was planning to move in that direction. A large initial download that is kept up to date with further sync requests. The nature of the application also feels like it doesn’t suffer too much from these issues. Slow churn on the data I guess. But I can’t rely on that forever.

Part of the issue is that the Locations data is like 20-30mb for some of our large clients. I never thought it would be that large (foolish me.) Having it client side is handy for doing quick searches in various search fields but maybe those searches should be moved to the server and the locations removed from the client side given the size of the data and the impacts that it has.

5

Thanks @John_Grogan, I think I can see what is going on there. My concern though is figuring out whether or not you have requested all the locations yet. If the user navigates to the ‘all locations’ page half way through some session then how do I figure out if I should request all the locations from the server or if I already have them in the cache? I can see that your approach works for just viewing one location at a time but I’m not sure how it would work for a list of all locations.

6

Yeah, that’s a lot of data. Just to be clear, I didn’t mean to imply ‘just use http caching’ is the magical answer for everything. I’m going to be facing a similar situation soon re. top-level search functionality and would love to hear how others are handling it.

7

If you have control over the API and the responses it can create, I would perhaps request a simple, lightweight list of strings of the locations, request that on load and store it in the model. That way when you go to the ‘all locations’ page, you can map over that list, and call the Task.attempt function as on the GetItem case.

I’ve updated the code above to reflect this.

8

Thanks for the thought. That’s an option though I’d be concerned that it could trigger 100s or 1,000s of simultaneous single location requests rather than a single bulk request. I’ll consider it though.

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.