Handling pagination and http request chaining

Learn
1

:wave: everyone.

I’m working on an interface using Spotify API.

When an User is logged, I use a command to fetch the playlist using: Get a list of current user’s playlists

As we can see, the playlist list is wrapped in a paging object. I use this decoder when firing my request:

listPlaylistDecoder : Decode.Decoder PlaylistsPagined
listPlaylistDecoder =
        decode PlaylistsPagined
        |> required "items" (Decode.list playlistDecoder)
        |> optional "next"  Decode.string "end"

When I’m back in the update loop, I check if I have “end” or the next url and I call the API until I get all playlists:

        LoadPlaylists (Ok playlist) ->
            case playlist.nextPage of
                "end" ->
                    ( { model | playlists = List.append model.playlists (List.map (\p -> { playlist = p, tracks = [] }) playlist.playlists) }, Cmd.batch (loadTracksBatch model.playlists)

                url ->
                    ( { model | playlists = List.append model.playlists <| List.map (\p -> { playlist = p, tracks = [] }) playlist.playlists }, (getPlaylists LoadPlaylists playlist.nextPage))

Then, I will start to get every tracks for each playlist with a Cmd.batch using: Get playlist’s tracks

We also have a paginated object wrapping the tracks… so I do the same thing with this decoder:

listTrackDecoder : Decode.Decoder TracksPagined
listTrackDecoder =
        decode TracksPagined
        |> required "items" (Decode.list trackDecoder)
        |> optional "next"  Decode.string "end"
        |> required "href" Decode.string

I keep href around to be able to link a track to its playlist.
When I’m in the update loop, I check once again if I have to use next until I finished.

        LoadTracksPlaylist (Ok tracks) ->
            case tracks.nextPage of
                "end" ->
                    ( { model | playlists = addTracksToPlaylist model.playlists tracks }, loadFeaturesBatch model.playlists)

                url ->
                    ( { model | playlists = addTracksToPlaylist model.playlists tracks }, (fetchTracks tracks.nextPage LoadTracksPlaylist) )

When I got all the tracks, I want to get the audio features from each track.

I get the id for all my tracks and start to call this endpoint with the maximum tracks id allowed (100).

I feel like I should accumulate the response and ask for more result without having to go back in the update loop but I don’t see how I can implement this. :thinking:

If you have any ideas, it would be great! :smile:

1 Like
2

This is a fun problem! The answer to your problem is to use Tasks so you can treat a whole chain of requests as a single command. I think you may want some recursive calls to Task.andThen :sweat_smile: .

Ideal world:

Ideally you can do something like this in your update

case msg of
  TriggerFetch ->
    (model, Task.attempt ReceivePlaylists (fetchAllPlaylistsWithTracks model.firstPlaylistUrl))

  ReceivePlaylists (Ok playlists) ->
    ({ model |   playlists = playlists }, Cmd.none)

Fetching a playlist + tracks

Assuming we have a task to fetch a list of of playlists/tracks from the API (more on that later), we can combo the two of them using Task.sequence and Task.andThen

fetchAllPlaylistsWithTracks : String -> Task Http.Error (List PlaylistWithTracks)
fetchAllPlaylistsWithTracks firstUrl =
  fetchAllPlaylists firstUrl
    |> Task.andThen (Task.sequence << List.map fetchTracksAndCombo)

fetchTracksAndCombo : Playlist -> Task Http.Error PlaylistWithTracks
fetchTracksAndCombo playlist =
  fetchTracks playlist.trackUrl
    |> Task.map (\tracks -> { playlist = playlist, tracks = tracks })

Recursive fetching of entities

That’s all great but we need a way to get all the playlists/tracks by making multiple requests to the API.

Both paginated APIs return data in the same format. We can create a parameterized type representing that:

type alias Paginated a = { items : List a, next : String }

Once we have this type, we can write a generic function that recursively combines tasks using Task.andThen until we reach last item.

fetchAll : (String -> Http.Request (Paginated a)) -> String -> Task (List a)
fetchAll requestFunction initialUrl =
  fetchNextPlaylist requestFunction [] initialUrl
    |> Task.andThen (fetchNextOrEnd requestFunction)
    |> Task.map Tuple.first

fetchNextOrEnd : (String -> Http.Request (Paginated a)) -> (List a, String) -> Task (List a, String)
fetchNextOrEnd requestFunction (items, next) =
  case next of
    "end" ->
      Task.succeed (items, "end")

    _ ->
      fetchNext requestFunction items next

fetchNext : (String -> Http.Request (Paginated a)) -> List a -> String -> Task (List a, String)
fetchNext requestFunction itemsSoFar url =
  requestFunction url
    |> Http.toTask
    |> Task.map (\response -> (response.items ++ itemsSoFar, response.next))

Putting it all together

We now can now define the fetchAllPlaylists and fetchTracks functions.

fetchAllPlaylists : String -> Task Http.Error (List Playlist)
fetchAllPlaylists firstUrl =
  fetchAll requestPlaylists firstUrl

fetchTracks : String -> Task Http.Error (List Track)
fetchTracks firstUrl =
  fetchAll requestTracks firstUrl

where requestPlaylists and requestTracks are the usual functions you’d write to make a single request

5 Likes
3

Thanks Joël, you are insightful as always. :pray:

It works wonders and the code is waaaayyy cleaner now. :+1:

1 Like
4

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.