I want to preface this by saying I’m in a bit of a pickle and don’t know what to do. I began this project back in September as an Elm-Exploration in consultation with Evan, but he’s gotten very busy and I haven’t been able to get further response from him. I’m trying to get ready for my talk at Oslo Elm Day and I’m not sure what to do, so here goes:
One of the discussions around the original native/kernel code decision was that our code would be better in consultation with the community, so I’m publishing this here to try to get feedback and discussion. This work is substantially changed, for the better, from my previous attempts, and radically simplified, thanks in large part to Evan’s guidance.
Media Primitives
First of all, I want to cover what the Elm-Media-Primitives design is and isn’t, in its current incarnation. The eventual goal is to come up with a nice way to wrap the Html5 Media API, a pervasive and widely used api on the web. The media API covers several areas, including Web Audio (really useful for all you game makers), but generally all of these areas rest on media state management and playback control found in the plain ol’ DOM API.
The goal of Elm-Media-Primitives is to cover the most fundamental parts of these APIs, starting just with the parts in the DOM API. This is not a full featured package abstracting all of the features of the Media API away, it simply wraps the primitives that require kernel code. I am building a full featured library on top of this, and others can do the same if they have different needs.
The goal is to keep it simple, and hopefully eventually expand it to cover the necessary primitives in the other parts of the Media API: Media Stream, Media Source Extensions, and probably of most interest in Elm-world, Web Audio. All of these APIs will have to rest on the foundation laid by this one, though, and they don’t have the same drawbacks to using ports as playback control and state management do.
The fundamental challenge
HtmlMediaElement objects in javascript maintain their own state. This whole task would be a lot easier if I could feed the media elements my own timecode, in which case this package would work more like the various animation libraries in our ecosystem. Alas, it doesn’t work like this.
Obviously giving a media element a source, or clicking play, or pause, or changing the currentTime property changes the state of a media element. But even after you’ve caused that side effect, the state can continue to change. The currentTime updates continuously. The amount of video buffered changes as the browser loads the source, and it can be important to know how much data there is. The player can reach the end of the file and stop playing. And keeping track of that state in a pure, functional language is a challenge.
The new approach
My original approach was to rely on the Media API events firing to track changes in state. But based on conversations with Evan, I’ve abandoned this approach, at least for this primitives library.
Instead, I introduce a new function, getCurrentState : String -> Task Error State
, which takes a media elements id, and returns it’s current state.
State is a fairly complex record looking like this:
type alias State =
{ id : String
, mediaType : MediaType
, playbackStatus : PlaybackStatus
, readyState : ReadyState
, source : String
, currentTime : Float
, duration : Float
, networkState : NetworkState
, buffered : TimeRanges
, seekable : TimeRanges
, played : TimeRanges
, textTracks : List TextTrack
}
Some of the fields are their own types, but I think it’s mostly self explanatory and don’t want to get bogged down in the weeds here.
A Question for the Community
I’d like to hear some opinions about this way of doing things. The thing I’m most uncertain about, and one I probably need to do some benchmarking to resolve, is should I just have one function that returns the whole state, or should I develop a separate way to query for individual fields.
If the performance implications aren’t too bad, my preference would be to return the whole state, but if it isn’t performant enough, I suppose I would need some sort of design where individual attributes are queried.
Any thoughts on what the best design for that might be.
Playback Control
I won’t belabor this too much, but my design also includes tasks that take the media element’s id as a string and effect playback, the one’s you’d expect, play, pause, seek
.
One Miscellaneous
I also include a function that requires kernel code but is really more filling in an edge case, enabling the “mode” attribute on a TextTrack
textTrackMode : Types.TextTrackMode -> Html.Attribute msg
The problem is that while mode is a writeable attribute, it’s exposed from the HtmlMediaElement.textTracks field, which is read only.
To put it more succinctly, this exists to deal with a weird JavaScript API design quirk, and you can ignore. I just wanted to explain so no one’s wondering what it is.
If anyone has a solution that would let me take this out, I will be forever your friend for it, it annoys me that this is in here.
Why Should Media be an Elm Exploration
There are two reasons, one is that while most of these tasks can work as Ports instead, tasks are probably better from an API usability standpoint…but that’s minor.
The real reason is reusability. This is probably not a package that most users will ever use, instead, other packages wrapping the broader Media API will be developed. Even then, most users will probably not touch it, but use the products of those libraries: audio and video players, written in pure elm.
Some of that is overcome-able with web components, but the apis covered here are also basic work that needs to be done to work with the other Media APIs. Once implemented, we can grab images from cameras, sound from microphones, build music apps, complex 3D sounds for games, and more.
What I’m Seeking Feedback On
Overall, I’ve already gotten a lot of advice from Evan that I really appreciate, which has really let me simplify this design. My big question is the one I posed above, but any other feedback is wildly appreciated.
You can see my basic design in more depth here: https://github.com/danabrams/elm-media-primitives
Next Steps
I’ll be talking about this at Oslo Elm Day exactly one month from today, and hope to release a ports version of this package this week, as well as an accompanying, more all-inclusive wrapper on top of it, perhaps next week.
I’m worried about how hard the ports version will be to setup, if anyone have any advice on setting up an NPM package for the ports side, NPM is not something I know that I’m that expert in.
Thanks all!
EDIT: PS, thanks for reading this way too long post!