Elm Media Primitives

[Dan_Abrams]

I want to preface this by saying I’m in a bit of a pickle and don’t know what to do. I began this project back in September as an Elm-Exploration in consultation with Evan, but he’s gotten very busy and I haven’t been able to get further response from him. I’m trying to get ready for my talk at Oslo Elm Day and I’m not sure what to do, so here goes:

One of the discussions around the original native/kernel code decision was that our code would be better in consultation with the community, so I’m publishing this here to try to get feedback and discussion. This work is substantially changed, for the better, from my previous attempts, and radically simplified, thanks in large part to Evan’s guidance.

Media Primitives
First of all, I want to cover what the Elm-Media-Primitives design is and isn’t, in its current incarnation. The eventual goal is to come up with a nice way to wrap the Html5 Media API, a pervasive and widely used api on the web. The media API covers several areas, including Web Audio (really useful for all you game makers), but generally all of these areas rest on media state management and playback control found in the plain ol’ DOM API.

The goal of Elm-Media-Primitives is to cover the most fundamental parts of these APIs, starting just with the parts in the DOM API. This is not a full featured package abstracting all of the features of the Media API away, it simply wraps the primitives that require kernel code. I am building a full featured library on top of this, and others can do the same if they have different needs.

The goal is to keep it simple, and hopefully eventually expand it to cover the necessary primitives in the other parts of the Media API: Media Stream, Media Source Extensions, and probably of most interest in Elm-world, Web Audio. All of these APIs will have to rest on the foundation laid by this one, though, and they don’t have the same drawbacks to using ports as playback control and state management do.

The fundamental challenge
HtmlMediaElement objects in javascript maintain their own state. This whole task would be a lot easier if I could feed the media elements my own timecode, in which case this package would work more like the various animation libraries in our ecosystem. Alas, it doesn’t work like this.

Obviously giving a media element a source, or clicking play, or pause, or changing the currentTime property changes the state of a media element. But even after you’ve caused that side effect, the state can continue to change. The currentTime updates continuously. The amount of video buffered changes as the browser loads the source, and it can be important to know how much data there is. The player can reach the end of the file and stop playing. And keeping track of that state in a pure, functional language is a challenge.

The new approach
My original approach was to rely on the Media API events firing to track changes in state. But based on conversations with Evan, I’ve abandoned this approach, at least for this primitives library.

Instead, I introduce a new function, getCurrentState : String -> Task Error State, which takes a media elements id, and returns it’s current state.

State is a fairly complex record looking like this:

type alias State =
    { id : String
    , mediaType : MediaType
    , playbackStatus : PlaybackStatus
    , readyState : ReadyState
    , source : String
    , currentTime : Float
    , duration : Float
    , networkState : NetworkState
    , buffered : TimeRanges
    , seekable : TimeRanges
    , played : TimeRanges
    , textTracks : List TextTrack
    }

Some of the fields are their own types, but I think it’s mostly self explanatory and don’t want to get bogged down in the weeds here.

A Question for the Community
I’d like to hear some opinions about this way of doing things. The thing I’m most uncertain about, and one I probably need to do some benchmarking to resolve, is should I just have one function that returns the whole state, or should I develop a separate way to query for individual fields.

If the performance implications aren’t too bad, my preference would be to return the whole state, but if it isn’t performant enough, I suppose I would need some sort of design where individual attributes are queried.

Any thoughts on what the best design for that might be.

Playback Control
I won’t belabor this too much, but my design also includes tasks that take the media element’s id as a string and effect playback, the one’s you’d expect, play, pause, seek.

One Miscellaneous
I also include a function that requires kernel code but is really more filling in an edge case, enabling the “mode” attribute on a TextTrack

textTrackMode : Types.TextTrackMode -> Html.Attribute msg

The problem is that while mode is a writeable attribute, it’s exposed from the HtmlMediaElement.textTracks field, which is read only.

To put it more succinctly, this exists to deal with a weird JavaScript API design quirk, and you can ignore. I just wanted to explain so no one’s wondering what it is.

If anyone has a solution that would let me take this out, I will be forever your friend for it, it annoys me that this is in here.

Why Should Media be an Elm Exploration
There are two reasons, one is that while most of these tasks can work as Ports instead, tasks are probably better from an API usability standpoint…but that’s minor.

The real reason is reusability. This is probably not a package that most users will ever use, instead, other packages wrapping the broader Media API will be developed. Even then, most users will probably not touch it, but use the products of those libraries: audio and video players, written in pure elm.

Some of that is overcome-able with web components, but the apis covered here are also basic work that needs to be done to work with the other Media APIs. Once implemented, we can grab images from cameras, sound from microphones, build music apps, complex 3D sounds for games, and more.

What I’m Seeking Feedback On
Overall, I’ve already gotten a lot of advice from Evan that I really appreciate, which has really let me simplify this design. My big question is the one I posed above, but any other feedback is wildly appreciated.

You can see my basic design in more depth here: https://github.com/danabrams/elm-media-primitives

Next Steps
I’ll be talking about this at Oslo Elm Day exactly one month from today, and hope to release a ports version of this package this week, as well as an accompanying, more all-inclusive wrapper on top of it, perhaps next week.

I’m worried about how hard the ports version will be to setup, if anyone have any advice on setting up an NPM package for the ports side, NPM is not something I know that I’m that expert in.

Thanks all!

EDIT: PS, thanks for reading this way too long post!

[mrozbarry]

Every now and then, I come out of my lurking slumber

Just going to address your last point about ports. I’d recommend something like this:

const withMediaPrimitives = (elmApp) => {
  elmApp.ports.yourStuffHere.subscribe(() => {
    // more of your stuff
  });
  return elmApp;
};

So if someone were to use your package, it would look like:

const app = withMediaPrimitives(Elm.Main.init({
  node: document.getElementById('elm'),
  // ... other stuff?
}));

That’ll let people compose their own stuff in, too.

In terms of the publishing process, it’s pretty straight forward. Set up an npmjs account, then in your terminal, npm login. I really recommend setting up 2fa with something like google authenticator when you create your account.

You would have to build up a bit of a package.json file, but that’s not too hard. npm init will start that process for you. Add things like npm add --peer elm@^0.19.0 will be helpful for people who installed elm via npm, too!

Your package.json file might look something like:

{
  "name": "elm-media-primitives",
  "version": "0.0.1",
  "main": "ports/with-media-primitives.js",
  "peerDependencies": {
    "elm": "^0.19.0"
  }
}

I recommend a path like ./ports/whatever.js so you can keep all your code in a single repo. I also think you might want to set up .npmignore and ignore your elm sources, so the npm install is just the ports code that you need.

In terms of actually making your port work, you’ll need to export your function. I recommend adding a package bundler (I really like rollup, but there are many options) to export your port augmenting function, since that will cover most cases like commonjs (allowing you to do something like const myFn = require('my-package');).

When you’re happy with what you have npm publish will publish to npmjs. You can, of course, publish as often as you want, just remember to update your version number

Hopefully I wasn’t too granular on that, not sure how familiar you are with npm/javascript package deployment.

• Alex

[Dan_Abrams]

Alex, thank you for this. Exactly the guidance I was looking for. I wish I had the power to award you a badge or something for it. Really great.

[marcw]

Hello @Dan_Abrams,

I think the choice to return the full state is a good one.
I have never used textTracks, but I understand that some people might, and every separation I tried to come up with has serious drawbacks for different use cases, and thus needing multiple Tasks to get all data.
I think Adding logic to ‘compose’ which values are returned should be part of a higher-level module (if at all).

Have you also considered exposing the muted state? It might change unknown to the main program, especially when the controls are displayed.
Also, the volume level might be useful.

Cheers, marc

[Dan_Abrams]

@marcw

Have you also considered exposing the muted state?

Oh yes, of course. The point of this post is to propose the pets that require kernel code.

Mute is well worth having but can be done in normal Html Attributes, and would exist in a higher-order library on top of this.

[marcw]

I don’t remember for which browser we had to implement this, but at work we needed to set both the muted html attribute and the js property for one of the less common browsers (I guess Safari or Edge) and introduce a port for this.
Edit: Maybe I’ll have time tomorrow to look for it.

[Dan_Abrams]

I’ve definitely used muted as an attribute on several versions of Safari…

Worth noting that both attributes and properties can be set in the current version of elm/html.