Elm-Media - a Wrapper on the HTML5 Media API

[Dan_Abrams]

UPDATE: In response to Luke’s feedback, I’ve written out my proposal in a little bit more detail here (Ok, a lot more detail). Would love more feedback on the proposal, rather than the process. If people think it wise, I’ll start a new post and just do a do-over.

I’ve been working diligently on a wrapper for the HTML5 Media API for several months, and am ready to release it to this fine group for them to rip it to shreds and tell me what I can do better.

It can be found here.

Let me state at the top:

1. It uses native code…yes, I know, I know.

2. Using native code means I can’t publish to Elm-Packages. Tell me about it.

3. This potentially makes more work for Evan. Yes, it does. I’m sorry, Evan. Not my intention, I’m just passionate about this API and am hoping to push the ball forward in making it easier in Elm (about which I’m also passionate).

4. Just use ports…This is fine for one off media apps like a podcast player or even a youtube clone, but one of, if not THE most used scenario for the media API is a reusable player. My goal is that this package will someday be used by myself and others to make best-in-class audio and video players that are easy to configure, reusable views. To that end, I’ve created an example here

I understand the safety concerns for not allowing native modules and agree with them. I also strongly agree with not just wrapping an existing JavaScript library in Elm, but rather rewriting in Elm. To that end, my hope, if I’m not laughed out of the room with this package, is to also add a good Elm API to the Media Source Extensions API so that we can have our own implementations of HLS and MPEG-DASH without relying on JavaScript.

However, this package only uses modules to wrap native Media API calls. This is the only way to do it, other than ports.

I’m not sure what the submission process will be like, but this would be the exact kind of thing one might submit to elm-explorations someday. I hope that it can become, or at least inspire, a package that can help us use HTML5 Media in elm apps, hopefully sooner rather than later.

Thanks all, and don’t be afraid to really be brutal in your critiques. I have lots of Japanese Whiskey on hand, and I want this to be as good as it possibly can be. Just please, no criticisms of my hair cut.

Thanks for your time and input.

[luke]

Meta-reply:

It uses native code…yes, I know, I know.

Even with closing off access to Native in 0.19, these kinds of experiments are still useful and encouraged, and will get more attention after 0.19 is out and the focus turns more to the web platform. If you have an idea for an API it’s of course understandable that kernel code will be needed to try it out. A post like this is more likely to get attention if it is framed more like a proposal than as a package that “breaks the rules”, and so people shouldn’t or can’t use it. For my part, I started reading this thinking I was going to have another conversation about Native, and then found out it’s actually a nice example implementation of the media API!

With that said, API ideas make the biggest impact when they are heavily evidence-based and have the weight of a lot of preliminary discussion and design work. It’s hard to evaluate this code on its own without knowing the design constraints and the tradeoffs that you chose while putting it together. There are too many web APIs for Evan and the folks on the core team to know them by heart, so teaching that by telling the story of why you think this design is the best will be the most effective at drawing useful feedback, because there’s very little additional investigative work that needs to happen in order to just begin taking a look. I think the very best of this process is attained when that story is built up over several conversations and design iterations before presenting code at all.

To summarize, framing your web platform API idea as a package-that-actually-uses-native-so-you-can’t-install-it-or-use-it-in-0.19 is probably a good way to get a “just use a port” type of reply, and even though writing the code is fun it’ll stunt any long-term potential. There’s no official process for putting things in elm-explorations at this time, but my guess would be that it ends up looking more like the process described above. You clearly care a lot about this and put in a lot of effort to make this and present it here. Give it the presentation that it deserves!

---

Edit:
Also, if you have any recommendations for Japanese Whiskeys, I would love to receive a DM pointing me in the right direction there.

[Dan_Abrams]

Thanks for the feedback, Luke.

I agree somewhat, but I got carried away, especially since this was my first native module and wanted to make sure I understood some things about Elm API design. If I had made the proposal back in November when I started working on this, the design really would have been terrible and un-Elm-like. When I look back over my notes from that period…just yuck.

With the understanding that I may have to abandon/totally redesign this package, I plan to layout the story of how to use this API very soon. It was partially a design proposal and partially a learning exercise.

In the meantime, if you have any feedback on what a good way to lay that out, even an example of where someone else did, I’m happy to do the extra work.

In the future I promise to get others involved sooner, now that I have some more mastery over what I’m doing, and a network of friends in the community to offer ideas, criticisms, etc.

[Dan_Abrams]

In regards to the Japanese whiskey, I just had a great one last week have to look up the name tho.

[Dan_Abrams]

Update, it was this: https://www.masterofmalt.com/whiskies/yoichi/yoichi-single-malt-whisky/

It was expensive, but we had a discount and it was half off (at a bar).

[luke]

I do think I might have overemphasized the “talk first, code later” aspect. That is the one-liner description of how stuff gets done in elm-lang, but for this I understand that you’ve been talking with Richard at Elm meetups about this design. So in that case, all I would hope to see is:

• None of the caveats about Native stuff. Like I mentioned, this is a proposal and so the goal is not to side-step Elm’s process, but to eventually have some official kernel code come into place to provide a web platform feature.
• Some mention about how you’ve been working on this at meetups and how you talked with Richard about your design and maybe some of the interesting highlights of those conversations. (Talking to Richard isn’t a required part of any official process, it just happens to be a notable part of yours!)
• Brief explanation of the choices you made in the design, or a link to them if you wrote them out somewhere else
• I didn’t bring this up before because you did it quite well, but delineating what you want to happen next is really important. Your requests for feedback from the community accomplish that nicely.

With the understanding that I may have to abandon/totally redesign this package

I didn’t mean to give the impression that you should expect to throw away your code, and I apologize for not being clear enough! There are times where I’ve written code too early and had to throw it away, but in those cases I was way, way, way behind what you’ve already done in terms of design effort. If you decide to redo the whole API it will be because of feedback that leads to that conclusion and not because of arbitrary process stuff.

When I look back over my notes from that period

This is also really cool! We would benefit greatly from understanding the evolution you went through while working on this.

In the future I promise to get others involved sooner

It sounds like you totally did! My request is just that a post with these goals focuses on how doing that got you to the point we’re at here, and not so much on the code itself and the fact that it contains native. I’m really excited to see how this turns out, and not the least of all because of how excited you are about it!

If you want to talk more about this privately you can always reach out to me personally at @luke on the slack, too

---

https://www.masterofmalt.com/whiskies/yoichi/yoichi-single-malt-whisky/

Thank you!

[Dan_Abrams]

I hear those suggestions completely and will try to accomplish them in the next few days.

[Dan_Abrams]

Also, I’m just lucky to have in-person access to Richard (among others) by virtue of living in the Northeast and not being in the least bit hesitant to be pushy pushy pushy.

[Dan_Abrams]

I’ve updated the first post in response to Luke’s feedback with some more information, located here: (https://github.com/danabrams/elm-media/blob/master/Proposal.md).

I went to art school. I want to hear your brutal critiques, because the work is what matters, and criticism will make it better.

[Dan_Abrams]

The proposal has been updated with a few typo fixes and some clarification for a question I was asked…In answering this question, I also decided to remove the everyFrame subscription.

Really hoping to get some feedback to make this better. Is my way of dealing with the object-oriented nature of the API decent? Is there a better way?

Can anyone find a better way to decode the TimeRange objects, preferably not using native?

[mcordova47]

It doesn’t look to me like you could decode TimeRanges without native code based on the spec. It only exposes start and end methods, so even if it’s possible, anything else would be relying on implementation details, right?

[Dan_Abrams]

Yes. But maybe someone has a trick up their sleeve.

[Brian_Carroll]

Can’t think of a way to avoid JS code but have you looked into custom elements?
I’ve seen them come up in a few discussions here as an alternative way to write native code. Perhaps you could wrap the video element in a custom element with a more Array-like API for the TimeRanges?

I’m also wondering if the Proxy API might come in handy to create the new JS class for TimeRange. Maybe you could proxy Array and replace its getter function.

I haven’t actually used any of these techniques yet myself but they might be worth reading about!

[Dan_Abrams]

I’ve gotten some really great feedback on documentation and my explanation, but bumping to try to get some more feedback on the api design itself.

What do people think: decoding the state of the mediaElement object after a mediaEvent. Ok way of doing this? Should it be a subscription? Am I missing something?

Is the Media.State.State record too complex?

Is my Organization Into modules too much? Would we better served with a Media and a Media.events module only?

Using tasks to control playback? I know we don’t want to just put tasks around everything, but as I worked through, it really felt like they should be tasks, similar to the DOM library, as they can fail, and they create side effects…

With canPlayType…is there a better way to do this than a task? Can we query a browser on I itializiation to give us all the types it can play instead?

Anything else?

[Dan_Abrams]

I’ve done some work to add some missing features to my implementation here…Nothing new requiring native, all in pure elm:

1. Basic Support for Subtitles – A major accessibility win, and even better since Apple just added subtitle support into their NLE. Fairly confident in the cross-browser compatibility of WebVTT at this point.

2. Support for multiple audio and video tracks – Mostly a Safari/Edge feature, need to do more testing…

3. Added the work from my Media-Source-Fragments package into this one, and create the playableType function within it. This makes it easier to do something like this:

Task.attempt ResultHandler Media.canPlayType <| playableType [ mediaType OggOpus ]

I’ll have more work coming on better documentation and some tutorials in the next two weeks, and some improvements around this new stuff.

Still looking for feedback on API design, want to make this the best it can be.

[brian]

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.