Major update to Elm-Canvas

[Chadtech]

Hello,

Could I get some feedback on a package I have been working on quite a while? Its called Elm-Canvas. Here is the link to the repo: https://github.com/Elm-Canvas/elm-canvas . At the bottom of this post I have some links of code examples and the docs.

Its primarily meant to expose the Html Canvas api into Elm. At this point, after a year of use, I am ready to iterate on the package and make a major update. I would appreciate any thoughts on how I and the other contributors to the project have gone about making this package.

Use Case
My primary use case was a drawing application. Ive been working on that drawing application for a year and a half now, and I have been using Elm-Canvas continuously since the beginning. Aside from myself, the next two biggest contributors (in terms of loc) to the project were also working on drawing applications.

So I suppose the narrow use case is drawing applications, but the better and more abstract use case is for low-level high-performance graphics that cant be summarized as shapes, image assets, or document elements. Another way of putting this is that virtual doms only work when the source data is much smaller than the rendered data (like how “hi” is smaller than <p class=\"greeting\">Hi</p>, or “./mega-man.png” is much smaller than the full pixel data of how mega man looks), but not when the source data is equally large to the rendered data (like when a list of pixel values in memory correspond exactly to whats getting rendered).

Our approach
We have just exposed the Canvas api in the most direct possible way, much like how Elm-Css exposes css directly. We have a Canvas type, and a toHtml function, and along the way you can modify Canvas using another type called Ctx, which represents all the methods and properties in the Canvas api. Originally Elm-Canvas was fairly indirect and abstract, but it wasnt very performant and it couldnt accomidate use cases that didnt want pixel perfect drawing, so thats when we changed to this direct approach.

Future
I just hope this will be a better project that others will find useful. I would like to make something good enough to one day be a part of the core programming language, or could greatly inform how Canvas will eventually be incorporated into Elm (Im not that optimistic about this however).

Feedback
Ill keep most of my thoughts to myself so as not to contaminate whatever feedback you all might offer, but I do have one question. Currently there are some possible run time errors, like if you initialize a canvas with a size of { width = 0, height = 0 }. What should be done about that? I suppose thats no worse than the existing run time errors in core; but still.

Links
Here are the docs:
http://elm-canvas-docs.surge.sh/

Here are two examples:
Drawing images - http://elm-canvas-docs.surge.sh/draw-image
Animation frame - http://elm-canvas-docs.surge.sh/animation-frame

Here is the code to the examples:

Elm-Canvas/elm-canvas

[timbrenner]

I wrote this some time ago - https://gitlab.com/timbrenner/elm-viewport/

Would I be able to use Elm-Canvas for this type of game?

[Chadtech]

You could, but I dont think it would be a good idea. I notice you say in your repo that you cant use Elm-Graphics for this. That would have been my recommendation. Why not Elm Graphics? Ive made a side scrolling video game using Elm Graphics.

[ashishsc]

Elm-Canvas/elm-canvas

I think the author states that there are some specific performance considerations involved.

[kevinyank]

At a conference workshop today I experimented with implementing some generative art using Elm-Canvas. I’ve pushed these to a branch, and am happy to contribute them to the project if that would help.

Although these specific examples would have been straightforward to render with SVG or some other existing graphics library, some of the other examples that were presented in the workshop involved large numbers of translucent render passes for which Canvas is ideally suited.

As far as my feedback, I found the library felt about the same to work with as the JavaScript Canvas API, which is to say not very nice but it gets the job done. Since the mission of Elm seems to be to encapsulate this kind of ugly API with something much nicer, I wonder if this might be useful as an implementation layer for experiments with nicer API designs (a final iteration of which might then be implemented natively in Core). In the meantime, this makes the Canvas API available to experimental projects, and so, I personally would love to see this published as-is under elm-explorations (which will allow native modules in Elm 0.19).

I didn’t encounter any runtime errors while working with the library, but for the specific case of a 0×0 canvas, since this is a valid HTML element, I would expect that this special case be handled gracefully by the library, just as JavaScript would. In a more error-proof API, though, I could imagine something like Canvas.initialize returning a Maybe Canvas or a Result Canvas.Error Canvas to allow for invalid dimensions to be rejected.

I didn’t get around to implementing any animation, but I wondered if this library already handles the statefulness of canvas elements efficiently. For example, if my view has already rendered a bunch of drawing instructions to the DOM, if my program later updates my canvas state, are only the new changes rendered to the existing canvas (I’m hoping for this!), or does the entire drawing history replay?

[Chadtech]

@kevinyank Thanks so much for trying Elm-Canvas out! Just seeing your code and how you chose to use Elm-Canvas is really helpful. Thank you.

In regards to your idea about using Elm-Canvas as it exists today as an implementation layer, I think thats a great idea. I hadnt thought of that. I agree that the native canvas api is ugly. Im just not sure how to abstract it in a way that simplifies the best use-cases of Elm-Canvas. But a module for experimental abstracts could be a start. There are only so many ways to draw a line, or edit a single pixel, and this abstraction module could contain those functions.

Regarding your question about canvas state, Im not sure if I understand the question, but I believe the answer is “the entire drawing history is replayed”. If you invoke MoveTo and BeginPath in your Elm code, then they are invoked in the canvas api too. If you invoke them in your view function they are invoked every time your page re-renders.

[Dan_Abrams]

I’m taking a look and will try to provide what feedback I can. I really am a big believer in playing with the API and trying it out helps me understand what’s awesome and what needs improvement.

But a question that immediately pops out at me, why is DrawOp a huge ADT, instead of using functions, many other packages, such as Html.Attribute & SVG do?

Also, for someone who’s not as familiar with Canvas, some of the arguments for the DrawOp type seem like they might be more readable as records. For instance:

Transform Float Float Float Float Float Float

seems like it might be easier to use as

Transform {hScale: Float, hSkew: Float, vScale: Float, vSkew: Float, hMoving: Float, vMoving: Float }

This seems easier, especially for those of us with little to no experience with canvas. This is assuming I’ve figured out what those Floats represent.

I do like the feel of initializing the canvas and then feeding through toHtml, and at first I didn’t think I would like that.

I’ll play with it some over the next week or two and give you any extra feedback I have.

[Dan_Abrams]

Taking a cue from Jimmy Kimmel, I made a branch with a new example, unnecessary-censorship.elm.

Overall, I would say my initial impressions remained true. In particular, using named records would have been a lot easier for someone like me without a lot of canvas experience. For example:

Arc (Point 0 0) {radius: Float, start: Float, end: Float}

or something similar.

Also, I was looking for imageSmoothing Enabled/Disabled. Making this pixelated would have been awesome.

Passing the Canvas types around took a little getting used to, but I did get it eventually, and it did work.

One nagging thought, given that it’s a functional language and that we may not have an object to pass to these functions, is there a way to grab other media to put into canvas. I’m working with the Media API, so how would one pass a video into canvas, for instance?

I’m glad to see this progress made and I think this is going to be a great API.

[kevinyank]

Regarding your question about canvas state, Im not sure if I understand the question, but I believe the answer is “the entire drawing history is replayed”.

I’ll try to clarify, but maybe I just need to have more of a play with it.

My question boils down to this: the whole point of canvas is to have a stateful set of pixels that you can perform persistent drawing operations, mutating the pixel values. When your browser needs to repaint the canvas, it doesn’t have to replay all the drawing operations that led to the current pixels; it just paints the pixels stored in the current state of the canvas.

For canvas to be useful in Elm (it seems to me), we need to preserve this ability to use the canvas as a mutable set of pixels. For example, if my Elm app paints to the canvas on each animation frame to create an evolving image, on each of those animation frames Elm shouldn’t need to reset the canvas to its initial blank state and replay every frame of the animation to this point with one extra frame added on the end; it should be able to paint just the change required for this frame of the animation.

So my question was, does this library achieve this, and if so how? If it does, then I’m guessing that I’d need my program’s init function to paint the initial frame, and then each update function call for an animation frame would need to paint the changes for that frame. But those paints that I add to my canvas in update are only applied to the canvas in the DOM once, on the next view call, right?

A related question: if the Elm Virtual DOM replaces a previously-rendered Canvas, is its accumulated pixel state lost?

Sorry if I’m misunderstanding and the questions above don’t make sense. You have a “bouncing ball” animation in the examples that I suspect will answer all my questions if I pull it apart, so I’m happy to wait until I’ve had the time to do that and then come back to you with any lingering questions I have.

[Chadtech]

Hey @Dan_Abrams thanks for having a look, and for your kind words!

Functions instead of union type constructors
That sounds like a good idea. Sorry, I probably didnt make this clear enough in my OP, but there latest version of Elm-Canvas (0.4.0) hasnt been merged into master yet, and it uses functions instead of union type constructors.

Records instead of many params
Hey cool, I like that idea, but I would think these records need to be aliased. What do you think? And if you agree do you think TransformParams be a good alias for that record?

ImageSmoothing
That shouldnt be too hard to implement. That one must have slipped by us.

[Chadtech]

Ah okay, I think I understand what you are asking now. Let me describe how we solved this from our perspective as we first encountered these mutability problems in implementation.

The Canvas data type in Elm just wraps an actual <canvas/> html element. That element obviously contains pixel state and that pixel state gets dragged around with it wherever the canvas goes in the Elm program. The first problem we faced was that if you had one canvas, and you rendered it in two places using toHtml, you would break everything. This was because one html element cant be in two places in the DOM, and toHtml was simply taking the canvas element and appending it to the DOM. The way we resolved this was that toHtml actually makes a perfect copy of the canvas and renders the copy. This means that the canvas that floats around in your program is not the canvas that shows up in your html; it just serves as a source of state for what does show up in the html.

We ran into the same problem when performing draw operations. Per canvas, there is only one mutable pixel state, even if the reference to it might be floating around in multiple places in your code. The problem we had was that if you drew onto the canvas at any point then all canvases were effected. For example…

    [ toHtml [] (draw letterA canvas)
    , toHtml [] canvas
    ]

… would lead to both canvases having the letter ‘A’ drawn onto them, even tho in the code we clearly only intended that the first one have an ‘A’. We solved this problem the same way as the html problem: when you do any drawing, a copy of the canvas is first made, and then the copy is mutated and returned.

So the short answer is that we actually treat the canvas as a mutable stateful thing, but we narrowed down all the contexts in which this violates the promise of immutability, and in those cases we duplicate the entire state and mutate that instead. We can get away with this because the process of duplicating a canvas is surprisingly efficient.