Potential Scenario for packed array of integers (Capnproto)

Hey everyone,
I’ve been pretty interested in Capnproto (capnp) recently and have wanted to build a web app with it for fun. Capnp is a data interchange solution very much akin to ProtoBufs. It uses a schema in a custom language which is used to compile a library in a target language that handles the data. I was thinking of implementing the capnp-elm compiler plugin to allow the capnp compiler to target Elm.

Capnp was designed to be incredibly fast by having an in memory representation that matches the over the wire one. Additionally, the spec requires the use of arena allocation where we manually allocate a large chunk of memory in which we produce and maintain our protos. I believe this arena allocation would only be possible in pure Elm if we had access to a presized byte array that we could allocate. When reading through elm/bytes, I was directed to this forum.

Let me know if this scenario accurately fits the concerns listed in elm bytes. I’m also somewhat curious if any are interested in the use of Capnp in Elm, but that’s a bit secondary to this question. Thanks in advance for any feedback you may provide.


What elm-bytes is good at is decoding binary serialized data. So it seems possible to (automatically) write capnproto decoders using elm-bytes.

But because elm is a more high-level language, the performance characteristics will be different. In elm, you cannot put some bytes in a piece of memory and say “you are of type X now” like you could in C, so the in-memory representation will be different and there is a cost to going back and forth between bytes and actual elm data structures.

But I think this would be an interesting project. This kind of tooling really helps when you have a lot of decoders to write, and it can do optimizations that would just be too tedious if done by hand. The performance will be reasonable (though that does depend on the kind of data you send).

So I’d say see if you can make a prototype that feels nice to work with, and then it can be optimized.

So I understand the point of elm-bytes and I understand that elm being a higher level language does have to suffer a performance penalty for the runtime and certain abstractions.

However, it does still need to worry about performance. Java’s implementation of capnp for example uses a java.nio.ByteBuffer to back the data for the proto. Accessing the data is done through methods. The equivalent in Javascript would be to use the uint8array from my understanding.

The reason this byte buffer backing is so important - as opposed to decoding it into some abstract data type defined in elm - is that the entire point of capnproto/flatbuffers is to enable this kind of interaction with the data. If you are going to go through a decode step, then you’d rather just use protobuf.

I think prototyping this based on an Elm Array Int is probably a good idea. Thanks for the advice. I think I can get it to be somewhat intuitive to use - though it will never be as nice as accessing a record field. That said, I think it would be best to have a way to reserve a chunk of memory in Elm.

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.