How can I send large arrays through ports?

Learn
#1

Hi folks!

New Elm user here. I’m a very experienced front-end engineer with a preference for functional languages. (I’ve studied Haskell and Clojure in my free time for the fun of it.) I also do generative art, normally in JS. I was excited to try out Elm as an alternative for my art projects, but ran into a problem sending large arrays through ports.

Here’s the background:

I’ve got a library that loads images into arrays of integers in JS. In this particular case, I was trying to load an image of a 500x500 heightmap. The image contains 500 x 500 = 250,000 pixels, and each pixel has four channels (red, green, blue, and alpha). So the whole array contained a million elements.

I’ve worked with arrays this size in JS with no problem, but when I tried to pass it through a port to my Elm code, I got a “maximum call stack size exceeded” error. When I tried passing a shorter array through, it worked fine.

Interacting with the image data is a key part of the code for this art project, so I don’t want to just handle that part of the app in JS. There’d be too much back and forth through ports.

Do you all have any thoughts or ideas for getting around this? Maybe I could send smaller pieces of the array through one at a time, and gradually concatenate the full array on the Elm side? I also tried passing the data through as a flag, and ran into the same problem.

1 Like
#2

Hello, welcome to Elm.

Can you show your code? I’ve just sent through an array of 50 million integers as a flag, and traced out List.length on the incoming array, so it doesn’t look like it’s the size of the array that’s the problem.

Might be how you’re manipulating the data when it’s received? Code examples would help others help you :slight_smile:

Edit: I’ve also just sent through a List (List Int) of length 250,000, where each of the inner Lists contain 100 Ints, so a total of 25,000,000 elements.

When received by Elm I ran the following without a problem, albeit it a took a few seconds:

Debug.log "a" (List.concat flags.a |> List.map String.fromInt |> List.length)

This logged a: 25000000 to the console.

#3

Before going down the route described below, I would check that the “maximum call stack size exceeded” error is not coming from your own Elm code. What are you doing with the 250K elements once you receive them through the port?

The trick that may help is - I believe it is possible to pass a File through a port:

I have never done this myself so cannot confirm it works. Perhaps someone else can verify?

#4

Hello,

Elm ports will check the type of every item in your list if you specify the type, and convert js arrays to linked list if you declare a List (whereas Elm Array are plain js arrays). The trick might be to declare your data as Json Value, an Array of Json Values or even using Bytes, then you can arrange the parsing so that it’s suiting your needs. Not an easy task, but that’s what I ended up doing that one time I had to face the same problem.

#5

Worth noting that you cannot pass Bytes through a port.

#6

Hi everyone! Thanks for the suggestions. Sounds like there should be a way to do this. I’ll try out some of your suggestions and report back with a small, self-contained example if I still can’t get it to work. Thanks for the help!

#7

Hi everyone!

Thanks for your help – I’ve got it working now.

I had a Debug.log first thing in my main function that wasn’t getting printed before the error happened, so I assumed the problem was as the array crossed the border into Elm. But after reducing my Elm code down, it looks like the actual issue was somewhere else.

Also, good to know that there’s a difference between Arrays and Lists! I think Arrays are better for my use case since I need random access.

Thanks again!

#8

It may also be worth pointing out that an Elm Array is not implemented as a plain JS Array.

An Elm Array is a functional data structure which is immutable. To avoid copying the entire array on every Array.set, it is actually structured as a tree (with a high branching factor). This means that updates to the array can alter just one branch of the tree, but much of the new version of the tree can share the same memory as the original.

So when you pass a JS array into a port and decode it as an Elm array, it won’t simply pass through unchanged. The JS array will be restructured into an Elm array.

Just thought you might like to know in case you were thinking that using an array would carry zero performance cost.

2 Likes
#9

right, I forgot that this changed. my use case dated from the 0.17 era o som’thin’. my bad !

#10

Good to know! Thanks for the heads up!

#11

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.