Bytes, ports and Uint8Array

Learn
1

My goal is to get an image form user encrypt it in the browser and upload it to the server. All of these things by themselves work. The only problem is when I start to put all of this together. When I get the file it’s represented as File type. And I want to push it through port to JS where I encrypt it. I want to use Bytes for this becasue Base 64 wastes 30% of space and I don’t want to waste it just because of ports.

My initial idea was to Base 64 encode it decode it in JS to Uint8Array encrypt it and do the encoding all over again in reverse. But I don’t think that it’s a good idea to do it on files because they are a little bit bigger. The overhead might be acceptable for uploads but I’m not sure if it would be acceptable for processing multiple files when viewing more images on the same page. Is there a better way to make it work?

For encryption I use Tweet nacl-js.

2

It might be a little nicer (and easier) to decode the Bytes as an Array Int. Arrays can be sent through ports.

When you consider size waste also consider encoding/decoding performance. To and from base64 encoded string is probably much slower than using an Array Int (even if that array is slightly larger).

I think it would be nice to be able to transfer Bytes through a port (where it would be a javascript DataView), and have seen several people asking for it so far.

We then tie our Bytes type to the semantics of javascript DataView (possibly forever), so it is not an obviously good idea but would be very practical.

1 Like
3

This code does the trick.

import Bytes exposing (Bytes)
import Bytes.Decode as BDecode

decodeImage : Bytes -> List Int
decodeImage encodedImage =
    let
        listDecode =
            bytesListDecode BDecode.unsignedInt8 (Bytes.width encodedImage)
    in
    Maybe.withDefault [] (BDecode.decode listDecode encodedImage)


bytesListDecode : BDecode.Decoder a -> Int -> BDecode.Decoder (List a)
bytesListDecode decoder len =
    BDecode.loop ( len, [] ) (listStep decoder)


listStep : BDecode.Decoder a -> ( Int, List a ) -> BDecode.Decoder (BDecode.Step ( Int, List a ) (List a))
listStep decoder ( n, xs ) =
    if n <= 0 then
        BDecode.succeed (BDecode.Done xs)

    else
        BDecode.map (\x -> BDecode.Loop ( n - 1, x :: xs )) decoder

On the JS side I need to transform it into Uint8Array like this: Uint8Array.from(data); Unfortunately it is not as fast as I would want it even on my powerful desktop machine.

4

Did you run benchmarks? I actually think that would be really valuable to see how much of a degradation it is. Also for context: how big are the arrays you are sending?

RE your code: I think you’re missing a List.reverse somewhere, this decoder will reverse all the bytes. In general it might be better to use an Array on the elm side instead of List

5

So I gave up. Support for bytes is not as good as I would liked it would be. File.toUrl almost as fast as the code above. Even if that means decoding Base64 in JS again. But it is easier to read so I choose this path.

Much bigger problem is decoding of Base64 in Elm. For 2 MB file it takes ridiculous 10s to decode to bytes and send it to the server. I tried both Base64 libraries listed on Elm Finder that are able to decode to Bytes. But both are really slow. Until this is fixed I’m not able to do it any other way.

6

One way around it could be to make a custom element that gets the file, encrypts it as a new file object, then sends that into Elm through a custom event where you can treat it as a File on the Elm side, that way you don’t have to pay so much is serialization costs

7

Your advice is great I got it working all the way to the

I cannot get custom objects through Elm to HTTP file upload. I cannot get that value from JS to a elm/File. It would be perfect if I could. The best thing I was able to achieve is array of numbers which I serialize on the backend to a file.

8

Sorry for the delayed response, I created an Ellie (https://ellie-app.com/5v8nRD2Kvwwa1) of getting an encrypted File through a custom event, some of it is hacky for expedience, but the meat of it is

node "custom-file"
  [ on "file-selected"
      (Decode.field "detail"
          (Decode.map FileSelected File.decoder)
      )
  ]

And a custom element that handles the dirty work

class CustomFile extends HTMLElement {
  constructor() {
    super();
    this.processFile = this.processFile.bind(this);
  }
  
  processFile(evt) {
    const file = evt.target.files[0];
    const filename = file.name;
    const reader = new FileReader();

    reader.onload = (e) => {
      const data = e.target.result;
      const iv = crypto.getRandomValues(new Uint8Array(16));
  
      crypto.subtle.generateKey({ 'name': 'AES-CBC', 'length': 256 }, false, ['encrypt', 'decrypt'])
        .then(key => crypto.subtle.encrypt({ 'name': 'AES-CBC', iv }, key, data) )
        .then(encrypted => {
          const evt = new CustomEvent("file-selected", {
            detail: new File([encrypted.buffer], filename)
          });
          this.dispatchEvent(evt);
        })
        .catch(console.error);
    }
    
    reader.readAsArrayBuffer(file);   
  }
  
  connectedCallback() {
    const input = document.createElement('input');
    input.type = "file";
    input.addEventListener('change', this.processFile)
    input.innerHtml = "Open file";
    this.appendChild(input);
  }
}
customElements.define('custom-file', CustomFile);
2 Likes
9

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.