# Two use cases of the new elm/bytes

**URL:** <https://discourse.elm-lang.org/t/two-use-cases-of-the-new-elm-bytes/2518>\
**Category:** Request Feedback\
**Created:** [November 15, 2018, 11:40am UTC](https://discourse.elm-lang.org/t/two-use-cases-of-the-new-elm-bytes/2518 "2018-11-15T11:40:03Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![norpan](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/norpan/32/548_2.png) [@norpan](https://discourse.elm-lang.org/u/norpan)\
**Post date:** [November 15, 2018, 11:40am UTC](https://discourse.elm-lang.org/t/two-use-cases-of-the-new-elm-bytes/2518/1 "2018-11-15T11:40:03Z")

</div>

First of all, it’s great to finally have files and bytes in Elm!

- [https://package.elm-lang.org/packages/elm/file/latest/](https://package.elm-lang.org/packages/elm/file/latest/)
- [https://package.elm-lang.org/packages/elm/bytes/latest/](https://package.elm-lang.org/packages/elm/bytes/latest/)

We have two use cases for files and bytes (that are somewhat connected) that I want to share.

1. We want to calculate a hash for a large file without having to load the whole file in memory at once. We calculate the hash so that we don’t have to upload a file that is already uploaded.
2. We want to upload a file in chunks, to have better control over the upload process. See rationale at [https://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html](https://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html).

Both these cases could be solved in Elm with the new `bytes`, I think, if there was a way to get a slice of a file as bytes.

---

<div class="post-metadata">

**Author:** ![Hector](https://avatars.discourse-cdn.com/v4/letter/h/43a26b/32.png) [@Hector](https://discourse.elm-lang.org/u/Hector)\
**Post date:** [November 15, 2018, 1:54pm UTC](https://discourse.elm-lang.org/t/two-use-cases-of-the-new-elm-bytes/2518/2 "2018-11-15T13:54:55Z")

</div>

The way this is done in the javascript libraries that provide resumable uploads is with the [slice function](https://developer.mozilla.org/en-US/docs/Web/API/Blob/slice)

AFAICT, in this first version of the pacakge, toBytes is the only function provided and it loads the entire file blob into memory.

---

<div class="post-metadata">

**Author:** ![norpan](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/norpan/32/548_2.png) [@norpan](https://discourse.elm-lang.org/u/norpan)\
**Post date:** [November 15, 2018, 4:33pm UTC](https://discourse.elm-lang.org/t/two-use-cases-of-the-new-elm-bytes/2518/3 "2018-11-15T16:33:36Z")

</div>

Yes, we use the `Blob.slice()` currently in our javascript implementation, but I’m describing a suggested improvement api that could make a move to Elm possible.

---

<div class="post-metadata">

**Author:** ![evancz](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/evancz/32/16_2.png) [@evancz](https://discourse.elm-lang.org/u/evancz)\
**Post date:** [November 15, 2018, 10:07pm UTC](https://discourse.elm-lang.org/t/two-use-cases-of-the-new-elm-bytes/2518/4 "2018-11-15T22:07:27Z")

</div>

Can you share the JS code you are using to process files in chunks?

And any links you have on this approach that help me understand what is going on behind the scenes? (If there is anything tricky!)

---

<div class="post-metadata">

**Author:** ![Warry](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/warry/32/3998_2.png) [@Warry](https://discourse.elm-lang.org/u/Warry)\
**Post date:** [November 16, 2018, 8:34am UTC](https://discourse.elm-lang.org/t/two-use-cases-of-the-new-elm-bytes/2518/5 "2018-11-16T08:34:11Z")

</div>

I too am tempted to migrate my upload code to Elm to upload 100’s GB files. I personnaly use Evaporate.js. AWS S3 upload in chunks is fairly complex, and Evaporate contains a lot of code for backward compatibility (browser, and s3 itself) and makes extra requests, so I wouldn’t recommand as an inspiration. The process is described here in the docs: [https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html)

roughly, here are the main informations:

- Init the upload with AWS cryptographic methods
- Calculate each chunk signature (using previous chunk’s signature)
- Crypto method uses MD5 and HMAC-SHA256, which is my show stopper for now.

> **[TTLabs/EvaporateJS](https://github.com/TTLabs/EvaporateJS)**
>
> Javascript library for browser to S3 multipart resumable uploads - TTLabs/EvaporateJS

  

> <https://github.com/aws/aws-sdk-js/blob/master/lib/s3/managed_upload.js>

---

<div class="post-metadata">

**Author:** ![norpan](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/norpan/32/548_2.png) [@norpan](https://discourse.elm-lang.org/u/norpan)\
**Post date:** [November 16, 2018, 8:50am UTC](https://discourse.elm-lang.org/t/two-use-cases-of-the-new-elm-bytes/2518/6 "2018-11-16T08:50:46Z")

</div>

Certainly!

If you want me to help making a pull request for the things I suggest below, do let me know.

Reading a file in chunks in javascript is done by first doing `slice()` on the `File`, producing a `Blob` (files are also `Blob`s, by the way). The you can just use [`FileReader`](https://developer.mozilla.org/en-US/docs/Web/API/FileReader) as usual, but on the `Blob` instead, and you get the slice you wanted.

So here is our code that does the chunking:

```javascript
function readChunk(file, start, end, callback) {
  var reader = new FileReader();
  var blob = file.slice(start, end);
  reader.onloadend = function() {
    callback(reader.error, reader.result);
  }
  reader.readAsArrayBuffer(blob);
}

```

The calling code does `readChunk` with a [_error-first_ callback](https://nodejs.org/api/errors.html#errors_error_first_callbacks) that gets the array buffer, processes it, and then if there is still data to read (by looking at the file length) it calls `readChunk` again.

Looking at the `elm/file` implementation of [`File.toBytes`](https://github.com/elm/file/blob/bbd2ebdc93b3e6d3fd3226bf7f3bd612a1fb7c60/src/Elm/Kernel/File.js#L130-L141) it looks like it works the same way (but you’d probably want to catch errors too and have a `Task` that can return errors. Common errors are read permissions or that the file has been deleted. The error is an [DOMError/DOMException](https://developer.mozilla.org/en-US/docs/Web/API/DOMException) with `name` and `message` fields.

If this is functionality that should be included, there are two main options:

1. Add a `File.sliceToBytes : { from : Int, to : Int, file : File } -> Bytes`
2. Add a `File.slice : { from : Int, to : Int, file : File } -> File` (just returning `File.slice(...)`

The second one is more general (you can slice and then call `File.toBytes`, `File.toString`, `Http.fileBody` etc.) and in the Http case also more efficient, you don’t have to copy the bytes before sending them.

The drawback with the second solution is that some functions on the resulting `File` will be undefined: `name`, `mime`, `lastModified`, so that has to be taken into account (by returning empty values or `Maybe a`) or by copying these values from the original `File` object when creating the slice.

---

<div class="post-metadata">

**Author:** ![jxxcarlson](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/jxxcarlson/32/866_2.png) [@jxxcarlson](https://discourse.elm-lang.org/u/jxxcarlson)\
**Post date:** [November 16, 2018, 11:48am UTC](https://discourse.elm-lang.org/t/two-use-cases-of-the-new-elm-bytes/2518/7 "2018-11-16T11:48:45Z")

</div>

Is it feasible to implement unencrypted uncompressed zip using elm bytes? There is a description of the file format on wikipedia.

---

<div class="post-metadata">

**Author:** ![system](https://canada1.discourse-cdn.com/flex035/uploads/elm_lang/original/1X/50a05e53677a2c3b47776d7abd0f113eb50193a1.png) [@system](https://discourse.elm-lang.org/u/system)\
**Post date:** [November 26, 2018, 11:48am UTC](https://discourse.elm-lang.org/t/two-use-cases-of-the-new-elm-bytes/2518/8 "2018-11-26T11:48:51Z")

</div>

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.