# Working around "not enough bytes" problems?

**URL:** <https://discourse.elm-lang.org/t/working-around-not-enough-bytes-problems/3359>\
**Category:** Learn\
**Created:** [March 20, 2019, 11:53am UTC](https://discourse.elm-lang.org/t/working-around-not-enough-bytes-problems/3359 "2019-03-20T11:53:10Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![michaeljones](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/michaeljones/32/152_2.png) [@michaeljones](https://discourse.elm-lang.org/u/michaeljones)\
**Post date:** [March 20, 2019, 11:53am UTC](https://discourse.elm-lang.org/t/working-around-not-enough-bytes-problems/3359/1 "2019-03-20T11:53:10Z")

</div>

Has anyone come up with a strategy for managing the compiler crashes? In particular the `elm: not enough bytes.` one?

I’ve adapted my elm-vim plugin to look for that in the output and touch all the sources files before re-compiling which makes it less of a pain to deal with. However my colleagues uses VSCode so I can’t share the same approach easily. Are there any other strategies that?

Reading [this github ticket](https://github.com/elm/compiler/issues/1837) it seems there are suggestions around not running the compiler twice at the same time and making sure that elmi and elmo files are in sync. Interestingly I had a script to run a [make server](https://github.com/michaeljones/elm-make-server) process that make help make sure that only one compile was run at a time (because the 0.18 compiler could go quite slowly when there were multiple instances running) but I got rid of it for Elm 0.19 and hadn’t considered re-instating it.

I was hoping that people might share strategies that they’ve been using to deal with this?

As some additional information, I tend to have webpack running that runs compiler. I do compiles in my editor and on the command line at various times. I hit this error 10s of times a day.

---

<div class="post-metadata">

**Author:** ![Sidney](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/sidney/32/3290_2.png) [@Sidney](https://discourse.elm-lang.org/u/Sidney)\
**Post date:** [March 20, 2019, 7:09pm UTC](https://discourse.elm-lang.org/t/working-around-not-enough-bytes-problems/3359/2 "2019-03-20T19:09:22Z")

</div>

> [@michaeljones](#):
>
> I tend to have webpack running that runs compiler. I do compiles in my editor and on the command line at various times. I hit this error 10s of times a day

Similar experience here. I use Atom + Elmjutsu.

> [@michaeljones](#):
>
> I’ve adapted my elm-vim plugin to look for that in the output and touch all the sources files before re-compiling which makes it less of a pain to deal with.

Interesting, could you elaborate on how that fixes the `not enough bytes` error? I tend to just delete `elm-stuff` and recompile. I’m under the impression that `not enough bytes` is caused by corrupt elmi/elmo files, but for some reason the `data files are corrupt` message is not triggered.

---

<div class="post-metadata">

**Author:** ![evancz](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/evancz/32/16_2.png) [@evancz](https://discourse.elm-lang.org/u/evancz)\
**Post date:** [March 20, 2019, 7:32pm UTC](https://discourse.elm-lang.org/t/working-around-not-enough-bytes-problems/3359/3 "2019-03-20T19:32:20Z")

</div>

My understanding is that the root cause is when `elm` is run alongside other `elm` instances. This means the first instance may be writing a `.elmi` file while the second instance is reading it. So the second instance is like “hey, there are not enough bytes!”

People who are experiencing this problem more often _should_ be people who have particularly inefficient combinations of secondary tools, that are all calling `elm make` multiple times at once on the same project.

In an ideal world, tooling would be written more efficiently, but it seems that people producing binaries can use something like [`flock`](https://linux.die.net/man/1/flock) so that inefficient usage does not cause data races like this.

So I am looking into [filelock](https://hackage.haskell.org/package/filelock) to do something in the compiler itself, but in the meantime, possible workarounds include:

1. Have plugins do things more efficiently.
2. According to the internet [here](https://stackoverflow.com/questions/13551840/bash-flock-why-200), it may be possible to have a wrapper around the normal `elm` binary like this:

```auto
( 
  flock -e 200
  elm $@
) 200>/tmp/elm.lockfile 

```

I have not used `flock` in bash scripts personally, but something like this may work! It will also mean that when you try to run `elm make` multiple times at once, the later calls will block and skip any work that was done by the first one. (You’ll still be reading files and stuff though, so it’s still worse than just having the plugins avoid multiple simultaneous calls!)

---

<div class="post-metadata">

**Author:** ![michaeljones](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/michaeljones/32/152_2.png) [@michaeljones](https://discourse.elm-lang.org/u/michaeljones)\
**Post date:** [March 21, 2019, 12:34pm UTC](https://discourse.elm-lang.org/t/working-around-not-enough-bytes-problems/3359/4 "2019-03-21T12:34:25Z")

</div>

Thank you for the reply. I appreciate the information. I’ve run some tests with an error I have at the moment that seems to trigger the `elm: not enough bytes` issue. My steps are (having killed my webpack process):

### Not enough bytes error

```auto
$ rm -fr node_modules
$ yarn
...
$ which elm
/home/michael/root/projects/hive/server/portal/node_modules/.bin//elm
$ file /home/michael/root/projects/hive/server/portal/node_modules/.bin//elm
/home/michael/root/projects/hive/server/portal/node_modules/.bin//elm: symbolic link to ../elm/bin/elm
$ cd server/apps/business
$ rm -fr elm-stuff
$ elm make src/Business.elm
Dependencies loaded from local cache.
Dependencies ready!                
Detected errors in 1 module.                                         
-- TYPE MISMATCH ------------------------------- ./src/Business/Report/Tasks.elm

This function cannot handle the argument sent through the (|>) pipe:

 91| getPostIdFromReportId session reportId
 92|> |> Task.andThen
 93|> (\postId ->
 94|> Api.readTask
 95|> { endpoint = Api.endpoint ["api", "posts", postId, "comments"] []
 96|> , body = Http.emptyBody
 97|> , success = Api.JsonBody <| Decode.field "comments" (Decode.list commentDecoder)
 98|> , failure = Zap.Http.noExpectedErrorsHandler
 99|> , session = session
100|> }
101| )

The argument is:

    Api.Task () Post.Id

But (|>) is piping it a function that expects:

    Task.Task (Zap.Http.Error ()) String

$ elm make src/Business.elm
elm: not enough bytes
CallStack (from HasCallStack):
  error, called at src/Data/Binary.hs:212:21 in binary-0.8.6.0-1SWOkUSZj1B1k0FQ5fyFqI:Data.Binary

```

This error seems quite reproducible as I experienced it, fixed it, worked on another branch, returned to this, removed the fix and I get it again.

### The State of Elmi & Elmo

If I run a script to detect missing elmi or elmo files:

```auto
#!/bin/bash -e

for elmo in $(ls elm-stuff/0.19.0/*.elmo); do
	# echo $elmo
	elmi=$(echo $elmo | sed 's/elmo$/elmi/')
	if [! -e "$elmi"]; then
		echo "$elmi doesn't exist"
	fi
done

for elmi in $(ls elm-stuff/0.19.0/*.elmi); do
	# echo $elmo
	elmo=$(echo $elmi | sed 's/elmi$/elmo/')
	if [! -e "$elmo"]; then
		echo "$elmo doesn't exist"
	fi
done

```

Then I get:

```auto
$ bash ./check-stuff
elm-stuff/0.19.0/Zap-View-Action.elmo doesn't exist

```

### Deleting unmatched files

This takes a bit of time so I’ve attempted to improve it to something that runs quicker and cleans up the problematic files. I’m no expert though so this could have issues and I’m sure there is room for improvement:

```auto
elmoFiles=$(mktemp /tmp/elmo.XXXX)
elmiFiles=$(mktemp /tmp/elmo.XXXX)

ls -1 elm-stuff/0.19.0/*.elmo > $elmoFiles
ls -1 elm-stuff/0.19.0/*.elmi | sed 's/elmi/elmo/' > $elmiFiles

# Include tmp files so that we always have something and we clean up after ourselves
diff $elmoFiles $elmiFiles | tail -n +2 | sed 's/> //' | sed 's/elmo/elmi/' | xargs rm -f $elmoFiles $elmiFiles

```

### Wrapping the compiler with flock

I’ve also created script to replace the `node_modules/.bin/elm` symbolic link with a script that uses flock as you’ve suggested. The final result incorporate the cleanup above and looks like this:

```auto
#!/bin/bash -e

nodeModulesFolder=$(dirname $(dirname $0))
(
flock -e 200

# Delete all elmi files that don't have matching elmo file
elmoFiles=$(mktemp /tmp/elmo.XXXX)
elmiFiles=$(mktemp /tmp/elmo.XXXX)

ls -1 elm-stuff/0.19.0/*.elmo > $elmoFiles
ls -1 elm-stuff/0.19.0/*.elmi | sed 's/elmi/elmo/' > $elmiFiles

# Include tmp files so that we always have something and we clean up after ourselves
diff $elmoFiles $elmiFiles | tail -1 | sed 's/> //' | sed 's/elmo/elmi/' | xargs rm -f

$nodeModulesFolder/elm/bin/elm $@
) 200>/tmp/elm.lockfile

```

I don’t know how flock is meant to work so I’m just guessing from your example but it seems promising (note the ampersand after the first command):

```auto
$ elm make src/Business.elm &; elm make src/Business.elm
[1] 10755
Detected errors in 1 module.
-- TYPE MISMATCH ------------------------------- ./src/Business/Report/Tasks.elm

This function cannot handle the argument sent through the (|>) pipe:

 91| getPostIdFromReportId session reportId
 92|> |> Task.andThen
 93|> (\postId ->
 94|> Api.readTask
 95|> { endpoint = Api.endpoint ["api", "posts", postId, "comments"] []
 96|> , body = Http.emptyBody
 97|> , success = Api.JsonBody <| Decode.field "comments" (Decode.list commentDecoder)
 98|> , failure = Zap.Http.noExpectedErrorsHandler
 99|> , session = session
100|> }
101| )

The argument is:

    Api.Task () Post.Id

But (|>) is piping it a function that expects:

    Task.Task (Zap.Http.Error ()) String
Detected errors in 1 module.
-- TYPE MISMATCH ------------------------------- ./src/Business/Report/Tasks.elm

This function cannot handle the argument sent through the (|>) pipe:

 91| getPostIdFromReportId session reportId
 92|> |> Task.andThen
 93|> (\postId ->
 94|> Api.readTask
 95|> { endpoint = Api.endpoint ["api", "posts", postId, "comments"] []
 96|> , body = Http.emptyBody
 97|> , success = Api.JsonBody <| Decode.field "comments" (Decode.list commentDecoder)
 98|> , failure = Zap.Http.noExpectedErrorsHandler
 99|> , session = session
100|> }
101| )

The argument is:

    Api.Task () Post.Id

But (|>) is piping it a function that expects:

    Task.Task (Zap.Http.Error ()) String
[1] + exit 1 elm make src/Business.elm

```

Without the wrapper this looks much less happy.

### Helper script to wrap compiler

Here is a script that I run to wrap the compiler. Useful as a yarn re-install will lose it.

```auto
#!/bin/bash -e

rm -f node_modules/.bin/elm

cat << ENDCAT > node_modules/.bin/elm
#!/bin/bash -e

nodeModulesFolder=\$(dirname \$(dirname \$0))
(
flock -e 200

# Delete all elmi files that don't have matching elmo file
elmoFiles=\$(mktemp /tmp/elmo.XXXX)
elmiFiles=\$(mktemp /tmp/elmo.XXXX)

ls -1 elm-stuff/0.19.0/*.elmo > \$elmoFiles
ls -1 elm-stuff/0.19.0/*.elmi | sed 's/elmi/elmo/' > \$elmiFiles

# Include tmp files so that we always have something and we clean up after ourselves
diff \$elmoFiles \$elmiFiles | tail -n +2 | sed 's/> //' | sed 's/elmo/elmi/' | xargs rm -f \$elmoFiles \$elmiFiles

\$nodeModulesFolder/elm/bin/elm \$@
) 200>/tmp/elm.lockfile
ENDCAT

chmod +x node_modules/.bin/elm

```

I’ve only just been playing around with them and maybe there are other kinds of this error that this doesn’t handle but it seems promising and perhaps this will be useful to someone. I’ll update I experience issues. I fear that my excitement at messing around with this stuff as won over my better judgement that I should only post about this when it has been tested a bit more.

---

<div class="post-metadata">

**Author:** ![michaeljones](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/michaeljones/32/152_2.png) [@michaeljones](https://discourse.elm-lang.org/u/michaeljones)\
**Post date:** [March 21, 2019, 2:46pm UTC](https://discourse.elm-lang.org/t/working-around-not-enough-bytes-problems/3359/5 "2019-03-21T14:46:21Z")

</div>

> [@Sidney](#):
>
> Interesting, could you elaborate on how that fixes the `not enough bytes` error? I tend to just delete `elm-stuff` and recompile.

My guess is that if you `touch` all the files than the compiler is going to try to recompile all of them so it doesn’t matter what the previous state of the `elmo` and `elmi` files are as they are going to be replaced. I felt that this approach might be quicker than removing `elm-stuff` as it saves elm from copying over the libraries & dependencies again. Though I imagine that is pretty quick.

---

<div class="post-metadata">

**Author:** ![michaeljones](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/michaeljones/32/152_2.png) [@michaeljones](https://discourse.elm-lang.org/u/michaeljones)\
**Post date:** [March 21, 2019, 6:18pm UTC](https://discourse.elm-lang.org/t/working-around-not-enough-bytes-problems/3359/6 "2019-03-21T18:18:03Z")

</div>

I’ve made some further changes after encountering some issues. I’m not sure yet how useful it is. I feel like I’m still seeing the error with it sometimes.

The latest version of the replacement script I have is:

```auto
#!/bin/bash -e

nodeModulesFolder=$(dirname $(dirname $0))

(
    flock -e 200

    # Open sub-shell to contain directory change
    (
        # Search upwards for the elm.json folder - assuming elm-stuff will be next to it but
        # might not exist yet - so that this will work when run by elm-test
        path=$(pwd)
        while [["$path" != "" && ! -e "$path/elm.json"]]; do
          path=${path%/*}
        done

        # Change to parent of elm-stuff directory
        cd $path

        # Only try to clean up elm-stuff if it exists
        if [-e "$path/elm-stuff"]; then
            # Delete all elmi & elmo files that don't have a matching pair
            elmoFiles=$(mktemp /tmp/elmo.XXXX)
            elmiFiles=$(mktemp /tmp/elmo.XXXX)

            ls -1 elm-stuff/0.19.0/*.elmo > $elmoFiles
            ls -1 elm-stuff/0.19.0/*.elmi | sed 's/elmi/elmo/' > $elmiFiles

            # Include tmp files so that we always have something and we clean up after ourselves
            diff $elmoFiles $elmiFiles | grep elm-stuff | sed 's/\(>\|<\) //' | sed 's/\(.*\).elm./\1.elmi \1.elmo/g' | xargs rm -f $elmoFiles $elmiFiles
        fi
    )

    $nodeModulesFolder/elm/bin/elm $@

) 200>/tmp/elm.lockfile

```

---

<div class="post-metadata">

**Author:** ![system](https://canada1.discourse-cdn.com/flex035/uploads/elm_lang/original/1X/50a05e53677a2c3b47776d7abd0f113eb50193a1.png) [@system](https://discourse.elm-lang.org/u/system)\
**Post date:** [March 31, 2019, 6:18pm UTC](https://discourse.elm-lang.org/t/working-around-not-enough-bytes-problems/3359/7 "2019-03-31T18:18:09Z")

</div>

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.