Working around "not enough bytes" problems?

#1

Has anyone come up with a strategy for managing the compiler crashes? In particular the elm: not enough bytes. one?

I’ve adapted my elm-vim plugin to look for that in the output and touch all the sources files before re-compiling which makes it less of a pain to deal with. However my colleagues uses VSCode so I can’t share the same approach easily. Are there any other strategies that?

Reading this github ticket it seems there are suggestions around not running the compiler twice at the same time and making sure that elmi and elmo files are in sync. Interestingly I had a script to run a make server process that make help make sure that only one compile was run at a time (because the 0.18 compiler could go quite slowly when there were multiple instances running) but I got rid of it for Elm 0.19 and hadn’t considered re-instating it.

I was hoping that people might share strategies that they’ve been using to deal with this?

As some additional information, I tend to have webpack running that runs compiler. I do compiles in my editor and on the command line at various times. I hit this error 10s of times a day.

#2

Similar experience here. I use Atom + Elmjutsu.

Interesting, could you elaborate on how that fixes the not enough bytes error? I tend to just delete elm-stuff and recompile. I’m under the impression that not enough bytes is caused by corrupt elmi/elmo files, but for some reason the data files are corrupt message is not triggered.

#3

My understanding is that the root cause is when elm is run alongside other elm instances. This means the first instance may be writing a .elmi file while the second instance is reading it. So the second instance is like “hey, there are not enough bytes!”

People who are experiencing this problem more often should be people who have particularly inefficient combinations of secondary tools, that are all calling elm make multiple times at once on the same project.

In an ideal world, tooling would be written more efficiently, but it seems that people producing binaries can use something like flock so that inefficient usage does not cause data races like this.

So I am looking into filelock to do something in the compiler itself, but in the meantime, possible workarounds include:

  1. Have plugins do things more efficiently.
  2. According to the internet here, it may be possible to have a wrapper around the normal elm binary like this:
( 
  flock -e 200
  elm $@
) 200>/tmp/elm.lockfile 

I have not used flock in bash scripts personally, but something like this may work! It will also mean that when you try to run elm make multiple times at once, the later calls will block and skip any work that was done by the first one. (You’ll still be reading files and stuff though, so it’s still worse than just having the plugins avoid multiple simultaneous calls!)

#4

Thank you for the reply. I appreciate the information. I’ve run some tests with an error I have at the moment that seems to trigger the elm: not enough bytes issue. My steps are (having killed my webpack process):

Not enough bytes error

$ rm -fr node_modules
$ yarn
...
$ which elm
/home/michael/root/projects/hive/server/portal/node_modules/.bin//elm
$ file /home/michael/root/projects/hive/server/portal/node_modules/.bin//elm
/home/michael/root/projects/hive/server/portal/node_modules/.bin//elm: symbolic link to ../elm/bin/elm
$ cd server/apps/business
$ rm -fr elm-stuff
$ elm make src/Business.elm
Dependencies loaded from local cache.
Dependencies ready!                
Detected errors in 1 module.                                         
-- TYPE MISMATCH ------------------------------- ./src/Business/Report/Tasks.elm

This function cannot handle the argument sent through the (|>) pipe:

 91|     getPostIdFromReportId session reportId
 92|>        |> Task.andThen
 93|>            (\postId ->
 94|>                Api.readTask
 95|>                    { endpoint = Api.endpoint [ "api", "posts", postId, "comments" ] []
 96|>                    , body = Http.emptyBody
 97|>                    , success = Api.JsonBody <| Decode.field "comments" (Decode.list commentDecoder)
 98|>                    , failure = Zap.Http.noExpectedErrorsHandler
 99|>                    , session = session
100|>                    }
101|             )

The argument is:

    Api.Task () Post.Id

But (|>) is piping it a function that expects:

    Task.Task (Zap.Http.Error ()) String

$ elm make src/Business.elm
elm: not enough bytes
CallStack (from HasCallStack):
  error, called at src/Data/Binary.hs:212:21 in binary-0.8.6.0-1SWOkUSZj1B1k0FQ5fyFqI:Data.Binary

This error seems quite reproducible as I experienced it, fixed it, worked on another branch, returned to this, removed the fix and I get it again.

The State of Elmi & Elmo

If I run a script to detect missing elmi or elmo files:

#!/bin/bash -e

for elmo in $(ls elm-stuff/0.19.0/*.elmo); do
	# echo $elmo
	elmi=$(echo $elmo | sed 's/elmo$/elmi/')
	if [ ! -e "$elmi" ]; then
		echo "$elmi doesn't exist"
	fi
done

for elmi in $(ls elm-stuff/0.19.0/*.elmi); do
	# echo $elmo
	elmo=$(echo $elmi | sed 's/elmi$/elmo/')
	if [ ! -e "$elmo" ]; then
		echo "$elmo doesn't exist"
	fi
done

Then I get:

$ bash ./check-stuff
elm-stuff/0.19.0/Zap-View-Action.elmo doesn't exist

Deleting unmatched files

This takes a bit of time so I’ve attempted to improve it to something that runs quicker and cleans up the problematic files. I’m no expert though so this could have issues and I’m sure there is room for improvement:

elmoFiles=$(mktemp /tmp/elmo.XXXX)
elmiFiles=$(mktemp /tmp/elmo.XXXX)

ls -1 elm-stuff/0.19.0/*.elmo > $elmoFiles
ls -1 elm-stuff/0.19.0/*.elmi | sed 's/elmi/elmo/' > $elmiFiles

# Include tmp files so that we always have something and we clean up after ourselves
diff $elmoFiles $elmiFiles | tail -n +2 | sed 's/> //' | sed 's/elmo/elmi/' | xargs rm -f $elmoFiles $elmiFiles

Wrapping the compiler with flock

I’ve also created script to replace the node_modules/.bin/elm symbolic link with a script that uses flock as you’ve suggested. The final result incorporate the cleanup above and looks like this:

#!/bin/bash -e

nodeModulesFolder=$(dirname $(dirname $0))
(
flock -e 200

# Delete all elmi files that don't have matching elmo file
elmoFiles=$(mktemp /tmp/elmo.XXXX)
elmiFiles=$(mktemp /tmp/elmo.XXXX)

ls -1 elm-stuff/0.19.0/*.elmo > $elmoFiles
ls -1 elm-stuff/0.19.0/*.elmi | sed 's/elmi/elmo/' > $elmiFiles

# Include tmp files so that we always have something and we clean up after ourselves
diff $elmoFiles $elmiFiles | tail -1 | sed 's/> //' | sed 's/elmo/elmi/' | xargs rm -f

$nodeModulesFolder/elm/bin/elm $@
) 200>/tmp/elm.lockfile

I don’t know how flock is meant to work so I’m just guessing from your example but it seems promising (note the ampersand after the first command):

$ elm make src/Business.elm &; elm make src/Business.elm
[1] 10755
Detected errors in 1 module.
-- TYPE MISMATCH ------------------------------- ./src/Business/Report/Tasks.elm

This function cannot handle the argument sent through the (|>) pipe:

 91|     getPostIdFromReportId session reportId
 92|>        |> Task.andThen
 93|>            (\postId ->
 94|>                Api.readTask
 95|>                    { endpoint = Api.endpoint [ "api", "posts", postId, "comments" ] []
 96|>                    , body = Http.emptyBody
 97|>                    , success = Api.JsonBody <| Decode.field "comments" (Decode.list commentDecoder)
 98|>                    , failure = Zap.Http.noExpectedErrorsHandler
 99|>                    , session = session
100|>                    }
101|             )

The argument is:

    Api.Task () Post.Id

But (|>) is piping it a function that expects:

    Task.Task (Zap.Http.Error ()) String
Detected errors in 1 module.
-- TYPE MISMATCH ------------------------------- ./src/Business/Report/Tasks.elm

This function cannot handle the argument sent through the (|>) pipe:

 91|     getPostIdFromReportId session reportId
 92|>        |> Task.andThen
 93|>            (\postId ->
 94|>                Api.readTask
 95|>                    { endpoint = Api.endpoint [ "api", "posts", postId, "comments" ] []
 96|>                    , body = Http.emptyBody
 97|>                    , success = Api.JsonBody <| Decode.field "comments" (Decode.list commentDecoder)
 98|>                    , failure = Zap.Http.noExpectedErrorsHandler
 99|>                    , session = session
100|>                    }
101|             )

The argument is:

    Api.Task () Post.Id

But (|>) is piping it a function that expects:

    Task.Task (Zap.Http.Error ()) String
[1]  + exit 1     elm make src/Business.elm

Without the wrapper this looks much less happy.

Helper script to wrap compiler

Here is a script that I run to wrap the compiler. Useful as a yarn re-install will lose it.

#!/bin/bash -e

rm -f node_modules/.bin/elm

cat << ENDCAT > node_modules/.bin/elm
#!/bin/bash -e

nodeModulesFolder=\$(dirname \$(dirname \$0))
(
flock -e 200

# Delete all elmi files that don't have matching elmo file
elmoFiles=\$(mktemp /tmp/elmo.XXXX)
elmiFiles=\$(mktemp /tmp/elmo.XXXX)

ls -1 elm-stuff/0.19.0/*.elmo > \$elmoFiles
ls -1 elm-stuff/0.19.0/*.elmi | sed 's/elmi/elmo/' > \$elmiFiles

# Include tmp files so that we always have something and we clean up after ourselves
diff \$elmoFiles \$elmiFiles | tail -n +2 | sed 's/> //' | sed 's/elmo/elmi/' | xargs rm -f \$elmoFiles \$elmiFiles

\$nodeModulesFolder/elm/bin/elm \$@
) 200>/tmp/elm.lockfile
ENDCAT

chmod +x node_modules/.bin/elm

I’ve only just been playing around with them and maybe there are other kinds of this error that this doesn’t handle but it seems promising and perhaps this will be useful to someone. I’ll update I experience issues. I fear that my excitement at messing around with this stuff as won over my better judgement that I should only post about this when it has been tested a bit more.

#5

My guess is that if you touch all the files than the compiler is going to try to recompile all of them so it doesn’t matter what the previous state of the elmo and elmi files are as they are going to be replaced. I felt that this approach might be quicker than removing elm-stuff as it saves elm from copying over the libraries & dependencies again. Though I imagine that is pretty quick.

#6

I’ve made some further changes after encountering some issues. I’m not sure yet how useful it is. I feel like I’m still seeing the error with it sometimes.

The latest version of the replacement script I have is:

#!/bin/bash -e

nodeModulesFolder=$(dirname $(dirname $0))

(
    flock -e 200

    # Open sub-shell to contain directory change
    (
        # Search upwards for the elm.json folder - assuming elm-stuff will be next to it but
        # might not exist yet - so that this will work when run by elm-test
        path=$(pwd)
        while [[ "$path" != "" && ! -e "$path/elm.json" ]]; do
          path=${path%/*}
        done

        # Change to parent of elm-stuff directory
        cd $path

        # Only try to clean up elm-stuff if it exists
        if [ -e "$path/elm-stuff" ]; then
            # Delete all elmi & elmo files that don't have a matching pair
            elmoFiles=$(mktemp /tmp/elmo.XXXX)
            elmiFiles=$(mktemp /tmp/elmo.XXXX)

            ls -1 elm-stuff/0.19.0/*.elmo > $elmoFiles
            ls -1 elm-stuff/0.19.0/*.elmi | sed 's/elmi/elmo/' > $elmiFiles

            # Include tmp files so that we always have something and we clean up after ourselves
            diff $elmoFiles $elmiFiles | grep elm-stuff | sed 's/\(>\|<\) //' | sed 's/\(.*\).elm./\1.elmi \1.elmo/g' | xargs rm -f $elmoFiles $elmiFiles
        fi
    )

    $nodeModulesFolder/elm/bin/elm $@

) 200>/tmp/elm.lockfile
closed #7

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.