Bytes.Encode can't fail

Request Feedback
1

I have a moderate sized Elm application that deals with binary protocols (specifically SysEx data dumps and messages that MIDI instruments send). Back in 0.18, I wrote my own ByteArray type, and associated parser, builder, and codec style combinator libraries.

I’ve moved the app to 0.19, and am starting to replace my functionality with elm/bytes types…

First thing I’ve run into is that Byte.Encoder objects can’t fail to encode. A reality of many “bytes on the wire” protocols is that the logical Elm types that they decode into, can have values that can’t be correctly re-encoded. This happens commonly in two cases:

  1. Fields that have limited valid ranges: For example, it might encode a value as a 16bit integer, but perhaps only values between 0 and 4,000 are legal. If it is decoding into an Int, then corresponding encoder has no way of signaling failure if encoding a value outside the range.

  2. Repeated structures: For example, a data dump might have exactly 80 copies of some other structure in a row. It would be best to decode that into Array Thing or [Thing] – and it is easy to make the decoder decode exactly 80… But the encoder has no way of signaling failure if the passed array or list value is the wrong size.

Now, you might argue that these kinds of constraints should be checked - like other structural constraints - before passing to the Encoder. But if you are building up the encoder from parts - you’d have to replicate the whole structure again building up a checker. It generally makes sense - esp. for these low level field checks or simple structural checks - to build them up as you are building the encoder.

I think I might be able to build a “checked” encoder API on top of the existing one… but I thought this observation might be interesting to discuss.

1 Like
2

This follows the same pattern as other Encoders in Elm. The idea is that you decode things into types which strongly enforce their constraints, such that when you encode it, it will be valid.

So, instead of decoding into an Int, decode into a type MySpecialNumber = MySpecialNumber Int opaque type, and only allow that decoder to succeed if the number is within the accepted range. Try to make it so that the only way to get a MySpecialNumber value, is if it’s valid.

That way you don’t have to worry about invalid data when encoding, as it will always be good.

2 Likes
3

Those are not unreasonable ideas… but in a very large protocol surface, that can become quite unwieldy to have to create wrapper types, and then unwrapper functions for them all.

For example - I have a single data message that has four different arrays of fixed sizes. There are dozens of messages. Since there is no good way to write a higher order type encoding the size, this will get verbose rather quickly. I’ll probably take the coward’s way out… sigh…

4

Another solution is to use a wrapper module like this. It has the same API as the Bytes.Encode module but the encode function returns Result Error Bytes instead of just Bytes.

It also has an succeed and a fail function to build custom encoders.

Example

You define a custom encoder function like this:

import Bytes.EncoderThatCanFail as Encode exposing (Encoder)

below4k : Endianness -> Int -> Encoder
below4k e i =
    if i <= 4000 then
        Encode.unsignedInt16 e i
    else
        Encode.fail "Must be below 4k"

And when you encode a value you’ll get a result instead of Bytes

Encode.encode (below4k 3000) == Ok <2 bytes>
Encode.encode (below4k 5000) == Err (Other "Must be below 4k")

Wrapper module source

This is the source code of the wrapper module

module Bytes.EncoderThatCanFail exposing (..)

import Bytes exposing (Bytes, Endianness)
import Bytes.Encode as BE
import Result.Extra as Result


type alias Encoder =
    Result Error BE.Encoder


type Error
    = OutOfRange Where Int
    | Other String


type Where
    = SInt8
    | SInt16
    | SInt32
    | UInt8
    | UInt16
    | UInt32


encode : Encoder -> Result Error Bytes
encode encoder =
    Result.map BE.encode encoder


sequence : List Encoder -> Encoder
sequence list =
    Result.combine list
        |> Result.map BE.sequence


succeed : BE.Encoder -> Encoder
succeed encoder =
    Ok encoder


fail : String -> Encoder
fail err =
    Err (Other err)


signedInt8 : Int -> Encoder
signedInt8 i =
    if i >= -128 && i <= 127 then
        BE.signedInt8 i
            |> Ok
    else
        OutOfRange SInt8 i
            |> Err

signedInt16 : Endianness -> Int -> Encoder
signedInt16 e i =
    if i >= -32768 && i <= 32767 then
        BE.signedInt16 e i
            |> Ok
    else
        OutOfRange SInt16 i
            |> Err

signedInt32 : Endianness -> Int -> Encoder
signedInt32 e i =
    if i >= -2147483648 && i <= 2147483647 then
        BE.signedInt32 e i
            |> Ok
    else
        OutOfRange SInt32 i
            |> Err


unsignedInt8 : Int -> Encoder
unsignedInt8 i =
    if i >= 0 && i <= 255 then
        BE.unsignedInt8 i
            |> Ok
    else
        OutOfRange UInt8 i
            |> Err


unsignedInt16 : Endianness -> Int -> Encoder
unsignedInt16 e i =
    if i >= 0 && i <= 65535  then
        BE.unsignedInt16 e i
            |> Ok
    else
        OutOfRange UInt16 i
            |> Err


unsignedInt32 : Endianness -> Int -> Encoder
unsignedInt32 e i =
    if i >= 0 && i <= 4294967295 then
        BE.unsignedInt32 e i
            |> Ok
    else
        OutOfRange UInt32 i
            |> Err


float32 : Endianness -> Float -> Encoder
float32 e f =
    BE.float32 e f
        |> Ok

float64 : Endianness -> Float -> Encoder
float64 e f =
    BE.float64 e f
        |> Ok


bytes : Bytes -> Encoder
bytes =
    BE.bytes
        >> Ok

string : String -> Encoder
string =
    BE.string
        >> Ok

getStringWidth : String -> Int
getStringWidth =
    BE.getStringWidth
5

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.