Container of generic container in Elm

[arthur_teisseire]

I’m running into a design issue and I need your help.
I have a database containing my tables (Dict):

type alias Database =
    { table1 : Dict Int Int
    , table2 : Dict Int String
    }

I want to remove an id in each table. To do this I can call Dict.remove on each table:

remove : Int -> Database -> Database
remove id database =
    { table1 = Dict.remove id database.table1
    , table2 = Dict.remove id database.table2
    }

The issue is that when I add a new table I have to add it to the remove function.

Now I want to know the size of my database. I can call Dict.size for each table like I did for remove:

size : Database -> Int
size database =
    Dict.size database.table1
        + Dict.size database.table2

So now, when I add a new table, I need to add it to the remove and size function.

And so on… In fact, what I want, like for any other container, is to have a foldl function which could look like:

foldl : (Dict Int a -> result -> result) -> result -> Database -> result
foldl func result database =
    result
        |> func database.table1
        |> func database.table2

With foldl, when I add a new table I have to add it to the foldl function but, only in foldl function. But this doesn’t compile for an understandable reason: Type mismatch . Required: Dict Int a Found: Dict Int Int.

In fact, all my tables (Dict) have a common thing: Their id (Int). If they have a “common thing” I want to do “common actions” on them. But, I can’t figure how to.
The best would be to iterate over my tables in foldl to avoid adding a line each time I add a table. But to do this I think we need reflection.

I love elm because it helps me to generalize my local design issues to the simplest form. A big thanks to everyone in the Elm community!

I’m really curious to hear your answers !

[rupert]

You could use Value to make it more reflective:

type alias Database = Dict Int Value

getString : Int -> Database -> Maybe String
getString key db =
    Dict.get key db
        |> Maybe.andThen (Decode.decodeValue (Decode.string) >> Result.withDefault Nothing)

getInt : Int -> Database -> Maybe Int
getInt key db =
    Dict.get key db
        |> Maybe.andThen (Decode.decodeValue (Decode.int) >> Result.withDefault Nothing)

[arthur_teisseire]

Oh I didn’t know about Value, is it a built-in feature made for reflection purpose ?
In fact, doing this is not exactly the same because I can’t have a String and an Int with the same id in my database. Or I missed something.

[dta]

Based on your remove function, it seems like all of your tables share a common id space (meaning that no two values in any tables can share an id). If that’s the case, I might recommend using a single table with custom type to store the value:

type DatabaseEntry = IntEntry Int | StringEntry String

type alias Database = Dict Int DatabaseEntry

remove : Int -> Database -> Database
remove = Dict.remove

size : Database -> Int
size = Dict.size

foldl : (Int -> DatabaseEntry -> result -> result) -> result -> Database -> result
foldl = Dict.foldl

You may want to add some helpers to make querying the dictionary easier:

getIntEntry : Int -> Database -> Maybe Int
getIntEntry key =
    Database.get key
        |> Maybe.andThen (\value ->
            case value of
                IntEntry entry -> Just entry
                StringEntry _ -> Nothing
        )

getStringEntry : Int -> Database -> Maybe String
getStringEntry key =
    Database.get key
        |> Maybe.andThen (\value ->
            case value of
                IntEntry _ -> Nothing
                StringEntry entry -> Just entry
        )

[arthur_teisseire]

My example is maybe not enough exhaustive. In fact in my case this is the opposite. I only have one-to-one tables relations. So all the id are shared.
To be more precise on my intentions, here is the concrete case behind this issue: I’m creating an entity component system game engine. My entities are only represented by an id, and each component type is stored in its Dict. Example Dict Int Transform. The Dict key is my entity id in each of those tables. This allows me to add new component types easily and my systems just query different components, join them by id and update them.

[rupert]

@arthur_teisseire

Oh I didn’t know about Value, is it a built-in feature made for reflection purpose ?

From elm/json encoders/decoders: Json.Encode - json 1.1.3

Not really intended for reflection but it can fulfil that role.

@arthur_teisseire

In fact, doing this is not exactly the same because I can’t have a String and an Int with the same id in my database. Or I missed something.

Ok I mean like this:

import Dict exposing (Dict)
import Json.Decode as Decode exposing (Value)

type alias Database =
    { table1 : Table
    , table2 : Table
    }

type alias Table = Dict Int Value

getString : Int -> Table -> Maybe String
getString key db =
    Dict.get key db
        |> Maybe.andThen (Decode.decodeValue (Decode.string) >> Result.withDefault Nothing)

getInt : Int -> Table -> Maybe Int
getInt key db =
    Dict.get key db
        |> Maybe.andThen (Decode.decodeValue (Decode.int) >> Result.withDefault Nothing)

[arthur_teisseire]

Oh ! With this it seems I can create the foldl function. I’ll will test it asap (not at home currently). Thank you for your time !

[arthur_teisseire]

Moreover I can create a List Table and don’t have to implement it !

[arthur_teisseire]

But, it seems it will have a huge impact on performances if I need to access all my fields each frame. Isn’t it ?

[pit]

Maybe I’ve been writing too much about Jeremy’s interface technique recently and now every problem seems to scream “hide the type, hide the type”

Here’s another idea:

In foldl, you could transform each of your Dict Int a into the following type which lists the “common actions” you want to perform on the Dicts:

type alias GenericDictOps =
    { size : Int -> Int
    , remove : Int -> Database -> Database
    }

This way you could keep your current Database type for (maybe) faster access.

Here’s an Ellie with the complete code: https://ellie-app.com/mfj7D445ZrVa1

But: if your database is a record of Dicts with identical keys, why don’t you change the database into a single Dict where the values behind the ids are records holding an entity’s components in individual fields?

[rupert]

@pit

In fold, you could transform each of your Dict Int a into the following type which lists the “common actions” you want to perform on the Dicts:

type alias GenericDictOps =
    { size : Int -> Int
    , remove : Int -> Database -> Database
    }

I was also thinking of the interfaces approach. But how do you deal with the get operations? One gets an Int and the other a String, so they are not the same interface:

type alias GenericDictOps =
    { size : Int -> Int
    , remove : Int -> Database -> Database
    , get : Int -> ?
    }

[pit]

Hi Rupert, you can see in the Ellie that I build the GenericDictOps on the fly and only inside of the foldl function.

I don’t think that a get operation is something you can use in a fold. Arthur’s “common actions” he wanted to pass to foldl are of the form Dict Int a -> result -> result. This only works if the result type is independent of type a.

Because I don’t store the GenericDictOps anywhere, Arthur’s Database type is unchanged:

type alias Database =
    { table1 : Dict Int Int
    , table2 : Dict Int String
    }

so get operations can still be performed on the individual Dict Int a record fields.

[rupert]

type alias GenericDictOps =
    { size : Int -> Int
    , remove : Int -> Database -> Database
    }

Ok I get it. Your remove operation acts on Database, so the Database model itself is not a hidden type, like it would be if using the interfaces approach.

[arthur_teisseire]

Very good suggestion, thank you @pit I will try it out asap. If I understand well, GenericDictOps is a kind of typeclass ?

[arthur_teisseire]

It solves my issue, I’m sincerely grateful for your help. Thank you @pit, @dta and @rupert for your time and excellent suggestions !

[arthur_teisseire]

@pit

But: if your database is a record of Dicts with identical keys, why don’t you change the database into a single Dict where the values behind the ids are records holding an entity’s components in individual fields?

Some entities can have a Transform and Visual components and others can have Transform and Collider but no Visual. With a simple Dict, I don’t see how to represent this.

[dta]

type alias Database =
    Dict Int
        { transform : Maybe Transform
        , visual : Maybe Visual
        , collider : Maybe Collider
        }

[arthur_teisseire]

Yes, this is another way to do it. But I prefer the other way for many reasons. First, it is easier to serialize many Dict with their component rather than one with any components, I can use an existing sql database. Then, for memory reasons. With one Dict with all maybe components each entity will have the size of all components, which is a sufficient reason to prefer the approach of one table by component. Then for performance reasons, cache misses will be huge with your approach. Moreover, my systems don’t need to depend (indirectly but still) of each component, but only the one it uses with my approach, letting me share same systems across different games.
On top of that, my engine, as you see, is still work in progress and I’m not sure yet I will only need one-to-one table relations. Maybe I will need many-to-one which will be impossible with the design you propose.
That was a hard decision from start that I absolutely don’t regret. But, yes, this design deserve to be challenged, thank you for doing so !

[arthur_teisseire]

I just looked at the code you provided and hiding types this way is brilliant. I’ve never seen this before. To understand better this concept, I will look closer to your post and first watch videos you linked. I feel this will open my mind to a new way of doing things. Thanks a lot for sharing this !