Hi, I’m about to publish a package to make it easier to use IndexedDB from elm apps. Would love some feedback before hitting the publish button of version 1.0.0.
Love that you’re doing this! I don’t have a concrete use case right now, but definitely something I’ve wanted in the past. elm-concurrent-task wasn’t around when I last looked at this and it definitely seems to be a key piece of the puzzle. Again, thank you!
With all the new LLM powers we have, I’ve been doing a lot of vibecoding for small utility mobile apps, local-first. And it pains me that I’ve been using SolidJS instead of Elm for it (even though solid is quite nice). So I’m trying to fill the gaps to be able to unleash claude with elm powers ^^. This means better support of a few things I need. And for local-first PWAs, IndexedDB is a must have!
So this IndexedDB package is a first step in my larger goal
Don’t want to diverge too much from the topic, happy to discuss in a separate thread/space as well. I’m curious if you’ve hit any issues with your PWA work and iOS or Android? Either IndexedDB related or anything else.
I have a couple of suggestions to improve type safety and ergonomics but I am pretty sure you intentionally decided against what I’ll suggest. Some (maybe all) of my suggestions may be based on ignorance of IndexedDb or the intended use cases.
The Store type has a phantom type argument to protect portions of the Api related to the three different key categories of Explicit, Inline, and Generated which is really great. However
When using get one can pass an Int, Float or compound key to a store that has a String key and that seems like an outright run-time error.
When using get or putAt one is limited to simple data types like string, int, etc. so one cannot use a semantically rich id type like type PersonId = PersonId String or type PetId = PetId String.
Users have to include the decoder for every get and provide a Value for write operations which feels type-unsafe and a little tedious. Though perhaps this is a feature and not a bug in that it allows for flexibility? If so then discussing the use cases in the documentation might be nice.
What if we enhanced the type signature of store a wee bit?
type alias KeyCodec key =
{ encode : key -> Key -- this makes it so that you can only produce appropriate keys
, decode : Key -> Result String key
}
-- keyCategory would be your marker types ExplicitKey, InineKey, GeneratedKey
type Store keyCategory key model
= Store
{ config : StoreConfig -- holds the name, key path and auto increment
, keyCodec : KeyCodec key
, modelCodec :
{ encoder : model -> Value
, decoder : Decoder model
}
}
Now there would be enough type information in the store to rigorously type the get and putAt operations with respect to the key and users wouldn’t have to pass the encoder/decoder every time.
Of course, this makes a couple of assumptions
This is to be a high level library that is used by the application code directly. If this is intended as a lower level library put behind an opaque data type (PetStoreDb or TodoDb) then you might want the flexibility and both the type safety and ergonomic would be moot because 96% of the code would go through the explicitly defined / strict application or domain Api.
I am assuming that all keys in a store should have a homogeneous type.
When using get one can pass an Int, Float or compound key to a store that has a String key
That’s actually not forbidden by IndexedDB, and though I thought about limiting it, I eventually decided on not doing it. So you can use multiple types of keys for the same store.
When using get or putAt one is limited to simple data types like string, int, etc.
Indeed. For keys, IndexedDB support numbers, dates, strings, binary, and arrays. Among those, it was easy to support numbers, strings, and arrays. Adding more complex keys would require you first converting these keys into one of the supported types. So it felt like it was simpler and versatile enough to do as currently.
Users have to include the decoder for every get and provide a Value for write operations which feels type-unsafe and a little tedious.
Indeed, it’s on purpose. I often have stores that are generic “stuff” stores where I put things of different nature in it. Thus it is required to be able to use different decoders for each of the read into the DB. I suppose we could make that more typesafe by defining at the store creation if a store has a fixed “shape” or is “multi-shape”. In the former, we could provide the codecs at creation time. In the latter, use Value and codecs at execution time.
Regarding keys/phantom types, I’ve been thinking about how I’m going to use this in a project I’m working on and I’ve been thinking that I could have my own API on top of this IndexedDB package. Similar to how I might have a IdDict value module on top of Dict key value where internally it’s
type Id key = Id String
type IdDict key value = IdDict (Dict String value)
This separation would let me play a little more lose with IndexedDB during my initial exploration phase and then harden it later when I know what I want. Not sure if that’s a good idea or not.
I’ve been playing around a bit the raw package and was wondering about the GeneratedKey stores. If possible I’d prefer the DB does the ID/key generation but I don’t see a way with the current API of being able to do a put/upsert on a store that uses GeneratedKeys. Am I missing something or is this currently unsupported?
Ah, it’s missing indeed. I’ve mostly used indexeddb as an immutable store in my apps, so I didn’t think of it. How would you suggest to handle that? Different function, making the phantom type more advanced with capabilities (extensible records) seems a bit overkill for a small number of functions.
I’m not sure I know enough about the differences with the existing put vs putAt functions beyond them operating on different kinds of stores. I can take a look later this evening maybe.
My initial thought was that autoIncrement means the key is an Int so we should be able to assume it’s an IntKey, but then we can’t use the existing put or putAt because they’re restricted to only a single type of store.
Yeah, the auto-generated keys are numbers. In theory, you can use something like put or add even on stores that have auto-generated keys, so where it’s not needed to provide keys.
Doing that separation of store types in the package was a design decision to make it a clear split between stores were you manage keys VS stores that you don’t. But I forgot about edits on the auto-increment store.
Maybe a function named replace would make sense.
replace : Db -> Store GeneratedKey -> Key -> Value -> ConcurrentTask Error ()
People could abuse it to also just put stuff in a store with auto-increment with a custom key. Internally this would essentially be the same JS implementation as putAt anyway. But at least the naming of this function conveys better the intent. With Store GeneratedKey you use either insert or replace.
Another thought occurred to me. When I’m loading a store, is it possible to load both the data and the key for the data? Right now I think it’d be 2 separate requests. 1 to get all of the keys and another to load their respective values. The downside to this is it would mean N + 1 requests to load all data.
Another option would be to store the keys in the object I’m saving. This feels a little wrong as I’m storing the key twice essentially. Also not sure how to handle the case where I’m saving a new instance of an object. I’d need to have a placeholder key to store in the object and then immediately overwrite if after saving? Something like
So first, this isn’t an issue for Store InlineKey since the key is literally in the JS data, and I suppose also in the equivalent Elm data. So this is mostly a problem for ExplicitKey and GeneratedKey.
IndexedDB will answer with the keys and with the values in the same order. So in JS this isn’t a problem because we can make both these store calls in the same transaction to guaranty consistency when mapping the keys with the values. Here however, for simplicity I’ve made each function be executed on its own transaction. This may cause issues since the keys and values retrievals are done in two different transactions.
The simplest fix seems to change the getAll function to have the following signature:
getAll : Db -> Store k -> Decoder a -> ConcurrentTask Error (List (Key, a))
A few changes to close gaps in the API around GeneratedKey stores and key retrieval.
getAll now returns keys alongside values
-- before
getAll : Db -> Store k -> Decoder a -> ConcurrentTask Error (List a)
-- after
getAll : Db -> Store k -> Decoder a -> ConcurrentTask Error (List ( Key, a ))
Previously, retrieving both keys and values required two separate calls (getAll + getAllKeys), each running in its own transaction with no consistency guarantee. Now getAll fetches both in a single transaction. The overhead of also returning keys when you only need values is minimal. For InlineKey stores the key is already embedded in the value so this is mostly relevant for ExplicitKey and GeneratedKey stores.
insertMany now returns the generated keys
-- before
insertMany : Db -> Store GeneratedKey -> List Value -> ConcurrentTask Error ()
-- after
insertMany : Db -> Store GeneratedKey -> List Value -> ConcurrentTask Error (List Key)
The returned keys are in the same order as the input values, so you can associate each value with its generated key.
New replace and replaceMany for GeneratedKey stores
replace : Db -> Store GeneratedKey -> Key -> Value -> ConcurrentTask Error ()
replaceMany : Db -> Store GeneratedKey -> List ( Key, Value ) -> ConcurrentTask Error ()
These let you update existing records in a GeneratedKey store using a key previously obtained from insert or insertMany. replaceMany runs all updates in a single transaction.
That totally makes sense. It makes sense that this should be a general purpose low-level library designed to grant Elm developers access to IndexedDb so it should focus on abilities and flexibility over premature type safety. I have a tendency to want to lock things down with types perhaps a little too soon.
The Api that I suggested would be appropriate only for homogeneous stores, which is a subset of the total. IndexedDb supports heterogeneous data in a store so that engineers don’t need to do migrations - among other use cases.
Perhaps in a future version you could add support for
Indexes would be nice for a future version indeed.
For upgrade on open, I’m wondering if this would require a full featured DSL because we can’t just give it JS code from Elm. (Well I guess we could but I’d rather not do that ahah)
Currently, I feel like the only way to control it is to always keep old stores and just deprecate them in new versions. It would work this way:
Always have a “version” store that just keep the last version that was used before the current open. Check the previous version used just after open call.
If we are on a new schema version:
Apply some logic to manually process a migration
Upgrade the schema version in the version store
Clear the old stores of their data to free up space
Encode all that logic into a migrateData task and call it after open.
