SurrealDB, the perfect backend for Elm apps?

Atlewee · February 12, 2023, 1:12pm

Anyone tried SurrealDB? I just tried it and find a lot of things reminding me of elm.
New and radical ideas. Simple and super performant with a small footprint.
It is made by two dedicated brilliant brothers, not a company.
The main goal is to be able to develop applications with no API or backends, you just need the database and a frontend.

It is like Firebase, but with a better database, better access control and can run on an embedded device or scale up to large clusters. I think it is just a few MB self contained thing. Written in Rust
And can also run inside the browser itself if you want offline support (wasm)
It takes the best from SQL, Document and graph databases. Has native json support so you can easily work with arrays and structured objects. You can choose to use schema or not. Or only schema for some tables. As it is a database made for webapps its primary interface is HTTP and websockets, both a perfect fit for elm. The database can even hold javascript functions to have calculated fields and rules. (maybe we someday could compile those from elm functions?)

They say they have been using it in production themself for several years, but it is released to the public only the last couple of months.
It’s geting raving reviews by anyone that has looked at it.

Here is the Fireship “SurrealDB in 100 seconds”:

The website also looks amazing:

Just wanted to share this with others looking for a nice elm backend

dirkbj · February 13, 2023, 6:23pm

Thanks for pointing this out. I’ve not looked at it, but certainly intend to.

kanishka · February 13, 2023, 10:27pm

I’m curious.

This comment about nested documents has me hesitating a little - https://www.reddit.com/r/rust/comments/wt3ygg/comment/ilnkdv4/?utm_source=reddit&utm_medium=web2x&context=3

Laurent · February 14, 2023, 9:07am

@Atlewee From your description I first though it was yet another Firebase alternative, but SurrealDB, as it name says, is simply a database without authentication or hosting or serverless functions. So it could merely replace Firebase Firestore or Firebase Realtime DB, except that it is a SQL database. And without Firebase offline capabilities.

Atlewee · February 14, 2023, 9:59am

Yes you are correct, I was thinking of the Firebase realtime database. Not the whole firebase solution
But it also seems to have built in user account management / signup and stuff + functions that can run on triggers inside the database. So sort of cloud-functions capabilities as well.
And you can write your own js/wasm traversing functions so it happens inside the database instead of sending data to the client before the next step in treaversal.

I would not call surreal an SQL database… It seems to be mostly a json document database but with additional graph and SQL features + schema possibilities.

As for offline, they are definetly aiming for offline first webapps. That’s the whole reason why SurrealDB is designed to run embedded in the browser, as well as in a server environment.

timinou · February 14, 2023, 12:36pm

I’m currently checking it out but the documentation has a lot of Lorem Ipsums… @Atlewee Did you find good docs to get you started on auth and Websockets connections? Or a gist of using it with Elm? Thanks!

Atlewee · February 14, 2023, 2:09pm

Yes docs is not complete.

Websocket looks simple. Connect unauthenticated and run the signin command passing username,password or a Token.
All websocket commands is a json object like this:

{
  "id": <an id so you can identify responses later on>,
  "method": <one of the available commands>,
  "params": <an array of parameters>
}

So the websocket message to select a namespace and database would look like this in elm:

E.object
  [ ( "id", E.string "1" )
  , ( "method", E.string "use" )
  , ( "params", E.list identity [ E.string "testNamespace", E.string "testDatbase" ] )                                    
  ]

As the websocket docs is not created yet I looked at the source of the javascript client to see how it uses the websocket:

github.com

surrealdb/surrealdb.js/blob/ce949aeddd2b451b3b7b473705e62fbbc58e095b/src/index.ts#L198


      
          
          
		if (url) {
          			this.connect(url);
          		}
          	}
          
          
	/**
          	 * Connects to a local or remote database endpoint.
          	 * @param url - The url of the database endpoint to connect to.
          	 */
          	connect(url: string): Promise<void> {
          		// Next we setup the websocket connection
          		// and listen for events on the socket,
          		// specifying whether logging is enabled.
          
          
		this.#ws = new Socket(url);
          
          
		// Setup the interval pinger so that the
          		// connection is kept alive through
          		// loadbalancers and proxies.

The source on the server will also tell you all the available commands:

github.com

surrealdb/surrealdb/blob/1dd08bedaa997855c4c28dfc03ca23c650f6b2c6/src/net/rpc.rs#L128


      
          		let method = match req.pick(&*METHOD) {
          			Value::Strand(v) => v.to_raw(),
          			_ => return Response::failure(id, Failure::INVALID_REQUEST).send(chn).await,
          		};
          		// Fetch the 'params' argument
          		let params = match req.pick(&*PARAMS) {
          			Value::Array(v) => v,
          			_ => return Response::failure(id, Failure::INVALID_REQUEST).send(chn).await,
          		};
          		// Match the method to a function
          		let res = match &method[..] {
          			"ping" => Ok(Value::True),
          			"info" => match params.len() {
          				0 => rpc.read().await.info().await,
          				_ => return Response::failure(id, Failure::INVALID_PARAMS).send(chn).await,
          			},
          			"use" => match params.take_two() {
          				(Value::Strand(ns), Value::Strand(db)) => rpc.write().await.yuse(ns, db).await,
          				_ => return Response::failure(id, Failure::INVALID_PARAMS).send(chn).await,
          			},
          			"signup" => match params.take_one() {

Best description I found about auth (external JWT and how access-control works) was this answer:

Once you get that running the rest of the documentation you need seems nice and complete:

timinou · February 14, 2023, 8:39pm

Wow thanks! This is very thorough. I’ll use it for my next project

supermario · February 17, 2023, 4:22pm

Am I missing something here, or are there no features related to migrating/changing your schema?

I can’t seem to find any mention of schema migrations in the docs nor github

Atlewee · February 18, 2023, 12:27pm

Seeing your name, I now see that my title is incorrect. Should be something like. “If for any reason you cant use lamdera yet, would SurrealDB be the next best thing for elm-apps?”
I guess nothing beats the lamdera experience when it fits your requirements.

Great question, I have not looked into migrations yet. ( I just discovered surrealDB 6days ago )
Found a discussion on discord about migrations:

Where the answer from one of the makers is this:

So no tools for migrations yet. But I guess one could turn of schema for the effected tables, change data and turn on new schema? But not sure if that is possible in a single transaction, have not tried.
Or map the complete db or tables over to a new db/table with transform functions, immutable style.

Talking about immutability:
I forgot to mention one of the biggest difference to other databases: SurrealDB is immutable. It is meant to tell you how a single field in a table has changed over time in a performant way and with compact storage of changes.
For those interested I found this really interesting whitepaper written by one of the authors comparing performant immutable database structures: (it is more or less describing design decisions and how the previous version worked under the hood, it had a different name then and was written in go. They have later found even more performant structures and changed to rust + renamed to surrealDB after that)

supermario · February 18, 2023, 4:35pm

Very interesting, thanks for sharing!

I’m blown away how convincingly it’s presented as a production ready system with no data migration concepts at first I thought maybe you were expected to make new tables but their foreign key system is effectively hardcoded to a table name if I’m understanding correctly.

timinou · February 19, 2023, 1:56pm

That’s my understanding as well. Especially knowing how well you handle Evergreen migrations in Lamdera, it’s a bit harrowing to build a production app with no migration tooling at all…

In their defence, they do use SurrealDB in their production apps, and they are technically in beta. This feature request seems to have been picked up, though: Feature: `RENAME` statement for fields, databases and events · Issue #1615 · surrealdb/surrealdb · GitHub

I’ve been using it for a few days now and I’m thoroughly enjoying the parts that just work. I think I’ll stick with it for this project, have a few makeshift scripts to test database health, and hope they come up with the rename field before I need it

supermario · February 21, 2023, 7:17am

Cool, interested to hear your experience report once you feel you’ve got a handle on it.

Also if you come across documentation on the supported types (I.e byte sizes, lengths, encodings, limitations, etc) that would be cool to see.

timinou · February 24, 2023, 8:22am

Sounds good! In the meantime, I found this project with a lot of cool patterns. I’m learning a lot from them!

system · March 6, 2023, 8:23am

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Recommendations for Easing Elm into a Project Learn	5	881	January 21, 2022
Elm, meet Streamlit 🤝 Show and Tell	9	1175	December 27, 2020
Elm 0.19 - From a production perspective Request Feedback	6	7200	August 31, 2018
Elm-conf videos are published! Show and Tell	6	1572	November 6, 2018
Realworld Hasura + Elm Show and Tell	13	2243	November 20, 2020

Related Topics