A new Zokka version (bringing private package repositories)

[yosteden]

A new release of Zokka with private package repositories

For those who haven’t heard of Zokka before, see Zokka: Bug fixes for the Elm compiler and core packages

We have our first non-alpha (but still beta) release of Zokka! From here on out Zokka’s various configuration file formats and command line interface should not change up until its first production release.

The big feature of this release is a new kind of package repository. This release combines both changes to the Zokka compiler as well as the introduction of a new package repository server that anyone can deploy on their own machines, or can use my publically deployed version.

If you’ve ever wanted to be able to just experiment with publishing some packages before making them available to the whole wide world, or there’s just some packages you don’t want to expose, read on!

The package repository website is in extreme alpha! It is very ugly and it will change substantially! Do not use this for any critical packages! All package and user data will be deleted sometime past Jun 15th, 2024 at midnight UTC time. A deployment of a new server version will take place then.

From its inception, Zokka has had the ability to use custom packages from custom locations. However, apart from single ZIP files, it was actually pretty hard in practice to set up your own custom package repository, even if it was possible in theory. Even though the standard Elm package repository is open source, it doesn’t actually compile out of the box (some modifications are first required) and it’s tied to (publicly available) GitHub so that you can end up with a problem if you have packages you’d like to eventually publish to the standard Elm package, or if you don’t want to use GitHub.

This release of Zokka finally creates a new kind of repository that is different from the standard Elm and Zokka package repositories. These are repositories that:

1. Are not tied to GitHub or even git

2. Are aimed at teams and individuals with private packages that are not meant for the external world to consume. This can either be code meant to be private indefinitely, or alpha/beta releases you’d like to only share with a small group of people first

3. Are gated behind private API keys

4. Are easily created and destroyed in a few clicks on a third-party website. This allows for quick and easy experimentation of package publishing instead of committing to publishing a package forever on the standard Elm package repository.

   In particular I have an instance of this package server running that anyone can create an account on and upload their own packages gated behind an API key only they have access to. Read on for more information about that.

I emphasize again that these custom repositories are meant to be private repositories shared with a team or a small group of individuals. These are not meant to be repositories shared with the entire external world! That means that every package uploaded to this repository must have an API key used to authenticate it. There are no globally readable packages here!

This is because:

1. I don’t want to fragment the package community. If possible, if you have some package you’d like to share with the world please upload it to the standard Elm package repository. If for some reason that is not possible, reach out to me and we can talk about whether it would be possible to put the package on the standard Zokka package repository.
2. I don’t want to be on the hook for a large bandwidth bill for a free service because of some third-party CI process repeatedly downloading large ZIP files over and over again.
3. I’m allowing for deletion of repositories and packages and I believe small teams and individuals can handle deletions more gracefully than the dependency chaos that would cause for the external world at large to depend on a package and have it deleted out from under them.

To emphasize that this package service is not meant to be an authoritative source for globally readable packages, I purposefully have not used the usual zokka-lang.com domain name for this package server, but have made it on another domain that I use for various other projects I run. In particular the package server is located at https://zokka-custom-repos.igneousworkshop.com.

The package server to some extent, but especially the UI are very unpolished and very ugly. I’ve taken the work the work that @dmy did with https://github.com/dmy/elm.dmy.fr/ and
butchered it with all sorts of unstyled buttons and blobs of text.

However, all the core functionality should be there. If you’re willing to hold your nose a little bit at the ugliness, I hope that you’ll find this package server useful!

This is also the first beta release of Zokka because at this point, I don’t expect the configuration file format or user interface of the Zokka compiler to change. The package server and its UI may still change fairly drastically, but the API with the Zokka compiler should be unchanged. This same Zokka compiler version should work even with new versions of the package server coming in a few months.

Note before using the custom package manager

If you have used any version of Zokka in the past, make sure that you have the latest version of Zokka. You should also delete your $ELM_HOME/0.19.1/zokka and $ELM_HOME/0.19.1/zokka-cache-0.191.0 folders if they exist and were created by an older version of Zokka.

If you are using npx for Zokka, make sure that the npm version is 0.191.0-beta.0 or later.

Again: Do not use this for any critical packages! All packages will be deleted sometime past Jun 15th, 2024 at midnight UTC time. A deployment of a new server version will take place then.

After Jun 15th, 2024, I will likely institute pretty strict upload, download, and storage limits! Roughly speaking I’m imagining something like a per-user limit of a total of 100 packages (different versions of a package each count separately), a total of 10 MB in storage of ZIP files, and some download limit of around 1 GB per month and an upload limit of around 100 MB per month. These are very conservative and I will likely raise caps over time, but since now we need to store all the data associated with a package, I don’t want to wake up one morning to a high server bill. (Note: if you’re willing to pay, we can talk about setting up a custom server for your team, or you can set up a custom server yourself using the binaries that are published on the project GitHub page).

Quick start steps

1. First thing we’ll do is set up everything on https://zokka-custom-repos.igneousworkshop.com so that we have a working remote repository. You’ll need to make an account the first time you navigate to the page. Note that this part will likely change in the coming weeks to instead use Auth0 which provides things such as password recovery, OAuth, etc. (I also would prefer not to store things like email addresses on my own servers if at all possible).

2. Then you’ll need to create a repository. Click the “Create new repository button” to create a repository. Give it both a human readable name (one which can contain any Unicode characters) and a URL safe name (characters will be automatically percent encoded if necessary) which will form part of the URL that the Zokka compiler uses to hit this repository.

   Note that right now neither name is actually used anywhere. For now we just use the database ID, but that will change soon. The URL safe name, since it eventually is meant to form part of the URL used by the Zokka compiler, must be globally unique (if it is not the frontend will not have a good time at the moment. That will be fixed.)

3. Once you’ve created the repository copy the JSON fragment given in “How to use this repository” into

   $ELM_HOME/0.19.1/zokka/custom-package-repository-config.json

   If this file does not yet exist, run npx zokka make in any project you have to automatically generate this file. You can fill in $SOME_LOCAL_NAME with whatever name you want. This is the name that ultimately you will use for zokka publish. We’ll use my-custom-repo for now. We’ll fill in $FULL_TOKEN_VALUE in just a bit.

4. Create an API token by clicking “API Auth Tokens” and then clicking “Create new auth token.” For our purposes, we’ll want to create a token with package publish permissions.

5. Copy the token value that is displayed into your $ELM_HOME/0.19.1/zokka/custom-package-repository-config.json file. Now fill in $FULL_TOKEN_VALUE with the token value you got.

6. You can now publish to and read from your custom repository! You can run git clone https://github.com/Zokka-Dev/zokka-example-package for an example package which you can publish right away to your custom repository via

   cd zokka-example-package
   npx zokka publish my-custom-repo
   

   (or whatever name you used for $SOME_LOCAL_NAME instead of my-custom-repo).

   Then in whatever other project you have you can run npx zokka install example-zokka-author/example-zokka-package and use the amazing new Example.someConstant value!

Here are screenshots illustrating each of these steps:

Step 1: Create your user account and password

After logging in you’ll likely get this page.

It’s an ugly hack for the moment, but just refresh and the error page should disappear.

Step 2: Set up your repository by clicking the “Create new repository” button.

Click the “create new repository” button. Give whatever names you want for the repositories, keeping in mind that the URL safe name must be unique across all users. The frontend isn’t very graceful about handling this failure at the moment.

which leads to

Step 3: Copy the JSON fragment displayed into your configuration file

Expand “How to use this repository” to get the JSON fragment. Fill in $SOME_LOCAL_NAME with whatever you want (for the purposes of these instructions we’ll use my-custom-repo).

After this your configuration file at $ELM_HOME/0.19.1/zokka/custom-package-repository-config.json (where $ELM_HOME is usually ~/.elm) should look like the following. We’ll fill in $FULL_TOKEN_VALUE in just a bit.

{
    "repositories": [
        {
            "repository-type": "package-server-with-standard-elm-v0.19-package-server-api",
            "repository-url": "https://package.elm-lang.org",
            "repository-local-name": "standard-elm-repository"
        },
        {
            "repository-type": "package-server-with-standard-elm-v0.19-package-server-api",
            "repository-url": "https://package-server.zokka-lang.com",
            "repository-local-name": "standard-zokka-repository"
        },
        {
            "repository-type": "package-server-with-personal-zokka-repo-v1.0-package-server-api",
            "repository-url": "http://zokka-custom-repos.igneousworkshop.com/api/2",
            "repository-auth-token": "$FULL_TOKEN_VALUE",
            "repository-local-name": "my-custom-repo"
        }
    ],
    "single-package-locations": []
}

Step 4: Create the API auth token

This is when we create the token we’ll use for $FULL_TOKEN_VALUE. Click on “API Auth Tokens” and then click “Create new auth token”

You’ll want to create a token with both read and publish rights for our purposes.

Make sure to note that token value (it won’t appear again!) and replace $FULL_TOKEN_VALUE in the configuration file with the token value.

Now you should see the following:

Step 6: Publishing your first package

The following steps will publish your first private package

git clone https://github.com/Zokka-Dev/zokka-example-package/
cd zokka-example-package
# Verify the package compiles
npx zokka make
# If you didn't use my-custom-repo for your repository-local-name use whatever name you chose insetad
npx zokka publish my-custom-repo

You can then use it from some other package

# In some fresh new directory
npx zokka init
npx zokka install example-zokka-author/example-zokka-package

Then make a file src/Main.elm that looks like

module Main exposing (..)

import Example
import Html
import String

main = Html.text (String.fromInt Example.someConstant)

and go ahead and run

npx zokka make src/Main.elm

and you should see usage of your brand-new custom package in your own project!

[Magnetize2072]

Looks like this might solve a lot of headaches for people.

That being said, I would still caution against deploying your own package server for Zokka, even if it’s supposed for private use only. It might be a great solution, but it also might just put you on the hook for a single point of failure.

At the end of the day, I think the three needs people have are

• install packages as they normally do, from https://package.elm-lang.org/
• install packages from a public or private git repo that they have access to (could be on Github, any other platform, or even their own Git server)
• install packages from a filepath on their system

All three of those needs could be met without you having to pay for or operate a package manager - and it puts the bill on individuals (and Github)

[yosteden]

All three of those were previously possible with Zokka without custom package repos! You can use Zokka’s single-package-locations for this to install any Elm package available as a ZIP file (GitHub - Zokka-Dev/zokka-compiler: Fork of compiler for Elm, a functional language for reliable webapps.). The only note is that local packages from a filepath in the system require a local webserver to be running (because Zokka doesn’t yet support file URL schemes), so you’ll just first need to run python -m http.server in the directory with the package.

The reason for developing these repos was because publishing private packages is still quite a pain this way (especially if you have multiple packages depending on each other). It’s really nice to have the usual checks that elm bump gives you, nice documentation viewing, easy handling of transitive dependencies, etc. The custom package repository is therefore a strictly additive thing on top of custom single package locations and for truly critical use cases I would expect people to be running their own instance of the repository (at some point I’ll make things a single run.sh runner script that anyone can run on their own machines, but for now, the binary is published as part of the usual CI process Release 0.191.0-beta.0 · Zokka-Dev/zokka-compiler · GitHub or you can directly build it from source).

[yosteden]

Having a public instance of the repo is to let people easily play with the custom package repo to see whether it fits their needs.

[Magnetize2072]

All three of those were previously possible with Zokka without custom package repos!

Oh awesome, I wasn’t aware of that.

The custom package repository is therefore a strictly additive thing on top of custom single package locations and for truly critical use cases I would expect people to be running their own instance of the repository …

Ah yeah, that makes a lot of sense.

Having a public instance of the repo is to let people easily play with the custom package repo to see whether it fits their needs.

I gotcha. If it were me, I would make it so that it wipes everything after 24h, so folks can try it out and make sure it works for their use cases, but will quickly be incentivized to set up their own.

[andre-dietrich]

Awesome project and awesome work … I will definitely try this out … Thanks for the effort

[yosteden]

As mentioned, since it is now June 15th, I’ve wiped all the packages from the package server in preparation for some changes in the pipeline. Now the website is stable and that won’t happen again!

Also in the middle of moving out the custom repository server to a separate repo so that people can easily download and deploy if they would like.