Elm connection timeout while downloading packages

[mihkel]

I’ve been using Elm on DigitalOcean for 4 years and suddenly I get erros while downloading some packages.

I’ve talked with DigitalOcean support but they are telling me to stop trying downloading json and that I should download packages directly instead - like I had any control over how Elm downloads packages =D

Exapmles of two errors:

Dec 11 05:36:17 e[34m│e[0m -- PROBLEM LOADING PACKAGE LIST ------------------------------------------------

Dec 11 05:36:17 e[34m│e[0m 
Dec 11 05:36:17 e[34m│e[0m I need the list of published packages to verify your dependencies, so I tried to
Dec 11 05:36:17 e[34m│e[0m fetch:
Dec 11 05:36:17 e[34m│e[0m 
Dec 11 05:36:17 e[34m│e[0m   https://package.elm-lang.org/all-packages
Dec 11 05:36:17 e[34m│e[0m 
Dec 11 05:36:17 e[34m│e[0m But my HTTP library is giving me the following error message:
Dec 11 05:36:17 e[34m│e[0m 
Dec 11 05:36:17 e[34m│e[0m   ConnectionTimeout
Dec 11 05:36:17 e[34m│e[0m 
Dec 11 05:36:17 e[34m│e[0m Are you somewhere with a slow internet connection? Or no internet? Does the link
Dec 11 05:36:17 e[34m│e[0m I am trying to fetch work in your browser? Maybe the site is down? Does your
Dec 11 05:36:17 e[34m│e[0m internet connection have a firewall that blocks certain domains? It is usually
Dec 11 05:36:17 e[34m│e[0m something like that!
Dec 11 05:36:17 e[34m│e[0m 
Dec 11 05:36:17 e[34m│e[0m 

or

Dec 11 04:21:28  e[34m│e[0m Verifying dependencies (19/19)-- PROBLEM DOWNLOADING PACKAGE -------------------------------------------------
Dec 11 04:21:28  e[34m│e[0m 
Dec 11 04:21:28  e[34m│e[0m I need to find the latest download link for NoRedInk/elm-json-decode-pipeline
Dec 11 04:21:28  e[34m│e[0m 1.0.0, so I tried to fetch:
Dec 11 04:21:28  e[34m│e[0m 
Dec 11 04:21:28  e[34m│e[0m     https://package.elm-lang.org/packages/NoRedInk/elm-json-decode-pipeline/1.0.0/endpoint.json
Dec 11 04:21:28  e[34m│e[0m 
Dec 11 04:21:28  e[34m│e[0m But my HTTP library is giving me the following error message:
Dec 11 04:21:28  e[34m│e[0m 
Dec 11 04:21:28  e[34m│e[0m     ConnectionTimeout

[evancz]

There are basic instructions of how to avoid this in CI here:

Towards more reliable CI

Context It is possible to make CI much more reliable by caching the ~/.elm directory. The directory contains: Immutable files. All package content is hashed on publication, so downloading requires that the content exactly matches that hash. They will be the same for all downloads. Deterministic artifacts. Certain build artifacts can be pre-compiled for each package, so those are also stored in ~/.elm. These are deterministic based on the aforementioned immutable files. Caching these files mea…

I’ll hopefully know more about the cause in the coming days. Have to dig through logs to see if there’s some patterns in the requests.

Aside: I noticed some bursts of downloading lots of docs.json files in the same second. The pattern suggested “downloading docs for my project” rather than “scraping the whole catalog”, so if any editor tool is downloading lots and lots of documentation without caching anything, please figure out a way to avoid that! I am pretty certain that is not the root cause. Just figured I’d mention it in case it’s something easily fixable.

[rupert]

BTW, there does seem to be some extra flakiness with githubs service today, which is not helping.