The same argument could be made about elm/json. In fact, you cannot compile an Elm program without referencing elm/json as a (indirect) dependency, even though elm/core doesn’t depend on it.
The reason elm/json is such a fundamental package is because flags and ports can automatically generate code to decode/encode Javascript objects to Elm values and vice-versa. However, Javascript objects are infinitely more powerful than JSON. This thread for example asks for the ability to pass Bytes to Javascript, which have no native representation in JSON, but could be represented as a Javascript object (e.g. the native DataView or some TypedArray). Before elm/file existed, we (at work) passed raw Json.Encode.Values from the event handler through the port to process the Javascript File object there.
So, maybe we need to seperate communication with JSON things (like Web APIs) and communication with the host environment (basically the browser, so Javascript) more?