I suspect people will be able to help more if you can share more of your code. For example, is there a way to make a gist of the whole webpack setup? Directory structure? Etc.
Given that you can get the small sizes calling uglifyjs directly, it seems like its is something in webpack or some configuration detail. Or maybe that extra code is getting thrown in by the server code. It’s really hard to pin it down with the information we have so far.
Talking it through on the Elm slack may also be a better fit because the feedback loop on sharing additional information can be faster.