I’d like to self-host the fonts for elm-lang.org and package.elm-lang.org.
Right now it goes to fonts.googleapis.com but I’d like to avoid that. Articles like this argue that there are performance benefits, but my personal motivation was concern that Google Fonts could be used as a tracking tool by Google. Maybe it is not, but hey, it may be faster to self-host anyway.
Problem
I was self-hosting fonts when the package website got updated recently, but it was leading to strange visual artifacts in some browsers, like in Chrome on Windows:
Why do the files vary by browser? I assume it is some trick to reduce asset size, but what is the nature of that trick?
What is the full range of User-Agent fields that produce different files? Is it custom for each IE version or something wild?
What are our options for self-hosting in terms of asset sizes? What kind of savings can we get without detecting User-Agent?
This would be great for the Elm websites, but I suspect it’d be useful for lots of people in the community! So please blog about it and share in public spots if you end up looking into this!
My gut feeling from working with fonts for a while is that the artifacts in your screenshot are from using a file without hinting tables. Hinting tables basically tell the rendering engine what to do if a stroke would round to being wider than designed because of pixel boundaries (the linked article has a good visual.) Without this table, some rendering engines will color too many pixels and you’ll get over-stroked results, especially in horizontal and vertical lines. Benefit: better rendering. Drawback: they’re really big, almost doubling the font size in some cases.
macOS seems to be better than Windows about this, I’m not sure why.
Why do the files vary by browser? I assume it is some trick to reduce asset size, but what is the nature of that trick?
One reason is file compatibility. WOFF and WOFF2 are pretty much the standards now but they’re not uniformly supported. For example, .woff2 compresses a bit better than .woff but isn’t available in IE11.
There’s also the hinting issues mentioned above.
What is the full range of User-Agent fields that produce different files? Is it custom for each IE version or something wild?
I don’t have an answer for this as phrased but I have had good results from serving .woff files if I care about IE and .woff2 if I do not. I usually include hinting but remove tables which a site is unlikely to require (I usually remove CJK, Cyrillic, etc, but that may not be safe for package.elm-lang.org.)
What are our options for self-hosting in terms of asset sizes? What kind of savings can we get without detecting User-Agent ?
I’ve had good results by finding the original font files, either from the foundry or individual designer, and hosting them myself. As mentioned above, I usually remove lots of character tables I don’t need. fonttools is a great Swiss army knife here.
So please blog about it and share in public spots if you end up looking into this!
I’ve got a draft blog post on some of this, actually. I just need to publish.
It’s not just browser make and version either. Font hinting involves extra instructions in the font file which are then used to ensure the font is displayed the best - especially on low resolution screens or for really small sizes. Font hinting is used by Windows, but not MacOS so depending which you use to get your Google Fonts if using the browser (even if using Chrome on each platform), you’ll get font files with hints or without hints. If you then download the Windows version and serve those locally, you’ll actually make your MacOS users suffer with larger font files full of hints that will not be used, and if you do the opposite you’ll make Windows users potentially suffer with worse fonts as they will get no hints, when previously they did.
As fonts seem to have been manually downloaded with a browser from MacOS X, this likely confirms @brian hypothesis.
Can folks help confirm the font hinting theory by downloading with different user agents?
According to this theory, we should see only two possibilities. One for Windows users and one for macOS users. Is that true? And what happens on Linux?
I started a thread in #core-coordination on Slack to share info and fill in this table!
Note: People on Linux may need to replace shasum with sha1sum. They do the same thing! Not sure what needs to be done on Windows. Maybe just share a (User-Agent, OS version, browser version) tuple on Slack so other people can run this.
Running with no user-agent gives 838543040694e1bbe8d55a9119d7e0332d2e9a8
Trying different versions of Firefox:
v72.0 gives 2bb16ec6d4840c6a17c86d10049a1bb660579d8 v44.0 gives 2bb16ec6d4840c6a17c86d10049a1bb660579d8 v43.0 through v40.0 gives bda2936929cdc8b6c849c47f0e5b7268e48fbe24 v39.0 through v3.6.0 gives abd45dc9857779e8cd8f12ee9390862aa7e8e36e
And before that it’s 838543040694e1bbe8d55a9119d7e0332d2e9a8d. Same as with no user-agent at all.
So it is looking like modern browsers are cutting font hinting on macOS, and other stuff appears to be happening on pre-woff2 browsers. I assume serving other file types, and maybe similar things with font hinting.
Great information so far! Thanks everyone who is sharing info!
#!/bin/bash
DIR="$1"
FAMILY="$2"
mkdir -p $DIR
function download {
CSS="$DIR/$1.css"
curl -s -o "$CSS" -H "User-Agent: $2" "https://fonts.googleapis.com/css?family=$FAMILY"
for WOFF2_URL in $(sed -n 's!.*\(https.*\.woff2\).*!\1!p' $CSS)
do
WOFF2="$DIR/$(echo $WOFF2_URL | grep -o '[a-zA-Z0-9_-]*\.woff2')"
if [ ! -f $WOFF2 ]; then
curl -s -o "$WOFF2" "$WOFF2_URL"
fi
done
sed -i "" "s!https.*/\([a-zA-Z0-9_-]*\.woff2\)!/$DIR/\1!g" $CSS
}
download "_mac" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:73.0) Gecko/20100101 Firefox/73.0"
download "_win" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36"
download "_lin" "Mozilla/5.0 (X11; Linux x86_64; rv:74.0) Gecko/20100101 Firefox/74.0"
It’s not strictly necessary to serve _lin.css and _win.css but it’s easy to change. It also seems pretty easy to add an _old.css if you want to support that as well.
Thanks to everyone for helping sleuth and gather data on this!
This service might be handy if you want to directly download all .eot , .woff , .woff2 , .svg , .ttf files of a Google font (normally your User-Agent would determine the best format at Google’s CSS API). Furthermore it provides charset customization and CSS snippets, hence getting your fonts ready for local hosting should be finally a breeze.
@roovo, I found that when I realized I was getting weird rendering on Windows, but the font I need, roughly this, produces very heavy CSS. Nothing like what Google is serving using the User Agent detection:
/* source-sans-pro-regular - latin-ext_latin_cyrillic-ext_cyrillic */
@font-face {
font-family: 'Source Sans Pro';
font-style: normal;
font-weight: 400;
src: url('../fonts/source-sans-pro-v13-latin-ext_latin_cyrillic-ext_cyrillic-regular.eot'); /* IE9 Compat Modes */
src: local('Source Sans Pro Regular'), local('SourceSansPro-Regular'),
url('../fonts/source-sans-pro-v13-latin-ext_latin_cyrillic-ext_cyrillic-regular.eot?#iefix') format('embedded-opentype'), /* IE6-IE8 */
url('../fonts/source-sans-pro-v13-latin-ext_latin_cyrillic-ext_cyrillic-regular.woff2') format('woff2'), /* Super Modern Browsers */
url('../fonts/source-sans-pro-v13-latin-ext_latin_cyrillic-ext_cyrillic-regular.woff') format('woff'), /* Modern Browsers */
url('../fonts/source-sans-pro-v13-latin-ext_latin_cyrillic-ext_cyrillic-regular.ttf') format('truetype'), /* Safari, Android, iOS */
url('../fonts/source-sans-pro-v13-latin-ext_latin_cyrillic-ext_cyrillic-regular.svg#SourceSansPro') format('svg'); /* Legacy iOS */
}
No subsetting, no differentiation for font hinting on Mac. It doesn’t seem like it’s close to equivalent for perf. That’s part of why I am asking for help creating a better version of this!
Thanks everyone for the help! Self-hosted fonts are live on package.elm-lang.org as of about 24 hours ago, and it looks like it was a smooth transition!
