Help with self-hosting fonts on *.elm-lang.org

I’d like to self-host the fonts for elm-lang.org and package.elm-lang.org.

Right now it goes to fonts.googleapis.com but I’d like to avoid that. Articles like this argue that there are performance benefits, but my personal motivation was concern that Google Fonts could be used as a tracking tool by Google. Maybe it is not, but hey, it may be faster to self-host anyway.

Problem

I was self-hosting fonts when the package website got updated recently, but it was leading to strange visual artifacts in some browsers, like in Chrome on Windows:

Font weirdness in Chrome on Windows

So I reverted to using fonts.googleapis.com

My approach was to download the file given by https://fonts.googleapis.com/css?family=Source+Sans+Pro:400,700,400italic,700italic|Source+Code+Pro and then download the subsequent links. It appears that the initial file varies by browser, so people in Chrome actually needed different font files for some reason.

Request

Can someone figure out:

  1. Why do the files vary by browser? I assume it is some trick to reduce asset size, but what is the nature of that trick?

  2. What is the full range of User-Agent fields that produce different files? Is it custom for each IE version or something wild?

  3. What are our options for self-hosting in terms of asset sizes? What kind of savings can we get without detecting User-Agent?

This would be great for the Elm websites, but I suspect it’d be useful for lots of people in the community! So please blog about it and share in public spots if you end up looking into this!

6 Likes

My gut feeling from working with fonts for a while is that the artifacts in your screenshot are from using a file without hinting tables. Hinting tables basically tell the rendering engine what to do if a stroke would round to being wider than designed because of pixel boundaries (the linked article has a good visual.) Without this table, some rendering engines will color too many pixels and you’ll get over-stroked results, especially in horizontal and vertical lines. Benefit: better rendering. Drawback: they’re really big, almost doubling the font size in some cases.

macOS seems to be better than Windows about this, I’m not sure why.

  1. Why do the files vary by browser? I assume it is some trick to reduce asset size, but what is the nature of that trick?

One reason is file compatibility. WOFF and WOFF2 are pretty much the standards now but they’re not uniformly supported. For example, .woff2 compresses a bit better than .woff but isn’t available in IE11.

There’s also the hinting issues mentioned above.

  1. What is the full range of User-Agent fields that produce different files? Is it custom for each IE version or something wild?

I don’t have an answer for this as phrased but I have had good results from serving .woff files if I care about IE and .woff2 if I do not. I usually include hinting but remove tables which a site is unlikely to require (I usually remove CJK, Cyrillic, etc, but that may not be safe for package.elm-lang.org.)

  1. What are our options for self-hosting in terms of asset sizes? What kind of savings can we get without detecting User-Agent ?

I’ve had good results by finding the original font files, either from the foundry or individual designer, and hosting them myself. As mentioned above, I usually remove lots of character tables I don’t need. fonttools is a great Swiss army knife here.

So please blog about it and share in public spots if you end up looking into this!

I’ve got a draft blog post on some of this, actually. I just need to publish. :laughing:

11 Likes

I have personally used https://www.fontsquirrel.com/ before for finding fonts with open licenses and they also have a generator for making web fonts.

Also from the first article linked in Evan’s post:

It’s not just browser make and version either. Font hinting involves extra instructions in the font file which are then used to ensure the font is displayed the best - especially on low resolution screens or for really small sizes. Font hinting is used by Windows, but not MacOS so depending which you use to get your Google Fonts if using the browser (even if using Chrome on each platform), you’ll get font files with hints or without hints. If you then download the Windows version and serve those locally, you’ll actually make your MacOS users suffer with larger font files full of hints that will not be used, and if you do the opposite you’ll make Windows users potentially suffer with worse fonts as they will get no hints, when previously they did.

As fonts seem to have been manually downloaded with a browser from MacOS X, this likely confirms @brian hypothesis.

1 Like

Can folks help confirm the font hinting theory by downloading with different user agents?

According to this theory, we should see only two possibilities. One for Windows users and one for macOS users. Is that true? And what happens on Linux?

We can use a command like this to test:

curl -s -H "User-Agent: xxxxx" https://fonts.googleapis.com/css?family=Source+Sans+Pro:400,700,400italic,700italic|Source+Code+Pro | shasum

To make a table like this:

OS Browser SHA
Mac 10.15.3 Firefox 73.0.1 88c15c55bf28b7b56e1224db243b839e499a1210
Mac 10.15.3 Safari 13.0.5 88c15c55bf28b7b56e1224db243b839e499a1210
Mac 10.15.3 Chrome 80.0.3987.122 88c15c55bf28b7b56e1224db243b839e499a1210
Mac 10.15.2 Firefox 73.0.1 88c15c55bf28b7b56e1224db243b839e499a1210
Mac 10.14.6 Firefox 72.0.2 88c15c55bf28b7b56e1224db243b839e499a1210
Mac 10.13.6 Safari 13.0.5 88c15c55bf28b7b56e1224db243b839e499a1210
Ubuntu 18.04 Brave 1.4.95 2bb16ec6d4840c6a17c86d10049a1bb660579d8b
Ubuntu 18.04 Chrome 80.0.3987.122 2bb16ec6d4840c6a17c86d10049a1bb660579d8b
Debian 10.3 Chromium 79.0.3945.130 2bb16ec6d4840c6a17c86d10049a1bb660579d8b
Debian 10.3 Firefox 73.0.1 2bb16ec6d4840c6a17c86d10049a1bb660579d8b
Debian 10.3 Firefox (dev) 74.0b9 2bb16ec6d4840c6a17c86d10049a1bb660579d8b
Windows 10 Chrome 80.0.3987.122 2bb16ec6d4840c6a17c86d10049a1bb660579d8b
Windows 10 Firefox 73.0.1 2bb16ec6d4840c6a17c86d10049a1bb660579d8b
Windows 10 IE 11.1304.17134.0 abd45dc9857779e8cd8f12ee9390862aa7e8e36e
Windows 10 Edge 42.17134.1098.0 2bb16ec6d4840c6a17c86d10049a1bb660579d8b
Windows Server 2019 Chrome 80.0.3987.122 2bb16ec6d4840c6a17c86d10049a1bb660579d8b

I started a thread in #core-coordination on Slack to share info and fill in this table!

Note: People on Linux may need to replace shasum with sha1sum. They do the same thing! Not sure what needs to be done on Windows. Maybe just share a (User-Agent, OS version, browser version) tuple on Slack so other people can run this.

Recording some information from @drathier here:

Running with no user-agent gives 838543040694e1bbe8d55a9119d7e0332d2e9a8

Trying different versions of Firefox:

v72.0 gives 2bb16ec6d4840c6a17c86d10049a1bb660579d8
v44.0 gives 2bb16ec6d4840c6a17c86d10049a1bb660579d8
v43.0 through v40.0 gives bda2936929cdc8b6c849c47f0e5b7268e48fbe24
v39.0 through v3.6.0 gives abd45dc9857779e8cd8f12ee9390862aa7e8e36e

And before that it’s 838543040694e1bbe8d55a9119d7e0332d2e9a8d. Same as with no user-agent at all.

So it is looking like modern browsers are cutting font hinting on macOS, and other stuff appears to be happening on pre-woff2 browsers. I assume serving other file types, and maybe similar things with font hinting.

Great information so far! Thanks everyone who is sharing info!

That sounds accurate - if I look into the file returned for Internet Explorer it only has a small number of format('woff') fonts like

@font-face {
  font-family: 'Source Code Pro';
  font-style: normal;
  font-weight: 400;
  src: local('Source Code Pro Regular'), local('SourceCodePro-Regular'), url(https://fonts.gstatic.com/s/sourcecodepro/v11/HI_SiYsKILxRpg3hIP6sJ7fM7PqlPevQ.woff) format('woff');
}

instead of a much larger number of format('woff2') ones that are also split by Unicode range, like

/* latin */
@font-face {
  font-family: 'Source Code Pro';
  font-style: normal;
  font-weight: 400;
  src: local('Source Code Pro Regular'), local('SourceCodePro-Regular'), url(https://fonts.gstatic.com/s/sourcecodepro/v11/HI_SiYsKILxRpg3hIP6sJ7fM7PqlPevW.woff2) format('woff2');
  unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+2000-206F, U+2074, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD;
}

Based on the information here, I created a script that should download all the files needed for modern browsers. You would run it like this:

./get-fonts.sh "fonts" "Source+Sans+Pro|Source+Code+Pro"

The file itself is defined as follows:

#!/bin/bash

DIR="$1"
FAMILY="$2"

mkdir -p $DIR

function download {
	CSS="$DIR/$1.css"
	curl -s -o "$CSS" -H "User-Agent: $2" "https://fonts.googleapis.com/css?family=$FAMILY"

	for WOFF2_URL in $(sed -n 's!.*\(https.*\.woff2\).*!\1!p' $CSS)
	do
		WOFF2="$DIR/$(echo $WOFF2_URL | grep -o '[a-zA-Z0-9_-]*\.woff2')"
		if [ ! -f $WOFF2 ]; then
			curl -s -o "$WOFF2" "$WOFF2_URL"
		fi
	done

	sed -i "" "s!https.*/\([a-zA-Z0-9_-]*\.woff2\)!/$DIR/\1!g" $CSS
}

download "_mac" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:73.0) Gecko/20100101 Firefox/73.0"
download "_win" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36"
download "_lin" "Mozilla/5.0 (X11; Linux x86_64; rv:74.0) Gecko/20100101 Firefox/74.0"

It’s not strictly necessary to serve _lin.css and _win.css but it’s easy to change. It also seems pretty easy to add an _old.css if you want to support that as well.

Thanks to everyone for helping sleuth and gather data on this!

1 Like

This looks like it might be relevant/useful: google-webfonts-helper

A Hassle-Free Way to Self-Host Google Fonts

This service might be handy if you want to directly download all .eot , .woff , .woff2 , .svg , .ttf files of a Google font (normally your User-Agent would determine the best format at Google’s CSS API). Furthermore it provides charset customization and CSS snippets, hence getting your fonts ready for local hosting should be finally a breeze.

https://google-webfonts-helper.herokuapp.com/fonts/source-sans-pro?subsets=latin

@roovo, I found that when I realized I was getting weird rendering on Windows, but the font I need, roughly this, produces very heavy CSS. Nothing like what Google is serving using the User Agent detection:

/* source-sans-pro-regular - latin-ext_latin_cyrillic-ext_cyrillic */
@font-face {
  font-family: 'Source Sans Pro';
  font-style: normal;
  font-weight: 400;
  src: url('../fonts/source-sans-pro-v13-latin-ext_latin_cyrillic-ext_cyrillic-regular.eot'); /* IE9 Compat Modes */
  src: local('Source Sans Pro Regular'), local('SourceSansPro-Regular'),
       url('../fonts/source-sans-pro-v13-latin-ext_latin_cyrillic-ext_cyrillic-regular.eot?#iefix') format('embedded-opentype'), /* IE6-IE8 */
       url('../fonts/source-sans-pro-v13-latin-ext_latin_cyrillic-ext_cyrillic-regular.woff2') format('woff2'), /* Super Modern Browsers */
       url('../fonts/source-sans-pro-v13-latin-ext_latin_cyrillic-ext_cyrillic-regular.woff') format('woff'), /* Modern Browsers */
       url('../fonts/source-sans-pro-v13-latin-ext_latin_cyrillic-ext_cyrillic-regular.ttf') format('truetype'), /* Safari, Android, iOS */
       url('../fonts/source-sans-pro-v13-latin-ext_latin_cyrillic-ext_cyrillic-regular.svg#SourceSansPro') format('svg'); /* Legacy iOS */
}

No subsetting, no differentiation for font hinting on Mac. It doesn’t seem like it’s close to equivalent for perf. That’s part of why I am asking for help creating a better version of this!

Thanks everyone for the help! Self-hosted fonts are live on package.elm-lang.org as of about 24 hours ago, and it looks like it was a smooth transition!

Commits were add font downloader and serve based on user agent.

5 Likes

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.