Robots and the Elm SPA router

Learn
1

I have a request to add <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> on certain pages of the Elm app.

This got me thinking about crawling. Does any of you have more insight into what is the best way to handle this.

Right now, the navigation is done with a custom link that has an onWithOptions handler that disables the default behavior and generate a routing message that then writes the url using Navigation.newUrl.

Would adding a port call to JS to update the head be sufficient OR will this have no effect on the crawlers?

Any kind of insight into this is appreciated.

1 Like
2

Can you add it in robots.txt on the site root? You can have wildcards in there and any bot that will respect the meta tags should respect the robots.txt file. It will also work better in cases where the crawler doesn’t execute JS on the page.

2 Likes