medley Fix Hugo Build & Deploy to correctly generate robots.txt

Describe the bug robots.txt is supposed to be automatically generated when production is set to true. This would then overwrite the default robots.txt file -- which disallows indexing. This is not working.

Running locally the robots.txt file is correctly generated. This isn't happening in the github action. Figure out where the workflow is messed up.

Aug 08 '24 11:08 stumbo

Draft PR 239 resolves this issue. Need some final clean up and documentation before converting from draft.

Sep 18 '24 03:09 stumbo

A fix for this was approved and merged. Is there a way of checking whether it was effective? Do the web crawlers now crawl all our pages correctly?

Oct 05 '24 19:10 masinter

There are several online tools that claim to scan websites and identify issues.

From seomator.com

URL	Info	Crawlable	Indexable
/	Google crawl rule: Allow: Bing crawl rule: Allow: Google index rule: none Bing index rule: none	Google Bing	Google Bing

Google Search Console - access limited to Interlisp.org site maintainers - notes the new robots.txt file was successfully fetched on 10/3/2024.

https://interlisp.org/robots.txt	10/3/24, 2:07 AM	check_circle_outline Fetched	49 bytes

Oct 05 '24 22:10 stumbo

Google Page Indexing shows no pages being blocked by robots.txt.

Oct 24 '24 11:10 stumbo