discourse/app
Sam 758e160862
FEATURE: explicitly ban outlier traffic sources in robots.txt (#11553)
Googlebot handles no-index headers very elegantly. It advises to leave as many routes as possible open and uses headers for high fidelity rules regarding indexes.

Discourse adds special `x-robot-tags` noindex headers to users, badges, groups, search and tag routes.

Following up on b52143feff we now have it so Googlebot gets special handling.

Rest of the crawlers get a far more aggressive disallow list to protect against excessive crawling.
2020-12-23 08:51:14 +11:00
..
assets DEV: Add "before-manage-group-tags" plugin outlet (#11560) 2020-12-22 15:35:55 -05:00
controllers FEATURE: explicitly ban outlier traffic sources in robots.txt (#11553) 2020-12-23 08:51:14 +11:00
helpers UX: Use appropriate logo on static pages (#11211) 2020-11-12 10:50:55 -08:00
jobs FIX: Don't enqueue imported users when there're multiple custom fields. (#11559) 2020-12-22 14:28:07 -03:00
mailers FIX: Ensure group SMTP and message builder always uses from address for Reply-To when IMAP is enabled (#11037) 2020-10-28 07:01:58 +10:00
models Revert "FIX: use Redis replica host and port (#11543)" (#11552) 2020-12-22 16:16:50 +11:00
serializers FEATURE: Do not disturb (#11484) 2020-12-18 09:03:51 -06:00
services FIX: handle badge grant for deleted badge (#11558) 2020-12-22 20:24:51 +05:30
views Move CSS link tags above JS in the head 2020-11-17 09:57:30 -05:00