3a7b696703
FEATURE: allow for setting crawl delay per user agent
...
Also moved to default crawl delay bing so no more than a req every 5 seconds is allowed
New site settings:
"slow_down_crawler_user_agents" - list of crawlers that will be slowed down
"slow_down_crawler_rate" - how many seconds to wait between requests
Not enforced server side yet
2018-04-06 10:15:23 +10:00
5e4dd20795
Revert "Prevent robots from indexing uploads"
...
This reverts commit 0fd622e5d1
2018-04-02 21:29:29 +05:30
ced7e9a691
FEATURE: control which web crawlers can access using a whitelist or blacklist
2018-03-22 15:41:02 -04:00
0fd622e5d1
Prevent robots from indexing uploads
...
Although most user uploads are probably harmless, it's possible someone
has (either maliciously or not) uploaded sensitive information. Prevent
robots from indexing the uploads route.
2018-03-09 05:51:55 -06:00
e19ae6c55e
FEATURE: disallow groups from being indexed
2018-03-02 13:38:30 +11:00
0776340b29
SECURITY: Prevent robots from indexing more routes
...
These routes could contain sensitive material and should never be
indexed for content.
2018-02-04 13:24:36 -05:00
77d4c4d8dc
Fix all the errors to get our tests green on Rails 5.1.
2017-09-25 13:48:58 +08:00
14410b71fb
Convert server side paths to use `/u/`
2017-03-30 10:23:24 -04:00
08c14dd689
new: server plugin outlet for indexable robots.txt
2017-02-13 17:31:10 +05:30
ae671355da
FIX: add /tags routes to robots.txt
2017-02-03 11:57:00 -05:00
54645261aa
better disallow search ... this could get ugly
2015-04-02 17:08:00 +11:00
e66c53a4a7
Add /badges to robots.txt for now, we don't have a crawlable view so
...
it's better to exclude it.
2014-10-30 14:32:42 -04:00
8267a451b2
Disallow /users/ in robots.txt
2014-05-23 10:28:26 -04:00
9c4dc9a966
Block browser-update.js in robots.txt. Move noscript block above everything else in application layout.
2014-02-14 15:33:00 -05:00
7ad00f426c
FEATURE REMOVAL: persona login
...
see: https://meta.discourse.org/t/pulling-persona-out-of-discourse-core/12613
2014-02-11 16:56:48 +11:00
88d9f3a786
Disallow auth callbacks in robots.txt
2014-01-14 10:42:22 -05:00
c50a9e4d01
added support for disabling indexing by google using SiteSetting.allow_index_in_robots_txt = false
2013-02-11 11:02:57 +11:00
Sam
Arpit Jalan
Neil Lalonde
Dan Nicholson
Robin Ward
Guo Xiang Tan
Vinoth Kannan