30990006a9
DEV: enable frozen string literal on all files
...
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.
Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
baa72d18f8
FIX: simplify so we ban all auth paths
...
previously plugins that have auth paths were not disallowed and robots
tend to call them
2018-08-16 19:16:47 +10:00
030e322a39
FEATURE: block top level /my/ routes
2018-06-12 19:47:45 +10:00
3d7dbdedc0
FEATURE: An API to help sites build robots.txt files programatically
...
This is mainly useful for subfolder sites, who need to expose their
robots.txt contents to a parent site.
2018-04-16 15:43:20 -04:00
df7970a6f6
prefix the robots.txt rules with the directory when using subfolder
2018-04-11 22:05:02 +02:00
3a7b696703
FEATURE: allow for setting crawl delay per user agent
...
Also moved to default crawl delay bing so no more than a req every 5 seconds is allowed
New site settings:
"slow_down_crawler_user_agents" - list of crawlers that will be slowed down
"slow_down_crawler_rate" - how many seconds to wait between requests
Not enforced server side yet
2018-04-06 10:15:23 +10:00
ced7e9a691
FEATURE: control which web crawlers can access using a whitelist or blacklist
2018-03-22 15:41:02 -04:00
77d4c4d8dc
Fix all the errors to get our tests green on Rails 5.1.
2017-09-25 13:48:58 +08:00
d75cc67d86
FIX: robots.txt should be accessible even when login is required
2015-10-15 11:42:41 +02:00
e5888cf090
PERF: avoid preloading json in cases where it is not needed
...
(uploads / avatars / non GET requests)
2015-05-20 17:12:16 +10:00
a86b35c873
Remove the access_password site setting
2013-06-25 15:05:25 -04:00
d2596c3c4c
Remove unusued site_settings, show checkbox in UI for boolean values, remove restrict_access
...
boolean to avoid locking yourself out by setting access_password to empty string. Minor
UI tweaks.
2013-03-01 14:27:41 -05:00
cafc75b238
remove trailing whitespaces ❤️
2013-02-26 07:31:35 +03:00
1c12c91d0c
forgot to skip a filter
2013-02-11 17:14:36 +11:00
c50a9e4d01
added support for disabling indexing by google using SiteSetting.allow_index_in_robots_txt = false
2013-02-11 11:02:57 +11:00
Sam Saffron
Robin Ward
Régis Hanol
Neil Lalonde
Guo Xiang Tan
Gosha Arinich