Discource-C/discourse
1
0
Fork 0
mirror of https://github.com/discourse/discourse.git synced 2025-02-21 03:19:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
27,145 Commits 208 Branches 505 Tags
Commit Graph

3 Commits

Author SHA1 Message Date
Neil Lalonde
2bd44bbf13 WebCrawlerRequest.clear_cache needs to clear user agent list too 2018-03-27 15:11:48 -04:00
Neil Lalonde
4d12ff2e8a when writing cache, remove elements from the user agents list. also return a message and content type when blocking a crawler. 2018-03-27 13:44:14 -04:00
Neil Lalonde
ced7e9a691 FEATURE: control which web crawlers can access using a whitelist or blacklist 2018-03-22 15:41:02 -04:00