Arpit Jalan
|
9ca6ebe8fe
|
FEATURE: enforce tagging on categories
|
2018-04-11 07:15:24 +05:30 |
Neil Lalonde
|
2bd44bbf13
|
WebCrawlerRequest.clear_cache needs to clear user agent list too
|
2018-03-27 15:11:48 -04:00 |
|
Neil Lalonde
|
4d12ff2e8a
|
when writing cache, remove elements from the user agents list. also return a message and content type when blocking a crawler.
|
2018-03-27 13:44:14 -04:00 |
|
Neil Lalonde
|
ced7e9a691
|
FEATURE: control which web crawlers can access using a whitelist or blacklist
|
2018-03-22 15:41:02 -04:00 |