30990006a9
DEV: enable frozen string literal on all files
...
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.
Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
3dc4ab905c
DEV: annotate models
...
(also looks like renaming of bigint(8) -> bigint in annotate
which seems fine)
2019-05-03 08:34:12 +10:00
214dac05de
Update annotations.
2018-07-16 14:19:07 +08:00
9ca6ebe8fe
FEATURE: enforce tagging on categories
2018-04-11 07:15:24 +05:30
2bd44bbf13
WebCrawlerRequest.clear_cache needs to clear user agent list too
2018-03-27 15:11:48 -04:00
4d12ff2e8a
when writing cache, remove elements from the user agents list. also return a message and content type when blocking a crawler.
2018-03-27 13:44:14 -04:00
ced7e9a691
FEATURE: control which web crawlers can access using a whitelist or blacklist
2018-03-22 15:41:02 -04:00
Sam Saffron
Guo Xiang Tan
Arpit Jalan
Neil Lalonde