Robin Ward
|
b81e7be9a1
|
FEATURE: Rate limit how often we'll crawl a destination IP
|
2017-05-23 15:03:04 -04:00 |
|
Robin Ward
|
36e477750c
|
FIX: Use same code path for downloading images
|
2017-05-23 14:51:30 -04:00 |
|
Robin Ward
|
e5e7a15a85
|
SECURITY: Never crawl by IP
|
2017-05-23 13:07:18 -04:00 |
|
Robin Ward
|
93a5fc62bf
|
FEATURE: A site setting to prevent crawling on private IP blocks
|
2017-05-23 11:56:06 -04:00 |
|
Robin Ward
|
b8d78b33c6
|
FIX: Other content types like images are fine
|
2017-05-22 16:51:37 -04:00 |
|
Robin Ward
|
b23fc2bf84
|
Helper to find the final destination for a URL
|
2017-05-22 15:52:41 -04:00 |