4 Commits

Author	SHA1	Message	Date
Robin Ward	e5e7a15a85	SECURITY: Never crawl by IP	2017-05-23 13:07:18 -04:00
Robin Ward	93a5fc62bf	FEATURE: A site setting to prevent crawling on private IP blocks	2017-05-23 11:56:06 -04:00
Robin Ward	b8d78b33c6	FIX: Other content types like images are fine	2017-05-22 16:51:37 -04:00
Robin Ward	b23fc2bf84	Helper to find the final destination for a URL	2017-05-22 15:52:41 -04:00