Arpit Jalan
|
b059a0f789
|
extract url escaping to a dedicated class method and improved tests
|
2017-07-29 22:16:51 +05:30 |
|
Arpit Jalan
|
1fe553873c
|
FIX: preserve fragment identifier when escaping url
|
2017-07-29 17:22:45 +05:30 |
Guo Xiang Tan
|
b534778f46
|
FIX: Escape URL before attempting to resolve it.
|
2017-07-18 10:04:24 +09:00 |
Robin Ward
|
db485ae0da
|
FIX: Support for skipping redirects on certain domains (like steam)
|
2017-06-26 15:38:43 -04:00 |
|
Robin Ward
|
009f0921dc
|
FEATURE: Whitelist hosts for internal crawling
|
2017-06-13 12:59:54 -04:00 |
|
Robin Ward
|
a3729b51eb
|
FIX: Always allow the host the forum is hosted on
|
2017-06-12 13:22:51 -04:00 |
|
Robin Ward
|
53b95f009f
|
FIX: If HEAD is not supported, try GET. Also set cookies
|
2017-06-06 13:53:49 -04:00 |
|
Guo Xiang Tan
|
56f98de7b2
|
Use webmock to stub external web requests.
|
2017-05-26 15:19:09 +08:00 |
|
Guo Xiang Tan
|
f8f1548fd4
|
Revert "FIX: Use Excon to do its own stubbing"
This reverts commit 80af54460a.
|
2017-05-26 13:04:25 +08:00 |
|
Robin Ward
|
3b0cbf7013
|
FIX: Always allow downloads from CDN
|
2017-05-23 16:32:54 -04:00 |
|
Robin Ward
|
b81e7be9a1
|
FEATURE: Rate limit how often we'll crawl a destination IP
|
2017-05-23 15:03:04 -04:00 |
|
Robin Ward
|
36e477750c
|
FIX: Use same code path for downloading images
|
2017-05-23 14:51:30 -04:00 |
|
Robin Ward
|
e5e7a15a85
|
SECURITY: Never crawl by IP
|
2017-05-23 13:07:18 -04:00 |
|
Robin Ward
|
93a5fc62bf
|
FEATURE: A site setting to prevent crawling on private IP blocks
|
2017-05-23 11:56:06 -04:00 |
|
Robin Ward
|
80af54460a
|
FIX: Use Excon to do its own stubbing
|
2017-05-22 18:19:20 -04:00 |
|
Robin Ward
|
b51126dd5e
|
FIX: Reset the WebMock after before every test
|
2017-05-22 17:52:31 -04:00 |
|
Robin Ward
|
4c690f7089
|
Use `FinalDestination` to ensure public redirects for onebox
|
2017-05-22 16:42:49 -04:00 |
|
Robin Ward
|
b23fc2bf84
|
Helper to find the final destination for a URL
|
2017-05-22 15:52:41 -04:00 |