Sam f6fdc1ebe8 FEATURE: flexible crawler detection
You can use the crawler user agents site setting to amend what user agents
are considered crawlers based on a string match in the user agent

Also improves performance of crawler detection slightly
2017-09-29 12:31:50 +10:00
..
2017-08-10 22:54:52 +01:00
2017-07-28 10:20:09 +09:00
2017-09-15 17:33:31 +02:00
2017-07-29 13:12:45 +02:00
2017-07-28 10:20:09 +09:00
2017-07-28 10:20:09 +09:00
2016-02-25 12:13:59 +10:00
2017-09-08 14:07:24 +08:00
2017-07-28 10:20:09 +09:00
2016-07-15 11:36:06 +08:00
2017-07-28 10:20:09 +09:00
2017-07-28 10:20:09 +09:00
2017-07-03 15:26:57 -04:00
2017-07-28 10:20:09 +09:00
2017-07-28 10:20:09 +09:00
2017-07-28 10:20:09 +09:00
2017-09-28 15:27:46 -04:00
2017-07-28 10:20:09 +09:00