Commit Graph

12 Commits

Author SHA1 Message Date
Jay Pfaffman 3c72cbc5de
FIX: Google groups import changed login URL (#9432)
I'm not clear why changing only the `wait_for_url` address was necessary and not also the `get` a few lines above, but this change seems to work for me on both literatecomputing.com Groups and a public group.
2020-04-15 16:45:14 -04:00
Gerhard Schlager e825f47daa DEV: Better handling of incremental scrapes for Google Groups 2020-03-14 00:00:36 +01:00
Gerhard Schlager d7ccb58559 FIX: Google Groups scraper failed to login 2020-03-02 17:24:48 +01:00
Gerhard Schlager c218036107 FIX: Make Google Groups scraper work for G Suite users 2019-11-28 02:09:51 +01:00
Gerhard Schlager ed1e5ef6cc FIX: By default, don't abort Google Groups crawling on error 2019-09-18 18:14:04 +02:00
Gerhard Schlager ab96239f2a FIX: Google Groups crawler failed to login
Trying to automate the login into a Google account is quite hard. This makes the crawler use the content of a cookies.txt file instead. It also removes a couple of deprecation warnings and adds some color to the output.
2019-09-18 13:09:20 +02:00
Gerhard Schlager f0fea5991f FIX: Latest Selenium gem broke Google Groups import script
Selenium uses Keep-Alive since version 3.141, so the net-http-persistent gem shouldn't be needed anymore.
2019-07-10 09:45:33 +02:00
Sam Saffron 30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Gerhard Schlager 453ba2da7b Make Google Groups scraper work with latest chromedriver 2019-03-25 16:11:22 +01:00
Gerhard Schlager 2349ba3bc4 Improve Google Groups scraper
* Better error detection during login phase
* Experimental support for 2FA and SMS codes
* Detect missing permissions to scrape email addresses
2019-03-24 23:15:13 +01:00
Gerhard Schlager d6f89a85ef Make Rubocop happy 2018-10-31 01:30:14 +01:00
Gerhard Schlager 65db9326b4 FEATURE: Add download script for Google Groups 2018-10-31 01:12:05 +01:00