Commit Graph

20 Commits

Author SHA1 Message Date
David Taylor 436b3b392b
DEV: Apply syntax_tree formatting to `script/*` 2023-01-09 11:13:22 +00:00
Leonardo Mosquera 3e5faffb0d
DEV: mbox importer improvements (#16557)
* FIX: support specifying parent_category_id in mbox import metadata
* FIX: elide tabs from topic titles
* FIX: optionally fix Mailman from: addresses
* DEV: optionally elide anything up to the last = in email addresses
* Fix Mailmain broken from: detection
2022-04-29 13:24:29 -03:00
Gerhard Schlager 0a88232e87 DEV: Improve mbox import script
* Better documentation of settings
* Add option to exclude trimmed parts of emails (enabled by default) to not revail email addresses
2020-03-14 00:00:36 +01:00
Gerhard Schlager 0e752db411 DEV: Improve mbox import script
* Customizable email subject prefixes to remove "Re" and "Fwd" as well as localized prefixes.
* Configuration option for prefixes like [FOO] or (BAR) which can be replaced with tags during import.
* Bugfix: Import script might have skipped some users due to missing ORDER BY.
2020-03-09 10:26:45 +01:00
Gerhard Schlager 0f3c3bc309 Make import scripts work with frozen strings 2019-05-30 22:22:24 +02:00
Sam Saffron 30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Gerhard Schlager 341836eb42 Fix the rake task and importer instead 2018-10-17 16:48:09 +02:00
Gerhard Schlager ee18d9ace0 FIX: mbox importer and rake task were broken 2018-10-17 16:34:18 +02:00
Gerhard Schlager ac743dab10 Improve mbox import script
* emails weren't sorted in correct order
* better default regex for splitting mbox files
* output Message-ID if email is skipped because it doesn't have a Date
2018-08-23 09:46:28 +02:00
Gerhard Schlager dc32ee5cbf Improvements to mbox import script
* Ignore errors during indexing and show information about the message causing the problem
* Always activate imported users if they aren't staged
2018-03-06 11:32:12 +01:00
Gerhard Schlager bb54eb1192 Improvements to mbox importer
* store time it took to index message in DB (to find performance issues)
* ignore listserv specific files
* better examples for split_regex
* first email in mbox shouldn't contain the split string
* always lock the DB in exclusive mode
* save email within transaction
* messages can be grouped by subject and use original order (for Listserv)
* adds option to index emails without running the import
2018-01-17 12:04:57 +01:00
Yaw Anokwa 77a92e8878 Allow user staging via setting (#5468) 2018-01-04 09:17:35 +01:00
Gerhard Schlager cafe69caac Refactor mbox import script 2017-12-13 22:03:31 +01:00
Arpit Jalan 3190c13c22 import staged users as inactive in mbox import 2017-12-13 08:45:43 +05:30
Gerhard Schlager 16738cfb1b FEATURE: convert plain text emails to markdown 2017-12-06 01:47:51 +01:00
Gerhard Schlager 32dd1e66be improvements to the mbox import script
* ignores dot-files and empty emails
* new setting to prefer HTML over plaintext emails during import
* restore original site settings at the end of import
* elided content of HTML mails was not put inside details block
2017-11-18 17:16:44 +01:00
Gerhard Schlager 06a6ddc3ba handle plaintext and HTML emails in mbox importer 2017-11-15 20:22:11 +01:00
Gerhard Schlager 6c829c24d7 escaping the subject isn't needed in the mbox importer 2017-10-19 15:25:20 +02:00
Gerhard Schlager c41880ab19 Improvements to the experimental mbox importer
* Disable journaling to improve performance in Docker
* Use the email cooking method
* Store IncomingEmail in order find related posts by Message-ID
* Escape HTML in imported messages
2017-10-19 14:27:40 +02:00
Gerhard Schlager 8299e7e8c3
Add new, experimental version of mbox importer 2017-05-29 20:59:18 +02:00