Commit Graph

10 Commits

Author SHA1 Message Date
David Taylor 436b3b392b
DEV: Apply syntax_tree formatting to `script/*` 2023-01-09 11:13:22 +00:00
Leonardo Mosquera 3e5faffb0d
DEV: mbox importer improvements (#16557)
* FIX: support specifying parent_category_id in mbox import metadata
* FIX: elide tabs from topic titles
* FIX: optionally fix Mailman from: addresses
* DEV: optionally elide anything up to the last = in email addresses
* Fix Mailmain broken from: detection
2022-04-29 13:24:29 -03:00
Gerhard Schlager 0e752db411 DEV: Improve mbox import script
* Customizable email subject prefixes to remove "Re" and "Fwd" as well as localized prefixes.
* Configuration option for prefixes like [FOO] or (BAR) which can be replaced with tags during import.
* Bugfix: Import script might have skipped some users due to missing ORDER BY.
2020-03-09 10:26:45 +01:00
Sam Saffron 30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Gerhard Schlager ac743dab10 Improve mbox import script
* emails weren't sorted in correct order
* better default regex for splitting mbox files
* output Message-ID if email is skipped because it doesn't have a Date
2018-08-23 09:46:28 +02:00
Gerhard Schlager 9b651adadb FIX: mbox importer should ignore emails without date 2018-03-13 13:42:57 +01:00
Gerhard Schlager bb54eb1192 Improvements to mbox importer
* store time it took to index message in DB (to find performance issues)
* ignore listserv specific files
* better examples for split_regex
* first email in mbox shouldn't contain the split string
* always lock the DB in exclusive mode
* save email within transaction
* messages can be grouped by subject and use original order (for Listserv)
* adds option to index emails without running the import
2018-01-17 12:04:57 +01:00
Gerhard Schlager 06a6ddc3ba handle plaintext and HTML emails in mbox importer 2017-11-15 20:22:11 +01:00
Gerhard Schlager c41880ab19 Improvements to the experimental mbox importer
* Disable journaling to improve performance in Docker
* Use the email cooking method
* Store IncomingEmail in order find related posts by Message-ID
* Escape HTML in imported messages
2017-10-19 14:27:40 +02:00
Gerhard Schlager 8299e7e8c3
Add new, experimental version of mbox importer 2017-05-29 20:59:18 +02:00