Commit Graph

299 Commits

Author SHA1 Message Date
Robin Ward 602e4d0da4 FIX: Return `nil` when the `reply_key` is missing 2017-06-08 14:28:48 -04:00
Gerhard Schlager 8299e7e8c3
Add new, experimental version of mbox importer 2017-05-29 20:59:18 +02:00
Régis Hanol 9dddb81cf6 FIX: remove memoization on class method used in a job 2017-05-22 23:35:41 +02:00
Régis Hanol 4fb335f1f0 FIX: don't process the same incoming email more than once 2017-05-18 16:43:07 +02:00
Régis Hanol 80435cf2af Add mutex around incoming email processing 2017-05-18 01:09:51 +02:00
Régis Hanol 9641d2413d REFACTOR: upload workflow creation into UploadCreator
- Automatically convert large-ish PNG/BMP to JPEG
- Updated fast_image to latest version
2017-05-11 00:16:57 +02:00
Régis Hanol bf322281e3 Improve handling of inlined images in incoming emails 2017-05-03 23:02:02 +02:00
Claus Strasburger e9bb9a167b Fix reply-by-mail for 8-bit transfer encodings
The mail class seems to handle mails sent with Content-Transfer-Encoding: 8bit
somewhat weirdly: It decodes them (to utf-8), changes the raw source to base64,
and does not modify the Content-Type:charset= header.

This leads to Discourse trying the message encoding (in my example ISO-8859-1)
first, and if that does not contain any unparseable characters, it uses that.
Sadly, in ISO-8859-1, every byte sequence is valid.

Fix this by always trying to decode as UTF-8 first. The probability of someone
using another encoding that cleanly (but wrongly) decodes as UTF-8 should be
fairly low.
2017-04-30 23:30:40 +02:00
Régis Hanol aba76bace6 add support to keep img tags when converting to html 2017-04-28 22:14:46 +02:00
Régis Hanol 0ec15af970 restore the 'incoming_email_prefer_html' site setting 2017-04-27 14:31:11 +02:00
Régis Hanol b76674f640 FEATURE: convert incoming emails in HTML to markdown
- remove incoming_email_prefer_html site setting
- remove HtmlCleaner class
2017-04-26 16:49:06 +02:00
Guo Xiang Tan 423f2ab228 FIX: Processing incoming email should be done in a background job. 2017-04-24 13:57:28 +08:00
Ryan C. Gordon a51c191a66 Make Email::Receiver.check_address() into a class method. 2017-04-05 23:10:36 -04:00
Régis Hanol 91d09ebc08 post ids are always an integer 2017-02-08 23:46:11 +01:00
Régis Hanol e9e88a010f FIX: find replied-to post even when incoming email doesn't have a reply key 2017-02-08 21:38:52 +01:00
Régis Hanol c725225f69 FIX: log message when revoking email with too many bounces 2017-02-01 16:53:24 +01:00
Matt Palmer 04ae3539d0 FEATURE: Better error message when incoming e-mail is missing a Date: header 2017-01-13 11:05:00 +11:00
Neil Lalonde 42c39ab38e Don't display email addresses in staff action logs for revoked email 2017-01-10 17:51:22 -05:00
Régis Hanol 185dcb2ca1 handle emails with localized headers 😠 2017-01-09 22:59:30 +01:00
Régis Hanol 98c62bccb5 FIX: mark forwarded email as read by the forwarder
FIX: 'Re:' prefix is mostly used for replies and not forwarded emails
2017-01-06 15:33:55 +01:00
Régis Hanol eb453d0f82 the note in a FWed email should be a whisper only in PM and when the author is member of the group 2016-12-01 18:43:56 +01:00
Régis Hanol 62763f025c FIX: wasn't able to parse FROM email in the embedded email 2016-12-01 18:34:47 +01:00
Régis Hanol 2125a630d9 FIX: encoding issues with forwarded emails 2016-11-17 12:44:39 +01:00
Régis Hanol a69f45d0da FEATURE: new 'always_show_trimmed_content' site setting 2016-11-16 22:06:07 +01:00
Régis Hanol 17f2be9f88 FEATURE: new 'enable_forwarded_email' site setting 2016-11-16 19:42:11 +01:00
Régis Hanol 5c06076b5c FIX: strip blacklisted attachments before checking for empty email body 2016-08-08 22:20:33 +02:00
Régis Hanol e92f5e4fbf FEATURE: new email attachment blacklists site settings 2016-08-03 17:55:54 +02:00
Régis Hanol 323bd555c0 FIX: process all incoming email's destinations instead of only the first 2016-08-03 15:57:37 +02:00
Régis Hanol ca4c939e9a incoming mail is considered autogenerated when the subject match a list of 'out of office' keywords 2016-08-02 00:04:59 +02:00
Régis Hanol 7d0dc7cb7a FIX: don't create staged user(s) when receiving a bounce (mailer-deamon begone) 2016-08-01 23:37:59 +02:00
Régis Hanol b0f7e4ba00 FEATURE: deactive users after too many bounces 2016-07-25 18:57:06 +02:00
Régis Hanol 7648916314 new 'reset_bounce_score_after_days' site setting 2016-07-25 17:29:54 +02:00
Régis Hanol d2e22ab215 extract bounce scores into site settings 2016-07-25 17:27:28 +02:00
Régis Hanol 6ed03bf3d9 FIX: always update bounce_score even when email was not auto_generated 2016-07-15 18:00:40 +02:00
James Kiesel 3588780ac3 Don't reject likes by email for closed topics (#4311) 2016-07-05 17:33:08 +02:00
James Kiesel 949be591da FEATURE: Allow <3 and ❤ to trigger like via email (#4310) 2016-07-05 15:59:23 +02:00
Régis Hanol 9bf34ce49b FIX: support 'nil' alternative reply by email address 2016-07-02 11:09:06 +02:00
Régis Hanol 214f5bff5c don't send more than 1 reply per day to auto-generated emails 2016-06-28 16:42:05 +02:00
Régis Hanol 376881845c always strip s/mime signatures in incoming emails 2016-06-27 22:26:05 +02:00
Régis Hanol 589bae5c03 try to fix badly encoded emails 2016-06-26 13:27:34 +02:00
Régis Hanol 8c51d34100 FIX: receiving a bounce from a deleted user 2016-06-22 20:41:21 +02:00
Matt Palmer fde5ef4902 Merge pull request #4273 from mpalmer/log-useful-info-replyusernotmatchingerror
Log some useful info when raising ReplyUserNotMatchingError
2016-06-17 12:06:03 +10:00
Matt Palmer eaf111f3e1 Log some useful info when raising ReplyUserNotMatchingError 2016-06-17 12:01:08 +10:00
Régis Hanol dffe50a2e6 new alternative reply by email addresses 2016-06-10 16:14:42 +02:00
Régis Hanol ef8b652d1f FIX: elided part of incoming email was dismissed when creating a new topic 2016-06-06 10:30:04 +02:00
Régis Hanol 116efffdaa FEATURE: webhooks support for mailgun 2016-05-30 17:11:17 +02:00
David McClure a92fd9d701 Add Site Setting to use HTML from incoming email when available. (#4236) 2016-05-26 10:13:01 +02:00
Régis Hanol c75d58ab21 FEATURE: new 'ignore_by_title' site setting 2016-05-18 23:07:01 +02:00
Régis Hanol 5f76287b18 FEATURE: cap number of staged users (defaults to 10) created per incoming email 2016-05-16 21:45:34 +02:00
Régis Hanol 341d5e10b3 FIX: auto-reply to VERP should be considered as hard boucne 2016-05-16 09:01:15 +02:00
Régis Hanol 1e57bbf5c8 Lots bounce emails related fixes
- Show bounce score on user admin page
- Added reset bounce score button on user admin page
- Only whitelisted email types are sent to emails with high bounce score
- FIX: properly detect bounces even when there is no TO: header in the email
- Don't desactivate a user when reaching the bounce threshold
2016-05-06 19:34:33 +02:00
Régis Hanol 8e611ec7a1 FEATURE: handle bounced emails 2016-05-02 23:15:32 +02:00
Régis Hanol a244c33023 temporarily disable processing email replies to VERP 2016-04-29 18:30:05 +02:00
Régis Hanol 4960b62110 FEATURE: new 'block_auto_generated_emails' site setting 2016-04-20 21:29:27 +02:00
Régis Hanol 7d9f2265b9 FIX: improve support for handling emails coming from screened email addresses 2016-04-18 23:01:54 +02:00
Régis Hanol 4d9c81fde7 Merge pull request #4148 from tgxworld/dont_reply_to_emails_that_are_autogenerated
FIX: Don't send rejection mailer to bounced emails.
2016-04-13 15:36:14 +02:00
Régis Hanol 56089c4856 FEATURE: new 'auto_generated_whitelist' site setting 2016-04-11 22:47:34 +02:00
Régis Hanol e4980392da improve receiver specs to account for posts approval 2016-04-11 18:20:26 +02:00
Guo Xiang Tan 5734c7f3f3 FIX: Don't send rejection mailer to bounced emails. 2016-04-07 22:21:17 +08:00
Régis Hanol 79329c8e85 FIX: check we used the right encoding when parsing incoming emails 2016-03-30 19:54:38 +02:00
Régis Hanol 6c82fbebf5 detect more auto-generated emails 2016-03-30 18:41:09 +02:00
Régis Hanol 39863953cd new 'enable_staged_users' site setting 2016-03-23 18:56:03 +01:00
Régis Hanol 03a1aa0000 SECURITY: only add elided part of email in PM 2016-03-17 23:10:46 +01:00
Régis Hanol e09e807027 FIX: skip validation when replying via email for staged users 2016-03-14 22:21:18 +01:00
Régis Hanol 4a3cb4a000 FIX: use MD5 of the email_string when there's no 'Message-Id' 2016-03-14 18:18:58 +01:00
Régis Hanol 841cec1bc1 FIX: support emails in with weird encoding 2016-03-11 18:51:53 +01:00
Régis Hanol 6d84a8a1b3 FIX: don't send out elided message in email notifications
UX: improved details tag for elided messages
2016-03-11 17:51:16 +01:00
Régis Hanol 2747e14b4c FEATURE: hide elided part of incoming emails behind a [details] tag 2016-03-09 18:51:54 +01:00
Régis Hanol 622d804d46 FEATURE: Add rejection message on rejected IncomingEmail
FIX: Better RateLimit description in rejected IncomingEmail
FEATURE: Send email when hitting a rate limit
2016-03-07 16:56:17 +01:00
Régis Hanol be5a54d67d FEATURE: new 'allow_all_attachments_for_group_messages' site setting 2016-02-29 22:39:24 +01:00
Régis Hanol f7432d8ec9 FEATURE: add support for multiple incoming emails for groups & categories 2016-02-24 19:47:58 +01:00
Régis Hanol 54262cc9b2 FIX: properly handle invalid from header 2016-02-24 17:40:57 +01:00
Régis Hanol c3a393a11f let's try with 'normal' linebreaks 2016-02-24 15:16:29 +01:00
Régis Hanol 999e7ac4bc FIX: newlines around attachments in received emails 2016-02-24 11:38:12 +01:00
Régis Hanol 5415b2666d FIX: staged users couldn't create new topic in a private category via email 2016-02-22 19:57:53 +01:00
Régis Hanol 86819f08c3 FIX: use RFC-compliant previous replies separator 2016-02-11 18:48:09 +01:00
Régis Hanol cad7fc1062 FIX: don't allow blocked user to send emails in 2016-02-11 10:39:57 +01:00
Régis Hanol 91bb38626c FEATURE: new incoming email details modal 2016-02-10 22:00:27 +01:00
Régis Hanol cf4c256b17 FEATURE: new 'raw email' modal when listing rejected emails 2016-02-01 21:41:49 +01:00
Régis Hanol 49d1f88f6e FIX: support incoming emails with no subject 2016-02-01 12:16:15 +01:00
Régis Hanol 27aaed0aa5 FIX: don't use 'Return-Path' header to determine whether the email was auto generated 2016-02-01 10:21:58 +01:00
Régis Hanol 46b6c55197 FIX: replace 'discourse_email_parser' with 'email_reply_trimmer' to better trim replies from plain text emails
FIX: undefined method `number_to_human_size' when email contains attachments
2016-01-30 01:29:31 +01:00
Régis Hanol 4a17cdc1e4 FIX: don't invite users with emails configured as 'incoming' (reply, group our category) 2016-01-20 23:08:27 +01:00
Régis Hanol f145310cd5 FIX: associates email replies using both 'In-Reply-To' and 'References' headers 2016-01-20 22:52:08 +01:00
James Kiesel c7283751a3 Unsubscribe via email 2016-01-20 22:25:25 +13:00
Régis Hanol 735017dd9f FIX: don't invite users with the same email as 'reply_be_email_address' 2016-01-19 15:24:34 +01:00
Régis Hanol 3083657358 FEATURE: better email in support
FEATURE: new incoming_email model
FEATURE: infinite scrolling in emails admin
FEATURE: new 'emails:import' rake task
2016-01-19 00:57:55 +01:00
James Kiesel 6ceb108946 Add specs for post action guardian 2015-12-30 20:52:36 +01:00
James Kiesel a559754db3 Add post action creator 2015-12-30 20:04:05 +01:00
James Kiesel 86da47880a Allow +1 via email 2015-12-30 12:17:45 +01:00
Régis Hanol 3e923c7a41 FIX: ensure inactive users can't email in 2015-12-21 17:54:02 +01:00
Régis Hanol ca529a673a just use the address instead of building it ourselves 2015-12-18 17:54:42 +01:00
Régis Hanol 4bb31daa2e FIX: when getting a reply by email, ensure it's by the same user 2015-12-16 00:43:05 +01:00
Régis Hanol 02279c41cb Merge pull request #3918 from techAPJ/better-email-parsing
FEATURE: better email reply parsing
2015-12-11 14:51:24 +01:00
Régis Hanol 323e3cee22 when creating a staged account, use the display name provided in the email 2015-12-10 23:52:20 +01:00
Régis Hanol 93d1cc6294 add support for incoming emails in CC/BCC fields 2015-12-10 23:49:16 +01:00
Arpit Jalan 553248de86 FEATURE: new gem discourse_email_parser 2015-12-11 01:02:28 +05:30
Arpit Jalan 7a8e5a50ff FEATURE: better email reply parsing 2015-12-09 23:47:01 +05:30
Régis Hanol 1cde276656 FEATURE: ability to send emails to a group 2015-12-07 17:01:08 +01:00
Régis Hanol dd28a3c63a add support for inline images in emails in 2015-11-30 18:33:24 +01:00
Régis Hanol 0d54c18c8b new hidden 'allow_staged_accounts' setting 2015-11-26 18:31:23 +01:00
Robin Ward 2196160549 FIX: Allow emails to begin with a quote (but skip it!) 2015-11-18 15:23:10 -05:00
Robin Ward 33a5fce124 Importer for mbox format 2015-07-23 18:37:40 -04:00
Robin Ward ab50d039bb Importer for nabble 2015-05-22 15:41:26 -04:00
Robin Ward 2cbcd15789 FIX: Post Mannager wasn't being used for email replies 2015-04-28 13:04:34 -04:00
Arpit Jalan f3daae6dec Improve auto generated email reply error message 2015-02-18 18:58:51 +05:30
Régis Hanol 0e5c9b2590 small upload code refactor 2015-02-03 18:44:18 +01:00
Arpit Jalan 5287669116 💄 simplify utf-8 conversion 2015-01-19 15:21:39 +05:30
Arpit Jalan fad9ca7b36 FIX: convert UTF8 charset to UTF-8 2015-01-19 10:09:17 +05:30
Arpit Jalan d28515792a FIX: auto response email replies should not be accepted 2014-12-04 22:39:13 +05:30
Arpit Jalan 529014d56e FEATURE: switch to GitHub email_reply_parser library and parse plain text email content 2014-12-01 09:00:35 +05:30
Robin Ward c89064f7c0 Merge pull request #2901 from JKillian/patch-1
Add additional fields that indicate reply
2014-11-10 15:44:14 -05:00
Arpit Jalan b37d845dd3 FIX: email replies should not be accepted for deleted topics 2014-10-27 13:05:33 +05:30
Arpit Jalan 08dc0e6ee6 FIX: email replies to closed topic should not be accepted 2014-10-25 23:47:13 +05:30
Arpit Jalan 7a1731ce0e emails from microsoft outlook 14 were not getting parsed as text 2014-10-24 11:45:17 +05:30
JKillian d872d0f78a Add additional fields that indicate reply
When scanning for where a reply starts, three lines in a row matching common email reply headers indicate the start of a reply. Add 'Cc', 'Bcc', and 'Date' because these are also common.
2014-10-22 15:22:04 -04:00
Arpit Jalan 2131a37811 FEATURE: show raw email for replies/topics created via email 2014-10-17 22:22:19 +05:30
Arpit Jalan 24a0db3244 FIX: fix email replies for ms outlook 2014-10-14 16:20:35 +05:30
riking a0dccf7bc0 FIX: Add "On day, name wrote:" quote trigger for emails 2014-09-09 16:27:06 -07:00
Sam 59d04c0695 Internal renaming of elder,leader,regular,basic to numbers
Changed internals so trust levels are referred to with

TrustLevel[1], TrustLevel[2] etc.

This gives us much better flexibility naming trust levels, these names
are meant to be controlled by various communities.
2014-09-05 15:20:52 +10:00
Robin Ward 1c5e8efa68 FEATURE: Show an email icon beside posts that arrived via email 2014-09-04 13:04:41 -04:00
riking 8ddd90daa4 Have parse_body() recover from ASCII-8BIT encoding
Added a test to make sure that the result can be passed into TextCleaner
(which expects UTF-8)
2014-08-28 14:38:19 -07:00
riking 0a09593f3b FIX: Prefer HTML in incoming emails, heavily refactor email receiver
This commit heavily refactors Email::Receiver to both better handle
different emails and improve testability.

A primary focus of the refactor is reducing the usage of class
variables, in favor of actually passing parameters - making it possible
for multiple tests to use the same Receiver instance.

The EmailLog reported when a topic is created is reflected to put the
user's email in the to_address field, instead of the system address.

The discourse_email_parser function is renamed to
discourse_email_trimmer, and additional stopping conditions are added to
make up for EmailReplyParser's inability to deal with html at the start
of a line.

The force_encoding calls are refactored out to a 'fix_charset' method.

parse_body is renamed to select_body, and the scrub_html method is
dropped in favor of the new HtmlCleaner class.

A new parse_body method is added, which performs the job of the removed
lines of code in the 'process' method.

EmailUnparsableError is redefined again, to be encoding errors (when the
declared encoding is not what was delivered).
2014-08-28 14:36:10 -07:00
riking 0d0225133c FIX: Failed incoming emails could create empty topics
A failure condition is eliminated where a topic would be created, but post
creation would fail, leaving the forum with a topic without any posts.
By asking PostCreator to create the topic instead, inside of its
transaction, this failure condition is eliminated.

Additionally, attachments are restored to working status. Previously,
the attachment code would build up the post raw, but then drop it and
not do anything with the result (creating orphaned uploads). By actually
placing the raw value back in the options hash, it is included in the
created post.
2014-08-28 14:35:43 -07:00
riking cce8b4720a Have reciever handle multiple To addresses 2014-08-13 11:43:49 -07:00
Sam dcc9923e4b FIX: failed messages posted via email silently ignored
also... test suite over mocking
2014-07-31 18:46:02 +10:00
riking 17db265b41 Remove use of email_in_address and email_in_category 2014-07-17 10:25:13 -07:00
Régis Hanol 4a93b1b0bc missed a newline 2014-04-15 00:04:13 +02:00
Régis Hanol 2505d18aa9 FEATURE: support email attachments 2014-04-14 22:55:57 +02:00
Neil Lalonde fd504e741f FIX: reply by email can handle emails with attachments. Attachments are still ignored, but a post or topic can be created from the email now. 2014-03-28 09:57:20 -04:00
Benjamin Kampmann 024597e643 Switch to proper exception handling system for better user feedback
- Replace implicit return code-system in Email::Receiver with proper exception system
 - Update tests to check for exceptions instead
 - Test the PollMailbox for expected failures
 - Add proper email-handling of problematic emails
"
2014-02-28 17:02:58 +01:00
Benjamin Kampmann d32cb55837 Add public-inbox to Email-In-Feature
- Adds the advanced option to accept email from non-users per category email-address
 - Adds tests covering the new feature
 - Adds UI to configure this feature in the frontend
2014-02-28 16:53:58 +01:00
Benjamin Kampmann 37cea49459 Add Email-In-Per-Category
- allow the configuration of an inbox-email-address per category
 - post emails to that email into that category instead of global

 - Adds UI for configuration
 - Adds Documentation for configuration
 - Adds Tests for new feature
2014-02-28 16:53:58 +01:00
Benjamin Kampmann 4af2cf3f23 Refactor and clean up New-Topic via Email
With the new email_in admin configuration setting, emails to the email_in_address fetched via POP will now be processed and posted as new topics to the forum.

With the email_in_min_trust you can control the trust level the user needs to have at least to be able to post an email as a new topic.

Also contains tests for the email-in feature and minor clean ups
2014-02-28 16:53:58 +01:00
Allen Hancock dad43b9853 Optionally allow discourse to create new topics from email. 2014-02-28 16:53:58 +01:00
Robin Ward 318e692290 FIX: If there is an exception when parsing one email, don't stop all
work, return an error code and continue.
2014-02-12 12:08:34 -05:00
Herb Jiang 5fa464e10b force_encoding according to the charset in mail, this solve encoding problem when not using ‘utf-8’ charset. 2014-01-19 02:12:38 +08:00
Sam 9738c4ff48 FEATURE: backport a minimal String#scrub
BUGFIX: invalid byte sequence in email would explode all processing
2013-12-30 14:05:25 +11:00
Robin Ward 5055da76e6 Abort triggers for email body processing 2013-11-20 13:30:01 -05:00
Robin Ward c86b06e3fa Fix for non-english email polling. 2013-11-04 15:30:24 -05:00
Robin Ward ff3b163a78 FIX: Another email regexp for quote lines in the format we can recognize including
the site name.
2013-08-21 16:54:51 -04:00
Sam 61b330abb4 odd formatting 2013-08-01 16:02:43 +10:00
Sam df3a4c05cd upgrade rack mini profiler
fixes for ruby 1.9.3
2013-07-30 17:36:34 +10:00
Robin Ward 2319a824f8 Special work to never include previous posts in parsed emails. Also a new attempt
to catch reply strings in different languages.
2013-07-24 14:22:32 -04:00
Robin Ward bb908d5913 Email parsing uses Traditional Markdown Linebreaks by default. Added JS tests for line breaks. 2013-06-21 11:37:26 -04:00
Robin Ward 0d10b5c9c4 More email receiving logic 2013-06-20 12:38:37 -04:00
Robin Ward 82d1465d18 Ugly Hack: Remove improperly parsed headers from Mail::Message 2013-06-19 12:14:01 -04:00
Robin Ward 8acdc18bc8 First stab at polling support for POP3S / reply by email 2013-06-17 16:49:02 -04:00
Robin Ward e29f4a3496 Work in Progress: Reply By Email:
- Add support classes and settings to enable reply by email
- Split out Email builder to be more OO, add many specs
2013-06-13 12:39:56 -04:00