Commit Graph

360 Commits

Author SHA1 Message Date
Krzysztof Kotlarek e0d9232259
FIX: use allowlist and blocklist terminology (#10209)
This is a PR of the renaming whitelist to allowlist and blacklist to the blocklist.
2020-07-27 10:23:54 +10:00
Guo Xiang Tan c6202af005
Update rubocop to 2.3.1. 2020-07-24 17:19:21 +08:00
Daniel Waterworth 9313706649 DEV: Enable preserve_email_structure_when_styling by default
In 1bd8a075, a hidden site setting was added that causes Email::Styles
to treat its input as a complete document in all cases.

This commit enables that setting by default.

Some tests were removed that were broken by this change. They tested the
behaviour of applying email styles to empty strings. They weren't useful
because:

 * Sending empty email is not something we ever intend to do,
 * They were testing incidental behaviour - there are lots of
   valid ways to process the empty string,
 * Their intent wasn't clear from their descriptions,
2020-07-20 10:21:32 +01:00
Dan Ungureanu c72bc27888
FEATURE: Implement support for IMAP and SMTP email protocols. (#8301)
Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
2020-07-10 12:05:55 +03:00
Bianca Nenciu bd842cd2b0
FEATURE: Parse images in email signatures (#10137)
* FEATURE: Parse images in email signatures

* DEV: Fix tests

* Code review
2020-07-08 15:50:30 +10:00
Michael Brown d9a02d1336
Revert "Revert "Merge branch 'master' of https://github.com/discourse/discourse""
This reverts commit 20780a1eee.

* SECURITY: re-adds accidentally reverted commit:
  03d26cd6: ensure embed_url contains valid http(s) uri
* when the merge commit e62a85cf was reverted, git chose the 2660c2e2 parent to land on
  instead of the 03d26cd6 parent (which contains security fixes)
2020-05-23 00:56:13 -04:00
Jeff Atwood 20780a1eee Revert "Merge branch 'master' of https://github.com/discourse/discourse"
This reverts commit e62a85cf6f, reversing
changes made to 2660c2e21d.
2020-05-22 20:25:56 -07:00
Guo Xiang Tan 96c02caba7
DEV: Change use of Redis `flushall` to `flushdb`.
FLUSHALL removes all keys from all databases. Instead we only want to
remove keys from the current Redis database.
2020-05-19 10:20:00 +08:00
Mark VanLandingham a047004c9a
FIX: Specific email error for replies to digest emails (#9770) 2020-05-14 09:04:58 -05:00
Krzysztof Kotlarek 9bff0882c3
FEATURE: Nokogumbo (#9577)
* FEATURE: Nokogumbo

Use Nokogumbo HTML parser.
2020-05-05 13:46:57 +10:00
Martin Brennan 2237ba8c9d
FIX: Add topic deleted check to email/sender (#9166)
It already had a deleted post check and log reason, add a topic one too to avoid errors
2020-03-13 10:04:15 +10:00
Daniel Waterworth 59578dfc5b FIX: Notification emails with attachments are incorrectly structured
Two behaviors in the mail gem collide:

 1. Attachments are added as extra parts at the top level,
 2. When there are both text and html parts, the content type is set to
    'multipart/alternative'.

Since attachments aren't alternative renderings, for emails that contain
attachments and both html and text parts, some coercing is necessary.
2020-03-12 15:42:24 +00:00
Jarek Radosz 29b35aa64c
DEV: Improve flaky time-sensitive specs (#9141) 2020-03-10 22:13:17 +01:00
Gerhard Schlager 453bec9394 FEATURE: Add small action post to indicate forwarded email
This happens only when the sender of the email didn't write anything in their email.
2020-02-12 16:23:17 +01:00
Gerhard Schlager e84d88ddea FIX: Email attachments with a size of 0 bytes caused error 2020-02-04 12:30:43 +01:00
Arpit Jalan 62c21ba649 Remove `bounce_score_threshold_deactivate` setting.
Removed `bounce_score_threshold_deactivate` setting as the deactivate
threshold is not getting reached.
2020-01-30 16:17:31 +05:30
Martin Brennan 1150cd4621
FIX: Stop secure media URLs being censored too liberally in emails (#8817)
For example /t/ URLs were being replaced if they contained secure-media-uploads so if you made a topic called "Secure Media Uploads Are Cool" the View Topic link in the user notifications would be stripped out.

Refactored code so this secure URL detection happens in one place.
2020-01-30 16:19:14 +10:00
Neil Lalonde 34f564acd6 FIX: customized email subjects was ignored for some notifications
Email templates for notifications about new posts and topics weren't
using customized values.
2020-01-29 14:49:08 -05:00
Leo McArdle 8883cca373 enqueue spam/dmarc failing emails instead of hiding (#8674)
* enqueue spam/dmarc failing emails instead of hiding

* add translations for dmarc/spam enqueued reasons

* unescape quote

* if email_in_authserv_id is blank return gray for all emails
2020-01-21 11:12:00 -05:00
Gerhard Schlager c88797bf0e FIX: Ignore DMARC for emails sent to mailing list mirror 2019-12-06 13:29:39 +01:00
Joffrey JAFFEUX 0d3d2c43a0
DEV: s/\$redis/Discourse\.redis (#8431)
This commit also adds a rubocop rule to prevent global variables.
2019-12-03 10:05:53 +01:00
Leo McArdle 2714149fd2 FEATURE: hide posts from incoming email based on dmarc verdict (#8333) 2019-11-26 15:55:22 +01:00
Penar Musaraj 102909edb3 FEATURE: Add support for secure media (#7888)
This PR introduces a new secure media setting. When enabled, it prevent unathorized access to media uploads (files of type image, video and audio). When the `login_required` setting is enabled, then all media uploads will be protected from unauthorized (anonymous) access. When `login_required`is disabled, only media in private messages will be protected from unauthorized access. 

A few notes: 

- the `prevent_anons_from_downloading_files` setting no longer applies to audio and video uploads
- the `secure_media` setting can only be enabled if S3 uploads are already enabled and configured
- upload records have a new column, `secure`, which is a boolean `true/false` of the upload's secure status
- when creating a public post with an upload that has already been uploaded and is marked as secure, the post creator will raise an error
- when enabling or disabling the setting on a site with existing uploads, the rake task `uploads:ensure_correct_acl` should be used to update all uploads' secure status and their ACL on S3
2019-11-18 11:25:42 +10:00
Krzysztof Kotlarek c32bd8ae48 FEATURE: Remove attachments and truncate raw field for incoming emails (#8253)
Adds the settings: 

raw_email_max_length, raw_rejected_email_max_length, delete_rejected_email_after_days. 

These settings control retention of the "raw" emails logs.

raw_email_max_length ensures that if we get incoming email that is huge we will truncate it removing uploads from the raw log.

raw_rejected_email_max_length introduces an even more aggressive truncation for rejected incoming mail. 

delete_rejected_email_after_days controls how many days we will keep rejected emails for (default 90)
2019-10-30 16:54:35 +11:00
Leo McArdle e7ff6809a3 FEATURE: add SES spam header to recognised spam headers (#8254) 2019-10-28 12:46:53 -04:00
Leo McArdle 6296ae3d31 FEATURE: add setting to show content of forwarded emails in topics (#7935) 2019-08-07 12:32:19 +02:00
Gerhard Schlager fd12c414e7 DEV: Refactor helper methods for upload markdown
Follow-up to a61ff167
2019-07-25 16:36:35 +02:00
Gerhard Schlager 7e0eeed292 FEATURE: Add attachments to outgoing emails
This feature is off by default and can can be configured with the `email_total_attachment_size_limit_kb` site setting.

Co-authored-by: Maja Komel <maja.komel@gmail.com>
2019-07-25 15:57:45 +02:00
Guo Xiang Tan 42ab016856 FIX: Use markdown for images and attachments in `Email::Receiver`. 2019-06-11 14:49:46 +08:00
Guo Xiang Tan 6ad58913c8 DEV: Prefer `save!` over `save`. 2019-06-04 16:59:29 +08:00
Maja Komel 87d3b86484 FIX: better error message when user without permissions replies via email 2019-06-04 16:39:41 +08:00
Gerhard Schlager 4a1755b780 FIX: Don't create staged users when processing forwarded email fails 2019-05-09 23:47:47 +02:00
Daniel Waterworth e219588142 DEV: Prefabrication (test optimization) (#7414)
* Introduced fab!, a helper that creates database state for a group

It's almost identical to let_it_be, except:

 1. It creates a new object for each test by default,
 2. You can disable it using PREFABRICATION=0
2019-05-07 13:12:20 +10:00
David Taylor 0644c10bfe FIX: Prioritize VERP key over final_recipient header in bounced email
Per RFC3464 2.3.2, the final_recipient header may not match the address we originally sent the email to.
2019-05-03 12:12:44 +01:00
Sam Saffron 4ea21fa2d0 DEV: use #frozen_string_literal: true on all spec
This change both speeds up specs (less strings to allocate) and helps catch
cases where methods in Discourse are mutating inputs.

Overall we will be migrating everything to use #frozen_string_literal: true
it will take a while, but this is the first and safest move in this direction
2019-04-30 10:27:42 +10:00
Arpit Jalan 230e111e87 DEV: more accurate spec for email iframe handling 2019-04-25 22:32:53 +05:30
Arpit Jalan bd1db1860a FIX: prefer data-original-href attribute to get iframe URL 2019-04-24 13:53:27 +05:30
Dan Ungureanu 35a866fe22
FIX: Do not strip email lines having lists. 2019-04-16 11:39:35 +03:00
Dan Ungureanu e92cd5318b FEATURE: Add setting to strip whitespaces from incoming emails. (#7375)
Some email clients add leading whitespaces which get are transformed in
code blocks when processed.
2019-04-15 16:26:00 +10:00
David Taylor 6a05f190c6
PERF: Do not create staged users for most rejected incoming emails (#7301)
Previously we would create users, then destroy them at the end of the job if the post was rejected. Now we do not create users unless required.
2019-04-08 10:36:39 +01:00
David Taylor a9d5ffbe3d FIX: Prevent critical emails bypassing disable, and improve email test logic
- The test_email job is removed, because it was always being run synchronously (not in sidekiq)
- 34b29f62 added a bypass for critical emails, to match the spec. This removes the bypass, and removes the spec.
- This adapts the specs for 72ffabf6, so that they check for emails being sent
- This reimplements c2797921, allowing test emails to be sent even when emails are disabled
2019-03-22 17:28:43 +08:00
David Taylor 3f9e7eb326 FIX: Respect the disable_emails=non-staff site setting correctly
This reverts commit c279792130.

This commit inadvertently removed all of the non-staff email logic, rather than just for the 'test email' button. 

https://meta.discourse.org/t/112231/5
2019-03-21 21:44:14 +00:00
Penar Musaraj 9334d2f4f7
FEATURE: add more granular user option levels for email notifications (#7143)
Migrates email user options to a new data structure, where `email_always`, `email_direct` and `email_private_messages` are replace by

* `email_messages_level`, with options: `always`, `only_when_away` and `never` (defaults to `always`)
* `email_level`, with options: `always`, `only_when_away` and `never` (defaults to `only_when_away`)
2019-03-15 10:55:11 -04:00
David Taylor 420c6f8102
FEATURE: Skip sending emails to domains on the `.invalid` TLD (#7162)
This is a reserved TLD which we use when importing users without an email address. https://tools.ietf.org/html/rfc2606
2019-03-13 16:17:59 +00:00
Sam b2187301fd FEATURE: allow users to easily track/watch/mute topics via email
If you reply to an email with the word "mute" a topic will be muted
If you reply to an email with the word "track" a topic will be tracked
If you reply to an email with the word "watch" a topic will be watched

These ninja command can help advanced mailing list ex-users, saves a trip
to the website
2019-03-06 18:38:49 +11:00
Robin Ward 720e896e17 FIX: `PostActionCreator` was not checking the guardian properly
It also exposed a bug in the EmailReceiver spec, where a test had a user
liking their own post and was not failing.
2019-01-31 14:48:42 -05:00
Régis Hanol 51fdf7a11d FIX: don't duplicate attachments 2019-01-28 18:40:52 +01:00
Régis Hanol db76e5a368 FIX: properly handle attachments in received emails
This regressed in 1ac3e547 when we added support for eml attachments.
2019-01-25 19:13:34 +01:00
Rishabh c279792130 FIX: Allow sending test e-mails to any email address when disable_email is set to non-staff (#6792) 2018-12-18 16:12:05 +01:00
Vinoth Kannan fc0b7c9e26 FIX: incoming email matches the wrong user if null bounce key available in db 2018-11-30 12:29:51 +05:30
Vinoth Kannan bfb3c4d9f9 DEV: create bounce alert earlier if email_log detected from bounce_key 2018-11-28 21:13:06 +05:30
Vinoth Kannan 25253dec56 FIX: Get email address from email_log if bounced with verp
We can not access mail.final_recipient attr if it bounced with verp
2018-11-28 19:04:09 +05:30
Vinoth Kannan 7dbf709467 FIX: create whisper post in PMs when bounces with verp and user is staged 2018-11-28 08:24:23 +05:30
Guo Xiang Tan 68bef91dd6 DEV: Minor clean up of specs. 2018-11-27 13:09:58 +08:00
Vinoth Kannan cedd2118c4
FEATURE: If PM email bounced for staged user then alert in whisper reply (#6648) 2018-11-27 00:29:37 +05:30
Régis Hanol 6b51d84dc5 FIX: Don't enqueue topics if the user can't create them
Co-authored-by: Vinoth Kannan <vinothkannan@vinkas.com>
2018-11-09 18:24:28 +01:00
Maja Komel 1ac3e5473a FIX: don't strip eml attachments from received emails 2018-11-05 09:35:22 +01:00
Penar Musaraj b06dccac49 FIX: force enable a user's email_private_messages option when user replies via email (#6478)
* Enable user email PM when posting to group or replying to topic via email

* remove extra line

* Add test and fix snake_case

* Only reenable email_private_messages for PM replies
2018-10-16 10:51:57 +11:00
Gerhard Schlager 7a41a783a4 FIX: Don't reply to Unsubscribe email sent to mailing list mirror 2018-10-11 16:09:22 +02:00
Guo Xiang Tan 5039a6c3f1 FIX: Strip null bytes in mail subjects. 2018-10-11 09:46:32 +08:00
Guo Xiang Tan f26804394a DEV: Remove the use of stubs on `Rails.logger` in our test suite. 2018-10-10 09:34:50 +08:00
Maja Komel 361ad7ed2b FEATURE: add indication if incoming email attachment was rejected and inform sender about it (#6376)
* FEATURE: add indication if incoming email attachment was rejected and inform sender about it

* include errors for rejected attachments in email

* don't send warning email to staged users

* use user object instead of user_id in add_attachments method
2018-10-04 22:08:28 +08:00
Penar Musaraj da9eee5262 FIX: Force enable user PM emails option when user posts to a group by email. 2018-10-02 12:38:10 +08:00
Gerhard Schlager eeedc3901e FIX: Replying to deleted post via email should create new reply to topic 2018-09-03 23:06:40 +02:00
Guo Xiang Tan 36a7028f19 FEATURE: Clean up `PostReplyKey` records.
* Default retention of 90 days.
2018-08-23 10:40:02 +08:00
Régis Hanol f01169d6ff FIX: don't send email when the post was deleted 2018-08-22 13:13:58 +02:00
Guo Xiang Tan 7dcc69aef4 DEV: Refactor test to not call private method. 2018-08-21 14:29:58 +08:00
Guo Xiang Tan 48f499b324 DEV: Refactor some mail receiver related specs. 2018-08-21 14:00:45 +08:00
Guo Xiang Tan d104de2a09 Remove line that is no longer required. 2018-08-21 11:48:58 +08:00
Régis Hanol ac2513b0f2 FEATURE: automatic PM when a user's email is revoked 2018-08-03 16:39:22 +02:00
OsamaSayegh a157dfd418 UX: better rejection message when reply via email is too short 2018-08-02 22:43:53 +03:00
Guo Xiang Tan fad9c2b971 PERF: Move `EmailLog#reply_key` into new `post_reply_keys` table. 2018-07-24 13:51:53 +08:00
Guo Xiang Tan 3874d40910 Prepare to drop `EmailLog#topic_id`. 2018-07-18 10:22:24 +08:00
Leo McArdle 21ebb1cd54 FEATURE: Secondary emails support. 2018-07-16 11:09:49 +08:00
Patrick Gansterer 28dd7fb562 FEATURE: Create hidden posts for received spam emails (#6010)
* Add possibility to add hidden posts with PostCreator

* FEATURE: Create hidden posts for received spam emails

Spamchecker usually have 3 results: HAM, SPAM and PROBABLY_SPAM
SPAM gets usually directly rejected and needs no further handling.
HAM is good message and usually gets passed unmodified.
PROBABLY_SPAM gets an additional header to allow further processing.
This change addes processing capabilities for such headers and marks
new posts created as hidden when received via email.
2018-07-05 11:07:46 +02:00
Jeff Wong 4599cc8435 FIX: PM participants listed inline 2018-06-11 18:14:25 -07:00
Arpit Jalan f9ab3848ed FEATURE: support disabling emails for non-staff users 2018-06-07 18:31:08 +05:30
Guo Xiang Tan 7fc8a36529 DEV: Take 2 Queue jobs in tests by default.
On my machine this cuts the time taken to run our test suite
from ~11mins to ~9mins.
2018-05-31 16:23:23 +08:00
Guo Xiang Tan 56e9ff6853 Revert "DEV: Queue jobs in tests by default."
Too risky for now

This reverts commit be28154d3b.
2018-05-31 15:34:46 +08:00
Guo Xiang Tan be28154d3b DEV: Queue jobs in tests by default. 2018-05-31 14:45:47 +08:00
Ryan Mulligan fac4bf2f85 ignore emails that are from the reply by email addresses (#5843) 2018-05-23 10:04:45 +02:00
Régis Hanol 86eb3528ec FEATURE: clearer error message when receiving a reply to an old notification 2018-05-09 18:51:01 +02:00
Régis Hanol 6b1ff0edd3 FIX: always update bounce score (instead of doing it once per day) 2018-05-09 16:40:52 +02:00
Jeff Wong 62a8904729
Feature: Include participants at the bottom of PM emails (#5797)
* Feature: Include participants at the bottom of PM emails

... as undecorated links.

https://meta.discourse.org/t/email-notification-recipients-unclear-when-pm-is-sent-to-multiple-users/26934/13?u=featheredtoast

Fix: missing translation for PM mentions

* display membership count as `group (count)`
2018-05-03 15:50:06 -07:00
Régis Hanol ddb092f397 FIX: update mail gem to fix UTF-8 parsing issue 2018-04-25 21:53:37 +02:00
Gerhard Schlager f2d00e5eff FEATURE: Use Message-ID for detecting email replies to group
Ignores the site setting "find_related_post_with_key" and always tries to honor the `In-Reply-To` and `References` header for emails sent to a group.

The senders email address must be included in the `To` or `CC` header of a previous email sent to the group and the `Message-ID` of that email must be included in the current email's `In-Reply-To` or `References` header.
2018-04-05 11:00:38 +02:00
Gerhard Schlager fcd352e089 FIX: Try fixing unparsable email addresses
The mail gem returns `UnstructuredField` when it fails to parse email addresses, but the `Receiver` always expects an `AddressList`.
2018-03-27 18:28:54 +02:00
Régis Hanol fd33090646 FEATURE: automatically elides gmail quotes 2018-02-26 23:54:02 +01:00
Régis Hanol 7a13e50aa6 fix build 2018-02-22 11:17:49 +01:00
Gerhard Schlager 97e19a7d02 Fix the build 2018-02-21 11:26:41 +01:00
Régis Hanol 61930e092a FIX: support incoming emails with just an attachment 2018-02-16 18:14:56 +01:00
Gerhard Schlager 9f7ae908d8 Add specs to check email domain whitelist/blacklist for To and Cc 2018-01-10 16:57:26 +01:00
Gerhard Schlager f086d28b30 FIX: Do not validate messages sent to mailing list mirror 2018-01-05 11:21:53 +01:00
Gerhard Schlager e0d73a957d FEATURE: Allow posting via email to read-only mailing list mirror category 2018-01-05 11:21:53 +01:00
Gerhard Schlager d7cd7e4dc7 FIX: Never mark emails sent to mailing list mirror as auto-generated 2018-01-05 11:21:53 +01:00
Gerhard Schlager ceb7590bcb FIX: bounced email can contain multiple status codes 2018-01-03 17:59:20 +01:00
Gerhard Schlager 16738cfb1b FEATURE: convert plain text emails to markdown 2017-12-06 01:47:51 +01:00
Gerhard Schlager 1a3ab7c02e ignore some site settings for emails sent to mailinglist mirror category 2017-11-17 15:29:14 +01:00
Gerhard Schlager aea161fabd suppress rejection email when email was sent to mailinglist mirror category 2017-11-17 15:29:14 +01:00
Robin Ward 971e302ff2 FEATURE: Support an end date for user silencing 2017-11-14 13:20:19 -05:00