Commit Graph

600 Commits

Author SHA1 Message Date
Martin Brennan 9ee8a01c3a
FIX: Change default for IncomingEmail#created_via to 0 (unknown) and make NOT NULL (#11782)
Follow up to https://review.discourse.org/t/dev-add-created-via-column-to-incomingemail-pr-11751/18366/6
2021-01-21 12:59:50 +10:00
Martin Brennan fb184fed06
DEV: Add created_via column to IncomingEmail (#11751)
This should make it easier to track down how the incoming email was created, which is one of four locations:

The POP3 poller (which picks up reply via email replies)
The admin email controller #handle_mail (which is where hosted mail is sent)
The IMAP sync tool
The group SMTP mailer, which sends emails when replying to IMAP topics, pre-emptively creating IncomingEmail records to avoid double syncing
2021-01-20 13:22:41 +10:00
Martin Brennan bd25627198
FIX: IMAP post alerter race condition and code improvements (#11711)
This PR fixes a race condition with the IMAP notification code. In the `Email::Receiver` we call the `NewPostManager` to create the post and enqueue jobs and sends alerts via `PostAlerter`. However, if the post alerter reaches the `notify_pm_users` and the `group_notifying_via_smtp` method _before_ the incoming email is updated with the post and topic, we unnecessarily send a notification to the person who just posted. The result of this is that the IMAP syncer re-imports the email sent to the user about their own post, which looks like this in the group inbox:

To fix this, we skip the jobs enqueued by `NewPostManager` and only enqueue them with `PostJobsEnqueuer` manually _after_ the incoming email record has been updated with the post and topic.

Other improvements:

* Moved code to calculate email addresses from `IncomingEmail` records into the topic, with a group passed in, for easier testing and debugging. It is not the responsibility of the post alerter to figure this stuff out.
* Add shortcut methods on `IncomingEmail` to split or provide an empty array for to and cc addresses to avoid repetition.
2021-01-15 10:54:46 +10:00
Martin Brennan 64ba5b1d21
FIX: Group SMTP email improvements (#11633)
Fixes a rare race condition causing the `Imap::Sync` class to create an incoming email and associated post/topic, which then kicks off the PostAlerter to notify others in the PM about a reply in the topic, but for the OP which is not necessary (because the person emailing the IMAP inbox already knows about the OP). Basically, we should never be sending the group SMTP email for the first post in a topic.

Also in this PR:

* Custom attribute accessors for the to/from/cc addresses on `IncomingEmail`, to parse them from an array to a joined string so the logic for this is only in one place.
* Store extra detail against the `IncomingEmail` created in `GroupSmtpMailer`
* regex test Mail header Reply-To as string instead of Field, which fixes `warning: deprecated Object#=~ is called on Mail::Field; it always returns nil`
* Add DEBUG_IMAP to log all IMAP logs as warnings for easier debugging
* Changed the Rails logging to `ImapSyncLog` in the `GroupSmtpMailer`
2021-01-05 15:32:04 +10:00
Martin Brennan 28db835c4c
FIX: Calculate email attachment size limit correctly (#11321)
When calculating whether the attached uploads went over the SiteSetting.email_total_attachment_size_limit_kb.kilobytes limit, we were using the original_upload for the calculation instead of the actually attached_upload, which will be smaller in most cases because it can be an optimized image.
2020-11-23 11:16:08 +10:00
Martin Brennan 879e4a9e29
FIX: Inline avatar style for onebox when embedding secure images (#11229)
When embedding secure images that are inline-avatars for oneboxes we weren't applying the correct sizing/style.
2020-11-16 09:58:40 +10:00
Martin Brennan 2bb7676ba4
FIX: Stop bypassing email checks for invite emails (#11113)
See #10794 for original context.

I did not mean to add invite to the BYPASS_TYPES for Email::Sender, it was supposed to be invite_password_instructions.
2020-11-04 09:18:22 +10:00
Rafael dos Santos Silva b70b3e867c
FIX: Safely skip secure_media steps when it's not enabled (#11110)
* FIX: Safely skip secure_media steps when it's not enabled

* DEV: Secure media tests should enable secure media
2020-11-03 13:53:15 -03:00
Kane York 789e3775df
FIX: Make all email subject vars available in notification subjects (#11064)
A site owner attempting to use both the email_subject site setting and translation overrides for normal post notification
email subjects would find themselves frusturated at the lack of template argument parity.
Make all the variables available for translation overrides by adding the subject variables to the custom interpolation keys list and applying them.

Reported at https://meta.discourse.org/t/customize-subject-format-for-standard-emails/20801/47?u=riking
2020-11-02 20:00:11 -08:00
Martin Brennan 3655062c60
FIX: Ensure oneboxed secure images which are optimized and also lightboxed optimized images are embedded in email (#11061)
We had an issue where onebox thumbnail was too large and thus was optimized, and we are using the image URLs in post to redact and re-embed, based on the sha1 in the URL. Optimized image URLs have extra stuff on the end like _99x99 so we were not parsing out the sha1 correctly. Another issue I found was for posts that have giant images, the original was being used to embed in the email and thus would basically never get included because it is huge.

For example the URL 787b17ea61_2_690x335.jpeg was not parsed correctly; we would end up with 787b17ea6140f4f022eb7f1509a692f2873cfe35_2_690x335.jpeg as the sha1 which would not find the image to re-embed that was already attached to the email.

This fix will use the first optimized image of the detected upload when we are redacting and then re-embedding to make sure we are not sending giant things in email. Also, I detect if it is a onebox thumbnail or the site icon and force appropriate sizes and styles.
2020-11-02 09:52:21 +10:00
Martin Brennan 632942e697
FIX: Ensure group SMTP and message builder always uses from address for Reply-To when IMAP is enabled (#11037)
There is a site setting reply_by_email_enabled which when combined with reply_by_email_address creates a Reply-To header in emails in the format "test+%{reply_key}@test.com" along with a PostReplyKey record, so when replying Discourse knows where to route the reply.

However this conflicts with the IMAP implementation. Since we are sending the email for a group via SMTP and from their actual email account, we want all replys to go to that email account as well so the IMAP sync job can pick them up and put them in the correct place. So if the group has IMAP enabled and configured, then the reply-to header will be correct.

This PR also makes a further fix to 64b0b50 by using the correct recipient user for the PostReplyKey record. If the post user is used we encounter this error:

if destination.user_id != user.id && !forwarded_reply_key?(destination, user)
  raise ReplyUserNotMatchingError, "post_reply_key.user_id => #{destination.user_id.inspect}, user.id => #{user.id.inspect}"
end
This is because the user above is found from the from_address, but the destination which is the PostReplyKey is made by the post.user, which will be different people.
2020-10-28 07:01:58 +10:00
Krzysztof Kotlarek b2481adb40
FIX: persist secure image width and height if is given (#10994)
`max-width: 50%; max-height: 400px;` is a good fallback, however, if width and height are given and are smaller than fallback -  we should persist that smaller size.
2020-10-22 13:25:09 +11:00
Martin Brennan 64b0b50ac0
FIX: Pass user to Email::Sender to avoid broken reply key for group_smtp email (#10978)
Our Email::Sender class accepts an optional user argument, which is used to create a PostReplyKey record when present. This record is used to sub out the %{reply_key} placeholder in the Reply-To mail header, so if we do not pass in the user we get a broken Reply-To header.

This is especially problematic in the IMAP group SMTP situation, because these emails go to customers that we are replying to, and when they reply to us the email bounces! This fixes the issue by passing user to the Email::Sender when sending a group_smtp email but there is still more to do in another PR.

This Email::Sender optional user is a bit of a footgun IMO, especially because most of the time we use it there is a user we can source. I would like to do another PR for this after this one to make the parameter not optional, so we don't end up with these reply issues down the line again.
2020-10-22 10:49:08 +10:00
Daniel Waterworth 48c3172913
DEV: Remove preserve_email_structure_when_styling setting (#10956)
This was made adjustable to allow rolling back quickly if problems came
up. The new behaviour was made default in 93137066 and no problems with
this have been reported.
2020-10-19 12:19:30 +01:00
Martin Brennan f63da1c146
FIX: Confirm new email not sent for staff if email disabled with "non-staff" option (#10794)
See https://meta.discourse.org/t/email-address-change-confirmation-email-not-sent-but-every-other-notification-emails-are/165358

In short: with disable emails set to non-staff, email address change confirmation emails (those sent to the new address) are not sent for staff or admin members.

This was happening because we were looking up the staff user with the to_address of the email, but the to address was the new email address because we are sending a confirm email change email, and thus the user could not be found. We didn't need to do this anyway because we are passing the user into the Email::Sender class anyway.
2020-10-08 13:52:17 +10:00
Gerhard Schlager 99181bb3b8
FIX: Stripping lines from incoming email shouldn't fail for blank body (#10800) 2020-10-02 15:44:35 +02:00
Martin Brennan a8ed0b4612
FIX: Correct corrupt encoding in emails containing attachments 2020-09-29 14:10:57 +10:00
Arpit Jalan 376255ed84 UX: add width and height to GitHub onebox inline avatar in emails. 2020-09-21 11:47:09 +05:30
Martin Brennan dede942007
FEATURE: Allow email image embed with secure media (#10563)
This PR introduces a few important changes to secure media redaction in emails. First of all, two new site settings have been introduced:

* `secure_media_allow_embed_images_in_emails`: If enabled we will embed secure images in emails instead of redacting them.
* `secure_media_max_email_embed_image_size_kb`: The cap to the size of the secure image we will embed, defaulting to 1mb, so the email does not become too big. Max is 10mb. Works in tandem with `email_total_attachment_size_limit_kb`.

`Email::Sender` will now attach images to the email based on these settings. The sender will also call `inline_secure_images` in `Email::Styles` after secure media is redacted and attachments are added to replace redaction messages with attached images. I went with attachment and `cid` URLs because base64 image support is _still_ flaky in email clients.

All redaction of secure media is now handled in `Email::Styles` and calls out to `PrettyText.strip_secure_media` to do the actual stripping and replacing with placeholders. `app/mailers/group_smtp_mailer.rb` and `app/mailers/user_notifications.rb` no longer do any stripping because they are earlier in the pipeline than `Email::Styles`.

Finally the redaction notice has been restyled and includes a link to the media that the user can click, which will show it to them if they have the necessary permissions.

![image](https://user-images.githubusercontent.com/920448/92341012-b9a2c380-f0ff-11ea-860e-b376b4528357.png)
2020-09-10 09:50:16 +10:00
Martin Brennan 95b71b35d6
FEATURE: IMAP delete email sync for group inboxes (#10392)
Adds functionality to reflect topic delete in Discourse to IMAP inbox (Gmail only for now) and reflecting Gmail deletes in Discourse.

Adding lots of tests, various refactors and code improvements.

When Discourse topic is destroyed in PostDestroyer mark the topic incoming email as imap_sync: true, and do the opposite when post is recovered.
2020-08-12 10:16:26 +10:00
Martin Brennan ed68ac054d
DEV: Cover other case for generated message ID in Email::Receiver 2020-08-03 13:37:06 +10:00
Martin Brennan 7f55ed1a4a
DEV: Clarify Email::Receiver existing incoming email behaviour 2020-08-03 13:31:34 +10:00
Martin Brennan 2920988b3a
FIX: IMAP sync email update uniqueness across groups and minor improvements (#10332)
Adds a imap_group_id column to IncomingEmail to deal with an issue where we were trying to update emails in the mailbox, calling IncomingEmail.where(imap_sync: true). However UID and UIDVALIDITY could be the same across accounts. So if group A used IMAP details for Gmail account A, and group B used IMAP details for Gmail account B, and both tried to sync changes to an email with UID of 3 (e.g. changing Labels), one account could affect the other. This even applied to Archiving!

Also in this PR:

* Fix error occurring if we do a uid_fetch and no emails are returned
* Allow for creating labels within the target mailbox (previously we would not do this, only use existing labels)
* Improve consistency for log messages
* Add specs for generic IMAP provider (Gmail specs still to come)
* Add custom archiving support for Gmail
* Only use Message-ID for uniqueness of IncomingEmail if it was generated by us
* Various refactors and improvements
2020-08-03 13:10:17 +10:00
Krzysztof Kotlarek e0d9232259
FIX: use allowlist and blocklist terminology (#10209)
This is a PR of the renaming whitelist to allowlist and blacklist to the blocklist.
2020-07-27 10:23:54 +10:00
Bianca Nenciu 22fdd5dfda
FIX: Improve email styling of code blocks (#10248)
Long numbered code lines were not rendered correctly in some email
clients.
2020-07-22 18:26:14 +03:00
Dan Ungureanu c72bc27888
FEATURE: Implement support for IMAP and SMTP email protocols. (#8301)
Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
2020-07-10 12:05:55 +03:00
Daniel Waterworth 8d5750d90a FIX: Catch all kinds of exceptions when processing email 2020-07-09 13:41:51 +01:00
Bianca Nenciu bd842cd2b0
FEATURE: Parse images in email signatures (#10137)
* FEATURE: Parse images in email signatures

* DEV: Fix tests

* Code review
2020-07-08 15:50:30 +10:00
Daniel Waterworth 1bd8a075d8 FIX: Make Email::Styles operate on html documents instead of fragments
`Nokogiri::HTML.fragment` is a huge hack (a comment in the source code
admits this). The current behavior of `Email::Styles` is to try to
emulate `fragment` using nokogumbo, but it misses some edge cases. In
particular, meta tags in a email template don't make it through to the
final email.

Instead of treating the provided HTML as an indeterminate fragment, this
commit makes `Email::Styles` treat the HTML as a complete document. This
means that the generated HTML for an email will now always contain top
level structure (a doctype, html, head and body tags).

This new behavior is behind a hidden site setting for now and defaults
off.
2020-07-06 11:45:39 +01:00
Gerhard Schlager 2da6faf281 FEATURE: Add `before_email_send` event
Plugins can use it to customize the message (e.g. add header) before the email is sent.
2020-06-18 18:58:51 +02:00
Mark VanLandingham a047004c9a
FIX: Specific email error for replies to digest emails (#9770) 2020-05-14 09:04:58 -05:00
Krzysztof Kotlarek 9bff0882c3
FEATURE: Nokogumbo (#9577)
* FEATURE: Nokogumbo

Use Nokogumbo HTML parser.
2020-05-05 13:46:57 +10:00
Penar Musaraj 5706cab897
FIX: Improve digest email styling on Outlook 2016 (#9626) 2020-05-04 14:07:03 -04:00
Sam Saffron d0d5a138c3
DEV: stop freezing frozen strings
We have the `# frozen_string_literal: true` comment on all our
files. This means all string literals are frozen. There is no need
to call #freeze on any literals.

For files with `# frozen_string_literal: true`

```
puts %w{a b}[0].frozen?
=> true

puts "hi".frozen?
=> true

puts "a #{1} b".frozen?
=> true

puts ("a " + "b").frozen?
=> false

puts (-("a " + "b")).frozen?
=> true
```

For more details see: https://samsaffron.com/archive/2018/02/16/reducing-string-duplication-in-ruby
2020-04-30 16:48:53 +10:00
Bianca Nenciu ba1a08510e
FIX: Use correct spacing in emails with code (#9274)
Follow-up-to 778454e26b
2020-03-26 14:24:07 +02:00
Bianca Nenciu 778454e26b
FIX: Condense line codes in emails (#9225) 2020-03-18 16:21:24 +02:00
Martin Brennan 2237ba8c9d
FIX: Add topic deleted check to email/sender (#9166)
It already had a deleted post check and log reason, add a topic one too to avoid errors
2020-03-13 10:04:15 +10:00
Daniel Waterworth 59578dfc5b FIX: Notification emails with attachments are incorrectly structured
Two behaviors in the mail gem collide:

 1. Attachments are added as extra parts at the top level,
 2. When there are both text and html parts, the content type is set to
    'multipart/alternative'.

Since attachments aren't alternative renderings, for emails that contain
attachments and both html and text parts, some coercing is necessary.
2020-03-12 15:42:24 +00:00
Arpit Jalan 7b92280b97 UX: use same styling for username and user-name 2020-02-20 17:51:37 +05:30
Arpit Jalan f36719c1f5 FIX: respect prioritize_username_in_ux setting in email
UX: only the first attribute should be hyperlinked
UX: add margin based on attribute position
2020-02-20 17:47:16 +05:30
Gerhard Schlager 453bec9394 FEATURE: Add small action post to indicate forwarded email
This happens only when the sender of the email didn't write anything in their email.
2020-02-12 16:23:17 +01:00
Gerhard Schlager e84d88ddea FIX: Email attachments with a size of 0 bytes caused error 2020-02-04 12:30:43 +01:00
Arpit Jalan 62c21ba649 Remove `bounce_score_threshold_deactivate` setting.
Removed `bounce_score_threshold_deactivate` setting as the deactivate
threshold is not getting reached.
2020-01-30 16:17:31 +05:30
Martin Brennan 1150cd4621
FIX: Stop secure media URLs being censored too liberally in emails (#8817)
For example /t/ URLs were being replaced if they contained secure-media-uploads so if you made a topic called "Secure Media Uploads Are Cool" the View Topic link in the user notifications would be stripped out.

Refactored code so this secure URL detection happens in one place.
2020-01-30 16:19:14 +10:00
Neil Lalonde 34f564acd6 FIX: customized email subjects was ignored for some notifications
Email templates for notifications about new posts and topics weren't
using customized values.
2020-01-29 14:49:08 -05:00
Martin Brennan 45b37a8bd1
FIX: Resolve pull hotlinked image and broken link issues for secure media URLs (#8777)
When pull_hotlinked_images tried to run on posts with secure media (which had already been downloaded from external sources) we were getting a 404 when trying to download the image because the secure endpoint doesn't allow anon downloads.

Also, we were getting into an infinite loop of pull_hotlinked_images because the job didn't consider the secure media URLs as "downloaded" already so it kept trying to download them over and over.

In this PR I have also refactored secure-media-upload URL checks and mutations into single source of truth in Upload, adding a SECURE_MEDIA_ROUTE constant to check URLs against too.
2020-01-24 11:59:30 +10:00
Leo McArdle 8883cca373 enqueue spam/dmarc failing emails instead of hiding (#8674)
* enqueue spam/dmarc failing emails instead of hiding

* add translations for dmarc/spam enqueued reasons

* unescape quote

* if email_in_authserv_id is blank return gray for all emails
2020-01-21 11:12:00 -05:00
Gerhard Schlager ab07b945c2
Merge pull request #8736 from gschlager/rename_reply_id_column
REFACTOR: Rename `post_replies.reply_id` column to `post_replies.reply_post_id`
2020-01-17 17:24:49 +01:00
Sam Saffron 0c52537f10 DEV: update rubocop to version 0.77
We like to stay as close as possible to latest with rubocop cause the cops
get better.

This update required some code changes, specifically the default is to avoid
explicit returns where implicit is done

Also this renames a few rules
2019-12-10 11:48:39 +11:00
Gerhard Schlager c88797bf0e FIX: Ignore DMARC for emails sent to mailing list mirror 2019-12-06 13:29:39 +01:00
Joffrey JAFFEUX 0d3d2c43a0
DEV: s/\$redis/Discourse\.redis (#8431)
This commit also adds a rubocop rule to prevent global variables.
2019-12-03 10:05:53 +01:00
Leo McArdle 2714149fd2 FEATURE: hide posts from incoming email based on dmarc verdict (#8333) 2019-11-26 15:55:22 +01:00
Penar Musaraj 102909edb3 FEATURE: Add support for secure media (#7888)
This PR introduces a new secure media setting. When enabled, it prevent unathorized access to media uploads (files of type image, video and audio). When the `login_required` setting is enabled, then all media uploads will be protected from unauthorized (anonymous) access. When `login_required`is disabled, only media in private messages will be protected from unauthorized access. 

A few notes: 

- the `prevent_anons_from_downloading_files` setting no longer applies to audio and video uploads
- the `secure_media` setting can only be enabled if S3 uploads are already enabled and configured
- upload records have a new column, `secure`, which is a boolean `true/false` of the upload's secure status
- when creating a public post with an upload that has already been uploaded and is marked as secure, the post creator will raise an error
- when enabling or disabling the setting on a site with existing uploads, the rake task `uploads:ensure_correct_acl` should be used to update all uploads' secure status and their ACL on S3
2019-11-18 11:25:42 +10:00
Krzysztof Kotlarek c32bd8ae48 FEATURE: Remove attachments and truncate raw field for incoming emails (#8253)
Adds the settings: 

raw_email_max_length, raw_rejected_email_max_length, delete_rejected_email_after_days. 

These settings control retention of the "raw" emails logs.

raw_email_max_length ensures that if we get incoming email that is huge we will truncate it removing uploads from the raw log.

raw_rejected_email_max_length introduces an even more aggressive truncation for rejected incoming mail. 

delete_rejected_email_after_days controls how many days we will keep rejected emails for (default 90)
2019-10-30 16:54:35 +11:00
Leo McArdle e7ff6809a3 FEATURE: add SES spam header to recognised spam headers (#8254) 2019-10-28 12:46:53 -04:00
Neil Lalonde f061aee818 FEATURE: support SCSS in custom email style
In the CSS tab of Admin > Customize > Email Style, SCSS can now be used.
2019-10-23 15:42:37 -04:00
David Taylor 815dcb25ba UX: Improve new GitHub onebox rendering in emails
Unfortunately SVGs are not supported in most email clients, so remove them
2019-10-17 10:54:04 +01:00
Sam Saffron 5aaf7e3316 FIX: during concurrent emails generation renderer should not be reused
Our instance used for template rendering needs a lock to ensure there is
no race condition where rendering happens on 2 threads at the same time.

This can lead to local poisoning which can cause unexpected results in
emails
2019-10-10 08:50:48 +11:00
Sam Saffron 71ea4ad7fc PERF: reuse renderer when rendering email templates
Previous to this fix we were leaking methods on the internal action view
template class per render.

This caused email generation to be very low and a steady memory leak in the
application in sidekiq when sending out emails

The behavior change is new to Rails 6 so this fix does not need to be
backported into stable.
2019-10-06 23:57:03 -04:00
Krzysztof Kotlarek 427d54b2b0 DEV: Upgrading Discourse to Zeitwerk (#8098)
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains. 

We no longer need to use Rails "require_dependency" anywhere and instead can just use standard 
Ruby patterns to require files.

This is a far reaching change and we expect some followups here.
2019-10-02 14:01:53 +10:00
Sam Saffron d05b724c4e FIX: correct race condition loading library
In some very rare cases CssParser could be loaded but CssParser::Parser not
this ensures we check for the actual constant we plan to call for concurrent
digest generations
2019-08-15 17:16:41 +10:00
Leo McArdle 6296ae3d31 FEATURE: add setting to show content of forwarded emails in topics (#7935) 2019-08-07 12:32:19 +02:00
Neil Lalonde 43365a2bf1 Fix some broken styles 2019-07-30 16:46:20 -04:00
Neil Lalonde 9656a21fdb
FEATURE: customization of html emails (#7934)
This feature adds the ability to customize the HTML part of all emails using a custom HTML template and optionally some CSS to style it. The CSS will be parsed and converted into inline styles because CSS is poorly supported by email clients. When writing the custom HTML and CSS, be aware of what email clients support. Keep customizations very simple.

Customizations can be added and edited in Admin > Customize > Email Style.

Since the summary email is already heavily styled, there is a setting to disable custom styles for summary emails called "apply custom styles to digest" found in Admin > Settings > Email.

As part of this work, RTL locales are now rendered correctly for all emails.
2019-07-30 15:05:08 -04:00
Gerhard Schlager fd12c414e7 DEV: Refactor helper methods for upload markdown
Follow-up to a61ff167
2019-07-25 16:36:35 +02:00
Gerhard Schlager 7e0eeed292 FEATURE: Add attachments to outgoing emails
This feature is off by default and can can be configured with the `email_total_attachment_size_limit_kb` site setting.

Co-authored-by: Maja Komel <maja.komel@gmail.com>
2019-07-25 15:57:45 +02:00
Gerhard Schlager a61ff16740 DEV: Make attachment markdown reusable 2019-07-25 14:04:18 +02:00
David Taylor b3e5f7a8c6 SECURITY: Sanitize email id for use as mutex key 2019-07-24 13:45:02 +01:00
Guo Xiang Tan 40e67971f9 DEV: Add spec for `Email::Sender` for upload links in plain text emails. 2019-06-11 16:02:24 +08:00
Guo Xiang Tan 42ab016856 FIX: Use markdown for images and attachments in `Email::Receiver`. 2019-06-11 14:49:46 +08:00
Maja Komel 87d3b86484 FIX: better error message when user without permissions replies via email 2019-06-04 16:39:41 +08:00
Sam Saffron 30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Gerhard Schlager 4a1755b780 FIX: Don't create staged users when processing forwarded email fails 2019-05-09 23:47:47 +02:00
Guo Xiang Tan 152238b4cf DEV: Prefer `public_send` over `send`. 2019-05-07 09:33:21 +08:00
David Taylor 0644c10bfe FIX: Prioritize VERP key over final_recipient header in bounced email
Per RFC3464 2.3.2, the final_recipient header may not match the address we originally sent the email to.
2019-05-03 12:12:44 +01:00
Sam Saffron 1be01f8dd4 DEV: Add support for Rails 6
Minor fixes to add Rails 6 support to Discourse, we now will boot
with RAILS_MASTER=1, all specs pass

Only one tiny deprecation left

Largest change was the way ActiveModel:Errors changed interface a
bit but there is a simple backwards compat way of working it
2019-05-02 16:23:25 +10:00
Sam Saffron 0a5a6dfded DEV: stop mutating inputs as a side effect
We had quite a few cases in core where inputs are being mutated as a side
effect of calling a method.

This handles all the cases where specs caught this.

Mutating inputs makes code harder to reason about. Eg:

```
frog = "frog"
jump(frog)
puts frog
"fly" # ?????
```

This commit is part of a followup commit that adds # frozen_string_literal
to all our specs.
2019-04-30 10:25:53 +10:00
Sam Saffron 45285f1477 DEV: remove update_attributes which is deprecated in Rails 6
See: https://github.com/rails/rails/pull/31998

update_attributes is a relic of the past, it should no longer be used.
2019-04-29 17:32:25 +10:00
Arpit Jalan bd1db1860a FIX: prefer data-original-href attribute to get iframe URL 2019-04-24 13:53:27 +05:30
Dan Ungureanu 35a866fe22
FIX: Do not strip email lines having lists. 2019-04-16 11:39:35 +03:00
Dan Ungureanu e92cd5318b FEATURE: Add setting to strip whitespaces from incoming emails. (#7375)
Some email clients add leading whitespaces which get are transformed in
code blocks when processed.
2019-04-15 16:26:00 +10:00
Guo Xiang Tan bac1bcc79f PERF: Add `index_reply_id_on_post_replies`.
Speeds up the reference posts query in `Email::Sender#send`.
2019-04-10 13:58:29 +08:00
David Taylor 6a05f190c6
PERF: Do not create staged users for most rejected incoming emails (#7301)
Previously we would create users, then destroy them at the end of the job if the post was rejected. Now we do not create users unless required.
2019-04-08 10:36:39 +01:00
Robin Ward b58867b6e9 FEATURE: New 'Reviewable' model to make reviewable items generic
Includes support for flags, reviewable users and queued posts, with REST API
backwards compatibility.

Co-Authored-By: romanrizzi <romanalejandro@gmail.com>
Co-Authored-By: jjaffeux <j.jaffeux@gmail.com>
2019-03-28 12:45:10 -04:00
Gerhard Schlager 4f04ae5692 FIX: Failed to show details about some bounced emails
Bounces sent to reply_by_email_address could not be found.
2019-03-26 18:00:27 +01:00
David Taylor a9d5ffbe3d FIX: Prevent critical emails bypassing disable, and improve email test logic
- The test_email job is removed, because it was always being run synchronously (not in sidekiq)
- 34b29f62 added a bypass for critical emails, to match the spec. This removes the bypass, and removes the spec.
- This adapts the specs for 72ffabf6, so that they check for emails being sent
- This reimplements c2797921, allowing test emails to be sent even when emails are disabled
2019-03-22 17:28:43 +08:00
David Taylor 3f9e7eb326 FIX: Respect the disable_emails=non-staff site setting correctly
This reverts commit c279792130.

This commit inadvertently removed all of the non-staff email logic, rather than just for the 'test email' button. 

https://meta.discourse.org/t/112231/5
2019-03-21 21:44:14 +00:00
Penar Musaraj 9334d2f4f7
FEATURE: add more granular user option levels for email notifications (#7143)
Migrates email user options to a new data structure, where `email_always`, `email_direct` and `email_private_messages` are replace by

* `email_messages_level`, with options: `always`, `only_when_away` and `never` (defaults to `always`)
* `email_level`, with options: `always`, `only_when_away` and `never` (defaults to `only_when_away`)
2019-03-15 10:55:11 -04:00
David Taylor 420c6f8102
FEATURE: Skip sending emails to domains on the `.invalid` TLD (#7162)
This is a reserved TLD which we use when importing users without an email address. https://tools.ietf.org/html/rfc2606
2019-03-13 16:17:59 +00:00
Guo Xiang Tan 34b29f62db DEV: Remove the use of stubs and mocks in `Jobs::UserEmail` tests.
We can only be sure that an email is sent when we get a mailer in
`ActionMailer::Deliveries`. A couple of tests were actually incorrect
because it didn't flow through our email sender where there are more
conditions in determining whether an email is sent or not.
2019-03-12 09:39:16 +08:00
Sam b2187301fd FEATURE: allow users to easily track/watch/mute topics via email
If you reply to an email with the word "mute" a topic will be muted
If you reply to an email with the word "track" a topic will be tracked
If you reply to an email with the word "watch" a topic will be watched

These ninja command can help advanced mailing list ex-users, saves a trip
to the website
2019-03-06 18:38:49 +11:00
Arpit Jalan 1b2bd36061 UX: limit onebox thumbnail image size in emails 2019-03-05 23:07:44 +05:30
Régis Hanol 51fdf7a11d FIX: don't duplicate attachments 2019-01-28 18:40:52 +01:00
Régis Hanol db76e5a368 FIX: properly handle attachments in received emails
This regressed in 1ac3e547 when we added support for eml attachments.
2019-01-25 19:13:34 +01:00
Gerhard Schlager 197e3fd722 FIX: Keep original subject in emails to staged users
Renaming a topic shouldn't be visibile to staged users when the topic was created via email.
2019-01-18 11:07:54 +01:00
Gerhard Schlager c0a8bb9a91 FEATURE: Include "via <site_name>" in email From header 2019-01-04 17:06:19 +01:00
Rishabh c279792130 FIX: Allow sending test e-mails to any email address when disable_email is set to non-staff (#6792) 2018-12-18 16:12:05 +01:00
Vinoth Kannan fc0b7c9e26 FIX: incoming email matches the wrong user if null bounce key available in db 2018-11-30 12:29:51 +05:30
Vinoth Kannan bfb3c4d9f9 DEV: create bounce alert earlier if email_log detected from bounce_key 2018-11-28 21:13:06 +05:30
Vinoth Kannan 25253dec56 FIX: Get email address from email_log if bounced with verp
We can not access mail.final_recipient attr if it bounced with verp
2018-11-28 19:04:09 +05:30