Commit Graph

546 Commits

Author SHA1 Message Date
Leo McArdle e7ff6809a3 FEATURE: add SES spam header to recognised spam headers (#8254) 2019-10-28 12:46:53 -04:00
Neil Lalonde f061aee818 FEATURE: support SCSS in custom email style
In the CSS tab of Admin > Customize > Email Style, SCSS can now be used.
2019-10-23 15:42:37 -04:00
David Taylor 815dcb25ba UX: Improve new GitHub onebox rendering in emails
Unfortunately SVGs are not supported in most email clients, so remove them
2019-10-17 10:54:04 +01:00
Sam Saffron 5aaf7e3316 FIX: during concurrent emails generation renderer should not be reused
Our instance used for template rendering needs a lock to ensure there is
no race condition where rendering happens on 2 threads at the same time.

This can lead to local poisoning which can cause unexpected results in
emails
2019-10-10 08:50:48 +11:00
Sam Saffron 71ea4ad7fc PERF: reuse renderer when rendering email templates
Previous to this fix we were leaking methods on the internal action view
template class per render.

This caused email generation to be very low and a steady memory leak in the
application in sidekiq when sending out emails

The behavior change is new to Rails 6 so this fix does not need to be
backported into stable.
2019-10-06 23:57:03 -04:00
Krzysztof Kotlarek 427d54b2b0 DEV: Upgrading Discourse to Zeitwerk (#8098)
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains. 

We no longer need to use Rails "require_dependency" anywhere and instead can just use standard 
Ruby patterns to require files.

This is a far reaching change and we expect some followups here.
2019-10-02 14:01:53 +10:00
Sam Saffron d05b724c4e FIX: correct race condition loading library
In some very rare cases CssParser could be loaded but CssParser::Parser not
this ensures we check for the actual constant we plan to call for concurrent
digest generations
2019-08-15 17:16:41 +10:00
Leo McArdle 6296ae3d31 FEATURE: add setting to show content of forwarded emails in topics (#7935) 2019-08-07 12:32:19 +02:00
Neil Lalonde 43365a2bf1 Fix some broken styles 2019-07-30 16:46:20 -04:00
Neil Lalonde 9656a21fdb
FEATURE: customization of html emails (#7934)
This feature adds the ability to customize the HTML part of all emails using a custom HTML template and optionally some CSS to style it. The CSS will be parsed and converted into inline styles because CSS is poorly supported by email clients. When writing the custom HTML and CSS, be aware of what email clients support. Keep customizations very simple.

Customizations can be added and edited in Admin > Customize > Email Style.

Since the summary email is already heavily styled, there is a setting to disable custom styles for summary emails called "apply custom styles to digest" found in Admin > Settings > Email.

As part of this work, RTL locales are now rendered correctly for all emails.
2019-07-30 15:05:08 -04:00
Gerhard Schlager fd12c414e7 DEV: Refactor helper methods for upload markdown
Follow-up to a61ff167
2019-07-25 16:36:35 +02:00
Gerhard Schlager 7e0eeed292 FEATURE: Add attachments to outgoing emails
This feature is off by default and can can be configured with the `email_total_attachment_size_limit_kb` site setting.

Co-authored-by: Maja Komel <maja.komel@gmail.com>
2019-07-25 15:57:45 +02:00
Gerhard Schlager a61ff16740 DEV: Make attachment markdown reusable 2019-07-25 14:04:18 +02:00
David Taylor b3e5f7a8c6 SECURITY: Sanitize email id for use as mutex key 2019-07-24 13:45:02 +01:00
Guo Xiang Tan 40e67971f9 DEV: Add spec for `Email::Sender` for upload links in plain text emails. 2019-06-11 16:02:24 +08:00
Guo Xiang Tan 42ab016856 FIX: Use markdown for images and attachments in `Email::Receiver`. 2019-06-11 14:49:46 +08:00
Maja Komel 87d3b86484 FIX: better error message when user without permissions replies via email 2019-06-04 16:39:41 +08:00
Sam Saffron 30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Gerhard Schlager 4a1755b780 FIX: Don't create staged users when processing forwarded email fails 2019-05-09 23:47:47 +02:00
Guo Xiang Tan 152238b4cf DEV: Prefer `public_send` over `send`. 2019-05-07 09:33:21 +08:00
David Taylor 0644c10bfe FIX: Prioritize VERP key over final_recipient header in bounced email
Per RFC3464 2.3.2, the final_recipient header may not match the address we originally sent the email to.
2019-05-03 12:12:44 +01:00
Sam Saffron 1be01f8dd4 DEV: Add support for Rails 6
Minor fixes to add Rails 6 support to Discourse, we now will boot
with RAILS_MASTER=1, all specs pass

Only one tiny deprecation left

Largest change was the way ActiveModel:Errors changed interface a
bit but there is a simple backwards compat way of working it
2019-05-02 16:23:25 +10:00
Sam Saffron 0a5a6dfded DEV: stop mutating inputs as a side effect
We had quite a few cases in core where inputs are being mutated as a side
effect of calling a method.

This handles all the cases where specs caught this.

Mutating inputs makes code harder to reason about. Eg:

```
frog = "frog"
jump(frog)
puts frog
"fly" # ?????
```

This commit is part of a followup commit that adds # frozen_string_literal
to all our specs.
2019-04-30 10:25:53 +10:00
Sam Saffron 45285f1477 DEV: remove update_attributes which is deprecated in Rails 6
See: https://github.com/rails/rails/pull/31998

update_attributes is a relic of the past, it should no longer be used.
2019-04-29 17:32:25 +10:00
Arpit Jalan bd1db1860a FIX: prefer data-original-href attribute to get iframe URL 2019-04-24 13:53:27 +05:30
Dan Ungureanu 35a866fe22
FIX: Do not strip email lines having lists. 2019-04-16 11:39:35 +03:00
Dan Ungureanu e92cd5318b FEATURE: Add setting to strip whitespaces from incoming emails. (#7375)
Some email clients add leading whitespaces which get are transformed in
code blocks when processed.
2019-04-15 16:26:00 +10:00
Guo Xiang Tan bac1bcc79f PERF: Add `index_reply_id_on_post_replies`.
Speeds up the reference posts query in `Email::Sender#send`.
2019-04-10 13:58:29 +08:00
David Taylor 6a05f190c6
PERF: Do not create staged users for most rejected incoming emails (#7301)
Previously we would create users, then destroy them at the end of the job if the post was rejected. Now we do not create users unless required.
2019-04-08 10:36:39 +01:00
Robin Ward b58867b6e9 FEATURE: New 'Reviewable' model to make reviewable items generic
Includes support for flags, reviewable users and queued posts, with REST API
backwards compatibility.

Co-Authored-By: romanrizzi <romanalejandro@gmail.com>
Co-Authored-By: jjaffeux <j.jaffeux@gmail.com>
2019-03-28 12:45:10 -04:00
Gerhard Schlager 4f04ae5692 FIX: Failed to show details about some bounced emails
Bounces sent to reply_by_email_address could not be found.
2019-03-26 18:00:27 +01:00
David Taylor a9d5ffbe3d FIX: Prevent critical emails bypassing disable, and improve email test logic
- The test_email job is removed, because it was always being run synchronously (not in sidekiq)
- 34b29f62 added a bypass for critical emails, to match the spec. This removes the bypass, and removes the spec.
- This adapts the specs for 72ffabf6, so that they check for emails being sent
- This reimplements c2797921, allowing test emails to be sent even when emails are disabled
2019-03-22 17:28:43 +08:00
David Taylor 3f9e7eb326 FIX: Respect the disable_emails=non-staff site setting correctly
This reverts commit c279792130.

This commit inadvertently removed all of the non-staff email logic, rather than just for the 'test email' button. 

https://meta.discourse.org/t/112231/5
2019-03-21 21:44:14 +00:00
Penar Musaraj 9334d2f4f7
FEATURE: add more granular user option levels for email notifications (#7143)
Migrates email user options to a new data structure, where `email_always`, `email_direct` and `email_private_messages` are replace by

* `email_messages_level`, with options: `always`, `only_when_away` and `never` (defaults to `always`)
* `email_level`, with options: `always`, `only_when_away` and `never` (defaults to `only_when_away`)
2019-03-15 10:55:11 -04:00
David Taylor 420c6f8102
FEATURE: Skip sending emails to domains on the `.invalid` TLD (#7162)
This is a reserved TLD which we use when importing users without an email address. https://tools.ietf.org/html/rfc2606
2019-03-13 16:17:59 +00:00
Guo Xiang Tan 34b29f62db DEV: Remove the use of stubs and mocks in `Jobs::UserEmail` tests.
We can only be sure that an email is sent when we get a mailer in
`ActionMailer::Deliveries`. A couple of tests were actually incorrect
because it didn't flow through our email sender where there are more
conditions in determining whether an email is sent or not.
2019-03-12 09:39:16 +08:00
Sam b2187301fd FEATURE: allow users to easily track/watch/mute topics via email
If you reply to an email with the word "mute" a topic will be muted
If you reply to an email with the word "track" a topic will be tracked
If you reply to an email with the word "watch" a topic will be watched

These ninja command can help advanced mailing list ex-users, saves a trip
to the website
2019-03-06 18:38:49 +11:00
Arpit Jalan 1b2bd36061 UX: limit onebox thumbnail image size in emails 2019-03-05 23:07:44 +05:30
Régis Hanol 51fdf7a11d FIX: don't duplicate attachments 2019-01-28 18:40:52 +01:00
Régis Hanol db76e5a368 FIX: properly handle attachments in received emails
This regressed in 1ac3e547 when we added support for eml attachments.
2019-01-25 19:13:34 +01:00
Gerhard Schlager 197e3fd722 FIX: Keep original subject in emails to staged users
Renaming a topic shouldn't be visibile to staged users when the topic was created via email.
2019-01-18 11:07:54 +01:00
Gerhard Schlager c0a8bb9a91 FEATURE: Include "via <site_name>" in email From header 2019-01-04 17:06:19 +01:00
Rishabh c279792130 FIX: Allow sending test e-mails to any email address when disable_email is set to non-staff (#6792) 2018-12-18 16:12:05 +01:00
Vinoth Kannan fc0b7c9e26 FIX: incoming email matches the wrong user if null bounce key available in db 2018-11-30 12:29:51 +05:30
Vinoth Kannan bfb3c4d9f9 DEV: create bounce alert earlier if email_log detected from bounce_key 2018-11-28 21:13:06 +05:30
Vinoth Kannan 25253dec56 FIX: Get email address from email_log if bounced with verp
We can not access mail.final_recipient attr if it bounced with verp
2018-11-28 19:04:09 +05:30
Vinoth Kannan 7dbf709467 FIX: create whisper post in PMs when bounces with verp and user is staged 2018-11-28 08:24:23 +05:30
Vinoth Kannan cedd2118c4
FEATURE: If PM email bounced for staged user then alert in whisper reply (#6648) 2018-11-27 00:29:37 +05:30
Régis Hanol 6b51d84dc5 FIX: Don't enqueue topics if the user can't create them
Co-authored-by: Vinoth Kannan <vinothkannan@vinkas.com>
2018-11-09 18:24:28 +01:00
Maja Komel 1ac3e5473a FIX: don't strip eml attachments from received emails 2018-11-05 09:35:22 +01:00
Gerhard Schlager 341836eb42 Fix the rake task and importer instead 2018-10-17 16:48:09 +02:00
Gerhard Schlager ee18d9ace0 FIX: mbox importer and rake task were broken 2018-10-17 16:34:18 +02:00
Penar Musaraj b06dccac49 FIX: force enable a user's email_private_messages option when user replies via email (#6478)
* Enable user email PM when posting to group or replying to topic via email

* remove extra line

* Add test and fix snake_case

* Only reenable email_private_messages for PM replies
2018-10-16 10:51:57 +11:00
Gerhard Schlager 7a41a783a4 FIX: Don't reply to Unsubscribe email sent to mailing list mirror 2018-10-11 16:09:22 +02:00
Guo Xiang Tan c92bda95e1 Fix the build. 2018-10-11 10:45:01 +08:00
Guo Xiang Tan 5039a6c3f1 FIX: Strip null bytes in mail subjects. 2018-10-11 09:46:32 +08:00
Kyle Zhao 9bbc1ae7b2 FIX: raise if staged user creation failed (#6461) 2018-10-08 15:45:23 +08:00
Maja Komel 361ad7ed2b FEATURE: add indication if incoming email attachment was rejected and inform sender about it (#6376)
* FEATURE: add indication if incoming email attachment was rejected and inform sender about it

* include errors for rejected attachments in email

* don't send warning email to staged users

* use user object instead of user_id in add_attachments method
2018-10-04 22:08:28 +08:00
Penar Musaraj da9eee5262 FIX: Force enable user PM emails option when user posts to a group by email. 2018-10-02 12:38:10 +08:00
Guo Xiang Tan e1b16e445e Rename `FileHelper.is_image?` -> `FileHelper.is_supported_image?`. 2018-09-12 09:22:28 +08:00
Guo Xiang Tan 8dc1463ab3 Enable `Lint/ShadowingOuterLocalVariable` for Rubocop. 2018-09-04 10:16:42 +08:00
Gerhard Schlager eeedc3901e FIX: Replying to deleted post via email should create new reply to topic 2018-09-03 23:06:40 +02:00
Guo Xiang Tan 36a7028f19 FEATURE: Clean up `PostReplyKey` records.
* Default retention of 90 days.
2018-08-23 10:40:02 +08:00
Régis Hanol f01169d6ff FIX: don't send email when the post was deleted 2018-08-22 13:13:58 +02:00
Guo Xiang Tan 791ddb8630 PERF: Memoize `EmailReceiver#sent_to_mailinglist_mirror?`. 2018-08-21 15:44:47 +08:00
Guo Xiang Tan 7dcc69aef4 DEV: Refactor test to not call private method. 2018-08-21 14:29:58 +08:00
Sam ca5a6f0a9d FIX: handle concurrently creating post reply keys
In some very rare conditions this would be called concurrently and fail
2018-08-21 10:59:18 +10:00
Régis Hanol de92913bf4 FIX: store the topic links using the cooked upload url 2018-08-14 12:23:32 +02:00
Régis Hanol ac2513b0f2 FEATURE: automatic PM when a user's email is revoked 2018-08-03 16:39:22 +02:00
OsamaSayegh a157dfd418 UX: better rejection message when reply via email is too short 2018-08-02 22:43:53 +03:00
Guo Xiang Tan fad9c2b971 PERF: Move `EmailLog#reply_key` into new `post_reply_keys` table. 2018-07-24 13:51:53 +08:00
Guo Xiang Tan ae8b0a517f PERF: Split skipped email logs into a seperate table. 2018-07-24 13:14:37 +08:00
Guo Xiang Tan 3874d40910 Prepare to drop `EmailLog#topic_id`. 2018-07-18 10:22:24 +08:00
Neil Lalonde 7b3ef4d13f FIX: use email color settings consistently in notification emails 2018-07-16 12:30:42 -04:00
Maja Komel 0942e2c795 allow adding tags as a custom subject format for emails (#5846)
allow adding tags as a custom subject format for emails
2018-07-11 12:24:07 +10:00
Patrick Gansterer 28dd7fb562 FEATURE: Create hidden posts for received spam emails (#6010)
* Add possibility to add hidden posts with PostCreator

* FEATURE: Create hidden posts for received spam emails

Spamchecker usually have 3 results: HAM, SPAM and PROBABLY_SPAM
SPAM gets usually directly rejected and needs no further handling.
HAM is good message and usually gets passed unmodified.
PROBABLY_SPAM gets an additional header to allow further processing.
This change addes processing capabilities for such headers and marks
new posts created as hidden when received via email.
2018-07-05 11:07:46 +02:00
Régis Hanol 8a53941fe0 FIX: less aggressive gmail eliding 2018-07-04 20:04:46 +02:00
Jeff Wong 4599cc8435 FIX: PM participants listed inline 2018-06-11 18:14:25 -07:00
Arpit Jalan f9ab3848ed FEATURE: support disabling emails for non-staff users 2018-06-07 18:31:08 +05:30
Sam 89ad2b5900 DEV: Rails 5.2 upgrade and global gem upgrade
This updates tests to use latest rails 5 practice
and updates ALL dependencies that could be updated

Performance testing shows that performance has not regressed
if anything it is marginally faster now.
2018-06-07 14:21:33 +10:00
Ryan Mulligan fac4bf2f85 ignore emails that are from the reply by email addresses (#5843) 2018-05-23 10:04:45 +02:00
Régis Hanol 86eb3528ec FEATURE: clearer error message when receiving a reply to an old notification 2018-05-09 18:51:01 +02:00
Régis Hanol 6b1ff0edd3 FIX: always update bounce score (instead of doing it once per day) 2018-05-09 16:40:52 +02:00
Régis Hanol bae7203f42 FIX: proper category > subcategory order in List-Id description header 2018-05-05 01:51:53 +02:00
Jeff Wong 62a8904729
Feature: Include participants at the bottom of PM emails (#5797)
* Feature: Include participants at the bottom of PM emails

... as undecorated links.

https://meta.discourse.org/t/email-notification-recipients-unclear-when-pm-is-sent-to-multiple-users/26934/13?u=featheredtoast

Fix: missing translation for PM mentions

* display membership count as `group (count)`
2018-05-03 15:50:06 -07:00
Régis Hanol 7d223c0e55
FEATURE: add description in 'List-Id' mail headers
https://meta.discourse.org/t/minor-feature-request-descriptions-in-list-id-headers/86681
2018-05-03 16:39:25 +02:00
Régis Hanol 2d561a0422 FIX: don't extract divs with a 'gmail_default' class 2018-05-03 12:29:21 +02:00
Régis Hanol 2b0e505121 Extract signatures from emails sent with Newton 2018-04-19 12:39:55 +02:00
Régis Hanol fe32733a57 extract signatures from emails sent using Zimbra 2018-04-13 19:04:27 +02:00
Gerhard Schlager f2d00e5eff FEATURE: Use Message-ID for detecting email replies to group
Ignores the site setting "find_related_post_with_key" and always tries to honor the `In-Reply-To` and `References` header for emails sent to a group.

The senders email address must be included in the `To` or `CC` header of a previous email sent to the group and the `Message-ID` of that email must be included in the current email's `In-Reply-To` or `References` header.
2018-04-05 11:00:38 +02:00
Guo Xiang Tan 142571bba0 Remove use of `rescue nil`.
* `rescue nil` is a really bad pattern to use in our code base.
  We should rescue errors that we expect the code to throw and
  not rescue everything because we're unsure of what errors the
  code would throw. This would reduce the amount of pain we face
  when debugging why something isn't working as expexted. I've
  been bitten countless of times by errors being swallowed as a
  result during debugging sessions.
2018-04-02 13:52:51 +08:00
Régis Hanol a0386655a6 Add support to elide content from ProtonMail emails 2018-03-30 10:41:32 +02:00
Gerhard Schlager fcd352e089 FIX: Try fixing unparsable email addresses
The mail gem returns `UnstructuredField` when it fails to parse email addresses, but the `Receiver` always expects an `AddressList`.
2018-03-27 18:28:54 +02:00
Régis Hanol 20ba54d536
FIX: extracting mail content from exchange emails 2018-03-14 22:02:43 +01:00
Guo Xiang Tan a9713ca20b REFACTOR: Always prefer to raise an error if record fails to save. 2018-03-13 12:01:01 +08:00
Gerhard Schlager 832f0a9c4d FIX: Typo prevented extraction of email signatures 2018-03-06 11:34:47 +01:00
Régis Hanol 482c615ef8 FEATURE: extract signatures from most popular email services/software 2018-03-02 01:51:15 +01:00
Régis Hanol 3c430a3949 FEATURE: begone gmail signatures! 2018-02-27 15:19:34 +01:00
Régis Hanol 73ee62f55f FEATURE: automatically elide forwarded emails and signature from outlook 2018-02-27 15:00:50 +01:00
Régis Hanol fd33090646 FEATURE: automatically elides gmail quotes 2018-02-26 23:54:02 +01:00