Commit Graph

623 Commits

Author SHA1 Message Date
Alessio Cosenza 56718504ac
FEATURE: Add hooks for email poller plugins (#21384)
While we are unable to support OAUTH2 with pop3 (due to upstream dependency ruby/net-pop#16), we are adding the support for mail pollers plugin. Doing so, it would be possible to write a plugin which then uses other ways (microsoft graph sdk for example) to poll emails from a mailbox.

The idea is that a plugin would define a class which inherits from Email::Poller and defines a poll_mailbox static method which returns an array of strings. Then the plugin could call register_mail_poller(<class_name>) to have it registered. All the configuration (oauth2 tokens, email, etc) could be managed by sitesettings defined in the plugin.
2023-06-26 13:16:03 +08:00
Régis Hanol db9d998de3
FIX: improve mailman email parsing (#21627)
https://meta.discourse.org/t/improving-mailman-email-parsing/253041

When mirroring a public mailling list which uses mailman, there were some cases where the incoming email was not associated to the proper user.

As it happens, for various (undertermined) reasons, the email from the sender is often not in the `From` header but can be in any of the following headers: `Reply-To`, `CC`, `X-Original-From`, `X-MailFrom`.

It might be in other headers as well, but those were the ones we found the most reliable.
2023-05-19 10:33:48 +02:00
David Battersby 1de8361d2e
FIX: Prevent Email Processor errors when mail is blank or nil (#21292)
Currently processing emails that are blank or have a nil value for the mail will cause several errors.

This update allows emails with blank body or missing sender to log the blank email error to the mail logs rather than throwing an error.
2023-05-18 10:39:37 +08:00
Ted Johansson a2bc24456f
DEV: Update call sites using BaseStore#download but expecting exceptions (#21551)
In #21498, we split `BaseStore#download` into a "safe" version which returns nil on errors, and an "unsafe" version which raises an exception, which was the old behaviour of `#download`.

This change updates call sites that used the old `#download`, which raised exceptions, to use the new `#download!` to preserve behaviour (and silence deprecation warnings.)

It also silences the deprecation warning in tests.
2023-05-17 16:03:33 +08:00
Michael Brown 076def505e
FIX: email receiver should ignore x-auto-response-suppress
This header is used by Microsoft Exchange to indicate when certain types of
autoresponses should not be generated for an email.

It triggers our "is this mail autogenerated?" detection, but should not be used
for this purpose.
2023-05-03 12:20:00 -04:00
Jarek Radosz f63d18d6ca
DEV: Update syntax_tree to 6.0.1 (#20466) 2023-02-27 17:20:00 +01:00
natiz 9c30ae93b3
DEV: Add `after_email_send` app event (#20178) 2023-02-15 11:23:45 -05:00
Ted Johansson 25a226279a
DEV: Replace #pluck_first freedom patch with AR #pick in core (#19893)
The #pluck_first freedom patch, first introduced by @danielwaterworth has served us well, and is used widely throughout both core and plugins. It seems to have been a common enough use case that Rails 6 introduced it's own method #pick with the exact same implementation. This allows us to retire the freedom patch and switch over to the built-in ActiveRecord method.

There is no replacement for #pluck_first!, but a quick search shows we are using this in a very limited capacity, and in some cases incorrectly (by assuming a nil return rather than an exception), which can quite easily be replaced with #pick plus some extra handling.
2023-02-13 12:39:45 +08:00
Martin Brennan 63fdb6dd65
FIX: Do not add empty use/svg tags in ExcerptParser (#19969)
There was an issue where if hashtag-cooked HTML was sent
to the ExcerptParser without the keep_svg option, we would
end up with empty </use> and </svg> tags on the parts of the
excerpt where the hashtag was, in this case when a post
push notification was sent.

Fixed this, and also added a way to only display a plaintext
version of the hashtag for cases like this via PrettyText#excerpt.
2023-01-24 14:40:24 +10:00
Daniel Waterworth 666536cbd1
DEV: Prefer \A and \z over ^ and $ in regexes (#19936) 2023-01-20 12:52:49 -06:00
Martin Brennan 1f59a8299d
DEV: Address TODOs for email Message-ID generation (#19842)
Remove some old deprecated methods and update docs. Will leave
the backwards-compatible Message-ID formats a little while longer
just to be sure.
2023-01-12 13:54:15 +10:00
David Taylor 6417173082
DEV: Apply syntax_tree formatting to `lib/*` 2023-01-09 12:10:19 +00:00
Martin Brennan c4ea158656
FIX: Improve tags in email subjects and add filter headers (#19760)
This commit does a couple of things:

1. Changes the limit of tags to include a subject for a
   notification email to the `max_tags_per_topic` setting
   instead of the arbitrary 3 limit
2. Adds both an X-Discourse-Tags and X-Discourse-Category
   custom header to outbound emails containing the tags
   and category from the subject, so people on mail clients
   that allow advanced filtering (i.e. not Gmail) can filter
   mail by tags and category, which is useful for mailing
   list mode users

c.f. https://meta.discourse.org/t/headers-for-email-notifications-so-that-gmail-users-can-filter-on-tags/249982/17
2023-01-06 10:03:02 +10:00
Alan Guo Xiang Tan ab3a032b4b
SECURITY: BCC active user emails from group SMTP (#19725)
When sending emails out via group SMTP, if we
are sending them to non-staged users we want
to mask those emails with BCC, just so we don't
expose them to anyone we shouldn't. Staged users
are ones that have likely only interacted with
support via email, and will likely include other
people who were CC'd on the original email to the
group.

Co-authored-by: Martin Brennan <martin@discourse.org>
2023-01-05 06:07:50 +08:00
Bianca Nenciu b80765f1f4
DEV: Remove enable_whispers site setting (#19196)
* DEV: Remove enable_whispers site setting

Whispers are enabled as long as there is at least one group allowed to
whisper, see whispers_allowed_groups site setting.

* DEV: Always enable whispers for admins if at least one group is allowed.
2022-12-16 18:42:51 +02:00
Martin Brennan 9513e7be6d
FIX: Email hashtag-cooked text replacement error (#19278)
In some cases (e.g. user notification emails) we
are passing an excerpted/stripped version of the
post HTML to Email::Styles, at which point the
<span> elements surrounding the hashtag text have
been stripped. This caused an error when trying to
remove that element to replace the text.

Instead we can just remove all elements inside
a.hashtag-cooked and replace with the raw #hashtag
text which will work in more cases.
2022-12-01 19:48:24 +10:00
Martin Brennan d3f02a1270
FEATURE: Generic hashtag autocomplete lookup and markdown cooking (#18937)
This commit fleshes out and adds functionality for the new `#hashtag` search and
lookup system, still hidden behind the `enable_experimental_hashtag_autocomplete`
feature flag.

**Serverside**

We have two plugin API registration methods that are used to define data sources
(`register_hashtag_data_source`) and hashtag result type priorities depending on
the context (`register_hashtag_type_in_context`). Reading the comments in plugin.rb
should make it clear what these are doing. Reading the `HashtagAutocompleteService`
in full will likely help a lot as well.

Each data source is responsible for providing its own **lookup** and **search**
method that returns hashtag results based on the arguments provided. For example,
the category hashtag data source has to take into account parent categories and
how they relate, and each data source has to define their own icon to use for the
hashtag, and so on.

The `Site` serializer has two new attributes that source data from `HashtagAutocompleteService`.
There is `hashtag_icons` that is just a simple array of all the different icons that
can be used for allowlisting in our markdown pipeline, and there is `hashtag_context_configurations`
that is used to store the type priority orders for each registered context.

When sending emails, we cannot render the SVG icons for hashtags, so
we need to change the HTML hashtags to the normal `#hashtag` text.

**Markdown**

The `hashtag-autocomplete.js` file is where I have added the new `hashtag-autocomplete`
markdown rule, and like all of our rules this is used to cook the raw text on both the clientside
and on the serverside using MiniRacer. Only on the server side do we actually reach out to
the database with the `hashtagLookup` function, on the clientside we just render a plainer
version of the hashtag HTML. Only in the composer preview do we do further lookups based
on this.

This rule is the first one (that I can find) that uses the `currentUser` based on a passed
in `user_id` for guardian checks in markdown rendering code. This is the `last_editor_id`
for both the post and chat message. In some cases we need to cook without a user present,
so the `Discourse.system_user` is used in this case.

**Chat Channels**

This also contains the changes required for chat so that chat channels can be used
as a data source for hashtag searches and lookups. This data source will only be
used when `enable_experimental_hashtag_autocomplete` is `true`, so we don't have
to worry about channel results suddenly turning up.

------

**Known Rough Edges**

- Onebox excerpts will not render the icon svg/use tags, I plan to address that in a follow up PR
- Selecting a hashtag + pressing the Quote button will result in weird behaviour, I plan to address that in a follow up PR
- Mixed hashtag contexts for hashtags without a type suffix will not work correctly, e.g. #ux which is both a category and a channel slug will resolve to a category when used inside a post or within a [chat] transcript in that post. Users can get around this manually by adding the correct suffix, for example ::channel. We may get to this at some point in future
- Icons will not show for the hashtags in emails since SVG support is so terrible in email (this is not likely to be resolved, but still noting for posterity)
- Additional refinements and review fixes wil
2022-11-21 08:37:06 +10:00
Martin Brennan 8ebd5edd1e
DEV: Rename secure_media to secure_uploads (#18376)
This commit renames all secure_media related settings to secure_uploads_* along with the associated functionality.

This is being done because "media" does not really cover it, we aren't just doing this for images and videos etc. but for all uploads in the site.

Additionally, in future we want to secure more types of uploads, and enable a kind of "mixed mode" where some uploads are secure and some are not, so keeping media in the name is just confusing.

This also keeps compatibility with the `secure-media-uploads` path, and changes new
secure URLs to be `secure-uploads`.

Deprecated settings:

* secure_media -> secure_uploads
* secure_media_allow_embed_images_in_emails -> secure_uploads_allow_embed_images_in_emails
* secure_media_max_email_embed_image_size_kb -> secure_uploads_max_email_embed_image_size_kb
2022-09-29 09:24:33 +10:00
Martin Brennan e3d495850d
FEATURE: Overhaul email threading (#17996)
See https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499
for thorough reasoning.

This commit changes how we generate Message-IDs and do email
threading for emails sent from Discourse. The main changes are
as follows:

* Introduce an outbound_message_id column on Post that
  is either a) filled with a Discourse-generated Message-ID
  the first time that post is used for an outbound email
  or b) filled with an original Message-ID from an external
  mail client or service if the post was created from an
  incoming email.
* Change Discourse-generated Message-IDs to be more consistent
  and static, in the format `discourse/post/:post_id@:host`
* Do not send References or In-Reply-To headers for emails sent
  for the OP of topics.
* Make sure that In-Reply-To is filled with either a) the OP's
  Message-ID if the post is not a direct reply or b) the parent
  post's Message-ID
* Make sure that In-Reply-To has all referenced post's Message-IDs
* Make sure that References is filled with a chain of Message-IDs
  from the OP down to the parent post of the new post.

We also are keeping X-Discourse-Post-Id and X-Discourse-Topic-Id,
headers that we previously removed, for easier visual debugging
of outbound emails.

Finally, we backfill the `outbound_message_id` for posts that have
a linked `IncomingEmail` record, using the `message_id` of that record.
We do not need to do that for posts that don't have an incoming email
since they are backfilled at runtime if `outbound_message_id` is missing.
2022-09-26 09:14:24 +10:00
Bianca Nenciu 707034bc75
FIX: Limit new and existent staged users for email topics (#17970)
The maximum_staged_users_per_email site setting controls how many
staged users will be invited to the topic created from an incoming
email. Previously, it counted only the new staged users.
2022-08-18 18:19:20 +03:00
Bianca Nenciu b082f459c9
FEATURE: Limit maximum recipients for group emails (#17971)
New maximum_recipients_per_new_group_email site setting can be used to
prevent spam group emails with many recipients.
2022-08-18 18:18:58 +03:00
Martin Brennan 4d3c1ceb44
FEATURE: Log the SMTP response in EmailLog (#17056)
When sending emails with delivery_method_options -> return_response
set to true, the SMTP sending code inside Mail will return the SMTP
response when calling deliver! for mail within the app. This commit
ensures that Email::Sender captures this response if it is returned
and stores it against the EmailLog created for the sent email.

A follow up PR will make this visible within the admin email UI.
2022-06-15 10:28:30 +10:00
Martin Brennan a6be4972a8
FIX: Use our header value instead of custom header on duplicates (#16711)
When we build and send emails using MessageBuilder and Email::Sender
we add custom headers defined in SiteSetting.email_custom_headers.
However this was causing errors in cases where the custom headers
defined a header that we already specify in outbound emails (e.g.
the Precedence: list header for topic/post emails).

This commit makes it so we always use the header value defined in Discourse
core if there is a duplicate, discarding the custom header value
from the site setting.

cf. https://meta.discourse.org/t/email-notifications-fail-if-duplicate-headers-exist/222960/14
2022-05-11 13:47:12 +10:00
ValdikSS f7540aa52f
FIX: properly clean Thunderbird emails, don't remove links (#16543)
Mozilla Thunderbird email client add links into HTML as follows:

`<p>The link: <a class="moz-txt-link-freetext" href="https://google.com">https://google.com</a></p>`

Current filtering rules strip out the link, leaving only `<p>The link: </p>`.
Properly strip only unnecessary information: quote prefix, signature, forwarded message header.
2022-04-25 12:57:56 -04:00
Isaac Janzen 692e0140e2
FEATURE: Enables support for dark mode emails (#16520)
This PR enables custom email dark mode styles by default that were added here.

There is currently poor support for dark mode queries in mail clients. The main beneficiary of these changes will be Apple Mail and Outlook.

Enjoy the darkness 🕶️
2022-04-20 13:00:04 -05:00
Isaac Janzen 10a4bbfa72
update email dark mode styles (#16484)
Update dark mode styles for:

- secure media
- code blocks (they were too dark and had bad contrast)
2022-04-14 15:08:09 -05:00
Penar Musaraj e164ff3085
FIX: Email styles for Gmail app dark mode (#16482) 2022-04-14 15:03:06 -04:00
Isaac Janzen 0acbd63320
update dark mode email styles (#16480)
Update dark mode styles for:

secure media borders
code blocks
quote blocks
2022-04-14 12:51:09 -05:00
Isaac Janzen 5f9c98dfc1
FIX: update dark mode emails styles (#16461)
- divider sometimes is too bright
- lists have dark text on dark copy
- border around the “secure media” message is bright
2022-04-12 13:13:34 -05:00
Isaac Janzen 6c0abe15e0
FEATURE: Add email dark mode (#16104)
implement dark mode emails when `SiteSetting.dark_mode_emails_active` is active.
2022-04-11 12:27:50 -05:00
Loïc Guitaut 357011eb3b DEV: Clean up freedom patches
This patch removes some of our freedom patches that have been deprecated
for some time now.
Some of them have been updated so we’re not shipping code based on an
old version of Rails.
2022-04-06 10:07:14 +02:00
Arpit Jalan 567be512c9
FIX: save email rejection error class names for incoming email logs (#16095) 2022-03-03 18:58:13 +05:30
Jarek Radosz 2fc70c5572
DEV: Correctly tag heredocs (#16061)
This allows text editors to use correct syntax coloring for the heredoc sections.

Heredoc tag names we use:

languages: SQL, JS, RUBY, LUA, HTML, CSS, SCSS, SH, HBS, XML, YAML/YML, MF, ICS
other: MD, TEXT/TXT, RAW, EMAIL
2022-02-28 20:50:55 +01:00
Martin Brennan 4086ee551e
DEV: Add bounce_error_code to EmailLog (#15948)
Whenever we got a bounced email in the Email::Receiver we
previously would just set bounced: true on the EmailLog and
discard the status/diagnostic code. This commit changes this
flow to store the bounce error code (defined in the RFC at
https://www.iana.org/assignments/smtp-enhanced-status-codes/smtp-enhanced-status-codes.xhtml)
not just in the Email::Receiver, but also via webhook events
from other mail services and from SNS.

This commit does not surface the bounce error in the UI,
we can do that later if necessary.
2022-02-15 14:17:26 +10:00
Martin Brennan 82cb67e67b
FIX: Canonical Message-ID was incorrect for some cases (#15701)
When creating a direct message to a group with group SMTP
set up, and adding another person to that message in the OP,
we send an email to the second person in the OP via the group_smtp
job. This in turn creates an IncomingEmail record to guard against
IMAP double sync.

The issue with this was that this IncomingEmail (which is essentialy
a placeholder/dummy one) was having its Message-ID used as the canonical
References Message-ID for subsequent emails sent out to user_private_message
recipients (such as members of the group), causing threading issues in
the mail client. The canonical <topic/ID@HOST> format should be used
instead for these cases.

This commit fixes the issue by only using the IncomingEmail for the
OP's Message-ID if the OP was created via our handle_mail email receiver
pipeline. It does not make sense to use it in other cases.
2022-02-03 10:36:32 +10:00
Vinoth Kannan f614b30032
FIX: don't trigger `topic_created` event for reply posts via email. (#15485)
Previously, we incorrectly triggered `topic_created` events when the posts are created via email since we didn't check the post number.
2022-01-10 13:54:10 +05:30
David Taylor cd1d0322f7
DEV: Add email bounce DiscourseEvent for plugins (#15398) 2022-01-06 16:50:37 +00:00
Daniel Waterworth ec36cddd2f
FIX: Defer topic/post created events until emails have been added (#15252) 2021-12-09 14:45:07 -06:00
David Taylor f799b8bfb1
FIX: Ensure MessageIdService can handle hostname changes and multisite (#15231) 2021-12-08 11:17:20 +00:00
Martin Brennan f26b8b448d
FIX: References header leading to broken email threading (#15206)
Since 3b13f1146b the email threading
in mail clients has been broken, because the random suffix meant
that the References header would always be different for non-group
SMTP email notifications sent out.

This commit fixes the issue by always using the "canonical" topic
reference ID inside the References header in the format:

topic/TOPIC_ID@HOST

Which was the old format. We also add the References header to
notifications sent for the first post arriving, so the threading
works for subsequent emails. The Message-ID header is still random
as per the previous change.
2021-12-08 08:14:48 +10:00
Martin Brennan 3b13f1146b
FIX: Add random suffix to outbound Message-ID for email (#15179)
Currently the Message-IDs we send out for outbound email
are not unique; for a post they look like:

topic/TOPIC_ID/POST_ID@HOST

And for a topic they look like:

topic/TOPIC_ID@HOST

This commit changes the outbound Message-IDs to also have
a random suffix before the host, so the new format is
like this:

topic/TOPIC_ID/POST_ID.RANDOM_SUFFIX@HOST

Or:

topic/TOPIC_ID.RANDOM_SUFFIX@HOST

This should help with email deliverability. This change
is backwards-compatible, the old Message-ID format will
still be recognized in the mail receiver flow, so people
will still be able to reply using Message-IDs, In-Reply-To,
and References headers that have already been sent.

This commit also refactors Message-ID related logic
to a central location, and adds judicious amounts of
tests and documentation.
2021-12-06 10:34:39 +10:00
Martin Brennan 44be79f095
FIX: Strip Auto-Submitted email header from group SMTP emails (#15057)
Remove Auto-Submitted header for group private message emails, it does
not make sense there and may hurt deliverability.

From https://www.iana.org/assignments/auto-submitted-keywords/auto-submitted-keywords.xhtml:

> Indicates that a message was generated by an automatic process, and is not a direct response to another message.
2021-11-24 10:54:01 +10:00
Bianca Nenciu 79e55ec3f0
FIX: Remove List-Post email header (#14554)
* FIX: Remove List-Post email header

This header is used for mailing lists and can confuse some email clients
such as Thunderbird to display wrong replying options.

* FIX: Replace reply_key in email custom headers

Admins can add custom email headers from site settings. Email sender
will try to replace the reply key if %{reply_key} exists or remove the
header if a reply key does not exist.
2021-10-11 20:57:42 +03:00
Bianca Nenciu f58ab2283d
FIX: Parse address lists in embedded emails (#14514)
Same fix is applied to emails immediately after being parsed because
long headers are sometimes in an invalid format.
2021-10-06 15:07:29 +03:00
Dan Ungureanu 74a9c0509b
FIX: Use addresses to compare email header (#14509)
Usually, when an email is received a user lookup is performed using the
email address found in the `From` header. When an email has an
`X-Original-From` header, if it is equal to `Reply-To` then it uses that
one instead. The comparison was sensitive to whitespaces and other
insignificant characters such as quotes because it reconstructed the
`From` header.

For the fixture added in this commit, it compared the reconstructed
`From` header `John Doe <johndoe@example.com>` with the `Reply-To`
header `"John Doe"    <johndoe@example.com>`.
2021-10-05 12:42:19 +03:00
Martin Brennan 4fb7d045a0
FIX: Handle forwarded email quotes around Reply-To display name (#14384)
The display name can have quotes around it, which does not work
with our current comparison of a from field (in this case Reply-To)
and another header (X-Original-From), because we are not comparing
the two values in the same way. This causes an issue where the
commit here: b88d8c8 will not
work properly; the forwarded email gets the From address instead
of the Reply-To address as intended.
2021-09-20 16:26:18 +10:00
Martin Brennan 9f36d8ad43
FIX: Capture CC addresses for forwarded emails (#14254)
When forwarding emails into the group inbox, we now use the
original sender email as the from_address since
2ac9fd9dff. However, we have not
been saving the original CC addresses of the forwarded email,
which are needed to include those recipients in on the conversation
when replying via the group inbox.

This commit captures the CC addresses on the incoming email, and
makes sure the emails are created as staged users and added to the
list of topic allowed users so they are included on CC's sent by
the GroupSmtpEmail and other jobs.
2021-09-07 08:46:28 +10:00
Martin Brennan 7b392cee50
FIX: Correct the forwarded by user small post for group inbox (#14252)
When 2ac9fd9dff was done, this
affected the small post that is created when forwarding an email
into the group inbox. Instead of using the name and the email of
the user who forwarded the email, it used the original from email
and name to create the small post. So instead of something like
"Discourse Team forwarded the above email" we ended up with
"John Smith forwarded the above email" which is incorrect.

This fixes the issue by creating a staged user for the forwarding
email address (if such a user does not yet exist) and uses that
for the "forwarded" small post instead.
2021-09-06 15:02:13 +10:00
Bianca Nenciu f0005025c4
FIX: Correctly display GitHub code oneboxes (#14149)
Code fragments without the code lines were not displayed correctly.

Follow-up-to 721a946d77.
2021-08-26 12:55:46 +03:00
Bianca Nenciu 721a946d77
FIX: Remove spacing from GitHub oneboxes (#14147)
Each code line was surrounded by a lot of white space. The new styles
are similar to what the web interface is using.
2021-08-25 16:34:01 +03:00