Commit Graph

119 Commits

Author SHA1 Message Date
Alan Guo Xiang Tan 7079698cdf
FIX: Use MaxMind supplied permalinks to download MaxMind databases (#26847)
This commit switches `DiscourseIpInfo.mmdb_download` to use the
permalinks supplied by MaxMind to download the MaxMind databases as
specified in
https://dev.maxmind.com/geoip/updating-databases#directly-downloading-databases
which states:

```
To directly download databases, follow these steps:

1. In the "Download Links" column, click "Get Permalink(s)" for the desired database.
2. Copy the permalink(s) provided in the modal window.
3. Provide your account ID and your license key using Basic Authentication to authenticate.
```

Previously we are downloading from `https://download.maxmind.com/app/geoip_download` but this is not
documented anyway on MaxMind's docs so this URL can in theory break
in the future without warning. Therefore, we are taking a proactive
approach to download the databases from MaxMind the recommended way
instead of relying on a hidden URL. This old way of downloading the
databases with only a license key will be deprecated in 3.3 and be
removed in 3.4.
2024-05-09 15:11:56 +08:00
Charley Sheets 6c9f04a56c
DEV: Fix confusing comment (#26781) 2024-04-29 09:16:20 +08:00
David Taylor dcd994a9f1
DEV: Drop workbox dependency (#26735)
This service-worker caching functionality was disabled by default in 1c58395bca, and the setting to re-enable was marked as experimental. Now we are dropping all the related logic.
2024-04-24 10:19:12 +01:00
Alan Guo Xiang Tan 9182501366
DEV: Introduce `maxmind_mirror_url` GlobalSetting (#26458)
Why this change?

This allows downloading the MaxMind databases from a mirror in cases
where downloading directly from MaxMind's API endpoint is problematic
due to API limits.
2024-04-02 14:53:53 +08:00
David Taylor 1cc8c72a98
DEV: Consolidate experimental 'Link' header implementations (#26377)
This commit removes the 'experimental_preconnect_link_header' site setting, and the 'preload_link_header' site setting, and introduces two new global settings: early_hint_header_mode and early_hint_header_name.

We don't actually send 103 Early Hint responses from Discourse. However, upstream proxies can be configured to cache a response header from the app and use that to send an Early Hint response to future clients.

- `early_hint_header_mode` specifies the mode for the early hint header. Can be nil (disabled), "preconnect" (lists just CDN domains) or "preload" (lists all assets).
- `early_hint_header_name` specifies which header name to use for the early hint. Defaults to "Link", but can be changed to support different proxy mechanisms.
2024-03-27 09:06:50 +00:00
Alan Guo Xiang Tan 36cdb1444c
EXPERIMENTAL: preconnect and dns-prefetch resource hints for CDN domains (#26215)
Why this change?

In https://web.dev/articles/preconnect-and-dns-prefetch, it describes
how hinting to the browser to preconnect to domains which we will
eventually use the connection for can help improve the time it takes to
load a page.

We are putting this behind an experimental flag so that we can test and
profile this in a production environment.

What does this change introduce?

Introduce a hidden experimental `experimental_preconnect_link_header`
site setting which when enabled will add the `preconnect` and
`dns-prefetch` resource hints to the response headers for full page load
requests.
2024-03-18 13:45:41 +08:00
Martin Brennan ed47b55026
DEV: Increase default SMTP read timeout to 30s (#25763)
A while ago we increased group SMTP read and open timeouts
to address issues we were seeing with Gmail sometimes giving
really long timeouts for these values. The commit was:

3e639e4aa7

Now, we want to increase all SMTP read timeouts to 30s,
since the 5s is too low sometimes, and the ruby Net::SMTP
stdlib also defaults to 30s.

Also, we want to slightly tweak the group smtp email job
not to fail if the IncomingEmail log fails to create, or if
a ReadTimeout is encountered, to avoid retrying the job in sidekiq
again and sending the same email out.
2024-02-21 07:13:18 +10:00
David Taylor 1c58395bca
DEV: Disable service worker caching by default (#25723)
Followup to c4559ae575
2024-02-16 15:03:05 +00:00
David Taylor c4559ae575
DEV: Add `disable_service_worker_cache` global setting (#25589)
When enabled, the workbox caching logic in the service worker will be replaced with a very simple offline error page. We plan to use this as an experiment to see how it affects performance and stability of Discourse.
2024-02-07 10:44:12 +00:00
Michael Brown 7157798938 FIX: clear db_backup_port default value
The presence of this value with the postgresql default keeps tripping up
people. We shouldn't set this here.

examples:
* https://meta.discourse.org/t/84439/2
* https://meta.discourse.org/t/100604/7
* https://meta.discourse.org/t/223144/6
* https://meta.discourse.org/t/291992/10
2024-01-22 11:39:15 -05:00
David Taylor 5783f231f8
DEV: Introduce `DISCOURSE_ASSET_URL_SALT` (#24596)
This value is included when generating static asset URLs. Updating the value will allow site operators to invalidate all asset urls to recover from configuration issues which may have been cached by CDNs/browsers.
2023-11-28 11:28:40 +00:00
Martin Brennan 3e639e4aa7
FIX: Add higher read & open timeouts for group SMTP emails (#24593)
When sending SMTP for group SMTP functionality, we
are running into timeouts for both read and open
when sending mail occassionally, which can cause issues
like the email only being sent to _some_ of the recipients
or to fail altogether.

The defaults of 5s are too low, so bumping them up to
the defaults of the `net-smtp` gem.
2023-11-28 15:32:59 +10:00
Martin Brennan 731dffdf92
DEV: Align S3 transfer acceleration global settings (#24302)
Followup to fe05fdae24

For consistency with other S3 settings, make the global setting
the same name as the site setting and use SiteSetting.Upload
too so it reads from the correct place.
2023-11-10 09:50:23 +10:00
Martin Brennan fe05fdae24
DEV: Introduce S3 transfer acceleration for uploads behind hidden setting (#24238)
This commit adds an `enable_s3_transfer_acceleration` site setting,
which is hidden to begin with. We are adding this because in certain
regions, using https://aws.amazon.com/s3/transfer-acceleration/ can
drastically speed up uploads, sometimes as much as 70% in certain
regions depending on the target bucket region. This is important for
us because we have direct S3 multipart uploads enabled everywhere
on our hosting.

To start, we only want this on the uploads bucket, not the backup one.
Also, this will accelerate both uploads **and** downloads, depending
on whether a presigned URL is used for downloading. This is the case
when secure uploads is enabled, not anywhere else at this time. To
enable the S3 acceleration on downloads more generally would be a
more in-depth change, since we currently store S3 Upload record URLs
like this:

```
 url: "//test.s3.dualstack.us-east-2.amazonaws.com/original/2X/6/123456.png"
```

For acceleration, `s3.dualstack` would need to be changed to `s3-accelerate.dualstack`
here.

Note that for this to have any effect, Transfer Acceleration must be enabled
on the S3 bucket used for uploads per https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration-examples.html.
2023-11-07 11:50:40 +10:00
Alan Guo Xiang Tan cbbe3a808b
SECURITY: Add a default limit as to when logs should be truncated
Why this change?

This ensures that malicious requests cannot end up causing the logs to
quickly fill up. The default chosen is sufficient for most legitimate
requests to the Discourse application.

When truncation happens, parsing of logs in supported format like
lograge may break down.
2023-10-16 10:34:38 -04:00
Sam 98dfc9df3f
DEV: relative_url_root is used for subfolder installs add comment (#23403)
The word subfolder is used commonly to describe subdirectory installs
2023-09-05 17:06:31 +10:00
Daniel Waterworth 50d527b80c
DEV: Remove db_timeout setting (#22912)
It doesn't actually do anything.
2023-08-01 14:17:43 -05:00
David Taylor d37ecd4764
PERF: Set default global regex timeout to 2 seconds (#20933)
Followup to a0140f6f75
2023-04-03 10:43:28 +01:00
Sam d87e78616d
FEATURE: allow site owners to disable impersonation (#20783)
Site owners can now disable impersonation using the global setting
`allow_impersonation` (Eg: DISCOURSE_ALLOW_IMPERSONATION: false)

see:

https://meta.discourse.org/t/thoughts-about-impersonate-user/258795
2023-03-23 15:16:05 +11:00
David Taylor a0140f6f75
DEV: Introduce regex_timeout_seconds global setting (#20774)
This is introduced as `nil` by default. We intend to set a sensible default once we have performed some real-world testing.
2023-03-22 12:01:35 +00:00
Leonardo Mosquera 509fee0f5a
FIX: allow changing default DNS query timeout of 2s via GlobalSetting (#20383)
The current default timeout is hardcoded to 2 seconds which is proving
too low for certain cases, and resulting in sporadic timeouts due to slow DNS queries.
2023-02-21 09:54:29 +11:00
David Taylor 5406e24acb
FEATURE: Introduce pg_force_readonly_mode GlobalSetting (#19612)
This allows the entire cluster to be forced into pg readonly mode. Equivalent to running `Discourse.enable_pg_force_readonly_mode` on the console.
2023-01-19 13:59:11 +00:00
David Taylor 03fadf91f0
FEATURE: Optionally allow a separate `s3_asset_cdn_url` to be specified (#19284)
If configured, this will be used for static JS assets which are stored on S3. This can be useful if you want to use different CDN providers/configuration for Uploads and JS
2022-12-08 10:36:20 +00:00
David Taylor badd64ceee
PERF: Add GlobalSetting to redirect avatars instead of proxying (#19281)
When uploads are stored on S3, by default Discourse will fetch the avatars and proxy them through to the requesting client. This is simple, but it can lead to significant inbound/outbound network load in the hosting environment.

This commit adds an optional redirect_avatar_requests GlobalSetting. When enabled, requests for user avatars will be redirected to the S3 asset instead of being proxied. This adds an extra round-trip for clients, but it should significantly reduce server load. To mitigate that extra round-trip for clients, a CDN with 'follow redirect' capability could be used.
2022-12-02 10:07:25 +00:00
Rafael dos Santos Silva 6888eb5c2d
Revert "Revert "FEATURE: Preload resources via link header (#18475)" (#18511)" (#18531)
* Revert "Revert "FEATURE: Preload resources via link header (#18475)" (#18511)"

This reverts commit 95a57f7e0c.

* put behind feature flag

* env -> global setting

* declare global setting

* forgot one spot
2022-10-11 20:11:44 -03:00
Rafael dos Santos Silva ef505c0b73
DEV: Declare new SMTP settings from 3a37a7f (#17932)
Follow up to 3a37a7f where adding those new SMTP settings forgot to
declare it in our defaults file.
2022-08-15 13:45:23 -03:00
Jarek Radosz 79c854d9be
DEV: Remove RTLit gem (#16620)
Its only use was removed 7 years ago in #3377.
2022-05-04 14:11:12 +08:00
David Taylor 98a7fa3d1a
PERF: Bump message_bus to 4.2 (#16026)
This includes upstream performance improvements. For details, see 1baa1ea4a5
2022-02-22 16:16:02 +00:00
Osama Sayegh b86127ad12
FEATURE: Apply rate limits per user instead of IP for trusted users (#14706)
Currently, Discourse rate limits all incoming requests by the IP address they
originate from regardless of the user making the request. This can be
frustrating if there are multiple users using Discourse simultaneously while
sharing the same IP address (e.g. employees in an office).

This commit implements a new feature to make Discourse apply rate limits by
user id rather than IP address for users at or higher than the configured trust
level (1 is the default).

For example, let's say a Discourse instance is configured to allow 200 requests
per minute per IP address, and we have 10 users at trust level 4 using
Discourse simultaneously from the same IP address. Before this feature, the 10
users could only make a total of 200 requests per minute before they got rate
limited. But with the new feature, each user is allowed to make 200 requests
per minute because the rate limits are applied on user id rather than the IP
address.

The minimum trust level for applying user-id-based rate limits can be
configured by the `skip_per_ip_rate_limit_trust_level` global setting. The
default is 1, but it can be changed by either adding the
`DISCOURSE_SKIP_PER_IP_RATE_LIMIT_TRUST_LEVEL` environment variable with the
desired value to your `app.yml`, or changing the setting's value in the
`discourse.conf` file.

Requests made with API keys are still rate limited by IP address and the
relevant global settings that control API keys rate limits.

Before this commit, Discourse's auth cookie (`_t`) was simply a 32 characters
string that Discourse used to lookup the current user from the database and the
cookie contained no additional information about the user. However, we had to
change the cookie content in this commit so we could identify the user from the
cookie without making a database query before the rate limits logic and avoid
introducing a bottleneck on busy sites.

Besides the 32 characters auth token, the cookie now includes the user id,
trust level and the cookie's generation date, and we encrypt/sign the cookie to
prevent tampering.

Internal ticket number: t54739.
2021-11-17 23:27:30 +03:00
Daniel Waterworth fa3c4ad28b
DEV: Deprecate message bus site settings (#14465)
It makes much more sense for these to be GlobalSettings, since, in
multisite clusters, only the default site's settings would be respected.

Co-authored-by: David Taylor <david@taylorhq.com>
2021-11-11 11:12:25 -06:00
Daniel Waterworth ceb234c2e9
FEATURE: Make the multisite config path configurable (#14308) 2021-09-10 14:19:52 -05:00
David Taylor 4134173bbf
FEATURE: Add global admin api key rate limiter (#12527) 2021-06-03 10:52:43 +01:00
Josh Soref 59097b207f
DEV: Correct typos and spelling mistakes (#12812)
Over the years we accrued many spelling mistakes in the code base. 

This PR attempts to fix spelling mistakes and typos in all areas of the code that are extremely safe to change 

- comments
- test descriptions
- other low risk areas
2021-05-21 11:43:47 +10:00
David Taylor 7970d1d99f
FEATURE: Allow a cluster_name to be configured and used for /srv/status (#12365)
The cluster name can be configured by setting the `DISCOURSE_CLUSTER_NAME` environment variable. If set, you can then call /srv/status with a `?cluster=` parameter. If the cluster does not match, an error will be returned. This is useful if you need a load balancer to be able to verify the identity, as well as the presence, of an application container.
2021-03-15 15:41:59 +11:00
godmar 9aeece465f
FEATURE: support DISCOURSE_SMTP_FORCE_TLS option (#11733)
Background: RFC 8314 3.3 asks that:

clients and servers SHOULD implement both STARTTLS on
port 587 and Implicit TLS on port 465

Discourse currently cannot be configured this way.
With this patch, it's possible to set
DISCOURSE_SMTP_FORCE_TLS=true to use implicit TLS on port 465
2021-01-18 11:56:18 -05:00
Krzysztof Kotlarek f84ff26aa9
FIX: use Redis replica host and port (#11566)
Introduce Redis `replica` config and deprecate `slave`
2020-12-23 13:14:19 +11:00
Krzysztof Kotlarek 07bf7a91f4
Revert "FIX: use Redis replica host and port (#11543)" (#11552)
This reverts commit b0e1210b0c.
2020-12-22 16:16:50 +11:00
Krzysztof Kotlarek b0e1210b0c
FIX: use Redis replica host and port (#11543)
* FIX: use Redis replica host and port

Introduce Redis `replica` config and deprecate `slave`

* FIX: move deprecations to separate file
2020-12-22 15:52:00 +11:00
Osama Sayegh a04c300495
DEV: Add optional ENV variables for MiniProfiler snapshots transporter (#10985) 2020-10-21 19:37:28 +03:00
Sam 32393f72b1
PERF: backoff background requests when overloaded (#10888)
When the server gets overloaded and lots of requests start queuing server
will attempt to shed load by returning 429 errors on background requests.

The client can flag a request as background by setting the header:
`Discourse-Background` to `true`

Out-of-the-box we shed load when the queue time goes above 0.5 seconds.

The only request we shed at the moment is the request to load up a new post
when someone posts to a topic.

We can extend this as we go with a more general pattern on the client.

Previous to this change, rate limiting would "break" the post stream which
would make suggested topics vanish and users would have to scroll the page
to see more posts in the topic.

Server needs this protection for cases where tons of clients are navigated
to a topic and a new post is made. This can lead to a self inflicted denial
of service if enough clients are viewing the topic.

Due to the internal security design of Discourse it is hard for a large
number of clients to share a channel where we would pass the full post body
via the message bus.

It also renames (and deprecates) triggerNewPostInStream to triggerNewPostsInStream

This allows us to load a batch of new posts cleanly, so the controller can
keep track of a backlog

Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
2020-10-13 16:56:03 +11:00
Sam 120fa8ad2f
PERF: Introduce absolute limit of digests per 30 minutes (#10845)
To avoid blocking the sidekiq queue a limit of 10,000 digests per 30 minutes
is introduced.

This acts as a safety measure that makes sure we don't keep pouring oil on
a fire.

On multisites it is recommended to set the number way lower so sites do not
dominate the backlog. A reasonable default for multisites may be 100-500.

This can be controlled with the environment var

DISCOURSE_MAX_DIGESTS_ENQUEUED_PER_30_MINS_PER_SITE
2020-10-07 17:30:15 +11:00
Osama Sayegh a92d88747e
DEV: Add ENV variable for enabling MiniProfiler snapshots (#10690)
* DEV: Add ENV variable for enabling MiniProfiler snapshots

* MiniProfiler is not loaded in test env
2020-09-17 18:18:35 +03:00
Krzysztof Kotlarek e0d9232259
FIX: use allowlist and blocklist terminology (#10209)
This is a PR of the renaming whitelist to allowlist and blacklist to the blocklist.
2020-07-27 10:23:54 +10:00
Martin Brennan a9905ef7e5
FIX: Add enable_email_sync_demon global variable and disable EmailSync demon by default (#10304)
Demon::EmailSync is used in conjunction with the SiteSetting.enable_imap to sync N IMAP mailboxes with specific groups. It is a process started in unicorn.conf, and it spawns N threads (one for each multisite connection) and for each database spans another N threads (one for each configured group).

We want this off by default so the process is not started when it does not need to be (e.g. development, test, certain hosting tiers)
2020-07-24 17:09:29 +10:00
Guo Xiang Tan d8cd912769
DEV: Switch to db config to disable advisory locks. 2020-06-15 14:33:41 +08:00
Sam Saffron 57a3d4e0d2
FEATURE: whitelist theme repo mode (experimental)
In some restricted setups all JS payloads need tight control.

This setting bans admins from making changes to JS on the site and
requires all themes be whitelisted to be used.

There are edge cases we still need to work through in this mode
hence this is still not supported in production and experimental.

Use an example like this to enable:

`DISCOURSE_WHITELISTED_THEME_REPOS="https://repo.com/repo.git,https://repo.com/repo2.git"`

By default this feature is not enabled and no changes are made.

One exception is that default theme id was missing a security check
this was added for correctness.
2020-06-03 13:19:57 +10:00
Rafael dos Santos Silva b48299f81c
FEATURE: Add setting to disable automatic CORS rule install in S3 buckets (#9872) 2020-05-25 17:09:34 -03:00
Daniel Waterworth 497dc6eaa7 Add a global setting for CDN origin
This is so that, on a multisite cluster, when we handle a CDN request,
the hostname that is requested corresponds to one of the sites -
specifically the default site.
2020-05-12 16:43:40 +01:00
Rafael dos Santos Silva 08e4af6636 FEATURE: Add setting to controle the Expect header on S3 calls
Some providers don't implement the Expect: 100-continue support,
which results in a mismatch in the object signature.

With this settings, users can disable the header and use such providers.
2020-04-30 12:12:00 -03:00
Rafael dos Santos Silva 54f67661ac FEATURE: Option to connect to Redis using SSL 2020-03-05 20:49:05 -03:00