We have a couple of site setting, `slow_down_crawler_user_agents` and `slow_down_crawler_rate`, that are meant to allow site owners to signal to specific crawlers that they're crawling the site too aggressively and that they should slow down.
When a crawler is added to the `slow_down_crawler_user_agents` setting, Discourse currently adds a `Crawl-delay` directive for that crawler in `/robots.txt`. Unfortunately, many crawlers don't support the `Crawl-delay` directive in `/robots.txt` which leaves the site owners no options if a crawler is crawling the site too aggressively.
This PR replaces the `Crawl-delay` directive with proper rate limiting for crawlers added to the `slow_down_crawler_user_agents` list. On every request made by a non-logged in user, Discourse will check the User Agent string and if it contains one of the values of the `slow_down_crawler_user_agents` list, Discourse will only allow 1 request every N seconds for that User Agent (N is the value of the `slow_down_crawler_rate` setting) and the rest of requests made within the same interval will get a 429 response.
The `slow_down_crawler_user_agents` setting becomes quite dangerous with this PR since it could rate limit lots if not all of anonymous traffic if the setting is not used appropriately. So to protect against this scenario, we've added a couple of new validations to the setting when it's changed:
1) each value added to setting must 3 characters or longer
2) each value cannot be a substring of tokens found in popular browser User Agent. The current list of prohibited values is: apple, windows, linux, ubuntu, gecko, firefox, chrome, safari, applewebkit, webkit, mozilla, macintosh, khtml, intel, osx, os x, iphone, ipad and mac.
Currently when a user creates posts that are moderated (for whatever
reason), a popup is displayed saying the post needs approval and the
total number of the user’s pending posts. But then this piece of
information is kind of lost and there is nowhere for the user to know
what are their pending posts or how many there are.
This patch solves this issue by adding a new “Pending” section to the
user’s activity page when there are some pending posts to display. When
there are none, then the “Pending” section isn’t displayed at all.
When this setting is turned on, it will check that normalized emails
are unique. Normalized emails are emails without any dots or plus
aliases.
This setting can be used to block use of aliases of the same email
address.
Use @here to mention all users that were allowed to topic directly or
through group, who liked topics or read the topic. Only first 10 users
will be notified.
When rendering the markdown code blocks we replace the
offending characters in the output string with spans highlighting a textual
representation of the character, along with a title attribute with
information about why the character was highlighted.
The list of characters stripped by this fix, which are the bidirectional
characters considered relevant, are:
U+202A
U+202B
U+202C
U+202D
U+202E
U+2066
U+2067
U+2068
U+2069
* FEATURE: Always show advanced invite options
The UI is more simple and more efficient than how it was when the
advanced options toggle was introduced. It does not make sense to keep
it anymore.
* UX: Minor copy edits
* UX: Merge expire invite controls
There were two controls in the create invite modal. One was a static
text that displayed how much time is left until the invite expires. The
other one was a datetime selector that set the time the invite expires.
This commit merges the two controls in a single one: staff users will
continue to see the datetime selector without the static text and
regular users will only see the static text because they cannot set
when the invite expires.
* UX: Remove invite link
It should only be visible after the invite was created.
* FEATURE: display warning when sharing a topic in a restricted category
If a topic belongs to a category that is not readable by everyone, display a text warning of "Only visible to members of groups: [group_a], [group_b]"
* DEV: Adding a new category means we need to bump this value
* DEV: pass category to showModal
* FIX: small copy fix for embedding.sample
* FIX: improve copy for site_description and short_site_description
As part of the setup wizard, improve the description of these two
strings to add context on where they will be used, so that it is
clearer how to write each one.
Meta discussion: https://meta.discourse.org/t/unclear-double-question-in-setup-wizard/208344/
This commit changes the emoji uploader to use the UppyUploadMixin,
and makes some minor changes to the emoji uploader (tightening the
copy for drag and drop and adding a percentage for the upload).
Since no other uppy upload mixin components have needed to upload
multiple files so far, this necessitated adding a tracker for the
in progress uploads so we know when to reset the uploader once all
uploads are complete.
At the moment, the emoji uploader cannot be used for direct S3 uploads
because the admin emoji controller creates other records and does other
magic with the emojis. At some point we need to factor this kind of thing
into the ExternalUploadManager.transform! action to complete external
uploads.
* DEV: Sanitize HTML admin inputs
This PR adds on-save HTML sanitization for:
Client site settings
translation overrides
badges descriptions
user fields descriptions
I used Rails's SafeListSanitizer, which [accepts the following HTML tags and attributes](018cf54073/lib/rails/html/sanitizer.rb (L108))
* Make sure that the sanitization logic doesn't corrupt settings with special characters
This commit makes the following change to the Edit Bookmark
modal window for clarity:
* If the user is editing an existing bookmark without a reminder set,
hide the "none needed" option. This will draw more attention to the
delete button.
* If the user is editing an existing bookmark with a reminder set for the
future, change the "none needed" option to say "remove reminder, keep bookmark"
To do this, I needed to provide an option to override the labels
for time shortcuts in certain cases, so I could keep the NONE shortcut
but have the different wording.
Sometimes administrators want to permanently delete posts and topics
from the database. To make sure that this is done for a good reasons,
administrators can do this only after one minute has passed since the
post was deleted or immediately if another administrator does it.
Reimplemented following the revert in ce0daae636
This approach uses the global `e`/`q` shortcuts, rather than shifting focus to the `quote-button` component. The current `quoteState` is used to determine whether the quote-button is currently visible. If yes, an appEvent transmits the intention to the quote-button component. If no, the old behavior is maintained.
Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
This reverts the new e and q shortcuts for quick-edit, and quote. The current implementation of these is causing issues with quoting on mobile devices.
We intend restore these new shortcuts soon.
* Revert "FIX: Apply quote selection workaround to all browsers (#14558)"
This reverts commit 488f716c16.
* Revert "FIX: selection going missing in Safari (#14557)"
This reverts commit 538fe2cc31.
* Revert "UX: adds shortcuts for quote (q) and fast edit (e) (#14552)"
This reverts commit 2af6052307.
We aren't translating these settings, so it makes more sense to move them into the code. I added an instance method so plugins can add mappings for custom reasons.
Krzysztof Kotlarek
Discourse Translator Bot
Osama Sayegh
Loïc Guitaut
Penar Musaraj
janzenisaac
Bianca Nenciu
Arpit Jalan
Natalie Tay
RogerBW
Martin Brennan
jbrw
Dan Ungureanu
Joffrey JAFFEUX
tshenry
Leonardo Mosquera
Jean
Vinoth Kannan
Roman Rizzi
Joshua Rosenfeld
Simon Cossar
Gavin
Gerhard Schlager
David Taylor