Commit Graph

5759 Commits

Author SHA1 Message Date
Kyle Zhao dec8e5879a FEATURE: set CSP base-uri and object-src to none (#6863) 2019-01-09 15:04:50 -05:00
Sam 4232d32699 PERF: reduce workload when optimizing images
Previously, we would initialize an ImageOptim object each time we resize.

This object init is mega expensive (170ms on a VERY fast machine):

```
[1] pry(main)> Benchmark.measure { FileHelper.image_optim   }
=> #<Benchmark::Tms:0x00007f55440c1de0
 @cstime=0.055742,
 @cutime=0.141031,
 @label="",
 @real=0.17165619300794788,
 @stime=0.0002750000000000252,
 @total=0.19890400000000008,
 @utime=0.0018560000000000798>

```

This happens cause during init it hunts for all the right binaries and sets
up internals.

We now memoize this object to avoid a huge amount of pointless work.
2019-01-09 12:28:18 +11:00
Sam df460b4abd PERF: run sidekiq with nice 5
This ensures that unicorn master forks of sidekiq run with a lower priority
than the webs. It means that a busy sidekiq is less likely to impact web
performance
2019-01-09 09:29:14 +11:00
Régis Hanol 3ec38f5a3b Revert "FIX: migrate_to_s3 rake task with folder path"
This reverts commit 97fd12e8af.
2019-01-08 19:44:31 +01:00
Régis Hanol 97fd12e8af
FIX: migrate_to_s3 rake task with folder path 2019-01-08 18:56:18 +01:00
Rishabh f181e9cc08
FIX: Add compatibility for bucket folder paths in migrate_to_s3 task (#6855)
* FIX: Add compatibility for bucket folder paths in migrate_to_s3 task
* Refactor bucket_name split logic into S3Helper
2019-01-08 20:04:48 +05:30
Guo Xiang Tan ec27db78be FIX: Set unique post key for a user outside of transaction.
Previously, the Redis key was set within the transaction and the key
isn't deleted if the transaction is not successful.

Note that this isn't tested because we don't have a repro of what can
raise an error within the transaction.

https://meta.discourse.org/t/body-is-too-similar-to-what-you-previously-posted-even-when-previous-post-didnt-go-through/105436
2019-01-08 15:22:22 +08:00
Rishabh efc481d9c0 DEV: Use puts instead of printing newline (follow up on c5b7bda1) 2019-01-05 01:20:00 +05:30
Robin Ward 5eaf3cb104 Adjusts the `minimum_flag_threshold` for TL3/TL4 actions
Before this patch, a high trust level user could flag something
and have an action be taken, as well as skipping the flag queue.

Now, if a TL3/TL4 cause an action, the flag will skip the minimum
visibility check and allow staff to review it.
2019-01-04 13:16:44 -05:00
Gerhard Schlager c0a8bb9a91 FEATURE: Include "via <site_name>" in email From header 2019-01-04 17:06:19 +01:00
Régis Hanol 788719d271 DEV: speed up posts base imports 2019-01-04 15:30:17 +01:00
Régis Hanol 95e5f8380d FEATURE: Allow plugins to add custom emoji translations
FIX: buildTranslationTree was erroring when translations overlapped (ie. ":-)" and ":-))")
FIX: emoji translations wasn't working properly when translations overlapped
2019-01-04 15:27:46 +01:00
Rishabh c5b7bda198 DEV: Show migrate_to_s3 output on a new line 2019-01-04 18:09:54 +05:30
Vinoth Kannan 902f535111 FIX: upload method in S3Helper will expect a file object param 2019-01-04 15:30:45 +05:30
Vinoth Kannan 82d7f9ce5e fix the build
Checking size for a file object directly will cause issue if it is a closed stream
2019-01-04 13:25:11 +05:30
Vinoth Kannan 940a61037c DEV: Add option to pass s3 client in param 2019-01-04 12:16:09 +05:30
Vinoth Kannan 75dbb98cca FEATURE: Add S3 etag value to uploads table (#6795) 2019-01-04 14:16:22 +08:00
Guo Xiang Tan 5f0f7f909d FIX: Incorrect CDN URL for site setting uploads when s3 is enabled. 2019-01-04 07:52:13 +08:00
Sam e2dca641c6 handle exceptions in s3:correct_acl task
We need to handle arbitrary exceptions in this task, especially since the
task is not easily resumable.

Simply output problem uploads as you hit them for now.
2019-01-04 08:32:09 +11:00
Sam 05a3e3670f FEATURE: add rake task that resets ACL on every object in S3
Some previous migrations to S3 may have bad ACLs set on objects. This
introduces a new rake task (`rake s3:correct_acl`) that will reset ACL on
every S3 object.

Vast majority of users will never have to run it, but if you have ACL issues
this is the atomic solution.
2019-01-04 08:13:43 +11:00
Gerhard Schlager b089ac1537 FIX: Posting without bump raised an error for TL4 2019-01-03 14:14:02 +01:00
Neil Lalonde 9ee08908ad Version bump to v2.2.0.beta7 2019-01-02 15:32:45 -05:00
Gerhard Schlager c30996129f FEATURE: Allow TL4 users to reset bump date 2019-01-02 16:57:05 +01:00
Gerhard Schlager e8053d6e7d FIX: Polls didn't work in imported posts
Imports skip validation of posts, but polls are only created during the validation phase.
2019-01-02 15:26:57 +01:00
Gerhard Schlager a474bf966c DEV: Skip prettier in single plugin test if patterns aren't found
This reverts d06ca90c94
It didn't work because yarn doesn't return the original exit code of the failed command.
2019-01-02 14:36:09 +01:00
Gerhard Schlager d06ca90c94 DEV: Ignore prettier error when there are no ES6 files 2019-01-02 11:30:09 +01:00
Sam 766e67ce57 FEATURE: introduce lossy color optimization on resized pngs
This feature ensures optimized images run via pngquant, this results extreme amounts of savings for resized images. Effectively the only impact is that the color palette on small resized images is reduced to 256.

To ensure safety we only apply this optimisation to images smaller than 500k.

This commit also makes a bunch of image specs less fragile.
2019-01-02 17:20:02 +11:00
Guo Xiang Tan 9e50813252 FIX: Pass all necessary options in `migrate_to_s3` rake task. 2019-01-02 09:11:23 +08:00
Guo Xiang Tan 0daaae1cf3 DEV: Fix rubocop after update. 2019-01-02 05:47:29 +08:00
David Taylor 23c65feb6c DEV: Prevent deprecation warnings being incorrectly logged 2018-12-31 14:41:30 +00:00
Sam a19170a4c2 DEV: avoid require_dependency for some libs
This avoids require dependency on method_profiler and anon cache.

It means that if there is any change to these files the reloader will not pick it up.

Previously the reloader was picking up the anon cache twice causing it to double load on boot.

This caused warnings.

Long term my plan is to give up on require dependency and instead use:

https://github.com/Shopify/autoload_reloader
2018-12-31 10:53:30 +11:00
Régis Hanol 5381096bfd PERF: new 'migrate_to_s3' rake task 2018-12-26 17:34:49 +01:00
Guo Xiang Tan 2cbb513c98 FIX: Don't use `Redis#keys` in production.
As per the documentation for KEYS

```
Warning: consider KEYS as a command that should only be used in production environments with extreme care. It may ruin performance when it is executed against large databases. This command is intended for debugging and special operations, such as changing your keyspace layout.
```

Instead SCAN

```
Since these commands allow for incremental iteration, returning only a small number of elements per call, they can be used in production without the downside of commands like KEYS or SMEMBERS that may block the server for a long time (even several seconds) when called against big collections of keys or elements.
```
2018-12-25 15:12:59 +02:00
Jeff Atwood a74e49c87c use proper typographical × instead of x 2018-12-24 20:33:17 -08:00
David Taylor 7feabd9e49 PERF: Eradicate N+1 queries from the theme admin page 2018-12-21 11:03:58 +02:00
Gerhard Schlager 0bc1fa8aa4 FEATURE: Don't create PM for successful automatic backups 2018-12-20 13:34:24 +01:00
Arpit Jalan e5fd018f44 DEV: assign constant to `preserve_fragment_url_hosts` 2018-12-19 17:37:39 +05:30
Maja Komel 2fcbbead45 FIX: move sso provider into its own class so it doesn't interfere with sso client (#6767) 2018-12-19 10:22:10 +01:00
Bianca Nenciu 2eefe6d5d6 FIX: Use CDN for logos and icons. (#6698) 2018-12-19 10:20:48 +01:00
Arpit Jalan 1ab91f0474 FIX: preserve github fragment URL 2018-12-19 12:34:47 +05:30
Rishabh cae5ba7356 FIX: Ensure that multisite s3 uploads are tombstoned correctly (#6769)
* FIX: Ensure that multisite uploads are tombstoned into the correct paths

* Move multisite specs to spec/multisite/s3_store_spec.rb
2018-12-19 13:32:32 +08:00
Robin Ward 662cfc416b FEATURE: Show a blurry preview when lazy loading images
This generates a 10x10 PNG thumbnail for each lightboxed image.
If Image Lazy Loading is enabled (IntersectionObserver API) then
we'll load the low res version when offscreen. As the image scrolls
in we'll swap it for the high res version.

We use a WeakMap to track the old image attributes. It's much less
memory than storing them as `data-*` attributes and swapping them
back and forth all the time.
2018-12-19 01:57:30 +08:00
Robin Ward e593d68beb Use an options hash instead of boolean parameters 2018-12-19 01:57:30 +08:00
Rishabh c279792130 FIX: Allow sending test e-mails to any email address when disable_email is set to non-staff (#6792) 2018-12-18 16:12:05 +01:00
Gerhard Schlager 86546bb5c6 DEV: Use locale fallback even in development mode
Making dev behave differently than prod is a bad idea.
2018-12-18 16:01:14 +01:00
Bianca Nenciu 825ae86857 FEATURE: Remove full quote only if first paragraph. (#6793) 2018-12-18 15:46:20 +01:00
Gerhard Schlager 577af81e76 FIX: Font tag resulted in wrong email trimming 2018-12-18 11:40:54 +01:00
Gerhard Schlager 37461a6398 FIX: Weird mixture of line breaks resulted in wrong email trimming 2018-12-18 11:40:54 +01:00
Guo Xiang Tan 5c2e194d01 SECURITY: Users can pick non-avatar uploads.
https://meta.discourse.org/t/bug-report-idor-on-avatar-pick-function-discussions-udacity-com/103564
2018-12-18 13:38:25 +08:00
Guo Xiang Tan 899caf35ba Revert "SECURITY: User could non-avatar uploads."
This reverts commit 89581fa301.
2018-12-18 13:37:31 +08:00
Guo Xiang Tan 89581fa301 SECURITY: User could non-avatar uploads.
https://meta.discourse.org/t/bug-report-idor-on-avatar-pick-function-discussions-udacity-com/103564
2018-12-18 13:35:33 +08:00
Guo Xiang Tan d7660dfe40 FIX: Enabling readonly mode should clear anon cache as well. 2018-12-18 11:56:25 +08:00
Sam Saffron abbbcb2622 DEV: allow creating users with random passwords via rake
Use: `RANDOM_PASSWORD=1 bin/rake admin:create`

Handy in conjunction with dev mode /session/username/become.
2018-12-18 11:43:16 +08:00
Gerhard Schlager 01cdbd3a13 FEATURE: Prohibit S3 bucket reusage
This validation makes sure that the s3_upload_bucket and the
s3_backup_bucket have different values. The backup bucket is
allowed to be a subfolder of the upload bucket. The other way
around is forbidden because the backup system searches by
prefix and would return all files stored within the backup
bucket and its subfolders.
2018-12-17 11:35:28 +01:00
Gerhard Schlager 1a8ca68ea3 FEATURE: Improve backup stats on admin dashboard
* Dashboard doesn't timeout anymore when Amazon S3 is used for backups
* Storage stats are now a proper report with the same caching rules
* Changing the backup_location, s3_backup_bucket or creating and deleting backups removes the report from the cache
* It shows the number of backups and the backup location
* It shows the used space for the correct backup location instead of always showing used space on local storage
* It shows the date of the last backup as relative date
2018-12-17 11:35:11 +01:00
Guo Xiang Tan c0aae16f6b FIX: Clear anon cache when disabling readonly mode.
`SiteSerializer#is_readonly` is cached for an anonymous user so we have
to clear the cache when disabling readonly mode. Otherwise, the site may
appear to be in readonly mode for an extended period of time.
2018-12-17 17:27:44 +08:00
Guo Xiang Tan 9e795b9d90 Follow up to 704a122656. 2018-12-17 16:39:46 +08:00
Guo Xiang Tan e75ad37aaf FIX: Switch recently readonly mdoe cache to a `DistributedCache`.
A per process cache is hard to reason about. During PostgreSQL
failovers. The site may bounce in and out of readonly mode depending on
which server and process that a request hits.
2018-12-17 13:28:36 +08:00
Guo Xiang Tan 704a122656 Remove Ruby warning due to assignment in conditional. 2018-12-17 13:08:12 +08:00
Gerhard Schlager 7e1f20b07f FIX: Create CORS rule on S3 only before a backup upload 2018-12-17 00:15:37 +01:00
Vinoth Kannan 0d3c1cde90 FIX: Use find_by_id method to prevent record not found exception 2018-12-15 03:19:45 +05:30
Neil Lalonde 4ddaceda1a Version bump to v2.2.0.beta6 2018-12-14 12:21:55 -05:00
David Taylor 77fe57f923 FEATURE: Rake task to list users which have been staff in the past month 2018-12-14 16:29:12 +00:00
Kyle Zhao b0c2e9bb05
minor changes to default script-src (#6770)
- add report-sample to force require a sample of the violating code
- do not whitelist GA/GTM's entire domain
2018-12-14 08:17:31 -05:00
Joffrey JAFFEUX 03014b0d05
FEATURE: adds security tab to dashboard (#6768)
This commit also includes the new staff_logins report
2018-12-14 13:47:59 +01:00
Sam 939b82ef0c DEV: correct intermittent test failure
ActionController::BadRequest can not be re-dispatched, under some conditions
we are getting this vs InvalidParameterError in the following test

59c56bd20f/spec/requests/application_controller_spec.rb (L34-L62)
2018-12-13 18:27:13 +11:00
Sam 94b8ba4f8f FIX: remove slow platform detection from server side
Historically due to https://meta.discourse.org/t/why-is-discourse-so-slow-on-android/8823
we decreased page sizes of both home page and topic page on android by half.

This was done on the server side and as a side effect and caused page sizes on android
to mismatch between Android and non Android.

Unfortunately about a year ago googlebot started pretending it is Android,
this cause Google to start indexing pages as what android would see. So
it saw double the amount of pages in the index as what exists on desktop.
This in turn caused double the amount of indexing work and a large amount
of broken links on long topics.

This fix removes all special behavior which is no longer needed due to
other performance work in Discourse including raw handlebars on home page
and virtual dom on topic pages.

I tested we do not need this on Blu Advance 5.0 it has 1.3 GHZ mediatec mt6580
This phone retails for around $50 USD.

If we decide long term that we want any hacks like this we will shift them
to the client side. It can just hold data in memory without rendering.
2018-12-13 13:57:05 +11:00
Bianca Nenciu 7cac04e1a8 * FEATURE: Adds site setting to let quotes on direct replies.
* DEV: Added test.
* FIX: Do not bump topic when removing full quotes.
2018-12-12 15:42:53 +01:00
Maja Komel dbbadb5c35 FEATURE: add short_site_description setting to be included in title tag on homepage 2018-12-12 11:46:58 +01:00
Sam 0ca61242b8 FEATURE: polyfill intersection observer for IE11 / iOS Safari
This feature is used for defer loading of images and in future for post cloaking

This gives us a polyfill so we can safely use the feature in problem browsers

The polyfill supports "polling" but it does not appear we need it yet.

If we discover anything odd here, consider setting poll interval per:

https://github.com/w3c/IntersectionObserver/tree/master/polyfill

```
var io = new IntersectionObserver(callback);
io.POLL_INTERVAL = 100; // Time in milliseconds.
```

Keeping the mutation observer cause we often mutate the DOM
2018-12-12 15:36:08 +11:00
David Taylor c7c56af397
FEATURE: Allow connecting associated accounts when two-factor is enabled (#6754)
Previously the 'reconnect' process was a bit magic - IF you were already logged into discourse, and followed the auth flow, your account would be reconnected and you would be 'logged in again'.

Now, we explicitly check for a reconnect=true parameter when the flow is started, store it in the session, and then only follow the reconnect logic if that variable is present. Setting this parameter also skips the 'logged in again' step, which means reconnect now works with 2fa enabled.
2018-12-11 13:19:00 +00:00
Sam 671469bcc7 FIX: URLs containing two # would fail to work
Some URLs in browsers are non compliant and contain twos `#` this commit adds
special handling for this edge case by auto encoding any fragments containing `#`
2018-12-11 18:03:13 +11:00
David Taylor 9db829134c
FIX: Use database to persist metadata during social registration (#6750)
Previously was using the cookie_store, which is limited to 4kb. This caused issues for providers sending large volumes of metadata about a user.
2018-12-10 15:10:06 +00:00
Rishabh 314c084e5d DEV: nuke code dependent comments from scheduler.rake 2018-12-10 16:16:19 +05:30
Rishabh 4767a5a26e
DEV: Update incorrect Job frequency 2018-12-10 13:26:19 +05:30
Sam e17d0c0526 FIX: don't double add users to topic allowed users
In latest rails building the same relation twice results in an error
Previous versions would deduplicate
2018-12-10 14:45:02 +11:00
Sam 955cdad649 FIX: exec_params needs instrumentation
the method no longer routes to "exec" or "async_exec" in latest PG so we
need to explicitly intercept
2018-12-10 14:28:10 +11:00
Sam 502a0fe778 FIX: support connecting GitHub with existing accounts 2018-12-10 09:27:00 +11:00
David Taylor 25aed5a47c FIX: Add missing android icon
Used in the "Recently Used Devices" section of user preferences
2018-12-07 15:49:40 +00:00
David Taylor 160d29b18a
REFACTOR: Migrate TwitterAuthenticator to use ManagedAuthenticator (#6739)
No changes to functionality. TwitterAuthenticator goes from 136 lines to 24, and all twitter-specific logic elsewhere has been deleted 🎉
2018-12-07 15:39:06 +00:00
David Taylor 86f8734bc0 FIX: Prioritize explicit 'connect' over matching by email
This is an edge case that was previously handled by TwitterAuthenticator, but not FacebookAuthenticator.
2018-12-07 15:05:51 +00:00
David Taylor 3cad3f9df1 DEV: Add profile fetching support to `ManagedAuthenticator` 2018-12-07 15:05:51 +00:00
David Taylor f7ce607e5d
FIX: Return 422 instead of 500 for invalid SSO signature (#6738) 2018-12-07 15:01:44 +00:00
David Taylor 6c71395bf6
FIX: Only hide shared draft topics from `latest` (#6737)
Previously we were hiding them from all topic lists, which can result in
topics being "stuck" in an unread state with no easy way to clear them.
2018-12-07 12:44:23 +00:00
Bianca Nenciu 41e184280d FEATURE: Remove full quotes of direct replies. (#6729) 2018-12-07 13:07:11 +01:00
Gerhard Schlager df3bb07ec6 UX: Add missing icons 2018-12-07 13:00:02 +01:00
David Taylor 5e09398c5b
FIX: Do not serialize user fields unless they are specified for display (#6736) 2018-12-07 10:57:28 +00:00
David Taylor 0b1d660876
UX: Make shared drafts behaviour consistent for non-staff users (#6734)
This makes it easier to diagnose the problem when a public category
is set as the 'shared drafts category'. Doing this is not recommended.
2018-12-06 18:59:29 +00:00
David Taylor ded3639f87 DEV: Add openid-connect as an official plugin 2018-12-06 16:16:47 +00:00
Xiao Guan 7ec124fc89 FEATURE: Improved deprecation warnings (#6722)
* FEATURE: Discourse.deprecate can report version

* Ember counterpart for deprecation
2018-12-06 11:38:01 +00:00
Gerhard Schlager 43cfdb1cb9 FIX: Wizard tries harder to find existing Welcome Topic
The wizard searches for:

* a topic that with the "is_welcome_topic" custom field
* a topic with the correct slug for the current default locale
* a topic with the correct slug for the English locale
* the oldest globally pinned topic

It gives up if it didn't find any of the above.
2018-12-06 10:27:22 +01:00
Guo Xiang Tan 978f0db109 SECURITY: Require groups to be given when inviting to a restricted category. (#6715) 2018-12-05 16:43:07 +01:00
Gerhard Schlager 99117d664c FEATURE: Multisite support for S3 backup store (#6700) 2018-12-05 10:10:39 +08:00
David Taylor e117deb2ba FIX: Improve avatar loading, and add tests
Follow-up from 4e2cc9c
2018-12-04 15:09:32 +00:00
Régis Hanol 3c9c95ac83 Update Rubocop to 0.60 2018-12-04 10:48:16 +01:00
David Taylor 9248ad1905 DEV: Enable `Style/SingleLineMethods` and `Style/Semicolon` in Rubocop (#6717) 2018-12-04 11:48:13 +08:00
Penar Musaraj 56948896ff UX: replace FA5 compress/expand icons
Adds a new SVG sprite file that contains non-FontAwesome 5 icons to our subset.

Adds the FontAwesome 4.7.0 icons for expand / collapse.
2018-12-03 22:29:20 -05:00
David Taylor 4831c4c2ee FIX: Use safe navigation operator throughout statement 2018-12-03 20:00:28 +00:00
David Taylor e32cdb00cb Fix the build 2018-12-03 18:51:07 +00:00
David Taylor b373ff47e2 FIX: Topic is nil when first post is being created 2018-12-03 18:30:52 +00:00