Commit Graph

200 Commits

Author SHA1 Message Date
Sam 766e67ce57 FEATURE: introduce lossy color optimization on resized pngs
This feature ensures optimized images run via pngquant, this results extreme amounts of savings for resized images. Effectively the only impact is that the color palette on small resized images is reduced to 256.

To ensure safety we only apply this optimisation to images smaller than 500k.

This commit also makes a bunch of image specs less fragile.
2019-01-02 17:20:02 +11:00
Régis Hanol 5381096bfd PERF: new 'migrate_to_s3' rake task 2018-12-26 17:34:49 +01:00
Gerhard Schlager 2bdbca3801 DEV: Remove unnecessary `to_not raise_error` from specs
Follow-up to 01cdbd3a13
2018-12-17 16:10:10 +01:00
Gerhard Schlager 01cdbd3a13 FEATURE: Prohibit S3 bucket reusage
This validation makes sure that the s3_upload_bucket and the
s3_backup_bucket have different values. The backup bucket is
allowed to be a subfolder of the upload bucket. The other way
around is forbidden because the backup system searches by
prefix and would return all files stored within the backup
bucket and its subfolders.
2018-12-17 11:35:28 +01:00
Gerhard Schlager 1a8ca68ea3 FEATURE: Improve backup stats on admin dashboard
* Dashboard doesn't timeout anymore when Amazon S3 is used for backups
* Storage stats are now a proper report with the same caching rules
* Changing the backup_location, s3_backup_bucket or creating and deleting backups removes the report from the cache
* It shows the number of backups and the backup location
* It shows the used space for the correct backup location instead of always showing used space on local storage
* It shows the date of the last backup as relative date
2018-12-17 11:35:11 +01:00
Kyle Zhao b0c2e9bb05
minor changes to default script-src (#6770)
- add report-sample to force require a sample of the violating code
- do not whitelist GA/GTM's entire domain
2018-12-14 08:17:31 -05:00
Gerhard Schlager 43cfdb1cb9 FIX: Wizard tries harder to find existing Welcome Topic
The wizard searches for:

* a topic that with the "is_welcome_topic" custom field
* a topic with the correct slug for the current default locale
* a topic with the correct slug for the English locale
* the oldest globally pinned topic

It gives up if it didn't find any of the above.
2018-12-06 10:27:22 +01:00
Gerhard Schlager 99117d664c FEATURE: Multisite support for S3 backup store (#6700) 2018-12-05 10:10:39 +08:00
David Taylor 9248ad1905 DEV: Enable `Style/SingleLineMethods` and `Style/Semicolon` in Rubocop (#6717) 2018-12-04 11:48:13 +08:00
Kyle Zhao 488fba3c5f
FEATURE: allow plugins and themes to extend the default CSP (#6704)
* FEATURE: allow plugins and themes to extend the default CSP

For plugins:

```
extend_content_security_policy(
  script_src: ['https://domain.com/script.js', 'https://your-cdn.com/'],
  style_src: ['https://domain.com/style.css']
)
```

For themes and components:

```
extend_content_security_policy:
  type: list
  default: "script_src:https://domain.com/|style_src:https://domain.com"
```

* clear CSP base url before each test

we have a test that stubs `Rails.env.development?` to true

* Only allow extending directives that core includes, for now
2018-11-30 09:51:45 -05:00
Penar Musaraj 03deda2147
Upgrade to FontAwesome 5 (take two) (#6673)
* Add missing icons to set

* Revert FA5 revert

 This reverts commit 42572ff

* use new SVG syntax in locales

* Noscript page changes (remove login button, center "powered by" footer text)

* Cast wider net for SVG icons in settings

- include any _icon setting for SVG registry (offers better support for plugin settings)

- let themes store multiple pipe-delimited icons in a setting

- also replaces broken onebox image icon with SVG reference in cooked post processor

* interpolate icons in locales

* Fix composer whisper icon alignment

* Add support for stacked icons

* SECURITY: enforce hostname to match discourse hostname

This ensures that the hostname rails uses for various helpers always matches
the Discourse hostname

* load SVG sprite with pre-initializers

* FIX: enable caching on SVG sprites

* PERF: use JSONP for SVG sprites so they are served from CDN

This avoids needing to deal with CORS for loading of the SVG

Note, added the svg- prefix to the filename so we can quickly tell in
dev tools what the file is

* Add missing SVG sprite JSONP script to CSP

* Upgrade to FA 5.5.0

* Add support for all FA4.7 icons

- adds complete frontend and backend for renamed FA4.7 icons

- improves performance of SvgSprite.bundle and SvgSprite.all_icons

* Fix group avatar flair preview

- adds an endpoint at /svg-sprites/search/:keyword

- adds frontend ajax call that pulls icon in avatar flair preview even when it is not in subset

* Remove FA 4.7 font files
2018-11-26 16:49:57 -05:00
Sam 86255faa08 FEATURE: do not switch to JPEG unless you meet 75k byte savings
This also adjusts the algorithm to expect

- 30% saving for JPEG conversion

AND

- Minimum of 75K bytes saved

The reasoning for increase of saving requirements is cause PNG may have been
uploaded unoptimized, 30% saving on PNG is very possible
2018-11-21 11:01:08 +11:00
Kyle Zhao 962fbd1ec7 include '/plugins/' directory for script-src and blob for worker-src
- plugins may include additional static JS assets
- ACE.js editor register a service worker with a blob for syntax
checking
2018-11-16 16:31:01 -05:00
Kyle Zhao 055d59373a
CSP: drop 'self' in `script-src` (#6611) 2018-11-15 12:14:16 -05:00
Guo Xiang Tan 44391ee8ab
FEATURE: Upload Site Settings. (#6573) 2018-11-14 15:03:02 +08:00
Guo Xiang Tan 57f92ac808 Revert "Swtich to regexp for `DbHelper.remap`."
Regexp is so much slower.

This reverts commit c3f89e3cd7.
2018-11-08 14:20:09 +08:00
Guo Xiang Tan c3f89e3cd7 Swtich to regexp for `DbHelper.remap`. 2018-11-08 14:08:38 +08:00
Guo Xiang Tan 9737938a4a Add option to skip tabels when using `DbHelper.remap`. 2018-11-08 12:29:37 +08:00
Guo Xiang Tan 3365753bd0 PERF: Reduce number of database queries for `DbHelper.remap`
* Cuts number of queries from 273 to 89
* Add some specs
* For a table with 500 posts, benchmarks locally shows a runtime
  reduction from 0.046929135 to 0.032694705.
2018-11-08 10:54:39 +08:00
Kyle Zhao e9a971a2b6
FEATURE: [Experimental] Content Security Policy (#6514)
do not register new MIME type, parse raw body instead
2018-10-22 13:22:23 -04:00
Bianca Nenciu 99b43f281b FIX: Fix browser detection for Microsoft Edge. (#6516)
cool!
2018-10-22 23:15:41 +11:00
Kyle Zhao dca830cb73 Revert "FEATURE: [Experimental] Content Security Policy (#6504)"
This reverts commit fb8231077a.
2018-10-19 11:53:29 -04:00
Kyle Zhao fb8231077a
FEATURE: [Experimental] Content Security Policy (#6504) 2018-10-19 10:39:22 -04:00
Guo Xiang Tan 84d4c81a26 FEATURE: Support backup uploads/downloads directly to/from S3.
This reverts commit 3c59106bac.
2018-10-15 09:43:31 +08:00
Guo Xiang Tan 3c59106bac Revert "FEATURE: Support backup uploads/downloads directly to/from S3."
This reverts commit c29a4dddc1.

We're doing a beta bump soon so un-revert this after that is done.
2018-10-11 11:08:23 +08:00
Gerhard Schlager c29a4dddc1 FEATURE: Support backup uploads/downloads directly to/from S3. 2018-10-11 10:38:43 +08:00
Bianca Nenciu 1d26a473e7 FEATURE: Show "Recently used devices" in user preferences (#6335)
* FEATURE: Added MaxMindDb to resolve IP information.

* FEATURE: Added browser detection based on user agent.

* FEATURE: Added recently used devices in user preferences.

* DEV: Added acceptance test for recently used devices.

* UX: Do not show 'Show more' button if there aren't more tokens.

* DEV: Fix unit tests.

* DEV: Make changes after code review.

* Add more detailed unit tests.

* Improve logging messages.

* Minor coding style fixes.

* DEV: Use DropdownSelectBoxComponent and run Prettier.

* DEV: Fix unit tests.
2018-10-09 22:21:41 +08:00
Guo Xiang Tan de85bb0a39 FIX: Don't update user_profile URLs unless upload is persisted. 2018-10-01 14:21:39 +08:00
Guo Xiang Tan e262a08350 Add `UploadRecovery#recover_user_profile_backgrounds`. 2018-10-01 10:51:54 +08:00
Guo Xiang Tan 1a64b3a487 FIX: Don't try to recover an invalid sha1. 2018-09-20 14:21:57 +08:00
Sam df45e82377 SECURITY: only allow picking of avatars created by self (#6417)
* SECURITY: only allow picking of avatars created by self

Also adds origin tracking to all uploads including de-duplicated uploads
2018-09-19 22:33:10 -07:00
Guo Xiang Tan d403883d16 DEV: Improve specs for 293cf600f0. 2018-09-19 16:03:52 +08:00
Guo Xiang Tan bc7f58191e FIX: `UploadRecovery` should look at links too. 2018-09-19 11:52:57 +08:00
Guo Xiang Tan 4a92c5b2d6 `UploadRecovery` should recover attachments too. 2018-09-19 10:44:36 +08:00
Guo Xiang Tan 0a06b3d977 Accept custom AR relation for `UploadRecovery`. 2018-09-13 16:33:14 +08:00
Guo Xiang Tan 5eb65ad612 FIX: Do not try to recover invalid `Upload#short_url` in `UploadRecovery`. 2018-09-13 13:59:17 +08:00
Guo Xiang Tan 1afe7162e1 Fix the build. 2018-09-13 13:41:38 +08:00
Guo Xiang Tan d99dd840e4 Add basic test case for `UploadRecovery`. 2018-09-13 13:26:23 +08:00
Guo Xiang Tan 71caf7521d Drop `tif`, `tiff`, `webp` and `bmp` from supported images.
https://meta.discourse.org/t/cr2-raw-files-are-being-treated-as-tiff-files/96775/3?u=tgxworld
2018-09-12 09:29:54 +08:00
Régis Hanol d1607a387a FIX: only allow printable characters in uploads filename 2018-08-21 18:11:01 +02:00
Sam f5fe58384f correct regression around file renaming 2018-08-20 16:08:05 +10:00
Sam d7b1919ead correct specs 2018-08-20 12:46:14 +10:00
Sam 8b5e42ea16 FIX: always test and coerce to image on upload
In the past the filename of the origin was used as the source
for the extension of the file when optimizing on upload.

We now use the actual calculated extension based on upload data.
2018-08-20 12:18:59 +10:00
Guo Xiang Tan 4b7c5ba162 Update `UploadCreator` specs to include cropping code path. 2018-08-17 15:22:12 +08:00
Guo Xiang Tan a9e502936f FIX: Converting PNG to JPEG does not set the correct extension. 2018-08-17 13:09:48 +08:00
Gerhard Schlager a115aae45f Use rchardet instead of charlock_holmes gem 2018-08-01 10:41:20 +02:00
Gerhard Schlager ff942ed2f3 FIX: Try detecting encoding of RSS feed 2018-08-01 10:41:20 +02:00
David Taylor abddb48260 Rubocop fix 2018-07-24 10:49:20 +01:00
David Taylor 20a21b1240 Move into MiniSQLMultisiteConnection, and add test for rollback 2018-07-24 09:41:55 +01:00
Michael Brown 63a1e9b60a backup restorer: tidy pg_dump schema portability logic, add test 2018-03-20 10:32:39 +08:00