Commit Graph

220 Commits

Author SHA1 Message Date
Guo Xiang Tan 52c4711e80 DEV: Fix method that was incorrectly made private. 2019-04-03 12:37:50 +08:00
Guo Xiang Tan 4037a2fd0b FIX: Restrict scope of old scheme upload migration.
Some sites have external URLs that don't even match `%/uploads/%' and
some sites surprise me with URLs that contains the default path when it
is a site in a multisite cluster. We can't do anything about those.
2019-04-03 11:56:41 +08:00
Guo Xiang Tan feb731bffd FIX: Regenerate optimized images instead of migrating from old scheme.
`OptimizedImage.migrate_to_new_scheme` was optimizing optimized images
which we don't need to do. Regnerating the optimized image is way easier.
2019-04-03 09:45:02 +08:00
Guo Xiang Tan d0a1922ad9 FIX: `Upload.migrate_to_new_scheme` undefined error when external image fails to download. 2019-03-28 16:00:13 +08:00
Guo Xiang Tan 161a3c3870 DEV: Don't optimize image when migrating to new scheme.
The image has already been uploaded, the migrate to new scheme job's is
just to correct the scheme and not the content of the upload.
2019-03-26 15:12:10 +08:00
Guo Xiang Tan 0ac6a58b09 FIX: Skip validation when saving in `Upload#migrate_to_new_scheme`. 2019-03-26 14:28:39 +08:00
Guo Xiang Tan 40b03e717b FIX: `Upload#migrate_to_new_scheme` should not migrate system uploads. 2019-03-14 12:45:03 +08:00
Guo Xiang Tan b0c8fdd7da FIX: Properly support defaults for upload site settings. 2019-03-13 16:36:57 +08:00
Guo Xiang Tan 58b0e945bd
UX: Lightbox support for image uploader. (#7034) 2019-02-21 10:13:37 +08:00
Robin Ward 95f263995d FIX: Previous annotations were broken 2019-01-11 14:30:19 -05:00
Robin Ward a3839495e0 Update annotations 2019-01-11 12:19:43 -05:00
Vinoth Kannan 75dbb98cca FEATURE: Add S3 etag value to uploads table (#6795) 2019-01-04 14:16:22 +08:00
Sam Saffron 303a535dba PERF: automatic upload size calculation not persisted
Previously if upload had missing width and height we would calculate
on first use BUT we (me) forgot to save this to the database

This was particularly bad on home page cause category images (when old)
miss dimensions.
2018-12-26 17:19:29 +02:00
Robin Ward 662cfc416b FEATURE: Show a blurry preview when lazy loading images
This generates a 10x10 PNG thumbnail for each lightboxed image.
If Image Lazy Loading is enabled (IntersectionObserver API) then
we'll load the low res version when offscreen. As the image scrolls
in we'll swap it for the high res version.

We use a WeakMap to track the old image attributes. It's much less
memory than storing them as `data-*` attributes and swapping them
back and forth all the time.
2018-12-19 01:57:30 +08:00
Robin Ward e593d68beb Use an options hash instead of boolean parameters 2018-12-19 01:57:30 +08:00
Penar Musaraj f8e6a37858 FIX: raise exception when getting dimensions of missing image
- follow-up on 0eacd45ab1
2018-12-03 10:19:49 -05:00
Penar Musaraj 0eacd45ab1 FIX: refactor ImageSizer.resize
reverts 140d9c2
2018-11-29 15:28:45 -05:00
Penar Musaraj 140d9c2910 FIX: call ImageSizer only if width/height are available 2018-11-29 15:03:02 -05:00
Guo Xiang Tan 44391ee8ab
FEATURE: Upload Site Settings. (#6573) 2018-11-14 15:03:02 +08:00
Arpit Jalan 3e17ef0507
Merge pull request #6414 from techAPJ/user-field-unique-key
FEATURE: add external details to user fields
2018-09-20 22:29:39 +05:30
Sam df45e82377 SECURITY: only allow picking of avatars created by self (#6417)
* SECURITY: only allow picking of avatars created by self

Also adds origin tracking to all uploads including de-duplicated uploads
2018-09-19 22:33:10 -07:00
Arpit Jalan 72be638728 FEATURE: add external details to user fields 2018-09-20 08:10:51 +05:30
Guo Xiang Tan 4a92c5b2d6 `UploadRecovery` should recover attachments too. 2018-09-19 10:44:36 +08:00
Guo Xiang Tan 3884e99e88 Add extra protection in `Upload#get_from_url`.
In case the extension goes missing from the URL.
2018-09-12 00:12:14 -07:00
Guo Xiang Tan b3469bea2d FIX: Uploads not being linked correctly to posts.
Regression due to 1f636c445b.
2018-09-11 23:50:23 -07:00
Guo Xiang Tan e1b16e445e Rename `FileHelper.is_image?` -> `FileHelper.is_supported_image?`. 2018-09-12 09:22:28 +08:00
Guo Xiang Tan 04d26c65e2 Refactor `Upload.get_from_url` to check length of sha1. 2018-09-10 10:10:39 +08:00
Guo Xiang Tan 1f636c445b PERF: Add fast path to find uploads before resorting to `LIKE` query.
For a normal upload url

Before

```
Warming up --------------------------------------
                       264.000  i/100ms
Calculating -------------------------------------
                          2.754k (± 8.4%) i/s -     13.728k in   5.022066s
```

After

```
Warming up --------------------------------------
                       341.000  i/100ms
Calculating -------------------------------------
                          3.435k (±11.6%) i/s -     17.050k in   5.045676s
```
2018-09-06 14:44:24 +08:00
Sam e1975e293f FIX: when uploads are destroyed clear up avatar refs in user table
This also auto corrects twice daily when we ensure consistency
2018-08-31 14:46:42 +10:00
Sam 9ab1fb7dfc FEATURE: correctly store width and height on uploads
Previously we used width and height for thumbnails, new code ensures

1. We auto correct width and height
2. We added extra columns for thumbnail_width and height, this is determined
 by actual upload and no longer passed in as a side effect
3. Optimized Image now stores filesize which can be used for analysis, decisions

Also

- fixes Android image manifest as a side effect
- fixes issue where a thumbnail generated that is smaller than the upload is no longer used
2018-08-28 12:59:22 +10:00
Sam 9628c3cf97 FEATURE: automatically correct extension for bad uploads
This fixes with post thumbnails on the fly
2018-08-17 14:00:27 +10:00
Sam 796164b58c FIX: automatically correct bad avatars on access
Also start relying on upload extension for optimized images
2018-08-16 16:32:56 +10:00
Régis Hanol de92913bf4 FIX: store the topic links using the cooked upload url 2018-08-14 12:23:32 +02:00
Régis Hanol 0cd9e2acb9 fix build 2018-08-04 01:56:26 +02:00
Régis Hanol bf4d98e89d FIX: always store topic links using the upload url 2018-08-04 01:29:32 +02:00
Guo Xiang Tan 875008522d FIX: `Discourse.S3BaseUrl` did not account for subfolder bucket names. 2018-07-06 15:53:57 +08:00
Guo Xiang Tan 7163bf9323 FIX: `Upload.get_from_url` not respective subfolder in s3 bucket names. 2018-07-06 11:37:11 +08:00
Maja Komel 6c9190357c fix get from url for external uploads without CDN 2018-06-13 16:21:28 +10:00
Arpit Jalan 290ee312e6 FIX: handle invalid mailto links 2018-05-18 18:02:52 +05:30
Arpit Jalan 539cf32f87 FIX: handle encoded mailto links when looking for upload record 2018-05-17 12:44:53 +05:30
Arpit Jalan 91bf10bd12 FIX: create upload record for exported csv files 2018-04-20 00:27:49 +05:30
Guo Xiang Tan 142571bba0 Remove use of `rescue nil`.
* `rescue nil` is a really bad pattern to use in our code base.
  We should rescue errors that we expect the code to throw and
  not rescue everything because we're unsure of what errors the
  code would throw. This would reduce the amount of pain we face
  when debugging why something isn't working as expexted. I've
  been bitten countless of times by errors being swallowed as a
  result during debugging sessions.
2018-04-02 13:52:51 +08:00
Guo Xiang Tan bfe38b2118 FIX: `limit` wasn't correctly applied to `Upload.migrate_to_new_scheme`. 2018-03-22 10:56:06 +08:00
Guo Xiang Tan 226ace1643 Update annotations. 2018-02-20 14:28:58 +08:00
Rafael dos Santos Silva b9a343afe7 FIX: Allow shared CDN for s3 and assets 2018-01-12 01:08:15 -02:00
Arpit Jalan daeb7694bc update annotations 2017-12-05 21:03:20 +05:30
Sam 4ea87b5ab8 Merge branch 's3_refactor' 2017-10-09 10:27:52 +11:00
Sam 70bb2aa426 FEATURE: allow specifying s3 config via globals
This refactors handling of s3 so it can be specified via GlobalSetting

This means that in a multisite environment you can configure s3 uploads
without actual sites knowing credentials in s3

It is a critical setting for situations where assets are mirrored to s3.
2017-10-06 16:20:01 +11:00
Guo Xiang Tan d67f0b39ae Update annotations. 2017-10-06 11:13:01 +08:00
Sam f766ea4257 Correct short url decoding for sha1s leading with zero
This was picked up cause we had a flaky test!
2017-08-23 11:08:18 -04:00
Sam bcf7dc38c2 FEATURE: server side support for upload:// markdown
This allows uploads to be specified using short sha1 hash instead of full
URL

Client side change is pending
2017-08-22 11:46:23 -04:00
Sam c68999e128 annotate models
WARNING annotators out there, be to run bin/annotate on RAILS_ENV=test on a clean db
2017-08-16 10:38:11 -04:00
Neil Lalonde 5d528f0d15 Merge pull request #4958 from dmacjam/search_posts_by_filetype
FEATURE: Search posts by filetype
2017-07-31 11:55:34 -04:00
Guo Xiang Tan 5012d46cbd Add rubocop to our build. (#5004) 2017-07-28 10:20:09 +09:00
Neil Lalonde d8c27e3871 Merge branch 'master' into search_posts_by_filetype 2017-07-25 14:41:20 -04:00
Régis Hanol c7c93e7159 FEATURE: new 'strip image metadata' site setting 2017-07-25 11:48:39 +02:00
Jakub Macina 8c445e9f17 Fix backend code for searching by a filetype as a combination of uploads and topic links. Add rspec test for extracting file extension in upload. 2017-07-06 19:19:31 +02:00
Régis Hanol 54e8fb0d89 FEATURE: new 'allow_staff_to_upload_any_file_in_pm' site setting 2017-06-12 22:41:29 +02:00
Robin Ward cdbe027c1c Refactor `FileHelper` to use keyword arguments. 2017-05-24 13:54:26 -04:00
Régis Hanol 13e489b4ca replace the upload type whitelist with a sanitizer 2017-05-18 12:13:13 +02:00
Arpit Jalan 8e5b0c79ae FIX: allow images to be uploaded in wizard 2017-05-18 13:53:23 +05:30
Arpit Jalan 8c337ecf82 FIX: allow uploading of category logo and background 2017-05-11 12:48:54 +05:30
Régis Hanol 214939bb87 freeze array constants 2017-05-11 09:08:59 +02:00
Régis Hanol 9641d2413d REFACTOR: upload workflow creation into UploadCreator
- Automatically convert large-ish PNG/BMP to JPEG
- Updated fast_image to latest version
2017-05-11 00:16:57 +02:00
Sam bc0b9af576 FEATURE: support uploads for themes
This allows themes to bundle various assets
2017-05-10 15:47:11 -04:00
Guo Xiang Tan 2af1b9e93c Add time out when optimizing images. 2017-04-20 15:21:43 +08:00
Guo Xiang Tan 60f1169077 REFACTOR: Reduce repetition in code. 2017-04-18 17:03:49 +08:00
Guo Xiang Tan e7c972ac89 FIX: Don't use backticks that take in inputs. 2017-03-17 15:33:51 +08:00
Guo Xiang Tan 1a7e954e09 FIX: Store custom emojis as uploads.
* Depending on a hardcoded directory was a flawed design
  which made it impossible to debug when custom emojis go
  missing.
2017-03-14 13:07:18 +08:00
Régis Hanol 887e9af84f FEATURE: new 'max_image_megapixels' site setting 2017-01-11 23:37:12 +01:00
Régis Hanol 8d48779b5c FIX: don't 💥 with an invalid URI 2016-10-20 12:34:42 +02:00
Guo Xiang Tan 7db33cc512 FIX: Videos and audio files were not associated to the post. 2016-10-18 16:13:39 +08:00
Guo Xiang Tan e4b75f604c FIX: Make clean up upload script a safer task to run. 2016-09-05 10:06:02 +08:00
Guo Xiang Tan 1a4a0d7e89 FIX: Don't fail silently. 2016-09-02 11:59:03 +08:00
Guo Xiang Tan 692ecff3eb Revert "FIX: Don't fail silently."
This reverts commit baa6af93a2.
2016-09-02 11:58:56 +08:00
Guo Xiang Tan baa6af93a2 FIX: Don't fail silently. 2016-09-02 11:53:53 +08:00
Guo Xiang Tan efd7cbd887 Remove limit default.
Having the limit makes it harder to migrate all `Upload`/`OptimizedImage`
since the count has to be figured out and passed to the method.
2016-09-02 10:55:11 +08:00
Régis Hanol 5169bcdb6e FIX: httpshttps ultra secure URLs 2016-06-30 16:55:01 +02:00
Régis Hanol 5e2545a578 FEATURE: improve support for (whitelisted) SVGs as images 2016-06-20 10:22:13 +02:00
Régis Hanol 32d253d484 remove 'crop_tall_images' site setting but keep the behavior 2016-05-23 16:42:19 +02:00
Régis Hanol 667dd54a23 FEATURE: new 'crop_tall_images' site setting 2016-05-23 16:18:30 +02:00
Guo Xiang Tan 0634834009 Some fixes related to optimized images (#4233)
* FIX: No need to manually include relation.

* FIX: OR instead of chaining relation.
2016-05-20 09:12:25 +02:00
Régis Hanol df14926e42 SECURITY: check magic bytes before using ImageMagick tools 2016-05-03 21:54:07 +02:00
Régis Hanol be5a54d67d FEATURE: new 'allow_all_attachments_for_group_messages' site setting 2016-02-29 22:39:24 +01:00
Sam 32c681c96b annotate models 2016-02-23 10:33:53 +11:00
Régis Hanol 4d981cec53 FIX: don't try to optimize large PNGs (takes too much time) 2016-02-22 12:57:24 +01:00
Régis Hanol a9099f9e23 SECURITY: ensure we never accept fake images 2015-12-21 16:08:14 +01:00
Régis Hanol 09bfe49254 FIX: don't automagically downsize uploaded images that are larger than 10MB
FIX: don't optimize GIFs since ImageOption was disabled for GIFs (too slow)
2015-11-26 18:16:47 +01:00
Régis Hanol fb62a7c0c3 FIX: only downsize user card & profile backgrounds 2015-09-24 21:04:06 +02:00
Régis Hanol a3831a7003 FIX: uploading an animated user card/profile background was converted to a still image 2015-09-20 22:01:03 +02:00
Régis Hanol 93f9dcfcec FIX: don't overwrite custom uploaded avatar when selecting gravatar
FIX: remove unecessary serialized fields
2015-09-11 15:10:56 +02:00
Sam cd8d82aa31 correct file size and add note about impending breakage of image_optim 2015-09-10 14:37:46 +10:00
Régis Hanol d456460d33 FIX: don't butcher GIFs
Use 'gifsicle' instead of 'convert' to resize & optimize GIFs

FIX: don't even try to fix GIFs orientation
FIX: use 'allow_animated_thumbnails' site setting for user profile backgrounds & user cards
2015-07-22 17:10:42 +02:00
Régis Hanol b0802abae2 FIX: crop & optimize user background profile/card images 2015-07-15 17:15:43 +02:00
Gerhard Schlager 2e8838a0cd FIX: Disable validation during thumbnail creation 2015-06-27 01:26:16 +02:00
Régis Hanol bc9fd2c46d don't silence these errors 2015-06-12 20:11:23 +02:00
Régis Hanol 189cb3ff12 FEATURE: move migrate_to_new_scheme into a background job
- new hidden site setting 'migrate_to_new_scheme' (defaults to false)
- new rake tasks to toggle migration to new scheme
- FIX: migrate_to_new_scheme also works with CDN
- PERF: improve perf of the DbHelper.remap method
- REFACTOR: UrlHelper is now a class
2015-06-12 12:07:57 +02:00
Régis Hanol 64e73e98fb FIX: allow the cooked_post_processor to download external uploads 2015-06-01 20:08:41 +02:00
Régis Hanol 61d85206ee FIX: optimize uploaded images using lossy but very fast compression 2015-05-29 15:57:24 +02:00
Régis Hanol e101396ea1 FEATURE: add support for device pixel ratio = 3 2015-05-28 01:48:07 +02:00