Commit Graph

109 Commits

Author SHA1 Message Date
Vinoth Kannan 914ada1c74 DEV: convert scheduled job EnsurePostUploadsExistence into a rake task 2019-04-09 02:07:35 +05:30
Vinoth Kannan b3fb0a7039 FEATURE: ensure consistency of post uploads in cooked content 2019-04-04 02:23:28 +05:30
Guo Xiang Tan feb731bffd FIX: Regenerate optimized images instead of migrating from old scheme.
`OptimizedImage.migrate_to_new_scheme` was optimizing optimized images
which we don't need to do. Regnerating the optimized image is way easier.
2019-04-03 09:45:02 +08:00
Guo Xiang Tan 839a54b97b FIX: Destroy `OptimizedImage` record even if `Upload` record is invalid. 2019-03-22 16:47:06 +08:00
Régis Hanol ad12b2a23d
FIX: only call 'unlink' for tempfiles 2019-03-08 11:22:20 +01:00
Penar Musaraj c50db76f5d FIX: do not treat TIFF, BMP, WEBP as images
Treating TIFF and BMP as images cause us to add them to IMG tags, this is very inconsistent across browsers.

You can still upload these files they will simply not be displayed in IMG tags.
2019-02-11 16:28:43 +11:00
Robin Ward 95f263995d FIX: Previous annotations were broken 2019-01-11 14:30:19 -05:00
Robin Ward a3839495e0 Update annotations 2019-01-11 12:19:43 -05:00
Sam 7e4a43afc6 PERF: run ImageMagick conversions with nice 10
This lowers the priority of all image resizes so they do not clog CPU on
machines running low on resources
2019-01-09 09:21:58 +11:00
Sam 9c91e68351 PERF: remove image optimization throttling from Sidekiq
Previously we only allowed one image optimization per machine, this meant there
was cross talk between avatar resizing and Sidekiq. This could lead to large
amounts of starvation when optimized image version changed which in turn could
block the Sidekiq queue.

This increases amount of allowed load on machines but this is preferable to
having crosstalk between avatar resizing and Sidekiq.
2019-01-04 18:44:44 +11:00
Sam 570877da3c FEATURE: store thumbnail algorithm version in optimized image table
Previously we had no idea what algorithm generated thumbnails, this starts tracking the version.

We also bumped up the version to force all optimized images to be generated. This is important cause we recently introduced pngquant which results in much smaller images.
2019-01-03 17:07:30 +11:00
Sam 766e67ce57 FEATURE: introduce lossy color optimization on resized pngs
This feature ensures optimized images run via pngquant, this results extreme amounts of savings for resized images. Effectively the only impact is that the color palette on small resized images is reduced to 256.

To ensure safety we only apply this optimisation to images smaller than 500k.

This commit also makes a bunch of image specs less fragile.
2019-01-02 17:20:02 +11:00
Robin Ward 6080e3a2c0 FIX: Broken spec 2018-12-18 14:55:09 -05:00
Robin Ward 662cfc416b FEATURE: Show a blurry preview when lazy loading images
This generates a 10x10 PNG thumbnail for each lightboxed image.
If Image Lazy Loading is enabled (IntersectionObserver API) then
we'll load the low res version when offscreen. As the image scrolls
in we'll swap it for the high res version.

We use a WeakMap to track the old image attributes. It's much less
memory than storing them as `data-*` attributes and swapping them
back and forth all the time.
2018-12-19 01:57:30 +08:00
Sam 0a442e319c FIX: correct svg handling for images
We regressed and optimized images no longer worked with svg

The following adds the correct logic to simply copy file for svgs
and bypasses resizing for svg avatars
2018-11-07 15:29:26 +11:00
Arpit Jalan 72be638728 FEATURE: add external details to user fields 2018-09-20 08:10:51 +05:30
Sam e0e6dae6a7 minor cleanup to previous commit from code review 2018-09-19 16:07:29 +10:00
Sam 5302709343 FIX: in redis readonly raise an exception from DistributedMutex
If we detect redis is in readonly we can not correctly get a mutex
raise an exception to notify caller

When getting optimized images avoid the distributed mutex unless
for some reason it is the first call and we need to generate a thumb

In redis readonly no thumbnails will be generated
2018-09-19 15:50:58 +10:00
Sam 9ab1fb7dfc FEATURE: correctly store width and height on uploads
Previously we used width and height for thumbnails, new code ensures

1. We auto correct width and height
2. We added extra columns for thumbnail_width and height, this is determined
 by actual upload and no longer passed in as a side effect
3. Optimized Image now stores filesize which can be used for analysis, decisions

Also

- fixes Android image manifest as a side effect
- fixes issue where a thumbnail generated that is smaller than the upload is no longer used
2018-08-28 12:59:22 +10:00
Sam 8b5e42ea16 FIX: always test and coerce to image on upload
In the past the filename of the origin was used as the source
for the extension of the file when optimizing on upload.

We now use the actual calculated extension based on upload data.
2018-08-20 12:18:59 +10:00
Sam 9628c3cf97 FEATURE: automatically correct extension for bad uploads
This fixes with post thumbnails on the fly
2018-08-17 14:00:27 +10:00
Sam f62073a22a correct regression uploading images 2018-08-16 18:49:08 +10:00
Sam 796164b58c FIX: automatically correct bad avatars on access
Also start relying on upload extension for optimized images
2018-08-16 16:32:56 +10:00
Sam 33b5cc6bbd FIX: Add gif to list of allowed decoders 2018-08-15 13:56:57 +10:00
Sam cf788582cc DEV: split optimize image failure warnings in logs 2018-08-15 13:27:24 +10:00
Sam 83fd308963 FEATURE: group error message regarding image optimization failures 2018-08-13 14:48:25 +10:00
Gerhard Schlager 6d813c2b52 FIX: Importers failed to import avatars 2018-08-12 22:02:17 +02:00
Guo Xiang Tan f8b367cc9c Fix the build. 2018-07-26 10:17:51 +08:00
Guo Xiang Tan 30242972d3 Add spec for 4bf3bf6786. 2018-07-26 09:16:14 +08:00
Régis Hanol f94aeaf6cf SECURITY: force IM decoder based on file extension - part 3 2018-07-25 23:55:06 +02:00
Régis Hanol 800c57c6ab SECURITY: force IM decoder based on file extension - part 2 2018-07-25 23:08:02 +02:00
Régis Hanol 4bf3bf6786 SECURITY: force IM decoder based on file extension 2018-07-25 22:00:04 +02:00
Guo Xiang Tan 487ef9773f Better logging when optimizing image fails. 2018-07-18 14:11:23 +08:00
Guo Xiang Tan 1d74ccaaf8 Add compatibility for ImageMagick7. 2018-07-17 15:50:58 +08:00
Guo Xiang Tan 142571bba0 Remove use of `rescue nil`.
* `rescue nil` is a really bad pattern to use in our code base.
  We should rescue errors that we expect the code to throw and
  not rescue everything because we're unsure of what errors the
  code would throw. This would reduce the amount of pain we face
  when debugging why something isn't working as expexted. I've
  been bitten countless of times by errors being swallowed as a
  result during debugging sessions.
2018-04-02 13:52:51 +08:00
Guo Xiang Tan 226ace1643 Update annotations. 2018-02-20 14:28:58 +08:00
Sam c42bb94362 PERF: slow down optimizing images per host 2018-01-24 08:53:17 +11:00
Sam bfd052a317 PERF: guard so we only create one optimized image per process
This protectd sidekiq and other cases where optimized images are created on
demand so they do not dominate the machine.
2018-01-24 08:22:28 +11:00
Arpit Jalan daeb7694bc update annotations 2017-12-05 21:03:20 +05:30
Guo Xiang Tan 5012d46cbd Add rubocop to our build. (#5004) 2017-07-28 10:20:09 +09:00
Régis Hanol c7c93e7159 FEATURE: new 'strip image metadata' site setting 2017-07-25 11:48:39 +02:00
Régis Hanol aa69f78fed let's not use progressing rendering on images 2017-06-26 23:19:48 +02:00
Régis Hanol 6e5c59e5ec FIX: image orientation wasn't properly working 2017-06-22 16:53:49 +02:00
Robin Ward cdbe027c1c Refactor `FileHelper` to use keyword arguments. 2017-05-24 13:54:26 -04:00
Régis Hanol 9641d2413d REFACTOR: upload workflow creation into UploadCreator
- Automatically convert large-ish PNG/BMP to JPEG
- Updated fast_image to latest version
2017-05-11 00:16:57 +02:00
Robin Ward 24e1892d28 Silence logs for OptimizedImage failures 2017-05-09 14:59:46 -04:00
Guo Xiang Tan ddc8abf9d8 Fix broken optimize image command. 2017-03-20 15:07:38 +08:00
Guo Xiang Tan e7c972ac89 FIX: Don't use backticks that take in inputs. 2017-03-17 15:33:51 +08:00
Régis Hanol c4552ef8c1 slightly shorter regexp for OptimizedImage.safe_path? 2017-01-02 16:28:14 +01:00
Sam 15b5fddd49 SECURITY: protect upload params, only allow very strict filenames 2016-12-19 10:16:18 +11:00