9db05a895a
DEV: Add job_id to sidkiq log
...
This makes it easier to correlate 'pending' logs from the same job
2019-03-08 09:16:13 +00:00
1121514799
UX: Localize date format in "new user of the month" message
2019-03-06 21:58:25 +01:00
df474bceee
DEV: Further sidekiq logging stability improvements
...
- Open the log file in "append" mode. This avoids issues if the file does not exist (and matches standard rails log behavior)
- Correctly parse the interval logging environment variable
2019-03-06 12:50:15 +00:00
fe62de68dd
DEV: Correct sidekiq logging to avoid thread leak
2019-03-06 10:11:31 +00:00
8b30ed5b7a
DEV: Serialize the job parameters in sidekiq logs
...
Otherwise this can lead to some very large data structures when processing the logs later
2019-03-05 17:44:49 +00:00
8963f1af30
FEATURE: Optional detailed performance logging for Sidekiq jobs ( #7091 )
...
By default, this does nothing. Two environment variables are available:
- `DISCOURSE_LOG_SIDEKIQ`
Set to `"1"` to enable logging. This will log all completed jobs to `log/rails/sidekiq.log`, along with various db/redis/network statistics. This is useful to track down poorly performing jobs.
- `DISCOURSE_LOG_SIDEKIQ_INTERVAL`
(seconds) Check running jobs periodically, and log their current duration. They will appear in the logs with `status:pending`. This is useful to track down jobs which take a long time, then crash sidekiq before completing.
2019-03-05 11:19:11 +00:00
326d892f5e
Aadd 'secondary_emails' field in users export
...
FIX: escape_comma wasn't working in CSV exports
FIX: group_names field wasn't properly serialized
2019-02-27 10:12:20 +01:00
e3a23116d2
FIX: don't update gravatar if the user has no email
2019-02-20 22:34:43 +01:00
fc14847c14
PERF: only require aws-sdk-sns gem when it's being used
2019-02-14 11:08:21 +01:00
484bd82278
FIX: Add onceoff job to remove double quotes from s3 etags
2019-02-14 05:19:41 +05:30
4d674acc25
FEATURE: AWS SNS bounce notifications webhooks
2019-02-13 21:26:40 +01:00
b4f713ca52
FEATURE: Use amazon s3 inventory to manage upload stats ( #6867 )
2019-02-01 10:10:48 +05:30
78ddc82952
FIX: Respect min_flags_staff_visibility for new flags too
...
There was a situation where if:
* There were new flags to review that met the visibility threshold
AND
* There were old flags that *didn't* meet the threshold
THEN
a pending flags notification would be sent out. This fixes that case.
Staff should not be notified of flags if they do not meet the threshold
and are old.
2019-01-25 11:27:43 -05:00
96b2585a91
REFACTOR: Remove unncessary stubs from pending flags reminder
...
They seem to be calculated fine by the application, and stubbing
makes the tests more brittle and prone to regression.
2019-01-24 13:45:58 -05:00
f32de88dfc
FIX: Don't notify of pending flags if min_flags_staff_visibility not met
2019-01-22 11:01:18 -05:00
95e3841974
FIX: Remove old reference and use `MiniScheduler::Stat`.
2019-01-18 16:36:11 +08:00
384135845b
FEATURE: introduce ultra_low priority queue
...
This commit introduces an ultra low priority queue for post rebakes. This
way rebakes can never interfere with regular sidekiq processing for cases
where we perform a large scale rebake.
Additionally it allows Post.rebake_old to be run with rate_limiter: false
to avoid triggering the limiter when rebaking. This is handy for cases
where you want to just force the full rebake and not wait for it to trickle
2019-01-17 14:53:19 +11:00
7d84648d11
FEATURE: Remove full quotes only from new posts. ( #6862 )
2019-01-17 13:24:32 +11:00
b63b399799
DEV: remove uploaded_meta_id column from category ( #6725 )
...
* DEV: remove uploaded_meta_id column from category
* remove uploaded_meta part
2019-01-10 09:37:21 +08:00
e08a3f719c
FEATURE: push post rebake regular task to low priority queue
...
This allows us to run regular rebakes without starving the normal queue.
It additionally adds the ability to specify queue with `Jobs.enqueue` so
we can specifically queue a job with lower priority using the `queue` arg.
2019-01-09 08:57:20 +11:00
8f602be2fe
FEATURE: keep the topic in closed status until the community flags are handled
2019-01-08 16:13:10 +05:30
5bf16d7d10
FEATURE: Topic timer for bumping a topic in the future
2019-01-04 13:08:04 +00:00
1a8ca68ea3
FEATURE: Improve backup stats on admin dashboard
...
* Dashboard doesn't timeout anymore when Amazon S3 is used for backups
* Storage stats are now a proper report with the same caching rules
* Changing the backup_location, s3_backup_bucket or creating and deleting backups removes the report from the cache
* It shows the number of backups and the backup location
* It shows the used space for the correct backup location instead of always showing used space on local storage
* It shows the date of the last backup as relative date
2018-12-17 11:35:11 +01:00
f74ef71130
DEV: add transaction and active check to invalidate job
...
Follows up on 3b76f19668
2018-12-13 14:59:56 +11:00
3b76f19668
FIX: invalidating inactive admin emails should mark them as not active
2018-12-12 17:07:49 -05:00
a1db15fead
FEATURE: require admins to re-validate their email addresses if they haven't been seen for a number of days, configurable with the invalidate_inactive_admin_email_after_days site setting. Social logins are also revoked. Default is 365 days.
2018-12-12 15:32:38 -05:00
9db829134c
FIX: Use database to persist metadata during social registration ( #6750 )
...
Previously was using the cookie_store, which is limited to 4kb. This caused issues for providers sending large volumes of metadata about a user.
2018-12-10 15:10:06 +00:00
9e3143445b
DEV:add uploaded_meta option in category for category meta image ( #6724 )
2018-12-07 16:24:07 +01:00
8e307e633e
FIX: posts would not auto rebake unless gravatar download was enabled
2018-12-07 17:03:22 +11:00
d33d031742
FEATURE: Filter topic and post web hook events by tags ( #6726 )
...
* FEATURE: Filter topic and post web hook events by tags
* Add a spec test with unmatched tags
2018-12-05 14:44:06 +05:30
3c9c95ac83
Update Rubocop to 0.60
2018-12-04 10:48:16 +01:00
eecd1a7d8c
FIX: `Jobs::CleanUpUploads` fails when value of upload data_type is an empty string.
2018-11-30 10:46:39 +08:00
ba280f9cf3
FIX: Give up migrating url site setting if endpoint doesn't return data.
2018-11-22 12:16:52 +08:00
a8603e04ab
FIX: Give up migration of URL site settings if there is an SSL error.
2018-11-22 12:14:27 +08:00
df72674f24
Warn when migrate url site settings job encounters an error.
2018-11-21 07:55:50 +08:00
81b3bdaabd
FIX: Remove site settings override for deprecated url site settings.
2018-11-20 11:42:39 +08:00
9fd704735e
Rescue from `SocketError` in migrate url onceoff job.
2018-11-19 13:07:34 +08:00
9e86b425bc
FIX: Job to clean up old URL settings when new setting has been set.
...
Related to 44391ee8ab
2018-11-16 09:33:31 +08:00
901611cb4e
Fix migrate upload url site settings to rescue from invalid URLs.
2018-11-15 15:16:14 +08:00
5b320e6652
DEV: Don't warn about using deprecated site settings in clean up uploads job.
2018-11-14 20:58:06 +08:00
df111259fe
More URL site settings into a onceoff job.
...
* Doing it in a post migration was a bad idea
because the migration will fail if the site
is down while trying to download uploads
which points to the instance. This mainly
affects self-hosters using `discourse_docker`
where `./launcher rebuild` will take the
existing container down.
2018-11-14 20:29:20 +08:00
44391ee8ab
FEATURE: Upload Site Settings. ( #6573 )
2018-11-14 15:03:02 +08:00
7b44339529
FIX: Prevent uploads used in site settings from being deleted.
2018-11-13 09:15:16 +08:00
24e5be3f0c
FIX: Relative links in translations should work with subfolder
2018-11-08 23:31:05 +00:00
2070edf889
FIX: Clarify User.group_locked_trust_level.
...
* Rename User.group_locked_trust_level to User.group_granted_trust_level.
* Remove the column from users table.
2018-11-07 10:27:44 +08:00
e0ccd36dbe
FEATURE: Suspicious logins report. ( #6544 )
2018-10-30 22:51:58 +00:00
6a3767cde7
FEATURE: Warn users via email about suspicious logins. ( #6520 )
...
* FEATURE: Warn users via email about suspicious logins.
* DEV: Move suspicious login check to a job.
2018-10-25 09:45:31 +00:00
de6b585368
minor, bypass gravatar update if user does not match
...
this protects against a race condition that can happen when a user record
is destroyed reasonably quickly
2018-10-23 12:20:41 +11:00
0f1afad6da
FIX: extracted theme JavaScripts for multisite ( #6502 )
...
* FIX: extracted theme javascripts for multisite
* onceoff to rebake all theme fields
2018-10-18 17:05:34 +11:00
8d06731484
FIX: reduce amount of work onceoff does
...
In the past onceoff was forcing inline download of gravatars,
this can be so expensive that it will never finish
This fix ensures it only marks avatars stale which will be picked
up by regular schedules
2018-10-16 10:29:16 +11:00
84d4c81a26
FEATURE: Support backup uploads/downloads directly to/from S3.
...
This reverts commit 3c59106bac
2018-10-15 09:43:31 +08:00
09961fb425
FIX: properly escape name of custom emoji
2018-10-11 09:35:23 +02:00
3c59106bac
Revert "FEATURE: Support backup uploads/downloads directly to/from S3."
...
This reverts commit c29a4dddc1
2018-10-11 11:08:23 +08:00
c29a4dddc1
FEATURE: Support backup uploads/downloads directly to/from S3.
2018-10-11 10:38:43 +08:00
e0e0a91e7d
FIX: Retry sending email in case of temporary issue. ( #6375 )
2018-10-04 22:56:24 +08:00
3a6ab02563
FEATURE: Notify admins when scheduling of backup fails
2018-10-02 15:48:16 +02:00
d8fa7fbbab
Fix onceoff job in cfa7173da3 not running.
2018-10-01 17:14:04 +08:00
cfa7173da3
FIX: Onceoff job to fix missing user profile backgrounds.
2018-10-01 16:26:40 +08:00
02da022c70
PERF: Quit out of the email job quickly if disabled ( #6423 )
...
This prevents sidekiq from doing a bunch of queries when email is
disabled.
Critical emails are a special case and will be sent.
2018-10-01 01:15:45 +08:00
767f27929d
Rename `Jobs::RecoverPostUploads` to rerun the job take 2.
2018-09-19 22:40:32 +08:00
6dbe1d832d
Rename `Jobs::RecoverPostUploads` to rerun the job.
...
Running a migration to destroy the onceoff job doesn't work
because an instance running the old code may execute the job.
2018-09-19 21:59:44 +08:00
667e68ec58
Prefer `create` over `create!` when not checking for response.
2018-09-19 18:22:54 +08:00
aa1af9fc22
FIX: Onceoff job to recover missing post uploads.
...
This fixes the regression due to 1f636c445b
2018-09-14 09:04:01 +08:00
e1b16e445e
Rename `FileHelper.is_image?` -> `FileHelper.is_supported_image?`.
2018-09-12 09:22:28 +08:00
797cbf8653
FIX: Remove user fields when anonymizing user
2018-09-07 00:02:56 +02:00
b6a139b581
Fix broken spec.
2018-09-06 12:41:43 +08:00
d9c0dc8687
correct prev commit
...
s3. did not exists it is s3-
2018-09-05 16:11:44 +10:00
83e1315e42
FIX: correct urls in uploads table to point at dualstack
...
Last week we added support for dual stack urls but did not remap the
the old records in the uploads and optimized images table
This caused a few minor edge cases worst was that if you rebaked old
images S3 CDN was not repopulated.
2018-09-05 15:58:04 +10:00
e4498d2a8a
FIX: keep db and job correctly in multisite logs
...
This ensures we report job and db correctly, previously we were
only reporting this on default
2018-09-04 16:05:44 +10:00
e1975e293f
FIX: when uploads are destroyed clear up avatar refs in user table
...
This also auto corrects twice daily when we ensure consistency
2018-08-31 14:46:42 +10:00
4f81d4cc8e
Minor fixes to `Jobs::BulkInvite`.
2018-08-30 15:35:16 +08:00
b2cf725700
FIX: Don't try to send invite email when invite was deleted
2018-08-29 12:43:12 +02:00
44cf3cf975
FIX: queue heartbeats in readonly modes
...
If sidekiq is paused or Discourse is in readonly continue to queue
heartbeats
If we do not do that then a master process can end up reaping sidekiq
workers and causing various badness
This also impacts restore which can do weird stuff TM in cases like this
2018-08-29 12:36:59 +10:00
740308675b
FEATURE: erode bounce score every time an email is sent
...
Introduces a hidden setting (default is 0.1) that erodes bounce score
every time we send an email. This means that erratic failures are less
painful cause system auto corrects
2018-08-28 17:02:12 +10:00
9ab1fb7dfc
FEATURE: correctly store width and height on uploads
...
Previously we used width and height for thumbnails, new code ensures
1. We auto correct width and height
2. We added extra columns for thumbnail_width and height, this is determined
by actual upload and no longer passed in as a side effect
3. Optimized Image now stores filesize which can be used for analysis, decisions
Also
- fixes Android image manifest as a side effect
- fixes issue where a thumbnail generated that is smaller than the upload is no longer used
2018-08-28 12:59:22 +10:00
36a7028f19
FEATURE: Clean up `PostReplyKey` records.
...
* Default retention of 90 days.
2018-08-23 10:40:02 +08:00
8bdf14834b
PERF: Restrict number of skipped email log for `Jobs::UserEmail`.
2018-08-21 11:14:43 +08:00
2c70d3f443
Take 2 on ba6f11c521.
2018-08-21 10:06:36 +08:00
860c1c3dcd
FEATURE: Automatically expire keys if not used for a configurable amount of time. ( #6264 )
2018-08-20 17:36:14 +02:00
16c0ebe8a8
Fix the build.
2018-08-17 16:53:07 +08:00
796164b58c
FIX: automatically correct bad avatars on access
...
Also start relying on upload extension for optimized images
2018-08-16 16:32:56 +10:00
1f17b84b63
FEATURE: more context for error reporting on jobs fails
2018-08-16 12:38:49 +10:00
de92913bf4
FIX: store the topic links using the cooked upload url
2018-08-14 12:23:32 +02:00
ba022234c6
Add onceoff job to fix incorrect upload extensions.
2018-08-13 10:44:08 +08:00
b73950692b
FIX: Parsing non-existent feed should not fail
2018-08-10 18:37:14 +02:00
3cd4dc0f5f
Allow users with group_locked_trust_level to be promoted to tl3 ( #6249 )
2018-08-10 10:42:23 +10:00
ba6f11c521
PERF: Only log the first skipped email when user exceeds daily limit.
...
https://meta.discourse.org/t/cleaning-up-e-mail-logs/39132
2018-08-08 16:25:00 +08:00
d5a7b818ba
FIX: Move Jobs::CreateAvatarThumbnails to low priority queue.
2018-08-07 16:05:54 +08:00
4e11811321
FIX: `UserAvatar#update_gravatar!` does not update `User#uploaded_avatar`.
...
https://meta.discourse.org/t/missing-user-profile-pictures/93844/4
2018-08-06 13:29:44 +08:00
880462a41c
FEATURE: display out of date themes on admin dashboard
...
* FEATURE: disaply out of date themes on admin dashboard
* Update copy
2018-08-03 09:53:48 +10:00
b829452c75
Merge pull request #6209 from discourse/mini_scheduler
...
REFACTOR: extract scheduler to the mini_scheduler gem
2018-08-01 10:28:24 -04:00
0b9437cee7
FIX: more resilient/consistent dashboard caching ( #6223 )
2018-08-01 09:45:50 -04:00
a115aae45f
Use rchardet instead of charlock_holmes gem
2018-08-01 10:41:20 +02:00
5d421fb946
FIX: Try respecting charset in HTTP header of RSS feed
2018-08-01 10:41:20 +02:00
ff942ed2f3
FIX: Try detecting encoding of RSS feed
2018-08-01 10:41:20 +02:00
4ad7ce70ce
REFACTOR: extract scheduler to the mini_scheduler gem
2018-07-31 17:12:55 -04:00
acde8d4323
Fix the build.
2018-07-30 15:07:03 +08:00
96a0448c52
FIX: Add onceoff job to fix incorrect extension for gravatar uploads.
2018-07-30 14:44:02 +08:00
87537b679c
Drop `reply_key`, `skipped` and `skipped_reason` from `email_logs`.
2018-07-30 11:39:28 +08:00
ccf76d45f2
FIX: Missing variable outside of `begin` block.
2018-07-27 08:19:11 +08:00
796639a797
FIX: makes disk_space computation more resilient ( #6172 )
2018-07-25 11:04:01 -04:00
c3b6811651
PERF: Split loading of posts to speed up user renames
2018-07-24 11:57:04 +02:00
fad9c2b971
PERF: Move `EmailLog#reply_key` into new `post_reply_keys` table.
2018-07-24 13:51:53 +08:00
ae8b0a517f
PERF: Split skipped email logs into a seperate table.
2018-07-24 13:14:37 +08:00
f8e9190617
FEATURE: Retry web hook when it is failed
2018-07-23 10:12:04 +08:00
06323f9c89
FIX: takes old dashboard out of caching job
2018-07-21 12:42:03 -04:00
0972dd2552
FIX: takes dashboard-next out of caching job
2018-07-21 11:13:15 -04:00
1d5096eb46
FIX: lazy load more reports in dashboard
2018-07-20 23:35:53 -04:00
1a78e12f4e
FEATURE: part 2 of dashboard improvements
...
- moderation tab
- sorting/pagination
- improved third party reports support
- trending charts
- better perf
- many fixes
- refactoring
- new reports
Co-Authored-By: Simon Cossar <scossar@users.noreply.github.com>
2018-07-19 14:33:11 -04:00
6d6e026e3c
FEATURE: selectable avatars
2018-07-18 12:57:43 +02:00
91266cdabb
correct auto bump topic logic
2018-07-17 09:33:33 +10:00
ac0053f491
FEATURE: navigate to first post and auto bump category settings
...
### navigate_to_first_post_after_read setting for categories
When enabled on categories logged on users will return to OP after
reading the entire category. (useful for documentation categories)
### num_auto_bump_daily
Set a number of topics that will automatically bump daily on a category.
- Every 15 minutes we will check if any category has this setting
- Categories with the setting are shuffled
- We exclude pinned, closed, category description and archived topics
- Maximum of 1 topic for the list of categories is bumped till limit reached per category
- We always try to bump oldest first
- Limit is elastic using a RateLimiter that ensures that we only bump N per day
Also some minor organisation on category settings
Froze strings on category.rb
2018-07-16 18:10:35 +10:00
4172e1dd52
FIX: Rename `User#usernames` that clashes with `Group#name`. ( #6069 )
2018-07-09 16:54:57 +08:00
f7322c05ad
FIX: Avoid `destroy_all` in `Jobs::CleanUpUploads`.
...
`destroy_all` loads all the relation into memory as once.
See https://github.com/rails/rails/issues/22510
2018-07-02 12:44:36 +08:00
7550e9ff95
FIX: purge unactivated users with a message from non-human users
2018-06-29 13:03:04 +05:30
c5e24da5dd
Update poll_mailbox.rb
2018-06-25 16:18:07 +10:00
cbaf521fc1
correct regression and add integrity spec for onceoffs
2018-06-20 09:09:31 +10:00
5f64fd0a21
DEV: remove exec_sql and replace with mini_sql
...
Introduce new patterns for direct sql that are safe and fast.
MiniSql is not prone to memory bloat that can happen with direct PG usage.
It also has an extremely fast materializer and very a convenient API
- DB.exec(sql, *params) => runs sql returns row count
- DB.query(sql, *params) => runs sql returns usable objects (not a hash)
- DB.query_hash(sql, *params) => runs sql returns an array of hashes
- DB.query_single(sql, *params) => runs sql and returns a flat one dimensional array
- DB.build(sql) => returns a sql builder
See more at: https://github.com/discourse/mini_sql
2018-06-19 16:13:36 +10:00
12fd0b827e
PERF: release results in a couple of cases
2018-06-13 15:11:43 +10:00
2564a8285d
REFACTOR: Run some parts of user anonymizing in background job
2018-06-08 15:50:07 +02:00
ad5082d969
Make rubocop happy again.
2018-06-07 13:28:18 +08:00
d600e71b3d
FIX: Clean up stale `UserExport` records daily.
...
* Add tests for `UserExport.remove_old_exports`
2018-06-05 09:43:40 +08:00
6024fb84cc
make rubocop happy
2018-06-04 19:06:52 +02:00
ab8cdd6b5b
Upload.url can't be NULL
2018-06-04 18:43:00 +02:00
085eaaf18d
FIX: always delete invalid upload records
2018-06-04 18:40:57 +02:00
587a9c82f5
FIX: Use the username formatter when updating cooked usernames
2018-05-25 17:29:06 -04:00
71f66cd679
FIX: ensure PostAlerter is always run in sidekiq
2018-05-24 17:27:43 +02:00
cafd1241b9
FIX: make report data export more resilient to inconsistent date format
...
The DAU/MAU report export was failing because of date being a string
and not a valid Date value
2018-05-24 17:14:08 +05:30
43f7cb05c9
FIX: Broken ping event for web hooks due to missing payload.
2018-05-24 15:16:52 +08:00
95db5f0c8c
FIX: Do not replace wrong avatars when renaming user
2018-05-22 13:41:51 +02:00
24abf38d38
PERF: lower the priority on user emails
...
User emails can wait behind other regular jobs, they are usually somewhat
slower as they involve smtp transactions
2018-05-22 15:20:55 +10:00
bf84037f79
FIX: Payload for webhooks should be current as of the time the event was triggered.
...
https://meta.discourse.org/t/group-category-tag-user-deleted-webhooks-not-firing/87752
2018-05-21 17:29:58 +08:00
1841dd48dc
FIX: revert utf-8 encode changes
2018-05-20 17:50:36 +05:30
9512796ef6
FIX: check for blank response when polling feed
2018-05-18 17:00:44 +05:30
1ff767559d
Fix incorrect param.
2018-05-18 14:05:45 +08:00
c42b65df5f
`find` raises an error if the record is missing.
2018-05-18 13:37:07 +08:00
238a13643d
FIX: handle missing users when sending push notifications
2018-05-17 12:53:19 +05:30
39aceed63d
FIX: don't purge unactivated users with a message
2018-05-16 18:24:11 +02:00
146f634c8f
FIX: UTF-8 encode feed response body
2018-05-16 12:59:23 +05:30
21e0b7c818
avoid async report pattern and replace with simpler hijack
2018-05-16 16:05:03 +10:00
74c4af279a
Improvements to user renaming
...
* don't update search index if post belongs to deleted topic
* log errors instead of crashing when updating post or revision fails
* update mentions even when the href attribute is missing
* run the background job with low priority
* replace username in all notifications
* update `action_code_who` used by small action posts
2018-05-15 21:05:51 +02:00
2e1b5bc8d3
FIX: Transaction in UserAnonymizer prevented avatar from updating
2018-05-15 20:47:58 +02:00
9947c38e1c
UX: support for multiple datasets in one chart
2018-05-15 20:12:03 +02:00
193b6d5651
UX: improve new dashboard
...
- top referred topics
- limit search logs to 8 results
2018-05-15 15:08:36 +10:00
a28c58feb1
FIX: automatic group membership when using SSO
2018-05-15 01:48:30 +02:00
c67c2dc638
FIX: Username update should ignore revisions without raw
2018-05-14 15:22:42 +02:00
8232aba743
FIX: Update search index after renaming user within posts
2018-05-14 13:20:30 +02:00
02cb84847c
FIX: Update avatar in oneboxed posts when renaming user
2018-05-14 13:20:29 +02:00
6332d5040d
UX: switch dashboard to be the new dashboard
...
Also:
- add pageviews
- add problems and version sections
2018-05-14 13:07:59 +10:00
bc9e0d46af
PERF: use cached reports for dashboard if available
2018-05-14 12:01:44 +10:00
David Taylor
Gerhard Schlager
Régis Hanol
Vinoth Kannan
Vinoth Kannan
Robin Ward
Guo Xiang Tan
Sam
Bianca Nenciu
Saurabh Patel
Neil Lalonde
Guo Xiang Tan
Bianca Nenciu
Kyle Zhao
Simon Cossar
Osama Sayegh
Joffrey JAFFEUX
Arpit Jalan
Ernesto Serrano