Discource-C
/
discourse
mirror of https://github.com/discourse/discourse.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
discourse/app/jobs/regular
History
Alan Guo Xiang Tan 9812407f76
FIX: Redo Sidekiq monitoring to restart stuck sidekiq processes (#30198) 
This commit reimplements how we monitor Sidekiq processes that are
forked from the Unicorn master process. Prior to this change, we rely on
`Jobs::Heartbeat` to enqueue a `Jobs::RunHeartbeat` job every 3 minutes.
The `Jobs::RunHeartbeat` job then sets a Redis key with a timestamp. In
the Unicorn master process, we then fetch the timestamp that has been set
by the job from Redis every 30 minutes. If the timestamp has not been
updated for more than 30 minutes, we restart the Sidekiq process. The
fundamental flaw with this approach is that it fails to consider
deployments with multiple hosts and multiple Sidekiq processes. A
sidekiq process on a host may be in a bad state but the heartbeat check
will not restart the process because the `Jobs::RunHeartbeat` job is
still being executed by the working Sidekiq processes on other hosts.

In order to properly ensure that stuck Sidekiq processs are restarted,
we now rely on the [Sidekiq::ProcessSet](https://github.com/sidekiq/sidekiq/wiki/API#processes)
API that is supported by Sidekiq. The API provides us with "near real-time (updated every 5 sec)
info about the current set of Sidekiq processes running". The API
provides useful information like the hostname, pid and also when Sidekiq
last did its own heartbeat check. With that information, we can easily
determine if a Sidekiq process needs to be restarted from the Unicorn
master process.
 		2024-12-18 12:48:50 +08:00
..
admin_confirmation_email.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
anonymize_user.rb PERF: limit anonymization to 1 per cluster (#21992) 2023-06-14 08:30:23 +10:00
automatic_group_membership.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
backfill_sidebar_site_settings.rb FIX: Set `Jobs::BackfillSidebarSiteSettings` cluster concurrency to 1 (#22316) 2023-06-28 13:07:46 +08:00
backup_chunks_merger.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
bulk_grant_trust_level.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
bulk_invite.rb DEV: Update bulk-invite logs and PM template (#27057) 2024-05-17 12:21:21 -06:00
bulk_user_title_update.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
bump_topic.rb DEV: Change Topic Timer from enqueue_at scheduled jobs to incrementally executed jobs (#11698) 2021-01-19 13:30:58 +10:00
change_display_name.rb DEV: Update to lastest rubocop-discourse 2024-05-27 18:06:14 +02:00
clear_slow_mode.rb DEV: Change Topic Timer from enqueue_at scheduled jobs to incrementally executed jobs (#11698) 2021-01-19 13:30:58 +10:00
close_topic.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
confirm_sns_subscription.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
crawl_topic_link.rb DEV: Update to lastest rubocop-discourse 2024-05-27 18:06:14 +02:00
create_avatar_thumbnails.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
create_backup.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
create_linked_topic.rb DEV: Update to lastest rubocop-discourse 2024-05-27 18:06:14 +02:00
create_user_reviewable.rb DEV: Update to lastest rubocop-discourse 2024-05-27 18:06:14 +02:00
critical_user_email.rb DEV: Fix various rubocop lints (#24749) 2023-12-06 23:25:00 +01:00
delete_inaccessible_notifications.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
delete_replies.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
delete_topic.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
download_avatar_from_url.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
download_backup_email.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
download_profile_background_from_url.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
emit_web_hook_event.rb DEV: Update to lastest rubocop-discourse 2024-05-27 18:06:14 +02:00
enable_bootstrap_mode.rb UX: make first admin a moderator to review user approvals. (#26588) 2024-04-10 20:59:03 +05:30
export_csv_file.rb DEV: Remove logical OR assignment of constants (#29201) 2024-10-16 10:09:07 +08:00
export_user_archive.rb DEV: Remove logical OR assignment of constants (#29201) 2024-10-16 10:09:07 +08:00
feature_topic_users.rb DEV: Update to lastest rubocop-discourse 2024-05-27 18:06:14 +02:00
generate_topic_thumbnails.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
group_pm_alert.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
group_pm_update_summary.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
group_smtp_email.rb DEV: Increase default SMTP read timeout to 30s (#25763) 2024-02-21 07:13:18 +10:00
index_category_for_search.rb FIX: don't error-index category job when missing category 2020-12-01 09:30:53 +08:00
index_user_fields_for_search.rb FIX: Move user reindexing into a job (#26753) 2024-04-25 20:58:34 +08:00
invite_email.rb DEV: Update to lastest rubocop-discourse 2024-05-27 18:06:14 +02:00
invite_password_instructions_email.rb DEV: Update to lastest rubocop-discourse 2024-05-27 18:06:14 +02:00
make_embedded_topic_visible.rb FIX: Better tracking of topic visibility changes (#26709) 2024-04-29 10:34:46 +10:00
mass_award_badge.rb FEATURE: Add option to grant badge multiple times to users using Bulk Award (#13571) 2021-07-15 05:53:26 +03:00
merge_user.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
notify_category_change.rb FIX: Notify mailing list subscribers on category change (#28811) 2024-10-11 14:47:39 +02:00
notify_mailing_list_subscribers.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
notify_moved_posts.rb DEV: Update to lastest rubocop-discourse 2024-05-27 18:06:14 +02:00
notify_post_revision.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
notify_reviewable.rb FEATURE: Support designating multiple groups as mods on category (#28655) 2024-09-04 04:38:46 +03:00
notify_tag_change.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
open_topic.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
post_alert.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
post_update_topic_tracking_state.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
process_bulk_invite_emails.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
process_email.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
process_post.rb DEV: Update to lastest rubocop-discourse 2024-05-27 18:06:14 +02:00
process_sns_notification.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
publish_group_membership_updates.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
publish_topic_to_category.rb DEV: Change Topic Timer from enqueue_at scheduled jobs to incrementally executed jobs (#11698) 2021-01-19 13:30:58 +10:00
pull_hotlinked_images.rb DEV: Cleanup empty method in job (#28395) 2024-08-16 14:10:46 +08:00
pull_user_profile_hotlinked_images.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
push_notification.rb DEV: Fix undefined variable (#29876) 2024-11-21 20:23:20 +02:00
rebake_custom_emoji_posts.rb
…
rebake_posts_for_upload.rb
…
refresh_users_reviewable_counts.rb DEV: Exclude system users when calculating group user count (#25400) 2024-01-25 08:13:58 +08:00
remove_banner.rb DEV: Update to lastest rubocop-discourse 2024-05-27 18:06:14 +02:00
retrieve_topic.rb DEV: Update to lastest rubocop-discourse 2024-05-27 18:06:14 +02:00
run_heartbeat.rb FIX: Redo Sidekiq monitoring to restart stuck sidekiq processes (#30198) 2024-12-18 12:48:50 +08:00
run_problem_check.rb DEV: Database backed admin notices (#26192) 2024-05-23 09:29:08 +08:00
send_push_notification.rb DEV: refactor live notifications setting in user preferences (#28145) 2024-08-02 17:25:15 +04:00
send_system_message.rb DEV: Update to lastest rubocop-discourse 2024-05-27 18:06:14 +02:00
suspicious_login.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
sync_acls_for_uploads.rb DEV: Resolve TODO comments for martin-brennan 2024-07-01 15:32:30 +10:00
sync_topic_user_bookmarked.rb DEV: Update to lastest rubocop-discourse 2024-05-27 18:06:14 +02:00
toggle_topic_closed.rb DEV: Replace raw comments with deprecation warnings (#22617) 2023-07-18 10:13:40 +08:00
topic_action_converter.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
topic_timer_base.rb DEV: Change Topic Timer from enqueue_at scheduled jobs to incrementally executed jobs (#11698) 2021-01-19 13:30:58 +10:00
truncate_user_flag_stats.rb DEV: Update to lastest rubocop-discourse 2024-05-27 18:06:14 +02:00
unpin_topic.rb DEV: Update to lastest rubocop-discourse 2024-05-27 18:06:14 +02:00
update_gravatar.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
update_group_mentions.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
update_hotlinked_raw.rb PERF: Skip validations when hotlinked images are replaced in posts 2024-01-16 14:35:24 +01:00
update_post_uploads_secure_status.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
update_top_redirection.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
update_topic_upload_security.rb DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
update_username.rb DEV: Update to lastest rubocop-discourse 2024-05-27 18:06:14 +02:00
user_email.rb DEV: Remove logical OR assignment of constants (#29201) 2024-10-16 10:09:07 +08:00