discourse/lib
Sam 29fac1ac18 PERF: improve performance of unread queries
Figuring out what unread topics a user has is a very expensive
operation over time.

Users can easily accumulate 10s of thousands of tracking state rows
(1 for every topic they ever visit)

When figuring out what a user has that is unread we need to join
the tracking state records to the topic table. This can very quickly
lead to cases where you need to scan through the entire topic table.

This commit optimises it so we always keep track of the "first" date
a user has unread topics. Then we can easily filter out all earlier
topics from the join.

We use pg functions, instead of nested queries here to assist the
planner.
2017-05-25 15:07:30 -04:00
..
active_record/connection_adapters Use a different Redis key when PG failover sets site to readonly mode. 2017-01-11 16:38:49 +08:00
auth add some explicit scoping to help avoid erratic failure in test 2017-03-07 16:00:51 -05:00
autospec detect dead qunit 2017-05-24 12:24:34 -04:00
backup_restore Fix typo. 2017-03-24 20:59:34 +08:00
common_passwords fix build & add migration to clear common passwords cache 2016-03-03 19:39:22 +01:00
demon Allow a sidekiq queue to be configured to only run on a certain hostname. 2017-04-27 15:32:16 +08:00
email FIX: remove memoization on class method used in a job 2017-05-22 23:35:41 +02:00
emoji add missing flags emoji 2016-09-19 19:39:46 +02:00
es6_module_transpiler FEATURE: upgrade sprockets to latest stable 2017-04-17 10:12:05 -04:00
file_store Refactor `FileHelper` to use keyword arguments. 2017-05-24 13:54:26 -04:00
freedom_patches FEATURE: Native theme support 2017-04-12 10:53:49 -04:00
guardian FIX: Don't let users edit wiki posts unless they can reply 2017-05-08 16:23:11 -04:00
highlight_js Updated highlight.js 2016-03-09 08:58:44 +01:00
i18n FIX: i18n integrity specs 2017-02-24 11:35:33 +01:00
import
import_export FEATURE: Category setting to make all topics wikis 2016-12-19 06:42:18 -06:00
javascripts Update moment locales as well. 2017-04-20 12:24:20 +08:00
middleware FEATURE: Native theme support 2017-04-12 10:53:49 -04:00
onebox UX: show expand button on internal topic onebox 2017-03-08 21:02:38 +05:30
plugin FIX: don't search for plugins in nested subdirectories 2017-05-16 17:28:56 -04:00
pretty_text Fix `Fixnum` is deprecated in Ruby 2.4. 2017-04-25 15:19:12 +08:00
rate_limiter FIX: do not raise exception if user not found while rate limiting 2016-04-13 20:08:03 +05:30
scheduler Log error for all exceptions in scheduler stats. 2017-04-26 09:33:05 +08:00
search Remove SearchObserver, aim is to remove all observers 2016-12-22 13:13:14 +11:00
sidekiq FIX: disable scheduled jobs when in readonly mode 2016-01-11 18:31:28 +01:00
site_settings fix the build 2015-09-09 15:34:53 +02:00
stylesheet FIX: CDN wasn't properly applied to category background images when using S3 2017-05-22 18:37:01 +02:00
tasks Refactor `FileHelper` to use keyword arguments. 2017-05-24 13:54:26 -04:00
validators FEATURE: support uploads for themes 2017-05-10 15:47:11 -04:00
wizard FIX: Invalid creation of `Theme` in wizard builder. 2017-05-04 11:44:23 +08:00
admin_confirmation.rb SECURITY: Confirm new administrator accounts via email 2017-04-04 15:59:01 -04:00
admin_constraint.rb FIX: Don't raise error when admin access is invalid. 2016-12-05 17:36:45 +08:00
admin_user_index_query.rb Add pagination to /admin/users/list API 2017-05-12 13:28:35 -07:00
age_words.rb reused value of Time.now in a method 2015-12-01 21:06:37 +00:00
archetype.rb
auth.rb Added Instagram login method 2016-02-25 12:13:59 +10:00
avatar_lookup.rb FIX: don't overwrite custom uploaded avatar when selecting gravatar 2015-09-11 15:10:56 +02:00
badge_queries.rb Change Anniversary badge to be multiple grant, once per year 2017-04-28 14:22:54 -04:00
cache.rb
canonical_url.rb
category_badge.rb FIX: box style category badges in emails, and shrink text and some spacing in summary email 2016-12-09 16:59:54 -05:00
column_dropper.rb rename on_remove, on_drop 2017-05-04 14:03:17 -04:00
comment_migration.rb
composer_messages_finder.rb Convert server side paths to use `/u/` 2017-03-30 10:23:24 -04:00
configurable_urls.rb
content_buffer.rb
cooked_post_processor.rb FIX: Don't raise an error when the img attribute cannot be found 2017-05-08 15:35:31 -04:00
crawler_detection.rb Update crawler_detection.rb 2017-05-16 11:17:05 -04:00
current_user.rb SECURITY: don't grant same privileges to user_api and api access 2016-12-16 12:05:43 +11:00
custom_renderer.rb
custom_setting_providers.rb Plugins can register providers for global settings 2017-01-09 17:18:58 -05:00
db_helper.rb
directory_helper.rb
discourse.rb FIX: Don't use backticks that take in inputs. 2017-03-17 15:33:51 +08:00
discourse_cookie_store.rb FEATURE: add support for same site cookies 2017-02-23 12:01:28 -05:00
discourse_diff.rb FIX: Don't diplay character reference in HTML diffs (#4204) 2016-05-09 08:44:21 +02:00
discourse_event.rb Remove `DiscourseEvent.clear`. 2016-09-05 15:17:49 +08:00
discourse_hub.rb FEATURE: Send anonymized usage statistics to Discourse if Discourse Hub can't reach the site 2017-03-28 09:07:23 +05:30
discourse_iife.rb Allow the loader to understand when files are moved 2016-11-15 11:41:16 -05:00
discourse_plugin.rb
discourse_plugin_registry.rb Allow plugins to add vendored files for the text pipeline 2017-04-18 17:59:05 -04:00
discourse_redis.rb FIX: Set master to true before forcing slave connections to reconnect. 2016-11-23 14:04:43 +08:00
discourse_tagging.rb use named param in tag_topic_by_names 2017-02-28 12:08:06 -05:00
discourse_updates.rb FIX: latest-version tag is shown by git-describe 2017-03-08 11:05:11 +00:00
disk_space.rb FIX: Allow for large free disk space 2016-01-25 13:22:36 +11:00
distributed_cache.rb FIX: DistributedCache would fail serialization in some cases 2016-01-30 09:01:15 +11:00
distributed_memoizer.rb FIX: select appropriate period when redirecting to top 2015-09-21 20:28:20 +02:00
distributed_mutex.rb
edit_rate_limiter.rb Add better error messages for rate limits. 2015-09-24 13:52:46 -04:00
email.rb UX: display text & html parts alongside raw email in incoming email modal 2017-03-08 23:15:42 +01:00
email_backup_token.rb FEATURE: further restrict downloading of backups 2017-03-01 08:28:34 -07:00
email_cook.rb Email Cooker can onebox posts too 2016-07-18 15:00:12 -04:00
email_updater.rb FIX: Don't allow invalid email to be saved. 2016-12-21 17:47:11 +08:00
enum.rb Enums that are used in tables need to be stable 2016-01-08 20:43:11 +05:30
enum_site_setting.rb
excerpt_parser.rb FIX: Quotes should be ignored when parsing for onebox source. 2017-04-11 15:22:21 +08:00
file_helper.rb FIX: Don't rate limit gravatar downloads 2017-05-24 13:54:26 -04:00
filter_best_posts.rb
final_destination.rb FIX: Don't rate limit gravatar downloads 2017-05-24 13:54:26 -04:00
flag_query.rb FIX: Don't notify or return flags on system users 2017-05-08 13:13:35 -04:00
gaps.rb
git_importer.rb FEATURE: support uploads for themes 2017-05-10 15:47:11 -04:00
global_path.rb Allow theme field object model to support uploads 2017-05-10 15:46:42 -04:00
guardian.rb SECURITY: Validate the `entity` when downloading a CSV 2017-05-19 16:00:51 -04:00
has_errors.rb Add `PostCreator#create!`. 2016-07-15 11:36:06 +08:00
headless-ember.js
homepage_constraint.rb FIX: Suppress error when acccess is invalid. 2017-01-06 13:18:04 +08:00
html_prettify.rb FEATURE: stop escaping special chars in title prettify 2017-03-13 10:02:20 -04:00
html_to_markdown.rb FIX: supports bare <li> when converting html to markdown 2017-05-17 15:05:11 +02:00
image_sizer.rb FIX: cropping GIF wasn't working 2016-07-27 18:48:02 +02:00
introduction_updater.rb Add new welcome message step 2016-09-22 09:52:19 -04:00
ip_addr.rb Replace certain uses of 'gsub' with 'tr' or 'chomp' for a speed 2016-06-10 22:08:37 -05:00
js_locale_helper.rb FIX: admin locales were not getting converted to message format 2017-03-25 01:12:23 +05:30
json_error.rb
letter_avatar.rb Fix rspec tests. 2017-03-20 12:35:08 +08:00
markdown_linker.rb
mem_info.rb
memory_diagnostics.rb compact! only returns array if changes were made 2015-10-27 12:27:37 -04:00
message_bus_diags.rb
mobile_detection.rb radically simplify our mobile user agent detection 2016-03-26 02:02:23 -07:00
new_post_manager.rb FEATURE: log reason staff auto blocks a user 2017-03-10 15:45:48 -05:00
new_post_result.rb
notification_levels.rb Server side code for Watching First Post Only 2016-07-07 11:21:50 -04:00
oneboxer.rb Use `FinalDestination` to ensure public redirects for onebox 2017-05-22 16:42:49 -04:00
onpdiff.rb
pbkdf2.rb
permalink_constraint.rb
pinned_check.rb
plugin_gem.rb Plugins can register providers for global settings 2017-01-09 17:18:58 -05:00
post_action_creator.rb Add specs for post action guardian 2015-12-30 20:52:36 +01:00
post_creator.rb PERF: improve performance of unread queries 2017-05-25 15:07:30 -04:00
post_destroyer.rb REFACTOR: Load `Post` records in batches when destroying stubs. 2017-04-25 10:19:21 +08:00
post_enqueuer.rb
post_jobs_enqueuer.rb FIX: whisper no longer experimental 2016-12-02 17:03:31 +11:00
post_merger.rb FIX: Revised post not updated correctly when merging posts. 2016-08-11 09:01:54 +08:00
post_revisor.rb FIX: Do not update user stats like counts for private messages. 2017-01-16 11:07:53 +08:00
pretty_text.rb Allow plugins to add vendored files for the text pipeline 2017-04-18 17:59:05 -04:00
primary_group_lookup.rb FEATURE: primary group class on avatars in topic list 2017-02-20 15:55:10 -05:00
promotion.rb
rate_limiter.rb FEATURE: Warn a user when they have few likes remaining 2016-03-18 11:30:29 -04:00
s3_helper.rb REFACTOR: Get bucket name from S3Helper. 2016-08-19 14:08:37 +08:00
score_calculator.rb PERF: make score calculator cheaper when site has long topics 2016-07-22 09:48:44 +10:00
screening_model.rb
search.rb FIX: order:latest not working for search within topic 2017-05-24 11:24:41 -04:00
secure_session.rb SECURITY: prevent reuse of password reset 2016-12-19 18:00:22 +11:00
single_sign_on.rb tiny refactor 2017-03-27 16:21:38 +02:00
site_setting_extension.rb Support Ruby 2.4. 2017-04-15 12:29:00 +08:00
site_setting_validations.rb improve default_categories_watching_first_post copy 2016-11-10 00:15:14 +05:30
slug.rb FIX: prevent creation of tags with invalid characters 2016-10-12 15:44:36 -04:00
socket_server.rb Merge pull request #4870 from tgxworld/fix_cleanup_sockets_servers 2017-05-24 12:33:34 -04:00
source_url.rb FEATURE: upgrade sprockets to latest stable 2017-04-17 10:12:05 -04:00
spam_handler.rb
sql_builder.rb
staff_constraint.rb
stats_socket.rb FEATURE: in vim dev you can focus on spec line in autospec 2017-04-25 09:13:29 -07:00
suggested_topics_builder.rb PERF: cache category topic ids 2016-07-19 12:34:54 +10:00
system_message.rb Convert server side paths to use `/u/` 2017-03-30 10:23:24 -04:00
table_migration_helper.rb fix deploy problems due to renamed table 2017-05-12 15:17:25 -04:00
text_cleaner.rb Test for existing title 2016-03-21 23:06:39 -03:00
text_sentinel.rb FIX: allow long words if they contain periods 2016-09-13 09:15:05 +05:30
timeline_lookup.rb FIX: Deleted posts were showing the wrong dates in the timeline 2016-05-31 10:51:55 -04:00
topic_creator.rb Support Ruby 2.4. 2017-04-15 12:29:00 +08:00
topic_list_responder.rb
topic_query.rb PERF: improve performance of unread queries 2017-05-25 15:07:30 -04:00
topic_query_sql.rb FEATURE: digests choose topics you're watching or tracking first 2016-08-18 17:16:52 -04:00
topic_retriever.rb FIX: More errors with non-ascii URLs 2017-03-07 11:21:41 -05:00
topic_subtype.rb
topic_view.rb FIX: when entering topics "tracking" would not be set 2017-05-16 15:04:21 -04:00
topics_bulk_action.rb FEATURE: bulk relist 2017-05-10 14:53:54 -04:00
trust_level.rb FIX: admin trust level user list URL should be consistent with new naming 2015-11-23 14:28:56 +05:30
twitter_api.rb Add support for oneboxing twitter videos 2017-03-17 20:49:29 +00:00
unread.rb FIX: whisper no longer experimental 2016-12-02 17:03:31 +11:00
upload_creator.rb replace the upload type whitelist with a sanitizer 2017-05-18 12:13:13 +02:00
url_helper.rb FIX: absolute URL for CDN should always be rooted with a protocol 2016-09-05 15:57:46 +10:00
user_name_suggester.rb FIX: suggested username should not be more than setting max_username_length 2017-03-29 18:19:28 +05:30
version.rb Version bump to v1.8.0.beta13 2017-05-22 13:49:35 -04:00
wizard.rb FIX: wizard was not showing up if more than 1 system user 2017-05-23 11:09:50 -04:00