Commit Graph

182 Commits

Author SHA1 Message Date
Guo Xiang Tan 385372e384 Revert "PERF: Reduce number of topics to filter while querying for unread."
This reverts commit c06b782cab.
2017-11-20 11:49:09 +08:00
Guo Xiang Tan c06b782cab PERF: Reduce number of topics to filter while querying for unread. 2017-09-26 12:36:52 +08:00
Guo Xiang Tan 77d4c4d8dc Fix all the errors to get our tests green on Rails 5.1. 2017-09-25 13:48:58 +08:00
Guo Xiang Tan 1a9762a9c0 Fixed private group messages being exposed in suggested topics.
Revert "Revert "PERF: Avoid unnecessary expensive joins if possible.""

This reverts commit d9714c21c8.
2017-09-15 23:40:53 +08:00
Guo Xiang Tan d9714c21c8 Revert "PERF: Avoid unnecessary expensive joins if possible."
This reverts commit f3fadf41b7.

* This ended up exposing group pms to users that are not part
  of a group.
2017-09-15 22:22:07 +08:00
Guo Xiang Tan de46d59a94 Remove comment that is no longer relevant. 2017-09-14 19:57:50 +08:00
Guo Xiang Tan 195982fd41 PERF: Remove N+1 query when generating posters summary. 2017-09-14 12:07:35 +08:00
Guo Xiang Tan f3fadf41b7 PERF: Avoid unnecessary expensive joins if possible.
```
EXPLAIN ANALYZE SELECT  "topics".* FROM "topics"
LEFT JOIN topic_users tu ON topics.id = tu.topic_id AND tu.user_id =
13455
WHERE ("topics"."deleted_at" IS NULL)
AND (topics.archetype = 'private_message')
AND (
  topics.id IN (
    SELECT topic_id
    FROM topic_allowed_groups tg
    JOIN group_users gu ON gu.user_id = 13455 AND gu.group_id =
tg.group_id
    WHERE gu.group_id IN (47)
  )
)
AND (
  topics.id IN (
    SELECT ta.topic_id
    FROM topic_allowed_users ta
    WHERE ta.user_id IN (32852,-10)
  )
  OR
  topics.id IN (
    SELECT tg.topic_id
    FROM topic_allowed_groups tg
    WHERE tg.group_id IN (-10)
  )
)
AND (topics.id NOT IN (69933,69995,69988,69984,69968,69973,69971,69952))
AND "topics"."visible" = 't'
ORDER BY topics.bumped_at DESC
LIMIT 3;
```

Planning time: 1.277 ms
Execution time: 71.577 ms

```
EXPLAIN ANALYZE SELECT  "topics".* FROM "topics"
LEFT JOIN topic_users tu ON topics.id = tu.topic_id AND tu.user_id =
13455
LEFT JOIN (
  SELECT * FROM topic_allowed_groups _tg
  LEFT JOIN group_users gu
  ON gu.user_id = 13455
  AND gu.group_id = _tg.group_id
  AND gu.group_id IN (47)
) tg ON topics.id = tg.topic_id
LEFT JOIN topic_allowed_users ta2 ON topics.id = ta2.topic_id AND
ta2.user_id IN (32852)
WHERE ("topics"."deleted_at" IS NULL)
AND (topics.archetype = 'private_message')
AND (tg.topic_id IS NOT NULL)
AND (ta2.topic_id IS NOT NULL)
AND (topics.id NOT IN (69933,69995,69988,69984,69968,69973,69971,69952))
AND "topics"."visible" = 't'
ORDER BY topics.bumped_at DESC
LIMIT 3;
```

Planning time: 1.191 ms
Execution time: 0.129 ms
2017-09-14 11:18:58 +08:00
Guo Xiang Tan 28148197d6 PERF: Avoid `NOT IN (<subquery>>` which can get really slow.
```
EXPLAIN ANALYZE SELECT  "topics".*
FROM "topics" LEFT JOIN topic_users tu ON topics.id = tu.topic_id AND
tu.user_id = 13455
WHERE ("topics"."deleted_at" IS NULL)
AND (topics.archetype = 'private_message')
AND (
  topics.id IN (
    SELECT topic_id
    FROM topic_allowed_users
    WHERE user_id = 13455
   )
)
AND (
  topics.id IN (
    SELECT ta.topic_id
    FROM topic_allowed_users ta
    WHERE ta.user_id IN (2,1995,8307,17621,22980,-10)
  )
  OR
  topics.id IN (
    SELECT tg.topic_id
    FROM topic_allowed_groups tg
    WHERE tg.group_id IN (-10)
  )
)
AND (topics.id NOT IN (68559,60069,42145))
AND "topics"."visible" = 't'
ORDER BY topics.bumped_at
DESC LIMIT 5;
```

Planning time: 1.196 ms
Execution time: 21.176 ms

```
EXPLAIN ANALYZE SELECT  "topics".*
FROM "topics"
LEFT JOIN topic_users tu ON topics.id = tu.topic_id AND tu.user_id =
13455
LEFT JOIN topic_allowed_users ta ON topics.id = ta.topic_id AND
ta.user_id = 13455
LEFT JOIN topic_allowed_users ta2 ON topics.id = ta2.topic_id AND
ta2.user_id IN (2,1995,8307,17621,22980,-10)
LEFT JOIN topic_allowed_groups tg ON topics.id = tg.topic_id AND
tg.group_id IN (-10)
WHERE ("topics"."deleted_at" IS NULL)
AND (topics.archetype = 'private_message')
AND (ta.topic_id IS NOT NULL)
AND (ta2.topic_id IS NOT NULL OR tg.topic_id IS NOT NULL)
AND (topics.id NOT IN (68559,60069,42145))
AND "topics"."visible" = 't'
ORDER BY topics.bumped_at DESC
LIMIT 5;
```

Planning time: 1.792 ms
Execution time: 2.546 ms
2017-09-13 22:28:33 +08:00
Guo Xiang Tan 5012d46cbd Add rubocop to our build. (#5004) 2017-07-28 10:20:09 +09:00
Bianca Nenciu 06366b5379 latest.json: last topic from first page appears on the second page as well. 2017-07-11 13:59:37 -04:00
Sam b839493fea PERF: Limit new messages in suggested for PMs
unlimited the query will hunt for ancient old PMs which is very
expensive
2017-05-26 11:00:31 -04:00
Sam 0aed2533ac Revert unread optimisation, has too many edge cases 2017-05-26 09:04:13 -04:00
Sam 29fac1ac18 PERF: improve performance of unread queries
Figuring out what unread topics a user has is a very expensive
operation over time.

Users can easily accumulate 10s of thousands of tracking state rows
(1 for every topic they ever visit)

When figuring out what a user has that is unread we need to join
the tracking state records to the topic table. This can very quickly
lead to cases where you need to scan through the entire topic table.

This commit optimises it so we always keep track of the "first" date
a user has unread topics. Then we can easily filter out all earlier
topics from the join.

We use pg functions, instead of nested queries here to assist the
planner.
2017-05-25 15:07:30 -04:00
Guo Xiang Tan 604aaf8686 FIX: N+1 query in suggested topics list. 2017-05-22 18:05:38 +08:00
Vinoth Kannan 1241660c2c FIX: 'read' filter in top menu showing new topics 2017-04-29 14:37:45 +05:30
Sam abc4dff0fe FEATURE: add bumped_before query param for topic list 2017-03-02 15:11:50 -05:00
Sam 4dac4c69a6 FEATURE: add `before` topic list filter 2017-03-02 14:54:33 -05:00
Sam 872d9eae45 FEATURE: add :all filter for messages 2017-03-02 14:54:33 -05:00
Sam Saffron 040e10a627 reduce duplication 2017-02-15 17:27:10 -05:00
Sam 74d4209d24 FEATURE: allow plugins to register custom topic list filters 2017-02-15 15:25:43 -05:00
Sam c04d4171ff FIX: whisper no longer experimental
- Regular users are not notified of whispers
- Regular users no longer have "stuck" topics in unread
- Additional tracking for staff highest post number
- Remove a bunch of unused columns in topics table
2016-12-02 17:03:31 +11:00
Neil Lalonde 9ef1688a76 FEATURE: per-category default topic list sort order 2016-11-01 12:18:41 -04:00
James Kiesel 386b8b8498 Don't join on tags unnecessarily when matching all tags 2016-08-19 10:37:32 -05:00
Régis Hanol 6d1d7b7c8f UX: new /categories layout 2016-08-17 23:23:16 +02:00
James Kiesel 037e9bb7b8 Support any number of tag intersections 2016-08-15 15:30:17 -04:00
James Kiesel e14f3c802b Cleanup 2016-08-12 15:56:56 -04:00
James Kiesel 7e73b933c7 First pass 2016-08-12 15:28:46 -04:00
Neil Lalonde f10c4682cd FIX: muted tags showing in latest topic list 2016-08-04 11:54:48 -04:00
Neil Lalonde 7c092b0fe0 FEATURE: add filter to show topics that have not been tagged 2016-07-20 16:21:51 -04:00
Sam 1c964bf730 PERF: cache category topic ids 2016-07-19 12:34:54 +10:00
Guo Xiang Tan 8fd0414cdf
WIP: Tags which are not allowed in a category showing in drop down. 2016-07-08 10:27:56 +08:00
Guo Xiang Tan e221414935
PERF: Remove N+1 queries on user messages page. 2016-06-29 09:30:54 +08:00
Neil Lalonde 884779b5c1 FIX: N+1 query when tagging enabled and no tags in topic list query. Topic query ignored tags input when tagging is disabled. 2016-05-26 18:03:50 -04:00
Neil Lalonde f13470b96b Use db schema for tags instead of plugin store and custom fields 2016-05-26 14:29:48 -04:00
Robin Ward 1714019cf7
FIX: OFFSET wasn't being applied correctly 2016-05-11 13:39:21 -04:00
Robin Ward 49a6d0b789
FIX: Don't bother with negative offsets 2016-05-09 16:33:55 -04:00
Neil Lalonde e5918c7d00 FEATURE: Merge tagging plugin into core 2016-04-27 11:58:53 -04:00
Joe Buhlig 8d6579c729 Added ability to create topic sorts in plugins with custom_fields 2016-03-04 11:07:39 -06:00
Sam f0e942f647 PERF: move 3 more option columns out of the user table 2016-02-18 16:57:22 +11:00
Sam Saffron bc75010b20 FEATURE: suggested messages for messages
FEATURE: clicking envelope takes you to inbox

Suggested messages works somewhat like suggested topics.

- New show up first (in either group inbox or inbox)
- Then unread (in either group inbox or inbox)
- Finally "related" which are messages with same participants as the current pm.
2016-02-03 18:50:05 +11:00
Régis Hanol e2744fc19f FIX: adding the 'contains_messages' column back 2016-01-12 12:06:51 +01:00
Sam 03ea0bfe22 FEATURE: allow users to archive messages
Messages are now in 3 buckets

- Inbox for all new messages
- Sent for all sent messages
- Archive for all messages you are done with

You can select messages from your Inbox or Sent and move them to your Archive,
you can move messages out of your Archive similarly

Similar concept applied to group messages, except that archiving and unarchiving
will apply to all group members
2015-12-23 11:09:30 +11:00
Neil Lalonde b3198d7a6a callback in TopicQuery for plugins to add result set filters 2015-12-21 11:43:17 -05:00
Sam 8cf4d52cb6 FEATURE: split up group PMS on user page 2015-12-10 11:39:33 +11:00
Régis Hanol a37d575d7d FEATURE: new 'My Groups' messages filter in user page 2015-12-07 18:37:03 +01:00
Sam 48ef609003 correct implementation add tests 2015-11-02 15:05:08 +11:00
Sam 3575012a47 correct implementation 2015-11-02 14:59:10 +11:00
Sam Saffron 606b10445e FEATURE: remove muted topics from suggested and latest 2015-11-02 09:20:22 +11:00
Sam Saffron 8a0a87812b FIX: navigating back to latest was not respecting pinned topics 2015-10-14 13:03:55 +11:00