discourse-ai/db/post_migrate
Roman Rizzi 46fcdb6ba5
FIX: Make summaries backfill job more resilient. (#1071)
To quickly select backfill candidates without comparing SHAs, we compare the last summarized post to the topic's highest_post_number. However, hiding or deleting a post and adding a small action will update this column, causing the job to stall and re-generate the same summary repeatedly until someone posts a regular reply. On top of this, this is not always true for topics with `best_replies`, as this last reply isn't necessarily included.

Since this is not evident at first glance and each summarization strategy picks its targets differently, I'm opting to simplify the backfill logic and how we track potential candidates.

The first step is dropping `content_range`, which serves no purpose and it's there because summary caching was supposed to work differently at the beginning. So instead, I'm replacing it with a column called `highest_target_number`, which tracks `highest_post_number` for topics and could track other things like channel's `message_count` in the future.

Now that we have this column when selecting every potential backfill candidate, we'll check if the summary is truly outdated by comparing the SHAs, and if it's not, we just update the column and move on
2025-01-16 09:42:53 -03:00
..
20240119152348_explicit_provider_backwards_compat.rb FEATURE: GPT4o support and better auditing (#618) 2024-05-14 13:28:46 +10:00
20240207144910_fix_llm_backed_setting_defaults.rb FIX: Explicit check for empty string in compat migration (#463) 2024-02-07 14:51:51 -03:00
20240528144216_seed_open_ai_models.rb DEV: Rewire AI bot internals to use LlmModel (#638) 2024-06-18 14:32:14 -03:00
20240531205234_seed_claude_models.rb DEV: Rewire AI bot internals to use LlmModel (#638) 2024-06-18 14:32:14 -03:00
20240603133432_seed_other_propietary_models.rb DEV: Rewire AI bot internals to use LlmModel (#638) 2024-06-18 14:32:14 -03:00
20240603143158_seed_oss_models.rb FEATURE: allow access to private topics if tool permits (#673) 2024-06-19 15:49:36 +10:00
20240606152117_copy_summary_sections_to_ai_summaries.rb DEV: Add summarization logic from core (#658) 2024-07-02 08:51:59 -07:00
20240609232736_drop_commands_from_ai_personas.rb FEATURE: optional tool detail blocks (#662) 2024-06-11 18:14:14 +10:00
20240610232040_copy_summarization_strategy_to_ai_summarization_strategy.rb DEV: Add summarization logic from core (#658) 2024-07-02 08:51:59 -07:00
20240610232546_copy_custom_summarization_allowed_groups_to_ai_custom_summarization_allowed_groups.rb DEV: Add summarization logic from core (#658) 2024-07-02 08:51:59 -07:00
20240611170906_drop_old_embeddings_tables.rb DEV: Move to single table per embeddings type (#561) 2024-08-08 11:55:20 -03:00
20240619193057_choose_llm_model_setting_migration.rb DEV: Transition "Select model" settings to only use LlmModels (#675) 2024-06-19 18:01:35 -03:00
20240619211337_update_automation_script_models.rb DEV: Use LlmModels as options in automation rules (#676) 2024-06-21 08:07:17 +10:00
20240624202602_add_provider_specific_params_to_llm_models.rb DEV: Use Rails 7.0 instead of 7.1 in post-migrations 2024-06-26 18:41:38 +02:00
20240703135444_llm_models_for_summarization.rb FEATURE: move summary to use llm_model (#699) 2024-07-04 10:48:18 +10:00
20240708193243_fix_vllm_model_name.rb FIX: Flaky SRV-backed model seeding. (#708) 2024-07-08 18:47:10 -03:00
20240724174343_migrate_vision_llms.rb DEV: Remove old code now that features rely on LlmModels. (#729) 2024-07-30 13:44:57 -03:00
20240729202857_migrate_persona_llm_override.rb DEV: Remove old code now that features rely on LlmModels. (#729) 2024-07-30 13:44:57 -03:00
20240809162837_rename_ai_helper_enabled_setting.rb DEV: Clearly separate post/composer helper settings (#747) 2024-08-12 15:40:23 -07:00
20240809163303_rename_ai_helper_allowed_groups_setting.rb DEV: Clearly separate post/composer helper settings (#747) 2024-08-12 15:40:23 -07:00
20240912055831_drop_persona_id_from_rag_document_fragments.rb FEATURE: Make tool support polymorphic (#798) 2024-09-16 08:17:17 +10:00
20241008055831_drop_old_embeddings_indexes.rb FEATURE: Index embeddings using bit vectors (#824) 2024-10-14 13:26:03 -03:00
20241014041242_ai_persona_post_migrate_drop_cols.rb FEATURE: smarter persona tethering (#832) 2024-10-16 07:20:31 +11:00
20241031041242_migrate_sentiment_classification_result_format.rb Migrate sentiment to a TEI backend (#886) 2024-11-04 09:14:34 -03:00
20241206115958_rebake_shared_ai_conversation_oneboxes.rb SECURITY: Fix XSS on Shared AI Conversations local Onebox (#1069) 2025-01-14 18:05:37 -03:00
20250113171444_drop_old_embedding_tables.rb FIX: Split backfill into separate migrations to use independent transactions (#1063) 2025-01-14 13:30:52 -03:00
20250114184356_drop_old_embedding_tables2.rb FIX: Missing table check in post_migration (#1068) 2025-01-14 17:33:01 -03:00
20250115181147_drop_ai_summaries_content_range.rb FIX: Make summaries backfill job more resilient. (#1071) 2025-01-16 09:42:53 -03:00