Commit Graph

52 Commits

Author SHA1 Message Date
Rafael dos Santos Silva 3c9513e754
Refinements to embeddings and tokenizers (#61)
* Refinements to embeddings and tokenizers

* lint

* Truncate with tokenizers for summary

* fix
2023-05-15 15:10:42 -03:00
Rafael dos Santos Silva 97124b30de
FEATURE: Update summarization token count and add Claude 100k (#58) 2023-05-11 15:35:58 -03:00
Rafael dos Santos Silva 66bf4c74c6
FEATURE: Handle invalid media in NSFW module (#57)
* FEATURE: Handle invalid media in NSFW module

* fix lint
2023-05-11 15:35:39 -03:00
Roman Rizzi 7e3cb0ea16
FEATURE: Multi-model support for the AI Bot module. (#56)
We'll create one bot user for each available model. When listed in the `ai_bot_enabled_chat_bots` setting, they will reply.

This PR lets us use Claude-v1 in stream mode.
2023-05-11 10:03:03 -03:00
Rafael dos Santos Silva e5537d4c77
FEATURE: Allow excluding closed topics from semantic related (#55) 2023-05-09 15:30:50 -03:00
Rafael dos Santos Silva f1133f66a6
Updates to embedding rake tasks (#54)
- Creates embeddings in topic ID order, so it's easier to stop and
restart from where we stopped

- Update index parameters with current best practices
2023-05-09 13:45:16 -03:00
Sam e76fc77189
fixes (#53)
* Minor... use username suggester in case username already exists

* FIX: ensure we truncate long prompts

Previously we

1. Used raw length instead of token counts for counting length
2. We totally dropped a prompt if it was too long

New implementation will truncate "raw" if it gets too long maintaining
meaning.
2023-05-06 07:31:53 -03:00
Roman Rizzi 71b105a1bb
FEATURE: Introduce the ai-bot module (#52)
This module lets you chat with our GPT bot inside a PM. The bot only replies to members of the groups listed on the ai_bot_allowed_groups setting and only if you invite it to participate in the PM.
2023-05-05 15:28:31 -03:00
Rafael dos Santos Silva c96edc8a72
FIX: Pass correct API Key to summarization service (#50) 2023-05-02 21:41:11 -03:00
Rafael dos Santos Silva 89ac5d720a
FIX: Only send supported image types for classification (#49)
* FIX: Only send supported image types for classification
2023-04-27 17:52:20 -03:00
Sam 2cd60a4b3b
FEATURE: add a table to audit OpenAI usage (#45)
Still need to build a job to purge logs
2023-04-26 11:44:29 +10:00
David Taylor a0542d1859
DEV: Resolve add_to_serializer deprecations (#46)
26b7f8a63b
2023-04-24 16:07:17 +01:00
Sam 057fbe1ce6
FEATURE: add internal support for streaming mode (#42)
Also adds some tests around completions and supports additional params
such as top_p, temperature and max_tokens

This also migrates off Faraday to using Net::HTTP directly
2023-04-21 16:54:25 +10:00
Meghna 14b21b4f4d
UX: add a custom sparkles icon for AI action buttons (#44) 2023-04-20 20:41:24 +05:30
Roman Rizzi 38e007a3a5
FEATURE: Topic summarization (#41)
* FEATURE: Topic summarization

Summarize topics using the TopicView's "summary" filter. The UI is similar to what we do for chat, but we don't allow the user to select a timeframe.


Co-authored-by: Rafael dos Santos Silva <xfalcox@gmail.com>
2023-04-19 17:57:31 -03:00
Rafael dos Santos Silva 9783e3b025
FEATURE: Add a basic tokenizer API (#37)
* FEATURE: Add a basic tokenizer API

* Add tests

* lint
2023-04-19 11:55:59 -03:00
Rafael dos Santos Silva 4368ef29d8
FIX: Sometimes Claude sends all titles suggestions in a single ai tag (#40) 2023-04-10 16:02:44 -03:00
Rafael dos Santos Silva bb0b829634
FEATURE: Anthropic Claude for AIHelper and Summarization modules (#39) 2023-04-10 11:04:42 -03:00
Rafael dos Santos Silva 5549e4d5b3
FEATURE: Chat channel summarization. (#32)
* start summary module

* chat channel summarization

* FEATURE: modal for channel summarization

---------

Co-authored-by: Roman Rizzi <rizziromanalejandro@gmail.com>
2023-04-04 11:24:09 -03:00
Roman Rizzi 7a54455cf6
FIX: Use correct variable and method for embeddings (#35) 2023-03-31 16:15:10 -03:00
Roman Rizzi 4e05763a99
FEATURE: Semantic assymetric full-page search (#34)
Depends on discourse/discourse#20915

Hooks to the full-page-search component using an experimental API and performs an assymetric similarity search using our embeddings database.
2023-03-31 15:29:56 -03:00
Sam 6543c50758
FIX: stop returning self as a candidate for related topics (#31) 2023-03-31 11:04:17 +10:00
Sam 0d80d9ec49
FEATURE: allow limiting results in related topics section (#30)
Also:

- Normalizes behavior between logged in and anon,
 we only show related topics in the related topic section

- Renames "suggested" to "related" given this only exists in related section
- Adds a spec section to ensure anon does not regress
- Adds `ai_embeddings_semantic_related_topics` to limit related topics

Renamed settings:

ai_embeddings_semantic_suggested_model -> ai_embeddings_semantic_related_model
ai_embeddings_semantic_suggested_topics_enabled -> ai_embeddings_semantic_related_topics_enabled

Plugins is still in an experimental phase and not much is overidden hence
avoiding adding site setting migrations.


Co-authored-by: Krzysztof Kotlarek <kotlarek.krzysztof@gmail.com>
2023-03-31 11:04:34 +11:00
Sam 1d097b9d82
FEATURE: attempt to include related topics above suggested (#28)
Allows related topics to show up for logged on users

- Introduces a new "Related Topics" block above suggested when related topics exist
- Renames `ai_embeddings_semantic_suggested_topics_anons_enabled` -> `ai_embeddings_semantic_suggested_topics_enabled` (given it is only deployed on 1 site not bothering with a migration)
- Adds an integration test to ensure data arrives correctly on the client
2023-03-31 09:07:22 +11:00
Rafael dos Santos Silva b942a18298
FEATURE: Support for GPT-4 in AI Helper module (#29) 2023-03-28 23:22:34 -03:00
Rafael dos Santos Silva 45950f1bb4
FIX: Only show public visible topics as suggested for anons (#27)
* FIX: Only show public visible topics as suggested for anons

* DEV: Add tests for embeddings

* Update spec/lib/modules/embeddings/semantic_suggested_spec.rb

Co-authored-by: Bianca Nenciu <nbianca@users.noreply.github.com>

* Update spec/lib/modules/embeddings/semantic_suggested_spec.rb

Co-authored-by: Bianca Nenciu <nbianca@users.noreply.github.com>

* move to top

---------

Co-authored-by: Bianca Nenciu <nbianca@users.noreply.github.com>
2023-03-23 17:28:01 -03:00
Roman Rizzi 4c960970fa
DEV: Log information about errors from the completions OpenAI API (#26) 2023-03-22 16:00:28 -03:00
Sam 1d14f7ffaf
FEATURE: Add a markdown table AI helper (#25) 2023-03-22 13:16:29 -03:00
Rafael dos Santos Silva bd342f538d
FEATURE: Try to generate embeddings for a topic when those aren't found (#23) 2023-03-21 18:20:46 -03:00
Roman Rizzi 39f7f1f29e
FEATURE: Prompts can consist of multiple messages. (#21)
A prompt with multiple messages leads to better results, as the AI can learn for given examples. Alongside this change, we provide a better default proofreading prompt.
2023-03-21 12:04:59 -03:00
Rafael dos Santos Silva 6bdbc0e32d
FIX: Proper flow when a topic doesn't have embeddings (#20) 2023-03-20 16:44:55 -03:00
Roman Rizzi fea9041ee1
DEV: Use 10s timeout when using the completions API (#19) 2023-03-20 16:43:51 -03:00
Roman Rizzi 320ac6e84b
REFACTOR: Store prompts in a dedicated table. (#14)
This change makes it easier to add new prompts to our AI helper. We don't have a UI for it yet. You'll have to do it through a console.
2023-03-17 15:14:19 -03:00
Joffrey JAFFEUX edfdc6dfae
DEV: applies chat namespacing (#12) 2023-03-17 15:15:38 +01:00
Roman Rizzi 75aa595105
FIX: Use cooked suggestion when generating a diff (#13) 2023-03-16 11:09:28 -03:00
Rafael dos Santos Silva 80d662e9e8
FEATURE: Semantic Suggested Topics (#10) 2023-03-15 17:21:45 -03:00
Roman Rizzi f99fe7e1ed
FEATURE: Composer AI helper (#8)
* FEATURE: Composer AI helper

This change introduces a new composer button for the group members listed in the `ai_helper_allowed_groups` site setting.

Users can use chatGPT to review, improve, or translate their posts to English.

* Add a safeguard for PMs and don't rely on parentView
2023-03-15 17:02:20 -03:00
Roman Rizzi aa2fca6086
DEV: DiscourseAI -> DiscourseAi rename to have consistent folders and files (#9) 2023-03-14 16:03:50 -03:00
Rafael dos Santos Silva 510c6487e3
DEV: Preparation work for multiple inference providers (#5) 2023-03-07 16:14:39 -03:00
Roman Rizzi a838116cd5
FEATURE: Use dedicated reviewables for AI flags. (#4)
This change adds two new reviewable types: ReviewableAIPost and ReviewableAIChatMessage. They have the same actions as their existing counterparts: ReviewableFlaggedPost and ReviewableChatMessage.

We'll display the model used and their accuracy when showing these flags in the review queue and adjust the latter after staff performs an action, tracking a global accuracy per existing model in a separate table.


* FEATURE: Dedicated reviewables for AI flags

* Store and adjust model accuracy

* Display accuracy in reviewable templates
2023-03-07 15:39:28 -03:00
Roman Rizzi 676d3ce6b2
DEV: Rename XClassification --> XClassificator to make it more obvious (#3) 2023-02-28 11:17:03 -03:00
Roman Rizzi b9a650fde4
DEV: Dedicated table for saving classification results (#1) 2023-02-27 16:21:40 -03:00
Roman Rizzi 5f9597474c
REFACTOR: Streamline flag and classification process 2023-02-24 13:25:02 -03:00
Roman Rizzi 85768cfb1c
FEATURE: Classify posts looking for NSFW images 2023-02-24 09:11:58 -03:00
Roman Rizzi 94933f3c58
DEV: Add missing specs for the toxicity module 2023-02-24 07:53:43 -03:00
Roman Rizzi e8bffcdd64
DEV: Add tests for the sentiment module 2023-02-23 15:50:10 -03:00
Roman Rizzi ef6c785aca
DEV: Move jobs undear each module lib directory 2023-02-23 14:09:52 -03:00
Roman Rizzi 1afa274b99
DEV: Reorganize files and add an entry point for each module 2023-02-23 12:25:00 -03:00
Rafael dos Santos Silva a73931c151
Refactoring of nsfw and flagger 2023-02-23 12:13:26 -03:00
Roman Rizzi 6f0c141062
FEATURE: Introduce NSFW content detection basic flow. 2023-02-23 11:08:34 -03:00