Commit Graph

371 Commits

Author SHA1 Message Date
Rafael dos Santos Silva 2ed1f874c2
Use correct API signature for instructor embeddings (#62) 2023-05-15 17:18:11 -03:00
Rafael dos Santos Silva 3c9513e754
Refinements to embeddings and tokenizers (#61)
* Refinements to embeddings and tokenizers

* lint

* Truncate with tokenizers for summary

* fix
2023-05-15 15:10:42 -03:00
Sam 93d9d9ea91
FEATURE: hide various api keys in site settings (#59)
Also... ensures that plugin has a link to more info and proper description
2023-05-12 12:54:25 +10:00
Rafael dos Santos Silva 97124b30de
FEATURE: Update summarization token count and add Claude 100k (#58) 2023-05-11 15:35:58 -03:00
Rafael dos Santos Silva 66bf4c74c6
FEATURE: Handle invalid media in NSFW module (#57)
* FEATURE: Handle invalid media in NSFW module

* fix lint
2023-05-11 15:35:39 -03:00
Roman Rizzi 7e3cb0ea16
FEATURE: Multi-model support for the AI Bot module. (#56)
We'll create one bot user for each available model. When listed in the `ai_bot_enabled_chat_bots` setting, they will reply.

This PR lets us use Claude-v1 in stream mode.
2023-05-11 10:03:03 -03:00
Rafael dos Santos Silva e5537d4c77
FEATURE: Allow excluding closed topics from semantic related (#55) 2023-05-09 15:30:50 -03:00
Rafael dos Santos Silva f1133f66a6
Updates to embedding rake tasks (#54)
- Creates embeddings in topic ID order, so it's easier to stop and
restart from where we stopped

- Update index parameters with current best practices
2023-05-09 13:45:16 -03:00
Sam e76fc77189
fixes (#53)
* Minor... use username suggester in case username already exists

* FIX: ensure we truncate long prompts

Previously we

1. Used raw length instead of token counts for counting length
2. We totally dropped a prompt if it was too long

New implementation will truncate "raw" if it gets too long maintaining
meaning.
2023-05-06 07:31:53 -03:00
Roman Rizzi 71b105a1bb
FEATURE: Introduce the ai-bot module (#52)
This module lets you chat with our GPT bot inside a PM. The bot only replies to members of the groups listed on the ai_bot_allowed_groups setting and only if you invite it to participate in the PM.
2023-05-05 15:28:31 -03:00
Rafael dos Santos Silva 29b22243de
FIX: Toxicity automatic flagging should be disabled by default (#51) 2023-05-03 16:44:33 -03:00
Rafael dos Santos Silva c96edc8a72
FIX: Pass correct API Key to summarization service (#50) 2023-05-02 21:41:11 -03:00
Rafael dos Santos Silva 89ac5d720a
FIX: Only send supported image types for classification (#49)
* FIX: Only send supported image types for classification
2023-04-27 17:52:20 -03:00
Sam 2cd60a4b3b
FEATURE: add a table to audit OpenAI usage (#45)
Still need to build a job to purge logs
2023-04-26 11:44:29 +10:00
Jarek Radosz f6c30e8df9
DEV: Fix minor js issues (#48)
1. Add missing `pluginId` to `modifyClass` calls
2. Replace deprecated `site-settings:main` lookups with `service:site-settings`
3. Use "kebab-case" for initializer names
2023-04-25 13:19:57 +02:00
Joffrey JAFFEUX daacd2ffe3
FIX: chat-composer is now using glimmer (#47)
As a result of this change the channel is now accessible through `this.args.channel` and not `this.chatChannel`.

Longer term we want a better solution here, but this should fix the issue for now.
2023-04-25 11:25:07 +02:00
David Taylor a0542d1859
DEV: Resolve add_to_serializer deprecations (#46)
26b7f8a63b
2023-04-24 16:07:17 +01:00
Sam 057fbe1ce6
FEATURE: add internal support for streaming mode (#42)
Also adds some tests around completions and supports additional params
such as top_p, temperature and max_tokens

This also migrates off Faraday to using Net::HTTP directly
2023-04-21 16:54:25 +10:00
Meghna 14b21b4f4d
UX: add a custom sparkles icon for AI action buttons (#44) 2023-04-20 20:41:24 +05:30
Roman Rizzi 38e007a3a5
FEATURE: Topic summarization (#41)
* FEATURE: Topic summarization

Summarize topics using the TopicView's "summary" filter. The UI is similar to what we do for chat, but we don't allow the user to select a timeframe.


Co-authored-by: Rafael dos Santos Silva <xfalcox@gmail.com>
2023-04-19 17:57:31 -03:00
Rafael dos Santos Silva 9783e3b025
FEATURE: Add a basic tokenizer API (#37)
* FEATURE: Add a basic tokenizer API

* Add tests

* lint
2023-04-19 11:55:59 -03:00
Rafael dos Santos Silva 5a78b6a953
DEV: Fix test (#43)
* DEV: Fix test

* lint
2023-04-18 15:43:10 -03:00
Rafael dos Santos Silva 4368ef29d8
FIX: Sometimes Claude sends all titles suggestions in a single ai tag (#40) 2023-04-10 16:02:44 -03:00
Rafael dos Santos Silva bb0b829634
FEATURE: Anthropic Claude for AIHelper and Summarization modules (#39) 2023-04-10 11:04:42 -03:00
Roman Rizzi f2e52f7f24
FIX: Check for chat channel permissions (#38) 2023-04-06 09:07:31 -03:00
Rafael dos Santos Silva 5549e4d5b3
FEATURE: Chat channel summarization. (#32)
* start summary module

* chat channel summarization

* FEATURE: modal for channel summarization

---------

Co-authored-by: Roman Rizzi <rizziromanalejandro@gmail.com>
2023-04-04 11:24:09 -03:00
Roman Rizzi 333cb8f212
FIX: Don't try to use pg headlines for semantic search. (#36)
Depends on discourse/discourse#20939.
2023-04-03 11:48:38 -03:00
Roman Rizzi 7a54455cf6
FIX: Use correct variable and method for embeddings (#35) 2023-03-31 16:15:10 -03:00
Roman Rizzi 4e05763a99
FEATURE: Semantic assymetric full-page search (#34)
Depends on discourse/discourse#20915

Hooks to the full-page-search component using an experimental API and performs an assymetric similarity search using our embeddings database.
2023-03-31 15:29:56 -03:00
Rafael dos Santos Silva 99886fb64d
Update default thresholds for Toxicity (#33) 2023-03-31 13:03:44 -03:00
Sam 6543c50758
FIX: stop returning self as a candidate for related topics (#31) 2023-03-31 11:04:17 +10:00
Sam 0d80d9ec49
FEATURE: allow limiting results in related topics section (#30)
Also:

- Normalizes behavior between logged in and anon,
 we only show related topics in the related topic section

- Renames "suggested" to "related" given this only exists in related section
- Adds a spec section to ensure anon does not regress
- Adds `ai_embeddings_semantic_related_topics` to limit related topics

Renamed settings:

ai_embeddings_semantic_suggested_model -> ai_embeddings_semantic_related_model
ai_embeddings_semantic_suggested_topics_enabled -> ai_embeddings_semantic_related_topics_enabled

Plugins is still in an experimental phase and not much is overidden hence
avoiding adding site setting migrations.


Co-authored-by: Krzysztof Kotlarek <kotlarek.krzysztof@gmail.com>
2023-03-31 11:04:34 +11:00
Sam 1d097b9d82
FEATURE: attempt to include related topics above suggested (#28)
Allows related topics to show up for logged on users

- Introduces a new "Related Topics" block above suggested when related topics exist
- Renames `ai_embeddings_semantic_suggested_topics_anons_enabled` -> `ai_embeddings_semantic_suggested_topics_enabled` (given it is only deployed on 1 site not bothering with a migration)
- Adds an integration test to ensure data arrives correctly on the client
2023-03-31 09:07:22 +11:00
Rafael dos Santos Silva b942a18298
FEATURE: Support for GPT-4 in AI Helper module (#29) 2023-03-28 23:22:34 -03:00
Rafael dos Santos Silva 45950f1bb4
FIX: Only show public visible topics as suggested for anons (#27)
* FIX: Only show public visible topics as suggested for anons

* DEV: Add tests for embeddings

* Update spec/lib/modules/embeddings/semantic_suggested_spec.rb

Co-authored-by: Bianca Nenciu <nbianca@users.noreply.github.com>

* Update spec/lib/modules/embeddings/semantic_suggested_spec.rb

Co-authored-by: Bianca Nenciu <nbianca@users.noreply.github.com>

* move to top

---------

Co-authored-by: Bianca Nenciu <nbianca@users.noreply.github.com>
2023-03-23 17:28:01 -03:00
Roman Rizzi 4c960970fa
DEV: Log information about errors from the completions OpenAI API (#26) 2023-03-22 16:00:28 -03:00
Sam 1d14f7ffaf
FEATURE: Add a markdown table AI helper (#25) 2023-03-22 13:16:29 -03:00
Roman Rizzi 1916bafca0
FIX: Remove `null: false` from dropped column (#24)
* FIX: Remove `null: false` from dropped column

It's causing an issue when attempting to seed data in fresh installs.
The column was dropped in a post-migration.

* Update db/migrate/20230322142028_make_dropped_value_column_nullable.rb

Co-authored-by: David Taylor <david@taylorhq.com>

---------

Co-authored-by: David Taylor <david@taylorhq.com>
2023-03-22 11:40:20 -03:00
Rafael dos Santos Silva bd342f538d
FEATURE: Try to generate embeddings for a topic when those aren't found (#23) 2023-03-21 18:20:46 -03:00
Roman Rizzi 55182ca0f5
FIX: Allow null messages to migrate existing rows (#22) 2023-03-21 12:33:30 -03:00
Roman Rizzi 39f7f1f29e
FEATURE: Prompts can consist of multiple messages. (#21)
A prompt with multiple messages leads to better results, as the AI can learn for given examples. Alongside this change, we provide a better default proofreading prompt.
2023-03-21 12:04:59 -03:00
Rafael dos Santos Silva 6bdbc0e32d
FIX: Proper flow when a topic doesn't have embeddings (#20) 2023-03-20 16:44:55 -03:00
Roman Rizzi fea9041ee1
DEV: Use 10s timeout when using the completions API (#19) 2023-03-20 16:43:51 -03:00
Roman Rizzi 54a1694637
FIX: Remove old seeded prompts. (#17)
We switched to negative integers for seeded prompts, but we also
need to delete the old ones using IDs 1/2/3 if they exist,
 or deploys will fail due to the unique index on the name column.
2023-03-20 12:55:38 -03:00
Roman Rizzi 95d7692ae8
FIX: Fix reviewable title translations (#18) 2023-03-20 10:09:20 -03:00
Roman Rizzi 258eb7e78f
DEV: Use negative ids for seeded prompts. (#16)
This makes easier to add new defaults in the feature without clashing with user created ones.
2023-03-17 15:44:45 -03:00
Roman Rizzi 01a08f41cf
FIX: Change gen titles prompt to only return a list (#15) 2023-03-17 15:36:38 -03:00
Roman Rizzi 320ac6e84b
REFACTOR: Store prompts in a dedicated table. (#14)
This change makes it easier to add new prompts to our AI helper. We don't have a UI for it yet. You'll have to do it through a console.
2023-03-17 15:14:19 -03:00
Joffrey JAFFEUX edfdc6dfae
DEV: applies chat namespacing (#12) 2023-03-17 15:15:38 +01:00
Roman Rizzi 75aa595105
FIX: Use cooked suggestion when generating a diff (#13) 2023-03-16 11:09:28 -03:00