Commit Graph

579 Commits

Author SHA1 Message Date
Sam Saffron 6351c401f8
work in progress adding roles to personas
# Conflicts:
#	app/models/ai_persona.rb
2024-05-01 15:39:13 +10:00
Sam Saffron acd3ac2371
minor fix to read ... complete is just confusing 2024-04-30 17:08:23 +10:00
Sam Saffron c61af76d86
Responder for categories 2024-04-30 17:08:23 +10:00
Sam Saffron cd7790f39a
message responder backend now works 2024-04-30 17:08:21 +10:00
Sam Saffron 4fcf12287f
work in progress adding roles to personas 2024-04-30 17:07:22 +10:00
Sam 32b3004ce9
FEATURE: Add Question Consolidator for robust Upload support in Personas (#596)
This commit introduces a new feature for AI Personas called the "Question Consolidator LLM". The purpose of the Question Consolidator is to consolidate a user's latest question into a self-contained, context-rich question before querying the vector database for relevant fragments. This helps improve the quality and relevance of the retrieved fragments.

Previous to this change we used the last 10 interactions, this is not ideal cause the RAG would "lock on" to an answer. 

EG:

- User: how many cars are there in europe
- Model: detailed answer about cars in europe including the term car and vehicle many times
- User: Nice, what about trains are there in the US

In the above example "trains" and "US" becomes very low signal given there are pages and pages talking about cars and europe. This mean retrieval is sub optimal. 

Instead, we pass the history to the "question consolidator", it would simply consolidate the question to "How many trains are there in the United States", which would make it fare easier for the vector db to find relevant content. 

The llm used for question consolidator can often be less powerful than the model you are talking to, we recommend using lighter weight and fast models cause the task is very simple. This is configurable from the persona ui.

This PR also removes support for {uploads} placeholder, this is too complicated to get right and we want freedom to shift RAG implementation. 

Key changes:

1. Added a new `question_consolidator_llm` column to the `ai_personas` table to store the LLM model used for question consolidation.

2. Implemented the `QuestionConsolidator` module which handles the logic for consolidating the user's latest question. It extracts the relevant user and model messages from the conversation history, truncates them if needed to fit within the token limit, and generates a consolidated question prompt.

3. Updated the `Persona` class to use the Question Consolidator LLM (if configured) when crafting the RAG fragments prompt. It passes the conversation context to the consolidator to generate a self-contained question.

4. Added UI elements in the AI Persona editor to allow selecting the Question Consolidator LLM. Also made some UI tweaks to conditionally show/hide certain options based on persona configuration.

5. Wrote unit tests for the QuestionConsolidator module and updated existing persona tests to cover the new functionality.

This feature enables AI Personas to better understand the context and intent behind a user's question by consolidating the conversation history into a single, focused question. This can lead to more relevant and accurate responses from the AI assistant.
2024-04-30 13:49:21 +10:00
Sam 85734fef52
FIX: properly cache user locale (#593)
This blob is localized according to user locale, so we can end up
bleeding incorrect data in the cache
2024-04-26 09:28:35 -03:00
Roman Rizzi 283445cf81
FIX: RAG uploader must support multi-file indexing. (#592)
Updating the editing model's rag_uploads in the editor component broke multi-file uploading. Instead, we'll keep the uploads in the uploader and update the model when we finish.

This PR also fast-tracks the initial update so we can show feedback to the user quickly, and allows uploading MD files.

Bug reported on https://meta.discourse.org/t/discourse-ai-persona-upload-support/304049/11
2024-04-25 10:48:55 -03:00
Roman Rizzi 0c4069ab3f
DEV: Remove non-LLM-based summarization strategies. (#589)
We removed these services from our hosting two weeks ago. It's safe to assume everyone has moved to other LLM-based options.
2024-04-23 12:11:04 -03:00
Discourse Translator Bot 66804bc13c
Update translations (#587) 2024-04-23 16:22:37 +02:00
Sam 4d8b7742da
FIX: many missing topics when categories excluded (#585)
We were forgetting about the NULL parent_category_id handling in
our check for sub categories
2024-04-23 08:53:51 +10:00
Rafael dos Santos Silva 595cde0fd6
FIX: Users with empty locales would error out during prompt localization (#584) 2024-04-22 13:55:10 -03:00
Sam 5ab86923ff
FIX: when excluding categories also exclude children (#583)
This allows you to exclude trees of categories in a simple way

It also means you can no longer exclude "just the parent" but
this is a fair compromise.
2024-04-22 16:05:24 +10:00
Sam bd6f5caeac
FEATURE: Stable diffusion 3 support (#582)
- Adds support for sd3 and sd3 turbo models - this requires new endpoints
- Adds a hack to normalize arrays in the tool calls
- Removes some leftover code
- Adds support for aspect ratio as well so you can generate wide or tall images
2024-04-19 18:08:16 +10:00
Sam a223d18f1a
FIX: more robust function call support (#581)
For quite a few weeks now, some times, when running function calls
on Anthropic we would get a "stray" - "calls" line.

This has been enormously frustrating!

I have been unable to find the source of the bug so instead decoupled
the implementation and create a very clear "function call normalizer"

This new class is extensively tested and guards against the type of
edge cases we saw pre-normalizer.

This also simplifies the implementation of "endpoint" which no longer
needs to handle all this complex logic.
2024-04-19 06:54:54 +10:00
Sam 50be66ee63
FEATURE: Gemini 1.5 pro support and Claude Opus bedrock support (#580)
- Updated AI Bot to only support Gemini 1.5 (used to support 1.0) - 1.0 was removed cause it is not appropriate for Bot usage
- Summaries and automation can now lean on Gemini 1.5 pro
- Amazon added support for Claude 3 Opus, added internal support for it on bedrock
2024-04-17 15:37:19 +10:00
Sam a5e4ab2825
FIX: blank metadata leading to errors (#578)
blank metadata block in RAG was leading to an error, this handles the edge case
2024-04-17 13:46:40 +10:00
Discourse Translator Bot c2b2741f3d
Update translations (#579) 2024-04-16 17:38:00 +02:00
Sam 4a29f8ed1c
FEATURE: Enhance AI debugging capabilities and improve interface adjustments (#577)
* FIX: various RAG edge cases

- Nicer text to describe RAG, avoids the word RAG
- Do not attempt to save persona when removing uploads and it is not created
- Remove old code that avoided touching rag params on create

* FIX: Missing pause button for persona users

* Feature: allow specific users to debug ai request / response chains

This can help users easily tune RAG and figure out what is going
on with requests.

* discourse helper so it does not explode

* fix test

* simplify implementation
2024-04-15 23:22:06 +10:00
Bianca Nenciu 3e54697c5a
FIX: Load categories for related topics (#570)
This is necessary when "lazy load categories" feature is enabled to
make sure the categories are rendered for all related topics.
2024-04-15 09:31:07 +10:00
Rafael dos Santos Silva 6090580e36
FEATURE: Add basic connection check to DNS SRV resources (#563) 2024-04-12 10:39:19 -03:00
Sam f6ac5cd0a8
FEATURE: allow tuning of RAG generation (#565)
* FEATURE: allow tuning of RAG generation

- change chunking to be token based vs char based (which is more accurate)
- allow control over overlap / tokens per chunk and conversation snippets inserted
- UI to control new settings

* improve ui a bit

* fix various reindex issues

* reduce concurrency

* try ultra low queue ... concurrency 1 is too slow.
2024-04-12 10:32:46 -03:00
Sam b906046aad
FEATURE: Add Cohere models to AI helper and automation (#576) 2024-04-12 14:46:58 +10:00
Rafael dos Santos Silva 253e0b7b39
FEATURE: Mixtral/Mistral/Haiku Automation Support (#571)
Adds new models to automation, and makes LLM output parsing more robust.
2024-04-11 09:50:46 -03:00
Sam 23d12c8927
FEATURE: GPT-4 turbo vision support (#575)
Recent release of GPT-4 turbo adds vision support, this adds
the pipeline for sending images to Open AI.
2024-04-11 16:22:59 +10:00
Sam a77658e2b1
FIX: tools broke on Claude with no params (#574)
Some tools may have no params, allow that
2024-04-11 15:17:56 +10:00
Sam 3b0cfdbe5c
FIX: throwing away first file in diff (#573)
We were chucking out the first file in a PR diff due to a
logic bug
2024-04-11 13:26:58 +10:00
Sam 0cbbf130b9
FIX: never mention the word JSON in tool preamble (#572)
Just having the word JSON can confuse models when we expect them
to deal solely in XML

Instead provide an example of how string arrays should be returned

Technically the tool framework supports int arrays and more, but
our current implementation only does string arrays.

Also tune the prompt construction not to give any tips about arrays
if none exist
2024-04-11 11:24:22 +10:00
Martin Brennan bab5e52e38
FIX: Secure/unsecure uploads when sharing AI conversations (#554)
This commit uses a new plugin modifier introduced in https://github.com/discourse/discourse/pull/26508
to mark all uploads as _not_ secure in shared PM AI conversations.
This is so images created by the AI bot (or uploaded by the user)
do not end up as broken URLs because of the security requirements
around them.

This relies on the UpdateTopicUploadSecurity job in core as well,
which is fired when an AI conversation is shared or deleted.
2024-04-11 10:00:41 +10:00
Sam 7f16d3ad43
FEATURE: Cohere Command R support (#558)
- Added Cohere Command models (Command, Command Light, Command R, Command R Plus) to the available model list
- Added a new site setting `ai_cohere_api_key` for configuring the Cohere API key
- Implemented a new `DiscourseAi::Completions::Endpoints::Cohere` class to handle interactions with the Cohere API, including:
   - Translating request parameters to the Cohere API format
   - Parsing Cohere API responses 
   - Supporting streaming and non-streaming completions
   - Supporting "tools" which allow the model to call back to discourse to lookup additional information
- Implemented a new `DiscourseAi::Completions::Dialects::Command` class to translate between the generic Discourse AI prompt format and the Cohere Command format
- Added specs covering the new Cohere endpoint and dialect classes
- Updated `DiscourseAi::AiBot::Bot.guess_model` to map the new Cohere model to the appropriate bot user

In summary, this PR adds support for using the Cohere Command family of models with the Discourse AI plugin. It handles configuring API keys, making requests to the Cohere API, and translating between Discourse's generic prompt format and Cohere's specific format. Thorough test coverage was added for the new functionality.
2024-04-11 07:24:17 +10:00
Rafael dos Santos Silva eb93b21769
FEATURE: Add BGE-M3 embeddings support (#569)
BAAI/bge-m3 is an interesting model, that is multilingual and with a
context size of 8192. Even with a 16x larger context, it's only 4x slower
to compute it's embeddings on the worst case scenario.

Also includes a minor refactor of the rake task, including setting model
and concurrency levels when running the backfill task.
2024-04-10 17:24:01 -03:00
Sam 6de9c53a71
FEATURE: remove gpt-4-turbo-0125 preview swap with gpt-4-turbo (#568)
Open AI just released gpt-4-turbo (with vision)

This change stops using the old preview model and swaps with the
officially released gpt-4-turbo

To come is an implementation of vision.
2024-04-10 09:53:20 -03:00
Kris f20ada00db
UX: adjust AI share styles for images and quotes (#567) 2024-04-09 15:36:45 -04:00
Keegan George 8444789974
UX: Update suggestion radii to accommodate for new default (#564) 2024-04-09 10:48:18 -07:00
Discourse Translator Bot 310238d38a
Update translations (#566) 2024-04-09 18:48:54 +02:00
Roman Rizzi aa8918911d
UX: Display the indexing progress for RAG uploads (#557) 2024-04-09 11:03:07 -03:00
Keegan George 35fbf5c836
FIX: Ask AI highlight fixes (#562) 2024-04-08 11:00:03 -07:00
Bianca Nenciu 505650205d
FIX: Fetch categories data using specific endpoint (#543)
It used to fetch it from /site.json, but /categories.json is the more
appropriate one. This one also implements pagination, so we have to do
one request per page.
2024-04-08 11:33:20 +03:00
Keegan George 55d6e1cdf8
UX: Update Ask AI highlight styling (#560) 2024-04-07 10:30:59 -07:00
Rafael dos Santos Silva 969fbae21e
DEV: Hide quick search setting since it's experimental (#559) 2024-04-05 12:12:37 -03:00
Keegan George cb4d438506
UX: Highlight AI post helper selection (#520) 2024-04-04 11:35:01 -07:00
Keegan George fc6b937df7
FIX: Revert length issue (#556) 2024-04-04 10:28:17 -07:00
Sam 830cc26075
FEATURE: Add metadata support for RAG (#553)
* FEATURE: Add metadata support for RAG

You may include non indexed metadata in the RAG document by using

[[metadata ....]]

This information is attached to all the text below and provided to
the retriever.

This allows for RAG to operate within a rich amount of contexts
without getting lost

Also:

- re-implemented chunking algorithm so it streams
- moved indexing to background low priority queue

* Baran gem no longer required.

* tokenizers is on 4.4 ... upgrade it ...
2024-04-04 11:02:16 -03:00
Régis Hanol bc561eb332
DEV: replace diffhtml with morphlex (#555)
morphlex is a lighter and faster morphing library 🚀
2024-04-04 16:00:16 +02:00
Sam 6f5f34184b
FEATURE: add Claude 3 Haiku bot support (#552)
it is close in performance to GPT 4 at a fraction of the cost,
nice to add it to the mix.

Also improves a test case to simulate streaming, I am hunting for
the "calls" word that is jumping into function calls and can't quite
find it.
2024-04-03 16:06:27 +11:00
Discourse Translator Bot 08b53e851c
Update translations (#551) 2024-04-02 18:19:55 +02:00
Roman Rizzi 1f1c94e5c6
FEATURE: AI Bot RAG support. (#537)
This PR lets you associate uploads to an AI persona, which we'll split and generate embeddings from. When building the system prompt to get a bot reply, we'll do a similarity search followed by a re-ranking (if available). This will let us find the most relevant fragments from the body of knowledge you associated with the persona, resulting in better, more informed responses.

For now, we'll only allow plain-text files, but this will change in the future.

Commits:

* FEATURE: RAG embeddings for the AI Bot

This first commit introduces a UI where admins can upload text files, which we'll store, split into fragments,
and generate embeddings of. In a next commit, we'll use those to give the bot additional information during
conversations.

* Basic asymmetric similarity search to provide guidance in system prompt

* Fix tests and lint

* Apply reranker to fragments

* Uploads filter, css adjustments and file validations

* Add placeholder for rag fragments

* Update annotations
2024-04-01 13:43:34 -03:00
Kris a2018d4a04
UX: add general onebox styles (#550) 2024-03-29 09:50:28 -04:00
Sam fb81307c59
FEATURE: web browsing tool (#548)
This pull request makes several improvements and additions to the GitHub-related tools and personas in the `discourse-ai` repository:

1. It adds the `WebBrowser` tool to the  `Researcher` persona, allowing the AI to visit web pages, retrieve HTML content, extract the main content, and convert it to plain text.

2. It updates the `GithubFileContent`, `GithubPullRequestDiff`, and `GithubSearchCode` tools to handle HTTP responses more robustly (introducing size limits). 

3. It refactors the `send_http_request` method in the `Tool` class to follow redirects when specified, and to read the response body in chunks to avoid memory issues with large responses. (only for WebBrowser)

4. It updates the system prompt for the `Researcher` persona to provide more detailed guidance on when to use Google search vs web browsing, and how to optimize tool usage and reduce redundant requests.

5. It adds a new `web_browser_spec.rb` file with tests for the `WebBrowser` tool, covering various scenarios like handling different HTML structures and following redirects.
2024-03-28 16:01:58 +11:00
Sam 74c6725c75
FIX: avoid error when attempting to show ai helper to anon (#549)
Also adds an extra guard around ai helper context positioning.
2024-03-28 12:24:00 +11:00