Commit Graph

145 Commits

Author SHA1 Message Date
Sam 98c89953d3
FEATURE: remember previously selected persona (#299)
People tend to keep to 1 persona when working with the bot,
this adds local browser memory for the last persona you interacted
with so you do not need to select it over and over again.

This is per browser, not per user memory.

Also... clean up tests so they do not need to require stubs which
were breaking the build

---------

Co-authored-by: Martin Brennan <martin@discourse.org>
2023-11-21 17:02:27 +10:00
Sam 5b5edb22c6
FEATURE: UI to update ai personas on admin page (#290)
Introduces a UI to manage customizable personas (admin only feature)

Part of the change was some extensive internal refactoring:

- AIBot now has a persona set in the constructor, once set it never changes
- Command now takes in bot as a constructor param, so it has the correct persona and is not generating AIBot objects on the fly
- Added a .prettierignore file, due to the way ALE is configured in nvim it is a pre-req for prettier to work
- Adds a bunch of validations on the AIPersona model, system personas (artist/creative etc...) are all seeded. We now ensure
- name uniqueness, and only allow certain properties to be touched for system personas.
- (JS note) the client side design takes advantage of nested routes, the parent route for personas gets all the personas via this.store.findAll("ai-persona") then child routes simply reach into this model to find a particular persona.
- (JS note) data is sideloaded into the ai-persona model the meta property supplied from the controller, resultSetMeta
- This removes ai_bot_enabled_personas and ai_bot_enabled_chat_commands, both should be controlled from the UI on a per persona basis
- Fixes a long standing bug in token accounting ... we were doing to_json.length instead of to_json.to_s.length
- Amended it so {commands} are always inserted at the end unconditionally, no need to add it to the template of the system message as it just confuses things
- Adds a concept of required_commands to stock personas, these are commands that must be configured for this stock persona to show up.
- Refactored tests so we stop requiring inference_stubs, it was very confusing to need it, added to plugin.rb for now which at least is clearer
- Migrates the persona selector to gjs

---------

Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
Co-authored-by: Martin Brennan <martin@discourse.org>
2023-11-21 16:56:43 +11:00
Sam a4f419f54f
FEATURE: basic infrastructure for custom personas (#288)
- New AiPersona model which can store custom personas
- Persona are restricted via group security
- They can contain custom system messages
- They can support a list of commands optionally

To avoid expensive DB calls in the serializer a Multisite friendly Hash was introduced (which can be expired on transaction commit)
2023-11-10 11:39:49 +11:00
Roman Rizzi d0198c5c5b
FIX: Changes to the sentiment reports. (#289)
This PR aims to clarify sentiment reports by replacing averages with a count of posts that have one of their values above a threshold (60), meaning we have some level of confidence they are, in fact, positive or negative.

Same thing happen with post emotions, with the difference that a post can have multiple values above it (30). Additionally, we dropped the "Neutral" axis.

We also reworded the tooltip next to each report title, and added an early return to signal we have no data available instead of displaying an empty chart.
2023-11-09 17:23:25 -03:00
Roman Rizzi 458e66aef9
FIX: Filter classification type using the correct column (#286) 2023-11-08 14:58:35 -03:00
Roman Rizzi 231cf91cc2
FIX: Don't divide by zero if there is no emotion data for TL group (#285) 2023-11-08 13:05:36 -03:00
Roman Rizzi b172ef11c4
FEATURE: Expose sentiment classifications via the admin dashboard. (#284)
This PR adds new reports for displaying information about post sentiments grouped by date and emotions group by TL.

Depends on discourse/discourse#24274
2023-11-08 10:50:37 -03:00
Sam fc65404896
FEATURE: support topic_id and post_id logging in ai audit log (#274)
This makes it easier to track who is responsible for a completion
in logs

Note: ai helper and summarization are not yet implemented
2023-11-01 08:41:31 +11:00
Sam 0b62c0fa02
FIX: keep parity of shape for image command (#275)
Function calling will start hallucinating if you reshape results.

Previously we were morphing from:

`{ prompts: ["prompt 1", "prompt 2"] }`

to

`{ prompts: { prompt: "prompt 1", seed: 222}, { ... `

This meant that over a few call sequences function_call starts hallucinating an incorrect shape.

This change grounds us even on GPT-3.5
2023-10-31 19:12:25 +11:00
Sam b06380d9fa
FIX: avoid semicolons at the end of queries for SQL Helper (#268)
This makes it easier to cut and paste snippets it is producing

Also fine tune the prompt in an attempt to hone gpt 3.5 which is very finicky
2023-10-27 16:21:09 +11:00
Sam 6add06af8f
FEATURE: Make artist more creative (#266)
This allows for 2 big features:

1. Artist can ship up to 4 prompts for image generation
2. Artist can regenerate images cause it is aware of seed

This allows for iteration on images maintaining visual style
2023-10-27 14:48:12 +11:00
David Battersby cda5cb6e9c
DEV: update test to use new chat message service (#254)
Update test to use the new chat update message service when editing chat messages.
2023-10-23 17:50:25 +08:00
David Battersby abd3da3383
DEV: skip edit message test until new service added to chat (#259)
Temporarily skip the edit chat message test as we are moving MessageUpdate to a new service in Chat.
2023-10-23 15:30:12 +08:00
Sam 1500308437
FEATURE: defer creation of bot users (#258)
Also fixes it so users without bot in header can send it messages.

Previous to this change we would seed all bots with database seeds.

This lead to lots of confusion for people who do not enable ai bot.

Instead:

1. We do not seed any bots **until** user enables the ai_bot_enabled setting
2. If it is disabled we will
  a. If no messages were created by bot - delete it
  b. Otherwise we will deactivate account
2023-10-23 17:00:58 +11:00
Sam f65e50bd9e
FIX: allow for blank fields in Google results (#255)
Under certain cases, for example:

```
there is this japanese band called kirimi, tell me more about them, try searching 3 times and at least 2 times in japanese before answering.
```

Results come back with blank snippets. This adds protection so this
is allowed and code does not simply blow up.
2023-10-19 14:44:59 +11:00
Sam aa463d64f1
FEATURE: Add creative persona (#231)
This adds a new creative persona that has access to the underlying
model and no external integrations.

It allows people to use Claude/GPT models in a Discourse agnostic
way.
2023-09-27 10:48:38 +10:00
Keegan George 2e5a39360a
FEATURE: Create custom prompts with composer AI helper (#214)
* DEV: Add icon support

* DEV: Add basic setup for custom prompt menu

* FEATURE: custom prompt backend

* fix custom prompt param check

* fix custom prompt replace

* WIP

* fix custom prompt usage

* fixes

* DEV: Update front-end

* DEV: No more custom prompt state

* DEV: Add specs

* FIX: Title/Category/Tag suggestions

Suggestion dropdowns broke because it `messages_with_user_input(user_input)` expects a hash now.

* DEV: Apply syntax tree

* DEV: Restrict custom prompts to configured groups

* oops

* fix tests

* lint

* I love tests

* lint is cool tho

---------

Co-authored-by: Rafael dos Santos Silva <xfalcox@gmail.com>
2023-09-25 15:12:54 -03:00
Sam 316ea9624e
FIX: properly truncate !command prompts (#227)
* FIX: properly truncate !command prompts

### What is going on here?

Previous to this change where a command was issued by the LLM it
could hallucinate a continuation eg:

```
This is what tags are

!tags

some nonsense here
```

This change introduces safeguards so `some nonsense here` does not
creep in to the prompt history, poisoning the llm results

This in effect grounds the llm a lot better and results in the llm
forgetting less about results.

The change only impacts Claude at the moment, but will also improve
stuff for llama 2 in future.

Also, this makes it significantly easier to test the bot framework
without an llm cause we avoid a whole bunch of complex stubbing

* blank is not a valid bot response, do not inject into prompt
2023-09-15 07:02:37 +10:00
Roman Rizzi f57c1bb0f6
FEATURE: AI Helper endpoint to generate a thumbnail from text. (#224)
We pass the text to the current LLM and ask them to generate a StableDifussion prompt.
We'll use that to generate 4 samples, temporarily creating uploads and returning their short URLs.
2023-09-14 12:53:44 -03:00
Jarek Radosz 1eb70c4f0a
DEV: Fix rspec-expectations warnings (#228) 2023-09-14 17:50:13 +02:00
Sam 9e94457154
FIX: Made bot more robust (#226)
* FIX: Made bot more robust

This is a collection of small fixes

- Display "Searching for: ..." while searching instead of showing found 0 results.
- Only allow 5 commands in lang chain - 6 feels like too much
- On the 5th command stop informing the engine about functions, so it is forced to complete
- Add another 30 tokens of buffer and explain why
- Typo in command prompt


Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2023-09-14 16:46:56 +10:00
Sam cdd6faa648
FEATURE: add filter support to ai bot semantic search (#222)
Previously we would bypass semantic search if any filters were
present

Also shows progress now.
2023-09-13 14:59:45 +10:00
Sam d75e3ca82b
FEATURE: include tag and category context in search (#217)
Previous to this we just included title/body.. tags and category
structure can be very critical for decision making.
2023-09-12 16:09:28 +10:00
Sam b0310f90d3
FEATURE: add tags and categories to read context (#215)
Note, we perform permission checks on tag list against anon
to ensure we do not disclose information about private tags
to the llm which could get extracted.
2023-09-12 11:06:55 +10:00
Roman Rizzi 0828254d61
FIX: Generate embeddings job was broken (#211)
* FIX: Use correct methods to generate embeddings

* FIX: Generate embeddings job was broken
2023-09-07 11:54:43 -03:00
Sam 615eb8b440
FEATURE: add semantic search with hyde bot (#210)
In specific scenarios (no special filters or limits) we will also
always include 5 semantic results (at least) with every query.

This effectively means that all very wide queries will always return
20 results, regardless of how complex they are.

Also: 

FIX: embedding backfill rake task not working
We renamed internals, this corrects the implementation
2023-09-07 13:25:26 +10:00
Roman Rizzi 13d63f1f30
FIX: filter allowed categories from semantic search results (#206) 2023-09-06 10:00:20 -03:00
Rafael dos Santos Silva 4b42c09814
FEATURE: Tweak HyDE prompts for better grounding in forum subject and limit response size (#200)
* FEATURE: Tweak HyDE prompts for better grounding in forum subject and limit response size

* fix test

* lint
2023-09-05 16:11:07 -03:00
Rafael dos Santos Silva 2c0f535bab
FEATURE: HyDE-powered semantic search. (#136)
* FEATURE: HyDE-powered semantic search.

It relies on the new outlet added on discourse/discourse#23390 to display semantic search results in an unobtrusive way.

We'll use a HyDE-backed approach for semantic search, which consists on generating an hypothetical document from a given keywords, which gets transformed into a vector and used in a asymmetric similarity topic search.

This PR also reorganizes the internals to have less moving parts, maintaining one hierarchy of DAOish classes for vector-related operations like transformations and querying.

Completions and vectors created by HyDE will remain cached on Redis for now, but we could later use Postgres instead.

* Missing translation and rate limiting

---------

Co-authored-by: Roman Rizzi <rizziromanalejandro@gmail.com>
2023-09-05 11:08:23 -03:00
Sam 38af2ca63e
FIX: cut completion short after function call is found (#182)
Previous to this change we would keep completing and throw away
result
2023-09-05 10:37:58 +10:00
Rafael dos Santos Silva 3c4a53b2cb
FEATURE: Better link in Claude summaries (#183)
* FEATURE: Better link in Claude summaries

* lint
2023-09-04 12:04:47 -03:00
Sam e3abbd9f46
FEATURE: add researcher persona (#181)
The researcher persona has access to Google and can perform
various internet research tasks. At the moment it can not read
web pages, but that is under consideration
2023-09-04 12:05:27 +10:00
Sam 181113159b
FIX: setting explorer was exceeding token budget
This refactor changes it so we only include minimal data in the
system prompt which leaves us lots of tokens for specific searches

The new search command allows us to pull in settings on demand

Descriptions are include in short search results, and names only
in longer results

Also: 

* In dev it is important to tell when calls are made to open ai
this adds a console log to increase awareness around token usage

* PERF: stop counting tokens so often

This changes it so we only count tokens once per response

Previously each time we heard back from open ai we would count
tokens, leading to uneeded delays

* bug fix, commands may reach in for tokenizer

* add logging to console for anthropic calls as well

* Update lib/shared/inference/openai_completions.rb

Co-authored-by: Martin Brennan <mjrbrennan@gmail.com>
2023-09-01 11:48:51 +10:00
Loïc Guitaut 65091690eb DEV: Don’t use `Chat::MessageCreator` in specs
As message creation is being rewritten in
https://github.com/discourse/discourse/pull/22390, a new way of using
the underlying service to create chat messages has been implemented in
https://github.com/discourse/discourse/pull/23222.

This patch uses the new fabricator option which will prevent breaking
specs from this plugin when the main PR will be merged.
2023-08-31 11:30:07 +02:00
Sam 00d69b463e
FEATURE: new site setting explorer persona (#178)
Also adds ai_bot_enabled_personas so admins can tweak which stock
personas are enabled.

The new persona has a full listing of all site settings and is
able to get context for each setting.

This means you can ask it to search through settings for something
relevant.

Security wise there is no access to actual configuration of settings
just to the names / description and implementation.

Previously this was part of the forum helper persona however it
just clashes too much with other behaviors, isolating it makes
it far more powerful.

* sneaking this one in, user_emails is a non obvious table in our
structure.

usually one would assume users has emails so the clarifies a bit
better. plus it is a very common table to hit.
2023-08-31 17:02:03 +10:00
Sam db19e37748
FEATURE: add initial support for personas (#172)
This splits out a bunch of code that used to live inside bots
into a dedicated concept called a Persona.

This allows us to start playing with multiple personas for the bot

Ships with:

artist - for making images
sql helper - for helping with data explorer
general - for everything and anything
 
Also includes a few fixes that make the generic LLM function implementation  more robust
2023-08-30 16:15:03 +10:00
Sam 8fdb88604f
FIX: trim first space when getting a reply from anthropic (#164)
Anthropic loves sending a pointless leading space with completions
this throws off the command framework.
2023-08-29 10:57:36 +10:00
Sam b14cb864dc
FEATURE: add setting_context experimental command (#160)
This command can be used to extract information about a discourse
site setting directly from source.

To operate it needs the rg binary in the container.
2023-08-29 10:43:58 +10:00
Sam 7d943be7b2
FIX: automatic bot titles missing sometime (#151)
This fixes 2 big issues:

1. No matter how hard you try, grounding anthropic title prompt
is just too hard. This works around by only looking at the last
sentence it returns and treating as title

2. Non English locales would be stuck with "generic" title, this
ensures every bot message gets a title, using a custom field to
track

Also, slightly tunes some anthropic prompts.
2023-08-24 07:20:24 +10:00
Sam f0e1c72aa7
FEATURE: implement command framework for non Open AI (#147)
Open AI support function calling, this has a very specific shape
that other LLMs have not quite adopted.

This simulates a command framework using system prompts on LLMs
that are not open AI.

Features include:

- Smart system prompt to steer the LLM
- Parameter validation (we ensure all the params are specified correctly)

This is being tested on Anthropic at the moment and intial results
are promising.
2023-08-23 07:49:36 +10:00
Rafael dos Santos Silva ea5a443588
FEATURE: Try to generate OpenAI Summaries in current language (#146)
* FEATURE: Try to generate OpenAI Summaries in current language

* lint
2023-08-21 15:40:32 -03:00
Rafael dos Santos Silva 49f2453c2d
FEATURE: Tweaks to Anthropic Summarization (#138)
* FEATURE: Tweaks to Anthropic Summarization

* fix specs
2023-08-16 15:09:52 -03:00
Rafael dos Santos Silva 0738f67fa4
FIX: Fix embeddings truncation strategy (#139) 2023-08-16 15:09:41 -03:00
Sam 20c1f2d788
FEATURE: basic progress for image generation (#133)
previously you would have to wait quite a while to see the prompt this implements
a very basic implementation of progress so you can see the API is working.

Also: 

- Fix google progress.
- Handle the incredibly rare, zero results from google.
- Simplify command so it is less error prone
- replace invoke and attache results with a invoke
- ensure invoke can only ever be run once
- pass in all the information a command needs in constructor
- use new pattern throughout
- test invocation in isolation
2023-08-14 16:30:12 +10:00
Sam 7eedbf29e0
FIX: refine image and read command (#131)
- Attempt to hint reading is done by sending complete:true
- Do not include post_number in result unless it was sent in
- Rush visual feedback when a command is run (ensure we always revise)
- Include hyperlink in read command description
- Stop round tripping to GPT after image generation (speeds up images by a lot)
- Add a test for image command
2023-08-09 16:01:48 +10:00
Sam 958dfc360e
FEATURE: experimental read command for bot (#129)
This command is useful for reading a topics content. It allows us to perform
critical analysis or suggest answers.

Given 8k token limit in GPT-4 I hardcoded reading to 1500 tokens, but we can
follow up and allow larger windows on models that support more tokens.

On local testing even in this limited form this can be very useful.
2023-08-09 07:19:56 +10:00
Rafael dos Santos Silva 8318c4374c
FIX: Remove muted from Similar list (#127)
* FIX: Remove muted from Similar list
2023-08-08 15:44:10 -03:00
Sam 03e689deb7
FIX: Google command was including full payload (#128)
* FIX: Google command was including full payload

Additionally there was no truncating happening meaning you could blow token
budget easily on a single search.

This made Google search mostly useless and it would mean that after using
Google we would revert to a clean slate which is very confusing.

* no need for nil there
2023-08-08 15:41:57 +10:00
Sam 7edb57c005
DEV: simplify command framework (#125)
The command framework had some confusing dispatching where it would dispatch
JSON blobs, this meant there was lots of parsing required in every command

The refactor handles transforming the args prior to dispatch which makes
consuming far simpler

This is also general prep to supporting some basic command framework in other
llms.
2023-08-04 09:37:58 +10:00
Roman Rizzi 58b96eda6c
REFACTOR: Build related topics using TopicQuery. (#124)
TopicQuery already provides a lot of safeguards and options for filtering topic, and enforcing permissions. It makes sense to rely on it as other plugins like discourse-assign do.

As a bonus, we now have access to the current_user while serializing these topics, so users will see things like unread posts count just like we do for the lists.
2023-08-02 16:58:09 -03:00
Sam 4b0c077ce5
FEATURE: port to use claude-2 for chat bot (#114)
Claude 1 costs the same and is less good than Claude 2. Make use of Claude
2 in all spots ...

This also fixes streaming so it uses the far more efficient streaming protocol.
2023-07-27 11:24:44 +10:00
Roman Rizzi 5f0c617880
REFACTOR: Cohesive narrative for single-chunk summaries. (#103)
Single and multi-chunk summaries end using different prompts for the last summary. This change detects when the summarized content fits in a single chunk and uses a slightly different prompt, which leads to more consistent summary formats.

This PR also moves the chunk-splitting step to the `FoldContent` strategy as preparation for implementing streamed summaries.
2023-07-13 17:05:41 -03:00
Rafael dos Santos Silva 5e3f4e1b78
FEATURE: Embeddings to main db (#99)
* FEATURE: Embeddings to main db

This commit moves our embeddings store from an external configurable PostgreSQL
instance back into the main database. This is done to simplify the setup.

There is a migration that will try to import the external embeddings into
the main DB if it is configured and there are rows.

It removes support from embeddings models that aren't all_mpnet_base_v2 or OpenAI
text_embedding_ada_002. However it will now be easier to add new models.

It also now takes into account:
  - topic title
  - topic category
  - topic tags
  - replies (as much as the model allows)

We introduce an interface so we can eventually support multiple strategies
for handling long topics.

This PR severely damages the semantic search performance, but this is a
temporary until we can get adapt HyDE to make semantic search use the same
embeddings we have for semantic related with good performance.

Here we also have some ground work to add post level embeddings, but this
will be added in a future PR.

Please note that this PR will also block Discourse from booting / updating if 
this plugin is installed and the pgvector extension isn't available on the 
PostgreSQL instance Discourse uses.
2023-07-13 12:41:36 -03:00
Roman Rizzi 1b568f2391
FIX: Claude's max_tookens_to_sample is a required field (#97) 2023-06-27 14:42:33 -03:00
Roman Rizzi 9a79afcdbf
DEV: Better strategies for summarization (#88)
* DEV: Better strategies for summarization

The strategy responsibility needs to be "Given a collection of texts, I know how to summarize them most efficiently, using the minimum amount of requests and maximizing token usage".

There are different token limits for each model, so it all boils down to two different strategies:

Fold all these texts into a single one, doing the summarization in chunks, and then build a summary from those.
Build it by combining texts in a single prompt, and truncate it according to your token limits.

While the latter is less than ideal, we need it for "bart-large-cnn-samsum" and "flan-t5-base-samsum", both with low limits. The rest will rely on folding.

* Expose summarized chunks to users
2023-06-27 12:26:33 -03:00
Sam 9390fba768
FIX: adjust token limits to account for functions (#96)
Reduce maximum replies to 2500 tokens and make them even for both GPT-3.5
and 4

Account for 400+ tokens in function definitions (this was unaccounted for)
2023-06-23 10:02:04 +10:00
Sam a028309cbd
FEATURE: add ai_bot_enabled_chat commands and tune search (#94)
* FEATURE: add ai_bot_enabled_chat commands and tune search

This allows admins to disable/enable GPT command integrations.

Also hones search results which were looping cause the result did not denote
the failure properly (it lost context)

* include more context for google command
include more context for time command

* type
2023-06-21 17:10:30 +10:00
Sam 30778d8af8
FIX: avoid storing corrupt prompts (#92)
```
prompt << build_message(bot_user.username, reply)
```

Would store a "cooked" prompt which is invalid, instead just store the raw
values which are later passed to build_message

Additionally:

1. Disable summary command which needs honing
2. Stop storing decorations (searched for X) in prompt which leads to straying
3. Ship username directly to model, avoiding "user: content" in prompts. This
 was causing GPT to stray
2023-06-20 15:44:03 +10:00
Sam 70c158cae1
FEATURE: add full bot support for GPT 3.5 (#87)
Given latest GPT 3.5 16k which is both better steered and supports functions
we can now support rich bot integration.

Clunky system message based steering is removed and instead we use the
function framework provided by Open AI
2023-06-20 08:45:31 +10:00
Roman Rizzi 3364fec425
DEV: Remove the summarization feature (#83)
* DEV: Remove the summarization feature

Instead, we'll register summarization implementations for OpenAI, Anthropic, and Discourse AI using the API defined in discourse/discourse#21813.

Core and chat will implement features on top of these implementations instead of this plugin extending them.

* Register instances that contain the model, requiring less site settings
2023-06-13 14:32:26 -03:00
Sam 081231a6eb
FIX: support multiple command executions (#85)
Previous to this change we were chaining stuff too late and would execute
commands serially leading to very unexpected results

This corrects this and allows us to run stuff like:

> Search google 3/4 times on various permutations of
QUERY and answer this question.

We limit at 5 commands to ensure there are not pathological user cases
where you lean on the LLM to flood us with results.
2023-06-06 07:09:33 +10:00
Sam 840968630e
FEATURE: disable smart commands on Claude and GPT 3.5 (#84)
For the time being smart commands only work consistently on GPT 4.
Avoid using any smart commands on the earlier models.

Additionally adds better error handling to Claude which sometimes streams
partial json and slightly tunes the search command.
2023-06-01 09:10:33 +10:00
Sam 96d521198b
FIX: missing localization (#81)
blog.start_gpt_chat -> was on my blog

This also slightly tunes the search prompt to support filtering by oldest
and try a tiny bit harder to guide GPT 3.5 which is a bit of a losing battle

Co-authored-by: Krzysztof Kotlarek <kotlarek.krzysztof@gmail.com>
2023-05-25 11:05:02 +10:00
Roman Rizzi c582e3b848
DEV: Fix toxicity test (#78) 2023-05-23 11:02:11 -03:00
Sam d85b503ed4
FIX: guide GPT 3.5 better (#77)
* FIX: guide GPT 3.5 better

This limits search results to 10 cause we were blowing the whole token
budget on search results, additionally it includes a quick exchange at
the start of a session to try and guide GPT 3.5 to follow instructions

Sadly GPT 3.5 drifts off very quickly but this does improve stuff a bit.

It also attempts to correct some issues with anthropic, though it still is
surprisingly hard to ground

* add status:public, this is a bit of a hack but ensures that we can search
for any filter provided

* fix specs
2023-05-23 23:08:17 +10:00
Sam b82fc1e692
FIX: ensure we only attempt embedding once every 15 minutes (#76)
This also heavily reduced log noise and ensures our exception handling is
more surgical.
2023-05-23 10:43:24 +10:00
Sam e0cf7b7d70
FIX: results will be nil for invalid queries (#74)
Previous to this change invalid searches would break the command.
2023-05-22 15:14:26 +10:00
Sam 92fb84e24d
iterate commands (#73)
* FEATURE: introduce a more efficient formatter

Previous formatting style was space inefficient given JSON consumes lots
of tokens, the new format is now used consistently across commands

Also fixes

- search limited to 10
- search breaking on limit: non existent directive

* Slight improvement to summarizer
Stop blowing up context with custom prompts

* ensure we include the guiding message

* correct spec

* langchain style summarizer ...

much more accurate (albeit more expensive)

* lint
2023-05-22 12:09:14 +10:00
Sam d59ed1091b
FEATURE: add support for GPT <-> Forum integration
This change-set connects GPT based chat with the forum it runs on. Allowing it to perform search, lookup tags and categories and summarize topics. 

The integration is currently restricted to public portions of the forum. 

Changes made:

- Do not run ai reply job for small actions
- Improved composable system prompt
- Trivial summarizer for topics
- Image generator 
- Google command for searching via Google
- Corrected trimming of posts raw (was replacing with numbers) 
- Bypass of problem specs

The feature works best with GPT-4


---------

Co-authored-by: Roman Rizzi <rizziromanalejandro@gmail.com>
2023-05-20 17:45:54 +10:00
Roman Rizzi 362f6167d1
FEATURE: Less friction for starting a conversation with an AI bot. (#63)
* FEATURE: Less friction for starting a conversation with an AI bot.

This PR adds a new header icon as a shortcut to start a conversation with one of our AI Bots. After clicking and selecting one from the dropdown menu, we'll open the composer with some fields already filled (recipients and title).

If you leave the title as is, we'll queue a job after five minutes to update it using a bot suggestion.

* Update assets/javascripts/initializers/ai-bot-replies.js

Co-authored-by: Rafael dos Santos Silva <xfalcox@gmail.com>

* Update assets/javascripts/initializers/ai-bot-replies.js

Co-authored-by: Rafael dos Santos Silva <xfalcox@gmail.com>

---------

Co-authored-by: Rafael dos Santos Silva <xfalcox@gmail.com>
2023-05-16 14:38:21 -03:00
Rafael dos Santos Silva 66bf4c74c6
FEATURE: Handle invalid media in NSFW module (#57)
* FEATURE: Handle invalid media in NSFW module

* fix lint
2023-05-11 15:35:39 -03:00
Roman Rizzi 7e3cb0ea16
FEATURE: Multi-model support for the AI Bot module. (#56)
We'll create one bot user for each available model. When listed in the `ai_bot_enabled_chat_bots` setting, they will reply.

This PR lets us use Claude-v1 in stream mode.
2023-05-11 10:03:03 -03:00
Rafael dos Santos Silva e5537d4c77
FEATURE: Allow excluding closed topics from semantic related (#55) 2023-05-09 15:30:50 -03:00
Roman Rizzi 71b105a1bb
FEATURE: Introduce the ai-bot module (#52)
This module lets you chat with our GPT bot inside a PM. The bot only replies to members of the groups listed on the ai_bot_allowed_groups setting and only if you invite it to participate in the PM.
2023-05-05 15:28:31 -03:00
Rafael dos Santos Silva bb0b829634
FEATURE: Anthropic Claude for AIHelper and Summarization modules (#39) 2023-04-10 11:04:42 -03:00
Roman Rizzi 4e05763a99
FEATURE: Semantic assymetric full-page search (#34)
Depends on discourse/discourse#20915

Hooks to the full-page-search component using an experimental API and performs an assymetric similarity search using our embeddings database.
2023-03-31 15:29:56 -03:00
Sam 6543c50758
FIX: stop returning self as a candidate for related topics (#31) 2023-03-31 11:04:17 +10:00
Sam 0d80d9ec49
FEATURE: allow limiting results in related topics section (#30)
Also:

- Normalizes behavior between logged in and anon,
 we only show related topics in the related topic section

- Renames "suggested" to "related" given this only exists in related section
- Adds a spec section to ensure anon does not regress
- Adds `ai_embeddings_semantic_related_topics` to limit related topics

Renamed settings:

ai_embeddings_semantic_suggested_model -> ai_embeddings_semantic_related_model
ai_embeddings_semantic_suggested_topics_enabled -> ai_embeddings_semantic_related_topics_enabled

Plugins is still in an experimental phase and not much is overidden hence
avoiding adding site setting migrations.


Co-authored-by: Krzysztof Kotlarek <kotlarek.krzysztof@gmail.com>
2023-03-31 11:04:34 +11:00
Sam 1d097b9d82
FEATURE: attempt to include related topics above suggested (#28)
Allows related topics to show up for logged on users

- Introduces a new "Related Topics" block above suggested when related topics exist
- Renames `ai_embeddings_semantic_suggested_topics_anons_enabled` -> `ai_embeddings_semantic_suggested_topics_enabled` (given it is only deployed on 1 site not bothering with a migration)
- Adds an integration test to ensure data arrives correctly on the client
2023-03-31 09:07:22 +11:00
Rafael dos Santos Silva 45950f1bb4
FIX: Only show public visible topics as suggested for anons (#27)
* FIX: Only show public visible topics as suggested for anons

* DEV: Add tests for embeddings

* Update spec/lib/modules/embeddings/semantic_suggested_spec.rb

Co-authored-by: Bianca Nenciu <nbianca@users.noreply.github.com>

* Update spec/lib/modules/embeddings/semantic_suggested_spec.rb

Co-authored-by: Bianca Nenciu <nbianca@users.noreply.github.com>

* move to top

---------

Co-authored-by: Bianca Nenciu <nbianca@users.noreply.github.com>
2023-03-23 17:28:01 -03:00
Roman Rizzi 4c960970fa
DEV: Log information about errors from the completions OpenAI API (#26) 2023-03-22 16:00:28 -03:00
Roman Rizzi 320ac6e84b
REFACTOR: Store prompts in a dedicated table. (#14)
This change makes it easier to add new prompts to our AI helper. We don't have a UI for it yet. You'll have to do it through a console.
2023-03-17 15:14:19 -03:00
Joffrey JAFFEUX edfdc6dfae
DEV: applies chat namespacing (#12) 2023-03-17 15:15:38 +01:00
Roman Rizzi f99fe7e1ed
FEATURE: Composer AI helper (#8)
* FEATURE: Composer AI helper

This change introduces a new composer button for the group members listed in the `ai_helper_allowed_groups` site setting.

Users can use chatGPT to review, improve, or translate their posts to English.

* Add a safeguard for PMs and don't rely on parentView
2023-03-15 17:02:20 -03:00
Roman Rizzi aa2fca6086
DEV: DiscourseAI -> DiscourseAi rename to have consistent folders and files (#9) 2023-03-14 16:03:50 -03:00
Roman Rizzi cbaa40edc5
FIX: Do not inherit from classes defined by plugins (#6) 2023-03-08 12:39:03 -03:00
Roman Rizzi a838116cd5
FEATURE: Use dedicated reviewables for AI flags. (#4)
This change adds two new reviewable types: ReviewableAIPost and ReviewableAIChatMessage. They have the same actions as their existing counterparts: ReviewableFlaggedPost and ReviewableChatMessage.

We'll display the model used and their accuracy when showing these flags in the review queue and adjust the latter after staff performs an action, tracking a global accuracy per existing model in a separate table.


* FEATURE: Dedicated reviewables for AI flags

* Store and adjust model accuracy

* Display accuracy in reviewable templates
2023-03-07 15:39:28 -03:00
Roman Rizzi b9a650fde4
DEV: Dedicated table for saving classification results (#1) 2023-02-27 16:21:40 -03:00
Roman Rizzi 5f9597474c
REFACTOR: Streamline flag and classification process 2023-02-24 13:25:02 -03:00
Roman Rizzi 85768cfb1c
FEATURE: Classify posts looking for NSFW images 2023-02-24 09:11:58 -03:00
Roman Rizzi 94933f3c58
DEV: Add missing specs for the toxicity module 2023-02-24 07:53:43 -03:00
Roman Rizzi e8bffcdd64
DEV: Add tests for the sentiment module 2023-02-23 15:50:10 -03:00
Roman Rizzi ef6c785aca
DEV: Move jobs undear each module lib directory 2023-02-23 14:09:52 -03:00
Roman Rizzi 1afa274b99
DEV: Reorganize files and add an entry point for each module 2023-02-23 12:25:00 -03:00
Roman Rizzi 6f0c141062
FEATURE: Introduce NSFW content detection basic flow. 2023-02-23 11:08:34 -03:00