FIX: blank metadata leading to errors (#578)

blank metadata block in RAG was leading to an error, this handles the edge case
This commit is contained in:
Sam 2024-04-17 13:46:40 +10:00 committed by GitHub
parent c2b2741f3d
commit a5e4ab2825
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
4 changed files with 14 additions and 2 deletions

View File

@ -98,6 +98,11 @@ module ::Jobs
to_chunk = after_metadata to_chunk = after_metadata
buffer = buffer.split(metadata_regex, 2).last buffer = buffer.split(metadata_regex, 2).last
overlap = "" overlap = ""
else
current_metadata = new_metadata
buffer = buffer.split(metadata_regex, 2).last
overlap = ""
next
end end
chunk, split_char = first_chunk(to_chunk, tokenizer: tokenizer, chunk_tokens: chunk_tokens) chunk, split_char = first_chunk(to_chunk, tokenizer: tokenizer, chunk_tokens: chunk_tokens)

View File

@ -47,3 +47,6 @@ It was a masterpiece of nonsense, a symphony of absurdity that would leave even
And so, with a flourish of her keyboard and a triumphant grin, Jane unleashed her creation upon the world. The response was immediate and overwhelming. People from all walks of life flocked to converse with Janes AI, marveling at its intelligence, its charm, and its uncanny ability to make even the most mundane of topics seem fascinating. And so, with a flourish of her keyboard and a triumphant grin, Jane unleashed her creation upon the world. The response was immediate and overwhelming. People from all walks of life flocked to converse with Janes AI, marveling at its intelligence, its charm, and its uncanny ability to make even the most mundane of topics seem fascinating.
In the end, Janes discourse AI became the stuff of legend, a shining example of what can be achieved when brilliance, determination, and a healthy dose of eccentricity come together. And as for Jane herself? Well, lets just say that shes already hard at work on her next project: a robot that can make the perfect cup of tea. But that, dear reader, is a story for another day. In the end, Janes discourse AI became the stuff of legend, a shining example of what can be achieved when brilliance, determination, and a healthy dose of eccentricity come together. And as for Jane herself? Well, lets just say that shes already hard at work on her next project: a robot that can make the perfect cup of tea. But that, dear reader, is a story for another day.
[[metadata blank1]]
[[metadata blank2]]
abc

View File

@ -116,3 +116,7 @@ number: 19
the most mundane of topics seem fascinating. the most mundane of topics seem fascinating.
In the end, Janes discourse AI became the stuff of legend, a shining example of what can be achieved when brilliance, determination, and a healthy dose of eccentricity come together. And as for Jane herself? Well, lets just say that shes already hard at work on her next project: a robot that can make the perfect cup of tea. But that, dear reader, is a story for another day. In the end, Janes discourse AI became the stuff of legend, a shining example of what can be achieved when brilliance, determination, and a healthy dose of eccentricity come together. And as for Jane herself? Well, lets just say that shes already hard at work on her next project: a robot that can make the perfect cup of tea. But that, dear reader, is a story for another day.
metadata: blank2
number: 20
abc

View File

@ -57,7 +57,7 @@ RSpec.describe Jobs::DigestRagUpload do
end end
# to rebuild parsed # to rebuild parsed
# File.write("/tmp/testing", parsed) #File.write("/tmp/testing", parsed)
expect(parsed).to eq(parsed_document_with_metadata.read) expect(parsed).to eq(parsed_document_with_metadata.read)
end end