FIX: blank metadata leading to errors (#578)
blank metadata block in RAG was leading to an error, this handles the edge case
This commit is contained in:
parent
c2b2741f3d
commit
a5e4ab2825
|
@ -98,6 +98,11 @@ module ::Jobs
|
||||||
to_chunk = after_metadata
|
to_chunk = after_metadata
|
||||||
buffer = buffer.split(metadata_regex, 2).last
|
buffer = buffer.split(metadata_regex, 2).last
|
||||||
overlap = ""
|
overlap = ""
|
||||||
|
else
|
||||||
|
current_metadata = new_metadata
|
||||||
|
buffer = buffer.split(metadata_regex, 2).last
|
||||||
|
overlap = ""
|
||||||
|
next
|
||||||
end
|
end
|
||||||
|
|
||||||
chunk, split_char = first_chunk(to_chunk, tokenizer: tokenizer, chunk_tokens: chunk_tokens)
|
chunk, split_char = first_chunk(to_chunk, tokenizer: tokenizer, chunk_tokens: chunk_tokens)
|
||||||
|
|
|
@ -47,3 +47,6 @@ It was a masterpiece of nonsense, a symphony of absurdity that would leave even
|
||||||
And so, with a flourish of her keyboard and a triumphant grin, Jane unleashed her creation upon the world. The response was immediate and overwhelming. People from all walks of life flocked to converse with Jane’s AI, marveling at its intelligence, its charm, and its uncanny ability to make even the most mundane of topics seem fascinating.
|
And so, with a flourish of her keyboard and a triumphant grin, Jane unleashed her creation upon the world. The response was immediate and overwhelming. People from all walks of life flocked to converse with Jane’s AI, marveling at its intelligence, its charm, and its uncanny ability to make even the most mundane of topics seem fascinating.
|
||||||
|
|
||||||
In the end, Jane’s discourse AI became the stuff of legend, a shining example of what can be achieved when brilliance, determination, and a healthy dose of eccentricity come together. And as for Jane herself? Well, let’s just say that she’s already hard at work on her next project: a robot that can make the perfect cup of tea. But that, dear reader, is a story for another day.
|
In the end, Jane’s discourse AI became the stuff of legend, a shining example of what can be achieved when brilliance, determination, and a healthy dose of eccentricity come together. And as for Jane herself? Well, let’s just say that she’s already hard at work on her next project: a robot that can make the perfect cup of tea. But that, dear reader, is a story for another day.
|
||||||
|
[[metadata blank1]]
|
||||||
|
[[metadata blank2]]
|
||||||
|
abc
|
||||||
|
|
|
@ -115,4 +115,8 @@ metadata: Jane's story
|
||||||
number: 19
|
number: 19
|
||||||
the most mundane of topics seem fascinating.
|
the most mundane of topics seem fascinating.
|
||||||
|
|
||||||
In the end, Jane’s discourse AI became the stuff of legend, a shining example of what can be achieved when brilliance, determination, and a healthy dose of eccentricity come together. And as for Jane herself? Well, let’s just say that she’s already hard at work on her next project: a robot that can make the perfect cup of tea. But that, dear reader, is a story for another day.
|
In the end, Jane’s discourse AI became the stuff of legend, a shining example of what can be achieved when brilliance, determination, and a healthy dose of eccentricity come together. And as for Jane herself? Well, let’s just say that she’s already hard at work on her next project: a robot that can make the perfect cup of tea. But that, dear reader, is a story for another day.
|
||||||
|
|
||||||
|
metadata: blank2
|
||||||
|
number: 20
|
||||||
|
abc
|
|
@ -57,7 +57,7 @@ RSpec.describe Jobs::DigestRagUpload do
|
||||||
end
|
end
|
||||||
|
|
||||||
# to rebuild parsed
|
# to rebuild parsed
|
||||||
# File.write("/tmp/testing", parsed)
|
#File.write("/tmp/testing", parsed)
|
||||||
|
|
||||||
expect(parsed).to eq(parsed_document_with_metadata.read)
|
expect(parsed).to eq(parsed_document_with_metadata.read)
|
||||||
end
|
end
|
||||||
|
|
Loading…
Reference in New Issue