Commit Graph

5 Commits

Author SHA1 Message Date
Mark Payne 754baf0a37
NIFI-12308: Create Python Environment in background thread instead of during Processor creation
This closes #7971

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2023-11-02 15:45:43 -05:00
Mark Payne 5bcad9eef3
NIFI-12240 Added Python Processors for Docs, ChatGPT, Chroma, and Pinecone
Created new python processors for text embeddings, inserting into Chroma, querying Chroma, querying ChatGPT, inserting into and querying Pinecone. Fixed some bugs in the Python framework. Added Python extensions to assembly. Also added ability to load dependencies from a requirements.txt as that was important for making the different vectorstore implementations play more nicely together.

Excluded nifi-python-extensions-bundle from GitHub build because it requires Maven to use unpack-resources goal, which will not work in GitHub because it uses mvn compile instead of mvn install

- ParseDocument
- ChunkDocument
- PromptChatGPT
- PutChroma
- PutPinecone
- QueryChroma
- QueryPinecone

NIFI-12195 Added support for requirements.txt to define Python dependencies

This closes #7894

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2023-11-01 16:17:36 -05:00
Mark Payne cbdf32ab79
NIFI-12205: Moved loading of Python dependencies into background thread when processor created instead of during startup. Some code cleanup.
This closes #7863

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2023-10-14 15:05:17 -05:00
Mark Payne c68f5ec9a2
NIFI-11793 Added Python debugging properties and documentation
Added documentation to indicate how to debug Python side of nifi framework, as well as debugging Python processors themselves using VSCode's Remote debugger.
This also provides the ability to launch the Controller process in such a way that it will listen to incoming remote debug connections.

This closes #7469

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2023-07-24 13:23:19 -05:00
Mark Payne 7f7cccfdc2
NIFI-11241 Initial implementation of Python-based Processor API with Py4J
- Updated GitHub workflow so that system tests include Python 3.9
- Updated GitHub actions to build necessary modules for system tests

This closes #7003

Co-authored-by: David Handermann <exceptionfactory@apache.org>
Signed-off-by: David Handermann <exceptionfactory@apache.org>
2023-04-14 15:03:34 -05:00