NIFI-13042 Support Python 3.12 for Python Processors

This closes #8644.

- Updated Controller.py main function to join non-daemon threads avoiding RuntimeError on Python 3.12
- Replaced deprecated find_module method with find_spec
- Updated documentation to include support for Python 3.12

Signed-off-by: Joseph Witt <joewitt@apache.org>
This commit is contained in:
exceptionfactory 2024-04-12 17:41:15 -05:00 committed by Joseph Witt
parent 28a9026479
commit 2d9943e2d3
No known key found for this signature in database
GPG Key ID: 9093BF854F811A1A
4 changed files with 13 additions and 6 deletions

View File

@ -24,7 +24,7 @@ Apache NiFi Team <dev@nifi.apache.org>
Apache NiFi can run on something as simple as a laptop, but it can also be clustered across many enterprise-class servers. Therefore, the amount of hardware and memory needed will depend on the size and nature of the dataflow involved. The data is stored on disk while NiFi is processing it. So NiFi needs to have sufficient disk space allocated for its various repositories, particularly the content repository, flowfile repository, and provenance repository (see the <<system_properties>> section for more information about these repositories). NiFi has the following minimum system requirements:
* Requires Java 21
* Use of Python-based Processors (beta feature) requires Python 3.9, 3.10 or 3.11
* Use of Python-based Processors (beta feature) requires Python 3.9, 3.10, 3.11, or 3.12
* Supported Operating Systems:
** Linux
** Unix
@ -231,7 +231,7 @@ The `name` attribute must start with `deprecation`, followed by the component cl
NiFi is a Java-based application. NiFi 2.0 introduces support for a Python-based Processor API. This capability is still
considered to be in "Beta" mode and should not be used in production. By default, support for Python-based Processors is disabled. In order to enable it,
Python 3.9, 3.10 or 3.11 must be installed on the NiFi node (Python 3.12 is not supported yet).
Python 3.9, 3.10, 3.11, or 3.12 must be installed on the NiFi node.
The following properties may be used to configure the Python 3 installation and process management. These properties are all located under the
"Python Extensions" heading in the _nifi.properties_ file:

View File

@ -509,7 +509,7 @@ because we know that the ListFile Processor will produce these attributes for us
[[requirements]]
== Requirements
The Python API requires that Python 3.9, 3.10 or 3.11 is available on the machine hosting NiFi (Python 3.12 is not supported yet).
The Python API requires that Python 3.9, 3.10, 3.11, or 3.12 is available on the machine hosting NiFi.
Each Processor may have its own list of requirements / dependencies. These are made available to the Processor by creating a separate
environment for each Processor implementation (not for each instance of a Processor on the canvas). PyPI is then used to install these

View File

@ -128,3 +128,10 @@ if __name__ == "__main__":
gateway.java_gateway_server.resetCallbackClient(
gateway.java_gateway_server.getCallbackClient().getAddress(),
python_port)
# Join main thread to non-daemon threads in order to avoid RuntimeError on Python 3.12 blocking new thread creation in Py4J
import threading
for thread in threading.enumerate():
if thread.daemon or thread is threading.current_thread():
continue
thread.join()

View File

@ -188,8 +188,8 @@ class ExtensionManager:
if not require_nifi_prefix or name.startswith('nifi_'):
module_file = '<Unknown Module File>'
try:
module = finder.find_module(name)
module_file = module.path
module = finder.find_spec(name)
module_file = module.origin
# Ignore any packaged dependencies
if 'NAR-INF/bundled-dependencies' in module_file: