If a Python processor defines dependencies both inline and in a
requirements.txt file, then we need to install the two groups of
dependencies in a single `pip install` command, otherwise pip is
not able to resolve the web of dependencies correctly.
- Added setup-python step with Python 3.12 to ci-workflow for consistent version behavior
This closes#9429
Signed-off-by: David Handermann <exceptionfactory@apache.org>
* NIFI-13632: Updating content access.
- Removing existing content viewer application.
- Introduced new front end based content viewer applicaiton.
- Introduced new bundled content viewers for hex and images.
- Deleted previous image content viewer.
- Migrated existing standard content viewer (which handles json, xml, yaml, csv, and text) to utilize new content access interface.
- Moved standard content viewer into its own NAR.
- Moved and renamed custom ui utils and content viewer utils into nifi-common.
- Added mime type to FlowFileSummary response payload to help drive the initially opened content viewer.
* NIFI-13632: Fixing rat issue.
* NIFI-13632: Fixing CI issue.
* NIFI-13632: Removing standard content viewer war from code coverage.
* NIFI-13632: Addressing review feedback.
* NIFI-13632: Fixing import.
* NIFI-13632: Further simplification of content viewer state.
* NIFI-13632: Removing unneeded code.
* NIFI-13632: Adding a min height around codemirror.
* NIFI-13632: Addressing review feedback.
- Rendering mime type in the content viewer.
- Preventing additions to the browser history as the user changes how the content is viewed.
- Appending forward slash to content viewer UIs to prevent unnecessary redirect.
- Removing Standard Content Viewer title based on review feedback.
- Fixing double scroll bar in the Standard Content Viewer on small browser heights.
- Removed unnecessary application.yml.
- Bumping spring boot version.
This closes#9221
- Removed references to nifi-property-utils under nifi-extension-bundles
This closes#9169
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Rescheduled docker-tests to weekly execution
- Removed scheduled execution of other testing workflows
Signed-off-by: Chris Sampson <chris.sampson82@gmail.com>
This closes#9126
This closes#8978
- Removed nifi-property-protection-api and implementation modules
- Removed nifi-toolkit-encrypt-config and minifi-toolkit-encrypt-config modules
- Removed extra bootstrap.conf configuration files for property protection implementations
Signed-off-by: Joseph Witt <joewitt@apache.org>
- Moving NiFi front end source into a top level maven module. This prepares for the introduction of other UIs that NiFi offers (like Custom UIs, Data Viewers, etc) to be colocated and share common components, styles, and dependencies.
- The nifi-web-frontend module which produces the war that is included in the server nar now no longer contains any source. It simply depends on the new nifi-frontend artifact that bundles all built UIs and unpacks its contents into the resulting war.
- Renaming nifi-web-frontend to nifi-ui. Now nifi-frontend at the top level will hold all frontend apps. In this commit the nifi app in nifi-frontend is bundled into a war called nifi-ui.
This closes#8936
* NIFI-13313:
- Use nifi-web-frontend as the default UI hosted at /nifi no longer deploying nifi-web-ui.
* NIFI-13313:
- Adding logout complete page.
- Updating backend to redirect to new logout complete page.
* NIFI-13313:
- Remove nifi-web-ui module.
* NIFI-13313:
- Updating LICENSE and NOTICE files for dependencies that are no longer included.
* NIFI-13313:
- Updating README.
- Updating proxy config to mirror actual context path.
* NIFI-13313:
- Establishing rewrite rules for redirecting logout complete.
- Setting the default handler for when a request isn't handled to redirect the user to /nifi.
* NIFI-13313:
- Removing nifi-web-error module.
* NIFI-13313:
- Restoring /nifi/logout-complete path.
* NIFI-13313:
- Adding an error handler for the ui which handles redirects to a 404 page.
This closes#8906
* NIFI-13211:
- Decoupling the content viewer from the NiFi UI which is currently pulling in shared dependencies, scripts, styles, and images.
* NIFI-13211:
- Decoupling nifi documentation from the NiFi UI which is currently pulling in shared dependencies, scripts, styles, and images.
* NIFI-13211:
- Decoupling nifi image viewer from the NiFi UI which is currently pulling in shared dependencies, scripts, styles, and images.
* NIFI-13211:
- Decoupling nifi standard content viewer from the NiFi UI which is currently pulling in shared dependencies, scripts, styles, and images.
* NIFI-13211:
- Decoupling nifi jolt transform json from the NiFi UI which is currently pulling in shared dependencies, scripts, styles, and images.
* NIFI-13211:
- Decoupling nifi update attribute from the NiFi UI which is currently pulling in shared dependencies, scripts, styles, and images.
* NIFI-13211:
- Updating rat config.
* NIFI-13211:
- Bumping dependency check timeout.
* NIFI-13211:
- Since this page is included through the context viewer, the dependency imports must directly reference through its context path.
* NIFI-13211:
- Since this page is included through the context viewer, the dependency imports must directly reference through its context path.
This closes#8828
Removed the entire integration-tests-ci profile and reference in GitHub workflow.
Removed all tests which require specific local env configs/cloud credentials/ or simply do not work and thus have been blocked from the build for a long time.
Set Disabled annotation on a couple key tests which are by design manual and must be retained.
The move to testcontainers has been a dramatic improvement in test reliability/behaviors and should be a model for all such tests going forward.
This closes#8818
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Set Codecov action to be conditional based on running in the context of the Apache organization
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#8808.
This closes#8757
- Added frontend.skipTests property to nifi-web-frontend Maven configuration with default value of false
- Configured lint and unit-test execution goals to depend on frontend.skipTests
- Added paths-filter GitHub Actions step for conditional enabling of frontend tests
Signed-off-by: Joseph Witt <joewitt@apache.org>
Simplified dependency management into Bill of Material dependencies as parents that align to classloader structures.
- Moved nifi-framework-bundle to root level
- Moved nifi-nar-bundle modules to root nifi-extension-bundles
- Added nifi-extension-bom for internal and external extensions
- Added nifi-framework-nar-bom and nifi-server-nar-bom for managing framework dependencies
- Renamed nifi-jetty-bundle to nifi-jetty-nar
This closes#8677
Signed-off-by: David Handermann <exceptionfactory@apache.org>
* NIFI-13089:
- Updating PR Template and README for new UI.
* Update nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-frontend/src/main/nifi/README.md
Reformatting the code structure layout.
Co-authored-by: Rob Fellows <rob.fellows@gmail.com>
---------
Co-authored-by: Rob Fellows <rob.fellows@gmail.com>
This closes#8693
- Upgraded cache from 3 to 4
- Upgraded CodeQL from 2 to 3
- Upgraded Codecov from 3 to 4
- Upgraded upload-artifact from 3 to 4
- Upgraded stale from 6 to 9
This closes#8373
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Upgraded setup-java action from v3 to v4
- Upgraded setup-python action from v4 to v5
- Updated system-tests to use Python 3.10 instead of Python 3.9
This closes#8371
Signed-off-by: David Handermann <exceptionfactory@apache.org>
Signed-off-by: Joe Witt <joewitt@apache.org>
- Added some Use Case docs for Python processors and updated Runtime Manifests to include Python based processors as well as Use Case/MultiProcessorUseCase documentation elements. Refactored/cleaned up some of the Python code and added unit tests.
- Added python-unit-tests profile and enabled on Ubuntu and macOS GitHub workflows
This closes#8253
Signed-off-by: David Handermann <exceptionfactory@apache.org>
* [NIFI-12419] - Only include the new UI module when building if the 'include-new-ui' profile is active. Also update the ci-workflow.yaml to activate that profile.
* add the profile activation to the Maven Build command
This closes#8069
- Added report-code-coverage profile to nifi-assembly for aggregation
- Added Codecov reporting to Ubuntu Linux Job in ci-workflow
- Added Codecov badge to README.md
This closes#7995
Signed-off-by: Chris Sampson <chris.sampson82@gmail.com>
- Updated StandardPythonBridge.onProcessorRemoved to avoid throwing exceptions when called with a Processor Type and Version that is not registered
- Updated system-tests workflow to run on changes in the nifi-py4j-bundle
Created new python processors for text embeddings, inserting into Chroma, querying Chroma, querying ChatGPT, inserting into and querying Pinecone. Fixed some bugs in the Python framework. Added Python extensions to assembly. Also added ability to load dependencies from a requirements.txt as that was important for making the different vectorstore implementations play more nicely together.
Excluded nifi-python-extensions-bundle from GitHub build because it requires Maven to use unpack-resources goal, which will not work in GitHub because it uses mvn compile instead of mvn install
- ParseDocument
- ChunkDocument
- PromptChatGPT
- PutChroma
- PutPinecone
- QueryChroma
- QueryPinecone
NIFI-12195 Added support for requirements.txt to define Python dependencies
This closes#7894
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Added dependency-check GitHub workflow
- Upgraded Janino Commons Compiler from 3.1.9 to 3.1.10
- Upgraded Azure SDK BOM from 1.2.16 to 1.2.17
- Upgraded GCP SDK BOM from 26.17.0 to 26.25.0
- Upgraded AWS SDK from 1.12.550 to 1.12.573
- Upgraded Hazelcast from 5.3.2 to 5.3.5
- Upgraded Jersey from 2.40 to 2.41
- Upgraded Camel Salesforce from 3.14.5 to 3.14.9
- Unified ZooKeeper versioning on 3.9.1
- Applied Groovy 2.4.21 to Hive 3 and Iceberg components
- Applied gRPC version 1.59.0 to Asana components
- Applied Jettison 1.5.4 to Atlas and Hive 3 components
- Managed JUnit 4 version to 4.13.2 for MockWebServer
- Excluded HBase libraries from Hive 3 following Iceberg approach
- Excluded Htrace from HBase components
- Upgraded OWASP Dependency Check from 8.4.0 to 8.4.2
- Removed non-applicable dependency check suppressions
- Added dependency check suppressions for non-applicable findings
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#7948.
Use Java Zulu distribution for all docker-tests and integration-tests Github Workflows
This closes#7858
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Replaced Maven Wagon properties to Maven Native HTTP properties as required for Maven 3.9
- Removed the deprecated disableXmlReport argument
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#7573
When waiting for a Processor to stop completely in system tests, also consider any terminated threads. Consider the processor stopped only when all active threads and all terminated threads have gone to 0. Additionally, when stopping all components in a Process Group, wait for the PG to reflect that there are no running components. Previously, we checked for processor active thread counts. However, this was problematic because the flow that was fetched was not fetched recursively and as a result, processors running in embedded groups were not always stopped when the waitForProcessorsStopped(String groupId) method returned. Finally, removed the step for gathering surefire-reports in failed system tests because it was not working as expected and was unneeded because the tests' logs were already gathered into the diagnostic dump that is uploaded.
Added fix for BatchFlowBetweenGroupsIT. The test would intermittently fail, based on timing. The test assumed that when Input Port in Group B was scheduled to run, it would trigger before FlowFiles in Group A finished processing. The Sleep processor caused a 10 ms delay for each of 5 FlowFiles which gives the Input Port about 50 ms to trigger. If it took longer to trigger the first time, the test failed. Adjusted the Sleep processor to use 2 seconds. So now instead of 50 milliseconds to start, it has 10 seconds. Overall this adds about 30-45 seconds to the run time of the test but ensures more-than-adequate time for the test to run, even on extremely slow hardware
This closes#7289
Signed-off-by: David Handermann <exceptionfactory@apache.org>
NIFI-11557: Added an additional system test and updated github actions to include surefire-report in order to help diagnose problem that occurred in one of the last system-test runs in Github. Could not replicate problem locally
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#7265
- Updated GitHub workflow so that system tests include Python 3.9
- Updated GitHub actions to build necessary modules for system tests
This closes#7003
Co-authored-by: David Handermann <exceptionfactory@apache.org>
Signed-off-by: David Handermann <exceptionfactory@apache.org>
* NIFI-11257 Enabled Maven HTTP pooling and additional retries
- Enabled HTTP connection pooling for Maven Wagon
- Configured 30 second timeout for HTTP pooled connection lifespan
- Enabled 5 retries for HTTP connections
- Set maximum connection per route to 5 instead of 20
- Enabled retry for sent HTTP requests
Ferenc Gerlits
Matt Gilman
Joseph Witt
exceptionfactory
Bryan Bende
Mark Payne
Rob Fellows
Chris Sampson
Paul Grey
Nandor Soma Abonyi