Created new python processors for text embeddings, inserting into Chroma, querying Chroma, querying ChatGPT, inserting into and querying Pinecone. Fixed some bugs in the Python framework. Added Python extensions to assembly. Also added ability to load dependencies from a requirements.txt as that was important for making the different vectorstore implementations play more nicely together.
Excluded nifi-python-extensions-bundle from GitHub build because it requires Maven to use unpack-resources goal, which will not work in GitHub because it uses mvn compile instead of mvn install
- ParseDocument
- ChunkDocument
- PromptChatGPT
- PutChroma
- PutPinecone
- QueryChroma
- QueryPinecone
NIFI-12195 Added support for requirements.txt to define Python dependencies
This closes#7894
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Added explicit round in FormatUtils.makeWholeNumberTime()
- Removed unnecessary boxing in component descriptors
- Maintained long number tracking for releasable counts in Wait and Notify Processors
Signed-off-by: Joseph Witt <joewitt@apache.org>
- Added dependency-check GitHub workflow
- Upgraded Janino Commons Compiler from 3.1.9 to 3.1.10
- Upgraded Azure SDK BOM from 1.2.16 to 1.2.17
- Upgraded GCP SDK BOM from 26.17.0 to 26.25.0
- Upgraded AWS SDK from 1.12.550 to 1.12.573
- Upgraded Hazelcast from 5.3.2 to 5.3.5
- Upgraded Jersey from 2.40 to 2.41
- Upgraded Camel Salesforce from 3.14.5 to 3.14.9
- Unified ZooKeeper versioning on 3.9.1
- Applied Groovy 2.4.21 to Hive 3 and Iceberg components
- Applied gRPC version 1.59.0 to Asana components
- Applied Jettison 1.5.4 to Atlas and Hive 3 components
- Managed JUnit 4 version to 4.13.2 for MockWebServer
- Excluded HBase libraries from Hive 3 following Iceberg approach
- Excluded Htrace from HBase components
- Upgraded OWASP Dependency Check from 8.4.0 to 8.4.2
- Removed non-applicable dependency check suppressions
- Added dependency check suppressions for non-applicable findings
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#7948.
- Added standard implementation for formatting Subject and Issuer using RFC 1779
- Replaced direct method references to maintain compatibility with historical getSubjectDN and getIssuerDN methods
This closes#7931
Signed-off-by: Chris Sampson <chris.sampson82@gmail.com>
NIFI-12245: Removed null check that is no longer necessary
NIFI-12245: Fixed checkstyle violation
Signed-off-by: Bence Simon <bsimon@apache.org>
This closes#7899
* NIFI-12206 Refactored Flow History using JetBrains Xodus
- Replaced H2 Database Engine with JetBrains Xodus for persistent storage of FlowConfigurationHistory
- Added EntityStoreAuditService implementation using Xodus PersistentEntityStore
- Removed nifi.h2.url.append from properties
- SensitiveValueEncoder and SecureHasher are no longer required following the removal of support for flow.xml.gz
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#7873.
- Created NIFI-12203 to evaluate issues with flow comparison surfaced in JoinClusterWithDifferentFlow
This closes#7661
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Imported nifi-bom to nifi-commons and nifi-nar-bundles parent modules
- Removed explicit version and scope properties for dependencies declared in nifi-bom
- Removed Security.addProvider() references from several tests
- Refactored KeyStoreUtils to use instance of BouncyCastleProvider instead of BC provider name string
- Refactored MiNiFi references to pass BouncyCastleProvider for BCFKS
Signed-off-by: Joseph Witt <joewitt@apache.org>
- Updated component references to remove use of Object[] wrapping for log methods
This closes#7748
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Upgraded SLF4J from 2.0.7 to 2.0.9
- Upgraded Logback from 1.3.8 to 1.3.11
- Upgraded Testcontainers from 1.18.3 to 1.19.0
- Upgraded Fabric8 Kubernetes from 6.5.1 to 6.8.1
- Upgraded AspectJ from 1.9.19 to 1.9.20.1
- Upgraded Caffeine from 3.1.6 to 3.1.8
- Upgraded AWS SDK from 2.20.103 to 2.20.148
- Upgraded Guava from 32.0.1 to 32.1.2
- Upgraded Nimbus JOSE JWT from 9.31 to 9.33
- Upgraded Apache Tika from 2.8.0 to 2.9.0
- Upgraded gRPC from 1.57.2 to 1.58.0
- Upgraded Google Libraries from 26.17.0 to 26.22.0
- Upgraded Azure SDK from 1.2.13 to 1.2.16
This closes#7733
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Added org.spockframework to the list of banned dependencies
- Removed several other Groovy tests and associated build profiles
Signed-off-by: Joseph Witt <joewitt@apache.org>
- Changed default value of nifi.web.https.application.protocols to include both h2 and http/1.1
- Changed default value of nifi.registry.web.https.application.protocols to include both h2 and http/1.1
- Updated HostHeaderHandler logging
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#7684.
- Added nifi-security-cert for reusable components without dependencies
- Added nifi-security-cert-builder for certificate generation
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#7651
- Added new extensible Component Type: FlowAnalysisRule
- Added DisallowComponentType Rule implementation
- Flow Analysis Rules can be managed from the UI under Controller Settings -> Flow Analysis Rules
- Flow Analysis Rules can be set up with an enforcement policy of WARN or ENFORCE
- Flow Analysis Rules can evaluate an individual Component or a Process Group
This closes#7191
Signed-off-by: David Handermann <exceptionfactory@apache.org>
Added documentation to indicate how to debug Python side of nifi framework, as well as debugging Python processors themselves using VSCode's Remote debugger.
This also provides the ability to launch the Controller process in such a way that it will listen to incoming remote debug connections.
This closes#7469
Signed-off-by: David Handermann <exceptionfactory@apache.org>
This closes#6816
Co-authored-by: Peter Gyori <peter.gyori.dev@gmail.com>
Co-authored-by: Chris Sampson <chris.sampson82@gmail.com>
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Adjusted nifi-repository-encryption to remove dependency on Bouncy Castle Provider
- Updated Google Cloud Provider dependencies to remove exclusions and dependencies on Bouncy Castle that no longer apply to current versions
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#7384
- Removed hamcrest-all from default dependencies
- Added groovy-test to selected modules with Groovy test classes
- Added junit-vintage-engine to selected modules with JUnit 4 test classes
- Corrected references to JUnit 4 assertions in JUnit 5 test classes
- Removed several unnecessary test classes from nifi-socket-utils
- Removed duplicative Registry toolkit test classes
- Removed Kudu integration tests
NIFI-11532 Corrected scope for junit-vintage-engine for Elasticsearch
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#7233
- Moved Java tests from groovy directory to java directory and removed groovy directory from nifi-web-utils
This closes#7333
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Removed NetworkUtils methods for getting available ports
- Updated Socket-based components to support using 0 to listen on a random available port for improved test reliability
This closes#7299
Signed-off-by: David Handermann <exceptionfactory@apache.org>
NIFI-11557: Added an additional system test and updated github actions to include surefire-report in order to help diagnose problem that occurred in one of the last system-test runs in Github. Could not replicate problem locally
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#7265
- Added methods to enumerate Stored Component Identifiers on State Provider interface and implementations
- Added nifi.state.management.provider.cluster.previous to nifi.properties
- Updated State Manager Provider to restore Cluster State from Previous Cluster Provider
- Updated Configuring State Providers documentation for new property
NIFI-10865 allow UpdateRecord to replace the Record root for relative paths, e.g. when a RecordPath function is used to modify selected field(s)
This closes#6708
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
- Updated GitHub workflow so that system tests include Python 3.9
- Updated GitHub actions to build necessary modules for system tests
This closes#7003
Co-authored-by: David Handermann <exceptionfactory@apache.org>
Signed-off-by: David Handermann <exceptionfactory@apache.org>
* NIFI-4890 Refactored OIDC with support for Refresh Tokens
- Implemented OIDC Authorization Code Grant Flow using Spring Security Filters
- Implemented OIDC RP-Initiated Logout 1.0
- Implemented OAuth2 Token Revocation RFC 7009 for Refresh Tokens
- Added OIDC Bearer Token Refresh Filter for updating application Bearer Tokens from Refresh Token exchanges
- Added configurable Token Refresh Window to application properties
- Removed original implementation and supporting classes
* NIFI-4890 Set Bearer Token expiration based on Access Token
* NIFI-4890 Corrected spelling and naming issues based on feedback
This closes#7013
Fixed issue in StandardContentClaimWriteCache in which inner OutputStream class did not have an idempotent close() method; as a result, the stream could be written to while already in use for another active FlowFile; fixed bug in ContentClaimInputStream in which skip() method ignored its own BufferedInputStream - this was discovered because it was causing failures in StandardProcessSessionIT; fixed bug in StandardProcessSessionIT in which the length of StandardContentClaim was being doubled because the OutputStream was setting the claim length but that is already handled at a lower level.
This closes#7087
Signed-off-by: David Handermann <exceptionfactory@apache.org>
* NIFI-10975 Added Kubernetes Leader Election and State Provider
- Added Kubernetes Leader Election Manager based on Kubernetes Leases
- Added Kubernetes State Provider based on Kubernetes ConfigMaps
- Added nifi-kubernetes-client for generalized access to Fabric8 Kubernetes Client
- Added nifi.cluster.leader.election.implementation Property defaulting to CuratorLeaderElectionManager
- Refactored LeaderElectionManager to nifi-framework-api for Extension Discovering Manager
- Refactored shared ZooKeeper configuration to nifi-framework-cluster-zookeeper
* NIFI-10975 Updated Kubernetes Client and StateMap
- Upgraded Kubernetes Client from 6.2.0 to 6.3.0
- Added getStateVersion to StateMap and deprecated getVersion
- Updated Docker start.sh with additional properties
* NIFI-10975 Corrected MockStateManager.assertStateSet()
* NIFI-10975 Upgraded Kubernetes Client from 6.3.0 to 6.3.1
* NIFI-10975 Corrected unregister leader and disabled release on cancel
* NIFI-10975 Corrected findLeader handling of Lease expiration
- Changed LeaderElectionManager.getLeader() return to Optional String
* NIFI-10975 Corrected StandardNiFiServiceFacade handling of Optional Leader
* NIFI-10975 Changed getLeader() to call findLeader() to avoid stale cached values
* NIFI-10975 Updated LeaderElectionCommand to run LeaderElector in loop
* NIFI-10975 Rebased on project version 2.0.0-SNAPSHOT
* NIFI-10975 Corrected Gson and AspectJ versions
- Updated versions to match current main branch and avoid reverting
- Replaced OSUtils.getProcessId() with java.lang.Process.pid() available on Java 9 and following
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#6989.
- Moved StringUtils from nifi-properties to nifi-property-utils
- Moved Peer Identity methods from CertificateUtils to specific Site-to-Site classes
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#6977.
- Restores behavior so that XML parsing failure details will be included in FlowFile error attribute from ValidateXml
This closes#6970
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Upgraded Google Drive library to v3-rev20221219-2.0.0
- Corrected test encryption keys to match expected encoding
Signed-off-by: Joe Witt <joewitt@apache.org>
- Replaced Groovy asserts with JUnit 5 assertions and Groovy shouldFail method Junit 5 with assertThrow method
This closes#6880
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Changed from Bouncy Castle to Sun JSSE Provider for Key Stores to improve reading and writing Trust Stores formatted in PKCS12
- Updated TLS Toolkit Key Password handling to remove setting null for PKCS12
Signed-off-by: Chris Sampson <chris.sampson82@gmail.com>
This closes#6881
- Updated impacted classes to remove redundant import lines
- Removed WebUtilsGroovyTest.groovy class due to use of internal sun.security classes
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#6804.
- Added Commons Net to managed dependencies in root Maven configuration
- Removed version references to 3.3 in MiNiFi and 3.6 in NiFi standard modules
Signed-off-by: Joe Witt <joewitt@apache.org>
- Added LongSupplier for TimedBuffer and ControlRate classes to support overriding System.currentTimeMillis()
This closes#6671
Signed-off-by: Paul Grey <greyp@apache.org>
- Added deprecation warnings for NiFi, Registry, and MiNiFi
- Added RuntimeVersionProvider for shared reference to deprecated and minimum versions
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#6648.
[NIFI-10612] Made suggested change to only test subject value where it is formatted like a Json array or object.
This closes#6574
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
- Added nifi-security-ssl for generalized SSLContext creation
- Removed static keystore and truststore test files from nifi-registry-jetty
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#6514.
- Added org.apache.commons.text.version property in root Maven configuration
- Removed direct version references in favor of managed dependency version
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#6531.
- Refactored nifi-vault-utils to nifi-hashicorp-vault-api and nifi-hashcorp-vault modules
- Added HashiCorpVaultClientService and Standard implementation
This closes#6304
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Replaced individual AWS SDK versions with root managed dependency version
- Set AWS SDK 1 version to 1.12.299
- Set AWS SDK 2 version to 2.17.270
- Suppressed false positive dependency vulnerability for aws-sdk-swf-libraries
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#6405.
- Added nifi-deprecation-log module with interface and implementation using SLF4J
- Updated standard logback.xml with nifi-deprecation.log appender
- Updated NiFiLegacyCipherProvider with deprecation logging
- Set Size, Time Policy, and Total Size Limit for Deprecation Log
This closes#6300
Signed-off-by: Paul Grey <greyp@apache.org>
- Added nifi-web-client implementation based on OkHttp
- Added WebClientServiceProvider Controller Service interface and implementation
- Corrected comments and added unmodifiableMap wrapper
- Added getHeaderNames() and corrected ProxyContext comments
This closes#6268
Signed-off-by: Paul Grey <greyp@apache.org>
- Removed extension of deprecated WebSecurityConfigurerAdapter
- Moved Filter bean configuration associated configuration classes
- Set default Spring Security log level to INFO
- Adjusted CSRF Token Repository to leverage simplified RequestUriBuilder for retrieving allowed context paths
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#6195
- Removed OSUtils.getWindowsProcessId() which removes JNA-based retrieval of Process ID retrieval for Windows on Java 8
- Removing jna-platform from nifi-bootstrap-utils eliminates the library from lib/bootstrap and allows it as a dependency in lib/properties
This closes#6199
Signed-off-by: Joey Frazee <jfrazee@apache.org>
- Bump version to 6.29.5 for arm64 compatibility
- Extract RocksDBFlowFileRepository and supporting code to its own module
- Mark RocksDBFlowFileRepository as deprecated, both in code and documentation
- Log deprecation warning at startup if RocksDBFlowFileRepository is used
- Move native RocksDB info logs to NiFi debug level logs
This closes#6155
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Updated SAML Authentication Configuration with Spring Security SAML 2 components
- Updated Administration Guide with REST Resources
- Replaced SAMLAccessResource methods with applicable Spring Security Filters
- Removed IDP Credential Service and supporting components
- Removed message.logging.enabled, metadata.signing.enabled, and signature.digest.algorithm properties
- Added Access Token Expiration resource method
- Removed Saml2AccessResource and replaced with Access Token Expiration to avoid unnecessary conflicts with SAML login consumer
- Corrected Resource URI handling to support proxy server access
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#6149.
- Excluded slf4j-reload4j implementation of Log4j 1
- Updated nifi-hive3-processors to leverage shared Hadoop version from 3.1.1
- Updated nifi-accumulo-bundle to leverage shared Hadoop version from 3.1.1
- Updated nifi-atlas-bundle to leverage shared Hadoop version from 3.3.2
- Updated nifi-spark-receiver to leverage shared Hadoop version from 3.3.2
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#6139.
Mark Payne
exceptionfactory
Chris Sampson
Bence Simon
Mike Moser
Matt Burgess
Pierre Villard
Peter Turcsanyi
Joseph Witt
Timea Barna
Tamas Palfy
Mike Thomsen
Tamas Neumer
mr1716
timeabarna
dan-s1
Bryan Bende
Arek Burdach
Emilio Setiadarma
Chris Sampson
krisztina-zsihovszki
dependabot[bot]
Peter Turcsanyi
nathluu
Lehel Boér
Hervé Boutemy
Malthe Borch
Joe Gresock
Mark Bathori
simonbence
Nathan Gough
Csaba Bejan
NissimShiman
Kevin Doran