- Moved commons-lang3 version management to root Maven configuration
- Refactored limited usage of StringUtils is nifi-reporting-utils to remove commons-lang3
- Refactored limited usage of StringUtils in nifi-websocket-processors to remove commons-lang3
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5773
The NiFi and NiFi Bootstrap processes both bind to random ephemeral
ports to allow for inter-process communication (e.g. shutdown, port,
ping, etc.). However, the randomness of these ephemeral ports can pose
challenges for some security policies and firewall rules.
This adds two configuration options, nifi.bootstrap.listen.port and
nifi.listener.bootstrap.port, that allow an administrator to define
which ports the two processes should bind to for this communication,
making it easier to define security policies. The options default to
zero to maintain the current ephemeral port behavior.
NIFI-6740: Add configuration options to specify NiFi/Bootstrap communication ports
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5746
- Upgraded nifi-bootstrap-utils JNA from 4.4.0 to 5.10.0
- Upgraded nifi-windows-event-log-processors JNA from 4.5.2 to 5.10.0
- Upgraded nifi-toolkit-cli JLine from 3.5.2 to 3.21.0
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5766.
- Added okhttp-bom to root Maven configuration
- Removed repeated okhttp.version property from specific dependencies
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5737.
- Corrected several missing return statements
- Increased Maven heap size to 3 GB for ci-workflow
This closes#5700
Signed-off-by: David Handermann <exceptionfactory@apache.org>
This PR introduces 2 new properties for the ConsumeAMQP processor
And one new property for PublishAMQP
This allows to configure the processors to use escaping for commas and to consistently not use curly braces in the amqp$header attribute.
The default values ensure backwards compatibility.
This closes#5458.
Signed-off-by: Kevin Doran <kdoran@apache.org>
- Added nifi-property-protection-api with provider interfaces
- Added nifi-property-protection-factory with implementation references
- Added ProtectionSchemeResolver for abstracting conversion from command arguments
- Refactored PropertyProtectionScheme to package private visibility
- Refactored multiple unit test and removed provider integration tests
- Renamed AESSensitivePropertyProvider to AesGcmSensitivePropertyProvider
- Added getSupportedProtectionSchemes() to StandardProtectionSchemeResolver
- Updated command argument descriptions for protection schemes to include supported values
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5650.
- Upgraded Apache HttpCore to 4.4.15
- Added dependency management declarations in root Maven configuration for HttpClient and HttpCore
- Removed version numbers from multiple modules
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5647.
Ensure that we delegate calls to write(byte[]) and write(byte[], int, int) to the underlying OutputStream for NonFlushableOutputStream, instead of allowing FilterOutputStream to iterate over every byte
This closes#5642
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Centralized Spring Framework and Spring Security versions using BOM dependencies
- Upgraded Spring Security from 5.5.2 to 5.6.1
- Upgraded Spring Boot from 2.5.5 to 2.5.8 in Registry
- Upgraded Spring Integration from 5.5.2 to 5.5.7
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5631.
- Implemented override for Timestamp Record Field Type format handling to add support for optional microseconds
- Added FieldConverter and ObjectTimestampFieldConverter implementation for generalized Timestamp parsing using DateTimeFormatter
- Updated PutKudu unit tests for standard Timestamp and Timestamp with microseconds
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5589.
- Refactor background transaction extension to ExtendTransactionCommand
- Avoid closing S2S HTTP client for IllegalStateExceptions
- Avoid creating additional S2S HTTP client instance for transaction extension commands
- Add check for extend transaction requests received in client test class
- Add null check for Peer Persistence implementation in PeerSelector
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5577.
- Changed framework so that it serializes the dataflow into a VersionedDataflow using JSON as well as XML, and prefers the JSON representation on load, if it's available. This also results in the need for the cluster protocol to exchange its representation of the dataflow to using JSON. Rather than re-implementing all of the complex logic of Flow Fingerprinting, updated to just inherit the cluster's flow.
- Moved logic to synchronize Process Group with Versioned Process Group into a new ProcessGroupSynchronizer class instead of having all of the logic within StandardProcessGroup
- Reworked versioned components to use an instance id.
- Renamed StandardFlowSynchronizer to XmlFlowSynchronizer; introduced new StandardFlowSynchronizer that delegates to the appropriate (Xml or Versioned)FlowSynchronzer
- Updated to allow import of VersionedProcessGroup even if not all bundles are available - will now use ghost components
- Introduced a VersionedDataflow object to hold controller-level services, reporting tasks, parameter contexts, templates, etc.
- Allow mutable requests to be made while nodes are disconnected. Also fixed issue in AbstractPolicyBasedAuthorizer that caused ClassNotFoundException / NoClassDefFoundError if the authorizations were changed and then a node attempts to rejoin the cluster. The Authorizer was attempting to use XmlUtils, which is in nifi-security-utils and so so by madking nifi-security-utils a provided dependency of nifi-framework-api, but this doesn't work, because nifi-framework-api is loaded by a higher-level classloader, so the classloader that loads AbstractPolicyBasedAuthorizer will never have the appropriate classloader to provide nifi-security-utils. Addressed this by copying the code for creating a safe document builder from XmlUtils to AbstractPolicyBasedAuthorizer.
- Fixed bug that occurred when importing a Process Group that has 2 parameter contexts, one inheriting from another, where neither is pre-defined in the existing flow
- Fixed bug that was encountered when Updating a Versioned Process Group where one version had a disabled processor and the other had the processor running.
- Increased system-tests workflow timeout to 120 minutes
- Added additional exception handling to system tests
This closes#5514
Signed-off-by: David Handermann <exceptionfactory@apache.org>
* NIFI-9382: Created a new ClassloaderIsolationKey mechanism by which Hadoop related processors (and potentially others) can indicate that they need full classloaders to be cloned but can share with other instances in certain circumstances
- Added system tests
* NIFI-9382: Renamed interface based on review feedback
* NIFI-9382: Removed ReentrantKerberosUser.
- Added nifi.web.request.log.format property
- Added Filters to set and retrieve authenticated username for logging
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5527.
- Replaced Netty 3.6.9 and 3.7.1 with 3.10.6
- Replaced Netty 4.1 with 4.1.69
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5490.
- Removed GroovyTestCase super class
- Replaced shouldFail() with assertThrows()
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5481.
- Upgraded Spring Security from 5.5.1 to 5.5.2
- Upgraded Spring Boot from 2.5.2 to 2.5.5 in Registry
- Upgraded Spring Data Redis from 2.5.3 to 2.5.5 in Redis modules
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5463.
- Updated documentation with new properties
- Refactored cipher operations to common RepositoryEncryptor classes
- Abstracted record metadata serialization for better compatibility
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5407.
- Added unit tests to reproduce issues with available() method
- Changed available() to return size of application buffer
- Removed unused isDataAvailable()
- Refactored unwrap handling to read from channel for buffer underflow
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5421.
- Removed nifi.minifi.sensitive.props.provider Property from MiNiFi
- Removed property from example NiFi properties files
- Removed provider from MiNiFi SensitivePropsSchema
- Removed BC provider value from MiNiFi test cases
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5422.
- Set NIFI_PBKDF2_AES_GCM_256 as property encryption method
- Replaced static default sensitive properties key with random UUID
- Added unit test for PropertiesFileEngineConfigurationParser
- Added random encryption key generation method
- Changed Stateless to use PropertyEncryptionMethod enum
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5424
- Replaced use of Authorization header with custom Request-Token header for CSRF mitigation
- Added Request-Token cookie for CSRF mitigation
- Replaced session storage of JWT with expiration in seconds
- Removed and disabled CORS configuration
- Disabled HTTP OPTIONS method
- Refactored HTTP Proxy URI construction using RequestUriBuilder
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5417.
NIFI-9192: ResultSetRecordSet consider value of useLogicalType flag when sqlType is not handled separately (default) and readerSchema is not null
Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>
NIFI-9192: Unit tests added to test the use of useLogicalType flag
Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>
NIFI-9192: ResultSetRecordSet consider not using logical type when sqlType is OTHER
NIFI-9192: Fixed checkstyle violations
Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>
NIFI-9192: Renamed JdbcCommon.getUseLogicalTypes to isUseLogicalTypes
Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>
NIFI-9192: Added unit tests for Array types with and without using logical types
Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>
NIFI-9192: Review comments considered and unit tests refactored
NIFI-9192: Added further unit tests
NIFI-9192: Minor modifications based on review comments
Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5388
- Implemented ApplicationCookieService for adding and retrieving HTTP Cookies
- Added getCookieResourceUri() leveraging allowed proxy headers to support optional Cookie Paths
- Refactored Access Resources to use ApplicationCookieService for processing
- Changed __Host- prefix to __Secure- prefix for Bearer Token cookie to support Cookie Path processing
- Removed unnecessary jetty-http dependency from nifi-web-api
- Corrected NiFi path references in JavaScript to support prefixed paths
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5329.
- Added TemporaryKeyStoreBuilder with File.deleteOnExit() for KeyStore and TrustStore files
- Removed JKS files from nifi-security-utils tests
- Refactored usage of KeyStoreUtils.createKeyStoreAndGetX509Certificate() to TemporaryKeyStoreBuilder
- Removed unnecesary hadoop-minikdc test dependency in security-utils
- Replaced Mini KDC Hex utility with Bouncy Castle Hex utility in unit tests
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5406
- Added EncodedSensitivePropertiesProvider with Base64 encoding methods
- Added ClientBasedEncodedSensitivePropertiesProvider with validate method
- Abstracted client configuration to ClientProvider interface and implementations
- Added unit tests for AWS and Azure Property Providers
NIFI-9184 Adjusted abstract provider class names and updated documentation
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5363.
- Refactored multiple tests using KeyStoreUtils
- Removed static KeyStore and TrustStore files
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5401
- Replaced old com.sun.xml.bind:jaxb-impl and jaxb-core with current org.glassfish.jaxb:jaxb-runtime
- Replaced old javax.xml.bind:jaxb-api with current jakarta.xml.bind-api
- Removed unnecessary dependency references to javax.activation-api
This closes#5320
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
- Updated DataTypeUtils.getDateFormat() to use system default time zone
- Updated Record Path Guide to match Expression Language Guide regarding default time zone
- Updated impacted unit tests to expect localized dates instead of dates converted to GMT
NIFI-8749 Set user.timezone in TestRecordPath to avoid unexpected date conversions
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5361
- Updated DataTypeUtils.getDateFormat() to use system default time zone
- Updated Record Path Guide to match Expression Language Guide regarding default time zone
- Updated impacted unit tests to expect localized dates instead of dates converted to GMT
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5305
* NIFI-9082 Added nifi.zookeeper.jute.maxbuffer property
- Updated ZooKeeperStateProvider to check jute.maxbuffer instead of hard-coded number
- Updated internal ZooKeeperClientConfig class to include new juteMaxBuffer property
* NIFI-9082 Updated jute.maxbuffer documentation and added hexadecimal representation
- Renamed DefaultWatcher to NoOpWatcher
- Removed unnecessary validateDataSize() from createNode()
- Replaced per-user symmetric-key HS256 with shared and rotated RSA asymmetric-key RS512 implementation
- Added nifi.security.user.jws.key.rotation.period property for RSA Key Pair rotation
- Added JSON Web Tokens section to Administration Guide
- Implemented persistent storage of RSA Public Keys for verification using Local State Manager
- Implemented JWT revocation on logout with persistence using Local State Manager
- Refactored JWT implementation using Spring Security OAuth2 and Nimbus JWT
- Refactored Spring Security Provider configuration using Java instead of XML
- Removed H2 storage of per-user keys
- Upgraded nimbus-jose-jwt from 7.9 to 9.11.2
NIFI-8766 Corrected AuthenticationException handling in AccessResource.getAccessStatus
- Added nifi.user.security.jws.key.rotation.period to default nifi.properties
- Updated logging statements and clarified configuration and method documentation
NIFI-8766 Changed Algorithm to PS512 and updated documentation
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5262.
NIFI-8974 Integrate KerberosUserService with HDFS processors
NIFI-8980 Integrate KerberosUserService with Kafka 2.6 processors
- Introduced SelfContainerKerberosUserService to restrict which impls can be used with Kafka
- Add variations of KerberosUser doAs that allow setting the context ClassLoader
- Add additional unit tests for configurations
This closes#5277
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Upgraded direct and transitive dependencies from 1.20 and earlier to 1.21
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5273.
- Upgrades Spring Framework from 5.3.8 to 5.3.9
- Upgrades Spring Security from 5.4.6 to 5.5.1
- Upgrades Spring Boot from 2.5.1 to 2.5.2 in Registry
- Upgrades Spring Data Redis from 2.5.1 to 2.5.3
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5252.
- Added Jetty DoSFilter configured for /access/token
- Added nifi.web.max.access.token.requests.per.second property with default value of 25
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5215.
- Updated PutKudu to use DataTypeUtils.toLocalDate() for DATE fields
- Updated PutDatabaseRecord to remove convertDateToLocalTZ() since convertType() uses toLocalDate()
- Updated PutElasticsearchHttpRecord to use default time zone format for DATE fields
- Updated WriteXMLResult to use default time zone format for DATE fields
- Updated WriteJsonResult to use default time zone format for DATE fields
- Updated AvroTypeUtil to use toLocalDate() for logical DATE fields
- Updated JdbcCommon to avoid conversion to UTC for logical DATE fields
- Updated Processor and RecordReader unit tests for consistency in DATE comparison
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5210
- KeyStoreKeyProvider supports PKCS12 and BCFKS
- Refactored KeyProvider and implementations to nifi-security-kms
- Updated Admin Guide and User Guide with KeyStoreKeyProvider details
NIFI-8511 Improved documentation and streamlined several methods
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5110.
- Handling additional FINISHED Handshake Status for TLS 1.3 Post-Handshake Messages per RFC 8446 Section 4.6
- Removed clearing buffers after handshake to avoid losing packets
- Updated read() method to check Handshake Status after SSLEngine.unwrap()
- Changed SSLSocketChannelSender to close SSLSocketChannel before other resources
- Added ChannelStatus enum and convenience logging methods for tracing status
- Added unit tests for TLS 1.2 and 1.3 using Netty server and client handlers
NIFI-8704 Updated netty-handler to 4.1.65.Final
NIFI-7468 Corrected SSLSocketChannel.read() to return byte read
NIFI-7468 Adjusted comment formatting
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5152.
- Set default HTTPS Port to 9443
- Set default authorizer to single-user-authorizer
- Set default login-identity-provider to single-user-provider
- Updated README.md with authentication instructions using generated credentials
- Updated default URL and port information in Administration and User Guides
- Updated Getting Started Guide with authentication and URL changes
- Updated Docker images to set HTTPS as default configuration
- Updated default HTTPS port to 8443
- Set Cluster Protocol secure property in Docker start scripts
- Added set-single-user-credentials command
- Refactored shared classes to nifi-single-user-utils
- Updated Getting Started documentation and logging
- Updated documentation and TLS Toolkit default ports
- Updated Toolkit Guide and Administration Guide
- Updated README.md with HTTPS links
NIFI-8658: Addressed issue where the RecordField that was provided from Function Filters were not accurate
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5125
* NIFI-8137 Record Path EscapeJson/UnescapeJson functions
* Correct jackson-databind dependency version
* Add negative tests for RecordPath JSON handling; rename RecordPath JSON classes to better match existing functions
Signed-off-by: Otto Fowler <ottobackwards@gmail.com>
This closes#4756.
- Added nifi-event-transport module encapsulating Netty classes
- Refactored unit tests for PutSyslog and ListenSyslog
- Removed integration tests for PutSyslog and ListenSyslog
NIFI-8462 Added context.yield() in PutSyslog when no FlowFiles and addressed other issues
NIFI-8462 Removed unused import of ExpressionLanguageScope
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5044.
- Upgraded direct dependencies from 2.6 to 2.8.0
- Added dependency management configuration to use 2.8.0 for some modules
- Updated scripted Groovy tests to avoid copying unnecessary files
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5073
NIFI-6061: Force getBytes() in BLOB handling to use UTF-8 charset
NIFI-6061: Use setClob(), added unit tests, incorporated review comments
This closes#5049
Co-authored-by: zhangcheng <zhangchengk@foxmail.com>
Signed-off-by: David Handermann <exceptionfactory@apache.org>
exceptionfactory
Steve Lawrence
sedadgn
Mark Payne
Pierre Villard
Matthew Burgess
Joe Witt
Mike Thomsen
Nathan Gough
Joe Gresock
Paul Grey
Peter Gyori
denes
Emilio Setiadarma
Bryan Bende
Lehel Boér
Timea Barna
Bryan Bende
Peter Turcsanyi
rhavermans
Chris Sampson
Bence Simon
Sandro Tolaini