- Added dependency management override for Groovy 2.4.21 in Hive Processors
- Upgraded TinkerPop Gremlin to 3.5.1 in graph bundle to depend on Groovy 2.5.14
- Adjusted TlsHelperGroovyTest variable declaration to avoid compiler warnings
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5276
- Upgraded direct and transitive dependencies from 1.20 and earlier to 1.21
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5273.
- Updated MiNiFi references to use shared parent dependency version
- Updated direct dependencies from 1.28 to 1.29
- Added dependency management declarations where necessary to override transitive versions
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5275.
- Added debug logs and a new method to get schema info without making subjects API calls
This closes#5250
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Replaced expected SSLException with IOException in assertThrows
- Removed conditional override of jdk.tls.disabledAlgorithms in test SslContextUtils
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5260.
- Upgrades Spring Framework from 5.3.8 to 5.3.9
- Upgrades Spring Security from 5.4.6 to 5.5.1
- Upgrades Spring Boot from 2.5.1 to 2.5.2 in Registry
- Upgrades Spring Data Redis from 2.5.1 to 2.5.3
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5252.
* NIFI-8939: Ensure that when async/long-running flow updates are made, referencing controller services that are disabling are waited on but not attempted to be disabled
* NIFI-8939: Ensure that when waiting for Controller Services to reach desired state, we use correct URI for fetch service state. There was a typo that resulted in not getting all controller services' states.
This closes#5240
NIFI-8442 Put DateTimeFormatter as static and Add comments to explain why ZoneOffset.UTC is required
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5014.
- Upgraded version 4.0.1 to 5.0.0 in nifi-email-processors and nifi-poi-processors
- Removed unnecessary commons-logging dependency in nifi-email-processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5234.
- Refactored response handling to use shared sendError() method
- Standardized request logging to include HTTP Method and URI
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5218.
- Added FLOWFILE_ATTRIBUTES expression language support to the Storage Account Name and
and also to the Storage Account Key property to be consistent with
AzureStorageCredentialsControllerService
- ADLSCredentialControllerService.ACCOUNT_KEY and ADLSCredentialControllerService.SAS_TOKEN
PropertyDescriptor public constants are the same as AzureStorageUtils.ACCOUNT_KEY and
AzureStorageUtils.PROP_SAS_TOKEN respectively, but they haven't been removed to keep
backward compatibility.
NIFI-8762 Removed ADLSCredentialsControllerService.ACCOUNT_KEY and SAS_TOKEN static fields
NIFI-8762 Add test for EL in Account Name and Account Key
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5229.
- Remove reference to ongoing work for Java 11
- Remove references to Bower which is no longer used as of NIFI-2781
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5232
- Added Jetty DoSFilter configured for /access/token
- Added nifi.web.max.access.token.requests.per.second property with default value of 25
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5215.
- Updated PutKudu to use DataTypeUtils.toLocalDate() for DATE fields
- Updated PutDatabaseRecord to remove convertDateToLocalTZ() since convertType() uses toLocalDate()
- Updated PutElasticsearchHttpRecord to use default time zone format for DATE fields
- Updated WriteXMLResult to use default time zone format for DATE fields
- Updated WriteJsonResult to use default time zone format for DATE fields
- Updated AvroTypeUtil to use toLocalDate() for logical DATE fields
- Updated JdbcCommon to avoid conversion to UTC for logical DATE fields
- Updated Processor and RecordReader unit tests for consistency in DATE comparison
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5210
- UnpackContent uses Zip4j when configured with a password property
- UnpackContent uses Commons Compress when a password is not specified
NIFI-8764 Updated Password property description mentioning disabled algorithms
NIFI-8764 Adjusted Password property description
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5201
- KeyStoreKeyProvider supports PKCS12 and BCFKS
- Refactored KeyProvider and implementations to nifi-security-kms
- Updated Admin Guide and User Guide with KeyStoreKeyProvider details
NIFI-8511 Improved documentation and streamlined several methods
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5110.
- include new process group property support in NiFi Registry
- updated documentation to describe and show new feature
- added elements to XSD schema definition
NIFI-8195: update to DAO to fix PG move and copy/paste
update condition to not null vice null
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5192
- Upgraded Angular Material from 1.1.10 to 1.1.26
- Upgraded Moment from 2.24.0 to 2.29.1
- Upgraded JSON Lint from 1.6.2 to 1.6.3
- Upgraded Slickgrid from 2.4.27 to 2.4.38
- Upgraded frontend-maven-plugin from 1.4 to 1.12.0
- Upgraded frontend-maven-plugin NodeJS from 12.7.0 to 12.22.2
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5197.
- Refactored stateless to use this when creating a dataflow so that Parameter Provider implementations can be externalized into NARs. Also updated ExtensionDiscoveringManager such that callers are able to provide a new type of class to be discovered (e.g., ParameterProvider) so that the extensions will be automatically discovered
- Put specific command-line overrides as highest precedence for parameter overrides
- Make ParameterOverrideProvider valid by allowing for dynamically added parameters
- Fixed bug in validation logic, added new system tests to verify proper handling of Required and Optional properties
- Addressed review feedback and fixed some bugs. Also added system test to verify Parameter Providers are working as expected
This closes#5113
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Added getDateFormat() using default time zone instead of GMT time zone from DataTypeUtils.getDateFormat()
NIFI-8748 Adjusted Date Format to use DataType.getFormat()
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5194.
- Added jakarata.activation 2.0.1 to support jakarta.mail 2.0.1
- Clarified description of SMTP TLS property enabling STARTTLS
NIFI-8630 Added final to several variables and adjusted variable declaration
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5179.
As of KUDU-1884, Kudu supports custom Kerberos principals on server-side
and custom SASL protocol (service) names on client-side which must match
the SPN base, i.e. if the SPN is kudu/_HOST, SASL protocol name *must*
be "kudu" in the client to be able to connect to the cluster.
This patch adds the ability to configure this in the PutKudu processor.
Signed-off-by: Attila Bukor <abukor@apache.org>
NIFIDEVS-8195: fixed properties not properly inheriting from template/snippet values
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5094
- Updated secure FTP processors to configure which algorithms, ciphers and message authentication codes are allowed to be used by the SSH Client
- Included Expression Language Variable Registry support for properties
This closes#5061
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Handling additional FINISHED Handshake Status for TLS 1.3 Post-Handshake Messages per RFC 8446 Section 4.6
- Removed clearing buffers after handshake to avoid losing packets
- Updated read() method to check Handshake Status after SSLEngine.unwrap()
- Changed SSLSocketChannelSender to close SSLSocketChannel before other resources
- Added ChannelStatus enum and convenience logging methods for tracing status
- Added unit tests for TLS 1.2 and 1.3 using Netty server and client handlers
NIFI-8704 Updated netty-handler to 4.1.65.Final
NIFI-7468 Corrected SSLSocketChannel.read() to return byte read
NIFI-7468 Adjusted comment formatting
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5152.
Added unit test that is ignored so that it can be manually run for testing performance before/after changes to AvroTypeUtil. Updated AvroTypeUtil to be more efficient by not using Record.getValue() and instead iterating over the Map of values directly. getValue() is less efficient here because we know the RecordField's we are iterating over exist in the schema since they are retrieved from there directly; as a result, any null values still have be looked up by aliaases, but that step can be skipped in this situation. Also avoided looking for fields that exist in Avro Schema and not in RecordSchema just to set default values on GenericRecord - there's no need to set them if they are default values.
This closes#5080
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Set default HTTPS Port to 9443
- Set default authorizer to single-user-authorizer
- Set default login-identity-provider to single-user-provider
- Updated README.md with authentication instructions using generated credentials
- Updated default URL and port information in Administration and User Guides
- Updated Getting Started Guide with authentication and URL changes
- Updated Docker images to set HTTPS as default configuration
- Updated default HTTPS port to 8443
- Set Cluster Protocol secure property in Docker start scripts
- Added set-single-user-credentials command
- Refactored shared classes to nifi-single-user-utils
- Updated Getting Started documentation and logging
- Updated documentation and TLS Toolkit default ports
- Updated Toolkit Guide and Administration Guide
- Updated README.md with HTTPS links
- Overriding Calcite incubating versions avoids dependencies from third party repositories
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5142
NIFI-8658: Addressed issue where the RecordField that was provided from Function Filters were not accurate
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5125
NIFI-8133 pass null/empty field handling from PutElasticsearchRecord to ElasticSearchClientImpl
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4755
Also noticed a typo in the ControllerStatusReportingTask and found in comparing outputs
that it had a bug that caused it to log counters generated only by processors at the root level so fixed that.
This closes#5101
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Upgrade resolves issue unpacking Zip files with temporary spanning markers
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5103.
NIFI-8528 Migrate NiFi Registry fully codebase into NiFi as a module. No changes except certain dependency scopes to preserve the NiFi Registry original by overriding the new parent (nifi).
- Version adjustments. Removed distinct checkstye rules form nifi-registry. (Using nifi's instead.)
- Made some tests Windows-compatible.
- Consolidated LICENSE, NOTICE and README.md.
- Fixed CryptoKeyLoaderGroovyTest.groovy.
- Disable frontend-maven-plugin on Windows.
- Skipping all goals of the frontend-maven-plugin on Windows.
- Registry integration tests not to run in github jobs (same as the original settings). Skip all registry tests (build and run) on Windows.
- Removed Husky from registry.
- Added nifi-event-transport module encapsulating Netty classes
- Refactored unit tests for PutSyslog and ListenSyslog
- Removed integration tests for PutSyslog and ListenSyslog
NIFI-8462 Added context.yield() in PutSyslog when no FlowFiles and addressed other issues
NIFI-8462 Removed unused import of ExpressionLanguageScope
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5044.
Also allow to write them as such (byte-arrays) - again, instead of throwing an exception.
NIFI-8439 Fixed unit tests.
NIFI-8439 Allow writing parquet INT96 timestamps if they were read by the same parquet-avro library.
This closes#5006.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
NIFI-8618: Allow for parameters to be set for environment variables. Updated README to indicate this and also corrected some outdated information.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5087
- Unflattening a flattened json
- Preserving primitive arrays such as strings, numbers, booleans and null in a nested json
- Logging errors when failure
- Pretty printing resulted json
This closes#5083
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Implemented custom ThreadPoolExecutor with maximum pool size based on Worker Count property
- Refactored processing methods to ensure KuduSession is always closed
- Added SystemResourceConsideration for Memory
- Removed duplicate dependency on nifi-security-kerberos
- Adjusted method naming to clarify functionality
- Reverted addition of defaultAdminOperationTimeoutMs()
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5020.
- Updated Mock Framework to now fail tests that use ProcessSession.commit() unless they first call TestRunner.setAllowSynchronousSessionCommits(true)
- Updated stateless nifi in order to make use of async session commits
- Fixed bug that caused stateless to not properly handle Additional Classpath URLs and bug that caused warnings about validation to get generated when a flow that used controller services was initialized. While this is not really in scope of NIFI-8469, it was found when testing and blocked further progress so addresssed here.
- If Processor fails to progress when run from stateless, trigger from start of flow until that is no longer the case
- Introduced notion of TransactionThresholds that can limit the amount of data that a flow will bring in for a given invocation of stateless dataflow
- Several new system-level tests
- FileBasedClusterNodeFirewallFactoryBean returns PermitAllClusterNodeFirewall instead of null to avoid having the Spring Framework return a NullBean in Spring Framework 5
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5082
- Added totalActiveQueuedDuration and maxActiveQueuedDuration to the ConnectionStatus object
- Updated FlowFileQueue implementations and supporting code to properly calculate and provide the totalActiveQueuedDuration and maxActiveQueuedDuration for their active queues
- Fixing failing unit test. In examining this it appears that the unit test only accidentally passed in the past and that the object mocked to always throw an exception was not actually being used in the test.
- Adding UI component via ConnectionStatusDescriptor along with updates based on PR comments.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Upgraded direct dependencies from 2.6 to 2.8.0
- Added dependency management configuration to use 2.8.0 for some modules
- Updated scripted Groovy tests to avoid copying unnecessary files
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5073
- Upgraded Spring Framework references from version 4.3.30 to 5.3.6
- Upgraded Spring Security from version 4.2.20 to 5.4.6
- Upgraded Spring Data Redis from 2.1.16 to 2.5.0
- Upgraded Jedis from 2.9.0 to 3.6.0 to match Spring Data Redis 2.5.0
- Upgraded Easy Rules from 3.4.0 to 4.1.0 to support Spring 5
- Upgraded Hortonworks Schema Registry Client from 0.8.1 to 0.9.1 to support Spring 5
- Refactored ThreadPoolRequestReplicatorFactoryBean to implement DisposableBean to handle executor shutdown
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5066.
- Upgraded tika-core from 1.24 to 1.26 in nifi-framework-bundle and nifi-standard-bundle
- Upgraded tika-parsers from 1.24.1 to 1.26 in nifi-media-processors
NIFI-8515 Removed jsr-275 exclusion and confirmed it is no longer included
- TIKA-2535 Replaced the jsr-275 dependency to resolve licensing issues
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5056
NIFI-6061: Force getBytes() in BLOB handling to use UTF-8 charset
NIFI-6061: Use setClob(), added unit tests, incorporated review comments
This closes#5049
Co-authored-by: zhangcheng <zhangchengk@foxmail.com>
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Upgraded direct Spring Framework references to 4.3.30.RELEASE
- Upgraded direct Spring Security references to 4.2.20.RELEASE
NIFI-8513 Updated Spring Framework and Security Notices with major version
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5054.
When selecting run/stop on a process group/canvas/selection, it will try to enable/disable transmission of all involved remote process groups.
NIFI-7788 Supplied same functionality missed when selecting a process group.
NIFI-7788 Updated endpoint URL paths.
NIFI-7788 No need to return list of remote process groups when updating en masse.
NIFI-7788 Added some null checks in RemoteProcessGroupsEndpointMerger.merge.
NIFI-7788 Fix checkstyle violation.
This closes#4516.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
- Retained legacy default Sensitive Properties Key in ConfigEncryptionTool to support migration
- Streamlined default file path and moved key generation conditional
- Refactored with getDefaultProperties()
- Cleared System Property in ConfigEncryptionToolTest
- Added checking and error handling for clustered status
- Added set-sensitive-properties-key command
- Refactored PropertyEncryptor classes to nifi-property-encryptor
- Added nifi-flow-encryptor
- Refactored ConfigEncryptionTool to use FlowEncryptor for supporting AEAD algorithms
- Added Admin Guide section Updating the Sensitive Properties Key
This closes#4857.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Initial copy of MiNiFi Java into NiFi
- Checkpoint for basic integration, load empty flow without startup errors
- Refactor provenance repositories, replace some MiNiFi NARs with NiFi NARs
- Remove MiNiFi LICENSE, NOTICE, and KEYS, change file permissions on build.properties
- Updated MiNiFi LICENSE and NOTICE files
- Fix headless NiFi config, set krb5 file in MiNiFi for Kerberos support
- Removed commented-out lines from POMs, added nifi-kerberos-credentials-service NAR to MiNiFi assembly
- Update Git repo URLs, replace NiFi version property
This closes#4933.
Signed-off-by: Kevin Doran <kdoran@apache.org>
- During cleanup keep "tailingPostRollover" in the updated state.
- Skipping tests that can't run on Windows.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Added PGPPrivateKeyService and PGPPublicKeyService interfaces with standard implementations
- NIFI-7396 EncryptContentPGP writes encryption metadata attributes
- NIFI-6708 Controller Services support ElGamal Public and Private Keys
- NIFI-5346 Controller Services support Keyring Files and ASCII Key properties
- NIFI-5335 Controller Services support multiple public or private keys from keyrings
- NIFI-2983 DecryptContentPGP finds and decrypts Encrypted Data Packets regardless of signing
- NIFI-1694 Controller Services support individual key files or keyrings
NIFI-8251 Refactored Public Key ID Property to Public Key Search
NIFI-8251 Corrected handling of multiple Encrypted Data packets in DecryptContentPGP
- Added unit tests for encryption and decryption with both password-based and public key
- Added PGP NAR dependencies to nifi-assembly
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4842.
- AbstractJsonRowRecordReader - Handle (meaning log a warning and not fail completely) multi-array CHOICE type when data has extra fields (not defined by the schema) and can't determine correct type.
- AvroTypeUtil - Allow multiple different record types in avro union type. Minor refactors. Added documentation fro EqualsWrapper.
Lehel Boér
Mark Bean
Pierre Villard
exceptionfactory
Peter Turcsanyi
Timea Barna
Mohammed Nadeem
Kevin Silva
Joe Gresock
markap14
Denes Arvay
Bryan Bende
Knowles Atchison Jr
Juldrixx
Emilio Setiadarma
Nathan Gough
Peter Gyori
Joe Witt
Paul Grey
thenatog
Matthew Burgess
Guillaume Schaer
Lars Francke
Attila Bukor
Bence Simon
Jon Kessler
simonbence
Chris Sampson
rhavermans
Bernhard Geisberger
Abhishek Kumar
Kent Nguyen
Dominik Przybysz
s9514171
Chris Sampson
tpalfy
Tamas Palfy
Wouter de Vries
Chris Sampson
matthewlannan
Sushil Kumar
Paul Kelly
Otto Fowler