- Removed version declarations from multiple modules
- Adjusted PutDynamoDB to remove catch for IOException not thrown in Commons IO 2.11.0
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#6015.
NIFI-7234 Replaced Jackson 1.X references with Jackson 2.X references in various classes.
NIFI-7234 Added jackson-annotations to nifi-hdfs-processors.
NIFI-7234 Various updates to bring our test cases into better alignment with the Avro specification as of 1.11.
Fixed a checkstyle issue.
NIFI-7234 Made changes requested in a review.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5900
- Removed unnecessary references to jackson.version property
- Removed unnecessary dependency management references to Jackson libraries
This closes#5992
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
- Corrected No Tracking strategy Record Writer handling for ListSFTP
- Updated temporary test files to have last modified time of epoch to avoid intermittent issue with Minimum Age filtering
- Refactored MockCacheService to separate reusable class
This closes#5885
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- NIFI-9129 Refactored as much of nifi-kudu-bundle to use JUnit 5 as possible without breaking the integration tests which use test libraries from the Kudu project that depend on JUnit 4
- NIFI-9109 Refactored nifi-extension-utils to use JUnit 5. Left two tests with JUnit 4 because they used APIs that don't appear to have analogues in JUnit 5
- NIFI-9112 Refactored nifi-groovyx-bundle to use JUnit 5
- NIFI-9142 Refactored nifi-ranger-bundle to use JUnit 5
- NIFI-9136 Refactored nifi-network-bundle to use JUnit 5
This closes#5794
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Added nifi-h2-database module shading H2 1.4.200
- Implemented version checking and migration
This closes#5724
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Moved commons-lang3 version management to root Maven configuration
- Refactored limited usage of StringUtils is nifi-reporting-utils to remove commons-lang3
- Refactored limited usage of StringUtils in nifi-websocket-processors to remove commons-lang3
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5773
* NIFI-9655 Added Queue Logging to ListenUDP
- Added TrackingLinkedBlockingQueue to track the largest queue size
- Updated AbstractListenEventProcessor to write debug log at most once per minute
- AbstractListenEventProcessor updates support both ListenUDP and ListenUDPRecord
- Corrected several missing return statements
- Increased Maven heap size to 3 GB for ci-workflow
This closes#5700
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Added generic type to AbstractPutEventProcessor for compiler checking of event types
- Refactored createTransitUri to shared method in AbstractPutEventProcessor
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5658.
- Removed deprecation from ListenTCP Pool Receive Buffers property
- Added BufferAllocator configuration property for NettyEventServerFactory
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5653.
NIFI-9390: Addressed underlying condition in stateless framework that caused Merge-related processors and similar to not properly be triggered as necessary. Added several system tests to verify different configurations.
NIFI-9390: Simplified the logic for how to iterate over the components in a Stateless flow that are ready to be triggered
This closes#5634.
Co-authored-by: Peter Turcsanyi <turcsanyi@apache.org>
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
- Upgraded Apache HttpCore to 4.4.15
- Added dependency management declarations in root Maven configuration for HttpClient and HttpCore
- Removed version numbers from multiple modules
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5647.
- Removed incorrect usage of TriggerWhenEmpty
- Allow for 0 seconds of gracefully waiting for incoming threads on shutdown since we won't have any
- Updated unit tests to no longer have arbitrary sleep statements
This closes#5639
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Changed display name of Max Number of TCP Connections to Max Number of Worker Threads for ListenTCP
- Set Netty Socket Receive Buffer using Max Socket Buffer Size in ListenTCP
This closes#5599
Signed-off-by: David Handermann <exceptionfactory@apache.org>
NIFI-9382: Fixed issue with SharedInstanceClassLoader where the classloader may get closed but then get used again. When the SharedInstanceClassLoader is closed, we will now ensure that we don't use anymore and instead create a new one.
Signed-off-by: Joe Witt <joewitt@apache.org>
* NIFI-9382: Created a new ClassloaderIsolationKey mechanism by which Hadoop related processors (and potentially others) can indicate that they need full classloaders to be cloned but can share with other instances in certain circumstances
- Added system tests
* NIFI-9382: Renamed interface based on review feedback
* NIFI-9382: Removed ReentrantKerberosUser.
- Addressed compiler warnings in ListenTCP and EventBatcher
- Adjusted ListenTCP property order to match previous version
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5526.
Added <scope>test</scope> tag to the nifi-web-api pom.xml and corrected imports.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5447
- Replaced Netty 3.6.9 and 3.7.1 with 3.10.6
- Replaced Netty 4.1 with 4.1.69
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5490.
- Refactored NettyEventServerFactory to accept nullable InetAddress
- Updated unit tests referencing NettyEventServerFactory
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5426.
NIFI-9192: ResultSetRecordSet consider value of useLogicalType flag when sqlType is not handled separately (default) and readerSchema is not null
Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>
NIFI-9192: Unit tests added to test the use of useLogicalType flag
Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>
NIFI-9192: ResultSetRecordSet consider not using logical type when sqlType is OTHER
NIFI-9192: Fixed checkstyle violations
Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>
NIFI-9192: Renamed JdbcCommon.getUseLogicalTypes to isUseLogicalTypes
Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>
NIFI-9192: Added unit tests for Array types with and without using logical types
Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>
NIFI-9192: Review comments considered and unit tests refactored
NIFI-9192: Added further unit tests
NIFI-9192: Minor modifications based on review comments
Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5388
- Added TemporaryKeyStoreBuilder with File.deleteOnExit() for KeyStore and TrustStore files
- Removed JKS files from nifi-security-utils tests
- Refactored usage of KeyStoreUtils.createKeyStoreAndGetX509Certificate() to TemporaryKeyStoreBuilder
- Removed unnecesary hadoop-minikdc test dependency in security-utils
- Replaced Mini KDC Hex utility with Bouncy Castle Hex utility in unit tests
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5406
- Replaced old com.sun.xml.bind:jaxb-impl and jaxb-core with current org.glassfish.jaxb:jaxb-runtime
- Replaced old javax.xml.bind:jaxb-api with current jakarta.xml.bind-api
- Removed unnecessary dependency references to javax.activation-api
This closes#5320
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
NIFI-8974 Integrate KerberosUserService with HDFS processors
NIFI-8980 Integrate KerberosUserService with Kafka 2.6 processors
- Introduced SelfContainerKerberosUserService to restrict which impls can be used with Kafka
- Add variations of KerberosUser doAs that allow setting the context ClassLoader
- Add additional unit tests for configurations
This closes#5277
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Upgraded direct and transitive dependencies from 1.20 and earlier to 1.21
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5273.
- Updated PutKudu to use DataTypeUtils.toLocalDate() for DATE fields
- Updated PutDatabaseRecord to remove convertDateToLocalTZ() since convertType() uses toLocalDate()
- Updated PutElasticsearchHttpRecord to use default time zone format for DATE fields
- Updated WriteXMLResult to use default time zone format for DATE fields
- Updated WriteJsonResult to use default time zone format for DATE fields
- Updated AvroTypeUtil to use toLocalDate() for logical DATE fields
- Updated JdbcCommon to avoid conversion to UTC for logical DATE fields
- Updated Processor and RecordReader unit tests for consistency in DATE comparison
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5210
- Handling additional FINISHED Handshake Status for TLS 1.3 Post-Handshake Messages per RFC 8446 Section 4.6
- Removed clearing buffers after handshake to avoid losing packets
- Updated read() method to check Handshake Status after SSLEngine.unwrap()
- Changed SSLSocketChannelSender to close SSLSocketChannel before other resources
- Added ChannelStatus enum and convenience logging methods for tracing status
- Added unit tests for TLS 1.2 and 1.3 using Netty server and client handlers
NIFI-8704 Updated netty-handler to 4.1.65.Final
NIFI-7468 Corrected SSLSocketChannel.read() to return byte read
NIFI-7468 Adjusted comment formatting
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5152.
Added unit test that is ignored so that it can be manually run for testing performance before/after changes to AvroTypeUtil. Updated AvroTypeUtil to be more efficient by not using Record.getValue() and instead iterating over the Map of values directly. getValue() is less efficient here because we know the RecordField's we are iterating over exist in the schema since they are retrieved from there directly; as a result, any null values still have be looked up by aliaases, but that step can be skipped in this situation. Also avoided looking for fields that exist in Avro Schema and not in RecordSchema just to set default values on GenericRecord - there's no need to set them if they are default values.
This closes#5080
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Added nifi-event-transport module encapsulating Netty classes
- Refactored unit tests for PutSyslog and ListenSyslog
- Removed integration tests for PutSyslog and ListenSyslog
NIFI-8462 Added context.yield() in PutSyslog when no FlowFiles and addressed other issues
NIFI-8462 Removed unused import of ExpressionLanguageScope
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5044.
Also allow to write them as such (byte-arrays) - again, instead of throwing an exception.
NIFI-8439 Fixed unit tests.
NIFI-8439 Allow writing parquet INT96 timestamps if they were read by the same parquet-avro library.
This closes#5006.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
- Updated Mock Framework to now fail tests that use ProcessSession.commit() unless they first call TestRunner.setAllowSynchronousSessionCommits(true)
- Updated stateless nifi in order to make use of async session commits
- Fixed bug that caused stateless to not properly handle Additional Classpath URLs and bug that caused warnings about validation to get generated when a flow that used controller services was initialized. While this is not really in scope of NIFI-8469, it was found when testing and blocked further progress so addresssed here.
- If Processor fails to progress when run from stateless, trigger from start of flow until that is no longer the case
- Introduced notion of TransactionThresholds that can limit the amount of data that a flow will bring in for a given invocation of stateless dataflow
- Several new system-level tests
- Upgraded direct dependencies from 2.6 to 2.8.0
- Added dependency management configuration to use 2.8.0 for some modules
- Updated scripted Groovy tests to avoid copying unnecessary files
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5073
- AbstractJsonRowRecordReader - Handle (meaning log a warning and not fail completely) multi-array CHOICE type when data has extra fields (not defined by the schema) and can't determine correct type.
- AvroTypeUtil - Allow multiple different record types in avro union type. Minor refactors. Added documentation fro EqualsWrapper.
- Updated components to make use of new feature
NIFI-8206: Added a ResourceType of TEXT. This requires that the ResourceReferenceFactory know which types are allowed in order to create the ResourceReference. PropertyValue needs to then have the PropertyDescriptor available to it. This resulted in highlighting many bugs in unit tests where components were not exposing property descriptors via getSupportedPropertyDescriptors() or were evaluating Expression Language using the wrong scope, so fixed many unit tests/components to properly declare Expression Language scope when using it
NIFI-8206: Removed problematic unit test that required directory names with special characters that are not allowed on some operating systems
This closes#4890.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-8329 - Removed unnecessary jackson.version from azure bundle to use the global property instead.
NIFI-8329 - Updated jackson/jackson-databind version and removed the 'jackson-databind.version' pom property in favor of 'jackson.version'
Updated dependencies include the following:
- jackson-core
- jackson-databind
- icu4j
- snakeyaml
- spring-integration-mail
- spring-core and framework modules
- activemq-client
- activemq-broker
- xercesImpl
This closes#4911
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Created nifi-security-socket-ssl
- Created nifi-security-kerberos
- Removed nifi-security-utils dependency from nifi-processor-utils
- Updated modules to reference new dependencies
- Eliminated unnecessary transitive dependencies on bcprov-jdk15on from over 30 modules
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4881.
- Refactored TestPutTCP to single class
- Improved TestListenRELP
- Improved TestListenTCP
- Improved TestListenUDP
- Improved TestListenTCPRecord
- Changed OnUnscheduled to OnStopped in AbstractListenEventProcessor
Signed-off-by: Joe Witt <joewitt@apache.org>
NIFI-6752 Refactored type and value conversion logic. Added support for more types. Added more tests.
Removed 'parent' from 'Recursive'. (Caused issues. The recursive nature is still there as it has a child with the same type).
Updated jasn1 1.11.2 to asn1bean 1.12.0. If an asn field name is a Java reserved keyword, the field gets a trailing "_" but the getter remains normal. In JASN1Utils adjusted logic when looking for the getter.
Added support for inherited types. OctetStrings are converted to Strings instead of byte arrays.
Service takes care of the compilation of the ASN files. Test sources are generated and removed from source control.
NIFI-6752 Removed obsolete TODOs.
NIFI-6752 Updated nifi-asn1-nar version to 1.13.0-SNAPSHOT. Fixed checkstyle violations (unused imports).
NIFI-6752 ASN.1 reader - ASN.1 bundle requires 'include-asn1' profile to be active to be part of assembly.
NIFI-6752 ASN.1 reader - Updated ASN1.xml template.
NIFI-6752 ASN.1 reader - Updated versions.
NIFI-6752 ASN.1 reader - Update example generator. Updated ASN1.xml template. Updated (fixed) nifi-asn1-nar version in pom.xml.
NIFI-6752 ASN.1 reader - Added missing license for ASN1.xml.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4577
exceptionfactory
Mike Thomsen
Joe Witt
Peter Turcsanyi
Paul Grey
Nathan Gough
Peter Gyori
Bence Simon
Matthew Burgess
Mark Payne
Timea Barna
Tamas Palfy
noblenumbat360
Joe Gresock
Bryan Bende
hondawei
timeabarna
Denes Arvay
thenatog
simonbence
Mohammed Nadeem
Otto Fowler
tpalfy
Koji Kawamura
Christian Gumpert