379 Commits

Author SHA1 Message Date
Joe Witt
873d25585c
NIFI-9780 Merge branch 'NIFI-9780-RC3' 2022-04-04 08:35:23 -07:00
Nathan Gough
2a6e521549
NIFI-9702 ListSFTP will set mime.type when Record Writer configured
This closes #5891

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-03-28 16:03:00 -05:00
Peter Gyori
6ade47ac4f
NIFI-9838: Added Client Certificate attributes to ListenTCPRecord FlowFiles
This closes #5908

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-03-28 13:36:16 -05:00
Nathan Gough
2fbe10b4bc
NIFI-9701 - Corrected No Tracking strategy to create one flow file when using a Record Writer
- Corrected No Tracking strategy Record Writer handling for ListSFTP
- Updated temporary test files to have last modified time of epoch to avoid intermittent issue with Minimum Age filtering
- Refactored MockCacheService to separate reusable class

This closes #5885

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-03-28 12:47:30 -05:00
Joe Witt
0419dc2939
NIFI-9780-RC3 prepare for next development iteration 2022-03-21 11:58:27 -07:00
Joe Witt
b019a9191f
NIFI-9780-RC3 prepare release nifi-1.16.0-RC3 2022-03-21 11:58:09 -07:00
Nathan Gough
713e2fd03c
NIFI-9645 - Updated PutSplunk to allow idle connection timeouts
This closes #5841

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-03-14 11:15:16 -05:00
Bence Simon
b86466b4a5 NIFI-9669 Adding PutDynamoDBRecord processor
This closes #5761.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2022-03-09 16:20:33 +01:00
Mike Thomsen
ae1731c79e
NIFI-9134 nifi-metrics-reporting-bundle to use JUnit 5
- NIFI-9129 Refactored as much of nifi-kudu-bundle to use JUnit 5 as possible without breaking the integration tests which use test libraries from the Kudu project that depend on JUnit 4
- NIFI-9109 Refactored nifi-extension-utils to use JUnit 5. Left two tests with JUnit 4 because they used APIs that don't appear to have analogues in JUnit 5
- NIFI-9112 Refactored nifi-groovyx-bundle to use JUnit 5
- NIFI-9142 Refactored nifi-ranger-bundle to use JUnit 5
- NIFI-9136 Refactored nifi-network-bundle to use JUnit 5

This closes #5794

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-02-26 08:18:08 -06:00
Matthew Burgess
bcc8d03314
NIFI-9585 Upgraded H2 from 1.4 to 2.1.210
- Added nifi-h2-database module shading H2 1.4.200
- Implemented version checking and migration

This closes #5724

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-02-25 19:20:48 -06:00
Matthew Burgess
ab4cadc204
NIFI-9721: Support enum types in AvroTypeUtil.buildAvroSchema()
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5793.
2022-02-23 12:44:57 -05:00
Mark Payne
ce0122bd25
NIFI-9691: Added ForkEnrichment, JoinEnrichment processors
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5772.
2022-02-22 18:54:32 -05:00
Mark Payne
4d8c79d7f3 NIFI-9698: When creating an Avro schema, ensure that any default value is converted from what is returned by RecordField.getDefaultValue() to what Avro requires.
Signed-off-by: Chris Sampson <chriss@apache.org>

This closes #5776
2022-02-16 20:37:50 +00:00
exceptionfactory
fe1139b8bb
NIFI-9692 Upgraded Apache Commons Lang3 to 3.12.0
- Moved commons-lang3 version management to root Maven configuration
- Refactored limited usage of StringUtils is nifi-reporting-utils to remove commons-lang3
- Refactored limited usage of StringUtils in nifi-websocket-processors to remove commons-lang3

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5773
2022-02-16 09:28:21 -05:00
exceptionfactory
bab4309905
NIFI-9655 Add Queue Logging to ListenUDP (#5747)
* NIFI-9655 Added Queue Logging to ListenUDP

- Added TrackingLinkedBlockingQueue to track the largest queue size
- Updated AbstractListenEventProcessor to write debug log at most once per minute
- AbstractListenEventProcessor updates support both ListenUDP and ListenUDPRecord
2022-02-04 18:41:38 -05:00
exceptionfactory
dc7d9510cd NIFI-9610 Refactored nifi-processor-utils to separate modules
- Added nifi-bin-manager
- Added nifi-event-listen
- Added nifi-event-put
- Added nifi-listed-entity
- Added nifi-put-pattern
- Removed nifi-processor-utils

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5729.
2022-02-01 12:59:16 -05:00
Mark Payne
038811d0dd
NIFI-9629: Ensure that when we are setting default values on Avro GenericRecord objects that we convert from the schema's default value to the proper type
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5716.
2022-01-29 13:56:53 +01:00
Nathan Gough
8f2a9f94fe
NIFI-9632 - Removed nifi-lumberjack-bundle
- Removed several unused socket classes from nifi-processor-utils

This closes #5722

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-01-27 14:43:19 -06:00
Timea Barna
3ccc9d29b6
NIFI-9455 Added aggregated predictions to Prometheus Flow Metrics
This closes #5582

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-01-27 10:07:01 -06:00
exceptionfactory
e2243500d3
NIFI-9618 Upgraded Checkstyle to 9.2.1
- Corrected several missing return statements
- Increased Maven heap size to 3 GB for ci-workflow

This closes #5700

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-01-21 20:08:16 -06:00
Mark Payne
1d353435a5
NIFI-9594: When converting Record to Avro GenericRecord, ensure that any default values that are defined in the GenericRecord's schema get applied, regardless of whether or not the field exists in the associated RecordSchema.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5677.
2022-01-19 19:20:53 +01:00
Joe Witt
2f95eed8cb
NIFI-9575 This closes #5667. updating copyright year to 2022
Signed-off-by: Joe Witt <joewitt@apache.org>
2022-01-19 09:14:09 -07:00
exceptionfactory
405934dcd2
NIFI-9571 Corrected Session commit handling in PutTCP
- Added generic type to AbstractPutEventProcessor for compiler checking of event types
- Refactored createTransitUri to shared method in AbstractPutEventProcessor

Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5658.
2022-01-18 15:31:36 -05:00
exceptionfactory
7c50f1429e
NIFI-9563 Enabled ListenTCP Pool Receive Buffers property
- Removed deprecation from ListenTCP Pool Receive Buffers property
- Added BufferAllocator configuration property for NettyEventServerFactory

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5653.
2022-01-13 19:50:44 +01:00
Mark Payne
72e54f4fab NIFI-9390: Updates to MergeContent / MergeRecord so that they play nicely within Stateless
NIFI-9390: Addressed underlying condition in stateless framework that caused Merge-related processors and similar to not properly be triggered as necessary. Added several system tests to verify different configurations.

NIFI-9390: Simplified the logic for how to iterate over the components in a Stateless flow that are ready to be triggered

This closes #5634.

Co-authored-by: Peter Turcsanyi <turcsanyi@apache.org>
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2022-01-11 09:45:00 +01:00
Tamas Palfy
990285ba1c NIFI-8676 Added 'Tracking Entities' listing strategy to 'ListS3' and 'ListGCSBucket'
This closes #5413.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2022-01-10 20:35:04 +01:00
exceptionfactory
5832dff25e
NIFI-9556 Upgraded Apache HttpClient to 4.5.13
- Upgraded Apache HttpCore to 4.4.15
- Added dependency management declarations in root Maven configuration for HttpClient and HttpCore
- Removed version numbers from multiple modules

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5647.
2022-01-08 11:06:07 +01:00
Mark Payne
56593ad12f
NIFI-9546: Updated PutTCP and PutUDP with SupportsBatching annotations
- Removed incorrect usage of TriggerWhenEmpty
- Allow for 0 seconds of gracefully waiting for incoming threads on shutdown since we won't have any
- Updated unit tests to no longer have arbitrary sleep statements

This closes #5639

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-01-06 15:52:37 -06:00
exceptionfactory
12ba579b8f NIFI-9539 Upgraded Netty from 4.1.69 to 4.1.72
- Added Netty 4 BOM to root Maven configuration
- Removed individual Netty 4 dependency overrides
- Removed commented dependencies and unnecessary declarations from MiNiFi

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5635.
2022-01-05 18:11:03 -05:00
exceptionfactory
7c6bdcb035
NIFI-9491 Excluded commons-logging and added jcl-over-slf4j references
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5608
2021-12-17 07:21:03 -05:00
Nathan Gough
5e9c86885c
NIFI-9478 Moved Netty Log Exception Handler to end of pipeline
- Changed display name of Max Number of TCP Connections to Max Number of Worker Threads for ListenTCP
- Set Netty Socket Receive Buffer using Max Socket Buffer Size in ListenTCP

This closes #5599

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-12-15 16:59:07 -06:00
Mark Payne
97198e35a0
NIFI-9382: This closes #5584. Added system test that replicates issue in which a closed shared classloader causes issues when used again
NIFI-9382: Fixed issue with SharedInstanceClassLoader where the classloader may get closed but then get used again. When the SharedInstanceClassLoader is closed, we will now ensure that we don't use anymore and instead create a new one.

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-12-09 12:41:57 -07:00
Tamas Palfy
ff864266f5
NIFI-9436 - In AbstractPutHDFSRecord make sure the record writers use the FileSystem object the processor already has.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5565
2021-12-02 14:24:12 -05:00
markap14
839fbf7d19
NIFI-9382: Created a new ClassloaderIsolationKey mechanism by which H… (#5533)
* NIFI-9382: Created a new ClassloaderIsolationKey mechanism by which Hadoop related processors (and potentially others) can indicate that they need full classloaders to be cloned but can share with other instances in certain circumstances
- Added system tests

* NIFI-9382: Renamed interface based on review feedback

* NIFI-9382: Removed ReentrantKerberosUser.
2021-11-22 14:55:19 -05:00
exceptionfactory
0cf515c9c0 NIFI-9384 Corrected usage and generics in ListenTCP
- Addressed compiler warnings in ListenTCP and EventBatcher
- Adjusted ListenTCP property order to match previous version

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5526.
2021-11-17 11:54:51 -05:00
noblenumbat360
f410c8df0a
NIFI-8272 Delete stale metrics from REST API Prometheus endpoint.
Added <scope>test</scope> tag to the nifi-web-api pom.xml and corrected imports.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5447
2021-11-08 20:26:30 -05:00
Joe Witt
9e845d87ff
Merge branch 'NIFI-9344-RC3' 2021-11-07 16:23:10 -07:00
Mark Payne
705c65c86c
NIFI-9362: Ensure that we update the StateMap in AbstractListProcessor to hold any files whose date matches the latest before setting cluster-wide state
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5507
2021-11-04 13:46:41 -04:00
Joe Witt
c055895952
NIFI-9344-RC3 prepare for next development iteration 2021-11-03 08:53:37 -07:00
Joe Witt
7fdc07cccd
NIFI-9344-RC3 prepare release nifi-1.15.0-RC3 2021-11-03 08:53:32 -07:00
Bence Simon
75d1d7f6e7
NIFI-9265 Fixing path handling for HDFS processors when there are multiplied separators in the path
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5437.
2021-10-28 10:47:53 +02:00
exceptionfactory
60d6d469bf
NIFI-9342 Upgraded to Netty 3.10.6 and 4.1.69
- Replaced Netty 3.6.9 and 3.7.1 with 3.10.6
- Replaced Netty 4.1 with 4.1.69

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5490.
2021-10-28 09:34:31 +02:00
exceptionfactory
67ccdf6159
NIFI-9326 Added Socket Keep Alive property to ListenSyslog
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5479.
2021-10-26 19:44:56 +02:00
Mark Payne
1191e511a5
NIFI-9335: Updated AvroTypeUtil#createAvroRecord to ensure that if the given Avro Schema contains a field whose value is defaulted, the produced Avro Record has that value populated. Also added a unit test to verify behavior. Ran performance test (which is igonred and must be manually enabled) - about 2.1 seconds before the change and about 2.5 seconds after the change for each iteration.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5483
2021-10-26 12:15:54 -04:00
Nathan Gough
309ab8f4ab
NIFI-8792 - Modified ListenRELP to use Netty
- Refactored RELP encoders and decoders

This closes #5398

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-10-25 15:53:58 -05:00
Joe Gresock
d2f8f97b10
NIFI-9276: Adding config verification to AbstractListProcessor subclasses (#5453) 2021-10-21 12:22:50 -04:00
Bryan Bende
a380fb2964
NIFI-9297 Correct typo in documentation for listing strategy in AbstractListProcessor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5466.
2021-10-19 17:25:17 +02:00
exceptionfactory
4bcd03024a
NIFI-9283 Excluded Log4j 1.2 and upgraded Log4j2 to 2.14.1
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5440.
2021-10-06 11:17:36 -04:00
exceptionfactory
4943560521
NIFI-9223 Corrected ListenSyslog with default address of 0.0.0.0
- Refactored NettyEventServerFactory to accept nullable InetAddress
- Updated unit tests referencing NettyEventServerFactory

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5426.
2021-10-01 09:24:46 +02:00
Peter Gyori
38d26dafa3
NIFI-9192: ResultSetRecordSet considers value of useLogicalTypes flag when determining the object's schema.
NIFI-9192: ResultSetRecordSet consider value of useLogicalType flag when sqlType is not handled separately (default) and readerSchema is not null

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>

NIFI-9192: Unit tests added to test the use of useLogicalType flag

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>

NIFI-9192: ResultSetRecordSet consider not using logical type when sqlType is OTHER

NIFI-9192: Fixed checkstyle violations

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>

NIFI-9192: Renamed JdbcCommon.getUseLogicalTypes to isUseLogicalTypes

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>

NIFI-9192: Added unit tests for Array types with and without using logical types

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>

NIFI-9192: Review comments considered and unit tests refactored

NIFI-9192: Added further unit tests

NIFI-9192: Minor modifications based on review comments

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5388
2021-09-27 16:27:39 -04:00