Commit Graph

1358 Commits

Author SHA1 Message Date
Joe Witt 1a99b9611c
NIFI-10559 Clarified remote path can come from flowfile attributes and made inherited prop descriptors more obvious
This closes #6457

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-09-28 15:52:32 -05:00
Eduardo Fontes 9bc63920aa
NIFI-10289 This closes #6256. Change getConflictColumnsClause
NIFI-10289 Test upsert to catch this issue

Signed-off-by: Joe Witt <joewitt@apache.org>
2022-09-26 15:38:42 -07:00
Joe Witt 9e6ee656b5
NIFI-10532 ensuring client gets reset if any of the key values host/port/user/pw change on a per ff basis (#6445) 2022-09-26 17:00:23 -04:00
Bryan Bende 27e3ee1915 NIFI-10528 Create nifi-json-record-utils and updated Salesforce NAR dependencies to use it
This closes #6435.

Signed-off-by: Tamas Palfy <tpalfy@apache.org>
2022-09-23 17:11:13 +02:00
Emilio Setiadarma f586f8f4cf NIFI-9451 - Add 'Input Character Set' property for PutEmail and additional tests
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #6313.
2022-09-23 00:20:23 -04:00
Mark Payne ec09c56e93
NIFI-10508: When inferring data types for values, allow float and double to encapsulate byte/short/int/long values
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #6421
2022-09-16 17:58:19 -04:00
exceptionfactory 04c0df1723
NIFI-10471 Documented and applied standard deprecation logging
- Updated Admin Guide with section on Logging Configuration and Deprecation Logging
- Updated Developer Guide with section on Deprecating Components and Features
- Added Deprecation Logging when adding components marked with DeprecationNotice to the Flow Configuration
- Added Deprecation Logging on deprecated methods in standard Controller Service implementations
- Removed integration tests for deprecated PersistentProvenanceRepository
- Updated logging and added documentation on minor upgrade steps
- Updated logging for HDFSNarProvider and RocksDB Repository

This closes #6390
Signed-off-by: Paul Grey <greyp@apache.org>
2022-09-13 19:40:03 -04:00
exceptionfactory a91928e0e3
NIFI-10489 Upgraded zip4j from 2.10.0 to 2.11.2
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #6409.
2022-09-13 15:03:54 +02:00
ravinarayansingh 97b8f0e205
NIFI-10435 Masked sensitive arguments in ExecuteStreamCommand
This closes #6367

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-09-07 21:33:57 -05:00
krisztina-zsihovszki 822c4d473a NIFI-10404 TailFile processor persistent state not cleaned up
This closes #6349.

Signed-off-by: Tamas Palfy <tpalfy@apache.org>
2022-09-06 10:52:38 +02:00
Matthew Hawkins 64acd8b1c5
NIFI-2827 Adding zstd-jni to the necessary pom.xml. It's already in the LICENSE.
NIFI-2827 Update CompressContent.java to use zstd compression format
NIFI-2827 Update test cases for CompressContent.java to include zstd format
NIFI-2827 Update JsonRecordSetWriter.java to enable zstd compression format

This closes #6294

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2022-09-04 15:02:41 -04:00
exceptionfactory 4f87cd770b
NIFI-10410 Added InputRequirement annotation to ExtractGrok
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #6345.
2022-08-30 15:46:14 +02:00
Emilio Setiadarma f5fee4dda3
NIFI-10344 Refactored Groovy tests in standard-processor to Java and JUnit 5
This closes #6321

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-08-25 08:02:15 -05:00
exceptionfactory 8605d0a29f
NIFI-10336 Removed unnecessary lineage check from TestMonitorActivity
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #6285.
2022-08-10 15:34:44 +02:00
Emilio Setiadarma 5b9f096143
NIFI-10307 Refactored nifi-standard-processors tests to use JUnit 5
This closes #6272

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-08-08 12:35:19 -05:00
exceptionfactory fdf3925f81
NIFI-3964 Updated Grok Patterns to support TEXT and URL Resources
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #6260.
2022-08-08 14:48:21 +02:00
exceptionfactory 5cc2790891
NIFI-10302 Streamlined TestMonitorActivity to improve reliability
- Removed TestRunner.run() loops
- Reduced usage of Thread.sleep()
- Upgraded TestMonitorActivity to JUnit 5

Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes #6261.
2022-08-05 10:44:26 -04:00
exceptionfactory 2d73fd5631
NIFI-10318 Corrected HandleHttpRequest TLS client authentication
- Corrected setting of wantClientAuth instead of needClientAuth based on Client Authentication configuration

Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #6271.
2022-08-04 14:18:10 -04:00
Joe Witt 8c66bf948c
NIFI-10272-RC2 prepare for next development iteration 2022-07-27 13:24:09 -07:00
Joe Witt 8d256784d8
NIFI-10272-RC2 prepare release nifi-1.17.0-RC2 2022-07-27 13:24:06 -07:00
exceptionfactory 7767a5b85d
NIFI-10265 Route to Failure on Connect Exceptions in PutDatabaseRecord
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #6235
2022-07-25 09:36:54 -04:00
Eduardo Fontes c61efecbdc
NIFI-10252: Use correct field(s) for PutDatabaseRecord update keys
Not always UpdateKey comes in first position

Better readability

Include testUpdatePkNotFirst

Including usefull SQL log debug

Fix semicolon

NIFI-10252: Additional unit test for specified update keys
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #6226
2022-07-25 08:48:06 -04:00
NissimShiman 74b25c7306
NIFI-10154 ReplaceText AdminYielding on long line (#6151)
* NIFI-10154 ReplaceText AdminYielding on long line

* NIFI-10154 code review changes

Signed-off-by: Otto Fowler <otto@apache.org>
2022-07-22 11:14:23 -04:00
Nandor Soma Abonyi 302bdbfb11
NIFI-10260 Changed display name and description of Validate DTD properties
- Set Allow DTD as display name in EvaluateXPath and EvaluateXQuery

This closes #6230

Co-authored-by: David Handermann <exceptionfactory@apache.org>
Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-07-21 13:28:36 -05:00
Arpad Boda eab687ee50
NIFI-10255 Updated original relationship description on ExecuteStreamCommand
- Description of original relationship of ExecuteStreamCommand processor might be misleading

This closes #6229

Signed-off-by: David Handermann <exceptinofactory@apache.org>
2022-07-21 09:41:53 -05:00
Emilio Setiadarma cb0753b03c
NIFI-9758 Added Dynamic Properties to PutEmail
This closes #6204

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-07-15 10:58:31 -05:00
Paul Grey 274e4feeaa
NIFI-10143 Adjusted UTF-8 auto-detection in FTP Processors
This closes #6172

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-07-11 20:00:40 -05:00
Peter Turcsanyi 1685204cc1
NIFI-10206: Restored flushBuffer() call in HandleHttpResponse to avoid HTTP 404
This closes #6186

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-07-08 16:51:20 -05:00
exceptionfactory ea0e1b446e
NIFI-10187 Upgraded FtpServer from 1.1.1 to 1.2.0
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #6171.
2022-07-01 14:24:21 +02:00
Paul Grey 26a10d772f
NIFI-10186 Corrected PutFTP Remote Path Property EL
This closes #6173

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-06-30 19:12:28 -05:00
exceptionfactory 0458b6432d NIFI-10158 Corrected ListFTP expression support for Hostname and Port
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #6162.
2022-06-29 22:11:11 -04:00
exceptionfactory 7a47c8cfbd
NIFI-10162 Reorganized InvokeHTTP Properties
- Corrected handling of documentation for allowable values

NIFI-10162 Updated Multipart Form-Data Name description

This closes #6163.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2022-06-29 16:52:19 -04:00
Mark Payne 39c366eeef
NIFI-10169: When using the Insert Record Fields join strategy of JoinEnrichment, ensure that in order to combine schemas from the original record and the enrichment record we use incorporateSchema() so that even when the first enrichment record is null, we get the schema correct
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #6157.
2022-06-29 17:28:15 +02:00
Matthew Burgess 9d379121c4 NIFI-5005: Fix MergeRecord to honor writer's schema
This closes #6141.

Signed-off-by: Tamas Palfy <tpalfy@apache.org>
2022-06-27 21:53:20 +02:00
exceptionfactory d8ebfb25d0
NIFI-10161 Added Gzip Content-Encoding to InvokeHTTP and ListenHTTP
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #6150.
2022-06-26 22:23:32 +02:00
Juldrixx bc84532a8b
NIFI-9983 Add output relation name attribute in QueryRecord processor (#6011)
* NIFI-9983 Add output relation name attribute
2022-06-21 17:55:07 -04:00
greyp9 f9beb507fd
NIFI-9996 - ListFile does not list root directory when Recurse Subdirectories is active and Path Filter is specified (#6110) 2022-06-21 17:23:49 -04:00
tpalfy c08996515b
NIFI-10106 Avoid NPE in LookupRecord.createLookupCoordinates (#6111) 2022-06-21 17:03:44 -04:00
exceptionfactory 554f648f00
NIFI-9654 Added Queue Logging to ListenTCP
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #6116.
2022-06-14 15:47:38 +02:00
Emilio Setiadarma 58f93d1f8b
NIFI-10085 Removed CryptoUtility.isUnlimitedStrengthCryptoSupported() and references
This closes #6098

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-06-06 08:49:39 -05:00
exceptionfactory 4aa0d31d9f NIFI-10088 Set SSH Transport Timeout using SFTP Data Timeout property
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #6097.
2022-06-03 19:10:28 -04:00
exceptionfactory 1f9672c18b
NIFI-10075 Improved TestListSFTP configuration verification
- Changed assertVerificationSuccess() to separate testVerificationSuccessful() method
- Upgraded TestListSFTP to JUnit 5

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #6092.
2022-06-02 11:36:25 +02:00
exceptionfactory 45cc3cefe5 NIFI-10069 Updated multiple components to support Sensitive Dynamic Properties
- ExecuteScript
- ExecuteStreamCommand
- InvokeScriptedProcessor
- HikariCPConnectionPool

This closes #6085.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2022-06-01 11:26:41 -04:00
exceptionfactory 99be62be67
NIFI-9958 Add Framework Support for Sensitive Dynamic Properties (#6057)
* NIFI-9958 Implemented Sensitive Dynamic Properties

- Added SupportsSensitiveDynamicProperties annotation for components
- Added optional sensitive query parameter to Property Descriptor REST Resource methods
- Added system tests for components supporting sensitive dynamic properties
- Updated REST Resources to support Sensitive Dynamic Property Names
- Updated Documentation Writer to indicate component support for Sensitive Dynamic Properties
- Updated InvokeHTTP to support Sensitive Dynamic Properties
- Updated Auditor components to handle masking Sensitive Dynamic Properties

* Refactored Property Descriptor REST method handling

- Corrected AbstractDocumentationWriter evaluation of support for sensitive dynamic properties
- Refactored Controller Service Dynamic Properties system tests to new class

* Updated AbstractComponentNode.getProperties() to get canonical descriptor
2022-05-23 17:20:54 -04:00
exceptionfactory e0976f42d3 NIFI-3869 Added HTTP/2 support to ListenHTTP and HandleHttpRequest
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #6048.
2022-05-19 16:21:53 -04:00
Peter Gyori 3f16a41ca1
NIFI-10010: ListenTCP adds Certificate Subject and Issuer FlowFile attributes
This closes #6032

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-05-13 15:48:12 -05:00
exceptionfactory 30f7c1ba1e
NIFI-9990 This closes #6030. Improved FTP 550 file unavailable handling
- Improved File Not Found reply detection
- Added Permission Denied reply handling

Signed-off-by: Joe Witt <joewitt@apache.org>
2022-05-13 07:44:22 -07:00
Mike Thomsen 7271e8cea7
NIFI-7234 Standardized on Avro 1.11.0
NIFI-7234 Replaced Jackson 1.X references with Jackson 2.X references in various classes.
NIFI-7234 Added jackson-annotations to nifi-hdfs-processors.
NIFI-7234 Various updates to bring our test cases into better alignment with the Avro specification as of 1.11.
Fixed a checkstyle issue.

NIFI-7234 Made changes requested in a review.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5900
2022-04-29 17:48:59 -04:00
exceptionfactory ac2d7d1e54
NIFI-9968 Added null check before System.setProperty() in test methods
- Resolves build failures on Java 17 where the original user.timezone property returns null from System.getProperty()

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #6000
2022-04-27 14:38:33 -04:00
exceptionfactory 4b7f8cbe1f
NIFI-9949 This closes #5988. Corrected intermittent failures in TestPutSFTP
- Changed SSH server to start and stop after each method
- Replaced queued file with string FlowFile contents
- Refactored TestPutSFTP using JUnit 5

Signed-off-by: Joe Witt <joewitt@apache.org>
2022-04-21 12:23:07 -07:00
exceptionfactory b288810316
NIFI-9943 Added Transform Provider to nifi-xml-processing
- Refactored TransformerFactory references using StandardTransformerProvider

This closes #5986
Signed-off-by: Paul Grey <greyp@apache.org>
2022-04-21 15:14:54 -04:00
exceptionfactory 1e345db649
NIFI-9942 Removed load test from TestPutUDP
- Refactored TestPutUDP using JUnit 5
- Standardized test method naming

This closes #5984
Signed-off-by: Paul Grey <greyp@apache.org>
2022-04-21 09:45:00 -04:00
exceptionfactory e98b68cc3a
NIFI-9939 Upgraded SSHJ from 0.32.0 to 0.33.0
- Removed PatchedSFTPEngine with resolution of SFTP renaming in SSHJ 0.33.0

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5982.
2022-04-20 15:21:22 +02:00
Paul Grey 0a94762119
NIFI-9925 Adjust PutUDP and SNMP tests to run without preferring IPv4
This closes #5976

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-04-19 14:13:00 -05:00
exceptionfactory ecf0a93124
NIFI-9935 Upgraded Zip4j from 2.9.1 to 2.10.0
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5975.
2022-04-19 09:36:29 +02:00
Paul Grey 4310df0544
NIFI-9924 Corrected text encoding in PutEmail filenames and TestFTP
This closes #5967

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-04-14 14:58:22 -05:00
exceptionfactory 15f7590f7a
NIFI-9901 Added nifi-xml-processing to nifi-commons
- Refactored XML parsing to use providers from nifi-xml-processing
- Configured spotbugs-maven-plugin with findsecbugs-plugin in nifi-xml-processing
- Disabled Validate DTD in default configuration for EvaluateXPath and EvaluateXQuery
- Replaced configuration of DocumentBuilder and streaming XML Readers with shared components
- Removed XML utilities from nifi-security-utils
- Moved Commons Configuration classes to nifi-lookup-services

This closes #5962
Signed-off-by: Paul Grey <greyp@apache.org>
2022-04-13 19:56:57 -04:00
Mark Payne 05f3d7510f
NIFI-9903: This closes #5955. When using the 'success' relationship only for LookupRecord, lookup all records until a match is found, in order to determine the resultant schema. Refactored code to eliminate AbstractRouteRecord, because LookupRecord is the last processor that extended from it. Refactored code to use an inner interface to clean up code.
Signed-off-by: Joe Witt <joewitt@apache.org>
2022-04-13 13:49:26 -07:00
Joe Witt 873d25585c
NIFI-9780 Merge branch 'NIFI-9780-RC3' 2022-04-04 08:35:23 -07:00
exceptionfactory cf21bc47cd NIFI-9844 Refactored Encryptor tests using JUnit 5
- Refactored Keyed and Password Based Encryptor tests from Groovy to Java

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5913.
2022-03-29 17:53:13 -04:00
Nathan Gough 2a6e521549
NIFI-9702 ListSFTP will set mime.type when Record Writer configured
This closes #5891

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-03-28 16:03:00 -05:00
Peter Gyori 6ade47ac4f
NIFI-9838: Added Client Certificate attributes to ListenTCPRecord FlowFiles
This closes #5908

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-03-28 13:36:16 -05:00
Emilio Setiadarma 9566d3fa78
NIFI-9841 Improved Encryptor test reliability
- Changed expected exception to ProcessException in KeyedEncryptorGroovyTest and PasswordBasedEncryptorGroovyTest to avoid intermittent failures

This closes #5911

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-03-28 13:28:28 -05:00
Nathan Gough 2fbe10b4bc
NIFI-9701 - Corrected No Tracking strategy to create one flow file when using a Record Writer
- Corrected No Tracking strategy Record Writer handling for ListSFTP
- Updated temporary test files to have last modified time of epoch to avoid intermittent issue with Minimum Age filtering
- Refactored MockCacheService to separate reusable class

This closes #5885

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-03-28 12:47:30 -05:00
Mark Payne 63f02c99bf
NIFI-9835: Fixed threading bug in which NioAsyncLoadBalanceClient calls LoadBalanceSession.isComplete() followed by LoadBalanceSession.isCanceled() but it's possible for the complete flag to change before the canceled flag (they are not updated atomically). So changed to use a single LoadBalanceSessionState enum that represents the state. Also made the private StandardProcessSession.commit(boolean) method synchronized. When a processor is terminated (as is the case in Offload), we roll back sessions and both the commit() and rollback() need to be synchronized. Only the public commit() method was synchronized, and now with commitAsync() happening, we had the ability to commit without any synchronization. This addresses that concern. Also fixed a typo in docs for MergeRecord.
This closes #5902

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-03-25 12:42:03 -05:00
Marcus Ely 72435c3c6d
NIFI-8124 Added Cookie Strategy property to InvokeHTTP
This closes #5887

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-03-23 15:55:40 -05:00
Emilio Setiadarma 772adbc709
NIFI-1468 Added tests to handle invalid cipher streams missing Salt/IV
- Updated PasswordBasedEncryptorGroovyTest and KeyedEncryptorGroovyTest

This closes #5877

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-03-23 13:30:15 -05:00
Joe Witt 0419dc2939
NIFI-9780-RC3 prepare for next development iteration 2022-03-21 11:58:27 -07:00
Joe Witt b019a9191f
NIFI-9780-RC3 prepare release nifi-1.16.0-RC3 2022-03-21 11:58:09 -07:00
Matthew Burgess c84f438782
NIFI-9800: Unwrap SQLException in PutDatabaseRecord when table does not exist
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5871
2022-03-21 14:00:13 -04:00
Mike Thomsen 2bb1a06a8c
NIFI-9788 Updated commons-codec to 1.15 across all modules.
This closes #5881

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-03-21 09:13:05 -05:00
exceptionfactory 412e21be97
NIFI-9281 Enabled building on Java 17
- Added Ubuntu Zulu JDK 17 GitHub build
- Adjusted MiNiFi C2 FileSystemConfigurationCache test to avoid using environment variables
- Adjusted MiNiFi StatusLogger and StatusLoggerTest to avoid overriding private logger
- Adjusted failure reason attribute check in TestGetIgniteCache
- Adjusted TestRangerAuthorizer and TestRangerNiFiAuthorizer to avoid checking nested exceptions
- Adjusted encrypt-config TestUtil to avoid unnecessary comparison of different types
- Disabled Javascript tests on Java 15 and higher
- Disabled several Hive 3 tests on Java 17 for StringInternUtils illegal access
- Refactored nifi-enrich-processors tests to use Mockito without Powermock
- Refactored nifi-toolkit-tls tests to avoid illegal reflective access
- Removed deprecated X509Certificate test in CertificateUtilsTest
- Removed kryo serialization from nifi-site-to-site-client test
- Updated TestHashContent to use SHA-1 instead of SHA for hash algorithm
- Upgraded maven-war-plugin from 2.5 to 3.3.2
- Upgraded nifi-graph-bundle dependencies from Groovy 2.5.14 to 3.0.8
- Upgraded QuestDB from 4.2.1 to 6.2.1 in nifi-framework-core

This closes #5870

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2022-03-17 14:08:38 -04:00
Mike Thomsen df00cc6cb5
NIFI-6047 Cleaned up code to allow tests to run against 1.13.0-snapshot
Removed DMC.
NIFI-6047 Started integrating changes from NIFI-6014.
NIFI-6047 Added DMC tests.
NIFI-6047 Added cache identifier recordpath test.
NIFI-6047 Added additional details.
NIFI-6047 Removed old additional details.
NIFI-6047 made some changes requested in a follow up review.
NIFI-6047 latest.
NIFI-6047 Finished updates
First round of code review cleanup
Latest
Removed EL from the dynamic properties.
Finished code review requested refactoring.
Checkstyle fix.
Removed a Java 11 API
NIFI-6047 Renamed processor to DeduplicateRecord

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4646
2022-03-09 19:06:44 -05:00
Adam 23132fb89f
NIFI-6047
Added NiFi DetectDuplicateRecord standard processor.
Adding some documentation and PR review tweaks.
Exposing processor
Documentation updates, exception handling consolidation, added support for record path field variables.
Added tests.
Build bump.
Migrated cache service to groovy folder.
Moved declarations for properties to @BeforeClass lifecycle method.
Adding some documentation and PR review tweaks.
Documentation updates, exception handling consolidation, added support for record path field variables.
Added tests.
Build bump.
Migrated cache service to groovy folder.
Fixed variable type bug.
Fixed mapping of test params to usage.
Fixed potential illegal state exception bug.
2022-03-09 19:06:08 -05:00
Mark Payne 6a1c7c72d5
NIFI-9781: Fix handling when selecting array element via QueryRecord
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5853.
2022-03-09 13:07:15 -05:00
Mark Payne 65dd62716a
NIFI-9748: Added new property for Output Format to LogAttribute. Also made the FlowFile Properties (file size, entry date, lineage start date) optional and renamed from 'Standard FlowFile Attributes' to 'FlowFile Properties' because this has led to confusion many times in the past, around users wanting to reference these things as attributes via EL but they are not actually attributes.
This closes #5825

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2022-03-04 08:24:39 -05:00
exceptionfactory ddecfa201b
NIFI-9736 Improved TestRouteText to avoid intermittent failures (#5814)
- Refactored testPatternCache to avoid testing maximum cache size reached
2022-02-28 14:26:19 -05:00
exceptionfactory 1eede30974
NIFI-9732 Upgraded Zip4j from 2.8.0 to 2.9.1
This closes #5811

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2022-02-27 06:09:14 -05:00
markobean 4e3871fec7
NIFI-9072: improvements to ValidateXML including validate XML in attr… (#5324)
NIFI-9072: Improvements to ValidateXML including validate XML in attributes and non-schema, well-formed XML only validation, and additional documentation
- Updated description and added System Resource Consideration to discourage using large XML documents in attributes
- Migrate from XML DocumentBuilderFactory to SafeXMLConfiguration
2022-02-25 14:57:04 -05:00
Mark Payne 39483b9c12
NIFI-9716, NIFI-9577: Addressed issue in the PathFilter for GetFile / ListFile. For any file that is found in the Input Directory directly, it was previously being listed/fetched even if it didn't match the PathFilter. Additionally, updated the code to create a new File FIlter for every invocation of onTrigger. This was necessary for NIFI-9577 because the directory to monitor supports Expression Language and as a result may change from invocation to invocation, if using a function such as now() but the PathFilter would always relativize the path based on the value that was obtained when the processor was scheduled.
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5788.
2022-02-24 09:13:16 -05:00
Mark Payne ce0122bd25
NIFI-9691: Added ForkEnrichment, JoinEnrichment processors
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5772.
2022-02-22 18:54:32 -05:00
levilentz dfbf2e3cea
NIFI-9286: JOLT Expression Language
Fixes NIFI-6213 and adds in functionality to use expression language in class and module specification
NIFI-9286: adding JOLT unit tests
NIFI-9286: addressing PR feedback
Fixes a problem with the scope of the EL for module directory
NIFI-9286: alignment of JOLT processors
NIFI-9286: fix checkstyle

This closes #5444

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2022-02-19 14:11:44 -05:00
Matthew Burgess 03165ad817 NIFI-9607: Honor Update Keys when Quoting Identifiers in PutDatabaseRecord
NIFI-9607: Fixed wrong column name in WHERE clause for generateUpdate

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5701.
2022-02-16 16:07:34 -05:00
Gabor Gyimesi 6788076536
NIFI-9058 Corrected AttributesToJSON Core Attributes filtering
- Avoiding filtering Core Attributes from configured Attributes List

This closes #5312

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-02-15 15:51:39 -06:00
exceptionfactory e584d3cf04
NIFI-9660 Upgraded Apache Tika to 2.3.0
- Upgraded tika-core from 1.27 to 2.3.0
- Upgraded tika-parsers to tika-parsers-standard-package in nifi-media-processors
- Updated Tika metadata property references

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5754.
2022-02-09 11:11:46 +01:00
Noel Baek bbc78f1547
NIFI-9621: Added Ignore Reserved Characters to FlattenJson
- Upgraded to json-flattener from 0.12.0 to 0.13.0

This closes #5704

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-02-08 21:02:55 -06:00
exceptionfactory ce66cf41e2
NIFI-6699 Corrected SFTP symbolic link handling (#5744)
- Added SFTP stat() request on symbolic links to check directory status
- Refactored and renamed TestServerSFTPTransfer class
2022-02-04 18:43:42 -05:00
exceptionfactory ba0bf2ede0
NIFI-9650 Upgraded OkHttp from 4.9.2 to 4.9.3
- Added okhttp-bom to root Maven configuration
- Removed repeated okhttp.version property from specific dependencies

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5737.
2022-02-02 11:45:18 +01:00
exceptionfactory 43748a5523 NIFI-9638 Refactored Google Guava references
- Refactored nifi-framework and nifi-standard modules
- Replaced Google Cache with Caffeine Cache
- Replaced Google collections classes with standard Java collections

This closes #5730.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2022-02-01 15:17:50 -05:00
exceptionfactory dc7d9510cd NIFI-9610 Refactored nifi-processor-utils to separate modules
- Added nifi-bin-manager
- Added nifi-event-listen
- Added nifi-event-put
- Added nifi-listed-entity
- Added nifi-put-pattern
- Removed nifi-processor-utils

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5729.
2022-02-01 12:59:16 -05:00
Bence Simon 49978cdd91
NIFI-9341 Added CEF RecordReader
This closes #5530

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-01-28 12:42:51 -06:00
exceptionfactory 95abc35463 NIFI-6266 Corrected proxy FTP connect handling
- Added ProxyFTPClient with connect method supporting unresolved hosts
- Added FTPClientProvider with implementation handling configuration and connection
- Added support for SOCKS with authentication in FTP components
- Refactored FTPTransfer using FTPClientProvider

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5655.
2022-01-27 17:36:27 -05:00
Nathan Gough 08153b8260
NIFI-9453 Refactored ListenBeats using Netty
This closes #5669

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-01-26 14:40:32 -06:00
Peter Turcsanyi 96b53d8943
NIFI-9589: Support initial loading from the current max values in QueryDatabaseTable* processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5676.
2022-01-25 18:12:14 +01:00
Tamas Palfy aa61494fc3
NIFI-9065 Add support for OAuth2AccessTokenProvider in InvokeHTTP
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5319.
2022-01-19 13:29:11 +01:00
exceptionfactory 405934dcd2
NIFI-9571 Corrected Session commit handling in PutTCP
- Added generic type to AbstractPutEventProcessor for compiler checking of event types
- Refactored createTransitUri to shared method in AbstractPutEventProcessor

Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5658.
2022-01-18 15:31:36 -05:00
Pierre Villard 4847926a4b
NIFI-9544 - LookupRecord - fixed behavior when no matching value in the LRS
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5638.
2022-01-13 16:13:24 -05:00
exceptionfactory 7c50f1429e
NIFI-9563 Enabled ListenTCP Pool Receive Buffers property
- Removed deprecation from ListenTCP Pool Receive Buffers property
- Added BufferAllocator configuration property for NettyEventServerFactory

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5653.
2022-01-13 19:50:44 +01:00
Pierre Villard d1599eebcf
NIFI-9545 - Fix in-place replacement for LookupRecord processor
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5637.
2022-01-13 10:59:02 -05:00
exceptionfactory 64495e99e9
NIFI-7835 Added authenticated SOCKS proxy support for SFTP
NIFI-7749 Added authenticated HTTP proxy support for SFTP

- Added StandardSocketFactoryProvider to return SocketFactory based on credentials

Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5624.
2022-01-12 10:50:01 -05:00
Mark Payne 72e54f4fab NIFI-9390: Updates to MergeContent / MergeRecord so that they play nicely within Stateless
NIFI-9390: Addressed underlying condition in stateless framework that caused Merge-related processors and similar to not properly be triggered as necessary. Added several system tests to verify different configurations.

NIFI-9390: Simplified the logic for how to iterate over the components in a Stateless flow that are ready to be triggered

This closes #5634.

Co-authored-by: Peter Turcsanyi <turcsanyi@apache.org>
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2022-01-11 09:45:00 +01:00
exceptionfactory 6039095625
NIFI-9267 Replaced nifi-standard-web-test-utils with mockwebserver
- Replaced instances of custom Jetty TestServer with OkHttp MockWebServer
- Removed integration tests referencing nifi-standard-web-test-utils SslContextUtils
- Removed nifi-standard-web-test-utils

Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5620.
2022-01-09 05:25:32 -05:00
exceptionfactory ca8bc17c0a
NIFI-7089 Upgraded Apache SSHD from 1.7.0 to 2.8.0
- Replaced fake-sftp-server-rule with test Apache SSHD Server

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5646.
2022-01-08 11:02:12 +01:00
markobean 42626adab8
NIFI-8962: Add overflow strategy to DistributeLoad (#5267) 2022-01-07 14:45:18 -05:00
Joe Gresock 287e41d8a6
NIFI-9547: Allowing more datatypes in LookupRecord value substitution
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5640.
2022-01-07 12:30:13 +01:00
Mark Payne 56593ad12f
NIFI-9546: Updated PutTCP and PutUDP with SupportsBatching annotations
- Removed incorrect usage of TriggerWhenEmpty
- Allow for 0 seconds of gracefully waiting for incoming threads on shutdown since we won't have any
- Updated unit tests to no longer have arbitrary sleep statements

This closes #5639

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-01-06 15:52:37 -06:00
Peter Turcsanyi 4346dd8faf
NIFI-9391: Modified MergeRecord to process FlowFiles within a loop in… (#5550)
* NIFI-9391: Modified MergeRecord to process FlowFiles within a loop in a single onTrigger

MergeRecord processed the FlowFiles in multiple onTrigger-s and it needed an extra onTrigger call
(with no incoming FFs) to realize that no more FFs are available and it is time to send the merged FF downstream.
It was not compatible with Stateless Runtime which does not trigger the flow any more if no FFs available.

Also changed "unschedule" logic in StandardProcessorTestRunner: @OnUnscheduled methods were called immediately after
the 1st FlowFile was processed. Unschedule the processor only at the end of the execution (onTrigger finished)
and only if stopOnFinish has been requested by the test case.
2022-01-04 11:36:56 -05:00
Vibhath Ileperuma 0ac8f1b32c
NIFI-8605 Adding a new property for ExecuteSQL and ExecuteSQLRecord processors to enable/disable auto committing
change the default value of auto commit function to true
Changed the auto commit property name and add more details in the description
If the auto commit is set to false, commit() is called for consistency
adds unit tests
Fix the check style issue of having more than 200 characters in single line

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5554
2021-12-30 18:35:50 -05:00
exceptionfactory 9747d6a410
NIFI-9509 This closes #5618. Added PatchedSFTPEngine to correct SFTP rename flags handling
- SSHJ 0.32.0 introduced support for rename flags in SFTP commands without checking the protocol version
- PatchedSFTPEngine overrides the rename method to check the SFTP protocol version

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-12-21 12:14:23 -07:00
exceptionfactory 4f28b28ba2
NIFI-9507 This closes #5617. Corrected SSH Client handling on connect failures
- Refactored SSH Client configuration and connection to SSHClientProvider
- Implemented exception handling for configuration and connection failures
- Named SSH keep-alive thread for improved runtime tracking
- Closed SSH Client and interrupted keep-alive thread on configuration failures
- Added missing Compression Property to ListSFTP
- Corrected Hostname and Port property descriptors in ListSFTP

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-12-20 20:23:31 -07:00
exceptionfactory 7c6bdcb035
NIFI-9491 Excluded commons-logging and added jcl-over-slf4j references
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5608
2021-12-17 07:21:03 -05:00
Nathan Gough 5e9c86885c
NIFI-9478 Moved Netty Log Exception Handler to end of pipeline
- Changed display name of Max Number of TCP Connections to Max Number of Worker Threads for ListenTCP
- Set Netty Socket Receive Buffer using Max Socket Buffer Size in ListenTCP

This closes #5599

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-12-15 16:59:07 -06:00
Pierre Villard 90930ca197
NIFI-9476 - Fix QueryRecord when no result and with array type column
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5594
2021-12-15 17:37:24 -05:00
Pierre Villard e6b573e79e
NIFI-9419 ParseCEF - Upgraded parcefone and supported empty extensions
- Upgraded com.fluenda:parcefone from 2.0.0 to 2.1.0
- Added Accept empty extensions property to ParseCEF

This closes #5555

Co-authored-by: David Handermann <exceptionfactory@apache.org>
Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-11-30 11:27:50 -06:00
Paul Grey d6b6dd1eb4
NIFI-9389 - NPE guard for ExecuteProcess when not triggered
- Added support for TestRunner.run() with zero operations

This closes #5544

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-11-29 09:27:04 -06:00
exceptionfactory c033debdf3
NIFI-9399 This closes #5542. Apply Secure Processing to TransformXml XSLT
- Added XML Stream Reader processing for XSLT with external entities disabled
- Removed unused XsltValidator
- Upgraded Saxon-HE from 9.6.0-5 to 10.6

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-11-22 09:18:57 -07:00
exceptionfactory 0cf515c9c0 NIFI-9384 Corrected usage and generics in ListenTCP
- Addressed compiler warnings in ListenTCP and EventBatcher
- Adjusted ListenTCP property order to match previous version

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5526.
2021-11-17 11:54:51 -05:00
Nathan Gough f7466fd0f1
NIFI-8806 Refactored ListenTCP using Netty
- Deprecated Max Number of Receiving Message Handler Threads property
- Deprecated Pool Receive Buffers property
- Updated TestListenTCP using Netty EventSender

This closes #5493

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-11-16 11:27:38 -06:00
Nandor Soma Abonyi 6fd1f03bd6
NIFI-830 Added FlowFile Naming Strategy to InvokeHTTP
This closes #5475

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-11-15 07:46:37 -06:00
Roberto Santos e906c1d70f
NIFI-9194: Upsert for Oracle12+
Fixes pr #5366.

Fixes pr #5366. Replace tabchars fot whitespaces.

Fixes pr #5366. Replaced tabchars for whitespaces. Removed unnecessary comments.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5366
2021-11-08 20:45:25 -05:00
Joe Witt c055895952
NIFI-9344-RC3 prepare for next development iteration 2021-11-03 08:53:37 -07:00
Joe Witt 7fdc07cccd
NIFI-9344-RC3 prepare release nifi-1.15.0-RC3 2021-11-03 08:53:32 -07:00
Pierre Villard decfad394d
NIFI-9349 - add attribute with request duration in InvokeHTTP (#5501)
This closes #9349

Signed-off-by: Otto Fowler <ottobackwards@gmail.com>
2021-11-03 09:25:57 -04:00
exceptionfactory 16e6045d13 NIFI-9346 Added closing of EventSender to TestListenRELP
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5492.
2021-10-28 13:10:04 -04:00
exceptionfactory 67ccdf6159
NIFI-9326 Added Socket Keep Alive property to ListenSyslog
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5479.
2021-10-26 19:44:56 +02:00
Lehel d7e41e2005
NIFI-9328: Transfer cleanup and reuse added to FetchFileTransfer in case of FileNotFound and PermissionDenied exceptions.
- Corrects connection handling in FetchFTP and FetchSFTP

This closes #5478

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-10-26 11:17:33 -05:00
Nathan Gough 309ab8f4ab
NIFI-8792 - Modified ListenRELP to use Netty
- Refactored RELP encoders and decoders

This closes #5398

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-10-25 15:53:58 -05:00
Pierre Villard 8fe7f372d6
NIFI-9329 - Expose event validation in ParseCEF processor (#5477) 2021-10-25 14:37:39 +02:00
Joe Gresock d2f8f97b10
NIFI-9276: Adding config verification to AbstractListProcessor subclasses (#5453) 2021-10-21 12:22:50 -04:00
Lehel 207894ebe0 NIFI-9277: Add Record Reader and Writer to ListenHTTP.
This closes #5446.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-10-15 18:27:53 +02:00
Lehel Boér f7df326521 NIFI-9183: Add a command-line option to save status history
Fixed typo error.

Fixed error.

Fixed review items.

Fix contrib-check. Added missing test file to excludes.

Fix review items.

Fix review items: error handling, input validation, added more unit tests.

Improved status history repository creation in HeadlessNiFiServer.
2021-10-12 16:35:20 +02:00
Pierre Villard 9588758aac
NIFI-9231 - Add support for custom extensions in ParseCEF
This closes #5451

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-10-11 10:40:05 -05:00
exceptionfactory 70432507df
NIFI-6617 Refactored Encrypted Repository configuration
- Updated documentation with new properties
- Refactored cipher operations to common RepositoryEncryptor classes
- Abstracted record metadata serialization for better compatibility

Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5407.
2021-10-08 11:57:10 -04:00
Mark Payne 607480c1a0
NIFI-9239: Updated Consume/Publish Kafka processors to support Exactly Once Semantics. Added ExecuteStateless processor/bundle
- A few minor updates to Stateless in order to surface some concepts from the stateless engine up to the caller, such as bulletins, counters, etc.

Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5412.
2021-10-04 15:32:21 -04:00
exceptionfactory 4943560521
NIFI-9223 Corrected ListenSyslog with default address of 0.0.0.0
- Refactored NettyEventServerFactory to accept nullable InetAddress
- Updated unit tests referencing NettyEventServerFactory

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5426.
2021-10-01 09:24:46 +02:00
Peter Turcsanyi 1379674240
NIFI-9255: Support asynchronous session commits in ListenHTTP (#5419)
NIFI-9255: Support asynchronous session commits in ListenHTTP
2021-09-29 18:16:47 -04:00
Peter Gyori 38d26dafa3
NIFI-9192: ResultSetRecordSet considers value of useLogicalTypes flag when determining the object's schema.
NIFI-9192: ResultSetRecordSet consider value of useLogicalType flag when sqlType is not handled separately (default) and readerSchema is not null

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>

NIFI-9192: Unit tests added to test the use of useLogicalType flag

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>

NIFI-9192: ResultSetRecordSet consider not using logical type when sqlType is OTHER

NIFI-9192: Fixed checkstyle violations

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>

NIFI-9192: Renamed JdbcCommon.getUseLogicalTypes to isUseLogicalTypes

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>

NIFI-9192: Added unit tests for Array types with and without using logical types

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>

NIFI-9192: Review comments considered and unit tests refactored

NIFI-9192: Added further unit tests

NIFI-9192: Minor modifications based on review comments

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5388
2021-09-27 16:27:39 -04:00
exceptionfactory b4e0416b5e
NIFI-9228 Refactored tests using TemporaryKeyStoreBuilder
- Added TemporaryKeyStoreBuilder with File.deleteOnExit() for KeyStore and TrustStore files
- Removed JKS files from nifi-security-utils tests
- Refactored usage of KeyStoreUtils.createKeyStoreAndGetX509Certificate() to TemporaryKeyStoreBuilder
- Removed unnecesary hadoop-minikdc test dependency in security-utils
- Replaced Mini KDC Hex utility with Bouncy Castle Hex utility in unit tests

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5406
2021-09-21 16:07:13 -04:00
exceptionfactory ba775d28de
NIFI-9228 Refactored tests to use generated KeyStores
- Refactored multiple tests using KeyStoreUtils
- Removed static KeyStore and TrustStore files

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5401
2021-09-21 11:06:19 -04:00
Gabor Gyimesi 1fdae088cf
NIFI-4692 Fix AttributesToJSON property docs to match functionality
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5313.
2021-09-18 15:13:24 +02:00
Paul Grey 1ab47ce931
NIFI-9212 Add timeout and quiet period for Netty shutdown
This closes #5386

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-09-16 09:38:49 -05:00
exceptionfactory f5a421ccb1
NIFI-9067 Upgraded JAXB Runtime from 2.3.0 to 2.3.5
- Replaced old com.sun.xml.bind:jaxb-impl and jaxb-core with current org.glassfish.jaxb:jaxb-runtime
- Replaced old javax.xml.bind:jaxb-api with current jakarta.xml.bind-api
- Removed unnecessary dependency references to javax.activation-api

This closes #5320

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2021-09-07 13:27:53 -04:00
exceptionfactory 20804ff2b1
NIFI-8749 Removed implicit time zone conversion to GMT
- Updated DataTypeUtils.getDateFormat() to use system default time zone
- Updated Record Path Guide to match Expression Language Guide regarding default time zone
- Updated impacted unit tests to expect localized dates instead of dates converted to GMT

NIFI-8749 Set user.timezone in TestRecordPath to avoid unexpected date conversions

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5361
2021-08-31 16:07:53 -04:00
Matthew Burgess dfa6592a47
NIFI-8749 Revert "Removed implicit time zone conversion to GMT"
This reverts commit 758e1850ed.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>
2021-08-31 13:33:53 -04:00
exceptionfactory 758e1850ed
NIFI-8749 Removed implicit time zone conversion to GMT
- Updated DataTypeUtils.getDateFormat() to use system default time zone
- Updated Record Path Guide to match Expression Language Guide regarding default time zone
- Updated impacted unit tests to expect localized dates instead of dates converted to GMT

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5305
2021-08-31 10:40:57 -04:00
Paul Grey 0a827d6cd7 NIFI-9025 Fixed reference in TestTailFile unit test
This closes #5294

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-08-07 15:40:26 -07:00
Mark Payne bf52973d62
NIFI-8773: Implemented Line Start Pattern in TailFile
Each message encountered in the tailed file will be buffered (up to some configurable max) until the subsequent message arrives. At that point, the previous message will be flushed.

This closes #5251

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-08-06 15:52:55 -05:00
Pierre Villard 6139c51063
NIFI-8951 - fixed scale vs precision reference in QueryDatabaseTable processors
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5256
2021-08-04 09:23:35 -04:00
exceptionfactory 3621a2182c
NIFI-8424 Removed SeeAlso references to service interfaces
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5269.
2021-08-02 20:20:11 +02:00
exceptionfactory f539d9886e
NIFI-8956 Changed TestListenHTTP to expect IOException
- Replaced expected SSLException with IOException in assertThrows
- Removed conditional override of jdk.tls.disabledAlgorithms in test SslContextUtils

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5260.
2021-07-29 09:44:43 +02:00
exceptionfactory cb020072f7 NIFI-8770 Use queue drainTo() on shutdown in HandleHttpRequest
- Refactored response handling to use shared sendError() method
- Standardized request logging to include HTTP Method and URI

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5218.
2021-07-20 13:21:48 -04:00
exceptionfactory ebdc5eb92b
NIFI-7443 Corrected SFTP Keep Alive behavior
- Configured SSHJ Keep Alive Interval of 5 seconds
- Updated Send Keep Alive On Timeout property description

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5223.
2021-07-19 20:26:45 +02:00
exceptionfactory 00f385b51b
NIFI-8789 Corrected TestListenUDP to use getAvailableUdpPort()
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5219.
2021-07-16 15:13:05 +02:00
Joe Witt 7356332852
Merge branch 'NIFI-8767-RC2' 2021-07-14 09:07:22 -05:00
exceptionfactory 05d7867a16
NIFI-8768 Added toLocalDate() for convertType() handling of DATE fields
- Updated PutKudu to use DataTypeUtils.toLocalDate() for DATE fields
- Updated PutDatabaseRecord to remove convertDateToLocalTZ() since convertType() uses toLocalDate()
- Updated PutElasticsearchHttpRecord to use default time zone format for DATE fields
- Updated WriteXMLResult to use default time zone format for DATE fields
- Updated WriteJsonResult to use default time zone format for DATE fields
- Updated AvroTypeUtil to use toLocalDate() for logical DATE fields
- Updated JdbcCommon to avoid conversion to UTC for logical DATE fields
- Updated Processor and RecordReader unit tests for consistency in DATE comparison

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5210
2021-07-13 16:17:36 -04:00
Joe Witt 97feacc181
NIFI-8767-RC2 prepare for next development iteration 2021-07-10 12:17:09 -07:00
Joe Witt fcbf1d5f97
NIFI-8767-RC2 prepare release nifi-1.14.0-RC2 2021-07-10 12:17:05 -07:00
exceptionfactory 75de68e013
NIFI-8764 Refactored UnpackContent to use both Commons Compress and Zip4j
- UnpackContent uses Zip4j when configured with a password property
- UnpackContent uses Commons Compress when a password is not specified

NIFI-8764 Updated Password property description mentioning disabled algorithms

NIFI-8764 Adjusted Password property description

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5201
2021-07-09 13:15:44 -04:00
Nathan Gough 9a125dcaaf
NIFI-8616 Refactored PutTCP, PutUDP, and PutSplunk to use Netty nifi-event-transport classes
This closes #5182

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-07-07 16:24:49 -05:00
Matthew Burgess e16016b4ab
NIFI-8530: Improved DELETE handling in PutDatabaseRecord for non-nullable columns
This closes #5173

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-07-07 14:51:40 -05:00
exceptionfactory 6c7c3b3c1b
NIFI-8711 Added SFTP support for keyboard-interactive authentication
- Included debug logging of configured Authentication Methods

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5166.
2021-06-30 14:03:28 +02:00
exceptionfactory 786f17182b
NIFI-8630 Upgraded javax.mail 1.4.7 to jakarta.mail 2.0.1 for PutEmail
- Added jakarata.activation 2.0.1 to support jakarta.mail 2.0.1
- Clarified description of SMTP TLS property enabling STARTTLS

NIFI-8630 Added final to several variables and adjusted variable declaration

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5179.
2021-06-25 19:18:16 +02:00
Jon Kessler 115bba9ab0
NIFI-8523 Added SFTP algorithm and cipher properties
- Updated secure FTP processors to configure which algorithms, ciphers and message authentication codes are allowed to be used by the SSH Client
- Included Expression Language Variable Registry support for properties

This closes #5061

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-06-24 11:51:04 -05:00
Mark Payne 6fc9f33ba2
NIFI-8720: Ensure that when we use Expression Language for Records Per Split property of SplitRecord, we treat the result as a value of 1 if it's less than 1.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5171
2021-06-21 13:09:20 -04:00
exceptionfactory ab851b9cde NIFI-8672 Added syncUninterruptibly() to Netty shutdown and close methods
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5139.
2021-06-12 17:14:34 -04:00
Mark Payne c9dee30294
NIFI-8658: Allow Filter Functions and expressions to be specified as a RecordPaths
NIFI-8658: Addressed issue where the RecordField that was provided from Function Filters were not accurate

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5125
2021-06-04 17:17:33 -04:00
exceptionfactory 7fb0293c3e NIFI-8626 Upgraded okhttp-digest to 2.5
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5096.
2021-05-27 12:41:11 -04:00
exceptionfactory 1e6161c0aa
NIFI-8361 Upgraded Zip4j to 2.8.0
- Upgrade resolves issue unpacking Zip files with temporary spanning markers

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5103.
2021-05-26 20:21:11 +02:00
exceptionfactory a3365c8833 NIFI-8462 Refactored PutSyslog and ListenSyslog using Netty
- Added nifi-event-transport module encapsulating Netty classes
- Refactored unit tests for PutSyslog and ListenSyslog
- Removed integration tests for PutSyslog and ListenSyslog

NIFI-8462 Added context.yield() in PutSyslog when no FlowFiles and addressed other issues

NIFI-8462 Removed unused import of ExpressionLanguageScope

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5044.
2021-05-25 10:54:15 -04:00
Tamas Palfy 5108d7cdd0 NIFI-8439 Update parquet-avro to allow reading parquet INT96 timestamps as byte arrays (instead of throwing an exception).
Also allow to write them as such (byte-arrays) - again, instead of throwing an exception.

NIFI-8439 Fixed unit tests.
NIFI-8439 Allow writing parquet INT96 timestamps if they were read by the same parquet-avro library.

This closes #5006.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-05-25 13:18:46 +02:00
Mohammed Nadeem c113960b81
NIFI-8613: Improve FlattenJson Processor
- Unflattening a flattened json
- Preserving primitive arrays such as strings, numbers, booleans and null in a nested json
- Logging errors when failure
- Pretty printing resulted json

This closes #5083

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-05-20 15:04:23 -05:00
Mark Payne ecacfdaa4c NIFI-8469: Introduced ProcessSession.commitAsync and updated processors to use it. Deprecated ProcessSession.commit()
- Updated Mock Framework to now fail tests that use ProcessSession.commit() unless they first call TestRunner.setAllowSynchronousSessionCommits(true)
- Updated stateless nifi in order to make use of async session commits
- Fixed bug that caused stateless to not properly handle Additional Classpath URLs and bug that caused warnings about validation to get generated when a flow that used controller services was initialized. While this is not really in scope of NIFI-8469, it was found when testing and blocked further progress so addresssed here.
- If Processor fails to progress when run from stateless, trigger from start of flow until that is no longer the case
- Introduced notion of TransactionThresholds that can limit the amount of data that a flow will bring in for a given invocation of stateless dataflow
- Several new system-level tests
2021-05-19 08:51:16 -04:00
Matthew Burgess f812dfdfc0
NIFI-8535: Better error message in PutDatabaseRecord when table does not exist (#5070)
* NIFI-8535: Better error message in PutDatabaseRecord when table does not exist
2021-05-14 15:13:00 -04:00
exceptionfactory 340b13033f NIFI-8367 Removed joda-time from nifi-standard-processors 2021-05-11 14:50:49 -04:00
matthewlannan 86d5b08d23 Adding addititonal documentation to EvalluateJsonPath per https://issues.apache.org/jira/browse/NIFI-8518 2021-05-11 14:40:12 -04:00
Matthew Burgess 30fc26647e
NIFI-6061: Fix CLOB/BLOB handling in PutDatabaseRecord
NIFI-6061: Force getBytes() in BLOB handling to use UTF-8 charset

NIFI-6061: Use setClob(), added unit tests, incorporated review comments

This closes #5049

Co-authored-by: zhangcheng <zhangchengk@foxmail.com>
Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-05-04 14:58:14 -05:00
Paul Grey cc554a6b11
NIFI-8478 Addressed i18n test issues; CI supplies locale to surefire
This closes #5040

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-05-03 13:40:01 -05:00
Matthew Burgess 3963f66dff
NIFI-8320: Fix column mismatch in PutDatabaseRecord
This closes #5024

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-30 10:41:36 -05:00
Tamas Palfy 16dc61e151 NIFI-8458 TailFile - Fix some bugs. Add more tests.
- During cleanup keep "tailingPostRollover" in the updated state.
- Skipping tests that can't run on Windows.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-04-27 09:44:39 -04:00
Peter Turcsanyi 525bfe00b3
NIFI-8474: Added new Replacement Strategy for variable substitution in ReplaceText.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5032.
2021-04-27 13:03:21 +02:00
Paul Grey 835f50c83a
NIFI-8465 Handle bcrypt legacy decrypt failures in testing
This closes #5029

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-26 11:34:20 -05:00
Paul Kelly aedacdf86f
NIFI-8454: Allow InvokeHTTP to output final URL from response request property
This closes #5016

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-21 18:18:25 -05:00
Mark Payne 1ef1905461 NIFI-8344: Addressed corner case in which we didn't handle a case where we tailed a file after being rolled over and then processor encounters a newline followed by NUL characters
This closes #5009.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-04-20 14:51:24 +02:00
Joe Gresock be716c1621 NIFI-3862: Adding Issuer DN support to ListenHTTP
This closes #4976.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-04-20 13:09:53 +02:00
exceptionfactory ef794c56b6
NIFI-8398 Updated dependencies to build using Apache Maven 3.8.1
- Updated ParCEFone to 1.2.8
- Added conjars.org HTTPS repository for Hive transitive dependencies

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4989
2021-04-12 14:45:02 -04:00
Mark Payne 7d1d536da6
NIFI-8206: Added identifiesExternalResource() method to PropertyDescriptor.Builder and implemented functionality.
- Updated components to make use of new feature

NIFI-8206: Added a ResourceType of TEXT. This requires that the ResourceReferenceFactory know which types are allowed in order to create the ResourceReference. PropertyValue needs to then have the PropertyDescriptor available to it. This resulted in highlighting many bugs in unit tests where components were not exposing property descriptors via getSupportedPropertyDescriptors() or were evaluating Expression Language using the wrong scope, so fixed many unit tests/components to properly declare Expression Language scope when using it

NIFI-8206: Removed problematic unit test that required directory names with special characters that are not allowed on some operating systems

This closes #4890.

Signed-off-by: Bryan Bende <bbende@apache.org>
2021-04-12 11:48:04 -04:00
Mark Payne 2edf5514b7 NIFI-8344: If configured to tail a file for some period of time post-rollover, ensure that we only consume lines that are fully written (i.e., end with a newline). Once we stop tailing that file post-rollover, consume any data from that file that has not yet been consumed, up to the end of the file, even if there is no newline.
This closes #4937.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-03-31 13:40:08 +02:00
exceptionfactory 1cd3fbb4eb NIFI-8288 Removed OkHttpClientUtils to reduce reliance on nifi-security-utils
- Added createTrustManager() on SSLContextService
- Removed nifi-security-utils and okhttp dependencies from nifi-web-utils

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4869.
2021-03-25 15:38:24 -04:00
Hsin-Ying Lee 0f28702b47
NIFI-8354 ExecuteStreamCommand processor doesn't delete the temp file… (#4923)
* NIFI-8354 ExecuteStreamCommand processor doesn't delete the temp file if the process start failed
* NIFI-8354 Record the log when delete file failed


This closes #4923

Signed-off-by: Otto Fowler <otto@apache.org>
2021-03-24 12:26:56 -04:00
exceptionfactory e505e8b42d
NIFI-8352 This closes #4922. Changed assertServerConnections to loop and sleep while evaluating
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-23 07:15:48 -07:00
Mark Payne e1b9548ab6 NIFI-8344: Introduced new Rollover Tail Period property
This closes #4916.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-03-22 18:22:00 +01:00
exceptionfactory a7ba5bb4f8
NIFI-8319 Added AES/CBC/NoPadding for decryption in EncryptContent Processor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4894.
2021-03-18 01:40:18 +04:00
exceptionfactory 8202bffc98
NIFI-8304 This closes #4907. Replaced HttpsURLConnection with OkHttpClient in TestListenHTTP
NIFI-8304 Updated TestPutTCP to shutdown server before checking connections
NIFI-8304 Changed TestListenTCP to send messages in one byte array
NIFI-8304 Added check for expected jdk.tls.disabledAlgorithms
2021-03-17 14:15:25 -07:00
exceptionfactory be7233d252
NIFI-8304 Changed TestListenTCPRecord to loop until files found
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4904.
2021-03-17 11:41:21 +04:00
exceptionfactory 2ad88bbfff
NIFI-8304 This closes #4900. Improved Socket test reliability for several Processors
- Refactored TestPutTCP to single class
- Improved TestListenRELP
- Improved TestListenTCP
- Improved TestListenUDP
- Improved TestListenTCPRecord
- Changed OnUnscheduled to OnStopped in AbstractListenEventProcessor

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-16 15:30:11 -07:00
exceptionfactory b1f513a2ca
NIFI-8304 Refactored InvokeHTTP unit tests using OkHttp MockWebServer
- Replaced Joda Time with java.time for date formatting
- Replaced Guava Files with java.nio.file.Files for cache directory
- Updated PutTCP test server to close connection when testing connection per FlowFile

NIFI-8304 Removed Thread wrapper for TestListenHTTP client requests

NIFI-8304 Disabled InvokeHTTP Connection Pooling for testing

NIFI-8304 Set 60 second timeout for testing TLS connections

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4892.
2021-03-15 21:28:40 +04:00
exceptionfactory 09692c5f69
NIFI-8313 Upgraded net.lingala.zip4j:zip4j to 2.7.0
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4889.
2021-03-12 12:21:07 +04:00
Otto Fowler 9f0b47af77
NIFI-2702 Support named captures in ExtractText
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4384.
2021-03-12 12:11:09 +04:00
Mark Payne b523698534
NIFI-8212: Refactored StandardExtensionDiscoveringManager to avoid using ServiceLoader
Instead, it will look at the ServiceLoader file and read the names of the classes but avoid instantiating all of the objects or loading the classes into memory.
- Updated Doc Generation so that if the documentation for a given NAR already exists, it doesn't delete it and re-generate it. This was necessary because we are no longer instantiating an instance of each component and instead lazily creating the components as necessary.
- Removed stateless version of extension registry because it's no longer necessary

This closes #4852

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-08 12:21:19 -06:00
Peter Gyori 4a27b23b1f
NIFI-8263: Maximum Thread Pool Size property introduced in ListenHTTP
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4847.
2021-03-01 20:14:34 +04:00