Commit Graph

402 Commits

Author SHA1 Message Date
Mark Payne 56593ad12f
NIFI-9546: Updated PutTCP and PutUDP with SupportsBatching annotations
- Removed incorrect usage of TriggerWhenEmpty
- Allow for 0 seconds of gracefully waiting for incoming threads on shutdown since we won't have any
- Updated unit tests to no longer have arbitrary sleep statements

This closes #5639

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-01-06 15:52:37 -06:00
exceptionfactory 12ba579b8f NIFI-9539 Upgraded Netty from 4.1.69 to 4.1.72
- Added Netty 4 BOM to root Maven configuration
- Removed individual Netty 4 dependency overrides
- Removed commented dependencies and unnecessary declarations from MiNiFi

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5635.
2022-01-05 18:11:03 -05:00
exceptionfactory 7c6bdcb035
NIFI-9491 Excluded commons-logging and added jcl-over-slf4j references
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5608
2021-12-17 07:21:03 -05:00
Nathan Gough 5e9c86885c
NIFI-9478 Moved Netty Log Exception Handler to end of pipeline
- Changed display name of Max Number of TCP Connections to Max Number of Worker Threads for ListenTCP
- Set Netty Socket Receive Buffer using Max Socket Buffer Size in ListenTCP

This closes #5599

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-12-15 16:59:07 -06:00
Mark Payne 97198e35a0
NIFI-9382: This closes #5584. Added system test that replicates issue in which a closed shared classloader causes issues when used again
NIFI-9382: Fixed issue with SharedInstanceClassLoader where the classloader may get closed but then get used again. When the SharedInstanceClassLoader is closed, we will now ensure that we don't use anymore and instead create a new one.

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-12-09 12:41:57 -07:00
Tamas Palfy ff864266f5
NIFI-9436 - In AbstractPutHDFSRecord make sure the record writers use the FileSystem object the processor already has.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5565
2021-12-02 14:24:12 -05:00
markap14 839fbf7d19
NIFI-9382: Created a new ClassloaderIsolationKey mechanism by which H… (#5533)
* NIFI-9382: Created a new ClassloaderIsolationKey mechanism by which Hadoop related processors (and potentially others) can indicate that they need full classloaders to be cloned but can share with other instances in certain circumstances
- Added system tests

* NIFI-9382: Renamed interface based on review feedback

* NIFI-9382: Removed ReentrantKerberosUser.
2021-11-22 14:55:19 -05:00
exceptionfactory 0cf515c9c0 NIFI-9384 Corrected usage and generics in ListenTCP
- Addressed compiler warnings in ListenTCP and EventBatcher
- Adjusted ListenTCP property order to match previous version

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5526.
2021-11-17 11:54:51 -05:00
noblenumbat360 f410c8df0a
NIFI-8272 Delete stale metrics from REST API Prometheus endpoint.
Added <scope>test</scope> tag to the nifi-web-api pom.xml and corrected imports.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5447
2021-11-08 20:26:30 -05:00
Joe Witt 9e845d87ff
Merge branch 'NIFI-9344-RC3' 2021-11-07 16:23:10 -07:00
Mark Payne 705c65c86c
NIFI-9362: Ensure that we update the StateMap in AbstractListProcessor to hold any files whose date matches the latest before setting cluster-wide state
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5507
2021-11-04 13:46:41 -04:00
Joe Witt c055895952
NIFI-9344-RC3 prepare for next development iteration 2021-11-03 08:53:37 -07:00
Joe Witt 7fdc07cccd
NIFI-9344-RC3 prepare release nifi-1.15.0-RC3 2021-11-03 08:53:32 -07:00
Bence Simon 75d1d7f6e7
NIFI-9265 Fixing path handling for HDFS processors when there are multiplied separators in the path
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5437.
2021-10-28 10:47:53 +02:00
exceptionfactory 60d6d469bf
NIFI-9342 Upgraded to Netty 3.10.6 and 4.1.69
- Replaced Netty 3.6.9 and 3.7.1 with 3.10.6
- Replaced Netty 4.1 with 4.1.69

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5490.
2021-10-28 09:34:31 +02:00
exceptionfactory 67ccdf6159
NIFI-9326 Added Socket Keep Alive property to ListenSyslog
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5479.
2021-10-26 19:44:56 +02:00
Mark Payne 1191e511a5
NIFI-9335: Updated AvroTypeUtil#createAvroRecord to ensure that if the given Avro Schema contains a field whose value is defaulted, the produced Avro Record has that value populated. Also added a unit test to verify behavior. Ran performance test (which is igonred and must be manually enabled) - about 2.1 seconds before the change and about 2.5 seconds after the change for each iteration.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5483
2021-10-26 12:15:54 -04:00
Nathan Gough 309ab8f4ab
NIFI-8792 - Modified ListenRELP to use Netty
- Refactored RELP encoders and decoders

This closes #5398

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-10-25 15:53:58 -05:00
Joe Gresock d2f8f97b10
NIFI-9276: Adding config verification to AbstractListProcessor subclasses (#5453) 2021-10-21 12:22:50 -04:00
Bryan Bende a380fb2964
NIFI-9297 Correct typo in documentation for listing strategy in AbstractListProcessor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5466.
2021-10-19 17:25:17 +02:00
exceptionfactory 4bcd03024a
NIFI-9283 Excluded Log4j 1.2 and upgraded Log4j2 to 2.14.1
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5440.
2021-10-06 11:17:36 -04:00
exceptionfactory 4943560521
NIFI-9223 Corrected ListenSyslog with default address of 0.0.0.0
- Refactored NettyEventServerFactory to accept nullable InetAddress
- Updated unit tests referencing NettyEventServerFactory

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5426.
2021-10-01 09:24:46 +02:00
Peter Gyori 38d26dafa3
NIFI-9192: ResultSetRecordSet considers value of useLogicalTypes flag when determining the object's schema.
NIFI-9192: ResultSetRecordSet consider value of useLogicalType flag when sqlType is not handled separately (default) and readerSchema is not null

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>

NIFI-9192: Unit tests added to test the use of useLogicalType flag

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>

NIFI-9192: ResultSetRecordSet consider not using logical type when sqlType is OTHER

NIFI-9192: Fixed checkstyle violations

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>

NIFI-9192: Renamed JdbcCommon.getUseLogicalTypes to isUseLogicalTypes

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>

NIFI-9192: Added unit tests for Array types with and without using logical types

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>

NIFI-9192: Review comments considered and unit tests refactored

NIFI-9192: Added further unit tests

NIFI-9192: Minor modifications based on review comments

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5388
2021-09-27 16:27:39 -04:00
hondawei 30cf49db7e
NIFI-4542 - add target.dir.created to indicate if the target directory created
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5397.
2021-09-26 11:48:18 +02:00
exceptionfactory b4e0416b5e
NIFI-9228 Refactored tests using TemporaryKeyStoreBuilder
- Added TemporaryKeyStoreBuilder with File.deleteOnExit() for KeyStore and TrustStore files
- Removed JKS files from nifi-security-utils tests
- Refactored usage of KeyStoreUtils.createKeyStoreAndGetX509Certificate() to TemporaryKeyStoreBuilder
- Removed unnecesary hadoop-minikdc test dependency in security-utils
- Replaced Mini KDC Hex utility with Bouncy Castle Hex utility in unit tests

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5406
2021-09-21 16:07:13 -04:00
Paul Grey 1ab47ce931
NIFI-9212 Add timeout and quiet period for Netty shutdown
This closes #5386

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-09-16 09:38:49 -05:00
exceptionfactory f5a421ccb1
NIFI-9067 Upgraded JAXB Runtime from 2.3.0 to 2.3.5
- Replaced old com.sun.xml.bind:jaxb-impl and jaxb-core with current org.glassfish.jaxb:jaxb-runtime
- Replaced old javax.xml.bind:jaxb-api with current jakarta.xml.bind-api
- Removed unnecessary dependency references to javax.activation-api

This closes #5320

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2021-09-07 13:27:53 -04:00
Bryan Bende e436381c3a
NIFI-8975 Integrate KerberosUserService into HBase processors/services
This closes #5322

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-08-27 17:41:58 -05:00
timeabarna 9ebdd4bdf1
NIFI-8761 Enable not setting a value for Escape Character in CSVReade… (#5249)
NIFI-8761 Enable not setting a value for Escape Character in CSVReader controller service

Co-authored-by: Pierre Villard <pierre.villard.fr@gmail.com>
2021-08-24 14:38:16 +02:00
Bryan Bende 21c2fb95d3
NIFI-8973 Implement KerberosUserService API and keytab, password, and ticket cache implementations
NIFI-8974 Integrate KerberosUserService with HDFS processors

NIFI-8980 Integrate KerberosUserService with Kafka 2.6 processors
- Introduced SelfContainerKerberosUserService to restrict which impls can be used with Kafka
- Add variations of KerberosUser doAs that allow setting the context ClassLoader
- Add additional unit tests for configurations

This closes #5277

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-08-18 19:54:56 -05:00
Bence Simon c3ecf2fea8 NIFI-9032 Refactoring HDFS processors in order to increase flexibility
This closes #5295.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-08-17 16:57:27 +02:00
exceptionfactory d93e9f152c
NIFI-8986 Upgraded Commons Compress to 1.21
- Upgraded direct and transitive dependencies from 1.20 and earlier to 1.21

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5273.
2021-08-03 15:52:45 +02:00
Timea Barna 0497129aa6
NIFI-8763 Hide some properties in CSVReader when built-in CSV Format is selected
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5263.
2021-08-03 10:14:47 +02:00
exceptionfactory 49cbc9bd6b
NIFI-8791 Removed unused ChannelSender and implementations - Removed unused PruneResult inner class from AbstractPutEventProcessor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5221.
2021-07-19 20:24:03 +02:00
Joe Witt 7356332852
Merge branch 'NIFI-8767-RC2' 2021-07-14 09:07:22 -05:00
exceptionfactory 05d7867a16
NIFI-8768 Added toLocalDate() for convertType() handling of DATE fields
- Updated PutKudu to use DataTypeUtils.toLocalDate() for DATE fields
- Updated PutDatabaseRecord to remove convertDateToLocalTZ() since convertType() uses toLocalDate()
- Updated PutElasticsearchHttpRecord to use default time zone format for DATE fields
- Updated WriteXMLResult to use default time zone format for DATE fields
- Updated WriteJsonResult to use default time zone format for DATE fields
- Updated AvroTypeUtil to use toLocalDate() for logical DATE fields
- Updated JdbcCommon to avoid conversion to UTC for logical DATE fields
- Updated Processor and RecordReader unit tests for consistency in DATE comparison

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5210
2021-07-13 16:17:36 -04:00
Denes Arvay 4fde9648dd
NIFI-8759 ExecuteSQL and ExecuteSQLRecord unnecessarily fall back to default decimal scale
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5212
2021-07-13 13:27:58 -04:00
Joe Witt 97feacc181
NIFI-8767-RC2 prepare for next development iteration 2021-07-10 12:17:09 -07:00
Joe Witt fcbf1d5f97
NIFI-8767-RC2 prepare release nifi-1.14.0-RC2 2021-07-10 12:17:05 -07:00
thenatog 775130580a
NIFI-8616 - Updated DelimitedInputStream to extend the FilterInputStream class. (#5203) 2021-07-07 18:36:58 -04:00
Nathan Gough 9a125dcaaf
NIFI-8616 Refactored PutTCP, PutUDP, and PutSplunk to use Netty nifi-event-transport classes
This closes #5182

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-07-07 16:24:49 -05:00
Peter Turcsanyi 39ffa4a8ac
NIFI-8737: Fixed incorrect provenance data in HDFS processors when Directory property is inconsistent with core-site.xml 2021-06-25 14:19:57 +02:00
exceptionfactory 6a83115d6a NIFI-7468 Updated SSLSocketChannel to support TLS 1.3
- Handling additional FINISHED Handshake Status for TLS 1.3 Post-Handshake Messages per RFC 8446 Section 4.6
- Removed clearing buffers after handshake to avoid losing packets
- Updated read() method to check Handshake Status after SSLEngine.unwrap()
- Changed SSLSocketChannelSender to close SSLSocketChannel before other resources
- Added ChannelStatus enum and convenience logging methods for tracing status
- Added unit tests for TLS 1.2 and 1.3 using Netty server and client handlers

NIFI-8704 Updated netty-handler to 4.1.65.Final

NIFI-7468 Corrected SSLSocketChannel.read() to return byte read

NIFI-7468 Adjusted comment formatting

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5152.
2021-06-22 22:29:16 -04:00
simonbence 73363d9aea
NIFI-8717 Refactoring PutHDFS (#5175)
NIFI-8717 Refactoring PutHDFS
2021-06-22 13:13:14 -04:00
exceptionfactory f2698a4e1e
NIFI-8723 Upgraded Jackson to 2.12.3
- Updated MiNiFi to reference root jackson.version
- Updated Registry assembly to reference root jackson.version
- Added dependency management in NAR modules to override transitive versions

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5174
2021-06-22 11:21:45 -04:00
exceptionfactory 2efc2b6b8b
NIFI-8718 Upgraded Apache Commons IO to 2.10.0
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5169
2021-06-21 11:29:27 -04:00
Mark Payne e06afbdd22
NIFI-8609: Optimized AvroTypeUtil Record creation and conversion
Added unit test that is ignored so that it can be manually run for testing performance before/after changes to AvroTypeUtil. Updated AvroTypeUtil to be more efficient by not using Record.getValue() and instead iterating over the Map of values directly. getValue() is less efficient here because we know the RecordField's we are iterating over exist in the schema since they are retrieved from there directly; as a result, any null values still have be looked up by aliaases, but that step can be skipped in this situation. Also avoided looking for fields that exist in Avro Schema and not in RecordSchema just to set default values on GenericRecord - there's no need to set them if they are default values.

This closes #5080

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-06-15 09:05:38 -05:00
exceptionfactory ab851b9cde NIFI-8672 Added syncUninterruptibly() to Netty shutdown and close methods
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5139.
2021-06-12 17:14:34 -04:00
exceptionfactory a3365c8833 NIFI-8462 Refactored PutSyslog and ListenSyslog using Netty
- Added nifi-event-transport module encapsulating Netty classes
- Refactored unit tests for PutSyslog and ListenSyslog
- Removed integration tests for PutSyslog and ListenSyslog

NIFI-8462 Added context.yield() in PutSyslog when no FlowFiles and addressed other issues

NIFI-8462 Removed unused import of ExpressionLanguageScope

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5044.
2021-05-25 10:54:15 -04:00
Tamas Palfy 5108d7cdd0 NIFI-8439 Update parquet-avro to allow reading parquet INT96 timestamps as byte arrays (instead of throwing an exception).
Also allow to write them as such (byte-arrays) - again, instead of throwing an exception.

NIFI-8439 Fixed unit tests.
NIFI-8439 Allow writing parquet INT96 timestamps if they were read by the same parquet-avro library.

This closes #5006.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-05-25 13:18:46 +02:00
Mohammed Nadeem 622507f994
NIFI-8627: Upgrade apache derby to 10.14.2.0 to mitigate CVE_2015_1832
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5095
2021-05-24 14:35:27 -04:00
Mark Payne ecacfdaa4c NIFI-8469: Introduced ProcessSession.commitAsync and updated processors to use it. Deprecated ProcessSession.commit()
- Updated Mock Framework to now fail tests that use ProcessSession.commit() unless they first call TestRunner.setAllowSynchronousSessionCommits(true)
- Updated stateless nifi in order to make use of async session commits
- Fixed bug that caused stateless to not properly handle Additional Classpath URLs and bug that caused warnings about validation to get generated when a flow that used controller services was initialized. While this is not really in scope of NIFI-8469, it was found when testing and blocked further progress so addresssed here.
- If Processor fails to progress when run from stateless, trigger from start of flow until that is no longer the case
- Introduced notion of TransactionThresholds that can limit the amount of data that a flow will bring in for a given invocation of stateless dataflow
- Several new system-level tests
2021-05-19 08:51:16 -04:00
Bence Simon 51aae5bcf6 NIFI-8519 Adding HDFS support for NAR autoload
- Refining classloader management with the help of @markap14

This closes #5059

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-05-14 13:14:06 -04:00
exceptionfactory 6776765a92
NIFI-8538 Upgraded Apache Commons IO to 2.8.0
- Upgraded direct dependencies from 2.6 to 2.8.0
- Added dependency management configuration to use 2.8.0 for some modules
- Updated scripted Groovy tests to avoid copying unnecessary files

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5073
2021-05-14 09:31:48 -04:00
Mark Payne 2895bac2c0
NIFI-8512: When converting to/from Avro UNION type, we can be more efficient when the UNION consists of a Null type and one other type by determinine the non-null type and just using that. Also eliminated a call to List.stream() and related .collect() call by using an existing method that performs the logic without the very expensive call to stream()
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5051
2021-05-06 16:06:23 -04:00
Otto Fowler a44ab0d683
NIFI-8397 update to simple-syslog-5424 0.0.16
resolves issue working with syslog with BOM markers

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4978.
2021-04-20 09:28:29 +02:00
Tamas Palfy a50957161c NIFI-8365 Fix JSON AbstractJsonRowRecordReader to handle deep CHOICE-typed records properly: change the logic that selects the first compatible schema which can have missing fields compared to the real value and search for a more strict match first and fallback to the existing logic only if not one found.
- AbstractJsonRowRecordReader - Handle (meaning log a warning and not fail completely) multi-array CHOICE type when data has extra fields (not defined by the schema) and can't determine correct type.
- AvroTypeUtil - Allow multiple different record types in avro union type. Minor refactors. Added documentation fro EqualsWrapper.
2021-04-19 12:56:09 -04:00
Mark Payne 7d1d536da6
NIFI-8206: Added identifiesExternalResource() method to PropertyDescriptor.Builder and implemented functionality.
- Updated components to make use of new feature

NIFI-8206: Added a ResourceType of TEXT. This requires that the ResourceReferenceFactory know which types are allowed in order to create the ResourceReference. PropertyValue needs to then have the PropertyDescriptor available to it. This resulted in highlighting many bugs in unit tests where components were not exposing property descriptors via getSupportedPropertyDescriptors() or were evaluating Expression Language using the wrong scope, so fixed many unit tests/components to properly declare Expression Language scope when using it

NIFI-8206: Removed problematic unit test that required directory names with special characters that are not allowed on some operating systems

This closes #4890.

Signed-off-by: Bryan Bende <bbende@apache.org>
2021-04-12 11:48:04 -04:00
Matthew Burgess a5dbf56114
NIFI-8368: If decimal scale > precision, set precision = scale
This closes #4938

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-29 10:53:46 -05:00
Nathan Gough 07a4966d10
NIFI-8329 - Updated dependencies with no build failures
NIFI-8329 - Removed unnecessary jackson.version from azure bundle to use the global property instead.

NIFI-8329 - Updated jackson/jackson-databind version and removed the 'jackson-databind.version' pom property in favor of 'jackson.version'

Updated dependencies include the following:

- jackson-core
- jackson-databind
- icu4j
- snakeyaml
- spring-integration-mail
- spring-core and framework modules
- activemq-client
- activemq-broker
- xercesImpl

This closes #4911

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-19 14:46:33 -05:00
exceptionfactory 0e659981d4 NIFI-8298 Refactored Kerberos and Socket classes from security-utils to new modules
- Created nifi-security-socket-ssl
- Created nifi-security-kerberos
- Removed nifi-security-utils dependency from nifi-processor-utils
- Updated modules to reference new dependencies
- Eliminated unnecessary transitive dependencies on bcprov-jdk15on from over 30 modules

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4881.
2021-03-19 13:12:19 -04:00
exceptionfactory 2ad88bbfff
NIFI-8304 This closes #4900. Improved Socket test reliability for several Processors
- Refactored TestPutTCP to single class
- Improved TestListenRELP
- Improved TestListenTCP
- Improved TestListenUDP
- Improved TestListenTCPRecord
- Changed OnUnscheduled to OnStopped in AbstractListenEventProcessor

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-16 15:30:11 -07:00
tpalfy f2a03fca2a
NIFI-8285 Prevent HBase client services to throw NPE in non-kerberized environment. (#4868)
* NIFI-8285 Prevent HBase client services to throw NPE in non-kerberized environment.

* NIFI-8285 Improve exception handling.
2021-03-08 09:10:12 -05:00
Matthew Burgess 29ebec071e
NIFI-4974: Add try/catch around blob.free() to support JDBC spec
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4861.
2021-03-07 18:51:07 +04:00
Mark Payne e2e137fced
NIFI-8291: Support escaping unicode characters for fields that previously allowed escaping only of \t, \n\, \r
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4871.
2021-03-03 21:26:18 +04:00
Koji Kawamura 68d38dd0a6
NIFI-6752 Add ASN.1 RecordReader
NIFI-6752 Refactored type and value conversion logic. Added support for more types. Added more tests.
Removed 'parent' from 'Recursive'. (Caused issues. The recursive nature is still there as it has a child with the same type).
Updated jasn1 1.11.2 to asn1bean 1.12.0. If an asn field name is a Java reserved keyword, the field gets a trailing "_" but the getter remains normal. In JASN1Utils adjusted logic when looking for the getter.
Added support for inherited types. OctetStrings are converted to Strings instead of byte arrays.
Service takes care of the compilation of the ASN files. Test sources are generated and removed from source control.

NIFI-6752 Removed obsolete TODOs.

NIFI-6752 Updated nifi-asn1-nar version to 1.13.0-SNAPSHOT. Fixed checkstyle violations (unused imports).

NIFI-6752 ASN.1 reader - ASN.1 bundle requires 'include-asn1' profile to be active to be part of assembly.

NIFI-6752 ASN.1 reader - Updated ASN1.xml template.

NIFI-6752 ASN.1 reader - Updated versions.

NIFI-6752 ASN.1 reader - Update example generator. Updated ASN1.xml template. Updated (fixed) nifi-asn1-nar version in pom.xml.

NIFI-6752 ASN.1 reader - Added missing license for ASN1.xml.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4577
2021-02-25 12:58:05 -05:00
Christian Gumpert f2555f27f1
NIFI-8119: properly free database resources when done with the processing of BLOBs
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4791
2021-02-23 16:47:32 -05:00
Chris Sampson 3cb26aec72
NIFI-8232 CSV Parsers optionally allow/reject duplicate header names
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4828.
2021-02-23 09:50:18 +04:00
Joe Witt 88fab00e29
NIFI-7873 merging release branch to latest and updating to 1.14.0-SNAPSHOT 2021-02-15 12:09:32 -07:00
Joe Witt 4afb2ba743
NIFI-7873-RC4 prepare for next development iteration 2021-02-15 12:09:31 -07:00
Joe Witt 487280bee9
NIFI-7873-RC4 prepare release nifi-1.13.0-RC4 2021-02-15 12:09:30 -07:00
Tamas Palfy b55998afc1 NIFI-8081 Added new Listing Strategy to ListFTP and ListSFTP: Time Window
NIFI-8081 Added new Listing Strategy to ListFTP and ListSFTP: Adjusted Time Window. User can specify the time zone or time difference (compared to where NiFi runs) of the system hosting the files and based on the calculates the current time there. Lists files modified before this adjusted current time (and after the last listing).
NIFI-8081 'Time Adjustment' validated not to be set if listing strategy is not 'Adjusted Time Window'. Extracted validator to a separate class. Added more tests. Minor refactor. Typo fix.
NIFI-8081 Improved validation.
NIFI-8081 'Time Adjustment' is not necessary - in fact it can cause problems. SFTP (and usually FTP - which has a more general bug at the moment) returns a timestamp that doesn't really need adjustment. (SFTP in particular returns the an 'epoch' time.) Everything remains the same - the new listing strategy relies on a sliding time window, but without the unnecessary option to adjust for the modification time.
NIFI-8081 Resolved conflicts after rebasing to main.
NIFI-8081 Renamed 'AbstractListProcessor.listByAdjustedSlidingTimeWindow' to 'listByTimeWindow'. Post main rebase correction.
NIFI-8081 Updated user doc for the BY_TIME_WINDOW strategy to warn user on it's reliance of accurate time.

This closes #4721.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-02-04 17:04:37 +01:00
Joe Witt d826416217
NIFI-8192 updating Copyright years for nifi things to include 2021
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4802.
2021-02-03 12:56:28 +04:00
exceptionfactory 8c1fe84f62
NIFI-8177 This closes #4786. Added timeout to Sender Pool polling to avoid blocking indefinitely
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-01-27 15:04:23 -07:00
Waleed Al Aibani 813ac539b3
NIFI-7263 This closes #4167. Add No Tracking Listing Strategy to ListFile/ListFTP processors
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-01-26 22:13:26 -07:00
Peter Turcsanyi 67d06003b7 NIFI-8023: Convert java.sql.Date between UTC/local time zone normalized forms before/after database operations
This closes #4781

Signed-off-by: David Handermann <exceptionfactory@gmail.com>
2021-01-26 14:39:02 -06:00
Mark Payne 930e8d9e0e
NIFI-7698: Added a merge.reason attribute for flowfiles indicating why they were merged by MergeContent. Also updated logs to indicate the reason and added additional documentation for processor
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4764
2021-01-19 17:20:06 -05:00
Mark Payne f02079c2fa
NIFI-8146: Added RecordPath for Statement Type and to specify where to find the data within the incoming Record. Also performed some minor code refactoring to avoid lots of large lambdas, as it results in code that is more readable and more maintainable. Updated some properties to use dependsOn() notation now that this is available.
NIFI-8146: Updated PutDatabaseRecord to avoid using the functional style framework that had previously been used. Doing so resulted in code that was difficult to understand and maintain. Added additional unit tests and improved MockRecordParser so that it could throw configurable types of exceptions

NIFI-8146: Fixed checkstyle issues

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4763
2021-01-19 13:23:54 -05:00
Mark Payne 097edf4f7c
NIFI-8136: Added getState/setState/replaceState/clearState methods to ProcessSession, updated processors to use these methods instead of StateManager version where appropriate
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4757
2021-01-15 10:02:09 -05:00
Matthew Burgess 382439c1d0 NIFI-7973: Add default precision and scale properties to remaining SQL-based components
This closes #4682.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-01-13 08:41:12 -05:00
exceptionfactory 0af736dc0b
NIFI-7885 Added Environment Variable to deny LFS access using Hadoop (#4712)
* NIFI-7885 Added Environment Variable to deny LFS access using Hadoop

* NIFI-7885 Changed Hadoop Processor validation to check LFS access
2020-12-09 09:29:42 -05:00
Bryan Bende 039a09923a
NIFI-8073 Correcting isBackPressureEnabled in PrometheusMetricsUtil 2020-12-07 10:03:16 -05:00
Bence Simon 39f8a008d4 NIFI-8039 Adding properties to ListenTCP in order to allow refine behaviour under higher load; Refining thread pool for better scalability
NIFI-8039 Review findings; refining thread pool to be able to scale down properly when not under load
NIFI-8039 Answers to PR comments

This closes #4689.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-11-30 10:17:15 +01:00
Denes Arvay 83948db989
NIFI-7996 Conversion with ConvertRecord to avro results in invalid date
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4666.
2020-11-16 17:40:40 +01:00
tpalfy 940bc3056c
NIFI-7954 Wrapping HBase_*_ClientService calls in getUgi().doAs() (#4629)
* NIFI-7954 Wrapping HBase_*_ClientService calls in getUgi().doAs() and taking care of TGT renewal.

* NIFI-7954 Simplified SecurityUtil.callWithUgi a little.

* NIFI-7954 Simplified SecurityUtil.callWithUgi more.

* NIFI-7954 Removed unnecessary code.
2020-11-09 09:00:20 -05:00
Pierre Villard 14ec02f21d
NIFI-7981 - add support for enum type in avro schema
This closes #4648

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-11-05 18:19:55 -05:00
Bryan Bende 1366d017b8
NIFI-7968 Ensure the status login user is set in UserGroupInformation after creating a UGI from a Subject
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4643.
2020-11-05 13:18:35 +01:00
Bryan Bende d773521ee0
NIFI-1121 Fix Schema Name and Schema Branch properties
This closes #4512.

Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-03 15:38:41 -05:00
Mark Payne 4b9014b959
NIFI-1121: Updated backend to perform appropriate validation. Added tests. Updated documentation writer. Updated dev guide to explain how PropertyDescriptor.Builder#dependsOn affects validation. Updated JavaDocs for PropertyDescriptor.Builder#dependsOn
Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-03 15:37:42 -05:00
mtien 535cab3167
NIFI-1121: Added an additional check for hidden properties to account for transitive dependent properties.
- Added a 'dependent' attribute to determine whether or not to save dependent property values

Co-authored-by: Scott Aslan <scottyaslan@gmail.com>
Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-03 15:37:33 -05:00
Mark Payne f7f336a4b0
NIFI-1121: Added API changes for having one Property depend on another
Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-03 15:37:08 -05:00
Denes Arvay d05d0c6240 NIFI-7925: ValidateRecord reports false positive for avro arrays with null elements 2020-10-21 09:17:29 -04:00
Matthew Burgess 7cc3713389
NIFI-7796: Add Prometheus counters for total bytes sent/received (#4522)
* NIFI-7796: Add Prometheus metrics for total bytes sent/received, fixed read/written metrics

* NIFI-7796: Incorporated review comments
2020-10-06 09:26:27 -04:00
Denes Arvay f73a019f36 NIFI-7843 Recursive avro schemas fail to write with RecordWriter
NIFI-7843 Recursive avro schemas fail to write with RecordWriter
Add new test case to TestSimpleRecordSchema to test the scenario
when schema name and schema namespace match.

This closes #4550.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-09-28 19:52:41 +02:00
Matthew Burgess 7e145142e1
NIFI-7799: Relogin with Kerberos on connect exception in DBCPConnectionPool (#4519) 2020-09-17 13:33:54 -04:00
Andy LoPresto 9370571131
NIFI-7804 Split nifi-security-utils into sub-module for nifi-security… (#4533)
* NIFI-7804 Split nifi-security-utils into sub-module for nifi-security-utils-api (no external dependencies).
Separated interface and implementation of TlsConfiguration.
Reabsorbed nifi-security-xml-config into nifi-security-utils.

* NIFI-7804 Resolved failing unit test on Java 8.
Removed accidental module dependency.

* NIFI-7804 Resolved failing unit test.

* NIFI-7804 Removed legacy dependency.

* NIFI-7804 Marked nifi-security-utils-api as provided and overrode with compile scope in specific modules which are not children of nifi-standard-services-api-nar.
2020-09-17 12:52:22 -04:00
Joe Witt 8baa5c9940
NIFI-7692 updating for next dev release 1.13.0 2020-08-18 14:48:02 -07:00
Joe Witt fb57bcbc11
NIFI-7692-RC1 prepare for next development iteration 2020-08-13 09:20:39 -07:00
Joe Witt 303d6c59ba
NIFI-7692-RC1 prepare release nifi-1.12.0-RC1 2020-08-13 09:20:36 -07:00
Mark Payne 869c4236c4
NIFI-7729: Ensure that if a script used in the ScriptedTransformRecord processor introduces a new field to the schema that the field gets incorporated into the schema.
NIFI-7729: Updated docs to explain how to add new fields to Records via ScriptedTransformRecord and added example. Fixed checkstyle violation.

NIFI-7729: Always call Record.incorporateInactiveFields with ScriptedTransformRecord

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4470
2020-08-11 18:06:55 -04:00
zhangcheng e18b4f0c75
NIFI-7410 Update JdbcCommon.java when javaSqlType is CLOB or NCLOB in convertToAvroStream method, use the DataTypeUtils.toString(clob/nClob,(String) null, StandardCharsets.UTF_8)) method to get the String
NIFI-7410 Update JdbcCommon.java when javaSqlType is CLOB or NCLOB in convertToAvroStream method, use the CharacterStream rto read the value of CLOB
NIFI-7410 Add a unit test. validate if it's unreadable when the clob value is Chinese, Japanese, and Korean.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4243.
2020-07-17 14:39:30 +02:00
Mark Payne 6cece9cad7
NIFI-7572: Added ScriptedTransformRecord processor. Addressed a couple of minor bugs in mock classes that were encountered during testing.
NIFI-7572: Addressed review feedback
NIFI-7572: Fixed bug that resulted in constantly recompiling Jython script. Updated documentation showing performance difference. Fixed problematic unit tests for TestResizeImage

This closes #4374

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-07-08 15:52:06 -04:00
Mark Payne 0a16002076
NIFI-7509: Added optional Record Writer property to all List* Processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4315.
2020-06-19 17:30:40 +02:00
Bence Simon 5c2bfcf7d3 NIFI-7369 Adding decimal support for record handling in order to avoid missing precision when reading in records
Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-06-02 15:13:14 -04:00
zhangcheng 1ec6675f38
NIFI-7403:Add a function that adjust if the result is failed before we call the onFailed or onCompleted function. If the result is failed, return true and do sth
NIFI-7403:Add an extension point to adjust the result, if the result is failed then process onFailed function

NIFI-7403:Implement the AdjustFailed Function, if PutSQL set the SUPPORT_TRANSACTIONS true, then check whether the result contains REL_RETRY or REL_FAILURE.If it contains that, reroute the result and return true.

NIFI-7403: fix reroute logic in AdjustFailed function

NIFI-7403:Add and modify some unit test for PutSQL's SUPPORT_TRANSACTIONS property

NIFI-7403:Update for PR recheck

NIFI-7403:Add documentation on the Support Fragmented Transactions property to indicate the transactions rollback behavior

NIFI-7403: Fix Checkstyle issue

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4266
2020-06-01 11:33:10 -04:00
Bence Simon 66b175f405
NIFI-7390 Covering Avro type conversion in case of map withing Record
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4256
2020-05-08 13:11:19 -04:00
Yolanda M. Davis 7784178abd
NIFI-7408 - added percent used metrics for connections
NIFI-7408 - return double value for utilization

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4240
2020-04-29 13:11:06 -04:00
Matthew Burgess 1259bd5bd1
NIFI-7379: Support multiple instances of Prometheus registries/metrics (#4229)
* NIFI-7379: Support multiple instances of Prometheus registries/metrics

* NIFI-7379: Refactored Prometheus objects to support multiple instances
2020-04-28 16:56:49 -04:00
Matthew Burgess cd10435b9f
NIFI-7378: Ensure label values are not null in Prometheus metrics (#4219) 2020-04-22 10:37:04 -04:00
Bence Simon 923a07a5db NIFI-7300 Allowing narrow numeric types to fit againt schema check with wider type; Allowing doubles with value within float precision to be considered as valid floats (NIFI-7302) 2020-04-20 11:19:21 -04:00
Alexandre Vautier b113a022e4
NIFI-7359 Fix parent id on process metrics for Prometheus
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4209
2020-04-15 20:27:44 -04:00
Matthew Burgess a093af2d42
NIFI-7273: Add flow metrics REST endpoint with for Prometheus scraping (#4156)
* NIFI-7273: Add flow metrics REST endpoint with for Prometheus scraping

* NIFI-7273: Changed method name, fix handling when analytics not enabled

* NIFI-7273: Removed attachment header from Prometheus metrics endpoint

* NIFI-7273: Removed unused variable
2020-04-03 16:12:53 -04:00
Bryan Bende 5784888082
NIFI-7281 This closes #4159. Use BufferedInputStream in StandardSocketChannelRecordReader in order to support mark/reset
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-03-25 17:39:44 -04:00
Bryan Bende 2feeb57159
NIFI-7221 Support v2 and v3 protocol version for Hortonworks Schema Registry
- Update nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/serialization/SchemaRegistryRecordSetWriter.java
- Addressing review feedback

This closes #4120.
2020-03-20 10:56:47 -04:00
Pierre Villard 1fe79021b5
NIFI-7221 Initial work 2020-03-20 10:56:32 -04:00
Joe Witt f694e6464f NIFI-7187 adding missing version strings from accumulo bundle pom
- Removed Cat X JSON.org dep inclusion which seems to not be necessary
- Updated a ton of easier/safer looking deps
- Updated tika due to CVE

This closes #4086

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-03-20 10:07:56 -04:00
Matthew Burgess ace5754524 NIFI-7208: Restore default timezone in JdbcCommon 2020-03-13 15:00:24 -04:00
Matthew Burgess 798a8eeb50
NIFI-7249: Force String keys in maps in DataTypeUtils.inferDataType()
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4139.
2020-03-13 17:20:45 +01:00
Matthew Burgess 74b1b2fc59 NIFI-7208: Fixed PutSQL/JdbcCommon handling of timestamps (nanoseconds, e.g.) 2020-03-04 11:46:47 -05:00
jstorck 1678531638
NIFI-7025: Initial commit adding Kerberos Password feature for Hive components
Kerberos Password property should not support EL, this includes a change to KerberosProperties which is also used by the HDFS processors (AbstractHadoopProcessor)
Added wiring in a KerberosContext to a TestRunner's MockProcessorInitializationContext
Removed synchronization blocks around KerberosUser.checkTGTAndRelogin, since that method is already synchronized
Updated AbstractHadoopProcessor to have a boolean accessor method to determine if explicit keytab configuration is allowed
Removed synchronization block from HiveConnectionPool's getConnection method (in Hive, Hive_1_1, Hive3 modules), since new TGT ticket acquisition is handled by the KerberosUser implementation.  If UGI is used to relogin, synchronization is handled internally by UGI.
Added Kerberos Principal and Kerberos Password properties to Hive, Hive_1_1, and Hive3 components
Hive, Hive_1_1, and Hive3 components now use KerberosUser implementations to authenticate with a KDC

Updated handling of the NIFI_ALLOW_EXPLICIT_KEYTAB environment variable in Hive and Hive3 components.  An accessor method has been added that uses Boolean.parseBoolean, which returns true if the environment variable is set to true, and false otherwise (including when the environment variable is unset).

Addressing PR feedback

Addressing PR feedback

This closes #4102.
2020-03-02 11:28:59 -05:00
jstorck 614136ce51
NIFI-7018: Initial commit of processors extending AbstractHadoopProcessor supporting kerberos passwords
AbstractHadoopProcessor will always authenticate the principal with a KerberosUser implementation and a UGI will be acquired from the Subject associated with the KerberosUser implementation
AbstractHadoopProcessor's getUserGroupInformation method will now attempt to check the TGT and relogin if a KerberosUser impelmentation is available, otherwise it will return the UGI referenced in the HdfsResource instance
Updated AbstractHadoopProcessor's customValidate method to consider the provided password and updated validation failure explanations when a KerberosCredentialsService is specified together with a principal, password, or keytab
Added toString method override to AbstractKerberosUser
Updated Hive/HBase components to be compatible with the KerberosProperties.validatePrincipalWithKeytabOrPassword method
Fixed null ComponentLog in GetHDFSSequenceFileTest

Added package-protected accessor method (getAllowExplicitKeytabEnvironmentVariable) to AbstractHadoopProcessor for determining if the environment variable "NIFI_ALLOW_EXPLICIT_KEYTAB" has been set
AbstractHadoopProcessor will now only fail validation when the NIFI_ALLOW_EXPLICIT_KEYTAB environment variable is set to false if a keytab is provided to allow the user to specify a principal and password
Added AbstractHadoopProcessorSpec to verify validation of principal/keytab/password/kerberos credential service combinations

This closes #4095.
2020-02-28 10:10:19 -05:00
Joe Witt 3de77ebacc
NIFI-7021-RC3 prepare for next development iteration 2020-01-19 14:14:40 -05:00
Joe Witt 633408bce7
NIFI-7021-RC3 prepare release nifi-1.11.0-RC3 2020-01-19 14:14:38 -05:00
Joe Witt 23c8234586
NIFI-7031 updating copyright year on NOTICES 2020-01-15 16:10:31 -05:00
Tamas Palfy b7fb94723c NIFI-6884 - Native library loading fixed/improved: NarClassLoader and InstanceClassLoader can load libraries from their own or their ancestors' NAR-INF/bundled-dependencies/native directory.
They also scan directories defined via java.library.path system property.
InstanceClassLoader also checks additional classpath resources defined by PropertyDescriptors with "dynamicallyModifiesClasspath(true)".
Added tests for loading native libraries. Supports mac only.
Added support for loading native libs from additional resources in AbstractHadoopProcessor.
Updated javadoc for PropertyDescriptor.dynamicallyModifiesClasspath.

This closes #3894.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-01-13 13:59:11 -05:00
Matthew Burgess f4ef45678e
NIFI-6963: Fix ValidateRecord handling of missing required array fields
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3948.
2020-01-08 02:02:04 +07:00
KovalevIV 8f9fc49ca0 NIFI-6955 close streams in Reader/writer mock 2019-12-19 22:26:31 +03:00
Pierre Villard ac5bacccb8
NIFI-6839 - Upgrade jackson-databind direct dependencies
This closes #3870

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-11-25 10:58:22 -05:00
Bryan Bende 5cfc68d26d
NIFI-6896 Standardize hadoop.version for everything that uses nifi-hadoop-libraries
NIFI-6896 Adding hadoop-aws, hadoop-azure, and hadoop-azure-datalake to nifi-hadoop-libraries-nar and nifi-ranger-nar so other filesystem impls will be available

NIFI-6896 Introduce profiles to optionally include additional hadoop modules to reduce default build size

NIFI-6896 Updating licenses and notices to account for new optional dependencies

NIFI-6896 Add profile for hadoop-cloud-storage

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3900
2019-11-22 11:14:46 -05:00
Koji Kawamura b3880a4a06
NIFI-5970 Handle multiple input FlowFiles at Put.initConnection
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3583
2019-11-19 16:48:14 -05:00
Atul Mahajan e3f72c5ed0
NIFI-6771 : Added fix to support Oracle Binary float and Binary double conversion to Avro type
Fixed CheckStyle errors

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3815
2019-11-06 08:57:31 -05:00
Atul Mahajan 7ac41576e3
NIFI-6775 Added fix to support Oracle Float type conversion to Avro
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3812
2019-11-05 16:52:20 -05:00
Tamas Palfy 84a05c8595 NIFI-6729 - Created AbstractSingleAttributeBasedControllerServiceLookup and updated DBCPConnectionPoolLookup and AzureStorageCredentialsControllerServiceLookup to inherit from it.
This closes #3774.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2019-11-05 14:02:03 +01:00
Joe Witt f8c3d877cf
NIFI-6733 updating to next release version for master branch 2019-11-04 13:31:39 -05:00
Joe Witt 418179f5b2
NIFI-6733-RC3 prepare for next development iteration 2019-10-28 15:13:13 -07:00
Joe Witt b217ae20ad
NIFI-6733-RC3 prepare release nifi-1.10.0-RC3 2019-10-28 15:12:57 -07:00
Jan Hentschel ccf85777c4
NIFI-6816 Removed duplicated dependency declarations
This closes #3851

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-10-28 12:49:08 -04:00
Joe Witt 0f02de6002
NIFI-6733 updating key apache commons dependencies and apache base dependency for build
This closes #3791.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2019-10-07 22:20:48 -04:00
Tamas Palfy 34112519c2 NIFI-6640 - UNION/CHOICE types not handled correctly
3 important changes:
1. FieldTypeInference had a bug when dealing with multiple datatypes for
 the same field where some (but not all) were in a wider-than-the-other
 relationship.
 Before: Some datatypes could be lost. String was wider than any other.
 After: Consistent behaviour. String is NOT wider than any other.
2. Choosing a datatype for a value from a ChoiceDataType:
 Before it chose the first compatible datatype as the basis of conversion.
 After change it tries to find the most suitable datatype.
3. Conversion of a value of avro union type:
 Before it chose the first compatible datatype as the basis of conversion.
 After change it tries to find the most suitable datatype.

Change: In the RecordFieldType enum moved TIMESTAMP ahead of DATE.

This closes #3724.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-09-19 11:02:30 -04:00
Bryan Bende ff6a7d9561
NIFI-5816 This closes #3726. Remove unused SFTP classes that reference Jsch
NIFI-5816 Switching SFTP processors from JSCH to SSHJ
NIFI-5816 LICENSE/NOTICIE updates to reflect changing from JSch to SSHJ

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-19 10:58:53 -04:00
archon 8a8852e73d
NIFI-6480: PutORC/PutParquet can't overwrite file even if set 'Overwrite Files' to true
This closes #3599.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-13 12:08:27 -04:00
Bryan Bende 76c2c3fee2
NIFI-6089 Add Parquet record reader and writer
NIFI-5755 Allow PutParquet processor to specify avro write configuration
Review feedback
Additional review feedback

This closes #3679

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-09-04 08:52:17 -04:00
Yolanda Davis 07baf723f7 NIFI-6429 - Added optional field to allow variables with null values to be included in reporting records. Default functionality will exclude variables with null values for backwards compatibility.
NIFI-6429 - checkstyle corrections
NIFI-6429 - change property descriptor name
2019-08-30 13:38:38 -04:00
Bryan Bende cc88dd428f
NIFI-6158 Fix conversion of Avro fixed type with logicalType decimal
This closes #3665.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-08-26 10:57:28 +09:00
Mark Payne bcf373a049
NIFI-6382: Allow Parameters to have null values
Signed-off-by: Bryan Bende <bbende@apache.org>
2019-08-16 16:47:27 -04:00
Jeff Storck 33187f8058 NIFI-5176 NiFI builds on Java 11
Added "jigsaw" profile to multiple modules, which auto-activates when building with Java 11 and adds several dependencies: jaxb, activation, and annotation-api.
Updated SslContextFactory to return tuple of socket factory with trust manager for issue with okhttp api changes in java 9+
Updated TestGet/PostHTTPGroovy to use default SSL context to get default cipher suites
Updated StandardTemplateDAOSpec.groovy, was using a 37-character UUID, shortened to 36 characters.
Multiple tests updated to specifically use TLSv1.2, since two-way TLSv1.3 for some component tests fail during the Java 11 build.  Needs more investigation.
Replaced GString with String concatenation for map key's value in ScriptedReportingTaskGroovyTest to avoid runtime error of casting GString to String when retrieved from a map that is storing the GString
Removed nifi-toolkit-api plugin configuration of maven-compiler-plugin, it is inherited
Resolved deprecation errors during Groovy compile for bouncycastle, Extension should be used in place of X509Extension
Fixed JNAOverridingJUnitRunner's creation of the classpath for the custom classloader created to be able to mock jna classes
Removed import of IOUtils (from the wrong package) from InferenceSchemaStrategy
Updated maven-compiler-plugin version to 3.8.1
TLS (default in Java 11 is TLSv1.3) working for Site-to-Site client tests after upgrading JDK installation to JDK 11.0.3, httpclient5 5.0-beta4, and httpasyncclient 4.1.4
HttpNotificationService updated to find and use the first found X509TrustManager rather than casting directly from the array of TrustManagers returned from SslContextFactory
  Removed unnecessary throws declaration from getSslSocketFactory method
Replaced DnsContextFactory.class.getName() with a string to avoid having to export/open the jdk.naming.dns module
Updated TestGetIgniteCache and TestPutIgniteCache.java to skip tests on Java 11+ (via Assume), and noted the check should be removed once a version of Ignite is released that supports Java 11
Updated SpringContextProcessor to create proper ClassLoader and uses URLClassloader.getResource() instead of URLClassloader.findResource() in isConfigResolvable.
  Due to the application classloader no longer being a URLClassLoader in Java 9+, the URLClassLoader created in isConfigResolvable no longer has explicit access to the parent's resources as URLs.
  URLClassLoader.getResource() searches the parent classloaders, and must be used instead of URLClassLoader.findResource() which only searches the URLs in the URLClassLoader and does not search the parent classloaders.

NIFI-5176 Moved exclusion of jdk.tools to the jigsaw profile in the POMs of nifi-hbase_1_1_2-client-service and nifi-hbase_2-client-service

NIFI-5176 Updated site-to-site-client's POM to use properties to establish the correct httpclient dependency for when building with Java 11

This closes #3404.
2019-08-13 18:41:39 -04:00
Jeff Storck b938fc20a3
NIFI-6479 Fixes timezone issues in TestJdbcCommon
Dates calculated for method testConvertToAvroStreamForDateTimeAsLogicalType now handle timezone correctly
Added debug logging of date/time calculations and expectations

This closes #3630.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-08-09 16:29:40 +09:00
Peter Turcsanyi c6e6a418aa
NIFI-6318: Support EL in CSV formatting properties
CSVReader and CSVRecordSetWriter controller services and also ConvertExcelToCSVProcessor
support EL for Value Separator, Quote Character and Escape Character properties.

NIFI-6318: Fixed null checks and compound OR expression.

NIFI-6318: RecordSetWriterFactory.createWriter() changes.

NIFI-6318: Initialize CSVFormat in onEnabled() if there are no dynamic formatting properties.

NIFI-6318: Comment Marker supports EL.

NIFI-6318: Various review changes.

This closes #3504.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-07-25 17:04:22 +09:00
archon 3bfef3356e
NIFI-6442: ExecuteSQL/ExecuteSQLRecord convert to Avro date type incorrectly when set 'Use Avro Logical Types' to true
This closes #3584.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-07-19 14:47:07 +09:00
mike 3fb4454375
NIFI-6443 - ParseSyslog5424 n-th SD-ELEMENT isn't parsed
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3588.
2019-07-18 16:01:32 +02:00