Commit Graph

5995 Commits

Author SHA1 Message Date
Matthew Burgess 798a8eeb50
NIFI-7249: Force String keys in maps in DataTypeUtils.inferDataType()
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4139.
2020-03-13 17:20:45 +01:00
Pierre Villard dc95cff3b7
NIFI-7229 - Upgrade jackson-databind direct dependencies
This closes #4113
2020-03-13 12:02:24 -04:00
Kotaro Terada dbda8c7f7e
NIFI-7251: Upgrade hadoop-client version to 3.2.1 to avoid the regression bug
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4141.
2020-03-13 10:07:34 +01:00
Adam Taft decb5d062e
NIFI-6742 Use JUnit TemporaryFolder when creating test databases
- Add @Rule for TemporaryFolder
- Replace use of previous target/db with TemporaryFolder/db
- Remove use of ~/test db (in home directory)
- Remove System.out lines

Signed-off-by: Marc Parisi <phrocker@apache.org>

This closes #4137.
2020-03-12 20:48:52 -04:00
Joe Witt 97e250cdaa
NIFI-7244 Updated all tests which dont run well on windows to either work or be ignored on windows
Also dealt with unreliable tests which depend on timing by ignoring them or converting to IT.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4132.
2020-03-12 19:13:59 +01:00
Peter Turcsanyi dfaef38805
NIFI-7248: Atlas reporting task handles PutHive3Streaming
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4138.
2020-03-12 10:44:48 +01:00
Mark Payne d4a2afc25c
NIFI-7241: When updating Process Group to match VersionedProcessGroup, remove any connections before recursing into child groups. This ensures that if a Port exists in child group A and is connected to a port in child group B, if the VersionedProcessGroup indicates to remove the port, that connection will be removed before attempting to remove the port. Updating and adding connections must still be done last, after all components have been added. But missing connections can be removed earlier.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4136.
2020-03-12 10:25:04 +01:00
M Tien 290bd378d5
NIFI-7119 Implement boundary checking for Argon2 cost parameters (#4111)
* NIFI-7119 Implemented parameter boundary enforcement for Argon2SecureHasher constructor.
Added unit tests for validating each parameter check.

* NIFI-7119 Refactored parameter validations. Added more test sizes to boundary checkers. Changed logger severity to error and added bounds to messages.

* NIFI-7119 Refactored Argon2 parameter data types to handle unsigned integer boundary values.
Updated unit tests.

Co-authored-by: Andy LoPresto <alopresto@apache.org>

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-03-11 15:51:15 -07:00
Mark Payne d68720920f
NIFI-7242: When a Parameter is changed, any property referencing that parameter should have its #onPropertyModified method called. Also renamed Accumulo tests to integration tests because they start embedded servers and connect to them, which caused failures in my environment. Also fixed a bug in TestLengthDelimitedJournal because it was resulting in failures when building locally as well.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4134.
2020-03-11 21:00:43 +01:00
Mark Payne 67676ba5b4 Fixed bug in JsonRowRecordReader when reading a 'raw' record with a schema that indicates that a field should be a Map. Also updated unit test to explicitly define schema, since schema inference will never return a Map but rather a Record 2020-03-11 14:29:44 -04:00
Pierre Villard cd83e70b91 unit test reproducing the issue 2020-03-11 14:29:44 -04:00
Peter Turcsanyi 3de3ad4029
NIFI-7245: JMS processors yield when connection factory initialisation failed
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4133.
2020-03-11 19:24:10 +01:00
Gardella Juan Pablo 89d8b877f9 NIFI-7050 ConsumeJMS is not yielded in case of exception
This closes #4004.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-03-11 10:54:20 +01:00
Peter Turcsanyi 9cb85a4fd0
NIFI-7239: Upgrade the Hive 3 bundle to use Apache Hive 3.1.2
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4129.
2020-03-11 10:26:10 +01:00
Pierre Villard a679e88b6f
NIFI-4970 - Add a property to deal with empty GZIP HTTP response
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4127.
2020-03-11 10:08:51 +01:00
Mark Payne afad982e91
NIFI-7200: Revert "NIFI-6530 - HTTP SiteToSite server returns 201 in case no data is available"
This reverts commit f01668e66a.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-03-10 14:32:43 -04:00
Andrew Lim 7b0ae56a2c
NIFI-7179 Documented Download flow option in Process group context menu (#4124)
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-03-09 16:54:56 -07:00
Pierre Villard 12c8402ac3
NIFI-7210 - added PG path in bulletins for S2S Bulletin RT
Added group path to BULLETIN table for QueryNiFiReportingTask

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4100
2020-03-09 16:29:22 -04:00
Mike a33a5e35b3
Merge pull request #4119 from eduardofontes/patch-4
NIFI-7233 - Fix Mongo Processors to avoid cycle in provenance graph
2020-03-08 13:18:12 -04:00
Pierre Villard 0f775f3a57
NIFI-7195 - Catch MongoException to route flow files to failure
This closes #4089

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-03-08 08:10:54 -04:00
Bryan Bende ac4d52b6ca
NIFI-7224 Protecting against possible NPE in ImportFlowVersion command in CLI
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4121.
2020-03-07 15:21:31 +01:00
Pierre Villard 578430c9d9 NIFI-7197 - In-place replacement in LookupRecord processor
This closes #4088

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-03-06 15:35:59 -05:00
Mark Bean f4b65afb64 NIFI-7231: move controller service validation out of synchronized block for enabling
This closes #4118.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-03-06 15:01:19 -05:00
Eduardo Fontes 57947cd2a2
Fiz writeBatch to avoid cycle in provenance graph 2020-03-06 16:06:21 -03:00
Kevin Doran cca54f7ff2 NIFI-7191 Conditionally disable docker integration tests
Honor Maven properties skipTests and maven.test.skip for the
shell script integration tests that verifiy docker images.
2020-03-06 13:34:23 -05:00
Mark Payne f283c1191c
Fixed unit test failed with NIFI-7232
This closes #4117.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-03-06 12:46:17 -05:00
Joe Witt 20dda05f26
NIFI-7232 if users do not supply a remote path PutSFTP with conflict resolution will fail with an NPE 2020-03-06 12:46:02 -05:00
Peter Turcsanyi 7c57e75da4 NIFI-7226: Add Connection Factory configuration properties to PublishJMS and ConsumeJMS processors
Some JMS client libraries may not work with the existing controller services due to incompatible
classloader handling between the 3rd party library and NiFi.
Via configuring the Connection Factory on the processor itself, only the processor's and its
children's classloaders will be used which eliminates the mentioned incompatibility.

This closes #4110.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-03-06 09:51:37 -05:00
Joe Witt 040c8a0af9
NIFI-7222 Cleaned up API for FTP/SFTP remote file retrieval and ensure we close remote file resources for SFTP pulls in particular
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4115.
2020-03-06 12:48:15 +01:00
Matthew Burgess b82fec41d9
NIFI-7055: Removed unit test that is now covered by ListValidator
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4114.
2020-03-05 11:01:48 +01:00
Otto Fowler f1c6e92df5
NIFI-7055 handle empty split evaluations, which contain only ,
add explict test for " , "

updated with counting validator

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4012
2020-03-04 18:34:18 -05:00
Andy LoPresto 7773681eea
NIFI-7227 Fixed typo in Global Access Policy table (#4112)
Co-authored-by: spius <57421336+spius@users.noreply.github.com>

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-03-04 12:59:05 -08:00
Matthew Burgess 74b1b2fc59 NIFI-7208: Fixed PutSQL/JdbcCommon handling of timestamps (nanoseconds, e.g.) 2020-03-04 11:46:47 -05:00
jstorck 4b6de8d164
NIFI-7025: Wrap Hive 3 calls with UGI.doAs
Updated PutHive3Streaming to wrap calls to Hive in UGI.doAs methods
Fixed misleading logging message after the principal has been authenticated with the KDC
When connecting to unsecured Hive 3, a UGI with "simple" auth will be used

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4108
2020-03-04 11:21:27 -05:00
Eduardo Fontes bad254ec5b
NIFI-4970 - EOF Exception in InvokeHttp when body's response is empty with gzip
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4109.
2020-03-04 10:36:06 +01:00
M Tien 0b2816baa4
NIFI-7121 Updated comment to state a 'static' salt is used in the constructor. (#4098)
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-03-03 15:50:49 -08:00
Andy LoPresto cb08382da4
NIFI-7218 Fixed typo in Overview docs. (#4107) 2020-03-03 15:47:42 -08:00
karthik.kadajji 2cf4fde686
NIFI-5644 Fixed typo in getWrappedQuery method of AbstractDatabaseFetchProcessor class
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4106.
2020-03-03 16:43:56 +01:00
Bryan Bende 852d2f3210 NIFI-7030 Add Kerberos principal and password properties to Solr processors
Updating validation logic to be consistent with other password-based kerberos processors, removing getPassword from KerberosPasswordUser as it was only used from testing

This closes #4062.
2020-03-02 16:01:39 -05:00
Bryan Bende e0fc75a963 NIFI-7019 Add kerberos principal and password properties to NiFi DBPCConnectionPool
This closes #4087.
2020-03-02 14:20:20 -05:00
Bryan Bende 5b93e537d5 NIFI-7029 Add kerberos password property to NiFi Kudu components
This closes #4097.
2020-03-02 14:03:37 -05:00
Jeff Storck c74e71306e
Merge pull request #4071 from bbende/NIFI-7026
NIFI-7026 Add kerberos password property to NiFi HortonworksSchemaReg…
2020-03-02 12:50:20 -05:00
jstorck ab1658130d
NIFI-7024: Added Kerberos Password support to HBase_1_1_2_ClientService and HBase_2_ClientService
This closes #4103.
2020-03-02 11:43:13 -05:00
jstorck 1678531638
NIFI-7025: Initial commit adding Kerberos Password feature for Hive components
Kerberos Password property should not support EL, this includes a change to KerberosProperties which is also used by the HDFS processors (AbstractHadoopProcessor)
Added wiring in a KerberosContext to a TestRunner's MockProcessorInitializationContext
Removed synchronization blocks around KerberosUser.checkTGTAndRelogin, since that method is already synchronized
Updated AbstractHadoopProcessor to have a boolean accessor method to determine if explicit keytab configuration is allowed
Removed synchronization block from HiveConnectionPool's getConnection method (in Hive, Hive_1_1, Hive3 modules), since new TGT ticket acquisition is handled by the KerberosUser implementation.  If UGI is used to relogin, synchronization is handled internally by UGI.
Added Kerberos Principal and Kerberos Password properties to Hive, Hive_1_1, and Hive3 components
Hive, Hive_1_1, and Hive3 components now use KerberosUser implementations to authenticate with a KDC

Updated handling of the NIFI_ALLOW_EXPLICIT_KEYTAB environment variable in Hive and Hive3 components.  An accessor method has been added that uses Boolean.parseBoolean, which returns true if the environment variable is set to true, and false otherwise (including when the environment variable is unset).

Addressing PR feedback

Addressing PR feedback

This closes #4102.
2020-03-02 11:28:59 -05:00
jstorck 614136ce51
NIFI-7018: Initial commit of processors extending AbstractHadoopProcessor supporting kerberos passwords
AbstractHadoopProcessor will always authenticate the principal with a KerberosUser implementation and a UGI will be acquired from the Subject associated with the KerberosUser implementation
AbstractHadoopProcessor's getUserGroupInformation method will now attempt to check the TGT and relogin if a KerberosUser impelmentation is available, otherwise it will return the UGI referenced in the HdfsResource instance
Updated AbstractHadoopProcessor's customValidate method to consider the provided password and updated validation failure explanations when a KerberosCredentialsService is specified together with a principal, password, or keytab
Added toString method override to AbstractKerberosUser
Updated Hive/HBase components to be compatible with the KerberosProperties.validatePrincipalWithKeytabOrPassword method
Fixed null ComponentLog in GetHDFSSequenceFileTest

Added package-protected accessor method (getAllowExplicitKeytabEnvironmentVariable) to AbstractHadoopProcessor for determining if the environment variable "NIFI_ALLOW_EXPLICIT_KEYTAB" has been set
AbstractHadoopProcessor will now only fail validation when the NIFI_ALLOW_EXPLICIT_KEYTAB environment variable is set to false if a keytab is provided to allow the user to specify a principal and password
Added AbstractHadoopProcessorSpec to verify validation of principal/keytab/password/kerberos credential service combinations

This closes #4095.
2020-02-28 10:10:19 -05:00
Joey Frazee ddfc425d98 Bump commons-lang3 and commons-codec versions 2020-02-27 14:57:55 -06:00
Joey Frazee ccb62826b7 NIFI-6791 Add UUID3 and UUID5 functions to Expression Language
This closes #4031
2020-02-27 14:57:55 -06:00
Justin Miller 23b04ae968
NIFI-6856 - Make client ID a non-required field for the MQTTConsume and MQTTProduce processors. Generates a
random ID if not set.

Also add group ID field to ConsumeMQTT processor. Allows consumer to join consumer group at $share/<group_id>/<topic_filter>

add expression language support for the MQTT client ID

Setting client id in publish test fails because it is not a flowfile attribute.
Remove client id and autogenerate it when testing.

Since the evaluation is done in onScheduled, there is no flow file available and we're not using the attributes to make the expression language evaluation. You can change the scope to use the Variable Registry.

Co-Authored-By: Pierre Villard <pierre.villard.fr@gmail.com>
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3879.
2020-02-26 15:54:53 -08:00
Joe Witt 4cd63c99e8
NIFI-7205 NIFI-7206
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4093.
2020-02-26 14:02:49 -08:00
Shawn Weeks 0bb8ce7438
NIFI-7139 Add record.error.message on failure of a record reader or writer
Handle scenario where message might be null.

Update to test case that was failing because adding attributes modified a flow file even if you don't change the contents.

Fixed Style Issues and Updated WritesAttributes.

Added Test Case for Error Message

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4052
2020-02-26 08:50:32 -05:00