NIFI-7250 fix a test which appears brittle at least on windows builds on slow environments
NIFI-7250 activated a timezone run for AU Australia/Melbourne which exposed a poor magic number and needless assertion but interesting results worth keeping
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4140.
- Add @Rule for TemporaryFolder
- Replace use of previous target/db with TemporaryFolder/db
- Remove use of ~/test db (in home directory)
- Remove System.out lines
Signed-off-by: Marc Parisi <phrocker@apache.org>
This closes#4137.
Also dealt with unreliable tests which depend on timing by ignoring them or converting to IT.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4132.
Some JMS client libraries may not work with the existing controller services due to incompatible
classloader handling between the 3rd party library and NiFi.
Via configuring the Connection Factory on the processor itself, only the processor's and its
children's classloaders will be used which eliminates the mentioned incompatibility.
This closes#4110.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Updated PutHive3Streaming to wrap calls to Hive in UGI.doAs methods
Fixed misleading logging message after the principal has been authenticated with the KDC
When connecting to unsecured Hive 3, a UGI with "simple" auth will be used
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4108
Updating validation logic to be consistent with other password-based kerberos processors, removing getPassword from KerberosPasswordUser as it was only used from testing
This closes#4062.
Kerberos Password property should not support EL, this includes a change to KerberosProperties which is also used by the HDFS processors (AbstractHadoopProcessor)
Added wiring in a KerberosContext to a TestRunner's MockProcessorInitializationContext
Removed synchronization blocks around KerberosUser.checkTGTAndRelogin, since that method is already synchronized
Updated AbstractHadoopProcessor to have a boolean accessor method to determine if explicit keytab configuration is allowed
Removed synchronization block from HiveConnectionPool's getConnection method (in Hive, Hive_1_1, Hive3 modules), since new TGT ticket acquisition is handled by the KerberosUser implementation. If UGI is used to relogin, synchronization is handled internally by UGI.
Added Kerberos Principal and Kerberos Password properties to Hive, Hive_1_1, and Hive3 components
Hive, Hive_1_1, and Hive3 components now use KerberosUser implementations to authenticate with a KDC
Updated handling of the NIFI_ALLOW_EXPLICIT_KEYTAB environment variable in Hive and Hive3 components. An accessor method has been added that uses Boolean.parseBoolean, which returns true if the environment variable is set to true, and false otherwise (including when the environment variable is unset).
Addressing PR feedback
Addressing PR feedback
This closes#4102.
AbstractHadoopProcessor will always authenticate the principal with a KerberosUser implementation and a UGI will be acquired from the Subject associated with the KerberosUser implementation
AbstractHadoopProcessor's getUserGroupInformation method will now attempt to check the TGT and relogin if a KerberosUser impelmentation is available, otherwise it will return the UGI referenced in the HdfsResource instance
Updated AbstractHadoopProcessor's customValidate method to consider the provided password and updated validation failure explanations when a KerberosCredentialsService is specified together with a principal, password, or keytab
Added toString method override to AbstractKerberosUser
Updated Hive/HBase components to be compatible with the KerberosProperties.validatePrincipalWithKeytabOrPassword method
Fixed null ComponentLog in GetHDFSSequenceFileTest
Added package-protected accessor method (getAllowExplicitKeytabEnvironmentVariable) to AbstractHadoopProcessor for determining if the environment variable "NIFI_ALLOW_EXPLICIT_KEYTAB" has been set
AbstractHadoopProcessor will now only fail validation when the NIFI_ALLOW_EXPLICIT_KEYTAB environment variable is set to false if a keytab is provided to allow the user to specify a principal and password
Added AbstractHadoopProcessorSpec to verify validation of principal/keytab/password/kerberos credential service combinations
This closes#4095.
random ID if not set.
Also add group ID field to ConsumeMQTT processor. Allows consumer to join consumer group at $share/<group_id>/<topic_filter>
add expression language support for the MQTT client ID
Setting client id in publish test fails because it is not a flowfile attribute.
Remove client id and autogenerate it when testing.
Since the evaluation is done in onScheduled, there is no flow file available and we're not using the attributes to make the expression language evaluation. You can change the scope to use the Variable Registry.
Co-Authored-By: Pierre Villard <pierre.villard.fr@gmail.com>
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#3879.
Handle scenario where message might be null.
Update to test case that was failing because adding attributes modified a flow file even if you don't change the contents.
Fixed Style Issues and Updated WritesAttributes.
Added Test Case for Error Message
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4052
JsonContentConversionExceptionMapper, JsonMappingExceptionMapper, JsonParseExceptionMapper.
Registered the custom ExceptionMappers.
Added unit tests to throw Exception for string port value and sanitize script input. Handled null or empty JsonMappingException reference path.
Added the Apache license to Groovy Test.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
NIFI-6363 Additional fixes.
NIFI-6363 Fix Hadoop compile problem. Add GCP IT instructions.
NIFI-6363 - Removed GCP provider due to dependency conflicts with GRPC processors. Fixed unit test to match master branch after rebase.
NIFI-6363 - Added some docs and experimental tag to the relevant classes.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4080.
NIFI-7007: Add additional unit tests that hit non-happy path
NIFI-7007: Use AllowableValue instead of string
NIFI-7007: Add the use of attributes for the update method, statement type and batch statement type
NIFI-7007: Add additional tests, mainly for the use of attributes
NIFI-7007: add some ReadsAttribute properties to the PutCassandraRecord processor
NIFI-7007: additional update keys validation logic
NIFI-7007: fix imports
NIFI-7007: Convert fieldValue to long in separate method
NIFI-7007: Add new style of tests checking actual CQL output
NIFI-7007: add license to new test file
NIFI-7007: add customValidate to check for certain incompatible property combinations
NIFI-7007: remove check on updateMethod and replace Set.of with java 8 compatible replacmenet
NIFI-7007: Add test for failure with empty update method via attributes
NIFI-7007: remove unused variable
NIFI-7007: Fix customValidate that incorrectly invalidated a valid config
Fix Checkstyle
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3977
Uses `DataTypeUtils.toTimestamp` when writing to Kudu
timestamp (`UNIXTIME_MICROS`) columns. This allows
us to use the `row.addTimestamp` API and get much more
intuitive and predictable timestamp ingest behavior.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4054.
This patch adds validation to the PutKudu operation type property.
It also improves the description to include the valid values and
adjusts the inputs to be case insensitive.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4063.
Adds a boolean property to the PutKudu processor to optionally
enable automatic schema drift handling.
If set to true, when fields with names that are not in the target
Kudu table are encountered, the Kudu table will be altered to
include new columns for those fields.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4053.
- decoupled flow update request behavior from VersionsResource into new abstract FlowUpdateResource
- added replace process group functionality in ProcessGroupResource
- parameterized FlowUpdateResource and created entity hierarchies to allow for maximum code sharing across different update types
- refactored flow update methods to make use of commonality across different update types whenever possible
- fixed issues in StandardProcessGroup verify update methods where same components existed in different ancestry chains but were considered a match when they shouldn't be
- improved StandardProcessGroup to properly match up components on update using generated versioned component ids, when necessary to allow for update flow to efficiently match common components on flow import
This closes#4023.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Work in progress adding support for array based queries
updated calcite dependency
tidy up unused imports highlighted by checkstyle in travis build
tidy up }s highlighted by checkstyle in travis build
Add test for use case referenced in NIFI-4792
Bumped Calcite version to 1.21.0
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4015
Add two new properties to IdentifyMimeType (Config File and Config Body).
Specifying one of these properties will override the default NiFi MIME
type configuration and use the configured property's MIME config instead.
Add additional runtime documentation for IdentifyMimeType's usage. The
default behavior of IdentifyMimeType is unchanged.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4016.
NIFI-7051 Fixing issue where identity was being used instead of identifier, making a flag to control legacy id behavior, increasing timeout of shell command runner, and changing the NSS system check command to return less info
NIFI-7051 Updating command for getSystemCheck in NSS impl to use getent --version to improve performance
This closes#4003.
Revert NIFI-4834 enhancement for durable non shared consumers only.
Updated also AbstractJMSProcessor class to be public. The testing are not working
without that change, as org.apache.nifi.jms.processors.PublishJMSIT and
org.apache.nifi.jms.processors.ConsumeJMSIT are not working, as @OnSchedule
method is not called (because it is not public).
The method org.apache.nifi.util.StandardProcessorTestRunner.run(int iterations, boolean stopOnFinish, boolean initialize, long runWait) uses ReflectionUtils.invokeMethodsWithAnnotation which does not call non public
methods.
Signed-off-by: Joe Witt <joewitt@apache.org>
Added no-password keystore for tests
System NiFi truststore now allows a passwordless truststore. Added a unit test to prove this.
Forgot no-password-truststore.jks file for the unit test.
Refactored utility method from CertificateUtils to KeyStoreUtils.
Added utility methods to verify keystore and key passwords.
Added unit tests.
Implemented different keystore and truststore validation logic.
Refactored internal custom validation in StandardSSLContextService.
Added unit test resource for keystore with different key and keystore passwords.
Added unit test to generate passwordless truststore for https://nifi.apache.org for live testing.
Resolved NPE in SSLContext generation in StandardSSLContextService
Added unit test to generate passwordless truststore for localhost for InvokeHTTP testing.
Resolved TrustManagerFactoryImpl initialization error.
Fixed unit test without proper cleanup which caused RAT failures.
Co-authored-by: Andy LoPresto <alopresto@apache.org>
This closes#3823.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
NIFI-6919: Cleaned up docs
NIFI-6919: Cleanup
NIFI-6919: Cleanup
NIFI-6919: added negative unit test
NIFI-6919: Removed unnecesary feature flag
Updated attribute description
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3939
NIFI-7011: Addressed corner case where data could be inserted out of order still if added while swapping was taking place
NIFI-7011: Fixed ordering issue with swap queue that can occur if data is migrated from swap queue to active queue instead of being swapped out
They also scan directories defined via java.library.path system property.
InstanceClassLoader also checks additional classpath resources defined by PropertyDescriptors with "dynamicallyModifiesClasspath(true)".
Added tests for loading native libraries. Supports mac only.
Added support for loading native libs from additional resources in AbstractHadoopProcessor.
Updated javadoc for PropertyDescriptor.dynamicallyModifiesClasspath.
This closes#3894.
Signed-off-by: Mark Payne <markap14@hotmail.com>
NIFI-6992 - Add "Batch Size" property to GetHDFSFileInfo processor - Added validation for 'Batch Size' in 'GetHDFSFileInfo'.
NIFI-6992 - Changed 'GetHDFSFileInfo.BATCH_SIZE' validator from 'NON_NEGATIVE_INTEGER_VALIDATOR' to 'POSITIVE_INTEGER_VALIDATOR'. Added more tests.
NIFI-6992 - Removed 'AllowEmptyValidator'. 'Batch Size' in 'GetHDFSFileInfo' allows null but not empty String.
NIFI-6992 - 'Batch Size' in 'GetHDFSFileInfo' allows null but not empty String - cont.
NIFI-6992 - Fix: Unused import.
This closes#3966.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
Added EncryptedSchemaRepositoryRecordSerde.
Refactored CryptoUtils utility methods for repository encryption configuration validation checks to RepositoryEncryptorUtils.
Added FlowFile repo encryption config container.
Added more logging in cryptographic and serialization operations.
Generalized log messages in shared encryption services.
Added encrypted serde factory.
Added marker impl for encrypted WAL.
Moved validation of FF repo encryption config earlier in startup process.
Refactored duplicate property lookup code in NiFiProperties.
Added title case string helper.
Added validation and warning around misformatted encryption repo properties.
Added unit tests.
Added documentation to User Guide & Admin Guide.
Added screenshot for docs.
Added links to relevant sections of NiFi In-Depth doc to User Guide.
Added flowfile & content repository encryption configuration properties to default nifi.properties.
Signed-off-by: Joe Witt <joewitt@apache.org>
Signed-off-by: Mark Payne <markap14@hotmail.com>
This closes#3968.
Attribute peerPersistence can be null generating Bulletin WARNs "Unable to refresh Remote Group's peers due to null".
Rollback
The fix is inside site-to-site-reporting-task-bundle
Modify getClient()
Get ConfigurationContext and ReportingContext to provide a StateManager.
Modify OnScheduled setup()
The OnSchedule setup() now save the ConfigurationContext to lazily create a SiteToSiteClient with ReportingContext through an overloaded setup().
Modify OnTrigger
Lazily creates SiteToSiteClient to provide a StateManager
Modify OnTrigger
Lazily create SiteToSiteClient to provide a StateManager
Modify OnTrigger
Lazily create SiteToSiteClient to provide a StateManager
Modify OnTrigger
Lazily create SiteToSiteClient to provide a StateManager
Retry compile
Fix maven-checkstyle-plugin
Fix maven-checkstyle-plugin
Fix maven-checkstyle-plugin
Fix maven-checkstyle-plugin
Update AbstractSiteToSiteReportingTask.java
Removed the OnSchedule setup(ConfigContext) because it is not needed.
Update SiteToSiteUtils.java
Removed ConfigContext from getClient parameters because ReportContext share the same properties.
Mark Payne
Joe Witt
Matthew Burgess
Adam Taft
Peter Turcsanyi
Pierre Villard
Gardella Juan Pablo
Mike
Mark Bean
Eduardo Fontes
jstorck
karthik.kadajji
Bryan Bende
Justin Miller
Shawn Weeks
Przemysław Dubaniewicz
Andy LoPresto
Yolanda M. Davis
Troy Melhase
MatthewKnight-NG
Wouter de Vries
Zoltan Kornel Torok
Grant Henke
M Tien
Joe Ferner
David Savage
John Highcock
nagasivanath
Matt Gilman
Andy LoPresto
mdayakar
mans2singh
Marc Parisi
Dirk Ernst van den Biggelaar
Gal Shinder
Tamas Palfy
Joe Gresock
Nathan Gough
bahlulh
wagggz
Michael Hogue
Tom
Martin Šimek
John Pierce
mtien
Ubuntu
r65535
KovalevIV