Uses `DataTypeUtils.toTimestamp` when writing to Kudu
timestamp (`UNIXTIME_MICROS`) columns. This allows
us to use the `row.addTimestamp` API and get much more
intuitive and predictable timestamp ingest behavior.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4054.
This patch adds validation to the PutKudu operation type property.
It also improves the description to include the valid values and
adjusts the inputs to be case insensitive.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4063.
Adds a boolean property to the PutKudu processor to optionally
enable automatic schema drift handling.
If set to true, when fields with names that are not in the target
Kudu table are encountered, the Kudu table will be altered to
include new columns for those fields.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4053.
- decoupled flow update request behavior from VersionsResource into new abstract FlowUpdateResource
- added replace process group functionality in ProcessGroupResource
- parameterized FlowUpdateResource and created entity hierarchies to allow for maximum code sharing across different update types
- refactored flow update methods to make use of commonality across different update types whenever possible
- fixed issues in StandardProcessGroup verify update methods where same components existed in different ancestry chains but were considered a match when they shouldn't be
- improved StandardProcessGroup to properly match up components on update using generated versioned component ids, when necessary to allow for update flow to efficiently match common components on flow import
This closes#4023.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Work in progress adding support for array based queries
updated calcite dependency
tidy up unused imports highlighted by checkstyle in travis build
tidy up }s highlighted by checkstyle in travis build
Add test for use case referenced in NIFI-4792
Bumped Calcite version to 1.21.0
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4015
Add two new properties to IdentifyMimeType (Config File and Config Body).
Specifying one of these properties will override the default NiFi MIME
type configuration and use the configured property's MIME config instead.
Add additional runtime documentation for IdentifyMimeType's usage. The
default behavior of IdentifyMimeType is unchanged.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4016.
NIFI-7051 Fixing issue where identity was being used instead of identifier, making a flag to control legacy id behavior, increasing timeout of shell command runner, and changing the NSS system check command to return less info
NIFI-7051 Updating command for getSystemCheck in NSS impl to use getent --version to improve performance
This closes#4003.
Revert NIFI-4834 enhancement for durable non shared consumers only.
Updated also AbstractJMSProcessor class to be public. The testing are not working
without that change, as org.apache.nifi.jms.processors.PublishJMSIT and
org.apache.nifi.jms.processors.ConsumeJMSIT are not working, as @OnSchedule
method is not called (because it is not public).
The method org.apache.nifi.util.StandardProcessorTestRunner.run(int iterations, boolean stopOnFinish, boolean initialize, long runWait) uses ReflectionUtils.invokeMethodsWithAnnotation which does not call non public
methods.
Signed-off-by: Joe Witt <joewitt@apache.org>
Added no-password keystore for tests
System NiFi truststore now allows a passwordless truststore. Added a unit test to prove this.
Forgot no-password-truststore.jks file for the unit test.
Refactored utility method from CertificateUtils to KeyStoreUtils.
Added utility methods to verify keystore and key passwords.
Added unit tests.
Implemented different keystore and truststore validation logic.
Refactored internal custom validation in StandardSSLContextService.
Added unit test resource for keystore with different key and keystore passwords.
Added unit test to generate passwordless truststore for https://nifi.apache.org for live testing.
Resolved NPE in SSLContext generation in StandardSSLContextService
Added unit test to generate passwordless truststore for localhost for InvokeHTTP testing.
Resolved TrustManagerFactoryImpl initialization error.
Fixed unit test without proper cleanup which caused RAT failures.
Co-authored-by: Andy LoPresto <alopresto@apache.org>
This closes#3823.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
NIFI-6919: Cleaned up docs
NIFI-6919: Cleanup
NIFI-6919: Cleanup
NIFI-6919: added negative unit test
NIFI-6919: Removed unnecesary feature flag
Updated attribute description
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3939
NIFI-7011: Addressed corner case where data could be inserted out of order still if added while swapping was taking place
NIFI-7011: Fixed ordering issue with swap queue that can occur if data is migrated from swap queue to active queue instead of being swapped out
They also scan directories defined via java.library.path system property.
InstanceClassLoader also checks additional classpath resources defined by PropertyDescriptors with "dynamicallyModifiesClasspath(true)".
Added tests for loading native libraries. Supports mac only.
Added support for loading native libs from additional resources in AbstractHadoopProcessor.
Updated javadoc for PropertyDescriptor.dynamicallyModifiesClasspath.
This closes#3894.
Signed-off-by: Mark Payne <markap14@hotmail.com>
NIFI-6992 - Add "Batch Size" property to GetHDFSFileInfo processor - Added validation for 'Batch Size' in 'GetHDFSFileInfo'.
NIFI-6992 - Changed 'GetHDFSFileInfo.BATCH_SIZE' validator from 'NON_NEGATIVE_INTEGER_VALIDATOR' to 'POSITIVE_INTEGER_VALIDATOR'. Added more tests.
NIFI-6992 - Removed 'AllowEmptyValidator'. 'Batch Size' in 'GetHDFSFileInfo' allows null but not empty String.
NIFI-6992 - 'Batch Size' in 'GetHDFSFileInfo' allows null but not empty String - cont.
NIFI-6992 - Fix: Unused import.
This closes#3966.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
Added EncryptedSchemaRepositoryRecordSerde.
Refactored CryptoUtils utility methods for repository encryption configuration validation checks to RepositoryEncryptorUtils.
Added FlowFile repo encryption config container.
Added more logging in cryptographic and serialization operations.
Generalized log messages in shared encryption services.
Added encrypted serde factory.
Added marker impl for encrypted WAL.
Moved validation of FF repo encryption config earlier in startup process.
Refactored duplicate property lookup code in NiFiProperties.
Added title case string helper.
Added validation and warning around misformatted encryption repo properties.
Added unit tests.
Added documentation to User Guide & Admin Guide.
Added screenshot for docs.
Added links to relevant sections of NiFi In-Depth doc to User Guide.
Added flowfile & content repository encryption configuration properties to default nifi.properties.
Signed-off-by: Joe Witt <joewitt@apache.org>
Signed-off-by: Mark Payne <markap14@hotmail.com>
This closes#3968.
Attribute peerPersistence can be null generating Bulletin WARNs "Unable to refresh Remote Group's peers due to null".
Rollback
The fix is inside site-to-site-reporting-task-bundle
Modify getClient()
Get ConfigurationContext and ReportingContext to provide a StateManager.
Modify OnScheduled setup()
The OnSchedule setup() now save the ConfigurationContext to lazily create a SiteToSiteClient with ReportingContext through an overloaded setup().
Modify OnTrigger
Lazily creates SiteToSiteClient to provide a StateManager
Modify OnTrigger
Lazily create SiteToSiteClient to provide a StateManager
Modify OnTrigger
Lazily create SiteToSiteClient to provide a StateManager
Modify OnTrigger
Lazily create SiteToSiteClient to provide a StateManager
Retry compile
Fix maven-checkstyle-plugin
Fix maven-checkstyle-plugin
Fix maven-checkstyle-plugin
Fix maven-checkstyle-plugin
Update AbstractSiteToSiteReportingTask.java
Removed the OnSchedule setup(ConfigContext) because it is not needed.
Update SiteToSiteUtils.java
Removed ConfigContext from getClient parameters because ReportContext share the same properties.
- Added UI versioned flow supportsDownload functionality with download flow menu item
- Added VersionsResource endpoint for downloading versioned flow with registry-related info removed
- Added ProcessGroupResource endpoint for downloading current flow with registry-related info removed
- Added StandardNifiServiceFacade functionality for downloading both current and versioned flow
- Added XmlTransient markers on variables introduced by Instantiated model classes so they do not appear in serialized download
- Updated NiFiRegistryFlowMapper.mapParameterContexts to handle mapping nested parameter contexts for use in producing a complete VersionedFlowSnapshot
- Added ability for NiFiRegistryFlowMapper to map nested process groups ignoring versioning for use in producing a complete VersionedFlowSnapshot
- Added unit tests where helpful
NIFI-6872: PR response...
- Updated mapParameterContext to return a Map to handle uniqueness of contexts by name since ultimately everything converted it to a map anyway. The VersionedParameterContext class from the registry model doesn't support hashcode/equals currently so returning a Set wouldn't work.
- Updated assert calls to put expected value as first parameter and actual as second parameter
- Added one time password (OTP) support for flow download endpoint to support non cert based authentication
This closes#3931
* NIFI-6925: Fixed JoltTransformRecord for RecordReaders, improved MockProcessSession
* Fixed logic for no records, added unit test
* Fixed PutElasticsearchHttpRecord and PutHive3Streaming, same bug as JoltTransformRecord
* Added null checks
After: Go over all partitions and see if all of them are full.
Wrapping RoundRobinPartitioner in a (new) AvailableSeekingPartitioner which selects a new partition if the currently selected one is full.
NIFI-6926: Use new instance of list instead of clearing it
NIFI-6926: Logging the number of messages to be sent to Atlas.
NIFI-6926: Pass a copy of the messages list to send() and clear the original list.
This closes#3915
Calls to `trigger()` may be called concurrently from different threads,
however the PutKudu processor is storing the `kuduSession`
in a class level field. This can result in the logging issue reported in
NIFI-6895 and likely other unusual anomolies including performace
issues depending on the processor configuration.
Additionally the `operationType` was also stored in a class level field
and could be set concurrently resulting in the incorrect operation type
used.
This patch fixes the issue by moving both kuduSession and operationType
to be local. Additionaly some minor code cleanup was included.
An integration test, ITPutKudu, was added and used to manual verify the
logging issue existed and is fixed by this patch. I ran the test using
`mvn -Pintegration-tests verify -Dtest=ITPutKudu`
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#3910.
NIFI-6896 Adding hadoop-aws, hadoop-azure, and hadoop-azure-datalake to nifi-hadoop-libraries-nar and nifi-ranger-nar so other filesystem impls will be available
NIFI-6896 Introduce profiles to optionally include additional hadoop modules to reduce default build size
NIFI-6896 Updating licenses and notices to account for new optional dependencies
NIFI-6896 Add profile for hadoop-cloud-storage
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3900
- Fixed bug when calling logUnbuildableDestination with "jms_destination" instead of actual destination name value
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3895
NIFI-6855 - added setting to support ignore, warn or throwing exception for unsupported action types. added EL support for defining types to enforce.
NIFI-6855 - fix checkstyle violation
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3886
NIFI-5248 Fixed a few stray 1.7.0 references.
NIFI-5248 Removed build helper plugin.
NIFI-5248 Made changes requested in a review.
NIFI-5248 Updated dummy service.
NIFI-5248 Made a few changes from a code review.
NIFI-5248 Added logic for removing nulls so record paths can be removed when no longer needed.
NIFI-5248 Switched from variable registry to flowfile level EL.
NIFI-5248 Added JsonPath code to remove index, id and type path statements.
NIFI-5248 Updated validation.
NIFI-5248 Set the field to null instead of empty string when nulling records.
NIFI-5248 Fixed TestElasticSearchClientService.
NIFI-5248 Removed high level client and switched over to low level client for everything.
NIFI-5248 Added profiles for ES 6 and ES 7 integration testing.
NIFI-5248 Updated integration tests to support 5 and 6.
NIFI-5248 Fixed some style check breaks.
NIFI-5248 Added create operation type.
NIFI-5248 Updated documentation.
NIFI-5248 Added error handling to PutElasticsearchRecord.
NIFI-5248 Added error logging to PutElasticsearchJson.
NIFI-5248 Added split failed records option to PutElasticsearchJson.
NIFI-5248 Added documentation for PutElasticsearchRecord.
NIFI-5248 Updated import to not use * import.
NIFI-5248 Removed processor that is no longer relevant due to schema inference.
NIFI-5248 Renamed ElasticSearch instances to Elasticsearch where we can within API guidelines.
NIFI-5248 Added groovy-json test dependency.
NIFI-5248 Updated PutElasticsearchRecord to only do index operations.
NIFI-5248 Added batch size property and refactored the way relationships and properties are added.
NIFI-5248 Added batch processing support.
NIFI-5248 Updated error handling.
NIFI-5248 Updated to 1.11.0-SNAPSHOT.
NIFI-5248 Made changes requested in a code review.
NIFI-5248 Made a few more changes from a code review.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2861
proper line break
more proper line break :)
Link to external javadocs and add some code comments
Test that properties is set on ConnectionFactory
cleanup
made two static properties optional and elaborated on 'Additional Details' page
minor corrections to user doc
open external links in new tab
Do broker accommodations only if property is set. Add fallback accommodation.
fix test
Add support for colon pair(s) for IBM MQ. In fallback broker handling use first pair if multiple given.
This closes#3246.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
* Added additional unit tests and updated ParameterEvaluator to make sure that it follows the same logic as the ParameterExpression in that sensitive parameters cannot be access from within EL
This closes#3889
Signed-off-by: Rob Fellows <rob.fellows@gmail.com>
NIFI-6707: Added System Test to verify behavior when services depend on one another
NIFI-6707: Ensure that when tearing down flow after test, we wait for all processors and controller services to complete stop/disable before attempting to delete them
NIFI-6707: Fixed bug in RemoteProcessGroupIT that caused the test to fail intermittently based on timing. Improved logging for Process Group, Standard Process Group
NIFI-6707: Updated to include java11 subdirectory under lib/
NIFI-6707: Rebased against master and changed dependencies from 1.10.0-SNAPSHOT to 1.11.0-SNAPSHOT
NIFI-6707: Updates to see if it will help travis build correctly
NIFI-6707: Commenting out java commands in bootstrap.conf files
This closes#3831.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-6842 - Added AlertHandler for bulletin reporting. Update ReportingTask meta data.
NIFI-6842 - corrected display names in action handlers, included metrics option for alert handlers, small refactor in reporting task
NIFI-6842 - updated docs and tags
NIFI-6842 - Added documentation for handlers.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3874
Update CountText.java
Local variable changes by AtomicInteger
NIFI-6395 - Fix line is longer than 200 characters
This closes#3552.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
Matthew Burgess
Bryan Bende
Joe Witt
Troy Melhase
Grant Henke
Peter Turcsanyi
M Tien
Joe Ferner
Wouter de Vries
Mark Payne
David Savage
Pierre Villard
John Highcock
nagasivanath
Matt Gilman
Andy LoPresto
mdayakar
Gardella Juan Pablo
mans2singh
Marc Parisi
Dirk Ernst van den Biggelaar
Gal Shinder
Tamas Palfy
Joe Gresock
Nathan Gough
bahlulh
wagggz
Shawn Weeks
Michael Hogue
Tom
Martin Šimek
John Pierce
mtien
Yolanda Davis
Eduardo Fontes
Ubuntu
r65535
KovalevIV
Matthew Formosa
Iván Rodriguez
Kourge
Matthew Knight
John Fortin
Andrew Lim
sjyang18
Koji Kawamura
Mike Thomsen
Shayne Burgess
Veli Kerim Celik
kevinmccarthy
Atul Mahajan
Ivan Ezequiel Rodriguez
Nicholas Zhan