NIFI-5406 introduced the issue by trying to use the resetState variable for
different purposes. AbstractListProcessor should have had a different variable
to control whether to clear state for tracking entity strategy.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#3189.
Adding followings:
- Use separate DistributedMapCache for tracking entities to avoid
conflict with existing code
- Added more validation
- Delete listed entities from cache if reset is needed
- Support Local scope
- Added Initial Listing Target
This closes#2876.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- take into account input requirement for documentation rendering
- Renamed variable registry scope and added comments
- Doc + change in mock framework to check scope + update of components + UI
Before this fix, MergeContent only processed the first bin even if there
were multiple bins.
There were two unit tests marked with Ignore those had been
failing because of this.
This closes#2444.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Upgrading to Jersey 2.x.
- Updating NOTICE files where necessary.
- Fixing checkstyle issues.
This closes#2206.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Upgraded immediately actionable dependency versions from Meterian report.
- Upgraded jackson-core test dependencies for HBase and Elasticsearch modules.
- Only 3 instances of jackson-core < 2.8.6 (Google Cloud Platform and Spark Receiver modules).
- Upgraded version of poi dependency in nifi-email-processors to 3.16.
- Resolving dependency issues after rebasing against 1.5.0-SNAPSHOT.
- Removed jackson-databind from <dependencyManagement> block in nifi/pom.xml and added explicit reference to ${jackson.version} in all referenced artifacts.
- Removed jackson-mapper-asl from <dependencyManagement> block in nifi/pom.xml and added explicit reference to ${jackson.old.version} in all referenced artifacts.
- Removed Jasypt from <dependencyManagement> and added explicit version in test dependency for legacy compatibility.
- This closes#2084
Before this fix, it's possible that ListXXX processors can miss files those have the same timestamp as the one which was the latest processed timestamp at the previous cycle. Since it only used timestamps, it was not possible to determine whether a file is already processed or not.
However, storing every single processed identifier as we used to will not perform well.
Instead, this commit makes ListXXX to store only identifiers those have the latest timestamp at a cycle to minimize the amount of state data to store.
NIFI-3332: ListXXX to not miss files with the latest processed timestamp
- Fixed TestAbstractListProcessor to use appropriate time precision.
Without this fix, arbitrary test can fail if generated timestamp does
not have the desired time unit value, e.g. generated '10:51:00' where
second precision is tested.
- Fixed TestFTP.basicFileList to use millisecond time precision explicitly
because FakeFtpServer's time precision is in minutes.
- Changed junit dependency scope to 'provided' as it is needed by
ListProcessorTestWatcher which is shared among different modules.
This closes#1975.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Refactored variable names to better represents what those are meant for.
- Added deterministic logic which detects target filesystem timestamp precision and adjust lag time based on it.
- Changed from using System.nanoTime() to System.currentTimeMillis in test because Java File API reports timestamp in milliseconds at the best granularity. Also, System.nanoTime should not be used in mix with epoch milliseconds because it uses arbitrary origin and measured differently.
- Changed TestListFile to use more longer interval between file timestamps those are used by testFilterAge to provide more consistent test result because sleep time can be longer with filesystems whose timestamp in seconds precision.
- Added logging at TestListFile.
- Added TestWatcher to dump state in case assertion fails for further investigation.
- Added Timestamp Precision property so that user can set if auto-detect is not enough
- Adjust timestamps for ages test
This closes#1915.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-4060: Addressed threading issue with RecordBin being updated after it is completed; fixed issue that caused mime.type attribute not to be written properly if all incoming flowfiles already have a different value for that attribute
NIFI-4060: Bug fixes; improved documentation; added a lot of debug information; updated StandardProcessSession to produce more accurate logs in case of a session being committed/rolled back with open input/output streams
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1958
NIFI-1833 Moved AbstractListProcessor.java, EntityListing.java, and ListableEntity.java from nifi-standard-processors into nifi-processor-utils
Moved TestAbstractListProcessor.java into nifi-processor-utils
Set nifi-azure-nar's nar dependency back to nifi-standard-services-api-nar
Fixed failing integration tests (ITFetchAzureBlobStorage.java, ITListAzureBlobStorage.java, and ITPutAzureStorageBlob.java) and refactored them to be able to run in parallel
NIFI-1833 Moved security notice info in the additional details documentation into the descriptions of the specific attributes for which those notices are intended
Added displayName usage to properties
Updated exception handling in FetchAzureBlobStorage.java and PutAzureBlobStorage.java to cause flowfiles with Output/InputStreamCallback failures to be routed to the processor's failure relationship
Cleaned up dependencies in pom
NIFI-1833 Removed unnecessary calls to map on Optional in the onTrigger exception handling of FetchAzureBlobStorage.java and PutAzureBlobStorage.java
NIFI-1833 Updates due to nifi-processor-utils being moved under nifi-nar-bundles
This closes#1719.
Signed-off-by: Bryan Rosander <brosander@apache.org>
- Creating nifi-records-utils to share utility code from record services
- Refactoring Parquet tests to use MockRecorderParser and MockRecordWriter
- Refactoring AbstractPutHDFSRecord to use schema access strategy
- Adding custom validate to AbstractPutHDFSRecord and adding handling of UNION types when writing Records as Avro
- Refactoring project structure to get CS API references out of nifi-commons, introducing nifi-extension-utils under nifi-nar-bundles
- Updating abstract put/fetch processors to obtain the WriteResult and update flow file attributes
This closes#1712.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Endre Zoltan Kovacs
Koji Kawamura
Jeff Storck
joewitt
Andy LoPresto
zenfenan
Pierre Villard
Max Viazovskyi
Matthew Burgess
sbouchex
Mark Bean
Derek Straka
Matt Gilman
Bryan Bende
Mark Payne