Apache
/
nifi
mirror of https://github.com/apache/nifi.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
nifi/nifi-nar-bundles
History
Koji Kawamura e68ff153e8
NIFI-3332: ListXXX to not miss files with the latest processed timestamp 
Before this fix, it's possible that ListXXX processors can miss files those have the same timestamp as the one which was the latest processed timestamp at the previous cycle. Since it only used timestamps, it was not possible to determine whether a file is already processed or not.

However, storing every single processed identifier as we used to will not perform well.
Instead, this commit makes ListXXX to store only identifiers those have the latest timestamp at a cycle to minimize the amount of state data to store.

NIFI-3332: ListXXX to not miss files with the latest processed timestamp

- Fixed TestAbstractListProcessor to use appropriate time precision.
  Without this fix, arbitrary test can fail if generated timestamp does
  not have the desired time unit value, e.g. generated '10:51:00' where
  second precision is tested.
- Fixed TestFTP.basicFileList to use millisecond time precision explicitly
  because FakeFtpServer's time precision is in minutes.
- Changed junit dependency scope to 'provided' as it is needed by
  ListProcessorTestWatcher which is shared among different modules.

This closes #1975.

Signed-off-by: Bryan Bende <bbende@apache.org>
 		2017-08-28 11:31:04 -04:00
..
nifi-ambari-bundle NIFI-4019: 2017-06-15 14:17:35 -04:00
nifi-amqp-bundle NIFI-4154 Fixing line endings in .java,.html files 2017-07-09 12:13:15 -07:00
nifi-avro-bundle NIFI-3997: 2017-06-08 15:22:51 -04:00
nifi-aws-bundle NIFI-4161: Adding expression evaluation to AWS PROXY_HOST and PROXY_HOST_PORT property usage 2017-07-07 13:28:33 +02:00
nifi-azure-bundle NIFI-4007 - Update azure EventHubs client to latest version 2017-06-29 09:30:39 +02:00
nifi-beats-bundle NIFI-3997: 2017-06-08 15:22:51 -04:00
nifi-cassandra-bundle NIFI-3997: 2017-06-08 15:22:51 -04:00
nifi-ccda-bundle NIFI-3997: 2017-06-08 15:22:51 -04:00
nifi-cdc NIFI-4217 - Retain original DDL in CaptureChangeMySQL 2017-07-25 17:54:11 +02:00
nifi-confluent-platform-bundle NIFI-1763: Initial implementation of ConfluentSchemaRegistry. 2017-07-06 15:52:07 -04:00
nifi-couchbase-bundle NIFI-3997: 2017-06-08 15:22:51 -04:00
nifi-cybersecurity-bundle NIFI-4019: 2017-06-15 14:17:35 -04:00
nifi-datadog-bundle NIFI-4019: 2017-06-15 14:17:35 -04:00
nifi-elasticsearch-bundle NIFI-4250 - Elasticsearch 5 delete processor 2017-08-21 08:49:57 -04:00
nifi-email-bundle NIFI-3992: Added Original Headers to Include/Exclude 2017-06-23 17:38:19 +09:00
nifi-enrich-bundle NIFI-3997: 2017-06-08 15:22:51 -04:00
nifi-evtx-bundle NIFI-4019: 2017-06-15 14:17:35 -04:00
nifi-extension-utils NIFI-3332: ListXXX to not miss files with the latest processed timestamp 2017-08-28 11:31:04 -04:00
nifi-flume-bundle NIFI-4019: 2017-06-15 14:17:35 -04:00
nifi-framework-bundle NIFI-4324: 2017-08-25 16:20:32 -04:00
nifi-gcp-bundle NIFI-3997: 2017-06-08 15:22:51 -04:00
nifi-grpc-bundle NIFI-4037 added InvokeGRPC processor, with proto service IDL 2017-06-29 22:01:22 -04:00
nifi-hadoop-bundle NIFI-4311 Allowing umask to get set properly before initializing the FileSystem 2017-08-22 22:40:26 +02:00
nifi-hadoop-libraries-bundle NIFI-3997: 2017-06-08 15:22:51 -04:00
nifi-hbase-bundle NIFI-4275 Adding support for specifying the timestamp on PutHBase processors 2017-08-11 18:04:13 +02:00
nifi-hive-bundle NIFI-2923 Added evaluation of attribute expressions for Kerberos principal and keytab 2017-06-21 17:14:28 -04:00
nifi-hl7-bundle NIFI-3997: 2017-06-08 15:22:51 -04:00
nifi-html-bundle NIFI-3997: 2017-06-08 15:22:51 -04:00
nifi-ignite-bundle NIFI-3997: 2017-06-08 15:22:51 -04:00
nifi-jetty-bundle NIFI-3997: 2017-06-08 15:22:51 -04:00
nifi-jms-bundle NIFI-3983 - Support ability to make JMS 2.0 durable subscriptions on Topic 2017-06-29 10:23:15 +09:00
nifi-kafka-bundle NIFI-4142: This closes #2015. Refactored Record Reader/Writer to allow for reading/writing "raw records". Implemented ValidateRecord. Updated Record Reader to take two parameters for nextRecord: (boolean coerceTypes) and (boolean dropUnknownFields) 2017-08-11 22:01:46 -07:00
nifi-kerberos-iaa-providers-bundle NIFI-3997: 2017-06-08 15:22:51 -04:00
nifi-kite-bundle NIFI-4019: 2017-06-15 14:17:35 -04:00
nifi-kudu-bundle [NiFi-3973] Add PutKudu processor to ingest data to Kudu 2017-08-25 10:53:31 -04:00
nifi-language-translation-bundle NIFI-3997: 2017-06-08 15:22:51 -04:00
nifi-ldap-iaa-providers-bundle NIFI-4059: 2017-06-19 19:25:33 +02:00
nifi-lumberjack-bundle NIFI-3997: 2017-06-08 15:22:51 -04:00
nifi-media-bundle NIFI-3677 - ExtractMediaMetadata should close TikaInputStream 2017-07-11 18:04:50 +02:00
nifi-mongodb-bundle NIFI-4124 Added org.apache.nifi.mongo.PutMongoRecord. 2017-08-07 22:10:27 -07:00
nifi-mqtt-bundle NIFI-3997: 2017-06-08 15:22:51 -04:00
nifi-parquet-bundle NIFI-3921: Allow Record Writers to inherit schema from Record 2017-06-09 16:13:25 -04:00
nifi-poi-bundle NIFI-4112: Fix ConvertExcelToCSV to handle empty cells. 2017-07-13 10:40:01 -04:00
nifi-provenance-repository-bundle NIFI-4023, NIFI-4077 This closes #2075. Addressed issue where repository was aging off the wrong index. When it should age off Index 1, it was removing Index 2. As a result, the earliest index is never aged off, and the newest index could potentially be aged off before it is ready to be. Also addressed issue where a query that attempts to read an event that has aged off throws FileNotFoundException (NIFI-4077) instead of skipping over the event. The JavaDocs indicate that the EventIterator should skip records that it cannot find, but SelectiveRecordReaderEventIterator throw FileNotFoundException instead 2017-08-11 22:01:46 -07:00
nifi-ranger-bundle NIFI-4032: - Introducing the ManagedRangerAuthorizer. - Introducing the AuthorizationAuditor. - Updating authorization requests to utilize Authorizable where ever possible so allow for a singular place to audit resource not found as denied when the parent authorizable is null (no more inheritance). - Updating unit tests as appropriate. - Addressing issues with broken web-api integration tests. 2017-07-24 15:51:27 -04:00
nifi-redis-bundle NIFI-4043 Initial commit of nifi-redis-bundle 2017-06-21 12:47:19 -04:00
nifi-registry-bundle NIFI-1763: Initial implementation of ConfluentSchemaRegistry. 2017-07-06 15:52:07 -04:00
nifi-rethinkdb-bundle NIFI-4212 - RethinkDB Delete Processor 2017-07-29 14:49:22 -07:00
nifi-riemann-bundle NIFI-3997: 2017-06-08 15:22:51 -04:00
nifi-scripting-bundle NIFI-4142: This closes #2015. Refactored Record Reader/Writer to allow for reading/writing "raw records". Implemented ValidateRecord. Updated Record Reader to take two parameters for nextRecord: (boolean coerceTypes) and (boolean dropUnknownFields) 2017-08-11 22:01:46 -07:00
nifi-site-to-site-reporting-bundle NIFI-4019: 2017-06-15 14:17:35 -04:00
nifi-slack-bundle NIFI-4019: 2017-06-15 14:17:35 -04:00
nifi-snmp-bundle NIFI-4019: 2017-06-15 14:17:35 -04:00
nifi-social-media-bundle NIFI-3997: 2017-06-08 15:22:51 -04:00
nifi-solr-bundle NIFI-4019: 2017-06-15 14:17:35 -04:00
nifi-splunk-bundle NIFI-3997: 2017-06-08 15:22:51 -04:00
nifi-spring-bundle NIFI-4019: 2017-06-15 14:17:35 -04:00
nifi-standard-bundle NIFI-3332: ListXXX to not miss files with the latest processed timestamp 2017-08-28 11:31:04 -04:00
nifi-standard-services NIFI-4304 Extracting HWX Schema Registry client version to a property and bumping to latest 0.3.0 release 2017-08-17 16:07:03 -04:00
nifi-stateful-analysis-bundle NIFI-3997: 2017-06-08 15:22:51 -04:00
nifi-tcp-bundle NIFI-3997: 2017-06-08 15:22:51 -04:00
nifi-update-attribute-bundle NIFI-11 Fixing checkstyle issue - usused import 2017-07-19 22:46:15 -07:00
nifi-websocket-bundle NIFI-3997: 2017-06-08 15:22:51 -04:00
nifi-windows-event-log-bundle NIFI-4019: 2017-06-15 14:17:35 -04:00
pom.xml [NiFi-3973] Add PutKudu processor to ingest data to Kudu 2017-08-25 10:53:31 -04:00