- Upgrading to Jersey 2.x.
- Updating NOTICE files where necessary.
- Fixing checkstyle issues.
This closes#2206.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
remove redundant additionalDetails.html and add docs to CapabilityDescription in HDFS processors
revert the modified CapabilityDescriptions in HDFS processors and add it to AbstractHadoopProcessor
- Upgraded immediately actionable dependency versions from Meterian report.
- Upgraded jackson-core test dependencies for HBase and Elasticsearch modules.
- Only 3 instances of jackson-core < 2.8.6 (Google Cloud Platform and Spark Receiver modules).
- Upgraded version of poi dependency in nifi-email-processors to 3.16.
- Resolving dependency issues after rebasing against 1.5.0-SNAPSHOT.
- Removed jackson-databind from <dependencyManagement> block in nifi/pom.xml and added explicit reference to ${jackson.version} in all referenced artifacts.
- Removed jackson-mapper-asl from <dependencyManagement> block in nifi/pom.xml and added explicit reference to ${jackson.old.version} in all referenced artifacts.
- Removed Jasypt from <dependencyManagement> and added explicit version in test dependency for legacy compatibility.
- This closes#2084
- Removed FlowFile from RecordReaderFactory, RecordSetWriterFactory and SchemaAccessStrategy.
- Renamed variable 'allowableValue' to 'strategy' to represent its meaning better.
- Removed creation of temporal FlowFile to resolve Record Schema from ConsumerLease.
- Removed unnecessary 'InputStream content' argument from
RecordSetWriterFactory.getSchema method.
This closes#1877.
Before this fix, it's possible that ListXXX processors can miss files those have the same timestamp as the one which was the latest processed timestamp at the previous cycle. Since it only used timestamps, it was not possible to determine whether a file is already processed or not.
However, storing every single processed identifier as we used to will not perform well.
Instead, this commit makes ListXXX to store only identifiers those have the latest timestamp at a cycle to minimize the amount of state data to store.
NIFI-3332: ListXXX to not miss files with the latest processed timestamp
- Fixed TestAbstractListProcessor to use appropriate time precision.
Without this fix, arbitrary test can fail if generated timestamp does
not have the desired time unit value, e.g. generated '10:51:00' where
second precision is tested.
- Fixed TestFTP.basicFileList to use millisecond time precision explicitly
because FakeFtpServer's time precision is in minutes.
- Changed junit dependency scope to 'provided' as it is needed by
ListProcessorTestWatcher which is shared among different modules.
This closes#1975.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Refactored variable names to better represents what those are meant for.
- Added deterministic logic which detects target filesystem timestamp precision and adjust lag time based on it.
- Changed from using System.nanoTime() to System.currentTimeMillis in test because Java File API reports timestamp in milliseconds at the best granularity. Also, System.nanoTime should not be used in mix with epoch milliseconds because it uses arbitrary origin and measured differently.
- Changed TestListFile to use more longer interval between file timestamps those are used by testFilterAge to provide more consistent test result because sleep time can be longer with filesystems whose timestamp in seconds precision.
- Added logging at TestListFile.
- Added TestWatcher to dump state in case assertion fails for further investigation.
- Added Timestamp Precision property so that user can set if auto-detect is not enough
- Adjust timestamps for ages test
This closes#1915.
Signed-off-by: Bryan Bende <bbende@apache.org>
NiFi can now parse an Avro schema of a record that references an already defined record, including itself.
Signed-off-by: James Wing <jvwing@gmail.com>
This closes#2034.
NIFI-4060: Addressed threading issue with RecordBin being updated after it is completed; fixed issue that caused mime.type attribute not to be written properly if all incoming flowfiles already have a different value for that attribute
NIFI-4060: Bug fixes; improved documentation; added a lot of debug information; updated StandardProcessSession to produce more accurate logs in case of a session being committed/rolled back with open input/output streams
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1958
NIFI-1763: Fixed bug where the Confluent Schema Registry Schema Access Writer was not being created
Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>
This closes#1938
- Oracle NUMBER can return 0 precision and -127 or 0 scale with variable scale NUMBER such as ROWNUM or function result
- Added 'Default Decimal Precision' and 'Default Decimal Scale' property to ExecuteSQL and QueryDatabaseTable to apply default precision and scale if those are unknown
- Coerce BigDecimal scale to field schema logical type, so that BigDecimals having different scale can be written
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1851
- Removed remaining duplicate lines of code left by NIFI-3861 refactoring.
- Added test case that writes Avro record having union field.
This closes#1813.
- Added Logical type support for DECIMAL/NUMBER, DATE, TIME and TIMESTAMP column types.
- Added Logical type 'decimal' to AvroReader so that Avro records with logical types written by ExecuteSQL and QueryDatabaseTable can be consumed by AvroReader.
- Added JdbcCommon.AvroConversionOptions to consolidate conversion options.
- Added 'Use Avro Logical Types' property to ExecuteSQL and QueryDatabaseTable to toggle whether to use Logical types.
- Added 'mime.type' FlowFile attribute as 'application/avro-binary' so that output FlowFiles can be displayed by content viewer.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1798
NIFI-3838: Updated version from 1.2.0-SNAPSHOT to 1.3.0-SNAPSHOT; removed unneeded value from AttributeExpression.ResultType enum
NIFI-3838: Addressed PR Review feedback
NIFI-3838: Allow for schemas to be merged together for a record; refactored RecordSetWriterFactory so that there is a method to obtain the schema and then the writer is created with that schema. Added additional unit tests
NIFI-3838: Addressed problems with documentation based on PR Review
NIFI-3838: Fixed checkstyle violation
NIFI-3838: Addressed issue of comparing different types of Number objects
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1772
Previous fix#1779 refactored the way to check Logical type to use string constants.
One of those refactoring used wrong constant mistakenly in normalizeValue method.
Fortunately, this defect is harmless since even though normalizeValue did not convert int to Time, DataTypeUtils.convertType does the same conversion.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1782
This closes#1787.
- When converting from a raw value to an Avro object, convert the values
of any Avro map types so that they can be complex types like other
records.
- AvroReader did not convert logical types if those are defined with union
- Consolidated createSchema method in AvroSchemaRegistry and AvroTypeUtil as both has identical implementation and mai
ntaining both would be error-prone
This closes#1779.
Koji Kawamura
patricker
Matt Gilman
Bryan Bende
Takanobu Asanuma
Andy LoPresto
Jeff Storck
Mark Payne
Wesley-Lawrence
James Wing
Maurizio Colleluori
Steve Champagne
Pierre Villard