Forced HandleHTTPRequest to use RestrictedSSLContextService and removed extraneous SSL algorithm checks
Throw RuntimeException if the chosen SSL protocol isn't supported by HandleHttpRequest
This closes#1985.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Added external compatibility regression test for StringEncryptor to ensure continued functionality during removal of Jasypt.
Documents custom salt lengths and iteration counts for each encryption method.
Added (ignored) failing tests for keyed encryption (Jasypt does not support keyed encryption).
Changed StringEncryptor to non-final class and added protected default constructor.
Added failing test for initialization status.
Added utility methods in CipherUtility.
Moved PBE cipher providers (and tests) from nifi-standard-processors to nifi-security-utils module.
Implemented PBE and keyed encryption/decryption logic.
Moved Scrypt unit test back into scrypt package.
Resolved test failures in limited strength cryptographic environment.
Implemented keyed encryption/decryption and enabled unit tests.
Removed Jasypt dependency from production scope (kept in test scope for backward compatibility tests).
Signed-off-by: joewitt <joewitt@apache.org>
The unit test for DATE type used GMT timezone, that causes an assertion error in timezones such as EST (-5).
We need to use local timezone instead of GMT, as Derby and PutSQL uses local timezone.
The unit test failed before as follows:
- Unit test code, passed: '2002-02-02 GMT'
- PutSQL code convertedi it to local: '2002-02-01 EST', and stored as '2002-02-01' in Derby database without timezone info
- Unit test code SELECT the inserted value, passed a GMT calender, then got epoch timestamp, which was '2002-01-31'
Support negative long value for timezones ahead of UTC.
- For timezones such as '+0800', it's possible that a local time e.g. '02:03:04' can be a negative epoch value. This commit changes LONG_PATTERN so that it can accept nevative values.
- Changed time values in unit tests to verify negative epoch values, and avoid using the same digits among different time unit for better readability.
This closes#2082
- Unmatched fields were ignored, but the number of prepared statement
place holders were not correct.
- Added unit test code for generateUpdate.
- Added unit test code with "Ignore Unmatched Columns".
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#2165.
- Removed FlowFile from RecordReaderFactory, RecordSetWriterFactory and SchemaAccessStrategy.
- Renamed variable 'allowableValue' to 'strategy' to represent its meaning better.
- Removed creation of temporal FlowFile to resolve Record Schema from ConsumerLease.
- Removed unnecessary 'InputStream content' argument from
RecordSetWriterFactory.getSchema method.
This closes#1877.
fix checkstyle issue, and added unit test showing data duplication issue, removed property
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2091
- Refactored variable names to better represents what those are meant for.
- Added deterministic logic which detects target filesystem timestamp precision and adjust lag time based on it.
- Changed from using System.nanoTime() to System.currentTimeMillis in test because Java File API reports timestamp in milliseconds at the best granularity. Also, System.nanoTime should not be used in mix with epoch milliseconds because it uses arbitrary origin and measured differently.
- Changed TestListFile to use more longer interval between file timestamps those are used by testFilterAge to provide more consistent test result because sleep time can be longer with filesystems whose timestamp in seconds precision.
- Added logging at TestListFile.
- Added TestWatcher to dump state in case assertion fails for further investigation.
- Added Timestamp Precision property so that user can set if auto-detect is not enough
- Adjust timestamps for ages test
This closes#1915.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-4028: Refactored Wait processor.
- Consolidated implementation for the cases of releasableFlowCount is 1 or more, in order to reduce complexity and behavior differences
- Added 'consumed' counter when total counter is used to release incoming FlowFiles
- Fixed method name typo, releaseCandidates
This closes#2055.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
The properties are already evaluated against EL. Also added an
integration test that is failing without the change.
This closes#1968.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-3281 - Review - handle completePendingCommand return and added a unit test for ListFTP
NIFI-3281 - Review - Added flow file for EL evaluation in other methods and added unit test for NIFI-3590
This closes#1974.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
Before this fix, it is possible that TailFile to produce duplicated data
if an already tailed file has newer timestamp and fewer or the same
amout of data.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#2021.
Fix unit test for Date and Time type time zone problem
Enhance Time type to record milliseconds
This closes#1983.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Changed boolean value conversion to use Boolean.valueOf.
- Updated comments in source code to reflect current situation more clearly.
- Updated tests those have been added since the original commits were made.
NIFI-4060: Addressed threading issue with RecordBin being updated after it is completed; fixed issue that caused mime.type attribute not to be written properly if all incoming flowfiles already have a different value for that attribute
NIFI-4060: Bug fixes; improved documentation; added a lot of debug information; updated StandardProcessSession to produce more accurate logs in case of a session being committed/rolled back with open input/output streams
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1958
NIFI-4061 Initial version of RedisStateProvider
- Adding PropertyContext and updating existing contexts to extend it
- Added embedded Redis for unit testing
- Added wrapped StateProvider with NAR ClassLoader in StandardStateManagerProvider
- Updating state-management.xml with config for RedisStateProvider
- Renaming tests that use RedisServer to be IT tests so they don't run all the time
This closes#1918.
To be used with cache engines that does not have revision number.
NIFI-4049: Refactor AtomicDistributedMapCacheClient
Removed old methods completely.
This closes#1904.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Oracle NUMBER can return 0 precision and -127 or 0 scale with variable scale NUMBER such as ROWNUM or function result
- Added 'Default Decimal Precision' and 'Default Decimal Scale' property to ExecuteSQL and QueryDatabaseTable to apply default precision and scale if those are unknown
- Coerce BigDecimal scale to field schema logical type, so that BigDecimals having different scale can be written
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1851
- Added dependency notice.
- Added EL evaluation at SimpleKeyValueLookupService.
- Updated documentation.
- Updated CommonsConfigurationLookupService to throw LookupFailureException if it fails to get configuration so that error messages can be displayed at each processor bulletin.
- Added calling getConfiguration at OnEnabled of CommonsConfigurationLookupService, so that the service will stay in Enabling state if there is any issue.
Signed-off-by: Joey Frazee <jfrazee@apache.org>
- Added Logical type support for DECIMAL/NUMBER, DATE, TIME and TIMESTAMP column types.
- Added Logical type 'decimal' to AvroReader so that Avro records with logical types written by ExecuteSQL and QueryDatabaseTable can be consumed by AvroReader.
- Added JdbcCommon.AvroConversionOptions to consolidate conversion options.
- Added 'Use Avro Logical Types' property to ExecuteSQL and QueryDatabaseTable to toggle whether to use Logical types.
- Added 'mime.type' FlowFile attribute as 'application/avro-binary' so that output FlowFiles can be displayed by content viewer.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1798
NIFI-3838: Updated version from 1.2.0-SNAPSHOT to 1.3.0-SNAPSHOT; removed unneeded value from AttributeExpression.ResultType enum
NIFI-3838: Addressed PR Review feedback
NIFI-3838: Allow for schemas to be merged together for a record; refactored RecordSetWriterFactory so that there is a method to obtain the schema and then the writer is created with that schema. Added additional unit tests
NIFI-3838: Addressed problems with documentation based on PR Review
NIFI-3838: Fixed checkstyle violation
NIFI-3838: Addressed issue of comparing different types of Number objects
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1772
NIFI-1833 Moved AbstractListProcessor.java, EntityListing.java, and ListableEntity.java from nifi-standard-processors into nifi-processor-utils
Moved TestAbstractListProcessor.java into nifi-processor-utils
Set nifi-azure-nar's nar dependency back to nifi-standard-services-api-nar
Fixed failing integration tests (ITFetchAzureBlobStorage.java, ITListAzureBlobStorage.java, and ITPutAzureStorageBlob.java) and refactored them to be able to run in parallel
NIFI-1833 Moved security notice info in the additional details documentation into the descriptions of the specific attributes for which those notices are intended
Added displayName usage to properties
Updated exception handling in FetchAzureBlobStorage.java and PutAzureBlobStorage.java to cause flowfiles with Output/InputStreamCallback failures to be routed to the processor's failure relationship
Cleaned up dependencies in pom
NIFI-1833 Removed unnecessary calls to map on Optional in the onTrigger exception handling of FetchAzureBlobStorage.java and PutAzureBlobStorage.java
NIFI-1833 Updates due to nifi-processor-utils being moved under nifi-nar-bundles
This closes#1719.
Signed-off-by: Bryan Rosander <brosander@apache.org>
Added src/test/resources/logback-test.xml files resetting log level from DEBUG (in nifi-data-provenance-utils) to WARN because later tests depend on MockComponentLog recording a certain number of messages and this number is different than expected if the log level is DEBUG.
This closes#1686.
Signed-off-by: Bryan Bende, Yolanda M. Davis, and Mark Payne
- Added org.apache.nifi.processor.util.pattern package in nifi-processor-utils containing reusable functions to mix-in 'Rollback on Failure' capability.
- Created a process pattern classes, Put and PutGroup. It will be helpful to standardize Processor implementations.
- Applied Rollback on Failure to PutSQL, PutHiveQL, PutHiveStreaming and PutDatabaseRecord.
- Stop using AbstractProcessor for these processors, as it penalizes FlowFiles being processed when it rollback a process session. If FlowFiles are penalized, it will not be fetched again until penalization expires.
- Yield processor when a failure occurs and RollbackOnFailure is enabled. If we do not penalize nor yield, a failed FlowFile retries too frequently.
- When Rollback on Failure is enabled but processor is not transactional, discontinue when an error occurred after successful processes.
- Fixed existing issues on PutHiveStreaming:
- Output FlowFile Avro format was corrupted by concatenating multiple Avro files.
- Output FlowFile records had incorrect values because of reusing GenericRecord instance.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1658
NIFI-3704: Incorporated review comments
NIFI-3704: Updated PutDatabaseRecord documentation, including limitations on Update records
NIFI-3704: Added retry paths for transient SQL errors
NIFI-3704: Updated to use renamed RecordReaderFactory and catch SchemaNotFoundException
NIFI-3704: Added better error handling and rollback
This closes#1677.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-3738 Fixed NPE when ParseSyslog UDP datagram has zero length.
Added unit test.
NIFI-3738 Added licenses to new unit tests.
This closes#1694.
Signed-off-by: Bryan Bende <bbende@apache.org>
m-hogue
Andy LoPresto
Bryan Bende
Koji Kawamura
Deon Huang
Mark Payne
Gino Lisignoli
Joe Percivall
Pierre Villard
patricker
Arun Manivannan
Matt Burgess
Wayne Steel
Wil Selwood
Leah Anderson
Andre F de Miranda
deonhuang
Toivo Adams
mans2singh
Tony Kurc
yuri1969
Jeff Storck
Matt Gilman
Joey Frazee
Peter G. Horvath
Jon Anderson III