- Updates to make UpdateRecord and RecordPath automatically update Record schema when performing update and perform the updates on the first record in UpdateRecord before obtaining Writer Schema. This allows the Writer to to inherit the Schema of the updated Record instead of the Schema of the Record as it was when it was read.
- Updated JoltTransformRecord so that schema is inferred on the first transformed object before passing the schema to the Record Writer, so that if writer inherits schema from record, the schema that is inherited is the trans transformed schema
- Updated LookupRecord to allow for Record fields to be arbitrarily added
- Implemented ContentClaimInputStream
- Added controller service for caching schemas
- UpdatedQueryRecord to cache schemas automatically up to some number of schemas, which will significantly inprove throughput in many cases, especially with inferred schemas.
NIFI-5938: Updated AvroTypeUtil so that if creating an Avro Schema using a field name that is not valid for Avro, it creates a Schema that uses a different, valid field name and adds an alias for the given field name so that the fields still are looked up appropriately. Fixed a bug in finding the appropriate Avro field when aliases are used. Updated ContentClaimInputStream so that if mark() is called followed by multiple calls to reset(), that each reset() call is successful instead of failing after the first one (the JavaDoc for InputStream appears to indicate that the InputStream is free to do either and in fact the InputStream is even free to allow reset() to reset to the beginning of file if mark() is not even called, if it chooses to do so instead of requiring a call to mark()).
NIFI-5938: Added another unit test for AvroTypeUtil
NIFI-5938: If using inferred schema in CSV Reader, do not consider first record as a header line. Also addressed a bug in StandardConfigurationContext that was exposed by CSVReader, in which calling getProperty(PropertyDescriptor) did not properly lookup the canonical representation of the Property Descriptor from the component before attempting to get a default value
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3253
Unit test is implemented: testNoSplitterInString
NIFI-5974: Fixed Checkstyle violations
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3275
Clear the wait timestamp when transferring to failur or success
replace explicit attribute clear with function call, refactor and integrate into existing tests per review
This closes#3233.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-4892 - ValidateCSV: no doublequote escaping in invalid output
NIFI-5907 - unit test
This closes#2481.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-5788: Introduce batch size limit in PutDatabaseRecord processor
Renamed 'batch size' to 'Maximum Batch Size'.
Changed default value of max_batch_size to zero (INFINITE)
Fixed parameter validation.
Added unit tests
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3128
making the database session autocommit value a configurable property
adding custom validation to PutSQL processor so as to disallow 'supports transaction' and 'rollback on failure' to be true when the autocommit value has been set to true
fixing some style issues to conform to standards
This closes#3113.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-5761 Code review.
Remove startsWidth to check an exception. Added the dependency as provided.
NIFI-5761 Code review.
Remove provided.
NIFI-5761 Code review.
Improve logging.
This closes#3112.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-5686: Fixed unit test in TestSocketLoadBalancedFlowFileQueue; renamed TestProcessorLifecycle to ProcessorLifecycleIT as it is testing integration between many components and largely focuses on high numbers of concurrent tasks to see if it can trigger any threading bugs that may get introduced
NIFI-5686: Extended unit test timeouts
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3062
- introducing a in-memory-file-size-threashold, above which the incoming file is written to local disk
- using java.io.tmpdir for such file writes
- enhancing documentation
- documenting how to avoid premature HTTP response
- fix and UT for unsuccessful request registration
This closes#2991.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- introducing a in-memory-file-size-threashold, above which the incoming file is written to local disk
- using java.io.tmpdir for such file writes
- enhancing documentation
- Updated FlowFile Repo / Write Ahead Log so that any update that writes more than 1 MB of data is written to a file inside the FlowFile Repo rather than being buffered in memory
- Update SplitText so that it does not hold FlowFiles that are not the latest version in heap. Doing them from being garbage collected, so while the Process Session is holding the latest version of the FlowFile, SplitText is holding an older version, and this results in two copies of the same FlowFile object
NIFI-5533: Checkpoint
NIFI-5533: Bug Fixes
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2974
Refactored some test code to be clearer.
Renamed some resources to be consistent across modules.
Changed passwords to meet new minimum length requirements.
This closes#3018
Refactors tests in order to share code repeated in tests and to enable
some parameterized testing.
MySQL Connector/J 5.1.x in conjunction with MySQL 5.0.x will return
a Long for ResultSet#getObject when the SQL type is an unsigned integer.
This change prevents that error from occurring while implementing a more
informational exception describing what the failing object's POJO type
is in addition to its string value.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3032
NIFI-5514: Do not rely on ProcessSession.getQueueSize() to return a queue size of 0 objects because if the processor is holding onto data, the queue size won't be 0.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#2954.
Deprecated HashContent processor.
Added documentation to HashAttribute processor.
Added shared HashService and HashAlgorithm enum.
Added unit tests.
Added #clearProperties() to TestRunner, StandardProcessorTestRunner, and MockProcessContext.
Updated processor manifest.
Updated Javadoc.
Added documentation about deprecated/renamed components.
Added logic handling for UTF-16 encoding where UTF-16BE is overridden so the BOM is not inserted prior to hashing.
This closes#2983.
Co-authored-by: Otto Fowler <ottobackwards@gmail.com>
Signed-off-by: Kevin Doran <kdoran@apache.org>
This reverts commit f7f809c3d3.
NIFI-5474, NIFI-4272: When using Regex Replace with ReplaceText, and there are capturing groups, ensure that we populate the 'additionalVariables' map for each match of the regex. This allows Expression Language to reference the back-references properly even when there are multiple matches
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2951
Adding followings:
- Use separate DistributedMapCache for tracking entities to avoid
conflict with existing code
- Added more validation
- Delete listed entities from cache if reset is needed
- Support Local scope
- Added Initial Listing Target
This closes#2876.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Create nifi-syslog-utils to move syslog parsing functionalty to a central location shared by the processors and serialization/record system.
- Refactor Processors to use these utils
- Update 5424 syslog classes using simple-syslog-5424 to pick up new changes to support this work, as well as keep dependencies/types from bleeding out to the
processors or readers
- Refactor Syslog5424Event and Parser
- Create Syslog5424RecordReader
- per review, handle blank message differently from eof
- name schema per review
This closes#2816.
Signed-off-by: Bryan Bende <bbende@apache.org>
Updated Javadoc for SiteToSiteClient#createTransaction() and HttpClient implementation.
Reverted exception listing in method contract for SiteToSiteClient#createTransaction and HttpClient tion of same.
Reverted import ordering in TestSiteToSiteClient.
Reverted exception listing in TestGetHDFSFileInfo, TestListHDFS, and StandardHttpFlowFileServerProtocol.
Restored @SuppressWarnings annotation and removed unnecessary "public static" keywords from inner classes in SiteToSiteClient.
This closes#2841.
Signed-off-by: Joe Witt <joewitt@apache.org>
- address review comments
- per review, add property for including body in attributes
- per review, create processor in OnScheduled
This closes#2805.
Signed-off-by: Bryan Bende <bbende@apache.org>
forgot to penalize flowfiles sent to failure
NIFI-5274 avoid rollback on StackOverflowError in ReplaceText
add comment
removed penalize on failure, doesn't really apply here
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2767
NIFI-5214 Added support for the new ProxyConfigurationService
NIFI-5214 Integration tests added.
NIFI-5214 Added missing pom.xml and a change from a code review.
NIFI-5214 Added another tag based on code review.
NIFI-5214 Added user-defined header support.
NIFI-5214 Added Basic Auth support.
NIFI-5214 Moved documentation.
NIFI-5214 Fixed checkstyle issues; added changes requested in a review.
NIFI-5214 Added changes requested in a code review.
NIFI-5214 Added verb test and @DynamicProperties
NIFI-5214 Added templated URL support to RestLookupService.
NIFI-5214 Fixed documentation based on code review changes.
NIFI-5214 Changed RestLookupService to use a property descriptor.
NIFI-5214 Updated documentation.
NIFI-5214 Made changes requested in code review.
NIFI-5214 Renamed nifi-standard-web-utils to reflect that it is for tests.
NIFI-5214: Refactor RestLookupService.
1. Added 'Base URL' property to address environment specific part of URL.
2. Removed 'Record Path Property Name' property, because the name of
a resulted record field of a record path can be obtained by field name.
3. Lower cased HTTP method name should be used throughout.
4. Added mimeType require check when body is specified.
5. Added debug log to print HTTP response code.
6. Prepare for NIFI-5287.
7. Fixed that mime.type being used regardless of whether body is
specified or not, caused NullPointerException when 'mime.type' is not
specified when it is not required.
8. Updated documentation.
NIFI-5214 Fixed AWS processors that broke w/ change to TestServer location.
NIFI-5214 Added changes requested in a code review.
Refactored the way to evaluate EL for URL property
- Use PropertyValue instead of PreparedQuery to utilize Variable
Registry.
- Removed BASE_URL because Variable Registry can be used at URL
NIFI-5214 Rebased to use new LookupService method.
This closes#2723.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-5287 Removed unneeded property descriptor.
NIFI-5287 Added additional changes from a code review.
This closes#2777.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-5231 Minor fixes from code review.
NIFI-5231 Added documentation and made record_count a constant.
NIFI-5231 Added changes requested in a code review.
NIFI-5231 Added changes based on code review.
This closes#2737.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
commit 9c31e45d3f
Author: Joe Trite <joetrite@gmail.com>
Date: Mon Mar 13 07:55:19 2017 -0400
NIFI-1705 Adding AttributesToCSV processor
commit 5e9afa9ccf
Merge: 3177eb18574cbfc4b6
Author: Joe Trite <joetrite@gmail.com>
Date: Sat Mar 4 08:12:39 2017 -0500
Merge remote-tracking branch 'origin/master'
# Conflicts:
# nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ScanAttribute.java
commit 3177eb1852
Author: Joe Trite <joetrite@gmail.com>
Date: Sat Mar 4 08:09:06 2017 -0500
NIFI-3497 Converted properties to use displayName, changed validator on demarcator property and created a method for duplicate code.
commit 74cbfc4b69
Merge: a974f78e0f8cad0f8c
Author: Joe Trite <joetrite@gmail.com>
Date: Sat Mar 4 07:47:46 2017 -0500
Merge branch 'master' into master
commit a974f78e03
Author: Joe Trite <joetrite@gmail.com>
Date: Sat Mar 4 07:43:02 2017 -0500
NIFI-3497 Converted properties to use displayName, changed validator on demarcator property and created a method for duplicate code.
commit 1bfaef8e87
Merge: 65ed46de989ec68d14
Author: Joe Trite <joetrite@gmail.com>
Date: Fri Mar 3 08:01:59 2017 -0500
Merge branch 'master' of https://github.com/joetrite/nifi
commit 65ed46de9a
Author: Joe Trite <joetrite@gmail.com>
Date: Fri Feb 24 18:09:36 2017 -0500
NIFI-3497 - fixing Pcontrib issues
commit c5d52cf6f0
Author: Joe Trite <joetrite@gmail.com>
Date: Thu Feb 23 10:19:01 2017 -0500
NIFI-3497 - excluding test files
Adding new test data files to exclude list.
commit b195934113
Author: Joe Trite <joetrite@gmail.com>
Date: Wed Feb 22 16:48:10 2017 -0500
NIFI-3497 - New dictionary files for test
Adding new dictionary files to support metadata dictionary option.
commit e296268f39
Author: Joe Trite <joetrite@gmail.com>
Date: Wed Feb 22 16:46:13 2017 -0500
NIFI-3497 test cases for metadata updates
Adding test cases to support metadata option update.
commit de7e348e62
Author: Joe Trite <joetrite@gmail.com>
Date: Wed Feb 22 16:36:08 2017 -0500
NIFI-3497 - Added metadata option
Added optional to post additional metadata as new attributed if a match is found in the dictionary.
commit 89ec68d14b
Author: Joe Trite <joetrite@gmail.com>
Date: Fri Feb 24 18:09:36 2017 -0500
NIFI-3497 - fixing Pcontrib issues
commit d71426037b
Author: Joe Trite <joetrite@gmail.com>
Date: Thu Feb 23 10:19:01 2017 -0500
NIFI-3497 - excluding test files
Adding new test data files to exclude list.
commit a7a7b6ace8
Author: Joe Trite <joetrite@gmail.com>
Date: Wed Feb 22 16:48:10 2017 -0500
NIFI-3497 - New dictionary files for test
Adding new dictionary files to support metadata dictionary option.
commit 8eb54a5019
Author: Joe Trite <joetrite@gmail.com>
Date: Wed Feb 22 16:46:13 2017 -0500
NIFI-3497 test cases for metadata updates
Adding test cases to support metadata option update.
commit f52e1f2a06
Author: Joe Trite <joetrite@gmail.com>
Date: Wed Feb 22 16:36:08 2017 -0500
NIFI-3497 - Added metadata option
Added optional to post additional metadata as new attributed if a match is found in the dictionary.
NIFI-1705 Adding AttributesToCSV processor.
NIFI-1705 Adding AttributesToCSV processor.
NIFI-1705 Adding AttributesToCSV processor.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2711
- PR Fix - 'Execution' dropdown will now be shown in all cases
- Annotated ListGCSBucket with PrimaryNodeOnly
This closes#2509.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Added ProxyConfigurationService to manage centralized proxy
configurations
- Adopt ProxyConfigurationService at FTP and HTTP processors
NIFI-4175 - Add HTTP proxy support to *SFTP processors
This closes#2018.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-4199: Add ProxyConfigurationService to SFTP processors
- Fixed check style issue
- Use the same proxy related PropertyDescriptors from FTPTransfer and
SFTPTransfer
- Dropped FlowFile EL evaluation support to make it align with other
processors spec, Now it supports VARIABLE_REGISTRY
- Added ProxyConfigurationService to SFTP processors
- Added SOCKS proxy support to SFTP processors
NIFI-4199: Added ProxyConfigurationService to ElasticsearchHttp processors
- ElasticsearchHttp processors now support SOCKS proxy, too
- Added proxy support to PutElasticsearchHttpRecord
- Moved more common property descriptors to
AbstractElasticsearchHttpProcessor and just return static unmodifiable
property descriptor list at each implementation processors
NIFI-4196 - Expose AWS proxy authentication settings
NIFI-4196 - Fix jUnit errors
This closes#2016.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-4199: Add ProxyConfigService to AWS processors
- Applied ProxyConfigService to S3 processors
- Added proxy support to following processors:
- PutKinesisFirehose, PutKinesisStream
- PutDynamoDB, DeleteDynamoDB, GetDynamoDB
- PutKinesisStream
- All AWS processors support HTTP proxy now
NIFI-4199: Add ProxyConfigService to Azure processors
NIFI-4199: More explicit validation and docs for Proxy spec
- Each processor has different supporting Proxy specs
- Show supported spec to ProxyConfigurationService property doc
- Validate not only Proxy type, but also with Authentication
NIFI-4199: Incorporated review comments
- Fixed TestListS3 property descriptor check
- Separate name and displayName
This closes#2016
This closes#2018
This closes#2704
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
NIFI-5204: If processor joins cluster and inherits 'disabled' state but is still stopping, ensure that the state becomes disabled when the processor finishes stopping
This closes#2713
NIFI-950: Still seeing some slow response times when instantiating a large template in cluster mode so making some minor tweaks based on the results of CPU profiling
NIFI-5112: Refactored FlowSerializer so that it creates the desired intermediate data model that can be serialized, separate from serializing. This allows us to hold the FlowController's Read Lock only while creating the data model, not while actually serializing the data. Configured Jersey Client in ThreadPoolRequestReplicator not to look for features using the Service Loader for every request. Updated Template object to hold a DOM Node that represents the template contents instead of having to serialize the DTO, then parse the serialized form as a DOM object each time that it needs to be serialized.
NIFI-5112: Change ThreadPoolRequestReplicator to use OkHttp client instead of Jersey Client
NIFI-5111: Ensure that if a node is no longer cluster coordinator, that it clears any stale heartbeats.
NIFI-5110: Notify StandardProcessScheduler when a component is removed so that it will clean up any resource related to component lifecycle.
NIFI-950: Avoid gathering the Status objects for entire flow when we don't need them; removed unnecessary code
NIFI-950: Bug fixes
NIFI-950: Bug fix; added validation status to ProcessorDTO, ControllerServiceDTO, ReportingTaskDTO; updated DebugFlow to allow for pause time to be set in the customValidate method for testing functionality
NIFI-950: Addressing test failures
NIFI-950: Bug fixes
NIFI-950: Addressing review feedback
NIFI-950: Fixed validation logic in mock framework
This closes#2693
Support for the new option to keep empty captures. I did not add to the GrokReader because I am not sure of the effect on
the schema extraction.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#2698.
QueryDatabaseTable and GenerateTableFetch processors were not able to
use max value state as expected, if max value column was wrapped with
escape characters. Due to a mis-match between computed state keys
and actual keys used in the managed state. State keys computed by
getStateKey method included escape characters while actual stored keys
did not. Resulted querying the same dataset again and again.
This commit added unwrapIdentifier method to DatabaseAdapter class to
remove database specific escape characters for identifiers such as table
and column names, so that max value state keys are populated correctly
even if identifiers are wrapped with escape characters.
This commit also added new DatabaseAdapter for MySQL, to handle MySQL
specific identifier escape with back-ticks.
This closes#2424
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
Support creating target parent directories even if directory listing is disabled.
fixed typo in doc
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2642
- Only include Maximum Value columns in the type map.
- Squashed commits in the previous PR
- Rebased against the latest master
- Added stop method to GenerateTableFetch so that it refreshes the
column type map when it gets restarted
- Fixed whitespacing around if/for statement
- Updated expressionLanguageSupported value since it is not auto-merged
correctly
This closes#2618.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- take into account input requirement for documentation rendering
- Renamed variable registry scope and added comments
- Doc + change in mock framework to check scope + update of components + UI
- Fixing required permission for PutParquet.
NIFI-5008:
- Ensuring all restricted components are tagged as such.
This closes#2583.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Initial set of components marked with the HighResourceUsageScenario annotation.
- Added customized descriptions to SystemResourceConsideration annotations for MergeContent, SplitContent, SplitJson, SplitText, and SplitXml.
This closes#2475.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- When Jetty initializes fails, performing a shutdown sequence to ensure all allocated resources are released.
This closes#2455.
Signed-off-by: Bryan Bende <bbende@apache.org>
Before this fix, MergeContent only processed the first bin even if there
were multiple bins.
There were two unit tests marked with Ignore those had been
failing because of this.
This closes#2444.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2307
Replaced star imports, removed unused import
Added explanation for invalid Expression
- Added check for keystore properties and only initialized keystore when necessary.
- Added TestInvokeHttpTwoWaySSL test class to test with two-way SSL
- Modified TestInvokeHttpSSL to test with one-way SSL
Signed-off-by: joewitt <joewitt@apache.org>
NIFI-4559: Removed Penalize Non-zero Status property and updated doc per review comments
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2246
- Ensuring the reason behind the SchemaNotFoundException is reported when interacting with the configured SchemaRegistry.
Signed-off-by: joewitt <joewitt@apache.org>
- Upgrading to Jersey 2.x.
- Updating NOTICE files where necessary.
- Fixing checkstyle issues.
This closes#2206.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Upgraded immediately actionable dependency versions from Meterian report.
- Upgraded jackson-core test dependencies for HBase and Elasticsearch modules.
- Only 3 instances of jackson-core < 2.8.6 (Google Cloud Platform and Spark Receiver modules).
- Upgraded version of poi dependency in nifi-email-processors to 3.16.
- Resolving dependency issues after rebasing against 1.5.0-SNAPSHOT.
- Removed jackson-databind from <dependencyManagement> block in nifi/pom.xml and added explicit reference to ${jackson.version} in all referenced artifacts.
- Removed jackson-mapper-asl from <dependencyManagement> block in nifi/pom.xml and added explicit reference to ${jackson.old.version} in all referenced artifacts.
- Removed Jasypt from <dependencyManagement> and added explicit version in test dependency for legacy compatibility.
- This closes#2084
Forced HandleHTTPRequest to use RestrictedSSLContextService and removed extraneous SSL algorithm checks
Throw RuntimeException if the chosen SSL protocol isn't supported by HandleHttpRequest
This closes#1985.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Added external compatibility regression test for StringEncryptor to ensure continued functionality during removal of Jasypt.
Documents custom salt lengths and iteration counts for each encryption method.
Added (ignored) failing tests for keyed encryption (Jasypt does not support keyed encryption).
Changed StringEncryptor to non-final class and added protected default constructor.
Added failing test for initialization status.
Added utility methods in CipherUtility.
Moved PBE cipher providers (and tests) from nifi-standard-processors to nifi-security-utils module.
Implemented PBE and keyed encryption/decryption logic.
Moved Scrypt unit test back into scrypt package.
Resolved test failures in limited strength cryptographic environment.
Implemented keyed encryption/decryption and enabled unit tests.
Removed Jasypt dependency from production scope (kept in test scope for backward compatibility tests).
Signed-off-by: joewitt <joewitt@apache.org>
The unit test for DATE type used GMT timezone, that causes an assertion error in timezones such as EST (-5).
We need to use local timezone instead of GMT, as Derby and PutSQL uses local timezone.
The unit test failed before as follows:
- Unit test code, passed: '2002-02-02 GMT'
- PutSQL code convertedi it to local: '2002-02-01 EST', and stored as '2002-02-01' in Derby database without timezone info
- Unit test code SELECT the inserted value, passed a GMT calender, then got epoch timestamp, which was '2002-01-31'
Support negative long value for timezones ahead of UTC.
- For timezones such as '+0800', it's possible that a local time e.g. '02:03:04' can be a negative epoch value. This commit changes LONG_PATTERN so that it can accept nevative values.
- Changed time values in unit tests to verify negative epoch values, and avoid using the same digits among different time unit for better readability.
This closes#2082
- Unmatched fields were ignored, but the number of prepared statement
place holders were not correct.
- Added unit test code for generateUpdate.
- Added unit test code with "Ignore Unmatched Columns".
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#2165.
- Removed FlowFile from RecordReaderFactory, RecordSetWriterFactory and SchemaAccessStrategy.
- Renamed variable 'allowableValue' to 'strategy' to represent its meaning better.
- Removed creation of temporal FlowFile to resolve Record Schema from ConsumerLease.
- Removed unnecessary 'InputStream content' argument from
RecordSetWriterFactory.getSchema method.
This closes#1877.
fix checkstyle issue, and added unit test showing data duplication issue, removed property
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2091
Before this fix, it's possible that ListXXX processors can miss files those have the same timestamp as the one which was the latest processed timestamp at the previous cycle. Since it only used timestamps, it was not possible to determine whether a file is already processed or not.
However, storing every single processed identifier as we used to will not perform well.
Instead, this commit makes ListXXX to store only identifiers those have the latest timestamp at a cycle to minimize the amount of state data to store.
NIFI-3332: ListXXX to not miss files with the latest processed timestamp
- Fixed TestAbstractListProcessor to use appropriate time precision.
Without this fix, arbitrary test can fail if generated timestamp does
not have the desired time unit value, e.g. generated '10:51:00' where
second precision is tested.
- Fixed TestFTP.basicFileList to use millisecond time precision explicitly
because FakeFtpServer's time precision is in minutes.
- Changed junit dependency scope to 'provided' as it is needed by
ListProcessorTestWatcher which is shared among different modules.
This closes#1975.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Refactored variable names to better represents what those are meant for.
- Added deterministic logic which detects target filesystem timestamp precision and adjust lag time based on it.
- Changed from using System.nanoTime() to System.currentTimeMillis in test because Java File API reports timestamp in milliseconds at the best granularity. Also, System.nanoTime should not be used in mix with epoch milliseconds because it uses arbitrary origin and measured differently.
- Changed TestListFile to use more longer interval between file timestamps those are used by testFilterAge to provide more consistent test result because sleep time can be longer with filesystems whose timestamp in seconds precision.
- Added logging at TestListFile.
- Added TestWatcher to dump state in case assertion fails for further investigation.
- Added Timestamp Precision property so that user can set if auto-detect is not enough
- Adjust timestamps for ages test
This closes#1915.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-4028: Refactored Wait processor.
- Consolidated implementation for the cases of releasableFlowCount is 1 or more, in order to reduce complexity and behavior differences
- Added 'consumed' counter when total counter is used to release incoming FlowFiles
- Fixed method name typo, releaseCandidates
This closes#2055.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
The properties are already evaluated against EL. Also added an
integration test that is failing without the change.
This closes#1968.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-3281 - Review - handle completePendingCommand return and added a unit test for ListFTP
NIFI-3281 - Review - Added flow file for EL evaluation in other methods and added unit test for NIFI-3590
This closes#1974.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Moved key provider interface and implementations from nifi-data-provenance-utils module to nifi-security-utils module.
- Refactored duplicate byte[] concatenation methods from utility classes and removed deprecation warnings from CipherUtility.
- Created KeyProviderFactory to encapsulate key provider instantiation logic.
- Added logic to handle legacy package configuration values for key providers.
- Added unit tests.
- Added resource files for un/limited strength cryptography scenarios.
- Added ASL to test resources.
- Moved legacy FQCN handling logic to CryptUtils.
- Added unit tests to ensure application startup logic handles legacy FQCNs.
- Moved master key extraction/provision out of FBKP.
- Removed nifi-security-utils dependency on nifi-properties-loader module.
- Added unit tests.
Before this fix, it is possible that TailFile to produce duplicated data
if an already tailed file has newer timestamp and fewer or the same
amout of data.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#2021.
Fix unit test for Date and Time type time zone problem
Enhance Time type to record milliseconds
This closes#1983.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Changed boolean value conversion to use Boolean.valueOf.
- Updated comments in source code to reflect current situation more clearly.
- Updated tests those have been added since the original commits were made.
NIFI-4060: Addressed threading issue with RecordBin being updated after it is completed; fixed issue that caused mime.type attribute not to be written properly if all incoming flowfiles already have a different value for that attribute
NIFI-4060: Bug fixes; improved documentation; added a lot of debug information; updated StandardProcessSession to produce more accurate logs in case of a session being committed/rolled back with open input/output streams
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1958
NIFI-4061 Initial version of RedisStateProvider
- Adding PropertyContext and updating existing contexts to extend it
- Added embedded Redis for unit testing
- Added wrapped StateProvider with NAR ClassLoader in StandardStateManagerProvider
- Updating state-management.xml with config for RedisStateProvider
- Renaming tests that use RedisServer to be IT tests so they don't run all the time
This closes#1918.
To be used with cache engines that does not have revision number.
NIFI-4049: Refactor AtomicDistributedMapCacheClient
Removed old methods completely.
This closes#1904.
Signed-off-by: Bryan Bende <bbende@apache.org>
Mark Payne
a.durov
erichanson5
Koji Kawamura
Otto Fowler
SavtechSolutions
Pierre Villard
Matthew Burgess
zenfenan
joewitt
Kumara M S Hemantha
yjhyjhyjh0
Colin Dean
vadimar
Arek Burdach
Vish Uma
Kotaro Terada
Gardella Juan Pablo
Jeff Storck
juliansniff
Bryan Bende
rednikotin
patricker
Endre Zoltan Kovacs
Brad Hards
Endre Zoltan Kovacs
Andy LoPresto
Nick Lewis
Lars Francke
Mike Moser
Marco Gaido
Mike Thomsen
Bartlomiej Tartanus
Dustin Rodrigues
Joe Trite
Nicolas Sanglard
Andrew Psaltis
Joe Percivall
Mark Bean
Matt Gilman
sbouchex
m-hogue
Andy LoPresto
Marco Gaido
Norito Agetsuma
trixpan
Josh Anderton
Julian Gimbel
Andrew Lim
Gino Lisignoli
Arun Manivannan
Wayne Steel
Wil Selwood
Leah Anderson
deonhuang
Toivo Adams
mans2singh
Tony Kurc
yuri1969