- Updated FlowFile Repo / Write Ahead Log so that any update that writes more than 1 MB of data is written to a file inside the FlowFile Repo rather than being buffered in memory
- Update SplitText so that it does not hold FlowFiles that are not the latest version in heap. Doing them from being garbage collected, so while the Process Session is holding the latest version of the FlowFile, SplitText is holding an older version, and this results in two copies of the same FlowFile object
NIFI-5533: Checkpoint
NIFI-5533: Bug Fixes
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2974
Refactored some test code to be clearer.
Renamed some resources to be consistent across modules.
Changed passwords to meet new minimum length requirements.
This closes#3018
Refactors tests in order to share code repeated in tests and to enable
some parameterized testing.
MySQL Connector/J 5.1.x in conjunction with MySQL 5.0.x will return
a Long for ResultSet#getObject when the SQL type is an unsigned integer.
This change prevents that error from occurring while implementing a more
informational exception describing what the failing object's POJO type
is in addition to its string value.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3032
NIFI-5514: Do not rely on ProcessSession.getQueueSize() to return a queue size of 0 objects because if the processor is holding onto data, the queue size won't be 0.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#2954.
Deprecated HashContent processor.
Added documentation to HashAttribute processor.
Added shared HashService and HashAlgorithm enum.
Added unit tests.
Added #clearProperties() to TestRunner, StandardProcessorTestRunner, and MockProcessContext.
Updated processor manifest.
Updated Javadoc.
Added documentation about deprecated/renamed components.
Added logic handling for UTF-16 encoding where UTF-16BE is overridden so the BOM is not inserted prior to hashing.
This closes#2983.
Co-authored-by: Otto Fowler <ottobackwards@gmail.com>
Signed-off-by: Kevin Doran <kdoran@apache.org>
This reverts commit f7f809c3d3.
NIFI-5474, NIFI-4272: When using Regex Replace with ReplaceText, and there are capturing groups, ensure that we populate the 'additionalVariables' map for each match of the regex. This allows Expression Language to reference the back-references properly even when there are multiple matches
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2951
Adding followings:
- Use separate DistributedMapCache for tracking entities to avoid
conflict with existing code
- Added more validation
- Delete listed entities from cache if reset is needed
- Support Local scope
- Added Initial Listing Target
This closes#2876.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Create nifi-syslog-utils to move syslog parsing functionalty to a central location shared by the processors and serialization/record system.
- Refactor Processors to use these utils
- Update 5424 syslog classes using simple-syslog-5424 to pick up new changes to support this work, as well as keep dependencies/types from bleeding out to the
processors or readers
- Refactor Syslog5424Event and Parser
- Create Syslog5424RecordReader
- per review, handle blank message differently from eof
- name schema per review
This closes#2816.
Signed-off-by: Bryan Bende <bbende@apache.org>
Updated Javadoc for SiteToSiteClient#createTransaction() and HttpClient implementation.
Reverted exception listing in method contract for SiteToSiteClient#createTransaction and HttpClient tion of same.
Reverted import ordering in TestSiteToSiteClient.
Reverted exception listing in TestGetHDFSFileInfo, TestListHDFS, and StandardHttpFlowFileServerProtocol.
Restored @SuppressWarnings annotation and removed unnecessary "public static" keywords from inner classes in SiteToSiteClient.
This closes#2841.
Signed-off-by: Joe Witt <joewitt@apache.org>
- address review comments
- per review, add property for including body in attributes
- per review, create processor in OnScheduled
This closes#2805.
Signed-off-by: Bryan Bende <bbende@apache.org>
forgot to penalize flowfiles sent to failure
NIFI-5274 avoid rollback on StackOverflowError in ReplaceText
add comment
removed penalize on failure, doesn't really apply here
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2767
NIFI-5214 Added support for the new ProxyConfigurationService
NIFI-5214 Integration tests added.
NIFI-5214 Added missing pom.xml and a change from a code review.
NIFI-5214 Added another tag based on code review.
NIFI-5214 Added user-defined header support.
NIFI-5214 Added Basic Auth support.
NIFI-5214 Moved documentation.
NIFI-5214 Fixed checkstyle issues; added changes requested in a review.
NIFI-5214 Added changes requested in a code review.
NIFI-5214 Added verb test and @DynamicProperties
NIFI-5214 Added templated URL support to RestLookupService.
NIFI-5214 Fixed documentation based on code review changes.
NIFI-5214 Changed RestLookupService to use a property descriptor.
NIFI-5214 Updated documentation.
NIFI-5214 Made changes requested in code review.
NIFI-5214 Renamed nifi-standard-web-utils to reflect that it is for tests.
NIFI-5214: Refactor RestLookupService.
1. Added 'Base URL' property to address environment specific part of URL.
2. Removed 'Record Path Property Name' property, because the name of
a resulted record field of a record path can be obtained by field name.
3. Lower cased HTTP method name should be used throughout.
4. Added mimeType require check when body is specified.
5. Added debug log to print HTTP response code.
6. Prepare for NIFI-5287.
7. Fixed that mime.type being used regardless of whether body is
specified or not, caused NullPointerException when 'mime.type' is not
specified when it is not required.
8. Updated documentation.
NIFI-5214 Fixed AWS processors that broke w/ change to TestServer location.
NIFI-5214 Added changes requested in a code review.
Refactored the way to evaluate EL for URL property
- Use PropertyValue instead of PreparedQuery to utilize Variable
Registry.
- Removed BASE_URL because Variable Registry can be used at URL
NIFI-5214 Rebased to use new LookupService method.
This closes#2723.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-5287 Removed unneeded property descriptor.
NIFI-5287 Added additional changes from a code review.
This closes#2777.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-5231 Minor fixes from code review.
NIFI-5231 Added documentation and made record_count a constant.
NIFI-5231 Added changes requested in a code review.
NIFI-5231 Added changes based on code review.
This closes#2737.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
commit 9c31e45d3f
Author: Joe Trite <joetrite@gmail.com>
Date: Mon Mar 13 07:55:19 2017 -0400
NIFI-1705 Adding AttributesToCSV processor
commit 5e9afa9ccf
Merge: 3177eb18574cbfc4b6
Author: Joe Trite <joetrite@gmail.com>
Date: Sat Mar 4 08:12:39 2017 -0500
Merge remote-tracking branch 'origin/master'
# Conflicts:
# nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ScanAttribute.java
commit 3177eb1852
Author: Joe Trite <joetrite@gmail.com>
Date: Sat Mar 4 08:09:06 2017 -0500
NIFI-3497 Converted properties to use displayName, changed validator on demarcator property and created a method for duplicate code.
commit 74cbfc4b69
Merge: a974f78e0f8cad0f8c
Author: Joe Trite <joetrite@gmail.com>
Date: Sat Mar 4 07:47:46 2017 -0500
Merge branch 'master' into master
commit a974f78e03
Author: Joe Trite <joetrite@gmail.com>
Date: Sat Mar 4 07:43:02 2017 -0500
NIFI-3497 Converted properties to use displayName, changed validator on demarcator property and created a method for duplicate code.
commit 1bfaef8e87
Merge: 65ed46de989ec68d14
Author: Joe Trite <joetrite@gmail.com>
Date: Fri Mar 3 08:01:59 2017 -0500
Merge branch 'master' of https://github.com/joetrite/nifi
commit 65ed46de9a
Author: Joe Trite <joetrite@gmail.com>
Date: Fri Feb 24 18:09:36 2017 -0500
NIFI-3497 - fixing Pcontrib issues
commit c5d52cf6f0
Author: Joe Trite <joetrite@gmail.com>
Date: Thu Feb 23 10:19:01 2017 -0500
NIFI-3497 - excluding test files
Adding new test data files to exclude list.
commit b195934113
Author: Joe Trite <joetrite@gmail.com>
Date: Wed Feb 22 16:48:10 2017 -0500
NIFI-3497 - New dictionary files for test
Adding new dictionary files to support metadata dictionary option.
commit e296268f39
Author: Joe Trite <joetrite@gmail.com>
Date: Wed Feb 22 16:46:13 2017 -0500
NIFI-3497 test cases for metadata updates
Adding test cases to support metadata option update.
commit de7e348e62
Author: Joe Trite <joetrite@gmail.com>
Date: Wed Feb 22 16:36:08 2017 -0500
NIFI-3497 - Added metadata option
Added optional to post additional metadata as new attributed if a match is found in the dictionary.
commit 89ec68d14b
Author: Joe Trite <joetrite@gmail.com>
Date: Fri Feb 24 18:09:36 2017 -0500
NIFI-3497 - fixing Pcontrib issues
commit d71426037b
Author: Joe Trite <joetrite@gmail.com>
Date: Thu Feb 23 10:19:01 2017 -0500
NIFI-3497 - excluding test files
Adding new test data files to exclude list.
commit a7a7b6ace8
Author: Joe Trite <joetrite@gmail.com>
Date: Wed Feb 22 16:48:10 2017 -0500
NIFI-3497 - New dictionary files for test
Adding new dictionary files to support metadata dictionary option.
commit 8eb54a5019
Author: Joe Trite <joetrite@gmail.com>
Date: Wed Feb 22 16:46:13 2017 -0500
NIFI-3497 test cases for metadata updates
Adding test cases to support metadata option update.
commit f52e1f2a06
Author: Joe Trite <joetrite@gmail.com>
Date: Wed Feb 22 16:36:08 2017 -0500
NIFI-3497 - Added metadata option
Added optional to post additional metadata as new attributed if a match is found in the dictionary.
NIFI-1705 Adding AttributesToCSV processor.
NIFI-1705 Adding AttributesToCSV processor.
NIFI-1705 Adding AttributesToCSV processor.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2711
- PR Fix - 'Execution' dropdown will now be shown in all cases
- Annotated ListGCSBucket with PrimaryNodeOnly
This closes#2509.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Added ProxyConfigurationService to manage centralized proxy
configurations
- Adopt ProxyConfigurationService at FTP and HTTP processors
NIFI-4175 - Add HTTP proxy support to *SFTP processors
This closes#2018.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-4199: Add ProxyConfigurationService to SFTP processors
- Fixed check style issue
- Use the same proxy related PropertyDescriptors from FTPTransfer and
SFTPTransfer
- Dropped FlowFile EL evaluation support to make it align with other
processors spec, Now it supports VARIABLE_REGISTRY
- Added ProxyConfigurationService to SFTP processors
- Added SOCKS proxy support to SFTP processors
NIFI-4199: Added ProxyConfigurationService to ElasticsearchHttp processors
- ElasticsearchHttp processors now support SOCKS proxy, too
- Added proxy support to PutElasticsearchHttpRecord
- Moved more common property descriptors to
AbstractElasticsearchHttpProcessor and just return static unmodifiable
property descriptor list at each implementation processors
NIFI-4196 - Expose AWS proxy authentication settings
NIFI-4196 - Fix jUnit errors
This closes#2016.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-4199: Add ProxyConfigService to AWS processors
- Applied ProxyConfigService to S3 processors
- Added proxy support to following processors:
- PutKinesisFirehose, PutKinesisStream
- PutDynamoDB, DeleteDynamoDB, GetDynamoDB
- PutKinesisStream
- All AWS processors support HTTP proxy now
NIFI-4199: Add ProxyConfigService to Azure processors
NIFI-4199: More explicit validation and docs for Proxy spec
- Each processor has different supporting Proxy specs
- Show supported spec to ProxyConfigurationService property doc
- Validate not only Proxy type, but also with Authentication
NIFI-4199: Incorporated review comments
- Fixed TestListS3 property descriptor check
- Separate name and displayName
This closes#2016
This closes#2018
This closes#2704
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
NIFI-5204: If processor joins cluster and inherits 'disabled' state but is still stopping, ensure that the state becomes disabled when the processor finishes stopping
This closes#2713
NIFI-950: Still seeing some slow response times when instantiating a large template in cluster mode so making some minor tweaks based on the results of CPU profiling
NIFI-5112: Refactored FlowSerializer so that it creates the desired intermediate data model that can be serialized, separate from serializing. This allows us to hold the FlowController's Read Lock only while creating the data model, not while actually serializing the data. Configured Jersey Client in ThreadPoolRequestReplicator not to look for features using the Service Loader for every request. Updated Template object to hold a DOM Node that represents the template contents instead of having to serialize the DTO, then parse the serialized form as a DOM object each time that it needs to be serialized.
NIFI-5112: Change ThreadPoolRequestReplicator to use OkHttp client instead of Jersey Client
NIFI-5111: Ensure that if a node is no longer cluster coordinator, that it clears any stale heartbeats.
NIFI-5110: Notify StandardProcessScheduler when a component is removed so that it will clean up any resource related to component lifecycle.
NIFI-950: Avoid gathering the Status objects for entire flow when we don't need them; removed unnecessary code
NIFI-950: Bug fixes
NIFI-950: Bug fix; added validation status to ProcessorDTO, ControllerServiceDTO, ReportingTaskDTO; updated DebugFlow to allow for pause time to be set in the customValidate method for testing functionality
NIFI-950: Addressing test failures
NIFI-950: Bug fixes
NIFI-950: Addressing review feedback
NIFI-950: Fixed validation logic in mock framework
This closes#2693
Support for the new option to keep empty captures. I did not add to the GrokReader because I am not sure of the effect on
the schema extraction.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#2698.
QueryDatabaseTable and GenerateTableFetch processors were not able to
use max value state as expected, if max value column was wrapped with
escape characters. Due to a mis-match between computed state keys
and actual keys used in the managed state. State keys computed by
getStateKey method included escape characters while actual stored keys
did not. Resulted querying the same dataset again and again.
This commit added unwrapIdentifier method to DatabaseAdapter class to
remove database specific escape characters for identifiers such as table
and column names, so that max value state keys are populated correctly
even if identifiers are wrapped with escape characters.
This commit also added new DatabaseAdapter for MySQL, to handle MySQL
specific identifier escape with back-ticks.
This closes#2424
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
Support creating target parent directories even if directory listing is disabled.
fixed typo in doc
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2642
- Only include Maximum Value columns in the type map.
- Squashed commits in the previous PR
- Rebased against the latest master
- Added stop method to GenerateTableFetch so that it refreshes the
column type map when it gets restarted
- Fixed whitespacing around if/for statement
- Updated expressionLanguageSupported value since it is not auto-merged
correctly
This closes#2618.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- take into account input requirement for documentation rendering
- Renamed variable registry scope and added comments
- Doc + change in mock framework to check scope + update of components + UI
- Fixing required permission for PutParquet.
NIFI-5008:
- Ensuring all restricted components are tagged as such.
This closes#2583.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Initial set of components marked with the HighResourceUsageScenario annotation.
- Added customized descriptions to SystemResourceConsideration annotations for MergeContent, SplitContent, SplitJson, SplitText, and SplitXml.
This closes#2475.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- When Jetty initializes fails, performing a shutdown sequence to ensure all allocated resources are released.
This closes#2455.
Signed-off-by: Bryan Bende <bbende@apache.org>
Before this fix, MergeContent only processed the first bin even if there
were multiple bins.
There were two unit tests marked with Ignore those had been
failing because of this.
This closes#2444.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2307
Replaced star imports, removed unused import
Added explanation for invalid Expression
- Added check for keystore properties and only initialized keystore when necessary.
- Added TestInvokeHttpTwoWaySSL test class to test with two-way SSL
- Modified TestInvokeHttpSSL to test with one-way SSL
Signed-off-by: joewitt <joewitt@apache.org>
NIFI-4559: Removed Penalize Non-zero Status property and updated doc per review comments
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2246
- Ensuring the reason behind the SchemaNotFoundException is reported when interacting with the configured SchemaRegistry.
Signed-off-by: joewitt <joewitt@apache.org>
- Upgrading to Jersey 2.x.
- Updating NOTICE files where necessary.
- Fixing checkstyle issues.
This closes#2206.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Upgraded immediately actionable dependency versions from Meterian report.
- Upgraded jackson-core test dependencies for HBase and Elasticsearch modules.
- Only 3 instances of jackson-core < 2.8.6 (Google Cloud Platform and Spark Receiver modules).
- Upgraded version of poi dependency in nifi-email-processors to 3.16.
- Resolving dependency issues after rebasing against 1.5.0-SNAPSHOT.
- Removed jackson-databind from <dependencyManagement> block in nifi/pom.xml and added explicit reference to ${jackson.version} in all referenced artifacts.
- Removed jackson-mapper-asl from <dependencyManagement> block in nifi/pom.xml and added explicit reference to ${jackson.old.version} in all referenced artifacts.
- Removed Jasypt from <dependencyManagement> and added explicit version in test dependency for legacy compatibility.
- This closes#2084
Forced HandleHTTPRequest to use RestrictedSSLContextService and removed extraneous SSL algorithm checks
Throw RuntimeException if the chosen SSL protocol isn't supported by HandleHttpRequest
This closes#1985.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Added external compatibility regression test for StringEncryptor to ensure continued functionality during removal of Jasypt.
Documents custom salt lengths and iteration counts for each encryption method.
Added (ignored) failing tests for keyed encryption (Jasypt does not support keyed encryption).
Changed StringEncryptor to non-final class and added protected default constructor.
Added failing test for initialization status.
Added utility methods in CipherUtility.
Moved PBE cipher providers (and tests) from nifi-standard-processors to nifi-security-utils module.
Implemented PBE and keyed encryption/decryption logic.
Moved Scrypt unit test back into scrypt package.
Resolved test failures in limited strength cryptographic environment.
Implemented keyed encryption/decryption and enabled unit tests.
Removed Jasypt dependency from production scope (kept in test scope for backward compatibility tests).
Signed-off-by: joewitt <joewitt@apache.org>
The unit test for DATE type used GMT timezone, that causes an assertion error in timezones such as EST (-5).
We need to use local timezone instead of GMT, as Derby and PutSQL uses local timezone.
The unit test failed before as follows:
- Unit test code, passed: '2002-02-02 GMT'
- PutSQL code convertedi it to local: '2002-02-01 EST', and stored as '2002-02-01' in Derby database without timezone info
- Unit test code SELECT the inserted value, passed a GMT calender, then got epoch timestamp, which was '2002-01-31'
Support negative long value for timezones ahead of UTC.
- For timezones such as '+0800', it's possible that a local time e.g. '02:03:04' can be a negative epoch value. This commit changes LONG_PATTERN so that it can accept nevative values.
- Changed time values in unit tests to verify negative epoch values, and avoid using the same digits among different time unit for better readability.
This closes#2082
- Unmatched fields were ignored, but the number of prepared statement
place holders were not correct.
- Added unit test code for generateUpdate.
- Added unit test code with "Ignore Unmatched Columns".
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#2165.
- Removed FlowFile from RecordReaderFactory, RecordSetWriterFactory and SchemaAccessStrategy.
- Renamed variable 'allowableValue' to 'strategy' to represent its meaning better.
- Removed creation of temporal FlowFile to resolve Record Schema from ConsumerLease.
- Removed unnecessary 'InputStream content' argument from
RecordSetWriterFactory.getSchema method.
This closes#1877.
fix checkstyle issue, and added unit test showing data duplication issue, removed property
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2091
Before this fix, it's possible that ListXXX processors can miss files those have the same timestamp as the one which was the latest processed timestamp at the previous cycle. Since it only used timestamps, it was not possible to determine whether a file is already processed or not.
However, storing every single processed identifier as we used to will not perform well.
Instead, this commit makes ListXXX to store only identifiers those have the latest timestamp at a cycle to minimize the amount of state data to store.
NIFI-3332: ListXXX to not miss files with the latest processed timestamp
- Fixed TestAbstractListProcessor to use appropriate time precision.
Without this fix, arbitrary test can fail if generated timestamp does
not have the desired time unit value, e.g. generated '10:51:00' where
second precision is tested.
- Fixed TestFTP.basicFileList to use millisecond time precision explicitly
because FakeFtpServer's time precision is in minutes.
- Changed junit dependency scope to 'provided' as it is needed by
ListProcessorTestWatcher which is shared among different modules.
This closes#1975.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Refactored variable names to better represents what those are meant for.
- Added deterministic logic which detects target filesystem timestamp precision and adjust lag time based on it.
- Changed from using System.nanoTime() to System.currentTimeMillis in test because Java File API reports timestamp in milliseconds at the best granularity. Also, System.nanoTime should not be used in mix with epoch milliseconds because it uses arbitrary origin and measured differently.
- Changed TestListFile to use more longer interval between file timestamps those are used by testFilterAge to provide more consistent test result because sleep time can be longer with filesystems whose timestamp in seconds precision.
- Added logging at TestListFile.
- Added TestWatcher to dump state in case assertion fails for further investigation.
- Added Timestamp Precision property so that user can set if auto-detect is not enough
- Adjust timestamps for ages test
This closes#1915.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-4028: Refactored Wait processor.
- Consolidated implementation for the cases of releasableFlowCount is 1 or more, in order to reduce complexity and behavior differences
- Added 'consumed' counter when total counter is used to release incoming FlowFiles
- Fixed method name typo, releaseCandidates
This closes#2055.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
The properties are already evaluated against EL. Also added an
integration test that is failing without the change.
This closes#1968.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-3281 - Review - handle completePendingCommand return and added a unit test for ListFTP
NIFI-3281 - Review - Added flow file for EL evaluation in other methods and added unit test for NIFI-3590
This closes#1974.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Moved key provider interface and implementations from nifi-data-provenance-utils module to nifi-security-utils module.
- Refactored duplicate byte[] concatenation methods from utility classes and removed deprecation warnings from CipherUtility.
- Created KeyProviderFactory to encapsulate key provider instantiation logic.
- Added logic to handle legacy package configuration values for key providers.
- Added unit tests.
- Added resource files for un/limited strength cryptography scenarios.
- Added ASL to test resources.
- Moved legacy FQCN handling logic to CryptUtils.
- Added unit tests to ensure application startup logic handles legacy FQCNs.
- Moved master key extraction/provision out of FBKP.
- Removed nifi-security-utils dependency on nifi-properties-loader module.
- Added unit tests.
Before this fix, it is possible that TailFile to produce duplicated data
if an already tailed file has newer timestamp and fewer or the same
amout of data.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#2021.
Fix unit test for Date and Time type time zone problem
Enhance Time type to record milliseconds
This closes#1983.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Changed boolean value conversion to use Boolean.valueOf.
- Updated comments in source code to reflect current situation more clearly.
- Updated tests those have been added since the original commits were made.
NIFI-4060: Addressed threading issue with RecordBin being updated after it is completed; fixed issue that caused mime.type attribute not to be written properly if all incoming flowfiles already have a different value for that attribute
NIFI-4060: Bug fixes; improved documentation; added a lot of debug information; updated StandardProcessSession to produce more accurate logs in case of a session being committed/rolled back with open input/output streams
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1958
NIFI-4061 Initial version of RedisStateProvider
- Adding PropertyContext and updating existing contexts to extend it
- Added embedded Redis for unit testing
- Added wrapped StateProvider with NAR ClassLoader in StandardStateManagerProvider
- Updating state-management.xml with config for RedisStateProvider
- Renaming tests that use RedisServer to be IT tests so they don't run all the time
This closes#1918.
To be used with cache engines that does not have revision number.
NIFI-4049: Refactor AtomicDistributedMapCacheClient
Removed old methods completely.
This closes#1904.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Oracle NUMBER can return 0 precision and -127 or 0 scale with variable scale NUMBER such as ROWNUM or function result
- Added 'Default Decimal Precision' and 'Default Decimal Scale' property to ExecuteSQL and QueryDatabaseTable to apply default precision and scale if those are unknown
- Coerce BigDecimal scale to field schema logical type, so that BigDecimals having different scale can be written
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1851
- Added dependency notice.
- Added EL evaluation at SimpleKeyValueLookupService.
- Updated documentation.
- Updated CommonsConfigurationLookupService to throw LookupFailureException if it fails to get configuration so that error messages can be displayed at each processor bulletin.
- Added calling getConfiguration at OnEnabled of CommonsConfigurationLookupService, so that the service will stay in Enabling state if there is any issue.
Signed-off-by: Joey Frazee <jfrazee@apache.org>
- Added Logical type support for DECIMAL/NUMBER, DATE, TIME and TIMESTAMP column types.
- Added Logical type 'decimal' to AvroReader so that Avro records with logical types written by ExecuteSQL and QueryDatabaseTable can be consumed by AvroReader.
- Added JdbcCommon.AvroConversionOptions to consolidate conversion options.
- Added 'Use Avro Logical Types' property to ExecuteSQL and QueryDatabaseTable to toggle whether to use Logical types.
- Added 'mime.type' FlowFile attribute as 'application/avro-binary' so that output FlowFiles can be displayed by content viewer.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1798
NIFI-3838: Updated version from 1.2.0-SNAPSHOT to 1.3.0-SNAPSHOT; removed unneeded value from AttributeExpression.ResultType enum
NIFI-3838: Addressed PR Review feedback
NIFI-3838: Allow for schemas to be merged together for a record; refactored RecordSetWriterFactory so that there is a method to obtain the schema and then the writer is created with that schema. Added additional unit tests
NIFI-3838: Addressed problems with documentation based on PR Review
NIFI-3838: Fixed checkstyle violation
NIFI-3838: Addressed issue of comparing different types of Number objects
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1772
NIFI-1833 Moved AbstractListProcessor.java, EntityListing.java, and ListableEntity.java from nifi-standard-processors into nifi-processor-utils
Moved TestAbstractListProcessor.java into nifi-processor-utils
Set nifi-azure-nar's nar dependency back to nifi-standard-services-api-nar
Fixed failing integration tests (ITFetchAzureBlobStorage.java, ITListAzureBlobStorage.java, and ITPutAzureStorageBlob.java) and refactored them to be able to run in parallel
NIFI-1833 Moved security notice info in the additional details documentation into the descriptions of the specific attributes for which those notices are intended
Added displayName usage to properties
Updated exception handling in FetchAzureBlobStorage.java and PutAzureBlobStorage.java to cause flowfiles with Output/InputStreamCallback failures to be routed to the processor's failure relationship
Cleaned up dependencies in pom
NIFI-1833 Removed unnecessary calls to map on Optional in the onTrigger exception handling of FetchAzureBlobStorage.java and PutAzureBlobStorage.java
NIFI-1833 Updates due to nifi-processor-utils being moved under nifi-nar-bundles
This closes#1719.
Signed-off-by: Bryan Rosander <brosander@apache.org>
Added src/test/resources/logback-test.xml files resetting log level from DEBUG (in nifi-data-provenance-utils) to WARN because later tests depend on MockComponentLog recording a certain number of messages and this number is different than expected if the log level is DEBUG.
This closes#1686.
Signed-off-by: Bryan Bende, Yolanda M. Davis, and Mark Payne
- Creating nifi-records-utils to share utility code from record services
- Refactoring Parquet tests to use MockRecorderParser and MockRecordWriter
- Refactoring AbstractPutHDFSRecord to use schema access strategy
- Adding custom validate to AbstractPutHDFSRecord and adding handling of UNION types when writing Records as Avro
- Refactoring project structure to get CS API references out of nifi-commons, introducing nifi-extension-utils under nifi-nar-bundles
- Updating abstract put/fetch processors to obtain the WriteResult and update flow file attributes
This closes#1712.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Added org.apache.nifi.processor.util.pattern package in nifi-processor-utils containing reusable functions to mix-in 'Rollback on Failure' capability.
- Created a process pattern classes, Put and PutGroup. It will be helpful to standardize Processor implementations.
- Applied Rollback on Failure to PutSQL, PutHiveQL, PutHiveStreaming and PutDatabaseRecord.
- Stop using AbstractProcessor for these processors, as it penalizes FlowFiles being processed when it rollback a process session. If FlowFiles are penalized, it will not be fetched again until penalization expires.
- Yield processor when a failure occurs and RollbackOnFailure is enabled. If we do not penalize nor yield, a failed FlowFile retries too frequently.
- When Rollback on Failure is enabled but processor is not transactional, discontinue when an error occurred after successful processes.
- Fixed existing issues on PutHiveStreaming:
- Output FlowFile Avro format was corrupted by concatenating multiple Avro files.
- Output FlowFile records had incorrect values because of reusing GenericRecord instance.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1658
NIFI-3704: Incorporated review comments
NIFI-3704: Updated PutDatabaseRecord documentation, including limitations on Update records
NIFI-3704: Added retry paths for transient SQL errors
NIFI-3704: Updated to use renamed RecordReaderFactory and catch SchemaNotFoundException
NIFI-3704: Added better error handling and rollback
This closes#1677.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-3738 Fixed NPE when ParseSyslog UDP datagram has zero length.
Added unit test.
NIFI-3738 Added licenses to new unit tests.
This closes#1694.
Signed-off-by: Bryan Bende <bbende@apache.org>
Signed-off-by: Mark Payne <markap14@hotmail.com>
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-1280: Updated SimpleKeyValueSchemaRegistry to make use of new CHOICE RecordFieldType - Update Record Readers to use SchemaRegistry controller service. Moved SchemaRegistry api into its own maven module and added to standard-services-api so that we can properly add dependencies on it. Code cleanup and bug fixes
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-1280: Fixed checkstyle violations and license exclusions for RAT plugin
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-1280: Addressed feedback from PR Review
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-1280: Additional changes/doc to support QueryFlowFile and Record Readers/Writers
This closes#1652
Use it with FirstInFirstOutPrioritizer, it can enforce original ordering
of 'out-of-order' FlowFiles.
nifi-mock is modified to support FlowFile assertion using Prioritizer.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3414: Added EnforceOrder processor
Incorporated review comments, added displayNames.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3414: Added EnforceOrder processor
Incorporate review comments:
- Moved nifi-standard-prioritizers dependency to top level nifi/pom.xml.
- Changed default initial order from 1 to 0.
- Fixed typos.
- Use session.get(batchCount).
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3414: Added EnforceOrder processor
When a FlowFile is transferred to success, remove attributes previously set when it was transferred to wait or failure.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1496
Signed-off-by: Matt Burgess <mattyb149@apache.org>
Updated test to check selected column names
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1596
- Refactoring NarDetails to include all info from MANIFEST
- Adding the concept of a Bundle and refactoring NarClassLoaders to pass Bundles to ExtensionManager
- Adding logic to fail start-up when multiple NARs with same coordinates exist, moving Bundle classes to framework API
- Refactoring bundle API to classes and creating BundleCoordinate
- Updating FlowController to use BundleCoordinate
- Updating the UI and DTO model to support showing bundle details that loaded an extension type.
- Adding bundle details for processor canvas node, processor dialogs, controller service dialogs, and reporting task dialogs.
- Updating the formating of the bundle coordinates.
- Addressing text overflow in the configuration/details dialog.
- Fixing self referencing functions.
- Updating extension UI mapping to incorporate bundle coordinates.
- Discovering custom UIs through the supplied bundles.
- Adding verification methods for creating extensions through the rest api.
- Only returning extensions that are common amongst all nodes.
- Rendering the ghost processors using a dotted border.
- Adding bundle details to the flow.xml.
- Loading NiFi build and version details from the framework NAR.
- Removing properties for build and version details.
- Wiring together front end and back end changes.
- Including bundle coordinates in the component data model.
- Wiring together component data model and flow.xml.
- Addressing issue when resolve unvesioned dependent NARs.
Updating unit tests to pass based on framework changes
- Fixing logging of extension types during start up
- Allowing the application to start if there is a compatible bundle found. - Reporting missing bundle when the a compatible bundle is not found. - Fixing table height in new component dialogs.
Fixing chechstyle error and increasing test timeout for TestStandardControllerServiceProvider
- Adding ability to change processor type at runtime
- Adding backend code to change type for controller services
- Cleaning up instance classloaders for temp components.
- Creating a dialog for changing the version of a component.
- Updating the formatting of the component type and bundle throughout.
- Updating the new component dialogs to support selecting source group.
- Cleaning up new component dialogs.
- Cleaning up documentation in the cluster node endpoint.
Adding missing include in nifi-web-ui pom compressor plugin
- Refactoring so ConfigurableComponent provides getLogger() and so the nodes provide the ConfigurableComponent
- Creating LoggableComponent to pass around the component, logger, and coordinate with in the framework
- Finishing clean up following rebase.
Calling lifecycle methods for add and remove when changing versions of a component
- Introducing verifyCanUpdateBundle(coordinate) to ConfiguredComponent, and adding unit tests
- Ensuring documentation is available for all components. Including those of the same type that are loaded from different bundles.
Adding lookup from ClassLoader to Bundle, adding fix for instance class loading to include all parent NARs, and adding additional unit tests for FlowController
- Adding validation to ensure referenced controller services implement the required API
- Fixing template instantiation to look up compatible bundle
- Requiring services/reporting tasks to be disabled/stopped.
- Only supporting a change version option when the item has multiple versions available.
- Limiting the possible new controller services to the applicable API version.
- Showing the implemented API versions for Controller Services.
- Updating the property descriptor tooltip to indicate the required service requirements.
- Introducing version based sorting in the new component dialog, change version dialog, and new controller service dialog.
- Addressing remainder of the issues from recent rebase.
Ensuring bundles have been added to the flow before proposing a flow, and incorporating bundle information into flow fingerprinting
- Refactoring the way missing bundles work to retain the desired bundle if available
- Fixing logger.isDebugEnabled to be logger.isTraceEnabled
- Auditing when user changes the bundle. - Ensuring bundle details are present in templates.
Moving standard prioritizers to framework NAR and refactoring ExtensionManager logic to handle cases where an extension is in a JAR directly in the lib directory
- Ensuring all nodes attempt to instantiate the same template instance when the available bundles may differ. - Fixing the auditing of copy/paste and template instantiation. - Running addtional verification methods when running standalone.
Refactoring controller service invocation handler to allow updating the node used by the invocation handler
- Ensuring the bundles in a proposed flow are compatible with the current instance when the current instance has no flow is going to accept the proposed flow
- Merging whether multiple versions of the component are available
- Setting NAR plugin back to current released version
- Cleaning up DocGenerator to not process multiple times
Addressing incorrect usage of nf.Common. - Using formatType in the new component type dialogs.
Improving error messages when looking for bundles
Addressing comments from PR. - Fixing references to global nf namespace. - Fixing injection of nfProcessGroupConfiguration in nfComponentVersion. - Fixing web api integration tests.
Not rendering unversioned in help documentation. - Ensuring the isExtentionMissing flag is correct after changing the component type.
Adding synchronization in node classes to ensure changing component can't occur when component is running, introducing MissingBundleException for better reporting when a node can't join cluster due to a missing bundle, and bumping NAR plugin to released version 1.2.0
Adding concept of missing components to fingerprinting to ensure nodes agree on missing components when joining a cluster
NIFI-3380: NIFI-3520: - Fixing hive nar dependency. - Marking DBCPService as provided. - Skipping services that require instance classloading and are cobundled with their service API. - Skipping components that require instance classloading and reference service APIs that are cobundled. - Addressing UI issues in the new component dialogs when re-opening with a filter applied.
Fixing checkstyles issue and adding back assume checks to distributed cache server test
Ensuring new component types are sorted correctly when shown initially.
This closes#1585.
- Refactored TestGetHTTPGroovy for componentization & customization of tests.
- Ignored some TestGetHTTPGroovy tests because they cannot connect to a Jetty server running TLSv1 or TLSv1.1 (the new version of Jetty only supports TLSv1.2+). The tests are still valuable to demonstrate GetHTTP's capacity to connect to various TLS protocols, and should not be removed unless necessary.
- Refactored TestPostHTTPGroovy for componentization & customization of tests.
- Ignored some TestPostHTTPGroovy tests because they cannot connect to a Jetty server running TLSv1 or TLSv1.1 (the new version of Jetty only supports TLSv1.2+). The tests are still valuable to demonstrate PostHTTP's capacity to connect to various TLS protocols, and should not be removed unless necessary.
- Restored one of the TestGetHTTPGroovy unit tests (GetHTTP vs. TLSv1/1.1/1.2) by connecting to https://nifi.apache.org which still supports all three protocol versions (and uses JVM cacerts as truststore).
- This closes#1601
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3481 Added support for Database Adapter Descriptions
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3481 Updated Adapter Name
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1510
* Add test for failure not clear context map.
* Add handler to remove context map entry if ProcessException occurs during while exporting response.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#1567.
- Support multiplle incoming FlowFiles to Wait processor, up to Wait
Buffer Count
- Added Releasable FlowFile Count, which controls how many FlowFiles can
be released when wait condition is met
- Added special meaning to Notify delta Zero(0) to clear a signal
counter back to zero
This closes#1554
Signed-off-by: Aldrin Piri <aldrin@apache.org>
Before this fix, files with the latest timestamp within a listing
iteration are always be held back one cycle no matter how old it is.
Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
- Added Signal Counter Delta property
- Added Signal Buffer Count property
- Added processor property name and display name
- Changed IOException handling from routing it to failure to throw
RuntimeException, so that NiFi framework can yield the processor for a while and try again
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#1466.
use the FileNameFilter when not passing down explit jar paths
Filter out ^. files when reading lists of files from directories
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
Mark Payne
Pierre Villard
Matthew Burgess
Andy LoPresto
joewitt
Colin Dean
Nick Lewis
Otto Fowler
Lars Francke
Mike Moser
patricker
Koji Kawamura
Marco Gaido
Mike Thomsen
Bartlomiej Tartanus
Dustin Rodrigues
Joe Trite
Nicolas Sanglard
zenfenan
Andrew Psaltis
Gardella Juan Pablo
Joe Percivall
Mark Bean
Matt Gilman
sbouchex
Bryan Bende
Deon Huang
m-hogue
Jeff Storck
Andy LoPresto
Marco Gaido
Norito Agetsuma
trixpan
Josh Anderton
Julian Gimbel
Andrew Lim
Gino Lisignoli
Arun Manivannan
Wayne Steel
Wil Selwood
Leah Anderson
deonhuang
Toivo Adams
mans2singh
Tony Kurc
yuri1969
Joey Frazee
Peter G. Horvath
Jon Anderson III
Yolanda M. Davis
Oleg Zhurakousky
ambud
Peddy
Joe Skora
Pere Urbon-Bayes
Giovanni Lanzani
josephxsxn
Nick Carenza
Selim Namsi
Otto Fowler
Ryan Persaud
Giovanni Lanzani
Joe Gresock