This change is based on https://github.com/apache/nifi/pull/2489
I have reproduced the issue with some additional test cases and test files then applied the original fix.
commit message from the original change:
Updated the EVTX FileHeader class to correctly check if there are more chunks in the file. Previously this would not process the last chunk.
Updated the EVTX ChunkHeader class to correctly check if there are additional records in the chunk. Previously this would only process the first record of each chunk. It was using the fileLastRecordNumber where it should have been using the logLastRecordNumber value.
Updated the EVTX unit tests to have the correct expected number of events and use the logLastRecordNumber.
refactoring duplicated code and magic numbers
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2489
This closes#3379
- Allowing user or group existence enforcement to be parameterized.
- Fixing error handling when loading user groups which may have resulted in stack trace leaking.
This closes#3377.
Signed-off-by: Kevin Doran <kdoran@apache.org>
Added an unit test representing the fixed issue.
And updated existing testDefragment test to illustrate
the remaining FlowFiles those did not meet the threshold.
- Make enriched fields nullable at LookupRecord.
- Removed unnecessary AvroConversionOptions and reader schema creation,
because ResultSetRecordSet can generate NiFi Record Schema from RS
directly. No Avro schema is needed to do that.
* Documentation : ConsumerKafka_2_0 - updated configuration details related to SASL_PLAINTEXT - SCRAM
* Updated rest of the related documents with SCRAM configurations and few NOTEs
* incorporated review comments
* incorporated review comments
- Added snap alignment for nf-label, label resize events, and nf-connection
- Shift key now disables snap alignment during the drag event.
- nf-connection load-balance-icon updated
This closes#3335
Processor for sending messages on Slack and optionally upload and attach
the FlowFile content (e.g. an image) to the message.
NIFI-6078: Remove username/icon properties.
NIFI-6078: Make Text property optional.
NIFI-6078: Documentation changes.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3339
Moved test classes into the proper folder.
Fixed outdated mocking and added assertions that actually tests the content.
This closes#3346.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-6056 Fixing cassandra bundle so nifi-mock is a test scoped and removing incorrect usage of ProviderCreation exception from nifi-framework-api
This closes#3322.
Signed-off-by: Mark Payne <markap14@hotmail.com>
NIFI-5660: Added lines to integration test to verify these header properties (added by Mike Moser)
Signed-off-by: Mike Moser <mosermw@apache.org>
This closes#3053
Suppressed deprecation warnings in TestGetHTTP, TestPostHTTP, TestGetHTTPGroovy, and TestPostHTTPGroovy. The annotation suppresses some warnings, but until Java 9, explicit FQCN imports are needed to avoid all warnings.
This closes#3312.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Updated exception message thrown when accessing the ListHDFS state to refer to the State Manager rather than the "Distributed Cache Service"
Removed usage and references to MockCacheClient from TestListHDFS
NIFI-6023: Minor update of the "Distributed Cache Service" description
This closes#3313.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-5943 Added another unit test to verify list + map conversion to list of records. (Mike Thomsen)
This closes#3267
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
NIFI-5947 Made the lookup key for ElasticSearchStringLookupService more unique to the service (was previously 'id'); change by Mike Thomsen
This closes#3268.
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
NIFI-4975 Added changes requested in a code review.
NIFI-4975 Reverted some base Mongo changes.
NIFI-4975 Moved connection configuration to using Mongo client service.
NIFI-4975 Fixed a lot of style issues.
NIFI-4975 Removed an EL statement that was causing problems with the UI.
NIFI-4975 Added changes from code review.
NIFI-4975 Added additional details for FetchGridFS.
NIFI-4975 Added documentation for DeleteGridFS.
NIFI-4975 Added documentation for PutGridFS.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2546
resets worker if it doesn't work anymore for any reason. this will add "reconnect" capabilities. Will solve issues for following use cases:
- authentication changed after successful connection
- JNDI mapping changed and requires recaching.
- JMS server isn't available anymore or restarted.
improved controller reset on exception
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3261
NIFI-5903: Removed TODO comments that were done
NIFI-5903: Added support for working with MAP types to QueryRecord and associated RPATH functions
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3223
Use temporary values for port names when updated process groups as part of a Change Flow Version operation.
This avoids the potential for a name conflict between a ports during the update process.
Add a final step to the update process group logic to set the final name on all ports.
This closes#3301.
- Updates to make UpdateRecord and RecordPath automatically update Record schema when performing update and perform the updates on the first record in UpdateRecord before obtaining Writer Schema. This allows the Writer to to inherit the Schema of the updated Record instead of the Schema of the Record as it was when it was read.
- Updated JoltTransformRecord so that schema is inferred on the first transformed object before passing the schema to the Record Writer, so that if writer inherits schema from record, the schema that is inherited is the trans transformed schema
- Updated LookupRecord to allow for Record fields to be arbitrarily added
- Implemented ContentClaimInputStream
- Added controller service for caching schemas
- UpdatedQueryRecord to cache schemas automatically up to some number of schemas, which will significantly inprove throughput in many cases, especially with inferred schemas.
NIFI-5938: Updated AvroTypeUtil so that if creating an Avro Schema using a field name that is not valid for Avro, it creates a Schema that uses a different, valid field name and adds an alias for the given field name so that the fields still are looked up appropriately. Fixed a bug in finding the appropriate Avro field when aliases are used. Updated ContentClaimInputStream so that if mark() is called followed by multiple calls to reset(), that each reset() call is successful instead of failing after the first one (the JavaDoc for InputStream appears to indicate that the InputStream is free to do either and in fact the InputStream is even free to allow reset() to reset to the beginning of file if mark() is not even called, if it chooses to do so instead of requiring a call to mark()).
NIFI-5938: Added another unit test for AvroTypeUtil
NIFI-5938: If using inferred schema in CSV Reader, do not consider first record as a header line. Also addressed a bug in StandardConfigurationContext that was exposed by CSVReader, in which calling getProperty(PropertyDescriptor) did not properly lookup the canonical representation of the Property Descriptor from the component before attempting to get a default value
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3253
Refactored KerberosAction to return a result from execute()
Removed usage of ProcessContext.yield() from KerberosAction, which should instead be handled the component using the KerberosCredentialsService.
Updated SolrProcessor to yield a flowfile on error, rather than the KerberosAction invoking the yield.
NIFI-5985: Updated TestPutSolrContentStream.testUpdateWithKerberosAuth test case to match on PrivilegedExceptionAction instead of PrivilegedAction doAs arguments.
NIFI-5985: Moved kerberosUser logout after closing the datasource in the shutdown method.
NIFI-5985: Removed catching exceptions in DBCPConnectionPool.shutdown
Exception when closing the datasource is prioritized over an exception when logging out the kerberos principal
Added GroovyDBCPServiceTest tests to verify prioritizing datasource.close() exception over kerberosUser.logout() exception
This closes#3288.
Signed-off-by: Bryan Bende <bbende@apache.org>
Removed bad test.
Refactored filter creation method.
Ensure HSTS header is only applied if NiFi is secured with HTTPS
Small changes to header array list.
Fixed checkstyle errors.
This closes#3273.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
MQTT over websockets uses ws (and wss for secure websocket connections) schemes in the broker URI.
Added support for the same.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#3289.
Unit test is implemented: testNoSplitterInString
NIFI-5974: Fixed Checkstyle violations
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3275
NIFI-5909 added content checks to the unit tests
NIFI-5937 use explicit long value for test dates/times (to not depend on the timezone of test executor)
NIFI-5937 tabs to spaces
Fixing checkstyle violations introduced by https://github.com/apache/nifi/pull/3249 PR)
NIFI-5937 adjusted property descriptions for consistency; limited EL scope to variable registry; added an appropriate validator along with its Maven dependency; moved format initialization to @OnScheduled
NIFI-5909 tabs to spaces
Signed-off-by: Ed <edward.berezitsky@gmail.com>
This closes#3227
NIFI-5937 added tests to verify that accented characters are preserved correctly
NIFI-5937 unfolding starred imports
NIFI-5937 unfolding starred imports (now with statics)
Signed-off-by: Ed <edward.berezitsky@gmail.com>
This closes#3250
- Ensuring exceptions are handled in the ldap user/group sync background thread.
- Adding additional logging around what users/groups were discovered.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#3247.
Unit tests and functionality for tagging an object in S3.
Set FlowFile attributes directly from tags retrieved from S3.
Add guard clauses to ensure evaluated properties are not blank.
This closes#3239.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
Clear the wait timestamp when transferring to failur or success
replace explicit attribute clear with function call, refactor and integrate into existing tests per review
This closes#3233.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-4892 - ValidateCSV: no doublequote escaping in invalid output
NIFI-5907 - unit test
This closes#2481.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Fixed proxy header support to use X-Forwarded-Host instead of X-ForwardedServer
- Added support for the context path header used by Traefik when proxying a service (X-Forwarded-Prefix)
- Added tests to ApplicationResourceTest for X-Forwarded-Context and X-Forwarded-Prefix
- Updated administration doc to include X-Forwarded-Prefix
- Added NIFI_WEB_PROXY_CONTEXT_PATH env var to dockerhub and dockermaven start.sh scripts
- Added documentation for NIFI_WEB_PROXY_CONTEXT_PATH to dockerhub README.md
- Updated ApplicationResource to handle a port specified in X-ProxyPort and X-Forwarded-Port headers
This closes#3129.
Signed-off-by: Kevin Doran <kdoran@apache.org>
- Found several instances of nifi-framework-api's ProviderException being thrown from processors. Changed those to IllegalStateException, as ProviderException is not an appropriate Exception in those cases, and extensions should not depend on nifi-framework-api.
- Performed some cleanup, moving Property Descriptors from Controller Service API's/specs into the implementations. Adding to the Service API results in bringing in nifi-utils to the nifi-standard-services-api-nar, which is a bad practice. The 'main' service api nar should not have a dependency on a util class.
NIFI-5859: Added javadocs. Fixed pom.xml that was left pointing to snapshot version of nar maven plugin
NIFI-5859: Addressing review feedback: adding component type, multiple additional details into separate file(s)
This closes#3192.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-5406 introduced the issue by trying to use the resetState variable for
different purposes. AbstractListProcessor should have had a different variable
to control whether to clear state for tracking entity strategy.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#3189.
NIFI-5833 Added unit test to demonstrate arbitrary decryption of sensitive values regardless of processor property sensitive status.
NIFI-5833 Updated GetTwitter documentation with note about 1.9.0+ marking Consumer Key and Access Token as sensitive.
Signed-off-by: joewitt <joewitt@apache.org>
NIFI-5788: Introduce batch size limit in PutDatabaseRecord processor
Renamed 'batch size' to 'Maximum Batch Size'.
Changed default value of max_batch_size to zero (INFINITE)
Fixed parameter validation.
Added unit tests
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3128
making the database session autocommit value a configurable property
adding custom validation to PutSQL processor so as to disallow 'supports transaction' and 'rollback on failure' to be true when the autocommit value has been set to true
fixing some style issues to conform to standards
This closes#3113.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-5752: Remove an unnecessary String.format
NIFI-5752: Remove an unnecessary block
This closes#3110.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
ListS3 used to update currentKeys within listing loop, that causes
duplicates. Because S3 returns object list in lexicographic order, if we
clear currentKeys during the loop, we cannot tell if the object has been
listed or not, in a case where newer object has a lexicographically
former name.
Signed-off-by: James Wing <jvwing@gmail.com>
This closes#3116, closes#2361.
- Set up NarAutoLoader to watch directory for new files
- Move NarAutoLoader to JettyServer since it will need access to ExtensionManager
- Created NarLoader to shared between NarAutoLoader and the framework
- Created nifi-framework-nar-loading-utils so we can use nifi-documentation to call DocGenerator
- Add additional bundles to overall map in NarClassLoaders as they are loaded
- Added handling of skipped NARs to include them in next iteration
- Added check of last modified timestamp on NARs
- Refactored JettyServer so we can load additional web contexts while the application is running
- Setting up unit tests
- Remove static use of ExtensionManager
- Adding unit tests for NarLoader
- Extracting interface for ExtensionManager and splitting discovery into it's own interface
This closes#3119.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Moved module appending (aka classpath in python) into init stage instead of running each time onTrigger.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3117
NIFI-5761 Code review.
Remove startsWidth to check an exception. Added the dependency as provided.
NIFI-5761 Code review.
Remove provided.
NIFI-5761 Code review.
Improve logging.
This closes#3112.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
Pierre Villard
Joe Witt
Matthew Burgess
Ferenc Szabó
Matt Gilman
Koji Kawamura
Andres Garagiola
Bryan Bende
Peter Turcsanyi
Rob Fellows
Sandish Kumar
pushpavanthar
Ryan Bower
Mark Payne
Kevin Doran
thenatog
Arpad Boda
Otto Fowler
Joe Percivall
Mike Thomsen
Mark Bean
Jan Hentschel
Denes Arvay
Wietze
Corey Fritz
Aleksandr Salatich
Jeff Storck
Mike Moser
Endre Zoltan Kovacs
Patrice Freydiere
Alex Savitsky
Key Miyauchi
Ed
Dorian Bugeja
Alex Goos
Peter Wicks
Sujesh Menon
a.durov
erichanson5
SavtechSolutions
Stephen Goodman
Ryan Whittington
zenfenan
gkkorir
Kemix Koo
Alexander Bukarev
luup2k
Kotaro Terada
Kumara M S Hemantha
Aaron Leon
Daniel Jimenez
Andy LoPresto
yjhyjhyjh0
Colin Dean
Colin Dean
vadimar
Ed B
Arek Burdach
Vish Uma
Andrew Lim
Adam Lamar
Gardella Juan Pablo
juliansniff