NIFI-5943 Added another unit test to verify list + map conversion to list of records. (Mike Thomsen)
This closes#3267
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
NIFI-5947 Made the lookup key for ElasticSearchStringLookupService more unique to the service (was previously 'id'); change by Mike Thomsen
This closes#3268.
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
NIFI-4975 Added changes requested in a code review.
NIFI-4975 Reverted some base Mongo changes.
NIFI-4975 Moved connection configuration to using Mongo client service.
NIFI-4975 Fixed a lot of style issues.
NIFI-4975 Removed an EL statement that was causing problems with the UI.
NIFI-4975 Added changes from code review.
NIFI-4975 Added additional details for FetchGridFS.
NIFI-4975 Added documentation for DeleteGridFS.
NIFI-4975 Added documentation for PutGridFS.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2546
resets worker if it doesn't work anymore for any reason. this will add "reconnect" capabilities. Will solve issues for following use cases:
- authentication changed after successful connection
- JNDI mapping changed and requires recaching.
- JMS server isn't available anymore or restarted.
improved controller reset on exception
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3261
NIFI-5903: Removed TODO comments that were done
NIFI-5903: Added support for working with MAP types to QueryRecord and associated RPATH functions
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3223
Use temporary values for port names when updated process groups as part of a Change Flow Version operation.
This avoids the potential for a name conflict between a ports during the update process.
Add a final step to the update process group logic to set the final name on all ports.
This closes#3301.
- Updates to make UpdateRecord and RecordPath automatically update Record schema when performing update and perform the updates on the first record in UpdateRecord before obtaining Writer Schema. This allows the Writer to to inherit the Schema of the updated Record instead of the Schema of the Record as it was when it was read.
- Updated JoltTransformRecord so that schema is inferred on the first transformed object before passing the schema to the Record Writer, so that if writer inherits schema from record, the schema that is inherited is the trans transformed schema
- Updated LookupRecord to allow for Record fields to be arbitrarily added
- Implemented ContentClaimInputStream
- Added controller service for caching schemas
- UpdatedQueryRecord to cache schemas automatically up to some number of schemas, which will significantly inprove throughput in many cases, especially with inferred schemas.
NIFI-5938: Updated AvroTypeUtil so that if creating an Avro Schema using a field name that is not valid for Avro, it creates a Schema that uses a different, valid field name and adds an alias for the given field name so that the fields still are looked up appropriately. Fixed a bug in finding the appropriate Avro field when aliases are used. Updated ContentClaimInputStream so that if mark() is called followed by multiple calls to reset(), that each reset() call is successful instead of failing after the first one (the JavaDoc for InputStream appears to indicate that the InputStream is free to do either and in fact the InputStream is even free to allow reset() to reset to the beginning of file if mark() is not even called, if it chooses to do so instead of requiring a call to mark()).
NIFI-5938: Added another unit test for AvroTypeUtil
NIFI-5938: If using inferred schema in CSV Reader, do not consider first record as a header line. Also addressed a bug in StandardConfigurationContext that was exposed by CSVReader, in which calling getProperty(PropertyDescriptor) did not properly lookup the canonical representation of the Property Descriptor from the component before attempting to get a default value
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3253
Refactored KerberosAction to return a result from execute()
Removed usage of ProcessContext.yield() from KerberosAction, which should instead be handled the component using the KerberosCredentialsService.
Updated SolrProcessor to yield a flowfile on error, rather than the KerberosAction invoking the yield.
NIFI-5985: Updated TestPutSolrContentStream.testUpdateWithKerberosAuth test case to match on PrivilegedExceptionAction instead of PrivilegedAction doAs arguments.
NIFI-5985: Moved kerberosUser logout after closing the datasource in the shutdown method.
NIFI-5985: Removed catching exceptions in DBCPConnectionPool.shutdown
Exception when closing the datasource is prioritized over an exception when logging out the kerberos principal
Added GroovyDBCPServiceTest tests to verify prioritizing datasource.close() exception over kerberosUser.logout() exception
This closes#3288.
Signed-off-by: Bryan Bende <bbende@apache.org>
Removed bad test.
Refactored filter creation method.
Ensure HSTS header is only applied if NiFi is secured with HTTPS
Small changes to header array list.
Fixed checkstyle errors.
This closes#3273.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
MQTT over websockets uses ws (and wss for secure websocket connections) schemes in the broker URI.
Added support for the same.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#3289.
Unit test is implemented: testNoSplitterInString
NIFI-5974: Fixed Checkstyle violations
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3275
NIFI-5909 added content checks to the unit tests
NIFI-5937 use explicit long value for test dates/times (to not depend on the timezone of test executor)
NIFI-5937 tabs to spaces
Fixing checkstyle violations introduced by https://github.com/apache/nifi/pull/3249 PR)
NIFI-5937 adjusted property descriptions for consistency; limited EL scope to variable registry; added an appropriate validator along with its Maven dependency; moved format initialization to @OnScheduled
NIFI-5909 tabs to spaces
Signed-off-by: Ed <edward.berezitsky@gmail.com>
This closes#3227
NIFI-5937 added tests to verify that accented characters are preserved correctly
NIFI-5937 unfolding starred imports
NIFI-5937 unfolding starred imports (now with statics)
Signed-off-by: Ed <edward.berezitsky@gmail.com>
This closes#3250
- Ensuring exceptions are handled in the ldap user/group sync background thread.
- Adding additional logging around what users/groups were discovered.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#3247.
Unit tests and functionality for tagging an object in S3.
Set FlowFile attributes directly from tags retrieved from S3.
Add guard clauses to ensure evaluated properties are not blank.
This closes#3239.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
Clear the wait timestamp when transferring to failur or success
replace explicit attribute clear with function call, refactor and integrate into existing tests per review
This closes#3233.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-4892 - ValidateCSV: no doublequote escaping in invalid output
NIFI-5907 - unit test
This closes#2481.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Fixed proxy header support to use X-Forwarded-Host instead of X-ForwardedServer
- Added support for the context path header used by Traefik when proxying a service (X-Forwarded-Prefix)
- Added tests to ApplicationResourceTest for X-Forwarded-Context and X-Forwarded-Prefix
- Updated administration doc to include X-Forwarded-Prefix
- Added NIFI_WEB_PROXY_CONTEXT_PATH env var to dockerhub and dockermaven start.sh scripts
- Added documentation for NIFI_WEB_PROXY_CONTEXT_PATH to dockerhub README.md
- Updated ApplicationResource to handle a port specified in X-ProxyPort and X-Forwarded-Port headers
This closes#3129.
Signed-off-by: Kevin Doran <kdoran@apache.org>
- Found several instances of nifi-framework-api's ProviderException being thrown from processors. Changed those to IllegalStateException, as ProviderException is not an appropriate Exception in those cases, and extensions should not depend on nifi-framework-api.
- Performed some cleanup, moving Property Descriptors from Controller Service API's/specs into the implementations. Adding to the Service API results in bringing in nifi-utils to the nifi-standard-services-api-nar, which is a bad practice. The 'main' service api nar should not have a dependency on a util class.
NIFI-5859: Added javadocs. Fixed pom.xml that was left pointing to snapshot version of nar maven plugin
NIFI-5859: Addressing review feedback: adding component type, multiple additional details into separate file(s)
This closes#3192.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-5406 introduced the issue by trying to use the resetState variable for
different purposes. AbstractListProcessor should have had a different variable
to control whether to clear state for tracking entity strategy.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#3189.
NIFI-5833 Added unit test to demonstrate arbitrary decryption of sensitive values regardless of processor property sensitive status.
NIFI-5833 Updated GetTwitter documentation with note about 1.9.0+ marking Consumer Key and Access Token as sensitive.
Signed-off-by: joewitt <joewitt@apache.org>
NIFI-5788: Introduce batch size limit in PutDatabaseRecord processor
Renamed 'batch size' to 'Maximum Batch Size'.
Changed default value of max_batch_size to zero (INFINITE)
Fixed parameter validation.
Added unit tests
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3128
making the database session autocommit value a configurable property
adding custom validation to PutSQL processor so as to disallow 'supports transaction' and 'rollback on failure' to be true when the autocommit value has been set to true
fixing some style issues to conform to standards
This closes#3113.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-5752: Remove an unnecessary String.format
NIFI-5752: Remove an unnecessary block
This closes#3110.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
ListS3 used to update currentKeys within listing loop, that causes
duplicates. Because S3 returns object list in lexicographic order, if we
clear currentKeys during the loop, we cannot tell if the object has been
listed or not, in a case where newer object has a lexicographically
former name.
Signed-off-by: James Wing <jvwing@gmail.com>
This closes#3116, closes#2361.
- Set up NarAutoLoader to watch directory for new files
- Move NarAutoLoader to JettyServer since it will need access to ExtensionManager
- Created NarLoader to shared between NarAutoLoader and the framework
- Created nifi-framework-nar-loading-utils so we can use nifi-documentation to call DocGenerator
- Add additional bundles to overall map in NarClassLoaders as they are loaded
- Added handling of skipped NARs to include them in next iteration
- Added check of last modified timestamp on NARs
- Refactored JettyServer so we can load additional web contexts while the application is running
- Setting up unit tests
- Remove static use of ExtensionManager
- Adding unit tests for NarLoader
- Extracting interface for ExtensionManager and splitting discovery into it's own interface
This closes#3119.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Moved module appending (aka classpath in python) into init stage instead of running each time onTrigger.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3117
NIFI-5761 Code review.
Remove startsWidth to check an exception. Added the dependency as provided.
NIFI-5761 Code review.
Remove provided.
NIFI-5761 Code review.
Improve logging.
This closes#3112.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Removing needClientAuth property since cluster comms now requires two way ssl. Jetty client auth settings are based on configured features.
- Removing dead code.
- Updating documentation.
- Removing references to needClientAuth property in all test resources.
- Removing overloaded util method with strict parameter.
This closes#3102.
NIFI-5686: Fixed unit test in TestSocketLoadBalancedFlowFileQueue; renamed TestProcessorLifecycle to ProcessorLifecycleIT as it is testing integration between many components and largely focuses on high numbers of concurrent tasks to see if it can trigger any threading bugs that may get introduced
NIFI-5686: Extended unit test timeouts
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3062
- introducing a in-memory-file-size-threashold, above which the incoming file is written to local disk
- using java.io.tmpdir for such file writes
- enhancing documentation
- documenting how to avoid premature HTTP response
- fix and UT for unsuccessful request registration
This closes#2991.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- introducing a in-memory-file-size-threashold, above which the incoming file is written to local disk
- using java.io.tmpdir for such file writes
- enhancing documentation
This commit also contains following refactoring:
- Catch URISyntaxException inside subscribe when constructing provenance
URI as it does not affect the core responsibility of this processor.
Even if it fails to be a proper URI, if the query works for consuming
logs, the processor should proceed forward.
Upgrade JNA version.
Do not update lastActivityTimestamp when subscribe failed.
This closes#3037
Added Spock test for NonLocalPartitionPartitioner
Updated NOTICE files for FontAwesome with the updated version (4.7.0) and URL to the free license
Updated package-lock.json with the updated version of FontAwesome (4.7.0)
Added method to FlowFileQueue interface to reset an offloaded queue
Queues that are now immediately have the offloaded status reset once offloading finishes
SocketLoadBalancedFlowFileQueue now ignores back-pressure when offloading flowfiles
Cleaned up javascript in nf-cluster-table.js when creating markup for the node operation icons
Fixed incorrect handling of a heartbeat from an offloaded node. Heartbeats from offloading or offloaded nodes will now be reported as an event, the heartbeat will be removed and ignored.
Added unit tests and integration tests to cover offloading nodes
Updated Cluster integration test class with accessor for the current cluster coordinator
Updated Node integration test class's custom NiFiProperties implementation to return the load balancing port and a method to assert an offloaded node
Added exclusion to top-level pom for ITSpec.class
Updated NodeClusterCoordinator to allow idempotent requests to offload a cluster
Added capability to connect/delete/disconnect/offload a node from the cluster to the Toolkit CLI
Added capability to get the status of nodes from the cluster to the Toolkit CLI
Upgraded FontAwesome to 4.7.0 (from 4.6.1)
Added icon "fa-upload" for offloading nodes in the cluster table UI
- Updated FlowFile Repo / Write Ahead Log so that any update that writes more than 1 MB of data is written to a file inside the FlowFile Repo rather than being buffered in memory
- Update SplitText so that it does not hold FlowFiles that are not the latest version in heap. Doing them from being garbage collected, so while the Process Session is holding the latest version of the FlowFile, SplitText is holding an older version, and this results in two copies of the same FlowFile object
NIFI-5533: Checkpoint
NIFI-5533: Bug Fixes
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2974
Explicitly allow methods GET, HEAD. These are the Spring defaults when the allowedMethods is empty but now it is explicit. This will require other methods like POST etc to be from the same origin (for the template/upload URL).
This closes#3024.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Refactoring StandardFlowFileQueue to have an AbstractFlowFileQueue
Refactored more into AbstractFlowFileQueue
Added documentation, cleaned up code some
Refactored FlowFileQueue so that there is SwappablePriorityQueue
Several unit tests written
Added REST API Endpoint to allow PUT to update connection to use load balancing or not. When enabling load balancing, though, I saw the queue size go from 9 to 18. Then was only able to process 9 FlowFiles.
Bug fixes
Code refactoring
Added integration tests, bug fixes
Refactored clients to use NIO
Bug fixes. Appears to finally be working with NIO Client!!!!!
NIFI-5516: Refactored some code from NioAsyncLoadBalanceClient to LoadBalanceSession
Bug fixes and allowed load balancing socket connections to be reused
Implemented ability to compress Nothing, Attributes, or Content + Attributes when performing load-balancing
Added flag to ConnectionDTO to indicate Load Balance Status
Updated Diagnostics DTO for connections
Store state about cluster topology in NodeClusterCoordinator so that the state is known upon restart
Code cleanup
Fixed checkstyle and unit tests
NIFI-5516: Updating logic for Cluster Node Firewall so that the node's identity comes from its certificate, not from whatever it says it is.
NIFI-5516: FIxed missing License headers
NIFI-5516: Some minor code cleanup
NIFI-5516: Adddressed review feedback; Bug fixes; some code cleanup. Changed dependency on nifi-registry from SNAPSHOT to official 0.3.0 release
NIFI-5516: Take backpressure configuration into account
NIFI-5516: Fixed ConnectionDiagnosticsSnapshot to include node identifier
NIFI-5516: Addressed review feedback
This closes#2947
This closes#3044
- Revert 87cf474e54 to enable connection
pooling
- Changes the expected HTTP status code for the 1st request of a
two-phase commit transaction from 150 (NiFi custom) to 202 Accepted
- Corrected RevisionManager Javadoc about revision varidation protocol
Refactored some test code to be clearer.
Renamed some resources to be consistent across modules.
Changed passwords to meet new minimum length requirements.
This closes#3018
NIFI-5051 Fixed checkstyle issue.
NIFI-5051 Converted ES lookup service to use a SchemaRegistry.
NIFI-5051 Cleaned up POM and added a simple unit test that uses a mock client service.
NIFI-5051 Added change; waiting for feedback.
NIFI-5051 Changed query setup based on code review. Changed tests to Groovy to make them easier to read with all of the inline JSON.
NIFI-5051 fixed a checkstyle issue.
NIFI-5051 Rebased to cleanup merge issues
NIFI-5051 Added changes from a code review.
NIFI-5051 Fixed a checkstyle issue.
NIFI-5051 Added coverage generator for tests.
Rebased.
NIFI-5051 Updated service and switched it over to JsonInferenceSchemaRegistryService.
NIFI-5051 Removed dead code.
NIFI-5051 Fixed checkstyle errors.
NIFI-5051 Refactored query builder.
NIFI-5051 Added placeholder gitignore to force test compile.
NIFI-5051 Added note explaining why the .gitignore file was needed.
NIFI-5051 Made constructor public.
NIFI-5051 Fixed path issue in client service integration tests.
NIFI-5051 Added additional mapping capabilities to let users massage the result set into the fields they want.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2615
Refactors tests in order to share code repeated in tests and to enable
some parameterized testing.
MySQL Connector/J 5.1.x in conjunction with MySQL 5.0.x will return
a Long for ResultSet#getObject when the SQL type is an unsigned integer.
This change prevents that error from occurring while implementing a more
informational exception describing what the failing object's POJO type
is in addition to its string value.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3032
NIFI-5514: Do not rely on ProcessSession.getQueueSize() to return a queue size of 0 objects because if the processor is holding onto data, the queue size won't be 0.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#2954.
The operation policy allows that a user to operate components even if they does not have direct READ/WRITE
permission of the component.
Following operations are controlled by the new operate policy:
- Start/stop/enable/disable Processors, ControllerServices,
ReportingTasks, Input/OuputPorts
- Enable/disable transmission of RemoteInput/OutputPorts and
RemoteProcessGroups
- Terminate Processor threads
Refactored what API exposes
The previous commit let API exposes few fields in DTO. But we should
avoid returning partial DTO as it complicates authorization logic.
Instead, this commit adds StatusDTO for ReportingTaskEntity and
ControllerServiceEntity, so that it can be returned regardless of having
READ permission. Component DTO can only be returned with a READ
permission.
Refactor RPG same as ControllerService.
WIP incorporating review comments.
Incorporated review comments
- Cleaned up merger classes
- Recreate DTO instance at each function during two phase commmit
Restrict enabling ControllerService without read permission
Revert the last commit.
Fix review comments.
- Renamed confusing static method names and its parameters
- Removed unnecessary permission checks from UI condition
Fixed delete action display condition.
Fixed NPE at Summary.
Apply operation policy to activateControllerServices.
Removed OperationPermissible from ComponentEntity.
This closes#2990
Deprecated HashContent processor.
Added documentation to HashAttribute processor.
Added shared HashService and HashAlgorithm enum.
Added unit tests.
Added #clearProperties() to TestRunner, StandardProcessorTestRunner, and MockProcessContext.
Updated processor manifest.
Updated Javadoc.
Added documentation about deprecated/renamed components.
Added logic handling for UTF-16 encoding where UTF-16BE is overridden so the BOM is not inserted prior to hashing.
This closes#2983.
Co-authored-by: Otto Fowler <ottobackwards@gmail.com>
Signed-off-by: Kevin Doran <kdoran@apache.org>
Mark Payne
Mike Moser
Aleksandr Salatich
Endre Zoltan Kovacs
Patrice Freydiere
Alex Savitsky
Mike Thomsen
Jeff Storck
Key Miyauchi
Kevin Doran
Ed
Dorian Bugeja
Matthew Burgess
Bryan Bende
Alex Goos
thenatog
Peter Wicks
Sujesh Menon
a.durov
Matt Gilman
erichanson5
Koji Kawamura
SavtechSolutions
Stephen Goodman
Pierre Villard
Ryan Whittington
Otto Fowler
zenfenan
gkkorir
Kemix Koo
Alexander Bukarev
luup2k
Kotaro Terada
joewitt
Kumara M S Hemantha
Aaron Leon
Daniel Jimenez
Andy LoPresto
yjhyjhyjh0
Colin Dean
Colin Dean
vadimar
Ed B
Arek Burdach
Vish Uma
Andrew Lim
Adam Lamar
Gardella Juan Pablo
Joe Percivall
juliansniff
rednikotin
Mohit Garg
Christian Englert
Endre Zoltan Kovacs
Brad Hards
pepov
Brandon Devries
Nick Lewis
Andrew I. Christianson
“PrashanthVenkatesan”
uday