NIFI-6280 - Updated terminology in JwtAuthenticationFilter to authentication instead of authorization. Added stricter token parsing using an explicit regex pattern. Added tests.
NIFI-6280 - Updated terminology from Authorization to Authentication.
NIFI-6280 - Updated the access logout method to use getNiFiUserIdentity(). Updated javascript logout method to handle errors.
NIFI-6280 - Fixing checkstyle issues.
NIFI-6280 - Added some javadoc comments and logging. Renamed some variables for clarity. Fixed handling of exception when JWT does not match expected format.
NIFI-6280 - Cleaned up checkstyle, increased log severity level for logout action, and cleaned up Groovy syntax in test.
This closes#3482.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Specify remote access at port creation.
Incorporated comments, and finished refactoring.
Renamed RootGroupPort to PublicPort.
Fix error message for creating a connection from a child PG having only PublicPorts.
Enhanced ProcessGroup instances rendered in the parent ProcessGroup
Loosen Port move check, allow moving public port between PG.
Show 'Remote NiFi Instance' info on Connection dialogs
Make labels narrative.
'Within Remote Group'.
Fixed DTO (de)serialization.
Return null only if all values are null.
This closes#3351.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Separated into nifi-fn-core, nifi-fn-bootstrap, nifi-fn-nar, nifi-fn-assembly; fully unpacks nars and runs flows
- Rebased against master and updated to version 1.10.0-SNAPSHOT
- Removed dependency on nifi-framework-core
- Added LICENSE/NOTICE files
Updated NOTICE with current copyright year for Jetty dependencies
Updated Jetty SSLContextFactory usage, invoke setEndpointIdentificationAlgorithm(null) on server SslContextFactory instances
Updated TestInvokeHttpSSL to provide a separate client keystore, rather than reusing the server's keystore
Regenerated nifi-standard-processors keystore and truststore, added client keystore
Updated ITestHandleHttpRequest, TestInvokeHttpSSL, TestInvokeHttpTwoWaySSL, and TestListenHTTP to use a separate client keystore instead of reusing the server's keystore. Also updated the tests to separately test one-way and two-way SSL
NIFI-6196 - Setting the endpoint identification algorithm to null for sockets to ensure certificates without SANs applied correctly still work. We can change this in a new NiFi release with other potentially breaking changes.
This closes#3426.
NIFI-6085 - Updated logOut method to use NiFiUserUtils and updated tests.
NIFI-6085 - Added some more integration tests.
NIFI-6085 Suppressed stacktrace when token is used after being invalidated.
This closes#3362.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
NIFI-6253 - simplifying loading of a new process group, handle failure.
NIFI-6253 - Expose nfCanvas.loadProcessGroup functionality through nfCanvasUtils.loadProcessGroup
NIFI-6253 - optional groupId approach to reload
This closes#3458
NIFI-6171 re-added lookupEmail() as fallback
NIFI-6171 additional OIDC scopes via nifi.properties
NIFI-6171 alternative user identification (instead of email) via nifi.properties
NIFI-6171 changed lookupEmail() so that any configured claim can be fetched fro the UserInfo endpoint
This closes#3398
This closes#2346
NIFI-5986 Refactored Stop & Configure feature to enable bulletin/thread notifications and terminate capability in processor dialogs. Also added feature as a menu item to the canvas context menu.
NIFI-5986 Refactored Stop & Configure feature to enable a status bar in a dialog that conveys bulletin/thread notifications and buttons.
NIFI-5986 Refactored Stop & Configure feature to decouple status bar from modal component, updated styling and revised graph synchronization process.
NIFI-5986 Refactored Stop & Configure feature to improve status bar button hide/show functionality.
NIFI-5986: Rebased and resolved conflicts.
NIFI-5986 - Refactored Stop & Configure statusbar observer, as well as processor dialogs to remove duplicative code.
This closes#3281
What this PR changed:
- Standardized required name error message to 'The name of the xxxx must
be specified'
- Added nfErrorHandler.handleConfigurationUpdateAjaxError
- Replaced existing custom error handling for 400 status code
- Replaced handleAjxError with this where appropriate
- Standardized config error dialog title to 'Configuration Error'
- Close the configuration dialog when configuration is successfully
updated
Above convention applied to following components:
- ProcessGroup
- Input/OutputPort
- RemoteProcessGroup
- RemoteGroupPort
- Template
- Connection
- ControllerService
- Label
- Processor
- ReportingTask
- NiFi Settings/GENERAL tab
- Variable
- User
NIFI-6170 Provide consistent UX at Connection config
Fixed Label.
Incorporated review comments
Removed unnecessary else block.
Disable canvas when the message pane is shown
This closes#3401
- Before this fix, NiFi loses information about created versioned flow in case of subsequent snapshot creation failure, and NiFi API returned an error response
- This commit makes:
- The created versioned Flow information is stored even if subsequent snapshot creation fails
- NiFi API to return a successful 200 response in that case, but return versioned flow status as SYNC_FAILURE with an explanation. NiFi UI shows a popup error dialog with the explanation.
- Versioned flow status will be LOCALLY_MODIFIED if the latest version is 0.
This closes#3134.
Signed-off-by: Mark Payne <markap14@hotmail.com>
NIFI-6201 Fixed existing checkstyle complaint on missing Javadoc element.
NIFI-6201 Restored "connection refused" text to relevant registry sync error message.
NIFI-6201 Simplified error logging for bundle not found issues and removed unused bundle coordinate methods.
This closes#3427.
Signed-off-by: Kevin Doran <kdoran@apache.org>
- handle multi-select copy/paste better.
- remove dimension info from getOrigin function
- handle pasting better when pasted items should be centered and not offset from the original.
- fixed strict mode and removed console.logs
- Fixing pasting logic when trying to center the items on the canvas.
This closes#3383
* Added lodash utility library to leverage its throttle capability (and many other useful functions in the future).
* Made lodash available in all JSP's so it could be imported into nf-common (or any component for that matter).
* Added a throttle function to nf-common that just wraps _.throttle
This closes#3393
- Allowing user or group existence enforcement to be parameterized.
- Fixing error handling when loading user groups which may have resulted in stack trace leaking.
This closes#3377.
Signed-off-by: Kevin Doran <kdoran@apache.org>
- Added snap alignment for nf-label, label resize events, and nf-connection
- Shift key now disables snap alignment during the drag event.
- nf-connection load-balance-icon updated
This closes#3335
NIFI-6056 Fixing cassandra bundle so nifi-mock is a test scoped and removing incorrect usage of ProviderCreation exception from nifi-framework-api
This closes#3322.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Use temporary values for port names when updated process groups as part of a Change Flow Version operation.
This avoids the potential for a name conflict between a ports during the update process.
Add a final step to the update process group logic to set the final name on all ports.
This closes#3301.
- Updates to make UpdateRecord and RecordPath automatically update Record schema when performing update and perform the updates on the first record in UpdateRecord before obtaining Writer Schema. This allows the Writer to to inherit the Schema of the updated Record instead of the Schema of the Record as it was when it was read.
- Updated JoltTransformRecord so that schema is inferred on the first transformed object before passing the schema to the Record Writer, so that if writer inherits schema from record, the schema that is inherited is the trans transformed schema
- Updated LookupRecord to allow for Record fields to be arbitrarily added
- Implemented ContentClaimInputStream
- Added controller service for caching schemas
- UpdatedQueryRecord to cache schemas automatically up to some number of schemas, which will significantly inprove throughput in many cases, especially with inferred schemas.
NIFI-5938: Updated AvroTypeUtil so that if creating an Avro Schema using a field name that is not valid for Avro, it creates a Schema that uses a different, valid field name and adds an alias for the given field name so that the fields still are looked up appropriately. Fixed a bug in finding the appropriate Avro field when aliases are used. Updated ContentClaimInputStream so that if mark() is called followed by multiple calls to reset(), that each reset() call is successful instead of failing after the first one (the JavaDoc for InputStream appears to indicate that the InputStream is free to do either and in fact the InputStream is even free to allow reset() to reset to the beginning of file if mark() is not even called, if it chooses to do so instead of requiring a call to mark()).
NIFI-5938: Added another unit test for AvroTypeUtil
NIFI-5938: If using inferred schema in CSV Reader, do not consider first record as a header line. Also addressed a bug in StandardConfigurationContext that was exposed by CSVReader, in which calling getProperty(PropertyDescriptor) did not properly lookup the canonical representation of the Property Descriptor from the component before attempting to get a default value
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3253
Removed bad test.
Refactored filter creation method.
Ensure HSTS header is only applied if NiFi is secured with HTTPS
Small changes to header array list.
Fixed checkstyle errors.
This closes#3273.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Fixed proxy header support to use X-Forwarded-Host instead of X-ForwardedServer
- Added support for the context path header used by Traefik when proxying a service (X-Forwarded-Prefix)
- Added tests to ApplicationResourceTest for X-Forwarded-Context and X-Forwarded-Prefix
- Updated administration doc to include X-Forwarded-Prefix
- Added NIFI_WEB_PROXY_CONTEXT_PATH env var to dockerhub and dockermaven start.sh scripts
- Added documentation for NIFI_WEB_PROXY_CONTEXT_PATH to dockerhub README.md
- Updated ApplicationResource to handle a port specified in X-ProxyPort and X-Forwarded-Port headers
This closes#3129.
Signed-off-by: Kevin Doran <kdoran@apache.org>
- Found several instances of nifi-framework-api's ProviderException being thrown from processors. Changed those to IllegalStateException, as ProviderException is not an appropriate Exception in those cases, and extensions should not depend on nifi-framework-api.
- Performed some cleanup, moving Property Descriptors from Controller Service API's/specs into the implementations. Adding to the Service API results in bringing in nifi-utils to the nifi-standard-services-api-nar, which is a bad practice. The 'main' service api nar should not have a dependency on a util class.
NIFI-5859: Added javadocs. Fixed pom.xml that was left pointing to snapshot version of nar maven plugin
NIFI-5859: Addressing review feedback: adding component type, multiple additional details into separate file(s)
This closes#3192.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-5833 Added unit test to demonstrate arbitrary decryption of sensitive values regardless of processor property sensitive status.
NIFI-5833 Updated GetTwitter documentation with note about 1.9.0+ marking Consumer Key and Access Token as sensitive.
Signed-off-by: joewitt <joewitt@apache.org>
NIFI-5752: Remove an unnecessary String.format
NIFI-5752: Remove an unnecessary block
This closes#3110.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Set up NarAutoLoader to watch directory for new files
- Move NarAutoLoader to JettyServer since it will need access to ExtensionManager
- Created NarLoader to shared between NarAutoLoader and the framework
- Created nifi-framework-nar-loading-utils so we can use nifi-documentation to call DocGenerator
- Add additional bundles to overall map in NarClassLoaders as they are loaded
- Added handling of skipped NARs to include them in next iteration
- Added check of last modified timestamp on NARs
- Refactored JettyServer so we can load additional web contexts while the application is running
- Setting up unit tests
- Remove static use of ExtensionManager
- Adding unit tests for NarLoader
- Extracting interface for ExtensionManager and splitting discovery into it's own interface
This closes#3119.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Removing needClientAuth property since cluster comms now requires two way ssl. Jetty client auth settings are based on configured features.
- Removing dead code.
- Updating documentation.
- Removing references to needClientAuth property in all test resources.
- Removing overloaded util method with strict parameter.
This closes#3102.
NIFI-5686: Fixed unit test in TestSocketLoadBalancedFlowFileQueue; renamed TestProcessorLifecycle to ProcessorLifecycleIT as it is testing integration between many components and largely focuses on high numbers of concurrent tasks to see if it can trigger any threading bugs that may get introduced
NIFI-5686: Extended unit test timeouts
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3062
Added Spock test for NonLocalPartitionPartitioner
Updated NOTICE files for FontAwesome with the updated version (4.7.0) and URL to the free license
Updated package-lock.json with the updated version of FontAwesome (4.7.0)
Added method to FlowFileQueue interface to reset an offloaded queue
Queues that are now immediately have the offloaded status reset once offloading finishes
SocketLoadBalancedFlowFileQueue now ignores back-pressure when offloading flowfiles
Cleaned up javascript in nf-cluster-table.js when creating markup for the node operation icons
Fixed incorrect handling of a heartbeat from an offloaded node. Heartbeats from offloading or offloaded nodes will now be reported as an event, the heartbeat will be removed and ignored.
Added unit tests and integration tests to cover offloading nodes
Updated Cluster integration test class with accessor for the current cluster coordinator
Updated Node integration test class's custom NiFiProperties implementation to return the load balancing port and a method to assert an offloaded node
Added exclusion to top-level pom for ITSpec.class
Updated NodeClusterCoordinator to allow idempotent requests to offload a cluster
Added capability to connect/delete/disconnect/offload a node from the cluster to the Toolkit CLI
Added capability to get the status of nodes from the cluster to the Toolkit CLI
Upgraded FontAwesome to 4.7.0 (from 4.6.1)
Added icon "fa-upload" for offloading nodes in the cluster table UI
- Updated FlowFile Repo / Write Ahead Log so that any update that writes more than 1 MB of data is written to a file inside the FlowFile Repo rather than being buffered in memory
- Update SplitText so that it does not hold FlowFiles that are not the latest version in heap. Doing them from being garbage collected, so while the Process Session is holding the latest version of the FlowFile, SplitText is holding an older version, and this results in two copies of the same FlowFile object
NIFI-5533: Checkpoint
NIFI-5533: Bug Fixes
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2974
Explicitly allow methods GET, HEAD. These are the Spring defaults when the allowedMethods is empty but now it is explicit. This will require other methods like POST etc to be from the same origin (for the template/upload URL).
This closes#3024.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Refactoring StandardFlowFileQueue to have an AbstractFlowFileQueue
Refactored more into AbstractFlowFileQueue
Added documentation, cleaned up code some
Refactored FlowFileQueue so that there is SwappablePriorityQueue
Several unit tests written
Added REST API Endpoint to allow PUT to update connection to use load balancing or not. When enabling load balancing, though, I saw the queue size go from 9 to 18. Then was only able to process 9 FlowFiles.
Bug fixes
Code refactoring
Added integration tests, bug fixes
Refactored clients to use NIO
Bug fixes. Appears to finally be working with NIO Client!!!!!
NIFI-5516: Refactored some code from NioAsyncLoadBalanceClient to LoadBalanceSession
Bug fixes and allowed load balancing socket connections to be reused
Implemented ability to compress Nothing, Attributes, or Content + Attributes when performing load-balancing
Added flag to ConnectionDTO to indicate Load Balance Status
Updated Diagnostics DTO for connections
Store state about cluster topology in NodeClusterCoordinator so that the state is known upon restart
Code cleanup
Fixed checkstyle and unit tests
NIFI-5516: Updating logic for Cluster Node Firewall so that the node's identity comes from its certificate, not from whatever it says it is.
NIFI-5516: FIxed missing License headers
NIFI-5516: Some minor code cleanup
NIFI-5516: Adddressed review feedback; Bug fixes; some code cleanup. Changed dependency on nifi-registry from SNAPSHOT to official 0.3.0 release
NIFI-5516: Take backpressure configuration into account
NIFI-5516: Fixed ConnectionDiagnosticsSnapshot to include node identifier
NIFI-5516: Addressed review feedback
This closes#2947
This closes#3044
- Revert 87cf474e54 to enable connection
pooling
- Changes the expected HTTP status code for the 1st request of a
two-phase commit transaction from 150 (NiFi custom) to 202 Accepted
- Corrected RevisionManager Javadoc about revision varidation protocol
Refactored some test code to be clearer.
Renamed some resources to be consistent across modules.
Changed passwords to meet new minimum length requirements.
This closes#3018
The operation policy allows that a user to operate components even if they does not have direct READ/WRITE
permission of the component.
Following operations are controlled by the new operate policy:
- Start/stop/enable/disable Processors, ControllerServices,
ReportingTasks, Input/OuputPorts
- Enable/disable transmission of RemoteInput/OutputPorts and
RemoteProcessGroups
- Terminate Processor threads
Refactored what API exposes
The previous commit let API exposes few fields in DTO. But we should
avoid returning partial DTO as it complicates authorization logic.
Instead, this commit adds StatusDTO for ReportingTaskEntity and
ControllerServiceEntity, so that it can be returned regardless of having
READ permission. Component DTO can only be returned with a READ
permission.
Refactor RPG same as ControllerService.
WIP incorporating review comments.
Incorporated review comments
- Cleaned up merger classes
- Recreate DTO instance at each function during two phase commmit
Restrict enabling ControllerService without read permission
Revert the last commit.
Fix review comments.
- Renamed confusing static method names and its parameters
- Removed unnecessary permission checks from UI condition
Fixed delete action display condition.
Fixed NPE at Summary.
Apply operation policy to activateControllerServices.
Removed OperationPermissible from ComponentEntity.
This closes#2990
NIFI-5366 - Added unit test. Added single quotes around 'self' for frame-ancestors CSP header.
NIFI-5366 - Fixed dependencies.
This closes#2989.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
NIFI-5540 Added failing unit test and ignored regression test to demonstrate missing default sensitive properties key in flow election encryptor creation.
NIFI-5540 Added equality logic to StringEncryptor and utility equality methods to CryptoUtils.
NIFI-5540 Added default sensitive properties key population logic and log warning to StringEncryptor.
NIFI-5540 Cleaned up formatting.
NIFI-5540 Cleaned up boolean logic.
NIFI-5540 Added Javadoc to StringEncryptor.
NIFI-5540 Added unit test for StringEncryptor#equals().
NIFI-5540 Added performance benchmarking unit tests for constantTimeEquals methods for String, byte[], and char[].
NIFI-5540 Fixed checkstyle issue.
NIFI-5540 Fixed unit tests for default key population.
This closes#2959.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Refactored VolatileComponentStatusRepository to avoid holding on to ProcessorStatus objects, etc, and only keep what they need
- Updated VolatileComponentStatusRepository to ensure that we are efficiently storing metrics for processors, etc. that are not running
This closes#2939
Signed-off-by: Mark Payne <markap14@hotmail.com>
NIFI-5442 Populate request contextPath attribute during AccessResource before displaying on message-page.jsp.
Refactored shared code from CatchAllFilter to WebUtils.
NIFI-5442 Refactored filter and context path code to shared parent filter and subclass.
NIFI-5442 Removed unnecessary initParams from nifi-web-ui web.xml.
NIFI-5442 Added explicit dispatchers to nifi-web-ui web.xml and removed unnecessary code from AccessResource.
This closes#2908
Fixed tests in NiFiGroovyTest in the nifi-runtime module
Updated NiFi.createBootstrapClassLoader to log a warning if lib/bootstrap does not exist rather than throwing a FileNotFoundException, since it already catches MalformedUrlException if there's an issue adding one of the bootstrap JARs to the bootstrap classpath
Explicitly handling InvocationTargetException in NiFi.initializeProperties to unwrap the cause and rewrap as an IllegalArgumentException to propogate the real cause of the underlying exception thrown by NiFiPropertiesLoader
This closes#2821.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Updated Javadoc for SiteToSiteClient#createTransaction() and HttpClient implementation.
Reverted exception listing in method contract for SiteToSiteClient#createTransaction and HttpClient tion of same.
Reverted import ordering in TestSiteToSiteClient.
Reverted exception listing in TestGetHDFSFileInfo, TestListHDFS, and StandardHttpFlowFileServerProtocol.
Restored @SuppressWarnings annotation and removed unnecessary "public static" keywords from inner classes in SiteToSiteClient.
This closes#2841.
Signed-off-by: Joe Witt <joewitt@apache.org>
Added integration test for checking the ExceptionFilter catches malicious string exceptions.
Made minor changes to PR 2840 for code style.
This closes#2840.
Co-authored-by: Andy LoPresto <alopresto@apache.org>
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Minor adjustments following PR.
- Avoiding additional find operation when authorizing components when populating component details.
- Requiring access to provenance events when downloading content or submitting a replay as they may provide events details.
- Updating the REST API docs detailing the required permissions.
- Updating the wording in the documentation regarding the provenance and data policies.
- Removed the event attributes from the authorization calls that were verifying access to provenance events.
- Only checking content availability when the user is authorized for the components data.
- Addressing typo in JavaDoc.
This closes#2703
- Ensuring the proxy headers are considered when redirecting the user following a OIDC or Knox login exchange.
This closes#2763.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Disabling the resize of SlickGrid when there is an active edit occuring. This will prevent the accidental closure of the current edit.
- Triggering a resize when the active editor closes.
This cloese #2766
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
- Preventing duplicate bulletins that are registried under different contexts (e.g. Controller and Component).
This closes#2771.
Signed-off-by: Mark Payne <markap14@hotmail.com>
NIFI-5241: Extended timeout that Jetty uses before closing an active HTTP connection. Because the UI refreshes every 30 seconds by default, and the Jetty connection pool times out every 30 seconds by default, we very frequently saw new HTTP connections being created for the UI refreshes. This resulted in 4 new connections and 4 SSL handshakes occurring every 30 seconds. By extending the timeout, we now see those connections being reused and SSL Handshakes no longer occurring frequently
NIFI-5241: Set Jetty idle timeout to double the amount of time for browser to refresh
NIFI-5241: Fixed synchronization issue with EventSumValue
This closes#2752
- Fixing missing message when a node is disconnected from a cluster.
- Updating endpoints to accept a flag to allow for changes to be made to a disconnected node.
- Updating custom UIs to acknowledge disconnected nodes prior to performing modifications.
- Avoid triggering async validation for each update to component when instantiating a template (such as copy/paste or templates). Added debug logging to indicate when and why we are triggering validation; removed unit test that made poor assumptions about the inner workings of the FlowSynchronizer that resulted in failures when we make calls into processors that the unit test doesn't know about"
This closes#2731.
- PR Fix - 'Execution' dropdown will now be shown in all cases
- Annotated ListGCSBucket with PrimaryNodeOnly
This closes#2509.
Signed-off-by: Mark Payne <markap14@hotmail.com>
NIFI-5204: If processor joins cluster and inherits 'disabled' state but is still stopping, ensure that the state becomes disabled when the processor finishes stopping
This closes#2713
The bootstrap process (RunNiFi) detects Java 9 and adds "--add-modules=java.xml.bind" to the command to start NiFi
Updated OSUtils to detect Java 9 and reflectively invoke the Process.pid() method to get the PID of the NiFi process
Added java debug variable to nifi.sh to allow debugging of the bootstrap process (RunNiFi)
This closes#2708
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
NIFI-950: Still seeing some slow response times when instantiating a large template in cluster mode so making some minor tweaks based on the results of CPU profiling
NIFI-5112: Refactored FlowSerializer so that it creates the desired intermediate data model that can be serialized, separate from serializing. This allows us to hold the FlowController's Read Lock only while creating the data model, not while actually serializing the data. Configured Jersey Client in ThreadPoolRequestReplicator not to look for features using the Service Loader for every request. Updated Template object to hold a DOM Node that represents the template contents instead of having to serialize the DTO, then parse the serialized form as a DOM object each time that it needs to be serialized.
NIFI-5112: Change ThreadPoolRequestReplicator to use OkHttp client instead of Jersey Client
NIFI-5111: Ensure that if a node is no longer cluster coordinator, that it clears any stale heartbeats.
NIFI-5110: Notify StandardProcessScheduler when a component is removed so that it will clean up any resource related to component lifecycle.
NIFI-950: Avoid gathering the Status objects for entire flow when we don't need them; removed unnecessary code
NIFI-950: Bug fixes
NIFI-950: Bug fix; added validation status to ProcessorDTO, ControllerServiceDTO, ReportingTaskDTO; updated DebugFlow to allow for pause time to be set in the customValidate method for testing functionality
NIFI-950: Addressing test failures
NIFI-950: Bug fixes
NIFI-950: Addressing review feedback
NIFI-950: Fixed validation logic in mock framework
This closes#2693
- Removing unnecessary logic in the zoom handler since the zoom event is no longer triggered during onClick.
This closes#2692
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
- Updating how the nf-reporting-task module is injected to the nf-controller-service module.
This closes#2688
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
- Added logic to check for simultaneous configuration of HTTP and HTTPS connectors in JettyServer.
- Added test logging resources. Added unit tests.
- Refactored shared functionality to generic method which accepts lambdas.
Fixed unit test with logging side effects.
- Added note about exclusive HTTP/HTTPS behavior to Admin Guide. Fixed typos.
This closes#2683.
Signed-off-by: Kevin Doran <kdoran@apache.org>
- Forcing FileSystem statistics thread to be interrupted when HDFS processors are stopped
- Stop creating temp components during import from registry, use bundle info instead
This closes#2668.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Fixed bug in unit test
Addressed review feedback/addressed issue where if a group is moved inside another group, the descendant processors of the moved group did not have their service references updated properly. Also addressed an issue where if a service is defined in Group A, then Group B lives within Group A and has a processor that references a service at the level of Group A, we allowed user to move Group B outside of Group A (but wouldn't allow the processor to be moved out of scope by itself).
This closes#2678
- Adding UI controls for terminating hung threads.
- Showing current number of terminated threads.
- Fixing issue when replicating terminate threads request throughout the cluster.
This closes#2607.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Allowing the enable/disable buttons to be active under the same conditions as the start/stop buttons.
This closes#2633.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Added dedicated conditions for Funnels
- Fixed stale Javadoc
- Stopped caching hasNonLoopConnection variable
- Grouped some conditions to isSourceComponent variable for better
readability
This closes#2634.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Existing web error index page suggests user may have meant /nifi/ and crafts a contextPath link, may as well redirect them automatically. Non-human (i.e. curl/wget) can still retrieve the original source without being redirected.
Signed-off-by: joewitt <joewitt@apache.org>
- Fixing process group audit advice.
- Setting spring security user in background threads.
- Removing unnecessary overloaded methods.
This closes#2626.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- take into account input requirement for documentation rendering
- Renamed variable registry scope and added comments
- Doc + change in mock framework to check scope + update of components + UI
- Processing properties and property descriptors in Controller Service referencing components unconditionally.
This closes#2602.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Ensuring appropriate response in checkAuthorization when user is null.
- Ensuring the user reference is passed down when applying variable changes.
This closes#2598.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Fixing required permission for PutParquet.
NIFI-5008:
- Ensuring all restricted components are tagged as such.
This closes#2583.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Fixing RPG port merging.
- Adding unit tests.
- Removing unecessary sorting that wasn't maintained while clustered.
This closes#2551.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Implemented review feedback. Refactored data model to make the API cleaner and delineate more along the lines of what permissions are required in order to see which details
Implementing review feedback
Removed sensitive information from the diagnostics reports
Fixed bug in merging logic for GCDiagnosticsSnapshots
This closes#2468
- Addressing memory leak from lingering authorization results that did not represent actual access attempts. This closes#2511.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Initial set of components marked with the HighResourceUsageScenario annotation.
- Added customized descriptions to SystemResourceConsideration annotations for MergeContent, SplitContent, SplitJson, SplitText, and SplitXml.
This closes#2475.
Signed-off-by: Mark Payne <markap14@hotmail.com>
...Search results
* Separated the search functionality.
* Added a unit test.
* Added the PG info to UI (a mere draft).
* Introduce the nearest versioned group
* Removed the top level group results in favour of the nearest versioned group.
* This closes#2364
NIFI-4772: Introduced a new thread-pool with 2 threads that will be used for monitoring lifecycle task. This means that if all threads in the timer-driven thead pool are blocked by processors that don't complete their @OnScheduled methods, we have a separate thread pool that at least gives us a chance of interrupting those threads
NIFI-4772: Remove unused import
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2403
- Everywhere that we ignore adding remote ports we should ignore removing remote ports as well in flow diffs
This closes#2462.
Signed-off-by: Bryan Bende <bbende@apache.org>
Adds stricter checks in AuthorizerFactoryBean for unique ids within
a given type of provider and requires unique providers in composite
and composite-configurable user group providers. Failed validation
checks cause startup to fail. Adds test cases for these new rules.
This closes#2419.
Signed-off-by: Bryan Bende <bbende@apache.org>
Added HostHeaderHandler regression unit tests before adding new functionality.
Added logic for parsing nifi.web.proxy.host property.
Added default property in nifi.properties and pom.xml.
Added logic for IPv6 and custom default hostnames.
Improved error messaging.
Added HostHeaderHandler unit tests.
Disabled HostHeaderSanitizationCustomizer in HTTP mode.
Fixed HTML escaping in error message.
Improved error message.
Added failing unit test for parsing custom hostnames.
Fixed custom hostname parsing.
Fixed unit tests.
Added TODO for IPv6 custom hostname parsing and unit test.
Added IPv6 custom hostname parsing and unit tests.
Fixed checkstyle issues.
Removed empty element in host list when no value defined.
Improved error message formatting.
Added unit tests.
Removed HostHeaderSanitizationCustomizer.
Removed InvalidPropertiesFormatException from NiFiProperties.
Removed InvalidPropertiesFormatException from HostHeaderHandler.
This closes#2415
- Only showing the import link in the new Process Group dialog when initiating from the toolbar.
This closes#2395.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Adds support for NiFI Registry config files to the encrypt-config tool
in NiFi Toolkit.
Also adds decryption capability to encrypt-config tool.
This closes#2376.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
NIFI-4436: Fixed a bug that caused a flow not to successfully change version if a connection is added to an existing component and that component is running at time of version change
NIFI-4436: Fixed bug with ordering of controller services being enabled and disabled
NIFI-4436: Fixed bug that prevented local input and output ports from being stopped and started as needed
NIFI-4436: Fixed bugs around referencing controller services that are at a higher level than the versioned flow
NIFI-4436: Ensure that we clear components from FlowController's cache when removed and that they are added to cache when created.
NIFI-4436: Fixed error message coming back if component is invalid when trying to be restarted/re-enabled
NIFI-4436: Addressed issue with children of a removed process group not being considered 'affected components' and as a result not being stopped/disabled/restarted/re-enabled
This closes#2219.
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
- Minor tweak to when the max height for the options list is calculated per PR comments.
- Resolving logic issue in two phase commit when updating variable registry.
- Fixing variable visibility.
- Updating buckets permissions based on new model.
- Adding check to ensure that flow name is non null before checking the length.
- Adding versioned flow state to the Process Group tab in the Summary table.
- Fixing issue with navigating to Controller Services from the local changes dialog.
- Code clean up.
- Improved error handling.
- Minor UX improvements.
- Always showing Process Group state to complement the aggregation counts.
- Adding the Process Group state to the top status bar.
- Code clean up.
- Improved error handling.
- Minor UX improvements.
- Adding message to indicate that variables do not support sensitive values.
- Preventing a user from changing the flow version to the current version.
- Only presenting buckets a user has appropriate permissions to.
- Adding basic auditing to the version control actions.
- Updating front end to use version control state/status.
- Fixing copy/paste issue during revert local changes.
- Code clean up in the breadcrumbs.
- Update VersionsResource authorization and two phase commit object usage.
- Addressing miscellaneous minor UX issues.
- Updating comments UX for all components.
- Updating the styling of PG and RPG to be more consistent.
- Adding the icons for nested versioned process groups.
- Calculating the number/states of nested versioned process groups.
- Fixing default border radius.
- Code clean up.
- Ensuring component visibility is updated after updating/reverting.
- Fixing sort on component name in local changes dialog
NIFI-4526:
- Added front end controls for updating RPG target URL.
- Clearing bucket/flow/versions when changing the selected registry/bucket.
- Using the versioned flow to get the group name when importing.
- Adding menu items for viewing local changes.
- Showing local changes during revert request.
- Adding the version number to the start version control, commit, and change version dialog.
- Showing a loading item in the combo's while querying for the registries, buckets, and flows.
- Adding tooltips to display version control information on the canvas.
- Adding progress bar dialogs for changing version and reverting local changes.
- Updating canvas and breadcrumb according to the version control state.
- Updating to use registry name, bucket name, and flow name where appropriate.
- Added the import dialog for importing a versioned flow into a new process group.
- Added the change version dialog for upgrading/downgrading a versioned flow.
- Adding support to save a version of a flow based on a selected Process Group.
- Adding support for revert changes back to the most recent version.
- Adding support to disconnect from version control.
- Moving the version control information out of the entity objects and into the dto's.
- Fixing checkstyle issues.
NIFI-4502:
- Updating the UI to allow for the user to register registry clients.
- Updating the version control menu item names.
- Initial checkpoint: able ot start version control and detect changes, in standalone mode, still 'crude' implementation
- Checkpoint: Can place flow under version control and can determine if modified
- Checkpoint: Change version working in some cases. Does not work if processor removed because COMPONENT_REMOVED type has ComponentA whose ID is the VersionedComponentID but we are trying to call ProcessorDAO.get() with this ID
- Checkpoint: Able to change flow from Version 1 to Version 2 and back. Not yet tested with controller services. Have not tried changing/removing connections. Not cluster-friendly yet. All inline, not in background. Have not taken into account ports, funnels, remote ports, etc. Have not tested with Labels yet
- Checkpoint after implementing ClusterReplicationComponentLifecycle instead of JerseyClientComponentLifecycle
- Checkpoint: Updated to allow starting version control and updating version in clustered mode
- Checkpoint: Updated versioning endpoint so that when version of a flow is updated, the bundle information is populated and the snapshot is replicated to the cluster.
- Checkpoint: Implemented endpoint for reverting to previously sync'ed version of a flow and updated version control endpoint so that Process Group can be pushed as a new version to existing flow instead of only creating a new flow
- Checkpoint: Updated so that if a Process Group is under Version Control and it has a child Process Group, which is also under Version Control, we can handle that gracefully. Not yet tested because it depends on updates to the nifi-registry module, which can't be compiled due to maven dependency conflicts
Removes user existence check from FileUserGroupProvider when
group is created or updated. Replaces it with check in the
Authorizer Decorator class created by Authorizer Factory, so
that all providers are used.
Also fixes bug when searching for group membership by user
that returns results across all providers.
NIFI-3472 NIFI-4350 Removed explicit relogin code from HDFS/Hive/HBase components and updated SecurityUtils.loginKerberos to use UGI.loginUserFromKeytab. This brings those components in line with daemon-process-style usage, made possible by NiFi's InstanceClassloader isolation. Relogin (on ticket expiry/connection failure) can now be properly handled by hadoop-client code implicitly.
NIFI-3472 Added default value (true) for javax.security.auth.useSubjectCredsOnly to bootstrap.conf
NIFI-3472 Added javadoc explaining the removal of explicit relogin threads and usage of UGI.loginUserFromKeytab
Readded Relogin Period property to AbstractHadoopProcessor, and updated its documentation to indicate that it is now a deprecated property
Additional cleanup of code that referenced relogin periods
Marked KerberosTicketRenewer is deprecated
NIFI-3472 Cleaned up imports in TestPutHiveStreaming
- Ensure the /nifi-api/controller redirection filter executes before matching.
This closes # 2358.
Signed-off-by: Andy LoPresto <alopresto.apache@gmail.com>
Adds authorizers.xml to the files understood by the encrypt-config
tool in the NiFi Toolkit. If enabled, then the sensitive properties
for LdapUserGroupProvider in authorizers.xml will be encrypted.
Also fixes a bug wherein encrypt-config replaces multiple XML nodes
in login-indentity-providers.xml when LdapProvider is not the first
provider listed in the file.
Enable properties in authorizers.xml to be encrypted by the master key.
This closes#2350.
Signed-off-by: Andy LoPresto <alopresto.apache@gmail.com>
Corrects time unit conversion for the Sync Interval config property
for LdapUserGroupProvider in authorizers.xml.
Also enforces a minimum value of 10 secs for the Sync Interval to help
catch unintentional misconfigurations, for example users upgrading
from previous versions, where tiny Sync Interval values could be set
as a workaround for NIFI-4667.
This closes#2341
Ensure that when ProcessSession.clone(FlowFile) is called, we obtain the most recent version of the FlowFile before attempting to obtain FlowFile size.
Signed-off-by: joewitt <joewitt@apache.org>
Rob Fellows
thenatog
Koji Kawamura
Matt Gilman
Mark Payne
Scott Aslan
Mark Bean
Jeff Storck
simonl
Alex Aversa
Peter Wicks
Andrew Lim
aeaversa
Andy LoPresto
Brandon Devries
Joe Witt
Bryan Bende
Ryan Bower
Kevin Doran
Jan Hentschel
Denes Arvay
Mike Moser
Dorian Bugeja
Kemix Koo
Kotaro Terada
pepov
Andrew I. Christianson
Lars Francke
Matthew Burgess
Damian Czaja
Marco Gaido
Aldrin Piri
zenfenan
Frederik Petersen
Pierre Villard
Alex Moundalexis
Vi On
Derek Straka
Jorge Machado
Takanobu Asanuma
yuri1969
Peter G. Horvath
sbouchex
Senthilannaswamy
Kevin Doran
Joe Percivall
Joey Frazee