JsonContentConversionExceptionMapper, JsonMappingExceptionMapper, JsonParseExceptionMapper.
Registered the custom ExceptionMappers.
Added unit tests to throw Exception for string port value and sanitize script input. Handled null or empty JsonMappingException reference path.
Added the Apache license to Groovy Test.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- decoupled flow update request behavior from VersionsResource into new abstract FlowUpdateResource
- added replace process group functionality in ProcessGroupResource
- parameterized FlowUpdateResource and created entity hierarchies to allow for maximum code sharing across different update types
- refactored flow update methods to make use of commonality across different update types whenever possible
- fixed issues in StandardProcessGroup verify update methods where same components existed in different ancestry chains but were considered a match when they shouldn't be
- improved StandardProcessGroup to properly match up components on update using generated versioned component ids, when necessary to allow for update flow to efficiently match common components on flow import
This closes#4023.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Added UI versioned flow supportsDownload functionality with download flow menu item
- Added VersionsResource endpoint for downloading versioned flow with registry-related info removed
- Added ProcessGroupResource endpoint for downloading current flow with registry-related info removed
- Added StandardNifiServiceFacade functionality for downloading both current and versioned flow
- Added XmlTransient markers on variables introduced by Instantiated model classes so they do not appear in serialized download
- Updated NiFiRegistryFlowMapper.mapParameterContexts to handle mapping nested parameter contexts for use in producing a complete VersionedFlowSnapshot
- Added ability for NiFiRegistryFlowMapper to map nested process groups ignoring versioning for use in producing a complete VersionedFlowSnapshot
- Added unit tests where helpful
NIFI-6872: PR response...
- Updated mapParameterContext to return a Map to handle uniqueness of contexts by name since ultimately everything converted it to a map anyway. The VersionedParameterContext class from the registry model doesn't support hashcode/equals currently so returning a Set wouldn't work.
- Updated assert calls to put expected value as first parameter and actual as second parameter
- Added one time password (OTP) support for flow download endpoint to support non cert based authentication
This closes#3931
- Fixed checkstyle errors.
- Added PeerPersistence interface.
- Expose RemoteProcessGroup state via REST API
- Made stateManager transient.
This closes#3677.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-6589: Updated CuratorLeaderElectionManager to cache results for no more than 5 seconds per review feedback
Signed-off-by: Joe Witt <joewitt@apache.org>
* NIFI-6510 Implement initial analytic engine
* NIFI-6510 Implemented basic linear regression model for queue counts
* NIFI-6510 Initial analytics REST endpoint and supporting objects
* NIFI-6510 Connect the dots for StatusAnalytics -> API
* NIFI-6510 Added poc engine with prediction model caching
(cherry picked from commit e013b91)
DFA-9 - updated logging and corrected logic for checking if not in backpressure
(cherry picked from commit a1f8e70)
* NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics
(cherry picked from commit 050e0fc)
(cherry picked from commit 9fd365f)
* NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id
(cherry picked from commit 14854ff)
DFA-9 - reduced snapshot interval to 1 minute
(cherry picked from commit 36abb0a)
* NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions
* NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly
* NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly"
This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1.
* NIFI-6510 Added prediction fields for use by UI, still need to be populated
* NIFI-6510 Analytics Framework Introduction (#10)
* DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model.
DFA-9 - added logging
* DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval
* DFA-9 - checkstyle fixes
* NIFI-6510 Add prediction percent values and predicted interval seconds
(cherry picked from commit e60015d)
* NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not.
(cherry picked from commit 6fae058)
* NIFI-6510 Added tests for engine
(cherry picked from commit 6d7a13b)
* NIFI-6150 Added tests for connection status analytics class, corrected variable names
(cherry picked from commit 58c7c81)
* NIFI-6150 Make checkstyle happy
(cherry picked from commit b6e35ac)
* NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing
* NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics
* NIFI-6510 Adjusted interval and incorporated R-squared check
Updates to support multiple variables for features, clearing cached regression model based on r-squared values
Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support).
Ratcheck fixes
Added test for SimpleRegression. Minor fix for OLS model
fixed test errors
fixed checkstyle errors
(cherry picked from commit fab411b)
* NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11)
* NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties
NIFI-6566 - changes to allow scoring configurations for model in nifi.properties
NIFI-6566 - added default implementation value to NiFiProperties
NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3663
* NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type
* NIFI-6586 - documentation and comments
This closes NIFI-6586
Signed-off-by: Andrew I. Christianson <andy@andyic.org>
* NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI
* Add multi-line tooltips with detail for connection queue back pressure graphics.
* Add estimated time to back pressure to connections summary table.
* Add back pressure prediction ticks.
* add moment.js to format predicted time to back pressure
* tweak summary table headings to match data displayed. re-order connection summary columns
* NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment.
* NIFI-6510 - add an enable/disable property for analytics
* NIFI-6510 - documentation updates for enable/disable property
* NIFI-6510 - UI: handle the scenario where backpressure predictions are disabled (#3685)
* NIFI-6510 - admin guide updates to further describe model functionality
* NIFI-6510 - code quality fixes (if statement and constructor)
* NIFI-6510 - log warnings when properties could not be retrieved. fixed incorrect property retrieval for score threshold
* NIFI-6510 Extract out predictions into their own DTO
* NIFI-6510 Optimize imports
* NIFI-6510 Fix formatting
* NIFI-6510 Optimize imports
* NIFI-6510 Optimize imports
* NIFI-6510 - Notice updates for Commons math and Caffeine
* NIFI-6510 - UI updates to account for minor API changes for back pressure predictions (#3697)
* NIFI-6510 - Fix issue displaying estimated time to back pressure in connection summary table when only one of the predictions is known.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3705
* NIFI-6510 Rip out useless members
* NIFI-6510 - dto updates to check for -1 value
* NIFI-6510 - checkstyle fix
* NIFI-6510 - rolled back last change and applied minNonNegative method
* NIFI-6510 Rip out useless members
[NIFI-6282] when creating a parameter context inline during PG configuration set the newly created parameter context as the selected option
[NIFI-6282] If a request to update a parameter context fails, then update the button model to give the user the ability to Apply or Cancel again.
[NIFI-6282] address pr review comments
[NIFI-6282] remove es6 let
[NIFI-6282] update marshall parameters logic and add comments
[NIFI-6282] deterministic parameter usage listing, update CS status on PG PC change, expand all twisties by default, remove es6 const
[NIFI-6282] update regex and
[NIFI-6282] update parameter loading/serialization/marshalling
[NIFI-6282] use referencingComponents instead of affectedComponents
[NIFI-6282] activate Apply button for sensitive parameter set empty string change
[NIFI-6282] fix bug with PG parameters context menu enable
[NIFI-6282] only allow delete and recreate of a parameter with equivalent sensitivity
[NIFI-6282] display referencing components during parameter management as well as during the parameter context update
[NIFI-6282] display no value set in parameter table when parameter value is null
[NIFI-6282]
- Add ellipsis to referencing component names.
- Addressing issues canceling update requests.
- Addressing issues with incorrect service scope.
- Addressing issue showing the affected parameters.
NIFI-6382: Fixed bugs reported in code review: sensitive properties referencing parameters should be exported to registry; when importing a versioned flow, any new parameters should be added to the parameter context
NIFI-6382: Updated logic for handling parameter references with sensitive properties when integrating with nifi registry
NIFI-6382: If user does not have permissions to READ a parameter context with a given name when importing a flow or changing flow version, ensure that NiFi properly generates an AccessDenied messages instead of throwing NullPointerException
This closes#3642.
Signed-off-by: Bryan Bende <bbende@apache.org>
Added "jigsaw" profile to multiple modules, which auto-activates when building with Java 11 and adds several dependencies: jaxb, activation, and annotation-api.
Updated SslContextFactory to return tuple of socket factory with trust manager for issue with okhttp api changes in java 9+
Updated TestGet/PostHTTPGroovy to use default SSL context to get default cipher suites
Updated StandardTemplateDAOSpec.groovy, was using a 37-character UUID, shortened to 36 characters.
Multiple tests updated to specifically use TLSv1.2, since two-way TLSv1.3 for some component tests fail during the Java 11 build. Needs more investigation.
Replaced GString with String concatenation for map key's value in ScriptedReportingTaskGroovyTest to avoid runtime error of casting GString to String when retrieved from a map that is storing the GString
Removed nifi-toolkit-api plugin configuration of maven-compiler-plugin, it is inherited
Resolved deprecation errors during Groovy compile for bouncycastle, Extension should be used in place of X509Extension
Fixed JNAOverridingJUnitRunner's creation of the classpath for the custom classloader created to be able to mock jna classes
Removed import of IOUtils (from the wrong package) from InferenceSchemaStrategy
Updated maven-compiler-plugin version to 3.8.1
TLS (default in Java 11 is TLSv1.3) working for Site-to-Site client tests after upgrading JDK installation to JDK 11.0.3, httpclient5 5.0-beta4, and httpasyncclient 4.1.4
HttpNotificationService updated to find and use the first found X509TrustManager rather than casting directly from the array of TrustManagers returned from SslContextFactory
Removed unnecessary throws declaration from getSslSocketFactory method
Replaced DnsContextFactory.class.getName() with a string to avoid having to export/open the jdk.naming.dns module
Updated TestGetIgniteCache and TestPutIgniteCache.java to skip tests on Java 11+ (via Assume), and noted the check should be removed once a version of Ignite is released that supports Java 11
Updated SpringContextProcessor to create proper ClassLoader and uses URLClassloader.getResource() instead of URLClassloader.findResource() in isConfigResolvable.
Due to the application classloader no longer being a URLClassLoader in Java 9+, the URLClassLoader created in isConfigResolvable no longer has explicit access to the parent's resources as URLs.
URLClassLoader.getResource() searches the parent classloaders, and must be used instead of URLClassLoader.findResource() which only searches the URLs in the URLClassLoader and does not search the parent classloaders.
NIFI-5176 Moved exclusion of jdk.tools to the jigsaw profile in the POMs of nifi-hbase_1_1_2-client-service and nifi-hbase_2-client-service
NIFI-5176 Updated site-to-site-client's POM to use properties to establish the correct httpclient dependency for when building with Java 11
This closes#3404.
- Added nifi-parameter module
- Added Parameter to nifi-api
- Added ParameterContext and ParameterLookup
- Updated EL to support Parameters
- Updated backend so that any property can make use of Parameters
- Added web endpoint for Parameter Contexts
- Updated Templates to make use of Parameter Contexts
- Updated Versioned Flows to make use of Parameter Contexts
- Updated Stateless NiFi to support Parameters and take Parameters as part of the configuration, not Variables.
NIFI-6380: Addressed review feedback and fixed bugs; added additional unit and integration tests to verify
NIFI-6380: Added Description to Parameter Context
NIFI-6380: Fixed checkstyle violations
NIFI-6380: Fixed bug that caused updating a Parameter Context to fail if the name is provided and unchanged
NIFI-6380: If parameter is being deleted, don't worry about its sensitivity flag. This addresses a bug where the deletion of a Sensitive Parameter would require that the Parameter be submitted with a value of null and a sensitivity flag of true; else it would provide an error indicating that the parameter can't be changed from sensitive to non-sensitive. Now, the sensitivity flag is ignored.
NIFI-6380: Fixed bug around unsetting Process Group's Parameter Context
NIFI-6380: Moved lastRefreshed timestamp from ParameterContextDTO to ParameterContextsEntity and renamed to currentTime to match the pattern of ControllerServicesEntity rather than FlowHistory. Added parameterContextId to ProcessGroupFlowDTO
NIFI-6380: Added additional integration tests around escaped parameter references
NIFI-6380: Additional tests and bug fix for referencing EL from within another EL Expression
NIFI-6380: Created ParameterEntity to house a 'canWrite' flag for parameters and updated ParameterContext to use it. Updated ParameterContextUpdateRequestDTO to include a Set<AffectedComponentEntity> indicating the components that are affected by the update
NIFI-6380: Addressed review feedback
NIFI-6380: Addressed additional review feedback, mostly around code cleanup
NIFI-6380: Bug fix
NIFI-6380: Addressed more review feedback; fixed a couple of minor bugs encountered when testing
NIFI-6380: Bug fix around Parameter escaping for properties that support Expression Language. Fixed inconcsistency in Authorization hierarchy between /parameter-contexts/1234 and /parameter-contexts to ensure that the parent of /parameter-contexts is /controller, regardless of how we arrive at the /parameter-contexts resource
NIFI-6380: Fixed but around using ProcessContext#newPropertyValue(String) that previously resulted in causing the Parameters to be evaluated a second time, thereby ignoring escape characters. Updated Integration Test to verify behavior. Also fixed bug in LoadBalanceQueueIT as it was a one-liner and noticed it while running tests to verify other behavior
NIFI-6380: Ensure that if Processor or Controller Services references or de-references a Parameter that user making the change has READ policy on the Parameter Context
NIFI-6380: Verify permissions on Parameter Context when creating processor/controller service and when moving between groups, insantiating template, reverting/change flow version
NIFI-6380: Addressed problem that resulted from rebasing against master; fixed bug in showing the Affected Components for a particular Parameter Context Update
NIFI-6380: Fixed NPE that occurs when attempting to update controller-level Controller Service
NIFI-6380: Updated validation error text when referencing parameter from controller-level controller service or reporting task
NIFI-6380: Ensuring permissions for copy/paste when components reference parameters.
NIFI-6380: Do not require READ policy on Parameter Context in order to un-reference a parameter, only to reference one.
NIFI-6380: Restricted parameter names to match those in the Feature Proposal. Fixed bug that allowed copy & paste of a PG for which user does not have READ policy for a child/descendant PG's parameter context
This closes#3536
NIFI-5856: Fixed checkstyle violation
NIFI-5856: Addressed review feedback
NIFI-5856: When assigning a matching Controller Service at a higher level for Process Group import/update, ensure that we consider the User's read permissions when selecting the Controller Service
This closes#3553.
Signed-off-by: Bryan Bende <bbende@apache.org>
Fixed test failures in nifi-couchbase-processors, BinaryDocument matcher replaced with ByteArrayDocument
Fixed test failures in nifi-riemann-processors, anyInt() matcher replaced with anyLong() matcher, calling method passes a long, not int
Removed unnecessary method mocks from nifi-toolkit-tls tests, TlsCertificateAuthorityServiceHandlerTest and TlsCertificateSigningRequestPerformerTest, since those were flagged by Mockito as unnecessary (they're unused)
Removed explicit mockito dependency version in nifi-gcp-processors pom to inherit version from nifi's pom.xml
Updated ArgumentMatchers in Kafka 0.10, 0.11, 1.0, and 2.0 processor tests, since in Mockito 2.x, the "any" matchers no longer allow nulls
Updated ArgumentMatchers in nifi-jolt-transform-json-ui, since in Mockito 2.x, the "any" matchers no longer allow nulls
Removed unnecessary method mocks from MetricsReportingTaskTest
Updated TestStandardRemoteGroupPort to return Long instead of Integer for test flowfile.size() invocations
Updated AbstractCassandraProcessor to include keyspaceProperty.getValue() in null check
Updated SimpleProcessLogger and TestSimpleProcessLogger, vararg matching does not work the same in Java 8 and 11
Updated TestStandardProcessScheduler to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated TestPutHiveStreaming to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated FetchParquetTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated ControllerSearchServiceTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Removed usage of Whitebox from GetAzureEventHubTest due to Mockito 2.x, replaced with FieldUtils
Removed usage of Whitebox from StandardOidcIdentityProviderTest due to Mockito 2.x, replaced with FieldUtils
Updated apache-rat-plugin configuration in root POM to make use of useIdeaDefaultExcludes which makes the rat plugin exclude IntelliJ artifacts
Updated several modules to use mockito-core instead of mockito-all (discontinued in Mockito 2.x)
Updated nifi-site-to-site-reporting-task tests to be compatible with Mockito 2.x
Ignored TestPutJMS tests; the tests need to be refactored to work with Mockito 2.x, but the processor is deprecated. Refactor may be done in a separate PR.
Adjusted several mock interaction iterations to 0 for TestPublishKafkaRecord_* tests. Mockito 2.x flagged several interactions as unused and were adjusted to 0 interactions.
Updated PowerMock and Mockito dependencies to exclude transitive dependency on bytebuddy, added explicit dependency on bytebuddy 1.9.10 so that PowerMock and Mockito use the same version. Bytebuddy 1.9.3 (used by PowerMock 2.0.2) did not allow for the mocking of final/private classes, bytebuddy 1.9.10 (used by Mockito 2.28.2) does.
Updated TestSiteToSiteProvenanceReportingTask use of InvocationOnMock.getArgument to use objects for the resulting object rather than primitives
Removed unnecessary stubs from evtx tests, Mockito 2.x defaults to strict mocks
Fixed classloader issue with tests in nifi-windows-event-log-processors module that use JNAJUnitRunner when Mockito mocked JNA classes (Kernel32)
Addressed Mockito-related deprecation warnings
Import cleanup
This closes#3533
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
NIFI-6362 - Upgraded both checkstyle plugins to latest. Added a checkstyleCache file and property.
NIFI-6362 - Downgraded maven-checkstyle-plugin to 3.0.0 instead as there was an issue with the latest version and @throws Javadoc checkstyle module. Fixed some checkstyle issues. Changed checkstyleCache file to .checkstyleCache. Added to .gitignore.
NIFI-6362 - Changed checkstyleCache file location into target/, and is no longer a hidden file. Removed .checkstyleCache from .gitignore as it is no longer required.
NIFI-6362 - Grep pattern no longer required.
This closes#3522.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-6315 Renaming public ports during import/change-version when name alreadye exists, and ignorning name change from flow diffs
NIFI-6315 Ensuring public ports get renamed during copy/paste
NIFI-6315 Set max concurrent tasks on Port based on value from VersionedPort
This closes#3488.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-6323 Changed URLs for splunk.artifactoryonline.com to use HTTPS (certificate validity warning in browsers, but command-line connection using openssl s_client is successful).
NIFI-6323 Changed URLs for XMLNS schema locations to use HTTPS (the XMLNS and schema identifier remain http:// because they are not designed to be resolvable).
NIFI-6323 Fixed Maven XML schema descriptor URLs.
This closes#3497
NIFI-6280 - Updated terminology in JwtAuthenticationFilter to authentication instead of authorization. Added stricter token parsing using an explicit regex pattern. Added tests.
NIFI-6280 - Updated terminology from Authorization to Authentication.
NIFI-6280 - Updated the access logout method to use getNiFiUserIdentity(). Updated javascript logout method to handle errors.
NIFI-6280 - Fixing checkstyle issues.
NIFI-6280 - Added some javadoc comments and logging. Renamed some variables for clarity. Fixed handling of exception when JWT does not match expected format.
NIFI-6280 - Cleaned up checkstyle, increased log severity level for logout action, and cleaned up Groovy syntax in test.
This closes#3482.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Specify remote access at port creation.
Incorporated comments, and finished refactoring.
Renamed RootGroupPort to PublicPort.
Fix error message for creating a connection from a child PG having only PublicPorts.
Enhanced ProcessGroup instances rendered in the parent ProcessGroup
Loosen Port move check, allow moving public port between PG.
Show 'Remote NiFi Instance' info on Connection dialogs
Make labels narrative.
'Within Remote Group'.
Fixed DTO (de)serialization.
Return null only if all values are null.
This closes#3351.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Updated NOTICE with current copyright year for Jetty dependencies
Updated Jetty SSLContextFactory usage, invoke setEndpointIdentificationAlgorithm(null) on server SslContextFactory instances
Updated TestInvokeHttpSSL to provide a separate client keystore, rather than reusing the server's keystore
Regenerated nifi-standard-processors keystore and truststore, added client keystore
Updated ITestHandleHttpRequest, TestInvokeHttpSSL, TestInvokeHttpTwoWaySSL, and TestListenHTTP to use a separate client keystore instead of reusing the server's keystore. Also updated the tests to separately test one-way and two-way SSL
NIFI-6196 - Setting the endpoint identification algorithm to null for sockets to ensure certificates without SANs applied correctly still work. We can change this in a new NiFi release with other potentially breaking changes.
This closes#3426.
NIFI-6085 - Updated logOut method to use NiFiUserUtils and updated tests.
NIFI-6085 - Added some more integration tests.
NIFI-6085 Suppressed stacktrace when token is used after being invalidated.
This closes#3362.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
What this PR changed:
- Standardized required name error message to 'The name of the xxxx must
be specified'
- Added nfErrorHandler.handleConfigurationUpdateAjaxError
- Replaced existing custom error handling for 400 status code
- Replaced handleAjxError with this where appropriate
- Standardized config error dialog title to 'Configuration Error'
- Close the configuration dialog when configuration is successfully
updated
Above convention applied to following components:
- ProcessGroup
- Input/OutputPort
- RemoteProcessGroup
- RemoteGroupPort
- Template
- Connection
- ControllerService
- Label
- Processor
- ReportingTask
- NiFi Settings/GENERAL tab
- Variable
- User
NIFI-6170 Provide consistent UX at Connection config
Fixed Label.
Incorporated review comments
Removed unnecessary else block.
Disable canvas when the message pane is shown
This closes#3401
- Before this fix, NiFi loses information about created versioned flow in case of subsequent snapshot creation failure, and NiFi API returned an error response
- This commit makes:
- The created versioned Flow information is stored even if subsequent snapshot creation fails
- NiFi API to return a successful 200 response in that case, but return versioned flow status as SYNC_FAILURE with an explanation. NiFi UI shows a popup error dialog with the explanation.
- Versioned flow status will be LOCALLY_MODIFIED if the latest version is 0.
This closes#3134.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Allowing user or group existence enforcement to be parameterized.
- Fixing error handling when loading user groups which may have resulted in stack trace leaking.
This closes#3377.
Signed-off-by: Kevin Doran <kdoran@apache.org>
- Fixed proxy header support to use X-Forwarded-Host instead of X-ForwardedServer
- Added support for the context path header used by Traefik when proxying a service (X-Forwarded-Prefix)
- Added tests to ApplicationResourceTest for X-Forwarded-Context and X-Forwarded-Prefix
- Updated administration doc to include X-Forwarded-Prefix
- Added NIFI_WEB_PROXY_CONTEXT_PATH env var to dockerhub and dockermaven start.sh scripts
- Added documentation for NIFI_WEB_PROXY_CONTEXT_PATH to dockerhub README.md
- Updated ApplicationResource to handle a port specified in X-ProxyPort and X-Forwarded-Port headers
This closes#3129.
Signed-off-by: Kevin Doran <kdoran@apache.org>
- Set up NarAutoLoader to watch directory for new files
- Move NarAutoLoader to JettyServer since it will need access to ExtensionManager
- Created NarLoader to shared between NarAutoLoader and the framework
- Created nifi-framework-nar-loading-utils so we can use nifi-documentation to call DocGenerator
- Add additional bundles to overall map in NarClassLoaders as they are loaded
- Added handling of skipped NARs to include them in next iteration
- Added check of last modified timestamp on NARs
- Refactored JettyServer so we can load additional web contexts while the application is running
- Setting up unit tests
- Remove static use of ExtensionManager
- Adding unit tests for NarLoader
- Extracting interface for ExtensionManager and splitting discovery into it's own interface
This closes#3119.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Removing needClientAuth property since cluster comms now requires two way ssl. Jetty client auth settings are based on configured features.
- Removing dead code.
- Updating documentation.
- Removing references to needClientAuth property in all test resources.
- Removing overloaded util method with strict parameter.
This closes#3102.
Added Spock test for NonLocalPartitionPartitioner
Updated NOTICE files for FontAwesome with the updated version (4.7.0) and URL to the free license
Updated package-lock.json with the updated version of FontAwesome (4.7.0)
Added method to FlowFileQueue interface to reset an offloaded queue
Queues that are now immediately have the offloaded status reset once offloading finishes
SocketLoadBalancedFlowFileQueue now ignores back-pressure when offloading flowfiles
Cleaned up javascript in nf-cluster-table.js when creating markup for the node operation icons
Fixed incorrect handling of a heartbeat from an offloaded node. Heartbeats from offloading or offloaded nodes will now be reported as an event, the heartbeat will be removed and ignored.
Added unit tests and integration tests to cover offloading nodes
Updated Cluster integration test class with accessor for the current cluster coordinator
Updated Node integration test class's custom NiFiProperties implementation to return the load balancing port and a method to assert an offloaded node
Added exclusion to top-level pom for ITSpec.class
Updated NodeClusterCoordinator to allow idempotent requests to offload a cluster
Added capability to connect/delete/disconnect/offload a node from the cluster to the Toolkit CLI
Added capability to get the status of nodes from the cluster to the Toolkit CLI
Upgraded FontAwesome to 4.7.0 (from 4.6.1)
Added icon "fa-upload" for offloading nodes in the cluster table UI
Explicitly allow methods GET, HEAD. These are the Spring defaults when the allowedMethods is empty but now it is explicit. This will require other methods like POST etc to be from the same origin (for the template/upload URL).
This closes#3024.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Refactoring StandardFlowFileQueue to have an AbstractFlowFileQueue
Refactored more into AbstractFlowFileQueue
Added documentation, cleaned up code some
Refactored FlowFileQueue so that there is SwappablePriorityQueue
Several unit tests written
Added REST API Endpoint to allow PUT to update connection to use load balancing or not. When enabling load balancing, though, I saw the queue size go from 9 to 18. Then was only able to process 9 FlowFiles.
Bug fixes
Code refactoring
Added integration tests, bug fixes
Refactored clients to use NIO
Bug fixes. Appears to finally be working with NIO Client!!!!!
NIFI-5516: Refactored some code from NioAsyncLoadBalanceClient to LoadBalanceSession
Bug fixes and allowed load balancing socket connections to be reused
Implemented ability to compress Nothing, Attributes, or Content + Attributes when performing load-balancing
Added flag to ConnectionDTO to indicate Load Balance Status
Updated Diagnostics DTO for connections
Store state about cluster topology in NodeClusterCoordinator so that the state is known upon restart
Code cleanup
Fixed checkstyle and unit tests
NIFI-5516: Updating logic for Cluster Node Firewall so that the node's identity comes from its certificate, not from whatever it says it is.
NIFI-5516: FIxed missing License headers
NIFI-5516: Some minor code cleanup
NIFI-5516: Adddressed review feedback; Bug fixes; some code cleanup. Changed dependency on nifi-registry from SNAPSHOT to official 0.3.0 release
NIFI-5516: Take backpressure configuration into account
NIFI-5516: Fixed ConnectionDiagnosticsSnapshot to include node identifier
NIFI-5516: Addressed review feedback
This closes#2947
This closes#3044
- Revert 87cf474e54 to enable connection
pooling
- Changes the expected HTTP status code for the 1st request of a
two-phase commit transaction from 150 (NiFi custom) to 202 Accepted
- Corrected RevisionManager Javadoc about revision varidation protocol
Refactored some test code to be clearer.
Renamed some resources to be consistent across modules.
Changed passwords to meet new minimum length requirements.
This closes#3018
The operation policy allows that a user to operate components even if they does not have direct READ/WRITE
permission of the component.
Following operations are controlled by the new operate policy:
- Start/stop/enable/disable Processors, ControllerServices,
ReportingTasks, Input/OuputPorts
- Enable/disable transmission of RemoteInput/OutputPorts and
RemoteProcessGroups
- Terminate Processor threads
Refactored what API exposes
The previous commit let API exposes few fields in DTO. But we should
avoid returning partial DTO as it complicates authorization logic.
Instead, this commit adds StatusDTO for ReportingTaskEntity and
ControllerServiceEntity, so that it can be returned regardless of having
READ permission. Component DTO can only be returned with a READ
permission.
Refactor RPG same as ControllerService.
WIP incorporating review comments.
Incorporated review comments
- Cleaned up merger classes
- Recreate DTO instance at each function during two phase commmit
Restrict enabling ControllerService without read permission
Revert the last commit.
Fix review comments.
- Renamed confusing static method names and its parameters
- Removed unnecessary permission checks from UI condition
Fixed delete action display condition.
Fixed NPE at Summary.
Apply operation policy to activateControllerServices.
Removed OperationPermissible from ComponentEntity.
This closes#2990
NIFI-5442 Populate request contextPath attribute during AccessResource before displaying on message-page.jsp.
Refactored shared code from CatchAllFilter to WebUtils.
NIFI-5442 Refactored filter and context path code to shared parent filter and subclass.
NIFI-5442 Removed unnecessary initParams from nifi-web-ui web.xml.
NIFI-5442 Added explicit dispatchers to nifi-web-ui web.xml and removed unnecessary code from AccessResource.
This closes#2908
Added integration test for checking the ExceptionFilter catches malicious string exceptions.
Made minor changes to PR 2840 for code style.
This closes#2840.
Co-authored-by: Andy LoPresto <alopresto@apache.org>
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Minor adjustments following PR.
- Avoiding additional find operation when authorizing components when populating component details.
- Requiring access to provenance events when downloading content or submitting a replay as they may provide events details.
- Updating the REST API docs detailing the required permissions.
- Updating the wording in the documentation regarding the provenance and data policies.
- Removed the event attributes from the authorization calls that were verifying access to provenance events.
- Only checking content availability when the user is authorized for the components data.
- Addressing typo in JavaDoc.
This closes#2703
- Ensuring the proxy headers are considered when redirecting the user following a OIDC or Knox login exchange.
This closes#2763.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Fixing missing message when a node is disconnected from a cluster.
- Updating endpoints to accept a flag to allow for changes to be made to a disconnected node.
- Updating custom UIs to acknowledge disconnected nodes prior to performing modifications.
- Avoid triggering async validation for each update to component when instantiating a template (such as copy/paste or templates). Added debug logging to indicate when and why we are triggering validation; removed unit test that made poor assumptions about the inner workings of the FlowSynchronizer that resulted in failures when we make calls into processors that the unit test doesn't know about"
This closes#2731.
- PR Fix - 'Execution' dropdown will now be shown in all cases
- Annotated ListGCSBucket with PrimaryNodeOnly
This closes#2509.
Signed-off-by: Mark Payne <markap14@hotmail.com>
NIFI-950: Still seeing some slow response times when instantiating a large template in cluster mode so making some minor tweaks based on the results of CPU profiling
NIFI-5112: Refactored FlowSerializer so that it creates the desired intermediate data model that can be serialized, separate from serializing. This allows us to hold the FlowController's Read Lock only while creating the data model, not while actually serializing the data. Configured Jersey Client in ThreadPoolRequestReplicator not to look for features using the Service Loader for every request. Updated Template object to hold a DOM Node that represents the template contents instead of having to serialize the DTO, then parse the serialized form as a DOM object each time that it needs to be serialized.
NIFI-5112: Change ThreadPoolRequestReplicator to use OkHttp client instead of Jersey Client
NIFI-5111: Ensure that if a node is no longer cluster coordinator, that it clears any stale heartbeats.
NIFI-5110: Notify StandardProcessScheduler when a component is removed so that it will clean up any resource related to component lifecycle.
NIFI-950: Avoid gathering the Status objects for entire flow when we don't need them; removed unnecessary code
NIFI-950: Bug fixes
NIFI-950: Bug fix; added validation status to ProcessorDTO, ControllerServiceDTO, ReportingTaskDTO; updated DebugFlow to allow for pause time to be set in the customValidate method for testing functionality
NIFI-950: Addressing test failures
NIFI-950: Bug fixes
NIFI-950: Addressing review feedback
NIFI-950: Fixed validation logic in mock framework
This closes#2693
- Adding UI controls for terminating hung threads.
- Showing current number of terminated threads.
- Fixing issue when replicating terminate threads request throughout the cluster.
This closes#2607.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Allowing the enable/disable buttons to be active under the same conditions as the start/stop buttons.
This closes#2633.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Fixing process group audit advice.
- Setting spring security user in background threads.
- Removing unnecessary overloaded methods.
This closes#2626.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- take into account input requirement for documentation rendering
- Renamed variable registry scope and added comments
- Doc + change in mock framework to check scope + update of components + UI
- Processing properties and property descriptors in Controller Service referencing components unconditionally.
This closes#2602.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Ensuring appropriate response in checkAuthorization when user is null.
- Ensuring the user reference is passed down when applying variable changes.
This closes#2598.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Fixing required permission for PutParquet.
NIFI-5008:
- Ensuring all restricted components are tagged as such.
This closes#2583.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Fixing RPG port merging.
- Adding unit tests.
- Removing unecessary sorting that wasn't maintained while clustered.
This closes#2551.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Implemented review feedback. Refactored data model to make the API cleaner and delineate more along the lines of what permissions are required in order to see which details
Implementing review feedback
Removed sensitive information from the diagnostics reports
Fixed bug in merging logic for GCDiagnosticsSnapshots
This closes#2468
...Search results
* Separated the search functionality.
* Added a unit test.
* Added the PG info to UI (a mere draft).
* Introduce the nearest versioned group
* Removed the top level group results in favour of the nearest versioned group.
* This closes#2364
- Everywhere that we ignore adding remote ports we should ignore removing remote ports as well in flow diffs
This closes#2462.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-4436: Fixed a bug that caused a flow not to successfully change version if a connection is added to an existing component and that component is running at time of version change
NIFI-4436: Fixed bug with ordering of controller services being enabled and disabled
NIFI-4436: Fixed bug that prevented local input and output ports from being stopped and started as needed
NIFI-4436: Fixed bugs around referencing controller services that are at a higher level than the versioned flow
NIFI-4436: Ensure that we clear components from FlowController's cache when removed and that they are added to cache when created.
NIFI-4436: Fixed error message coming back if component is invalid when trying to be restarted/re-enabled
NIFI-4436: Addressed issue with children of a removed process group not being considered 'affected components' and as a result not being stopped/disabled/restarted/re-enabled
This closes#2219.
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
- Minor tweak to when the max height for the options list is calculated per PR comments.
- Resolving logic issue in two phase commit when updating variable registry.
- Fixing variable visibility.
- Updating buckets permissions based on new model.
- Adding check to ensure that flow name is non null before checking the length.
- Adding versioned flow state to the Process Group tab in the Summary table.
- Fixing issue with navigating to Controller Services from the local changes dialog.
- Code clean up.
- Improved error handling.
- Minor UX improvements.
- Always showing Process Group state to complement the aggregation counts.
- Adding the Process Group state to the top status bar.
- Code clean up.
- Improved error handling.
- Minor UX improvements.
- Adding message to indicate that variables do not support sensitive values.
- Preventing a user from changing the flow version to the current version.
- Only presenting buckets a user has appropriate permissions to.
- Adding basic auditing to the version control actions.
- Updating front end to use version control state/status.
- Fixing copy/paste issue during revert local changes.
- Code clean up in the breadcrumbs.
- Update VersionsResource authorization and two phase commit object usage.
- Addressing miscellaneous minor UX issues.
- Updating comments UX for all components.
- Updating the styling of PG and RPG to be more consistent.
- Adding the icons for nested versioned process groups.
- Calculating the number/states of nested versioned process groups.
- Clearing bucket/flow/versions when changing the selected registry/bucket.
- Using the versioned flow to get the group name when importing.
- Adding menu items for viewing local changes.
- Showing local changes during revert request.
- Adding the version number to the start version control, commit, and change version dialog.
- Showing a loading item in the combo's while querying for the registries, buckets, and flows.
- Adding tooltips to display version control information on the canvas.
- Adding progress bar dialogs for changing version and reverting local changes.
- Updating canvas and breadcrumb according to the version control state.
- Updating to use registry name, bucket name, and flow name where appropriate.
- Added the import dialog for importing a versioned flow into a new process group.
- Added the change version dialog for upgrading/downgrading a versioned flow.
- Adding support to save a version of a flow based on a selected Process Group.
- Adding support for revert changes back to the most recent version.
- Adding support to disconnect from version control.
- Moving the version control information out of the entity objects and into the dto's.
- Fixing checkstyle issues.
NIFI-4502:
- Updating the UI to allow for the user to register registry clients.
- Updating the version control menu item names.
- Initial checkpoint: able ot start version control and detect changes, in standalone mode, still 'crude' implementation
- Checkpoint: Can place flow under version control and can determine if modified
- Checkpoint: Change version working in some cases. Does not work if processor removed because COMPONENT_REMOVED type has ComponentA whose ID is the VersionedComponentID but we are trying to call ProcessorDAO.get() with this ID
- Checkpoint: Able to change flow from Version 1 to Version 2 and back. Not yet tested with controller services. Have not tried changing/removing connections. Not cluster-friendly yet. All inline, not in background. Have not taken into account ports, funnels, remote ports, etc. Have not tested with Labels yet
- Checkpoint after implementing ClusterReplicationComponentLifecycle instead of JerseyClientComponentLifecycle
- Checkpoint: Updated to allow starting version control and updating version in clustered mode
- Checkpoint: Updated versioning endpoint so that when version of a flow is updated, the bundle information is populated and the snapshot is replicated to the cluster.
- Checkpoint: Implemented endpoint for reverting to previously sync'ed version of a flow and updated version control endpoint so that Process Group can be pushed as a new version to existing flow instead of only creating a new flow
- Checkpoint: Updated so that if a Process Group is under Version Control and it has a child Process Group, which is also under Version Control, we can handle that gracefully. Not yet tested because it depends on updates to the nifi-registry module, which can't be compiled due to maven dependency conflicts
- Ensure the /nifi-api/controller redirection filter executes before matching.
This closes # 2358.
Signed-off-by: Andy LoPresto <alopresto.apache@gmail.com>
- Upgrading to Jersey 2.x.
- Updating NOTICE files where necessary.
- Fixing checkstyle issues.
This closes#2206.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Adding support for KnoxSSO.
- Updated the docs for nifi.security.user.knox.audiences.
- The KnoxSSO cookie is removed prior to request replication.
This closes#2177
- Added XmlUtils class.
- Added unit test.
- Added XXE test resource.
- Refactored JAXB unmarshalling globally to prevent XXE attacks.
- Refactored duplicated/legacy code.
- Cleaned up commented code.
- Switched from FileInputStream back to StreamSource in AuthorizerFactoryBean.
- This closes#2134
- Initial implementation of Process Group level Variable Registry
- Updated to incorporate PR Feedback
- Changed log message because slf4j-simple apparently has a memory leak; passing a String instead of passing in the Controller Service object as an argument addresses this.
- This closes#2051
- Introducing support for OpenId Connect.
- Updating REST API and UI to support the authorization code flow.
- Adding/fixing documentation.
- Implementing time constant equality checks where appropriate.
- Corrected error handling during startup and throughout the OIDC login sequence.
- Redacting the token values from the user log.
- Defaulting to RS256 when not preferred algorithm is specified.
- Marking the OIDC endpoints as non-guaranteed in to allow for minor adjustments if/when additional SSO techniques are introduced.
This closes#2047.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
NIFI-4032: - Generating the appropriate fingerprint for the ManagedRangerAuthorizer based on whether the UserGroupProvider is configurable. - Adding unit tests.
Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>
This closes#2019
- Expose processors' counters in Stats History
- Only include counters in Processors' Status History if user has read access to corresponding Processor
- Addressed review feedback. Found and addressed bug where a counter is not present in all of the aggregate snaphot values for status history, resulting in the UI not rendering the chart properly
- This closes#1872
- Introducing composite ConfigurableUserGroupProvider and UserGroupProvider.
- Adding appropriate unit tests.
- Updating object model to support per resource (user/group/policy) configuration.
- Updating UI to support per resource (user/group/policy) configuration.
- Adding necessary documentation.
- Updating documentation to clarify integrity checks.
- Providing an example of configuring a composite implementation.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#1978.
- Addressing issues causing the eventId to not be relayed when submitting a lineage request under certain conditions.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#1903.
- Introducing FileUserGroupProvider and FileAccessPolicyProvider.
- Refactoring FileAuthorizer to utilize the file based implementations.
- Introducing the StandardManagedAuthorizer.
- Decorating the configured ManagedAuthorizer to ensure integrity checks are still performed.
- Loading user groups if possible to use during access decisions.
- Merging responses for requests for AccessPolicies, Users, and UserGroups.
- Adding unit tests as appropriate.
- Adding methods to the User, Group, and AccessPolicy builder that more easily supports generating UUIDs.
- Fixing typo when seeding policies during startup.
- Fixing type in documentation and error messages.
This closes#1897.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Filtering out the EventTime searchable field for all implementation of ProvenanceRepository. Prefer user query using the startDate/endDate fields.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#1844.
Andy LoPresto
MatthewKnight-NG
Joe Ferner
Joe Witt
Andy LoPresto
Andrew Lim
Mark Payne
Matt Gilman
Rob Fellows
Koji Kawamura
Andy I. Christianson
Arpad Boda
Bryan Bende
Scott Aslan
Jeff Storck
Peter Wicks
thenatog
Jan Hentschel
Kevin Doran
Kemix Koo
Damian Czaja
Mark Bean
zenfenan
Pierre Villard
Mike Moser
Takanobu Asanuma
yuri1969
sbouchex
combine
m-hogue