- Added TemporaryKeyStoreBuilder with File.deleteOnExit() for KeyStore and TrustStore files
- Removed JKS files from nifi-security-utils tests
- Refactored usage of KeyStoreUtils.createKeyStoreAndGetX509Certificate() to TemporaryKeyStoreBuilder
- Removed unnecesary hadoop-minikdc test dependency in security-utils
- Replaced Mini KDC Hex utility with Bouncy Castle Hex utility in unit tests
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5406
- Refactored multiple tests using KeyStoreUtils
- Removed static KeyStore and TrustStore files
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5401
NIFI-9170 Add two more 1.9.4 references to close out the few things identified by the Maven dependency plugin.
This closes#5351
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
- Replaced old com.sun.xml.bind:jaxb-impl and jaxb-core with current org.glassfish.jaxb:jaxb-runtime
- Replaced old javax.xml.bind:jaxb-api with current jakarta.xml.bind-api
- Removed unnecessary dependency references to javax.activation-api
This closes#5320
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
- Added JavaScript Authorization Storage component for storing and retrieving JSON Web Tokens
- Added access status request to remove Session Cookie when Token not found
NIFI-9049 Updated Jolt JavaScript application to use AuthorizationStorage
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5344.
- Replaced per-user symmetric-key HS256 with shared and rotated RSA asymmetric-key RS512 implementation
- Added nifi.security.user.jws.key.rotation.period property for RSA Key Pair rotation
- Added JSON Web Tokens section to Administration Guide
- Implemented persistent storage of RSA Public Keys for verification using Local State Manager
- Implemented JWT revocation on logout with persistence using Local State Manager
- Refactored JWT implementation using Spring Security OAuth2 and Nimbus JWT
- Refactored Spring Security Provider configuration using Java instead of XML
- Removed H2 storage of per-user keys
- Upgraded nimbus-jose-jwt from 7.9 to 9.11.2
NIFI-8766 Corrected AuthenticationException handling in AccessResource.getAccessStatus
- Added nifi.user.security.jws.key.rotation.period to default nifi.properties
- Updated logging statements and clarified configuration and method documentation
NIFI-8766 Changed Algorithm to PS512 and updated documentation
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5262.
NIFI-8671 Moved versioned components class into nifi-api
- Removed @XmlRootElement from VersionedProcessGroup.
- Fixed nifi-api dependency version in nifi-registry-data-model. Changed logic of handling instances of un-annotated classes during xml serialization in JAXBSerializer.
* NIFI-8939: Ensure that when async/long-running flow updates are made, referencing controller services that are disabling are waited on but not attempted to be disabled
* NIFI-8939: Ensure that when waiting for Controller Services to reach desired state, we use correct URI for fetch service state. There was a typo that resulted in not getting all controller services' states.
This closes#5240
- Remove reference to ongoing work for Java 11
- Remove references to Bower which is no longer used as of NIFI-2781
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5232
- Added Jetty DoSFilter configured for /access/token
- Added nifi.web.max.access.token.requests.per.second property with default value of 25
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5215.
- include new process group property support in NiFi Registry
- updated documentation to describe and show new feature
- added elements to XSD schema definition
NIFI-8195: update to DAO to fix PG move and copy/paste
update condition to not null vice null
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5192
- Upgraded Angular Material from 1.1.10 to 1.1.26
- Upgraded Moment from 2.24.0 to 2.29.1
- Upgraded JSON Lint from 1.6.2 to 1.6.3
- Upgraded Slickgrid from 2.4.27 to 2.4.38
- Upgraded frontend-maven-plugin from 1.4 to 1.12.0
- Upgraded frontend-maven-plugin NodeJS from 12.7.0 to 12.22.2
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5197.
NIFIDEVS-8195: fixed properties not properly inheriting from template/snippet values
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5094
- Upgraded Spring Framework references from version 4.3.30 to 5.3.6
- Upgraded Spring Security from version 4.2.20 to 5.4.6
- Upgraded Spring Data Redis from 2.1.16 to 2.5.0
- Upgraded Jedis from 2.9.0 to 3.6.0 to match Spring Data Redis 2.5.0
- Upgraded Easy Rules from 3.4.0 to 4.1.0 to support Spring 5
- Upgraded Hortonworks Schema Registry Client from 0.8.1 to 0.9.1 to support Spring 5
- Refactored ThreadPoolRequestReplicatorFactoryBean to implement DisposableBean to handle executor shutdown
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5066.
When selecting run/stop on a process group/canvas/selection, it will try to enable/disable transmission of all involved remote process groups.
NIFI-7788 Supplied same functionality missed when selecting a process group.
NIFI-7788 Updated endpoint URL paths.
NIFI-7788 No need to return list of remote process groups when updating en masse.
NIFI-7788 Added some null checks in RemoteProcessGroupsEndpointMerger.merge.
NIFI-7788 Fix checkstyle violation.
This closes#4516.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
- Updated components to make use of new feature
NIFI-8206: Added a ResourceType of TEXT. This requires that the ResourceReferenceFactory know which types are allowed in order to create the ResourceReference. PropertyValue needs to then have the PropertyDescriptor available to it. This resulted in highlighting many bugs in unit tests where components were not exposing property descriptors via getSupportedPropertyDescriptors() or were evaluating Expression Language using the wrong scope, so fixed many unit tests/components to properly declare Expression Language scope when using it
NIFI-8206: Removed problematic unit test that required directory names with special characters that are not allowed on some operating systems
This closes#4890.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Added nifi.web.request.ip.whitelist property to set DoSFilter.ipWhitelist
- Added nifi.web.request.timeout property to set DoSFilter.maxRequestMs with default of 60 seconds
This closes#4972
Signed-off-by: David Handermann <exceptionfactory@apache.org>
* [NIFI-8387] - Use the bulletins from the referencing components rather than making extraneous http calls to get them.
* Moving the spinner next to the Parameter/Variable lables
This closes#4969
NIFI-8386: Addressed review feedback: removed unused call to determine permissions, null out bulletins in standalone mode if permissions not allowed. Also fixed automated tests that were failing due to changes
This closes#4955
Instead, it will look at the ServiceLoader file and read the names of the classes but avoid instantiating all of the objects or loading the classes into memory.
- Updated Doc Generation so that if the documentation for a given NAR already exists, it doesn't delete it and re-generate it. This was necessary because we are no longer instantiating an instance of each component and instead lazily creating the components as necessary.
- Removed stateless version of extension registry because it's no longer necessary
This closes#4852
Signed-off-by: David Handermann <exceptionfactory@apache.org>
NIFI-8260 [WIP] Fixed server side logic to upload a flow file. Cleaned up the front end logic.
NIFI-8260 [WIP] Finished the server side upload logic.
Added a client ID parameter to the endpoint.
Added JSON parsing error response.
Fixed the client side file form to reset after submit.
Fixed the canvas to instantly update and show the process group after submitting the file.
Changed the Add Processor Group dialog UI based on design notes.
Changed the Upload File link to an icon and moved to the process group name input.
Changed the Registry Import link to say 'Import from Registry' and moved to the bottom of the dialog.
Display the filename when a file is selected.
NIFI-8260 [WIP] Added a cancel file button to the Process Group dialog.
Fixed some CSS styles.
NIFI-8260 - Removed accessing the snapshot metadata to avoid an NPE.
Added a title attribute to the html of the dialog file cancel button.
NIFI-8260 - Disabled the dialog 'Add' button.
Revised based on PR feedback.
Refactored the upload file endpoint and client side filename extraction methods.
Fixed some CSS.
Reverted some unnecessary changes.
NIFI-8260 - Revised based on PR feedback.
Refactored uploadProcessGroup.
Fixed some exception handling.
Hid the Upload File button when grouping components.
Refactored nf-ng-group-component.js replacing jquery selectors with variables.
Extracted the resetValues function to clear dialog values.
NIFI-8260 - Fixed the cluster replicate request.
Created a new endpoint to handle the cluster replicate request.
Created ProcessGroupUploadEntity.
Renamed positionX and positionY parameters.
NIFI-8260 - Fixed a checkstyle error.
Removed unnecessary httpServletRequest parameter.
Reverted some re-ordering of imports.
NIFI-8260 - Changed the ProcessGroupUploadEntity to be consistent with other Entity and DTO models.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4846.
Changed to check the length of all unfiltered properties instead of only filtered properties.
Added additional check if descriptor is a dynamic property.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4707
- Refactored nifi-stateless to make use of nifi-framework-components
- Removed requirement for nifi-framework-nar to be provided.
- Refactored stateless nifi into api, engine, nar, and bootstrap modules, with a parent 'bundle' module
- Creation of nifi-stateless-system-tests
- Added unit tests and logging
- Changed flow configuration to use properties file instead of json
- Allow for -p parameter to specify parameters on command line
- Moved implementations of Authorizer, NiFiUser, and UserGroupProviders to new module named nifi-framework-authorization-providers so that those that depend on nifi-framework-authorization don't have to bring in the providers. This way, we can have stateless not bring in those providers, as we otherwise get warnings on startup about the provider already being registered. Additionally, it avoids needing dependencies on spring-security-core
- Updated bin/nifi.sh script to run new stateless bootstrap
- Added Reporting Tasks to stateless.
- Download bundles as necessary on stateless nifi startup
NIFI-7897: Addressing review feedback
NIFI-7897: Fixed typos in README and also addressed issue that caused parameters with spaces not to be parsed properly
This closes#4669.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Add dependency on spring-security-saml2-core
- Updated AccessResource with new SAML end-points
- Updated Login/Logout filters to handle SAML scenario
- Updated logout process to track a logout request using a cookie
- Added database storage for cached SAML credential and user groups
- Updated proxied requests when clustered to send IDP groups in a header
- Updated X509 filter to process the IDP groups from the header if present
- Updated admin guide
- Fixed logout action on error page
- Updated UserGroupProvider with a default method for getGroupByName
- Updated StandardManagedAuthorizer to combine groups from request with groups from lookup
- Updated UserGroupProvider implementations with more efficient impl of getGroupByName
- Added/updated unit tests
- Ensure signing algorithm is applied to all signatures and not just metadata signatures
- Added property to specify signature digest algorithm
- Added option to specify whether JDK truststore or NiFi's truststore should be used when connecting to IDP over https
- Added properties to configure connect and read timeouts for http client
- Added URL encoding of issuer when generating JWT to prevent potential issue with the frontend performing base64 decoding
- Made atomic replace methods for storing groups and saml credential in database
- Added properties to control AuthnRequestsSigned and WantAssertionsSigned in the generated service provider metadata
- Dynamically determine the private key alias from the keystore and remove the property for specifying the signing key alias
- Fixed unit test
- Added property to specify an optional identity attribute which would be used instead of NameID
- Cleaned up logging
- Fallback to keystore password when key password is blank
- Make signature and digest default to SHA-256 when no value provided in nifi.properties
This closes#4614
- Added a 'dependent' attribute to determine whether or not to save dependent property values
Co-authored-by: Scott Aslan <scottyaslan@gmail.com>
Signed-off-by: Bryan Bende <bbende@apache.org>
Bumped icu4j dependency to 60.2.
Replaced jackson-mapper-asl dependency with jackson-databind.
Fixed an error comparing key identities in TestKeyService.
Replaced jackson-mapper-asl ObjectMapper with jackson-databind ObjectMapper in LivySessionController.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4640.
Added method to validate the OIDC Access Token for the revoke endpoint.
Created a new callback URI of oidc/logoutCallback to handle certain OIDC logout cases.
Changed method to exchange the Authorization Code for a Login Authentication Token.
Added a new method to exchange the AuthN Code for an Access Token.
Changed method to convert OIDC Token to a Login AuthN Token instead of a NiFi JWT.
Created new OidcServiceGroovyTest class.
NIFI-7584-rebase Added test.
NIFI-7584 Fixed a checkstyle issue.
NIFI-7584 Removed a dependency not in use.
NIFI-7584 Made revisions based on PR review.
Refactored revoke endpoint POST request to a private method.
Removed unnecessary dependencies.
Fixed Regex Pattern to search for literal dot character.
Fixed logging the Exception message.
Fixed caught Exception.
Changed timeout value to a static variable.
Changed repeating error messages to a static string.
Reduced sleep duration in unit test.
Refactored cookie generation to private method.
NIFI-7584 Fixed the snapshot version.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4593.
Use an AspectJ aspect and agent to intercept the load native library calls
at runtime, copy the native library file to temp folder and proceed with
the newly created file in order to provide classloader isolation.
Remove AspectJ jars from lib directory, move the necessary jar to lib/aspectj subdirectory.
This closes#4540.
Signed-off-by: Bryan Bende <bbende@apache.org>
* NIFI-7592: Allow NiFi to be started without a GUI/REST interface
* NIFI-7592: Enable all controller services when starting headless
* NIFI-7592: Marked duplicate dependencies as provided
* NIFI-7592: Incorporated additional review comments
* NIFI-7804 Split nifi-security-utils into sub-module for nifi-security-utils-api (no external dependencies).
Separated interface and implementation of TlsConfiguration.
Reabsorbed nifi-security-xml-config into nifi-security-utils.
* NIFI-7804 Resolved failing unit test on Java 8.
Removed accidental module dependency.
* NIFI-7804 Resolved failing unit test.
* NIFI-7804 Removed legacy dependency.
* NIFI-7804 Marked nifi-security-utils-api as provided and overrode with compile scope in specific modules which are not children of nifi-standard-services-api-nar.
Cleaned up JettyServer code.
Changed test logging severity to include debug statements.
Added test resources.
This closes#4498.
Co-authored-by: Kotaro Terada <kotarot@apache.org>
NIFI-7663 Minor changes (variable name refactor, javadoc, GUI message). Merging Drop All Flowfiles responses across all nodes in a cluster.
NIFI-7663 Reloading the canvas after completing a Drop All Flowfiles request.
NIFI-7663 Fixed typos.
This closes#4425.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
Logout now deletes signing key by key ID rather than identity.
Validate token expiration now uses mapped identity instead, which allows logging of the mapped identity.
Updated delete key to expect only 0 or 1 keys deleted.
This closes#4416.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Added exception mapper.
NIFI-7657 Renamed exception & exception mapper to reflect scope of authentication not supported.
Registered exception mapper.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4418.
Added Bundle#toString() method.
Refactored implementation of filter addition logic.
Added logging.
Added unit tests to check for filter enablement.
Introduced content-length exception handling in StandardPublicPort.
Added filter bypass functionality for framework requests in ContentLengthFilter.
Updated property documentation in Admin Guide.
Renamed methods & added Javadoc to clarify purpose of filters in JettyServer.
Cleaned up conditional logic in StandardPublicPort.
Moved ContentLengthFilterTest to correct module.
Refactored unit tests for accuracy and clarity.
Fixed remaining merge conflict due to method renaming.
Signed-off-by: Joe Witt <joe.witt@gmail.com>
Added new StandardOidcIdentityProviderGroovyTest file.
Updated deprecated methods in StandardOidcIdentityProvider. Changed log output to print all available claim names from JWTClaimsSet. Added unit test.
Added comments in getAvailableClaims() method.
Fixed typos in NiFi Docs Admin Guide.
Added license to Groovy test.
Fixed a checkstyle error.
Refactor exchangeAuthorizationCode method.
Added unit tests.
Verified all unit tests added so far are passing.
Refactored code. Added unit tests.
Refactored OIDC provider to decouple constructor & network-dependent initialization.
Added unit tests.
Added unit tests.
Refactored OIDC provider to separately authorize the client. Added unit tests.
Added unit tests.
NIFI-7332 Refactored exchangeAuthorizationCode method to separately retrieve the NiFi JWT.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4344.
- Ensuring the group id is always set in the properties table when loading properties.
- Using a common approach to getting parameters in nfControllerService.
- Code clean up.
- Addressing review feedback.
- Ensuring the service dialog is closed when navigating to the parameter context dialog.
This closes#4322
Cleaned up code style.
Unit test was failing on Windows 1.8 GitHub Actions build but no other environment. Increased artificial delay to avoid timing issues.
Co-authored-by: Andy LoPresto <alopresto@apache.org>
This closes#4271.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Removed list structure for peer selection as it was unnecessary and often wasteful (most clusters are 3 - 7 nodes, the list was always 128 elements).
Changed integer percentages to double to allow for better normalization.
Removed 80% cap on remote peers as it was due to legacy requirements.
Added unit tests for non-deterministic distribution calculations.
Added unit tests for edge cases due to rounding errors, single valid remotes, unbalanced clusters, and peer queue consecutive selection tracking.
Migrated all legacy PeerSelector unit tests to new API.
Removed unused System time manipulation as tests no longer need it.
Added class-level Javadoc to PeerSelector.
Removed S2S details request replication, as the responses were not being merged, which led to incorrect ports being returned and breaking S2S peer retrieval.
Fixed copy/paste error where input ports were being listed as output ports during remote flow refresh.
Fixed comments and added unbalanced cluster test scenarios.
Removed unnecessary marker interface.
Removed commented code.
Changed weighting & penalization behavior.
Changed dependency scope to test.
This closes#4289.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Added FlowFileOutboundPolicy to ProcessGroups and updated LocalPort to make use of it
Persisted FlowFile Concurrency and FlowFile Output Policy to flow.xml.gz and included in flow fingerprint
Added configuration for FlowFile concurrency and outbound policy to UI for configuration of Process Groups
Added system tests. Fixed a couple of bugs that were found
Fixed a couple of typos in the RecordPath guide
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4306.
Changed JettyServer default SSL initialization and updated unit test.
Removed SecurityStoreTypes (unused).
Added StringUtils inverted blank and empty checks.
Added TlsConfiguration container object.
Enhanced KeystoreType enum.
Added clean #createSSLContext() method to serve as base method for special cases/other method signatures.
Added utility methods in KeyStoreUtils.
Added generic TlsException for callers that cannot resolve TLS-specific exceptions.
Added utility methods for component object debugging.
Enforced TLS protocol version on cluster comms socket creation.
Added utility method for SSL server socket creation.
Refactored (Server)SocketConfigurationFactoryBean to store relevant NiFiProperties in TlsConfiguration instead of stateful SSLContextFactory (Cluster comms now enforce modern TLS protocol version).
Removed duplicate SSLContextFactory.
Switched duplicate SslContextFactory to wrap shared SSLContextFactory.
Refactored SslContextFactoryTest for clarity (will move any unique tests to nifi-security-utils class test).
Added further validation & boundary checking in uses of TlsConfiguration.
Provided SSLSocketFactory accessor in SslContextFactory.
Refactored OkHttpReplicationClient tuple method.
Refactored OcspCertificateValidator TLS logic.
Added utility method to apply TLS configs to OkHttpClientBuilder.
Removed references to duplicate SslContextFactory.
Removed unnecessary SslContextFactory.
Moved OkHttpClientUtils to nifi-web-util module.
Updated module dependencies.
Removed now empty nifi-security module.
Enforced TLS protocol selection on LB server socket.
Enforced TLS protocol selection on S2S server socket.
Applied specified TLS protocol versions to S2S socket creation.
Completed removal of legacy SSLContext creation methods from only remaining SslContextFactory.
Replaced references to creation methods throughout codebase.
Replaced references to unnecessary NiFiProperties file reads throughout tests.
Removed duplicate ClientAuth enum from SSLContextService and changed all references to SslContextFactory.ClientAuth.
Suppressed repeated TLS exceptions in cluster, S2S, and load balance socket listeners.
Cleaned up legacy code.
Added external timing check to timing test assertion.
Made RestrictedSSLContextService TLS protocol versions allowable values explicit.
Enabled TLSv1.3 on Java 11.
Added explanations of TLS protocol versions in StandardSSLContextService and StandardRestrictedSSLContextService.
Resolved additional Java 11 test failures for NiFi internal classes that don't support TLSv1.3. Filed NIFI-7468 as follow on task.
This closes#4263.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
Signed-off-by: Mark Payne <markap14@hotmail.com>
Fixed a checkstyle error.
Added property to nifi.properties.
Changed property to a variable that is set with the pom.xml.
Added setting the version variable to another HTTPConfiguration to fix the version being sent in docs context.
Fixed typo error.
This closes#4192.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
* NIFI-7319 Added first draft of walkthroughs doc.
* NIFI-7319 Added instructions and screenshots for securing standalone NiFi instance.
* NIFI-7319 Added instructions and screenshots for instructing OS & browser to trust self-signed certificate.
* NIFI-7319 Added instructions and screenshots for securing NiFi with externally-provided certificates.
* NIFI-7319 Added instructions and screenshots for building NiFi from source.
* NIFI-7319 [WIP] Converting secure cluster instructions to match format.
Fixed instructions regarding embedded ZooKeeper configuration.
* NIFI-7319 Completed secure cluster walkthrough.
* NIFI-7319 Added walkthroughs to documentation navigation list.
* NIFI-7319 Incorporated PR feedback on broken links.
* NIFI-7319 Removed line number helpers from update sections.
* NIFI-7319 Incorporated final PR review items.
Co-authored-by: Sandra Pius <spiusapache@gmail.com>
- Refactored Flow Synchronization to make code cleaner
- Updated Authorizers to forcibly inherit Users, Groups, and Access Policies if the local flow is empty.
- Updated FlowFileRepositories to use SerializedRepositoryRecord instead of RepositoryRecord, so that we have the ability to read records without already knowing the Queue objects. Updated StandardFlowSynchronizer so that if the flow is not inheritable but the controller has not yet been initialized, the flow is backed up and replaced instead of NiFi failing to start
- Added system tests. Updated FlowController so that if it fails to inherit flow due to flow uninheritability that it notifies the cluster of this instead of remaining in the 'CONNECTING' state.
- Added additional log statements to aid in debugging
NIFI-6849: Rebased against master. Updated Admin Guide to describe new cluster flow inheritance behavior
NIFI-6849: Addressed review feedback
NIFI-6849: Addressed review feedback: Relocated logic for bundle compatibility into the BundleCompatibilityCheck class. Fixed logic that prevented users/groups/policies from being forcibly inherited during startup
This closes#3891
Adds DoSFilter to enforce configurable maximum on incoming HTTP requests per second.
Redirected log messages for ContentLengthFilter to nifi-app.log in logback.xml.
This closes#4125.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Removed Cat X JSON.org dep inclusion which seems to not be necessary
- Updated a ton of easier/safer looking deps
- Updated tika due to CVE
This closes#4086
Signed-off-by: Mark Payne <markap14@hotmail.com>
Also dealt with unreliable tests which depend on timing by ignoring them or converting to IT.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4132.
JsonContentConversionExceptionMapper, JsonMappingExceptionMapper, JsonParseExceptionMapper.
Registered the custom ExceptionMappers.
Added unit tests to throw Exception for string port value and sanitize script input. Handled null or empty JsonMappingException reference path.
Added the Apache license to Groovy Test.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
NIFI-6363 Additional fixes.
NIFI-6363 Fix Hadoop compile problem. Add GCP IT instructions.
NIFI-6363 - Removed GCP provider due to dependency conflicts with GRPC processors. Fixed unit test to match master branch after rebase.
NIFI-6363 - Added some docs and experimental tag to the relevant classes.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4080.
- decoupled flow update request behavior from VersionsResource into new abstract FlowUpdateResource
- added replace process group functionality in ProcessGroupResource
- parameterized FlowUpdateResource and created entity hierarchies to allow for maximum code sharing across different update types
- refactored flow update methods to make use of commonality across different update types whenever possible
- fixed issues in StandardProcessGroup verify update methods where same components existed in different ancestry chains but were considered a match when they shouldn't be
- improved StandardProcessGroup to properly match up components on update using generated versioned component ids, when necessary to allow for update flow to efficiently match common components on flow import
This closes#4023.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Added UI versioned flow supportsDownload functionality with download flow menu item
- Added VersionsResource endpoint for downloading versioned flow with registry-related info removed
- Added ProcessGroupResource endpoint for downloading current flow with registry-related info removed
- Added StandardNifiServiceFacade functionality for downloading both current and versioned flow
- Added XmlTransient markers on variables introduced by Instantiated model classes so they do not appear in serialized download
- Updated NiFiRegistryFlowMapper.mapParameterContexts to handle mapping nested parameter contexts for use in producing a complete VersionedFlowSnapshot
- Added ability for NiFiRegistryFlowMapper to map nested process groups ignoring versioning for use in producing a complete VersionedFlowSnapshot
- Added unit tests where helpful
NIFI-6872: PR response...
- Updated mapParameterContext to return a Map to handle uniqueness of contexts by name since ultimately everything converted it to a map anyway. The VersionedParameterContext class from the registry model doesn't support hashcode/equals currently so returning a Set wouldn't work.
- Updated assert calls to put expected value as first parameter and actual as second parameter
- Added one time password (OTP) support for flow download endpoint to support non cert based authentication
This closes#3931
- Fixing issue opening a users policy listing when there is a policy for a specific parameter context.
This closes#3805.
Signed-off-by: Joe Witt <joewitt@apache.org>
- Ensuring policy label is properly escaped when populating the user's access policy listing.
This closes#3804.
Signed-off-by: Joe Witt <joewitt@apache.org>
- Fixing the identifier on the user table. In a previous task, this was changed to utilize the URI but that does not work with other code interacting with this table.
This closes#3798.
- Fixed checkstyle errors.
- Added PeerPersistence interface.
- Expose RemoteProcessGroup state via REST API
- Made stateManager transient.
This closes#3677.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-6589: Updated CuratorLeaderElectionManager to cache results for no more than 5 seconds per review feedback
Signed-off-by: Joe Witt <joewitt@apache.org>
NIFI-6630 - base convert and goto parameter logic on any reference to a parameter contained in the text of the property value.
This closes#3718
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
NIFI-6644 - when deciding to show the convert prop option: enforce user has read and write permissions to the param context & the props are not for a controller service defined in the controller settings.
This closes#3713
* NIFI-6510 Implement initial analytic engine
* NIFI-6510 Implemented basic linear regression model for queue counts
* NIFI-6510 Initial analytics REST endpoint and supporting objects
* NIFI-6510 Connect the dots for StatusAnalytics -> API
* NIFI-6510 Added poc engine with prediction model caching
(cherry picked from commit e013b91)
DFA-9 - updated logging and corrected logic for checking if not in backpressure
(cherry picked from commit a1f8e70)
* NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics
(cherry picked from commit 050e0fc)
(cherry picked from commit 9fd365f)
* NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id
(cherry picked from commit 14854ff)
DFA-9 - reduced snapshot interval to 1 minute
(cherry picked from commit 36abb0a)
* NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions
* NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly
* NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly"
This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1.
* NIFI-6510 Added prediction fields for use by UI, still need to be populated
* NIFI-6510 Analytics Framework Introduction (#10)
* DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model.
DFA-9 - added logging
* DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval
* DFA-9 - checkstyle fixes
* NIFI-6510 Add prediction percent values and predicted interval seconds
(cherry picked from commit e60015d)
* NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not.
(cherry picked from commit 6fae058)
* NIFI-6510 Added tests for engine
(cherry picked from commit 6d7a13b)
* NIFI-6150 Added tests for connection status analytics class, corrected variable names
(cherry picked from commit 58c7c81)
* NIFI-6150 Make checkstyle happy
(cherry picked from commit b6e35ac)
* NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing
* NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics
* NIFI-6510 Adjusted interval and incorporated R-squared check
Updates to support multiple variables for features, clearing cached regression model based on r-squared values
Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support).
Ratcheck fixes
Added test for SimpleRegression. Minor fix for OLS model
fixed test errors
fixed checkstyle errors
(cherry picked from commit fab411b)
* NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11)
* NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties
NIFI-6566 - changes to allow scoring configurations for model in nifi.properties
NIFI-6566 - added default implementation value to NiFiProperties
NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3663
* NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type
* NIFI-6586 - documentation and comments
This closes NIFI-6586
Signed-off-by: Andrew I. Christianson <andy@andyic.org>
* NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI
* Add multi-line tooltips with detail for connection queue back pressure graphics.
* Add estimated time to back pressure to connections summary table.
* Add back pressure prediction ticks.
* add moment.js to format predicted time to back pressure
* tweak summary table headings to match data displayed. re-order connection summary columns
* NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment.
* NIFI-6510 - add an enable/disable property for analytics
* NIFI-6510 - documentation updates for enable/disable property
* NIFI-6510 - UI: handle the scenario where backpressure predictions are disabled (#3685)
* NIFI-6510 - admin guide updates to further describe model functionality
* NIFI-6510 - code quality fixes (if statement and constructor)
* NIFI-6510 - log warnings when properties could not be retrieved. fixed incorrect property retrieval for score threshold
* NIFI-6510 Extract out predictions into their own DTO
* NIFI-6510 Optimize imports
* NIFI-6510 Fix formatting
* NIFI-6510 Optimize imports
* NIFI-6510 Optimize imports
* NIFI-6510 - Notice updates for Commons math and Caffeine
* NIFI-6510 - UI updates to account for minor API changes for back pressure predictions (#3697)
* NIFI-6510 - Fix issue displaying estimated time to back pressure in connection summary table when only one of the predictions is known.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3705
* NIFI-6510 Rip out useless members
* NIFI-6510 - dto updates to check for -1 value
* NIFI-6510 - checkstyle fix
* NIFI-6510 - rolled back last change and applied minNonNegative method
* NIFI-6510 Rip out useless members
* fixed issues converting sensitive props to parameters
* only include parameter value if it has changed (even for non-sensitive params)
* do not allow null parameter values when editing parameters.
* open existing add param dialog from property table
* convert prop to param, update param context
* keep add parameter dialog open until update of the context completes when converting property to param
* Updating parameter context while converting property to parameter.
* Disable the Apply button while updating
* When Cancel is pressed and there is an update in progress, issue the DELETE request to cancel the update.
* Show some status in the Add Parameter dialog when parameter context is updating following a property conversion.
* enforce character restrictions on property names.
* Add convert property to parameter to controller service config.
* Properly set sensitive setting when converting sensitive properties
* Allow converting of properties from controller services configuration into parameters
* Refactor addNewParameter method to be able to share common parts with convertPropertyToParameter
[NIFI-6282] when creating a parameter context inline during PG configuration set the newly created parameter context as the selected option
[NIFI-6282] If a request to update a parameter context fails, then update the button model to give the user the ability to Apply or Cancel again.
[NIFI-6282] address pr review comments
[NIFI-6282] remove es6 let
[NIFI-6282] update marshall parameters logic and add comments
[NIFI-6282] deterministic parameter usage listing, update CS status on PG PC change, expand all twisties by default, remove es6 const
[NIFI-6282] update regex and
[NIFI-6282] update parameter loading/serialization/marshalling
[NIFI-6282] use referencingComponents instead of affectedComponents
[NIFI-6282] activate Apply button for sensitive parameter set empty string change
[NIFI-6282] fix bug with PG parameters context menu enable
[NIFI-6282] only allow delete and recreate of a parameter with equivalent sensitivity
[NIFI-6282] display referencing components during parameter management as well as during the parameter context update
[NIFI-6282] display no value set in parameter table when parameter value is null
[NIFI-6282]
- Add ellipsis to referencing component names.
- Addressing issues canceling update requests.
- Addressing issues with incorrect service scope.
- Addressing issue showing the affected parameters.
NIFI-6382: Fixed bugs reported in code review: sensitive properties referencing parameters should be exported to registry; when importing a versioned flow, any new parameters should be added to the parameter context
NIFI-6382: Updated logic for handling parameter references with sensitive properties when integrating with nifi registry
NIFI-6382: If user does not have permissions to READ a parameter context with a given name when importing a flow or changing flow version, ensure that NiFi properly generates an AccessDenied messages instead of throwing NullPointerException
This closes#3642.
Signed-off-by: Bryan Bende <bbende@apache.org>
Mark Payne
exceptionfactory
Mohammed Nadeem
Joe Gresock
tpalfy
Mark Bean
Denes Arvay
Joe Witt
Matthew Burgess
Nathan Gough
Abhishek Kumar
s9514171
Timea Barna
Bence Simon
Tamas Palfy
Paul Grey
Pierre Villard
Bryan Bende
Rob Fellows
mtien
Lehel Boér
Jon Kessler
exceptionfactory
Andrew Lim
Nissim Shiman
tlsmith
sjyang18
Shane Ardell
humpfhumpf
Kevin Doran
Peter Turcsanyi
Andy LoPresto
M Tien
Andy LoPresto
Kotaro Terada
jmconte
Scott Aslan
shreeju
Karthikeyan Singaravelan
Vasily Makarov
thenatog
Matt Gilman
Peter Gyori
Troy Melhase
Michael Hogue
MatthewKnight-NG
Joe Ferner
nagasivanath
Jan Hentschel
Koji Kawamura
Andy I. Christianson
Arpad Boda