NIFI-11636: Change default log level from parquet internal reader to WARN as it logs excessively at INFO level
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#7334
- Removed NetworkUtils methods for getting available ports
- Updated Socket-based components to support using 0 to listen on a random available port for improved test reliability
This closes#7299
Signed-off-by: David Handermann <exceptionfactory@apache.org>
NIFI-11557: Added an additional system test and updated github actions to include surefire-report in order to help diagnose problem that occurred in one of the last system-test runs in Github. Could not replicate problem locally
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#7265
- Added Shibboleth repository for OpenSAML
- Replaced deprecated OpenSAML 3 Spring Security components with OpenSAML 4
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#7251.
* NIFI-11287: detect dependent properties when the property it depends on references a parameter
* address review feedback
* address more review comments
* - loadProperties checks type before getting the parameter context
- retrieve the parameter context with ajax call from inside the class
* add type parameter to all places calling loadProperties
* get reference parameters by invoking provided callbacks from the client
* check for user permissions before requesting for referenced parameters
* address review feedback
This closes#7117
- Added methods to enumerate Stored Component Identifiers on State Provider interface and implementations
- Added nifi.state.management.provider.cluster.previous to nifi.properties
- Updated State Manager Provider to restore Cluster State from Previous Cluster Provider
- Updated Configuring State Providers documentation for new property
- Restored previous behavior of sending openid and email scopes for OpenID Connect token requests
- Added offline_access scope as the default value in nifi.properties to support Refresh Tokens
This closes#7168
Signed-off-by: Paul Grey <greyp@apache.org>
* NIFI-11461 Improved User and Group Tenants Search
- Added searchTenants method to NiFiServiceFacade and removed unnecessary object creation
- Updated TenantsResource to use delegated NiFiServiceFacade.searchTenants method
- Changed autocomplete delay from default 300 ms to 500 ms
* NIFI-11461 Adjusted implementation to use EntityFactory.createTenantEntity
This closes#7181
- Updated GitHub workflow so that system tests include Python 3.9
- Updated GitHub actions to build necessary modules for system tests
This closes#7003
Co-authored-by: David Handermann <exceptionfactory@apache.org>
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Fixed system tests so that they work properly in Clustered version of RegistryClientIT
- Fixed system test - ensure that we wait for processors to become valid before attempting to start them; also added an additional system test around Controller Services in versioned flows
This closes#7095
Signed-off-by: David Handermann <exceptionfactory@apache.org>
* NIFI-4890 Refactored OIDC with support for Refresh Tokens
- Implemented OIDC Authorization Code Grant Flow using Spring Security Filters
- Implemented OIDC RP-Initiated Logout 1.0
- Implemented OAuth2 Token Revocation RFC 7009 for Refresh Tokens
- Added OIDC Bearer Token Refresh Filter for updating application Bearer Tokens from Refresh Token exchanges
- Added configurable Token Refresh Window to application properties
- Removed original implementation and supporting classes
* NIFI-4890 Set Bearer Token expiration based on Access Token
* NIFI-4890 Corrected spelling and naming issues based on feedback
This closes#7013
Fixed issue in StandardContentClaimWriteCache in which inner OutputStream class did not have an idempotent close() method; as a result, the stream could be written to while already in use for another active FlowFile; fixed bug in ContentClaimInputStream in which skip() method ignored its own BufferedInputStream - this was discovered because it was causing failures in StandardProcessSessionIT; fixed bug in StandardProcessSessionIT in which the length of StandardContentClaim was being doubled because the OutputStream was setting the claim length but that is already handled at a lower level.
This closes#7087
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Ensure that components are notified that primary node has changed in a background thread instead of the Leader Election thread and activate/deactivate the thread in the case of Processors so that they can be viewed in the UI and terminated
- Fixed system tests that would fail intermittently because they did not wait for node disconnection to complete and did not properly switch the client to look at the connected node before checking cluster status
This closes#7052
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Upgraded direct dependencies in Framework and Registry modules as well as Spark bundle
This closes#7073
This closes#7074
This closes#7075
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Replaced deprecated Matchers references with ArgumentMatchers
- Removed unnecessary Mockito versions for Registry
- Refactored test configuration to Java for mocking
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#7071.
NIFI-11310: Fixed META-INF/services file that was mistakenly listing an extra extension point, due to rebase
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#7061
* NIFI-10975 Added Kubernetes Leader Election and State Provider
- Added Kubernetes Leader Election Manager based on Kubernetes Leases
- Added Kubernetes State Provider based on Kubernetes ConfigMaps
- Added nifi-kubernetes-client for generalized access to Fabric8 Kubernetes Client
- Added nifi.cluster.leader.election.implementation Property defaulting to CuratorLeaderElectionManager
- Refactored LeaderElectionManager to nifi-framework-api for Extension Discovering Manager
- Refactored shared ZooKeeper configuration to nifi-framework-cluster-zookeeper
* NIFI-10975 Updated Kubernetes Client and StateMap
- Upgraded Kubernetes Client from 6.2.0 to 6.3.0
- Added getStateVersion to StateMap and deprecated getVersion
- Updated Docker start.sh with additional properties
* NIFI-10975 Corrected MockStateManager.assertStateSet()
* NIFI-10975 Upgraded Kubernetes Client from 6.3.0 to 6.3.1
* NIFI-10975 Corrected unregister leader and disabled release on cancel
* NIFI-10975 Corrected findLeader handling of Lease expiration
- Changed LeaderElectionManager.getLeader() return to Optional String
* NIFI-10975 Corrected StandardNiFiServiceFacade handling of Optional Leader
* NIFI-10975 Changed getLeader() to call findLeader() to avoid stale cached values
* NIFI-10975 Updated LeaderElectionCommand to run LeaderElector in loop
* NIFI-10975 Rebased on project version 2.0.0-SNAPSHOT
* NIFI-10975 Corrected Gson and AspectJ versions
- Updated versions to match current main branch and avoid reverting
- Moved StringUtils from nifi-properties to nifi-property-utils
- Moved Peer Identity methods from CertificateUtils to specific Site-to-Site classes
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#6977.
NIFI-11192: If a failure is encountered when changing the version of a flow from 1 version to another, attempt to rollback the changes instead of just failing with the flow in a bad state
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#6981
- Removed unnecessary references to PropertyEncryptor from multiple framework components
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#6962.
- Updated TestAttributesToCSV with LinkedHashMap for deterministic test behavior
- Updated TestHttpFlowFileServerProtocol with adjusted checksums to match expectations from LinkedHashMap changes
This closes#6748
Signed-off-by: David Handermann <exceptionfactory@apache.org>
This closes#6750
Signed-off-by: David Handermann <exceptionfactory@apache.org>
Co-authored-by: Chris Sampson <12159006+ChrisSamo632@users.noreply.github.com>
- Upgraded Logback from 1.2.11 to 1.3.5
- Updated Logback DelayingShutdownHook to DefaultShutdownHook
- Disabled Spring Boot Logging System in favor of standard Logback initialization
- Excluded logback-classic from ZooKeeper and other dependencies to avoid conflicts when running tests
- Excluded spring-boot-starter-logging to avoid failures related to Logback 1.2 and Spring Boot 2.7
- Removed ZooKeeperMigratorTest.groovy based on Apache Curator test server usage of Logback 1.2
NIFI-10580 Added logback-core as explicit dependency
- Set logback-core as provided in root configuration
- Added logback-core as compile dependency in assembly configurations
Signed-off-by: Joe Witt <joewitt@apache.org>
Correcting EmbeddedQuestDbStatusHistoryRepositoryForComponentsTest
Correcting TestQueryNiFiReportingTask
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#6869
- Updated impacted classes to remove redundant import lines
- Removed WebUtilsGroovyTest.groovy class due to use of internal sun.security classes
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#6804.
- NIFI-10981 Ensure NarAutoLoader starts after provider retrieves NARs, and ensure the auto-loader attempts to process any NARs available before starting to watch for new files
- Add system test with NAR Provider that copies NARs from a local directory and verifies the NARs were loaded and linked correctly
- Exclude new assembly from CI test-compile
- Update manifests assembly to include the additionaDetails.html files
- Update C2 ExtensionDefinition to add a new field for additional details content
- Update manifest generation to read the additional details and pass through to C2 runtime manifest
- Switch to boolean to indicate presence of additional details, instead of including content
Add additionalDetails files to nifi-runtime-manifest artifact, organized by group/artifact/version/type
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#6767
- Added Commons Net to managed dependencies in root Maven configuration
- Removed version references to 3.3 in MiNiFi and 3.6 in NiFi standard modules
Signed-off-by: Joe Witt <joewitt@apache.org>
* display description for selected flow
* NIFI-10814: add border when content overflows and increase height
* NIFI-10814: add padding to description
* NIFI-10814: remove duplicate CSS property from declaration block
* NIFI-10814: a few more CSS tweaks
This closes#6693
NifiRegistryFlowRegistryClient defines the PropertyDescriptor PROPERTY_URL name as small case "url". The map bases on the name property of the PropertyDescriptor object. Here searching with uppercase value of "URL" causes the map lookup to fail and cause a NPE later on. Therefore, it is changed as "url"
NIFI-10787 - Added constant for property descriptor "url" in NiFiRegistryFlowMapper to make it more clear.
NIFI-10787 - Added change to unit test.
NIFI-10787 - Updated unit test to validate that NiFi registry url is being set and retrieved.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#6655.
NIFI-10703 - Also setting the maxEventDrivenThreadCount on the controller when using a versioned flow.
NIFI-10703 - Corrected to use getMaxEventDrivenThreadCount(), set a default value for event driven thread count in VersionedDataflow
NIFI-10703 - Updated log message for setMaxThreadCount in FlowController.java
NIFI-10703 - Updated default value for maxEventDrivenThreadCount
NIFI-10703 - Set private for DEFAULT_MAX_EVENT_DRIVEN_THREAD_COUNT
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#6638
NIFI-9398 processor verification integration-tests for nifi-elasticsearch-restapi-processors using common nifi-elasticsearch-test-utils refactored from nifi-elasticsearch-client-service
NIFI-9398 fail index exists verification for Elasticsearch processors whose operations would not create the index if it doesn't already exist
This closes#6544
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
- Only restart ports if their scheduledState is Running.
- Adding unit tests.
- Fixing unit test to properly verify Port isn't restarted.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#6582.
- Upgraded Jettison in nifi-framework-bundle
- Upgraded Jettison in nifi-registry-ranger
- Upgraded Jettison in nifi-spark-bundle
This closes#6548
This closes#6549
This closes#6550
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Added org.apache.commons.text.version property in root Maven configuration
- Removed direct version references in favor of managed dependency version
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#6531.
- Updated XmlFlowSynchronizer to filter parsed Sensitive Dynamic Property Names using dynamic status of component Property Descriptor
- Lack of access to the Component Property Descriptor when parsing DOM elements required subsequent dynamic status filtering
- Corrected logging statements with placeholders instead of concatenation
- Removed unused NiFiServerStub
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#6513
* NIFI-10577 Fixed Sensitive Checkbox rendering for Fetch Parameters
- Corrected rendering of Sensitive status checkbox on Firefox using standard nf-checkbox
* NIFI-10577 Corrected canvas rendering in Safari
Merged #6473 in main.
* NIFI-10566: update availability text in Controller Settings
- fix link in controller service referencing a parameter provider
* add Registry Clients to the availability message
This closes#6463
Adding provided scope to api dependency in nifi-standard-parameter-providers module
Adding additional documentation, other minor code cleanup
Correcting error handling in StandardParameterProviderNode, updating additional details for DatabaseParameterProvider
Correcting null columm value handling
NIFI-9402: Fixed Checkstyle violation
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#6391
* NIFI-10542: set sensitivity for new parameters in a synced parameter context
* show removed or missing but referenced parameters with tooltip in the selectable parameters table
- remove default parameter value
* address review feedback
- fix showing a referencing icon for a missing but removed parameter
* fix discovering new parameters
This closes#6446
* init commit
* remove test string
* display only class name for types combo
* remove commented code
* add UI support for dynamic properties
* add support for controller services
* address review feedback:
* fix syntax error
* remove console logs
* move contents of editRegistry into callback
* add name and description fields back to edit dialog
* add referencing component logic for registry clients
* implement reload logic
* more review findings
* increase size of description textarea in config
* remove location column from registry table
* remove commented out code
* post-rebase tweaks
* change if condition from assignment to evaluation
* add missing semicolons
* refactor new registry initialization into function
* more feedback addressed
* fix rebase error around controller service reference
* render registry references in their own list
* remove duplicate updateValidationError function
* remove TODO comment
* expose reload through nfSettings public variable
* show add registry dialog only after available types returns
* fix css class name
This closes#6437
* NIFI-9514 - Add UI support for Parameter Providers in Controller Services
* - Fix a dialog closing issue
* - Hide fetch parameters button if there are config errors
* - Update Fetch Parameters dialog with new UX
- Implement checkbox logic for Selectable Parameters table
* - add createNewParameterContext property to track if group will be created as a new parameter context
- update fetch parameters dialog views for editing and after applying changes
- show affected referencing components after applying changes
- disable apply button if no parameter contexts will be updated
- refactor showing and hiding fetched parameters and selectable parameters tables logic
- add tooltip for parameter context icon
* - update parameter contexts view for provided contexts
* - add affected referencing components after applying fetched parameters
- add referencing parameter contexts to the provider settings tab
- update checkbox style in the selectable parameters table
- fix updating the referencing components border
- fix lint errors
* - fix selectable parameters table checkbox styles
* - fix hasParameterContextChanged checks
- checkbox style tweaks
* - add status bar to fetch parameters dialog
* - more style tweaks and remove unnecessary code
* - fix the selectable parameters table not saving and loading correctly
* - add tooltips
- fix styles
* - link from the parameter context to the associated parameter provider
* 9514: support navigating to the controller service created in parameter providers
* fix bulletins format in status bar
* address review feedback
- manage permissions for parameter providers
- disable/enable Fetch Parameters dialog Apply button
* manage authorizations to access parameter providers and parameter contexts
* fix lint errors
* fix and refactor hasGroupsChanged to determine the Apply button state
* new parameter context name input cannot be empty in Fetch Parameters dialog
* address review feedback
* fix duplicate code
* more fixes
* update fetch parameters checkbox states
* replace slickgrid checkbox selection plugin with custom implementation for different selection types
- check for affected parameters using the parameter status
* fixed showing the correct referencing components when switching groups
* fixed selectable parameters table shifting contents on row selection
* leverage slickgrid onClick to set user-selected parameter sensitivities in the selectable parameters table
* fix statusbar bulletins in processor details
- fix updating borders for referencing components
- update referencing parameter icon
- update table title in fetch parameters dialog
* address review feedback
* populate controller services referencing components of a referenced provider
- update controller services tab name
* showing parameter providers in CS controller service
* update tooltip messages in fetch parameters dialog
- fix dialog height to make room for the status bar
* reload the parameter providers table after a referenced parameter provider change in controller services
- fix conditions for enabling/disabling fetch parameters dialog Apply button
- add affected components no permissions message in the dialog
* fix always reloading the parameter providers table
- fix disableApplyButton checks
- add dialog message for a missing new parameter context name
* remove extra reload
* more fixes for disableApplyButton
- fixed styles for dialog message
* add a check in disableApplyButton for changed parameter values that are not referenced
- add changed value icon for parameters that are not referenced
- fix border on the referencing affected components container
* removed unnused plugins
This closes#5671
* NIFI-10473: Removing referencing components check on param provider fetch
* NIFI-10473: Adding parameter status DTO to ParameterProviderDTO
* Allowing parameterStatus to be populated even when no parameters were updated
* Adding ParameterStatus enum for parameter fetching
* Adding MISSING_BUT_REFERENCED ParameterStatus
This closes#6388
- NIFI-10518: Adding intended state to ScheduledStateChangeListener
- Notifying of scheduled state change when transitionComponentState is called
- Notifying scheduled state change when reporting task state is changed
- Notifying scheduledState changes for remote group port start/stop components calls
- Removing org.apache.httpcomponents:httpclient from nifi-web-api avoids Linkage Errors with Sensitive Property Providers
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#6423.
- Updated Admin Guide with section on Logging Configuration and Deprecation Logging
- Updated Developer Guide with section on Deprecating Components and Features
- Added Deprecation Logging when adding components marked with DeprecationNotice to the Flow Configuration
- Added Deprecation Logging on deprecated methods in standard Controller Service implementations
- Removed integration tests for deprecated PersistentProvenanceRepository
- Updated logging and added documentation on minor upgrade steps
- Updated logging for HDFSNarProvider and RocksDB Repository
This closes#6390
Signed-off-by: Paul Grey <greyp@apache.org>
- Updated standard user interface error handler to clear the Access-Token-Expiration item from Session Storage when receiving an error with the WWW-Authenticate Header indicating a problem with the Bearer Token
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#6334.
- Added nifi-deprecation-log module with interface and implementation using SLF4J
- Updated standard logback.xml with nifi-deprecation.log appender
- Updated NiFiLegacyCipherProvider with deprecation logging
- Set Size, Time Policy, and Total Size Limit for Deprecation Log
This closes#6300
Signed-off-by: Paul Grey <greyp@apache.org>
Mark Payne
Scott Aslan
Bence Simon
Michael Moser
NissimShiman
Joe Gresock
exceptionfactory
Nissim Shiman
M Tien
Timea Barna
Lehel Boér
Pierre Villard
Bryan Bende
Matthew Burgess
Christian Wahl
Joe Witt
dependabot[bot]
simonbence
Mark Bean
dan-s1
Priyanka Awatramani
Paul Grey
Peter Turcsanyi
krisztina-zsihovszki
nathluu
greyp9
mr1716
Shane Ardell
aksharau
krisztina-zsihovszki
sedadgn
Nathan Gough
Chris Sampson
Ferenc Erdei
Matt Gilman
Ferenc Kis
Denes Arvay
UcanInfosec
mtien
Malthe Borch
Nandor Soma Abonyi
Andrew Lim
Csaba Bejan