- Addressing issues causing the eventId to not be relayed when submitting a lineage request under certain conditions.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#1903.
When NiFi is clustered, and autopurge.purgeInterval is greater than 1, the DatadirCleanupManager will be started in order to automatically purge transaction log and snapshot files based on the autopurge settings in zookeeper.properties
This closes#1928.
NIFI-4061 Initial version of RedisStateProvider
- Adding PropertyContext and updating existing contexts to extend it
- Added embedded Redis for unit testing
- Added wrapped StateProvider with NAR ClassLoader in StandardStateManagerProvider
- Updating state-management.xml with config for RedisStateProvider
- Renaming tests that use RedisServer to be IT tests so they don't run all the time
This closes#1918.
- Introducing the LdapUserGroupProvider.
- Updating documentation accordingly.
- Moving the IdentityMapping utilities so they were accessible.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#1923.
- Introducing FileUserGroupProvider and FileAccessPolicyProvider.
- Refactoring FileAuthorizer to utilize the file based implementations.
- Introducing the StandardManagedAuthorizer.
- Decorating the configured ManagedAuthorizer to ensure integrity checks are still performed.
- Loading user groups if possible to use during access decisions.
- Merging responses for requests for AccessPolicies, Users, and UserGroups.
- Adding unit tests as appropriate.
- Adding methods to the User, Group, and AccessPolicy builder that more easily supports generating UUIDs.
- Fixing typo when seeding policies during startup.
- Fixing type in documentation and error messages.
This closes#1897.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Filtering out the EventTime searchable field for all implementation of ProvenanceRepository. Prefer user query using the startDate/endDate fields.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#1844.
- When monitoring heartbeats use the connected nodes as the basis for the check. This addresses the case when a node is terminated and no corresponding heartbeats exist.
This closes#1838.
- When monitoring heartbeats use the connected nodes as the basis for the check. This addresses the case when a node is terminated and no corresponding heartbeats exist.
Before this fix, 'NullPointerException: Inflater has been closed' can be thrown as the Inflater is closed before input stream is consumed.
Also, calling close from AbstractTransaction.receive is removed, because the DataPacket is exposed as its return value and this class will not be able to know when to close the stream.
Signed-off-by: joewitt <joewitt@apache.org>
Forces a node reconnecting to a cluster to serialize the updated flow to disk
Added most processor settings to the flow fingerprint (excluding name, style, comment, position, and schedule state)
Updated some test data for FingerprintFactoryTest to test for new fields added to the flow fingerprint
Updated StandardProcessorNode to allow processor comments and name to be settable while a processor is running
Updated StandardFlowSynchronizer to inherit non-fingerprinted processor settings (name, style, comment, and position) when flow is already synchronized
This closes#1812
NIFI-3900: Rebased against master and updated new unit test to use new method signature for FlowSerializer.serialize
This closes#1804.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Addressing response code of POST /provenance/lineage.
- Ensuring cluster node details are set accordingly.
This closes#1803.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
- Filtering out certain control characters and unpaired Unicode surrogate codepoints prior to saving the flow.xml.
This closes#1784.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- adjust testFullyDocumentedProcessor to correctly eval to false when needed
- Introduce the ability to deprecate a component
- Allow documentation to inform user about deprecation of a component
- This closes#1718
NIFI-391 - Add set/getDeprecationReson to DocumentedDTO and use it within DtoFactory'
NIFI-3838: Updated version from 1.2.0-SNAPSHOT to 1.3.0-SNAPSHOT; removed unneeded value from AttributeExpression.ResultType enum
NIFI-3838: Addressed PR Review feedback
NIFI-3838: Allow for schemas to be merged together for a record; refactored RecordSetWriterFactory so that there is a method to obtain the schema and then the writer is created with that schema. Added additional unit tests
NIFI-3838: Addressed problems with documentation based on PR Review
NIFI-3838: Fixed checkstyle violation
NIFI-3838: Addressed issue of comparing different types of Number objects
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1772
- Making the selector for the enable/disable controller service state more specific. Previously name was too general and was affect other parts of the application.
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
This closes#1759
Added src/test/resources/logback-test.xml files resetting log level from DEBUG (in nifi-data-provenance-utils) to WARN because later tests depend on MockComponentLog recording a certain number of messages and this number is different than expected if the log level is DEBUG.
This closes#1686.
Signed-off-by: Bryan Bende, Yolanda M. Davis, and Mark Payne
- Addressing issue when enabling/disabling controller services when referencing services are not in scope.
- Fixing enabled/disabled icons for referencing services.
- Rendering when a reference cycle is detected.
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
This closes#1725
- added proxy dn flag to support providing authorized username for secured environments
- addressed pr comments including fix to ensure proxy info added when getting cluster info, showing cleaner error messaging and improving help text. Also fixed potential issue with versioning comparison (mismatched lengths)
- Printing response body when requests fails.
- This closes#1697
- Added batchCount, batchSize, batchDuration to limit flow files to be
included in a single Site-to-Site transaction.
- Added batch throttling logic when StandardRemoteGroupPort transfers
flow files to a remote input port using the batch limit configurations,
so that users can limit batch not only for pulling data, but also pushing data.
- Added destination list shuffle to provide better load distribution.
Previously, the load distribution algorithm produced the same host consecutively.
- Added new batch settings to FlowConfiguration.xsd.
- Added new batch settings to Flow Fingerprint.
- Added new batch settings to Audit.
- Sort ports by name at 'Remote Process Group Ports' dialog.
- Show 'No value set' when a batch configuration is not set
- Updated batch settings tooltip to clearly explain how it works the configuration works differently for input and output ports.
- Updated DTO by separating batch settings to BatchSettingsDTO to indicate count, size and duration are a set of configurations.
- This closes#1306
This approaches a proper solution on how to hand over the key from
RunNiFi to NiFi. Insofar the password file is pruned as part of the
startup, NiFi processors can't read it.
See also: NIFI-3045.
This closes#1302.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Fixed SiteInfoProvider and HttpClient to use specified local address with its SiteToSiteRestApiClient
- Removed setupRequestConfig method call from connection and read timeout setter methods at SiteToSiteRestApiClient, because it created config object before local address was set
- Null clear StandardRemoteProcessGroup localAddress when user clears Local Network Interface
Signed-off-by: joewitt <joewitt@apache.org>
- Refactored ServerProtocol.sendPeerList method signature to clarify the
meaning of arguments, and avoid null pointer exception when converting null Integer to int.
- Refactored SocketRemoteSiteListener handleRequest method to make it
more unit test friendly.
- Added more unit tests.
- Using consistent logic when verifying connection creation. Removed some unecessary checks as verification has been changed to run in cluster and standalone mode.
This closes#1672.
If there is an unconnected remote port, right clicking a Remote Process Group
then select 'Remote Ports' causes a JS error saying 'Uncaught TypeError'.
Because there are few code paths those don't return the created
transmissionSwitch value from createPortOption function in
nf-remote-process-group-ports.js.
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
This closes#1679
Use it with FirstInFirstOutPrioritizer, it can enforce original ordering
of 'out-of-order' FlowFiles.
nifi-mock is modified to support FlowFile assertion using Prioritizer.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3414: Added EnforceOrder processor
Incorporated review comments, added displayNames.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3414: Added EnforceOrder processor
Incorporate review comments:
- Moved nifi-standard-prioritizers dependency to top level nifi/pom.xml.
- Changed default initial order from 1 to 0.
- Fixed typos.
- Use session.get(batchCount).
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3414: Added EnforceOrder processor
When a FlowFile is transferred to success, remove attributes previously set when it was transferred to wait or failure.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1496
NIFI-3678: If we have a transaction ID but then have no more data written to Partition file, we end up with a NPE. Added logic to avoid this and instead return null for the next record when this happens
This closes#1656.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Fixing FlowController to use appropriate class loader when instantiating processor
- Updating ExtensionManager to leverage new flag in MANIFEST from NAR plugin
- Adding ReloadComponent interface and refactoring instance class loading to use it
- Fixing FetchHDFS issue with TDE by using ugi.doAs
- Refactoring nifi-nar-utils so that ExtensionManager only lives in nifi-framework
- Caching temp components found during service loader in ExtensionManager
- Updating authorizables, docs, and fingerprinting to use the cached components
- Introducing a flag on @RequiresInstanceClassLoading to indicate if ancestor resources should be cloned
- Updating developer guide regarding cloneAncestorResources flag
- This closes#1635
This closes#1646.
Newly created async response is added before checking map size nor
purging expired ones. If there are already 100 remaining requests,
the added request will not be executed nor removed.
- On contextDestroyed, referencing beans created during contextInitialized to prevent successive attempts to create a bean if that bean failed to be created initially.
Signed-off-by: joewitt <joewitt@apache.org>
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3257: Added additional logging regarding timing information when replicating requests across cluster in order to glean insight as to what is taking so long when replicating some requests
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3649: Buffer node responses when replicating HTTP Requests up to a maximum buffer size
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3636: Added unit test to ensure that flowfile attribute maps are copied when appropriate
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3636: Removed patch file that should not have been in commit
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1612
- Refactoring NarDetails to include all info from MANIFEST
- Adding the concept of a Bundle and refactoring NarClassLoaders to pass Bundles to ExtensionManager
- Adding logic to fail start-up when multiple NARs with same coordinates exist, moving Bundle classes to framework API
- Refactoring bundle API to classes and creating BundleCoordinate
- Updating FlowController to use BundleCoordinate
- Updating the UI and DTO model to support showing bundle details that loaded an extension type.
- Adding bundle details for processor canvas node, processor dialogs, controller service dialogs, and reporting task dialogs.
- Updating the formating of the bundle coordinates.
- Addressing text overflow in the configuration/details dialog.
- Fixing self referencing functions.
- Updating extension UI mapping to incorporate bundle coordinates.
- Discovering custom UIs through the supplied bundles.
- Adding verification methods for creating extensions through the rest api.
- Only returning extensions that are common amongst all nodes.
- Rendering the ghost processors using a dotted border.
- Adding bundle details to the flow.xml.
- Loading NiFi build and version details from the framework NAR.
- Removing properties for build and version details.
- Wiring together front end and back end changes.
- Including bundle coordinates in the component data model.
- Wiring together component data model and flow.xml.
- Addressing issue when resolve unvesioned dependent NARs.
Updating unit tests to pass based on framework changes
- Fixing logging of extension types during start up
- Allowing the application to start if there is a compatible bundle found. - Reporting missing bundle when the a compatible bundle is not found. - Fixing table height in new component dialogs.
Fixing chechstyle error and increasing test timeout for TestStandardControllerServiceProvider
- Adding ability to change processor type at runtime
- Adding backend code to change type for controller services
- Cleaning up instance classloaders for temp components.
- Creating a dialog for changing the version of a component.
- Updating the formatting of the component type and bundle throughout.
- Updating the new component dialogs to support selecting source group.
- Cleaning up new component dialogs.
- Cleaning up documentation in the cluster node endpoint.
Adding missing include in nifi-web-ui pom compressor plugin
- Refactoring so ConfigurableComponent provides getLogger() and so the nodes provide the ConfigurableComponent
- Creating LoggableComponent to pass around the component, logger, and coordinate with in the framework
- Finishing clean up following rebase.
Calling lifecycle methods for add and remove when changing versions of a component
- Introducing verifyCanUpdateBundle(coordinate) to ConfiguredComponent, and adding unit tests
- Ensuring documentation is available for all components. Including those of the same type that are loaded from different bundles.
Adding lookup from ClassLoader to Bundle, adding fix for instance class loading to include all parent NARs, and adding additional unit tests for FlowController
- Adding validation to ensure referenced controller services implement the required API
- Fixing template instantiation to look up compatible bundle
- Requiring services/reporting tasks to be disabled/stopped.
- Only supporting a change version option when the item has multiple versions available.
- Limiting the possible new controller services to the applicable API version.
- Showing the implemented API versions for Controller Services.
- Updating the property descriptor tooltip to indicate the required service requirements.
- Introducing version based sorting in the new component dialog, change version dialog, and new controller service dialog.
- Addressing remainder of the issues from recent rebase.
Ensuring bundles have been added to the flow before proposing a flow, and incorporating bundle information into flow fingerprinting
- Refactoring the way missing bundles work to retain the desired bundle if available
- Fixing logger.isDebugEnabled to be logger.isTraceEnabled
- Auditing when user changes the bundle. - Ensuring bundle details are present in templates.
Moving standard prioritizers to framework NAR and refactoring ExtensionManager logic to handle cases where an extension is in a JAR directly in the lib directory
- Ensuring all nodes attempt to instantiate the same template instance when the available bundles may differ. - Fixing the auditing of copy/paste and template instantiation. - Running addtional verification methods when running standalone.
Refactoring controller service invocation handler to allow updating the node used by the invocation handler
- Ensuring the bundles in a proposed flow are compatible with the current instance when the current instance has no flow is going to accept the proposed flow
- Merging whether multiple versions of the component are available
- Setting NAR plugin back to current released version
- Cleaning up DocGenerator to not process multiple times
Addressing incorrect usage of nf.Common. - Using formatType in the new component type dialogs.
Improving error messages when looking for bundles
Addressing comments from PR. - Fixing references to global nf namespace. - Fixing injection of nfProcessGroupConfiguration in nfComponentVersion. - Fixing web api integration tests.
Not rendering unversioned in help documentation. - Ensuring the isExtentionMissing flag is correct after changing the component type.
Adding synchronization in node classes to ensure changing component can't occur when component is running, introducing MissingBundleException for better reporting when a node can't join cluster due to a missing bundle, and bumping NAR plugin to released version 1.2.0
Adding concept of missing components to fingerprinting to ensure nodes agree on missing components when joining a cluster
NIFI-3380: NIFI-3520: - Fixing hive nar dependency. - Marking DBCPService as provided. - Skipping services that require instance classloading and are cobundled with their service API. - Skipping components that require instance classloading and reference service APIs that are cobundled. - Addressing UI issues in the new component dialogs when re-opening with a filter applied.
Fixing checkstyles issue and adding back assume checks to distributed cache server test
Ensuring new component types are sorted correctly when shown initially.
This closes#1585.
- Preventing the editing of controller services that are defined in an ancestor process group.
- Adding a go to link for users to easily navigate to where services are actually defined.
This closes#1594
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
- Adding notes to the controller service tables to better define their availability.
- Clarifying support of Safari.
This closes#1593
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
- Updating Controller Service scope header.
- Rendering the name of the parent when possible.
- Fixing the check if the parent Process Group is write-able when the parent is selected on the current canvas.
This closes#1590
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
- Verify that when moving a snippet, we do not attempt to move a process group into itself.
- Update the UI to ensure a subsequent move event cannot be triggered while a previous move event is still in progress.
This closes#1582.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Added timeout, yieldPeriod, transportProtocol, proxyHost, proxyPort,
proxyUser and proxyPassword
- Removed unused fingerprint methods
FingerprintFactory has two types of fingerprinting method, from XML
elements and from DTO. However, the ones from DTO are not used by
anywhere. IDE didn't report those private methods unused because
addProcessGroupFingerprint and addSnippetFingerprint call each other,
but those are not used from outside actuallly.
This commit removes those private methods to keep the class clean to
avoid unnecessary code maintenance and tests.
This closes#1332.
- The idea behind NIFI-3356 was to improve the efficiency and throughput of the Provenance Repository, as it is often the bottleneck. While testing the newly designed repository,
a handful of other, fairly minor, changes were made to improve efficiency as well, as these came to light when testing the new repository:
- Use a BufferedOutputStream within StandardProcessSession (via a ClaimCache abstraction) in order to avoid continually writing to FileOutputStream when writing many small FlowFiles
- Updated threading model of MinimalLockingWriteAheadLog - now performs serialization outside of lock and writes to a 'synchronized' OutputStream
- Change minimum scheduling period for components from 30 microseconds to 1 nanosecond. ScheduledExecutor is very inconsistent with timing of task scheduling. With the bored.yield.duration
now present, this value doesn't need to be set to 30 microseconds. This was originally done to avoid processors that had no work from dominating the CPU. However, now that we will yield
when processors have no work, this results in slowing down processors that are able to perform work.
- Allow nifi.properties to specify multiple directories for FlowFile Repository
- If backpressure is engaged while running a batch of sessions, then stop batch processing earlier. This helps FlowFiles to move through the system much more smoothly instead of the
herky-jerky queuing that we previously saw at very high rates of FlowFiles.
- Added NiFi PID to log message when starting nifi. This was simply an update to the log message that provides helpful information.
NIFI-3356: Fixed bug in ContentClaimWriteCache that resulted in data corruption and fixed bug in RepositoryConfiguration that threw exception if cache warm duration was set to empty string
NIFI-3356: Fixed NPE
NIFI-3356: Added debug-level performance monitoring
NIFI-3356: Updates to unit tests that failed after rebasing against master
NIFI-3356: Incorporated PR review feedback
NIFI-3356: Fixed bug where we would delete index directories that are still in use; also added additional debug logging and a simple util class that can be used to textualize provenance event files - useful in debugging
This closes#1493
The 'exec' command added by NIFI-2689 affected restart behavior
negatively as 'exec' command will not execute subsequent commands in the
shell script.
This commit changes 'exec' is added only when 'run' is specified.
This closes#1523.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
- Add 'nifi.flow.configuration.archive.max.count' in nifi.properties
- Change default archive limit so that it uses archive max time(30 days)
and storage (500MB) if no limitation is specified
- Simplified logic to delete old archives
This closes#1460.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
* add exec to RUN_NIFI_CMD
* remove subshell for else
* tested compatible with runit with these changes
This closes#966.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
- Added configure audits for Transport Protocol, HTTP Proxy Server Host,
Port, User and Password in RemoteProcessGroup configuration
- Added configure audits for enabling/disabling individual remote port
- Added configure audits for Concurrent Tasks and Compressed in Remote
Port configuration
- This closes#1476
- Requiring WRITE permissions to the parent resource when attempting to remove a component.
- Updating expired certificates in the REST API integration tests.
This closes#1399.
Signed-off-by: James Wing <jvwing@gmail.com>
- Adding additional parameters to be able to limit the size of the provenance response. Specifically, whether the events should be summarized and whether events should be returned incrementally before the query has completed.
- Ensuring the cluster node address is included in provenance events returned.
- Ensuring there is a cluster coordinator before attempting to get the cluster node address.
- Removing exponential back off between provenance requests.
- Ensuring the content viewer url is retrieve before initializing the provenance table.
This closes#1413.
Pierre Villard
Mark Payne
Matt Gilman
yuri1969
Scott Aslan
Koji Kawamura
Niels Zeilemaker
Jeff Storck
Joe Percivall
Bryan Bende
Wesley-Lawrence
Andre F de Miranda
Mike Moser
Joe Skora
James Wing
joewitt
Aldrin Piri
Peter G. Horvath
Andy LoPresto
Yolanda M. Davis
Bryan Rosander
Anders Breindahl
Oleg Zhurakousky
Puspendu Banerjee
Hunter Morgan