- Creating nifi-records-utils to share utility code from record services
- Refactoring Parquet tests to use MockRecorderParser and MockRecordWriter
- Refactoring AbstractPutHDFSRecord to use schema access strategy
- Adding custom validate to AbstractPutHDFSRecord and adding handling of UNION types when writing Records as Avro
- Refactoring project structure to get CS API references out of nifi-commons, introducing nifi-extension-utils under nifi-nar-bundles
- Updating abstract put/fetch processors to obtain the WriteResult and update flow file attributes
This closes#1712.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Added org.apache.nifi.processor.util.pattern package in nifi-processor-utils containing reusable functions to mix-in 'Rollback on Failure' capability.
- Created a process pattern classes, Put and PutGroup. It will be helpful to standardize Processor implementations.
- Applied Rollback on Failure to PutSQL, PutHiveQL, PutHiveStreaming and PutDatabaseRecord.
- Stop using AbstractProcessor for these processors, as it penalizes FlowFiles being processed when it rollback a process session. If FlowFiles are penalized, it will not be fetched again until penalization expires.
- Yield processor when a failure occurs and RollbackOnFailure is enabled. If we do not penalize nor yield, a failed FlowFile retries too frequently.
- When Rollback on Failure is enabled but processor is not transactional, discontinue when an error occurred after successful processes.
- Fixed existing issues on PutHiveStreaming:
- Output FlowFile Avro format was corrupted by concatenating multiple Avro files.
- Output FlowFile records had incorrect values because of reusing GenericRecord instance.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1658
- added proxy dn flag to support providing authorized username for secured environments
- addressed pr comments including fix to ensure proxy info added when getting cluster info, showing cleaner error messaging and improving help text. Also fixed potential issue with versioning comparison (mismatched lengths)
- Printing response body when requests fails.
- This closes#1697
- Added batchCount, batchSize, batchDuration to limit flow files to be
included in a single Site-to-Site transaction.
- Added batch throttling logic when StandardRemoteGroupPort transfers
flow files to a remote input port using the batch limit configurations,
so that users can limit batch not only for pulling data, but also pushing data.
- Added destination list shuffle to provide better load distribution.
Previously, the load distribution algorithm produced the same host consecutively.
- Added new batch settings to FlowConfiguration.xsd.
- Added new batch settings to Flow Fingerprint.
- Added new batch settings to Audit.
- Sort ports by name at 'Remote Process Group Ports' dialog.
- Show 'No value set' when a batch configuration is not set
- Updated batch settings tooltip to clearly explain how it works the configuration works differently for input and output ports.
- Updated DTO by separating batch settings to BatchSettingsDTO to indicate count, size and duration are a set of configurations.
- This closes#1306
NIFI-3704: Incorporated review comments
NIFI-3704: Updated PutDatabaseRecord documentation, including limitations on Update records
NIFI-3704: Added retry paths for transient SQL errors
NIFI-3704: Updated to use renamed RecordReaderFactory and catch SchemaNotFoundException
NIFI-3704: Added better error handling and rollback
This closes#1677.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-3738 Fixed NPE when ParseSyslog UDP datagram has zero length.
Added unit test.
NIFI-3738 Added licenses to new unit tests.
This closes#1694.
Signed-off-by: Bryan Bende <bbende@apache.org>
This approaches a proper solution on how to hand over the key from
RunNiFi to NiFi. Insofar the password file is pruned as part of the
startup, NiFi processors can't read it.
See also: NIFI-3045.
This closes#1302.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
connections (Think chat to all clients instead of individual person).
The core change was a change in WebSocketMessageRouter.java where if a
sessionId is not present the message is sent to all connected clients.
So the key is leaving the sessionId to empty or null to send to all
clients. If the sessionId is specified the message will be sent just to
that session specified.
This closes#1649.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Fixed SiteInfoProvider and HttpClient to use specified local address with its SiteToSiteRestApiClient
- Removed setupRequestConfig method call from connection and read timeout setter methods at SiteToSiteRestApiClient, because it created config object before local address was set
- Null clear StandardRemoteProcessGroup localAddress when user clears Local Network Interface
Signed-off-by: joewitt <joewitt@apache.org>
- Refactored ServerProtocol.sendPeerList method signature to clarify the
meaning of arguments, and avoid null pointer exception when converting null Integer to int.
- Refactored SocketRemoteSiteListener handleRequest method to make it
more unit test friendly.
- Added more unit tests.
- Using consistent logic when verifying connection creation. Removed some unecessary checks as verification has been changed to run in cluster and standalone mode.
This closes#1672.
If there is an unconnected remote port, right clicking a Remote Process Group
then select 'Remote Ports' causes a JS error saying 'Uncaught TypeError'.
Because there are few code paths those don't return the created
transmissionSwitch value from createPortOption function in
nf-remote-process-group-ports.js.
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
This closes#1679
Signed-off-by: Mark Payne <markap14@hotmail.com>
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-1280: Updated SimpleKeyValueSchemaRegistry to make use of new CHOICE RecordFieldType - Update Record Readers to use SchemaRegistry controller service. Moved SchemaRegistry api into its own maven module and added to standard-services-api so that we can properly add dependencies on it. Code cleanup and bug fixes
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-1280: Fixed checkstyle violations and license exclusions for RAT plugin
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-1280: Addressed feedback from PR Review
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-1280: Additional changes/doc to support QueryFlowFile and Record Readers/Writers
This closes#1652
Updated L&N in nifi-gcp-nar and nifi-assembly and fixed nifi-gcp-processors
to avoid jsr305 and json.org cat-x dependencies
Signed-off-by: James Wing <jvwing@gmail.com>
This closes#1665.
Fixed service instance lookup to use local port so that it works when
it is accessed through port forwarding.
This fixes: #1568
Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
Use it with FirstInFirstOutPrioritizer, it can enforce original ordering
of 'out-of-order' FlowFiles.
nifi-mock is modified to support FlowFile assertion using Prioritizer.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3414: Added EnforceOrder processor
Incorporated review comments, added displayNames.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3414: Added EnforceOrder processor
Incorporate review comments:
- Moved nifi-standard-prioritizers dependency to top level nifi/pom.xml.
- Changed default initial order from 1 to 0.
- Fixed typos.
- Use session.get(batchCount).
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3414: Added EnforceOrder processor
When a FlowFile is transferred to success, remove attributes previously set when it was transferred to wait or failure.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1496
Enabled the ability to specify wildcard topics as a regular expression
as supported in the Kafka client library.
Signed-off-by: joewitt <joewitt@apache.org>
NIFI-3678: If we have a transaction ID but then have no more data written to Partition file, we end up with a NPE. Added logic to avoid this and instead return null for the next record when this happens
This closes#1656.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Fixing FlowController to use appropriate class loader when instantiating processor
- Updating ExtensionManager to leverage new flag in MANIFEST from NAR plugin
- Adding ReloadComponent interface and refactoring instance class loading to use it
- Fixing FetchHDFS issue with TDE by using ugi.doAs
- Refactoring nifi-nar-utils so that ExtensionManager only lives in nifi-framework
- Caching temp components found during service loader in ExtensionManager
- Updating authorizables, docs, and fingerprinting to use the cached components
- Introducing a flag on @RequiresInstanceClassLoading to indicate if ancestor resources should be cloned
- Updating developer guide regarding cloneAncestorResources flag
- This closes#1635
NIFI-3413: Incorporated review comments
NIFI-3413: Changed GetChangeDataCaptureMySQL to CaptureChangeMySQL, fixed some bugs
NIFI-3413: Refactored setup() for better error handling, more review comments incorporated
NIFI-3413: Refactored CDC into its own module(s), updated assembly and top-level POMs
NIFI-3413: Added RECEIVE prov event and Server ID property
Signed-off-by: ijokarumawak <ijokarumawak@apache.org>
This closes#1646.
Newly created async response is added before checking map size nor
purging expired ones. If there are already 100 remaining requests,
the added request will not be executed nor removed.
Currently, NiFi Kafka consumer processors have following issue.
While downstream connections are full, ConsumeKafka is not scheduled to run onTrigger.
It stopps executing poll to tell Kafka server that this client is alive.
Thus, after a while in that situation, Kafka server rebalances the client.
When downstream connections back to normal, although ConsumeKafka is scheduled again,
the client is no longer a part of a consumer group.
If this happens, Kafka client succeeds polling messages when ConsumeKafka processor resumes, but fails to commit offset.
Received messages are already committed into NiFi flow, but since consumer offset is not updated, those will be consumed again, duplicated.
In order to address above issue:
- For ConsumeKafka_0_10, use latest client library
Above issue has been addressed by KIP-62.
The latest Kafka consumer poll checks if the client instance is still valid, and rejoin the group if not, before consuming messages.
- For ConsumeKafka (0.9), added manual retention logic using pause/resume
Kafka client 0.9 doesn't have background thread heartbeat, so similar machanism is added manually.
Use Kafka pause/resume consumer API to tell Kafka server that the client stops consuming messages but is still alive.
Another internal thread is used to perform paused poll periodically based on the time passed since the last onTrigger(poll) is executed.
This closes#1527.
Signed-off-by: Bryan Bende <bbende@apache.org>
- On contextDestroyed, referencing beans created during contextInitialized to prevent successive attempts to create a bean if that bean failed to be created initially.
Signed-off-by: joewitt <joewitt@apache.org>
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3257: Added additional logging regarding timing information when replicating requests across cluster in order to glean insight as to what is taking so long when replicating some requests
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3649: Buffer node responses when replicating HTTP Requests up to a maximum buffer size
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3636: Added unit test to ensure that flowfile attribute maps are copied when appropriate
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3636: Removed patch file that should not have been in commit
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1612
Signed-off-by: Matt Burgess <mattyb149@apache.org>
Updated test to check selected column names
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1596
- Refactoring NarDetails to include all info from MANIFEST
- Adding the concept of a Bundle and refactoring NarClassLoaders to pass Bundles to ExtensionManager
- Adding logic to fail start-up when multiple NARs with same coordinates exist, moving Bundle classes to framework API
- Refactoring bundle API to classes and creating BundleCoordinate
- Updating FlowController to use BundleCoordinate
- Updating the UI and DTO model to support showing bundle details that loaded an extension type.
- Adding bundle details for processor canvas node, processor dialogs, controller service dialogs, and reporting task dialogs.
- Updating the formating of the bundle coordinates.
- Addressing text overflow in the configuration/details dialog.
- Fixing self referencing functions.
- Updating extension UI mapping to incorporate bundle coordinates.
- Discovering custom UIs through the supplied bundles.
- Adding verification methods for creating extensions through the rest api.
- Only returning extensions that are common amongst all nodes.
- Rendering the ghost processors using a dotted border.
- Adding bundle details to the flow.xml.
- Loading NiFi build and version details from the framework NAR.
- Removing properties for build and version details.
- Wiring together front end and back end changes.
- Including bundle coordinates in the component data model.
- Wiring together component data model and flow.xml.
- Addressing issue when resolve unvesioned dependent NARs.
Updating unit tests to pass based on framework changes
- Fixing logging of extension types during start up
- Allowing the application to start if there is a compatible bundle found. - Reporting missing bundle when the a compatible bundle is not found. - Fixing table height in new component dialogs.
Fixing chechstyle error and increasing test timeout for TestStandardControllerServiceProvider
- Adding ability to change processor type at runtime
- Adding backend code to change type for controller services
- Cleaning up instance classloaders for temp components.
- Creating a dialog for changing the version of a component.
- Updating the formatting of the component type and bundle throughout.
- Updating the new component dialogs to support selecting source group.
- Cleaning up new component dialogs.
- Cleaning up documentation in the cluster node endpoint.
Adding missing include in nifi-web-ui pom compressor plugin
- Refactoring so ConfigurableComponent provides getLogger() and so the nodes provide the ConfigurableComponent
- Creating LoggableComponent to pass around the component, logger, and coordinate with in the framework
- Finishing clean up following rebase.
Calling lifecycle methods for add and remove when changing versions of a component
- Introducing verifyCanUpdateBundle(coordinate) to ConfiguredComponent, and adding unit tests
- Ensuring documentation is available for all components. Including those of the same type that are loaded from different bundles.
Adding lookup from ClassLoader to Bundle, adding fix for instance class loading to include all parent NARs, and adding additional unit tests for FlowController
- Adding validation to ensure referenced controller services implement the required API
- Fixing template instantiation to look up compatible bundle
- Requiring services/reporting tasks to be disabled/stopped.
- Only supporting a change version option when the item has multiple versions available.
- Limiting the possible new controller services to the applicable API version.
- Showing the implemented API versions for Controller Services.
- Updating the property descriptor tooltip to indicate the required service requirements.
- Introducing version based sorting in the new component dialog, change version dialog, and new controller service dialog.
- Addressing remainder of the issues from recent rebase.
Ensuring bundles have been added to the flow before proposing a flow, and incorporating bundle information into flow fingerprinting
- Refactoring the way missing bundles work to retain the desired bundle if available
- Fixing logger.isDebugEnabled to be logger.isTraceEnabled
- Auditing when user changes the bundle. - Ensuring bundle details are present in templates.
Moving standard prioritizers to framework NAR and refactoring ExtensionManager logic to handle cases where an extension is in a JAR directly in the lib directory
- Ensuring all nodes attempt to instantiate the same template instance when the available bundles may differ. - Fixing the auditing of copy/paste and template instantiation. - Running addtional verification methods when running standalone.
Refactoring controller service invocation handler to allow updating the node used by the invocation handler
- Ensuring the bundles in a proposed flow are compatible with the current instance when the current instance has no flow is going to accept the proposed flow
- Merging whether multiple versions of the component are available
- Setting NAR plugin back to current released version
- Cleaning up DocGenerator to not process multiple times
Addressing incorrect usage of nf.Common. - Using formatType in the new component type dialogs.
Improving error messages when looking for bundles
Addressing comments from PR. - Fixing references to global nf namespace. - Fixing injection of nfProcessGroupConfiguration in nfComponentVersion. - Fixing web api integration tests.
Not rendering unversioned in help documentation. - Ensuring the isExtentionMissing flag is correct after changing the component type.
Adding synchronization in node classes to ensure changing component can't occur when component is running, introducing MissingBundleException for better reporting when a node can't join cluster due to a missing bundle, and bumping NAR plugin to released version 1.2.0
Adding concept of missing components to fingerprinting to ensure nodes agree on missing components when joining a cluster
NIFI-3380: NIFI-3520: - Fixing hive nar dependency. - Marking DBCPService as provided. - Skipping services that require instance classloading and are cobundled with their service API. - Skipping components that require instance classloading and reference service APIs that are cobundled. - Addressing UI issues in the new component dialogs when re-opening with a filter applied.
Fixing checkstyles issue and adding back assume checks to distributed cache server test
Ensuring new component types are sorted correctly when shown initially.
This closes#1585.
- Refactored TestGetHTTPGroovy for componentization & customization of tests.
- Ignored some TestGetHTTPGroovy tests because they cannot connect to a Jetty server running TLSv1 or TLSv1.1 (the new version of Jetty only supports TLSv1.2+). The tests are still valuable to demonstrate GetHTTP's capacity to connect to various TLS protocols, and should not be removed unless necessary.
- Refactored TestPostHTTPGroovy for componentization & customization of tests.
- Ignored some TestPostHTTPGroovy tests because they cannot connect to a Jetty server running TLSv1 or TLSv1.1 (the new version of Jetty only supports TLSv1.2+). The tests are still valuable to demonstrate PostHTTP's capacity to connect to various TLS protocols, and should not be removed unless necessary.
- Restored one of the TestGetHTTPGroovy unit tests (GetHTTP vs. TLSv1/1.1/1.2) by connecting to https://nifi.apache.org which still supports all three protocol versions (and uses JVM cacerts as truststore).
- This closes#1601
- Preventing the editing of controller services that are defined in an ancestor process group.
- Adding a go to link for users to easily navigate to where services are actually defined.
This closes#1594
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
- Adding notes to the controller service tables to better define their availability.
- Clarifying support of Safari.
This closes#1593
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
- Updating Controller Service scope header.
- Rendering the name of the parent when possible.
- Fixing the check if the parent Process Group is write-able when the parent is selected on the current canvas.
This closes#1590
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
- Removed NAR dependency on nifi-hadoop-libraries-nar from nifi-hadoop-nar so that hadoop-client dependencies will be included directly in nifi-hadoop-nar
- Added RequiresInstanceClassLoading annotation to AbstractHadoopProcessor and HiveConnectionPool
- UGI relogins are now performed using doAs
- Added debug-level logging for UGI relogins in KerberosTicketRenewer and AbstractHadoopProcessor
This closes#1539.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Verify that when moving a snippet, we do not attempt to move a process group into itself.
- Update the UI to ensure a subsequent move event cannot be triggered while a previous move event is still in progress.
This closes#1582.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3481 Added support for Database Adapter Descriptions
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3481 Updated Adapter Name
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1510
- Added timeout, yieldPeriod, transportProtocol, proxyHost, proxyPort,
proxyUser and proxyPassword
- Removed unused fingerprint methods
FingerprintFactory has two types of fingerprinting method, from XML
elements and from DTO. However, the ones from DTO are not used by
anywhere. IDE didn't report those private methods unused because
addProcessGroupFingerprint and addSnippetFingerprint call each other,
but those are not used from outside actuallly.
This commit removes those private methods to keep the class clean to
avoid unnecessary code maintenance and tests.
This closes#1332.
* Add test for failure not clear context map.
* Add handler to remove context map entry if ProcessException occurs during while exporting response.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#1567.
- Support multiplle incoming FlowFiles to Wait processor, up to Wait
Buffer Count
- Added Releasable FlowFile Count, which controls how many FlowFiles can
be released when wait condition is met
- Added special meaning to Notify delta Zero(0) to clear a signal
counter back to zero
This closes#1554
Signed-off-by: Aldrin Piri <aldrin@apache.org>
- The idea behind NIFI-3356 was to improve the efficiency and throughput of the Provenance Repository, as it is often the bottleneck. While testing the newly designed repository,
a handful of other, fairly minor, changes were made to improve efficiency as well, as these came to light when testing the new repository:
- Use a BufferedOutputStream within StandardProcessSession (via a ClaimCache abstraction) in order to avoid continually writing to FileOutputStream when writing many small FlowFiles
- Updated threading model of MinimalLockingWriteAheadLog - now performs serialization outside of lock and writes to a 'synchronized' OutputStream
- Change minimum scheduling period for components from 30 microseconds to 1 nanosecond. ScheduledExecutor is very inconsistent with timing of task scheduling. With the bored.yield.duration
now present, this value doesn't need to be set to 30 microseconds. This was originally done to avoid processors that had no work from dominating the CPU. However, now that we will yield
when processors have no work, this results in slowing down processors that are able to perform work.
- Allow nifi.properties to specify multiple directories for FlowFile Repository
- If backpressure is engaged while running a batch of sessions, then stop batch processing earlier. This helps FlowFiles to move through the system much more smoothly instead of the
herky-jerky queuing that we previously saw at very high rates of FlowFiles.
- Added NiFi PID to log message when starting nifi. This was simply an update to the log message that provides helpful information.
NIFI-3356: Fixed bug in ContentClaimWriteCache that resulted in data corruption and fixed bug in RepositoryConfiguration that threw exception if cache warm duration was set to empty string
NIFI-3356: Fixed NPE
NIFI-3356: Added debug-level performance monitoring
NIFI-3356: Updates to unit tests that failed after rebasing against master
NIFI-3356: Incorporated PR review feedback
NIFI-3356: Fixed bug where we would delete index directories that are still in use; also added additional debug logging and a simple util class that can be used to textualize provenance event files - useful in debugging
This closes#1493
The 'exec' command added by NIFI-2689 affected restart behavior
negatively as 'exec' command will not execute subsequent commands in the
shell script.
This commit changes 'exec' is added only when 'run' is specified.
This closes#1523.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
Added support for simple Key/Value Schema Registry as Controller Service
Added support for registering multiple schemas as dynamic properties of Schema Registry Controller Service
Added the following 8 processors
- ExtractAvroFieldsViaSchemaRegistry
- TransformAvroToCSVViaSchemaRegistry
- TransformAvroToJsonViaSchemaRegistry
- TransformCSVToAvroViaSchemaRegistry
- TransformCSVToJsonViaSchemaRegistry
- TransformJsonToAvroViaSchemaRegistry
- TransformJsonToCSVViaSchemaRegistry
- UpdateAttributeWithSchemaViaSchemaRegistry
polishing
NIFI-3354 Adding support for HDFS Schema Registry, unions and default values in the Avro Schema and NULL columns in the source CSV
NIFI-3354 Adding support for logicalTypes per the Avro 1.7.7 spec
NIFI-3354 polishing and restructuring CSVUtils
NIFI-3354 renamed processors to address PR comment
NIFI-3354 addressed latest PR comments
- removed HDFS-based ControllerService. It will be migrated into a separate bundle as a true extension.
- removed UpdateAttribute. . . processor
- added mime.type attribute to all Transform* processors
NIFI-3354 added missing L&N entries
This closes pr/1436
Before this fix, files with the latest timestamp within a listing
iteration are always be held back one cycle no matter how old it is.
Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
- Add 'nifi.flow.configuration.archive.max.count' in nifi.properties
- Change default archive limit so that it uses archive max time(30 days)
and storage (500MB) if no limitation is specified
- Simplified logic to delete old archives
This closes#1460.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Added Signal Counter Delta property
- Added Signal Buffer Count property
- Added processor property name and display name
- Changed IOException handling from routing it to failure to throw
RuntimeException, so that NiFi framework can yield the processor for a while and try again
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#1466.
- Added detailed description about how the URL property works with
GetHTMLElement
- Added Expression support with URL
- Made URL property dynamic with ModifyHTMLElement and PutHTMLElement,
since it won't be used to alter HTML element and need not to be
specified. Making it a dynamic property let existing processor configuration stays valid
* add exec to RUN_NIFI_CMD
* remove subshell for else
* tested compatible with runit with these changes
This closes#966.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
* Corrected handling of corrupt journal file records that prevents instance startup and loss of records from corrupt files. Specifically, exception handling was expanded to cover failures on records after the first the same as failures on the first record.
* Adjusted log messages to reflect that the remainder or all of the journal will be skipped, not just the current record.
This closes#1485.
H2 and Kafka broker uses the same default port 9092.
If an user is running Kafka broker on the same machine, or run the unit
tests in parallel, DBCPServiceTest can fail since some of its test
methods connects to port 9092.
This closes#1504.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
* Remove function based on JDK source.
* Add new function to find bytes based on RFC3629.
* Add field name to log entry when field is truncated.
Signed-off-by: Mike Moser <mosermw@apache.org>
This closes#1475
* Credentials service with tests
* Abstract processor definitions
* GCS-themed processors and their corresponding tests
Signed-off-by: James Wing <jvwing@gmail.com>
This closes#1482.
use the FileNameFilter when not passing down explit jar paths
Filter out ^. files when reading lists of files from directories
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Added configure audits for Transport Protocol, HTTP Proxy Server Host,
Port, User and Password in RemoteProcessGroup configuration
- Added configure audits for enabling/disabling individual remote port
- Added configure audits for Concurrent Tasks and Compressed in Remote
Port configuration
- This closes#1476
* Updated StandardRecordWriter, even though it is now deprecated to consider the encoding behavior of java.io.DataOutputStream.writeUTF() and truncate string values such that the UTF representation will not be longer than that DataOutputStream's 64K UTF format limit.
* Updated the new SchemaRecordWriter class to similarly truncate long Strings that will be written as UTF.
* Add tests to confirm handling of large UTF strings and various edge conditions of UTF string handling.
Signed-off-by: Mike Moser <mosermw@apache.org>
This closes#1469.
- Marked PutKafka Partition Strategy property as deprecated, as Kafka 0.8 client doesn't use 'partitioner.class' as producer property, we don't have to specify it.
- Changed Partition Strategy property from a required one to a dynamic property, so that existing processor config can stay in valid state.
- Fixed partition property to work.
- Route a flow file if it failed to be published due to invalid partition.
This closes#1425
NIFI-2615 Addressing changes from P/R. Specifically, removing .gitignore as it should not be there for a nar. Removed non-used class. Changed name in notice
- Requiring WRITE permissions to the parent resource when attempting to remove a component.
- Updating expired certificates in the REST API integration tests.
This closes#1399.
Signed-off-by: James Wing <jvwing@gmail.com>
- Support counters at Wait/Notify processors so that NiFi flow can be
configured to wait for N signals
- Extract Wait/Notify logics into WaitNotifyProtocol
- Added FragmentAttributes to manage commonly used fragment attributes
- Changed existing split processors to set 'fragment.identifier' and
'fragment.count', so that Wait can use those to wait for all splits
get processed
This closes#1420.
Signed-off-by: Bryan Bende <bbende@apache.org>
Add support in SelectHiveQL to get script content from the Flow File to bring consistency with patterns used for PutHiveQL and support extra query management.
Changed behavior of using Flowfile to match ExecuteSQL. Handle query delimiter when embedded. Added test case for embedded delimiter
Formatting and License Header
PutHiveQL and SelectHiveQL Processor enhancements. Added support for multiple statements in a script. Options for delimiters, quotes, escaping, include header and alternate header.
Add support in SelectHiveQL to get script content from the Flow File to bring consistency with patterns used for PutHiveQL and support extra query management.
Changed behavior of using Flowfile to match ExecuteSQL. Handle query delimiter when embedded. Added test case for embedded delimiter
Removing dead code.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
Comments to Clarify test case.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
Final whitespace/formatting/typo changes
This closes#1316
- Adding additional parameters to be able to limit the size of the provenance response. Specifically, whether the events should be summarized and whether events should be returned incrementally before the query has completed.
- Ensuring the cluster node address is included in provenance events returned.
- Ensuring there is a cluster coordinator before attempting to get the cluster node address.
- Removing exponential back off between provenance requests.
- Ensuring the content viewer url is retrieve before initializing the provenance table.
This closes#1413.
- Using fetch and replace together can provide optimistic locking for
concurrency control.
- Added fetch to get cache entry with its meta data such as revision
number.
- Added replace to update cache only if it has not been updated.
- Added Map Cache protocol version 2 for those new operations.
- Existing operations such as get or put can work with protocol version
1.
This closes#1410.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-3004 Added logic to expire StandardSSLContextService customValidate cache after 5 invocations.
Updated unit test to demonstrate this logic.
This closes#1375.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Removing unnecessary authorization check during second phase of connection creation.
- Ensuring that the remote group port returns the correct resource type though not super critical since it is not possible to create policies for remote ports.
This closes#1353.
- Fixing the importUri when replicating a Template upload request. Previously was leveraging getResourceUri which considers proxy headers meant for client responses.
This closes#1334.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Removed host and port field from Peer since the same information is
available in PeerDescription
- Refactored variable names in SocketRemoteSiteListener to improve readability
- Changed how SocketRemoteSiteListener constructs PeerDescription
instance. It used to use hard-coded 'localhost' as hostname, and
getPort() which returns server's port. Since the peer is a remote peer,
i.e the client, it should be client hostname and port.
- Added hostname resolution at DataTransferResource to make s2s.host
value consistent with RAW transport. Without this, RAW uses hostname
while HTTP uses IP address. It will be hard to be used from downstream flows.
- Replaced heavy use of mockito which was difficult to maintain, with
nifi-mock
- Added SiteToSiteAttributes and more assertions in unit tests
This closes#1342.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Ensure that controller services are added to child process groups when creating snippet
- Addressed issue related to modifying higher-level process groups' controller services in snippet after having already visited the process group
This closes#1318
Signed-off-by: jpercivall <JPercivall@apache.org>
NIFI-1582 removing the option to use cluster state
NIFI-1582 addressing Oleg's comments
NIFI-1582 No longer forcing numbers as the init value and adding getStateValue() to EL instead of using 'ATTRIBUTE_state'
NIFI-1582 Removing init state value
NIFI-1582 Adding documentation for the changes to Init State value
This closes#319
NIFI-3133:
- Using getSourceAuthorizable() when accessing flow files and content.
NIFI-3133:
- Decouple local and remote connectable's to avoid ambiguity with self referencing RPGs.
NIFI-3133:
- Addressing comments from the PR.
NIFI-3133:
- Fixed check verifying source/destination when creating a connection.
NIFI-3133:
- Only showing the go to link when the source component is not a remote port.
NIFI-3133:
- Removing unnecessary checking of remote group port authorization since it's handled by the parent RPG.
NIFI-3133:
- Fixing issue showing the connection details dialog when the source component is a RPG.
NIFI-3133:
- Ensuring the local connectable was found.
This closes#1297
Signed-off-by: jpercivall <JPercivall@apache.org>
- Added urls in addition to the existing url, to support multiple target
URLs
- Backward compatibility is provided by returning the first url if
multipe urls are specified, but component accessing the url doesn't
support multiple urls
- UI is not fully updated yet. Following UI components are planned to be updated
by different commits
- Search component: only the first URL is searchable and shown
- Component status: RPG status shows only the first URL
- Component action history: only the first URL is searchable and shown
- Updated Search component to use URLs.
This closes#1208.
- Updating REST API docs accordingly.
- Adding some integration tests to exercise the additional restricted component checks.
This closes#1287.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Added unit test cases to simulate NiFi version update which fails without this fix.
- Added state object migration code, add file.0. prefix to state keys,
and add length from stored position.
This closes#1289
Fixed merging logic for root group status. Only consider a port transmitting when there is an active request, not when there is a 'queued' request
This closes#1279
- Updating the SnippetManager to automatically expire unused snippets.
- Making the checking of referenced services consistent across component removal.
- Adding checking of referenced services for all nested components in a snippet.
- Updating the REST API docs to clarify when a referenced service is authorized.
- Conditionally authorizing transitive service references.
This closes#1277.
Signed-off-by: Bryan Bende <bbende@apache.org>
- validate processors only when they are in STOPPED state
- report validation errors via REST API on processors/services/tasks/ports only when they are in the STOPPED state
- This closes#1192
- Addressing comments from PR.
- Adding restricted tags to relevant components.
- Showing a restricted icon overlay on the processor node on the canvas. (+1 squashed commit)
Squashed commits:
[f487682] NIFI-3050:
- Introducing a Restricted annotation for components that require elevated privileges to use.
- Updating the new Processor, Controller Service, and Reporting Task dialogs to include these details and prevent unauthorized selection.
- Including the Restricted description in the generated component documentation.
- Updating processor access control integration test to verify restricted component creation.
- Updating the developer, user, and admin guide to include the restricted component policy.
This closes#1247.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-2750
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-2750 Tweaking Property Retrieval
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1001
- Introducing a strategy for identifying users.
- Fixing issue with the referral strategy error message.
- Adding code to shutdown the application when the authorizer or login identity provider are not initialized successfully.
NIFI-3020:
- Updating the admin guide to document the identity strategy.
NIFI-3020:
- Ensuring the request replicator attempts to shutdown regardless of whether the flow service properly terminates.
This closes#1236
NIFI-2854: Incorporated PR review feedback
NIFI-2854: Implemented feedback from PR Review
NIFI-2854: Ensure that all resources are closed on CompressableRecordReader.close() even if an IOException is thrown when closing one of them
This closes#1202
- Returning the appropriate authorizable when accessing provenance events for a manual DROP event by emptying a queue.
- Populating the component details of a provenance event when the source is a connection.
NIFI-1002: Added WebSocket support.
- Reflecting review comments
- Added displayName to peroperty descriptors
NIFI-1002: Added WebSocket support.
This closes#1184
- Reflecting review comments:
- Removed unnecessary use of NarCloseable.withComponentNarLoader.
- Removed intermediate on memory queue to make it simpler and more
robust. Received messages in WebSocket layer now will be transferred
to downstream relationships directly.
Squashed commits:
[5dd22a9] NIFI-2654 Updated administration guide with login-identity-providers.xml flags.
Exposed master key retrieval code in NiFiPropertiesLoader.
Added logic to decrypt login identity providers XML configuration.
Updated login-identity-providers.xsd to include encryption scheme attribute.
Added unit tests. (+18 squashed commits)
Squashed commits:
[57c815f] NIFI-2654 Resolved issue where empty LIP property elements could not be encrypted.
Added unit test and resource.
[27d7309] NIFI-2654 Wired in serialization logic to write logic for LIP.
Added comprehensive unit test for LIP & NFP in same test.
[b450eb2] NIFI-2654 Finalized logic for preserving comments in LIP parsing.
[5aa6c9c] NIFI-2654 Added logic for maintaining XML formatting (comments and whitespace) for LIP.
Added unit tests (w/o encryption works; w/ does not).
[b53461f] NIFI-2654 Added unit test for full tool invocation migrating a login-identity-providers.xml file and updating file and bootstrap.conf with key.
[2d9686c] NIFI-2654 Updated tool description and various logging statements.
Added unit test for full tool invocation encrypting a login-identity-providers.xml file and updating file and bootstrap.conf with key.
[8c67cb2] NIFI-2654 Added logic to encrypt LIP XML content.
Added unit tests.
[8682d19] NIFI-2654 Added logic to handle "empty" (commented) LIP files.
Added unit tests.
[077230e] NIFI-2654 Fixed logic to decrypt multiline and multiple-per-line XML elements.
Added unit tests and resources.
[d5bb8da] NIFI-2654 Ignored unit test for unreadable conf directory because directory was causing Maven build issues.
Removed test resources.
[7e50506] NIFI-2654 Fixed AESSensitivePropertyProvider bug handling cipher text with whitespace.
Added unit test.
[b69a661] NIFI-2654 Fixed AESSensitivePropertyProviderFactoryTest to reflect absence of key causes errors.
[6f821b9] NIFI-2654 Added standard password to arbitrary encryption test for use in test resources.
[d289ffa] NIFI-2654 Added LIP XML decryption.
Added unit tests.
[a482245] NIFI-2654 Added LIP test resources.
[7204df4] NIFI-2654 Changed logic to only perform properties encryption when file path is provided.
[729e1df] NIFI-2654 Removed population of default file locations for bootstrap.conf, nifi.properties, and login-identity-providers.xml as not all files may be desired.
Added/updated unit tests.
[7dba5ef] NIFI-2654 Started LIP work (arguments & parsing).
Added unit tests.
Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>
This closes#1216
Added unit tests and resources.
NIFI-3032 Fixed bug in AESSensitivePropertyProvider#getIdentifierKey where the result was always the max available key size, not the size of the current key.
Added unit test.
This closes#1220
Signed-off-by: Bryan Rosander <brosander@apache.org>
DefaultSettings annotation should be use on Custom Processor to set the default penalty period, the yield duration or the bulletin log level for each instance of the processor
- introduced org.apache.nifi.stream.io.util.TextLineDemarcator
- refactored SplitText to use org.apache.nifi.stream.io.util.TextLineDemarcator
- updated SplitText's capability discription to provide more clarity around splits with headers.
1 - Replace a r/w test over $NIFI_HOME/lib that is performed
but never utilised by a RO test
2 - Rename ensureDirectoryExistAndCanAccess to
ensureDirectoryExistAndCanReadAndWrite and deprecate the
former
- Removing legacy code when retrieving site to site details when the target instance is this local NiFi.
- Ensures the UI accurately reflects available remote ports.
NIFI-2603:
- Fixing color of remote output port status in connection label.
This closes#1189.
NIFI-1712 Applying per-instance class loading to HBaseClientService to allow specifying Phoenix Client JAR
-Refactoring the ClassLoading so that every processor, controller service, and reporting task gets an InstanceClassLoader with a parent of the NAR ClassLoader, and only components with @RequiresInstanceClassLoading will make a copy of the NAR ClassLoader resources, and addressing some review feedback
This closes#1156
- Updating the content of the delete policy dialog to make it a little more clear that the permissions for the component will revert back to the inherit policy.
This closes#1183.
- Ensuring the selection is cleared when going to the Process Group where the selected component policy is defined.
- Ensure that the selection context is updated when going to a component or group.
This closes#1174.
* Java version and vendor
* OS name and version
* Release Tag
* Build revision (commit SHA), branch, and timestamp
* Handles formal releases, ad-hoc builds, and non-release source builds
* Standalone UI presence in About dialog, Summary -> System Diagnostics
* Cluster UI as Versions tab in Cluster dialog
* Reduce About Dialog Content
* Fix Missing Property Display Bugs
* Marking the build time as type string.
* This closes#583
Mark Payne
Bryan Bende
Andre F de Miranda
Jon Anderson III
Joe Gresock
Pierre Villard
Bryan Rosander
Matt Burgess
Andrew Lim
joewitt
Koji Kawamura
Yolanda M. Davis
Andy LoPresto
Anders Breindahl
Scott Aslan
Matt Gilman
Jeremy Dyer
ShellyLC
Oleg Zhurakousky
Toivo Adams
Joey Frazee
Joe Percivall
Francois Prunier
Jack Pickett
kedarchitale
Gahan Rakholia
James Wing
Puspendu Banerjee
Jeremy Dyer
ambud
Jeff Storck
Peddy
patricker
Aldrin Piri
Joe Skora
Pere Urbon-Bayes
Joe Trite
Giovanni Lanzani
josephxsxn
Nick Carenza
JJ
João Henrique Ferreira de Freitas
Hunter Morgan
Simon Elliston Ball
Mike Moser
Joe Skora
Selim Namsi
Gene Peters
Otto Fowler
Ryan Persaud
Giovanni Lanzani
Ben Schofield
Andrew Psaltis
David W. Streever
afucs
Davy De Waele
Randy Gelhausen
Wesley-Lawrence
Byunghwa Yun
jpercivall
Joey Frazee
Kathik Narayanan
Peter Wicks
Mathias Tiberghien
Brian Eugley
Joe N
simplesteph