- Added timeout, yieldPeriod, transportProtocol, proxyHost, proxyPort,
proxyUser and proxyPassword
- Removed unused fingerprint methods
FingerprintFactory has two types of fingerprinting method, from XML
elements and from DTO. However, the ones from DTO are not used by
anywhere. IDE didn't report those private methods unused because
addProcessGroupFingerprint and addSnippetFingerprint call each other,
but those are not used from outside actuallly.
This commit removes those private methods to keep the class clean to
avoid unnecessary code maintenance and tests.
This closes#1332.
* Add test for failure not clear context map.
* Add handler to remove context map entry if ProcessException occurs during while exporting response.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#1567.
- Support multiplle incoming FlowFiles to Wait processor, up to Wait
Buffer Count
- Added Releasable FlowFile Count, which controls how many FlowFiles can
be released when wait condition is met
- Added special meaning to Notify delta Zero(0) to clear a signal
counter back to zero
This closes#1554
Signed-off-by: Aldrin Piri <aldrin@apache.org>
- The idea behind NIFI-3356 was to improve the efficiency and throughput of the Provenance Repository, as it is often the bottleneck. While testing the newly designed repository,
a handful of other, fairly minor, changes were made to improve efficiency as well, as these came to light when testing the new repository:
- Use a BufferedOutputStream within StandardProcessSession (via a ClaimCache abstraction) in order to avoid continually writing to FileOutputStream when writing many small FlowFiles
- Updated threading model of MinimalLockingWriteAheadLog - now performs serialization outside of lock and writes to a 'synchronized' OutputStream
- Change minimum scheduling period for components from 30 microseconds to 1 nanosecond. ScheduledExecutor is very inconsistent with timing of task scheduling. With the bored.yield.duration
now present, this value doesn't need to be set to 30 microseconds. This was originally done to avoid processors that had no work from dominating the CPU. However, now that we will yield
when processors have no work, this results in slowing down processors that are able to perform work.
- Allow nifi.properties to specify multiple directories for FlowFile Repository
- If backpressure is engaged while running a batch of sessions, then stop batch processing earlier. This helps FlowFiles to move through the system much more smoothly instead of the
herky-jerky queuing that we previously saw at very high rates of FlowFiles.
- Added NiFi PID to log message when starting nifi. This was simply an update to the log message that provides helpful information.
NIFI-3356: Fixed bug in ContentClaimWriteCache that resulted in data corruption and fixed bug in RepositoryConfiguration that threw exception if cache warm duration was set to empty string
NIFI-3356: Fixed NPE
NIFI-3356: Added debug-level performance monitoring
NIFI-3356: Updates to unit tests that failed after rebasing against master
NIFI-3356: Incorporated PR review feedback
NIFI-3356: Fixed bug where we would delete index directories that are still in use; also added additional debug logging and a simple util class that can be used to textualize provenance event files - useful in debugging
This closes#1493
The 'exec' command added by NIFI-2689 affected restart behavior
negatively as 'exec' command will not execute subsequent commands in the
shell script.
This commit changes 'exec' is added only when 'run' is specified.
This closes#1523.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
Added support for simple Key/Value Schema Registry as Controller Service
Added support for registering multiple schemas as dynamic properties of Schema Registry Controller Service
Added the following 8 processors
- ExtractAvroFieldsViaSchemaRegistry
- TransformAvroToCSVViaSchemaRegistry
- TransformAvroToJsonViaSchemaRegistry
- TransformCSVToAvroViaSchemaRegistry
- TransformCSVToJsonViaSchemaRegistry
- TransformJsonToAvroViaSchemaRegistry
- TransformJsonToCSVViaSchemaRegistry
- UpdateAttributeWithSchemaViaSchemaRegistry
polishing
NIFI-3354 Adding support for HDFS Schema Registry, unions and default values in the Avro Schema and NULL columns in the source CSV
NIFI-3354 Adding support for logicalTypes per the Avro 1.7.7 spec
NIFI-3354 polishing and restructuring CSVUtils
NIFI-3354 renamed processors to address PR comment
NIFI-3354 addressed latest PR comments
- removed HDFS-based ControllerService. It will be migrated into a separate bundle as a true extension.
- removed UpdateAttribute. . . processor
- added mime.type attribute to all Transform* processors
NIFI-3354 added missing L&N entries
This closes pr/1436
Before this fix, files with the latest timestamp within a listing
iteration are always be held back one cycle no matter how old it is.
Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
- Add 'nifi.flow.configuration.archive.max.count' in nifi.properties
- Change default archive limit so that it uses archive max time(30 days)
and storage (500MB) if no limitation is specified
- Simplified logic to delete old archives
This closes#1460.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Added Signal Counter Delta property
- Added Signal Buffer Count property
- Added processor property name and display name
- Changed IOException handling from routing it to failure to throw
RuntimeException, so that NiFi framework can yield the processor for a while and try again
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#1466.
- Added detailed description about how the URL property works with
GetHTMLElement
- Added Expression support with URL
- Made URL property dynamic with ModifyHTMLElement and PutHTMLElement,
since it won't be used to alter HTML element and need not to be
specified. Making it a dynamic property let existing processor configuration stays valid
* add exec to RUN_NIFI_CMD
* remove subshell for else
* tested compatible with runit with these changes
This closes#966.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
* Corrected handling of corrupt journal file records that prevents instance startup and loss of records from corrupt files. Specifically, exception handling was expanded to cover failures on records after the first the same as failures on the first record.
* Adjusted log messages to reflect that the remainder or all of the journal will be skipped, not just the current record.
This closes#1485.
H2 and Kafka broker uses the same default port 9092.
If an user is running Kafka broker on the same machine, or run the unit
tests in parallel, DBCPServiceTest can fail since some of its test
methods connects to port 9092.
This closes#1504.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
* Remove function based on JDK source.
* Add new function to find bytes based on RFC3629.
* Add field name to log entry when field is truncated.
Signed-off-by: Mike Moser <mosermw@apache.org>
This closes#1475
* Credentials service with tests
* Abstract processor definitions
* GCS-themed processors and their corresponding tests
Signed-off-by: James Wing <jvwing@gmail.com>
This closes#1482.
use the FileNameFilter when not passing down explit jar paths
Filter out ^. files when reading lists of files from directories
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Added configure audits for Transport Protocol, HTTP Proxy Server Host,
Port, User and Password in RemoteProcessGroup configuration
- Added configure audits for enabling/disabling individual remote port
- Added configure audits for Concurrent Tasks and Compressed in Remote
Port configuration
- This closes#1476
Aldrin Piri
James Wing
Andre F de Miranda
Andy LoPresto
Pierre Villard
Koji Kawamura
Joe Skora
jpercivall
Matt Gilman
Mark Payne
Pere Urbon-Bayes
Joe Trite
Scott Aslan
Giovanni Lanzani
josephxsxn
patricker
Bryan Rosander
Oleg Zhurakousky
Nick Carenza
JJ
Jeff Storck
João Henrique Ferreira de Freitas
Bryan Bende
Hunter Morgan
Simon Elliston Ball
Matt Burgess
Mike Moser
Joe Skora
Selim Namsi
joewitt
Yolanda M. Davis
Gene Peters
Otto Fowler
Ryan Persaud