NIFI-5943 Added another unit test to verify list + map conversion to list of records. (Mike Thomsen)
This closes#3267
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
NIFI-5903: Removed TODO comments that were done
NIFI-5903: Added support for working with MAP types to QueryRecord and associated RPATH functions
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3223
- Updates to make UpdateRecord and RecordPath automatically update Record schema when performing update and perform the updates on the first record in UpdateRecord before obtaining Writer Schema. This allows the Writer to to inherit the Schema of the updated Record instead of the Schema of the Record as it was when it was read.
- Updated JoltTransformRecord so that schema is inferred on the first transformed object before passing the schema to the Record Writer, so that if writer inherits schema from record, the schema that is inherited is the trans transformed schema
- Updated LookupRecord to allow for Record fields to be arbitrarily added
- Implemented ContentClaimInputStream
- Added controller service for caching schemas
- UpdatedQueryRecord to cache schemas automatically up to some number of schemas, which will significantly inprove throughput in many cases, especially with inferred schemas.
NIFI-5938: Updated AvroTypeUtil so that if creating an Avro Schema using a field name that is not valid for Avro, it creates a Schema that uses a different, valid field name and adds an alias for the given field name so that the fields still are looked up appropriately. Fixed a bug in finding the appropriate Avro field when aliases are used. Updated ContentClaimInputStream so that if mark() is called followed by multiple calls to reset(), that each reset() call is successful instead of failing after the first one (the JavaDoc for InputStream appears to indicate that the InputStream is free to do either and in fact the InputStream is even free to allow reset() to reset to the beginning of file if mark() is not even called, if it chooses to do so instead of requiring a call to mark()).
NIFI-5938: Added another unit test for AvroTypeUtil
NIFI-5938: If using inferred schema in CSV Reader, do not consider first record as a header line. Also addressed a bug in StandardConfigurationContext that was exposed by CSVReader, in which calling getProperty(PropertyDescriptor) did not properly lookup the canonical representation of the Property Descriptor from the component before attempting to get a default value
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3253
NIFI-5406 introduced the issue by trying to use the resetState variable for
different purposes. AbstractListProcessor should have had a different variable
to control whether to clear state for tracking entity strategy.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#3189.
- Updated FlowFile Repo / Write Ahead Log so that any update that writes more than 1 MB of data is written to a file inside the FlowFile Repo rather than being buffered in memory
- Update SplitText so that it does not hold FlowFiles that are not the latest version in heap. Doing them from being garbage collected, so while the Process Session is holding the latest version of the FlowFile, SplitText is holding an older version, and this results in two copies of the same FlowFile object
NIFI-5533: Checkpoint
NIFI-5533: Bug Fixes
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2974
Added regression test for ProvenanceEventConsumer#isFilteringEnabled().
Changed isFilteringEnabled implementation to be expandable as other attributes are added using Streams.
EL + indentation.
This closes#2973.
Co-authored-by: Andy LoPresto <alopresto@apache.org>
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Adding followings:
- Use separate DistributedMapCache for tracking entities to avoid
conflict with existing code
- Added more validation
- Delete listed entities from cache if reset is needed
- Support Local scope
- Added Initial Listing Target
This closes#2876.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Create nifi-syslog-utils to move syslog parsing functionalty to a central location shared by the processors and serialization/record system.
- Refactor Processors to use these utils
- Update 5424 syslog classes using simple-syslog-5424 to pick up new changes to support this work, as well as keep dependencies/types from bleeding out to the
processors or readers
- Refactor Syslog5424Event and Parser
- Create Syslog5424RecordReader
- per review, handle blank message differently from eof
- name schema per review
This closes#2816.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-5059 Changed it to use a schema registry.
NIFI-5059 Updated MongoDBLookupService to be a SchemaRegistryService.
NIFI-5059 Added two changes from a code review.
NIFI-5059 Fixed two bad references.
NIFI-5059 Refactored schema strategy handling.
NIFI-5059 Moved schema strategy handling to JsonInferenceSchemaRegistryService.
NIFI-5059 Updated to use new LookupService method.
NIFI-5059 fixed schema inference bug.
NIFI-5059 Added test for schema text strategy
NIFI-5059 incremented version number to make the build work.
NIFI-5059 fixed a stray 1.7.0 reference.
NIFI-5059 Added getDatabase to client service.
NIFI-5059 Added changes requested in a code review.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2619
NIFI-5041: fixes http client version issue
Change-Id: I1b87ec4752ff6e1603025883a72113919aba5dd4
NIFI-5041: fixes Kerberos configuration
Change-Id: I868fdf3ea7cfd28cf415164e420f23bf3f6eefeb
NIFI-5041: adds new NOTICE entries
NIFI-5041: yields processor if no session is available, fixes error handling in session manager thread, fixes error returned in KerberosKeytabSPNegoScheme on authentication failure
Change-Id: I443e063ae21c446980087e5464a4b70373d730f6
NIFI-5041: makes the session manager thread exceptions visible to the users
Change-Id: I33fde5df6933cec2a87a4d82e681d4464f21b459
NIFI-5041: adds special SessionManagerException to identify error occurred on session manager thread
Change-Id: I25a52c025376a0cd238f14bda533d6f5f3e5fb4a
This closes#2630
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
NIFI-950: Still seeing some slow response times when instantiating a large template in cluster mode so making some minor tweaks based on the results of CPU profiling
NIFI-5112: Refactored FlowSerializer so that it creates the desired intermediate data model that can be serialized, separate from serializing. This allows us to hold the FlowController's Read Lock only while creating the data model, not while actually serializing the data. Configured Jersey Client in ThreadPoolRequestReplicator not to look for features using the Service Loader for every request. Updated Template object to hold a DOM Node that represents the template contents instead of having to serialize the DTO, then parse the serialized form as a DOM object each time that it needs to be serialized.
NIFI-5112: Change ThreadPoolRequestReplicator to use OkHttp client instead of Jersey Client
NIFI-5111: Ensure that if a node is no longer cluster coordinator, that it clears any stale heartbeats.
NIFI-5110: Notify StandardProcessScheduler when a component is removed so that it will clean up any resource related to component lifecycle.
NIFI-950: Avoid gathering the Status objects for entire flow when we don't need them; removed unnecessary code
NIFI-950: Bug fixes
NIFI-950: Bug fix; added validation status to ProcessorDTO, ControllerServiceDTO, ReportingTaskDTO; updated DebugFlow to allow for pause time to be set in the customValidate method for testing functionality
NIFI-950: Addressing test failures
NIFI-950: Bug fixes
NIFI-950: Addressing review feedback
NIFI-950: Fixed validation logic in mock framework
This closes#2693
- Forcing FileSystem statistics thread to be interrupted when HDFS processors are stopped
- Stop creating temp components during import from registry, use bundle info instead
This closes#2668.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Fixed dependency issue by providing a local JSON reader
Rebased + fixed conflict + updated versions in pom + EL scope
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2575
Joe Witt
Matthew Burgess
thenatog
Otto Fowler
Denes Arvay
Mark Payne
Endre Zoltan Kovacs
Alex Savitsky
Bryan Bende
Peter Wicks
Koji Kawamura
gkkorir
Arek Burdach
Ed B
Jeff Storck
Pierre Villard
Mike Thomsen
Andy LoPresto
zenfenan
Peter Toth
Max Viazovskyi
JohannesDaniel
sbouchex