Add the following functionalities:
- Custom value separator (default is comma)
- Custom quote char to use (default is " i.e. quote sign)
- Quote mode
- Escape character to use (default is no escape character)
- Comment marker
- Trim fields
- Character set to use
The above features use a common implementation with "CSVReader".
Also append a sentence to the capability description that first line of
csv file is considered header. Setting custom header instead of using
the first line is not supported (yet).
Also, a minor refactor: CSVRecordLoopkupService and
SimpleCsvFileLookupService now share common logic in implementation.
CSV Format is extended to the same list as CSVReader. In addition,
lookup services still have the "default" csv format for compatibility
reasons.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4494.
The TikaInputStream and FileInputStream instances utilized in IdentifyMimeType are now explicitly closed. The FileInputStream is additionally wrapped by a BufferedInputStream.
Signed-off-by: Joe Witt <joewitt@apache.org>
Cleaned up JettyServer code.
Changed test logging severity to include debug statements.
Added test resources.
This closes#4498.
Co-authored-by: Kotaro Terada <kotarot@apache.org>
Updated the patch based on @tpalfy's review
Updated the patch based on @mattyb149's review
Rename DATE_FORMAT_PATTERN to JSON_TIMESTAMP_FORMAT_PATTERN
Changed convertToJsonStream method's visibility to package private.
Removed json prefix from timestamp-format-pattern property to make it more generic
This closes#4463.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
Because even though permissions are only the 7 least significant bits of the file
mode but the file mode can be wider and can contain further info (like the
sticky bit).
Extend unit test for converting file mode with sticky bit into 'rwx' style
permission string.
Remove old test cases
This closes#4490.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
NIFI-7729: Updated docs to explain how to add new fields to Records via ScriptedTransformRecord and added example. Fixed checkstyle violation.
NIFI-7729: Always call Record.incorporateInactiveFields with ScriptedTransformRecord
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4470
NIFI-7663 Minor changes (variable name refactor, javadoc, GUI message). Merging Drop All Flowfiles responses across all nodes in a cluster.
NIFI-7663 Reloading the canvas after completing a Drop All Flowfiles request.
NIFI-7663 Fixed typos.
This closes#4425.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
Also added validator for Directory Name property in AbstractAzureDataLakeStorageProcessor
Fix Tracking Entities strategy: use milliseconds for lastModified
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4438.
Disable connection automatic recovery which can lead to uncontrolled/stale threads. Handle the recovery in the processors instead.
Use poisoning in case of errors, then discarding and recreating the poisoned consumer/publisher.
NIFI-6312: Use conventional exception handling instead of poisoning
Use component logger in workers.
Remove basicNack()/basicReject() calls as they are not needed because all unacknowledged messages will be redelivered.
NIFI-6312: Further improve exception handling and error logging.
NIFI-6312: Fix consumer closing in previous commit
NIFI-6312: Use custom executor with a single thread (no more is used by the processor)
Reviewed by tamas palfy and simon bence
Signed-off-by: Joe Witt <joewitt@apache.org>
Add new property 'Content Disposition' to allow user
to set the content-disposition http header on the S3 object.
Allowed values are 'inline' (default) and 'attachment'.
If 'attachment' is selected, the filename will be set to the S3 Object key.
Remove default value and keep backward compatibility
Update fetchS3Object filename attribute settin
Update constant names
Update order of if-else condition
NIFI-7664 Update condition in FetchS3Processor
NIFI-7664 Undo the unexpected indent
NIFI-7664 Update international chars unit test
NIFI-7664 Set fetchS3 file path name
NIFI-7664 Update code style
This closes#4423.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
