NIFI-6387 RetryFlowFile
NIFI-6387 Maximum Retries support FLOWFILE_ATTRIBUTES scope
NIFI-6387 Fixed reuses descriptions for clarity
This closes#3541.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-6004 Improve testing of PutFile file and directory permissions
NIFI-6004 Typo in regex
NIFI-6004 Updates based on review
This closes#3294.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
Fixed test failures in nifi-couchbase-processors, BinaryDocument matcher replaced with ByteArrayDocument
Fixed test failures in nifi-riemann-processors, anyInt() matcher replaced with anyLong() matcher, calling method passes a long, not int
Removed unnecessary method mocks from nifi-toolkit-tls tests, TlsCertificateAuthorityServiceHandlerTest and TlsCertificateSigningRequestPerformerTest, since those were flagged by Mockito as unnecessary (they're unused)
Removed explicit mockito dependency version in nifi-gcp-processors pom to inherit version from nifi's pom.xml
Updated ArgumentMatchers in Kafka 0.10, 0.11, 1.0, and 2.0 processor tests, since in Mockito 2.x, the "any" matchers no longer allow nulls
Updated ArgumentMatchers in nifi-jolt-transform-json-ui, since in Mockito 2.x, the "any" matchers no longer allow nulls
Removed unnecessary method mocks from MetricsReportingTaskTest
Updated TestStandardRemoteGroupPort to return Long instead of Integer for test flowfile.size() invocations
Updated AbstractCassandraProcessor to include keyspaceProperty.getValue() in null check
Updated SimpleProcessLogger and TestSimpleProcessLogger, vararg matching does not work the same in Java 8 and 11
Updated TestStandardProcessScheduler to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated TestPutHiveStreaming to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated FetchParquetTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated ControllerSearchServiceTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Removed usage of Whitebox from GetAzureEventHubTest due to Mockito 2.x, replaced with FieldUtils
Removed usage of Whitebox from StandardOidcIdentityProviderTest due to Mockito 2.x, replaced with FieldUtils
Updated apache-rat-plugin configuration in root POM to make use of useIdeaDefaultExcludes which makes the rat plugin exclude IntelliJ artifacts
Updated several modules to use mockito-core instead of mockito-all (discontinued in Mockito 2.x)
Updated nifi-site-to-site-reporting-task tests to be compatible with Mockito 2.x
Ignored TestPutJMS tests; the tests need to be refactored to work with Mockito 2.x, but the processor is deprecated. Refactor may be done in a separate PR.
Adjusted several mock interaction iterations to 0 for TestPublishKafkaRecord_* tests. Mockito 2.x flagged several interactions as unused and were adjusted to 0 interactions.
Updated PowerMock and Mockito dependencies to exclude transitive dependency on bytebuddy, added explicit dependency on bytebuddy 1.9.10 so that PowerMock and Mockito use the same version. Bytebuddy 1.9.3 (used by PowerMock 2.0.2) did not allow for the mocking of final/private classes, bytebuddy 1.9.10 (used by Mockito 2.28.2) does.
Updated TestSiteToSiteProvenanceReportingTask use of InvocationOnMock.getArgument to use objects for the resulting object rather than primitives
Removed unnecessary stubs from evtx tests, Mockito 2.x defaults to strict mocks
Fixed classloader issue with tests in nifi-windows-event-log-processors module that use JNAJUnitRunner when Mockito mocked JNA classes (Kernel32)
Addressed Mockito-related deprecation warnings
Import cleanup
This closes#3533
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
When PutFile uses 'replace' conflict resolution and max files, there is an issue when the folder has X files, and the limit is also X. The processor fails instead of replacing it, leaving X files. This commit fixes that issue.
This closes#3524.
Signed-off-by: Andres Garagiola <andresgaragiola@gmail.com>
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-6323 Changed URLs for splunk.artifactoryonline.com to use HTTPS (certificate validity warning in browsers, but command-line connection using openssl s_client is successful).
NIFI-6323 Changed URLs for XMLNS schema locations to use HTTPS (the XMLNS and schema identifier remain http:// because they are not designed to be resolvable).
NIFI-6323 Fixed Maven XML schema descriptor URLs.
This closes#3497
- set cleanup thread scheduling to 5 seconds in order to prevent stale request staying in the cache for a long time
- disable timeout handling in HandleHttpRequest in order to prevent a new flowfile being generated at request expiration
- use HttpServletResponse.SC* status codes everywhere in HandleHttpRequest, get rid of importing javax.ws.rs
- add some more error logging to make bug investigations easier
- add a short message to 503 error responses to make bug investigations easier
This closes#3490.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Updated NOTICE with current copyright year for Jetty dependencies
Updated Jetty SSLContextFactory usage, invoke setEndpointIdentificationAlgorithm(null) on server SslContextFactory instances
Updated TestInvokeHttpSSL to provide a separate client keystore, rather than reusing the server's keystore
Regenerated nifi-standard-processors keystore and truststore, added client keystore
Updated ITestHandleHttpRequest, TestInvokeHttpSSL, TestInvokeHttpTwoWaySSL, and TestListenHTTP to use a separate client keystore instead of reusing the server's keystore. Also updated the tests to separately test one-way and two-way SSL
NIFI-6196 - Setting the endpoint identification algorithm to null for sockets to ensure certificates without SANs applied correctly still work. We can change this in a new NiFi release with other potentially breaking changes.
This closes#3426.
Fixed indentation errors to pass checkstyle-checks
Added Evaluation Modes as per discussion in PR thread
Adding exclusions of test files on rat plugin
Added new property 'Line-by-Line Evaluation Mode' and refactored common code
This closes#3375.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
Added an unit test representing the fixed issue.
And updated existing testDefragment test to illustrate
the remaining FlowFiles those did not meet the threshold.
- Make enriched fields nullable at LookupRecord.
- Removed unnecessary AvroConversionOptions and reader schema creation,
because ResultSetRecordSet can generate NiFi Record Schema from RS
directly. No Avro schema is needed to do that.
Suppressed deprecation warnings in TestGetHTTP, TestPostHTTP, TestGetHTTPGroovy, and TestPostHTTPGroovy. The annotation suppresses some warnings, but until Java 9, explicit FQCN imports are needed to avoid all warnings.
This closes#3312.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
NIFI-5903: Removed TODO comments that were done
NIFI-5903: Added support for working with MAP types to QueryRecord and associated RPATH functions
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3223
- Updates to make UpdateRecord and RecordPath automatically update Record schema when performing update and perform the updates on the first record in UpdateRecord before obtaining Writer Schema. This allows the Writer to to inherit the Schema of the updated Record instead of the Schema of the Record as it was when it was read.
- Updated JoltTransformRecord so that schema is inferred on the first transformed object before passing the schema to the Record Writer, so that if writer inherits schema from record, the schema that is inherited is the trans transformed schema
- Updated LookupRecord to allow for Record fields to be arbitrarily added
- Implemented ContentClaimInputStream
- Added controller service for caching schemas
- UpdatedQueryRecord to cache schemas automatically up to some number of schemas, which will significantly inprove throughput in many cases, especially with inferred schemas.
NIFI-5938: Updated AvroTypeUtil so that if creating an Avro Schema using a field name that is not valid for Avro, it creates a Schema that uses a different, valid field name and adds an alias for the given field name so that the fields still are looked up appropriately. Fixed a bug in finding the appropriate Avro field when aliases are used. Updated ContentClaimInputStream so that if mark() is called followed by multiple calls to reset(), that each reset() call is successful instead of failing after the first one (the JavaDoc for InputStream appears to indicate that the InputStream is free to do either and in fact the InputStream is even free to allow reset() to reset to the beginning of file if mark() is not even called, if it chooses to do so instead of requiring a call to mark()).
NIFI-5938: Added another unit test for AvroTypeUtil
NIFI-5938: If using inferred schema in CSV Reader, do not consider first record as a header line. Also addressed a bug in StandardConfigurationContext that was exposed by CSVReader, in which calling getProperty(PropertyDescriptor) did not properly lookup the canonical representation of the Property Descriptor from the component before attempting to get a default value
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3253
Unit test is implemented: testNoSplitterInString
NIFI-5974: Fixed Checkstyle violations
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3275
Clear the wait timestamp when transferring to failur or success
replace explicit attribute clear with function call, refactor and integrate into existing tests per review
This closes#3233.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-4892 - ValidateCSV: no doublequote escaping in invalid output
NIFI-5907 - unit test
This closes#2481.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-5788: Introduce batch size limit in PutDatabaseRecord processor
Renamed 'batch size' to 'Maximum Batch Size'.
Changed default value of max_batch_size to zero (INFINITE)
Fixed parameter validation.
Added unit tests
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3128
making the database session autocommit value a configurable property
adding custom validation to PutSQL processor so as to disallow 'supports transaction' and 'rollback on failure' to be true when the autocommit value has been set to true
fixing some style issues to conform to standards
This closes#3113.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-5761 Code review.
Remove startsWidth to check an exception. Added the dependency as provided.
NIFI-5761 Code review.
Remove provided.
NIFI-5761 Code review.
Improve logging.
This closes#3112.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-5686: Fixed unit test in TestSocketLoadBalancedFlowFileQueue; renamed TestProcessorLifecycle to ProcessorLifecycleIT as it is testing integration between many components and largely focuses on high numbers of concurrent tasks to see if it can trigger any threading bugs that may get introduced
NIFI-5686: Extended unit test timeouts
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3062
- introducing a in-memory-file-size-threashold, above which the incoming file is written to local disk
- using java.io.tmpdir for such file writes
- enhancing documentation
- documenting how to avoid premature HTTP response
- fix and UT for unsuccessful request registration
This closes#2991.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- introducing a in-memory-file-size-threashold, above which the incoming file is written to local disk
- using java.io.tmpdir for such file writes
- enhancing documentation
- Updated FlowFile Repo / Write Ahead Log so that any update that writes more than 1 MB of data is written to a file inside the FlowFile Repo rather than being buffered in memory
- Update SplitText so that it does not hold FlowFiles that are not the latest version in heap. Doing them from being garbage collected, so while the Process Session is holding the latest version of the FlowFile, SplitText is holding an older version, and this results in two copies of the same FlowFile object
NIFI-5533: Checkpoint
NIFI-5533: Bug Fixes
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2974
Refactored some test code to be clearer.
Renamed some resources to be consistent across modules.
Changed passwords to meet new minimum length requirements.
This closes#3018
Refactors tests in order to share code repeated in tests and to enable
some parameterized testing.
MySQL Connector/J 5.1.x in conjunction with MySQL 5.0.x will return
a Long for ResultSet#getObject when the SQL type is an unsigned integer.
This change prevents that error from occurring while implementing a more
informational exception describing what the failing object's POJO type
is in addition to its string value.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3032
NIFI-5514: Do not rely on ProcessSession.getQueueSize() to return a queue size of 0 objects because if the processor is holding onto data, the queue size won't be 0.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#2954.
Deprecated HashContent processor.
Added documentation to HashAttribute processor.
Added shared HashService and HashAlgorithm enum.
Added unit tests.
Added #clearProperties() to TestRunner, StandardProcessorTestRunner, and MockProcessContext.
Updated processor manifest.
Updated Javadoc.
Added documentation about deprecated/renamed components.
Added logic handling for UTF-16 encoding where UTF-16BE is overridden so the BOM is not inserted prior to hashing.
This closes#2983.
Co-authored-by: Otto Fowler <ottobackwards@gmail.com>
Signed-off-by: Kevin Doran <kdoran@apache.org>
This reverts commit f7f809c3d3.
NIFI-5474, NIFI-4272: When using Regex Replace with ReplaceText, and there are capturing groups, ensure that we populate the 'additionalVariables' map for each match of the regex. This allows Expression Language to reference the back-references properly even when there are multiple matches
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2951
Adding followings:
- Use separate DistributedMapCache for tracking entities to avoid
conflict with existing code
- Added more validation
- Delete listed entities from cache if reset is needed
- Support Local scope
- Added Initial Listing Target
This closes#2876.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Create nifi-syslog-utils to move syslog parsing functionalty to a central location shared by the processors and serialization/record system.
- Refactor Processors to use these utils
- Update 5424 syslog classes using simple-syslog-5424 to pick up new changes to support this work, as well as keep dependencies/types from bleeding out to the
processors or readers
- Refactor Syslog5424Event and Parser
- Create Syslog5424RecordReader
- per review, handle blank message differently from eof
- name schema per review
This closes#2816.
Signed-off-by: Bryan Bende <bbende@apache.org>
Updated Javadoc for SiteToSiteClient#createTransaction() and HttpClient implementation.
Reverted exception listing in method contract for SiteToSiteClient#createTransaction and HttpClient tion of same.
Reverted import ordering in TestSiteToSiteClient.
Reverted exception listing in TestGetHDFSFileInfo, TestListHDFS, and StandardHttpFlowFileServerProtocol.
Restored @SuppressWarnings annotation and removed unnecessary "public static" keywords from inner classes in SiteToSiteClient.
This closes#2841.
Signed-off-by: Joe Witt <joewitt@apache.org>
- address review comments
- per review, add property for including body in attributes
- per review, create processor in OnScheduled
This closes#2805.
Signed-off-by: Bryan Bende <bbende@apache.org>
forgot to penalize flowfiles sent to failure
NIFI-5274 avoid rollback on StackOverflowError in ReplaceText
add comment
removed penalize on failure, doesn't really apply here
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2767
NIFI-5214 Added support for the new ProxyConfigurationService
NIFI-5214 Integration tests added.
NIFI-5214 Added missing pom.xml and a change from a code review.
NIFI-5214 Added another tag based on code review.
NIFI-5214 Added user-defined header support.
NIFI-5214 Added Basic Auth support.
NIFI-5214 Moved documentation.
NIFI-5214 Fixed checkstyle issues; added changes requested in a review.
NIFI-5214 Added changes requested in a code review.
NIFI-5214 Added verb test and @DynamicProperties
NIFI-5214 Added templated URL support to RestLookupService.
NIFI-5214 Fixed documentation based on code review changes.
NIFI-5214 Changed RestLookupService to use a property descriptor.
NIFI-5214 Updated documentation.
NIFI-5214 Made changes requested in code review.
NIFI-5214 Renamed nifi-standard-web-utils to reflect that it is for tests.
NIFI-5214: Refactor RestLookupService.
1. Added 'Base URL' property to address environment specific part of URL.
2. Removed 'Record Path Property Name' property, because the name of
a resulted record field of a record path can be obtained by field name.
3. Lower cased HTTP method name should be used throughout.
4. Added mimeType require check when body is specified.
5. Added debug log to print HTTP response code.
6. Prepare for NIFI-5287.
7. Fixed that mime.type being used regardless of whether body is
specified or not, caused NullPointerException when 'mime.type' is not
specified when it is not required.
8. Updated documentation.
NIFI-5214 Fixed AWS processors that broke w/ change to TestServer location.
NIFI-5214 Added changes requested in a code review.
Refactored the way to evaluate EL for URL property
- Use PropertyValue instead of PreparedQuery to utilize Variable
Registry.
- Removed BASE_URL because Variable Registry can be used at URL
NIFI-5214 Rebased to use new LookupService method.
This closes#2723.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-5287 Removed unneeded property descriptor.
NIFI-5287 Added additional changes from a code review.
This closes#2777.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-5231 Minor fixes from code review.
NIFI-5231 Added documentation and made record_count a constant.
NIFI-5231 Added changes requested in a code review.
NIFI-5231 Added changes based on code review.
This closes#2737.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
commit 9c31e45d3f
Author: Joe Trite <joetrite@gmail.com>
Date: Mon Mar 13 07:55:19 2017 -0400
NIFI-1705 Adding AttributesToCSV processor
commit 5e9afa9ccf
Merge: 3177eb18574cbfc4b6
Author: Joe Trite <joetrite@gmail.com>
Date: Sat Mar 4 08:12:39 2017 -0500
Merge remote-tracking branch 'origin/master'
# Conflicts:
# nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ScanAttribute.java
commit 3177eb1852
Author: Joe Trite <joetrite@gmail.com>
Date: Sat Mar 4 08:09:06 2017 -0500
NIFI-3497 Converted properties to use displayName, changed validator on demarcator property and created a method for duplicate code.
commit 74cbfc4b69
Merge: a974f78e0f8cad0f8c
Author: Joe Trite <joetrite@gmail.com>
Date: Sat Mar 4 07:47:46 2017 -0500
Merge branch 'master' into master
commit a974f78e03
Author: Joe Trite <joetrite@gmail.com>
Date: Sat Mar 4 07:43:02 2017 -0500
NIFI-3497 Converted properties to use displayName, changed validator on demarcator property and created a method for duplicate code.
commit 1bfaef8e87
Merge: 65ed46de989ec68d14
Author: Joe Trite <joetrite@gmail.com>
Date: Fri Mar 3 08:01:59 2017 -0500
Merge branch 'master' of https://github.com/joetrite/nifi
commit 65ed46de9a
Author: Joe Trite <joetrite@gmail.com>
Date: Fri Feb 24 18:09:36 2017 -0500
NIFI-3497 - fixing Pcontrib issues
commit c5d52cf6f0
Author: Joe Trite <joetrite@gmail.com>
Date: Thu Feb 23 10:19:01 2017 -0500
NIFI-3497 - excluding test files
Adding new test data files to exclude list.
commit b195934113
Author: Joe Trite <joetrite@gmail.com>
Date: Wed Feb 22 16:48:10 2017 -0500
NIFI-3497 - New dictionary files for test
Adding new dictionary files to support metadata dictionary option.
commit e296268f39
Author: Joe Trite <joetrite@gmail.com>
Date: Wed Feb 22 16:46:13 2017 -0500
NIFI-3497 test cases for metadata updates
Adding test cases to support metadata option update.
commit de7e348e62
Author: Joe Trite <joetrite@gmail.com>
Date: Wed Feb 22 16:36:08 2017 -0500
NIFI-3497 - Added metadata option
Added optional to post additional metadata as new attributed if a match is found in the dictionary.
commit 89ec68d14b
Author: Joe Trite <joetrite@gmail.com>
Date: Fri Feb 24 18:09:36 2017 -0500
NIFI-3497 - fixing Pcontrib issues
commit d71426037b
Author: Joe Trite <joetrite@gmail.com>
Date: Thu Feb 23 10:19:01 2017 -0500
NIFI-3497 - excluding test files
Adding new test data files to exclude list.
commit a7a7b6ace8
Author: Joe Trite <joetrite@gmail.com>
Date: Wed Feb 22 16:48:10 2017 -0500
NIFI-3497 - New dictionary files for test
Adding new dictionary files to support metadata dictionary option.
commit 8eb54a5019
Author: Joe Trite <joetrite@gmail.com>
Date: Wed Feb 22 16:46:13 2017 -0500
NIFI-3497 test cases for metadata updates
Adding test cases to support metadata option update.
commit f52e1f2a06
Author: Joe Trite <joetrite@gmail.com>
Date: Wed Feb 22 16:36:08 2017 -0500
NIFI-3497 - Added metadata option
Added optional to post additional metadata as new attributed if a match is found in the dictionary.
NIFI-1705 Adding AttributesToCSV processor.
NIFI-1705 Adding AttributesToCSV processor.
NIFI-1705 Adding AttributesToCSV processor.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2711
- PR Fix - 'Execution' dropdown will now be shown in all cases
- Annotated ListGCSBucket with PrimaryNodeOnly
This closes#2509.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Travis Neeley
adyoun2
Jeff Storck
Matthew Burgess
Andres Garagiola
Evan Reynolds
Andy LoPresto
Peter Turcsanyi
Otto Fowler
Troy Melhase
Dustin Rodrigues
Koji Kawamura
pushpavanthar
bdesert
Lars Francke
Pierre Villard
Joe Witt
Arpad Boda
Aleksandr Salatich
Mark Payne
a.durov
erichanson5
SavtechSolutions
zenfenan
Kumara M S Hemantha
yjhyjhyjh0
Colin Dean
vadimar
Arek Burdach
Vish Uma
Kotaro Terada
Gardella Juan Pablo
juliansniff
Bryan Bende
rednikotin
patricker
Endre Zoltan Kovacs
Brad Hards
Endre Zoltan Kovacs
Nick Lewis
Mike Moser
Marco Gaido
Mike Thomsen
Bartlomiej Tartanus
Joe Trite
Nicolas Sanglard