Commit Graph

5158 Commits

Author SHA1 Message Date
thenatog e7aac049ef
NIFI-6372 - This closes #3532. Upgrading jackson-databind version to 2.9.9.
Signed-off-by: Joe Witt <joewitt@apache.org>
2019-06-19 12:41:18 -04:00
thenatog 64b0d88d83
NIFI-6330 - This closes #3534.Updated Jetty version to 9.4.18.v20190429
NIFI-6330 - Removed jetty.version property declaration in nifi-prometheus-reporting-task pom.xml. This version should be inherited from the root pom.

NIFI-6330 - New release of Jetty came out 2 days ago. Upgraded and verified NiFi functionality again with smoke tests including ListenHTTP, secure clustering, site-to-site

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-06-19 12:39:15 -04:00
Jeff Storck 1d560e2b02 NIFI-6360 Updated Mockito to 2.28.2, PowerMock to 2.0.2
Fixed test failures in nifi-couchbase-processors, BinaryDocument matcher replaced with ByteArrayDocument
Fixed test failures in nifi-riemann-processors, anyInt() matcher replaced with anyLong() matcher, calling method passes a long, not int
Removed unnecessary method mocks from nifi-toolkit-tls tests, TlsCertificateAuthorityServiceHandlerTest and TlsCertificateSigningRequestPerformerTest, since those were flagged by Mockito as unnecessary (they're unused)
Removed explicit mockito dependency version in nifi-gcp-processors pom to inherit version from nifi's pom.xml
Updated ArgumentMatchers in Kafka 0.10, 0.11, 1.0, and 2.0 processor tests, since in Mockito 2.x, the "any" matchers no longer allow nulls
Updated ArgumentMatchers in nifi-jolt-transform-json-ui, since in Mockito 2.x, the "any" matchers no longer allow nulls
Removed unnecessary method mocks from MetricsReportingTaskTest
Updated TestStandardRemoteGroupPort to return Long instead of Integer for test flowfile.size() invocations
Updated AbstractCassandraProcessor to include keyspaceProperty.getValue() in null check
Updated SimpleProcessLogger and TestSimpleProcessLogger, vararg matching does not work the same in Java 8 and 11
Updated TestStandardProcessScheduler to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated TestPutHiveStreaming to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated FetchParquetTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated ControllerSearchServiceTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Removed usage of Whitebox from GetAzureEventHubTest due to Mockito 2.x, replaced with FieldUtils
Removed usage of Whitebox from StandardOidcIdentityProviderTest due to Mockito 2.x, replaced with FieldUtils
Updated apache-rat-plugin configuration in root POM to make use of useIdeaDefaultExcludes which makes the rat plugin exclude IntelliJ artifacts
Updated several modules to use mockito-core instead of mockito-all (discontinued in Mockito 2.x)
Updated nifi-site-to-site-reporting-task tests to be compatible with Mockito 2.x
Ignored TestPutJMS tests; the tests need to be refactored to work with Mockito 2.x, but the processor is deprecated.  Refactor may be done in a separate PR.
Adjusted several mock interaction iterations to 0 for TestPublishKafkaRecord_* tests.  Mockito 2.x flagged several interactions as unused and were adjusted to 0 interactions.
Updated PowerMock and Mockito dependencies to exclude transitive dependency on bytebuddy, added explicit dependency on bytebuddy 1.9.10 so that PowerMock and Mockito use the same version.  Bytebuddy 1.9.3 (used by PowerMock 2.0.2) did not allow for the mocking of final/private classes, bytebuddy 1.9.10 (used by Mockito 2.28.2) does.
Updated TestSiteToSiteProvenanceReportingTask use of InvocationOnMock.getArgument to use objects for the resulting object rather than primitives
Removed unnecessary stubs from evtx tests, Mockito 2.x defaults to strict mocks
Fixed classloader issue with tests in nifi-windows-event-log-processors module that use JNAJUnitRunner when Mockito mocked JNA classes (Kernel32)
Addressed Mockito-related deprecation warnings
Import cleanup

This closes #3533

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-06-17 12:21:07 -04:00
Bryan Bende c7768c909a
NIFI-6164 - Added CLI commands related to NiFi Registry extension bun… (#3492)
* NIFI-6164 - Added CLI commands related to NiFi Registry extension bundles
- Adding CLI commands fro uploading extension bundles
- Adding commands for listing extension bundles and their versions
- Adding checksums to UploadNarBundles command
- Adding sha256 to UploadExtensionBundle command, and refactoring result classes to be better organized
- Improvements for CLI commands to interact with extension registry.
- Added download-extension-bundle commands.
- Refactoring ProcessGroupBox

* Refactoring upload-nars command to be upload-bundles

* Rename ListTagCounts to ListExtensionTags and ListExtensionsWithTags to ListExtensions

* Update to catch more specific exception in UploadBundles

Co-Authored-By: Kevin Doran <kdoran@apache.org>
2019-06-17 08:22:19 -04:00
thenatog 0e1a37fcb9
NIFI-6362 - Upgraded com.puppycrawl.tools:checkstyle to 8.18. Made small checkstyle changes to nifi-kafka.
NIFI-6362 - Upgraded both checkstyle plugins to latest. Added a checkstyleCache file and property.

NIFI-6362 - Downgraded maven-checkstyle-plugin to 3.0.0 instead as there was an issue with the latest version and @throws Javadoc checkstyle module. Fixed some checkstyle issues. Changed checkstyleCache file to .checkstyleCache. Added to .gitignore.

NIFI-6362 - Changed checkstyleCache file location into target/, and is no longer a hidden file. Removed .checkstyleCache from .gitignore as it is no longer required.

NIFI-6362 - Grep pattern no longer required.

This closes #3522.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-17 17:19:09 +09:00
Mark Payne aab3af4b71 NIFI-6322: This closes #3518. 2019-06-13 11:36:33 -04:00
Mark Payne c456b30b9b NIFI-6322: Renamed EvaluationContext to EvaluatorState; added javadocs 2019-06-13 11:34:06 -04:00
Frederik Petersen 68d05ab575 NIFI-6322: Introduced EvaluationContext to store state while making evaluator tree reusable
- EvaluationContext gets passed through evalutors for each query
- evaluator tree is reused for multiple evaluation calls
- added extensive test cases for reusing prepared queries
- fixed tiny bug in DelineatedAttributeEvaluator
2019-06-13 11:34:06 -04:00
Bryan Bende 2102d8a0bd
NIFI-6328 Allowing force-commit when in LOCALLY_MODIFIED_AND_STALE state
NIFI-6328 Addressing review feedback
NIFI-6328 Additional validation on action

This closes #3499
2019-06-13 10:42:13 -04:00
thenatog 75fb34c8ee
NIFI-6301 - Added a SafeXMLConfiguration which disables XML DTDs which may call external entities.
NIFI-6301 - Fixed unit test. Added comments.
NIFI-6301 - Removed unused rule from test.
NIFI-6301 - Changed read() methods to use a boolean instead. Updated comments.
NIFI-6301 - Fixing checkstyle errors.
NIFI-6301 - Added an XXE Validator to do a simple regex check on files that contain XXE declarations. Added unit tests and related XML test files.
NIFI-6301 - Forgot license headers. Fixed a unit test.
NIFI-6301 - Refactored duplicate code for error handling.
NIFI-6301 - Removed unused imports.
NIFI-6301 - Slightly changed messaging language.
NIFI-6301 - Added logic to catch multiline XXE attacks.
Added logging messages.
Updated unit test.

This closes #3507.

Co-authored-by: Andy LoPresto <alopresto@apache.org>
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-06-12 16:40:49 -07:00
Mike Thomsen 3e7816da43
NIFI-5902 Added GeoEnrichIPRecord.
NIFI-5902 Updated GeoEnrichIPRecord with more error handling.

NIFI-5902 Made changes requested in a review.

NIFI-5902 Updated EL support on database path.

NIFI-5902 Made updates based on code review.

NIFI-5902 Made two changes from a code review.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3231
2019-06-12 15:25:42 -04:00
Mark Payne 8bed249f37
NIFI-6353: If LoadBalanceSession determines remote node's queue is full, mark session complete and penalize the Partition, rather than just penalizing the session. This allows the client to move on to another session, which avoids an issue where backpressure on one queue can prevent data from being transferred between nodes in another queue
This closes #3529.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-06-12 11:32:41 -04:00
Mark Payne 2465f67dfe
NIFI-6339: Only call @OnConfigurationRestored method on components when they are created if FlowController has been initialized
This closes #3530.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-06-12 10:56:04 -04:00
Shawn Weeks e913f5706f
NIFI-6243 Add Support for AtomicDistributedCache to the HBase 1.x and 2.x Map Cache Services
This closes #3462.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-12 15:31:39 +09:00
Koji Kawamura 9158498719
Renew my key.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-12 15:08:49 +09:00
Matthew Burgess 05f3cadee8 NIFI-6370: Allow multiple SQL statements in PutDatabaseRecord
This closes #3528.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-12 10:32:11 +09:00
Mark Payne f15332ff87 NIFI-6182: Updated dependency on Lucene to Lucene 8.0.0. Updated code necessary to use the new API. Updated WriteAheadProvenanceRepository so that upon startup if provenance indexes are written using the old Lucene format, , they are considered 'defunct' and the events are re-indexed in a background thread into a new index that uses the Lucene 8 format and the old index is then removed. Added Provenance Repository that consists of about 30 events and added integration test to ensure that the repo can be started up/initialized when pointing to a prov repo that was written using the old lucene format. 2019-06-11 09:45:46 -04:00
Andres Garagiola 1975101292 NIFI-6361: Add fix to PutFile processor
When PutFile uses 'replace' conflict resolution and max files, there is an issue when the folder has X files, and the limit is also X. The processor fails instead of replacing it, leaving X files. This commit fixes that issue.

This closes #3524.

Signed-off-by: Andres Garagiola  <andresgaragiola@gmail.com>

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-10 15:45:19 +09:00
Evan Reynolds 99350b761d NIFI-6349 Fix to MergeRecords when handling fragments over multiple iterations
Fixed fragment count attribute check

This closes #3517.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-10 14:33:11 +09:00
Andrew Lim 0b8d05aabd NIFI-6319 Update User Guide for RPG/S2S chnages for multiple URLs, batch settings and remote input/output ports
NIFI-6319 Improved Remote Process Group Ports screenshot

This closes #3503.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-10 14:18:55 +09:00
Matthew Burgess 99d6ed244c
NIFI-6352: Add ability to report all component metrics to Prometheus
This closes #3519.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2019-06-07 15:30:38 -04:00
Ferenc Szabó aefdc9377c NIFI-6218 Support setting transactional.id in PublishKafka/PublishKafkaRecord
Added new property to the processors. If it is empty, then the behaviour is the same as before.
Adde unit test to check if KafkaProducer receives the new property.

Testing Done:
Unit tests.
Connecting to kafka and verifying transactionalId-s in kafka server log.

fixing review findings

This closes #3439.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-07 17:55:52 +09:00
Peter Wicks 2d3fce81c1
NIFI-6350 Fixed misleading flow controller startup error message.
This closes #3516.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-06-05 19:25:13 -07:00
Matthew Burgess 1d18735076
NIFI-6348: Added Custom ORDER BY Column property to GenerateTableFetch
NIFI-6348: Fixed doc

This closes #3515.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-06-05 10:00:12 -04:00
Mark Payne 60b5c13ce9
NIFI-6341: Addressed bug that can result in a deadlock if Thread 1 adds a Controller Service to a Process Group (which obtains group's write lock and AbstractComponent's Lock) while at the same time Thread 2 attempts to peform validation on the service (which obtains Controller Service's write lock, then AbstractComponent's Lock)
This closes #3510.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-06-04 09:55:26 -04:00
Matthew Burgess 08d2e69878
Revert "NIFI-6244: Provide default ORDER BY clause in MSSQL adapters"
This reverts commit 7fc49c2cb8.

This closes #3512
2019-06-04 09:02:03 -04:00
Bryan Bende e7f4af6801 NIFI-6315 Ensuring remote ports get tracked correctly when saving/retrieving versioned flows
NIFI-6315 Renaming public ports during import/change-version when name alreadye exists, and ignorning name change from flow diffs

NIFI-6315 Ensuring public ports get renamed during copy/paste

NIFI-6315 Set max concurrent tasks on Port based on value from VersionedPort

This closes #3488.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-03 11:25:04 +09:00
Mike Thomsen 407add7847 NIFI-5916 Added an option to enable empty flowfiles to be sent if there are no results from a query.
NIFI-5916 Fixed potential NPE.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3315
2019-06-01 11:27:53 -04:00
Andy LoPresto ed19f61682
NIFI-6331 Removed nifi-testharness module.
This closes #3506.
2019-05-31 13:01:46 -07:00
Andy LoPresto a9d1dd7b08 NIFI-6337 Resolved groovy unit test execution problems in nifi-properties-loader and fixed failing unit tests.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3508
2019-05-31 08:44:17 -04:00
Joe Witt a951a8ec61
NIFI-6329 trust update did not help and oraclejdk is not working on xenial - removing it
This closes #3505.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-05-30 18:05:40 -07:00
slyouts 097b9c3075 NIFI-6022 Added try/catch to protect against secondary exception in a primary exception catch clause
NIFI-6022 mods per PR review

NIFI-6022 mods per PR review

This closes #3476.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-05-31 08:51:36 +09:00
Joe Witt de27ab94b0
NIFI-6329 added dist section to specify trusty 2019-05-30 09:47:00 -04:00
Satwik 8560eb995a NIFI-5435 Added Prometheus reporting task.
NIFI-5435 Fixed Rebase Issue
NIFI-5435 changes in capability description of reporting task
NIFI-5435 Changes in capability description of reporting task
NIFI-5435 Changes to build Property Descriptors in static constructor
NIFI-5435 Indentation changes
NIFI-5435 Test code changes
NIFI-5435 Minor code changes
NIFI-5435 Changes related to client authentication
NIFI-5435 Prometheus metrics changes
NIFI-5435 Added process group name and id to metrics

This closes #3257

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-05-30 08:08:07 -04:00
Matt Gilman 999cfdcac9
NIFI-6302:
- Updating integration tests to verify pruned results.

This closes #3487.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-05-29 14:45:00 -07:00
Andy LoPresto e6c843f465
NIFI-6323 Changed URLs for repositories, project description, and mailing lists to use HTTPS.
NIFI-6323 Changed URLs for splunk.artifactoryonline.com to use HTTPS (certificate validity warning in browsers, but command-line connection using openssl s_client is successful).
NIFI-6323 Changed URLs for XMLNS schema locations to use HTTPS (the XMLNS and schema identifier remain http:// because they are not designed to be resolvable).
NIFI-6323 Fixed Maven XML schema descriptor URLs.

This closes #3497
2019-05-29 14:36:40 -04:00
Mike Thomsen 6a06cd3094 NIFI-6304 added trim, toLowerCase and toUpperCase to record path operations.
NIFI-6304 Updated code based on code review.

NIFI-6304 Updated documentation.

NIFI-6304 Refactored to make it simpler

NIFI-6304 Reverted Concat to its last state.

This closes #3478.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-05-28 21:06:20 +09:00
Koji Kawamura 99b20ac2d1 NIFI-6303 CSVWriter should write schema defined fields first, then extra ones
This closes #3484

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-05-26 16:52:12 -04:00
Andy LoPresto 6be47de87e NIFI-6296 Improved PR template messaging around commit process. 2019-05-24 16:18:08 -04:00
Peter Turcsanyi 2bfffd35b2 NIFI-6317: HandleHttpRequest timeout handling issues
- set cleanup thread scheduling to 5 seconds in order to prevent stale request staying in the cache for a long time
- disable timeout handling in HandleHttpRequest in order to prevent a new flowfile being generated at request expiration
- use HttpServletResponse.SC* status codes everywhere in HandleHttpRequest, get rid of importing javax.ws.rs
- add some more error logging to make bug investigations easier
- add a short message to 503 error responses to make bug investigations easier

This closes #3490.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-05-24 14:43:50 -04:00
Rob Fellows 0a31c38790
NIFI-6253 - Remove invalid/unnecessary argument in call to nfCanvasUtil.reload
This closes #3491
2019-05-24 11:02:09 -04:00
Rob Fellows 410854f53e
NIFI-6316 - Upgrade jQuery to the latest version (3.4.1).
This closes #3489
2019-05-24 09:52:02 -04:00
Bryan Bende b09b84226c
NIFI-6311 Upgrade to nifi-registry-client 0.4.0
This closes #3485.
2019-05-23 12:48:33 -04:00
thenatog fe68d43e1d
NIFI-6280 - Broke out the matching for /access/knox/** and /access/oidc/** to allow the Jetty security filters to be applied in the /access/oidc/logout and /access/knox/logout cases.
NIFI-6280 - Updated terminology in JwtAuthenticationFilter to authentication instead of authorization. Added stricter token parsing using an explicit regex pattern. Added tests.
NIFI-6280 - Updated terminology from Authorization to Authentication.
NIFI-6280 - Updated the access logout method to use getNiFiUserIdentity(). Updated javascript logout method to handle errors.
NIFI-6280 - Fixing checkstyle issues.
NIFI-6280 - Added some javadoc comments and logging. Renamed some variables for clarity. Fixed handling of exception when JWT does not match expected format.
NIFI-6280 - Cleaned up checkstyle, increased log severity level for logout action, and cleaned up Groovy syntax in test.

This closes #3482.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-05-22 17:05:06 -07:00
Koji Kawamura 8a50cb10b2 NIFI-2933 Remote input/output ports at any PG
Specify remote access at port creation.
Incorporated comments, and finished refactoring.
Renamed RootGroupPort to PublicPort.
Fix error message for creating a connection from a child PG having only PublicPorts.
Enhanced ProcessGroup instances rendered in the parent ProcessGroup
Loosen Port move check, allow moving public port between PG.
Show 'Remote NiFi Instance' info on Connection dialogs
Make labels narrative.
'Within Remote Group'.
Fixed DTO (de)serialization.
Return null only if all values are null.

This closes #3351.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-05-22 15:44:16 -04:00
Matt Gilman a97766d62f NIFI-6302:
- Ensuring Process Group contents are pruned when appropriate.

This closes #3477.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-05-22 14:12:50 -04:00
Koji Kawamura c470608daf NIFI-6035 Fix check-style issue 2019-05-22 16:07:17 +09:00
Aaron Leon 81ddd02ca8 NIFI-6100 Use setBytes in JdbcCommon for binary types
This closes #3410

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-05-21 14:22:56 -04:00
Mark Payne 8245bc3f80 NIFI-5922: Ensure that we import any default variable values on flow import 2019-05-21 11:19:17 -04:00
samhjelmfelt 650c6aa820 Small fix after rebase 2019-05-21 11:19:17 -04:00