Commit Graph

3293 Commits

Author SHA1 Message Date
thenatog 0e1a37fcb9
NIFI-6362 - Upgraded com.puppycrawl.tools:checkstyle to 8.18. Made small checkstyle changes to nifi-kafka.
NIFI-6362 - Upgraded both checkstyle plugins to latest. Added a checkstyleCache file and property.

NIFI-6362 - Downgraded maven-checkstyle-plugin to 3.0.0 instead as there was an issue with the latest version and @throws Javadoc checkstyle module. Fixed some checkstyle issues. Changed checkstyleCache file to .checkstyleCache. Added to .gitignore.

NIFI-6362 - Changed checkstyleCache file location into target/, and is no longer a hidden file. Removed .checkstyleCache from .gitignore as it is no longer required.

NIFI-6362 - Grep pattern no longer required.

This closes #3522.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-17 17:19:09 +09:00
Bryan Bende 2102d8a0bd
NIFI-6328 Allowing force-commit when in LOCALLY_MODIFIED_AND_STALE state
NIFI-6328 Addressing review feedback
NIFI-6328 Additional validation on action

This closes #3499
2019-06-13 10:42:13 -04:00
thenatog 75fb34c8ee
NIFI-6301 - Added a SafeXMLConfiguration which disables XML DTDs which may call external entities.
NIFI-6301 - Fixed unit test. Added comments.
NIFI-6301 - Removed unused rule from test.
NIFI-6301 - Changed read() methods to use a boolean instead. Updated comments.
NIFI-6301 - Fixing checkstyle errors.
NIFI-6301 - Added an XXE Validator to do a simple regex check on files that contain XXE declarations. Added unit tests and related XML test files.
NIFI-6301 - Forgot license headers. Fixed a unit test.
NIFI-6301 - Refactored duplicate code for error handling.
NIFI-6301 - Removed unused imports.
NIFI-6301 - Slightly changed messaging language.
NIFI-6301 - Added logic to catch multiline XXE attacks.
Added logging messages.
Updated unit test.

This closes #3507.

Co-authored-by: Andy LoPresto <alopresto@apache.org>
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-06-12 16:40:49 -07:00
Mike Thomsen 3e7816da43
NIFI-5902 Added GeoEnrichIPRecord.
NIFI-5902 Updated GeoEnrichIPRecord with more error handling.

NIFI-5902 Made changes requested in a review.

NIFI-5902 Updated EL support on database path.

NIFI-5902 Made updates based on code review.

NIFI-5902 Made two changes from a code review.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3231
2019-06-12 15:25:42 -04:00
Mark Payne 8bed249f37
NIFI-6353: If LoadBalanceSession determines remote node's queue is full, mark session complete and penalize the Partition, rather than just penalizing the session. This allows the client to move on to another session, which avoids an issue where backpressure on one queue can prevent data from being transferred between nodes in another queue
This closes #3529.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-06-12 11:32:41 -04:00
Mark Payne 2465f67dfe
NIFI-6339: Only call @OnConfigurationRestored method on components when they are created if FlowController has been initialized
This closes #3530.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-06-12 10:56:04 -04:00
Shawn Weeks e913f5706f
NIFI-6243 Add Support for AtomicDistributedCache to the HBase 1.x and 2.x Map Cache Services
This closes #3462.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-12 15:31:39 +09:00
Matthew Burgess 05f3cadee8 NIFI-6370: Allow multiple SQL statements in PutDatabaseRecord
This closes #3528.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-12 10:32:11 +09:00
Mark Payne f15332ff87 NIFI-6182: Updated dependency on Lucene to Lucene 8.0.0. Updated code necessary to use the new API. Updated WriteAheadProvenanceRepository so that upon startup if provenance indexes are written using the old Lucene format, , they are considered 'defunct' and the events are re-indexed in a background thread into a new index that uses the Lucene 8 format and the old index is then removed. Added Provenance Repository that consists of about 30 events and added integration test to ensure that the repo can be started up/initialized when pointing to a prov repo that was written using the old lucene format. 2019-06-11 09:45:46 -04:00
Andres Garagiola 1975101292 NIFI-6361: Add fix to PutFile processor
When PutFile uses 'replace' conflict resolution and max files, there is an issue when the folder has X files, and the limit is also X. The processor fails instead of replacing it, leaving X files. This commit fixes that issue.

This closes #3524.

Signed-off-by: Andres Garagiola  <andresgaragiola@gmail.com>

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-10 15:45:19 +09:00
Evan Reynolds 99350b761d NIFI-6349 Fix to MergeRecords when handling fragments over multiple iterations
Fixed fragment count attribute check

This closes #3517.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-10 14:33:11 +09:00
Matthew Burgess 99d6ed244c
NIFI-6352: Add ability to report all component metrics to Prometheus
This closes #3519.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2019-06-07 15:30:38 -04:00
Ferenc Szabó aefdc9377c NIFI-6218 Support setting transactional.id in PublishKafka/PublishKafkaRecord
Added new property to the processors. If it is empty, then the behaviour is the same as before.
Adde unit test to check if KafkaProducer receives the new property.

Testing Done:
Unit tests.
Connecting to kafka and verifying transactionalId-s in kafka server log.

fixing review findings

This closes #3439.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-07 17:55:52 +09:00
Peter Wicks 2d3fce81c1
NIFI-6350 Fixed misleading flow controller startup error message.
This closes #3516.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-06-05 19:25:13 -07:00
Matthew Burgess 1d18735076
NIFI-6348: Added Custom ORDER BY Column property to GenerateTableFetch
NIFI-6348: Fixed doc

This closes #3515.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-06-05 10:00:12 -04:00
Mark Payne 60b5c13ce9
NIFI-6341: Addressed bug that can result in a deadlock if Thread 1 adds a Controller Service to a Process Group (which obtains group's write lock and AbstractComponent's Lock) while at the same time Thread 2 attempts to peform validation on the service (which obtains Controller Service's write lock, then AbstractComponent's Lock)
This closes #3510.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-06-04 09:55:26 -04:00
Matthew Burgess 08d2e69878
Revert "NIFI-6244: Provide default ORDER BY clause in MSSQL adapters"
This reverts commit 7fc49c2cb8.

This closes #3512
2019-06-04 09:02:03 -04:00
Bryan Bende e7f4af6801 NIFI-6315 Ensuring remote ports get tracked correctly when saving/retrieving versioned flows
NIFI-6315 Renaming public ports during import/change-version when name alreadye exists, and ignorning name change from flow diffs

NIFI-6315 Ensuring public ports get renamed during copy/paste

NIFI-6315 Set max concurrent tasks on Port based on value from VersionedPort

This closes #3488.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-03 11:25:04 +09:00
Mike Thomsen 407add7847 NIFI-5916 Added an option to enable empty flowfiles to be sent if there are no results from a query.
NIFI-5916 Fixed potential NPE.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3315
2019-06-01 11:27:53 -04:00
Andy LoPresto a9d1dd7b08 NIFI-6337 Resolved groovy unit test execution problems in nifi-properties-loader and fixed failing unit tests.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3508
2019-05-31 08:44:17 -04:00
slyouts 097b9c3075 NIFI-6022 Added try/catch to protect against secondary exception in a primary exception catch clause
NIFI-6022 mods per PR review

NIFI-6022 mods per PR review

This closes #3476.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-05-31 08:51:36 +09:00
Satwik 8560eb995a NIFI-5435 Added Prometheus reporting task.
NIFI-5435 Fixed Rebase Issue
NIFI-5435 changes in capability description of reporting task
NIFI-5435 Changes in capability description of reporting task
NIFI-5435 Changes to build Property Descriptors in static constructor
NIFI-5435 Indentation changes
NIFI-5435 Test code changes
NIFI-5435 Minor code changes
NIFI-5435 Changes related to client authentication
NIFI-5435 Prometheus metrics changes
NIFI-5435 Added process group name and id to metrics

This closes #3257

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-05-30 08:08:07 -04:00
Matt Gilman 999cfdcac9
NIFI-6302:
- Updating integration tests to verify pruned results.

This closes #3487.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-05-29 14:45:00 -07:00
Andy LoPresto e6c843f465
NIFI-6323 Changed URLs for repositories, project description, and mailing lists to use HTTPS.
NIFI-6323 Changed URLs for splunk.artifactoryonline.com to use HTTPS (certificate validity warning in browsers, but command-line connection using openssl s_client is successful).
NIFI-6323 Changed URLs for XMLNS schema locations to use HTTPS (the XMLNS and schema identifier remain http:// because they are not designed to be resolvable).
NIFI-6323 Fixed Maven XML schema descriptor URLs.

This closes #3497
2019-05-29 14:36:40 -04:00
Koji Kawamura 99b20ac2d1 NIFI-6303 CSVWriter should write schema defined fields first, then extra ones
This closes #3484

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-05-26 16:52:12 -04:00
Peter Turcsanyi 2bfffd35b2 NIFI-6317: HandleHttpRequest timeout handling issues
- set cleanup thread scheduling to 5 seconds in order to prevent stale request staying in the cache for a long time
- disable timeout handling in HandleHttpRequest in order to prevent a new flowfile being generated at request expiration
- use HttpServletResponse.SC* status codes everywhere in HandleHttpRequest, get rid of importing javax.ws.rs
- add some more error logging to make bug investigations easier
- add a short message to 503 error responses to make bug investigations easier

This closes #3490.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-05-24 14:43:50 -04:00
Rob Fellows 0a31c38790
NIFI-6253 - Remove invalid/unnecessary argument in call to nfCanvasUtil.reload
This closes #3491
2019-05-24 11:02:09 -04:00
Rob Fellows 410854f53e
NIFI-6316 - Upgrade jQuery to the latest version (3.4.1).
This closes #3489
2019-05-24 09:52:02 -04:00
thenatog fe68d43e1d
NIFI-6280 - Broke out the matching for /access/knox/** and /access/oidc/** to allow the Jetty security filters to be applied in the /access/oidc/logout and /access/knox/logout cases.
NIFI-6280 - Updated terminology in JwtAuthenticationFilter to authentication instead of authorization. Added stricter token parsing using an explicit regex pattern. Added tests.
NIFI-6280 - Updated terminology from Authorization to Authentication.
NIFI-6280 - Updated the access logout method to use getNiFiUserIdentity(). Updated javascript logout method to handle errors.
NIFI-6280 - Fixing checkstyle issues.
NIFI-6280 - Added some javadoc comments and logging. Renamed some variables for clarity. Fixed handling of exception when JWT does not match expected format.
NIFI-6280 - Cleaned up checkstyle, increased log severity level for logout action, and cleaned up Groovy syntax in test.

This closes #3482.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-05-22 17:05:06 -07:00
Koji Kawamura 8a50cb10b2 NIFI-2933 Remote input/output ports at any PG
Specify remote access at port creation.
Incorporated comments, and finished refactoring.
Renamed RootGroupPort to PublicPort.
Fix error message for creating a connection from a child PG having only PublicPorts.
Enhanced ProcessGroup instances rendered in the parent ProcessGroup
Loosen Port move check, allow moving public port between PG.
Show 'Remote NiFi Instance' info on Connection dialogs
Make labels narrative.
'Within Remote Group'.
Fixed DTO (de)serialization.
Return null only if all values are null.

This closes #3351.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-05-22 15:44:16 -04:00
Matt Gilman a97766d62f NIFI-6302:
- Ensuring Process Group contents are pruned when appropriate.

This closes #3477.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-05-22 14:12:50 -04:00
Aaron Leon 81ddd02ca8 NIFI-6100 Use setBytes in JdbcCommon for binary types
This closes #3410

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-05-21 14:22:56 -04:00
Mark Payne cbafd8f834 NIFI-5922: Addressed checkstyle violations; added to README Updates to nifi-fn proposal:
- Separated into nifi-fn-core, nifi-fn-bootstrap, nifi-fn-nar, nifi-fn-assembly; fully unpacks nars and runs flows
- Rebased against master and updated to version 1.10.0-SNAPSHOT
- Removed dependency on nifi-framework-core
- Added LICENSE/NOTICE files
2019-05-21 11:19:17 -04:00
Otto Fowler 4b509aa5a5
NIFI-3221 This closes #3396. Add a new property for setting the argument passing strategy, either the existing parameter, or by adding new dynamic parameters, along with implementation and tests
This allows for passing arguments with quotes.

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-05-15 15:03:07 -04:00
Michael Karpel d1fd1f5092 Support for flowfile attribute in TABLE_NAME
This closes #3472

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-05-15 12:31:11 -04:00
Mark Payne f08c2ee43f
NIFI-6285: Addressed issue that resulted in swapped data not being swapped back in if load balancing strategy changed while data was swapped out; added integration tests for swapping. In testing, also encountered an issue with data being swapped out while swap files were being recovered causing the queue size to be wrong and causing errors about not being able to swap data in, because it attempted to swap the data in twice.
This closes #3473.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-05-14 15:31:15 -04:00
Bryan Bende 7825e40d15
NIFI-6262 - Upgrade to nifi-nar-maven-plugin 1.3.1
This closes #3463.
2019-05-14 11:00:21 -04:00
Endre Zoltan Kovacs 3a6e28eaa4 NIFI-6289: using charset for byte encoding in ExecuteSparkInteractive
This closes #3468

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-05-14 06:35:45 -04:00
Scott Aslan 95fd14866e
[NIFI-6292] update variables slickgrid table row ids
This closes #3469
2019-05-13 09:57:19 -04:00
Jan Hentschel ecc014937a NIFI-5425 Removed duplicated declaration of nifi-utils and nifi-processor-utils
This closes #2888

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-05-11 18:26:34 -04:00
Alex Savitsky 060f4fe73f NIFI-6011 support for retrieving named schema versions
This closes #3297

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-05-10 18:20:18 -04:00
Mark Bean a2bacde62c
NIFI-6225: bug fix for listing relationships in sorted order
This closes #3459
2019-05-10 15:00:45 -04:00
Endre Zoltan Kovacs dc38a6bdce NIFI-6288: hooking in & using charset to HTTP based ES processors
This closes #3467

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-05-10 09:30:33 -04:00
Jeff Storck 65c41ab917 NIFI-6196: Upgrade Jetty version to 9.4.15.v20190215
Updated NOTICE with current copyright year for Jetty dependencies
Updated Jetty SSLContextFactory usage, invoke setEndpointIdentificationAlgorithm(null) on server SslContextFactory instances
Updated TestInvokeHttpSSL to provide a separate client keystore, rather than reusing the server's keystore
Regenerated nifi-standard-processors keystore and truststore, added client keystore
Updated ITestHandleHttpRequest, TestInvokeHttpSSL, TestInvokeHttpTwoWaySSL, and TestListenHTTP to use a separate client keystore instead of reusing the server's keystore.  Also updated the tests to separately test one-way and two-way SSL

NIFI-6196 - Setting the endpoint identification algorithm to null for sockets to ensure certificates without SANs applied correctly still work. We can change this in a new NiFi release with other potentially breaking changes.

This closes #3426.
2019-05-09 12:05:12 -04:00
thenatog cf6f517250
NIFI-6085 - Added /access/logout endpoint to allow JWT auth tokens to be removed correctly. Added some tests. Found an error in the KeyDAO which did not allow key deletion.
NIFI-6085 - Updated logOut method to use NiFiUserUtils and updated tests.
NIFI-6085 - Added some more integration tests.
NIFI-6085 Suppressed stacktrace when token is used after being invalidated.

This closes #3362.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-05-08 17:45:39 -07:00
Jeff Storck 376c344edb NIFI-5574 Removed usage of Paths.get() due to TEST_RES_PATH being resolved to a string from a URI, which results in platform-specific path information (C:\) when tests are run on Windows.
This closes #2988

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-05-08 07:38:40 -04:00
Tomer Ghelber ba642485cc NIFI-5933 Stopped using sun's IOUtils. Using apahce's instead.
NIFI-5933 Removed unused file with sun.* import
NIFI-5933 Removed unsafe of sun.*.
NIFI-5933 Added openjdk8 to test

This closes #3234

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-05-07 21:58:16 -04:00
Matthew Burgess f40b7fc6ba NIFI-2215: Added support for onScheduled and onStopped in InvokeScriptedProcessor
NIFI-2215: Incorporated review comments

This closes #3370

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-05-07 20:05:33 -04:00
Jeff Storck 25d8f64bed NIFI-6178: Implemented capability to add IP SANs to certs
Regenerated test certs for nifi-livy-processors and nifi-standard-processors to fix some broken SSL tests in those modules

This closes #3416.
2019-05-07 15:59:19 -04:00
Rob Fellows 4484565df1
NIFI-6253 - Fix: UI - Process Group ID display in Operate palette can get out of sync when process groups are removed from different sessions.
NIFI-6253 - simplifying loading of a new process group, handle failure.
NIFI-6253 - Expose nfCanvas.loadProcessGroup functionality through nfCanvasUtils.loadProcessGroup
NIFI-6253 - optional groupId approach to reload

This closes #3458
2019-05-07 15:34:12 -04:00