Commit Graph

3308 Commits

Author SHA1 Message Date
Mark Payne 1be0871473 NIFI-2854: Refactor repositories and swap files to use schema-based serialization so that nifi can be rolled back to a previous version after an upgrade.
NIFI-2854: Incorporated PR review feedback

NIFI-2854: Implemented feedback from PR Review

NIFI-2854: Ensure that all resources are closed on CompressableRecordReader.close() even if an IOException is thrown when closing one of them

This closes #1202
2016-11-18 14:53:13 -05:00
Andrew Lim 5a25884f52 NIFI-3052 Update screenshots in Admin Guide and Getting Starting guide for additional colors added to UI
This closes #1239
2016-11-18 14:05:17 -05:00
Andy LoPresto 8568d40cd8
NIFI-3051 Fixed issue serializing commented or empty login-identity-providers.xml.
Cleaned up commented pom.xml contents. (+4 squashed commits)
Squashed commits:
[725860b] NIFI-3051 Switched CET test logging dependency from logback to log4j as Zookeeper migration has an explicit dependency on log4j as the logging provider and the two libraries were causing classpath collisions. Now the tests run in both Maven and Intellij, the tools all build successfully, and the tools all run (TLS Toolkit, CET, and ZK) without logging provider warnings and print the expected output to the console.
[0e604c7] NIFI-3051 Changed provider element selection in serialize method to be by class (org.apache.nifi.ldap.LdapProvider) rather than identifier in case it has been modified.
Added unit tests.
[300a23d] NIFI-3051 Changed provider element selection to be by class (org.apache.nifi.ldap.LdapProvider) rather than identifier in case it has been modified.
Added unit tests.
[a0cdd40] NIFI-3051 Fixed issue serializing commented or empty login-identity-providers.xml.
Updated and added unit tests. (+1 squashed commit)
Squashed commits:
[b187202] NIFI-3051 - checked in test demonstrating failure to serialize commented ldap-provider section.

This closes #1238.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2016-11-18 01:12:00 -08:00
James Wing 7206318ecf NIFI-3033 GenerateFlowFile Dynamic Properties
This closes #1222.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2016-11-18 11:06:20 +09:00
Jeff Storck 06d7ecd324 NIFI-3059 Adds --ignore-source option to the ZooKeeper Migrator to allow data read from a source zookeeper to be written back to the same zookeeper
Added unit test to test the --ignore-source option

This closes #1242
2016-11-17 18:24:58 -05:00
jpercivall c4be800688 NIFI-2950 Adding support for whole number hex values and a fromRadix function
NIFI-2950 Fixing typo

This closes #1161
2016-11-17 07:27:51 -05:00
Bryan Rosander e5eda63705
NIFI-2943 - Toolkit uses JKS type over PKCS12 when creating truststore because non-Bouncy Castle providers cannot read certificates from PKCS12 truststore.
Peer review feedback (+2 squashed commits)
Squashed commits:
[0102c8e] NIFI-2943 - Peer review feedback
[9bcd495] NIFI-2943 - pkcs12 keystore improvements

1. loading pkcs12 keystores with bouncy castle everywhere
2. tls-toolkit client using jks truststore when keystore type is specified differently
3. tests

This closes #1165.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2016-11-16 17:13:10 -08:00
Jeff Storck fa13832a9c
NIFI-3049 Fixes logging issues due to logback and log4j being on the classpath
Removed logback usage from classpath
Added slf4j-log4j12 dependency in nifi-toolkit pom
Added logback-classic exclusion for nifi-properties-loader used by nifi-toolkit-encrypt-config
Updated log4j.properties logging pattern and logger config in nifi-toolkit-assembly and nifi-toolkit-zookeeper-migrator, filtering zookeeper messages below WARN
Removed logback.groovy since log4j is the single logging implementation
Updated ZooKeeperMigratorMain command line output to match standards established by other tools in nifi-toolkit

This closes #1237.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2016-11-16 15:33:11 -08:00
Scott Aslan 878db82375 [NIFI-2949] update remote process group port styles. This closes #1226 2016-11-16 17:08:06 -05:00
Oleg Zhurakousky f1eca0a09a NIFI-1002 added L&N entries for Bouncy Castle
This closes #1235
2016-11-16 15:30:47 -05:00
Andrew Lim 2c135c44f2 NIFI-2985 Update User Guide for Backpressure Visual indicator
This closes #1234.
2016-11-16 20:58:10 +01:00
Koji Kawamura 31ea50bce9 NIFI-1002: Added WebSocket support.
- Added LICENSE and NOTICE files.
- Fixed UnitTest errors.

This closes #1231
2016-11-16 12:17:37 -05:00
Bryan Bende 2f9ec03242 NIFI-2909 Fix logic in AbstractConfiguredComponent setProperties() for setting classpath resources in the InstanceClassLoader
This closes #1232
2016-11-16 11:34:52 -05:00
Matt Gilman 49afacc3ab NIFI-3022:
- Returning the appropriate authorizable when accessing provenance events for a manual DROP event by emptying a queue.
- Populating the component details of a provenance event when the source is a connection.
2016-11-16 10:13:18 -05:00
joewitt 45a5f5295c NIFI-2374 This closes #712. updated to latest tika versions where possible, updated doc, commented why cannot update media nar 2016-11-15 23:18:55 -05:00
Andre F de Miranda 3e571f9f1f NIFI-2374 - Adjust documentation wording to clarify IdentifyMimeType
is a non exhaustive list of mime type values
2016-11-15 22:42:21 -05:00
Koji Kawamura 26a5881d21 NIFI-1002: Added WebSocket support.
NIFI-1002: Added WebSocket support.

- Reflecting review comments
- Added displayName to peroperty descriptors

NIFI-1002: Added WebSocket support.

This closes #1184

- Reflecting review comments:
  - Removed unnecessary use of NarCloseable.withComponentNarLoader.
  - Removed intermediate on memory queue to make it simpler and more
    robust. Received messages in WebSocket layer now will be transferred
to downstream relationships directly.
2016-11-15 19:34:59 -05:00
joewitt b026f0bebe NIFI-2991 removed json.org references as appropriate for build. Twitter proc no longer deployed but reference kept in notice just in case someone uses it
This closes #1230

Signed-off-by: jpercivall <JPercivall@apache.org>
2016-11-15 18:09:27 -05:00
joewitt 6696529c99 Revert "NIFI-1893 Add processor for validating JSON"
This reverts commit f11682202b.

Signed-off-by: jpercivall <JPercivall@apache.org>
2016-11-15 18:09:14 -05:00
joewitt 1c9077334f NIFI-2991 removed usage of json lib and components still using it
Signed-off-by: jpercivall <JPercivall@apache.org>
2016-11-15 18:09:09 -05:00
Mark Payne 721964b7d8 NIFI-3036: When we replay a FlowFile, ensure that we are using the 'golden copy' of the associated Resource Claim, if the claim is still writable. Ensure that StandardResourceClaimManager retains the 'golden copy' of a Resource Claim until it is no longer writable and has a claim count of 0
This closes #1223

Signed-off-by: jpercivall <JPercivall@apache.org>
2016-11-15 16:51:34 -05:00
Scott Aslan 81357d4456 [NIFI-3042] Update class name for disabled combo option. This closes #1228 2016-11-15 16:43:27 -05:00
Mark Payne 9f3bf121f9 NIFI-3040: Fixed bug where we were generating a RepositoryRecord with an 'UPDATE' type instead of a 'CREATE' time for Replay of FlowFiles. This caused FlowFile to have no attributes when restored from the FlowFile Repository.
This closes #1224
2016-11-15 16:37:51 -05:00
jpercivall 2614b83543 NIFI-3043 Fixing Kafka unit test failures 2016-11-15 16:31:10 -05:00
Bryan Bende fe59b3415c NIFI-3003 Upgrading hadoop.version to 2.7.3 and fixing TDE issue with PutHDFS, ensuring clean up of instance class loaders, and adding classpath resource property to all HDFS processors
NIFI-3003 Addressing review feedback

NIFI-3003 added minor notice updates

This closes #1219
2016-11-15 16:23:27 -05:00
Mark Payne 3c694b641e NIFI-2751: This closes #1211. 2016-11-15 16:19:28 -05:00
Pierre Villard 45bc3e054d NIFI-2751 - Pull batch in round-robin manner - fix for / by 0 2016-11-15 16:19:00 -05:00
Andrew Lim 76b859c4ef NIFI-2953 Update Multi-tenant authorization doc for changes to policy management UI. This closes #1225 2016-11-15 16:10:46 -05:00
Scott Aslan 45bf8430fc [NIFI-2844] Update CSS styles for Cluster Summary Dialog icons and refresh button. This closes #1104 2016-11-15 11:55:42 -05:00
Matt Burgess 97cd94d260 NIFI-2684: Additional check for displayName
This closes #1002
2016-11-15 11:48:50 -05:00
Andre F de Miranda ef2c50577d NIFI-2684 - Change AbstractConfigurableComponent to use displayName when available 2016-11-15 11:48:33 -05:00
Scott Aslan 8f928e5d6e NIFI-2926: add policy action item to the user table and add user centric policy dialog with user policy listing table compolete with GoTo action.
This closes #1188.
2016-11-15 09:58:58 -05:00
Andy LoPresto 59fea1cb4e NIFI-2654 Enabled encryption coverage for login-identity-providers.xml.
Squashed commits:
[5dd22a9] NIFI-2654 Updated administration guide with login-identity-providers.xml flags.

Exposed master key retrieval code in NiFiPropertiesLoader.
Added logic to decrypt login identity providers XML configuration.
Updated login-identity-providers.xsd to include encryption scheme attribute.
Added unit tests. (+18 squashed commits)
Squashed commits:
[57c815f] NIFI-2654 Resolved issue where empty LIP property elements could not be encrypted.
Added unit test and resource.
[27d7309] NIFI-2654 Wired in serialization logic to write logic for LIP.
Added comprehensive unit test for LIP & NFP in same test.
[b450eb2] NIFI-2654 Finalized logic for preserving comments in LIP parsing.
[5aa6c9c] NIFI-2654 Added logic for maintaining XML formatting (comments and whitespace) for LIP.
Added unit tests (w/o encryption works; w/ does not).
[b53461f] NIFI-2654 Added unit test for full tool invocation migrating a login-identity-providers.xml file and updating file and bootstrap.conf with key.
[2d9686c] NIFI-2654 Updated tool description and various logging statements.
Added unit test for full tool invocation encrypting a login-identity-providers.xml file and updating file and bootstrap.conf with key.
[8c67cb2] NIFI-2654 Added logic to encrypt LIP XML content.
Added unit tests.
[8682d19] NIFI-2654 Added logic to handle "empty" (commented) LIP files.
Added unit tests.
[077230e] NIFI-2654 Fixed logic to decrypt multiline and multiple-per-line XML elements.
Added unit tests and resources.
[d5bb8da] NIFI-2654 Ignored unit test for unreadable conf directory because directory was causing Maven build issues.
Removed test resources.
[7e50506] NIFI-2654 Fixed AESSensitivePropertyProvider bug handling cipher text with whitespace.
Added unit test.
[b69a661] NIFI-2654 Fixed AESSensitivePropertyProviderFactoryTest to reflect absence of key causes errors.
[6f821b9] NIFI-2654 Added standard password to arbitrary encryption test for use in test resources.
[d289ffa] NIFI-2654 Added LIP XML decryption.
Added unit tests.
[a482245] NIFI-2654 Added LIP test resources.
[7204df4] NIFI-2654 Changed logic to only perform properties encryption when file path is provided.
[729e1df] NIFI-2654 Removed population of default file locations for bootstrap.conf, nifi.properties, and login-identity-providers.xml as not all files may be desired.
Added/updated unit tests.
[7dba5ef] NIFI-2654 Started LIP work (arguments & parsing).
Added unit tests.

Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>

This closes #1216
2016-11-14 22:39:06 -05:00
joewitt 8ad3125168 NIFI-2380 This closes #817. corrected licensing and notice and remove conflicting mail deps 2016-11-14 21:04:33 -05:00
Andre F de Miranda a347ebf87b NIFI-2380 - Introduce ExtractTNEFAttachment - A processor to extract
attachments embedded within a Transport Neutral Encapsulation Format (TNEF) attachment (i.e. winmail.dat)
2016-11-14 21:04:12 -05:00
Andy LoPresto 4d1bcc808f
NIFI-3032 Resolved issue where multiple invocations of NiFiPropertiesLoader.withKey() used cached key.
Added unit tests and resources.

NIFI-3032 Fixed bug in AESSensitivePropertyProvider#getIdentifierKey where the result was always the max available key size, not the size of the current key.
Added unit test.

This closes #1220

Signed-off-by: Bryan Rosander <brosander@apache.org>
2016-11-14 17:50:32 -05:00
Jeff Storck 7e23734181
NIFI-2957 Corrections of typos, documentation, and strings
Updates made regarding Intellij code inspections

This closes #1218

Signed-off-by: Bryan Rosander <brosander@apache.org>
2016-11-14 16:59:24 -05:00
Matt Gilman ae4f27a627 NIFI-3023:
- Ensuring there are no authorization issues prior to considering the remote process group's transmitting flag.
This closes #1205
2016-11-14 16:23:36 -05:00
Matt Gilman 2c91a1a33d NIFI-3023:
- Verifying permissions prior to checking Remote Process Group transmission status.
2016-11-14 16:03:47 -05:00
Mark Payne b0f6825e09 NIFI-1526: This closes #1107. 2016-11-14 12:35:18 -05:00
Mathias Tiberghien 65b7b377e3 NIFI-1526: DefaultSchedule annotation should be use on Custom Processor to set the default scheduling strategy, scheduling period or max number of concurrent task for each instance of the processor
DefaultSettings annotation should be use on Custom Processor to set the default penalty period, the yield duration or the bulletin log level for each instance of the processor
2016-11-14 12:04:21 -05:00
Randy Gelhausen 8d3177c38a NIFI-3025: Bump nifi-spark-receiver's jackson version to match Spark 2.0.1
This closes #1207
2016-11-14 09:33:55 -05:00
Koji Kawamura 13ea909122
NIFI-2851: Fixed CheckStyle error.
This closes #1215.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2016-11-14 00:44:16 -08:00
Mark Payne ad92474593 NIFI-2851: Added additional unit test to ensure correctness of demarcation when demarcator falls between buffered data
This closes #1116.
2016-11-11 15:45:40 -05:00
Oleg Zhurakousky 41f519e84c NIFI-2851 initial commit of perf improvements on SplitText
- introduced org.apache.nifi.stream.io.util.TextLineDemarcator
- refactored SplitText to use org.apache.nifi.stream.io.util.TextLineDemarcator
- updated SplitText's capability discription to provide more clarity around splits with headers.
2016-11-11 15:45:40 -05:00
Mark Payne b73ba7f8d4 NIFI-2999: When Cluster Coordinator changes, purge any old heartbeats so that we don't disconnect a node due to very old heartbeats
This closes #1210
2016-11-11 15:13:49 -05:00
joewitt b9ef0fb847 NIFI-2818 This closes #1059. aligned read method with read/write method 2016-11-11 14:52:13 -05:00
Andre F de Miranda f32bdf7be9 NIFI-2818 - Minimise fs permission required by NiFi
1 - Replace a r/w test over $NIFI_HOME/lib that is performed
            but never utilised by a RO test
            2 - Rename ensureDirectoryExistAndCanAccess to
            ensureDirectoryExistAndCanReadAndWrite and deprecate the
            former
2016-11-11 14:49:33 -05:00
Scott Aslan 52cde9ad13 [NIFI-2898] restore ellipsis for processor type, controller servies type, and reporting task type descriptions. This closes #1191 2016-11-11 14:46:20 -05:00
Mark Payne 57e6a2d515 This closes #1209. 2016-11-11 14:43:42 -05:00