147 Commits

Author SHA1 Message Date
Sivaprasanna
94c2b1e76e NIFI-6049: Updated MoveHDFS to support FF Attribute for Output Directory (#3433)
NIFI-6049: Updated MoveHDFS to support FF Attribute for Output Directory
2019-04-22 10:12:16 -06:00
joewitt
25cc7b4a1e
NIFI-6029 merging nifi 1.9.0 release into master 2019-02-19 22:55:49 -05:00
Denes Arvay
76e92c8682 NIFI-6052 Update NOTICE files to reflect 2019
This closes #3319

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-19 18:32:15 -05:00
joewitt
0e204f3576
NIFI-6029-RC2 prepare for next development iteration 2019-02-16 21:50:35 -05:00
joewitt
45bb53d2aa
NIFI-6029-RC2 prepare release nifi-1.9.0-RC2 2019-02-16 21:50:15 -05:00
Jeff Storck
b7ca59f568
NIFI-6023: Deprecated the "Distributed Cache Service" property in ListHDFS
Updated exception message thrown when accessing the ListHDFS state to refer to the State Manager rather than the "Distributed Cache Service"
Removed usage and references to MockCacheClient from TestListHDFS

NIFI-6023: Minor update of the "Distributed Cache Service" description

This closes #3313.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-02-15 14:12:04 -05:00
Jeff Storck
72244d09ff NIFI-5575: Added Hadoop configuration to MockFileSystem for umask tests in PutHDFSTest
Minor update to Permissions umask property description
Minor code cleanup/formatting

This closes #3252
2019-02-13 16:09:12 -05:00
Key Miyauchi
7a763d3c49 NIFI-5575 Make PutHDFS check fs.permissions.umask-mode if property "Permission umask" is empty. 2019-02-13 16:08:57 -05:00
Jeff Storck
c0182294ed NIFI-5720-RC3 prepare for next development iteration 2018-10-22 22:16:43 -04:00
Jeff Storck
98aabf2c50 NIFI-5720-RC3 prepare release nifi-1.8.0-RC3 2018-10-22 22:16:23 -04:00
Jeff Storck
e24388aa7f NIFI-5557 Added test in PutHDFSTest for IOException with a nested GSSException
Resolved most of the code warnings in PutHDFSTest

This closes #2971.
2018-09-27 16:24:31 -04:00
Endre Zoltan Kovacs
0f55cbfb9f NIFI-5557: handling expired ticket by rollback and penalization 2018-09-27 16:24:31 -04:00
uday
57ae9b65a0 NIFI-1184 Mock FileSystem for PutHDFSTest
This closes #2892

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-09-14 13:30:03 -04:00
Jeff Storck
451084e11f NIFI-4434 Fixed recursive listing with a custom regex filter.
Filter modes are now supported to perform listings based on directory and file names, file-names only, and full path.

This closes #2937

Signed-off-by: zenfenan <zenfenan@apache.org>
2018-08-08 22:06:50 +05:30
Andy LoPresto
f60585a9b6
NIFI-5376 Removed deprecation warnings.
Updated Javadoc for SiteToSiteClient#createTransaction() and HttpClient implementation.
Reverted exception listing in method contract for SiteToSiteClient#createTransaction and HttpClient tion of same.
Reverted import ordering in TestSiteToSiteClient.
Reverted exception listing in TestGetHDFSFileInfo, TestListHDFS, and StandardHttpFlowFileServerProtocol.
Restored @SuppressWarnings annotation and removed unnecessary "public static" keywords from inner classes in SiteToSiteClient.

This closes #2841.

Signed-off-by: Joe Witt <joewitt@apache.org>
2018-07-09 20:45:34 -07:00
Andy LoPresto
d42a1e8bf4
NIFI-5323-RC1 prepare for next development iteration 2018-06-19 20:02:21 -07:00
Andy LoPresto
99bcd1f88d
NIFI-5323-RC1 prepare release nifi-1.7.0-RC1 2018-06-19 20:02:01 -07:00
Ed
568959baf5
NIFI-4906 Add GetHDFSFileInfo
- Updated doc for write attributes
- Removing redundant files
- Updates based on review

Changes:
- changed validators for regex params (3)
- removed TriggerWhenEmpty and TriggerSerially.
- removed unused LISTING_LAG_NANOS
- removed unimplemented customValidate
- removed unsused error handling for building/updating request params
- added MIME_TYPE for content
- suppressed output to original relationship when scheduled
- added full path to original/new FF when not found
- added warning messages for huge attribute values.

This closes #2639.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-06-18 09:33:42 -04:00
zenfenan
0973c2d8d1 NIFI-543 Added annotation to restrict processor to run only on the primary node
- PR Fix - 'Execution' dropdown will now be shown in all cases
- Annotated ListGCSBucket with PrimaryNodeOnly

This closes #2509.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-05-25 11:50:29 -04:00
zenfenan
4544f3969d
NIFI-5152: MoveHDFS now works even with no upstream connection
This closes #2681.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-05-07 13:20:56 -04:00
Jeff Storck
88c079e447
NIFI-4923 Updated nifi-hadoop-libraries-nar, nifi-hdfs-processors, and nifi-hadoop-utils dependency on hadoop-client from 2.7.3 to 3.0.0
This closes #2582.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-04-25 15:34:14 -04:00
zenfenan
c118e96238 NIFI-5000 - ListHDFS properly lists files from updated directory path
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2576.
2018-04-25 18:03:44 +02:00
joewitt
59f625d3c2 Merge branch 'NIFI-4995-RC3' as part of NiFi 1.6.0 release process 2018-04-06 16:48:08 -07:00
Pierre Villard
4c787799ff NIFI-4149 - Indicate if EL is evaluated against FFs or not
- take into account input requirement for documentation rendering
- Renamed variable registry scope and added comments
- Doc + change in mock framework to check scope + update of components + UI
2018-04-06 12:40:54 -04:00
joewitt
d511fe3e4b NIFI-4995-RC3 prepare for next development iteration 2018-04-03 08:28:34 -07:00
joewitt
f8466cb16d NIFI-4995-RC3 prepare release nifi-1.6.0-RC3 2018-04-03 08:28:15 -07:00
joewitt
7b5bf265a6 NIFI-4995 updating copyright year on all notices 2018-03-26 21:54:10 -04:00
Matt Gilman
69a564e4c8
NIFI-5009:
- Fixing required permission for PutParquet.

NIFI-5008:
- Ensuring all restricted components are tagged as such.

This closes #2583.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-03-26 09:47:54 -04:00
Mark Payne
0b0aebe148
NIFI-4917: Externalize Keytab and Principal configuration from Processors to a Controller Service. This gives us the ability to allow users to interact with those Keytabs/Principals to which they've been given access without allowing them access to all Keytabs and Principals
- Addressed review feedback; rebased against master

This closes #2552.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-03-21 14:23:41 -04:00
Matt Gilman
b1217f529b NIFI-4885:
- Introducing more granular restricted component access policies.

This closes #2515.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-03-12 14:19:42 -04:00
joewitt
c71409fb5d
NIFI-4936 trying to quiet down the mvn output a bit so we dont exceed the travis-ci 4MB max
NIFI-4936 updated dependency handling pushing down delcarations where they belong
This closes #2512
2018-03-09 16:34:53 -05:00
Pierre Villard
dc67bd2fdd NIFI-4747 - Removed directory existence check in GetHDFS
This closes #2391

Signed-off-by: Jeremy Dyer <jeremydyer@apache.org>
2018-01-31 12:01:17 -05:00
joewitt
41ce788812 NIFI-4751 changed to next minor release version snapshot 2018-01-12 15:15:32 -05:00
joewitt
36405e888c NIFI-4751-RC1 prepare for next development iteration 2018-01-08 23:39:49 -07:00
joewitt
46d30c7e92 NIFI-4751-RC1 prepare release nifi-1.5.0-RC1 2018-01-08 23:39:32 -07:00
Jeff Storck
42a1ee011b NIFI-4323 This closes #2360. Wrapped Get/ListHDFS hadoop operations in ugi.doAs calls
NIFI-3472 NIFI-4350 Removed explicit relogin code from HDFS/Hive/HBase components and updated SecurityUtils.loginKerberos to use UGI.loginUserFromKeytab. This brings those components in line with daemon-process-style usage, made possible by NiFi's InstanceClassloader isolation.  Relogin (on ticket expiry/connection failure) can now be properly handled by hadoop-client code implicitly.
NIFI-3472 Added default value (true) for javax.security.auth.useSubjectCredsOnly to bootstrap.conf
NIFI-3472 Added javadoc explaining the removal of explicit relogin threads and usage of UGI.loginUserFromKeytab
Readded Relogin Period property to AbstractHadoopProcessor, and updated its documentation to indicate that it is now a deprecated property
Additional cleanup of code that referenced relogin periods
Marked KerberosTicketRenewer is deprecated

NIFI-3472 Cleaned up imports in TestPutHiveStreaming
2018-01-03 11:31:47 -05:00
Jeff Storck
600586d6be NIFI-3366 This closes #2332. Added parent/child flowfile relationship between the incoming flowfile and the files that are moved from the input directory to the output directory.
Updated to allow tests to check for evaluation of properties that support expression language.
Fixed bug with changeOwner attempting to operate on original file rather than the moved/copied file.
Added license header to MoveHDFSTest.java
Added test for moving a directory of files that contains a subdir, ensuring non-recursive behavior
Added to the description of the processor that it is non-recursive when a directory is used as input.
Added RAT exclude for test resource .dotfile to pom.xml

Signed-off-by: joewitt <joewitt@apache.org>
2017-12-11 08:41:36 -05:00
Gray Gwizdz
3731fbee88 NIFI-3366 MoveHDFS processor supports expressions language for input and copy operations
Signed-off-by: Jeff Storck <jtswork@gmail.com>
2017-12-11 08:41:25 -05:00
Koji Kawamura
2a69b35f12 NIFI-4548: Add REMOTE_INVOCATION provenance event type
This commit includes changes to DeleteHDFS to report REMOTE_INVOCATION
event. In order to do so, the processor had to be changed to create
output FlowFile because a provenance event needs a FlowFile it
associates with.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2234.
2017-11-02 19:32:22 +01:00
Koji Kawamura
77a51e1a9e NIFI-4544: Improve HDFS processors provenance transit URL
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2238.
2017-11-02 10:10:03 +01:00
Takanobu Asanuma
13e42678b6 NIFI-4338. This closes #2143. add docs for ssl configurations in HDFS processors
remove redundant additionalDetails.html and add docs to CapabilityDescription in HDFS processors

revert the modified CapabilityDescriptions in HDFS processors and add it to AbstractHadoopProcessor
2017-10-10 00:17:44 -04:00
Jeff Storck
a57911d3db NIFI-4412-RC2 prepare for next development iteration 2017-09-28 13:45:36 -04:00
Jeff Storck
e6508ba7d3 NIFI-4412-RC2 prepare release nifi-1.4.0-RC2 2017-09-28 13:45:21 -04:00
Bryan Bende
cf57639396 NIFI-4311 Allowing umask to get set properly before initializing the FileSystem
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2106.
2017-08-22 22:40:26 +02:00
Davide
d334532b16 NIFI-4184: Enabled EL on PutHDFS Remote Group and Owner
I needed to put some attributes on REMOTE_GROUP and REMOTE_OWNER, in order to achieve it i put expressionLanguageSupported(true) on the PropertyDescriptor of REMOTE_GROUP and REMOTE_OWNER

This closes #2007.

Signed-off-by: Davide <davidde85@hotmail.it>
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-07-21 16:31:46 +09:00
Pierre Villard
7843b885ee NIFI-4144 - added min/max age to ListHDFS processor
This closes #1966.

Signed-off-by: Tony Kurc <tkurc@apache.org>
2017-07-04 21:18:43 -04:00
Jeff Storck
c99100c934
NIFI-4010 Enables EL on Fetch/List/PutSFTP and List/Fetch/Put/DeleteHDFS processor properties
FetchSFTP/ListSFTP/PutSFTP: Private Key Path
ListHDFS/FetchHDFS/PutHDFS/DeleteHDFS: Hadoop Configuration Resources, Kerberos Principal, Kerberos Keytab, Kerberos Relogin Period

This closes #1148
This closes #1930.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-06-21 17:14:49 -04:00
Matt Gilman
490e1da5db NIFI-4019:
- Adding support for X-Forwarded-* headers.
- Unrelated code clean up.
- Addressing additional proxying issues.
- Addressing mistyped landing page.
- Handled trailing slashes in proxy headers for context path
2017-06-15 14:17:35 -04:00
Matt Gilman
cc741d2be6
NIFI-3997:
- Bumping to next minor version.
2017-06-08 15:22:51 -04:00
Matt Gilman
6ee12e9b47
NIFI-3997-RC1prepare for next development iteration 2017-06-05 11:07:43 -04:00