Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
22,286 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

5146 Commits

Author SHA1 Message Date
Chen Liang 8fb5ca3f40 HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang. 2019-07-11 13:23:25 -07:00
Steve Loughran 6a3433bffd
HADOOP-16357. TeraSort Job failing on S3 DirectoryStagingCommitter: destination path exists. 
Contributed by Steve Loughran.

This patch

* changes the default for the staging committer to append, as we get for the classic FileOutputFormat committer
* adds a check for the dest path being a file not a dir
* adds tests for this
* Changes AbstractCommitTerasortIT. to not use the simple parser, so fails if the file is present.

Change-Id: Id53742958ed1cf321ff96c9063505d64f3254f53
 2019-07-11 18:15:34 +01:00
Inigo Goiri aa9f0850e8 HADOOP-16385. Namenode crashes with 'RedundancyMonitor thread received Runtime exception'. Contributed by Ayush Saxena. 2019-07-02 13:22:30 -07:00
Christopher Gregorian 129576f628 HDFS-14403. Cost-based extension to the RPC Fair Call Queue. Contributed by Christopher Gregorian. 2019-06-24 12:09:17 -07:00
Ajay Kumar 95c94dcca7
HADOOP-16350. Ability to tell HDFS client not to request KMS Information from NameNode. Ccontributed by Greg Senia, Ajay Kumar. 2019-06-24 11:59:07 -07:00
Ayush Saxena b52fd05d42 HDFS-13404. Addendum: RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fail. Contributed by Takanobu Asanuma. 2019-06-24 22:03:04 +05:30
Ayush Saxena ba221dc50e HDFS-14526. RBF: Update the document of RBF related metrics. Contributed by Takanobu Asanuma. 2019-06-24 22:03:04 +05:30
Takanobu Asanuma 559cb11551 HDFS-13404. RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fails. 2019-06-24 22:03:03 +05:30
Surendra Singh Lilhore f40e10b349 HDFS-14129. RBF: Create new policy provider for router. Contributed by Ranith Sardar. 2019-06-24 22:03:03 +05:30
Prabhu Joseph 371452e260 HADOOP-15989. Synchronized at CompositeService#removeService is not required. Contributed by Prabhu Joseph. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-21 17:05:13 -07:00
Wei-Chiu Chuang 840d02ca5b HADOOP-14385. HttpExceptionUtils#validateResponse swallows exceptions. Contributed by Wei-Chiu Chuang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-20 19:57:48 -07:00
Steve Loughran e02eb24e0a
HADOOP-15183. S3Guard store becomes inconsistent after partial failure of rename. 
Contributed by Steve Loughran.

Change-Id: I825b0bc36be960475d2d259b1cdab45ae1bb78eb
 2019-06-20 09:56:40 +01:00
Inigo Goiri 3ab77d9bc9 HDFS-14201. Ability to disallow safemode NN to become active. Contributed by Xiao Liang and He Xiaoqiao. 2019-06-18 09:58:29 -07:00
Shweta Yakkali dd4a7633ec HADOOP-16156. [Clean-up] Remove NULL check before instanceof and fix checkstyle in InnerNodeImpl. Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-17 23:47:57 -07:00
Adam Antal 56c621c68f HADOOP-15914. hadoop jar command has no help argument. Contributed by Adam Antal. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-17 22:56:00 -07:00
Wei-Chiu Chuang f1c239c6a4 HADOOP-9157. Better option for curl in hadoop-auth-examples. Contributed by Andras Bokor. 2019-06-17 21:51:33 -07:00
Wei-Chiu Chuang 10311c30b0 HADOOP-14807. should prevent the possibility of NPE about ReconfigurableBase.java. Contributed by hu xiaodong. 2019-06-17 19:55:56 -07:00
Wei-Chiu Chuang 1e92db5a1e HDFS-11949. Add testcase for ensuring that FsShell cann't move file to the target directory that file exists. Contributed by legend. 2019-06-17 18:29:34 -07:00
Gabor Bota f9cc9e1621
HADOOP-16279. S3Guard: Implement time-based (TTL) expiry for entries (and tombstones). 
Contributed by Gabor Bota.

Change-Id: I73a2d2861901dedfe7a0e783b310fbb95e7c1af9
 2019-06-16 17:05:01 +01:00
cxorm 076618677d HADOOP-16336. finish variable is unused in ZStandardCompressor. Contributed by cxorm. 2019-06-14 20:05:20 -07:00
dineshchitlangia c7554ffd5c HADOOP-16373. Fix typo in FileSystemShell#test documentation (#968) 2019-06-14 14:36:04 -07:00
Takanobu Asanuma 54f9f75a44 HADOOP-16369. Fix zstandard shortname misspelled as zts. Contributed by Jonathan Eagles. 2019-06-14 10:17:25 +09:00
Eric Yang 4ea6c2f457 HADOOP-16354. Enable AuthFilter as default for WebHDFS. 
Contributed by Prabhu Joseph
 2019-06-11 18:41:08 -04:00
Wanqiang Ji b417a4c854
MAPREDUCE-7214. Remove unused pieces related to `mapreduce.job.userlog.retain.hours` 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-06-11 18:40:35 +09:00
Shweta Yakkali d6d95d2686 HDFS-14494. Move Server logging of StatedId inside receiveRequestState(). Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-10 10:23:18 -07:00
Erik Krogen 76b94c274f HADOOP-16345. Fix a potential NPE when instantiating FairCallQueue metrics. Contributed by Erik Krogen. 2019-06-07 14:20:44 -07:00
Steve Loughran 4e38dafde4
HADOOP-15563. S3Guard to support creating on-demand DDB tables. 
Contributed by Steve Loughran

Change-Id: I2262b5b9f52e42ded8ed6f50fd39756f96e77087
 2019-06-07 18:26:10 +01:00
Huan-Ping Su 14552d19e3
HADOOP-16344. Make DurationInfo public unstable. 
Second attempt: imports fixed up.

Contributed by Huan-Ping Su and Kai Xie.

Change-Id: Ib2a85dd4b2d12b840692cc854cc53ddb3bdde7d5
 2019-06-07 12:55:17 +01:00
Akira Ajisaka 3ea4f41d9f
MAPREDUCE-6794. Remove unused properties from TTConfig.java 2019-06-07 10:27:41 +09:00
Steve Loughran 1f244b4227
Revert "HADOOP-16344. Make DurationInfo public unstable." 
This reverts commit 829848ba2e.

Change-Id: Ied91250e191b2ba701a8fc697c78b3756ce76be8
 2019-06-06 17:49:31 +01:00
Huan-Ping Su 829848ba2e
HADOOP-16344. Make DurationInfo public unstable. 
Contributed by Huan-Ping Su

Change-Id: I64a94cf382f9db78b4ef49a3912f25f0d906af7c
 2019-06-06 12:52:49 +01:00
Eric Yang 294695dd57 HADOOP-16314. Make sure all web end points are covered by the same authentication filter. 
Contributed by Prabhu Joseph
 2019-06-05 18:55:13 -04:00
Sammi Chen d1aad44490 HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He. 2019-06-05 21:33:00 +08:00
Steve Loughran 309501c6fa
Revert "HADOOP-16050: s3a SSL connections should use OpenSSL" 
This reverts commit b067f8acaa.

Change-Id: I584b050a56c0e6f70b11fa3f7db00d5ac46e7dd8
 2019-06-05 13:54:55 +01:00
Steve Loughran 7724d8031b Revert "HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors" 
This reverts commit 5906268f0d.
 2019-06-05 13:54:42 +01:00
Steve Loughran 827a84778a
HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer -follow-on patch. 
This follow-on patch to HADOOP-16266 fixes up the problem where logs were being full of
stack traces because the timeout passed down to select was in nanos, whereas the API
expected millis.

Contributed by Erik Krogen.

Change-Id: I5c6e9ddf68127b1d7e0ca0e179d036eb9941e445
 2019-06-04 15:56:47 +01:00
Shweta Yakkali 6f5a36c13c HADOOP-13656. fs -expunge to take a filesystem. Contributed by Shweta. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-30 13:21:58 -07:00
Akira Ajisaka afd844059c HADOOP-16331. Fix ASF License check in pom.xml 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:25:13 +09:00
Akira Ajisaka 9f933e6446
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:24:59 +09:00
Shweta 93d7630062 HDFS-14451. Incorrect header or version mismatch log message. Contributed by Shweta. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-24 18:42:46 +02:00
Christopher Gregorian f96a2df38d HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 2019-05-23 10:28:37 -07:00
Eric Yang ea0b1d8fba HADOOP-16287. Implement ProxyUserAuthenticationFilter for web protocol impersonation. 
Contributed by Prabhu Joseph
 2019-05-23 11:36:32 -04:00
Akira Ajisaka a771e2a638
HADOOP-12948. Remove the defunct startKdc profile from hadoop-common. Contributed by Wei-Chiu Chuang. 2019-05-23 13:59:42 +09:00
Sahil Takiar 5906268f0d HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors 2019-05-21 11:30:45 -06:00
Ben Roling a36274d699
HADOOP-16085. S3Guard: use object version or etags to protect against inconsistent read after replace/overwrite. 
Contributed by Ben Roling.

S3Guard will now track the etag of uploaded files and, if an S3
bucket is versioned, the object version.

You can then control how to react to a mismatch between the data
in the DynamoDB table and that in the store: warn, fail, or, when
using versions, return the original value.

This adds two new columns to the table: etag and version.
This is transparent to older S3A clients -but when such clients
add/update data to the S3Guard table, they will not add these values.
As a result, the etag/version checks will not work with files uploaded by older clients.

For a consistent experience, upgrade all clients to use the latest hadoop version.
 2019-05-19 22:29:54 +01:00
Alexis Daboville 4cb3da6ac7
HADOOP-16248. MutableQuantiles leak memory under heavy load. 
Contributed by Alexis Daboville,
 2019-05-17 15:15:22 +01:00
Sahil Takiar b067f8acaa HADOOP-16050: s3a SSL connections should use OpenSSL 
(cherry picked from commit aebf229c17)
 2019-05-16 08:57:54 -06:00
David Mollitor 2713dcf6e9
HADOOP-16307. Intern User Name and Group Name in FileStatus. 
Author:    David Mollitor
 2019-05-16 16:02:07 +02:00
Bharat Viswanadham d4c8858586
HADOOP-16247. NPE in FsUrlConnection. Contributed by Karthik Palanisamy. 2019-05-15 17:41:36 -07:00
Inigo Goiri 389e640f0c HADOOP-16161. NetworkTopology#getWeightUsingNetworkLocation return unexpected result. Contributed by He Xiaoqiao. 2019-05-13 11:46:16 -07:00