Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,685 Commits 326 Branches 371 Tags
Commit Graph

5215 Commits

Author SHA1 Message Date
He Xiaoqiao
03151aa78b
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang. 2021-01-22 00:44:15 +08:00
He Xiaoqiao
a17793534c
Make upstream aware of 3.2.2 release. 2021-01-09 18:07:46 +08:00
Wei-Chiu Chuang
8af0fcf628 HDFS-15719. [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout (#2533) 
(cherry picked from commit 2b4febcf576e2da29ab86e2920302b82b47e435d)
(cherry picked from commit 94c126cc9e9aadaf9aa28ebd55a8712aca54e709)
 2021-01-04 20:56:32 -08:00
He Xiaoqiao
1c0ac5ed5c
HDFS-15751. Add documentation for msync() API to filesystem.md. Contributed by Konstantin V Shvachko. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Chao Sun <sunchao@apache.org>
 2021-01-03 16:31:01 +08:00
He Xiaoqiao
8d7600ae6b HADOOP-17068. Client fails forever when namenode ipaddr changed. Contributed by Sean Chow. 
(cherry picked from commit fa14e4bc001e28d9912e8d985d09bab75aedb87c)
(cherry picked from commit 3a860e876e799ef44fca34e3cb19a89dbd4f6744)
 2020-12-15 14:02:01 -08:00
Jim Brennan
f5a98b6000 HADOOP-13571. ServerSocketUtil.getPort() should use loopback address, not 0.0.0.0. Contributed by Eric Badger 
(cherry picked from commit 6de1a8eb678496393b625f430084a6c64e24b804)
 2020-12-11 20:42:59 +00:00
Ahmed Hussein
d9bf607066 HADOOP-17389. KMS should log full UGI principal. (#2476) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Jim Brennan <jbrennan@apache.org>
(cherry picked from commit 8c234fc0d4472b65ff555a79acb9a12f2dae3b27)
 2020-12-07 17:29:05 +00:00
Takanobu Asanuma
3f14715215
HADOOP-15775. [JDK9] Add missing javax.activation-api dependency. Contributed by Akira Ajisaka. 2020-12-04 22:28:19 +08:00
Hui Fei
6e215953e6
HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao. 2020-12-04 17:11:21 +08:00
Ayush Saxena
af437f1722
HDFS-13404. Addendum: RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fail. Contributed by Takanobu Asanuma. 
(cherry picked from commit b52fd05d42d9a76f6936a5d86c23fcd66244fe3d)
 2020-12-04 11:07:29 +09:00
Takanobu Asanuma
bdcca33e7c
HDFS-13404. RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fails. 
(cherry picked from commit 559cb11551d7045015aa60afe7e7998fe0b2ef13)
 2020-12-04 11:07:19 +09:00
Chao Sun
d82b0cc439
HADOOP-16080. hadoop-aws does not work with hadoop-client-api (#2510). Contributed by Chao Sun 2020-12-03 16:06:33 -08:00
Jim Brennan
9fe779d0e2 HADOOP-17392. Remote exception messages should not include the exception class (#2486). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 5bfb97bc7d5ff5e083aff90e5a380a850d9e4530)
 2020-12-03 18:08:37 +00:00
Andrea Scarpino
66e8f1ae09
YARN-10511. Update yarn.nodemanager.env-whitelist value in docs (#2512) 
Reviewed-by: Adam Antal <adamantal@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9170eb566b173472d9b71141142b31e5824357fb)
 2020-12-04 00:17:48 +09:00
Eric Payne
56ecf469f8 HADOOP-17346. Fair call queue is defeated by abusive service principals. Contributed by Ahmed Hussein (ahussein). 2020-11-23 20:48:07 +00:00
Jim Brennan
6b3d65b46e HADOOP-17367. Add InetAddress api to ProxyUsers.authorize (#2449). Contributed by Daryn Sharp and Ahmed Hussein 2020-11-19 21:55:28 +00:00
Ahmed Hussein
d2ae8bcc66 HADOOP-17360. Log the remote address for authentication success (#2441) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit 1ea3f74246294c280a2ccb0ff3e90b5721c5f0e2)
 2020-11-16 22:00:19 +00:00
Jim Brennan
4bdba35b9f HADOOP-17362. reduce RPC calls doing ls on HAR file (#2444). Contributed by Daryn Sharp and Ahmed Hussein 2020-11-13 21:49:35 +00:00
Ahmed Hussein
ac3e10af63 HADOOP-17358. Improve excessive reloading of Configurations (#2436) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit 71071e5c0fcaf73a3989000dbc60fa214a317da1)
(cherry picked from commit 23fe3bdab3410b8ed370fa6af9f2ed63817006ab)
 2020-11-12 10:35:47 -08:00
Stephen Jung
75c9aefe16 HADOOP-17096. Fix ZStandardCompressor input buffer offset (#2104). Contributed by Stephen Jung (Stripe). 
(cherry picked from commit 45434c93e865386de8c5f4036e795a3362557d56)
 2020-11-10 11:42:17 -08:00
Steve Loughran
b807fa2b3b HADOOP-17340. TestLdapGroupsMapping failing -string mismatch in exception validation. (#2427). Contributed by Steve Loughran. 2020-11-07 17:05:40 +05:30
hchaverr
92db25d64b HDFS-15623. Respect configured values of rpc.engine (#2403) Contributed by Hector Chaverri. 
(cherry picked from commit 6eacaffeea21e7e9286497ee17f89fd939d2eead)
 2020-11-06 14:43:54 -08:00
Eric Badger
0a90c2ee89 HADOOP-17342. Creating a token identifier should not do kerberos name 
resolution. Contributed by Jim Brennan.

(cherry picked from commit af389d989770c4218c24e640f572bc7492f7d4b3)
 2020-11-05 22:04:16 +00:00
Jim Brennan
431046d766 Revert "HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09 (#2387)" 
This reverts commit 8c6478b4058025e01e81a6c372ec3e9dfa36f196.
 2020-11-05 17:22:22 +00:00
Wei-Chiu Chuang
d6fd82ca56 Revert "HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. (#2291)" 
This reverts commit 1c7043b1ff89ba9c995cbf546a99bbbcccbe56b1.
 2020-11-04 16:18:41 -08:00
Akira Ajisaka
1c7043b1ff HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. (#2291) 
Reviewed-by: Steve Loughran <stevel@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7f5caca04cb8e5dab8410ef64daa0fe799e389fd)
(cherry picked from commit dd1634ec3b676d6c3da18af7da46734584ddd9f6)
 2020-11-03 11:23:02 -08:00
Sunil G
14a4606a9a HADOOP-17329. mvn site commands fails due to MetricsSystemImpl changes. Contributed by Xiaoqiao He. 
(cherry picked from commit f17e067d527b0089124a8ca69b36fb20ed118c55)
 2020-10-29 07:21:31 +05:30
Ayush Saxena
d2ecaf217f HADOOP-17119. Jetty upgrade to 9.4.x causes MR app fail with IOException. Contributed by Bilwa S T. 
(cherry picked from commit 8a58a12626f0332a4a828f312e2007d8c23564ee)
 2020-10-23 17:53:29 -07:00
Vinayakumar B
8c6478b405
HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09 (#2387) 2020-10-23 11:39:43 +05:30
Takanobu Asanuma
0d6958962c HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. (#2394) 
(cherry picked from commit 30f06e0c742806c7b9edf00a92c98b1e9cfcb0e1)
 2020-10-20 19:13:20 +09:00
He Xiaoqiao
3274fd139d
Preparing for 3.2.3 development 2020-10-16 14:52:41 +08:00
Konstantin V Shvachko
23b6ab67f7 HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 
(cherry picked from commit b3786d6c3cc13b0b92b9f42da1731c4ce35c9ded)
 2020-10-12 17:42:58 -07:00
maobaolong
4b1c0e4e5b HADOOP-17287. Support new Instance by non default constructor by ReflectionUtils (#2341) 2020-09-30 14:24:59 -07:00
Eric E Payne
947b0a154a YARN-9809. Added node manager health status to resource manager registration call. Contributed by Eric Badger (ebadger). 2020-09-28 18:50:44 +00:00
sguggilam
970b9a283b
HADOOP-17159. Make UGI support forceful relogin from keytab ignoring the last login time (#2249) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-26 23:58:34 -07:00
Mingliang Liu
acec431377
Revert "HADOOP-17159 Ability for forceful relogin in UserGroupInformation class (#2197)" 
This reverts commit d06f0de3affbd5e8232a6fcdb9a3c396934b6a05.
 2020-08-26 11:23:26 -07:00
Surendra Singh Lilhore
3bacea2e5e HDFS-14096. [SPS] : Add Support for Storage Policy Satisfier in ViewFs. Contributed by Ayush Saxena. 
(cherry picked from commit 788e7473a404fa074b3af522416ee3d2fae865a0)
 2020-08-26 10:31:39 -07:00
Uma Maheswara Rao G
0512b27172 HDFS-8631. WebHDFS : Support setQuota. Contributed by Chao Sun. (Backported) 2020-08-26 09:33:51 -07:00
Steve Loughran
134539f054
HADOOP-17199. S3A Directory Marker HADOOP-13230 backport #2210) 
This backports the listing-side changes of HADOOP-13230.

With this patch in, this branch of Hadoop is compatible with S3A clients
which do not delete directory markers when files are created underneath.

It does not allow this version to disable marker deletion; if the
fs.s3a.marker.retention option is changed to request this, a message
is printed at INFO and the policy remains at "delete"

The s3guard bucket-info command has been extended to support
probing for marker retention, as has the hasPathCapability method on
S3AFileSystem.

Read the documentation!
 2020-08-25 22:47:43 +01:00
Uma Maheswara Rao G
978ce5a1ee HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false (#2205) 
* HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false

* Used LambdaTestUtils#intercept in test

(cherry picked from commit 99b120a06e27add0b9070c829cd828d41a150e8c)
 2020-08-25 12:05:31 -07:00
sguggilam
d06f0de3af
HADOOP-17159 Ability for forceful relogin in UserGroupInformation class (#2197) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-24 23:58:36 -07:00
Joey
956c097f25 HADOOP-16925. MetricsConfig incorrectly loads the configuration whose value is String list in the properties file (#1896) 
Contributed by Jiayi Liu
 2020-08-24 14:06:14 +01:00
Mukund Thakur
28db29d32f HADOOP-17130. Configuration.getValByRegex() shouldn't be updating the results while fetching. (#2142) 
Contributed by Mukund Thakur

Change-Id: I35fe671d4026e8d3c04fc52012c3edcd4495e14a
 2020-08-24 14:05:03 +01:00
S O'Donnell
4163b43a83 HADOOP-17209. Erasure Coding: Native library memory leak. Contributed by Sean Chow 
(cherry picked from commit 17cd8a1b1627f4d87ddc5dcc2ec7c738ffdc576b)
 2020-08-24 13:06:15 +01:00
Steve Loughran
42c71a5790
HADOOP-15691. Add PathCapabilities to FileSystem and FileContext. 
Contributed by Steve Loughran.

This complements the StreamCapabilities Interface by allowing applications to probe for a specific path on a specific instance of a FileSystem client
to offer a specific capability.

This is intended to allow applications to determine

* Whether a method is implemented before calling it and dealing with UnsupportedOperationException.
* Whether a specific feature is believed to be available in the remote store.

As well as a common set of capabilities defined in CommonPathCapabilities,
file systems are free to add their own capabilities, prefixed with
 fs. + schema + .

The plan is to identify and document more capabilities -and for file systems which add new features, for a declaration of the availability of the feature to always be available.

Note

* The remote store is not expected to be checked for the feature;
  It is more a check of client API and the client's configuration/knowledge
  of the state of the remote system.
* Permissions are not checked.
 2020-08-19 17:15:06 +01:00
Akira Ajisaka
9aa78fe0d2
HADOOP-17204. Fix typo in Hadoop KMS document. Contributed by Xieming Li. 
(cherry picked from commit 141c62584bd253f0831d39557f54627f26cef7f7)
 2020-08-12 16:09:35 +09:00
Gautham B A
ff907b310d
HADOOP-17196. Fix C/C++ standard warnings (#2208) 
* Passing C/C++ standard flags -std is
  not cross-compiler friendly as not all
  compilers support all values.
* Thus, we need to make use of the
  appropriate flags provided by CMake in
  order to specify the C/C++ standards.

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 909f1e82d3e956bdd01b9d24ed2108f93bb94953)
 2020-08-11 16:47:26 +09:00
Sahil Takiar
eae1387f34
HDFS-14394: Add -std=c99 / -std=gnu99 to libhdfs compile flags 
Signed-off-by: Todd Lipcon <todd@apache.org>
(cherry picked from commit 3b0c5016b2931e3b87153c1cf368314d229e79ff)
 2020-08-11 16:45:27 +09:00
sguggilam
e484f5529c
HADOOP-17164. UGI loginUserFromKeytab doesn't set the last login time (#2178) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-04 10:47:31 -07:00
Uma Maheswara Rao G
cd5efe91d9 HDFS-15478: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. (#2160). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ac9a07b51aefd0fd3b4602adc844ab0f172835e3)
 2020-07-31 01:39:41 -07:00