Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,419 Commits 326 Branches 371 Tags
Commit Graph

4933 Commits

Author SHA1 Message Date
Xudong Cao
b4f39c2a8a HADOOP-16677. Recalculate the remaining timeout millis correctly while throwing an InterupptedException in SocketIOWithTimeout. (#1687) 
(cherry picked from commit df6b3162c11987ba5299c69cb251332228dacf36)
(cherry picked from commit 975f669141add885d631685d693e81c44c190751)
 2019-11-12 14:21:59 -08:00
Chen Liang
c6cbf67b82 HDFS-14941. Potential editlog race condition can cause corrupted file. Contributed by Chen Liang and Konstantin Shvachko. 2019-11-06 15:11:09 -08:00
Chen Liang
7b4f44573f HADOOP-16013. DecayRpcScheduler decay thread should run as a daemon. Contributed by Erik Krogen. 
(cherry picked from commit 01cb958af44b2376bcf579cc65d90566530f733d)
(cherry picked from commit a58f466d074becff11b0757b1b3cf9c8f6bb57a8)
 2019-11-04 17:18:37 -08:00
Jonathan Hung
f29481f046 Add 2.10.0 release notes for HDFS-12943 
(cherry picked from commit ef9d12df24c0db76fd37a95551db7920d27d740c)
(cherry picked from commit 9ccfc2d6d3f3819da58ea9f0ff056ed03970a6e1)
 2019-10-31 20:35:20 -07:00
Duo Zhang
07f59c023c HADOOP-16598. Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches 
Signed-off-by: stack <stack@apache.org>
 2019-10-30 21:30:34 -07:00
Jonathan Hung
4d274f60bc Make upstream aware of 2.10.0 release 
(cherry picked from commit 7663db59c097c82eeed2df7a91168a4d7123c96b)
(cherry picked from commit 5d2ffcc7aac608746946dc2ed5711549a01a2055)
 2019-10-30 21:00:55 -07:00
Steve Loughran
ad881214bc HADOOP-16669. TestRawLocalFileSystemContract.testPermission fails if no native library. 
Contributed by Steve Loughran.

Change-Id: I622e36f17a63c775c5e33e93b8b26e461d0bc451
 2019-10-29 17:11:25 +00:00
Rakesh Radhakrishnan
2db0b8f6ca HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. Contributed by Feilong He. 
(cherry picked from commit 659c88801d008bb352d10a1cb3bd0e401486cc9b)
 2019-10-27 22:24:44 +05:30
Sammi Chen
9281b72550 HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He. 
(cherry picked from commit d1aad444907e1fc5314e8e64529e57c51ed7561c)
 2019-10-27 22:23:44 +05:30
Fei Hui
287fab6346 HDFS-14923. Remove dead code from HealthMonitor. Contributed by Fei Hui. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7be5508d9b35892f483ba6022b6aced7648b8fa3)
(cherry picked from commit 54d3e212bfdcc6b473969a70faca339184849e61)
 2019-10-25 15:06:56 -07:00
Ayush Saxena
9434f36b43 HADOOP-16662. Remove unnecessary InnerNode check in NetworkTopology#add(). Contributed by Lisheng Sun. 2019-10-20 11:55:58 +05:30
Szilard Nemeth
f8c06f5a87 HADOOP-16580. Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException. Contributed by Adam Antal 
(cherry picked from commit c79a5f2d9930f58ad95864c59cd0a6164cd53280)
(cherry picked from commit 0bbd48c7a8c0d1dd57ffe5d372d749e5e7e41d34)
 2019-10-16 14:24:17 +02:00
Jonathan Hung
0dc85bc182 YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena 2019-10-08 11:18:52 -07:00
Szilard Nemeth
2c8cfb4f64 YARN-6715. Fix documentation about NodeHealthScriptRunner. Contributed by Peter Bacsko 2019-10-08 17:34:44 +02:00
Erik Krogen
56caacac1f HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen. 
(cherry picked from 64f28f9efa2ef3cd9dd54a6c5009029721e030ed)
(cherry picked from 69b0c513a9b11cc7f795747732173b36aacbe794)
 2019-10-04 10:48:17 -07:00
Tsz Wo Nicholas Sze
ce804be72e HADOOP-16127. In ipc.Client, put a new connection could happen after stop. 
(cherry picked from commit 9192f71e21847ad86bc9ff23847d8957dfe8ae58)
 2019-10-03 18:28:13 -07:00
Wei-Chiu Chuang
aa5dfb2f8a HADOOP-12282. Connection thread's name should be updated after address changing is detected. Contributed by Lisheng Sun. 
(cherry picked from commit b94eba9f11af66b10638dd255c224e946d842b8c)
 2019-10-03 17:30:00 -07:00
Giovanni Matteo Fumarola
76c30e4f3c HADOOP-16625: Backport HADOOP-14624 to branch-3.1. 
(cherry picked from commit 79c97f6a0bebc95ff81a8ef9b07d3619f05ed583)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
 2019-10-03 15:45:23 -07:00
Tsz Wo Nicholas Sze
8d71b508ab HADOOP-16126. ipc.Client.stop() may sleep too long to wait for all connections. 
(cherry picked from commit 0edb0c51dc2c4ae2f353e260f01912e28033d70f)
 2019-10-02 16:32:40 -07:00
Wei-Chiu Chuang
41c0757a1f HDFS-14523. Remove excess read lock for NetworkToplogy. Contributed by Wu Weiwei. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Chen Liang <cliang@apache.org>
(cherry picked from commit 971a4c8e8328a4bdea65de4a0e84c82b5b2de24b)
 2019-10-02 16:27:00 -07:00
Inigo Goiri
f1fe3abac9 HADOOP-16161. NetworkTopology#getWeightUsingNetworkLocation return unexpected result. Contributed by He Xiaoqiao. 
(cherry picked from commit 389e640f0cc7d8528e9b4411457f04a528601c69)
 2019-10-01 17:33:08 -07:00
Yiqun Lin
3aea2bcf12 HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. 
(cherry picked from commit a5bb1e8ee871df1111ff77d0f6921b13c8ffb50e)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/resources/core-default.xml

(cherry picked from commit c74027d9d34711c2c4baed7c98bc475d95097be0)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
 2019-10-01 17:16:55 -07:00
Stephen O'Donnell
9dfb26136b HADOOP-16140. hadoop fs expunge to add -immediate option to purge trash immediately. 
Contributed by Stephen O'Donnell.

Signed-off-by: Steve Loughran <stevel@apache.org>
(cherry picked from commit 686c0141eff0886c285b8e52fddade43c1ce4570)
 2019-10-01 13:13:59 -07:00
Surendra Singh Lilhore
5436df0c94 HDFS-14175. EC: Native XOR decoder should reset the output buffer before using it. Contributed by Ayush Saxena. 
(cherry picked from commit dacc1a759e3ba3eca000cbacc6145b231253b174)
 2019-10-01 09:47:11 -07:00
Wei-Chiu Chuang
57ed0ff225 HADOOP-15418. Hadoop KMSAuthenticationFilter needs to use getPropsByPrefix instead of iterator to avoid ConcurrentModificationException. Contributed by lqjack and Suma Shivaprasad 
(cherry picked from commit cd2158456db8c89eeea64b72654a736ea8607e23)
(cherry picked from commit ab3fb28d47e8fa260d2de7490aea59404d571c2e)
 2019-10-01 09:03:01 -07:00
Kihwal Lee
757c11e628 HADOOP-15909. KeyProvider class should implement Closeable. Contributed 
by Kuhu Shukla.

(cherry picked from commit 33c009a45d95c9b582db25acd205d087e01e2085)
 2019-10-01 08:53:47 -07:00
Erik Krogen
5e56914650 HADOOP-16531. Log more timing information for slow RPCs. Contributed by Chen Zhang. 
(cherry picked from commit a23417533e1ee052893baf207ec636c4993c5994)
 2019-09-30 17:35:38 -07:00
Wei-Chiu Chuang
72ef752876 HADOOP-15865. ConcurrentModificationException in Configuration.overlay() method. Contributed by Oleksandr Shevchenko. 
(cherry picked from commit e872ceb810a343da7fce7185dca78d3b9aad9b7b)
(cherry picked from commit 46e72775f5b2156f49b15beac8bffff9cd5aad7f)
 2019-09-30 17:32:23 -07:00
Zsombor Gegesy
db5b8358ca HADOOP-15014. Addendum: KMS should log the IP address of the clients. Contributed by Zsombor Gegesy. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit b0131bc265453051820e54908e70d39433c227ab)
 2019-09-30 17:15:51 -07:00
Zsombor Gegesy
b59911d841 HADOOP-15014. KMS should log the IP address of the clients. Contributed by Zsombor Gegesy. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 008766c119d9ed9d568f9458ed0c02136962da5b)
 2019-09-30 17:15:43 -07:00
Gopal V
b9dc2c17de HADOOP-16461. Regression: FileSystem cache lock parses XML within the lock. 
Contributed by Gopal V.

Change-Id: If6654f850e9c24ee0d9519a46fd6269b18e1a7a4
(cherry picked from commit aebac6d2d2e612e400a7d73be67dafb47e239211)
 2019-09-30 16:58:52 -07:00
Zhankun Tang
670d0657b4 Make upstream aware of 3.1.3 release. 2019-09-27 16:58:36 +08:00
Yuval Degani
b091abc471 HADOOP-16581. Revise ValueQueue to correctly replenish queues that go below the watermark (#1463) 
(cherry picked from dd0834696a694564af65a1355c9d13275f44df51)
(cherry picked from e31594f20ed43670a9ff1463a752d8b8e4b3b6f0)
 2019-09-20 09:58:57 -07:00
Takanobu Asanuma
6ef3204d54 HADOOP-16069. Support configure ZK_DTSM_ZK_KERBEROS_PRINCIPAL in ZKDelegationTokenSecretManager using principal with Schema /_HOST. Contributed by luhuachao. 
(cherry picked from commit 298f07abceb4bb854630c9058e1697d4ef3d2ae2)
 2019-09-20 13:58:19 +09:00
Kihwal Lee
56562b9117 HADOOP-16582. LocalFileSystem's mkdirs() does not work as expected under viewfs. Contributed by Kihwal Lee 
(cherry picked from commit d4205dce176287e863f567b333e0d408bf51ae6d)
 2019-09-19 08:26:36 -05:00
Wei-Chiu Chuang
6af1ecc8e4 HDFS-14569. Result of crypto -listZones is not formatted properly. Contributed by hemanthboyina. 
(cherry picked from 0f2dad6679b7fc35474a3d33dc40b0db89bb1d80)
(cherry picked from 9c9920897de182c3da3ad33ff53e506242bbd362)
 2019-09-18 10:30:52 -07:00
Weiwei Yang
1978317068 YARN-2255. YARN Audit logging not added to log4j.properties. Contributed by Aihua Xu. 
(cherry picked from commit f8c14326ee0026f2c57a4d061bff5fadec74b74d)
 2019-09-18 09:20:37 +08:00
Zhankun Tang
ef79d98788 Preparing for 3.1.4 development 2019-09-04 16:11:36 +08:00
Stephen O'Donnell
63b18d764a HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 915cbc91c0a12cc7b4d3ef4ea951941defbbcb33)
(cherry picked from commit 2c7f49743fba587e32398c81412db9eac426cf98)
 2019-09-02 09:58:48 -07:00
Chen Liang
0839cf1d4e HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang. 2019-08-23 10:11:22 -07:00
Chen Liang
ef9b0a1d9a HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang. 2019-08-23 10:11:22 -07:00
Chen Liang
3135cc35e3 HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang 2019-08-23 10:11:22 -07:00
Chen Liang
3835a6c4e8 HDFS-13566. Add configurable additional RPC listener to NameNode. Contributed by Chen Liang. 2019-08-23 10:11:22 -07:00
Chen Liang
14265b8f03 HDFS-13547. Add ingress port based sasl resolver. Contributed by Chen Liang. 2019-08-23 10:11:22 -07:00
David Mollitor
0b7f9a09c2 HADOOP-15246. SpanReceiverInfo - Prefer ArrayList over LinkedList. Contributed by David Mollitor. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 901652e26321832ac42ce54daa18ebc1813772b5)
(cherry picked from commit 1b30b3fbe7b5e130778c79465ac7986772a83ab8)
 2019-08-18 18:44:02 -07:00
Christopher Gregorian
96ea7f0562 HDFS-14403. Cost-based extension to the RPC Fair Call Queue. Contributed by Christopher Gregorian. 
(cherry picked from 129576f628d370def74e56112aba3a93e97bbf70)
(cherry picked from e4b650f91e7353f98fee2a725490ca45aa81e1a4)
 2019-08-14 13:37:01 -07:00
Masatake Iwasaki
e66ad19395 HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit da0006fe0473e353ee2d489156248a01aa982dfd)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java

(cherry picked from commit d7ca016d63d89e5c8377a035f93485a7c77c3430)
 2019-08-14 09:07:00 +09:00
Wei-Chiu Chuang
8501bed9cb HADOOP-15681. AuthenticationFilter should generate valid date format for Set-Cookie header regardless of default Locale. Contributed by Cao Manh Dat. 
(cherry picked from commit e20b19543ba148cf0242884a3e553ad2f39421b3)
(cherry picked from commit 490b18fd4f35a19dc2dd48249cb38a96dbf2f85c)
 2019-08-01 17:36:59 -07:00
He Xiaoqiao
5d35393912 HDFS-13529. Fix default trash policy emptier trigger time correctly. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit f86de6f76a3079c2655df9b242fc968edfb17b9d)
(cherry picked from commit 555b56c0421be84efd09221db2b3f26c8b44ed45)
 2019-08-01 14:56:17 -07:00
Christopher Gregorian
7b8f08f59e HADOOP-16459. Backport of HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 
This commit also includes the follow-on commit 827a84778a4e3b8f165806dfd2966f0951a5e575.

(cherry-picked from f96a2df38d889f29314c57f4d94227b2e419a11f)
(cherry-picked from d4492bdd9edec60c236aff85de50b963097e5a7f)
 2019-07-30 14:45:00 -07:00