Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,622 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

19622 Commits

Author SHA1 Message Date
Tsz Wo Nicholas Sze 8d71b508ab HADOOP-16126. ipc.Client.stop() may sleep too long to wait for all connections. 
(cherry picked from commit 0edb0c51dc)
 2019-10-02 16:32:40 -07:00
Wei-Chiu Chuang 41c0757a1f HDFS-14523. Remove excess read lock for NetworkToplogy. Contributed by Wu Weiwei. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Chen Liang <cliang@apache.org>
(cherry picked from commit 971a4c8e83)
 2019-10-02 16:27:00 -07:00
paulward24 ab7ecd6a0f HDFS-14618. Incorrect synchronization of ArrayList field (ArrayList is thread-unsafe). Contributed by Paul Ward. 
(cherry picked from commit d203045c30)
 2019-10-02 15:07:34 -07:00
paulward24 d95dd33ea1 HDFS-14610. HashMap is not thread safe. Field storageMap is typically synchronized by storageMap. However, in one place, field storageMap is not protected with synchronized. (#1015) 
(cherry picked from commit d8bac50e12)
(cherry picked from commit 76a9135947)
 2019-10-02 14:31:23 -07:00
Ayush Saxena 122b02e288 HDFS-14808. EC: Improper size values for corrupt ec block in LOG. Contributed by Ayush Saxena. 
(cherry picked from commit 66400c1cbb)
(cherry picked from commit 2bb4ac18ec)
 2019-10-02 13:49:20 -07:00
Surendra Singh Lilhore f0bdab02e7 HDFS-14699. Erasure Coding: Storage not considered in live replica when replication streams hard limit reached to threshold. Contributed by Zhao Yi Ming. 
(cherry picked from commit d1c303a497)
(cherry picked from commit eb1ddcd04c)
 2019-10-02 13:41:23 -07:00
Inigo Goiri f1fe3abac9 HADOOP-16161. NetworkTopology#getWeightUsingNetworkLocation return unexpected result. Contributed by He Xiaoqiao. 
(cherry picked from commit 389e640f0c)
 2019-10-01 17:33:08 -07:00
Yiqun Lin 3aea2bcf12 HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. 
(cherry picked from commit a5bb1e8ee8)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/resources/core-default.xml

(cherry picked from commit c74027d9d3)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
 2019-10-01 17:16:55 -07:00
Inigo Goiri 2a3eb04996 HDFS-14192. Track missing DFS operations in Statistics and StorageStatistics. Contributed by Ayush Saxena. 
(cherry picked from commit f048512bb8)
(cherry picked from commit e59ced9c60a7007551ee8a9f83ce8e266e4dbae1)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
 2019-10-01 16:56:29 -07:00
Ranith Sardar a400f396a6 HADOOP-16032. Distcp It should clear sub directory ACL before applying new ACL on. 
Contributed by Ranith Sardar.

(cherry picked from commit 546c5d70ef)
(cherry picked from commit c5eca3f7ee)

 Conflicts:
	hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestDistCpUtils.java
 2019-10-01 16:34:00 -07:00
Stephen O'Donnell 9dfb26136b HADOOP-16140. hadoop fs expunge to add -immediate option to purge trash immediately. 
Contributed by Stephen O'Donnell.

Signed-off-by: Steve Loughran <stevel@apache.org>
(cherry picked from commit 686c0141ef)
 2019-10-01 13:13:59 -07:00
Inigo Goiri 528dc8199b HDFS-14460. DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy configured. Contributed by CR Hota. 
(cherry picked from commit 865c328930)
 2019-10-01 09:53:31 -07:00
Surendra Singh Lilhore 5436df0c94 HDFS-14175. EC: Native XOR decoder should reset the output buffer before using it. Contributed by Ayush Saxena. 
(cherry picked from commit dacc1a759e)
 2019-10-01 09:47:11 -07:00
Inigo Goiri 23eed7b201 HDFS-14202. dfs.disk.balancer.max.disk.throughputInMBperSec property is not working as per set value. Contributed by Ranith Sardar. 
(cherry picked from commit 0e79a86582)
(cherry picked from commit fc8a7a9e5b)
 2019-10-01 09:35:14 -07:00
Inigo Goiri 7a915a0ba3 HDFS-14418. Remove redundant super user priveledge checks from namenode. Contributed by Ayush Saxena. 
(cherry picked from commit be6c8014e6)
(cherry picked from commit 06b2eceb76)
 2019-10-01 09:24:21 -07:00
Wei-Chiu Chuang 57ed0ff225 HADOOP-15418. Hadoop KMSAuthenticationFilter needs to use getPropsByPrefix instead of iterator to avoid ConcurrentModificationException. Contributed by lqjack and Suma Shivaprasad 
(cherry picked from commit cd2158456d)
(cherry picked from commit ab3fb28d47)
 2019-10-01 09:03:01 -07:00
Kihwal Lee 757c11e628 HADOOP-15909. KeyProvider class should implement Closeable. Contributed 
by Kuhu Shukla.

(cherry picked from commit 33c009a45d)
 2019-10-01 08:53:47 -07:00
Sunil G 933d81f0ce YARN-9801. SchedConfCli does not work wiwith https mode. Contributed by Prabhu Joseph 
(cherry picked from commit 99cd7572f1)
 2019-10-01 20:08:12 +05:30
Sunil G 71792f2122 YARN-9864. Format CS Configuration present in Configuration Store. Contributeed by Prabhu Joseph 2019-10-01 20:03:35 +05:30
bibinchundatt 6529a30d9e YARN-9858. Optimize RMContext getExclusiveEnforcedPartitions. Contributed by Jonathan Hung. 2019-10-01 16:06:38 +05:30
Erik Krogen 5e56914650 HADOOP-16531. Log more timing information for slow RPCs. Contributed by Chen Zhang. 
(cherry picked from commit a23417533e)
 2019-09-30 17:35:38 -07:00
Wei-Chiu Chuang 72ef752876 HADOOP-15865. ConcurrentModificationException in Configuration.overlay() method. Contributed by Oleksandr Shevchenko. 
(cherry picked from commit e872ceb810)
(cherry picked from commit 46e72775f5)
 2019-09-30 17:32:23 -07:00
Konstantin V Shvachko 8fc4e406ad HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by Konstantin V Shvachko. 2019-09-30 17:24:15 -07:00
Konstantin V Shvachko bece1a1f70 Revert "HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao." 
This reverts commit 0feba4396f.
 2019-09-30 17:24:15 -07:00
Zsombor Gegesy db5b8358ca HADOOP-15014. Addendum: KMS should log the IP address of the clients. Contributed by Zsombor Gegesy. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit b0131bc265)
 2019-09-30 17:15:51 -07:00
Zsombor Gegesy b59911d841 HADOOP-15014. KMS should log the IP address of the clients. Contributed by Zsombor Gegesy. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 008766c119)
 2019-09-30 17:15:43 -07:00
Gopal V b9dc2c17de HADOOP-16461. Regression: FileSystem cache lock parses XML within the lock. 
Contributed by Gopal V.

Change-Id: If6654f850e9c24ee0d9519a46fd6269b18e1a7a4
(cherry picked from commit aebac6d2d2)
 2019-09-30 16:58:52 -07:00
Surendra Singh Lilhore 1030ac85bb HDFS-14235. Handle ArrayIndexOutOfBoundsException in DataNodeDiskMetrics#slowDiskDetectionDaemon. Contributed by Ranith Sardar. 
(cherry picked from commit 41e18feda3)
(cherry picked from commit b93b127956)
 2019-09-30 16:47:08 -07:00
Xiao Chen 5167a29adb HADOOP-15849. Upgrade netty version to 3.10.6. 
(cherry picked from commit 8853fc8a55)
 2019-10-01 07:05:28 +09:00
Ayush Saxena 8cc702ad38 HDFS-14853. NPE in DFSNetworkTopology#chooseRandomWithStorageType() when the excludedNode is not present. Contributed by Ranith Sardar. 2019-09-28 08:30:26 +05:30
Zhankun Tang 670d0657b4 Make upstream aware of 3.1.3 release. 2019-09-27 16:58:36 +08:00
Chen Liang 3d7730266a HDFS-14785. [SBN read] Change client logging to be less aggressive. Contributed by Chen Liang. 2019-09-26 15:59:08 -07:00
Jonathan Hung 95ec7050b5 Addendum to YARN-9730. Support forcing configured partitions to be exclusive based on app node label 
(cherry picked from commit d86a1acc866cbda845fb3896dc824baf12217383)
(cherry picked from commit f4f210d2e5)
 2019-09-25 17:49:52 -07:00
Jonathan Hung 783cbced1d YARN-9730. Support forcing configured partitions to be exclusive based on app node label 
(cherry picked from commit 73a044a63822303f792183244e25432528ecfb1e)
(cherry picked from commit dd094d79023f6598e47146166aa8c213e03d41b7)
 2019-09-24 13:52:14 -07:00
Ayush Saxena 5c058549b7 HDFS-14655. [SBN Read] Namenode crashes if one of The JN is down. Contributed by Ayush Saxena. 2019-09-25 01:22:11 +05:30
Jonathan Hung 6a1d2d56bd YARN-9762. Add submission context label to audit logs. Contributed by Manoj Kumar 
(cherry picked from commit 3d78b1223d)
(cherry picked from commit a1fa9a8a7f)
 2019-09-23 13:13:09 -07:00
Yuval Degani b091abc471 HADOOP-16581. Revise ValueQueue to correctly replenish queues that go below the watermark (#1463) 
(cherry picked from dd0834696a)
(cherry picked from e31594f20e)
 2019-09-20 09:58:57 -07:00
Takanobu Asanuma 6ef3204d54 HADOOP-16069. Support configure ZK_DTSM_ZK_KERBEROS_PRINCIPAL in ZKDelegationTokenSecretManager using principal with Schema /_HOST. Contributed by luhuachao. 
(cherry picked from commit 298f07abce)
 2019-09-20 13:58:19 +09:00
Kihwal Lee 56562b9117 HADOOP-16582. LocalFileSystem's mkdirs() does not work as expected under viewfs. Contributed by Kihwal Lee 
(cherry picked from commit d4205dce17)
 2019-09-19 08:26:36 -05:00
Chen Liang d25b3339ed HDFS-14822. [SBN read] Revisit GlobalStateIdContext locking when getting server state id. Contributed by Chen Liang. 2019-09-18 17:20:05 -07:00
Kihwal Lee 2661c32c06 HDFS-13959. TestUpgradeDomainBlockPlacementPolicy is flaky. Contributed by Ayush Saxena. 
(cherry picked from commit 1851d06eb3)
 2019-09-18 16:45:29 -05:00
Wei-Chiu Chuang 6af1ecc8e4 HDFS-14569. Result of crypto -listZones is not formatted properly. Contributed by hemanthboyina. 
(cherry picked from 0f2dad6679)
(cherry picked from 9c9920897d)
 2019-09-18 10:30:52 -07:00
Wei-Chiu Chuang 0fcbeed49a HDFS-14836. FileIoProvider should not increase FileIoErrors metric in datanode volume metric. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit 5dd859a8a0)
(cherry picked from commit b906f53b0d)
 2019-09-18 09:36:55 -07:00
Anu Engineer eb5c4ce432 HDFS-14759. HDFS cat logs an info message. 
Contributed by Eric Badger.

(cherry picked from commit 8aaf5e1a14)
 2019-09-14 01:35:12 +09:00
Sunil G 45cf3de2e9 YARN-9833. Race condition when DirectoryCollection.checkDirs() runs during container launch. Contributed by Peter Bacsko. 
(cherry picked from commit c474e24c0b)
 2019-09-18 09:23:46 +05:30
Weiwei Yang 1978317068 YARN-2255. YARN Audit logging not added to log4j.properties. Contributed by Aihua Xu. 
(cherry picked from commit f8c14326ee)
 2019-09-18 09:20:37 +08:00
Eric Yang dae22c962d YARN-9837. Fixed reading YARN Service JSON spec file larger than 128k. 
Contributed by Tarun Parimi
 2019-09-17 13:17:13 -04:00
Ayush Saxena cab81f7f5f HDFS-14303. Addendum: check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 2019-09-14 10:20:49 +05:30
Yi Sheng 7d7b415409 HADOOP-16555. Update commons-compress to 1.19. (#1425) Contributed by YiSheng Lien. 
(cherry picked from commit 1843c4688a)

Conflicts:
	LICENSE-binary
(cherry picked from commit d39ebbfebc)
 2019-09-13 11:15:56 -07:00
Takanobu Asanuma 8d5460a045 HDFS-14838. RBF: Display RPC (instead of HTTP) Port Number in RBF web UI. Contributed by Xieming Li 
(cherry picked from commit c255333e20)
 2019-09-11 17:11:13 +09:00