Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,174 Commits 326 Branches 371 Tags
Commit Graph

21174 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Surendra Singh Lilhore
57f887a529 HDFS-15135. EC : ArrayIndexOutOfBoundsException in BlockRecoveryWorker#RecoveryTaskStriped. Contributed by Ravuri Sushma sree. 2020-03-10 23:56:42 +05:30
Szilard Nemeth
9e0d742025 YARN-9419. Log a warning if GPU isolation is enabled but LinuxContainerExecutor is disabled. Contribued by Andras Gyori 2020-03-10 16:39:03 +01:00
Eric E Payne
153eac1d21 YARN-942. TestContainerSchedulerQueuing.testKillOnlyRequiredOpportunisticContainers fails sporadically Contributed by Ahmed Hussein (ahussein) 
(cherry picked from commit ede05b19d1723147430fc426161326d46698507f)
 2020-03-10 14:28:13 +00:00
Brahma Reddy Battula
74fa55afc3 HADOOP-16871. Upgrade Netty version to 4.1.45.Final to handle CVE-2019-20444,CVE-2019-16869 
(cherry picked from commit c6b8a3038646697b77f6db54a2ef6266a9fc7888)
 2020-03-09 17:59:55 -07:00
Wei-Chiu Chuang
307abb7ce8 HADOOP-16643. Update netty4 to the latest 4.1.42. Contributed by Lisheng Sun. 
(cherry picked from commit 85af77c75768416db24ca506fd1704ce664ca92f)

 Conflicts:
	hadoop-project/pom.xml
 2020-03-09 17:59:14 -07:00
Weiwei Yang
5fca921fe3 HADOOP-16840. AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket. Contributed by wujinhu. 
(cherry picked from commit 6dfe00c71eb3721e9be3fc42349a81c4b013ada1)
 2020-03-08 21:14:41 -07:00
John Zhuge
2c3005824d
HADOOP-14206. TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature. Contributed by Jim Brennan. 
(cherry picked from commit 999096d82e6cc24023a77a69509af6820913f942)
 2020-03-08 00:00:50 -08:00
Inigo Goiri
1d89ba72f6 HDFS-15204. TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently. Contributed by Ahmed Hussein. 
(cherry picked from commit 1d4d0fcbe1e352b95c93dc016d3674db1126fb25)
 2020-03-04 10:04:22 -08:00
Akshay Nehe
369f4f9c58
HADOOP-16899. Update HdfsDesign.md to reduce ambiguity. (#1871) 
(cherry picked from commit bbd704bb828577a1f0afe5fb0ac358fb7c5af446)
 2020-03-04 11:12:00 +09:00
Siyao Meng
8a55178f49 HADOOP-16891. Upgrade jackson-databind to 2.9.10.3 (#1865) 
(cherry picked from commit e36b27260845c2eeb2211d01235cc6d3578b1942)
 2020-02-28 17:03:29 -08:00
Surendra Singh Lilhore
d33aec93f1 HDFS-15199. NPE in BlockSender. Contributed by Ayush Saxena. 2020-02-28 18:43:57 +05:30
Mukund Thakur
3937abddbd HDFS-13660. DistCp job fails when new data is appended in the file while the DistCp copy job is running 
This uses the length of the file known at the start of the copy to determine the amount of data to copy.

* If a file is appended to during the copy, the original bytes are copied.
* If a file is truncated during a copy, or the attempt to read the data fails with a truncated stream,
  distcp will now fail. Until now these failures were not detected.

Contributed by Mukund Thakur.

Change-Id: I576a49d951fa48d37a45a7e4c82c47488aa8e884
(cherry picked from commit 51c64b357d4bd1a0038e61df3d4b8ea0a3ad7449)
 2020-02-27 16:37:03 -08:00
Ayush Saxena
9acbbd885c HDFS-12999. When reach the end of the block group, it may not need to flush all the data packets(flushAllInternals) twice. Contributed by lufei and Fei Hui. 
(cherry picked from commit df622cf4a32ee172ded6c4b3b97a1e49befc4f10)
 2020-02-27 16:36:53 -08:00
Masatake Iwasaki
e1ff3b2f97 HDFS-15068. DataNode could meet deadlock if invoke refreshVolumes when register. Contributed by Aiphago. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 037ec8cfb1406ea3a8225a1b6306c2e78440353b)
 2020-02-27 16:36:46 -08:00
Konstantin V Shvachko
c82e20f9f8 HDFS-14731. [FGL] Remove redundant locking on NameNode. Contributed by Konstantin V Shvachko. 
(cherry picked from commit ecbcb058b8bc0fbc3903acb56814c6d9608bc396)
 2020-02-27 15:29:31 -08:00
Inigo Goiri
733f9b76b6 YARN-10161. TestRouterWebServicesREST is corrupting STDOUT. Contributed by Jim Brennan. 
(cherry picked from commit a43510e21d01e6c78e98e7ad9469cbea70a66466)
 2020-02-27 13:19:43 -08:00
Istvan Fajth
e42ac486e7 HDFS-14668 Support Fuse with Users from multiple Security Realms (#1739) 
(cherry picked from commit 57aa048516f5c5fe02441d213b52ce1bbeddf823)
 2020-02-27 07:49:31 -08:00
Kihwal Lee
27cfda708e HDFS-15147. LazyPersistTestCase wait logic is error-prone. Contributed 
by Ahmed Hussein.
 2020-02-27 09:17:44 -06:00
Kihwal Lee
de7edf58bd Revert "HADOOP-16888. [JDK11] Support JDK11 in the precommit job. Contributed by" 
Incorrect commit message

This reverts commit 749d7c0027b8978f9c07af04031a8fad6d7c18e1.
 2020-02-27 09:16:55 -06:00
Kihwal Lee
749d7c0027 HADOOP-16888. [JDK11] Support JDK11 in the precommit job. Contributed by 
Ahmed Hussein.
 2020-02-27 09:13:20 -06:00
Ayush Saxena
b182dd809c HDFS-15111. stopStandbyServices() should log which service state it is transitioning from. Contributed by Xieming Li. 2020-02-27 04:02:55 +05:30
Elixir Kook
7d0ff2dc85
YARN-10156. Fix typo 'complaint' which means quite different in Federation.md (#1856) 
(cherry picked from commit d608e94f9208354897a59dc8e533535b1b8a30b9)
 2020-02-26 17:32:09 +09:00
Stephen O'Donnell
2377649cdb HDFS-14861. Reset LowRedundancyBlocks Iterator periodically. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 900430b9907b590ed2d73a0d68f079c7f4d754b1)
 2020-02-25 13:29:03 -08:00
Takanobu Asanuma
fb0aba8521 HADOOP-16841. The description of hadoop.http.authentication.signature.secret.file contains outdated information. Contributed by Xieming Li. 
(cherry picked from commit 5cbc4c54611f062690c3bfcf78d09d37c78ffd12)
 2020-02-25 11:09:39 +09:00
Wei-Chiu Chuang
f9437b1817 HDFS-15174. Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operations. Contributed by Lisheng Sun. 
(cherry picked from commit cb4daeaad95470125c59ccb80b988261ae83d534)
 2020-02-24 13:37:48 -08:00
Szilard Nemeth
92ad3bd099 YARN-10143. YARN-10101 broke Yarn logs CLI. Contributed by Adam Antal 2020-02-24 21:27:31 +01:00
Ayush Saxena
af46658baf HDFS-15166. Remove redundant field fStream in ByteStringLog. Contributed by Xieming Li. 2020-02-24 20:17:31 +05:30
Konstantin V Shvachko
769397201a HDFS-15185. StartupProgress reports edits segments until the entire startup completes. Contributed by Konstantin V Shvachko. 
(cherry picked from commit 6f84269bcd5cdb08ca68b2d8276f66d34a2a7a0d)
 2020-02-21 10:54:20 -08:00
Masatake Iwasaki
610805ec72 HDFS-15052. WebHDFS getTrashRoot leads to OOM due to FileSystem object creation. (#1758) 
(cherry picked from commit 2338d25dc7150d75fbda84cc95422380b5622224)

 Conflicts:
        hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
 2020-02-21 13:40:11 +09:00
Akira Ajisaka
b52f201285
HADOOP-16869. Upgrade findbugs-maven-plugin to 3.0.5 to fix mvn findbugs:findbugs failure (#1855) 
(cherry picked from commit 7f35676f90f0730b8c9844cf00ee5a943f80d48d)

 Conflicts:
	hadoop-project/pom.xml
 2020-02-21 12:24:03 +09:00
Wei-Chiu Chuang
d411a849e1 HADOOP-16868. ipc.Server readAndProcess threw NullPointerException. Contributed by Tsz-wo Sze. 
(cherry picked from commit cb3f3cca01d6ab9f4befb1bcd31f384c37c0231a)
 2020-02-18 21:53:50 -08:00
Sunil G
de63115a2a YARN-10139. ValidateAndGetSchedulerConfiguration API fails when cluster max allocation > default 8GB. Contributed by Prabhu Joseph. 
(cherry picked from commit 6526f95bd281fc011f8776d21ff933087c5924de)
 2020-02-19 11:18:19 +05:30
xiaoxiaopan118
b5321f1600
HDFS-15173. RBF: Delete repeated configuration 'dfs.federation.router.metrics.enable' (#1849) 
(cherry picked from commit 439d935e1df601ed998521443fbe6752040e7a84)
 2020-02-17 15:57:15 +09:00
Ayush Saxena
9a351e837f HDFS-15164. Fix TestDelegationTokensWithHA. Contributed by Ayush Saxena. 2020-02-14 22:15:12 +05:30
Surendra Singh Lilhore
e13ea1b094 HDFS-15086. Block scheduled counter never get decremet if the block got deleted before replication. Contributed by hemanthboyina. 2020-02-13 22:52:50 +05:30
Ayush Saxena
9a1ce410f6 HDFS-15161. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException in ShortCircuitCache#close(). Contributed by Lisheng Sun 2020-02-12 20:29:09 +05:30
Akira Ajisaka
b80f020886
HADOOP-16849. start-build-env.sh behaves incorrectly when username is numeric only. Contributed by Jihyun Cho. 
(cherry picked from commit 9709afe67d8ed45c3dfb53e45fe1efdc0814ac6c)

 Conflicts:
	start-build-env.sh
 2020-02-12 14:07:56 +09:00
Kihwal Lee
f1840669cf HDFS-14758. Make lease hard limit configurable and reduce the default. 
Contributed by hemanthboyina.

(cherry picked from commit 9b8a78d97bfd825ce840c6033371c7f10e49a5b8)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend4.java
 2020-02-11 14:50:10 -06:00
testfixer
edad6f1d0f HADOOP-16847. Test can fail if HashSet iterates in a different order. 
Contributed by Testfixer
 2020-02-11 11:24:36 +00:00
Szilard Nemeth
6aec712c6c YARN-10101. Support listing of aggregated logs for containers belonging to an application attempt. Contributed by Adam Antal 2020-02-11 09:18:44 +01:00
Sunil G
95b1cbcbd4 YARN-10109. Allow stop and convert from leaf to parent queue in a single Mutation API call. Contributed by Prabhu Joseph 
(cherry picked from commit 28f730b317b23038bf9bd0775dd2cdb96518b13b)
 2020-02-09 21:15:31 +05:30
Ayush Saxena
7be104a9be HDFS-15115. Namenode crash caused by NPE in BlockPlacementPolicyDefault when dynamically change logger to debug. Contributed by wangzhixiang 2020-02-08 10:41:15 +05:30
Jonathan Hung
aca930402c YARN-10116. Expose diagnostics in RMAppManager summary 
(cherry picked from commit 314e2f9d2e888fae1e5bf669aeeead84a928d282)
 2020-02-05 11:17:03 -08:00
Kihwal Lee
0b19c8e086 HADOOP-15787. TestIPC.testRTEDuringConnectionSetup fails with jdk8u242. Contributed by Zsolt Venczel. 
(cherry picked from commit a463cf75a0ab1f0dbb8cfa16c39a4e698bc1a625)
 2020-02-05 13:02:21 -06:00
Chen Liang
226cd3977b HDFS-15148. dfs.namenode.send.qop.enabled should not apply to primary NN port. Contributed by Chen Liang. 2020-02-04 12:34:19 -08:00
Kihwal Lee
5257afb153 HDFS-12491. Support wildcard in CLASSPATH for libhdfs. Contributed by Muhammad Samir Khan. 
(cherry picked from commit 10a60fbe20bb08cdd71076ea9bf2ebb3a2f6226e)
 2020-02-04 12:24:12 -06:00
Prabhu Joseph
7136ebbb7a YARN-10022. Add RM Rest API to validate a CapacityScheduler Config with delta change 
Contributed by Kinga Marton.

(cherry-picked from commit 1ab9c692fa9107f10c0d8fbfbb9e74f54bddc960)
 2020-02-04 14:06:23 +05:30
Stephen O'Donnell
e89436b183 HDFS-7175. Client-side SocketTimeoutException during Fsck. Contributed by Stephen O'Donnell, Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 1e3a0b0d931676b191cb4813ed1a283ebb24d4eb)
 2020-01-31 16:13:42 -08:00
Inigo Goiri
2c8c6e8e22 HDFS-13179. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. Contributed by Ahmed Hussein. 
(cherry picked from commit 1839c467f60cbb8592d446694ec3d7710cda5142)
 2020-01-31 11:24:37 -08:00
Eric E Payne
f62c472944 MAPREDUCE-7079: JobHistory#ServiceStop implementation is incorrect. Contributed by Ahmed Hussein (ahussein) 
(cherry picked from commit b897f6834ba69d443c3acd1fab52261c00d675a1)
 2020-01-29 17:10:13 +00:00