Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,156 Commits 326 Branches 371 Tags
Commit Graph

21156 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Kihwal Lee
de7edf58bd Revert "HADOOP-16888. [JDK11] Support JDK11 in the precommit job. Contributed by" 
Incorrect commit message

This reverts commit 749d7c0027b8978f9c07af04031a8fad6d7c18e1.
 2020-02-27 09:16:55 -06:00
Kihwal Lee
749d7c0027 HADOOP-16888. [JDK11] Support JDK11 in the precommit job. Contributed by 
Ahmed Hussein.
 2020-02-27 09:13:20 -06:00
Ayush Saxena
b182dd809c HDFS-15111. stopStandbyServices() should log which service state it is transitioning from. Contributed by Xieming Li. 2020-02-27 04:02:55 +05:30
Elixir Kook
7d0ff2dc85
YARN-10156. Fix typo 'complaint' which means quite different in Federation.md (#1856) 
(cherry picked from commit d608e94f9208354897a59dc8e533535b1b8a30b9)
 2020-02-26 17:32:09 +09:00
Stephen O'Donnell
2377649cdb HDFS-14861. Reset LowRedundancyBlocks Iterator periodically. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 900430b9907b590ed2d73a0d68f079c7f4d754b1)
 2020-02-25 13:29:03 -08:00
Takanobu Asanuma
fb0aba8521 HADOOP-16841. The description of hadoop.http.authentication.signature.secret.file contains outdated information. Contributed by Xieming Li. 
(cherry picked from commit 5cbc4c54611f062690c3bfcf78d09d37c78ffd12)
 2020-02-25 11:09:39 +09:00
Wei-Chiu Chuang
f9437b1817 HDFS-15174. Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operations. Contributed by Lisheng Sun. 
(cherry picked from commit cb4daeaad95470125c59ccb80b988261ae83d534)
 2020-02-24 13:37:48 -08:00
Szilard Nemeth
92ad3bd099 YARN-10143. YARN-10101 broke Yarn logs CLI. Contributed by Adam Antal 2020-02-24 21:27:31 +01:00
Ayush Saxena
af46658baf HDFS-15166. Remove redundant field fStream in ByteStringLog. Contributed by Xieming Li. 2020-02-24 20:17:31 +05:30
Konstantin V Shvachko
769397201a HDFS-15185. StartupProgress reports edits segments until the entire startup completes. Contributed by Konstantin V Shvachko. 
(cherry picked from commit 6f84269bcd5cdb08ca68b2d8276f66d34a2a7a0d)
 2020-02-21 10:54:20 -08:00
Masatake Iwasaki
610805ec72 HDFS-15052. WebHDFS getTrashRoot leads to OOM due to FileSystem object creation. (#1758) 
(cherry picked from commit 2338d25dc7150d75fbda84cc95422380b5622224)

 Conflicts:
        hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
 2020-02-21 13:40:11 +09:00
Akira Ajisaka
b52f201285
HADOOP-16869. Upgrade findbugs-maven-plugin to 3.0.5 to fix mvn findbugs:findbugs failure (#1855) 
(cherry picked from commit 7f35676f90f0730b8c9844cf00ee5a943f80d48d)

 Conflicts:
	hadoop-project/pom.xml
 2020-02-21 12:24:03 +09:00
Wei-Chiu Chuang
d411a849e1 HADOOP-16868. ipc.Server readAndProcess threw NullPointerException. Contributed by Tsz-wo Sze. 
(cherry picked from commit cb3f3cca01d6ab9f4befb1bcd31f384c37c0231a)
 2020-02-18 21:53:50 -08:00
Sunil G
de63115a2a YARN-10139. ValidateAndGetSchedulerConfiguration API fails when cluster max allocation > default 8GB. Contributed by Prabhu Joseph. 
(cherry picked from commit 6526f95bd281fc011f8776d21ff933087c5924de)
 2020-02-19 11:18:19 +05:30
xiaoxiaopan118
b5321f1600
HDFS-15173. RBF: Delete repeated configuration 'dfs.federation.router.metrics.enable' (#1849) 
(cherry picked from commit 439d935e1df601ed998521443fbe6752040e7a84)
 2020-02-17 15:57:15 +09:00
Ayush Saxena
9a351e837f HDFS-15164. Fix TestDelegationTokensWithHA. Contributed by Ayush Saxena. 2020-02-14 22:15:12 +05:30
Surendra Singh Lilhore
e13ea1b094 HDFS-15086. Block scheduled counter never get decremet if the block got deleted before replication. Contributed by hemanthboyina. 2020-02-13 22:52:50 +05:30
Ayush Saxena
9a1ce410f6 HDFS-15161. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException in ShortCircuitCache#close(). Contributed by Lisheng Sun 2020-02-12 20:29:09 +05:30
Akira Ajisaka
b80f020886
HADOOP-16849. start-build-env.sh behaves incorrectly when username is numeric only. Contributed by Jihyun Cho. 
(cherry picked from commit 9709afe67d8ed45c3dfb53e45fe1efdc0814ac6c)

 Conflicts:
	start-build-env.sh
 2020-02-12 14:07:56 +09:00
Kihwal Lee
f1840669cf HDFS-14758. Make lease hard limit configurable and reduce the default. 
Contributed by hemanthboyina.

(cherry picked from commit 9b8a78d97bfd825ce840c6033371c7f10e49a5b8)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend4.java
 2020-02-11 14:50:10 -06:00
testfixer
edad6f1d0f HADOOP-16847. Test can fail if HashSet iterates in a different order. 
Contributed by Testfixer
 2020-02-11 11:24:36 +00:00
Szilard Nemeth
6aec712c6c YARN-10101. Support listing of aggregated logs for containers belonging to an application attempt. Contributed by Adam Antal 2020-02-11 09:18:44 +01:00
Sunil G
95b1cbcbd4 YARN-10109. Allow stop and convert from leaf to parent queue in a single Mutation API call. Contributed by Prabhu Joseph 
(cherry picked from commit 28f730b317b23038bf9bd0775dd2cdb96518b13b)
 2020-02-09 21:15:31 +05:30
Ayush Saxena
7be104a9be HDFS-15115. Namenode crash caused by NPE in BlockPlacementPolicyDefault when dynamically change logger to debug. Contributed by wangzhixiang 2020-02-08 10:41:15 +05:30
Jonathan Hung
aca930402c YARN-10116. Expose diagnostics in RMAppManager summary 
(cherry picked from commit 314e2f9d2e888fae1e5bf669aeeead84a928d282)
 2020-02-05 11:17:03 -08:00
Kihwal Lee
0b19c8e086 HADOOP-15787. TestIPC.testRTEDuringConnectionSetup fails with jdk8u242. Contributed by Zsolt Venczel. 
(cherry picked from commit a463cf75a0ab1f0dbb8cfa16c39a4e698bc1a625)
 2020-02-05 13:02:21 -06:00
Chen Liang
226cd3977b HDFS-15148. dfs.namenode.send.qop.enabled should not apply to primary NN port. Contributed by Chen Liang. 2020-02-04 12:34:19 -08:00
Kihwal Lee
5257afb153 HDFS-12491. Support wildcard in CLASSPATH for libhdfs. Contributed by Muhammad Samir Khan. 
(cherry picked from commit 10a60fbe20bb08cdd71076ea9bf2ebb3a2f6226e)
 2020-02-04 12:24:12 -06:00
Prabhu Joseph
7136ebbb7a YARN-10022. Add RM Rest API to validate a CapacityScheduler Config with delta change 
Contributed by Kinga Marton.

(cherry-picked from commit 1ab9c692fa9107f10c0d8fbfbb9e74f54bddc960)
 2020-02-04 14:06:23 +05:30
Stephen O'Donnell
e89436b183 HDFS-7175. Client-side SocketTimeoutException during Fsck. Contributed by Stephen O'Donnell, Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 1e3a0b0d931676b191cb4813ed1a283ebb24d4eb)
 2020-01-31 16:13:42 -08:00
Inigo Goiri
2c8c6e8e22 HDFS-13179. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. Contributed by Ahmed Hussein. 
(cherry picked from commit 1839c467f60cbb8592d446694ec3d7710cda5142)
 2020-01-31 11:24:37 -08:00
Eric E Payne
f62c472944 MAPREDUCE-7079: JobHistory#ServiceStop implementation is incorrect. Contributed by Ahmed Hussein (ahussein) 
(cherry picked from commit b897f6834ba69d443c3acd1fab52261c00d675a1)
 2020-01-29 17:10:13 +00:00
Kihwal Lee
7034482222 HDFS-15146. TestBalancerRPCDelay.testBalancerRPCDelay fails 
intermittently. Contributed by Ahmed Hussein.

(cherry picked from commit 799d4c1cf4e8fe78eb9ab607a0449cdd075041fb)
 2020-01-29 11:01:12 -06:00
Szilard Nemeth
351c64f327 MAPREDUCE-7260. Cross origin request support for Job history server web UI. Contributed by Adam Antal 2020-01-29 14:47:10 +01:00
Eric Badger
5736ecd123 YARN-10084. Allow inheritance of max app lifetime / default app lifetime. Contributed by Eric Payne. 2020-01-29 04:07:28 +00:00
Chen Liang
9813ba2ebc [SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Contributed by Chen Liang 2020-01-28 16:36:27 -08:00
Ayush Saxena
8a1b02f44c HDFS-14993. checkDiskError doesn't work during datanode startup. Contributed by Yang Yun. 2020-01-28 23:12:44 +05:30
Ahmed Hussein
96caaf7185 MAPREDUCE-7259. testSpeculateSuccessfulWithUpdateEvents fails Intermittently 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 08251538fe2550d9dd86f9daf79994f5b8bdf7fa)
 2020-01-28 10:58:42 -06:00
Ahmed Hussein
6538e57c41 MAPREDUCE-7262. MRApp helpers block for long intervals (500ms) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 3f01c481060585ccd37be9db8aa4d1e33d2e2d6b)
 2020-01-27 16:10:42 -06:00
Kihwal Lee
1bb9667137 HDFS-15119. Allow expiration of cached locations in DFSInputStream. 
Contributed by Ahmed Hussein.

(cherry picked from commit d10f77e3c91225f86ed9c0f0e6a9adf2e1434674)
 2020-01-24 09:21:09 -06:00
Abhishek Modi
be412546be YARN-9790. Failed to set default-application-lifetime if maximum-application-lifetime is less than or equal to zero. Contributed by kyungwan nam. 
(cherry picked from commit d2d963f3d4819704351c04dbeb90fc8154488f91)
 2020-01-23 15:23:45 +00:00
Szilard Nemeth
dcc16b07fa YARN-10083. Addendum to fix compilation error due to missing import 2020-01-22 17:18:03 +01:00
Szilard Nemeth
1e7679035f YARN-7913. Improve error handling when application recovery fails with exception. Contributed by Wilfred Spiegelenburg 2020-01-22 16:50:15 +01:00
Kihwal Lee
4842708320 HDFS-14968. Add ability to log stale datanodes. Contributed by Ahmed Hussein. 
(cherry picked from commit bd03053ea2f32ef982e37fbf2ffd679cb7dda797)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
 2020-01-22 09:26:47 -06:00
Szilard Nemeth
da416c826f YARN-10083. Provide utility to ask whether an application is in final status. Contributed by Adam Antal 2020-01-22 16:18:35 +01:00
Szilard Nemeth
bbdc39c13e YARN-9462. TestResourceTrackerService.testNodeRemovalGracefully fails sporadically. Contributed by Prabhu Joseph 2020-01-22 15:49:35 +01:00
sunlisheng
c36fbcbf17 Remove WARN log when ipc connection interrupted in Client#handleSaslConnectionFailure() 
Signed-off-by: sunlisheng <sunlisheng@xiaomi.com>
(cherry picked from commit d887e49dd4ed2b94bbb53b7608586f5da6cee037)
 2020-01-21 18:24:00 +09:00
Akira Ajisaka
d4f75e2798
HADOOP-16808. Use forkCount and reuseForks parameters instead of forkMode in the config of maven surefire plugin. Contributed by Xieming Li. 
(cherry picked from commit f6d20daf404fab28b596171172afa4558facb504)
 2020-01-21 18:03:56 +09:00
Steve Loughran
429d5db3d9
HADOOP-16785. followup to abfs close() fix. 
Adds one extra test to the ABFS close logic, to explicitly
verify that the close sequence of FilterOutputStream is
not going to fail.

This is just a due-diligence patch, but it helps ensure
that no regressions creep in in future.

Contributed by Steve Loughran.

Change-Id: Ifd33a8c322d32513411405b15f50a1aebcfa6e48
 2020-01-20 16:26:33 +00:00
Szilard Nemeth
c411a1468f YARN-8148. Update decimal values for queue capacities shown on queue status CLI. Contributed by Prabhu Joseph 2020-01-20 09:29:52 +01:00