Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,591 Commits 326 Branches 371 Tags
Commit Graph

19591 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Inigo Goiri
4924622e6e YARN-10161. TestRouterWebServicesREST is corrupting STDOUT. Contributed by Jim Brennan. 
(cherry picked from commit a43510e21d01e6c78e98e7ad9469cbea70a66466)
 2020-02-27 13:20:17 -08:00
Istvan Fajth
b5022b0515 HDFS-14668 Support Fuse with Users from multiple Security Realms (#1739) 
(cherry picked from commit 57aa048516f5c5fe02441d213b52ce1bbeddf823)
(cherry picked from commit e42ac486e7eecb6a24ac95f1ceaf61d24060adef)
 2020-02-27 07:49:45 -08:00
Kihwal Lee
b92477c638 HDFS-15147. LazyPersistTestCase wait logic is error-prone. Contributed 
by Ahmed Hussein.

(cherry picked from commit 27cfda708ef66dfbe5f52a5f1e716298a294f3f7)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java
 2020-02-27 09:45:12 -06:00
Elixir Kook
9ccefe2262
YARN-10156. Fix typo 'complaint' which means quite different in Federation.md (#1856) 
(cherry picked from commit d608e94f9208354897a59dc8e533535b1b8a30b9)
 2020-02-26 17:32:47 +09:00
Stephen O'Donnell
8aaa8d1b71 HDFS-14861. Reset LowRedundancyBlocks Iterator periodically. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 900430b9907b590ed2d73a0d68f079c7f4d754b1)
(cherry picked from commit 2377649cdb1307a72a946b95a9a04f316c6483ef)
 2020-02-25 13:29:15 -08:00
Takanobu Asanuma
241c5a14c5 HADOOP-16841. The description of hadoop.http.authentication.signature.secret.file contains outdated information. Contributed by Xieming Li. 
(cherry picked from commit 5cbc4c54611f062690c3bfcf78d09d37c78ffd12)
 2020-02-25 11:10:22 +09:00
Wei-Chiu Chuang
268e35d3ae HDFS-15174. Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operations. Contributed by Lisheng Sun. 
(cherry picked from commit cb4daeaad95470125c59ccb80b988261ae83d534)
(cherry picked from commit f9437b1817e09c5a3ee7b378a0f4faee9b853315)
 2020-02-24 13:37:59 -08:00
Ayush Saxena
d318b48e12 HDFS-15166. Remove redundant field fStream in ByteStringLog. Contributed by Xieming Li. 2020-02-24 20:17:52 +05:30
Konstantin V Shvachko
cd0525054d HDFS-15185. StartupProgress reports edits segments until the entire startup completes. Contributed by Konstantin V Shvachko. 
(cherry picked from commit 6f84269bcd5cdb08ca68b2d8276f66d34a2a7a0d)
 2020-02-21 10:56:31 -08:00
Masatake Iwasaki
8095e3b149 HDFS-15052. WebHDFS getTrashRoot leads to OOM due to FileSystem object creation. (#1758) 
(cherry picked from commit 2338d25dc7150d75fbda84cc95422380b5622224)

 Conflicts:
        hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java

(cherry picked from commit 610805ec7245769aebb36e52725522c42cb3dd88)
 2020-02-21 13:51:26 +09:00
Akira Ajisaka
3f8ac06cef
HADOOP-16869. Upgrade findbugs-maven-plugin to 3.0.5 to fix mvn findbugs:findbugs failure (#1855) 
(cherry picked from commit 7f35676f90f0730b8c9844cf00ee5a943f80d48d)

 Conflicts:
	hadoop-project/pom.xml

(cherry picked from commit b52f2012858d99c8959e63034dec947454363674)
 2020-02-21 12:24:41 +09:00
Wei-Chiu Chuang
b28347d282 HADOOP-16868. ipc.Server readAndProcess threw NullPointerException. Contributed by Tsz-wo Sze. 
(cherry picked from commit cb3f3cca01d6ab9f4befb1bcd31f384c37c0231a)
(cherry picked from commit d411a849e10910d24cf1bbc8d64405470e00f510)
 2020-02-18 21:54:00 -08:00
Sunil G
a6124cd2b8 YARN-10139. ValidateAndGetSchedulerConfiguration API fails when cluster max allocation > default 8GB. Contributed by Prabhu Joseph. 
(cherry picked from commit 6526f95bd281fc011f8776d21ff933087c5924de)
 2020-02-19 11:18:53 +05:30
Kihwal Lee
c54051def5 HDFS-12459. Fix revert: Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3ead525c71cba068e7abf1c76ad629bfeec10852)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
 2020-02-17 16:07:03 -06:00
xiaoxiaopan118
b59fa3e571
HDFS-15173. RBF: Delete repeated configuration 'dfs.federation.router.metrics.enable' (#1849) 
(cherry picked from commit 439d935e1df601ed998521443fbe6752040e7a84)
 2020-02-17 15:58:35 +09:00
Ayush Saxena
2190fb1fcf HDFS-15115. Namenode crash caused by NPE in BlockPlacementPolicyDefault when dynamically change logger to debug. Contributed by wangzhixiang 2020-02-16 10:09:40 +05:30
Ayush Saxena
cf5a7186e5 HDFS-15164. Fix TestDelegationTokensWithHA. Contributed by Ayush Saxena. 2020-02-14 22:11:39 +05:30
Surendra Singh Lilhore
6a4297adc4 HDFS-15086. Block scheduled counter never get decremet if the block got deleted before replication. Contributed by hemanthboyina. 2020-02-13 23:04:28 +05:30
Ayush Saxena
83ab853733 HDFS-15161. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException in ShortCircuitCache#close(). Contributed by Lisheng Sun 2020-02-12 20:30:13 +05:30
Akira Ajisaka
1409c9da7c
HADOOP-16849. start-build-env.sh behaves incorrectly when username is numeric only. Contributed by Jihyun Cho. 
(cherry picked from commit 9709afe67d8ed45c3dfb53e45fe1efdc0814ac6c)

Conflicts:
	start-build-env.sh
 2020-02-12 14:11:00 +09:00
Kihwal Lee
bf09bfa2db HDFS-14758. Make lease hard limit configurable and reduce the default. 
Contributed by hemanthboyina.

(cherry picked from commit 9b8a78d97bfd825ce840c6033371c7f10e49a5b8)
(cherry picked from commit f1840669cfc92c033521bf95989479523e2a649d)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
 2020-02-11 14:57:49 -06:00
Prabhu Joseph
00fde836f7 YARN-10022. RM Rest API to validate the CapacityScheduler Configuration change. 
Contributed by Kinga Marton.

(cherry-picked from commit 1ab9c692fa9107f10c0d8fbfbb9e74f54bddc960)
 2020-02-11 22:23:07 +05:30
Ayush Saxena
547f1798aa HDFS-15157. Fix compilation failure for branch-3.1. Contributed by Ayush Saxena 2020-02-09 08:14:33 +05:30
Jonathan Hung
5dfd1dcfe3 YARN-10116. Expose diagnostics in RMAppManager summary 
(cherry picked from commit 314e2f9d2e888fae1e5bf669aeeead84a928d282)
(cherry picked from commit 147897da4b420b4749f3c7b410f4c329632c3352)
 2020-02-05 11:17:24 -08:00
Kihwal Lee
a628ab0863 HADOOP-15787. TestIPC.testRTEDuringConnectionSetup fails with jdk8u242. Contributed by Zsolt Venczel. 
(cherry picked from commit a463cf75a0ab1f0dbb8cfa16c39a4e698bc1a625)
(cherry picked from commit 0b19c8e0860bf558d995951ab9ac4cd8a1674759)
 2020-02-05 13:16:34 -06:00
Chen Liang
466f77a5cf HDFS-15148. dfs.namenode.send.qop.enabled should not apply to primary NN port. Contributed by Chen Liang. 
(cherry picked from commit 226cd3977bfc5948afff6a2d38a41e341089d5fe)
 2020-02-04 12:36:46 -08:00
Kihwal Lee
a55a0a1f6d HDFS-12491. Support wildcard in CLASSPATH for libhdfs. Contributed by Muhammad Samir Khan. 
(cherry picked from commit 10a60fbe20bb08cdd71076ea9bf2ebb3a2f6226e)
 2020-02-04 12:24:58 -06:00
Stephen O'Donnell
8ea4787f9f HDFS-7175. Client-side SocketTimeoutException during Fsck. Contributed by Stephen O'Donnell, Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 1e3a0b0d931676b191cb4813ed1a283ebb24d4eb)
(cherry picked from commit e89436b1839aa87b4bedfc7e2d558c7541e73540)
 2020-01-31 16:13:53 -08:00
Inigo Goiri
ffaf0d3c82 HDFS-13179. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. Contributed by Ahmed Hussein. 
(cherry picked from commit 1839c467f60cbb8592d446694ec3d7710cda5142)
 2020-01-31 11:25:36 -08:00
Eric Badger
4af7d14ce2 YARN-10084. Allow inheritance of max app lifetime / default app lifetime. Contributed by Eric Payne. 2020-01-29 22:45:02 +00:00
Chen Liang
5e1d89b74d [SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Conntributed by Chen Liang 2020-01-29 11:15:35 -08:00
Eric E Payne
d4ecb65629 MAPREDUCE-7079: JobHistory#ServiceStop implementation is incorrect. Contributed by Ahmed Hussein (ahussein) 
(cherry picked from commit b897f6834ba69d443c3acd1fab52261c00d675a1)
 2020-01-29 17:21:40 +00:00
Kihwal Lee
b8d87cfc32 HDFS-15146. TestBalancerRPCDelay.testBalancerRPCDelay fails 
intermittently. Contributed by Ahmed Hussein.

(cherry picked from commit 799d4c1cf4e8fe78eb9ab607a0449cdd075041fb)
 2020-01-29 11:01:51 -06:00
Ayush Saxena
1293199532 HDFS-14993. checkDiskError doesn't work during datanode startup. Contributed by Yang Yun. 2020-01-28 23:17:21 +05:30
Ahmed Hussein
f3ff61865a MAPREDUCE-7259. testSpeculateSuccessfulWithUpdateEvents fails Intermittently 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 08251538fe2550d9dd86f9daf79994f5b8bdf7fa)
 2020-01-28 11:04:25 -06:00
Ahmed Hussein
1f315c28d9 MAPREDUCE-7262. MRApp helpers block for long intervals (500ms) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 3f01c481060585ccd37be9db8aa4d1e33d2e2d6b)
 2020-01-27 16:23:06 -06:00
Kihwal Lee
7dd43cec7f HDFS-15119. Allow expiration of cached locations in DFSInputStream. 
Contributed by Ahmed Hussein.

(cherry picked from commit d10f77e3c91225f86ed9c0f0e6a9adf2e1434674)
(cherry picked from commit 1bb9667137bc80e63dea3e53bb2746be011a2a9a)
 2020-01-24 09:53:26 -06:00
Abhishek Modi
868a0f5ef0 YARN-9790. Failed to set default-application-lifetime if maximum-application-lifetime is less than or equal to zero. Contributed by kyungwan nam. 
(cherry picked from commit d2d963f3d4819704351c04dbeb90fc8154488f91)
 2020-01-23 15:50:28 +00:00
Szilard Nemeth
9638985428 YARN-7913. Improve error handling when application recovery fails with exception. Contributed by Wilfred Spiegelenburg 2020-01-22 16:30:59 +01:00
Kihwal Lee
b4e9725955 HDFS-14968. Add ability to log stale datanodes. Contributed by Ahmed Hussein. 
(cherry picked from commit bd03053ea2f32ef982e37fbf2ffd679cb7dda797)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java

(cherry picked from commit 484270832064c84122348e663aaf5927ed411ebb)
 2020-01-22 09:28:01 -06:00
sunlisheng
96c653d0d5 Remove WARN log when ipc connection interrupted in Client#handleSaslConnectionFailure() 
Signed-off-by: sunlisheng <sunlisheng@xiaomi.com>
(cherry picked from commit d887e49dd4ed2b94bbb53b7608586f5da6cee037)
 2020-01-21 18:32:14 +09:00
Akira Ajisaka
2c84ea96e3
HADOOP-16808. Use forkCount and reuseForks parameters instead of forkMode in the config of maven surefire plugin. Contributed by Xieming Li. 
(cherry picked from commit f6d20daf404fab28b596171172afa4558facb504)
 2020-01-21 18:05:13 +09:00
Szilard Nemeth
2cdaeca84b YARN-8148. Update decimal values for queue capacities shown on queue status CLI. Contributed by Prabhu Joseph 2020-01-20 09:30:48 +01:00
kevinzhao1661
ad34d7032e
MAPREDUCE-7247. Modify HistoryServerRest.html content,change The job attempt id‘s datatype from string to int (#1684) 
(cherry picked from commit b8518e6dabc1cf73fa7c43b3f8904d2e2beb2bf3)
 2020-01-16 12:58:30 +09:00
Sunil G
af89b5b086 YARN-8373. RM Received RMFatalEvent of type CRITICAL_THREAD_CRASH. Contributed by Wilfred Spiegelenburg. 2020-01-15 17:07:34 +05:30
Masatake Iwasaki
ce52a66fcb
MAPREDUCE-7256. Fix javadoc error in SimpleExponentialSmoothing. (#1804) 
(cherry picked from commit a2fdd7c2b59463c39a4d4250f908a9e2d3653ead)
 2020-01-15 12:12:27 +09:00
Szilard Nemeth
3a09593a73 HADOOP-16683. Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException. Contributed by Adam Antal 2020-01-14 11:19:52 +01:00
Wei-Chiu Chuang
286c94f13d HDFS-15097. Purge log in KMS and HttpFS. Contributed by Doris Gu. 
(cherry picked from commit 6b86a5110e4b3f45bf55f97bd402680bf26cefb9)
(cherry picked from commit aa3c96474f52fba01d999fa75880e0948affbb01)
 2020-01-13 10:49:39 -08:00
Ayush Saxena
12b258115b HDFS-14578. AvailableSpaceBlockPlacementPolicy always prefers local node. Contributed by Ayush Saxena. 2020-01-11 10:34:37 +05:30
Chen Liang
5490061885 HDFS-15099. [SBN Read] checkOperation(WRITE) should throw ObserverRetryOnActiveException for ObserverNode. Contributed by Chen Liang. 
(cherry picked from commit 26a969ec734dbdbf1d544f486dfa33f15c291789)
 2020-01-10 17:19:05 -08:00