Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,150 Commits 326 Branches 371 Tags
Commit Graph

15150 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ahmed Hussein
8e22b80b5f MAPREDUCE-7262. MRApp helpers block for long intervals (500ms) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-01-28 09:13:54 -06:00
Kihwal Lee
8d4671b611 HADOOP-16683. Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException. Contributed by Adam Antal 
(cherry picked from commit 3d249301f42130de80ab3ecf8c783bb51b0fa8a3)
 2020-01-24 11:08:24 -06:00
Kihwal Lee
2873a705f4 HADOOP-16580. Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException. Contributed by Adam Antal 
(cherry picked from commit c79a5f2d9930f58ad95864c59cd0a6164cd53280)
 2020-01-24 11:07:06 -06:00
Abhishek Modi
296786a647 YARN-9790. Failed to set default-application-lifetime if maximum-application-lifetime is less than or equal to zero. Contributed by kyungwan nam. 
(cherry picked from commit d2d963f3d4819704351c04dbeb90fc8154488f91)
 2020-01-23 17:12:25 +00:00
Kihwal Lee
0ead38b59a HDFS-14968. Add ability to log stale datanodes. Contributed by Ahmed Hussein. 
(cherry picked from commit bd03053ea2f32ef982e37fbf2ffd679cb7dda797)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java

(cherry picked from commit 484270832064c84122348e663aaf5927ed411ebb)
 2020-01-22 09:30:41 -06:00
Kihwal Lee
c5d43b65a9 HDFS-15125. Pull back HDFS-11353, HDFS-13993, HDFS-13945, and HDFS-14324 
to branch-2.10. Contributed by Jim Brennan.
 2020-01-21 09:59:14 -06:00
sunlisheng
dc010b9844 HADOOP-16793. Redefine log level when ipc connection interrupted in Client#handleSaslConnectionFailure(). 
Signed-off-by: sunlisheng <sunlisheng@xiaomi.com>
(cherry picked from commit d887e49dd4ed2b94bbb53b7608586f5da6cee037)
 2020-01-21 18:56:41 +09:00
Akira Ajisaka
0c0be3b9f0
HADOOP-16808. Use forkCount and reuseForks parameters instead of forkMode in the config of maven surefire plugin. Contributed by Xieming Li. 
(cherry picked from commit f6d20daf404fab28b596171172afa4558facb504)
 2020-01-21 18:05:46 +09:00
kevinzhao1661
df16cea24f
MAPREDUCE-7247. Modify HistoryServerRest.html content,change The job attempt id‘s datatype from string to int (#1684) 
(cherry picked from commit b8518e6dabc1cf73fa7c43b3f8904d2e2beb2bf3)
 2020-01-16 12:58:57 +09:00
Wei-Chiu Chuang
d5f7aa2285 HDFS-13339. Volume reference can't be released and may lead to deadlock when DataXceiver does a check volume. Contributed by Jim Brennan, Zsolt Venczel. 
(cherry picked from commit c675719c3fb0fe5f6b0be624935fdf22bb228e0f)
 2020-01-15 11:06:15 -08:00
Masatake Iwasaki
7a309f1f91
MAPREDUCE-7256. Fix javadoc error in SimpleExponentialSmoothing. (#1804) 
(cherry picked from commit a2fdd7c2b59463c39a4d4250f908a9e2d3653ead)
 2020-01-15 12:13:12 +09:00
Chen Liang
6deb6be252 HDFS-15099. [SBN Read] checkOperation(WRITE) should throw ObserverRetryOnActiveException for ObserverNode. Contributed by Chen Liang. 
(cherry picked from commit 26a969ec734dbdbf1d544f486dfa33f15c291789)
 2020-01-10 17:19:59 -08:00
Daryn Sharp
23a77050e0 HADOOP-16749. Configuration parsing of CDATA values are blank 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-01-10 10:48:45 -06:00
Eric E Payne
c296e3c0aa YARN-9018. Add functionality to AuxiliaryLocalPathHandler to return all locations to read for a given path. Contributed by Kuhu Shukla (kshukla) 
(cherry picked from commit 93233a7d6e4d6b8098622a1aa830355cc18d9589)
 2020-01-09 17:52:26 +00:00
Ahmed Hussein
203f55fccc MAPREDUCE-7252. Handling 0 progress in SimpleExponential task runtime estimator 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-01-09 09:52:26 -06:00
Eric Badger
ab29b54b19 YARN-8672. TestContainerManager#testLocalingResourceWhileContainerRunning occasionally times out. Contributed by Chandni Singh and Jim Brennan. 2020-01-08 20:13:13 +00:00
Eric E Payne
5cca5ca81b YARN-7387: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer fails intermittently. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit b1e07d27cc1a26be4e5ebd1ab7b03ef15032bef0)
 2020-01-08 19:59:13 +00:00
Eric E Payne
2ae1b3568b YARN-10072: TestCSAllocateCustomResource failures. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit 6899be5a1729e49cff45090acd2cf4f54aeac089)
 2020-01-08 18:04:12 +00:00
Chen Liang
82bc4778ef HADOOP-16789. In TestZKFailoverController, restore changes from HADOOP-11149 that were dropped by HDFS-6440. Contributed by Jim Brennan. 2020-01-03 16:07:56 -08:00
Ayush Saxena
02642a6181 HADOOP-16784. Update the year to 2020. Contributed by Ayush Saxena. 2020-01-02 21:39:46 +05:30
Takanobu Asanuma
01a29f12c0 HDFS-14934. [SBN Read] Standby NN throws many InterruptedExceptions when dfs.ha.tail-edits.period is 0. Contributed by Ayush Saxena. 
(cherry picked from commit dc32f583afffc372f78fb45211c3e7ce13f6a4be)
 2019-12-28 22:24:28 +09:00
Duo Zhang
7db97a2326
HADOOP-16610. Upgrade to yetus 0.11.1 and use emoji vote on github pre commit (#1527). Contributed by Duo Zhang. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 3cecb2a46974d446a91435ac49c70742e3cbb6bd)
(cherry picked from commit 440b7abc049a7ea60cc2f72f33a792bdae523e03)
 2019-12-25 19:49:07 +09:00
Akira Ajisaka
eda4f5ed7c
HADOOP-16609. Add Jenkinsfile for all active branches (#1557) 
* Backport HADOOP-16035. Jenkinsfile for Hadoop
* Use Yetus 0.10.0

(cherry picked from commit ae8ca70fc0f83384800a249625285b657c3b3057)
 2019-12-25 17:26:06 +09:00
Konstantin V Shvachko
199c7cc62c HDFS-15076. Fix tests that hold FSDirectory lock, without holding FSNamesystem lock. Contributed by Konstantin V Shvachko. 
(cherry picked from commit b98ac2a3af50ccf2af07790ab0760d4c51820836)
 2019-12-24 12:48:10 -08:00
Eric Badger
cb5b80d6cb YARN-10009. In Capacity Scheduler, DRC can treat minimum user limit percent as a max when custom resource is defined. Contributed by Eric Payne. 2019-12-20 19:40:55 +00:00
Jonathan Hung
041fe5fb57 YARN-9894. CapacitySchedulerPerf test for measuring hundreds of apps in a large number of queues. Contributed by Eric Payne 
(cherry picked from commit 7b93575b92e8bad889c1ef15e0baaade6de6de4d)
(cherry picked from commit 0707d0a0ae36456f3467cbb408c3a9a0073c70f7)
(cherry picked from commit 750fb4c3212e7c197f418ea2df711be4467ee27a)
 2019-12-18 13:31:52 -08:00
Jonathan Hung
c4a8c834e5 YARN-10039. Allow disabling app submission from REST endpoints 
(cherry picked from commit 406c35dd1213ef354570f8337e54a4fb4bbf5bad)
 2019-12-18 11:01:29 -08:00
Shashikant Banerjee
abbf2a4446 HDFS-15012. NN fails to parse Edit logs after applying HDFS-13101. Contributed by Shashikant Banerjee. 
(cherry picked from commit fdd96e46d1f89f0ecdb9b1836dc7fca9fbb954fd)
 2019-12-18 23:30:34 +05:30
Ayush Saxena
0927041589 HDFS-14519. NameQuota is not update after concat operation, so namequota is wrong. Contributed by Ranith Sardar. 2019-12-17 21:55:58 +05:30
He Xiaoqiao
b91fda7f1d HDFS-15050. Optimize log information when DFSInputStream meet CannotObtainBlockLengthException. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 99af49878cc47f5750b6c0edadc8a4c2fcd7a7c6)
(cherry picked from commit 8ba4e13dcc4ad16e8483d33a31ed52e1b8d4e686)
(cherry picked from commit 974706c5fdd7c875922b38a2efffb6b7bda8cdf5)
 2019-12-12 03:25:28 -08:00
Xiao Chen
ba4de45e75 HDFS-13511. Provide specialized exception when block length cannot be obtained. Contributed by Gabor Bota. 
(cherry picked from commit 774c1f199e11d886d0c0a1069325f0284da35deb)
(cherry picked from commit 3b646a98365f0e7fb0690df63b1da6f7270c6e71)
 2019-12-12 03:25:28 -08:00
pingsutw
a969cad0a1
HADOOP-16754. Fix docker failed to build yetus/hadoop 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-12-12 11:05:43 +09:00
Mingliang Liu
a8044c75ad
HADOOP-16758. Refine testing.md to tell user better how to use auth-keys.xml (#1753) 
Contributed by Mingliang Liu
 2019-12-11 11:56:46 -08:00
Erik Krogen
a22698fa52 HDFS-15032. Properly handle InvocationTargetExceptions in the proxy created by ProxyCombiner. This fixes a bug encountered by the HDFS balancer when used with Observer Nodes. Contributed by Erik Krogen. 
(cherry picked from c174d50b30abc08a4642614fb35165e79792608b)
(cherry picked from 3402c87353c18730cb659ac1d8b43b4e21c634f6)
(cherry picked from b06965ae32c4be2e6923a2f057290fbdbda5f38a)
 2019-12-11 09:41:40 -08:00
Jonathan Hung
d74f64aa99 Setting version to 2.10.1-SNAPSHOT after branch-2 to branch-2.10 rename 2019-12-09 11:49:59 -08:00
Jonathan Hung
b04f1df4cf YARN-10012. Guaranteed and max capacity queue metrics for custom resources. Contributed by Manikandan R 
(cherry picked from commit 92bce918dc5d03560169642b71636800680a3292)
(cherry picked from commit 9228e3f0ad4e94cedd59c5b02a87ce6c267db5b5)
(cherry picked from commit 9b4368a62ff46db9529e333fdc2bc9db4863195d)
 2019-12-08 16:44:38 -08:00
Sunil G
5401d7f024 YARN-4901. QueueMetrics needs to be cleared before MockRM is initialized. Contributed by Peter Bacsko. 
(cherry picked from commit 002dcc4ebf79bbaa5e603565640d8289991d781f)
(cherry picked from commit 69dc329acc0174b133e8b68f16b3e894329a6fd0)
(cherry picked from commit 67cf1f94cd321761d676c5c9a8a07913fc3c2a70)
 2019-12-08 14:44:57 -08:00
Chao Sun
9a385e00a4 HDFS-15017. Remove redundant import of AtomicBoolean in NameNodeConnector. Contributed by Chao Sun. 2019-12-07 13:45:17 -08:00
Chao Sun
c6ce4011d7 HDFS-15005. Backport HDFS-12300 to branch-2. Contributed by Chao Sun. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-12-06 15:25:30 -08:00
Mingliang Liu
cdb0cafff5
HADOOP-16735. Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN. Contributed by Mingliang Liu 
This closes #1733
 2019-12-05 17:56:02 -08:00
Erik Krogen
0ac6dc79b0 HADOOP-16700. Track response time separately in Server.Call to avoid negative RpcQueueTime values. Contributed by xuzq. 
(cherry picked from be77231452baee2c3dd68f7c0991411cae2eea1b)
(cherry picked from c1ed2474595c672d95237c42a6e2a51913f14d25)
(cherry picked from 23f25418e822f90aa6bd0daa0ba65ab548b05eb1)
 2019-12-03 14:19:27 -08:00
Weiwei Yang
e7ecb4f51f YARN-9205. When using custom resource type, application will fail to run due to the CapacityScheduler throws InvalidResourceRequestException(GREATER_THEN_MAX_ALLOCATION). Contributed by Zhankun Tang. 
(cherry picked from commit bc6374f282dbff3b9ed91fb5d7825d57e6720f5e)
 2019-12-03 13:56:54 -08:00
Yiqun Lin
fc2cdb6cca HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago. 
(cherry picked from commit 2b452b4e6063072b2bec491edd3f412eb7ac21f3)
 2019-11-28 10:45:25 +08:00
Erik Krogen
403892264b HDFS-14973. More strictly enforce Balancer/Mover/SPS throttling of getBlocks RPCs to NameNodes. Contributed by Erik Krogen. 
(cherry picked from b2cc8b6b4a78f31cdd937dc4d1a2255f80c5881e)
(cherry picked from 60655bfe54e138957ef5bbf480a4541bd83152fd)
(cherry picked from 17779adb32569bd689a93802512003b6a6816bd4)
 2019-11-26 16:46:38 -08:00
Eric E Payne
3a7121b0b6 YARN-9915: Fix FindBug issue in QueueMetrics. Contributed by Prabhu Joseph. 
(cherry picked from commit 83d148074f9299de02d5c896a3ed4e11292cba73)
 2019-11-22 19:03:42 +00:00
Eric E Payne
bfb76e60cf YARN-9773: Add QueueMetrics for Custom Resources. Contributed by Manikandan R. 
(cherry picked from commit a5034c7988b6bf54bb7dab208100a2d205e3929e)
 2019-11-22 18:57:05 +00:00
Tao Yang
5b2a1b641a YARN-9838. Fix resource inconsistency for queues when moving app with reserved container to another queue. Contributed by jiulongzhu. 
(cherry picked from commit 8afabe41e4175551ae71217a45f7c4276f10b7da)
 2019-11-22 16:18:59 +08:00
Jason Lowe
dbbe23f269 YARN-7589. TestPBImplRecords fails with NullPointerException. Contributed by Daniel Templeton 
(cherry picked from commit 25df5054216a6a76d09d9c49984f8075ebc6a197)
 2019-11-21 11:55:20 -08:00
Eric E Payne
a9f51e0372 YARN-8842. Expose metrics for custom resource types in QueueMetrics. (Contributed by Szilard Nemeth) (refactored patch for branch-2) 2019-11-21 19:23:48 +00:00
Wangda Tan
835bc686e0 YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee. (Sunil G via wangda) 
Change-Id: I41b1d7558c20fc4eb2050d40134175a2ef6330cb
(cherry picked from commit 034b312d9f19024d2eabd377210d17d4080ef70e)

Conflicts:
  hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourcePBImpl.java
  I did not backport the changes to ResourcePBImpl.java because the code was removed by mistake and soon added back as part of YARN-7483 (Eric Payne).
 2019-11-19 22:39:14 +00:00