Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,134 Commits 326 Branches 371 Tags
Commit Graph

15134 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Eric E Payne
5cca5ca81b YARN-7387: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer fails intermittently. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit b1e07d27cc1a26be4e5ebd1ab7b03ef15032bef0)
 2020-01-08 19:59:13 +00:00
Eric E Payne
2ae1b3568b YARN-10072: TestCSAllocateCustomResource failures. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit 6899be5a1729e49cff45090acd2cf4f54aeac089)
 2020-01-08 18:04:12 +00:00
Chen Liang
82bc4778ef HADOOP-16789. In TestZKFailoverController, restore changes from HADOOP-11149 that were dropped by HDFS-6440. Contributed by Jim Brennan. 2020-01-03 16:07:56 -08:00
Ayush Saxena
02642a6181 HADOOP-16784. Update the year to 2020. Contributed by Ayush Saxena. 2020-01-02 21:39:46 +05:30
Takanobu Asanuma
01a29f12c0 HDFS-14934. [SBN Read] Standby NN throws many InterruptedExceptions when dfs.ha.tail-edits.period is 0. Contributed by Ayush Saxena. 
(cherry picked from commit dc32f583afffc372f78fb45211c3e7ce13f6a4be)
 2019-12-28 22:24:28 +09:00
Duo Zhang
7db97a2326
HADOOP-16610. Upgrade to yetus 0.11.1 and use emoji vote on github pre commit (#1527). Contributed by Duo Zhang. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 3cecb2a46974d446a91435ac49c70742e3cbb6bd)
(cherry picked from commit 440b7abc049a7ea60cc2f72f33a792bdae523e03)
 2019-12-25 19:49:07 +09:00
Akira Ajisaka
eda4f5ed7c
HADOOP-16609. Add Jenkinsfile for all active branches (#1557) 
* Backport HADOOP-16035. Jenkinsfile for Hadoop
* Use Yetus 0.10.0

(cherry picked from commit ae8ca70fc0f83384800a249625285b657c3b3057)
 2019-12-25 17:26:06 +09:00
Konstantin V Shvachko
199c7cc62c HDFS-15076. Fix tests that hold FSDirectory lock, without holding FSNamesystem lock. Contributed by Konstantin V Shvachko. 
(cherry picked from commit b98ac2a3af50ccf2af07790ab0760d4c51820836)
 2019-12-24 12:48:10 -08:00
Eric Badger
cb5b80d6cb YARN-10009. In Capacity Scheduler, DRC can treat minimum user limit percent as a max when custom resource is defined. Contributed by Eric Payne. 2019-12-20 19:40:55 +00:00
Jonathan Hung
041fe5fb57 YARN-9894. CapacitySchedulerPerf test for measuring hundreds of apps in a large number of queues. Contributed by Eric Payne 
(cherry picked from commit 7b93575b92e8bad889c1ef15e0baaade6de6de4d)
(cherry picked from commit 0707d0a0ae36456f3467cbb408c3a9a0073c70f7)
(cherry picked from commit 750fb4c3212e7c197f418ea2df711be4467ee27a)
 2019-12-18 13:31:52 -08:00
Jonathan Hung
c4a8c834e5 YARN-10039. Allow disabling app submission from REST endpoints 
(cherry picked from commit 406c35dd1213ef354570f8337e54a4fb4bbf5bad)
 2019-12-18 11:01:29 -08:00
Shashikant Banerjee
abbf2a4446 HDFS-15012. NN fails to parse Edit logs after applying HDFS-13101. Contributed by Shashikant Banerjee. 
(cherry picked from commit fdd96e46d1f89f0ecdb9b1836dc7fca9fbb954fd)
 2019-12-18 23:30:34 +05:30
Ayush Saxena
0927041589 HDFS-14519. NameQuota is not update after concat operation, so namequota is wrong. Contributed by Ranith Sardar. 2019-12-17 21:55:58 +05:30
He Xiaoqiao
b91fda7f1d HDFS-15050. Optimize log information when DFSInputStream meet CannotObtainBlockLengthException. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 99af49878cc47f5750b6c0edadc8a4c2fcd7a7c6)
(cherry picked from commit 8ba4e13dcc4ad16e8483d33a31ed52e1b8d4e686)
(cherry picked from commit 974706c5fdd7c875922b38a2efffb6b7bda8cdf5)
 2019-12-12 03:25:28 -08:00
Xiao Chen
ba4de45e75 HDFS-13511. Provide specialized exception when block length cannot be obtained. Contributed by Gabor Bota. 
(cherry picked from commit 774c1f199e11d886d0c0a1069325f0284da35deb)
(cherry picked from commit 3b646a98365f0e7fb0690df63b1da6f7270c6e71)
 2019-12-12 03:25:28 -08:00
pingsutw
a969cad0a1
HADOOP-16754. Fix docker failed to build yetus/hadoop 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-12-12 11:05:43 +09:00
Mingliang Liu
a8044c75ad
HADOOP-16758. Refine testing.md to tell user better how to use auth-keys.xml (#1753) 
Contributed by Mingliang Liu
 2019-12-11 11:56:46 -08:00
Erik Krogen
a22698fa52 HDFS-15032. Properly handle InvocationTargetExceptions in the proxy created by ProxyCombiner. This fixes a bug encountered by the HDFS balancer when used with Observer Nodes. Contributed by Erik Krogen. 
(cherry picked from c174d50b30abc08a4642614fb35165e79792608b)
(cherry picked from 3402c87353c18730cb659ac1d8b43b4e21c634f6)
(cherry picked from b06965ae32c4be2e6923a2f057290fbdbda5f38a)
 2019-12-11 09:41:40 -08:00
Jonathan Hung
d74f64aa99 Setting version to 2.10.1-SNAPSHOT after branch-2 to branch-2.10 rename 2019-12-09 11:49:59 -08:00
Jonathan Hung
b04f1df4cf YARN-10012. Guaranteed and max capacity queue metrics for custom resources. Contributed by Manikandan R 
(cherry picked from commit 92bce918dc5d03560169642b71636800680a3292)
(cherry picked from commit 9228e3f0ad4e94cedd59c5b02a87ce6c267db5b5)
(cherry picked from commit 9b4368a62ff46db9529e333fdc2bc9db4863195d)
 2019-12-08 16:44:38 -08:00
Sunil G
5401d7f024 YARN-4901. QueueMetrics needs to be cleared before MockRM is initialized. Contributed by Peter Bacsko. 
(cherry picked from commit 002dcc4ebf79bbaa5e603565640d8289991d781f)
(cherry picked from commit 69dc329acc0174b133e8b68f16b3e894329a6fd0)
(cherry picked from commit 67cf1f94cd321761d676c5c9a8a07913fc3c2a70)
 2019-12-08 14:44:57 -08:00
Chao Sun
9a385e00a4 HDFS-15017. Remove redundant import of AtomicBoolean in NameNodeConnector. Contributed by Chao Sun. 2019-12-07 13:45:17 -08:00
Chao Sun
c6ce4011d7 HDFS-15005. Backport HDFS-12300 to branch-2. Contributed by Chao Sun. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-12-06 15:25:30 -08:00
Mingliang Liu
cdb0cafff5
HADOOP-16735. Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN. Contributed by Mingliang Liu 
This closes #1733
 2019-12-05 17:56:02 -08:00
Erik Krogen
0ac6dc79b0 HADOOP-16700. Track response time separately in Server.Call to avoid negative RpcQueueTime values. Contributed by xuzq. 
(cherry picked from be77231452baee2c3dd68f7c0991411cae2eea1b)
(cherry picked from c1ed2474595c672d95237c42a6e2a51913f14d25)
(cherry picked from 23f25418e822f90aa6bd0daa0ba65ab548b05eb1)
 2019-12-03 14:19:27 -08:00
Weiwei Yang
e7ecb4f51f YARN-9205. When using custom resource type, application will fail to run due to the CapacityScheduler throws InvalidResourceRequestException(GREATER_THEN_MAX_ALLOCATION). Contributed by Zhankun Tang. 
(cherry picked from commit bc6374f282dbff3b9ed91fb5d7825d57e6720f5e)
 2019-12-03 13:56:54 -08:00
Yiqun Lin
fc2cdb6cca HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago. 
(cherry picked from commit 2b452b4e6063072b2bec491edd3f412eb7ac21f3)
 2019-11-28 10:45:25 +08:00
Erik Krogen
403892264b HDFS-14973. More strictly enforce Balancer/Mover/SPS throttling of getBlocks RPCs to NameNodes. Contributed by Erik Krogen. 
(cherry picked from b2cc8b6b4a78f31cdd937dc4d1a2255f80c5881e)
(cherry picked from 60655bfe54e138957ef5bbf480a4541bd83152fd)
(cherry picked from 17779adb32569bd689a93802512003b6a6816bd4)
 2019-11-26 16:46:38 -08:00
Eric E Payne
3a7121b0b6 YARN-9915: Fix FindBug issue in QueueMetrics. Contributed by Prabhu Joseph. 
(cherry picked from commit 83d148074f9299de02d5c896a3ed4e11292cba73)
 2019-11-22 19:03:42 +00:00
Eric E Payne
bfb76e60cf YARN-9773: Add QueueMetrics for Custom Resources. Contributed by Manikandan R. 
(cherry picked from commit a5034c7988b6bf54bb7dab208100a2d205e3929e)
 2019-11-22 18:57:05 +00:00
Tao Yang
5b2a1b641a YARN-9838. Fix resource inconsistency for queues when moving app with reserved container to another queue. Contributed by jiulongzhu. 
(cherry picked from commit 8afabe41e4175551ae71217a45f7c4276f10b7da)
 2019-11-22 16:18:59 +08:00
Jason Lowe
dbbe23f269 YARN-7589. TestPBImplRecords fails with NullPointerException. Contributed by Daniel Templeton 
(cherry picked from commit 25df5054216a6a76d09d9c49984f8075ebc6a197)
 2019-11-21 11:55:20 -08:00
Eric E Payne
a9f51e0372 YARN-8842. Expose metrics for custom resource types in QueueMetrics. (Contributed by Szilard Nemeth) (refactored patch for branch-2) 2019-11-21 19:23:48 +00:00
Wangda Tan
835bc686e0 YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee. (Sunil G via wangda) 
Change-Id: I41b1d7558c20fc4eb2050d40134175a2ef6330cb
(cherry picked from commit 034b312d9f19024d2eabd377210d17d4080ef70e)

Conflicts:
  hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourcePBImpl.java
  I did not backport the changes to ResourcePBImpl.java because the code was removed by mistake and soon added back as part of YARN-7483 (Eric Payne).
 2019-11-19 22:39:14 +00:00
He Xiaoqiao
ae7469cb67 HDFS-14952. Skip safemode if blockTotal is 0 in new NN. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Mukul Kumar Singh <msingh@apache.org>
(cherry picked from commit 0b50aa29fd5dc114b3e0fc54b5c393bbc9f3102e)
(cherry picked from commit 1d383436f09f8e490171c8f6af4605971ad2f904)
(cherry picked from commit 7909c5df572a5cbc729ebc6c7c6e59f17803d50f)
 2019-11-19 12:23:01 -08:00
Chen Liang
238781ab98 HDFS-14991. Backport better time precision of configuration#getTimeDuration to branch-2 to support SBN read. Contributed by Chen Liang. 2019-11-18 20:26:27 -08:00
Eric E Payne
a74411ebaa Revert "YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee. (Sunil G via wangda)" 
Pulling this back into branch-2 prevented nodemanagers from coming up if extended resources are present.

This reverts commit ab83765a6c587f8378daed741f7d05598d076188.
 2019-11-15 22:01:28 +00:00
Eric E Payne
6973f78a73 YARN-8179: Preemption does not happen due to natural_termination_factor when DRF is used. Contributed by Kyungwan Nam. 
(cherry picked from commit 0b4c44bdeef62945b592d5761666ad026b629c0b)
 2019-11-15 19:00:06 +00:00
Wangda Tan
ab83765a6c YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee. (Sunil G via wangda) 
Change-Id: I41b1d7558c20fc4eb2050d40134175a2ef6330cb
(cherry picked from commit 034b312d9f19024d2eabd377210d17d4080ef70e)
 2019-11-15 17:21:30 +00:00
Erik Krogen
792e2aecb3 HDFS-14979 Allow Balancer to submit getBlocks calls to Observer Nodes when possible. Contributed by Erik Krogen. 
(cherry picked from 586defe7113ed246ed0275bb3833882a3d873d70)
(cherry picked from dec765b329d3947f30273c0e7f0c4eb607ec42c9)
(cherry picked from 5cf36aa2b19aad8677e1a7553ff1f05805f772b7)
 2019-11-15 07:40:20 -08:00
Akira Ajisaka
81060b3413
HADOOP-15097. AbstractContractDeleteTest::testDeleteNonEmptyDirRecursive with misleading path. Contributed by Xieming Li. 
(cherry picked from commit 92c28c100ee1aa414948cd510321ad13cb8639bc)
 2019-11-15 13:53:39 +09:00
Mukul Kumar Singh
594307fe21 HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Yuval Degani, Mukul Kumar Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Yuval Degani <ydegani@linkedin.com>
 2019-11-14 20:50:51 -08:00
Robert Kanter
235d6c73e5 YARN-8202. DefaultAMSProcessor should properly check units of requested custom resource types against minimum/maximum allocation (snemeth via rkanter) 
(cherry picked from commit c8b53c43644b4ad22d5385c22cad8ed573c0b1ba)
(cherry picked from commit 0506c762b2f96ccb2c12cce8fd4659536236354e)
 2019-11-14 15:47:24 -08:00
Daniel Templeton
7f1d2e196c YARN-7541. Node updates don't update the maximum cluster capability for resources other than CPU and memory 
(cherry picked from commit 8498d287cd3beddcf8fe19625227e09982ec4be2)
 2019-11-14 14:56:23 -08:00
Wangda Tan
b5128b2ae2 YARN-7739. DefaultAMSProcessor should properly check customized resource types against minimum/maximum allocation. (wangda) 
Change-Id: I10cc9341237d9a2fc0f8c855efb98a36b91389e2
(cherry picked from commit d02e42cee4a08a47ed2835f7a4a100daaa95833f)
 2019-11-14 14:44:30 -08:00
Ayush Saxena
d1b91ede4c HDFS-14958. TestBalancerWithNodeGroup is not using NetworkTopologyWithNodeGroup. Contributed by Jim Brennan. 2019-11-07 22:32:13 +05:30
Akira Ajisaka
5e100ada27 HDFS-14590. [SBN Read] Add the document link to the top page. Contributed by Takanobu Asanuma. 
(cherry picked from commit 48d7f00da8aab25707dc08219122541867edfa65)
 2019-11-07 22:32:03 +09:00
Chen Liang
92c9b1de5b HDFS-14806. Bootstrap standby may fail if with in-progress tailing. Contributed by Chen Liang and Erik Krogen. 2019-11-06 16:23:51 -08:00
Chen Liang
db887f66d8 HDFS-14941. Potential editlog race condition can cause corrupted file. Contributed by Chen Liang and Konstantin Shvachko. 2019-11-06 15:47:57 -08:00
Ahmed Hussein
c59b1b66a5 MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2019-11-05 15:48:27 -06:00