Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
13,765 Commits 326 Branches 371 Tags
Commit Graph

13765 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Jonathan Hung
c8f9fe88d5 YARN-7046. Add closing logic to configuration store 2017-10-09 11:12:08 -07:00
Wangda Tan
887b366d5f YARN-6840. Implement zookeeper based store for scheduler configuration updates. (Jonathan Hung via wangda) 
Change-Id: I9debea674fe8c7e4109d4ca136965a1ea4c48bcc
 2017-10-09 11:12:07 -07:00
Xuan
36cda2ff80 YARN-7024: Fix issues on recovery in LevelDB store. Contributed by Jonathan Hung 2017-10-09 11:12:07 -07:00
Xuan
3a31c38885 YARN-6322: Disable queue refresh when configuration mutation is enabled. Contributed by Jonathan Hung 2017-10-09 11:12:07 -07:00
Xuan
cd58f5da5a YARN-5947: Create LeveldbConfigurationStore class using Leveldb as backing store. Contributed by Jonathan Hung 2017-10-09 11:12:07 -07:00
Xuan
16b27573ef YARN-5953 addendum: Move QueueConfigInfo and SchedConfUpdateInfo to package org.apache.hadoop.yarn.webapp.dao 2017-10-09 11:12:07 -07:00
Xuan
96f4cf7cde YARN-5953:Create CLI for changing YARN configurations. (Jonathan Hung via xgong) 2017-10-09 11:12:07 -07:00
Xuan
3c50084219 YARN-6575. Support global configuration mutation in MutableConfProvider. (Jonathan Hung via Xuan Gong) 2017-10-09 11:12:07 -07:00
Wangda Tan
a47823ca98 YARN-5949. Add pluggable configuration ACL policy interface and implementation. (Jonathan Hung via wangda) 
Change-Id: Ib98e82ff753bede21fcab2e6ca9ec1e7a5a2008f
 2017-10-09 11:12:07 -07:00
Wangda Tan
53a1268125 YARN-5952. Create REST API for changing YARN scheduler configurations. (Jonathan Hung via wangda) 2017-10-09 11:12:07 -07:00
Jonathan Hung
072527b31a YARN-5948. Implement MutableConfigurationManager for handling storage into configuration store 2017-10-09 11:12:07 -07:00
Xuan
38cf9ccd5b YARN-5946: Create YarnConfigurationStore interface and 
InMemoryConfigurationStore class. Contributed by Jonathan Hung
 2017-10-09 11:12:07 -07:00
Jonathan Hung
cd4bc54e6c YARN-5951. Changes to allow CapacityScheduler to use configuration store 2017-10-09 11:12:07 -07:00
Daniel Templeton
4d377c89bb YARN-2612 addendum: fixed javadoc error. 
(cherry picked from commit 6d6ca4c923142a893870fc8a6309c02f60dc2d24)
(cherry picked from commit bec03f7060d353e78a68bb64c88397cf7181727d)
 2017-10-09 14:16:50 +02:00
Sunil G
045526aca1 YARN-7245. Max AM Resource column in Active Users Info section of Capacity Scheduler UI page should be updated per-user. Contributed by Eric Payne. 
(cherry picked from commit 8137254a1f4367974bf1883f1982e31ab2865107)
 2017-10-09 14:44:33 +05:30
Steve Loughran
34040e8d73 HADOOP-14899. Restrict Access to setPermission operation when authorization is enabled in WASB 
Contributed by Kannapiran Srinivasan.
 2017-10-07 11:20:18 +01:00
Xiaoyu Yao
65b5e81752 HADOOP-14920. KMSClientProvider won't work with KMS delegation token retrieved from non-Java client. Contributed by Xiaoyu Yao. 
(cherry picked from commit 2b08a1fc644904a37545107666efc25b3552542d)
 2017-10-06 10:39:37 -07:00
Yufei Gu
da0a96968d YARN-7207. Cache the RM proxy server address. (Yufei Gu) 
(cherry picked from commit 72d22b753abde4d07a727479d3f3d5d84d5dd6b2)

# Conflicts:
#	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
#	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
 2017-10-06 09:56:24 -07:00
Yufei Gu
c949e8e83a YARN-2162. Add ability in Fair Scheduler to optionally configure maxResources in terms of percentage. (Yufei Gu) 2017-10-06 00:35:03 -07:00
Xiao Chen
dd0113d777 HADOOP-14521. KMS client needs retry logic. Contributed by Rushabh S Shah. 
(cherry picked from commit 212342d6db14673f8f5652a5209e1c4fcc0f0ede)
 2017-10-05 19:47:04 -07:00
Wangda Tan
73d1b02d5a HADOOP-13835. Move Google Test Framework code from mapreduce to hadoop-common. (Varun Vasudev via wangda) 
Change-Id: I1031a4fa4e0c5c76928d03c5e61971978ec7e6ab
 2017-10-05 17:02:00 -07:00
Arun Suresh
cb91bb3c96 YARN-7258. Add Node and Rack Hints to Opportunistic Scheduler. (Kartheek Muthyala via asuresh). 
(cherry picked from commit b733348dde18a242e6c9074c512116a8baf1d281)
 2017-10-05 15:22:20 -07:00
Naganarasimha
5cb3bd3b06 YARN-7285. ContainerExecutor always launches with priorities due to yarn-default property. Contributed by Jason Lowe. 
(cherry picked from commit d0046bb5c6919c9620ffa5330eb37b39484151ad)
 2017-10-06 00:09:46 +05:30
Chris Trezzo
d12cea2098 YARN-2960. Add documentation for the YARN shared cache. 
(cherry picked from commit 7e76f85bc68166b01b51fcf6ba4b3fd9281d4a03)
 2017-10-05 11:23:24 -07:00
Steve Loughran
fa6b8feb9e
HADOOP-14845. Azure wasb: getFileStatus not making any auth check. 
Final fixup of TestNativeAzureFileSystemAuthorization in branch-2 to match trunk's changes
Contributed by Sivaguru Sankaridurg
 2017-10-05 16:03:13 +01:00
Subru Krishnan
7fd4a997f4 YARN-5329. Placement Agent enhancements required to support recurring reservations in ReservationSystem. (Carlo Curino via Subru). 
(cherry picked from commit e6e614e380ed1d746973b50f666a9c40d272073e)
 2017-10-04 19:30:57 -07:00
Subru Krishnan
7836a6b59a YARN-2037. Add work preserving restart support for Unmanaged AMs. (Botong Huang via Subru) 2017-10-04 16:29:08 -07:00
Akira Ajisaka
5508b9b7c8
YARN-7044. TestContainerAllocation#testAMContainerAllocationWhenDNSUnavailable fails. 
(cherry picked from commit 2df1b2ac0509ba10fff606ada7e9b3562c12dd16)
 2017-10-05 06:08:46 +09:00
Sidharta S
3ecbfff937 YARN-7226. Whitelisted variables do not support delayed variable expansion. Contributed by Jason Lowe 2017-10-04 14:05:01 -07:00
Wangda Tan
175ff0585d YARN-7269. Tracking URL in the app state does not get redirected to ApplicationMaster for Running applications. (wangda via Jian He) 
Change-Id: I582566e09d5ef67e9cc9e9805231ccbabe57c3d0
 2017-10-03 12:19:11 -07:00
Steve Loughran
28e05dc1da
HADOOP-14220 Enhance S3GuardTool with bucket-info and set-capacity commands, tests. 
Contributed by Steve Loughran
 2017-10-03 20:06:45 +01:00
Wangda Tan
42e873d4ea YARN-6625. yarn application -list returns a tracking URL for AM that doesn't work in secured and HA environment. (Yufei Gu) 2017-10-03 12:04:56 -07:00
Wangda Tan
598329e6f8 YARN-6550. Capture launch_container.sh logs to a separate log file. (Suma Shivaprasad via wangda) 
Change-Id: I66c9b50ae3955dda0b5bf0674a9bfb728db27acb
 2017-10-03 09:46:35 -07:00
Wangda Tan
8beae14a07 YARN-7259. Add size-based rolling policy to LogAggregationIndexedFileController. (xgong via wangda) 
Change-Id: Ifaf82c0aee6b73b9b6ebf103aa72e131e3942f31
(cherry picked from commit 280080fad01304c85a9ede4d4f7b707eb36c0155)
 2017-10-02 15:36:11 -07:00
Steve Loughran
1eecf8a97c
HADOOP-14768. Honoring sticky bit during Deletion when authorization is enabled in WASB 
Contributed by Varada Hemeswari
 2017-10-02 14:28:59 +01:00
Xiao Chen
78d6dd414a HADOOP-14095. Document caveats about the default JavaKeyStoreProvider in KMS. 
(cherry picked from commit d7ecac379a02876919d3e6081d42f0937f54e664)

Conflicts:
	hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm
 2017-09-29 19:34:50 -07:00
Yufei Gu
e84d508c1b YARN-6333. Improve doc for minSharePreemptionTimeout, fairSharePreemptionTimeout and fairSharePreemptionThreshold. (Contributed by Chetna Chaudhari via Yufei Gu) 
(cherry picked from commit 66c417167ae0d24467360ae3b222a10f00fa2700)
 2017-09-29 18:08:55 -07:00
Wangda Tan
5fa5ba7f4d YARN-6509. Add a size threshold beyond which yarn logs will require a force option. (Xuan Gong via wangda) 
Change-Id: I755fe903337d4ff9ec35dae5b9cce638794e1d0f
(cherry picked from commit ec2ae3060a807c8754826af2135a68c08b2e4f13)
(cherry picked from commit d58d75ca501c6d1bb20605c1c964c0d351012763)
 2017-09-29 16:53:16 -07:00
Eric Payne
b0ba31c518 YARN-7084. TestSchedulingMonitor#testRMStarts fails sporadically. Contributed by Jason Lowe. 
(cherry picked from commit 7f6118f918e0fbe81cb64e8b9f6959b8d66679f6)
 2017-09-29 10:50:21 -05:00
Jason Lowe
c0bfb385a8 YARN-7260. yarn.router.pipeline.cache-max-size is missing in yarn-default.xml. Contributed by Jason Lowe 2017-09-29 09:53:05 -05:00
Steve Loughran
a1afc6aa59
HADOOP-13345 S3Guard: Improved Consistency for S3A. 
Contributed by: Chris Nauroth, Aaron Fabbri, Mingliang Liu, Lei (Eddy) Xu,
Sean Mackrory, Steve Loughran and others.

(cherry-picked from/based on commit 621b43e254afaff708cd6fc4698b29628f6abc33)
 2017-09-29 11:29:22 +01:00
Steve Loughran
a641bcec01
HADOOP-14845. Azure wasb: getFileStatus not making any auth check. 
Contributed by Sivaguru Sankaridurg
 2017-09-29 11:16:31 +01:00
Chris Trezzo
cba1891b61 YARN-7250. Update Shared cache client api to use URLs. 
(cherry picked from commit c114da5e64d14b1d9e614081c4171ea0391cb1aa)
 2017-09-28 15:32:18 -07:00
Jason Lowe
11ac10fe1d HADOOP-14902. LoadGenerator#genFile write close timing is incorrectly calculated. Contributed by Hanisha Koneru 
(cherry picked from commit 6f789fe05766a61b12ca10df3f26ee354eac84aa)
 2017-09-28 16:40:18 -05:00
Subru Krishnan
ffcf5ba1ce YARN-6962. Add support for updateContainers when allocating using FederationInterceptor. (Botong Huang via Subru). 
(cherry picked from commit ca669f9f8bc7abe5b7d4648c589aa1756bd336d1)
 2017-09-28 13:11:32 -07:00
Subru Krishnan
d6da014f67 YARN-6955. Handle concurrent register AM requests in FederationInterceptor. (Botong Huang via Subru). 
(cherry picked from commit c61f2c419830e40bb47fb2b1fe1f7d6109ed29a9)
 2017-09-28 13:11:19 -07:00
Jason Lowe
913a64e4c9 YARN-7248. NM returns new SCHEDULED container status to older clients. Contributed by Arun Suresh 
(cherry picked from commit 85d81ae58ec4361a944c84753a900460a0888b9b)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
 2017-09-28 14:25:01 -05:00
Arun Suresh
53c1115908 YARN-6691. Update YARN daemon startup/shutdown scripts to include Router service. (Giovanni Matteo Fumarola via asuresh) 2017-09-28 11:41:43 -07:00
Xuan
c143708acf YARN-7257. AggregatedLogsBlock reports a bad 'end' value as a bad 
'start' value. Contributed by Jason Lowe

(cherry picked from commit 28c4957fccebe2d7e63ec9fe9af58313b4f21d4f)
(cherry picked from commit 81e78727938a1c0369f164849cc3a08add31be46)
 2017-09-27 16:04:06 -07:00
Rohith Sharma K S
c570ddaa66 YARN-7157. Add admin configuration to filter per-user's apps in secure cluster. Contributed by Sunil G. 2017-09-27 11:49:59 +05:30