Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,716 Commits 326 Branches 371 Tags
Commit Graph

959 Commits

Author SHA1 Message Date
Sangjin Lee
e212e562ec YARN-5355. Backported YARN-2928 into our branch-2 feature branch. 2017-10-17 23:36:33 +05:30
Junping Du
3d2ef54450 YARN-7124. LogAggregationTFileController deletes/renames while file is open. Contributed by Jason Lowe. 
(cherry picked from commit 1fcbe7cf5f2f7c609cd4158912f151848268ca9e)
 2017-10-16 14:29:02 -07:00
Yufei Gu
f2350f5120 YARN-7270. Fix unsafe casting from long to int for class Resource and its sub-classes. (Yufei) 
(cherry picked from commit 7a27c2c367518e1bf6ee393391a2f9b412113819)
(cherry picked from commit dbf6c48af9aa2f3b41b1146643db6eb0140b38c5)
 2017-10-13 13:10:20 -07:00
Jonathan Hung
06c8938ab0 YARN-7251. Misc changes to YARN-5734 2017-10-09 11:12:08 -07:00
Wangda Tan
887b366d5f YARN-6840. Implement zookeeper based store for scheduler configuration updates. (Jonathan Hung via wangda) 
Change-Id: I9debea674fe8c7e4109d4ca136965a1ea4c48bcc
 2017-10-09 11:12:07 -07:00
Xuan
cd58f5da5a YARN-5947: Create LeveldbConfigurationStore class using Leveldb as backing store. Contributed by Jonathan Hung 2017-10-09 11:12:07 -07:00
Xuan
16b27573ef YARN-5953 addendum: Move QueueConfigInfo and SchedConfUpdateInfo to package org.apache.hadoop.yarn.webapp.dao 2017-10-09 11:12:07 -07:00
Xuan
96f4cf7cde YARN-5953:Create CLI for changing YARN configurations. (Jonathan Hung via xgong) 2017-10-09 11:12:07 -07:00
Wangda Tan
a47823ca98 YARN-5949. Add pluggable configuration ACL policy interface and implementation. (Jonathan Hung via wangda) 
Change-Id: Ib98e82ff753bede21fcab2e6ca9ec1e7a5a2008f
 2017-10-09 11:12:07 -07:00
Jonathan Hung
072527b31a YARN-5948. Implement MutableConfigurationManager for handling storage into configuration store 2017-10-09 11:12:07 -07:00
Naganarasimha
5cb3bd3b06 YARN-7285. ContainerExecutor always launches with priorities due to yarn-default property. Contributed by Jason Lowe. 
(cherry picked from commit d0046bb5c6919c9620ffa5330eb37b39484151ad)
 2017-10-06 00:09:46 +05:30
Wangda Tan
8beae14a07 YARN-7259. Add size-based rolling policy to LogAggregationIndexedFileController. (xgong via wangda) 
Change-Id: Ifaf82c0aee6b73b9b6ebf103aa72e131e3942f31
(cherry picked from commit 280080fad01304c85a9ede4d4f7b707eb36c0155)
 2017-10-02 15:36:11 -07:00
Wangda Tan
5fa5ba7f4d YARN-6509. Add a size threshold beyond which yarn logs will require a force option. (Xuan Gong via wangda) 
Change-Id: I755fe903337d4ff9ec35dae5b9cce638794e1d0f
(cherry picked from commit ec2ae3060a807c8754826af2135a68c08b2e4f13)
(cherry picked from commit d58d75ca501c6d1bb20605c1c964c0d351012763)
 2017-09-29 16:53:16 -07:00
Jason Lowe
c0bfb385a8 YARN-7260. yarn.router.pipeline.cache-max-size is missing in yarn-default.xml. Contributed by Jason Lowe 2017-09-29 09:53:05 -05:00
Jason Lowe
913a64e4c9 YARN-7248. NM returns new SCHEDULED container status to older clients. Contributed by Arun Suresh 
(cherry picked from commit 85d81ae58ec4361a944c84753a900460a0888b9b)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
 2017-09-28 14:25:01 -05:00
Xuan
c143708acf YARN-7257. AggregatedLogsBlock reports a bad 'end' value as a bad 
'start' value. Contributed by Jason Lowe

(cherry picked from commit 28c4957fccebe2d7e63ec9fe9af58313b4f21d4f)
(cherry picked from commit 81e78727938a1c0369f164849cc3a08add31be46)
 2017-09-27 16:04:06 -07:00
Rohith Sharma K S
c570ddaa66 YARN-7157. Add admin configuration to filter per-user's apps in secure cluster. Contributed by Sunil G. 2017-09-27 11:49:59 +05:30
Carlo Curino
bfd967d338 YARN-5412. Create a proxy chain for ResourceManager REST API in the Router. (Contributed by Giovanni Matteo Fumarola via curino) 
(cherry picked from commit b6240b92abf453affc5fd64e1eedf2d29842aa75)
(cherry picked from commit acda6b96a4e92e432bd1d97fa14004a11e70387e)
 2017-09-21 17:13:28 -07:00
Subru Krishnan
169037cbb4 YARN-3659. Federation: routing client invocations transparently to multiple RMs. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 52daa6d971ae408d121b3737ea8c0575e7e8516d)
(cherry picked from commit 43a97174fe49aa0c25d03b8a970a46d4bebf1aa8)
 2017-09-21 16:55:33 -07:00
Subru Krishnan
9476d86ce8 YARN-5531. UnmanagedAM pool manager for federating application across clusters. (Botong Huang via Subru). 
(cherry picked from commit 73bb2102ce4b82b3a3bed91319f7c8f067ddc3e8)
(cherry picked from commit 859aa1f9d621d07693825e610bdc0149f7a2770a)
 2017-09-21 16:47:43 -07:00
Subru Krishnan
7f00f938f2 YARN-5413. Create a proxy chain for ResourceManager Admin API in the Router. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 67846a5519b5905c2d925cf4c602f715b653e72c)
(cherry picked from commit 7444406d6d48846d06b04e82587885bb8632a16c)
 2017-09-21 16:47:14 -07:00
Subru Krishnan
dc0a2e6c59 YARN-5411. Create a proxy chain for ApplicationClientProtocol in the Router. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 4846069061b6baa06da3b524b9e36567dd368388)
(cherry picked from commit 80e1904000923819c98ae68456cb0452c255d5f3)
 2017-09-21 16:43:08 -07:00
Subru Krishnan
61c07e4f3f YARN-6247. Share a single instance of SubClusterResolver instead of instantiating one per AM. (Botong Huang via Subru) 
(cherry picked from commit 51aeb2ce0c599176aca9466a939c3ad55df30036)
(cherry picked from commit 86b2bec56e28a2d1ece53ab5a452860fd0444268)
 2017-09-21 16:25:09 -07:00
Jian He
615c912b61 YARN-3673. Create a FailoverProxy for Federation services. Contributed by Subru Krishnan 
(cherry picked from commit 3307564a5f8c8abc5fe84efcd05ee0f7dfdd921c)
(cherry picked from commit bdfad4523f1a5a776e20773c371ce99d0c538ac1)
 2017-09-21 16:13:06 -07:00
Jian He
fb3f9b48fb YARN-3674. Create Facade for Federation State and Policy Store. Contributed by Subru Krishnan 
(cherry picked from commit 5e976cd2b90ccf1bccb6039edf14140677804c4e)
(cherry picked from commit 5c843823972d030555e25729ae7b4b8fbafb59fd)
 2017-09-21 16:12:34 -07:00
Subru Krishnan
31d92378ba YARN-5390. Federation Subcluster Resolver. Contributed by Ellen Hui. 
(cherry picked from commit d3dc461a935c2af4ec3f0312ff0c26918c408467)
(cherry picked from commit d19b6773012d78cb20b92e98e13546013b6622c2)
 2017-09-21 16:00:36 -07:00
Jason Lowe
fc6d9612c7 YARN-4266. Allow users to enter containers as UID:GID pair instead of by username. Contributed by luhuichun, Zhankun Tang, and Eric Badger. 
(cherry picked from commit bfd1a72ba8fbb06da73fede2a85e0b544d6ab43f)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
 2017-09-21 17:52:11 -05:00
Sangjin Lee
a79422f717 YARN-6771. Use classloader inside configuration class to make new 
classes. Contributed by Jongyoul Lee.

(cherry picked from commit 8b336632acad10e45d029596c5e3196e1857d891)
 2017-09-20 22:46:10 -07:00
Arun Suresh
8a12a914fd YARN-7203. Add container ExecutionType into ContainerReport. (Botong Huang via asuresh) 
(cherry picked from commit 56ef5279c1db93d03b2f1e04badbfe804f548918)
 2017-09-18 15:51:19 -07:00
Jason Lowe
cd6cf0caf6 YARN-7192. Add a pluggable StateMachine Listener that is notified of NM Container State changes. Contributed by Arun Suresh 
(cherry picked from commit a4f9c7c9247801dd37beec6fc195622af1b884ad)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
 2017-09-18 10:34:53 -05:00
Arun Suresh
52bf458616 YARN-5216. Expose configurable preemption policy for OPPORTUNISTIC containers running on the NM. (Hitesh Sharma via asuresh) 
(cherry picked from commit 4f8194430fc6a69d9cc99b78828fd7045d5683e8)
 2017-09-14 10:29:54 -07:00
Junping Du
7e6cc17133 YARN-7194. Log aggregation status is always Failed with the newly added log aggregation IndexedFileFormat. Contributed by Xuan Gong. 
(cherry picked from commit c92c1d521eadfd8a4cd8205cc6aee74816f353f4)
 2017-09-13 22:18:39 -07:00
Xuan
02ae3f8202 YARN-7072. Add a new log aggregation file format controller (xgong) 2017-09-10 23:18:31 -07:00
Sunil G
a39829c456 YARN-6600. Introduce default and max lifetime of application at LeafQueue level. Contributed by Rohith Sharma K S. 2017-09-08 19:20:52 +05:30
Jason Lowe
c004d57d47 YARN-6930. Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager. Contributed by Shane Kumpf 2017-09-07 16:24:55 -05:00
Junping Du
1ef48f1d4b YARN-7144. Log Aggregation controller should not swallow the exceptions when it calls closeWriter and closeReader. Contributed by Xuan Gong. 
(cherry picked from commit 22de9449f8aa72c5b0bb586b8253390773502189)
 2017-09-06 14:54:29 -07:00
Junping Du
eaf134c46c YARN-7148. TestLogsCLI fails in trunk and branch-2 and javadoc error. Contributed by Xuan Gong. 
(cherry picked from commit 63231a2a3008794c5ebcbc9d3855638fc5f28216)
 2017-09-06 13:20:58 -07:00
Junping Du
119220b88f YARN-6877. Create an abstract log reader for extendability. Contributed by Xuan Gong. 2017-09-01 03:04:55 -07:00
Junping Du
b906bddd95 YARN-7037. Optimize data transfer with zero-copy approach for containerlogs REST API in NMWebServices. Contributed by Tao Yang. 
(cherry picked from commit ad45d19998c1b0da25754d0016854046731fa623)
 2017-08-29 15:17:21 -07:00
Junping Du
e12358c5ce YARN-6876. Create an abstract log writer for extendability. Contributed by Xuan Gong. 
(cherry picked from commit c2cb7ea1ef6532020b69031dbd18b0f9b8369f0f)
 2017-08-24 15:27:40 -07:00
Arun Suresh
f8f193c8e8 YARN-6979. Add flag to notify all types of container updates to NM via NodeHeartbeatResponse. (Kartheek Muthyala via asuresh) 
(cherry picked from commit 8410d862d3a72740f461ef91dddb5325955e1ca5)
 2017-08-20 08:31:17 -07:00
Subru Krishnan
a70efb6138 HADOOP-14741. Refactor curator based ZooKeeper communication into common library. (Íñigo Goiri via Subru). 2017-08-14 11:03:50 -07:00
Junping Du
e9c21d14b4 YARN-6811. [ATS1.5] All history logs should be kept under its own User Directory. Contributed by Rohith Sharma K S. 2017-08-08 12:45:55 -07:00
Daniel Templeton
675d368950 YARN-6757. Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path 
(Contributed by Miklos Szegedi via Daniel Templeton)

(cherry picked from commit 47b145b9b4e81d781891abce8a6638f0b436acc4)
 2017-08-08 11:12:34 -07:00
Arun Suresh
4cfd248a72 YARN-6777. Support for ApplicationMasterService processing chain of interceptors. (asuresh) 
(cherry picked from commit 077fcf6a96e420e7f36350931722b8603d010cf1)
 2017-08-04 16:50:21 -07:00
Arun Suresh
2d62af6545 YARN-5966. AMRMClient changes to support ExecutionType update. (asuresh) 
(cherry picked from commit aaf106fde35ec97e2e2ea4d7a67434038c4273ac)
 2017-08-04 16:19:44 -07:00
Arun Suresh
88d6adf419 YARN-5977. ContainerManagementProtocol changes to support change of container ExecutionType. (Kartheek Muthyala via asuresh) 
(cherry picked from commit 35dc7829236f92054d5ce6ea78d3a44ca6c8f3d3)
 2017-08-03 21:18:43 -07:00
Yufei Gu
ce609d06c0 YARN-6832. Tests use assertTrue(....equals(...)) instead of assertEquals()). (Daniel Templeton via Yufei Gu) 2017-08-03 13:52:58 -07:00
Jason Lowe
e76e7b9f0b YARN-6837. Null LocalResource visibility or resource type can crash the nodemanager. Contributed by Jinjiang Ling 
(cherry picked from commit c8df3668ecc37c2d58cad35520a762eaec3c8539)
 2017-07-20 11:06:39 -05:00
Sunil G
d83e8712f6 YARN-5892. Support user-specific minimum user limit percentage in Capacity Scheduler. Contributed by Eric Payne. 
(cherry picked from commit ca13b224b2feb9c44de861da9cbba8dd2a12cb35)
 2017-07-13 14:50:28 -05:00