Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,943 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

5502 Commits

Author SHA1 Message Date
Haibo Chen 5d4a432660 YARN-9025. TestFairScheduler#testChildMaxResources is flaky. (Contributed by Szilard Nemeth) 2018-12-05 16:36:13 -08:00
Haibo Chen 912b1f9d64 YARN-9019. Ratio calculation of ResourceCalculator implementations could return NaN. (Contributed by Szilard Nemeth) 2018-12-05 15:16:41 -08:00
Haibo Chen b3c75c1f1d YARN-8985. Improve debug log in FSParentQueue when assigning container. (Contributed by Wilfred Spiegelenburg) 2018-12-05 15:02:57 -08:00
Haibo Chen 2bba329c14 YARN-8994. Fix race condition between move app and queue cleanup in Fair Scheduler. (Contributed by Wilfred Spiegelenburg) 2018-12-05 14:38:15 -08:00
Eric Yang 1b790f4dd1 YARN-9071. Improved status update for reinitialized containers. 
Contributed by Chandni Singh
 2018-12-05 17:00:56 -05:00
Eric Yang 1dabb31cdf YARN-9057. Removed third party class bundle from CSI jar file. 
Contributed by Weiwei Yang
 2018-12-05 15:56:10 -05:00
Wangda Tan 228156cfd1 Revert "YARN-8870. [Submarine] Add submarine installation scripts. (Xun Liu via wangda)" 
This reverts commit 46d6e00166.
 2018-12-04 14:13:06 -08:00
Yufei Gu e89941fdbb YARN-9041. Performance Optimization of method FSPreemptionThread#identifyContainersToPreempt. Contributed by Wanqiang Ji. 2018-12-04 10:08:45 -08:00
Jonathan Hung 6d7b44c489 YARN-9036. Escape newlines in health report in YARN UI. Contributed by Keqiu Hu 2018-11-30 10:13:52 -08:00
Wangda Tan bad12031f6 YARN-9010. Fix the incorrect trailing slash deletion in constructor method of CGroupsHandlerImpl. (Zhankun Tang via wangda) 
Change-Id: Iaecc66d57781cc10f19ead4647e47fc9556676da
 2018-11-29 14:56:07 -08:00
bibinchundatt 07142f54a8 YARN-9069. Fix SchedulerInfo#getSchedulerType for custom schedulers. Contributed by Bilwa S T. 2018-11-29 22:02:59 +05:30
bibinchundatt a68d766e87 YARN-8948. PlacementRule interface should be for all YarnSchedulers. Contributed by Bibin A Chundatt. 2018-11-29 21:43:34 +05:30
Weiwei Yang efc4d91cbe YARN-9067. YARN Resource Manager is running OOM because of leak of Configuration Object. Contributed by Eric Yang. 2018-11-29 09:34:14 +08:00
Wangda Tan fe7dab8ef5 YARN-8989. [YARN-8851] Move DockerCommandPlugin volume related APIs' invocation from DockerLinuxContainerRuntime#prepareContainer to #launchContainer. (Zhankun Tang via wangda) 
Change-Id: Ia6d532c687168448416dfdf46f0ac34bff20e6ca
 2018-11-28 15:03:06 -08:00
Wangda Tan 897643928c YARN-8975. [Submarine] Use predefined Charset object StandardCharsets.UTF_8 instead of String UTF-8. (Zhankun Tang via wangda) 
Change-Id: If6c7904aa17895e543cfca245264249eb7328bdc
 2018-11-28 14:39:06 -08:00
Wangda Tan 8ebeda98a9 YARN-8974. Improve the assertion message in TestGPUResourceHandler. (Zhankun Tang via wangda) 
Change-Id: I4eb58e9d251d5f54e7feffc4fbb813b4f5ae4b1b
 2018-11-28 14:36:30 -08:00
Wangda Tan 9ed87567ad YARN-9061. Improve the GPU/FPGA module log message of container-executor. (Zhankun Tang via wangda) 
Change-Id: Iece9b47438357077a53984a820d4d6423f480518
 2018-11-28 14:31:31 -08:00
Wangda Tan 579ef4be06 YARN-8882. [YARN-8851] Add a shared device mapping manager (scheduler) for device plugins. (Zhankun Tang via wangda) 
Change-Id: I9435136642c3d556971a357bf687f69df90bb45e
 2018-11-28 14:09:52 -08:00
Wangda Tan 9de8e8d049 YARN-9030. Log aggregation changes to handle filesystems which do not support setting permissions. (Suma Shivaprasad via wangda) 
Change-Id: I80f1e8196b8624e24d74494719fdedfd7061dced
 2018-11-28 13:36:21 -08:00
Jason Lowe 3ce99e32f7 YARN-8812. Containers fail during creating a symlink which started with hyphen for a resource file. Contributed by Oleksandr Shevchenko 2018-11-28 08:46:11 -06:00
Suma Shivaprasad b3a052d199 YARN-9044. LogsCLI should contact ATSv2 for -am option. Contributed by Rohith Sharma K S 2018-11-28 00:46:53 -08:00
Suma Shivaprasad 7dc272199f YARN-9034. ApplicationCLI should have option to take clusterId. Contributed by Rohith Sharma K S. 2018-11-28 00:42:14 -08:00
Eric Yang f657a2a661 YARN-9056. Improved YARN service upgrade state logic for readiness check. 
Contributed by Chandni Singh
 2018-11-27 18:36:59 -05:00
Eric Yang 33e0df4b35 YARN-8986. Added port publish for Docker container running with bridge network. 
Contributed by Charo Zhang
 2018-11-26 19:45:05 -05:00
Giovanni Matteo Fumarola 0d8406135f YARN-9054. Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest. Contributed by Bibin A Chundatt. 2018-11-26 10:57:38 -08:00
Rohith Sharma K S e223a790a7 YARN-9042. Fix javadoc error in deviceplugin package. Contributed by Zhankun Tang. 2018-11-22 20:04:30 +05:30
Weiwei Yang 176bb3f812 YARN-8984. AMRMClient#OutstandingSchedRequests leaks when AllocationTags is null or empty. Contributed by Yang Wang. 2018-11-22 16:52:29 +08:00
Sunil G 5fe1dbf195 YARN-8964. [UI2] YARN ui2 should use clusters/{cluster name} for all ATSv2 REST APIs. Contributed by Akhil PB. 2018-11-22 10:25:22 +05:30
Haibo Chen a41b648e98 YARN-8992. Fair scheduler can delete a dynamic queue while an application attempt is being added to the queue. (Contributed by Wilfred Spiegelenburg) 2018-11-20 14:12:37 -08:00
Billie Rinaldi 49824ed260 YARN-8838. Check that container user is same as websocket user for interactive shell. Contributed by Eric Yang 2018-11-20 11:12:24 -08:00
Sunil G 5fb14e0635 YARN-8953. [CSI] CSI driver adaptor module support in NodeManager. Contributed by Weiwei Yang. 2018-11-20 08:02:11 +05:30
Billie Rinaldi 27ffec7ba7 YARN-8778. Add command line interface to invoke interactive docker shell. Contributed by Eric Yang 2018-11-19 17:59:12 -08:00
Wangda Tan 8b23814415 YARN-8960. [Submarine] Can't get submarine service status using the command of "yarn app -status" under security environment. (Zac Zhou via wangda) 
Change-Id: I21b1addc9c32817650ea744a8f2e6b5602f2f4d4
 2018-11-19 08:55:30 -08:00
Wangda Tan 6357803645 YARN-8881. [YARN-8851] Add basic pluggable device plugin framework. (Zhankun Tang via wangda) 
Change-Id: If9a2f68cd4713b4ec932cdeda68106f17437c3d3
 2018-11-19 08:54:31 -08:00
Weiwei Yang d027a24f03 YARN-8833. Avoid potential integer overflow when computing fair shares. Contributed by liyakun. 2018-11-18 23:18:26 +08:00
Giovanni Matteo Fumarola abd6d48c46 YARN-8860. Federation client intercepter class contains unwanted character. Contributed by Abhishek Modi. 2018-11-16 10:23:06 -08:00
Rohith Sharma K S ee3355be3c YARN-8303. YarnClient should contact TimelineReader for application/attempt/container report. 2018-11-16 17:58:19 +05:30
Inigo Goiri eeaf8edaa7 YARN-8856. TestTimelineReaderWebServicesHBaseStorage tests failing with NoClassDefFoundError. Contributed by Sushil Ks. 2018-11-15 09:25:25 -08:00
Eric Yang 21ec4bdaef YARN-8672. Improve token filename management for localization. 
Contributed by Chandni Singh
 2018-11-14 15:22:01 -05:00
Wangda Tan fcbd205cc3 YARN-9001. [Submarine] Use AppAdminClient instead of ServiceClient to sumbit jobs. (Zac Zhou via wangda) 
Change-Id: Ic3d6c1e439df9cdf74448b345b925343224efe51
 2018-11-13 13:13:27 -08:00
Wangda Tan 076b795b2e YARN-8918. [Submarine] Correct method usage of str.subString in CliUtils. (Zhankun Tang via wangda) 
Change-Id: Id1f11dbab3aa838dee3a0ec8b8fd5dc32f5dd946
 2018-11-13 11:24:15 -08:00
Eric Yang 4c465f5535 YARN-8761. Service AM support for decommissioning component instances. 
Contributed by Billie Rinaldi
 2018-11-12 19:53:10 -05:00
Giovanni Matteo Fumarola e269c3fb5a YARN-8997. [Submarine] Small refactors of modifier, condition check and redundant local variables. Contributed by Zhankun Tang. 2018-11-12 15:06:43 -08:00
Billie Rinaldi 1f9c4f32e8 YARN-8776. Implement Container Exec feature in LinuxContainerExecutor. Contributed by Eric Yang 2018-11-12 10:42:30 -08:00
Sunil G 42f3a7082a YARN-8877. [CSI] Extend service spec to allow setting resource attributes. Contributed by Weiwei Yang. 2018-11-12 21:09:30 +05:30
Weiwei Yang c741109522 YARN-8987. Usability improvements node-attributes CLI. Contributed by Bibin A Chundatt. 2018-11-12 18:18:23 +08:00
Sunil G 4e7284443e YARN-8902. [CSI] Add volume manager that manages CSI volume lifecycle. Contributed by Weiwei Yang. 2018-11-12 11:57:02 +05:30
Botong Huang b5ec85d966 YARN-8933. [AMRMProxy] Fix potential empty fields in allocation response, move SubClusterTimeout to FederationInterceptor. Contributed by Botong Huang. 2018-11-11 11:12:53 -08:00
Eric Yang 2664248797 YARN-9002. Improve keytab loading for YARN Service. 
Contributed by Gour Saha
 2018-11-10 01:52:19 -05:00
Haibo Chen 524a7523c4 YARN-8990. Fix fair scheduler race condition in app submit and queue cleanup. (Contributed by Wilfred Spiegelenburg) 2018-11-08 16:02:48 -08:00
Weiwei Yang e1bbf7dcdf YARN-8988. Reduce the verbose log on RM heartbeat path when distributed node-attributes is enabled. Contributed by Tao Yang. 2018-11-08 17:47:18 +08:00
Weiwei Yang f8c72d7b3a YARN-8880. Add configurations for pluggable plugin framework. Contributed by Zhankun Tang. 2018-11-08 12:23:00 +08:00
Weiwei Yang c96cbe8659 YARN-8977. Remove unnecessary type casting when calling AbstractYarnScheduler#getSchedulerNode. Contributed by Wanqiang Ji. 2018-11-07 22:42:22 +08:00
Takanobu Asanuma 8dc1f6dbf7 YARN-8866. Fix a parsing error for crossdomain.xml. 2018-11-07 18:26:07 +09:00
Weiwei Yang 482716e5a4 YARN-8976. Remove redundant modifiers in interface ApplicationConstants. Contributed by Zhankun Tang. 2018-11-07 10:48:07 +08:00
Akira Ajisaka 951c98f890
YARN-8233. NPE in CapacityScheduler#tryCommit when handling allocate/reserve proposal whose allocatedOrReservedContainer is null. Contributed by Tao Yang. 2018-11-07 11:17:35 +09:00
Eric Yang 08d69d91f2 YARN-8957. Add Serializable interface to ComponentContainers. 
Contributed by Zhankun Tang
 2018-11-06 14:46:05 -05:00
Jason Lowe ab6aa4c726 YARN-8865. RMStateStore contains large number of expired RMDelegationToken. Contributed by Wilfred Spiegelenburg 2018-11-06 08:40:59 -06:00
Weiwei Yang 5d6554c722 YARN-8970. Improve the debug message in CS#allocateContainerOnSingleNode. Contributed by Zhankun Tang. 2018-11-06 14:50:09 +08:00
Weiwei Yang c7fcca0d7e YARN-8969. AbstractYarnScheduler#getNodeTracker should return generic type to avoid type casting. Contributed by Wanqiang Ji. 2018-11-06 13:14:57 +08:00
bibinchundatt f84a278baa YARN-8905. [Router] Add JvmMetricsInfo and pause monitor. Contributed by Bilwa S T. 2018-11-03 20:35:31 +05:30
Giovanni Matteo Fumarola 989715ec50 YARN-8893. [AMRMProxy] Fix thread leak in AMRMClientRelayer and UAM client. Contributed by Botong Huang. 2018-11-02 15:30:08 -07:00
Giovanni Matteo Fumarola aed836efbf YARN-8897. LoadBasedRouterPolicy throws NPE in case of sub cluster unavailability. Contributed by Bilwa S T. 2018-11-02 11:27:11 -07:00
Giovanni Matteo Fumarola babc946d40 YARN-8954. Reservations list field in ReservationListInfo is not accessible. Contributed by Oleksandr Shevchenko. 2018-11-02 11:10:08 -07:00
Jonathan Hung 2ab611d48b YARN-7225. Add queue and partition info to RM audit log. Contributed by Eric Payne 2018-11-01 14:22:00 -07:00
Steve Loughran d174b91635 HADOOP-15895. [JDK9+] Add missing javax.annotation-api dependency to hadoop-yarn-csi. 
Contributed by Takanobu Asanuma.
 2018-11-01 12:56:20 +00:00
Akira Ajisaka f747f5b06c
HADOOP-15886. Fix findbugs warnings in RegistryDNS.java. 2018-10-31 10:01:31 +09:00
Shane Kumpf 277a3d8d9f YARN-6729. Clarify documentation on how to enable cgroup support. Contributed by Zhankun Tang 2018-10-30 11:36:55 -06:00
Sunil G d36012b69f YARN-8854. Upgrade jquery datatable version references to v1.10.19. Contributed by Akhil PB. 2018-10-30 22:56:46 +05:30
Rohith Sharma K S 486b9a4a75 YARN-7754. [Atsv2] Update document for running v1 and v2 TS. Contributed by Suma Shivaprasad. 2018-10-30 11:35:01 +05:30
Rohith Sharma K S a283da2167 YARN-8871. Document ATSv2 integrated LogWebService. Contributed by Suma Shivaprasad. 2018-10-30 11:34:20 +05:30
Rohith Sharma K S 4ec4ec6971 YARN-8950. Fix compilation issue due to dependency convergence error for hbase.profile=2.0. 2018-10-30 11:30:08 +05:30
Weiwei Yang 78ea897b67 YARN-8915. Update the doc about the default value of maximum-container-assignments for capacity scheduler. Contributed by Zhankun Tang. 2018-10-29 12:04:32 +08:00
Weiwei Yang 1d90a0dd23 YARN-8944. TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896. Contributed by Wilfred Spiegelenburg. 2018-10-29 11:53:10 +08:00
Billie Rinaldi d07e873b7d YARN-8569. Create an interface to provide cluster information to application. Contributed by Eric Yang 2018-10-26 17:57:05 -07:00
Robert Kanter f76e3c3db7 YARN-8930. CGroup-based strict container memory enforcement does not work with CGroupElasticMemoryController (haibochen via rkanter) 2018-10-25 11:09:47 -07:00
Weiwei Yang ddc1e0b727 YARN-8941. Fix findbugs warnings in hadoop-yarn-csi. Contributed by Weiwei Yang. 2018-10-25 07:55:51 +08:00
Eric Yang c16c49b8c3 YARN-8587. Added retries for fetching docker exit code. 
Contributed by Charo Zhang
 2018-10-24 17:29:46 -04:00
Robert Kanter 69b328943e YARN-8929. DefaultOOMHandler should only pick running containers to kill upon oom events (haibochen via rkanter) 2018-10-24 13:15:50 -07:00
Eric Yang bbc6dcd3d0 YARN-8938. Updated YARN service upgrade document. 
Contributed by Chandni Singh
 2018-10-24 11:51:44 -04:00
Haibo Chen 766b78ee07 YARN-8911. ContainerScheduler incorrectly uses percentage number as the cpu resource utlization. 2018-10-24 07:58:26 -07:00
Weiwei Yang 553044818d YARN-8939. Javadoc build fails in hadoop-yarn-csi. Contributed by Weiwei Yang. 2018-10-24 18:02:45 +08:00
Eric Yang a0c0b793c3 YARN-8814. Yarn Service Upgrade: Update the swagger definition. 
Contributed by Chandni Singh
 2018-10-23 19:04:58 -04:00
Haibo Chen 823bb5dda8 MAPREDUCE-4669. MRAM web UI does not work with HTTPS. (Contributed by Robert Kanter) 2018-10-23 15:28:13 -07:00
Jason Lowe 93fb3b4b9c YARN-8904. TestRMDelegationTokens can fail in testRMDTMasterKeyStateOnRollingMasterKey. Contributed by Wilfred Spiegelenburg 2018-10-23 12:49:15 -05:00
Rohith Sharma K S 0b62983c5a YARN-8826. Fix lingering timeline collector after serviceStop in TimelineCollectorManager. Contributed by Prabha Manepalli. 2018-10-23 12:58:41 +05:30
Sunil G 0efddd85f0 YARN-8873. [YARN-8811] Add CSI java-based client library. Contributed by Weiwei Yang. 2018-10-23 11:46:06 +05:30
Eric Yang e98a506227 YARN-8923. Cleanup references to ENV file type in YARN service code. 
Contributed by Suma Shivaprasad
 2018-10-22 18:39:57 -04:00
Eric Yang 292c9e017f YARN-8542. Added YARN service REST API to list containers. 
Contributed by Chandni Singh
 2018-10-22 15:59:52 -04:00
Eric Yang 66e7a2c787 YARN-8922. Fixed test-container-executor test setup and clean up. 
Contributed by Robert Kanter
 2018-10-22 15:28:58 -04:00
Sunil G e50334513c YARN-7502. Nodemanager restart docs should describe nodemanager supervised property. Contributed by Suma Shivaprasad. 2018-10-22 14:15:51 +05:30
Eric Yang e2a9fa8448 HADOOP-15821. Move YARN Registry to Hadoop Registry. 
Contributed by Íñigo Goiri
 2018-10-19 19:46:48 -04:00
Eric Yang 47ad98b2e1 YARN-8910. Fixed misleading log statement when container max retries is infinite. 
Contributed by Chandni Singh
 2018-10-19 13:49:04 -04:00
Wangda Tan 5e02b4915b YARN-8916. Define a constant docker string in ContainerRuntimeConstants.java for better maintainability. (Zhankun Tang via wangda) 
Change-Id: I1349e740037f81afdbe30edbe741f20e88fd0a90
 2018-10-19 09:49:26 -07:00
Wangda Tan 071b9e19aa YARN-8908. Fix errors in yarn-default.xml related to GPU/FPGA. (Zhankun Tang via wangda) 
Change-Id: I46f511a9b51b65b069e15d5b827530110c41edab
 2018-10-19 09:49:26 -07:00
Haibo Chen 285d2c0753 YARN-8449. RM HA for AM web server HTTPS Support. (Contributed by Robert Kanter) 2018-10-18 21:24:36 -07:00
Weiwei Yang 13cc0f50ea YARN-8907. Fix incorrect logging message in TestCapacityScheduler. Contributed by Zhankun Tang. 2018-10-19 09:55:59 +08:00
Eric Yang 66f059ed1d YARN-8618. Added detection for non-upgradable service. 
Contributed by Chandni Singh
 2018-10-18 19:59:11 -04:00
Wangda Tan 780be14f07 YARN-8896. Limit the maximum number of container assignments per heartbeat. (Zhankun Tang via wangda) 
Change-Id: I6e72f8362bd7f5c2a844cb9e3c4732492314e9f1
 2018-10-18 10:58:21 -07:00
Wangda Tan a457a8951a YARN-8456. Fix a configuration handling bug when user leave FPGA discover executable path configuration default but set OpenCL SDK path environment variable. (Zhankun Tang via wangda) 
Change-Id: Iff150ea98ba0c60d448474fd940eb121afce6965
 2018-10-18 10:57:11 -07:00
Wangda Tan bfb88b10f4 YARN-6098. Add documentation for Delete Queue. (Suma Shivaprasad via wangda) 
Change-Id: I23bf7ca0acbe1168a7f805ceccdd62fe41bfeb35
 2018-10-18 10:55:39 -07:00
Eric Yang ba7e81667c YARN-8687. Update YARN service file type in documentation. 
Contributed by Suma Shivaprasad
 2018-10-18 12:02:10 -04:00
Haibo Chen 32fe351bb6 YARN-8864. NM incorrectly logs container user as the user who sent a start/stop container request in its audit log. (Contributed by Wilfred Spiegelenburg) 2018-10-18 08:28:07 -07:00
Sunil G 2202e00ba8 YARN-8868. Set HTTPOnly attribute to Cookie. Contributed by Chandni Singh. 2018-10-18 15:23:20 +05:30
Sunil G 5085e5fa9e YARN-8759. Copy of resource-types.xml is not deleted if test fails, causes other test failures. Contributed by Antal Bálint Steinbach. 2018-10-17 16:05:08 +05:30
Haibo Chen 22f85f2927 YARN-8582. Document YARN support for HTTPS in AM Web server. (Contributed by Robert Kanter) 2018-10-16 17:23:33 -07:00
Eric Yang 3bfd214a59 YARN-8810. Fixed a YARN service bug in comparing ConfigFile object. 
Contributed by Chandni Singh
 2018-10-16 18:54:40 -04:00
Haibo Chen 84e22a6af4 YARN-8842. Expose metrics for custom resource types in QueueMetrics. (Contributed by Szilard Nemeth) 2018-10-16 14:14:30 -07:00
Wangda Tan 538250db26 YARN-8892. YARN UI2 doc changes to update security status (verified under security environment). (Sunil G via wangda) 
Change-Id: I8bc8622936861b8d6de3e42a0b75af86ad8a3961
 2018-10-16 13:51:02 -07:00
Wangda Tan 143d74775b YARN-8798. [Submarine] Job should not be submitted if --input_path option is missing. (Zhankun Tang via wangda) 
Change-Id: I7ae0e44eb5179b04a6ac861ec1c65f3b18c38f0f
 2018-10-16 13:51:02 -07:00
Wangda Tan 46d6e00166 YARN-8870. [Submarine] Add submarine installation scripts. (Xun Liu via wangda) 
Change-Id: I46e8d9fd32c7745c313030da62da41486a77b3ea
 2018-10-16 13:51:02 -07:00
Wangda Tan ed08dd3b0c YARN-8875. [Submarine] Add documentation for submarine installation script details. (Xun Liu via wangda) 
Change-Id: I1c8d39c394e5a30f967ea514919835b951f2c124
 2018-10-16 13:51:01 -07:00
Haibo Chen c2288ac45b YARN-8448. AM HTTPS Support for AM communication with RMWeb proxy. (Contributed by Robert Kanter) 2018-10-16 13:36:26 -07:00
Sunil G 753f149fd3 YARN-8879. Kerberos principal is needed when submitting a submarine job. Contributed by Zac Zhou. 2018-10-16 22:17:51 +05:30
Haibo Chen f880ff418c YARN-8775. TestDiskFailures.testLocalDirsFailures sometimes can fail on concurrent File modifications. (Contributed by Antal Bálint Steinbach) 2018-10-15 09:37:20 -07:00
Sunil G fa94d370b6 YARN-8869. YARN Service Client might not work correctly with RM REST API for Kerberos authentication. Contributed by Eric Yang. 2018-10-15 21:21:57 +05:30
Sunil G 8e5365e277 YARN-8836. Add tags and attributes in resource definition. Contributed by Weiwei Yang. 2018-10-15 15:38:42 +05:30
Vrushali C bca928d3c7 YARN-3879 [Storage implementation] Create HDFS backing storage implementation for ATS reads. Contributed by Abhishek Modi. 2018-10-11 21:14:06 -07:00
Vrushali C a3edfddcf7 YARN-8834 Provide Java client for fetching Yarn specific entities from TimelineReader. Contributed by Abhishek Modi 2018-10-11 20:21:00 -07:00
Vrushali C 8d1981806f YARN-5742 Serve aggregated logs of historical apps from timeline service. Contributed by Rohith Sharma KS 2018-10-11 16:26:07 -07:00
Billie Rinaldi 96d28b4750 YARN-8777. Container Executor C binary change to execute interactive docker command. Contributed by Eric Yang 2018-10-11 09:25:21 -07:00
Billie Rinaldi aeeb0389a5 YARN-8710. Service AM should set a finite limit on NM container max retries. Contributed by Suma Shivaprasad 2018-10-11 08:59:17 -07:00
Jason Lowe e787d65a08 YARN-8861. executorLock is misleading in ContainerLaunch. Contributed by Chandni Singh 2018-10-11 10:54:57 -05:00
Sunil G 8a37983990 YARN-8753. [UI2] Lost nodes representation missing from Nodemanagers Chart. Contributed by Yesha Vora. 2018-10-11 16:15:25 +05:30
Sunil G d3da741fa2 YARN-8666. [UI2] Remove application tab from YARN Queue Page. Contributed by Yesha Vora. 2018-10-11 16:05:53 +05:30
Vrushali C 045069efec YARN-6989 Ensure timeline service v2 codebase gets UGI from HttpServletRequest in a consistent way. Contributed by Abhishek Modi 2018-10-10 15:17:05 -07:00
Jason Lowe 5ce70e1211 YARN-7644. NM gets backed up deleting docker containers. Contributed by Chandni Singh 2018-10-10 09:52:19 -05:00
Weiwei Yang edce866489 YARN-8858. CapacityScheduler should respect maximum node resource when per-queue maximum-allocation is being used. Contributed by Wangda Tan. 2018-10-10 09:32:27 +08:00
Akira Ajisaka 6fa3feb577
HADOOP-15832. Upgrade BouncyCastle to 1.60. Contributed by Robert Kanter. 2018-10-10 10:16:57 +09:00
Eric Yang bf04f19456 YARN-8845. Removed unused hadoop.registry.rm reference. 
Contributed by Íñigo Goiri
 2018-10-09 15:54:45 -04:00
Inigo Goiri c3d22d3b45 YARN-7652. Handle AM register requests asynchronously in FederationInterceptor. Contributed by Botong Huang. 2018-10-09 10:29:40 -07:00
Wangda Tan a23ea68b97 YARN-8852. Add documentation for submarine installation details. (Zac Zhou via wangda) 
Change-Id: If5681d1ef37ff5dc916735eeef15a6120173d653
 2018-10-09 10:18:27 -07:00
Sunil G 6f11919beb YARN-8853. [UI2] Application Attempts tab is not shown correctly when there are no attempts. Contributed by Akhil PB. 2018-10-09 22:04:17 +05:30
Eric Yang 347ea38581 YARN-8763. Added node manager websocket API for accessing containers. 
Contributed by Zian Chen
 2018-10-08 13:56:47 -04:00
Inigo Goiri 745f64012a YARN-8843. updateNodeResource does not support units for memory. Contributed by Manikandan R. 2018-10-08 10:05:18 -07:00
Haibo Chen 7c13872cbb YARN-8659. RMWebServices returns only RUNNING apps when filtered with queue. (Contributed by Szilard Nemeth) 2018-10-08 09:40:37 -07:00
Sunil G 228dc192f1 YARN-7825. [UI2] Maintain constant horizontal application info bar for Application Attempt page. Contributed by Akhil PB. 2018-10-08 11:54:56 +05:30
Vidura Mudalige 907ef6c285
YARN-8788. mvn package -Pyarn-ui fails on JDK9 
This closes #421

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-10-07 16:48:30 +09:00
Vidura Mudalige b85ee0d75d
YARN-8819. Fix findbugs warnings in YarnServiceUtils 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-10-07 00:53:53 +09:00
Jason Lowe 9bb2801e8c YARN-4254. ApplicationAttempt stuck for ever due to UnknownHostException. Contributed by Bibin A Chundatt 2018-10-05 15:52:46 -05:00
Haibo Chen c968365650 YARN-8644. Improve unit test for RMAppImpl.FinalTransition. (Contributed by Szilard Nemeth) 2018-10-05 09:32:30 -07:00
Sunil G 94d82f4162 YARN-8797. [UI2] Improve error pages in new YARN UI. Contributed by Akhil PB. 2018-10-05 14:29:12 +05:30
Sunil G 751f626e50 YARN-7957. [UI2] YARN service delete option disappears after stopping application. Contributed by Akhil PB. 2018-10-05 12:28:09 +05:30
Haibo Chen e60b797c88 YARN-8750. Refactor TestQueueMetrics. (Contributed by Szilard Nemeth) 2018-10-04 13:00:31 -07:00
Haibo Chen b6d5d84e07 YARN-8732. Add unit tests of min/max allocation for custom resource types in FairScheduler. (Contributed by Szilard Nemeth) 2018-10-04 12:47:31 -07:00
Wangda Tan 2e9913caf2 YARN-8844. TestNMProxy unit test is failing. (Eric Yang via wangda) 
Change-Id: I241fa8701b6f1dbcad87fd2e9a429e32e7aa40f5
(cherry picked from commit b3ac886933)
 2018-10-04 10:53:39 -07:00
Wangda Tan 6926fd0ec6 YARN-8758. Support getting PreemptionMessage when using AMRMClientAsyn. (Zian Chen via wangda) 
Change-Id: Ibf5d165f49957b582eeadeb41dc285c84d2f05e7
 2018-10-04 10:31:33 -07:00
Akira Ajisaka 39b35036ba
YARN-8840. Add missing cleanupSSLConfig() call for TestTimelineClient test. Contributed by Aki Tanaka. 2018-10-04 09:51:42 +09:00
Sunil G 58fa96b697 Changed version in trunk to 3.3.0-SNAPSHOT. 2018-10-02 22:41:41 +05:30
Shane Kumpf 5edb9d3b97 YARN-8785. Improve the error message when a bind mount is not whitelisted. Contributed by Simon Prewo 2018-10-02 07:16:29 -06:00
Haibo Chen d0ee6fbe28 YARN-8621. Add test coverage of custom Resource Types for the apps/<appId> REST API endpoint. (Contributed by Szilard Nemeth) 2018-10-01 14:46:42 -07:00
Giovanni Matteo Fumarola 59d5af21b7 YARN-8760. [AMRMProxy] Fix concurrent re-register due to YarnRM failover in AMRMClientRelayer. Contributed by Botong Huang. 2018-10-01 13:12:38 -07:00
Weiwei Yang fd6be5898a YARN-8468. Enable the use of queue based maximum container allocation limit and implement it in FairScheduler. Contributed by Antal Bálint Steinbach. 2018-09-29 17:47:12 +08:00
Sunil G 19ad5be651 YARN-8800. Updated documentation of Submarine with latest examples. Contributed by Wangda Tan. 2018-09-29 00:01:04 +05:30
Billie Rinaldi d1c1dde309 YARN-8734. Readiness check for remote service belonging to the same user. Contributed by Eric Yang 2018-09-28 09:40:48 -07:00
Eric E Payne 8598b498bc YARN-8774. Memory leak when CapacityScheduler allocates from reserved container with non-default label. Contributed by Tao Yang. 2018-09-28 15:32:07 +00:00
bibinchundatt 7093afd874 YARN-8829. Cluster metrics can fail with IndexOutOfBoundsException. Contributed by Akshay Agarwal. 2018-09-28 12:35:33 +05:30
Vrushali C 90e2e493b3 YARN-8270 Adding JMX Metrics for Timeline Collector and Reader. Contributed by Sushil Ks. 2018-09-27 15:53:39 -07:00
Eric Yang b237a0dd44 YARN-6456. Added config to set default container runtimes. 
Contributed by Craig Condit
 2018-09-27 15:31:18 -04:00
Jason Lowe 6b988d821e YARN-8804. resourceLimits may be wrongly calculated when leaf-queue is blocked in cluster with 3+ level queues. Contributed by Tao Yang 2018-09-26 14:43:00 -07:00
Eric Yang 913f87dada YARN-8665. Added Yarn service cancel upgrade option. 
Contributed by Chandni Singh
 2018-09-26 14:51:35 -04:00
Rohith Sharma K S e5287a4fe0 YARN-8824. App Nodelabel missed after RM restart for finished apps. Contributed by Bibin A Chundatt. 2018-09-26 12:30:26 +05:30
Akira Ajisaka 44edcdfd6a
YARN-8745. Misplaced the TestRMWebServicesFairScheduler.java file. Contributed by Y. SREENIVASULU REDDY. 2018-09-26 10:09:11 +09:00
Akira Ajisaka 94b5c54e8a
YARN-8752. yarn-registry.md has wrong word ong-lived, it should be long-lived. Contributed by leiqiang. 2018-09-26 09:42:53 +09:00
Shane Kumpf 26c94a0fd0 YARN-8623. Update Docker examples to use image which exists. Contributed by Craig Condit 2018-09-25 06:38:25 -06:00
Rohith Sharma K S 50bc7746d7 YARN-8815. RM fails to recover finished unmanaged AM. Contributed by Bibin A Chundatt. 2018-09-25 11:31:14 +05:30
Sunil G d0bc6a27fd YARN-8817. [Submarine] In cases when user doesn't ask HDFS path while submitting job but framework requires user to set HDFS related environments. Contributed by Wangda Tan. 2018-09-25 11:23:34 +05:30
Haibo Chen 29dad7d258 YARN-8616. systemClock should be used in RMAppImpl instead of System.currentTimeMills(), to be consistent. (Contributed by Szilard Nemeth) 2018-09-24 16:04:28 -07:00
Haibo Chen cf62ff9a6a YARN-6338. Typos in Docker docs: contains => containers. (Contributed by Zoltan Siegl) 2018-09-24 15:54:02 -07:00
Giovanni Matteo Fumarola 3090922805 YARN-8696. [AMRMProxy] FederationInterceptor upgrade: home sub-cluster heartbeat async. Contributed by Botong Huang. 2018-09-24 11:37:05 -07:00
Sunil G 32a35dc94b YARN-8742. [UI2] Container logs on Application / Service pages on UI2 are not available many case, improve error messages in such cases. Contributed by Akhil PB. 2018-09-24 12:20:28 +05:30
Sunil G 0cd6346102 YARN-8769. [Submarine] Allow user to specify customized quicklink(s) when submit Submarine job. Contributed by Wangda Tan. 2018-09-21 23:39:22 +05:30
Sunil G a2752779ac YARN-8628. [UI2] Few duplicated or inconsistent information displayed in UI2. Contributed by Akhil PB. 2018-09-21 15:47:10 +05:30
Eric Yang aa4bd493c3 YARN-8801. Fixed header comments for docker utility functions. 
Contributed by Zian Chen
 2018-09-20 13:08:59 -04:00
Jason Lowe 6b5838ed32 YARN-8784. DockerLinuxContainerRuntime prevents access to distributed cache entries on a full disk. Contributed by Eric Badger 2018-09-19 16:44:51 -05:00
Eric Yang efdea85ad1 YARN-8791. Trim docker inspect output for line feed for STOPSIGNAL parsing. 
Contributed by Chandni Singh
 2018-09-19 13:16:11 -04:00
Sunil G 1824d5d1c4 YARN-8757. [Submarine] Add Tensorboard component when --tensorboard is specified. Contributed by Wangda Tan. 2018-09-19 22:18:55 +05:30
Weiwei Yang 0712537e79 YARN-8771. CapacityScheduler fails to unreserve when cluster resource contains empty resource type. Contributed by Tao Yang. 2018-09-19 19:31:07 +08:00
Jason Lowe 2df0a8dcb3 YARN-8648. Container cgroups are leaked when using docker. Contributed by Jim Brennan 2018-09-18 15:36:45 -05:00
Weiwei Yang 78a0d173e4 YARN-8787. Fix broken list items in PlacementConstraints documentation. Contributed by Masahiro Tanaka. 2018-09-18 17:19:39 +08:00
Sunil G bbeca0107e YARN-8652. [UI2] YARN UI2 breaks if getUserInfo REST API is not available in older versions. Contributed by Akhil PB. 2018-09-18 12:44:21 +05:30
Sunil G 0cc6e03945 YARN-8726. [UI2] YARN UI2 is not accessible when config.env file failed to load. Contributed by Akhil PB. 2018-09-18 12:31:55 +05:30
Weiwei Yang 9a265fa673 YARN-8782. Fix exception message in Resource.throwExceptionWhenArrayOutOfBound. Contributed by Gergely Pollak. 2018-09-17 22:15:24 +08:00
Sunil G 33d8327cff YARN-8715. Make allocation tags in the placement spec optional for node-attributes. Contributed by Weiwei Yang. 2018-09-17 10:07:45 +05:30
Shane Kumpf 144a55f0e3 YARN-8045. Reduce log output from container status calls. Contributed by Craig Condit 2018-09-14 10:41:55 -06:00
Shane Kumpf 78902f0250 YARN-8748. Javadoc warnings within the nodemanager package. Contributed by Craig Condit 2018-09-14 10:28:36 -06:00
Eric Yang 99237607bf YARN-8706. Allow additional flag in docker inspect call. 
Contributed by Chandni Singh
 2018-09-14 11:46:59 -04:00
Weiwei Yang f1a893fdbc YARN-8720. CapacityScheduler does not enforce max resource allocation check at queue level. Contributed by Tarun Parimi. 2018-09-14 16:33:51 +08:00
Andrew Purtell 568ebecdf4
YARN-8772. Annotation javax.annotation.Generated has moved 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-09-14 15:02:52 +09:00
Jason Lowe 250b50018e YARN-8680. YARN NM: Implement Iterable Abstraction for LocalResourceTracker state. Contributed by Pradeep Ambati 2018-09-13 13:28:54 -05:00
Weiwei Yang 39c1ea1ed4 YARN-8729. Node status updater thread could be lost after it is restarted. Contributed by Tao Yang. 2018-09-13 22:21:35 +08:00
Sunil G f4bda5e8e9 YARN-8630. ATSv2 REST APIs should honor filter-entity-list-by-user in non-secure cluster when ACls are enabled. Contributed by Rohith Sharma K S. 2018-09-13 17:47:21 +05:30
Shane Kumpf 8e9afbfb66 YARN-8768. Javadoc error in node attributes. Contributed by Sunil Govindan. 2018-09-12 15:12:28 -06:00
Giovanni Matteo Fumarola 02b9bfdf9e YARN-8658. [AMRMProxy] Metrics for AMRMClientRelayer inside FederationInterceptor. Contributed by Young Chen. 2018-09-12 11:46:35 -07:00
Sunil G 5e64e62dee YARN-8740. Clear node attribute path after each test run. Contributed by Bibin A Chundatt. 2018-09-12 16:01:01 +05:30
bibinchundatt c44088ac19 YARN-8739. Fix jenkins issues for Node Attributes branch. Contributed by Sunil Govindan. 2018-09-12 16:01:01 +05:30
Weiwei Yang 1e7d6e55a5 YARN-7865. Node attributes documentation. Contributed by Naganarasimha G R. 2018-09-12 16:01:01 +05:30
Weiwei Yang 52194351e7 YARN-8721. Relax NE node-attribute check when attribute doesn't exist on a node. Contributed by Sunil Govindan. 2018-09-12 16:01:01 +05:30
Naganarasimha 67ae81f0e0 YARN-7863. Modify placement constraints to support node attributes. Contributed by Sunil Govindan. 2018-09-12 16:01:01 +05:30
Naganarasimha 8c947398ab YARN-8574. Allow dot in attribute values. Contributed by Bibin A Chundatt. 2018-09-12 16:01:01 +05:30
Naganarasimha eb08543c7a YARN-8103. Add CLI interface to query node attributes. Contributed by Bibin A Chundatt. 2018-09-12 16:01:01 +05:30
Sunil G 76183428b7 YARN-8351. Node attribute manager logs are flooding RM logs. Contributed by Weiwei Yang. 2018-09-12 16:01:00 +05:30
bibinchundatt 8cf6a9a2bd YARN-7892. Revisit NodeAttribute class structure. Contributed by Naganarasimha G R. 2018-09-12 16:01:00 +05:30
Naganarasimha 5dc7d6e0f3 YARN-8104. Add API to fetch node to attribute mapping. Contributed by Bibin A Chundatt. 2018-09-12 16:01:00 +05:30
Naganarasimha 0a01b1350d YARN-8100. Support API interface to query cluster attributes and attribute to nodes. Contributed by Bibin A Chundatt. 2018-09-12 16:01:00 +05:30
Sunil G b9890d1f66 YARN-7875. Node Attribute store for storing and recovering attributes. Contributed by Bibin A Chundatt. 2018-09-12 16:01:00 +05:30
bibinchundatt a6590c1f1f YARN-8117. Fix TestRMWebServicesNodes test failure. Contributed by Bibin A Chundatt. 2018-09-12 16:01:00 +05:30
bibinchundatt 901e85238d YARN-8033. CLI Integration with NodeAttributesManagerImpl. Contributed by Naganarasimha G R. 2018-09-12 16:01:00 +05:30
Sunil G 89b3ebd11e YARN-8092. Expose Node Attributes info via RM nodes REST API. Contributed by Weiwei Yang. 2018-09-12 16:01:00 +05:30
Sunil G 440ff7f563 YARN-8094. Support configuration based Node Attribute provider. Contributed by Weiwei Yang. 2018-09-12 16:01:00 +05:30
Sunil G 6f4bc49c6d YARN-7988. Refactor FSNodeLabelStore code for Node Attributes store support. Contributed by Bibin A Chundatt. 2018-09-12 16:01:00 +05:30
Naganarasimha 3b3b6efe21 YARN-7871. Node attributes reporting from NM to RM. Contributed by Weiwei Yang. 2018-09-12 16:01:00 +05:30
Naganarasimha 86d024ef2a YARN-7965. NodeAttributeManager add/get API is not working properly. Contributed by Weiwei Yang. 2018-09-12 16:01:00 +05:30
Sunil G ffcabd24c3 YARN-7856. Validate Node Attributes from NM. Contributed by Weiwei Yang. 2018-09-12 16:01:00 +05:30
Sunil G 2f7712be09 YARN-6858. Attribute Manager to store and provide node attributes in RM. Contributed by Naganarasimha G R. 2018-09-12 16:01:00 +05:30
Naganarasimha d312b5cf9f YARN-7757. Refactor NodeLabelsProvider to be more generic and reusable for node attributes providers. Contributed by Weiwei Yang. 2018-09-12 16:01:00 +05:30
bibinchundatt 4458b2772f YARN-7840. Update PB for prefix support of node attributes. Contributed by Naganarasimha G R. 2018-09-12 16:01:00 +05:30
Weiwei Yang d9d93e3925 YARN-7842. PB changes to carry node-attributes in NM heartbeat. Contributed by Weiwei Yang. 2018-09-12 16:00:59 +05:30
Naganarasimha 2475fb0a1e YARN-6856. [YARN-3409] Support CLI for Node Attributes Mapping. Contributed by Naganarasimha G R. 2018-09-12 16:00:59 +05:30
Naganarasimha 1f42ce907a YARN-6855. [YARN-3409] CLI Proto Modifications to support Node Attributes. Contributed by Naganarasimha G R. 2018-09-12 16:00:59 +05:30
Eric E Payne 987d8191ad YARN-8709: CS preemption monitor always fails since one under-served queue was deleted. Contributed by Tao Yang. 2018-09-10 19:55:20 +00:00
Wangda Tan de85f841da YARN-8698. [Submarine] Failed to reset Hadoop home environment when submitting a submarine job. (Zac Zhou via wangda) 
Change-Id: If7fdc11fcc4e49bed2186ded76cd69164f34f502
 2018-09-09 17:45:31 -07:00
Eric Yang bf8a1750e9 YARN-8706. Updated docker container stop logic to avoid double kill. 
Contributed by Chandni Singh
 2018-09-07 20:18:09 -04:00
Eric Yang 7d62334387 YARN-8751. Reduce conditions that mark node manager as unhealthy. 
Contributed by Craig Condit
 2018-09-07 19:46:15 -04:00
Giovanni Matteo Fumarola 16333782c1 HADOOP-15731. TestDistributedShell fails on Windows. Contributed by Botong Huang. 2018-09-07 14:19:10 -07:00
Giovanni Matteo Fumarola 3dc2988a37 YARN-8699. Add Yarnclient#yarnclusterMetrics API implementation in router. Contributed by Bibin A Chundatt. 2018-09-07 11:32:03 -07:00
Giovanni Matteo Fumarola 9af96d4ed4 HADOOP-15707. Add IsActiveServlet to be used for Load Balancers. Contributed by Lukas Majercak. 2018-09-05 10:50:25 -07:00
Shane Kumpf dffb7bfe6c YARN-8638. Allow linux container runtimes to be pluggable. Contributed by Craig Condit 2018-09-05 06:47:54 -06:00
bibinchundatt eed8415dc1 YARN-8535. Fix DistributedShell unit tests. Contributed by Abhishek Modi. 2018-09-02 13:35:52 +05:30
Shane Kumpf 73625168c0 YARN-8642. Add support for tmpfs mounts with the Docker runtime. Contributed by Craig Condit 2018-08-29 07:08:37 -06:00
Weiwei Yang 3fa4639421 YARN-8723. Fix a typo in CS init error message when resource calculator is not correctly set. Contributed by Abhishek Modi. 2018-08-29 11:13:44 +08:00
Giovanni Matteo Fumarola 7ed458b255 YARN-8697. LocalityMulticastAMRMProxyPolicy should fallback to random sub-cluster when cannot resolve resource. Contributed by Botong Huang. 2018-08-28 16:01:35 -07:00
Eric Yang fd089caf69 YARN-8488. Added SUCCEEDED/FAILED states to YARN service. 
Contributed by Suma Shivaprasad
 2018-08-28 13:55:28 -04:00
Giovanni Matteo Fumarola 602d13844a HADOOP-15699. Fix some of testContainerManager failures in Windows. Contributed by Botong Huang. 2018-08-27 12:25:46 -07:00
Billie Rinaldi 05b2bbeb35 YARN-8675. Remove default hostname for docker containers when net=host. Contributed by Suma Shivaprasad 2018-08-27 11:34:33 -07:00
Giovanni Matteo Fumarola f152582562 YARN-8705. Refactor the UAM heartbeat thread in preparation for YARN-8696. Contributed by Botong Huang. 2018-08-27 10:32:22 -07:00
Weiwei Yang e8b063f630 YARN-8719. Typo correction for yarn configuration in OpportunisticContainers(federation) docs. Contributed by Y. SREENIVASULU REDDY. 2018-08-28 01:03:03 +08:00
Jason Lowe 585ebd873a YARN-8649. NPE in localizer hearbeat processing if a container is killed while localizing. Contributed by lujie 2018-08-23 09:29:46 -05:00
Sunil G 1ac01444a2 YARN-8015. Support all types of placement constraint support for Capacity Scheduler. Contributed by Weiwei Yang. 2018-08-23 10:05:43 +05:30
Eric Yang e557c6bd8d YARN-8298. Added express upgrade for YARN service. 
Contributed by Chandni Singh
 2018-08-21 19:49:26 -04:00
Weiwei Yang 9c3fc3ef28 YARN-7494. Add muti-node lookup mechanism and pluggable nodes sorting policies to optimize placement decision. Contributed by Sunil Govindan. 2018-08-21 22:42:28 +08:00
Weiwei Yang 54d0bf8935 YARN-8683. Support to display pending scheduling requests in RM app attempt page. Contributed by Tao Yang. 2018-08-21 19:00:31 +08:00
Rohith Sharma K S d3fef7a5c5 YARN-8129. Improve error message for invalid value in fields attribute. Contributed by Abhishek Modi. 2018-08-21 11:58:07 +05:30
Giovanni Matteo Fumarola e0f6ffdbad YARN-8581. [AMRMProxy] Add sub-cluster timeout in LocalityMulticastAMRMProxyPolicy. Contributed by Botong Huang. 2018-08-20 14:33:16 -07:00
Giovanni Matteo Fumarola 8736fc39ac YARN-8673. [AMRMProxy] More robust responseId resync after an YarnRM master slave switch. Contributed by Botong Huang. 2018-08-20 12:22:36 -07:00
Jason Lowe 65e7469712 YARN-8242. YARN NM: OOM error while reading back the state store on recovery. Contributed by Pradeep Ambati and Kanwaljeet Sachdev 2018-08-20 10:14:40 -05:00
Rohith Sharma K S 4aacbfff60 YARN-8679. [ATSv2] If HBase cluster is down for long time, high chances that NM ContainerManager dispatcher get blocked. Contributed by Wangda Tan. 2018-08-18 10:26:55 +05:30
Rohith Sharma K S 1697a02306 YARN-8612. Fix NM Collector Service Port issue in YarnConfiguration. Contributed by Prabha Manepalli. 2018-08-17 11:12:10 +05:30
Eric Yang d42806160e YARN-8667. Cleanup symlinks when container restarted by NM. 
Contributed by Chandni Singh
 2018-08-16 18:41:58 -04:00
Eric Yang 8990eaf592 YARN-8474. Fixed ApiServiceClient kerberos negotiation. 
Contributed by Billie Rinaldi
 2018-08-16 12:46:37 -04:00
Jason Lowe cb21eaa026 YARN-8656. container-executor should not write cgroup tasks files for docker containers. Contributed by Jim Brennan 2018-08-16 10:06:17 -05:00
Weiwei Yang 8dc07b4723 YARN-8614. Fix few annotation typos in YarnConfiguration. Contributed by Sen Zhao. 2018-08-15 12:15:54 +08:00
Wei-Chiu Chuang bdd0e01ed9 HADOOP-14212. Addendum patch: Expose SecurityEnabled boolean field in JMX for other services besides NameNode. Contributed by Adam Antal. 2018-08-14 18:25:38 -07:00
Wei-Chiu Chuang 363bd16e31 HADOOP-14212. Expose SecurityEnabled boolean field in JMX for other services besides NameNode. Contributed by Adam Antal. 2018-08-14 17:19:00 -07:00
Eric Yang 4cba0741d5 YARN-8160. Support upgrade of service that use docker containers. 
Contributed by Chandni Singh
 2018-08-14 11:36:26 -04:00
Jason Lowe d1d129aa9d YARN-8640. Restore previous state in container-executor after failure. Contributed by Jim Brennan 2018-08-14 10:21:03 -05:00
Eric Yang 74411ce0ce YARN-7417. Remove duplicated code from IndexedFileAggregatedLogsBlock 
and TFileAggregatedLogsBlock.
           Contributed by Zian Chen
 2018-08-13 16:50:00 -04:00
Sunil G cadbc8b57f YARN-8561. [Submarine] Initial implementation: Training job submission and job history retrieval. Contributed by Wangda Tan. 2018-08-13 14:22:55 +05:30
Shane Kumpf e7951c69cb YARN-8520. Document best practice for user management. Contributed by Eric Yang 2018-08-10 14:32:03 -06:00
Weiwei Yang 0a71bf1452 YARN-8575. Avoid committing allocation proposal to unavailable nodes in async scheduling. Contributed by Tao Yang. 2018-08-10 14:37:45 +08:00
Weiwei Yang 08d5060605 YARN-8521. NPE in AllocationTagsManager when a container is removed more than once. Contributed by Weiwei Yang. 2018-08-10 08:32:02 +08:00
Robert Kanter b2517dd66b YARN-4946. RM should not consider an application as COMPLETED when log aggregation is not in a terminal state (snemeth via rkanter) 2018-08-09 14:58:04 -07:00
Wangda Tan 8244abb7ae YARN-8136. Add version attribute to site doc examples and quickstart. (Eric Yang via wangda) 
Change-Id: I4541b239f490ca0a6edf9698e0d3deaf83669151
 2018-08-09 11:04:02 -07:00
Wangda Tan 344c335a92 YARN-8588. Logging improvements for better debuggability. (Suma Shivaprasad via wangda) 
Change-Id: I66aa4b0ec031ae5ce0fae558e2f8cbcbbfebc442
 2018-08-09 11:04:02 -07:00
Weiwei Yang d352f167eb YARN-8559. Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint. Contributed by Weiwei Yang. 2018-08-09 23:46:53 +08:00
Jason Lowe cd04e954d2 YARN-8331. Race condition in NM container launched after done. Contributed by Pradeep Ambati 2018-08-09 10:17:34 -05:00
Sunil G 00013d6ef7 YARN-8633. Update DataTables version in yarn-common in line with JQuery 3 upgrade. Contributed by Akhil PB. 2018-08-09 12:18:32 +05:30
Robert Kanter 8478732bb2 YARN-8568. Replace the deprecated zk-address property in the HA config example in ResourceManagerHA.md (bsteinbach via rkanter) 2018-08-08 15:08:55 -07:00
Wangda Tan 3214cd75ac Make 3.1.1 awared by other branches - adding missing files 2018-08-08 13:05:24 -07:00
Wangda Tan 49c687608b Make 3.1.1 awared by other branches 2018-08-08 13:02:12 -07:00
bibinchundatt ff06bd1be8 YARN-8601. Print ExecutionType in Container report CLI. Contributed by Bilwa S T. 2018-08-08 22:42:52 +05:30
Sunil G 5b898c176f Revert "YARN-8633. Update DataTables version in yarn-common in line with JQuery 3 upgrade. Contributed by Akhil PB." 
This reverts commit 64901abdfa.
 2018-08-08 19:48:49 +05:30
Sunil G 64901abdfa YARN-8633. Update DataTables version in yarn-common in line with JQuery 3 upgrade. Contributed by Akhil PB. 2018-08-08 19:43:29 +05:30
Giovanni Matteo Fumarola d838179d8d YARN-8626. Create HomePolicyManager that sends all the requests to the home subcluster. Contributed by Inigo Goiri. 2018-08-07 15:33:16 -07:00
Wangda Tan 861095f761 YARN-8407. Container launch exception in AM log should be printed in ERROR level. (Yesha Vora via wangda) 
Change-Id: I154e873df1df3503a09c41d6b3874ca195af91d9
 2018-08-07 13:01:13 -07:00
Wangda Tan c0599151bb YARN-7089. Mark the log-aggregation-controller APIs as public. (Zian Chen via wangda) 
Change-Id: I37851bdc5935d623a27d0973a206c997258716eb
 2018-08-07 12:37:32 -07:00
Wangda Tan d4258fcad7 YARN-8629. Container cleanup fails while trying to delete Cgroups. (Suma Shivaprasad via wangda) 
Change-Id: I392ef4f8baa84d5d7b1f2e438c560b5426b6d4f2
 2018-08-07 12:36:55 -07:00
Eric Yang ca20e0d7e9 YARN-8624. Updated verbiage around entry point support. 
Contributed by Craig Condit
 2018-08-06 12:37:59 -04:00
Shane Kumpf bcfc9851f4 YARN-8287. Update documentation and yarn-default related to the Docker runtime. Contributed by Craig Condit 2018-08-03 18:20:49 -06:00
Sunil G 3426f4062a YARN-8615. [UI2] Resource Usage tab shows only memory related info. No info available for vcores/gpu. Contributed by Akhil PB. 2018-08-03 19:28:16 +05:30
Sunil G 29417dbbf4 YARN-8620. [UI2] YARN Services UI new submission failures are not debuggable. Contributed by Akhil PB. 2018-08-03 19:12:34 +05:30
Rohith Sharma K S 022592ae79 YARN-8608. [UI2] No information available per application appAttempt about 'Total Outstanding Resource Requests'. Contributed by Akhil PB. 2018-08-03 15:30:39 +05:30
Rohith Sharma K S 33482d35e8 YARN-8603. [UI2] Latest run application should be listed first in the RM UI. Contributed by Akhil PB. 2018-08-03 15:30:39 +05:30
Haibo Chen 889df6f194 YARN-7948. Enable fair scheduler to refresh maximum allocation for multiple resource types. (Szilard Nemeth via Haibo Chen) 2018-08-02 10:10:12 -07:00
Sunil G 12a095a496 YARN-7159. Normalize unit of resource objects in RM to avoid unit conversion in critical path. Contributed by Manikandan R. 2018-08-02 22:29:21 +05:30
Jason Lowe 7526815e32 YARN-8263. DockerClient still touches hadoop.tmp.dir. Contributed by Craig Condit 2018-08-02 10:43:48 -05:00
Sunil G 5033d7da8f YARN-8318. [UI2] IP address in component page shows N/A. Contributed by Yesha Vora. 2018-08-02 20:09:24 +05:30
Sunil G 97870ec1f6 YARN-8592. [UI2] rmip:port/ui2 endpoint shows a blank page in windows OS and Chrome browser. Contributed by Akhil PB. 2018-08-02 16:10:54 +05:30
Sunil G 1ea81169ba YARN-8594. [UI2] Display current logged in user. Contributed by Akhil PB. 2018-08-02 12:41:06 +05:30
Sunil G 735b492556 YARN-8593. Add RM web service endpoint to get user information. Contributed by Akhil PB. 2018-08-02 08:34:09 +05:30
Eric Yang 23f394240e YARN-8610. Fixed initiate upgrade error message. 
Contributed by Chandni Singh
 2018-08-01 20:41:43 -04:00
Shane Kumpf 603a57476c YARN-8600. RegistryDNS hang when remote lookup does not reply. Contributed by Eric Yang 2018-08-01 12:22:01 -06:00
Billie Rinaldi 67c65da261 YARN-8403. Change the log level for fail to download resource from INFO to ERROR. Contributed by Eric Yang 2018-08-01 08:51:40 -07:00
Sunil G d920b9db77 YARN-8595. [UI2] Container diagnostic information is missing from container page. Contributed by Akhil PB. 2018-08-01 14:27:54 +05:30
Sunil G a48a0cc7fd YARN-8606. Opportunistic scheduling does not work post RM failover. Contributed by Bibin A Chundatt. 2018-08-01 12:17:18 +05:30
Wangda Tan 5cc8e99147 YARN-8522. Application fails with InvalidResourceRequestException. (Zian Chen via wangda) 
Change-Id: I34dd7fa49bd4d10580c4a78051033b1068d28f1e
 2018-07-31 22:34:53 -07:00
Sunil G 6310c0d17d YARN-8397. Potential thread leak in ActivitiesManager. Contributed by Rohith Sharma K S. 2018-08-01 08:33:30 +05:30
Eric Yang c7ebcd76bf YARN-8579. Recover NMToken of previous attempted component data. 
Contributed by Gour Saha
 2018-07-31 18:01:02 -04:00
Wangda Tan 4b540bbfcf YARN-8418. App local logs could leaked if log aggregation fails to initialize for the app. (Bibin A Chundatt via wangda) 
Change-Id: I29a23ca4b219b48c92e7975cd44cddb8b0e04104
 2018-07-31 12:08:00 -07:00
Haibo Chen 8aa93a575e YARN-8605. TestDominantResourceFairnessPolicy.testModWhileSorting is flaky. (Wilfred Spiegelenburg via Haibo Chen) 2018-07-31 11:32:40 -07:00
Inigo Goiri 9fea5c9ee7 YARN-8175. Add support for Node Labels in SLS. Contributed by Abhishek Modi. 2018-07-31 09:36:34 -07:00
Jonathan Hung 3e06a5dcea YARN-7974. Allow updating application tracking url after registration. Contributed by Jonathan Hung 2018-07-30 17:44:18 -07:00
bibinchundatt 2b39ad2698 YARN-8584. Several typos in Log Aggregation related classes. Contributed by Szilard Nemeth. 2018-07-30 23:25:19 +05:30
Sunil G 63e08ec071 YARN-8591. [ATSv2] NPE while checking for entity acl in non-secure cluster. Contributed by Rohith Sharma K S. 2018-07-30 14:48:04 +05:30
bibinchundatt 3d586841ab YARN-8558. NM recovery level db not cleaned up properly on container finish. Contributed by Bibin A Chundatt. 2018-07-28 20:52:39 +05:30
Eric Yang ed9d60e888 YARN-8508. Release GPU resource for killed container. 
Contributed by Chandni Singh
 2018-07-27 19:33:58 -04:00
Inigo Goiri 79091cf76f YARN-8596. Allow SQLFederationStateStore to submit the same app in the same subcluster. Contributed by Giovanni Matteo Fumarola. 2018-07-27 15:23:57 -07:00
Robert Kanter 2cccf4061c YARN-8517. getContainer and getContainers ResourceManager REST API methods are not documented (snemeth via rkanter) 2018-07-27 14:35:03 -07:00
Robert Kanter fecbac499e YARN-8566. Add diagnostic message for unschedulable containers (snemeth via rkanter) 2018-07-27 14:32:34 -07:00
Billie Rinaldi b429f19d32 YARN-8571. Validate service principal format prior to launching yarn service. Contributed by Eric Yang 2018-07-27 11:30:19 -07:00
Eric Yang 8d3c068e59 YARN-8429. Improve diagnostic message when artifact is not set properly. 
Contributed by Gour Saha
 2018-07-26 20:02:13 -04:00
Eric Yang 40fad32824 YARN-8545. Return allocated resource to RM for failed container. 
Contributed by Chandni Singh
 2018-07-26 18:22:57 -04:00
Eric Yang f93ecf5c1e YARN-8330. Improved publishing ALLOCATED events to ATS. 
Contributed by Suma Shivaprasad
 2018-07-25 18:49:30 -04:00
Eric E Payne 9485c9aee6 YARN-4606. CapacityScheduler: applications could get starved because computation of #activeUsers considers pending apps. Contributed by Manikandan R 2018-07-25 16:22:04 +00:00
Weiwei Yang 5be9f4a5d0 YARN-8546. Resource leak caused by a reserved container being released more than once under async scheduling. Contributed by Tao Yang. 2018-07-25 17:35:27 +08:00
Haibo Chen ea2c6c8c9a YARN-7133. Clean up lock-try order in fair scheduler. (Szilard Nemeth via Haibo Chen) 2018-07-24 12:46:59 -07:00
bibinchundatt e673dd1d4d YARN-8541. RM startup failure on recovery after user deletion. Contributed by Bibin A Chundatt. 2018-07-24 23:26:59 +05:30
Sunil G 35ce6eb1f5 YARN-7748. TestContainerResizing.testIncreaseContainerUnreservedWhenApplicationCompleted fails due to multiple container fail events. Contributed by Weiwei Yang. 2018-07-24 22:20:17 +05:30
bibinchundatt ff7c2eda34 YARN-8548. AllocationRespose proto setNMToken initBuilder not done. Contributed by Bilwa S T. 2018-07-24 16:17:20 +05:30
bibinchundatt 8461278833 YARN-8544. [DS] AM registration fails when hadoop authorization is enabled. Contributed by Bibin A Chundatt. 2018-07-24 13:09:17 +05:30
Eric Yang 8688a0c7f8 YARN-8380. Support bind propagation options for mounts in docker runtime. 
Contributed by Billie Rinaldi
 2018-07-23 20:12:04 -04:00
Haibo Chen 9d3c39e9dd YARN-6966. NodeManager metrics may return wrong negative values when NM restart. (Szilard Nemeth via Haibo Chen) 2018-07-23 11:07:24 -07:00
Eric Yang 84d7bf1eef YARN-8360. Improve YARN service restart policy and node manager auto restart policy. 
Contributed by Suma Shivaprasad
 2018-07-23 12:57:01 -04:00
Eric Yang 1622a4b810 YARN-8301. Added YARN service upgrade instructions. 
Contributed by Chandni Singh
 2018-07-20 19:46:35 -04:00
Haibo Chen 8a6bb8409c YARN-6964. Fair scheduler misuses Resources operations. (Daniel Templeton and Szilard Nemeth via Haibo Chen) 2018-07-20 10:46:28 -07:00
Weiwei Yang cbf2026483 YARN-8528. Final states in ContainerAllocation might be modified externally causing unexpected allocation results. Contributed by Xintong Song. 2018-07-20 22:34:06 +08:00
Haibo Chen e6873dfde0 YARN-7300. DiskValidator is not used in LocalDirAllocator. (Szilard Nemeth via Haibo Chen) 2018-07-19 16:27:11 -07:00
Haibo Chen f354f47f99 YARN-6995. Improve use of ResourceNotFoundException in resource types code. (Daniel Templeton and Szilard Nemeth via Haibo Chen) 2018-07-19 15:35:05 -07:00
Haibo Chen 2564884757 YARN-8436. FSParentQueue: Comparison method violates its general contract. (Wilfred Spiegelenburg via Haibo Chen) 2018-07-19 13:22:31 -07:00
Eric Yang 5836e0a46b YARN-8501. Reduce complexity of RMWebServices getApps method. 
Contributed by Szilard Nemeth
 2018-07-19 12:30:38 -04:00
Eric Yang 121865c3f9 YARN-8299. Added CLI and REST API for query container status. 
Contributed by Chandni Singh
 2018-07-16 17:41:23 -04:00
Eric Yang efb4e274e5 YARN-8538. Fixed memory leaks in container-executor and test cases. 
Contributed by Billie Rinaldi
 2018-07-16 17:38:49 -04:00
Wangda Tan 238ffff999 YARN-8524. Single parameter Resource / LightWeightResource constructor looks confusing. (Szilard Nemeth via wangda) 
Change-Id: I4ae97548b5b8d76a6bcebb2d3d70bf8e0be3c125
 2018-07-16 10:58:00 -07:00
Wangda Tan a2e49f41a8 YARN-8361. Change App Name Placement Rule to use App Name instead of App Id for configuration. (Zian Chen via wangda) 
Change-Id: I17e5021f8f611a9c5e3bd4b38f25e08585afc6b1
 2018-07-16 10:57:37 -07:00
Wangda Tan 752dcce5f4 YARN-8511. When AM releases a container, RM removes allocation tags before it is released by NM. (Weiwei Yang via wangda) 
Change-Id: I6f9f409f2ef685b405cbff547dea9623bf3322d9
 2018-07-16 10:54:41 -07:00
Eric E Payne 937ef39b3f YARN-8421: when moving app, activeUsers is increased, even though app does not have outstanding request. Contributed by Kyungwan Nam 2018-07-16 16:24:21 +00:00
bibinchundatt 4523cc5637 YARN-8434. Update federation documentation of Nodemanager configurations. Contributed by Bibin A Chundatt. 2018-07-15 13:53:53 +05:30
Akira Ajisaka 88625f5cd9
HADOOP-15531. Use commons-text instead of commons-lang in some classes to fix deprecation warnings. Contributed by Takanobu Asanuma. 2018-07-13 11:42:12 -04:00
Jason Lowe 17118f446c YARN-8515. container-executor can crash with SIGPIPE after nodemanager restart. Contributed by Jim Brennan 2018-07-13 10:05:25 -05:00
Robert Kanter 1bc106a738 YARN-8518. test-container-executor test_is_empty() is broken (Jim_Brennan via rkanter) 2018-07-12 16:38:46 -07:00
bibinchundatt 52e1bc8539 YARN-8491. TestServiceCLI#testEnableFastLaunch fail when umask is 077. Contributed by K G Bakthavachalam. 2018-07-11 16:20:29 +05:30
Sunil G 7f1d3d0e9d YARN-8512. ATSv2 entities are not published to HBase from second attempt onwards. Contributed by Rohith Sharma K S. 2018-07-11 12:26:32 +05:30
Giovanni Matteo Fumarola 82ac3aa6d0 YARN-8502. Use path strings consistently for webservice endpoints in RMWebServices. Contributed by Szilard Nemeth. 2018-07-10 10:36:17 -07:00
Sunil G 705e2c1f7c YARN-8473. Containers being launched as app tears down can leave containers in NEW state. Contributed by Jason Lowe. 2018-07-10 20:11:47 +05:30
Giovanni Matteo Fumarola ea9b608237 YARN-7899. [AMRMProxy] Stateful FederationInterceptor for pending requests. Contributed by Botong Huang. 2018-07-09 12:27:36 -07:00
Wangda Tan 83cd84b70b YARN-8506. Make GetApplicationsRequestPBImpl thread safe. (wangda) 
Change-Id: If304567abb77a01b686d82c769bdf50728484163
 2018-07-09 11:30:08 -07:00
Robert Kanter 0838fe8337 Only mount non-empty directories for cgroups (miklos.szegedi@cloudera.com via rkanter) 2018-07-09 10:37:20 -07:00
Rohith Sharma K S ba68320449 YARN-8302. ATS v2 should handle HBase connection issue properly. Contributed by Billie Rinaldi. 2018-07-06 15:19:01 -07:00
Sunil G e4bf38cf50 YARN-8492. ATSv2 HBase tests are failing with ClassNotFoundException. Contributed by Rohith Sharma K S. 2018-07-06 12:05:32 -07:00
Xiaoyu Yao 498e3bfb6b HADOOP-15571. Multiple FileContexts created with the same configuration object should be allowed to have different umask. Contributed by Vinod Kumar Vavilapalli. 2018-07-06 11:56:09 -07:00