20532 Commits

Author SHA1 Message Date
Pranay Singh
1f39eae7e6 HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit ecdeaa7e6ad43555031aed032e6ba7a14a17d7bc)
2019-01-03 09:57:08 -08:00
Weiwei Yang
a24cca11f2 YARN-9164. Shutdown NM may cause NPE when opportunistic container scheduling is enabled. Contributed by lujie.
(cherry picked from commit cfe89e6f963ba25b5fff1ce48cad36d74b3c789c)
2019-01-04 01:04:39 +08:00
Weiwei Yang
38ef85171d HADOOP-15323. AliyunOSS: Improve copy file performance for AliyunOSSFileSystemStore. Contributed wujinhu.
(cherry picked from commit 040a202b202a37f3b922cd321eb0a8ded457d88b)
2019-01-03 21:40:43 +08:00
Ayush Saxena
3783a35128
HADOOP-16025. Update the year to 2019.
Contributed by Ayush Saxena.

(cherry picked from commit cb26f154289ed065a967886b8eac04794907d643)
2019-01-02 22:24:57 +00:00
Weiwei Yang
0d78725fa8 MAPREDUCE-7174. Mapreduce example wordmedian should handle generic options. Contributed by Fei Hui.
(cherry picked from commit 9e7196749ee025db7668c648743d1deb72d3ca28)
2019-01-02 21:48:57 +08:00
Akira Ajisaka
eefe8ae960
HADOOP-16009. Replace the url of the repository in Apache Hadoop source code.
(cherry picked from commit eee29ed7066e9401c7530b8b6f3b221bacef9a8a)
2018-12-31 06:31:09 +09:00
Akira Ajisaka
6bc2781b39
HDFS-14183. [SPS] Remove the -w parameter from the -satisfystoragepolicy usage. Contributed by Ayush Saxena.
(cherry picked from commit f30473f2ef10482976fa49730dd9a139417150dc)
2018-12-31 03:17:27 +09:00
Wei-Chiu Chuang
c48640ffbd HDFS-14171. Performance improvement in Tailing EditLog. Contributed by Kenneth Yang.
(cherry picked from commit e9a005dfcf4d2249235aeff4da85b321dddc1ca9)
2018-12-28 10:38:55 -08:00
Sunil G
9bfea98746 YARN-9090. [Submarine] Update submarine installation script document. Contributed by Xun Liu.
(cherry picked from commit 742fd978b5b585fc14fc3fea7ec37f80c5d7dc97)
2018-12-26 08:27:50 +05:30
Surendra Singh Lilhore
d2ec1ff020 HDFS-14046. In-Maintenance ICON is missing in datanode info page. Contributed by Ranith Sardar.
(cherry picked from commit 686fcd4db34dfe8642ff4b25fffbc73e42217f30)
2018-12-24 11:09:07 +05:30
Surendra Singh Lilhore
5ab79f6377 HDFS-14166. Ls with -e option not giving the result in proper format. Contributed by Shubham Dewan.
(cherry picked from commit 763e96ea2b18f584ba102b68271263be8c5d621c)
2018-12-22 12:39:51 +05:30
Akira Ajisaka
d6baf9d5bf
YARN-9154. Fix itemization in YARN service quickstart document. Contributed by Ayush Saxena.
(cherry picked from commit e98e963e5143600017d2d227d504140b8802439d)
2018-12-22 13:26:41 +09:00
Weiwei Yang
7deef08eb8 YARN-8925. Updating distributed node attributes only when necessary. Contributed by Tao Yang. 2018-12-21 16:31:03 +08:00
Jason Lowe
9c89e2ea76 HADOOP-15973. Configuration: Included properties are not cached if resource is a stream. Contributed by Eric Payne
(cherry picked from commit 3961690037fa4274a3640104b3c344e8626caa10)
2018-12-20 09:33:22 -06:00
Takanobu Asanuma
1b941a680d HDFS-13661. Ls command with e option fails when the filesystem is not HDFS.
(cherry picked from commit d9635759182b614a1dd5034c30978e7c4be8d0dd)
2018-12-20 10:05:02 +09:00
Eric Yang
29c9c8a893 YARN-9126. Fix container clean up for reinitialization.
Contributed by Chandni Singh

(cherry picked from commit e815fd9c49e80b9200dd8852abe74fe219ad9110)
2018-12-19 14:58:19 -05:00
Eric Yang
4e1cef3625 YARN-9084. Reset container state and defer readiness check for upgrade.
Contributed by Chandni Singh

(cherry picked from commit ccdd982e51f4413bc2e98d03e0a05b1133042a31)
2018-12-18 18:05:40 -05:00
Eric Yang
28ca14e71b YARN-9040. Fixed memory leak in LevelDBCacheTimelineStore and DBIterator.
Contributed by Tarun Parimi

(cherry picked from commit 71e0b0d8005ea1952dc7e582db15c2ac09df7c91)
2018-12-17 12:08:09 -05:00
Da Zhou
f122ae7279
HADOOP-16004. ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException.
Contributed by Da Zhou.

(cherry picked from commit 346c0c8aff0b206d45f34dbce4fcc81364115d95)
2018-12-17 11:18:12 +00:00
Da Zhou
d09dbcc8fb
HADOOP-15972 ABFS: reduce list page size to to 500.
Contributed by Da Zhou.
2018-12-17 11:08:17 +00:00
Da Zhou
87d9a54968
HADOOP-15969. ABFS: getNamespaceEnabled can fail blocking user access thru ACLs.
Contributed by Da Zhou.

(cherry picked from commit b2523d8100844338e073531c47666d744a101caf)
2018-12-17 11:05:39 +00:00
Akira Ajisaka
71d8ffb551
YARN-9122. Add table of contents to YARN Service API document. Contributed by Zhankun Tang.
(cherry picked from commit 26e55d4ceb691f47801fa5f0ba1147a5bc413f96)
2018-12-15 10:30:37 +09:00
Eric Yang
52aafb9789 YARN-9125. Fixed Carriage Return detection in Docker container launch command.
Contributed by Billie Rinaldi

(cherry picked from commit b2d7204ed0b64405a0adbe07e3eaf385e046efa1)
2018-12-14 17:55:10 -05:00
Akira Ajisaka
495285e50e
MAPREDUCE-7170. Doc typo in PluggableShuffleAndPluggableSort.md. Contributed by Zhaohui Xin.
(cherry picked from commit e0d75088f24283985a4e4032fa547dad2ebb54db)
2018-12-13 16:27:52 +09:00
Wangda Tan
29e4e5f62c YARN-9112. [Submarine] Support polling applicationId when it's not ready in cluster. (Zhankun Tang via wangda)
Change-Id: I73d73f3d631b28fb9866faa56571839b13824a97
(cherry picked from commit 9fba6cc2471db8df9ca6da7b9df378e3072dfeb2)
2018-12-12 11:52:16 -08:00
Wangda Tan
ef0f835537 YARN-8714. [Submarine] Support files/tarballs to be localized for a training job. (Zhankun Tang via wangda)
Change-Id: I845131273e52a9d81dbc813ea6d4af06b205e334
(cherry picked from commit c771fe6e10bd3f8533a01026de3c33db2a983cf4)
2018-12-12 11:52:07 -08:00
Steve Loughran
bbe1ddd309
HADOOP-15808. Harden Token service loader use.
Contributed by Steve Loughran.

(cherry picked from commit 202926ac3301298753abd0e6e1f324caf0202ec6)
2018-12-11 17:34:16 +00:00
Vinayakumar B
abe14d32d4 HDFS-14124. EC : Support EC Commands (set/get/unset EcPolicy) via WebHdfs. Contributed by Ayush Saxena.
(cherry picked from commit 39dc7345b80e27ba8bd1ff4c19ca241aef5ac0fc)
2018-12-11 18:10:51 +05:30
Weiwei Yang
2b3c3d2a32 YARN-9009. Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs. Contributed by OrDTesters.
(cherry picked from commit 1c09a10e9601bd628f1e887f3bf92c5a4ac286ab)
2018-12-10 12:07:23 +08:00
Jonathan Hung
3ab6ea7aca YARN-9085. Add Guaranteed and MaxCapacity to CSQueueMetrics
(cherry picked from commit 978ab3e958227220cb6f1a08ae6e7cdb8a46628b)
2018-12-07 10:45:47 -08:00
Jason Lowe
298a1f29e8 MAPREDUCE-7159. FrameworkUploader: ensure proper permissions of generated framework tar.gz if restrictive umask is used. Contributed by Peter Bacsko
(cherry picked from commit 9886c27c71147a2bcef87b42ff21c2bbe98c8ab4)
2018-12-06 14:57:45 -06:00
Eric Yang
8c70728f7f YARN-9071. Improved status update for reinitialized containers.
Contributed by Chandni Singh

(cherry picked from commit 1b790f4dd1f682423d5dbb8e70c6225cbddce989)
2018-12-05 19:04:55 -05:00
Jason Lowe
78b90725a6 HADOOP-15974. Upgrade Curator version to 2.13.0 to fix ZK tests. Contributed by Akira Ajisaka
(cherry picked from commit 9287ab364292ce917fc120532681131821ac53ef)
2018-12-04 16:44:21 -06:00
Wangda Tan
559cb99bf6 Revert "YARN-8870. [Submarine] Add submarine installation scripts. (Xun Liu via wangda)"
This reverts commit 920401b2966c2dcd7cd1b3dfd7a764f08412323f.
2018-12-04 14:13:39 -08:00
Steve Loughran
05e41dd4fe
HADOOP-15966. Hadoop Kerberos broken on macos as java.security.krb5.realm is reset.
Contributed by Steve Loughran

(cherry picked from commit db2d8b01c65fdff3acf83b4c4f0a79fe48270487)
2018-12-04 15:36:50 +00:00
Akira Ajisaka
b5ebafa19d
HADOOP-15970. Upgrade plexus-utils from 2.0.5 to 3.1.0.
(cherry picked from commit ff31313d83ae26be5e29eaad7d9ce6184bae83e2)
2018-12-04 22:46:55 +09:00
Da Zhou
2d2212a508
HADOOP-15968. ABFS: add try catch for UGI failure when initializing ABFS.
Contributed by Da Zhou.

(cherry picked from commit a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b)
2018-12-04 13:40:03 +00:00
Giovanni Matteo Fumarola
7cfcabc473 HADOOP-15950. Failover for LdapGroupsMapping. Contributed by Lukas Majercak. 2018-12-03 12:44:19 -08:00
Jonathan Hung
6b01e4d2a8 YARN-9036. Escape newlines in health report in YARN UI. Contributed by Keqiu Hu
(cherry picked from commit 1c8bd7128c99d8215ef16438bd2ce6b1f025a966)
2018-11-30 10:16:00 -08:00
Eric Yang
69be6e7352 HADOOP-15922. Revert patch 004.
(cherry picked from commit 38ea3814bd0641d895e5d3b7415c6308e7f8491e)
2018-11-30 12:51:58 -05:00
Da Zhou
9bc1fd4721
HADOOP-15957. WASB: Add asterisk wildcard support for PageBlobDirSet.
Contributed by Da Zhou.

(cherry picked from commit 7ccb640a66bb5bb9f657a2db85bcc8ae0ded4892)
2018-11-30 10:13:57 +00:00
Akira Ajisaka
b722bfd6f5
MAPREDUCE-7165. mapred-site.xml is misformatted in single node setup document. Contributed by Zhaohui Xin.
(cherry picked from commit c9bfca217f4b15a3a367db51147d0dc2075ca274)
2018-11-30 13:30:32 +09:00
Tsz Wo Nicholas Sze
9d508f719b HDFS-14112. Avoid recursive call to external authorizer for getContentSummary. 2018-11-29 13:56:43 -08:00
Brahma Reddy Battula
e2fa9e8cdd HDFS-14095. EC: Track Erasure Coding commands in DFS statistics. Contributed by Ayush Saxena.
(cherry picked from commit f534736867eed962899615ca1b7eb68bcf591d17)
2018-11-30 00:28:04 +05:30
Eric Yang
a8f67ad7c9 YARN-9067. Fixed Resource Manager resource leak via YARN service.
Contributed by Eric Yang
2018-11-29 13:50:06 -05:00
Steve Loughran
1a448565a8
HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple"
This reverts commit 829a2e4d271f05afb209ddc834cd4a0e85492eda.

(cherry picked from commit d0edd37269bb40290b409d583bcf3b70897c13e0)
2018-11-29 17:53:23 +00:00
bibinchundatt
183ec39c4b YARN-9069. Fix SchedulerInfo#getSchedulerType for custom schedulers. Contributed by Bilwa S T.
(cherry picked from commit 07142f54a8c7f70857e99c041f3a2a5189c809b5)
2018-11-29 22:16:32 +05:30
Wangda Tan
ee9deb6e9c YARN-9030. Log aggregation changes to handle filesystems which do not support setting permissions. (Suma Shivaprasad via wangda)
Change-Id: I80f1e8196b8624e24d74494719fdedfd7061dced
(cherry picked from commit 9de8e8d0496a2628b63cc841b1fdee80e2912f7a)
2018-11-28 15:06:35 -08:00
Jason Lowe
7a78bdf7bb MAPREDUCE-7164. FileOutputCommitter does not report progress while merging paths. Contributed by Kuhu Shukla
(cherry picked from commit 4d8de7ab690ef919b392b12d856482a6a1f2bb3d)
2018-11-28 15:54:59 -06:00
Jason Lowe
df0e7766e4 YARN-8812. Containers fail during creating a symlink which started with hyphen for a resource file. Contributed by Oleksandr Shevchenko
(cherry picked from commit 3ce99e32f7d7887412cae8337cd4ebeb3b2ee308)
2018-11-28 08:50:18 -06:00