f4ee38df29
YARN-9519. TFile log aggregation file format is not working for yarn.log-aggregation.TFile.remote-app-log-dir config. Contributed by Adam Antal.
...
(cherry picked from commit 7d831eca64
2019-05-14 10:49:09 -07:00
b5b152aded
YARN-9504. [UI2] Fair scheduler queue view page does not show actual capacity. Contributed by Zoltan Siegl.
...
(cherry picked from commit 64c7f36ab1
2019-05-10 14:28:33 +05:30
12e0053932
HADOOP-16278. With S3A Filesystem, Long Running services End up Doing lot of GC and eventually die.
...
Contributed by Rajat Khandelwal
(cherry picked from commit 591ca69823
2019-05-09 21:14:37 +01:00
095815220d
HADOOP-16289. Allow extra jsvc startup option in hadoop_start_secure_daemon in hadoop-functions.sh. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 93f2283a69
2019-05-06 15:47:58 -07:00
c6573562cb
YARN-9529. Log correct cpu controller path on error while initializing CGroups. (Contributed by Jonathan Hung)
...
(cherry picked from commit 597fa47ad1
2019-05-06 11:58:31 -07:00
6fce24fb40
YARN-9285: RM UI progress column is of wrong type. Contributed by Ahmed Hussein.
...
(cherry picked from commit b094b94d43
2019-05-02 19:48:06 +00:00
bba8971897
HDFS-14463. Add Log Level link under NameNode and DataNode Web UI Utilities dropdown. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7cb46f035a
2019-05-01 16:49:45 -07:00
db4c0b357d
HDFS-13677. Dynamic refresh Disk configuration results in overwriting VolumeMap. Contributed by xuzq.
...
(cherry picked from commit 4b4200f1f8
2019-04-29 14:54:30 -07:00
cc0c85f04a
YARN-9325. TestQueueManagementDynamicEditPolicy fails intermittent. Contributed by Prabhu Joseph.
...
(cherry picked from commit 1c8046d67e
2019-04-23 14:24:15 +08:00
28ff96f367
HADOOP-16265. Fix bug causing Configuration#getTimeDuration to use incorrect units when the default value is used. Contributed by starphin.
...
(cherry-picked from 1ddb48872f
2019-04-22 08:18:17 -07:00
ac85aa80d9
YARN-8587. Added retries for fetching docker exit code.
...
Contributed by Charo Zhang
(cherry picked from commit c16c49b8c3
2019-04-19 15:40:23 -04:00
4a64dab0dd
YARN-8622. Fixed container-executor compilation on MacOSX.
...
Contributed by Siyao Meng
(cherry picked from commit ef97a20831
2019-04-18 19:01:11 -04:00
2503409977
YARN-6695. Fixed NPE in publishing appFinished events to ATSv2.
...
Contributed by Prabhu Joseph
(cherry picked from commit df76cdc895
2019-04-18 12:30:55 -04:00
742a3ad24b
YARN-9487. NodeManager native build shouldn't link against librt on macOS. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 6e4399ea61
2019-04-17 22:57:33 -07:00
d724f74f6b
HDFS-14432. dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml
...
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 13907d8479
2019-04-17 21:55:45 +09:00
df5d8f05d9
HADOOP-16227. Upgrade checkstyle to 8.19
...
(cherry picked from commit 4b4fef2f0e
2019-04-15 10:47:02 +09:00
03079be707
HADOOP-14544. DistCp documentation for command line options is misaligned. Contributed by Masatake Iwasaki.
...
(cherry picked from commit bbdbc7a9a1
2019-04-12 11:59:14 +09:00
db185de31c
YARN-9463. Add queueName info when failing with queue capacity sanity check. Contributed by Aihua Xu.
...
(cherry picked from commit 8c1bba375b
2019-04-10 23:02:24 +08:00
b28fdb98ee
HADOOP-14635. Javadoc correction for AccessControlList#buildACL. Contributed by Yeliang Cang.
...
(cherry picked from commit ab645b3caa
2019-04-08 15:19:17 +09:00
7a80b1b481
YARN-9413. Queue resource leak after app fail for CapacityScheduler. Contributed by Tao Yang.
...
(cherry picked from commit ec143cbf67
2019-04-06 20:19:03 +08:00
f020ac2c06
HDFS-14407. Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes. Contributed by Wanqiang Ji.
...
(cherry picked from commit 67020f0950
2019-04-05 12:30:46 +09:00
50fc9577c0
HADOOP-16208. Do Not Log InterruptedException in Client.
...
Contributed by David Mollitor.
(cherry picked from commit c90736350b
2019-04-04 16:16:57 +01:00
388f445dde
HDFS-14389. getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c528e427aa6745434672b1c1850738795ad1d6d2)
2019-04-04 04:06:15 -07:00
06cf8ce30f
YARN-9394. Use new API of RackResolver to get better performance. Contributed by Lantao Jin.
...
(cherry picked from commit 945e8c6064
2019-04-04 18:41:04 +08:00
b6ebe74526
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true ( #685 )
...
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.
Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.
Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up
Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.
Contributed by Steve Loughran.
Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
(cherry picked from commit 366186d999
2019-04-03 21:35:19 +01:00
c8703dda07
HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang.
...
(cherry picked from commit be488b6070
2019-04-03 11:01:44 -07:00
80a8d3310e
HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji.
...
(cherry picked from commit 8b6deebb1d
2019-04-03 19:36:17 +09:00
b6039e241d
HADOOP-16226. new Path(String str) does not remove all the trailing slashes of str
...
(cherry picked from commit aaaf856f4b
2019-04-03 13:18:36 +09:00
8037052c2b
HADOOP-16225. Fix links to the developer mailing lists in DownstreamDev.md. Contributed by Wanqiang Ji.
...
(cherry picked from commit ebd0d21538
2019-04-02 10:54:16 +09:00
ddf8859655
MAPREDUCE-7190. Add SleepJob additional parameter to make parallel runs distinguishable. Contributed by Adam Antal.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 856cbf62d3
2019-04-01 10:38:58 -07:00
67cdf807a2
HADOOP-16220. Add findbugs ignores for unjustified issues during update to guava to 27.0-jre in hadoop-project
...
This closes #665
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 53a86e2b8e
2019-04-01 13:52:19 +09:00
60c9042286
HADOOP-16058. S3A tests to include Terasort.
...
Contributed by Steve Loughran.
This includes
- HADOOP-15890. Some S3A committer tests don't match ITest* pattern; don't run in maven
- MAPREDUCE-7090. BigMapOutput example doesn't work with paths off cluster fs
- MAPREDUCE-7091. Terasort on S3A to switch to new committers
- MAPREDUCE-7092. MR examples to work better against cloud stores
2019-03-29 15:25:45 +00:00
52cfbc39cc
HADOOP-16037. DistCp: Document usage of Sync (-diff option) in detail.
...
Contributed by Siyao Meng
(cherry picked from commit ce4bafdf44
2019-03-26 18:43:43 +00:00
162e9999c7
HDFS-14037. Fix SSLFactory truststore reloader thread leak in URLConnectionFactory.
...
(cherry picked from commit 55fb3c32fb
2019-03-27 03:28:54 +09:00
10642a6205
YARN-9391. Fixed node manager environment leaks into Docker containers.
...
Contributed by Jim Brennan
(cherry picked from commit 3c45762a0b
2019-03-25 15:54:52 -04:00
ade3af6ef2
HADOOP-16147. Allow CopyListing sequence file keys and values to be more easily customized.
...
Author: Andrew Olson
(cherry picked from commit faba3591d3
2019-03-22 10:36:34 +00:00
397b63ad0b
HADOOP-16181. HadoopExecutors shutdown Cleanup.
...
Author: David Mollitor <david.mollitor@cloudera.com>
(cherry picked from commit d18d0859eb
2019-03-22 10:30:21 +00:00
9a449ac075
HADOOP-16196. Path Parameterize Comparable.
...
Author: David Mollitor <david.mollitor@cloudera.com>
(cherry picked from commit 246ab77f28
2019-03-22 10:27:17 +00:00
39f60faa60
HADOOP-16191. AliyunOSS: improvements for copyFile/copyDirectory and logging. Contributed by wujinhu.
...
(cherry picked from commit 568d3ab8b6
2019-03-19 10:08:11 +08:00
81a6ba1825
HADOOP-16124. Extend documentation in testing.md about S3 endpoint constants.
...
Contributed by Adam Antal.
(cherry picked from commit c0427c84dd
2019-03-18 19:14:43 +00:00
0de8b55a09
HADOOP-16192. Fix CallQueue backoff bugs: perform backoff when add() is used and update backoff when refreshed.
...
(cherry-picked from 8c95cb9d6b
2019-03-18 08:46:53 -07:00
4eb0497091
HDFS-14366. Improve HDFS append performance. Contributed by Chao Sun.
...
(cherry picked from commit ff06ef0631
2019-03-15 13:58:03 -07:00
43e8ac6097
HADOOP-15625. S3A input stream to use etags/version number to detect changed source files.
...
Author: Ben Roling <ben.roling@gmail.com>
Initial patch from Brahma Reddy Battula.
2019-03-14 19:46:34 +00:00
fec7c5f3eb
HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun.
...
(cherry picked from commit 66357574ae
2019-03-13 13:19:18 -07:00
3d86dd8931
MAPREDUCE-7192. JobHistoryServer attempts page support jump to containers log page in NM when logAggregation is disable. Contributed by Jiandan Yang.
...
(cherry picked from commit 159a715eef
2019-03-13 17:41:31 +08:00
1ceefa726e
HDFS-14081. hdfs dfsadmin -metasave metasave_test results NPE. Contributed by Shweta Yakkali.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 1bea785020
2019-03-12 16:05:55 -07:00
a21e2e4dbc
HDFS-14333. Datanode fails to start if any disk has errors during Namenode registration. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 34b14061b3
2019-03-12 10:18:56 -07:00
b6f6c34223
HADOOP-16109. Parquet reading S3AFileSystem causes EOF
...
Nobody gets seek right. No matter how many times they think they have.
Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
2019-03-11 11:15:25 +00:00
9a2b24642d
MAPREDUCE-7191. JobHistoryServer should log exception when loading/parsing history file failed. Contributed by Jiandan Yang.
...
(cherry picked from commit f0605146b3
2019-03-11 16:03:38 +08:00
cfaf21a4ba
HADOOP-16169. ABFS: Bug fix for getPathProperties.
...
Author: Da Zhou <da.zhou@microsoft.com>
(cherry picked from commit e0260417ad
2019-03-08 13:53:44 +00:00
Sunil G
Rajat Khandelwal
Siyao Meng
Haibo Chen
Eric E Payne
Arpit Agarwal
Weiwei Yang
Erik Krogen
Eric Yang
Puleya7
Akira Ajisaka
Masatake Iwasaki
David Mollitor
Stephen O'Donnell
Steve Loughran
Wei-Chiu Chuang
Adam Antal
Gabor Bota
Takanobu Asanuma
Andrew Olson
Inigo Goiri
Ben Roling
Shweta Yakkali
Da Zhou