Steve Loughran
b6ebe74526
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true ( #685 )
...
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.
Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.
Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up
Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.
Contributed by Steve Loughran.
Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
(cherry picked from commit 366186d999
)
2019-04-03 21:35:19 +01:00
Wei-Chiu Chuang
c8703dda07
HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang.
...
(cherry picked from commit be488b6070
)
2019-04-03 11:01:44 -07:00
Akira Ajisaka
80a8d3310e
HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji.
...
(cherry picked from commit 8b6deebb1d
)
2019-04-03 19:36:17 +09:00
Akira Ajisaka
b6039e241d
HADOOP-16226. new Path(String str) does not remove all the trailing slashes of str
...
(cherry picked from commit aaaf856f4b
)
2019-04-03 13:18:36 +09:00
Akira Ajisaka
8037052c2b
HADOOP-16225. Fix links to the developer mailing lists in DownstreamDev.md. Contributed by Wanqiang Ji.
...
(cherry picked from commit ebd0d21538
)
2019-04-02 10:54:16 +09:00
Adam Antal
ddf8859655
MAPREDUCE-7190. Add SleepJob additional parameter to make parallel runs distinguishable. Contributed by Adam Antal.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 856cbf62d3
)
2019-04-01 10:38:58 -07:00
Gabor Bota
67cdf807a2
HADOOP-16220. Add findbugs ignores for unjustified issues during update to guava to 27.0-jre in hadoop-project
...
This closes #665
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 53a86e2b8e
)
2019-04-01 13:52:19 +09:00
Steve Loughran
60c9042286
HADOOP-16058. S3A tests to include Terasort.
...
Contributed by Steve Loughran.
This includes
- HADOOP-15890. Some S3A committer tests don't match ITest* pattern; don't run in maven
- MAPREDUCE-7090. BigMapOutput example doesn't work with paths off cluster fs
- MAPREDUCE-7091. Terasort on S3A to switch to new committers
- MAPREDUCE-7092. MR examples to work better against cloud stores
2019-03-29 15:25:45 +00:00
Siyao Meng
52cfbc39cc
HADOOP-16037. DistCp: Document usage of Sync (-diff option) in detail.
...
Contributed by Siyao Meng
(cherry picked from commit ce4bafdf44
)
2019-03-26 18:43:43 +00:00
Takanobu Asanuma
162e9999c7
HDFS-14037. Fix SSLFactory truststore reloader thread leak in URLConnectionFactory.
...
(cherry picked from commit 55fb3c32fb
)
2019-03-27 03:28:54 +09:00
Eric Yang
10642a6205
YARN-9391. Fixed node manager environment leaks into Docker containers.
...
Contributed by Jim Brennan
(cherry picked from commit 3c45762a0b
)
2019-03-25 15:54:52 -04:00
Andrew Olson
ade3af6ef2
HADOOP-16147. Allow CopyListing sequence file keys and values to be more easily customized.
...
Author: Andrew Olson
(cherry picked from commit faba3591d3
)
2019-03-22 10:36:34 +00:00
David Mollitor
397b63ad0b
HADOOP-16181. HadoopExecutors shutdown Cleanup.
...
Author: David Mollitor <david.mollitor@cloudera.com>
(cherry picked from commit d18d0859eb
)
2019-03-22 10:30:21 +00:00
David Mollitor
9a449ac075
HADOOP-16196. Path Parameterize Comparable.
...
Author: David Mollitor <david.mollitor@cloudera.com>
(cherry picked from commit 246ab77f28
)
2019-03-22 10:27:17 +00:00
Weiwei Yang
39f60faa60
HADOOP-16191. AliyunOSS: improvements for copyFile/copyDirectory and logging. Contributed by wujinhu.
...
(cherry picked from commit 568d3ab8b6
)
2019-03-19 10:08:11 +08:00
Adam Antal
81a6ba1825
HADOOP-16124. Extend documentation in testing.md about S3 endpoint constants.
...
Contributed by Adam Antal.
(cherry picked from commit c0427c84dd
)
2019-03-18 19:14:43 +00:00
Erik Krogen
0de8b55a09
HADOOP-16192. Fix CallQueue backoff bugs: perform backoff when add() is used and update backoff when refreshed.
...
(cherry-picked from 8c95cb9d6b
)
2019-03-18 08:46:53 -07:00
Inigo Goiri
4eb0497091
HDFS-14366. Improve HDFS append performance. Contributed by Chao Sun.
...
(cherry picked from commit ff06ef0631
)
2019-03-15 13:58:03 -07:00
Ben Roling
43e8ac6097
HADOOP-15625. S3A input stream to use etags/version number to detect changed source files.
...
Author: Ben Roling <ben.roling@gmail.com>
Initial patch from Brahma Reddy Battula.
2019-03-14 19:46:34 +00:00
Erik Krogen
fec7c5f3eb
HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun.
...
(cherry picked from commit 66357574ae
)
2019-03-13 13:19:18 -07:00
Weiwei Yang
3d86dd8931
MAPREDUCE-7192. JobHistoryServer attempts page support jump to containers log page in NM when logAggregation is disable. Contributed by Jiandan Yang.
...
(cherry picked from commit 159a715eef
)
2019-03-13 17:41:31 +08:00
Shweta Yakkali
1ceefa726e
HDFS-14081. hdfs dfsadmin -metasave metasave_test results NPE. Contributed by Shweta Yakkali.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 1bea785020
)
2019-03-12 16:05:55 -07:00
Stephen O'Donnell
a21e2e4dbc
HDFS-14333. Datanode fails to start if any disk has errors during Namenode registration. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 34b14061b3
)
2019-03-12 10:18:56 -07:00
Steve Loughran
b6f6c34223
HADOOP-16109. Parquet reading S3AFileSystem causes EOF
...
Nobody gets seek right. No matter how many times they think they have.
Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
2019-03-11 11:15:25 +00:00
Weiwei Yang
9a2b24642d
MAPREDUCE-7191. JobHistoryServer should log exception when loading/parsing history file failed. Contributed by Jiandan Yang.
...
(cherry picked from commit f0605146b3
)
2019-03-11 16:03:38 +08:00
Da Zhou
cfaf21a4ba
HADOOP-16169. ABFS: Bug fix for getPathProperties.
...
Author: Da Zhou <da.zhou@microsoft.com>
(cherry picked from commit e0260417ad
)
2019-03-08 13:53:44 +00:00
Erik Krogen
6d076dd5e8
HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam.
...
(cherry-picked from commit 1bc282e0b3
)
2019-03-07 08:42:41 -08:00
Praveen Krishna
451844fee5
HADOOP-16114. NetUtils#canonicalizeHost gives different value for same host.
...
Author: Praveen Krishna <praveenkrishna@tutanota.com>
(cherry picked from commit 2b94e51a8f
)
2019-03-07 11:08:48 +00:00
Sunil G
aff5973401
YARN-8803. [UI2] Show flow runs in the order of recently created time in graph widgets. Contributed by Akhil PB.
...
(cherry picked from commit c79f139519
)
2019-03-06 16:49:49 +05:30
Sunil G
d721634fea
YARN-9138. Improve test coverage for nvidia-smi binary execution of GpuDiscoverer. Contributed by Szilard Nemeth.
...
(cherry picked from commit 46045c5cb3
)
2019-03-06 16:01:56 +05:30
Stephen O'Donnell
3fe31b36fa
HADOOP-16140. hadoop fs expunge to add -immediate option to purge trash immediately.
...
Contributed by Stephen O'Donnell.
(cherry picked from commit 686c0141ef
)
Signed-off-by: Steve Loughran <stevel@apache.org>
2019-03-05 14:11:49 +00:00
Da Zhou
dc38fc598d
HADOOP-16136. ABFS: Should only transform username to short name
...
Contributed by Da Zhou.
(cherry picked from commit 3988e75ca3
)
Signed-off-by: Steve Loughran <stevel@apache.org>
2019-03-05 10:47:58 +00:00
Da Zhou
075f6b061c
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth.
...
Contributed by Da Zhou and Junhua Gu.
(cherry picked from commit 1f1655028e
)
Signed-off-by: Steve Loughran <stevel@apache.org>
2019-03-05 10:44:46 +00:00
Da Zhou
ae832ccffe
HADOOP-16041. Include Hadoop version in User-Agent string for ABFS.
...
Contributed by Shweta Yakkali.
Signed-off-by: Sean Mackrory <mackrorysd@apache.org>
(cherry picked from commit 02eb91856e
)
Signed-off-by: Steve Loughran <stevel@apache.org>
2019-03-05 10:39:37 +00:00
Wei-Chiu Chuang
e58ccca3ce
HDFS-14314. fullBlockReportLeaseId should be reset after registering to NN. Contributed by star.
...
(cherry picked from commit 387dbe587a
)
2019-03-04 10:45:31 -08:00
bibinchundatt
63ed16e076
Revert "YARN-8132. Final Status of applications shown as UNDEFINED in ATS app queries. Contributed by Prabhu Joseph"
...
This reverts commit cf1944eb6e
.
2019-03-04 17:01:40 +05:30
Weiwei Yang
4ceb4e4f05
YARN-9332. RackResolver tool should accept multiple hosts. Contributed by Lantao Jin.
...
(cherry picked from commit fe6b2b2f23e69f0643e870d9c500117088983209)
2019-03-02 16:04:24 +00:00
Erik Krogen
af16db86d4
HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao.
2019-03-01 08:12:44 -08:00
Sunil G
d045f02a8d
YARN-9139. Simplify initializer code of GpuDiscoverer. Contributed by Szilard Nemeth.
2019-03-01 19:27:03 +05:30
Eric Yang
3f3548b66a
YARN-9334. Allow YARN Service client to send SPNEGO challenge header when authentication type is not simple.
...
Contributed by Billie Rinaldi
(cherry picked from commit 04b228e43b
)
2019-02-28 09:33:05 -08:00
Weiwei Yang
7575e3090d
YARN-9324. TestSchedulingRequestContainerAllocation(Async) fails with junit-4.11. Contributed by Prabhu Joseph.
2019-02-28 09:32:07 +08:00
Weiwei Yang
7fa5373ec4
YARN-9248. RMContainerImpl:Invalid event: ACQUIRED at KILLED. Contributed by lujie.
...
(cherry picked from commit 8c30114b00
)
2019-02-27 17:35:09 +08:00
Sunil G
809e3f2453
YARN-9121. Replace GpuDiscoverer.getInstance() to a readable object for easy access control. Contributed by Szilard Nemeth.
...
(cherry picked from commit 5e91ebd91a
)
2019-02-27 12:03:58 +05:30
Sunil G
a95a0cbf2f
YARN-9087. Improve logging for initialization of Resource plugins. Contributed by Szilard Nemeth.
2019-02-27 11:54:43 +05:30
Weiwei Yang
bdde6a612e
YARN-9329. updatePriority is blocked when using FairScheduler. Contributed by Jiandan Yang.
...
(cherry picked from commit 3e1739d589
)
2019-02-26 00:18:24 +08:00
Sunil G
359e459df1
YARN-9168. DistributedShell client timeout should be -1 by default. Contributed by Zhankun Tang.
...
(cherry picked from commit 6cec90653d
)
2019-02-25 15:29:31 +05:30
Sunil G
f282f9c362
YARN-9213. RM Web UI v1 does not show custom resource allocations for containers page. Contributed by Szilard Nemeth.
2019-02-25 11:37:42 +05:30
Weiwei Yang
cdce1c17a0
YARN-9316. TestPlacementConstraintsUtil#testInterAppConstraintsByAppID fails intermittently. Contributed by Prabhu Joseph.
...
(cherry picked from commit 9cd5c5447f
)
2019-02-24 22:48:55 +08:00
Weiwei Yang
604a915bab
YARN-9238. Avoid allocating opportunistic containers to previous/removed/non-exist application attempt. Contributed by lujie.
...
(cherry picked from commit 9c88695bcd
)
2019-02-24 22:21:53 +08:00
bibinchundatt
3e1bd53a37
YARN-9317. Avoid repeated YarnConfiguration#timelineServiceV2Enabled check. Contributed by Prabhu Joseph
2019-02-23 07:59:51 +05:30