Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
18,010 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

18010 Commits

Author SHA1 Message Date
Akira Ajisaka fbbb0e9919
HADOOP-16227. Upgrade checkstyle to 8.19 
(cherry picked from commit 4b4fef2f0e)
(cherry picked from commit b45813b259)
 2019-04-15 10:59:03 +09:00
Masatake Iwasaki 7985d9b1ce HADOOP-14544. DistCp documentation for command line options is misaligned. Contributed by Masatake Iwasaki. 
(cherry picked from commit bbdbc7a9a1)

Conflicts:
	hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm
 2019-04-12 12:18:19 +09:00
Weiwei Yang 5d85a34b1f YARN-9463. Add queueName info when failing with queue capacity sanity check. Contributed by Aihua Xu. 
(cherry picked from commit 8c1bba375b)
 2019-04-10 23:05:12 +08:00
Weiwei Yang 0e167fde55 YARN-9413. Queue resource leak after app fail for CapacityScheduler. Contributed by Tao Yang. 2019-04-08 13:44:33 +08:00
Akira Ajisaka f824f4dccb
HDFS-14407. Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes. Contributed by Wanqiang Ji. 
(cherry picked from commit 67020f0950)
 2019-04-05 12:31:38 +09:00
Stephen O'Donnell 79714fdfab HDFS-14389. getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c528e427aa6745434672b1c1850738795ad1d6d2)
(cherry picked from commit 388f445dde)
(cherry picked from commit d9899015eb)
 2019-04-04 04:06:57 -07:00
Steve Loughran bdf8be6374
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685) 
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without  using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
 2019-04-03 22:40:09 +01:00
Wei-Chiu Chuang cbac102566 HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang. 
(cherry picked from commit be488b6070)
(cherry picked from commit c8703dda07)
(cherry picked from commit 2a94603ae66d9000c0bb07df0d592279339af103)
 2019-04-03 11:02:14 -07:00
Akira Ajisaka 6f0896e087
HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji. 
(cherry picked from commit 8b6deebb1d)
(cherry picked from commit 875435dc7d)
 2019-04-03 19:40:52 +09:00
Akira Ajisaka 524dc4e662
HADOOP-16225. Fix links to the developer mailing lists in DownstreamDev.md. Contributed by Wanqiang Ji. 
(cherry picked from commit ebd0d21538)
 2019-04-02 10:55:27 +09:00
Gabor Bota 461df1b511
HADOOP-16220. Add findbugs ignores for unjustified issues during update to guava to 27.0-jre in hadoop-project 
This closes #665

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 53a86e2b8e)
 2019-04-01 13:53:11 +09:00
Takanobu Asanuma abb06c9d7c HDFS-14037. Fix SSLFactory truststore reloader thread leak in URLConnectionFactory. 
(cherry picked from commit 55fb3c32fb)
 2019-03-27 03:30:12 +09:00
Weiwei Yang 9b1f13fbd2 HADOOP-16191. AliyunOSS: improvements for copyFile/copyDirectory and logging. Contributed by wujinhu. 
(cherry picked from commit 568d3ab8b6)
 2019-03-19 10:13:57 +08:00
Erik Krogen e172fc62ce HADOOP-16192. Fix CallQueue backoff bugs: perform backoff when add() is used and update backoff when refreshed. 
(cherry-picked from 8c95cb9d6b)
(cherry-picked from 0de8b55a09)
(cherry-picked from d4fbbc83ad)
 2019-03-18 08:57:41 -07:00
Inigo Goiri 5a8445789b HDFS-14366. Improve HDFS append performance. Contributed by Chao Sun. 
(cherry picked from commit ff06ef0631)
 2019-03-15 13:59:51 -07:00
Erik Krogen 9a5daa5ec6 HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 
(cherry picked from commit 66357574ae)
(cherry picked from commit fec7c5f3eb)
(cherry picked from commit cb76fc5618)
 2019-03-13 13:21:40 -07:00
Steve Loughran 7f70a78ce3
HADOOP-16109. Parquet reading S3AFileSystem causes EOF (#589) 
Nobody gets seek right. No matter how many times they think they have.

Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran

Change-Id: I39b87f3d5daa98f65de2c0a44e348821a4930573
 2019-03-12 11:36:32 +00:00
Weiwei Yang 301ffee1bc MAPREDUCE-7191. JobHistoryServer should log exception when loading/parsing history file failed. Contributed by Jiandan Yang. 
(cherry picked from commit f0605146b3)
 2019-03-11 16:57:08 +08:00
Erik Krogen c924011f94 HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam. 
(cherry-picked from commit 1bc282e0b3)
(cherry-picked from commit 6d076dd5e8)
(cherry-picked from commit b0b8000240)
 2019-03-07 08:45:20 -08:00
Jason Lowe 32f96b6be4 YARN-5714. ContainerExecutor does not order environment map. Contributed by Remi Catherinot and Jim Brennan 
(cherry picked from commit 8e728f39c9)
 2019-03-07 15:10:52 +00:00
Wei-Chiu Chuang e23a448e0e HDFS-14314. fullBlockReportLeaseId should be reset after registering to NN. Contributed by star. 
(cherry picked from commit 387dbe587a)
(cherry picked from commit e58ccca3ce)
(cherry picked from commit d951497f57)
 2019-03-04 10:46:16 -08:00
Erik Krogen b170626c74 HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao. 2019-03-01 08:12:58 -08:00
Sunil G c139fe7a37 YARN-9168. DistributedShell client timeout should be -1 by default. Contributed by Zhankun Tang. 
(cherry picked from commit 6cec90653d)
 2019-02-25 15:30:17 +05:30
Daryn Sharp 4eccf2a3cc HADOOP-15813. Enable more reliable SSL connection reuse. Contributed by Daryn Sharp. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a87e458432)
(cherry picked from commit ae8839e6e8)
(cherry picked from commit 704330a616)
 2019-02-20 18:22:20 -08:00
Adam Antal bebbdd290e
YARN-9283. Javadoc of LinuxContainerExecutor#addSchedPriorityCommand has a wrong property name as reference 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9385ec45d7)
 2019-02-15 18:59:02 +09:00
Masatake Iwasaki ba035dda76 YARN-9282. Typo in javadoc of class LinuxContainerExecutor: hadoop.security.authetication should be 'authentication'. Contributed by Charan Hebri. 
(cherry picked from commit e0ab1bdece)
 2019-02-09 00:34:22 +09:00
Eric E Payne f3ff6d3aa7 YARN-7171: RM UI should sort memory / cores numerically. Contributed by Ahmed Hussein 
(cherry picked from commit d1ca9432dd)
 2019-02-07 21:03:43 +00:00
Sunil G ec0bed1008 YARN-9206. RMServerUtils does not count SHUTDOWN as an accepted state. Contributed by Kuhu Shukla. 
(cherry picked from commit 6ffe6ea899)
 2019-02-07 19:09:19 +05:30
Kitti Nanasi ad131757f2 HDFS-14231. DataXceiver#run() should not log exceptions caused by InvalidToken exception as an error. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 49ddd8a6ed)
(cherry picked from commit 38541b2375)
(cherry picked from commit c25b641cec)
 2019-02-05 15:15:53 -08:00
Todd Lipcon 02cfd5fa35
HADOOP-15549. Upgrade to commons-configuration 2.1 regresses task CPU consumption. 
Contributed by Todd Lipcon.

(cherry picked from commit 1f21be2d02)
 2019-02-05 19:57:08 +00:00
Akira Ajisaka 875b8c390b
HADOOP-16089. AliyunOSS: update oss-sdk version to 3.4.1. Contributed by wujinhu. 
(cherry picked from commit 0f9aa5b4d5)
 2019-02-04 14:48:25 +09:00
Eric E Payne 50ef25464f YARN-6616: YARN AHS shows submitTime for jobs same as startTime. Contributed by Prabhu Joseph 
(cherry picked from commit 04105bbfdb)
 2019-01-29 18:12:36 +00:00
Weiwei Yang d182c81938 YARN-9237. NM should ignore sending finished apps to RM during RM fail-over. Contributed by Jiandan Yang. 
(cherry picked from commit 4f63ffe444)
 2019-01-29 11:07:51 +08:00
Masatake Iwasaki 00eee37a93 HDFS-12729. Document special paths in HDFS. Contributed by Masatake Iwasaki. 
(cherry picked from commit dc5af4c6b7)
 2019-01-29 02:31:59 +09:00
Jonathan Hung bce8f28bec YARN-9222. Print launchTime in ApplicationSummary 
(cherry picked from commit 6cace58e21)
(cherry picked from commit bf760e7e81)
(cherry picked from commit 80a08c29adc9d07974bbe8087abcec750d5f7916)
 2019-01-25 13:51:21 -08:00
Dinesh Chitlangia 36ff7a4d02 HDFS-14228. Incorrect getSnapshottableDirListing() javadoc. Contributed by Dinesh Chitlangia. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 45caeee6cf)
(cherry picked from commit eb77e60163)
(cherry picked from commit b63941516f)
 2019-01-25 07:38:49 -08:00
Haibo Chen 3368f141b2 YARN-7088. Add application launch time to Resource Manager REST API. (Kanwaljeet Sachdev via Haibo Chen) 
(cherry picked from commit bb92bfb4ef)
(cherry picked from commit 3a764250262e745a6762dd6551b0d0a6af39e891)
 2019-01-24 15:58:27 -08:00
Sunil G d11fdbb7ac YARN-8961. [UI2] Flow Run End Time shows 'Invalid date'. Contributed by Akhil PB 
(cherry picked from commit c726445990)
 2019-01-24 15:03:16 +05:30
Takanobu Asanuma 5b8a99bf49 HDFS-14218. EC: Ls -e throw NPE when directory ec policy is disabled. Contributed by Ayush Saxena. 
(cherry picked from commit 7d6792e5d2)
 2019-01-23 10:06:30 +09:00
Weiwei Yang 0e3d5f3064 YARN-9210. RM nodes web page can not display node info. Contributed by Jiandan Yang. 
(cherry picked from commit d43df31751)
 2019-01-22 11:15:03 +08:00
Chen Liang 80b9fecaaf HADOOP-15481. Emit FairCallQueue stats as metrics. Contributed by Christopher Gregorian. 2019-01-18 11:45:22 -08:00
Akira Ajisaka 98345dcca4
HDFS-14213. Remove Jansson from BUILDING.txt. Contributed by Dinesh Chitlangia. 
(cherry picked from commit 96a84b61fb)
(cherry picked from commit dea75af9a7)
 2019-01-18 15:35:05 +09:00
Akira Ajisaka 1a38a61716
YARN-8747. [UI2] YARN UI2 page loading failed due to js error under some time zone configuration. Contributed by collinma. 
(cherry picked from commit 104ef5df36)
 2019-01-16 14:38:55 +09:00
Inigo Goiri 174fa73f99 HADOOP-16045. Don't run TestDU on Windows. Contributed by Lukas Majercak. 
(cherry picked from commit 35fa3bd685)
 2019-01-11 18:16:44 -08:00
Akira Ajisaka 9fd5a6b66b
HADOOP-16016. TestSSLFactory#testServerWeakCiphers fails on Java 1.8.0_191 or upper 
(cherry picked from commit d4ca907da6)
 2019-01-11 13:05:31 +09:00
Jason Lowe 642213a74a Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh." 
This reverts commit a4f8421324.
 2019-01-09 17:56:32 -06:00
Kai Xie a959aa61b4
HADOOP-16018. DistCp won't reassemble chunks when blocks per chunk > 0. 
Contributed by Kai Xie.

(cherry picked from commit 188bebbe7e)
 2019-01-08 13:35:41 +00:00
Weiwei Yang 009f5b55ad YARN-9173. FairShare calculation broken for large values after YARN-8833. Contributed Wilfred Spiegelenburg. 
(cherry picked from commit 91e9c9f96e)
 2019-01-08 14:09:49 +08:00
Weiwei Yang ad5dd32843 HADOOP-16030. AliyunOSS: bring fixes back from HADOOP-15671. Contributed by wujinhu. 
(cherry picked from commit f87b3b11c4)
 2019-01-07 17:03:22 +08:00
Sunil G a450d58b49
Revert "HADOOP-15759. AliyunOSS: Update oss-sdk version to 3.0.0. Contributed by Jinhu Wu." 
This reverts commit e4fca6aae4.

Revert "HADOOP-15671. AliyunOSS: Support Assume Roles in AliyunOSS. Contributed by Jinhu Wu."

This reverts commit 2b635125fb.

(cherry picked from commit 1f425271a7)
 2019-01-05 17:50:37 +09:00