Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
13,164 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

3335 Commits

Author SHA1 Message Date
Andrew Wang dab1deb9a0 HADOOP-14293. Initialize FakeTimer with a less trivial value. 
(cherry picked from commit be144117a8)
 2017-04-10 11:37:31 -07:00
John Zhuge ff2870310e HADOOP-14174. Set default ADLS access token provider type to ClientCredential. Contributed by John Zhuge. 
Signed-off-by: John Zhuge <jzhuge@apache.org>
(cherry picked from commit 56e81f2a20)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
 2017-04-07 15:01:29 -07:00
Zhe Zhang c85026038a HADOOP-14276. Add a nanosecond API to Time/Timer/FakeTimer. Contributed by Erik Krogen. 
(cherry picked from commit 95b7f1d29a)
 2017-04-07 10:39:40 -07:00
Andrew Wang ed8f738be5 HADOOP-14104. Client should always ask namenode for kms provider path. Contributed by Rushabh S Shah. 2017-04-06 14:38:22 -07:00
Xiaoyu Yao 9016614eb1 HDFS-11302. Improve Logging for SSLHostnameVerifier. Contributed by Chen Liang. 
(cherry picked from commit 32bb36b750)
 2017-04-05 18:30:33 -07:00
Jonathan Eagles 60e65a3e55 HADOOP-14216. Addendum to Improve Configuration XML Parsing Performance (jeagles) 2017-03-30 14:15:37 -05:00
Mingliang Liu 3d5e9e61d5 HADOOP-14247. FileContextMainOperationsBaseTest should clean up test root path. Contributed by Mingliang Liu 
(cherry picked from commit 13c766b62c)
 2017-03-29 11:19:28 -07:00
Jonathan Eagles 719ae5f2e8 HADOOP-14216. Improve Configuration XML Parsing Performance (jeagles) 2017-03-29 10:14:49 -05:00
John Zhuge ef43a62482 HADOOP-14038. Rename ADLS credential properties. Contributed by John Zhuge. 
(cherry picked from commit 253e3e78ab)
 2017-03-27 23:07:08 -07:00
Akira Ajisaka 643298b282
HADOOP-14218. Replace assertThat with assertTrue in MetricsAsserts. 
(cherry picked from commit 448ec81fd7)
 2017-03-28 14:33:36 +09:00
Andrew Wang 10deafee82 HDFS-11170. Add builder-based create API to FileSystem. Contributed by SammiChen and Wei Zhou. 2017-03-27 14:35:03 -07:00
Jonathan Eagles e41aac0cd4 Configuration#get return value optimization (jeagles) 2017-03-27 12:52:59 -05:00
Jonathan Eagles 675bc97f18 Delay construction of PreCondition.check failure message in Configuration#set (jeagles) 
(cherry picked from commit 858d597be0)
 2017-03-27 11:02:41 -05:00
Junping Du 3ad4d37363 Add COMMON/MAPREDUCE/YARN jdiff of 2.8.0 release. 
(cherry picked from commit 6df029db36)
 2017-03-25 14:33:20 -07:00
Junping Du e796105a6f Add 2.8.0 RELEASENOTEs.md and CHANGES.md. 
(cherry picked from commit d8e6d6df4c)
 2017-03-25 14:32:47 -07:00
Daniel Templeton bbd08bb7a4 YARN-6360. Prevent FS state dump logger from cramming other log files 
(Contributed by Yufei Gu via Daniel Templeton)

(cherry picked from commit d49d1862a6)
 2017-03-24 11:31:58 -07:00
Andrew Wang 96fe940e59 HADOOP-14211. FilterFs and ChRootedFs are too aggressive about enforcing 'authorityNeeded'. Contributed by Erik Krogen. 
(cherry picked from commit 0e556a5ba6)
 2017-03-24 11:12:31 -07:00
John Zhuge d946dfcbe1 HADOOP-14195. CredentialProviderFactory$getProviders is not thread-safe. Contributed by Vihang Karajgaonkar. 
(cherry picked from commit 128015584d)
 2017-03-23 13:57:41 -07:00
Mingliang Liu 1e64c582f5 HADOOP-14214. DomainSocketWatcher::add()/delete() should not self interrupt while looping await(). Contributed by Mingliang Liu 
(cherry picked from commit d35e79abc2)
 2017-03-23 13:34:53 -07:00
Ravi Prakash 87339d7691 HADOOP-14213. Move Configuration runtime check for hadoop-site.xml to initialization. Contributed by Jonathan Eagles 
(cherry picked from commit 595f62e362)
 2017-03-23 09:30:22 -07:00
Zhe Zhang f98f4bb6ca HADOOP-9631. ViewFs should use underlying FileSystem's server side defaults. Contributed by Lohit Vijayarenu and Erik Krogen. 
(cherry picked from commit 59d69257a8)
 2017-03-22 22:32:07 -07:00
John Zhuge 84e2c11f96 HADOOP-14205. No FileSystem for scheme: adl. Contributed by John Zhuge. 2017-03-21 08:23:30 -07:00
John Zhuge 2c5de0c60e HDFS-11132. Allow AccessControlException in contract tests when getFileStatus on subdirectory of existing files. Contributed by Vishwajeet Dusane 
(cherry picked from commit 19f373a46b)

Conflicts:
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
 2017-03-20 21:16:50 -07:00
Erik Krogen e44569b736 HADOOP-14169. Implement listStatusIterator, listLocatedStatus for ViewFs. Contributed by Erik Krogen. 
(cherry picked from commit e1a99802fc)
 2017-03-17 17:40:14 -07:00
Andrew Wang 7cc88eca43 HDFS-11516. Admin command line should print message to stderr in failure case. Contributed by Kai Sasaki. 
(cherry picked from commit 92ea6d74ec)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/cli/util/ErasureCodingCliCmdExecutor.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testErasureCodingConf.xml
 2017-03-17 11:34:43 -07:00
Mingliang Liu ed0d426a88 HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu 2017-03-14 15:00:59 -07:00
Ray Chiang fa59f4e490 YARN-6042. Dump scheduler and queue state information into FairScheduler DEBUG log. (Yufei Gu via rchiang) 2017-03-14 14:45:13 -07:00
Jing Zhao e03d8ff488 HDFS-11395. RequestHedgingProxyProvider#RequestHedgingInvocationHandler hides the Exception thrown from NameNode. Contributed by Nandakumar. 
(cherry picked from commit 55796a0946)
 2017-03-13 14:26:44 -07:00
Jian He 8c44a9db21 HADOOP-14062. ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled. Contributed by Steven Rand 2017-03-10 09:26:19 -08:00
Mingliang Liu 9ea6b24c49 HADOOP-13946. Document how HDFS updates timestamps in the FS spec; compare with object stores. Contributed by Steve Loughran 
(cherry picked from commit fd26783aaf)
 2017-03-10 00:21:58 -08:00
Lei Xu 01624f0c1b HADOOP-13956. Read ADLS credentials from Credential Provider. (John Zhuge via lei) 
(cherry picked from commit e015b56319)
 2017-03-09 15:16:28 -08:00
Arpit Agarwal 5cdcda34a2 HDFS-11477. Simplify file IO profiling configuration. Contributed by Hanisha Koneru. 2017-03-09 14:34:10 -08:00
Jian He ea8d53de6a Revert "HADOOP-14062. ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled. Contributed by Steven Rand" 
This reverts commit 36eda5af75.
 2017-03-08 13:20:45 -08:00
John Zhuge f1fd20be82 Revert "HADOOP-13606 swift FS to add a service load metadata file. Contributed by Steve Loughran" 
This reverts commit 53a12fa721.

(cherry picked from commit 98142d2f72)

Change-Id: Ia5fa901fa97391cd46d41f59cd05f7a19c7c7532
 2017-03-08 11:46:27 -08:00
Jian He 36eda5af75 HADOOP-14062. ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled. Contributed by Steven Rand 2017-03-08 10:49:07 -08:00
Mingliang Liu ff050866bf HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth 
(cherry picked from commit 686823529b)
 2017-03-06 17:35:06 -08:00
Mingliang Liu 8323e651a1 Revert "HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth" 
This reverts commit b26870c58e.
 2017-03-06 17:18:36 -08:00
Ray Chiang d943480147 YARN-5665. Enhance documentation for yarn.resourcemanager.scheduler.class property. (Yufei Gu via rchiang) 
(cherry picked from commit d9dc444dc7)
 2017-03-06 14:05:09 -08:00
Akira Ajisaka ed55bcee1f
HADOOP-14108. CLI MiniCluster: add an option to specify NameNode HTTP port. Contributed by Takanobu Asanuma. 
(cherry picked from commit 209ecd1a5c)
 2017-03-07 01:36:10 +09:00
Mingliang Liu b26870c58e HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth 2017-03-03 17:09:05 -08:00
Jason Lowe 9b949f49e3 YARN-6274. Documentation refers to incorrect nodemanager health checker interval property. Contributed by WeiWei Yang 
(cherry picked from commit 05237636d3)
 2017-03-03 08:01:40 -06:00
Harsh J d4c13df515 HADOOP-6801. io.sort.mb and io.sort.factor were renamed and moved to mapreduce but are still in CommonConfigurationKeysPublic.java and used in SequenceFile.java. 
This closes #146

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit eb5a17954a)
 2017-03-03 18:39:47 +09:00
Andrew Wang d737a26c5a Revert "HDFS-11477. Simplify file IO profiling configuration. Contributed by Hanisha Koneru." 
This reverts commit 5e760f8dc4.
 2017-03-02 16:21:32 -08:00
Arpit Agarwal 5e760f8dc4 HDFS-11477. Simplify file IO profiling configuration. Contributed by Hanisha Koneru. 2017-03-02 15:38:32 -08:00
Brahma Reddy Battula a545ee4d7f HDFS-11432. Federation : Support fully qualified path for Quota/Snapshot/cacheadmin/cryptoadmin commands. Contributed by Brahma Reddy Battula. 
(cherry picked from commit dcd03df9f9)
 2017-03-01 10:47:47 +05:30
Yiqun Lin 6cd39803ba HADOOP-14119. Remove unused imports from GzipCodec.java. Contributed by Yiqun Lin. 
(cherry picked from commit 9db2e0c8d4)
 2017-02-27 18:41:47 +08:00
Xuan 5c509f5f0c HADOOP-14116:FailoverOnNetworkExceptionRetry does not wait when failover 
on certain exception. Contributed by Jian He

(cherry picked from commit 289bc50e66)
 2017-02-24 11:43:22 -08:00
Harsh J c097d0343c HADOOP-13817. Add a finite shell command timeout to ShellBasedUnixGroupsMapping. (harsh) 
(cherry picked from commit e8694deb6a)
 2017-02-24 21:34:31 +05:30
Akira Ajisaka b5a8c49cf3 HADOOP-14097. Remove Java6 specific code from GzipCodec.java. Contributed by Elek, Marton. 
(cherry picked from commit 50decd3613)
 2017-02-25 00:29:40 +09:00
Wei-Chiu Chuang 12ad077aa6 HADOOP-13233. help of stat is confusing. Contributed by Attila Bukor. 
(cherry picked from commit cc45da79fd)
 2017-02-13 10:17:41 -08:00
Lei Xu 6d62d0ea87 HADOOP-13204. Add support for SSE-KMS and SSE-C in s3a filesystem. (Steve Moist via lei) 2017-02-11 15:02:19 -08:00
Arpit Agarwal 4c5655ef1a HDFS-11403. Addendum fix for branch-2. 2017-02-11 01:25:43 -08:00
Arpit Agarwal 32a0c2b5e7 HDFS-11403. Zookeper ACLs on NN HA enabled clusters should be handled consistently. Contributed by Hanisha Koneru. 2017-02-11 01:18:07 -08:00
Kihwal Lee 5de6f1b791 HADOOP-14033. Reduce fair call queue lock contention. Contributed by Daryn Sharp. 
(cherry picked from commit 0c01cf5798)
 2017-02-09 16:20:06 -06:00
Kihwal Lee d008b55153 HADOOP-14034. Allow ipc layer exceptions to selectively close connections. Contributed by Daryn Sharp. 2017-02-09 10:55:40 -06:00
Kihwal Lee 2039d021f9 HADOOP-14032. Reduce fair call queue priority inversion. Contributed by Daryn Sharp. 
(cherry picked from commit a0bfb41504)
 2017-02-09 10:13:37 -06:00
Junping Du 33e7c3044f YARN-6145. Improve log message on fail over. Contributed by Jian He. 
(cherry picked from commit eec52e158b)
 2017-02-08 02:15:11 -08:00
Junping Du b7d59e790b HADOOP-12097. Allow port range to be specified while starting webapp. Contributed by Varun Saxena. 
(cherry picked from commit cce35c3815)

Conflicts:

	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java
 2017-02-05 20:13:49 -08:00
Akira Ajisaka be9b02e0b6 HADOOP-14053. Update the link to HTrace SpanReceivers. Contributed by Yiqun Lin. 
(cherry picked from commit cb8f3f3adf)
 2017-02-04 10:59:18 +09:00
Xiao Chen 05ed48b75a HADOOP-14044. Synchronization issue in delegation token cancel functionality. Contributed by Hrishikesh Gadre. 
(cherry picked from commit ba75bc7593)
 2017-02-03 17:27:35 -08:00
Jason Lowe 4703f5d20b YARN-5641. Localizer leaves behind tarballs after container is complete. Contributed by Eric Badger 2017-01-27 15:25:57 +00:00
Xiaoyu Yao 6b602c6e34 HADOOP-14029. Fix KMSClientProvider for non-secure proxyuser use case. Contributed by Xiaoyu Yao. 
(cherry picked from commit 2034315763)
 2017-01-26 20:36:43 -08:00
Jason Lowe b806201f0a Revert "YARN-5641. Localizer leaves behind tarballs after container is complete. Contributed by Eric Badger" 
This reverts commit 45407acaed.
 2017-01-26 14:42:22 +00:00
Jason Lowe 45407acaed YARN-5641. Localizer leaves behind tarballs after container is complete. Contributed by Eric Badger 
(cherry picked from commit 9e19f758c1)
 2017-01-25 21:54:17 +00:00
Xiaoyu Yao 9fa98cc45e HADOOP-13988. KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser. Contributed by Greg Senia and Xiaoyu Yao. 
(cherry picked from commit a46933e8ce)
 2017-01-25 13:34:42 -08:00
Steve Loughran bb46d40558 HADOOP-13433 Race in UGI.reloginFromKeytab. Contributed by Duo Zhang. 2017-01-25 21:26:01 +00:00
Arpit Agarwal b4078e1d0e HDFS-11194. Maintain aggregated peer performance metrics on NameNode. 2017-01-24 19:34:33 -08:00
Eric Yang efe8f941a9 HADOOP-13119. Add ability to secure log servlet using proxy users. Contribute by Yuanbo Liu. 
(cherry picked from commit a847903b6e)
 2017-01-21 15:47:13 -08:00
Daniel Templeton be7e4a0805 YARN-6028. Add document for container metrics (Contributed by Weiwei Yang via Daniel Templeton) 
(cherry picked from commit f5839fd6df)
 2017-01-20 10:07:49 -08:00
Akira Ajisaka c6c29d0080 HADOOP-14001. Improve delegation token validity checking. 
(cherry picked from commit 1763467210)
 2017-01-19 17:58:44 +09:00
Arpit Agarwal 42a14b6db9 HDFS-10917. Collect peer performance statistics on DataNode. Contributed by Xiaobing Zhou. 2017-01-17 15:31:18 -08:00
Kihwal Lee 3017069a76 HADOOP-13976. Path globbing does not match newlines. Contributed by Eric Badger. 
(cherry picked from commit 706d630eb9)
 2017-01-17 15:12:50 -06:00
Arpit Agarwal 52fdd387ec HDFS-11299. Support multiple Datanode File IO hooks. Contributed by Hanisha Koneru. 2017-01-16 22:42:13 -08:00
Arpit Agarwal ae0082a3fc HDFS-11282. Document the missing metrics of DataNode Volume IO operations. Contributed by Yiqun Lin. 2017-01-16 22:24:49 -08:00
Akira Ajisaka 2d6be7ea23 HADOOP-13933. Add haadmin -getAllServiceState option to get the HA state of all the NameNodes/ResourceManagers. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit e407449ddb)
 2017-01-17 10:22:29 +09:00
Arpit Agarwal 954dae26cd HDFS-11337. (HDFS-10958 backport). Add instrumentation hooks around Datanode disk IO. 2017-01-16 13:53:30 -08:00
Lei Xu 00dec84cc2 HADOOP-13928. TestAdlFileContextMainOperationsLive.testGetFileContext1 runtime error. (John Zhuge via lei) 
(cherry picked from commit 15066a3e8ddacd36bf7be9fba0b10c350ae45668)
 2017-01-15 14:36:21 +08:00
Steve Loughran ec2f5b18ab HADOOP-13336 S3A to support per-bucket configuration. Contributed by Steve Loughran 2017-01-11 17:24:25 +00:00
Wei-Chiu Chuang 76d62b0632 HADOOP-13953. Make FTPFileSystem's data connection mode and transfer mode configurable. Contributed by Xiao Chen. 
(cherry picked from commit 0a212a40fc)

Conflicts:
   hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
 2017-01-09 15:59:36 -08:00
Jason Lowe db947fb870 HADOOP-13578. Add Codec for ZStandard Compression. Contributed by churro morales 2017-01-04 14:46:44 +00:00
Akira Ajisaka 3e2d26a97f HADOOP-12733. Remove references to obsolete io.seqfile configuration variables. Contributed by Ray Chiang. 
(cherry picked from commit 01d31fe938)
 2017-01-04 14:15:00 +09:00
Mingliang Liu 1b8cb0b9c9 Revert "HADOOP-13946. Document how HDFS updates timestamps in the FS spec; compare with object stores. Contributed by Steve Loughran" 
This reverts commit a447efa2ab.
 2017-01-03 14:45:19 -08:00
Mingliang Liu a447efa2ab HADOOP-13946. Document how HDFS updates timestamps in the FS spec; compare with object stores. Contributed by Steve Loughran 
(cherry picked from commit 451efb08fe)
 2017-01-03 13:09:29 -08:00
Robert Kanter 2f32a6aea2 YARN-5529. Create new DiskValidator class with metrics (yufeigu via rkanter) 
(cherry picked from commit 591fb15944)
 2017-01-03 12:14:13 -08:00
Brahma Reddy Battula d6e5e4288c HADOOP-13883. Addendum patch to correct the message and alphabetized with the earlier patch. Contributed by Yiqun Lin. 
(cherry picked from commit 6938b67711)
 2016-12-31 16:17:15 +05:30
Akira Ajisaka 74dd2770cc HADOOP-13943. TestCommonConfigurationFields#testCompareXmlAgainstConfigurationClass fails after HADOOP-13863. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 0665c5f09a)
 2016-12-28 01:41:56 +09:00
Daniel Templeton 28b605dd1a HADOOP-13940. Document the missing envvars commands (Contributed by Yiqun Lin via Daniel Templeton) 
(cherry picked from commit d432173737a532b7106682116013b751ba162ff0)
 2016-12-27 07:19:37 -08:00
Mingliang Liu edac82737a HADOOP-13863. Azure: Add a new SAS key mode for WASB. Contributed by Dushyanth 
(cherry picked from commit e92a77099b)
 2016-12-22 20:33:51 -08:00
Akira Ajisaka 897b4a6a77 HDFS-11261. Document missing NameNode metrics. Contributed by Yiqun Lin. 
(cherry picked from commit f6e2521eb2)
 2016-12-21 14:20:15 +09:00
Wei-Chiu Chuang 84ec079911 HADOOP-13508. FsPermission string constructor does not recognize sticky bit. Contributed by Atul Sikaria. 2016-12-15 13:49:02 -08:00
Jason Lowe ee12363f2d HADOOP-13709. Ability to clean up subprocesses spawned by Shell when the process exits. Contributed by Eric Badger 
(cherry picked from commit 631f1daee3)
 2016-12-15 20:54:42 +00:00
Jason Lowe 81be6644ce Revert "HADOOP-13709. Ability to clean up subprocesses spawned by Shell when the process exits. Contributed by Eric Badger" 
This reverts commit 0af67aa200.
 2016-12-15 14:52:43 +00:00
Xiaoyu Yao 85083567be HADOOP-13890. Maintain HTTP/host as SPNEGO SPN support and fix KerberosName parsing. Contributed by Xiaoyu Yao. 
(cherry picked from commit f5e0bd30fd)
 2016-12-14 13:59:24 -08:00
Akira Ajisaka 190b2dca62 HDFS-11204. Document the missing options of hdfs zkfc command. Contributed by Yiqun Lin. 
(cherry picked from commit 72bff192cd)
 2016-12-14 18:53:24 +09:00
Jason Lowe 0af67aa200 HADOOP-13709. Ability to clean up subprocesses spawned by Shell when the process exits. Contributed by Eric Badger 
(cherry picked from commit 9947aeb60c)
 2016-12-13 22:57:41 +00:00
Wei-Chiu Chuang 6c93c1870c HADOOP-13824. FsShell can suppress the real error if no error message is present. Contributed by John Zhuge. 
(cherry picked from commit b606e025f1)

Conflicts:
   hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShell.java
 2016-12-09 15:39:12 -08:00
Brahma Reddy Battula 73d5de5594 HADOOP-13883. Add description of -fs option in generic command usage. Contributed By Yiqun Lin. 
(cherry picked from commit 80b8023276)
 2016-12-09 18:42:49 +05:30
Brahma Reddy Battula c73d8399ff HADOOP-13867. FilterFileSystem should override rename(.., options) to take effect of Rename options called via FilterFileSystem implementations. Contributed By Vinayakumar B. 
(cherry picked from commit 0ef796174e)
 2016-12-08 18:58:54 +05:30
Ming Ma d2656dc5a6 HDFS-10206. Datanodes not sorted properly by distance when the reader isn't a datanode. (Nandakumar via mingma) 2016-12-07 08:27:17 -08:00
Karthik Kambatla 8773cd5f76 HADOOP-13859. TestConfigurationFieldsBase fails for fields that are DEFAULT values of skipped properties. (Haibo Chen via kasha) 
(cherry picked from commit c8d0a049b0)
 2016-12-06 14:16:38 -08:00