Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,845 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

3356 Commits

Author SHA1 Message Date
Kihwal Lee 3017069a76 HADOOP-13976. Path globbing does not match newlines. Contributed by Eric Badger. 
(cherry picked from commit 706d630eb9)
 2017-01-17 15:12:50 -06:00
Arpit Agarwal 52fdd387ec HDFS-11299. Support multiple Datanode File IO hooks. Contributed by Hanisha Koneru. 2017-01-16 22:42:13 -08:00
Arpit Agarwal ae0082a3fc HDFS-11282. Document the missing metrics of DataNode Volume IO operations. Contributed by Yiqun Lin. 2017-01-16 22:24:49 -08:00
Akira Ajisaka 2d6be7ea23 HADOOP-13933. Add haadmin -getAllServiceState option to get the HA state of all the NameNodes/ResourceManagers. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit e407449ddb)
 2017-01-17 10:22:29 +09:00
Jitendra Pandey ffb4c22f6e HDFS-11307. The rpc to portmap service for NFS has hardcoded timeout. Contributed by Mukul Kumar Singh. 2017-01-16 15:17:29 -08:00
Arpit Agarwal 954dae26cd HDFS-11337. (HDFS-10958 backport). Add instrumentation hooks around Datanode disk IO. 2017-01-16 13:53:30 -08:00
Lei Xu 00dec84cc2 HADOOP-13928. TestAdlFileContextMainOperationsLive.testGetFileContext1 runtime error. (John Zhuge via lei) 
(cherry picked from commit 15066a3e8ddacd36bf7be9fba0b10c350ae45668)
 2017-01-15 14:36:21 +08:00
Steve Loughran ec2f5b18ab HADOOP-13336 S3A to support per-bucket configuration. Contributed by Steve Loughran 2017-01-11 17:24:25 +00:00
Arun Suresh 8e5de45e08 HADOOP-13903. Improvements to KMS logging to help debug authorization errors. (Tristan Stevens via asuresh) 
(cherry picked from commit be529dade1)
 2017-01-11 00:27:14 -08:00
Wei-Chiu Chuang 76d62b0632 HADOOP-13953. Make FTPFileSystem's data connection mode and transfer mode configurable. Contributed by Xiao Chen. 
(cherry picked from commit 0a212a40fc)

Conflicts:
   hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
 2017-01-09 15:59:36 -08:00
Jason Lowe db947fb870 HADOOP-13578. Add Codec for ZStandard Compression. Contributed by churro morales 2017-01-04 14:46:44 +00:00
Akira Ajisaka 3e2d26a97f HADOOP-12733. Remove references to obsolete io.seqfile configuration variables. Contributed by Ray Chiang. 
(cherry picked from commit 01d31fe938)
 2017-01-04 14:15:00 +09:00
Mingliang Liu 1b8cb0b9c9 Revert "HADOOP-13946. Document how HDFS updates timestamps in the FS spec; compare with object stores. Contributed by Steve Loughran" 
This reverts commit a447efa2ab.
 2017-01-03 14:45:19 -08:00
Mingliang Liu a447efa2ab HADOOP-13946. Document how HDFS updates timestamps in the FS spec; compare with object stores. Contributed by Steve Loughran 
(cherry picked from commit 451efb08fe)
 2017-01-03 13:09:29 -08:00
Robert Kanter 2f32a6aea2 YARN-5529. Create new DiskValidator class with metrics (yufeigu via rkanter) 
(cherry picked from commit 591fb15944)
 2017-01-03 12:14:13 -08:00
Brahma Reddy Battula d6e5e4288c HADOOP-13883. Addendum patch to correct the message and alphabetized with the earlier patch. Contributed by Yiqun Lin. 
(cherry picked from commit 6938b67711)
 2016-12-31 16:17:15 +05:30
Akira Ajisaka 74dd2770cc HADOOP-13943. TestCommonConfigurationFields#testCompareXmlAgainstConfigurationClass fails after HADOOP-13863. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 0665c5f09a)
 2016-12-28 01:41:56 +09:00
Daniel Templeton 28b605dd1a HADOOP-13940. Document the missing envvars commands (Contributed by Yiqun Lin via Daniel Templeton) 
(cherry picked from commit d432173737a532b7106682116013b751ba162ff0)
 2016-12-27 07:19:37 -08:00
Mingliang Liu edac82737a HADOOP-13863. Azure: Add a new SAS key mode for WASB. Contributed by Dushyanth 
(cherry picked from commit e92a77099b)
 2016-12-22 20:33:51 -08:00
Akira Ajisaka 897b4a6a77 HDFS-11261. Document missing NameNode metrics. Contributed by Yiqun Lin. 
(cherry picked from commit f6e2521eb2)
 2016-12-21 14:20:15 +09:00
Wei-Chiu Chuang 84ec079911 HADOOP-13508. FsPermission string constructor does not recognize sticky bit. Contributed by Atul Sikaria. 2016-12-15 13:49:02 -08:00
Jason Lowe ee12363f2d HADOOP-13709. Ability to clean up subprocesses spawned by Shell when the process exits. Contributed by Eric Badger 
(cherry picked from commit 631f1daee3)
 2016-12-15 20:54:42 +00:00
Jason Lowe 81be6644ce Revert "HADOOP-13709. Ability to clean up subprocesses spawned by Shell when the process exits. Contributed by Eric Badger" 
This reverts commit 0af67aa200.
 2016-12-15 14:52:43 +00:00
Xiaoyu Yao 85083567be HADOOP-13890. Maintain HTTP/host as SPNEGO SPN support and fix KerberosName parsing. Contributed by Xiaoyu Yao. 
(cherry picked from commit f5e0bd30fd)
 2016-12-14 13:59:24 -08:00
Akira Ajisaka 190b2dca62 HDFS-11204. Document the missing options of hdfs zkfc command. Contributed by Yiqun Lin. 
(cherry picked from commit 72bff192cd)
 2016-12-14 18:53:24 +09:00
Jason Lowe 0af67aa200 HADOOP-13709. Ability to clean up subprocesses spawned by Shell when the process exits. Contributed by Eric Badger 
(cherry picked from commit 9947aeb60c)
 2016-12-13 22:57:41 +00:00
Xiaoyu Yao 48853c7319 HADOOP-13565. KerberosAuthenticationHandler#authenticate should not rebuild SPN based on client request. Contributed by Xiaoyu Yao. 
(cherry picked from commit 4c38f11cec)
 2016-12-09 21:38:14 -08:00
Wei-Chiu Chuang 6c93c1870c HADOOP-13824. FsShell can suppress the real error if no error message is present. Contributed by John Zhuge. 
(cherry picked from commit b606e025f1)

Conflicts:
   hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShell.java
 2016-12-09 15:39:12 -08:00
Brahma Reddy Battula 73d5de5594 HADOOP-13883. Add description of -fs option in generic command usage. Contributed By Yiqun Lin. 
(cherry picked from commit 80b8023276)
 2016-12-09 18:42:49 +05:30
Brahma Reddy Battula c73d8399ff HADOOP-13867. FilterFileSystem should override rename(.., options) to take effect of Rename options called via FilterFileSystem implementations. Contributed By Vinayakumar B. 
(cherry picked from commit 0ef796174e)
 2016-12-08 18:58:54 +05:30
Ming Ma d2656dc5a6 HDFS-10206. Datanodes not sorted properly by distance when the reader isn't a datanode. (Nandakumar via mingma) 2016-12-07 08:27:17 -08:00
Karthik Kambatla 8773cd5f76 HADOOP-13859. TestConfigurationFieldsBase fails for fields that are DEFAULT values of skipped properties. (Haibo Chen via kasha) 
(cherry picked from commit c8d0a049b0)
 2016-12-06 14:16:38 -08:00
Andrew Wang 6bd600eef2 HADOOP-13861. Spelling errors in logging and exceptions for code. Contributed by Grant Sohn. 
(cherry picked from commit 7b988e8899)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/util/GF256.java
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/KDiag.java
 2016-12-05 23:21:03 -08:00
Xiao Chen 7e58eec620 HADOOP-13847. KMSWebApp should close KeyProviderCryptoExtension. Contributed by John Zhuge. 
(cherry picked from commit 291df5c7fb)
 2016-12-05 09:35:51 -08:00
Akira Ajisaka 0c895e8a62 HADOOP-13812. Upgrade Tomcat to 6.0.48. Contributed by John Zhuge. 
(cherry picked from commit 00096dcc0c)
 2016-11-29 19:50:53 +09:00
Robert Kanter 285edf691d HADOOP-13838. KMSTokenRenewer should close providers (xiaochen via rkanter) 
(cherry picked from commit 47ca9e26fb)
 2016-11-28 18:12:53 -08:00
Brahma Reddy Battula 1c0fa25dc1 HADOOP-13820. Replace ugi.getUsername() with ugi.getShortUserName() in viewFS. Contributed By Brahma Reddy Battula. 
(cherry picked from commit ee3d437a33)
 2016-11-25 20:00:46 +05:30
Steve Loughran 2c2e77b3bb HADOOP-10776 Open up already widely-used APIs for delegation-token fetching & renewal to ecosystem projects. Contributed by Vinod Kumar Vavilapalli 2016-11-24 17:52:05 +00:00
Steve Loughran 36cc8dcdca HADOOP-13833 TestSymlinkHdfsFileSystem#testCreateLinkUsingPartQualPath2 fails after HADOOP-13605. Contributed by Brahma Reddy Battula. 2016-11-24 17:44:14 +00:00
Brahma Reddy Battula 1edabc4a46 HADOOP-11603. Metric Snapshot log can be changed #MetricsSystemImpl.java since all the services will be initialized. Contributed By Brahma Reddy Battula. 
(cherry picked from commit 13501053dd)
 2016-11-24 22:24:45 +05:30
Steve Loughran 89614f5355 HADOOP-13018. Make Kdiag check whether hadoop.token.files points to existent and valid files. Contributed by Ravi Prakash 2016-11-24 16:41:35 +00:00
Mingliang Liu 57b17ae800 HADOOP-13605. Clean up FileSystem javadocs, logging; improve diagnostics on FS load. Contributed by Steve Loughran 2016-11-23 17:15:55 -08:00
Jian He accd9136e4 HADOOP-11552. Allow handoff on the server side for RPC requests. Contributed by Siddharth Seth 
(cherry picked from commit 3d94da1e00)
 2016-11-23 16:17:36 -08:00
Mingliang Liu c11e0ef621 HADOOP-13166. Add getFileStatus("/") test to AbstractContractGetFileStatusTest. Contributed by Steve Loughran 
(cherry picked from commit 2bf9a15e8a)
 2016-11-22 14:31:25 -08:00
Mingliang Liu 9af7d1e298 HADOOP-13655. document object store use with fs shell and distcp. Contributed by Steve Loughran 2016-11-22 13:19:32 -08:00
Brahma Reddy Battula 2207a84a7d HADOOP-13646. Remove outdated overview.html. Contributed By Brahma Reddy Battula. 
(cherry picked from commit afcf8d38e7)
 2016-11-22 20:32:30 +05:30
Akira Ajisaka d29e3d0f52 HADOOP-13814. Sample configuration of KMS HTTP Authentication signature is misleading. Contributed by Masahiro Tanaka. 
(cherry picked from commit c65d6b6541)
 2016-11-21 11:28:33 +09:00
Mingliang Liu 92767f956f HADOOP-11601. Enhance FS spec & tests to mandate FileStatus.getBlocksize() >0 for non-empty files. Contributed by Steve Loughran 
(cherry picked from commit ae8849fe37)

This closes #50
 2016-11-17 14:26:44 -08:00
Kihwal Lee d3fa53a52b HADOOP-13742. Expose NumOpenConnectionsPerUser as a metric. Brahma Reddy Battula. 
(cherry picked from commit bd3735554f)
 2016-11-17 12:19:23 -06:00
Mingliang Liu 47526a25cc HADOOP-13822. Use GlobalStorageStatistics.INSTANCE.reset() at FileSystem#clearStatistics(). Contribute by Brahma Reddy Battula 
(cherry picked from commit aab9737a05)
 2016-11-16 20:09:07 -08:00
Xiao Chen efc9ffc3b6 HADOOP-13815. TestKMS#testDelegationTokensOpsSimple and TestKMS#testDelegationTokensOpsKerberized Fails in Trunk. 
(cherry picked from commit 61c0bedf77)
 2016-11-15 16:26:49 -08:00
Mingliang Liu 4e0fcff8ab HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu 
(cherry picked from commit 5af572b644)
 2016-11-15 11:36:08 -08:00
Mingliang Liu 0b428075a2 HADOOP-13810. Add a test to verify that Configuration handles &-encoded characters. Contributed by Steve Loughran 
This closes #158

(cherry picked from commit 04014c4c73)
 2016-11-14 19:41:32 -08:00
Zhe Zhang 818915aabb HDFS-10872. Add MutableRate metrics for FSNamesystemLock operations. Contributed by Erik Krogen. 2016-11-14 16:33:15 -08:00
Yongjun Zhang c684a00ed3 HADOOP-13720. Add more info to the msgs printed in AbstractDelegationTokenSecretManager. Contributed by Yongjun Zhang. 
(cherry picked from commit fd2f22adec)
 2016-11-10 23:06:09 -08:00
Xiao Chen ede909144d HADOOP-13590. Retry until TGT expires even if the UGI renewal thread encountered exception. 
(cherry picked from commit 367c3d4121)
 2016-11-09 09:09:13 -08:00
Subru Krishnan 6685a0a064 YARN-5833. Addendum patch to fix JDK7 compile issue. 2016-11-08 16:54:19 -08:00
Andrew Wang 4f25fd975e HADOOP-13789. Hadoop Common includes generated test protos in both jar and test-jar. Contributed by Sean Busbey. 
(cherry picked from commit e1c6ef2efa)
 2016-11-08 16:33:58 -08:00
Zhe Zhang 758638d08d HADOOP-13782. Make MutableRates metrics thread-local write, aggregate-on-read. Contributed by Erik Krogen. 
(cherry picked from commit 77c13c3857)
 2016-11-08 16:08:03 -08:00
Mingliang Liu 38b1ed18ef HADOOP-13802. Make generic options help more consistent, and aligned. Contributed by Grant Sohn 
(cherry picked from commit 2a65eb121e)
 2016-11-08 15:40:54 -08:00
Subru Krishnan 797c7bfab9 YARN-5833. Add validation to ensure default ports are unique in Configuration. (Konstantinos Karanasos via Subru). 
(cherry picked from commit 29e3b3417c)
 2016-11-08 14:39:09 -08:00
Zhe Zhang b245e9ce2f HADOOP-13804. MutableStat mean loses accuracy if add(long, long) is used. Contributed by Erik Krogen. 
(cherry picked from commit 3dbad5d823)
 2016-11-07 16:11:27 -08:00
Steve Loughran 29c6a0be0d HADOOP-13797 Remove hardcoded absolute path for ls. Contributed by Christine Koppelt 2016-11-07 12:36:25 +00:00
Xiao Chen f30d338766 HDFS-10756. Expose getTrashRoot to HTTPFS and WebHDFS. Contributed by Yuanbo Liu. 
(cherry picked from commit d8bab3dcb6)
 2016-11-04 18:08:46 -07:00
Xiaoyu Yao 9628f3da93 Revert "HADOOP-13565. KerberosAuthenticationHandler#authenticate should not rebuild SPN based on client request. Contributed by Xiaoyu Yao." 
This reverts commit 874402a295.
 2016-11-04 16:06:20 -07:00
Robert Kanter 0b36dcda7c HADOOP-7930. Kerberos relogin interval in UserGroupInformation should be configurable (xiaochen via rkanter) 2016-11-04 09:24:21 -07:00
Xiao Chen 1e29379244 HADOOP-12453. Support decoding KMS Delegation Token with its own Identifier. Contributed by Xiaoyu Yao. 
(cherry picked from commit 7154a20bcb)
 2016-11-03 13:10:37 -07:00
Andrew Wang 99d0b3eb2b HDFS-11080. Update HttpFS to use ConfigRedactor. Contributed by Sean Mackrory. 2016-11-02 19:11:22 -07:00
Arpit Agarwal 148bb3e737 HADOOP-13738. DiskChecker should perform some disk IO. 2016-11-01 18:09:12 -07:00
Ravi Prakash 045501a538 HADOOP-13773. Wrong HADOOP_CLIENT_OPTS in hadoop-env on branch-2. Contributed by Fei Hui 2016-11-01 10:49:04 -07:00
Steve Loughran c4ccafdaff HADOOP-13680. fs.s3a.readahead.range to use getLongBytes. Contributed by Abhishek Modi. 2016-10-31 20:52:49 +00:00
Wei-Chiu Chuang 9ba283c27d HDFS-11061. Update dfs -count -t command line help and documentation. Contributed by Yiqun Lin. 
(cherry picked from commit 112f04eb1b)
 2016-10-28 15:37:30 -07:00
Jason Lowe 0f224d405f HADOOP-13770. Shell.checkIsBashSupported swallowed an interrupted exception. Contributed by Wei-Chiu Chuang 
(cherry picked from commit c017171da0)
 2016-10-28 15:06:23 +00:00
Brahma Reddy Battula 5e760c89a1 HADOOP-13201. Print the directory paths when ViewFs denies the rename operation on internal dirs. Contributed by Tianyin Xiu 
(cherry picked from commit 0c837db8a8)
 2016-10-27 19:48:21 +05:30
Masatake Iwasaki 7e918637de HADOOP-13017. Implementations of InputStream.read(buffer, offset, bytes) to exit 0 if bytes==0. Contributed by Steve Loughran. 
(cherry picked from commit 0bdd263d82)
 2016-10-27 15:54:03 +09:00
Chris Nauroth 082d69ee66 HADOOP-13502. Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests. Contributed by Chris Nauroth. 
(cherry picked from commit 1f8490a5ba)
 2016-10-26 08:57:29 -07:00
Chris Nauroth 67e01f7218 HADOOP-13614. Purge some superfluous/obsolete S3 FS tests that are slowing test runs down. Contributed by Steve Loughran. 
(cherry picked from commit 9cad3e2350)
 2016-10-26 08:27:33 -07:00
Xiao Chen 4c5c6c8c44 HADOOP-13669. Addendum patch 2 for KMS Server should log exceptions before throwing. 
(cherry picked from commit 09ef97dccb)
 2016-10-25 09:59:45 -07:00
Chris Nauroth 05d772e297 HADOOP-13309. Document S3A known limitations in file ownership and permission model. Contributed by Chris Nauroth. 
(cherry picked from commit 309a43925c)
 2016-10-25 09:03:11 -07:00
Benoy Antony b913b62aaf HADOOP-12082. Support multiple authentication schemes via AuthenticationFilter. Contributed by Hrishikesh Gadre. 2016-10-25 08:30:02 -07:00
Chris Nauroth 5b7cbb5a3c HADOOP-13727. S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider. Contributed by Chris Nauroth. 
(cherry picked from commit d8fa1cfa67)
 2016-10-24 21:22:46 -07:00
Xiaoyu Yao 48c9064b33 HADOOP-13749. KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used. Contributed by Xiaoyu Yao. 
(cherry picked from commit d0a347984d)
 2016-10-23 10:59:01 -07:00
Xiaoyu Yao 68d91e0200 Revert "HDFS-10757. KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used. Contributed by Xiaoyu Yao." 
This reverts commit 0205ad5fa8.
 2016-10-23 08:26:01 -07:00
Xiaoyu Yao 0205ad5fa8 HDFS-10757. KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used. Contributed by Xiaoyu Yao. 
(cherry picked from commit be72372248)
 2016-10-21 14:25:29 -07:00
Chris Douglas 25f4327f0b HADOOP-13702. Add instrumented ReadWriteLock. Contributed by Jingcheng Du 
(cherry picked from commit ae8bccd509)
 2016-10-21 13:01:08 -07:00
Steve Loughran 385c1daa46 HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran. 2016-10-21 19:11:31 +01:00
Brahma Reddy Battula 2ab149f0e4 HADOOP-13236. truncate will fail when we use viewfilesystem. Contributed by Brahma Reddy Battula 
(cherry picked from commit f872c6bc03)
 2016-10-21 16:19:18 +05:30
Steve Loughran f0af3dee25 Revert "HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran." 
This reverts commit 42f8a1d6eb.
 2016-10-20 22:48:26 +01:00
Anu Engineer 1f384b617d HADOOP-13737. Cleanup DiskChecker interface. Contributed by Arpit Agarwal. 2016-10-20 13:40:01 -07:00
Anu Engineer 42f8a1d6eb HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran. 2016-10-20 12:47:57 -07:00
Steve Loughran ab36519b6f Revert "HADOOP-13236. truncate will fail when we use viewfilesystem. Contributed by Brahma Reddy Battula" 
This reverts commit bf6379fdd2.
 2016-10-20 19:32:10 +01:00
Brahma Reddy Battula bf6379fdd2 HADOOP-13236. truncate will fail when we use viewfilesystem. Contributed by Brahma Reddy Battula 
(cherry picked from commit f872c6bc03)
 2016-10-20 21:48:23 +05:30
Steve Loughran bc176961e6 HADOOP-13560. S3ABlockOutputStream to support huge (many GB) file writes. Contributed by Steve Loughran 2016-10-18 19:33:38 +01:00
Akira Ajisaka b2618685fb HADOOP-13522. Add %A and %a formats for fs -stat command to print permissions. Contributed by Alex Garbarini. 
(cherry picked from commit bedfec0c10)
 2016-10-18 15:03:33 +09:00
Andrew Wang 4ed7cf3b36 HADOOP-13724. Fix a few typos in site markdown documents. Contributed by Ding Fei. 
(cherry picked from commit 987ee51141)
 2016-10-17 13:32:39 -07:00
Allen Wittenauer fbdb23d2af HADOOP-11559. Add links to RackAwareness and InterfaceClassification to site index (Masatake Iwasaki via aw) 
(cherry picked from commit 7eeca90daa)

 Conflicts:
	hadoop-common-project/hadoop-common/CHANGES.txt
	hadoop-project/src/site/site.xml
 2016-10-17 13:32:28 -07:00
Eric Yang 7993fb5b81 HADOOP-13707. Skip authorization for anonymous user to access Hadoop 
web interface in non-secure environment.  (Yuanbo Liu via eyang)

    (cherry picked from commit dc308e98b9)
    With Addendum patch
 2016-10-17 07:57:45 -07:00
Eric Yang a4356bb7a5 HADOOP-13707. Skip authorization for anonymous user to access Hadoop 
web interface in non-secure environment.  (Yuanbo Liu via eyang)

    (cherry picked from commit dc308e98b9)
    With Addendum patch
 2016-10-16 20:32:48 -07:00
Eric Yang 439422fff9 HADOOP-13707. Skip authorization for anonymous user to access Hadoop 
web interface in non-secure environment.  (Yuanbo Liu via eyang)

(cherry picked from commit dc308e98b9)
With Addendum patch
 2016-10-15 21:23:21 -07:00
Masatake Iwasaki a13a607e20 HADOOP-13419. Fix javadoc warnings by JDK8 in hadoop-common package. Contributed by Kai Sasaki. 2016-10-16 12:44:19 +09:00