Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,752 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

3330 Commits

Author SHA1 Message Date
Chris Nauroth cf954d7a48 Revert "HADOOP-13081. add the ability to create multiple UGIs/subjects from one kerberos login. Contributed by Sergey Shelukhin." 
This reverts commit ec522a1907.
 2016-09-29 13:58:55 -07:00
Kihwal Lee 92336ea37e HADOOP-13537. Support external calls in the RPC call queue. Contributed by Daryn Sharp. 
(cherry picked from commit 236ac773c9)
 2016-09-29 13:54:30 -05:00
Steve Loughran 896df3f55a HADOOP-13164 Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories. Contributed by Rajesh Balamohan. 2016-09-29 17:00:34 +01:00
Steve Loughran 2b6b4fd779 HADOOP-13663 Index out of range in SysInfoWindows. Contributed by Inigo Goiri 2016-09-29 11:35:37 +01:00
Kihwal Lee baf8aac05c HADOOP-11780. Prevent IPC reader thread death. Contributed by Daryn Sharp. 
(cherry picked from commit e19b37ead2)
 2016-09-28 09:46:42 -05:00
Brahma Reddy Battula e9a34ae29c HDFS-9444. Add utility to find set of available ephemeral ports to ServerSocketUtil. Contributed by Masatake Iwasaki 2016-09-28 10:50:50 +05:30
Mingliang Liu 34ee18d73d HADOOP-13658. Replace config key literal strings with names I: hadoop common. Contributed by Chen Liang 
(cherry picked from commit 9a44a832a9)
 2016-09-27 17:39:36 -07:00
Wangda Tan 269401dc83 HADOOP-13544. JDiff reports unncessarily show unannotated APIs and cause confusion while our javadocs only show annotated and public APIs. (vinodkv via wangda) 
(cherry picked from commit 875062b5bc)
 2016-09-27 11:27:25 -07:00
Xiao Chen 06187e4f98 HADOOP-13638. KMS should set UGI's Configuration object properly. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit fa397e74fe)

Conflicts:
	hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
 2016-09-26 13:02:57 -07:00
Wei-Chiu Chuang 23984e1787 HADOOP-13535. Add jetty6 acceptor startup issue workaround to branch-2. Contributed by Min Shen. 2016-09-23 14:25:52 -07:00
Brahma Reddy Battula 347131fe8d HDFS-10886. Replace fs.default.name with fs.defaultFS in viewfs document. 
(cherry picked from commit bbdf350ff9)
 2016-09-23 19:32:37 +05:30
Mingliang Liu f98afb3bee HADOOP-13601. Fix a log message typo in AbstractDelegationTokenSecretManager. Contributed by Mehran Hassani. 
(cherry picked from commit e80386d69d)
 2016-09-20 13:23:17 -07:00
Xiao Chen b6a60d715d HDFS-10489. Deprecate dfs.encryption.key.provider.uri for HDFS encryption zones. Contributed by Xiao Chen. 
(cherry picked from commit ea839bd48e)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/HdfsConfiguration.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
 2016-09-19 16:38:18 -07:00
Andrew Wang b03a0be7a3 HDFS-10823. Implement HttpFSFileSystem#listStatusIterator. 
(cherry picked from commit 8a40953058)
 2016-09-16 15:37:49 -07:00
Wei-Chiu Chuang 031d5f6c5b HADOOP-13580. If user is unauthorized, log "unauthorized" instead of "Invalid signed text:". Contributed by Wei-Chiu Chuang. 
(cherry picked from commit f6f3a447bf)
 2016-09-16 14:53:50 -07:00
Akira Ajisaka e485ace0dd HADOOP-13616. Broken code snippet area in Hadoop Benchmarking. Contributed by Kai Sasaki. 
(cherry picked from commit 7cad7b7046)
 2016-09-15 23:43:20 +09:00
Steve Loughran 3b9268cfba HADOOP-13606 swift FS to add a service load metadata file. Contributed by Steve Loughran 2016-09-14 15:44:45 +01:00
Jing Zhao 55600b38a8 HADOOP-13546. Override equals and hashCode of the default retry policy to avoid connection leakage. Contributed by Xiaobing Zhou. 
(cherry picked from commit 08d8e0ba25)
 2016-09-13 11:13:37 -07:00
Yongjun Zhang 223ce108a1 HDFS-10657. testAclCLI.xml setfacl test should expect mask r-x. (John Zhuge via Yongjun Zhang) 
(cherry picked from commit d2466ec3e0)
 2016-09-12 21:46:09 -07:00
Arpit Agarwal 6948691ad1 HDFS-10830. FsDatasetImpl#removeVolumes crashes with IllegalMonitorStateException when vol being removed is in use. (Arpit Agarwal and Manoj Govindassamy) 2016-09-10 18:22:15 -07:00
Arpit Agarwal 012b266e5e HDFS-10742. Measure lock time in FsDatasetImpl. Contributed by Chen Liang. 2016-09-10 18:04:00 -07:00
Chris Douglas 806a8dea38 MAPREDUCE-6628. Potential memory leak in CryptoOutputStream. Contributed by Mariappan Asokan 
(cherry picked from commit 9f192cc5ac)
 2016-09-09 11:14:30 -07:00
Kihwal Lee d234990208 HADOOP-10940. RPC client does no bounds checking of responses. Contributed by Daryn Sharp. 
(cherry picked from commit d4d076876a)
 2016-09-09 10:46:37 -05:00
Xiao Chen d157733082 HADOOP-13558. UserGroupInformation created from a Subject incorrectly tries to renew the Kerberos ticket. Contributed by Xiao Chen. 
(cherry picked from commit 680be58aac)
 2016-09-06 20:54:38 -07:00
Kihwal Lee 614f9a62c4 HADOOP-13549. Eliminate intermediate buffer for server-side PB encoding. Contributed by Daryn Sharp. 
(cherry picked from commit 39d1b1d747)
 2016-09-06 10:15:57 -05:00
Kihwal Lee 28ea4122f0 HADOOP-13547. Optimize IPC client protobuf decoding. Contributed by Daryn Sharp. 2016-09-02 11:12:05 -05:00
Ray Chiang c142048084 Revert "YARN-5567. Fix script exit code checking in NodeHealthScriptRunner#reportHealthStatus. (Yufei Gu via rchiang)" 
This reverts commit 3b19bcb8f0.
 2016-09-01 15:10:53 -07:00
Kihwal Lee caef03d742 HADOOP-13465. Design Server.Call to be extensible for unified call queue. Contributed by Daryn Sharp. 
(cherry picked from commit 76cd81f4b6)
 2016-09-01 15:51:23 -05:00
Mingliang Liu 4d709be8c9 HADOOP-13375. o.a.h.security.TestGroupsCaching.testBackgroundRefreshCounters seems flaky. (Contributed by Weiwei Yang) 
(cherry picked from commit dcd21d083a)
 2016-09-01 11:14:25 -07:00
Ray Chiang 3b19bcb8f0 YARN-5567. Fix script exit code checking in NodeHealthScriptRunner#reportHealthStatus. (Yufei Gu via rchiang) 
(cherry picked from commit 05ede00386)
 2016-08-29 15:56:30 -07:00
Mingliang Liu d124af1a7c HADOOP-13559. Remove close() within try-with-resources in ChecksumFileSystem/ChecksumFs classes. (Contributed by Aaron Fabbri) 
(cherry picked from commit 6fcb04c178)
 2016-08-29 13:05:07 -07:00
Jason Lowe 039c831f5f HADOOP-13552. RetryInvocationHandler logs all remote exceptions. Contributed by Jason Lowe 
(cherry picked from commit 92d8f37155)
 2016-08-29 15:56:53 +00:00
Xiao Chen e31745e281 HADOOP-13396. Allow pluggable audit loggers in KMS. Contributed by Xiao Chen. 
(cherry picked from commit 3476156807)

Conflicts:
	hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMSAudit.java
 2016-08-24 11:09:54 -07:00
Wei-Chiu Chuang d19e29bd68 HDFS-8986. Add option to -du to calculate directory space usage excluding snapshots. Contributed by Xiao Chen. 2016-08-24 08:49:54 -07:00
Akira Ajisaka 2321b73037 HADOOP-13538. Deprecate getInstance and initialize methods with Path in TrashPolicy. Contributed by Yiqun Lin. 
(cherry picked from commit 092b4d5bfd)
 2016-08-24 16:50:11 +09:00
Zhe Zhang dfcbc12026 HADOOP-12765. HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections. Contributed by Min Shen. Branch-2 patch contributed by Wei-Chiu Chuang. 2016-08-23 14:46:08 -07:00
Masatake Iwasaki e1d3b8e71f HADOOP-13497. fix wrong command in CredentialProviderAPI.md. Contributed by Yuanbo Liu. 
(cherry picked from commit 8aae8d6bf0)
 2016-08-24 00:09:50 +09:00
Eric Yang 9faacbec2c HDFS-8312. Added permission check for moving file to Trash. (Weiwei Yang via Eric Yang) 2016-08-22 18:32:06 -07:00
Xiao Chen a4b296734b HADOOP-13526. Add detailed logging in KMS for the authentication failure of proxy user. Contributed by Suraj Acharya. 
(cherry picked from commit 98e7be769a7e69bdaec4f67143949f282f6761cf)
 2016-08-22 18:08:09 -07:00
Xiao Chen c76e0b93f0 HADOOP-13487. Hadoop KMS should load old delegation tokens from Zookeeper on startup. Contributed by Xiao Chen. 
(cherry picked from commit 2f21d53e3aaf97f425f954f52b50f36dddb6b886)
 2016-08-22 14:42:25 -07:00
Akira Ajisaka 62c7d09efa HDFS-10645. Make block report size as a metric and add this metric to datanode web ui. Contributed by Yuanbo Liu. 2016-08-22 14:22:37 +09:00
Mingliang Liu 50a8b0eb6b HADOOP-13527. Add Spark to CallerContext LimitedPrivate scope. (Contributed by Weiqing Yang) 2016-08-21 09:41:19 -07:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) a299146b6d HADOOP-13428. Fixed hadoop-common build files to generate jdiff by working around a jdiff-bug. Contributed by Wangda Tan. 
(cherry picked from commit 99603e9022)
 2016-08-19 19:10:35 -07:00
Chris Nauroth 6dba38f08b HADOOP-13252. Tune S3A provider plugin mechanism. Contributed by Steve Loughran. 
(cherry picked from commit 763f0497bb)
 2016-08-19 12:30:47 -07:00
Jing Zhao 9fdd1ea724 HADOOP-13503. Improve SaslRpcClient failure logging. Contributed by Xiaobing Zhou. 2016-08-18 14:57:38 -07:00
Mingliang Liu 3f3957ad66 HADOOP-13512. ReloadingX509TrustManager should keep reloading in case of exception. (Contributed by Mingliang Liu) 2016-08-18 14:15:01 -07:00
Junping Du d464483bf7 YARN-4676. Automatic and Asynchronous Decommissioning Nodes Status Tracking. Contributed by Diniel Zhi. 2016-08-18 07:23:29 -07:00
Steve Loughran e39d81be80 HADOOP-13405 doc for fs.s3a.acl.default indicates incorrect values. Contributed by Shen Yinjie 2016-08-18 14:36:41 +01:00
Anu Engineer ac8c3ae784 HADOOP-11786. Fix Javadoc typos in org.apache.hadoop.fs.FileSystem. Contributed by Andras Bokor. 2016-08-17 15:04:02 -07:00
Mingliang Liu 3ff4e6f3d9 HADOOP-13470. GenericTestUtils$LogCapturer is flaky. (Contributed by Mingliang Liu) 2016-08-16 17:36:21 -07:00
Mingliang Liu 9392fe3a6b Revert "HADOOP-13470. GenericTestUtils$LogCapturer is flaky. (Contributed by Mingliang Liu)" 
This reverts commit 23161c67cf.
 2016-08-16 16:27:02 -07:00
Andrew Wang c91e6f2828 HADOOP-13494. ReconfigurableBase can log sensitive information. Contributed by Sean Mackrory. 
(cherry picked from commit 4b689e7a75)
 2016-08-16 15:02:08 -07:00
Mingliang Liu 23161c67cf HADOOP-13470. GenericTestUtils$LogCapturer is flaky. (Contributed by Mingliang Liu) 2016-08-15 20:38:40 -07:00
Mingliang Liu f9a7e59066 HDFS-10678. Documenting NNThroughputBenchmark tool. (Contributed by Mingliang Liu) 2016-08-15 20:37:55 -07:00
Mingliang Liu e36a913663 HDFS-10724. Document caller context config keys. (Contributed by Mingliang Liu) 
(cherry picked from commit 4bcbef39f7)
 2016-08-15 20:36:11 -07:00
Xiao Chen 74156ee20a HADOOP-13437. KMS should reload whitelist and default key ACLs when hot-reloading. Contributed by Xiao Chen. 
(cherry picked from commit 9daa9979a1)
 2016-08-15 18:18:31 -07:00
Varun Saxena 5c2f2ae7bc HADOOP-13333. testConf.xml ls comparators in wrong order (Vrushali C via Varun Saxena) 2016-08-16 03:05:16 +05:30
Wei-Chiu Chuang 6d53e096b2 HADOOP-13190. Mention LoadBalancingKMSClientProvider in KMS HA documentation. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit db719ef125)
 2016-08-11 12:27:52 -07:00
Wei-Chiu Chuang d4501ad0d8 HADOOP-13441. Document LdapGroupsMapping keystore password properties. Contributed by Yuanbo Liu. 
(cherry picked from commit d892ae9576)
 2016-08-11 11:58:18 -07:00
Zhe Zhang 00ff3d737c HDFS-9888. Allow reseting KerberosName in unit tests. Contributed by Xiao Chen. 
(cherry picked from commit 3e8099a45a)
 2016-08-11 11:32:54 -07:00
Zhe Zhang 7d402692b4 HDFS-9804. Allow long-running Balancer to login with keytab. Contributed by Xiao Chen. 
(cherry picked from commit ccff6035f5)
 2016-08-11 10:53:16 -07:00
Xiao Chen 01fc975ed9 HADOOP-13461. NPE in KeyProvider.rollNewVersion. Contributed by Colm O hEigeartaigh. 
(cherry picked from commit e83be44af5)
 2016-08-10 16:28:34 -07:00
Wei-Chiu Chuang 8efd4959f3 HDFS-8224. Schedule a block for scanning if its metadata file is corrupt. Contributed by Rushabh S Shah. 2016-08-10 11:34:59 -07:00
Chris Nauroth f7ac0fd80a HADOOP-13476. CredentialProviderFactory fails at class loading from libhdfs (JNI). Contributed by Aaron Fabbri. 
(cherry picked from commit 4d4fe07c3c)
 2016-08-10 09:54:39 -07:00
Karthik Kambatla 2df34ab6e2 HADOOP-13299. JMXJsonServlet is vulnerable to TRACE. (Haibo Chen via kasha) 
(cherry picked from commit 85422bb7c5)
 2016-08-09 13:47:19 -07:00
Kihwal Lee 47293d951f HADOOP-13473. Tracing in IPC Server is broken. Contributed by Daryn Sharp. 
(cherry picked from commit caf800d529)
 2016-08-09 14:36:26 -05:00
Masatake Iwasaki 427b54086e HADOOP-13439. Fix race between TestMetricsSystemImpl and TestGangliaMetrics. Contributed by Chen Liang. 
(cherry picked from commit 8f9b61852b)
 2016-08-09 09:59:47 +09:00
Sangjin Lee 660e321523 HADOOP-12747. support wildcard in libjars argument (sjlee) 
(cherry picked from commit 0ad48aa2c8)
 2016-08-08 17:57:33 -07:00
Wei-Chiu Chuang cc20316b55 HADOOP-13395. Enhance TestKMSAudit. Contributed by Xiao Chen. 
(cherry picked from commit 070548943a)

Conflicts:
hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMSAudit.java
 2016-08-08 15:16:14 -07:00
Lei Xu 2cd4092c81 HADOOP-13380. TestBasicDiskValidator should not write data to /tmp (Yufei Gu via lei) 
(cherry picked from commit 6418edd6fe)
 2016-08-08 15:06:45 -07:00
Arpit Agarwal 2dbe16c024 HADOOP-13457. Remove hardcoded absolute path for shell executable. (Chen Liang) 2016-08-08 12:36:30 -07:00
Jitendra Pandey e9955b1806 HADOOP-10823. TestReloadingX509TrustManager is flaky. Contributed by Mingliang Liu. 2016-08-08 11:02:07 -07:00
Masatake Iwasaki 9cdd97a15e HADOOP-13418. Fix javadoc warnings by JDK8 in hadoop-nfs package. Contributed by Kai Sasaki. 
(cherry picked from commit 10ed06a0c9)
 2016-08-06 12:25:05 +09:00
Wei-Chiu Chuang d875dfef3d HADOOP-13353. LdapGroupsMapping getPassward shouldn't return null when IOException throws. Contributed by Zhaohao Liang and Wei-Chiu Chuang. 
(cherry picked from commit 49ba09a922)
 2016-08-05 16:35:28 -07:00
Arpit Agarwal 4f74b5e793 HADOOP-13466. Add an AutoCloseableLock class. (Chen Liang) 2016-08-04 12:32:23 -07:00
Arpit Agarwal 365a27099c HADOOP-13467. Shell#getSignalKillCommand should use the bash builtin on Linux. (Arpit Agarwal) 2016-08-04 10:08:23 -07:00
Kihwal Lee c7fe1f5ec5 HADOOP-13442. Optimize UGI group lookups. Contributed by Daryn Sharp. 
(cherry picked from commit 9422515239)
 2016-08-04 10:48:17 -05:00
Xiao Chen bb10750067 HADOOP-13443. KMS should check the type of underlying keyprovider of KeyProviderExtension before falling back to default. Contributed by Anthony Young-Garner. 
(cherry picked from commit 05db64913d)
 2016-08-04 16:08:25 +08:00
Wei-Chiu Chuang 5964d65a4d HADOOP-13458. LoadBalancingKMSClientProvider#doOp should log IOException stacktrace. Contributed by Chen Liang. 
(cherry picked from commit a1f6564d31)
 2016-08-03 14:27:33 -07:00
Jason Lowe 5a34fa670e HADOOP-10980. TestActiveStandbyElector fails occasionally in trunk. Contributed by Eric Badger 
(cherry picked from commit c82745432a)
 2016-08-03 20:19:52 +00:00
Kihwal Lee 9d62caa440 HADOOP-13483. Optimize IPC server protobuf decoding. Contributed by Daryn Sharp. 
(cherry picked from commit 580a833496)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java
 2016-08-03 14:12:13 -05:00
Kihwal Lee dd21d242eb HADOOP-13426. More efficiently build IPC responses. Contributed by Daryn Sharp. 
(cherry picked from commit 2d8227605f)
 2016-08-03 09:33:04 -05:00
Arpit Agarwal 9850cf07fa HDFS-9353. Code and comment mismatch in JavaKeyStoreProvider. (Andras Bokor) 2016-08-02 15:18:01 -07:00
Arpit Agarwal 158dd2d5a2 HADOOP-13434. Add bash quoting to Shell class. (Owen O'Malley) 2016-08-02 13:40:56 -07:00
Chris Nauroth ec522a1907 HADOOP-13081. add the ability to create multiple UGIs/subjects from one kerberos login. Contributed by Sergey Shelukhin. 
(cherry picked from commit 0458a2af6e)
 2016-08-02 12:43:51 -07:00
Kihwal Lee 41882bd5ff HADOOP-13429. Dispose of unnecessary SASL servers. Contributed by Daryn Sharp. 
(cherry picked from commit b3018e73cc)
 2016-08-02 10:52:10 -05:00
Akira Ajisaka 5469e128d3 HADOOP-13444. Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets. Contributed by Vincent Poon. 2016-08-02 10:17:33 +09:00
Xiao Chen 8fe4b2429a HADOOP-13381. KMS clients should use KMS Delegation Tokens from current UGI. Contributed by Xiao Chen. 
(cherry picked from commit 8ebf2e95d2)

Conflicts:
	hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
 2016-07-28 18:39:59 -07:00
Xiao Chen 24d464a150 HDFS-9276. Failed to Update HDFS Delegation Token for long running application in HA mode. Contributed by Liangliang Gu and John Zhuge 
(cherry picked from commit d9aae22fdf)
 2016-07-28 16:41:31 -07:00
Chris Nauroth df2ed6b2c4 HADOOP-13422. ZKDelegationTokenSecretManager JaasConfig does not work well with other ZK users in process. Contributed by Sergey Shelukhin. 
(cherry picked from commit 255ea45e50)
 2016-07-26 15:33:30 -07:00
Chris Nauroth e6ba69b4db HADOOP-13207. Specify FileSystem listStatus, listFiles and RemoteIterator. Contributed by Steve Loughran. 
(cherry picked from commit 5fda66e4a3)
 2016-07-22 15:38:34 -07:00
Chris Nauroth 382bcf21e7 HADOOP-13240. TestAclCommands.testSetfaclValidations fail. Contributed by John Zhuge. 
(cherry picked from commit 43cf6b101d)
 2016-07-21 14:12:39 -07:00
Jakob Homan 43a40fa009 HADOOP-12009. Clarify FileSystem.listStatus() sorting order & fix FileSystemContractBaseTest:testListStatus. backport to 2.8 2016-07-21 21:50:38 +01:00
Steve Loughran 5490ed5f87 HADOOP-13212 Provide an option to set the socket buffers in S3AFileSystem (Rajesh Balamohan) 2016-07-20 13:42:33 +01:00
Akira Ajisaka 8fe2c450b3 HADOOP-12991. Conflicting default ports in DelegateToFileSystem. Contributed by Kai Sasaki. 
(cherry picked from commit dc065dd64c)
 2016-07-19 16:06:16 -07:00
Allen Wittenauer 84c785036f HDFS-9852. hdfs dfs -setfacl error message is misleading (Wei-Chiu Chuang via aw) 
(cherry picked from commit b3649adf6a)
 2016-07-19 14:40:51 -07:00
Akira Ajisaka 09375baad1 HADOOP-13202. Avoid possible overflow in org.apache.hadoop.util.bloom.BloomFilter#getNBytes. Contributed by Kai Sasaki. 
(cherry picked from commit c2bcffb34e)
 2016-07-18 11:50:27 -07:00
Robert Kanter 04c8294b6f HADOOP-13254. Create framework for configurable disk checkers (yufeigu via rkanter) 
(cherry picked from commit 442162048a)
 2016-07-15 13:41:41 -07:00
Akira Ajisaka 2231ef22f5 HADOOP-13289. Remove unused variables in TestFairCallQueue. Contributed by Ye Zhou. 
(cherry picked from commit 1381483233)
 2016-07-14 23:15:34 -07:00
Zhe Zhang b95f1af8a9 HADOOP-13290. Appropriate use of generics in FairCallQueue. Contributed by Jonathan Hung. 
(cherry picked from commit 728bf7f698)
 2016-07-13 16:38:09 -07:00
Akira Ajisaka ee1e45e4b2 HADOOP-13366. Fix dead link in o.a.h.fs.CommonConfigurationKeysPublic javadoc. Contributed by Rakesh R. 
(cherry picked from commit 505ce88437)
 2016-07-13 13:08:06 -07:00
Tsuyoshi Ozawa 22e719b16e HADOOP-11361. Fix a race condition in MetricsSourceAdapter.updateJmxCache. Contributed by Vinayakumar B, Yongjun Zhang, and Brahma Reddy Battula. (ozawa) 
(cherry picked from commit 77ffe76212)
 2016-07-13 21:28:20 +09:00
Lei Xu 82c3c66dcb HADOOP-13315. FileContext#umask is not initialized properly. (John Zhuge via lei) 
(cherry picked from commit a290a98b6a)
 2016-07-12 16:50:12 -07:00
Steve Loughran 89121745c0 HADOOP-13139 Branch-2: S3a to use thread pool that blocks clients. Contributed by Pieter Reuse. 2016-07-11 17:21:00 +01:00
Akira Ajisaka f4b63a391b Revert "HADOOP-10465. Fix use of generics within SortedMapWritable. Contributed by Bertrand Dechoux." 
This reverts commit 1d175f0f3f.
 2016-07-09 21:00:13 -07:00
Jitendra Pandey e4023f8b13 HADOOP-13352. Make X-FRAME-OPTIONS configurable in HttpServer2. Contributed by Anu Engineer. 2016-07-08 14:20:00 -07:00
Jitendra Pandey 63b64a20c2 HADOOP-13283. Support reset operation for new global storage statistics and per FS storage stats. Contributed by Mingliang Liu. 2016-07-07 15:00:34 -07:00
Akira Ajisaka f78be97469 MAPREDUCE-6717. Remove deprecated StringUtils.getFormattedTimeWithDiff. Contributed by Shen Yinjie. 
(cherry picked from commit d1626a7b37)
 2016-07-07 11:19:33 -07:00
Jian He 46f8204992 YARN-5296. NMs going OutOfMemory because ContainerMetrics leak in ContainerMonitorImpl. Contributed by Junping Du 
(cherry picked from commit d792a90206)
 2016-07-05 18:06:03 -07:00
Jitendra Pandey 06b89f0d14 HADOOP-13305. Define common statistics names across schemes. Contributed by Mingliang Liu. 2016-07-01 15:35:50 -07:00
Vinayakumar B 1458e66593 HADOOP-13314. Remove 'package-info.java' from 'test\java\org\apache\hadoop\fs\shell\' to remove eclipse compile error (Contributed by Vinayakumar B) 
(cherry picked from commit c25021fb71)
 2016-07-01 12:21:47 +05:30
Andrew Wang af581c0e23 HADOOP-12345. Pad hostname correctly in CredentialsSys.java. Contributed by Pradeep Nayak Udupi Kadbet. 
(cherry picked from commit 846ada2de3)
 2016-06-29 17:32:49 -07:00
Masatake Iwasaki b7b5807d2c HADOOP-12242. Add in-page TOC to filesystem specification pages. (iwasakims) 
(cherry picked from commit 5c07c573db)
 2016-06-30 07:23:30 +09:00
Steve Loughran a82ea9275e HADOOP-12229 Fix inconsistent subsection titles in filesystem.md. Contributed by Masatake Iwasaki 2016-06-29 14:31:27 +01:00
Andrew Wang 22eb53eb71 HADOOP-13316. Enforce Kerberos authentication for required ops in DelegationTokenAuthenticator. Contributed by Xiao Chen. 
(cherry picked from commit 7d2070493e)
 2016-06-27 18:46:50 -07:00
Andrew Wang d8a69c8737 HADOOP-13251. Authenticate with Kerberos credentials when renewing KMS delegation token. Contributed by Xiao Chen. 
(cherry picked from commit 771f798edf)

 Conflicts:
	hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
 2016-06-27 18:29:39 -07:00
Arpit Agarwal f73e748754 HADOOP-13263. Reload cached groups in background after expiry. (Contributed bt Stephen O'Donnell) 2016-06-27 09:49:04 -07:00
Tsz-Wo Nicholas Sze 20ee619e32 HADOOP-13227. AsyncCallHandler should use an event driven architecture to handle async calls. 2016-06-25 15:46:33 +08:00
Sangjin Lee ae90d4dd90 HDFS-7959. WebHdfs logging is missing on Datanode (Kihwal Lee via sjlee) 2016-06-24 14:48:36 -07:00
Colin Patrick Mccabe e751bb4497 Fix accidental revert of HADOOP-13072: WindowsGetSpaceUsed constructor should be public (Vinayakumar B via cmccabe) 
(cherry picked from commit 975786492c)
 2016-06-24 09:47:25 -07:00
Andrew Wang 8a1ee4d2b5 Revert "HADOOP-13228. Add delegation token to the connection in DelegationTokenAuthenticator. Contributed by Xiao Chen." 
This reverts commit 55eda958cc.
 2016-06-23 11:08:52 -07:00
Allen Wittenauer 2a956baaab HADOOP-13245. Fix up some misc create-release issues (aw) 
(cherry picked from commit e2f640942b)
 2016-06-23 15:15:23 +09:00
Masatake Iwasaki 81497a4451 HADOOP-12588. addendum patch to fix intermittent failure of TestGangliaMetrics. (iwasakims) 
(cherry picked from commit 67089875f0)
 2016-06-22 23:48:06 +09:00
Steve Loughran fb9167fdef HADOOP-13203 S3A: Support fadvise "random" mode for high performance readPositioned() reads. Contributed by Rajesh Balamohan and stevel. 2016-06-22 15:42:30 +01:00
Akira Ajisaka 65b4f2682e HADOOP-13296. Cleanup javadoc for Path. Contributed by Daniel Templeton. 
(cherry picked from commit e15cd43369)
 2016-06-21 20:17:00 +09:00
Colin Patrick Mccabe 14df17383c HADOOP-13288. Guard null stats key in FileSystemStorageStatistics (Mingliang Liu via Colin P. McCabe) 
(cherry picked from commit 8c1f81d4bf)
 2016-06-20 14:30:21 -07:00
Colin Patrick Mccabe 7db1e57cc1 HADOOP-13280. FileSystemStorageStatistics#getLong(“readOps“) should return readOps + largeReadOps (Mingliang Liu via cmccabe) 
(cherry picked from commit 5370a6ffae)
 2016-06-20 10:50:30 -07:00
Akira Ajisaka 39ea0891d2 HADOOP-13192. org.apache.hadoop.util.LineReader cannot handle multibyte delimiters correctly. Contributed by binde. 
(cherry picked from commit fc6b50cc57)
 2016-06-20 17:08:02 +09:00
Brahma Reddy Battula 3f27f40503 HDFS-10474. hftp copy fails when file name with Chinese+special char in branch-2 (Contributed by Brahma Reddy Battula) 2016-06-20 12:38:18 +05:30
Arpit Agarwal bd33cc0179 HDFS-10437. ReconfigurationProtocol not covered by HDFSPolicyProvider. (Arpit Agarwal) 2016-06-18 16:17:10 -07:00
Chris Nauroth b726b5c52c Revert "HADOOP-12718. Incorrect error message by fs -put local dir without permission. (John Zhuge via Yongjun Zhang)" 
This reverts commit a1f913fb6e.

Conflicts:
	hadoop-common-project/hadoop-common/CHANGES.txt
 2016-06-17 23:50:55 -07:00
Xiaoyu Yao ed5d3a62ce HADOOP-13285. DecayRpcScheduler MXBean should only report decayed CallVolumeSummary. Contributed by Xiaoyu Yao. 
(cherry picked from commit 0761379fe4)
 2016-06-17 15:30:51 -07:00
Akira Ajisaka a36aa920f4 HADOOP-12943. Add -w -r options in dfs -test command. Contributed by Weiwei Yang. 
(cherry picked from commit 09e82acaf9)
 2016-06-17 16:21:05 +09:00
Xiaoyu Yao ddf66427ff HADOOP-13255. KMSClientProvider should check and renew tgt when doing delegation token operations. Contributed by Xiao Chen. 2016-06-16 20:12:17 -07:00
Akira Ajisaka 1ef89a767d MAPREDUCE-6542. HistoryViewer uses SimpleDateFormat, but SimpleDateFormat is not threadsafe. Contributed by zhangyubiao. 
(cherry picked from commit 51d497fa93)
 2016-06-17 11:05:13 +09:00
Konstantin V Shvachko 77ed9bafdd HADOOP-13189. FairCallQueue makes callQueue larger than the configured capacity. Contributed by Vinitha Gankidi. 2016-06-16 18:26:46 -07:00
Colin Patrick Mccabe 56c997a1a6 HADOOP-12975. Add jitter to CachingGetSpaceUsed's thread (Elliott Clark via Colin P. McCabe) 
(cherry picked from commit bf780406f2)
 2016-06-16 16:59:46 -07:00
Chris Nauroth e4b63ed7b4 HADOOP-13241. document s3a better. Contributed by Steve Loughran. 
(cherry picked from commit 127d2c7281)
 2016-06-16 11:18:40 -07:00
Vinayakumar B 6ab1f545fb HDFS-10256. Use GenericTestUtils.getTestDir method in tests for temporary directories (Contributed by Vinayakumar B) 
(cherry picked from commit cc6c265171)
 2016-06-16 16:56:27 +05:30
Jitendra Pandey 14b849489a HADOOP-12291. Add support for nested groups in LdapGroupsMapping. Contributed by Esther Kundin. 2016-06-15 13:29:29 -07:00
Masatake Iwasaki 9c66fffe4a HADOOP-13244. o.a.h.ipc.Server#Server should honor handlerCount when queueSizePerHandler is specified in consturctor. Contributed by Kai Sasaki. 
(cherry picked from commit 20b13d109a)
 2016-06-14 19:46:33 +09:00
Vinayakumar B 06b76f1586 HADOOP-13179. GenericOptionsParser is not thread-safe because commons-cli OptionBuilder is not thread-safe. (Contributed by hongbin ma) 
(cherry picked from commit 5cd19e429e7aef8751599dc5f54ffe29a9dc3ea9)
 2016-06-14 15:59:07 +05:30
Akira Ajisaka d219550e8b HADOOP-13270. BZip2CompressionInputStream finds the same compression marker twice in corner case, causing duplicate data blocks. Contributed by Kai Sasaki. 
(cherry picked from commit e3ba9ad3f1)
 2016-06-14 10:20:43 +09:00
Andrew Wang 6a8d3233fc HADOOP-13079. Add -q option to Ls to print ? instead of non-printable characters. Contributed by John Zhuge. 
(cherry picked from commit 0accc3306d)
 2016-06-13 11:43:56 -07:00
Karthik Kambatla 63da641910 HADOOP-13243. TestRollingFileSystemSink.testSetInitialFlushTime() fails intermittently. (Daniel Templeton via kasha) 
(cherry picked from commit 49b4064644)
 2016-06-12 14:28:19 -07:00
Akira Ajisaka 035eea4d12 HADOOP-13213. Small Documentation bug with AuthenticatedURL in hadoop-auth. Contributed by Tom Ellis. 
This closes #97.

(cherry picked from commit 8a1dccecce)
 2016-06-11 03:33:19 +09:00
Jing Zhao 9c7002b122 HADOOP-13249. RetryInvocationHandler need wrap InterruptedException in IOException when call Thread.sleep. Contributed by Zhihai Xu. 
(cherry picked from commit 0bbb4ddd79)
 2016-06-10 10:40:53 -07:00
Wangda Tan e91e9e596d YARN-3426. Add jdiff support to YARN. (vinodkv via wangda) 
(cherry picked from commit 03fc6b1bb0)
 2016-06-10 09:53:40 -07:00
Chris Nauroth 1925942294 Revert "HADOOP-12666. Support Microsoft Azure Data Lake - as a file system in Hadoop. Contributed by Vishwajeet Dusane." 
This reverts commit a8f03ef7ea.
 2016-06-09 15:17:18 -07:00
Chris Nauroth a8f03ef7ea HADOOP-12666. Support Microsoft Azure Data Lake - as a file system in Hadoop. Contributed by Vishwajeet Dusane. 
(cherry picked from commit 9581fb715c)

Conflicts:
	hadoop-tools/hadoop-tools-dist/pom.xml
	hadoop-tools/pom.xml
 2016-06-09 14:49:05 -07:00
Steve Loughran 6350e4b1fb HADOOP-12537 S3A to support Amazon STS temporary credentials. Contributed by Sean Mackrory. 2016-06-09 20:59:54 +01:00
Steve Loughran df29f77846 HADOOP-13237: s3a initialization against public bucket fails if caller lacks any credentials. Contributed by Chris Nauroth 2016-06-09 16:36:27 +01:00
Junping Du 19eb997f67 HADOOP-10048. LocalDirAllocator should avoid holding locks while accessing the filesystem. Contributed by Jason Lowe. 
(cherry picked from commit c14c1b298e)
 2016-06-07 09:20:51 -07:00
Tsz-Wo Nicholas Sze 074588d78b Revert "Revert "HADOOP-13226 Support async call retry and failover."" 
This reverts commit 5bfadb8e6b.
 2016-06-06 16:52:24 +08:00
Tsz-Wo Nicholas Sze 09a543fc86 Revert "Revert "HADOOP-13168. Support Future.get with timeout in ipc async calls."" 
This reverts commit 071aeab585.
 2016-06-06 16:52:02 +08:00
Tsz-Wo Nicholas Sze 7b9062f405 Revert "Revert "HADOOP-12957. Limit the number of outstanding async calls. Contributed by Xiaobing Zhou"" 
This reverts commit 7d4c16bae9.
 2016-06-06 16:51:48 +08:00
Tsz-Wo Nicholas Sze c1947a71ff Revert "Revert "HDFS-10224. Implement asynchronous rename for DistributedFileSystem. Contributed by Xiaobing Zhou"" 
This reverts commit c11c2ee649.
 2016-06-06 16:51:41 +08:00
Andrew Wang c11c2ee649 Revert "HDFS-10224. Implement asynchronous rename for DistributedFileSystem. Contributed by Xiaobing Zhou" 
This reverts commit d316f44059.
 2016-06-03 18:11:12 -07:00
Andrew Wang 7d4c16bae9 Revert "HADOOP-12957. Limit the number of outstanding async calls. Contributed by Xiaobing Zhou" 
This reverts commit 7e0824faa9.
 2016-06-03 18:10:49 -07:00
Andrew Wang 071aeab585 Revert "HADOOP-13168. Support Future.get with timeout in ipc async calls." 
This reverts commit 658072d62e.
 2016-06-03 18:10:49 -07:00
Andrew Wang 5bfadb8e6b Revert "HADOOP-13226 Support async call retry and failover." 
This reverts commit f7189d6a1c.
 2016-06-03 18:10:47 -07:00
Andrew Wang 03c4724c88 HADOOP-13155. Implement TokenRenewer to renew and cancel delegation tokens in KMS. Contributed by Xiao Chen. 
(cherry picked from commit 713cb71820)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/KeyProviderDelegationTokenExtension.java
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
 2016-06-03 17:01:11 -07:00
Chris Nauroth 5e6ee5aafb HADOOP-13105. Support timeouts in LDAP queries in LdapGroupsMapping. Contributed by Mingliang Liu. 
(cherry picked from commit d82bc85018)
 2016-06-03 16:38:38 -07:00
Jian He 7f865bf438 YARN-5190. Registering/unregistering container metrics in ContainerMonitorImpl and ContainerImpl causing uncaught exception in ContainerMonitorImpl. Contributed by Junping Du 
(cherry picked from commit 99cc439e29)
 2016-06-03 11:11:20 -07:00
Chris Nauroth 043a0c2e6c HADOOP-13171. Add StorageStatistics to S3A; instrument some more operations. Contributed by Steve Loughran. 2016-06-03 08:56:07 -07:00
Akira Ajisaka b6d5546e24 HDFS-10341. Add a metric to expose the timeout number of pending replication blocks. (aajisaka) 2016-06-03 16:04:11 +09:00
Masatake Iwasaki 3740974413 HDFS-10367. TestDFSShell.testMoveWithTargetPortEmpty fails with Address bind exception. Contributed by Brahma Reddy Battula. 
(cherry picked from commit aadb77e412)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
 2016-06-02 20:59:40 +09:00
Akira Ajisaka 6843e55208 HADOOP-13232. Typo in exception in ValueQueue.java. Contributed by Jiayi Zhou. 
(cherry picked from commit 99675e00df)
 2016-06-02 19:15:48 +09:00
Andrew Wang 55eda958cc HADOOP-13228. Add delegation token to the connection in DelegationTokenAuthenticator. Contributed by Xiao Chen. 
(cherry picked from commit 35356de1ba)
 2016-06-01 13:13:28 -07:00
Steve Loughran b29bf5d335 HADOOP-13162. Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs. (Rajesh Balamohan via stevel) 2016-06-01 14:17:50 +01:00
Colin Patrick Mccabe a64e1425ed HADOOP-13137. TraceAdmin should support Kerberized cluster (Wei-Chiu Chuang via cmccabe) 
(cherry picked from commit 8ceb06e239)
 2016-05-31 18:05:28 -07:00
Tsz-Wo Nicholas Sze f7189d6a1c HADOOP-13226 Support async call retry and failover. 2016-05-31 17:01:43 -07:00
Andrew Wang cbf87869b8 HADOOP-13132. Handle ClassCastException on AuthenticationException in LoadBalancingKMSClientProvider. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit bca31fe276)
 2016-05-31 09:35:28 -07:00
Xiaoyu Yao 001c8f5d7b HADOOP-13197. Add non-decayed call metrics for DecayRpcScheduler. Contributed by Xiaoyu Yao. 
(cherry picked from commit 4ca8859583)
 2016-05-27 18:48:55 -07:00
Yongjun Zhang 41c02757a5 HADOOP-12847. hadoop daemonlog should support https and SPNEGO for Kerberized cluster. (Wei-Chiu Chuang via Yongjun Zhang) 2016-05-26 22:54:07 -07:00
Karthik Kambatla b4c8729cf9 HDFS-10463. TestRollingFileSystemSinkWithHdfs needs some cleanup. (Daniel Templeton via kasha) 2016-05-26 14:39:07 -07:00
Kai Zheng 4c8ab5e162 HADOOP-12782. Faster LDAP group name resolution with ActiveDirectory. Contributed by Wei-Chiu Chuang 2016-05-27 17:16:32 +08:00
Karthik Kambatla afe603d973 HDFS-9782. RollingFileSystemSink should have configurable roll interval. (Daniel Templeton via kasha) 
(cherry picked from commit 57c31a3fef)
 2016-05-24 16:34:45 -07:00
Steve Loughran 15f997cca6 HADOOP-13130. s3a failures can surface as RTEs, not IOEs. (Steve Loughran) 2016-05-21 14:19:56 +01:00
Chris Nauroth d38f2090fa HDFS-10424. DatanodeLifelineProtocol not able to use under security cluster. Contributed by Chris Nauroth. 
(cherry picked from commit bcde1562d2)
 2016-05-20 12:47:52 -07:00
Steve Loughran eb160561e0 HADOOP-12723 S3A: Add ability to plug in any AWSCredentialsProvider. Contributed by Steven Wong. 2016-05-20 13:52:30 +01:00
Tsz-Wo Nicholas Sze 658072d62e HADOOP-13168. Support Future.get with timeout in ipc async calls. 2016-05-19 15:58:15 -07:00
Jian He 2d846fdcbd YARN-4002. Make ResourceTrackerService#nodeHeartbeat more concurrent. Contributed by Rohith Sharma K S & Zhiguo Hong 
(cherry picked from commit feb90ffcca)
 2016-05-19 13:02:05 -07:00
Steve Loughran 151ecdf26d HADOOP-12767. Update apache httpclient version to 4.5.2; httpcore to 4.4.4. Artem Aliev via stevel. 2016-05-19 14:44:44 +01:00
Andrew Wang 686691df60 HADOOP-13157. Follow-on improvements to hadoop credential commands. Contributed by Mike Yoder. 2016-05-18 14:55:31 -07:00
Chris Nauroth 42f64b3bcc HADOOP-13138. Unable to append to a SequenceFile with Compression.NONE. Contributed by Vinayakumar B. 
(cherry picked from commit 0a527196dcb35a699e12121165bc3738766b569e)
 2016-05-18 09:32:15 -07:00
Colin Patrick Mccabe feabfe9cda HADOOP-13140. FileSystem#initialize must not attempt to create StorageStatistics objects with null or empty schemes (Mingliang Liu via cmccabe) 
(cherry picked from commit f2c1d9181f)
 2016-05-18 09:19:45 -07:00
Akira Ajisaka e9dd0d84e5 Revert "HADOOP-13157. Follow-on improvements to hadoop credential commands. Contributed by Mike Yoder." 
This reverts commit e205421555.
 2016-05-18 17:57:46 +09:00
Andrew Wang e205421555 HADOOP-13157. Follow-on improvements to hadoop credential commands. Contributed by Mike Yoder. 
(cherry picked from commit 7154ace712)
 2016-05-17 17:44:34 -07:00
Xiaoyu Yao 937bc924cb HADOOP-13159. Fix potential NPE in Metrics2 source for DecayRpcScheduler. Contributed by Xiaoyu Yao. 
(cherry picked from commit 9478484845)
(cherry picked from commit 744d5edc87)
 2016-05-17 15:51:47 -07:00
Akira Ajisaka 7f03c1c90e HDFS-10242. Cannot create space quota of zero. Contributed by Takashi Ohnishi. 
(cherry picked from commit 9fe5828f05)
 2016-05-17 15:56:21 +09:00
Sangjin Lee 9330a7b4de HDFS-10208. Addendum for HDFS-9579: to handle the case when client machine can't resolve network path (Ming Ma via sjlee) 
(cherry picked from commit 61f46be071)
 2016-05-16 18:59:19 -07:00
Jing Zhao af70fc42e7 HADOOP-13146. Refactor RetryInvocationHandler. Contributed by Tsz Wo Nicholas Sze. 
(cherry picked from commit a9a8297cad)
 2016-05-16 15:26:48 -07:00
Sangjin Lee 8a316676b7 HADOOP-12971. FileSystemShell doc should explain relative path (John Zhuge via sale) 
(cherry picked from commit 1f2794b4fa)
 2016-05-13 10:02:47 -07:00
Andrew Wang d6f6b99c83 HADOOP-11180. Change log message "token.Token: Cannot find class for token kind kms-dt" to debug. Contributed by Yi Liu. 
(cherry picked from commit 3bfd29949f)
 2016-05-12 23:48:58 -07:00
Steve Loughran 1370dfc775 HADOOP-13028 add low level counter metrics for S3A; use in read performance tests. contributed by: stevel 
patch includes
HADOOP-12844 Recover when S3A fails on IOException in read()
HADOOP-13058 S3A FS fails during init against a read-only FS if multipart purge
HADOOP-13047 S3a Forward seek in stream length to be configurable
 2016-05-12 19:23:31 +01:00
Sangjin Lee d6d13ec670 YARN-4577. Enable aux services to have their own custom classpath/jar file (Xuan Gong via sale) 
(cherry picked from commit 0bbe01f8d5)
 2016-05-12 10:14:24 -07:00
Steve Loughran 257b2f5beb HADOOP-13122 Customize User-Agent header sent in HTTP requests by S3A. Chris Nauroth via stevel. 2016-05-12 13:57:05 +01:00
Colin Patrick Mccabe 15bfcde238 HADOOP-13065. Add a new interface for retrieving FS and FC Statistics (Mingliang Liu via cmccabe) 
(cherry picked from commit 687233f20d)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
 2016-05-11 14:02:03 -07:00
Larry McCay 788f80b617 HADOOP-12942. hadoop credential commands non-obviously use password of "none" (Mike Yoder via lmccay) 2016-05-11 14:38:21 -04:00