Commit Graph

4473 Commits

Author SHA1 Message Date
Brahma Reddy Battula 3e5a85f35e HDFS-11134. Fix bind exception threw in TestRenameWhileOpen. Contributed By Yiqun Lin.
(cherry picked from commit b2d4b7b1b9)
2016-11-17 10:24:21 +05:30
Arpit Agarwal da8c24bbad HDFS-11114. Support for running async disk checks in DataNode. 2016-11-16 13:38:15 -08:00
Mingliang Liu 57a0f84860 HDFS-11105. TestRBWBlockInvalidation#testRWRInvalidation fails intermittently. Contributed by Yiqun Lin
(cherry picked from commit c90891e7b3)
2016-11-16 13:06:25 -08:00
Kihwal Lee f08e227671 HDFS-8870. Lease is leaked on write failure. Contributed by Kuhu Shukla.
(cherry picked from commit 4fcea8a0c8)
2016-11-15 14:51:34 -06:00
Mingliang Liu 4e0fcff8ab HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu
(cherry picked from commit 5af572b644)
2016-11-15 11:36:08 -08:00
Zhe Zhang 818915aabb HDFS-10872. Add MutableRate metrics for FSNamesystemLock operations. Contributed by Erik Krogen. 2016-11-14 16:33:15 -08:00
Brahma Reddy Battula 399abe7c74 HDFS-11135. The tests in TestBalancer run fails due to NPE. Contributed By Yiqun Lin
(cherry picked from commit 6efb8c9c60)
2016-11-14 20:41:55 +05:30
Erik Krogen 98d9e9c4c8 HDFS-11087. NamenodeFsck should check if the output writer is still writable. Contributed by Erik Krogen. 2016-11-13 14:50:33 -08:00
Mingliang Liu fb3a5085e8 HADOOP-13813. TestDelegationTokenFetcher#testDelegationTokenWithoutRenewer is failing. Contributed by Mingliang Liu 2016-11-12 10:36:14 -08:00
Mingliang Liu 685d753d07 HDFS-11129. TestAppendSnapshotTruncate fails with bind exception. Contributed by Brahma Reddy Battula
(cherry picked from commit 2ee18fc15e)
2016-11-11 22:16:05 -08:00
Xiaoyu Yao 5613b0818c HDFS-10941. Improve BlockManager#processMisReplicatesAsync log. Contributed by Chen Liang.
(cherry picked from commit 4484b48498)
2016-11-11 14:31:54 -08:00
Mingliang Liu 289f03e49e HDFS-11122. TestDFSAdmin#testReportCommand fails due to timed out. Contributed by Yiqun Lin
(cherry picked from commit aa6010ccca)
2016-11-11 12:30:22 -08:00
Brahma Reddy Battula b51806b45f HDFS-11128. CreateEditsLog throws NullPointerException. Contributed by Hanisha Koneru.
(cherry picked from commit 1ae57f0f75)
2016-11-11 20:09:49 +05:30
Xiao Chen a6146a1026 HDFS-11120. TestEncryptionZones should waitActive. Contributed by John Zhuge.
(cherry picked from commit 907b1431c0ead934e26cf7dbc51bfab5d7cc5ddc)
2016-11-09 17:16:31 -08:00
Wei-Chiu Chuang 16430c64b7 HDFS-11056. Concurrent append and read operations lead to checksum error. Contributed by Wei-Chiu Chuang. 2016-11-09 13:28:16 -08:00
Brahma Reddy Battula 53ff3c9e03 HDFS-9482. Replace DatanodeInfo constructors with a builder pattern. Contributed by Brahma Reddy Battula. 2016-11-09 09:58:36 +05:30
Mingliang Liu f5c9f9f1b2 HDFS-11083. Add unit test for DFSAdmin -report command. Contributed by Xiaobing Zhou
(cherry picked from commit 62d8c17dfd)
2016-11-08 17:05:34 -08:00
Andrew Wang 4f25fd975e HADOOP-13789. Hadoop Common includes generated test protos in both jar and test-jar. Contributed by Sean Busbey.
(cherry picked from commit e1c6ef2efa)
2016-11-08 16:33:58 -08:00
Arpit Agarwal b77239b46d Revert "HDFS-11114. Support for running async disk checks in DataNode."
This reverts commit 732eaadddb.
2016-11-08 09:04:58 -08:00
Arpit Agarwal 732eaadddb HDFS-11114. Support for running async disk checks in DataNode.
This closes #153.
2016-11-07 18:46:37 -08:00
Xiao Chen f30d338766 HDFS-10756. Expose getTrashRoot to HTTPFS and WebHDFS. Contributed by Yuanbo Liu.
(cherry picked from commit d8bab3dcb6)
2016-11-04 18:08:46 -07:00
Mingliang Liu 4cee535aa5 HDFS-11085. Add unit test for NameNode failing to start when name dir is unwritable. Contributed by Xiaobing Zhou
(cherry picked from commit 0c0ab102ab)
2016-11-04 14:39:54 -07:00
Andrew Wang 9ede5ccfdd HADOOP-11088. Quash unnecessary safemode WARN message during NameNode startup. Contributed by Yiqun Lin.
(cherry picked from commit 5cad93d5c7)
2016-11-03 15:31:21 -07:00
Mingliang Liu 6d0cea4a5a HDFS-11076. Add unit test for extended Acls. Contributed by Chen Liang
(cherry picked from commit 7534aee09a)
2016-11-03 13:52:18 -07:00
Arpit Agarwal 458b636a39 HDFS-11097. Fix warnings for deprecated StorageReceivedDeletedBlocks constructor. Contributed by Yiqun Lin. 2016-11-03 10:32:11 -07:00
Andrew Wang 99d0b3eb2b HDFS-11080. Update HttpFS to use ConfigRedactor. Contributed by Sean Mackrory. 2016-11-02 19:11:22 -07:00
Mingliang Liu 7a662bea89 HDFS-11095. BlockManagerSafeMode should respect extension period default config value (30s). Contributed by Mingliang Liu
(cherry picked from commit 9b0c17f852)
2016-11-02 15:02:08 -07:00
Brahma Reddy Battula 35e6081020 HDFS-11031. Add additional unit test for DataNode startup behavior when volumes fail. Contributed by Mingliang Liu. 2016-11-02 10:39:12 +05:30
Mingliang Liu 1dd59042ac HDFS-11065. Add space quota tests for heterogenous storages. Contributed by Xiaobing Zhou
(cherry picked from commit fa1512a740)
2016-11-01 14:46:24 -07:00
Kihwal Lee 7c1a1834e4 HDFS-10997. Reduce number of path resolving methods. Contributed by Daryn Sharp. 2016-11-01 08:04:54 -05:00
Brahma Reddy Battula 1c5708f6de HDFS-10455. Logging the username when deny the setOwner operation. Contributed by Rakesh R. 2016-11-01 10:49:43 +05:30
Mingliang Liu 6bb12ecc3f HDFS-11030. TestDataNodeVolumeFailure#testVolumeFailure is flaky (though passing). Contributed by Mingliang Liu 2016-10-31 16:03:02 -07:00
Arpit Agarwal 56092e351b HDFS-11074. Remove unused method FsDatasetSpi#getFinalizedBlocksOnPersistentStorage. Contributed by Hanisha Koneru. 2016-10-31 12:39:54 -07:00
Mingliang Liu 2848a27700 HDFS-11047. Remove deep copies of FinalizedReplica to alleviate heap consumption on DataNode. Contributed by Xiaobing Zhou 2016-10-28 11:11:31 -07:00
Mingliang Liu d0da47ad9f HDFS-11048. Audit Log should escape control characters. Contributed by Eric Badger
(cherry picked from commit 8a9388e5f6)
2016-10-28 10:57:20 -07:00
Erik Krogen ce8ace372c HDFS-9500. Fix software version counts for DataNodes during rolling upgrade. Contributed by Erik Krogen.
(cherry picked from commit f3ac1f41b8)
2016-10-27 16:00:39 -07:00
Wei-Chiu Chuang 2cf3138d72 HDFS-11055. Update default-log4j.properties for httpfs to imporve test logging. Contributed by Wei-Chiu Chuang.
(cherry picked from commit 31ff42b510)
2016-10-27 13:38:02 -07:00
Kihwal Lee 5057abc390 HDFS-11069. Tighten the authorization of datanode RPC. Contributed by Kihwal Lee
(cherry picked from commit ae48c496dc)
2016-10-27 14:18:10 -05:00
Kihwal Lee 6c11a11918 Revert "HDFS-10455. Logging the username when deny the setOwner operation. Contributed by Tianyin Xiu"
This reverts commit f4300ea12a.
2016-10-27 14:10:07 -05:00
Brahma Reddy Battula f4300ea12a HDFS-10455. Logging the username when deny the setOwner operation. Contributed by Tianyin Xiu
(cherry picked from commit ac35ee9393)
2016-10-27 20:22:08 +05:30
Vinayakumar B 144f7a9e0e HDFS-8492. DN should notify NN when client requests a missing block (Contributed by Walter Su)
(cherry picked from commit 1cf6ec4ad4)
2016-10-27 16:46:09 +05:30
Akira Ajisaka 53ed19efec HDFS-11049. The description of dfs.block.replicator.classname is not clear. Contributed by Yiqun Lin.
(cherry picked from commit 9c680d0b05)
2016-10-27 16:37:32 +09:00
Masatake Iwasaki 7e918637de HADOOP-13017. Implementations of InputStream.read(buffer, offset, bytes) to exit 0 if bytes==0. Contributed by Steve Loughran.
(cherry picked from commit 0bdd263d82)
2016-10-27 15:54:03 +09:00
Vinayakumar B 72ce627533 HDFS-9929. Duplicate keys in NAMENODE_SPECIFIC_KEYS (Contributed by Akira Ajisaka)
(cherry picked from commit f0642b02996ae90d85499ddef2e20bdf88d98c7b)
2016-10-27 12:19:08 +05:30
Mingliang Liu 2aed61d887 HDFS-10921. TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode. Contributed by Eric Badger
(cherry picked from commit 55e1fb8e32)
2016-10-26 14:12:08 -07:00
Kihwal Lee 0cd43dd2b7 HDFS-11053. Unnecessary superuser check in versionRequest(). Contributed by Kihwal Lee.
(cherry picked from commit e90af4a89b)
2016-10-26 09:17:29 -05:00
Kihwal Lee 366ebb0822 HDFS-11050. Change log level to 'warn' when ssl initialization fails and defaults to DEFAULT_TIMEOUT_CONN_CONFIGURATOR. Contributed by Kuhu Shukla.
(cherry picked from commit ce6bbfb23c)
2016-10-26 08:11:12 -05:00
Arpit Agarwal a071e33a91 HDFS-11054. Suppress verbose log message in BlockPlacementPolicyDefault. Contributed by Chen Liang. 2016-10-25 19:51:32 -07:00
Xiaoyu Yao 590d19c0a8 HDFS-11011. Add unit tests for HDFS command 'dfsadmin -set/clrSpaceQuota'. Contributed by Xiaobing Zhou.
(cherry picked from commit 85f6fec0370b3cd94d6c2f19920c0b6d33f127c6)
2016-10-25 11:24:27 -07:00
Zhe Zhang ff806cbfc7 HDFS-11015. Enforce timeout in balancer. Contributed by Kihwal Lee.
(cherry picked from commit f6367c5f44)
2016-10-25 10:45:31 -07:00
Kihwal Lee 086577c67a HDFS-11042. Add missing cleanupSSLConfig() call for tests that use setupSSLConfig(). Contributed by Kuhu Shukla.
(cherry picked from commit b18f35f097)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
2016-10-24 08:56:14 -05:00
Chris Douglas 25f4327f0b HADOOP-13702. Add instrumented ReadWriteLock. Contributed by Jingcheng Du
(cherry picked from commit ae8bccd509)
2016-10-21 13:01:08 -07:00
Brahma Reddy Battula 2ab149f0e4 HADOOP-13236. truncate will fail when we use viewfilesystem. Contributed by Brahma Reddy Battula
(cherry picked from commit f872c6bc03)
2016-10-21 16:19:18 +05:30
Mingliang Liu c37c00b061 HDFS-10998. Add unit tests for HDFS command 'dfsadmin -fetchImage' in HA. Contributed by Xiaobing Zhou
(cherry picked from commit d7d87deece)
2016-10-20 19:54:35 -07:00
Aaron T. Myers 0f616cefeb HDFS-10423. Increase default value of httpfs maxHttpHeaderSize. Contributed by Nicolae Popa.
(cherry picked from commit aa1b583cf9)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/httpfs-config.sh
2016-10-20 13:34:52 -07:00
Wei-Chiu Chuang 6f4192d77d HDFS-11018. Incorrect check and message in FsDatasetImpl#invalidate. Contributed by Yiqun Lin.
(cherry picked from commit 6d2da38d16)

Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
2016-10-20 11:58:46 -07:00
Steve Loughran ab36519b6f Revert "HADOOP-13236. truncate will fail when we use viewfilesystem. Contributed by Brahma Reddy Battula"
This reverts commit bf6379fdd2.
2016-10-20 19:32:10 +01:00
Brahma Reddy Battula bf6379fdd2 HADOOP-13236. truncate will fail when we use viewfilesystem. Contributed by Brahma Reddy Battula
(cherry picked from commit f872c6bc03)
2016-10-20 21:48:23 +05:30
Kihwal Lee cacf83b6c3 HDFS-10627. Volume Scanner marks a block as "suspect" even if the exception is network-related. Contributed by Rushabh S Shah.
(cherry picked from commit 5c0bffddc0)
2016-10-20 11:02:02 -05:00
Brahma Reddy Battula 69a4b0c7d2 HDFS-10699. Log object instance get incorrectly in TestDFSAdmin. Contributed by Yiqun Lin
(cherry picked from commit f5e401de3e7ad6e73b07fef1cb5915753b08fbdb)
2016-10-20 21:08:39 +05:30
Brahma Reddy Battula df5671db74 HDFS-9480. Expose nonDfsUsed via StorageTypeStats. Contributed by Brahma Reddy Battula
(cherry picked from commit 4c73be135c)
2016-10-20 20:47:37 +05:30
Brahma Reddy Battula 5a37668d1b HDFS-11025. TestDiskspaceQuotaUpdate fails in trunk due to Bind exception. Contributed by Yiqun Lin
(cherry picked from commit 73504b1bdc)
2016-10-20 10:42:34 +05:30
Arpit Agarwal a3cbaf0c04 HDFS-10752. Several log refactoring/improvement suggestion in HDFS. Contributed by Hanisha Koneru. 2016-10-19 17:20:15 -07:00
Xiao Chen ad7d3c4db8 HDFS-11009. Add a tool to reconstruct block meta file from CLI.
(cherry picked from commit bc4a32aea50e86819730312e89315c0244ce64bf)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetUtil.java
2016-10-18 22:40:58 -07:00
Xiaoyu Yao 74f1c49d76 HDFS-10906. Add unit tests for Trash with HDFS encryption zones. Contributed by Hanisha Koneru.
(cherry picked from commit 0c79d5af74944363f3d04f7e1702411db564bbfd)
2016-10-18 14:07:57 -07:00
Kai Zheng 3972bb3bba HDFS-10920. TestStorageMover#testNoSpaceDisk is failing intermittently. Contributed by Rakesh R 2016-10-18 15:08:15 +06:00
Akira Ajisaka b2618685fb HADOOP-13522. Add %A and %a formats for fs -stat command to print permissions. Contributed by Alex Garbarini.
(cherry picked from commit bedfec0c10)
2016-10-18 15:03:33 +09:00
Vinitha Reddy Gankidi 278eeb1056 HDFS-10712. TestDataNodeVolumeFailure should pass not null BlockReportContext. Contributed by Vinitha Gankidi. 2016-10-17 18:55:26 -07:00
Ming Ma d55a7f8935 HDFS-9390. Block management for maintenance states. 2016-10-17 17:46:29 -07:00
Xiaoyu Yao 61aceb87df HDFS-10922. Adding additional unit tests for Trash (II). Contributed by Weiwei Yang.
(cherry picked from commit f1802d0be05ecc0b3248690b6f9efedbc7784112)
2016-10-17 14:21:51 -07:00
Brahma Reddy Battula 64e532393b HDFS-11003. Expose XmitsInProgress through DataNodeMXBean. Contributed By Brahma Reddy Battula
(cherry picked from commit 5f4ae85bd8)
2016-10-15 22:29:55 +05:30
Akira Ajisaka 35832901e8 HDFS-11012. Unnecessary INFO logging on DFSClients for InvalidToken. Contributed by Harsh J.
(cherry picked from commit 5ad037df25)
2016-10-15 22:17:55 +09:00
Vinitha Reddy Gankidi 863e1020c0 HDFS-10301. Remove FBR tracking state to fix false zombie storage detection for interleaving block reports. Contributed by Vinitha Gankidi.
(cherry picked from commit 391ce535a7)
2016-10-14 18:25:52 -07:00
Lei Xu 78cb79fa91 HDFS-10960. TestDataNodeHotSwapVolumes#testRemoveVolumeBeingWritten fails at disk error verification after volume remove. (Manoj Govindassamy via lei) 2016-10-14 15:00:27 -07:00
Mingliang Liu bbcbf78f0f HDFS-11008. Change unit test for testing parsing "-source" parameter in Balancer CLI. Contributed by Mingliang Liu
(cherry picked from commit 76cc84e6d4)
2016-10-14 14:43:58 -07:00
Kihwal Lee 1c0f2ef82f Revert "HDFS-10960. TestDataNodeHotSwapVolumes#testRemoveVolumeBeingWritten fails at disk error verification after volume remove. (Manoj Govindassamy via lei)"
This reverts commit aa8593f923.
2016-10-14 16:38:25 -05:00
Lei Xu aa8593f923 HDFS-10960. TestDataNodeHotSwapVolumes#testRemoveVolumeBeingWritten fails at disk error verification after volume remove. (Manoj Govindassamy via lei)
(cherry picked from commit 8c520a27cb)
2016-10-14 13:43:26 -07:00
Andrew Wang 86325e4346 HDFS-10883. 's behavior is not consistent in DFS after enabling EZ. Contributed by Yuanbo Liu.
(cherry picked from commit 0007360c33)
2016-10-14 11:41:36 -07:00
Benoy Antony 156b92e36b HDFS-10735 Distcp using webhdfs on secure HA clusters fails with StandbyException
(cherry picked from commit 701c27a776)
2016-10-14 10:29:42 -07:00
Kihwal Lee 6e0b55fe79 HDFS-10990. TestPendingInvalidateBlock should wait for IBRs. Contributed by Yiqun Lin.
(cherry picked from commit 8a9f6635a3)
2016-10-14 11:40:11 -05:00
Andrew Wang dc78e0e7fc Revert "HDFS-10990. TestPendingInvalidateBlock should wait for IBRs. Contributed by Yiqun Lin."
This reverts commit 2e153bc8ab.
2016-10-13 13:23:04 -07:00
Kihwal Lee c5a1303704 HDFS-10987. Make Decommission less expensive when lot of blocks present. Contributed by Brahma Reddy Battula.
(cherry picked from commit 332a61fd74)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
2016-10-13 14:55:22 -05:00
Andrew Wang 2e153bc8ab HDFS-10990. TestPendingInvalidateBlock should wait for IBRs. Contributed by Yiqun Lin.
(cherry picked from commit fdce515091)
2016-10-13 11:42:03 -07:00
Brahma Reddy Battula b60e545a08 10986. DFSAdmin should log detailed error message if any. Contributed by MingLiang Liu
(cherry picked from commit 29aa11b1a252e007ed62fad362096ca43aa408af)
2016-10-13 21:41:25 +05:30
Kihwal Lee a6197c1fa8 HDFS-11000. webhdfs PUT does not work if requests are routed to call queue. Contributed by Kihwal Lee.
(cherry picked from commit 9454dc5e80)
2016-10-13 08:48:33 -05:00
Akira Ajisaka 43cf0b2973 HDFS-11002. Fix broken attr/getfattr/setfattr links in ExtendedAttributes.md. Contributed by Mingliang Liu.
(cherry picked from commit 901eca004d)
2016-10-13 14:30:45 +09:00
Mingliang Liu b36aaa913c HDFS-10903. Replace config key literal strings with config key names II: hadoop hdfs. Contributed by Chen Liang 2016-10-12 17:26:11 -07:00
Kihwal Lee 5305a392c3 HDFS-10789. Route webhdfs through the RPC call queue. Contributed by Daryn Sharp and Rushabh S Shah.
(cherry picked from commit 85cd06f663)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
2016-10-12 15:22:51 -05:00
Mingliang Liu e341e5151a HDFS-10965. Add unit test for HDFS command 'dfsadmin -printTopology'. Contributed by Xiaobing Zhou
(cherry picked from commit f340d286d441d78c847722b529939b78abf9d2f3)
2016-10-11 19:59:36 -07:00
Xiaoyu Yao 2c77aca782 HDFS-10984. Expose nntop output as metrics. Contributed by Siddharth Wagle.
(cherry picked from commit 61f0490a73)
2016-10-11 15:56:12 -07:00
Andrew Wang a52c33fdb8 HDFS-10991. Export hdfsTruncateFile symbol in libhdfs. Contributed by Surendra Singh Lilhore.
(cherry picked from commit dacd3ec66b)
2016-10-11 15:07:22 -07:00
Wei-Chiu Chuang 402856a8a7 HDFS-10933. Refactor TestFsck. Contributed by Takanobu Asanuma. 2016-10-11 13:18:48 -07:00
Mingliang Liu 11e748394d HDFS-10988. Refactor TestBalancerBandwidth. Contributed by Brahma Reddy Battula
(cherry picked from commit b963818621)
2016-10-10 13:23:10 -07:00
Mingliang Liu 6c137ea94e HDFS-10972. Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'. Contributed by Xiaobing Zhou
(cherry picked from commit 3441c746b5)
2016-10-10 11:45:43 -07:00
Xiao Chen 5c96ef365d HDFS-10797. Disk usage summary of snapshots causes renamed blocks to get counted twice. Contributed by Sean Mackrory.
(cherry picked from commit 3ea027e970ec7c95a10e581435cdb6c3a6b64abb)
2016-10-07 17:37:23 -07:00
Kihwal Lee 7f05306895 HDFS-10980. Optimize check for existence of parent directory. Contributed by Daryn Sharp.
(cherry picked from commit e57fa81d95)
2016-10-07 17:21:28 -05:00
Kihwal Lee 5ea87e4aeb HDFS-10979. Pass IIP for FSDirDeleteOp methods. Contributed by Daryn Sharp.
(cherry picked from commit 3565c9af17)
2016-10-07 14:35:28 -05:00
Kihwal Lee 76c03acc6d HDFS-10939. Reduce performance penalty of encryption zones. Contributed by Daryn Sharp. 2016-10-07 13:20:05 -05:00
Brahma Reddy Battula cddd35e1f0 HDFS-10969. Fix typos in hdfs-default.xml Contributed by Yiqun Lin
(cherry picked from commit be3cb10f53)
2016-10-07 22:19:21 +05:30
Kihwal Lee e3a9666d28 HDFS-10955. Pass IIP for FSDirAttr methods. Contributed by Daryn Sharp. 2016-10-06 16:32:12 -05:00
Chris Nauroth a28ffd0fde HADOOP-13150. Avoid use of toString() in output of HDFS ACL shell commands. Contributed by Chris Nauroth.
(cherry picked from commit 1d330fbaf6)
2016-10-06 13:22:35 -07:00
Mingliang Liu 7aee005c03 HDFS-10893. Refactor TestDFSShell by setting up MiniDFSCluser once for all commands test. Contributed by Mingliang Liu 2016-10-06 08:59:49 -07:00
Brahma Reddy Battula d9e4ad78ca HDFS-10963. Reduce log level when network topology cannot find enough datanodes. Contributed by Xiao chen
(cherry picked from commit b90fc70d67)
2016-10-06 17:48:49 +05:30
Mingliang Liu a868cf4046 Revert "HDFS-10893. Refactor TestDFSShell by setting up MiniDFSCluser once for all commands test. Contributed by Mingliang Liu"
This reverts commit 14bacd2b99.
2016-10-05 23:10:28 -07:00
Mingliang Liu 14bacd2b99 HDFS-10893. Refactor TestDFSShell by setting up MiniDFSCluser once for all commands test. Contributed by Mingliang Liu 2016-10-05 22:54:27 -07:00
Wei-Chiu Chuang 434403a2a0 HDFS-10683. Make class Token$PrivateToken private. Contributed by John Zhuge.
(cherry picked from commit c5ca216915)
2016-10-05 17:46:46 -07:00
Andrew Wang 6b2a853dd7 HDFS-10962. TestRequestHedgingProxyProvider is flaky.
(cherry picked from commit e68c7b96c7)
2016-10-05 15:09:11 -07:00
Kihwal Lee b565fa8e05 HDFS-10956. Remove rename/delete performance penalty when not using snapshots. Contributed by Daryn Sharp.
(cherry picked from commit 44f48ee96e)
2016-10-04 15:26:42 -05:00
Kihwal Lee a9a2f64e56 HDFS-10878. TestDFSClientRetries#testIdempotentAllocateBlockAndClose throws ConcurrentModificationException. Contributed by Rushabh S Shah.
(cherry picked from commit 382307cbdd)
2016-10-04 11:10:29 -05:00
Brahma Reddy Battula 560e524f93 HDFS-10947. Correct the API name for truncate in webhdfs document. Contributed by Surendra Singh Lilhore
(cherry picked from commit ef7f06f7d1)
2016-10-04 18:51:40 +05:30
Brahma Reddy Battula c9df3c5f21 HDFS-10944. Correct the javadoc of dfsadmin#disallowSnapshot. Contributed by Jagadesh Kiran N
(cherry picked from commit 5ea0210c7f)
2016-10-04 18:37:43 +05:30
Mingliang Liu 612aa0cc0b HDFS-10934. TestDFSShell#testStat fails intermittently. Contributed by Eric Badger
(cherry picked from commit f61e3d13e9)
2016-10-03 22:47:09 -04:00
Mingliang Liu d963ecb918 HDFS-10810. setreplication removing block from underconstrcution temporarily. Contributed by Brahma Reddy Battula 2016-10-03 21:59:31 -04:00
Xiao Chen d9b686a2bd HDFS-10918. Add a tool to get FileEncryptionInfo from CLI. Contributed by Xiao Chen.
(cherry picked from commit 853d65a157)
2016-10-03 16:02:07 -07:00
Xiaoyu Yao 6f3744195d HDFS-10690. Optimize insertion/removal of replica in ShortCircuitCache. Contributed by Fenghua Hu.
(cherry picked from commit 607705c488)
2016-10-03 11:00:15 -07:00
Kihwal Lee 03b797a6ac HDFS-10940. Reduce performance penalty of block caching when not used. Contributed by Daryn Sharp.
(cherry picked from commit 744208431f)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
2016-10-03 11:41:19 -05:00
Kihwal Lee ebfbce3b5f HDFS-10945. Fix the Findbugwaring FSNamesystem#renameTo() in branch-2. Contributed by Brahma Reddy Battula. 2016-10-03 10:48:14 -05:00
Kihwal Lee 2ee47f02cf HDFS-10619. Cache path in InodesInPath. Contributed by Daryn Sharp.
(cherry picked from commit 90020624b0)
2016-10-03 09:20:08 -05:00
Arpit Agarwal 6dd747b6bc Revert "HDFS-10923. Make InstrumentedLock require ReentrantLock."
This reverts commit 9873ed32c6.
2016-09-30 23:12:07 -07:00
Arpit Agarwal 9873ed32c6 HDFS-10923. Make InstrumentedLock require ReentrantLock. 2016-09-30 23:04:22 -07:00
Kihwal Lee 2551ff80b7 HDFS-10851. FSDirStatAndListingOp: stop passing path as string. Contributed by Daryn Sharp. 2016-09-30 15:59:40 -05:00
Zhe Zhang dfca0f4abb HDFS-10896. Move lock logging logic from FSNamesystem into FSNamesystemLock. Contributed by Erik Krogen.
(cherry-picked from commit 434c5ea75d)
2016-09-30 13:30:31 -07:00
Kihwal Lee 10222cd54e HDFS-10850. getEZForPath should NOT throw FNF. Contributed by Andrew Wang.
(cherry picked from commit 0670149c88)
2016-09-30 08:58:38 -05:00
Mingliang Liu a7c1636296 HDFS-10892. Add unit tests for HDFS command 'dfs -tail' and 'dfs -stat'. Contributed by Mingliang Liu
(cherry picked from commit 84c6264079)
2016-09-28 16:09:09 -07:00
Andrew Wang b12f004474 HDFS-10914. Move remnants of oah.hdfs.client to hadoop-hdfs-client.
(cherry picked from commit 92e5e91598)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsAdmin.java
2016-09-28 16:03:16 -07:00
Kihwal Lee 087c61192d HDFS-10779. Rename does not need to re-solve destination. Contributed by Daryn Sharp.
(cherry picked from commit 5f34402ada)
2016-09-28 17:58:53 -05:00
Arpit Agarwal 50888471d9 HDFS-10824. MiniDFSCluster#storageCapacities has no effects on real capacity. Contributed by Xiaobing Zhou. 2016-09-28 15:56:08 -07:00
Brahma Reddy Battula e9a34ae29c HDFS-9444. Add utility to find set of available ephemeral ports to ServerSocketUtil. Contributed by Masatake Iwasaki 2016-09-28 10:50:50 +05:30
Mingliang Liu 54ca505c41 HDFS-10915. Fix time measurement bug in TestDatanodeRestart. Contributed by Xiaobing Zhou
(cherry picked from commit 6437ba18c5)
2016-09-27 20:42:47 -07:00
Wangda Tan 269401dc83 HADOOP-13544. JDiff reports unncessarily show unannotated APIs and cause confusion while our javadocs only show annotated and public APIs. (vinodkv via wangda)
(cherry picked from commit 875062b5bc)
2016-09-27 11:27:25 -07:00
Arpit Agarwal 80628ee2c3 HDFS-10828. Fix usage of FsDatasetImpl object lock in ReplicaMap. (Arpit Agarwal) 2016-09-27 10:02:22 -07:00
Masatake Iwasaki d17f03b625 HDFS-10426. TestPendingInvalidateBlock failed in trunk. Contributed by Yiqun Lin.
(cherry picked from commit f55eb981dd)
2016-09-28 01:40:55 +09:00
Arpit Agarwal b3c0656e90 HDFS-10713. Throttle FsNameSystem lock warnings. Contributed by Hanisha Koneru. 2016-09-27 09:00:39 -07:00
Brahma Reddy Battula 61368c8039 HDFS-10889. Remove outdated Fault Injection Framework documentaion. Contributed by Brahma Reddy Battula
(cherry picked from commit df1d0f5d56)
2016-09-27 19:34:38 +05:30
Arpit Agarwal 809a45a60c HDFS-9895. Remove unnecessary conf cache from DataNode. Contributed by Xiaobing Zhou. 2016-09-26 19:24:16 -07:00
Wei-Chiu Chuang bde787db23 HDFS-10609. Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications. Contributed by Wei-Chiu Chuang.
(cherry picked from commit 3ae652f821)
2016-09-26 14:44:48 -07:00
Brahma Reddy Battula 74962e3451 HDFS-10869. Remove the unused method InodeId#checkId(). Contributed by Jagadesh Kiran N
(cherry picked from commit 3e37e243ee)
2016-09-25 11:04:30 +05:30
Wei-Chiu Chuang 6c98455728 HDFS-10876. Dispatcher#dispatch should log IOException stacktrace. Contributed by Manoj Govindassamy.
(cherry picked from commit 74b3dd514c)
2016-09-23 13:28:18 -07:00
Konstantin V Shvachko c872ecaab4 HDFS-10843. Update space quota when a UC block is completed rather than committed. Contributed by Erik Krogen.
(cherry picked from commit a5bb88c8e0)
2016-09-23 10:40:09 -07:00
Brahma Reddy Battula 347131fe8d HDFS-10886. Replace fs.default.name with fs.defaultFS in viewfs document.
(cherry picked from commit bbdf350ff9)
2016-09-23 19:32:37 +05:30
Masatake Iwasaki 1b91ebb714 HDFS-9333. Some tests using MiniDFSCluster errored complaining port in use. (iwasakims)
(cherry picked from commit 964e546ab1)

 Conflicts:
  hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFSStriped.java
2016-09-21 10:40:56 +09:00
Xiao Chen 81df2be017 HDFS-10879. TestEncryptionZonesWithKMS#testReadWrite fails intermittently. Contributed by Xiao Chen.
(cherry picked from commit d31bef575478fe78c0600fb602e117e4b81c9887)
2016-09-20 16:57:34 -07:00
Xiao Chen 4e376f162f HDFS-10875. Optimize du -x to cache intermediate result. Contributed by Xiao Chen.
(cherry picked from commit e52d6e7a46)
2016-09-19 21:45:05 -07:00
Xiao Chen b6a60d715d HDFS-10489. Deprecate dfs.encryption.key.provider.uri for HDFS encryption zones. Contributed by Xiao Chen.
(cherry picked from commit ea839bd48e)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/HdfsConfiguration.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
2016-09-19 16:38:18 -07:00
Andrew Wang b03a0be7a3 HDFS-10823. Implement HttpFSFileSystem#listStatusIterator.
(cherry picked from commit 8a40953058)
2016-09-16 15:37:49 -07:00
Akira Ajisaka 214b6d16d5 HDFS-10862. Typos in 4 log messages. Contributed by Mehran Hassani.
(cherry picked from commit b09a03cd7d)
2016-09-16 16:12:19 +09:00
Arpit Agarwal 853e1e6946 HDFS-10805. Reduce runtime for append test. Contributed by Gergely Novak. 2016-09-14 09:31:24 -07:00
Andrew Wang abaad976cd HDFS-10837. Standardize serializiation of WebHDFS DirectoryListing.
(cherry picked from commit db6d243cf8)
2016-09-13 11:02:43 -07:00
Akira Ajisaka 764cf431ca HDFS-10856. Update the comment of BPServiceActor$Scheduler#scheduleNextBlockReport. Contributed by Yiqun Lin.
(cherry picked from commit f0876b8b60)
2016-09-13 14:51:04 +09:00
Yongjun Zhang 223ce108a1 HDFS-10657. testAclCLI.xml setfacl test should expect mask r-x. (John Zhuge via Yongjun Zhang)
(cherry picked from commit d2466ec3e0)
2016-09-12 21:46:09 -07:00
Arpit Agarwal 6948691ad1 HDFS-10830. FsDatasetImpl#removeVolumes crashes with IllegalMonitorStateException when vol being removed is in use. (Arpit Agarwal and Manoj Govindassamy) 2016-09-10 18:22:15 -07:00
Arpit Agarwal 012b266e5e HDFS-10742. Measure lock time in FsDatasetImpl. Contributed by Chen Liang. 2016-09-10 18:04:00 -07:00
Arpit Agarwal ad0ac6cced HDFS-10682. Replace FsDatasetImpl object lock with a separate lock object. (Contributed by Chen Liang) 2016-09-10 17:48:27 -07:00
Xiao Chen bb6d866207 HDFS-9781. FsDatasetImpl#getBlockReports can occasionally throw NullPointerException. Contributed by Manoj Govindassamy. 2016-09-09 18:25:35 -07:00
Andrew Wang f448ce2a89 HDFS-10832. Propagate ACL bit and isEncrypted bit in HttpFS FileStatus permissions.
(cherry picked from commit cba973f036)
2016-09-09 12:16:08 -07:00
Andrew Wang 72ea641468 HDFS-10831. Add log when URLConnectionFactory.openConnection failed. Contributed by yunjiong zhao.
(cherry picked from commit b07c266dca)
2016-09-08 18:30:26 -07:00
Akira Ajisaka c0756dfef6 HDFS-10847. Complete the document for FileDistribution processor in OfflineImageViewer. Contributed by Yiqun Lin.
(cherry picked from commit 20a20c2f6e)
2016-09-09 01:35:36 +09:00
Akira Ajisaka fe6dd66174 HDFS-10778. Add -format option to make the output of FileDistribution processor human-readable in OfflineImageViewer.
(cherry picked from commit 63f594892e)
2016-09-08 15:18:25 +09:00
Arpit Agarwal f9415b7a7e HDFS-9038. DFS reserved space is erroneously counted towards non-DFS used. (Brahma Reddy Battula) 2016-09-06 13:44:22 -07:00
Xiao Chen 3025992bd3 HDFS-10841. Remove duplicate or unused variable in appendFile(). Contributed by Kihwal Lee.
(cherry picked from commit f6c0b7543f)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java
2016-09-06 11:52:30 -07:00
Andrew Wang 40eab77562 HDFS-10833. Fix JSON errors in WebHDFS.md examples.
(cherry picked from commit cbd909ce2a)
2016-09-02 14:50:41 -07:00
Lei Xu d6bfb2176c HDFS-10822. Log DataNodes in the write pipeline. John Zhuge via Lei Xu
(cherry picked from commit 5a8c5064d1)
2016-09-02 11:02:38 -07:00
Allen Wittenauer b1466b8124 HDFS-9877. HDFS Namenode UI: Fix browsing directories that need to be encoded (Ravi Prakash via aw)
(cherry picked from commit 15f018434c)
2016-09-01 14:24:13 -07:00
Allen Wittenauer b9eedc24a6 HDFS-7987. Allow files / directories to be moved (Ravi Prakash via aw)
(cherry picked from commit d44f4745b4)
2016-09-01 14:23:42 -07:00
Allen Wittenauer 743c0ebd67 HDFS-7767. Use the noredirect flag in WebHDFS to allow web browsers to upload files via the NN UI (Ravi Prakash via aw)
(cherry picked from commit 99a771cd7a)
2016-09-01 14:22:23 -07:00
Zhe Zhang 776eccd873 Addendum fix for HDFS-10817 to fix failure of the added testFSReadLockLongHoldingReport in branch-2.
(cherry picked from commit 6c600360ca)
2016-08-31 23:44:54 -07:00
Zhe Zhang f87e2f5d05 HDFS-10817. Add Logging for Long-held NN Read Locks. Contributed by Erik Krogen.
(cherry picked from commit 6f4b0d33ca)
2016-08-31 15:54:03 -07:00
Andrew Wang 72d90cbe32 HDFS-10784. Implement WebHdfsFileSystem#listStatusIterator.
(cherry picked from commit 85bab5fb57)
2016-08-31 14:33:14 -07:00
Allen Wittenauer 72334a4be3 HDFS-7766. Add a flag to WebHDFS op=CREATE to not respond with a 307 redirect (Ravi Prakash via aw)
(cherry picked from commit 4b0f55b6ea)
2016-08-31 14:33:10 -07:00
Kihwal Lee 34042ccbb9 HDFS-10729. Improve log message for edit loading failures caused by FS limit checks. Contributed by Wei-Chiu Chuang.
(cherry picked from commit 01721dd88e)
2016-08-31 14:09:23 -05:00
Ming Ma 56c9a96a76 HDFS-9392. Admins support for maintenance state. Contributed by Ming Ma.
(cherry picked from commit 9dcbdbdb5a)
2016-08-30 14:09:18 -07:00
Wei-Chiu Chuang abbd95b790 HDFS-10760. DataXceiver#run() should not log InvalidToken exception as an error. Contributed by Pan Yuxuan.
(cherry picked from commit c4ee6915a1)
2016-08-30 10:44:27 -07:00
Zhe Zhang 320be1b3c6 HDFS-10814. Add assertion for getNumEncryptionZones when no EZ is created. Contributed by Vinitha Reddy Gankidi.
(cherry picked from commit 4bd45f54ee)
2016-08-29 23:42:46 -07:00
Xiao Chen 72863f0049 HDFS-4210. Throw helpful exception when DNS entry for JournalNode cannot be resolved. Contributed by Charles Lamb and John Zhuge.
(cherry picked from commit 046f6bf0f944311fc70de3a96fc6ab8211cbbbd8)
2016-08-29 17:48:49 -07:00
Yongjun Zhang 4bed2a4b9c HDFS-10625. VolumeScanner to report why a block is found bad. Contributed by Rushabh S Shah and Yiqun Lin.
(cherry picked from commit 5d1609ddf2)
2016-08-29 14:08:36 -07:00
Mingliang Liu 875c14004d HDFS-10807. Doc about upgrading to a version of HDFS with snapshots may be confusing. (Contributed by Mingliang Liu)
(cherry picked from commit 6742fb6e68)
2016-08-29 10:19:33 -07:00
Yongjun Zhang e813a3ea4a HDFS-10652. Add a unit test for HDFS-4660. Contributed by Vinayakumar B., Wei-Chiu Chuang, Yongjun Zhang.
(cherry picked from commit c25817159a)
2016-08-27 22:59:19 -07:00
Andrew Wang 1c47a356d8 HDFS-10793. Fix HdfsAuditLogger binary incompatibility introduced by HDFS-9184. Contributed by Manoj Govindassamy.
(cherry picked from commit a445b82baa)
2016-08-26 15:40:05 -07:00
Zhe Zhang f3a4cbcc60 HDFS-10798. Make the threshold of reporting FSNamesystem lock contention configurable. Contributed by Erik Krogen.
(cherry picked from commit 407b519fb1)
2016-08-26 14:20:48 -07:00
Kihwal Lee 442dfd1bbe HDFS-10768. Optimize mkdir ops. Contributed by Daryn Sharp.
(cherry picked from commit 8b7adf4ddf)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
2016-08-26 15:49:56 -05:00
Kihwal Lee 2a6497cc7c HDFS-8915. TestFSNamesystem.testFSLockGetWaiterCount fails intermittently. Contributed by Masatake Iwasaki.
(cherry picked from commit 13fb1b50e6)
2016-08-26 10:00:36 -05:00
Xiaoyu Yao fc67501393 HDFS-10748. TestFileTruncate#testTruncateWithDataNodesRestart runs sometimes timeout. Contributed by Yiqun Lin.
(cherry picked from commit 4da5000dd3)
2016-08-25 09:22:02 -07:00
Kihwal Lee ba3257baf5 HDFS-10772. Reduce byte/string conversions for get listing. Contributed by Daryn Sharp.
(cherry picked from commit a1f3293762)
2016-08-24 15:25:33 -05:00
Wei-Chiu Chuang d19e29bd68 HDFS-8986. Add option to -du to calculate directory space usage excluding snapshots. Contributed by Xiao Chen. 2016-08-24 08:49:54 -07:00
Daryn Sharp a30f6a68fa HDFS-10762. Pass IIP for file status related methods
(cherry picked from commit ec252ce0fc)
2016-08-24 08:51:05 -05:00
Zhe Zhang dfcbc12026 HADOOP-12765. HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections. Contributed by Min Shen. Branch-2 patch contributed by Wei-Chiu Chuang. 2016-08-23 14:46:08 -07:00
Jason Lowe 8cd3996246 HDFS-9745. TestSecureNNWithQJM#testSecureMode sometimes fails with timeouts. Contributed by Xiao Chen
(cherry picked from commit 126d165efd)
2016-08-23 14:48:35 +00:00
Akira Ajisaka 94a844b4f6 HDFS-10783. The option '-maxSize' and '-step' fail in OfflineImageViewer. Contributed by Yiqun Lin.
(cherry picked from commit e90f3359de)
2016-08-23 19:58:13 +09:00
Eric Yang 9faacbec2c HDFS-8312. Added permission check for moving file to Trash. (Weiwei Yang via Eric Yang) 2016-08-22 18:32:06 -07:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) 8a6a7c82f2 HDFS-10692. Update JDiff report's base version for HDFS from 2.6.0 to 2.7.2. Contributed by Wangda Tan.
(cherry picked from commit dc7a1c54f4)
2016-08-22 11:36:38 -07:00
Akira Ajisaka 62c7d09efa HDFS-10645. Make block report size as a metric and add this metric to datanode web ui. Contributed by Yuanbo Liu. 2016-08-22 14:22:37 +09:00
Jing Zhao fff0418458 HDFS-10764. Fix INodeFile#getBlocks to not return null. Contributed by Arpit Agarwal.
(cherry picked from commit 0faee62a0c)
2016-08-19 22:23:55 -07:00
Kihwal Lee 43d1279df0 HDFS-10711. Optimize FSPermissionChecker group membership check. Contributed by Daryn Sharp.
(cherry picked from commit 2550371f66)
2016-08-19 09:46:30 -05:00
Kihwal Lee 922d045e11 HDFS-10745. Directly resolve paths into INodesInPath. Contributed by Daryn Sharp. 2016-08-18 12:37:57 -05:00
Xiao Chen a20b943cf9 HDFS-10549. Correctly revoke file leases when closing files. Contributed by Yiqun Lin.
(cherry picked from commit 85aacaadb5a3f8c78b191867c0bde09b3c4b3c3c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
2016-08-17 15:50:13 -07:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) 2486c4c63a Revert "HDFS-9395. Make HDFS audit logging consistant. Contributed by Kuhu Shukla."
Reverting this on branch-2.* as it's an incompatible change.

This reverts commit 83f7f62be3.
2016-08-17 13:33:09 -07:00
Kihwal Lee 65a41696b2 HDFS-10342. BlockManager#createLocatedBlocks should not check corrupt replicas if none are corrupt. Contributed by Kuhu Shukla.
(cherry picked from commit b10c936020)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
2016-08-16 09:29:42 -05:00
Mingliang Liu 2f4660ce98 HDFS-10641. TestBlockManager#testBlockReportQueueing fails intermittently. (Contributed by Daryn Sharp) 2016-08-15 20:39:12 -07:00
Mingliang Liu 82623ea9c9 HDFS-10747. o.a.h.hdfs.tools.DebugAdmin usage message is misleading. (Contributed by Mingliang Liu) 2016-08-15 20:38:17 -07:00
Mingliang Liu f9a7e59066 HDFS-10678. Documenting NNThroughputBenchmark tool. (Contributed by Mingliang Liu) 2016-08-15 20:37:55 -07:00