Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
9,507 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

3214 Commits

Author SHA1 Message Date
Jing Zhao 725621d95a HDFS-7934. Update RollingUpgrade rollback documentation: should use bootstrapstandby for standby NN. Contributed by J. Andreina. 
(cherry picked from commit b172d03595)
 2015-04-15 13:11:15 -07:00
Jing Zhao d4dd97eabd HDFS-8151. Always use snapshot path as source when invalid snapshot names are used for diff based distcp. Contributed by Jing Zhao. 
(cherry picked from commit 4c097e473b)
 2015-04-15 12:39:09 -07:00
Arpit Agarwal 96e12fa46a HDFS-8144. Split TestLazyPersistFiles into multiple tests. (Arpit Agarwal) 2015-04-15 10:25:16 -07:00
Andrew Wang 871bf6a765 HDFS-8117. More accurate verification in SimulatedFSDataset: replace DEFAULT_DATABYTE with patterned data. Contributed by Zhe Zhang. 2015-04-15 08:43:42 -07:00
Jing Zhao 38b031d6ba HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to finalize upgrade. Contributed by Jing Zhao. 
(cherry picked from commit fddd55279d)
 2015-04-14 15:12:37 -07:00
cnauroth c7ebecfff5 HDFS-8055. NullPointerException when topology script is missing. Contributed by Anu Engineer. 
(cherry picked from commit fef596df03)
 2015-04-14 10:19:45 -07:00
cnauroth 9779b79ed8 HDFS-6666. Abort NameNode and DataNode startup if security is enabled but block access token is not enabled. Contributed by Vijay Bhat. 
(cherry picked from commit d45aa7647b)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSecureNameNode.java
 2015-04-14 10:03:53 -07:00
Arpit Agarwal d77705c980 HDFS-7701. Support reporting per storage type quota and usage with hadoop/hdfs shell. (Contributed by Peter Shi) 2015-04-13 21:20:30 -07:00
Allen Wittenauer b7a052c84a HADOOP-7713. dfs -count -q should label output column (Jonathan Allen via aw) 2015-04-13 21:18:59 -07:00
Tsz-Wo Nicholas Sze e4fe5fa2ee HDFS-8083. Move dfs.client.write.* conf from DFSConfigKeys to HdfsClientConfigKeys.Write. 2015-04-13 11:45:03 -07:00
Vinayakumar B 1071a57b20 HDFS-8111. NPE thrown when invalid FSImage filename given for 'hdfs oiv_legacy' cmd ( Contributed by surendra singh lilhore ) 
(cherry picked from commit 14384f5b51)
 2015-04-13 17:19:18 +05:30
cnauroth 293c763e82 HDFS-7933. fsck should also report decommissioning replicas. Contributed by Xiaoyu Yao. 
(cherry picked from commit f8f5887209)
 2015-04-11 13:24:06 -07:00
Tsz-Wo Nicholas Sze ef2bd4bd11 HDFS-8084. Move dfs.client.failover.* confs from DFSConfigKeys to HdfsClientConfigKeys.Failover and fix typos in the dfs.http.client.* configuration keys. 2015-04-10 19:58:42 -07:00
Haohui Mai 6f97b780df HDFS-8103. Move BlockTokenSecretManager.AccessMode into BlockTokenIdentifier. Contributed by Haohui Mai. 2015-04-10 16:36:57 -07:00
Tsz-Wo Nicholas Sze 1113aca7f8 HDFS-8100. Refactor DFSClient.Conf to a standalone class and separates short-circuit related conf to ShortCircuitConf. 2015-04-10 14:49:32 -07:00
Arun Suresh 3d17c50176 HDFS-7931. DistributedFileSystem should not look for keyProvider in cache if Encryption is disabled (asuresh) 
(cherry picked from commit 7660da95cb)
 2015-04-10 11:42:18 -07:00
Vinayakumar B 4aba069b37 HDFS-7939. Two fsimage_rollback_* files are created which are not deleted after rollback. (Contributed by J.Andreina) 
(cherry picked from commit 987c9e12e1)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2015-04-10 11:27:20 +05:30
Konstantin V Shvachko 0e99da620d HDFS-8081. Split getAdditionalBlock() into two methods. Contributed by Konstantin Shvachko 2015-04-09 22:02:14 -07:00
Haohui Mai e34afb48d4 HDFS-8102. Separate webhdfs retry configuration keys from DFSConfigKeys. Contributed by Haohui Mai. 2015-04-09 14:36:35 -07:00
Vinod Kumar Vavilapalli aec1c5ec7d Adding release 2.7.1 to CHANGES.txt 
(cherry picked from commit 623fd46c1e)
 2015-04-09 13:56:14 -07:00
Arun Suresh c846e84d55 HDFS-8091: ACLStatus and XAttributes should be presented to INodeAttributesProvider before returning to client (asuresh) 
(cherry picked from commit 922b7ed21d)
 2015-04-09 13:13:23 -07:00
Colin Patrick Mccabe 8dac245920 HDFS-8063: Fix intermittent test failures in TestTracing (Masatake Iwasaki via Colin P. McCabe) 
(cherry picked from commit 61dc2ea3fe)
 2015-04-09 11:42:00 -07:00
Colin Patrick Mccabe ca12088253 HDFS-8099. Change "DFSInputStream has been closed already" message to debug log level (Charles Lamb via Colin P. McCabe) 
(cherry picked from commit 30acb7372a)
 2015-04-09 11:23:01 -07:00
Aaron T. Myers edf2f52d6d HDFS-8101. DFSClient use of non-constant DFSConfigKeys pulls in WebHDFS classes at runtime. Contributed by Sean Busbey. 
(cherry picked from commit 3fe61e0bb0)
 2015-04-09 09:45:02 -07:00
Kihwal Lee 6d1cb34221 Fix CHANGES.txt for branch-2 2015-04-09 10:05:43 -05:00
Vinayakumar B 4e099c113b HDFS-8096. DatanodeMetrics#blocksReplicated will get incremented early and even for failed transfers (Contributed by Vinayakumar B) 
(cherry picked from commit 9d8952f97f)
 2015-04-09 11:59:05 +05:30
Andrew Wang 351fac25a9 HDFS-7979. Initialize block report IDs with a random number. 
(cherry picked from commit b1e059089d)
 2015-04-08 21:43:57 -07:00
Haohui Mai 88ef75f167 HDFS-8089. Move o.a.h.hdfs.web.resources.* to the client jars. Contributed by Haohui Mai. 2015-04-08 16:30:57 -07:00
Andrew Wang 069366e1be HDFS-8025. Addendum fix for HDFS-3087 Decomissioning on NN restart can complete without blocks being replicated. Contributed by Ming Ma. 2015-04-08 16:09:28 -07:00
Haohui Mai 6d2eca081d Revert HDFS-7813. 2015-04-08 16:03:20 -07:00
Haohui Mai 35816c438d Revert HDFS-7808. 2015-04-08 16:00:10 -07:00
Andrew Wang 8104d52269 HDFS-7725. Incorrect 'nodes in service' metrics caused all writes to fail. Contributed by Ming Ma. 2015-04-08 15:52:15 -07:00
Andrew Wang 9792500c54 HDFS-8076. Code cleanup for DFSInputStream: use offset instead of LocatedBlock when possible. Contributed by Zhe Zhang. 
(cherry picked from commit a42bb1cd91)
 2015-04-08 15:41:54 -07:00
Kihwal Lee 7e622076d4 HDFS-8046. Allow better control of getContentSummary. Contributed by Kihwal Lee. 
(cherry picked from commit 285b31e75e)
 2015-04-08 15:39:25 -05:00
Tsz-Wo Nicholas Sze de7f9a8bcc HDFS-8079. Move CorruptFileBlockIterator to a new hdfs.client.impl package. 2015-04-08 11:51:43 -07:00
Arpit Agarwal f0324738c9 HDFS-8072. Reserved RBW space is not released if client terminates while writing block. (Arpit Agarwal) 2015-04-08 11:38:30 -07:00
Haohui Mai f9fbde3074 HDFS-8080. Separate JSON related routines used by WebHdfsFileSystem to a package local class. Contributed by Haohui Mai. 2015-04-07 21:30:23 -07:00
Vinayakumar B 3cf7ac181b HDFS-8073. Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) 
(cherry picked from commit d505c8acd3)
 2015-04-08 09:57:49 +05:30
Tsz-Wo Nicholas Sze eb3ca514fb HDFS-8079. Move dfs.client.retry.* confs from DFSConfigKeys to HdfsClientConfigKeys.Retry. 2015-04-07 19:53:30 -07:00
Yongjun Zhang b76c321e94 HDFS-5215. dfs.datanode.du.reserved is not considered while computing 
available space ( Brahma Reddy Battula via Yongjun Zhang)

(cherry picked from commit 66763bb06f)
 2015-04-07 19:36:02 -07:00
cnauroth 8921526440 HDFS-8038. PBImageDelimitedTextWriter#getEntry output HDFS path in platform-specific format. Contributed by Xiaoyu Yao. 
(cherry picked from commit 672ed462965ea7da10ce3df48c02e2a3bd13f0ae)
 2015-04-07 14:23:29 -07:00
Tsz-Wo Nicholas Sze a4f0eea261 HDFS-8049. Add @InterfaceAudience.Private annotation to hdfs client implementation. Contributed by Takuya Fukudome 2015-04-07 14:01:55 -07:00
Konstantin V Shvachko 81b522790b HDFS-8071. Redundant checkFileProgress() in PART II of getAdditionalBlock(). Contributed by Konstantin Shvachko. 2015-04-06 22:23:33 -07:00
Colin Patrick Mccabe a827089905 HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time (sinago via cmccabe) 
(cherry picked from commit 28bebc81db)
 2015-04-06 09:10:15 -07:00
Vinod Kumar Vavilapalli 00854d2c2f HADOOP-11776. Fixed the broken JDiff support in Hadoop 2. Contributed by Li Lu. 
(cherry picked from commit 4b3948ea36)
 2015-04-04 13:52:51 -07:00
Vinod Kumar Vavilapalli 4eec6e09d0 HADOOP-11377. Added Null.java without which jdiff completely flops. Contributed by Tsuyoshi Ozawa. 
(cherry picked from commit 5370e7128b)
 2015-04-04 13:48:14 -07:00
Colin Patrick Mccabe b26ba22a90 HDFS-8051. FsVolumeList#addVolume should release volume reference if not put it into BlockScanner. (Lei (Eddy) Xu via Colin P. McCabe) 
(cherry picked from commit ef591b1d6a)
 2015-04-03 16:56:25 -07:00
Colin Patrick Mccabe 0c5069c432 HDFS-7996. After swapping a volume, BlockReceiver reports ReplicaNotFoundException (Lei (Eddy) Xu via Colin P. McCabe) 
(cherry picked from commit 023133cef9)
 2015-04-03 14:26:45 -07:00
cnauroth 28e0602e7b HDFS-8039. Fix TestDebugAdmin#testRecoverLease and testVerfiyBlockChecksumCommand on Windows. Contributed by Xiaoyu Yao. 
(cherry picked from commit db80e42891)
 2015-04-03 10:44:32 -07:00
cnauroth 0dde8c079d HDFS-8034. Fix TestDFSClientRetries#testDFSClientConfigurationLocateFollowingBlockInitialDelay for Windows. Contributed by Xiaoyu Yao. 
(cherry picked from commit dc5596c36a)
 2015-04-03 10:10:29 -07:00
Jing Zhao 2c1f33d178 HDFS-7811. Avoid recursive call getStoragePolicyID in INodeFile#computeQuotaUsage. Contributed by Xiaoyu Yao and Jing Zhao. 
(cherry picked from commit 72f6bd4893)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2015-04-02 21:22:43 -07:00
Haohui Mai ddb5bb8fcf Addendum for HDFS-7893 in branch-2. 2015-04-02 16:17:14 -07:00
Haohui Mai 1278310c21 HDFS-7893. Update the POM to create a separate hdfs-client jar. Contributed by Haohui Mai. 2015-04-02 15:39:40 -07:00
Haohui Mai 6e36dbf03a HDFS-8035. Move checkBlocksProperlyReplicated() in FSNamesystem to BlockManager. Contributed by Haohui Mai. 2015-04-02 11:38:02 -07:00
Tsz-Wo Nicholas Sze 81ed3a2bb8 HDFS-7954. TestBalancer#testBalancerWithPinnedBlocks should not be executed on Windows. Contributed by Xiaoyu Yao 2015-04-02 11:12:27 -07:00
Tsz-Wo Nicholas Sze ce3f32590d HDFS-7888. Change DFSOutputStream and DataStreamer for convenience of subclassing. Contributed by Li Bo 2015-04-02 11:00:47 -07:00
Vinayakumar B beb0fd0d60 HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) 
(cherry picked from commit 867d5d2675)
 2015-04-02 08:12:51 +05:30
Brandon Li 27970b3f46 HDFS-8001 RpcProgramNfs3 : wrong parsing of dfs.blocksize. Contributed by Remi Catherinot 
(cherry picked from commit 4d14816c26)
 2015-04-01 17:25:28 -07:00
Haohui Mai cfcf795492 HDFS-8008. Support client-side back off when the datanodes are congested. Contributed by Haohui Mai. 2015-04-01 16:54:53 -07:00
Haohui Mai 9e114ee607 HDFS-8036. Use snapshot path as source when using snapshot diff report in DistCp. Contributed by Jing Zhao. 2015-04-01 16:51:09 -07:00
Colin Patrick Mccabe 6f753da4a9 HDFS-7922. ShortCircuitCache#close is not releasing ScheduledThreadPoolExecutors (Rakesh R via Colin P. McCabe) 
(cherry picked from commit 3c7adaaf35)
 2015-04-01 16:11:44 -07:00
Colin Patrick Mccabe 4b74aa7182 HDFS-8026. Trace FSOutputSummer#writeChecksumChunks rather than DFSOutputStream#writeChunk (cmccabe) 
(cherry picked from commit c94d594a57)
 2015-04-01 14:10:25 -07:00
Andrew Wang b17d8a53fa HDFS-7978. Add LOG.isDebugEnabled() guard for some LOG.debug(..). Contributed by Walter Su. 
(cherry picked from commit ed72daa5df)
 2015-04-01 12:54:46 -07:00
Haohui Mai 2ca2515b33 Add the missing files for HDFS-8009. 2015-04-01 10:58:41 -07:00
Haohui Mai a5bcfe0d33 HDFS-8009. Signal congestion on the DataNode. Contributed by Haohui Mai. 2015-04-01 10:57:07 -07:00
Akira Ajisaka b85bbca745 HDFS-6945. BlockManager should remove a block from excessReplicateMap and decrement ExcessBlocks metric when the block is removed. (aajisaka) 
(cherry picked from commit 18a91fe4df)
 2015-04-01 09:08:22 +09:00
Akira Ajisaka 7f6811ad73 HDFS-7671. hdfs user guide should point to the common rack awareness doc. Contributed by Kai Sasaki. 
(cherry picked from commit 859cab2f22)
 2015-04-01 00:26:54 +09:00
yliu e4cd67e9b0 HDFS-7997. The first non-existing xattr should also throw IOException. (zhouyingchao via yliu) 2015-03-31 21:16:13 +08:00
Arpit Agarwal 312e50f9c4 HDFS-7944. Minor cleanup of BlockPoolManager#getAllNamenodeThreads. (Arpit Agarwal) 2015-03-30 21:24:59 -07:00
Harsh J 32766b6563 HDFS-3918. EditLogTailer shouldn't log WARN when other node is in standby mode. Contributed by Todd Lipcon. 
(cherry picked from commit cce66ba3c9)
 2015-03-31 08:17:22 +05:30
Arpit Agarwal abf3ad988d HDFS-7645. Rolling upgrade is restoring blocks from trash multiple times (Contributed by Vinayakumar B and Keisuke Ogiwara) 2015-03-30 15:25:26 -07:00
Haohui Mai 02fcf622fb Addendum for HDFS-7748. 2015-03-30 13:11:24 -07:00
Haohui Mai dd5b2dac5a HDFS-7748. Separate ECN flags from the Status in the DataTransferPipelineAck. Contributed by Anu Engineer and Haohui Mai. 2015-03-30 11:59:32 -07:00
Colin Patrick Mccabe 02ed22cd2d HDFS-7261. storageMap is accessed without synchronization in DatanodeDescriptor#updateHeartbeatState() (Brahma Reddy Battula via Colin P. McCabe) 
(cherry picked from commit 1feb9569f3)
 2015-03-30 10:53:57 -07:00
Akira Ajisaka d4bb9b2146 HDFS-8002. Website refers to /trash directory. Contributd by Brahma Reddy Battula. 
(cherry picked from commit e7ea2a8e8f)
 2015-03-31 00:28:27 +09:00
Kihwal Lee c4cedfc1d6 HDFS-7742. Favoring decommissioning node for replication can cause a block to stay 
underreplicated for long periods. Contributed by Nathan Roberts.
(cherry picked from commit 04ee18ed48)
 2015-03-30 10:11:25 -05:00
Harsh J c58357939f HDFS-4396. Add START_MSG/SHUTDOWN_MSG for ZKFC. Contributed by Liang Xie. 
(cherry picked from commit ae3e8c61ff)
 2015-03-30 15:22:57 +05:30
Vinayakumar B 9f49b3e93d HDFS-7890. Improve information on Top users for metrics in RollingWindowsManager and lower log level (Contributed by J.Andreina) 
(cherry picked from commit 1ed9fb7664)
 2015-03-30 10:04:11 +05:30
Akira Ajisaka 7aded71f3a HDFS-6408. Remove redundant definitions in log4j.properties. Contributed by Abhiraj Butala. 
(cherry picked from commit 232eca944a)
 2015-03-30 11:26:11 +09:00
Akira Ajisaka a79a38d096 HDFS-6263. Remove DRFA.MaxBackupIndex config from log4j.properties. Contributed by Abhiraj Butala. 
(cherry picked from commit 257c77f895)
 2015-03-30 10:52:45 +09:00
Harsh J b679dc5a8e HDFS-7501. TransactionsSinceLastCheckpoint can be negative on SBNs. Contributed by Gautam Gopalakrishnan. 
(cherry picked from commit 3d9132d434)
 2015-03-29 00:46:22 +05:30
Arpit Agarwal d0bc2eff1a HDFS-7700. Document quota support for storage types. (Contributed by Xiaoyu Yao) 2015-03-27 19:49:33 -07:00
Arun Suresh dadda2edca HDFS-8004. Use KeyProviderCryptoExtension#warmUpEncryptedKeys when creating an encryption zone. (awang via asuresh) 
(cherry picked from commit e97f8e44af)
 2015-03-27 19:25:32 -07:00
Kihwal Lee 577ea08865 HDFS-7990. IBR delete ack should not be delayed. Contributed by Daryn Sharp. 
(cherry picked from commit 60882ab26d)
 2015-03-27 09:06:23 -05:00
Haohui Mai c261d35b75 HADOOP-11748. The secrets of auth cookies should not be specified in configuration in clear text. Contributed by Li Lu and Haohui Mai. 2015-03-26 17:35:28 -07:00
Colin Patrick Mccabe 2a750c9aa6 HDFS-7410. Support CreateFlags with append() to support hsync() for appending streams (Vinayakumar B via Colin P. McCabe) 
(cherry picked from commit 61df1b27a7)
 2015-03-26 13:28:58 -07:00
Arpit Agarwal c035106c05 HDFS-7824. GetContentSummary API and its namenode implementation for Storage Type Quota/Usage. (Contributed by Xiaoyu Yao) 2015-03-26 10:47:43 -07:00
Kihwal Lee 071277b9ce HDFS-7963. Fix expected tracing spans in TestTracing. Contributed by Masatake Iwasaki. 2015-03-26 08:42:45 -05:00
Haohui Mai 6e9202c802 HDFS-7918. Clean up unused code for old web UI in branch-2. Contributed by Brahma Reddy Battula. 2015-03-25 17:46:36 -07:00
Kihwal Lee be4eabdcd4 HDFS-7928. Scanning blocks from disk during rolling upgrade startup takes a lot of time if disks are busy. Contributed by Rushabh Shah. 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
 2015-03-25 14:51:00 -05:00
Jitendra Pandey 456cec127b HDFS-6826. Plugin interface to enable delegation of HDFS authorization assertions. Contributed by Arun Suresh. 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
 2015-03-24 16:02:21 -07:00
Haohui Mai 161dae898f HDFS-7713. Implement mkdirs in the HDFS Web UI. Contributed by Ravi Prakash. 2015-03-24 15:49:08 -07:00
Jing Zhao 483f77b75b HDFS-7854. Separate class DataStreamer out of DFSOutputStream. Contributed by Li Bo. 
(cherry picked from commit a16bfff71b)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
 2015-03-24 11:21:10 -07:00
Brandon Li 260dbe96c3 HDFS-7977. NFS couldn't take percentile intervals. Contributed by Brandon Li 
(cherry picked from commit 570a83ae80)
 2015-03-24 10:50:50 -07:00
Harsh J 608ad6c2cd HDFS-7875. Improve log message when wrong value configured for dfs.datanode.failed.volumes.tolerated. Contributed by Nijel. 
(cherry picked from commit 2da3d2ed2ff2e9b48dbda7e029aa58261c729d35)
 2015-03-24 23:03:56 +05:30
Brandon Li 95bde88986 HDFS-7976. Update NFS user guide for mount option 'sync' to minimize or avoid reordered writes. Contributed by Brandon Li 
(cherry picked from commit 51f1f49374)
 2015-03-24 10:29:27 -07:00
Andrew Wang e39c58fd9d HDFS-7961. Trigger full block report after hot swapping disk. Contributed by Eddy Xu. 
(cherry picked from commit 6413d34986)
 2015-03-24 09:07:24 -07:00
Vinayakumar B c3e2c20c99 HDFS-3325. When configuring 'dfs.namenode.safemode.threshold-pct' to a value greater or equal to 1 there is mismatch in the UI report (Contributed by J.Andreina) 
(cherry picked from commit c6c396fcd6)
 2015-03-24 12:13:21 +05:30
Plamen Jeliazkov 75d6d9e022 HDFS-7956. Improve logging for DatanodeRegistration. Contributed by Plamen Jeliazkov. 2015-03-23 23:07:01 -07:00
Andrew Wang 2f46ee50bd HDFS-7960. The full block report should prune zombie storages even if they're not empty. Contributed by Colin McCabe and Eddy Xu. 
(cherry picked from commit 50ee8f4e67)
 2015-03-23 22:00:44 -07:00
Tsz-Wo Nicholas Sze fe693b72de HDFS-7884. Fix NullPointerException in BlockSender when the generation stamp provided by the client is larger than the one stored in the datanode. Contributed by Brahma Reddy Battula 2015-03-24 13:51:31 +09:00
cnauroth 01c0bcb176 HDFS-7917. Use file to replace data dirs in test to simulate a disk failure. Contributed by Lei (Eddy) Xu. 
(cherry picked from commit 2c238ae4e0)
 2015-03-23 16:30:33 -07:00
Akira Ajisaka fad8c78173 HDFS-7881. TestHftpFileSystem#testSeek fails in branch-2. Contributed by Brahma Reddy Battula. 2015-03-24 06:24:29 +09:00
Brandon Li 503d8e4164 HDFS-7942. NFS: support regexp grouping in nfs.exports.allowed.hosts. Contributed by Brandon Li 
(cherry picked from commit 36af4a913c)
 2015-03-23 10:09:43 -07:00
Kihwal Lee 29642b33cb HDFS-6841. Use Time.monotonicNow() wherever applicable instead of Time.now(). Contributed by Vinayakumar B 
(cherry picked from commit 99a8dcd19528b265d4fda9ae09a17e4af52f2782)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeInfo.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
 2015-03-20 14:05:02 -05:00
Jing Zhao 6be52e42a9 HDFS-7957. Truncate should verify quota before making changes. Contributed by Jing Zhao. 
(cherry picked from commit d368d3647a)
 2015-03-20 11:54:35 -07:00
Jing Zhao a3e16f414c HDFS-7829. Code clean up for LocatedBlock. Contributed by Takanobu Asanuma. 
(cherry picked from commit a6a5aae472)
 2015-03-20 10:51:06 -07:00
Yongjun Zhang 0a41b1bdca HDFS-7835. make initial sleeptime in locateFollowingBlock configurable for DFSClient. Contributed by Zhihai Xu. 
(cherry-picked from commit 15612313f5)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
 2015-03-20 09:25:59 -07:00
yliu 3fac95f48d HDFS-7962. Remove duplicated logs in BlockManager. (yliu) 2015-03-20 05:31:08 +08:00
yliu 90164ffd84 HDFS-7930. commitBlockSynchronization() does not remove locations. (yliu) 2015-03-19 23:24:55 +08:00
Kihwal Lee 2b9173059d HDFS-7932. Speed up the shutdown of datanode during rolling upgrade. Contributed by Kihwal Lee. 
(cherry picked from commit 61a4c7fc98)
 2015-03-19 12:28:36 -05:00
Vinayakumar B 786ba9784d HDFS.7849. Update documentation for enabling a new feature in rolling upgrade ( Contributed by J.Andreina ) 
(cherry picked from commit 5be5129d93)
 2015-03-19 19:06:47 +05:30
Kihwal Lee ceb39c1cc6 HDFS-7816. Unable to open webhdfs paths with "+". Contributed by Haohui Mai 
(cherry picked from commit e79be0ee12)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2015-03-19 08:03:47 -05:00
Vinayakumar B 25515d1421 HDFS-7867. Update action param from 'start' to 'prepare' in rolling upgrade ( Contributed by J.Andreina) Updated JIRA id. 
(cherry picked from commit 355e008ffb)
 2015-03-19 12:11:34 +05:30
Vinayakumar B 4527aa6f77 HDFS-7869. Update action param from 'start' to 'prepare' in rolling upgrade ( Contributed by J.Andreina) 
(cherry picked from commit 338b477ff7)
 2015-03-19 12:01:05 +05:30
Colin Patrick Mccabe 219eb22c15 HDFS-7929. inotify unable fetch pre-upgrade edit log segments once upgrade starts (Zhe Zhang via Colin P. McCabe) 
(cherry picked from commit 43b41f2241)
 2015-03-18 19:08:30 -07:00
Jing Zhao 5a5b244648 HDFS-7587. Edit log corruption can happen if append fails with a quota violation. Contributed by Jing Zhao. 
(cherry picked from commit c7c71cdba5)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2015-03-18 18:56:56 -07:00
Jing Zhao 6dcc79507d HDFS-7943. Append cannot handle the last block with length greater than the preferred block size. Contributed by Jing Zhao. 
(cherry picked from commit bee5a6a64a)
 2015-03-18 18:45:01 -07:00
Colin Patrick Mccabe 79c07bbaca HDFS-7054. Make DFSOutputStream tracing more fine-grained (cmccabe) 
(cherry picked from commit 8234fd0e10)
 2015-03-18 18:15:17 -07:00
Haohui Mai 8eb97ea14e HDFS-7945. The WebHdfs system on DN does not honor the length parameter. Contributed by Haohui Mai. 2015-03-18 16:19:57 -07:00
Arpit Agarwal ca629f4579 HDFS-7914. TestJournalNode#testFailToStartWithBadConfig fails when the default dfs.journalnode.http-address port 8480 is in use. (Contributed by Xiaoyu Yao) 2015-03-18 14:05:24 -07:00
Haohui Mai 572555c9f9 HDFS-7697. Mark the PB OIV tool as experimental. Contributed by Lei (Eddy) Xu. 2015-03-18 13:34:03 -07:00
Arpit Agarwal e7ceb434d0 Fix CHANGES.txt for HDFS-7722. 2015-03-18 12:52:50 -07:00
Arpit Agarwal 6b7cf6e096 HDFS-7951. Fix NPE for TestFsDatasetImpl#testAddVolumeFailureReleasesInUseLock on Linux. (Contributed by Xiaoyu Yao) 2015-03-18 12:46:09 -07:00
Arpit Agarwal 97341a4a1d HDFS-7950. Fix TestFsDatasetImpl#testAddVolumes failure on Windows. (Contributed by Xiaoyu Yao) 2015-03-18 12:35:35 -07:00
Arpit Agarwal a08b1b5703 HDFS-7948. TestDataNodeHotSwapVolumes#testAddVolumeFailures failed on Windows. (Contributed by Xiaoyu Yao) 2015-03-18 12:26:11 -07:00
Haohui Mai b547e6d6e0 HDFS-7953. NN Web UI fails to navigate to paths that contain #. Contributed by kanaka kumar avvaru. 2015-03-18 11:08:29 -07:00
Arpit Agarwal 7d0f84bd8d HDFS-7946. TestDataNodeVolumeFailureReporting NPE on Windows. (Contributed by Xiaoyu Yao) 2015-03-17 21:34:35 -07:00
Colin Patrick Mccabe 455d4aa8a1 HDFS-7940. Add tracing to DFSClient#setQuotaByStorageType (Rakesh R via Colin P. McCabe) 
(cherry picked from commit d8846707c5)
 2015-03-17 11:01:42 -07:00
Vinayakumar B 6ddb1bc857 HDFS-5356. MiniDFSCluster should close all open FileSystems when shutdown() (Contributed by Rakesh R) 
(cherry picked from commit 018893e81e)
 2015-03-17 15:34:48 +05:30
yliu 991ac04afc HDFS-7838. Expose truncate API for libhdfs. (yliu) 2015-03-17 07:24:20 +08:00
Harsh J cbb8858367 HDFS-2360. Ugly stacktrce when quota exceeds. (harsh) 
(cherry picked from commit 046521cd65)
 2015-03-17 10:28:52 +05:30
Konstantin V Shvachko 4568acaa62 HDFS-7886. Fix TestFileTruncate falures. Contributed by Plamen Jeliazkov and Konstantin Shvachko. 2015-03-16 12:55:16 -07:00
Colin Patrick Mccabe c71b54fa32 HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) 
(cherry picked from commit bc9cb3e271)
 2015-03-14 22:41:50 -07:00
Colin Patrick Mccabe f93a2dd94b Revert "HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe)" (Jenkins didn't run yet) 
This reverts commit 7550052b85.
 2015-03-13 18:39:43 -07:00
Colin Patrick Mccabe 7550052b85 HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) 
(cherry picked from commit 5aa892ed48)
 2015-03-13 18:30:36 -07:00
Konstantin V Shvachko 8afbfed5d3 HDFS-2605. Remove redundant "Release 0.21.1" section from CHANGES.txt. Contributed by Allen Wittenauer. 2015-03-13 13:45:13 -07:00
Konstantin V Shvachko b199081f54 HDFS-7903. Cannot recover block after truncate and delete snapshot. Contributed by Plamen Jeliazkov. 2015-03-13 12:40:38 -07:00
Kihwal Lee 464271a5ed HDFS-7435. PB encoding of block reports is very inefficient. Contributed by Daryn Sharp. 
(cherry picked from commit d324164a51)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
 2015-03-13 14:36:34 -05:00
Brandon Li af80a98ace HDFS-7926. NameNode implementation of ClientProtocol.truncate(..) is not idempotent. Contributed by Tsz Wo Nicholas Sze 
(cherry picked from commit f446669afb)
 2015-03-13 10:43:46 -07:00
Colin Patrick Mccabe 7455412a24 HDFS-7722. DataNode#checkDiskError should also remove Storage when error is found. (Lei Xu via Colin P. McCabe) 
(cherry picked from commit b49c3a1813)
 2015-03-12 12:40:38 -07:00
Tsz-Wo Nicholas Sze 981a1fa815 HDFS-6833. DirectoryScanner should not register a deleting block with memory of DataNode. Contributed by Shinichi Yamashita 2015-03-13 02:27:37 +08:00
Akira Ajisaka 6c80a3d34a HDFS-7880. Remove the tests for legacy Web UI in branch-2. Contributed by Brahma Reddy Battula. 2015-03-12 10:41:34 +09:00
Haohui Mai e75e6c66ea HADOOP-11633. Convert remaining branch-2 .apt.vm files to markdown. Contributed by Masatake Iwasaki. 2015-03-11 14:23:44 -07:00
cnauroth 5e99b7cd2f HDFS-7491. Add incremental blockreport latency to DN metrics. Contributed by Ming Ma. 
(cherry picked from commit fb34f45727)
 2015-03-11 14:11:39 -07:00
Colin Patrick Mccabe eefca23e8c HDFS-7830. DataNode does not release the volume lock when adding a volume fails. (Lei Xu via Colin P. McCabe) 
(cherry picked from commit 5c1036d598)
 2015-03-10 19:45:18 -07:00
Akira Ajisaka 2d3c5507ce HDFS-6806. HDFS Rolling upgrade document should mention the versions available. Contributed by J.Andreina. 
(cherry picked from commit 82db3341bf)
 2015-03-10 11:23:06 +09:00
Jing Zhao c7105fcff0 HDFS-7898. Change TestAppendSnapshotTruncate to fail-fast. Contributed by Tsz Wo Nicholas Sze. 
(cherry picked from commit e43882e84a)
 2015-03-09 10:52:39 -07:00
Tsuyoshi Ozawa b46f9e72db HADOOP-11602. Backport 'Fix toUpperCase/toLowerCase to use Locale.ENGLISH.' (ozawa) 2015-03-09 19:53:55 +09:00
Vinod Kumar Vavilapalli d9416317a9 Preparing for 2.8.0 development. Changing SNAPSHOT version to be 2.8 on branch-2. 2015-03-08 20:32:51 -07:00
Vinod Kumar Vavilapalli 8e6f8e9a04 Adding 2.8 section in CHANGES.txt 2015-03-08 20:26:57 -07:00
Chris Douglas 3bc4f3502a HDFS-7411. Change decommission logic to throttle by blocks rather 
than nodes in each interval. Contributed by Andrew Wang
 2015-03-08 18:32:20 -07:00
Brandon Li c8af3b2768 HDFS-6488. Support HDFS superuser in NFSv3 gateway. Contributed by Brandon Li 
(cherry picked from commit 0f8ecb1d0c)
 2015-03-06 15:22:20 -08:00
Haohui Mai ab251fd355 HDFS-7818. OffsetParam should return the default value instead of throwing NPE when the value is unspecified. Contributed by Eric Payne. 2015-03-06 14:26:47 -08:00
Jing Zhao 994dadb9ba HDFS-7885. Datanode should not trust the generation stamp provided by client. Contributed by Tsz Wo Nicholas Sze. 
(cherry picked from commit 24db0812be)
 2015-03-06 10:57:38 -08:00
Jing Zhao c5d52c7b52 HDFS-7855. Separate class Packet from DFSOutputStream. Contributed by Li Bo. 
(cherry picked from commit 952640fa4c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
 2015-03-05 11:11:08 -08:00
Tsuyoshi Ozawa f805d48b19 HADOOP-11648. Set DomainSocketWatcher thread name explicitly. Contributed by Liang Xie. 
(cherry picked from commit 74a4754d1c)
 2015-03-05 16:06:37 +09:00
Tsz-Wo Nicholas Sze f9a2007aff HDFS-7746. Add a test randomly mixing append, truncate and snapshot operations. 2015-03-05 10:23:14 +08:00
Dongming Liang ada545d3b7 HDFS-1522. Combine two BLOCK_FILE_PREFIX constants into one. Contributed by Dongming Liang. 2015-03-04 17:52:25 -08:00
Kihwal Lee f85530f649 HDFS-7434. DatanodeID hashCode should not be mutable. Contributed by Daryn Sharp. 
(cherry picked from commit 722b479469)
 2015-03-04 17:23:00 -06:00
Jing Zhao b2ccf54c14 HDFS-7535. Utilize Snapshot diff report for distcp. Contributed by Jing Zhao. 
(cherry picked from commit ed70fa142c)
 2015-03-04 10:32:39 -08:00
Haohui Mai d020544ef6 HDFS-7879. hdfs.dll does not export functions of the public libhdfs API. Contributed by Chris Nauroth. 2015-03-04 09:17:29 -08:00
Vinayakumar B c24448c27b HDFS-7869. Inconsistency in the return information while performing rolling upgrade ( Contributed by J.Andreina ) 
(cherry picked from commit 3560180b6e)
 2015-03-04 14:39:51 +05:30
Akira Ajisaka 31b3f84601 HADOOP-11666. Revert the format change of du output introduced by HADOOP-6857. Contributed by Byron Wong. 2015-03-03 21:46:28 -08:00
Aaron T. Myers bce3d442ff HDFS-7682. {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes non-snapshotted content. Contributed by Charles Lamb. 
(cherry picked from commit f2d7a67a2c)
 2015-03-03 18:09:31 -08:00
Haohui Mai 65bfde552a HDFS-6565. Use jackson instead jetty json in hdfs-client. Contributed by Akira AJISAKA. 2015-03-03 17:58:21 -08:00
Arpit Agarwal 7e6624c218 HDFS-7757. Misleading error messages in FSImage.java. (Contributed by Brahma Reddy Battula) 2015-03-03 10:55:32 -08:00
Jing Zhao 6090f51725 HDFS-7871. NameNodeEditLogRoller can keep printing 'Swallowing exception' message. Contributed by Jing Zhao. 
(cherry picked from commit b442aeec95)
 2015-03-02 20:23:10 -08:00
Haohui Mai 651aa8761e HDFS-7785. Improve diagnostics information for HttpPutFailedException. Contributed by Chengbing Liu. 2015-03-02 15:35:19 -08:00
Gera Shegalov 52d0724a20 HDFS-7789. DFSck should resolve the path to support cross-FS symlinks. (gera) 
(cherry picked from commit cbb492578e)
 2015-03-02 01:05:15 -08:00
Tsz-Wo Nicholas Sze a5f3156b30 HDFS-7439. Add BlockOpResponseProto's message to the exception messages. Contributed by Takanobu Asanuma 2015-03-02 15:05:06 +08:00
Akira Ajisaka d664705fd1 HDFS-5853. Add "hadoop.user.group.metrics.percentiles.intervals" to hdfs-default.xml (aajisaka) 
(cherry picked from commit aa55fd3096)
 2015-03-01 01:18:36 -08:00
Tsz-Wo Nicholas Sze c52636df3f recommit "HDFS-7769. TestHDFSCLI should not create files in hdfs project root dir." 
(cherry picked from commit acc172e371)
 2015-02-27 14:20:26 -08:00
Akira Ajisaka 0f9289e848 HDFS-7685. Document dfs.namenode.heartbeat.recheck-interval in hdfs-default.xml. Contributed by Kai Sasaki. 
(cherry picked from commit 8719cdd4f6)
 2015-02-27 12:18:46 -08:00
Tsz-Wo Nicholas Sze 657b027bb2 HDFS-7308. Change the packet chunk size computation in DFSOutputStream in order to enforce packet size <= 64kB. Contributed by Takuya Fukudome 2015-02-27 23:46:42 +08:00
Vinayakumar B bc60404eaf HDFS-6753. Initialize checkDisk when DirectoryScanner not able to get files list for scanning (Contributed by J.Andreina) 
(cherry picked from commit 4f75b15628)
 2015-02-27 16:37:03 +05:30
Tsz-Wo Nicholas Sze d223a4a594 Revert "HDFS-7769. TestHDFSCLI should not create files in hdfs project root dir." 
This reverts commit acc172e371.

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2015-02-27 18:27:19 +08:00
cnauroth 5dfb793b05 HDFS-7774. Unresolved symbols error while compiling HDFS on Windows 7/32 bit. Contributed by Kiran Kumar M R. 
(cherry picked from commit c6d5b37a1c)
 2015-02-26 13:37:58 -08:00
Colin Patrick Mccabe 8b3b9568b6 HDFS-7819. Log WARN message for the blocks which are not in Block ID based layout (Rakesh R via Colin P. McCabe) 
(cherry picked from commit f0c980abed)
 2015-02-26 12:03:11 -08:00
Vinayakumar B f0f3d1937d HDFS-7832. Show 'Last Modified' in Namenode's 'Browse Filesystem' (Contributed by Vinayakumar B) 
(cherry picked from commit 166eecf687)
 2015-02-26 14:37:03 +05:30
Tsz-Wo Nicholas Sze 22b13a0ddb HDFS-7537. Add "UNDER MIN REPL'D BLOCKS" count to fsck. Contributed by GAO Rui 2015-02-26 11:47:04 +08:00
Tsz-Wo Nicholas Sze 12fe3afcd3 HDFS-7843. A truncated file is corrupted after rollback from a rolling upgrade. 2015-02-26 10:15:49 +08:00
Benoy Antony 4e400030f6 HDFS-7467. Provide storage tier information for a directory via fsck. (Benoy Antony) 2015-02-25 16:46:41 -08:00
Colin Patrick Mccabe d389a1ae98 HDFS-7495. Remove updatePosition argument from DFSInputStream#getBlockAt() (cmccabe) 
(cherry picked from commit caa42adf20)
 2015-02-25 13:39:02 -08:00
Colin Patrick Mccabe 5c0073e706 HDFS-7668. Backport "Convert site documentation from apt to markdown" to branch-2 (Masatake Iwasaki via Colin P. McCabe) 
(backported from commit 2f1e5dc628)

Conflicts:
     hadoop-hdfs-project/hadoop-hdfs/src/site/apt/ExtendedAttributes.apt.vm
     hadoop-hdfs-project/hadoop-hdfs/src/site/apt/Federation.apt.vm
     hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm
     hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSHighAvailabilityWithNFS.apt.vm
     hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSHighAvailabilityWithQJM.apt.vm
     hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsDesign.apt.vm
     hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm
     hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsQuotaAdminGuide.apt.vm
 2015-02-24 16:18:18 -08:00
Andrew Wang efb7e287f4 HDFS-7763. fix zkfc hung issue due to not catching exception in a corner case. Contributed by Liang Xie. 
(cherry picked from commit 7105ebaa9f)
 2015-02-24 15:31:18 -08:00
Jing Zhao 8346427929 HDFS-7831. Fix the starting index and end condition of the loop in FileDiffList.findEarlierSnapshotBlocks(). Contributed by Konstantin Shvachko. 
(cherry picked from commit 73bcfa99af)
 2015-02-24 10:34:42 -08:00
Tsuyoshi Ozawa 280dda5bcd HDFS-7008. xlator should be closed upon exit from DFSAdmin#genericRefresh(). (ozawa) 
(cherry picked from commit b53fd7163b)
 2015-02-24 23:59:59 +09:00
Colin Patrick Mccabe 453b210d0a HDFS-7805. NameNode recovery prompt should be printed on console (Surendra Singh Lilhore via Colin P. McCabe) 
(cherry picked from commit faaddb6ecb)
 2015-02-23 19:06:01 -08:00
Colin Patrick Mccabe f4da6f7193 HDFS-7807. libhdfs htable.c: fix htable resizing, add unit test (cmccabe) 
(cherry picked from commit 585768667e)
 2015-02-23 18:45:44 -08:00
cnauroth 657a6e389b HDFS-7009. Active NN and standby NN have different live nodes. Contributed by Ming Ma. 
(cherry picked from commit 769507bd7a)
 2015-02-23 15:12:36 -08:00
Arpit Agarwal 83f779e49a HDFS-7806. Refactor: move StorageType from hadoop-hdfs to hadoop-common. (Contributed by Xiaoyu Yao) 2015-02-21 15:48:59 -08:00
yliu eca1588db8 HDFS-7740. Test truncate with DataNodes restarting. (yliu) 2015-02-21 06:33:58 +08:00
cnauroth 9f06fb3aef HDFS-7813. TestDFSHAAdminMiniCluster#testFencer testcase is failing frequently. Contributed by Rakesh R. 
(cherry picked from commit 0d6af574e0)
 2015-02-20 17:01:22 -08:00
cnauroth e09ba94f28 HDFS-7773. Additional metrics in HDFS to be accessed via jmx. Contributed by Anu Engineer. 2015-02-20 12:38:34 -08:00
cnauroth ad77dfc3e4 HDFS-7814. Fix usage string of storageType parameter for "dfsadmin -setSpaceQuota/clrSpaceQuota". Contributed by Xiaoyu Yao. 
(cherry picked from commit 8c6ae0d619)
 2015-02-20 12:22:00 -08:00
Kihwal Lee b9157f92fc HDFS-7788. Post-2.6 namenode may not start up with an image containing inodes created with an old release. Contributed by Rushabh Shah. 
(cherry picked from commit 7ae5255a16)
 2015-02-20 09:09:56 -06:00
Harsh J 5c2c6b00dd HDFS-7752. Improve description for "dfs.namenode.num.extra.edits.retained" and "dfs.namenode.num.checkpoints.retained" properties on hdfs-default.xml. Contributed by Wellington Chevreuil. 
(cherry picked from commit b9a17909ba)
 2015-02-20 19:21:34 +05:30
yliu fee29e4a4f HDFS-7656. Expose truncate API for HDFS httpfs. (yliu) 2015-02-19 08:38:25 +08:00
Haohui Mai 3605928bd0 HDFS-7808. Remove obsolete -ns options in in DFSHAAdmin.java. Contributed by Arshad Mohammad. 2015-02-18 15:14:53 -08:00
cnauroth b1b1e05e18 HDFS-7772. Document hdfs balancer -exclude/-include option in HDFSCommands.html. Contributed by Xiaoyu Yao. 2015-02-18 12:03:00 -08:00
Akira Ajisaka 6d6c684567 HDFS-7780. Update use of Iterator to Iterable in DataXceiverServer and SnapshotDiffInfo. Contributed by Ray Chiang. 
(cherry picked from commit 6dc8812a95)
 2015-02-17 14:49:03 -08:00
Haohui Mai f92a4904bf HDFS-6662. WebHDFS cannot open a file if its path contains "%". Contributed by Gerson Carlos. 2015-02-17 13:05:00 -08:00
Kihwal Lee 8b37b4a78b HDFS-7795. Show warning if not all favored nodes were chosen by namenode. Contributed by Kihwal Lee. 
(cherry picked from commit db6606223c)
 2015-02-17 13:06:41 -06:00
Uma Maheswara Rao G 2cbac36fd3 HDFS-4266. BKJM: Separate write and ack quorum (Rakesh R via umamahesh) 
(cherry picked from commit f0412de1c1)
 2015-02-17 21:31:43 +05:30
Uma Maheswara Rao G 35fecb5306 HDFS-7797. Add audit log for setQuota operation (Rakesh R via umamahesh) 
(cherry picked from commit f24a56787a)
 2015-02-17 20:16:34 +05:30
Tsuyoshi Ozawa 00fb0710b6 HADOOP-11600. Fix up source codes to be compiled with Guava 17.0. (ozawa) 
(cherry picked from commit 2f0f756b26)
 2015-02-17 21:57:58 +09:00
Vinayakumar B 91a5d92916 HDFS-7703. Support favouredNodes for the append for new blocks ( Contributed by Vinayakumar B) 
(cherry picked from commit 89a5449280)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2015-02-17 15:19:03 +05:30
yliu 54e33baaf6 HDFS-7798. Checkpointing failure caused by shared KerberosAuthenticator. (Chengbing Liu via yliu) 2015-02-17 07:50:14 +08:00
cnauroth 441dfa4867 HDFS-7604. Track and display failed DataNode storage locations in NameNode. Contributed by Chris Nauroth. 
(cherry picked from commit 9729b244de)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java
 2015-02-16 14:44:49 -08:00
Arpit Agarwal c6aa6cbfe5 HDFS-7775. Use consistent naming for NN-internal quota related types and functions. (Contributed bu Xiaoyu Yao) 2015-02-13 21:01:54 -08:00
cnauroth 751b834ad1 HDFS-7778. Rename FsVolumeListTest to TestFsVolumeList and commit it to branch-2. Contributed by Lei (Eddy) Xu. 
(cherry picked from commit 2efb2347a9)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java
 2015-02-13 16:30:53 -08:00
Akira Ajisaka 3ad1d8d533 HDFS-4265. BKJM doesn't take advantage of speculative reads. Contributed by Rakesh R. 
(cherry picked from commit 0d521e3326)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2015-02-13 15:25:13 -08:00
Colin Patrick Mccabe 7b9c978df2 HDFS-7686. Re-add rapid rescan of possibly corrupt block feature to the block scanner (cmccabe) 
(cherry picked from commit 8bb9a5000e)
 2015-02-13 15:19:04 -08:00
Colin Patrick Mccabe 6edcb9a503 HDFS-7721. The HDFS BlockScanner may run fast during the first hour (cmccabe) 
(cherry picked from commit 115428176e)
 2015-02-13 15:18:10 -08:00
Colin Patrick Mccabe a54b4a7fc6 HDFS-7430 backport addendum. Add missing change. (cmccabe) 2015-02-13 15:11:14 -08:00
Colin Patrick Mccabe a8c6a96f3b HDFS-7430. Refactor the BlockScanner to use O(1) memory and use multiple threads (cmccabe) 
(cherry picked from commit 6e62a1a672)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeListTest.java
 2015-02-13 15:04:06 -08:00
Arpit Agarwal f507791554 HDFS-7776. Adding additional unit tests for Quota By Storage Type. (Contributed by Xiaoyu Yao) 2015-02-13 12:02:54 -08:00
Allen Wittenauer 730597c20c HDFS-316. Balancer should run for a configurable # of iterations (Xiaoyu Yao via aw) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
	hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm
 2015-02-12 18:02:23 -08:00
Colin Patrick Mccabe a9a361c549 HDFS-7790. Do not create optional fields in DFSInputStream unless they are needed (cmccabe) 
(cherry picked from commit 871cb56152)
 2015-02-12 17:49:11 -08:00
Akira Ajisaka 5696376e29 HDFS-7684. The host:port settings of the deamons should be trimmed before use. Contributed by Anu Engineer. 
(cherry picked from commit 9235e2fb4e94a86671e1494f293c973cf572281c)
 2015-02-12 17:40:48 -08:00
Arpit Agarwal 676fc2d1dc HDFS-7723. Quota By Storage Type namenode implemenation. (Contributed by Xiaoyu Yao) 2015-02-12 10:50:14 -08:00
Colin Patrick Mccabe e35788aa5a HDFS-7694. FSDataInputStream should support "unbuffer" (cmccabe) 
(cherry picked from commit 6b39ad0865)
 2015-02-12 10:42:14 -08:00
Jing Zhao 2156e38d5e HDFS-3689. Add support for variable length block. Contributed by Jing Zhao. 
(cherry picked from commit 2848db814a)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend3.java
 2015-02-12 10:23:13 -08:00
Kihwal Lee 146f559097 Adding missing files from HDFS-7404 
(cherry picked from commit 8d7215d40f)
 2015-02-12 11:18:41 -06:00
Kihwal Lee 39594301e1 HDFS-7704. DN heartbeat to Active NN may be blocked and expire if connection to Standby NN continues to time out. Contributed by Rushabh Shah. 
(cherry picked from commit 38262779bb)
 2015-02-12 09:17:10 -06:00
yliu f8d0825f39 HDFS-7761. cleanup unnecssary code logic in LocatedBlock. (yliu) 2015-02-12 03:09:38 +08:00
Tsz-Wo Nicholas Sze 65a6cf47ec HDFS-6133. Add a feature for replica pinning so that a pinned replica will not be moved by Balancer/Mover. Contributed by zhaoyunjiong 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
 2015-02-11 15:16:34 -08:00
Colin Patrick Mccabe 2ca76df21a HDFS-7771. fuse_dfs should permit FILE: on the front of KRB5CCNAME (cmccabe) 
(cherry picked from commit 50625e660a)
 2015-02-11 12:14:56 -08:00
Haohui Mai 744db47872 HDFS-7736. Fix typos in dfsadmin/fsck/snapshotDiff usage messages. Contributed by Brahma Reddy Battula. 2015-02-11 11:24:44 -08:00
Tsz-Wo Nicholas Sze b49eeddbaf HDFS-7751. Fix TestHDFSCLI for quota with storage type. Contributed by Xiaoyu Yao 2015-02-11 11:10:05 -08:00
Arpit Agarwal 8fa0fc9574 HDFS-7720. Quota by Storage Type API, tools and ClientNameNode Protocol changes. (Contributed by Xiaoyu Yao) 2015-02-11 11:07:53 -08:00
Konstantin V Shvachko 04789db74c HDFS-7760. Document truncate for WebHDFS. Contributed by Konstantin Shvachko. 2015-02-11 10:05:53 -08:00
Konstantin V Shvachko 7f18176cd7 HDFS-7753. Fix Multithreaded correctness Warnings in BackupImage. Contributed by Rakesh R and Konstantin Shvachko. 2015-02-11 00:52:12 -08:00
Tsz-Wo Nicholas Sze acc172e371 HDFS-7769. TestHDFSCLI should not create files in hdfs project root dir. 2015-02-10 17:50:00 -08:00
Konstantin V Shvachko 2b722b904e HDFS-7058. Tests for truncate CLI. Contributed by Dasha Boudnik. 2015-02-09 21:31:02 -08:00
Vinayakumar B a1bf7aecf7 HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause DataNode to register successfully with only one NameNode.(Contributed by Vinayakumar B) 
(cherry picked from commit 3d15728ff5)
 2015-02-10 10:45:14 +05:30
Colin Patrick Mccabe 643a8892d1 HDFS-7718. Store KeyProvider in ClientContext to avoid leaking key provider threads when using FileContext (Arun Suresh via Colin P. McCabe) 
(cherry picked from commit 02340a24f2)
 2015-02-09 20:25:31 -08:00
Colin Patrick Mccabe 1465393d14 HDFS-7744. Fix potential NPE in DFSInputStream after setDropBehind or setReadahead is called (cmccabe) 
(cherry picked from commit a9dc5cd706)
 2015-02-09 20:19:48 -08:00
yliu cc0668ebe6 HDFS-7756. Restore method signature for LocatedBlock#getLocations(). (Ted Yu via yliu) 2015-02-10 02:00:32 +08:00
yliu ae316705bb HADOOP-11510. Expose truncate API via FileContext. (yliu) 2015-02-10 01:43:08 +08:00
Arpit Agarwal ff900eb64a HDFS-7647. DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs. (Contributed by Milan Desai) 2015-02-09 12:17:53 -08:00
Jing Zhao 1e49b40f1f HDFS-7743. Code cleanup of BlockInfo and rename BlockInfo to BlockInfoContiguous. Contributed by Jing Zhao. 
(cherry picked from commit 1382ae525c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/CreateEditsLog.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
 2015-02-08 12:16:09 -08:00
Tsz-Wo Nicholas Sze c5f18ba65b HDFS-7738. Revise the exception message for recover lease; add more truncate tests such as truncate with HA setup, negative tests, truncate with other operations and multiple truncates. 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAAppend.java
 2015-02-07 15:26:06 -08:00
Akira Ajisaka 964a3b0c3e HDFS-7710. Remove dead code in BackupImage.java. Contributed by Xiaoyu Yao. 
(cherry picked from commit cfb829ecd5)
 2015-02-06 19:00:58 -08:00
Akira Ajisaka 2ec796ba80 HDFS-7732. Fix the order of the parameters in DFSConfigKeys. Contributed by Brahma Reddy Battula. 
(cherry picked from commit eaab959a3e)
 2015-02-06 11:28:18 -08:00
yliu e2803f5a51 HDFS-7741. Remove unnecessary synchronized in FSDataInputStream and HdfsDataInputStream. (yliu) 2015-02-06 06:29:52 +08:00
yliu 44d800b353 HDFS-7698. Fix locking on HDFS read statistics and add a method for clearing them. (Colin P. McCabe via yliu) 2015-02-05 23:57:36 +08:00
yliu 3ffe5a1ed2 HDFS-7655. Expose truncate API for Web HDFS. (yliu) 2015-02-05 23:48:55 +08:00
Haohui Mai 67002a5fc0 HDFS-7270. Add congestion signaling capability to DataNode write protocol. Contributed by Haohui Mai. 2015-02-05 11:03:43 -08:00
Tsuyoshi Ozawa 970fdc3ad9 HDFS-7709. Fix findbug warnings in httpfs. Contributed by Rakesh R. 
(cherry picked from commit 20660b7a67)
 2015-02-05 19:57:40 +09:00
Arpit Agarwal 5807afed0a HDFS-7733. NFS: readdir/readdirplus return null directory attribute on failure. (Contributed by Arpit Agarwal) 2015-02-04 16:25:40 -08:00
Colin Patrick Mccabe 76c0757c6e HDFS-7719. BlockPoolSliceStorage#removeVolumes fails to remove some in-memory state associated with volumes. (Lei (Eddy) Xu via Colin P. McCabe) 
(cherry picked from commit 40a415799b)
 2015-02-04 12:46:39 -08:00
Andrew Wang 696266e99c HDFS-7734. Class cast exception in NameNode#main. Contributed by Yi Liu. 
(cherry picked from commit 9175105eea)
 2015-02-04 11:31:49 -08:00
Tsuyoshi Ozawa f92d4fa20f HADOOP-11546. Checkstyle failing: Unable to instantiate DoubleCheckedLockingCheck. (ozawa) 
(cherry picked from commit ba58da29f4)
 2015-02-05 00:51:40 +09:00
Andrew Wang debd63f0b0 HDFS-7712. Switch blockStateChangeLog to use slf4j. 
(cherry picked from commit 3ae38ec7df)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend3.java
 2015-02-03 15:04:16 -08:00
Kihwal Lee c428d303f6 HDFS-7707. Edit log corruption due to delayed block removal again. Contributed by Yongjun Zhang 
(cherry picked from commit 843806d03a)
 2015-02-03 14:46:24 -06:00
Haohui Mai 7cb70d3447 HDFS-6651. Deletion failure can leak inodes permanently. Contributed by Jing Zhao. 2015-02-02 16:32:56 -08:00
Tsz-Wo Nicholas Sze 8426c7d806 HDFS-7696. In FsDatasetImpl, the getBlockInputStream(..) and getTmpInputStreams(..) methods may leak file descriptors. 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
 2015-02-02 13:42:15 -08:00
Tsz-Wo Nicholas Sze a5568a276d HDFS-7681. Change ReplicaInputStreams constructor to take InputStream(s) instead of FileDescriptor(s). Contributed by Joe Pallas 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2015-02-02 13:30:07 -08:00
Tsz-Wo Nicholas Sze 2ecfd018f4 HDFS-5782. Change BlockListAsLongs constructor to take Replica as parameter type instead of concrete classes Block and ReplicaInfo. Contributed by David Powell and Joe Pallas 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2015-02-02 13:24:02 -08:00
Tsz-Wo Nicholas Sze e43b1428b4 HDFS-5631. Change BlockMetadataHeader.readHeader(..), ChunkChecksum class and constructor to public; and fix FsDatasetSpi to use generic type instead of FsVolumeImpl. Contributed by David Powell and Joe Pallas 2015-02-02 12:59:22 -08:00
Colin Patrick Mccabe 4bf14876b3 HADOOP-11498 addendum. Fix extra "import" statements in branch-2.7 patch backport 2015-01-30 16:07:12 -08:00
Colin Patrick Mccabe 80381b62aa HADOOP-11498. Bump the version of HTrace to 3.1.0-incubating (Masatake Iwasaki via Colin P. McCabe) 
(cherry picked from commit 09ad9a868a)
 2015-01-30 16:02:36 -08:00
Andrew Wang fcd23b61e3 HDFS-7706. Switch BlockManager logging to use slf4j. 
(cherry picked from commit 951b3608a8)
 2015-01-30 11:36:02 -08:00
Colin Patrick Mccabe e8d154baea HADOOP-11403. Avoid using sys_errlist on Solaris, which lacks support for it (Malcolm Kavalsky via Colin P. McCabe) 
(cherry picked from commit e36ef3b402)
 2015-01-29 15:48:59 -08:00
Kihwal Lee 3b030731a0 HDFS-7603. The background replication queue initialization may not let others run. Contributed by Kihwal Lee. 
(cherry picked from commit 89b07490f8)
 2015-01-29 12:37:34 -06:00
yliu dc7950ef18 HDFS-7423. various typos and message formatting fixes in nfs daemon and doc. (Charles Lamb via yliu) 2015-01-29 04:20:14 +08:00
Jing Zhao e8300957a7 HDFS-7611. deleteSnapshot and delete of a file can leave orphaned blocks in the blocksMap on NameNode restart. Contributed by Jing Zhao and Byron Wong. 
(cherry picked from commit d244574d03)
 2015-01-28 15:25:42 -08:00
Andrew Wang 55bb4eca73 HDFS-6673. Add delimited format support to PB OIV tool. Contributed by Eddy Xu. 
(cherry picked from commit caf7298e49)
 2015-01-28 12:36:44 -08:00
Colin Patrick Mccabe ad865b1cd0 HDFS-7675. Remove unused member DFSClient.spanReceiverHost (cmccabe) 
(cherry picked from commit d12dd47f45)
 2015-01-27 20:52:53 -08:00
yliu d483ba25d7 HDFS-7677. DistributedFileSystem#truncate should resolve symlinks. (yliu) 2015-01-27 23:47:52 +08:00
Allen Wittenauer 874e38d81f HDFS-7566. Remove obsolete entries from hdfs-default.xml (Ray Chiang via aw) 2015-01-27 14:32:21 -08:00
Haohui Mai b3a63fa8a2 HDFS-7683. Combine usages and percent stats in NameNode UI. Contributed by Vinayakumar B. 2015-01-27 11:09:42 -08:00
Steve Loughran aa16173a0d HDFS-49. MiniDFSCluster.stopDataNode will always shut down a node in the cluster if a matching name is not found. (stevel) 2015-01-27 11:10:46 +00:00
Kihwal Lee c081fa6ae8 HDFS-7224. Allow reuse of NN connections via webhdfs. Contributed by Eric Payne 
(cherry picked from commit 2b0fa20f69)
 2015-01-26 08:15:37 -06:00
Konstantin V Shvachko 24f0d56afb HDFS-7676. Fix TestFileTruncate to avoid bug of HDFS-7611. Contributed by Konstantin Shvachko. 2015-01-24 18:07:56 -08:00
yliu 5f70b7ecf0 HDFS-7659. truncate should check negative value of the new length. Contributed by Yi Liu. 2015-01-24 17:28:36 -08:00
Konstantin V Shvachko de66227a57 HADOOP-11490. Expose truncate API via FileSystem and shell command. Contributed by Milan Desai. 2015-01-24 17:28:34 -08:00
Konstantin V Shvachko 6b22170bee HDFS-7643. Test case to ensure lazy persist files cannot be truncated. (Contributed by Yi Liu) 2015-01-24 17:28:32 -08:00
yliu 8adffbb2c8 HDFS-7634. Disallow truncation of Lazy persist files. (Contributed by Yi Liu) 2015-01-24 16:34:08 -08:00
yliu 0fdd09a895 HDFS-7638. Small fix and few refinements for FSN#truncate. (yliu) 2015-01-24 16:29:50 -08:00
Konstantin V Shvachko a116e1fe9e HDFS-7606. Fix potential NPE in INodeFile.getBlocks(). Contributed by Byron Wong. 2015-01-24 16:16:58 -08:00
Konstantin V Shvachko 00a7ebab22 HDFS-7056. Snapshot support for truncate. Contributed by Konstantin Shvachko and Plamen Jeliazkov. 2015-01-24 16:06:41 -08:00
Plamen Jeliazkov 6ff9bde714 HDFS-3107. Introduce truncate. Contributed by Plamen Jeliazkov. 2015-01-24 16:01:07 -08:00
Allen Wittenauer 6f89692dc9 HDFS-7644. minor typo in HttpFS doc (Charles Lamb via aw) 2015-01-23 13:47:11 -08:00
cnauroth 639c7b0ddc HDFS-3519. Checkpoint upload may interfere with a concurrent saveNamespace. Contributed by Ming Ma. 2015-01-22 16:03:17 -08:00
Arpit Agarwal dcd14db1a8 HDFS-7575. Fix CHANGES.txt 2015-01-22 14:37:43 -08:00
Arpit Agarwal 1d9d166c0b HDFS-7575. Upgrade should generate a unique storage ID for each volume. (Contributed by Arpit Agarwal) 2015-01-22 14:12:56 -08:00
Kihwal Lee 95858db0c1 HDFS-7548. Corrupt block reporting delayed until datablock scanner thread detects it. Contributed by Rushabh Shah. 2015-01-21 14:42:45 -06:00
Colin Patrick Mccabe 5ec2b6caa9 HDFS-7496. Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances (Lei (Eddy) Xu via Colin P. McCabe) 
(cherry-picked from commit a17584936c)
 2015-01-21 11:43:42 -08:00
Tsz-Wo Nicholas Sze 1dc5483054 HDFS-3443. Fix NPE when namenode transition to active during startup by adding checkNNStartup() in NameNodeRpcServer. Contributed by Vinayakumar B 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
 2015-01-21 11:37:58 -08:00
Colin Patrick Mccabe 7779f38e68 HDFS-7610. Fix removal of dynamically added DN volumes (Lei (Eddy) Xu via Colin P. McCabe) 
(cherry picked from commit a17584936c)
 2015-01-20 20:15:33 -08:00
yliu 7b5aea8175 HDFS-7641. Update archival storage user doc for list/set/get block storage policies. (yliu) 2015-01-21 02:04:03 +08:00
yliu 3f37354daa HDFS-7623. Add htrace configuration properties to core-default.xml and update user doc about how to enable htrace. (yliu) 2015-01-21 00:57:51 +08:00
Haohui Mai 755731a6c1 HDFS-7640. print NFS Client in the NFS log. Contributed by Brandon Li. 2015-01-19 17:29:57 -08:00
Jing Zhao 5da33c5b5d HDFS-7637. Fix the check condition for reserved path. Contributed by Yi Liu. 
(cherry picked from commit e843a0a8ce)
 2015-01-19 10:29:02 -08:00
Haohui Mai 29551e19ef HDFS-7573. Consolidate the implementation of delete() into a single class. Contributed by Haohui Mai. 2015-01-17 12:56:44 -08:00
cnauroth 8e765b8513 HDFS-7632. MiniDFSCluster configures DataNode data directories incorrectly if using more than 1 DataNode and more than 2 storage locations per DataNode. Contributed by Chris Nauroth. 
(cherry picked from commit ec4389cf72)
 2015-01-16 10:52:13 -08:00
cnauroth b3d03980c4 HDFS-7635. Remove TestCorruptFilesJsp from branch-2. Contributed by Chris Nauroth. 2015-01-16 10:40:38 -08:00