Commit Graph

4834 Commits

Author SHA1 Message Date
Mingliang Liu b803641523 HDFS-12089. Fix ambiguous NN retry log message in WebHDFS. Contributed by Eric Badger
(cherry picked from commit 6436768baf)
2017-07-05 11:12:15 -07:00
Akira Ajisaka 063f2132ef
HDFS-12078. Add time unit to the description of property dfs.namenode.stale.datanode.interval in hdfs-default.xml. Contributed by Weiwei Yang.
(cherry picked from commit 186650d21d)
2017-07-04 14:52:39 +09:00
Akira Ajisaka 003b4a1675
HDFS-12079. Description of dfs.block.invalidate.limit is incorrect in hdfs-default.xml. Contributed by Weiwei Yang.
(cherry picked from commit b0560e0624)
2017-07-04 14:03:07 +09:00
Wei-Chiu Chuang 94bc5cdbb3 HDFS-12042. Lazy initialize AbstractINodeDiffList#diffs for snapshots to reduce memory consumption. Contributed by Misha Dmitriev.
(cherry picked from commit bcba844d11)
2017-06-30 11:42:51 -07:00
Wei-Chiu Chuang 6917c19167 HDFS-11881. NameNode consumes a lot of memory for snapshot diff report generation. Contributed by Manoj Govindassamy.
(cherry picked from commit 16c8dbde57)
2017-06-29 06:40:24 -07:00
Akira Ajisaka a896c0f11e
HDFS-12040. TestFsDatasetImpl.testCleanShutdownOfVolume fails. Contributed by hu xiaodong.
(cherry picked from commit 07defa4c09)
2017-06-27 20:49:56 +09:00
Ravi Prakash ef491f4637 HDFS-11993. Add log info when connect to datanode socket address failed. Contributed by chencan
(cherry picked from commit a9d3412b4c)
2017-06-26 13:25:32 -07:00
Akira Ajisaka b59b96deee
HDFS-11992. Replace commons-logging APIs with slf4j in FsDatasetImpl. Contributed by hu xiaodong. 2017-06-27 03:28:09 +09:00
Kihwal Lee 8f533ca76b HDFS-11960. Successfully closed files can stay under-replicated. Contributed by Kihwal Lee. 2017-06-20 09:16:37 -05:00
Brahma Reddy Battula 933fad2e17 HADOOP-14533. Size of args cannot be less than zero in TraceAdmin#run as its linkedlist. Contributed by Weisen Han.
(cherry picked from commit 2e9daa2e27)
2017-06-20 20:45:33 +08:00
Akira Ajisaka 6b3c13d543
HDFS-11345. Document the configuration key for FSNamesystem lock fairness. Contributed by Erik Krogen.
(cherry picked from commit 2c769167db)
2017-06-20 16:36:47 +09:00
Akira Ajisaka 71626fd0b9
HDFS-11736. OIV tests should not write outside 'target' directory. Contributed by Yiqun Lin. 2017-06-20 13:55:42 +09:00
Masatake Iwasaki b19ca498ae HDFS-11995. HDFS Architecture documentation incorrectly describes writing to a local temporary file. Contributed by Nandakumar.
(cherry picked from commit d954a64730)
2017-06-20 08:09:09 +09:00
Andrew Wang 1ec1aebd35 HDFS-10480. Add an admin command to list currently open files. Contributed by Manoj Govindassamy. 2017-06-19 09:52:07 -07:00
Lei Xu 33afa1fdca HADOOP-14395. Provide Builder pattern for DistributedFileSystem.append. Contributed by Lei (Eddy) Xu.
(cherry picked from commit 6460df21a0)

    Conflicts:
    hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
2017-06-16 17:32:29 -07:00
Arpit Agarwal 9529513f18 HADOOP-14503. Make RollingAverages a mutable metric. Contributed by Hanisha Koneru. 2017-06-16 16:41:08 -07:00
Xiao Chen 2afe9722af HADOOP-14524. Make CryptoCodec Closeable so it can be cleaned up proactively. 2017-06-16 09:49:41 -07:00
Lei Xu abac844c90 HADOOP-14394. Provide Builder pattern for DistributedFileSystem.create. (lei)
(cherry picked from commit 5fbec46525d6d49837d934556b59ba77bd2301a8)

    Conflicts:
    hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
    hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java
    hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
    hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingPolicies.java
2017-06-15 10:59:11 -07:00
Akira Ajisaka 0fd0612896
Revert "HDFS-11736. OIV tests should not write outside 'target' directory. Contributed by Yiqun Lin."
This reverts commit 88fa17c238.
2017-06-16 00:21:54 +09:00
Akira Ajisaka 88fa17c238
HDFS-11736. OIV tests should not write outside 'target' directory. Contributed by Yiqun Lin.
(cherry picked from commit 315f07700d)
2017-06-15 13:47:22 +09:00
Tsz-Wo Nicholas Sze 670224ae59 HDFS-11947. When constructing a thread name, BPOfferService may print a bogus warning message. Contributed by Weiwei Yang 2017-06-13 09:47:14 +08:00
Arpit Agarwal eda4bf5196 HDFS-11967. TestJMXGet fails occasionally. Contributed by Arpit Agarwal. 2017-06-12 16:04:08 -07:00
Arpit Agarwal 097b1f74e5 HDFS-11907. Add metric for time taken by NameNode resource check. Contributed by Chen Liang. 2017-06-12 15:22:36 -07:00
Mingliang Liu facb6812d9 HDFS-11945. Internal lease recovery may not be retried for a long time. Contributed by Kihwal Lee 2017-06-08 14:46:31 -07:00
Brahma Reddy Battula fca08f8362 HDFS-11711. DN should not delete the block On "Too many open files" Exception. Contributed by Brahma Reddy Battula. 2017-06-07 16:54:29 +08:00
Yiqun Lin 72f0fdfb59 HDFS-11929. Document missing processor of hdfs oiv_legacy command. Contributed by LiXin Ge.
(cherry picked from commit 5ec71639cf)
2017-06-07 13:44:33 +08:00
Vinayakumar B 9a26120809 HDFS-11708. Positional read will fail if replicas moved to different DNs after stream is opened. Contributed by Vinayakumar B. 2017-06-07 10:57:47 +05:30
Andrew Wang 5c819585f2 HDFS-11840. Log HDFS Mover exception message of exit to its own log. Contributed by LiXin Ge.
(cherry picked from commit c31cb879a3)
2017-06-06 14:57:51 -07:00
Arpit Agarwal 226688a12c HDFS-11932. BPServiceActor thread name is not correctly set. Contributed by Chen Liang. 2017-06-06 13:51:25 -07:00
Yongjun Zhang 1e52314117 HDFS-11914. Add more diagnosis info for fsimage transfer failure. Contributed by Yongjun Zhang.
(cherry picked from commit 6a28a2b914)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java
2017-06-06 12:31:11 -07:00
Kihwal Lee e2ea601231 HDFS-10816. TestComputeInvalidateWork#testDatanodeReRegistration fails due to race between test and replication monitor. Contributed by Eric Badger. 2017-06-05 15:24:54 -05:00
Arpit Agarwal baff06f019 HDFS-11928. Segment overflow in FileDistributionCalculator. Contributed by LiXin Ge. 2017-06-05 13:21:35 -07:00
Yiqun Lin 2e8557de12 HDFS-11899. ASF License warnings generated intermittently in trunk. Contributed by Yiqun Lin.
(cherry picked from commit 367da9b7b8)
2017-06-03 22:10:01 +08:00
Brahma Reddy Battula 8e119f1e90 HADOOP-14436. Remove the redundant colon in ViewFs.md. Contributed by maobaolong.
(cherry picked from commit 056cc72885)
2017-06-02 22:42:23 +08:00
Brahma Reddy Battula ac2af66e77 HDFS-11905. Fix license header inconsistency in hdfs. Contributed by Yeliang Cang. 2017-06-02 22:22:43 +08:00
Yiqun Lin 870b5e39d2 HDFS-11359. DFSAdmin report command supports displaying maintenance state datanodes. Contributed by Yiqun Lin. 2017-06-02 12:53:31 +08:00
Andrew Wang d0a4e7e95d HDFS-11383. Intern strings in BlockLocation and ExtendedBlock. Contributed by Misha Dmitriev.
(cherry picked from commit 7101477e47)
2017-06-01 15:20:22 -07:00
Xiao Chen eb98773ac8 HDFS-11904. Reuse iip in unprotectedRemoveXAttrs calls.
(cherry picked from commit 219f4c199e)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirErasureCodingOp.java
2017-06-01 14:16:58 -07:00
Xiao Chen 91b72a8aa5 HDFS-11741. Long running balancer may fail due to expired DataEncryptionKey. Contributed by Wei-Chiu Chuang and Xiao Chen.
(cherry picked from commit 068e23b896c63b0f817e6b91d73c994be1551eb2)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java
2017-06-01 14:05:43 -07:00
Brahma Reddy Battula 5cbdefbcc0 HDFS-11893. Fix TestDFSShell.testMoveWithTargetPortEmpty failure. Contributed by Brahma Reddy Battula.
(cherry picked from commit 0dcf843c00)
2017-06-01 22:39:22 +08:00
Kihwal Lee 212a566088 HDFS-11856. Ability to re-add Upgrading nodes to pipeline for future pipeline updates. Contributed by Vinayakumar B. 2017-05-31 12:45:33 -05:00
Kihwal Lee 5b0baeab5e HDFS-5042. Completed files lost after power failure. Contributed by Vinayakumar B. 2017-05-31 11:02:34 -05:00
Lei Xu a3ad1a39c5 HDFS-11659. TestDataNodeHotSwapVolumes.testRemoveVolumeBeingWritten fail due to no DataNode available for pipeline recovery. Contributed by Lei (Eddy) Xu.
(cherry picked from commit 91d6fe151f)
2017-05-30 11:11:00 -07:00
Yiqun Lin d36e64b8b2 HDFS-11446. TestMaintenanceState#testWithNNAndDNRestart fails intermittently. Contributed by Yiqun Lin.
(cherry picked from commit 31058b243e)
2017-05-28 11:27:09 +08:00
Inigo Goiri e336d92b99 Fix NPE in LazyPersistFileScrubber. Contributed by Inigo Goiri.
(cherry picked from commit 303c8dc9b6)
2017-05-26 13:40:16 -07:00
Xiao Chen 54971c4195 HDFS-11421. Make WebHDFS' ACLs RegEx configurable. Contributed by Harsh J. 2017-05-25 21:01:39 -07:00
Kihwal Lee 2cb63433ab HDFS-11817. A faulty node can cause a lease leak and NPE on accessing data. Contributed by Kihwal Lee. 2017-05-25 17:21:56 -05:00
Brahma Reddy Battula 7bad748091 HDFS-11445. FSCK shows overall health stauts as corrupt even one replica is corrupt. Contributed by Brahma Reddy Battula. 2017-05-25 22:38:12 +08:00
Yongjun Zhang 49dc547252 HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li. 2017-05-24 18:54:51 -07:00
Yongjun Zhang 97317b1aaa Revert "HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li."
This reverts commit c411adac32.
2017-05-24 18:54:51 -07:00
Wei-Chiu Chuang bd0138ea0a Revert "HDFS-10797. Disk usage summary of snapshots causes renamed blocks to get counted twice. Contributed by Sean Mackrory."
This reverts commit 5c96ef365d.
2017-05-24 18:08:30 -07:00
Wei-Chiu Chuang fe2c28df70 Revert "HDFS-11515. -du throws ConcurrentModificationException. Contributed by Istvan Fajth, Wei-Chiu Chuang."
This reverts commit 4f1f16f1a6.
2017-05-24 18:08:10 -07:00
Sunil G ec346d6377 YARN-6584. Correct license headers in hadoop-common, hdfs, yarn and mapreduce. Contributed by Yeliang Cang. 2017-05-23 07:57:36 +05:30
Brahma Reddy Battula ec2468ced1 HDFS-11849. JournalNode startup failure exception should be logged in log file. Contributed By Surendra Singh Lilhore.
(cherry picked from commit 9cab42cc79)
2017-05-23 02:40:11 +08:00
John Zhuge 85f7b7e8e4 HDFS-11579. Make HttpFS Tomcat SSL property sslEnabledProtocols and clientAuth configurable. Contributed by John Zhuge. 2017-05-18 18:04:05 -07:00
Arpit Agarwal 3db695d400 HDFS-11842. TestDataNodeOutlierDetectionViaMetrics UT fails intermittently. Controbuted by Hanisha Koneru. 2017-05-17 19:38:54 -07:00
Arpit Agarwal 0e95ff78c3 HDFS-11827. NPE is thrown when log level changed in BlockPlacementPolicyDefault#chooseRandom() method. Contributed by xupeng. 2017-05-16 19:51:17 -07:00
Kihwal Lee 82ea3f4545 HDFS-11641. Reduce cost of audit logging by using FileStatus instead of HdfsFileStatus. Contributed by Daryn Sharp. 2017-05-16 15:45:41 -05:00
Akira Ajisaka c17cb03a23
HDFS-11833. HDFS architecture documentation descibes outdated placement policy. Contributed by Chen Liang.
(cherry picked from commit 1d1c52b42f)
2017-05-16 11:53:55 -04:00
John Zhuge da7c900fe3 HDFS-11816. Update default SSL cipher list for HttpFS. Contributed by John Zhuge. 2017-05-15 16:26:39 -07:00
Andrew Wang 18b89e902b HDFS-11644. Support for querying outputstream capabilities. Contributed by Manoj Govindassamy. 2017-05-13 21:41:11 -07:00
Jason Lowe aeca471ec1 HDFS-11818. TestBlockManager.testSufficientlyReplBlocksUsesNewRack fails intermittently. Contributed by Nathan Roberts 2017-05-12 17:47:17 -05:00
Vinayakumar B 53d9f56a18 HDFS-11674. reserveSpaceForReplicas is not released if append request failed due to mirror down and replica recovered (Contributed by Vinayakumar B)
(cherry picked from commit 1411612aa4)
2017-05-12 07:44:11 +05:30
Kihwal Lee dfaaa14a47 HDFS-11755. Underconstruction blocks can be considered missing. Contributed by Nathan Roberts. 2017-05-10 14:20:26 -05:00
Jason Lowe 0a4021387f HDFS-11745. Increase HDFS test timeouts from 1 second to 10 seconds. Contributed by Eric Badger
(cherry picked from commit fab73748a1)
2017-05-10 13:03:33 -05:00
Kihwal Lee 0d9322aa5b HDFS-11702. Remove indefinite caching of key provider uri in DFSClient. Contributed by Rushabh S Shah.
(cherry picked from commit cef2815cf4)
2017-05-08 08:29:03 -05:00
Lei Xu 1f30a8a105 HDFS-11687. Add new public encryption APIs required by Hive. (lei)
Change-Id: I4a23a00de63ad18022312ceb1f306a87d032d07c
(cherry picked from commit 25f5d9ad5e)
2017-05-04 12:07:29 -07:00
Arpit Agarwal 89baea96d5 HDFS-11722. Change Datanode file IO profiling sampling to percentage. Contributed by Hanisha Koneru. 2017-05-03 16:30:01 -07:00
Arpit Agarwal 4b7d1ff326 Revert "HDFS-11488. JN log segment syncing should support HA upgrade. Contributed by Hanisha Koneru."
This reverts commit dfc6ef42c4.
2017-05-03 16:29:44 -07:00
Arpit Agarwal dfc6ef42c4 HDFS-11488. JN log segment syncing should support HA upgrade. Contributed by Hanisha Koneru. 2017-05-03 15:53:43 -07:00
Mingliang Liu 616413ecb2 HDFS-11739. Fix regression in tests caused by YARN-679. Contributed by Steve Loughran
(cherry picked from commit 83dded556d)
2017-05-03 11:23:15 -07:00
Erik Krogen 9d6b873d3c HDFS-11717. Add unit test for HDFS-11709 StandbyCheckpointer should handle non-existing legacyOivImageDir gracefully. Contributed by Erik Krogen.
(cherry picked from commit d9014bda93)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
2017-05-02 18:39:37 -07:00
Xiaoyu Yao 3e7f97fb0b HDFS-11593. Change SimpleHttpProxyHandler#exceptionCaught log level from info to debug. Contributed by Xiaobing Zhou.
(cherry picked from commit 1058b40843)
2017-05-01 16:13:30 -07:00
Kihwal Lee f77d138725 HDFS-11714. Newly added NN storage directory won't get initialized and cause space exhaustion. Contributed by Kihwal Lee. 2017-05-01 17:33:24 -05:00
Kihwal Lee 6a6d0ece99 HDFS-11609. Some blocks can be permanently lost if nodes are decommissioned while dead. Contributed by Kihwal Lee. 2017-05-01 14:20:37 -05:00
Zhe Zhang fc46f398d9 HDFS-11709. StandbyCheckpointer should handle an non-existing legacyOivImageDir gracefully. Contributed by Erik Krogen.
(cherry picked from commit d8a3309830)
2017-04-27 09:51:04 -07:00
Konstantin V Shvachko 4cbf5c5c41 HDFS-11384. Balancer disperses getBlocks calls to avoid NameNode's rpc queue saturation. Contributed by Konstantin V Shvachko.
(cherry picked from commit 28eb2aabeb)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
2017-04-26 19:02:48 -07:00
Akira Ajisaka cc66153e12
HDFS-11417. Add datanode admin command to get the storage info. Contributed by Surendra Singh Lilhore.
(cherry picked from commit 93fa48fcf2)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolTranslatorPB.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolServerSideTranslatorPB.java
2017-04-26 18:08:12 +09:00
Xiaoyu Yao fc0e6de929 HADOOP-14207. "dfsadmin -refreshCallQueue" fails with DecayRpcScheduler. Contributed by Surendra Singh Lihore.
(cherry picked from commit 918dee10f56861d15bb6644edcefb4246a3e00a4)
2017-04-25 18:49:44 -07:00
Jason Lowe 31af85c18e HDFS-11691. Add a proper scheme to the datanode links in NN web UI. Contributed by Kihwal Lee
(cherry picked from commit e4321ec843)
2017-04-25 17:31:31 -05:00
Yongjun Zhang 31d7d47070 HDFS-11689. New exception thrown by DFSClient%isHDFSEncryptionEnabled broke hacky hive code. Contributed by Yongjun Zhang.
(cherry picked from commit 5078df7be3)
2017-04-21 14:01:47 -07:00
John Zhuge 5046a45507 HDFS-11685. TestDistributedFileSystem.java fails to compile. Contributed by John Zhuge. 2017-04-19 22:10:18 -07:00
Andrew Wang efb790b0d6 HDFS-11660. TestFsDatasetCache#testPageRounder fails intermittently with AssertionError.
(cherry picked from commit 74a723852d)
2017-04-19 18:10:16 -07:00
Yiqun Lin e278eb3ab5 HDFS-11316. TestDataNodeVolumeFailure#testUnderReplicationAfterVolFailure fails in trunk. Contributed by Yiqun Lin.
(cherry picked from commit 724bb86351)
2017-04-19 12:15:52 -05:00
Arpit Agarwal 5c092097b6 HDFS-11603. Improve slow mirror/disk warnings in BlockReceiver. 2017-04-18 16:20:43 -07:00
Konstantin V Shvachko c3840bde54 HDFS-11634. Optimize BlockIterator when interating starts in the middle. Contributed by Konstantin V Shvachko.
(cherry picked from commit 8dfcd95d58)
2017-04-17 17:06:18 -07:00
Zhe Zhang 051ab77806 HDFS-11615. FSNamesystemLock metrics can be inaccurate due to millisecond precision. Contributed by Erik Krogen.
(cherry picked from commit ad49098eb3)
2017-04-17 16:28:58 -07:00
Yongjun Zhang c411adac32 HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li. 2017-04-14 10:14:02 -07:00
Hanisha Koneru 107f685f64 HDFS-11630. TestThrottledAsyncCheckerTimeout fails intermittently in Jenkins builds. Contributed by Hanisha Koneru. 2017-04-12 17:11:36 -07:00
Hanisha Koneru ed8bb51f59 HDFS-11560. Expose slow disks via NameNode JMX. Contributed by Hanisha Koneru. 2017-04-12 16:59:37 -07:00
Kihwal Lee 8ed230c805 HDFS-11648. Lazy construct the IIP pathname. Contributed by Daryn Sharp. 2017-04-12 13:28:26 -05:00
Chris Nauroth d5e2bd4096 HDFS-11163. Mover should move the file blocks to default storage once policy is unset. Contributed by Surendra Singh Lilhore.
(cherry picked from commit 00ed21a6fedb45a7c8992b8d45adaa83f14af34c)
2017-04-11 21:18:47 -07:00
Mingliang Liu f1a85b2be3 HDFS-11558. BPServiceActor thread name is too long. Contributed by Xiaobing Zhou 2017-04-11 18:09:43 -07:00
Hanisha Koneru f16bb79957 HDFS-11551. Support Timeout when checking single disk. Contributed by Hanisha Koneru. 2017-04-11 16:54:28 -07:00
Xiaoyu Yao 0391c92022 HDFS-11608. HDFS write crashed with block size greater than 2 GB. Contributed by Xiaobing Zhou.
(cherry picked from commit 0eacd4c13b)
2017-04-06 16:15:56 -07:00
Andrew Wang ed8f738be5 HADOOP-14104. Client should always ask namenode for kms provider path. Contributed by Rushabh S Shah. 2017-04-06 14:38:22 -07:00
Xiaoyu Yao 325be6e720 HDFS-11362. StorageDirectory should initialize a non-null default StorageDirType. Contribute by Hanisha Koneru.
(cherry picked from commit dde3bbf351ef3121ae80a96d06668d8fe33c788f)
2017-04-06 14:33:30 -07:00
Andrew Wang 832c2f52f8 HADOOP-13688. Stop bundling HTML source code in javadoc JARs.
(cherry picked from commit 72a2ae6452)
2017-04-06 13:14:59 -07:00
Arpit Agarwal 75e60b6bb7 HDFS-11131. TestThrottledAsyncChecker#testCancellation is flaky. 2017-04-05 17:54:27 -07:00
Xiaobing Zhou 9f20532ef5 HDFS-11628. Clarify the behavior of HDFS Mover in documentation. Contributed by Xiaobing Zhou. 2017-04-05 15:33:19 -07:00
Andrew Wang e154893d3b HDFS-11538. Move ClientProtocol HA proxies into hadoop-hdfs-client. Contributed by Huafeng Wang. 2017-04-05 12:52:34 -07:00
Hanisha Koneru 4cdda4ba87 HDFS-11545. Propagate DataNode's slow disks info to the NameNode via Heartbeat. Contributed by Hanisha Koneru. 2017-04-05 07:35:09 -07:00
Akira Ajisaka 88e1799034
HDFS-9651. Add missing robots.txt files.
(cherry picked from commit 8cb7325527)
2017-04-05 12:48:12 +09:00
Akira Ajisaka a563fa8ee9
HDFS-9651. All web UIs should include a robots.txt file. Contributed by Lars Francke.
(cherry picked from commit 09e6caec3d)
2017-04-05 12:46:09 +09:00
Hanisha Koneru fbe86471cf HDFS-11551. Support Timeout when checking single disk. Contributed by Hanisha Koneru. 2017-04-04 10:49:44 -07:00
Wei-Chiu Chuang 4f1f16f1a6 HDFS-11515. -du throws ConcurrentModificationException. Contributed by Istvan Fajth, Wei-Chiu Chuang.
(cherry picked from commit bc7aff7cec)
2017-04-03 07:36:08 -07:00
Mingliang Liu 9e7f8cbf9a HDFS-11592. Closing a file has a wasteful preconditions in NameNode. Contributed by Eric Badger
(cherry picked from commit e5f0622a6f)
2017-03-30 15:44:44 -07:00
Daniel Templeton e00ad6fca5 HDFS-11571. Typo in DataStorage exception message (Contributed by Anna Budai via Daniel Templeton)
(cherry picked from commit 15e3873dc3)
2017-03-29 10:57:52 -07:00
Andrew Wang 82b4a9c3d0 HDFS-9705. Refine the behaviour of getFileChecksum when length = 0. Contributed by SammiChen and Kai Zheng. 2017-03-28 21:40:53 -07:00
Masatake Iwasaki b7b9d74521 HDFS-11486. Client close() should not fail fast if the last block is being decommissioned. Contributed by Wei-Chiu Chuang and Yiqun Lin.
(cherry picked from commit 64ea62c1cc)
2017-03-28 09:19:42 +09:00
Andrew Wang 10deafee82 HDFS-11170. Add builder-based create API to FileSystem. Contributed by SammiChen and Wei Zhou. 2017-03-27 14:35:03 -07:00
Junping Du 2fefa1fa2e Add HDFS jdiff for 2.8.0 release.
(cherry picked from commit d174c06b01)
2017-03-25 14:33:06 -07:00
Hanisha Koneru a79ad2d122 HDFS-11570. Unit test for NameNodeStatusMXBean. Contributed by Hanisha Koneru. 2017-03-24 14:44:33 -07:00
Xiaobing Zhou 757d9ebcc5 HDFS-11534. Add counters for number of blocks in pending IBR. Contributed by Xiaobing Zhou. 2017-03-24 14:40:00 -07:00
Wei-Chiu Chuang 14414705f7 HDFS-10506. OIV's ReverseXML processor cannot reconstruct some snapshot details. Contributed by Akira Ajisaka. 2017-03-24 08:43:14 -07:00
Zhe Zhang f98f4bb6ca HADOOP-9631. ViewFs should use underlying FileSystem's server side defaults. Contributed by Lohit Vijayarenu and Erik Krogen.
(cherry picked from commit 59d69257a8)
2017-03-22 22:32:07 -07:00
John Zhuge 74fbdd557a HDFS-11561. HttpFS doc errors. Contributed by Yuanbo Liu.
(cherry picked from commit ed2d03f2a5)
2017-03-22 22:16:34 -07:00
Chris Douglas 69f34e812e HDFS-6648. Order of namenodes in ConfiguredFailoverProxyProvider is undefined. Contributed by Inigo Goiri
(cherry picked from commit b104f3a282)
2017-03-20 17:16:15 -07:00
Arpit Agarwal 5d7197b2e6 HDFS-11547. Add logs for slow BlockReceiver while writing data to disk. Contributed by Xiaobing Zhou. 2017-03-17 19:03:41 -07:00
Arpit Agarwal 2b4b85e06f HDFS-11517. Include missed change in previous commit. 2017-03-17 16:12:53 -07:00
Hanisha Koneru dbd673d516 HDFS-11517. Expose slow disks via DataNode JMX. Contributed by Hanisha Koneru 2017-03-17 15:50:00 -07:00
Andrew Wang 7cc88eca43 HDFS-11516. Admin command line should print message to stderr in failure case. Contributed by Kai Sasaki.
(cherry picked from commit 92ea6d74ec)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/cli/util/ErasureCodingCliCmdExecutor.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testErasureCodingConf.xml
2017-03-17 11:34:43 -07:00
Xiaobing Zhou dbdd97d057 HDFS-10394. move declaration of okhttp version from hdfs-client to hadoop-project POM. Contributed by Xiaobing Zhou. 2017-03-16 18:13:55 -07:00
Kihwal Lee a264d98017 HDFS-10601. Improve log message to include hostname when the NameNode is in safemode. Contributed by Kuhu Shukla.
(cherry picked from commit ba62b50eba)
2017-03-16 11:02:48 -05:00
Yiqun Lin c0e061b410 HDFS-11420. Edit file should not be processed by the same type processor in OfflineEditsViewer. Contributed by Yiqun Lin. 2017-03-15 22:37:25 +08:00
Yiqun Lin f731013c88 HDFS-11526. Fix confusing block recovery message. Contributed by Yiqun Lin. 2017-03-14 17:52:31 +08:00
Jing Zhao e03d8ff488 HDFS-11395. RequestHedgingProxyProvider#RequestHedgingInvocationHandler hides the Exception thrown from NameNode. Contributed by Nandakumar.
(cherry picked from commit 55796a0946)
2017-03-13 14:26:44 -07:00
Yiqun Lin 7a65601dd9 HDFS-11512. Increase timeout on TestShortCircuitLocalRead#testSkipWithVerifyChecksum. Contributed by Eric Badger.
(cherry picked from commit 79924266f8)
2017-03-13 18:24:50 +08:00
Lei Xu ee1d3105c2 HDFS-11340. DataNode reconfigure for disks doesn't remove the failed volumes. (Manoj Govindassamy via lei) 2017-03-10 14:37:58 -08:00
Arpit Agarwal 5cdcda34a2 HDFS-11477. Simplify file IO profiling configuration. Contributed by Hanisha Koneru. 2017-03-09 14:34:10 -08:00
Arpit Agarwal b6c4776911 HDFS-11461. DataNode Disk Outlier Detection. Contributed by Hanisha Koneru. 2017-03-09 12:59:48 -08:00
Masatake Iwasaki 60be2e5d8a HDFS-11499. Decommissioning stuck because of failing recovery. Contributed by Lukas Majercak and Manoj Govindassamy.
(cherry picked from commit 385d2cb777)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
2017-03-09 23:37:04 +09:00
Arpit Agarwal 8dc8ecbb91 HDFS-10838. Last full block report received time for each DN should be easily discoverable. Contributed by Surendra Singh Lilhore. 2017-03-06 16:40:00 -08:00
Andrew Wang 6e75c1e2f0 HDFS-11498. Make RestCsrfPreventionHandler and WebHdfsHandler compatible with Netty 4.0.
(cherry picked from commit 5e74196ede)
2017-03-06 15:04:29 -08:00
Yiqun Lin 5b49a96328 HDFS-8741. Proper error msg to be printed when invalid operation type is given to WebHDFS operations. Contributed by Surendra Singh Lilhore.
(cherry picked from commit 3536ce031c)
2017-03-06 19:05:43 +08:00
Yiqun Lin 2288857e3b HDFS-11494. Log message when DN is not selected for block replication. Contributed by Yiqun Lin. 2017-03-04 13:41:49 +08:00
Jing Zhao 30b43a2d4e HDFS-11476. Fix NPE in FsDatasetImpl#checkAndUpdate. Contributed by Xiaobing Zhou.
(cherry picked from commit ac5ae0065a)
2017-03-03 13:35:13 -08:00
Andrew Wang d737a26c5a Revert "HDFS-11477. Simplify file IO profiling configuration. Contributed by Hanisha Koneru."
This reverts commit 5e760f8dc4.
2017-03-02 16:21:32 -08:00
Arpit Agarwal 5e760f8dc4 HDFS-11477. Simplify file IO profiling configuration. Contributed by Hanisha Koneru. 2017-03-02 15:38:32 -08:00
Ming Ma 8ce4863103 HDFS-11412. Maintenance minimum replication config value allowable range should be [0, DefaultReplication]. (Manoj Govindassamy via mingma) 2017-03-02 11:06:48 -08:00
John Zhuge cacaa299cf HDFS-11418. HttpFS should support old SSL clients. Contributed by John Zhuge.
Change-Id: I05b1228f7f168e9b5904365ba9e81be3f29dce74
2017-03-01 19:45:25 -08:00
Brahma Reddy Battula a545ee4d7f HDFS-11432. Federation : Support fully qualified path for Quota/Snapshot/cacheadmin/cryptoadmin commands. Contributed by Brahma Reddy Battula.
(cherry picked from commit dcd03df9f9)
2017-03-01 10:47:47 +05:30
Zhe Zhang 5fb785e154 HDFS-11466. Change dfs.namenode.write-lock-reporting-threshold-ms default from 1000ms to 5000ms. Contributed by Andrew Wang.
(cherry picked from commit d269b488a7)
2017-02-28 09:02:59 -08:00
Wei-Chiu Chuang f3cdf29af4 HDFS-8498. Blocks can be committed with wrong size. Contributed by Jing Zhao. 2017-02-25 21:13:51 -08:00
Arpit Agarwal 98af813098 HDFS-11462. Fix occasional BindException in TestNameNodeMetricsLogger. 2017-02-24 19:53:45 -08:00
Yiqun Lin 31f9ef2203 HDFS-11433. Document missing usages of OfflineEditsViewer processors. Contributed by Yiqun Lin.
(cherry picked from commit 4a58870a04)
2017-02-25 10:03:02 +08:00
Arpit Agarwal fa86ec99a1 HDFS-11295. Check storage remaining instead of node remaining in BlockPlacementPolicyDefault.chooseReplicaToDelete(). Contributed by Marton Elek. 2017-02-24 15:44:19 -08:00
Akira Ajisaka bd3c2a2ee9 HDFS-10620. StringBuilder created and appended even if logging is disabled. Contributed by Staffan Friberg. 2017-02-25 04:17:17 +09:00
Andrew Wang 6c5fd82773 HDFS-11438. Fix typo in error message of StoragePolicyAdmin tool. Contributed by Alison Yu.
(cherry picked from commit d150f061f4)
2017-02-22 15:16:12 -08:00
Ming Ma 0add328c8c HDFS-11411. Avoid OutOfMemoryError in TestMaintenanceState test runs. (Manoj Govindassamy via mingma)
(cherry picked from commit cfcd527323)
2017-02-22 09:42:47 -08:00
Andrew Wang 4428c3da69 HDFS-11406. Remove unused getStartInstance and getFinalizeInstance in FSEditLogOp. Contributed by Alison Yu.
(cherry picked from commit 8ef7ebbc71)
2017-02-21 14:54:32 -08:00
Eric Payne 41d31588d1 HDFS-11404. Increase timeout on TestShortCircuitLocalRead.testDeprecatedGetBlockLocalPathInfoRpc. Contributed by Eric Badger
(cherry picked from commit a07ddef101)
2017-02-21 12:09:29 -06:00
Vinayakumar B 00ca9f1792 HDFS-11177. 'storagepolicies -getStoragePolicy' command should accept URI based path. (Contributed by Surendra Singh Lilhore)
(cherry picked from commit 4804050630)
2017-02-21 18:15:03 +05:30
Xiao Chen 06386b7e78 HDFS-11410. Use the cached instance when edit logging SetAclOp, SetXAttrOp and RemoveXAttrOp.
(cherry picked from commit 02c549484a)
2017-02-16 18:08:16 -08:00
Ming Ma 0f8b99fd34 HDFS-11265. Extend visualization for Maintenance Mode under Datanode tab in the NameNode UI. (Marton Elek via mingma)
(cherry picked from commit a136936d01)
2017-02-15 20:25:04 -08:00
Mingliang Liu 96be70513a HDFS-11375. Display the volume storage type in datanode UI. Contributed by Surendra Singh Lilhore
(cherry picked from commit 0741dd3b9a)
2017-02-15 12:34:43 -08:00
Wei-Chiu Chuang a8531d5d52 HDFS-11333. Print a user friendly error message when plugins are not found. Contributed by Wei-Chiu Chuang.
(cherry picked from commit 859bd159ae)
2017-02-15 02:59:45 -08:00
Yongjun Zhang eb5500f0e7 HDFS-11391. Numeric usernames do no work with WebHDFS FS write access. (Pierre Villard via Yongjun Zhang)
(cherry picked from commit 8e53f2b9b0)
2017-02-14 14:11:45 -08:00
Xiaoyu Yao d72f1c5976 HDFS-11409. DatanodeInfo getNetworkLocation and setNetworkLocation shoud use volatile instead of synchronized. Contributed by Chen Liang.
(cherry picked from commit aaf2713235)
2017-02-14 12:55:49 -08:00
Xiao Chen 92dbdbc490 HDFS-11390. Add process name to httpfs process. Contributed by Weiwei Yang. 2017-02-14 09:46:55 -08:00
Wei-Chiu Chuang 8e291e1015 HDFS-11084. Add a regression test for sticky bit support of OIV ReverseXML processor. Contributed by Wei-Chiu Chuang.
(cherry picked from commit 89cadb42111e4ffbd3f4bde8250013bba23eb51e)
2017-02-14 09:08:37 -08:00
Kihwal Lee 33c62d2d19 HDFS-11379. DFSInputStream may infinite loop requesting block locations. Contributed by Daryn Sharp. 2017-02-10 12:52:14 -06:00
Yiqun Lin 26bed5ce53 HDFS-11377. Balancer hung due to no available mover threads. Contributed by yunjiong zhao.
(cherry picked from commit 9cbbd1eae8)
2017-02-06 13:18:14 +08:00
Arpit Agarwal f17d3eddcf HDFS-11387. Socket reuse address option is not honored in PrivilegedNfsGatewayStarter. Contributed by Mukul Kumar Singh. 2017-02-03 12:28:17 -08:00
Xiao Chen b0a1703f10 HDFS-11363. Need more diagnosis info when seeing Slow waitForAckedSeqno.
(cherry picked from commit 0914fcca31)
2017-02-02 10:09:20 -08:00
Arpit Agarwal b64d9093e1 HDFS-11112. Journal Nodes should refuse to format non-empty directories. Contributed by Yiqun Lin. 2017-02-01 16:52:07 -08:00
Lei Xu d258077bc1 HDFS-11335. Remove HdfsClientConfigKeys.DFS_CLIENT_SLOW_IO_WARNING_THRESHOLD_KEY usage from DNConf. (Manoj Govindassamy via lei)
(cherry picked from commit bec9b7aa1d)
2017-02-01 14:44:08 +08:00
Karthik Kambatla 8b01d5cbba HDFS-11378. Addendum to fix branch-2 compilation. Verify multiple DataNodes can be decommissioned/maintenance at the same time. (Manoj Govindassamy via kasha) 2017-01-29 11:18:54 -08:00
Ming Ma 34f9ceab4a HDFS-11378. Verify multiple DataNodes can be decommissioned/maintenance at the same time. (Manoj Govindassamy via mingma)
(cherry picked from commit 312b36d113)
2017-01-27 16:17:51 -08:00
Masatake Iwasaki 528bff9c42 HDFS-9884. Use doxia macro to generate in-page TOC of HDFS site documentation. (iwasakims) 2017-01-28 08:11:40 +09:00
Anu Engineer d20dc8691e HDFS-9911. TestDataNodeLifeline Fails intermittently. Contributed by Yiqun Lin
(cherry picked from commit a95639068c)
2017-01-27 20:55:59 +00:00
Arpit Agarwal 17d4ba26fc HDFS-11374. Skip FSync in CreateEditsLog to speed up edit log generation. Contributed by Hanisha Koneru. 2017-01-26 16:21:38 -08:00
Xiao Chen 67d2875f8f Revert "HDFS-8377. Support HTTP/2 in datanode. Contributed by Duo Zhang."
This reverts commit f70b9d9241.
2017-01-26 14:21:17 -08:00
Xiao Chen 827cd30923 HDFS-11364. Add a test to verify Audit log entries for setfacl/getfacl commands over FS shell. Contributed by Manoj Govindassamy.
(cherry picked from commit 44606aa850)
2017-01-26 10:58:17 -08:00
Arpit Agarwal deb3ef4587 HDFS-11369. Change exception message in StorageLocationChecker. 2017-01-26 09:12:46 -08:00
Zhe Zhang 558fee285a HDFS-10534. NameNode WebUI should display DataNode usage histogram. Contributed by Kai Sasaki.
(cherry picked from commit 18e1d68209)
2017-01-25 10:39:11 -08:00
Jitendra Pandey 1af0a4d901 HDFS-11365. Log portnumber in PrivilegedNfsGatewayStarter. Contributed by Mukul Kumar Singh. 2017-01-24 21:59:41 -08:00
Arpit Agarwal b4078e1d0e HDFS-11194. Maintain aggregated peer performance metrics on NameNode. 2017-01-24 19:34:33 -08:00
Ming Ma bed700e98f HDFS-11296. Maintenance state expiry should be an epoch time and not jvm monotonic. (Manoj Govindassamy via mingma)
(cherry picked from commit f3fb94be05)
2017-01-19 22:33:43 -08:00
Lei Xu 1cc5f460ed HDFS-11259. Update fsck to display maintenance state info. (Manoj Govindassamy via lei) 2017-01-19 16:24:58 +08:00
Lei Xu 541efe18c7 HDFS-11287. Storage class member storageDirs should be private to avoid unprotected access by derived classes. (Manoj Govindassamy via lei)
(cherry picked from commit 63320d1daa)
2017-01-19 15:38:40 +08:00
Vinitha Reddy Gankidi e2b3eff641 HDFS-10733. NameNode terminated after full GC thinking QJM is unresponsive. Contributed by Vinitha Gankidi. 2017-01-18 12:52:58 -08:00
Akira Ajisaka db47bd15dd HDFS-11290. TestFSNameSystemMBean should wait until JMX cache is cleared. Contributed by Erik Krogen.
(cherry picked from commit b1a9ec856b)
2017-01-18 17:56:22 +09:00
Arpit Agarwal 42a14b6db9 HDFS-10917. Collect peer performance statistics on DataNode. Contributed by Xiaobing Zhou. 2017-01-17 15:31:18 -08:00
Xiaoyu Yao 420e4db98a HDFS-11209. SNN can't checkpoint when rolling upgrade is not finalized. Contributed by Xiaoyu Yao.
(cherry picked from commit b1fce2b8b1)
2017-01-17 14:36:03 -08:00
Xiaoyu Yao eae5bde007 HDFS-10913. Introduce fault injectors to simulate slow mirrors. Contributed by Xiaobing Zhou. 2017-01-17 10:30:34 -08:00
Arpit Agarwal be8e001f1f HDFS-11339. Support File IO sampling for Datanode IO profiling hooks. Contributed by Hanisha Koneru. 2017-01-16 22:49:14 -08:00
Arpit Agarwal 52fdd387ec HDFS-11299. Support multiple Datanode File IO hooks. Contributed by Hanisha Koneru. 2017-01-16 22:42:13 -08:00
Arpit Agarwal 8ec9dca2e1 HDFS-11279. Cleanup unused DataNode#checkDiskErrorAsync(). Contributed by Hanisha Koneru. 2017-01-16 22:14:06 -08:00
Xiaoyu Yao eafaddca1a HDFS-11274. Datanode should only check the failed volume upon IO errors. Contributed by Xiaoyu Yao. 2017-01-16 22:14:04 -08:00
Arpit Agarwal ec80de3ccc HDFS-11182. Update DataNode to use DatasetVolumeChecker. 2017-01-16 22:14:00 -08:00
Arpit Agarwal 413dccaf78 HDFS-10959. Update DataNode to use DatasetVolumeChecker. Contributed by Xiaoyu Yao. 2017-01-16 22:13:58 -08:00
Akira Ajisaka 2d6be7ea23 HADOOP-13933. Add haadmin -getAllServiceState option to get the HA state of all the NameNodes/ResourceManagers. Contributed by Surendra Singh Lilhore.
(cherry picked from commit e407449ddb)
2017-01-17 10:22:29 +09:00
Jitendra Pandey ffb4c22f6e HDFS-11307. The rpc to portmap service for NFS has hardcoded timeout. Contributed by Mukul Kumar Singh. 2017-01-16 15:17:29 -08:00
Arpit Agarwal 954dae26cd HDFS-11337. (HDFS-10958 backport). Add instrumentation hooks around Datanode disk IO. 2017-01-16 13:53:30 -08:00
Wei-Chiu Chuang 914eeb997b HDFS-11306. Print remaining edit logs from buffer if edit log can't be rolled. Contributed by Wei-Chiu Chuang.
(cherry picked from commit 1cde954a4f)
2017-01-13 11:46:57 -08:00
Andrew Wang b089e122b1 HDFS-11312. Fix incompatible tag number change for nonDfsUsed in DatanodeInfoProto. Contributed by Sean Mackrory.
(cherry picked from commit d51f8ba808)
2017-01-11 15:13:41 -08:00
Ming Ma b6258e2b15 HDFS-9391. Update webUI/JMX to display maintenance state info. (Manoj Govindassamy via mingma) 2017-01-10 20:20:13 -08:00
Mingliang Liu ce5ad0e88f HDFS-11253. FileInputStream leak on failure path in BlockSender. Contributed by Arpit Agarwal 2017-01-10 14:48:28 -08:00
Xiaoyu Yao 48fb796fcb HDFS-10930. Refactor: Wrap Datanode IO related operations. Contributed by Xiaoyu Yao. 2017-01-10 14:48:23 -08:00
Yongjun Zhang 137547579b HDFS-11292. log lastWrittenTxId etc info in logSyncAll. Contributed by Yongjun Zhang.
(cherry picked from commit 603cbcd513)
2017-01-09 16:16:42 -08:00
Arpit Agarwal 6f35700f04 HDFS-11149. Support for parallel checking of FsVolumes.
Change-Id: I6e7ada81562f380a260f8f561d0fa6a4cb102560
2017-01-06 07:55:19 -08:00
Arpit Agarwal fd3b1ca26c HDFS-11148. Update DataNode to use StorageLocationChecker at startup.
Change-Id: I61dfae840e767f9a33c0bb45a058f2b866cbffa4
2017-01-05 20:49:34 -08:00
Arpit Agarwal 76f1ab524c HDFS-11119. Support for parallel checking of StorageLocations on DataNode startup.
Change-Id: Iddedbeefd056af165be3557ccad199fe2878b591
2017-01-05 20:49:30 -08:00
Chris Nauroth 9c61858ee7 HDFS-9483. Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS. Contributed by Surendra Singh Lilhore.
(cherry picked from commit 4c8f9e1302)
2017-01-05 15:05:01 -08:00
Haohui Mai 20486730e8 HDFS-11280. Allow WebHDFS to reuse HTTP connections to NN. Contributed by Zheng Shao. 2017-01-04 21:03:05 -08:00
Andrew Wang d4ca1c5226 HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang. 2017-01-04 15:10:15 -08:00
Andrew Wang 2b5b62f9cc Revert "HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang."
This reverts commit 80e6640848.
2017-01-03 11:12:53 -08:00
Andrew Wang 80e6640848 HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang.
(cherry picked from commit 7fcc73fc0d)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
2017-01-03 09:58:45 -08:00
Brahma Reddy Battula 029b6fbe71 Revert "HDFS-11280. Allow WebHDFS to reuse HTTP connections to NN. Contributed by Zheng Shao."
This reverts commit 1bc9b316ba.
2017-01-03 19:26:44 +05:30
Haohui Mai 1bc9b316ba HDFS-11280. Allow WebHDFS to reuse HTTP connections to NN. Contributed by Zheng Shao. 2016-12-30 22:59:24 -08:00
Lei Xu db81cc3add HDFS-11267. Avoid redefinition of storageDirs in NNStorage and cleanup its accessors in Storage. (Manoj Govindassamy via lei)
(cherry picked from commit a4f66655ec)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
2016-12-29 17:06:21 +08:00
Lei Xu f349c7692d HDFS-11251. ConcurrentModificationException during DataNode#refreshVolumes. (Manoj Govindassamy via lei)
(cherry picked from commit e9f1396834)
2016-12-29 15:35:17 +08:00
Daniel Templeton af266c8c85 HDFS-11278. Add missing @Test annotation for TestSafeMode.testSafeModeUtils() (Contributed by Lukas Majercak via Daniel Templeton)
(cherry picked from commit ce3613c965)
2016-12-28 20:11:09 -08:00
Daniel Templeton 47f0a0e15b HADOOP-13932. Fix indefinite article in comments (Contributed by LiXin Ge via Daniel Templeton) 2016-12-28 07:03:51 -08:00
Brahma Reddy Battula 5d46d5d4c6 HDFS-11252. TestFileTruncate#testTruncateWithDataNodesRestartImmediately can fail with BindException. Contributed by Yiqun Lin.
(cherry picked from commit 0ddb8defad)
2016-12-28 10:37:21 +05:30
Mingliang Liu 74ee21a7b1 HDFS-11270. Document the missing options of NameNode bootstrap command. Contributed by Yiqun Lin
(cherry picked from commit c0e0ef2969)
2016-12-27 11:38:31 -08:00
Akira Ajisaka cd331556f6 HDFS-11272. Refine the assert messages in TestFSDirAttrOp. Contributed by Jimmy Xiang.
(cherry picked from commit ea547529cb)
2016-12-26 17:17:14 +09:00
Akira Ajisaka 1b39268233 HDFS-11250. Fix a typo in ReplicaUnderRecovery#setRecoveryID. Contributed by Yiqun Lin.
(cherry picked from commit 8f218ea284)
2016-12-26 16:52:43 +09:00
Akira Ajisaka a779b4ab8f HDFS-11271. Typo in NameNode UI. Contributed by Wei-Chiu Chuang.
(cherry picked from commit 483cd06ad4)
2016-12-26 16:37:52 +09:00
Akira Ajisaka 16c4dcad85 HDFS-11258. Addendum patch to fix compile issue. Contributed by Jimmy Xiang. 2016-12-23 03:56:12 +09:00
Haohui Mai de926f93c2 HDFS-11258. File mtime change could not save to editlog. Contributed by Jimmy Xiang. 2016-12-21 14:22:09 -08:00
Xiao Chen 1ab279c311 HDFS-11195. Return error when appending files by webhdfs rest api fails. Contributed by Yuanbo Liu.
(cherry picked from commit 5b7acdd206)
2016-12-20 12:27:40 -08:00
Brahma Reddy Battula 7a92345f16 HDFS-11263. ClassCastException when we use Bzipcodec for Fsimage compression. Contributed by Brahma Reddy Battula.
(cherry picked from commit 1b401f6a73)
2016-12-20 21:02:16 +05:30
Akira Ajisaka a95acbcb6e HDFS-11262. Remove unused variables in FSImage.java. Contributed by Jagadesh Kiran N.
(cherry picked from commit d2026ae9e6)
2016-12-20 14:36:03 +09:00
Wei-Chiu Chuang 0cb99db9d9 HDFS-11160. VolumeScanner reports write-in-progress replicas as corrupt incorrectly. Contributed by Wei-Chiu Chuang and Yongjun Zhang. 2016-12-15 16:35:37 -08:00
Mingliang Liu 012d28a1fa HDFS-11094. Send back HAState along with NamespaceInfo during a versionRequest as an optional parameter. Contributed by Eric Badger 2016-12-15 14:54:40 -08:00
Akira Ajisaka eaa50a1087 HDFS-11249. Redundant toString() in DFSConfigKeys.java. Contributed by Jagadesh Kiran N.
(cherry picked from commit 36947f79ba)
2016-12-16 00:34:43 +09:00
Akira Ajisaka 190b2dca62 HDFS-11204. Document the missing options of hdfs zkfc command. Contributed by Yiqun Lin.
(cherry picked from commit 72bff192cd)
2016-12-14 18:53:24 +09:00
Andrew Wang 6b795c34d0 HDFS-10684. WebHDFS DataNode calls fail without parameter createparent. Contributed by John Zhuge.
(cherry picked from commit fbdbbd57cd)
2016-12-13 18:01:35 -08:00
Mingliang Liu 282a562437 HDFS-11226. cacheadmin, cryptoadmin and storagepolicyadmin should support generic options. Contributed by Brahma Reddy Battula
(cherry picked from commit e91494c27a0b9a4a456d92fe7d632a608c4f11f3)
2016-12-12 18:17:54 -08:00
Vinayakumar B c6fb9f0d0b HDFS-10570. Remove classpath conflicts of netty-all jar in hadoop-hdfs-client (Contributed by Vinayakumar B) 2016-12-12 12:11:11 +05:30
Xiaoyu Yao 0bdaa23334 Revert "HDFS-10930. Refactor: Wrap Datanode IO related operations. Contributed by Xiaoyu Yao."
This reverts commit 4c9ca47386.
2016-12-09 21:37:13 -08:00
Xiaoyu Yao 4c9ca47386 HDFS-10930. Refactor: Wrap Datanode IO related operations. Contributed by Xiaoyu Yao.
(cherry picked from commit df983b524a)
2016-12-09 21:31:45 -08:00
Wei-Chiu Chuang 292bd78b44 HDFS-11229. HDFS-11056 failed to close meta file. Contributed by Wei-Chiu Chuang. 2016-12-09 16:07:56 -08:00
Andrew Wang 215abbdf5f HDFS-8630. WebHDFS : Support get/set/unset StoragePolicy. Contributed by Surendra Singh Lilhore. 2016-12-09 14:29:22 -08:00
Andrew Wang 85adc6677b HDFS-11217. Annotate NameNode and DataNode MXBean interfaces as Private/Stable. Contributed by Jagadesh Kiran N.
(cherry picked from commit 5bd7dece92)
2016-12-09 12:05:45 -08:00
Vinayakumar B 199d8b5594 HDFS-11224. Lifeline message should be ignored for dead nodes (Contributed by Vinayakumar B)
(cherry picked from commit d1d4aba71b)
2016-12-09 14:54:37 +05:30
Xiao Chen e51f32f74c HDFS-11197. Listing encryption zones fails when deleting a EZ that is on a snapshotted directory. Contributed by Wellington Chevreuil.
(cherry picked from commit 401c731872)
2016-12-08 12:41:17 -08:00
Akira Ajisaka 0478597ea9 HDFS-11223. Fix typos in HttpFs documentations. Contributed by Yiqun Lin.
(cherry picked from commit 4c2cf5560f)
2016-12-08 20:53:09 +09:00
Akira Ajisaka e00a124cb5 HDFS-11140. Directory Scanner should log startup message time correctly. Contributed by Yiqun Lin.
(cherry picked from commit 9ef89ede2f)
2016-12-08 13:39:58 +09:00
Vinayakumar B 4b21b47f56 HDFS-9057. allow/disallow snapshots via webhdfs (Contributed by Brahma Reddy Battula)
(cherry picked from commit 6d2332ae37)
2016-12-07 15:58:15 -08:00
Ming Ma d2656dc5a6 HDFS-10206. Datanodes not sorted properly by distance when the reader isn't a datanode. (Nandakumar via mingma) 2016-12-07 08:27:17 -08:00
Karthik Kambatla 8773cd5f76 HADOOP-13859. TestConfigurationFieldsBase fails for fields that are DEFAULT values of skipped properties. (Haibo Chen via kasha)
(cherry picked from commit c8d0a049b0)
2016-12-06 14:16:38 -08:00
Mingliang Liu eb9e7820a9 HDFS-11211. Add a time unit to the DataNode client trace format. Contributed by Jagadesh Kiran N
(cherry picked from commit 43cb1678cc)
2016-12-06 09:13:38 -08:00
Kihwal Lee f74dad7414 HDFS-11198. NN UI should link DN web address using hostnames. Contributed by Weiwei Yang. 2016-12-06 08:07:54 -06:00
Andrew Wang 6a2f239d9e Revert "HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang"
This reverts commit be969e5918.
2016-12-05 23:09:14 -08:00
Andrew Wang 2fdae7324b HDFS-10581. Hide redundant table on NameNode WebUI when no nodes are decomissioning. Contributed by Weiwei Yang.
(cherry picked from commit 8e63fa98ea)
2016-12-05 18:14:06 -08:00
Wei-Chiu Chuang b36af9b76c HDFS-11181. Fuse wrapper has a typo. Contributed by Wei-Chiu Chuang. 2016-12-05 08:46:57 -08:00
Mingliang Liu be969e5918 HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang
(cherry picked from commit c7ff34f8dc)
2016-12-02 11:36:11 -08:00
Akira Ajisaka 5a7941a4fc HDFS-11180. Intermittent deadlock in NameNode when failover happens. 2016-12-02 11:34:05 +09:00
Kihwal Lee 0d8a35bd6d HDFS-8674. Improve performance of postponed block scans. Contributed by Daryn Sharp. 2016-12-01 12:15:15 -06:00
Ravi Prakash fdf5394b2a HDFS-8678. Bring back the feature to view chunks of files in the HDFS file browser. Contributed by Ivo Udelsmann.
(cherry picked from commit 625df87c7b)
2016-11-30 09:14:36 -08:00
Akira Ajisaka 0c895e8a62 HADOOP-13812. Upgrade Tomcat to 6.0.48. Contributed by John Zhuge.
(cherry picked from commit 00096dcc0c)
2016-11-29 19:50:53 +09:00
Robert Kanter 285edf691d HADOOP-13838. KMSTokenRenewer should close providers (xiaochen via rkanter)
(cherry picked from commit 47ca9e26fb)
2016-11-28 18:12:53 -08:00
Akira Ajisaka 6fe5ffd05d HDFS-11174. Wrong HttpFS test command in doc. Contributed by John Zhuge.
(cherry picked from commit bb1b5e083b)
2016-11-25 21:29:28 +09:00
Akira Ajisaka 0df7f5cdf6 HDFS-11175. Document uppercase key names are not supported in TransparentEncryption.md. Contributed by Yiqun Lin.
(cherry picked from commit 636f7547b7)
2016-11-25 17:50:14 +09:00
Brahma Reddy Battula 2207a84a7d HADOOP-13646. Remove outdated overview.html. Contributed By Brahma Reddy Battula.
(cherry picked from commit afcf8d38e7)
2016-11-22 20:32:30 +05:30
Kihwal Lee 774b0cd845 HDFS-10966. Enhance Dispatcher logic on deciding when to give up a source DataNode. Contributed by Mark Wagner and Zhe Zhang. 2016-11-21 10:39:04 -06:00
Brahma Reddy Battula aa76fe4132 HDFS-11101. TestDFSShell#testMoveWithTargetPortEmpty fails intermittently. Contributed By Brahma Reddy Battula.
(cherry picked from commit f922067748)
2016-11-21 19:36:10 +05:30
Brahma Reddy Battula f3649deca5 HDFS-11144. TestFileCreationDelete#testFileCreationDeleteParent fails wind bind exception. Contributed By Brahma Reddy Battula.
(cherry picked from commit c68dad18ab)
2016-11-21 10:59:23 +05:30
Brahma Reddy Battula 3e5a85f35e HDFS-11134. Fix bind exception threw in TestRenameWhileOpen. Contributed By Yiqun Lin.
(cherry picked from commit b2d4b7b1b9)
2016-11-17 10:24:21 +05:30
Arpit Agarwal da8c24bbad HDFS-11114. Support for running async disk checks in DataNode. 2016-11-16 13:38:15 -08:00
Mingliang Liu 57a0f84860 HDFS-11105. TestRBWBlockInvalidation#testRWRInvalidation fails intermittently. Contributed by Yiqun Lin
(cherry picked from commit c90891e7b3)
2016-11-16 13:06:25 -08:00
Kihwal Lee f08e227671 HDFS-8870. Lease is leaked on write failure. Contributed by Kuhu Shukla.
(cherry picked from commit 4fcea8a0c8)
2016-11-15 14:51:34 -06:00
Mingliang Liu 4e0fcff8ab HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu
(cherry picked from commit 5af572b644)
2016-11-15 11:36:08 -08:00
Zhe Zhang 818915aabb HDFS-10872. Add MutableRate metrics for FSNamesystemLock operations. Contributed by Erik Krogen. 2016-11-14 16:33:15 -08:00
Brahma Reddy Battula 399abe7c74 HDFS-11135. The tests in TestBalancer run fails due to NPE. Contributed By Yiqun Lin
(cherry picked from commit 6efb8c9c60)
2016-11-14 20:41:55 +05:30
Erik Krogen 98d9e9c4c8 HDFS-11087. NamenodeFsck should check if the output writer is still writable. Contributed by Erik Krogen. 2016-11-13 14:50:33 -08:00
Mingliang Liu fb3a5085e8 HADOOP-13813. TestDelegationTokenFetcher#testDelegationTokenWithoutRenewer is failing. Contributed by Mingliang Liu 2016-11-12 10:36:14 -08:00
Mingliang Liu 685d753d07 HDFS-11129. TestAppendSnapshotTruncate fails with bind exception. Contributed by Brahma Reddy Battula
(cherry picked from commit 2ee18fc15e)
2016-11-11 22:16:05 -08:00
Xiaoyu Yao 5613b0818c HDFS-10941. Improve BlockManager#processMisReplicatesAsync log. Contributed by Chen Liang.
(cherry picked from commit 4484b48498)
2016-11-11 14:31:54 -08:00
Mingliang Liu 289f03e49e HDFS-11122. TestDFSAdmin#testReportCommand fails due to timed out. Contributed by Yiqun Lin
(cherry picked from commit aa6010ccca)
2016-11-11 12:30:22 -08:00
Brahma Reddy Battula b51806b45f HDFS-11128. CreateEditsLog throws NullPointerException. Contributed by Hanisha Koneru.
(cherry picked from commit 1ae57f0f75)
2016-11-11 20:09:49 +05:30
Xiao Chen a6146a1026 HDFS-11120. TestEncryptionZones should waitActive. Contributed by John Zhuge.
(cherry picked from commit 907b1431c0ead934e26cf7dbc51bfab5d7cc5ddc)
2016-11-09 17:16:31 -08:00
Wei-Chiu Chuang 16430c64b7 HDFS-11056. Concurrent append and read operations lead to checksum error. Contributed by Wei-Chiu Chuang. 2016-11-09 13:28:16 -08:00
Brahma Reddy Battula 53ff3c9e03 HDFS-9482. Replace DatanodeInfo constructors with a builder pattern. Contributed by Brahma Reddy Battula. 2016-11-09 09:58:36 +05:30
Mingliang Liu f5c9f9f1b2 HDFS-11083. Add unit test for DFSAdmin -report command. Contributed by Xiaobing Zhou
(cherry picked from commit 62d8c17dfd)
2016-11-08 17:05:34 -08:00
Andrew Wang 4f25fd975e HADOOP-13789. Hadoop Common includes generated test protos in both jar and test-jar. Contributed by Sean Busbey.
(cherry picked from commit e1c6ef2efa)
2016-11-08 16:33:58 -08:00
Arpit Agarwal b77239b46d Revert "HDFS-11114. Support for running async disk checks in DataNode."
This reverts commit 732eaadddb.
2016-11-08 09:04:58 -08:00
Arpit Agarwal 732eaadddb HDFS-11114. Support for running async disk checks in DataNode.
This closes #153.
2016-11-07 18:46:37 -08:00
Xiao Chen f30d338766 HDFS-10756. Expose getTrashRoot to HTTPFS and WebHDFS. Contributed by Yuanbo Liu.
(cherry picked from commit d8bab3dcb6)
2016-11-04 18:08:46 -07:00
Mingliang Liu 4cee535aa5 HDFS-11085. Add unit test for NameNode failing to start when name dir is unwritable. Contributed by Xiaobing Zhou
(cherry picked from commit 0c0ab102ab)
2016-11-04 14:39:54 -07:00
Andrew Wang 9ede5ccfdd HADOOP-11088. Quash unnecessary safemode WARN message during NameNode startup. Contributed by Yiqun Lin.
(cherry picked from commit 5cad93d5c7)
2016-11-03 15:31:21 -07:00
Mingliang Liu 6d0cea4a5a HDFS-11076. Add unit test for extended Acls. Contributed by Chen Liang
(cherry picked from commit 7534aee09a)
2016-11-03 13:52:18 -07:00
Arpit Agarwal 458b636a39 HDFS-11097. Fix warnings for deprecated StorageReceivedDeletedBlocks constructor. Contributed by Yiqun Lin. 2016-11-03 10:32:11 -07:00
Andrew Wang 99d0b3eb2b HDFS-11080. Update HttpFS to use ConfigRedactor. Contributed by Sean Mackrory. 2016-11-02 19:11:22 -07:00
Mingliang Liu 7a662bea89 HDFS-11095. BlockManagerSafeMode should respect extension period default config value (30s). Contributed by Mingliang Liu
(cherry picked from commit 9b0c17f852)
2016-11-02 15:02:08 -07:00
Brahma Reddy Battula 35e6081020 HDFS-11031. Add additional unit test for DataNode startup behavior when volumes fail. Contributed by Mingliang Liu. 2016-11-02 10:39:12 +05:30
Mingliang Liu 1dd59042ac HDFS-11065. Add space quota tests for heterogenous storages. Contributed by Xiaobing Zhou
(cherry picked from commit fa1512a740)
2016-11-01 14:46:24 -07:00
Kihwal Lee 7c1a1834e4 HDFS-10997. Reduce number of path resolving methods. Contributed by Daryn Sharp. 2016-11-01 08:04:54 -05:00
Brahma Reddy Battula 1c5708f6de HDFS-10455. Logging the username when deny the setOwner operation. Contributed by Rakesh R. 2016-11-01 10:49:43 +05:30
Mingliang Liu 6bb12ecc3f HDFS-11030. TestDataNodeVolumeFailure#testVolumeFailure is flaky (though passing). Contributed by Mingliang Liu 2016-10-31 16:03:02 -07:00
Arpit Agarwal 56092e351b HDFS-11074. Remove unused method FsDatasetSpi#getFinalizedBlocksOnPersistentStorage. Contributed by Hanisha Koneru. 2016-10-31 12:39:54 -07:00
Mingliang Liu 2848a27700 HDFS-11047. Remove deep copies of FinalizedReplica to alleviate heap consumption on DataNode. Contributed by Xiaobing Zhou 2016-10-28 11:11:31 -07:00
Mingliang Liu d0da47ad9f HDFS-11048. Audit Log should escape control characters. Contributed by Eric Badger
(cherry picked from commit 8a9388e5f6)
2016-10-28 10:57:20 -07:00
Erik Krogen ce8ace372c HDFS-9500. Fix software version counts for DataNodes during rolling upgrade. Contributed by Erik Krogen.
(cherry picked from commit f3ac1f41b8)
2016-10-27 16:00:39 -07:00
Wei-Chiu Chuang 2cf3138d72 HDFS-11055. Update default-log4j.properties for httpfs to imporve test logging. Contributed by Wei-Chiu Chuang.
(cherry picked from commit 31ff42b510)
2016-10-27 13:38:02 -07:00
Kihwal Lee 5057abc390 HDFS-11069. Tighten the authorization of datanode RPC. Contributed by Kihwal Lee
(cherry picked from commit ae48c496dc)
2016-10-27 14:18:10 -05:00
Kihwal Lee 6c11a11918 Revert "HDFS-10455. Logging the username when deny the setOwner operation. Contributed by Tianyin Xiu"
This reverts commit f4300ea12a.
2016-10-27 14:10:07 -05:00