HDFS-9182. Cleanup the findbugs and other issues after HDFS EC merged to trunk. Contributed by Uma Maheswara Rao G.
This commit is contained in:
Uma Mahesh
parent
3793cbe4c3
commit
8b7339312c
|
|
@ -1,74 +0,0 @@
|
|||
BREAKDOWN OF HADOOP-11264 SUBTASKS AND RELATED JIRAS (Common part of HDFS-7285)
|
||||
|
||||
HADOOP-11514. Raw Erasure Coder API for concrete encoding and decoding
|
||||
(Kai Zheng via umamahesh)
|
||||
|
||||
HADOOP-11534. Minor improvements for raw erasure coders
|
||||
( Kai Zheng via vinayakumarb )
|
||||
|
||||
HADOOP-11541. Raw XOR coder
|
||||
( Kai Zheng )
|
||||
|
||||
HADOOP-11542. Raw Reed-Solomon coder in pure Java. Contributed by Kai Zheng
|
||||
( Kai Zheng )
|
||||
|
||||
HADOOP-11643. Define EC schema API for ErasureCodec. Contributed by Kai Zheng
|
||||
( Kai Zheng )
|
||||
|
||||
HADOOP-11646. Erasure Coder API for encoding and decoding of block group
|
||||
( Kai Zheng via vinayakumarb )
|
||||
|
||||
HADOOP-11705. Make erasure coder configurable. Contributed by Kai Zheng
|
||||
( Kai Zheng )
|
||||
|
||||
HADOOP-11706. Refine a little bit erasure coder API. Contributed by Kai Zheng
|
||||
( Kai Zheng )
|
||||
|
||||
HADOOP-11707. Add factory to create raw erasure coder. Contributed by Kai Zheng
|
||||
( Kai Zheng )
|
||||
|
||||
HADOOP-11647. Reed-Solomon ErasureCoder. Contributed by Kai Zheng
|
||||
( Kai Zheng )
|
||||
|
||||
HADOOP-11782 Correct two thrown messages in ECSchema class. Contributed by Xinwei Qin
|
||||
( Xinwei Qin via Kai Zheng )
|
||||
|
||||
HADOOP-11740. Combine erasure encoder and decoder interfaces (Zhe Zhang)
|
||||
|
||||
HADOOP-11805 Better to rename some raw erasure coders. Contributed by Kai Zheng
|
||||
( Kai Zheng )
|
||||
|
||||
HADOOP-11645. Erasure Codec API covering the essential aspects for an erasure code
|
||||
( Kai Zheng via vinayakumarb )
|
||||
|
||||
HADOOP-11818. Minor improvements for erasurecode classes. (Rakesh R via Kai Zheng)
|
||||
|
||||
HADOOP-11841. Remove unused ecschema-def.xml files. (szetszwo)
|
||||
|
||||
HADOOP-11921. Enhance tests for erasure coders. (Kai Zheng via Zhe Zhang)
|
||||
|
||||
HADOOP-11920. Refactor some codes for erasure coders. (Kai Zheng via Zhe Zhang)
|
||||
|
||||
HADOOP-11566. Add tests and fix for erasure coders to recover erased parity
|
||||
units. (Kai Zheng via Zhe Zhang)
|
||||
|
||||
HADOOP-11938. Enhance ByteBuffer version encode/decode API of raw erasure
|
||||
coder. (Kai Zheng via Zhe Zhang)
|
||||
|
||||
HADOOP-12013. Generate fixed data to perform erasure coder test. (Kai Zheng)
|
||||
|
||||
HADOOP-12029. Remove chunkSize from ECSchema as its not required for coders
|
||||
(vinayakumarb)
|
||||
|
||||
HADOOP-11847. Enhance raw coder allowing to read least required inputs in decoding.
|
||||
(Kai Zheng)
|
||||
|
||||
HADOOP-12011. Allow to dump verbose information to ease debugging in raw erasure coders
|
||||
(Kai Zheng)
|
||||
|
||||
HADOOP-12065. Using more meaningful keys in EC schema. (Kai Zheng)
|
||||
|
||||
HDFS-8557. Allow to configure RS and XOR raw coders (Kai Zheng)
|
||||
|
||||
HADOOP-12060. Fix ByteBuffer usage for raw erasure coders. (Kai Zheng via
|
||||
jing9)
|
||||
|
|
@ -513,6 +513,81 @@ Trunk (Unreleased)
|
|||
|
||||
HADOOP-8589. ViewFs tests fail when tests and home dirs are nested (sanjay Radia)
|
||||
|
||||
BREAKDOWN OF HADOOP-11264 SUBTASKS AND RELATED JIRAS (Common part of HDFS-7285)
|
||||
|
||||
HADOOP-11514. Raw Erasure Coder API for concrete encoding and decoding
|
||||
(Kai Zheng via umamahesh)
|
||||
|
||||
HADOOP-11534. Minor improvements for raw erasure coders
|
||||
( Kai Zheng via vinayakumarb )
|
||||
|
||||
HADOOP-11541. Raw XOR coder
|
||||
( Kai Zheng )
|
||||
|
||||
HADOOP-11542. Raw Reed-Solomon coder in pure Java. Contributed by Kai Zheng
|
||||
( Kai Zheng )
|
||||
|
||||
HADOOP-11643. Define EC schema API for ErasureCodec. Contributed by Kai Zheng
|
||||
( Kai Zheng )
|
||||
|
||||
HADOOP-11646. Erasure Coder API for encoding and decoding of block group
|
||||
( Kai Zheng via vinayakumarb )
|
||||
|
||||
HADOOP-11705. Make erasure coder configurable. Contributed by Kai Zheng
|
||||
( Kai Zheng )
|
||||
|
||||
HADOOP-11706. Refine a little bit erasure coder API. Contributed by Kai Zheng
|
||||
( Kai Zheng )
|
||||
|
||||
HADOOP-11707. Add factory to create raw erasure coder. Contributed by Kai Zheng
|
||||
( Kai Zheng )
|
||||
|
||||
HADOOP-11647. Reed-Solomon ErasureCoder. Contributed by Kai Zheng
|
||||
( Kai Zheng )
|
||||
|
||||
HADOOP-11782 Correct two thrown messages in ECSchema class. Contributed by Xinwei Qin
|
||||
( Xinwei Qin via Kai Zheng )
|
||||
|
||||
HADOOP-11740. Combine erasure encoder and decoder interfaces (Zhe Zhang)
|
||||
|
||||
HADOOP-11805 Better to rename some raw erasure coders. Contributed by Kai Zheng
|
||||
( Kai Zheng )
|
||||
|
||||
HADOOP-11645. Erasure Codec API covering the essential aspects for an erasure code
|
||||
( Kai Zheng via vinayakumarb )
|
||||
|
||||
HADOOP-11818. Minor improvements for erasurecode classes. (Rakesh R via Kai Zheng)
|
||||
|
||||
HADOOP-11841. Remove unused ecschema-def.xml files. (szetszwo)
|
||||
|
||||
HADOOP-11921. Enhance tests for erasure coders. (Kai Zheng via Zhe Zhang)
|
||||
|
||||
HADOOP-11920. Refactor some codes for erasure coders. (Kai Zheng via Zhe Zhang)
|
||||
|
||||
HADOOP-11566. Add tests and fix for erasure coders to recover erased parity
|
||||
units. (Kai Zheng via Zhe Zhang)
|
||||
|
||||
HADOOP-11938. Enhance ByteBuffer version encode/decode API of raw erasure
|
||||
coder. (Kai Zheng via Zhe Zhang)
|
||||
|
||||
HADOOP-12013. Generate fixed data to perform erasure coder test. (Kai Zheng)
|
||||
|
||||
HADOOP-12029. Remove chunkSize from ECSchema as its not required for coders
|
||||
(vinayakumarb)
|
||||
|
||||
HADOOP-11847. Enhance raw coder allowing to read least required inputs in decoding.
|
||||
(Kai Zheng)
|
||||
|
||||
HADOOP-12011. Allow to dump verbose information to ease debugging in raw erasure coders
|
||||
(Kai Zheng)
|
||||
|
||||
HADOOP-12065. Using more meaningful keys in EC schema. (Kai Zheng)
|
||||
|
||||
HADOOP-12060. Fix ByteBuffer usage for raw erasure coders. (Kai Zheng via
|
||||
jing9)
|
||||
|
||||
HADOOP-11921. Enhance tests for erasure coders. (Kai Zheng)
|
||||
|
||||
Release 2.8.0 - UNRELEASED
|
||||
|
||||
INCOMPATIBLE CHANGES
|
||||
|
|
|
|||
|
|
@ -15,6 +15,9 @@
|
|||
<Class name="org.apache.hadoop.hdfs.protocol.SnapshottableDirectoryStatus"/>
|
||||
<Class name="org.apache.hadoop.hdfs.server.protocol.DatanodeStorageReport"/>
|
||||
<Class name="org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslResponseWithNegotiatedCipherOption"/>
|
||||
<Class name="org.apache.hadoop.hdfs.DFSPacket”/>
|
||||
<Class name="org.apache.hadoop.hdfs.protocol.LocatedStripedBlock”/>
|
||||
<Class name=“org.apache.hadoop.hdfs.util.StripedBlockUtil$ChunkByteArray”/>
|
||||
</Or>
|
||||
<Bug pattern="EI_EXPOSE_REP,EI_EXPOSE_REP2" />
|
||||
</Match>
|
||||
|
|
|
|||
|
|
@ -424,7 +424,7 @@ public class DFSOutputStream extends FSOutputSummer
|
|||
currentPacket = null;
|
||||
}
|
||||
|
||||
void enqueueCurrentPacketFull() throws IOException {
|
||||
synchronized void enqueueCurrentPacketFull() throws IOException {
|
||||
LOG.debug("enqueue full {}, src={}, bytesCurBlock={}, blockSize={},"
|
||||
+ " appendChunk={}, {}", currentPacket, src, getStreamer()
|
||||
.getBytesCurBlock(), blockSize, getStreamer().getAppendChunk(),
|
||||
|
|
|
|||
|
|
@ -531,7 +531,7 @@ public class DFSStripedOutputStream extends DFSOutputStream {
|
|||
}
|
||||
|
||||
@Override
|
||||
void enqueueCurrentPacketFull() throws IOException {
|
||||
synchronized void enqueueCurrentPacketFull() throws IOException {
|
||||
LOG.debug("enqueue full {}, src={}, bytesCurBlock={}, blockSize={},"
|
||||
+ " appendChunk={}, {}", currentPacket, src, getStreamer()
|
||||
.getBytesCurBlock(), blockSize, getStreamer().getAppendChunk(),
|
||||
|
|
|
|||
|
|
@ -1,460 +0,0 @@
|
|||
BREAKDOWN OF HDFS-7285 SUBTASKS AND RELATED JIRAS
|
||||
|
||||
HDFS-7347. Configurable erasure coding policy for individual files and
|
||||
directories ( Zhe Zhang via vinayakumarb )
|
||||
|
||||
HDFS-7339. Representing striped block groups in NameNode with hierarchical
|
||||
naming protocol ( Zhe Zhang )
|
||||
|
||||
HDFS-7652. Process block reports for erasure coded blocks (Zhe Zhang)
|
||||
|
||||
HDFS-7716. Erasure Coding: extend BlockInfo to handle EC info (Jing Zhao)
|
||||
|
||||
HDFS-7749. Erasure Coding: Add striped block support in INodeFile (Jing Zhao)
|
||||
|
||||
HDFS-7837. Erasure Coding: allocate and persist striped blocks in NameNode
|
||||
(Jing Zhao via Zhe Zhang)
|
||||
|
||||
HDFS-7872. Erasure Coding: INodeFile.dumpTreeRecursively() supports to print
|
||||
striped blocks (Takuya Fukudome via jing9)
|
||||
|
||||
HDFS-7853. Erasure coding: extend LocatedBlocks to support reading from
|
||||
striped files (Jing Zhao)
|
||||
|
||||
HDFS-7826. Erasure Coding: Update INodeFile quota computation for striped
|
||||
blocks ( Kai Sasaki via jing9 )
|
||||
|
||||
HDFS-7912. Erasure Coding: track BlockInfo instead of Block in
|
||||
UnderReplicatedBlocks and PendingReplicationBlocks (Jing Zhao)
|
||||
|
||||
HDFS-7369. Erasure coding: distribute recovery work for striped blocks to
|
||||
DataNode (Zhe Zhang)
|
||||
|
||||
HDFS-7864. Erasure Coding: Update safemode calculation for striped blocks
|
||||
(GAO Rui via jing9)
|
||||
|
||||
HDFS-7827. Erasure Coding: support striped blocks in non-protobuf fsimage
|
||||
( Hui Zheng via jing9 )
|
||||
|
||||
HDFS-7616. Add a test for BlockGroup support in FSImage.
|
||||
(Takuya Fukudome via szetszwo)
|
||||
|
||||
HDFS-7907. Erasure Coding: track invalid, corrupt, and under-recovery striped
|
||||
blocks in NameNode (Jing Zhao)
|
||||
|
||||
HDFS-8005. Erasure Coding: simplify striped block recovery work computation
|
||||
and add tests (Jing Zhao)
|
||||
|
||||
HDFS-7617. Add unit tests for editlog transactions for EC
|
||||
(Hui Zheng via Zhe Zhang)
|
||||
|
||||
HDFS-7839. Erasure coding: implement facilities in NameNode to create and
|
||||
manage EC zones (Zhe Zhang)
|
||||
|
||||
HDFS-7969. Erasure coding: NameNode support for lease recovery of striped
|
||||
block groups. (Zhe Zhang)
|
||||
|
||||
HDFS-7782. Erasure coding: pread from files in striped layout.
|
||||
(Zhe Zhang and Jing Zhao via Zhe Zhang)
|
||||
|
||||
HDFS-8023. Erasure Coding: retrieve eraure coding schema for a file from
|
||||
NameNode (vinayakumarb)
|
||||
|
||||
HDFS-8074. Define a system-wide default EC schema. (Kai Zheng)
|
||||
|
||||
HDFS-8077. Erasure coding: fix bugs in EC zone and symlinks.
|
||||
(Jing Zhao and Zhe Zhang via Jing Zhao)
|
||||
|
||||
HDFS-8104. Make hard-coded values consistent with the system default schema first before remove them. (Kai Zheng)
|
||||
|
||||
HDFS-7889. Subclass DFSOutputStream to support writing striping layout files. (Li Bo via Kai Zheng)
|
||||
|
||||
HDFS-8090. Erasure Coding: Add RPC to client-namenode to list all
|
||||
ECSchemas loaded in Namenode. (vinayakumarb)
|
||||
|
||||
HDFS-8122. Erasure Coding: Support specifying ECSchema during creation of ECZone.
|
||||
(Vinayakumar B via Zhe Zhang)
|
||||
|
||||
HDFS-8114. Erasure coding: Add auditlog FSNamesystem#createErasureCodingZone if this
|
||||
operation fails. (Rakesh R via Zhe Zhang)
|
||||
|
||||
HDFS-8123. Erasure Coding: Better to move EC related proto messages to a
|
||||
separate erasurecoding proto file (Rakesh R via vinayakumarb)
|
||||
|
||||
HDFS-7349. Support DFS command for the EC encoding (vinayakumarb)
|
||||
|
||||
HDFS-8120. Erasure coding: created util class to analyze striped block groups.
|
||||
(Contributed by Zhe Zhang and Li Bo via Jing Zhao)
|
||||
|
||||
HDFS-7994. Detect if resevered EC Block ID is already used during namenode
|
||||
startup. (Hui Zheng via szetszwo)
|
||||
|
||||
HDFS-8167. BlockManager.addBlockCollectionWithCheck should check if the block is a striped block. (Hui Zheng via zhz).
|
||||
|
||||
HDFS-8166. DFSStripedOutputStream should not create empty blocks. (Jing Zhao)
|
||||
|
||||
HDFS-7937. Erasure Coding: INodeFile quota computation unit tests.
|
||||
(Kai Sasaki via Jing Zhao)
|
||||
|
||||
HDFS-8145. Fix the editlog corruption exposed by failed TestAddStripedBlocks.
|
||||
(Jing Zhao)
|
||||
|
||||
HDFS-8146. Protobuf changes for BlockECRecoveryCommand and its fields for
|
||||
making it ready for transfer to DN (Uma Maheswara Rao G via vinayakumarb)
|
||||
|
||||
HDFS-8181. createErasureCodingZone sets retryCache state as false always
|
||||
(Uma Maheswara Rao G via vinayakumarb)
|
||||
|
||||
HDFS-8190. StripedBlockUtil.getInternalBlockLength may have overflow error.
|
||||
(szetszwo)
|
||||
|
||||
HDFS-8216. TestDFSStripedOutputStream should use BlockReaderTestUtil to
|
||||
create BlockReader. (szetszwo via Zhe Zhang)
|
||||
|
||||
HDFS-8212. DistributedFileSystem.createErasureCodingZone should pass schema
|
||||
in FileSystemLinkResolver. (szetszwo via Zhe Zhang)
|
||||
|
||||
HDFS-8024. Erasure Coding: ECworker frame, basics, bootstraping and configuration.
|
||||
(umamahesh)
|
||||
|
||||
HDFS-8156. Add/implement necessary APIs even we just have the system default
|
||||
schema. (Kai Zheng via Zhe Zhang)
|
||||
|
||||
HDFS-8136. Client gets and uses EC schema when reads and writes a stripping
|
||||
file. (Kai Sasaki via Kai Zheng)
|
||||
|
||||
HDFS-8233. Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when the last
|
||||
stripe is at the block group boundary. (jing9)
|
||||
|
||||
HDFS-8223. Should calculate checksum for parity blocks in DFSStripedOutputStream.
|
||||
(Yi Liu via jing9)
|
||||
|
||||
HDFS-8228. Erasure Coding: SequentialBlockGroupIdGenerator#nextValue may cause
|
||||
block id conflicts (Jing Zhao via Zhe Zhang)
|
||||
|
||||
HDFS-8033. Erasure coding: stateful (non-positional) read from files in
|
||||
striped layout (Zhe Zhang)
|
||||
|
||||
HDFS-8230. Erasure Coding: Ignore DatanodeProtocol#DNA_ERASURE_CODING_RECOVERY
|
||||
commands from standbynode if any (vinayakumarb)
|
||||
|
||||
HDFS-8189. ClientProtocol#createErasureCodingZone API was wrongly annotated
|
||||
as Idempotent (vinayakumarb)
|
||||
|
||||
HDFS-8235. Erasure Coding: Create DFSStripedInputStream in DFSClient#open.
|
||||
(Kai Sasaki via jing9)
|
||||
|
||||
HDFS-8272. Erasure Coding: simplify the retry logic in DFSStripedInputStream
|
||||
(stateful read). (Jing Zhao via Zhe Zhang)
|
||||
|
||||
HDFS-8282. Erasure coding: move striped reading logic to StripedBlockUtil.
|
||||
(Zhe Zhang)
|
||||
|
||||
HDFS-8183. Erasure Coding: Improve DFSStripedOutputStream closing of
|
||||
datastreamer threads. (Rakesh R via Zhe Zhang)
|
||||
|
||||
HDFS-8308. Erasure Coding: NameNode may get blocked in waitForLoadingFSImage()
|
||||
when loading editlog. (jing9)
|
||||
|
||||
HDFS-7949. WebImageViewer need support file size calculation with striped
|
||||
blocks. (Rakesh R via Zhe Zhang)
|
||||
|
||||
HDFS-8316. Erasure coding: refactor EC constants to be consistent with HDFS-8249.
|
||||
(Zhe Zhang via jing9)
|
||||
|
||||
HDFS-8281. Erasure Coding: implement parallel stateful reading for striped layout.
|
||||
(jing9)
|
||||
|
||||
HDFS-8137. Send the EC schema to DataNode via EC encoding/recovering command(umamahesh)
|
||||
|
||||
HDFS-8242. Erasure Coding: XML based end-to-end test for ECCli commands
|
||||
(Rakesh R via vinayakumarb)
|
||||
|
||||
HDFS-8324. Add trace info to DFSClient#getErasureCodingZoneInfo(..) (vinayakumarb via
|
||||
umamahesh)
|
||||
|
||||
HDFS-7672. Handle write failure for stripping blocks and refactor the
|
||||
existing code in DFSStripedOutputStream and StripedDataStreamer. (szetszwo)
|
||||
|
||||
HDFS-7348. Erasure Coding: DataNode reconstruct striped blocks.
|
||||
(Yi Liu via Zhe Zhang)
|
||||
|
||||
HADOOP-11921. Enhance tests for erasure coders. (Kai Zheng)
|
||||
|
||||
HDFS-8334. Erasure coding: rename DFSStripedInputStream related test
|
||||
classes. (Zhe Zhang)
|
||||
|
||||
HDFS-8129. Erasure Coding: Maintain consistent naming for Erasure Coding related classes - EC/ErasureCoding
|
||||
(umamahesh)
|
||||
|
||||
HDFS-8203. Erasure Coding: Seek and other Ops in DFSStripedInputStream.
|
||||
(Yi Liu via jing9)
|
||||
|
||||
HDFS-8289. Erasure Coding: add ECSchema to HdfsFileStatus. (Yong Zhang via
|
||||
jing9)
|
||||
|
||||
HDFS-8355. Erasure Coding: Refactor BlockInfo and BlockInfoUnderConstruction.
|
||||
(Tsz Wo Nicholas Sze via jing9)
|
||||
|
||||
HDFS-7678. Erasure coding: DFSInputStream with decode functionality (pread).
|
||||
(Zhe Zhang)
|
||||
|
||||
HDFS-8372. Erasure coding: compute storage type quotas for striped files,
|
||||
to be consistent with HDFS-8327. (Zhe Zhang via jing9)
|
||||
|
||||
HDFS-8368. Erasure Coding: DFS opening a non-existent file need to be
|
||||
handled properly (Rakesh R via zhz)
|
||||
|
||||
HDFS-8363. Erasure Coding: DFSStripedInputStream#seekToNewSource. (yliu)
|
||||
|
||||
HDFS-8195. Erasure coding: Fix file quota change when we complete/commit
|
||||
the striped blocks. (Takuya Fukudome via zhz)
|
||||
|
||||
HDFS-8364. Erasure coding: fix some minor bugs in EC CLI
|
||||
(Walter Su via vinayakumarb)
|
||||
|
||||
HDFS-8391. NN should consider current EC tasks handling count from DN while
|
||||
assigning new tasks. (umamahesh)
|
||||
|
||||
HDFS-8367. BlockInfoStriped uses EC schema. (Kai Sasaki via Kai Zheng)
|
||||
|
||||
HDFS-8352. Erasure Coding: test webhdfs read write stripe file. (waltersu4549)
|
||||
|
||||
HDFS-8417. Erasure Coding: Pread failed to read data starting from not-first stripe.
|
||||
(Walter Su via jing9)
|
||||
|
||||
HDFS-8418. Fix the isNeededReplication calculation for Striped block in NN.
|
||||
(Yi Liu via jing9)
|
||||
|
||||
HDFS-8320. Erasure coding: consolidate striping-related terminologies. (zhz)
|
||||
|
||||
HDFS-8366. Erasure Coding: Make the timeout parameter of polling blocking queue
|
||||
configurable in DFSStripedOutputStream. (Li Bo)
|
||||
|
||||
HDFS-8378. Erasure Coding: Few improvements for the erasure coding worker.
|
||||
(Rakesh R via waltersu4549)
|
||||
|
||||
HDFS-8375. Add cellSize as an XAttr to ECZone. ( Vinayakumar B via zhz).
|
||||
|
||||
HDFS-8428. Erasure Coding: Fix the NullPointerException when deleting file.
|
||||
(Yi Liu via zhz).
|
||||
|
||||
HDFS-8323. Bump GenerationStamp for write faliure in DFSStripedOutputStream.
|
||||
(Tsz Wo Nicholas Sze via jing9)
|
||||
|
||||
HDFS-8427. Remove dataBlockNum and parityBlockNum from BlockInfoStriped.
|
||||
(Kai Sasaki via jing9)
|
||||
|
||||
HDFS-8186. Erasure coding: Make block placement policy for EC file configurable.
|
||||
(Walter Su via zhz)
|
||||
|
||||
HDFS-8294. Erasure Coding: Fix Findbug warnings present in erasure coding.
|
||||
(Rakesh R via zhz)
|
||||
|
||||
HDFS-8441. Erasure Coding: make condition check earlier for setReplication.
|
||||
(waltersu4549)
|
||||
|
||||
HDFS-7768. Change fsck to support EC files. (Takanobu Asanuma via szetszwo)
|
||||
|
||||
HDFS-8382. Remove chunkSize and initialize from erasure coder. (Kai Zheng)
|
||||
|
||||
HDFS-8408. Revisit and refactor ErasureCodingInfo (vinayakumarb)
|
||||
|
||||
HDFS-8479. Erasure coding: fix striping related logic in FSDirWriteFileOp to
|
||||
sync with HDFS-8421. (Zhe Zhang via jing9)
|
||||
|
||||
HDFS-8481. Erasure coding: remove workarounds in client side stripped blocks
|
||||
recovering. (zhz)
|
||||
|
||||
HDFS-8336. Expose some administrative erasure coding operations to HdfsAdmin
|
||||
(Uma Maheswara Rao G via vinayakumarb)
|
||||
|
||||
HDFS-8444. Erasure Coding: fix cannot rename a zone dir
|
||||
(Walter Su via vinayakumarb)
|
||||
|
||||
HDFS-8517. Fix a decoding issue in stripped block recovering in client side.
|
||||
(Kai Zheng via jing9)
|
||||
|
||||
HDFS-8453. Erasure coding: properly handle start offset for internal blocks
|
||||
in a block group. (Zhe Zhang via jing9)
|
||||
|
||||
HDFS-7621. Erasure Coding: update the Balancer/Mover data migration logic.
|
||||
(Walter Su via zhz)
|
||||
|
||||
HDFS-8328. Follow-on to update decode for DataNode striped blocks
|
||||
reconstruction. (yliu)
|
||||
|
||||
HDFS-8319. Erasure Coding: support decoding for stateful read.
|
||||
(Jing Zhao via zhz)
|
||||
|
||||
HDFS-8460. Erasure Coding: stateful read result doesn't match data
|
||||
occasionally because of flawed test. (Walter Su via zhz)
|
||||
|
||||
HDFS-8556. Erasure Coding: Fix usage of 'createZone' (vinayakumarb)
|
||||
|
||||
HDFS-8571. Fix TestErasureCodingCli test (Vinayakumar B via waltersu4549)
|
||||
|
||||
HDFS-8450. Erasure Coding: Consolidate erasure coding zone related
|
||||
implementation into a single class (Rakesh R via vinayakumarb)
|
||||
|
||||
HDFS-8585. Erasure Coding: Remove dataBlockNum and parityBlockNum from
|
||||
StripedBlockProto. (Yi Liu via jing9)
|
||||
|
||||
HDFS-8559. Erasure Coding: fix non-protobuf fsimage for striped blocks.
|
||||
(Jing Zhao via yliu)
|
||||
|
||||
HDFS-8580. Erasure coding: Persist cellSize in BlockInfoStriped and
|
||||
StripedBlocksFeature. (Walter Su via jing9)
|
||||
|
||||
HDFS-8466. Refactor BlockInfoContiguous and fix NPE in
|
||||
TestBlockInfo#testCopyConstructor() (vinayakumarb)
|
||||
|
||||
HDFS-8254. Avoid assigning a leading streamer in StripedDataStreamer to
|
||||
tolerate datanode failure. (Tsz Wo Nicholas Sze via jing9)
|
||||
|
||||
HDFS-8543. Erasure Coding: processOverReplicatedBlock() handles striped block.
|
||||
(Walter Su via jing9)
|
||||
|
||||
HDFS-8602. Erasure Coding: Client can't read(decode) the EC files which have
|
||||
corrupt blocks. (jing9 and Kai Sasaki)
|
||||
|
||||
HDFS-8567. Erasure Coding: SafeMode handles file smaller than a full stripe.
|
||||
(Walter Su via jing9)
|
||||
|
||||
HDFS-8253. DFSStripedOutputStream.closeThreads releases cellBuffers
|
||||
multiple times. (Kai Sasaki via szetszwo)
|
||||
|
||||
HDFS-8468. 2 RPC calls for every file read in DFSClient#open(..) resulting in
|
||||
double Audit log entries (vinayakumarb)
|
||||
|
||||
HDFS-8684. Erasure Coding: fix some block number calculation for striped
|
||||
block. (yliu)
|
||||
|
||||
HDFS-8461. Erasure coding: fix priority level of UnderReplicatedBlocks for
|
||||
striped block. (Walter Su via jing9)
|
||||
|
||||
HDFS-8719. Erasure Coding: client generates too many small packets when
|
||||
writing parity data. (Li Bo via waltersu4549)
|
||||
|
||||
HDFS-8563. Erasure Coding: fsck handles file smaller than a full stripe.
|
||||
(Walter Su via jing9)
|
||||
|
||||
HDFS-8484. Erasure coding: Two contiguous blocks occupy IDs belong to same
|
||||
striped group. (Walter Su via jing9)
|
||||
|
||||
HDFS-8744. Erasure Coding: the number of chunks in packet is not updated
|
||||
when writing parity data. (Li Bo)
|
||||
|
||||
HDFS-8669. Erasure Coding: handle missing internal block locations in
|
||||
DFSStripedInputStream. (jing9)
|
||||
|
||||
HDFS-8702. Erasure coding: update BlockManager.blockHasEnoughRacks(..) logic
|
||||
for striped block. (Kai Sasaki via jing9)
|
||||
|
||||
HDFS-8734. Erasure Coding: fix one cell need two packets. (Walter Su via
|
||||
jing9)
|
||||
|
||||
HDFS-8619. Erasure Coding: revisit replica counting for striped blocks.
|
||||
(Jing Zhao via yliu)
|
||||
|
||||
HDFS-8058. Erasure coding: use BlockInfo[] for both striped and contiguous
|
||||
blocks in INodeFile. (Zhe Zhang and Yi Liu via zhz)
|
||||
|
||||
HDFS-8787. Erasure coding: rename BlockInfoContiguousUC and BlockInfoStripedUC
|
||||
to be consistent with trunk. (zhz)
|
||||
|
||||
HDFS-8433. Erasure coding: set blockToken in LocatedStripedBlock.(waltersu4549)
|
||||
|
||||
HDFS-8760. Erasure Coding: reuse BlockReader when reading the same block in pread.
|
||||
(jing9)
|
||||
|
||||
HDFS-8781. Erasure Coding: Correctly handle BlockManager#InvalidateBlocks for
|
||||
striped block. (Yi Liu via jing9)
|
||||
|
||||
HDFS-8813. Erasure Coding: Client no need to decode missing parity blocks.
|
||||
(Walter Su via jing9)
|
||||
|
||||
HDFS-8798. Erasure Coding: fix DFSStripedInputStream/DFSStripedOutputStream
|
||||
re-fetch token when expired. (Walter Su via jing9)
|
||||
|
||||
HDFS-8769. Erasure coding: unit test for SequentialBlockGroupIdGenerator.
|
||||
(Rakesh R via waltersu4549)
|
||||
|
||||
HDFS-8202. Improve end to end stirpping file test to add erasure recovering
|
||||
test. (Xinwei Qin via zhz)
|
||||
|
||||
HDFS-8804. Erasure Coding: use DirectBufferPool in DFSStripedInputStream for
|
||||
buffer allocation. (jing9)
|
||||
|
||||
HDFS-8399. Erasure Coding: unit test the behaviour of BlockManager recovery
|
||||
work for the deleted blocks. (Rakesh R via zhz)
|
||||
|
||||
HDFS-8857. Erasure Coding: Fix ArrayIndexOutOfBoundsException in
|
||||
TestWriteStripedFileWithFailure. (Li Bo)
|
||||
|
||||
HDFS-8827. Erasure Coding: Fix NPE when NameNode processes over-replicated
|
||||
striped blocks. (Walter Su and Takuya Fukudome via jing9)
|
||||
|
||||
HDFS-8854. Erasure coding: add ECPolicy to replace schema+cellSize in
|
||||
hadoop-hdfs. (Walter Su via zhz)
|
||||
|
||||
HDFS-8220. Erasure Coding: StripedDataStreamer fails to handle the
|
||||
blocklocations which doesn't satisfy BlockGroupSize. (Rakesh R via zhz)
|
||||
|
||||
HDFS-8838. Erasure Coding: Tolerate datanode failures in DFSStripedOutputStream
|
||||
when the data length is small. (szetszwo via waltersu4549)
|
||||
|
||||
HDFS-8909. Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC
|
||||
to use BlockUnderConstructionFeature. (Jing Zhao via waltersu4549)
|
||||
|
||||
HDFS-8937. Erasure coding: do not throw exception when setting replication on
|
||||
EC file. (Gao Rui via jing9)
|
||||
|
||||
HDFS-8978. Erasure coding: fix 2 failed tests of DFSStripedOutputStream.
|
||||
(Walter Su via jing9)
|
||||
|
||||
HDFS-8833. Erasure coding: store EC schema and cell size in INodeFile and
|
||||
eliminate notion of EC zones. (zhz)
|
||||
|
||||
HDFS-8853. Erasure Coding: Provide ECSchema validation when setting EC
|
||||
policy. (andreina via zhz)
|
||||
|
||||
HDFS-8975. Erasure coding : Fix random failure in TestSafeModeWithStripedFile
|
||||
(J.Andreina via vinayakumarb)
|
||||
|
||||
HDFS-7351. Document the HDFS Erasure Coding feature.
|
||||
(umamahesh and Zhe Zhang via wang)
|
||||
|
||||
HDFS-8899. Erasure Coding: use threadpool for EC recovery tasks on DataNode.
|
||||
(Rakesh R via zhz)
|
||||
|
||||
HDFS-9088. Cleanup erasure coding documentation. (wang via zhz)
|
||||
|
||||
HDFS-9097. Erasure coding: update EC command "-s" flag to "-p" when
|
||||
specifying policy. (zhz)
|
||||
|
||||
HDFS-9086. Rename dfs.datanode.stripedread.threshold.millis to
|
||||
dfs.datanode.stripedread.timeout.millis. (wang via zhz)
|
||||
|
||||
HDFS-8550. Erasure Coding: Fix FindBugs Multithreaded correctness Warning.
|
||||
(Rakesh R via zhz)
|
||||
|
||||
HDFS-9113. ErasureCodingWorker#processErasureCodingTasks should not fail to process
|
||||
remaining tasks due to one invalid ECTask (umamahesh)
|
||||
|
||||
HDFS-9091. Erasure Coding: Provide DistributedFilesystem API to
|
||||
getAllErasureCodingPolicies. (Rakesh R via zhz)
|
||||
|
||||
HDFS-8920. Erasure Coding: when recovering lost blocks, logs can be too
|
||||
verbose and hurt performance. (Rui Li via Kai Zheng)
|
||||
|
||||
HDFS-8882. Erasure Coding: Use datablocks, parityblocks and cell size from
|
||||
ErasureCodingPolicy (Vinayakumar B via zhz)
|
||||
|
||||
HDFS-9040. Erasure coding: coordinate data streamers in
|
||||
DFSStripedOutputStream. (jing9 and Walter Su)
|
||||
|
||||
HDFS-9185. Fix null tracer in ErasureCodingWorker. (Rakesh R via jing9)
|
||||
|
||||
HDFS-9180. Update excluded DataNodes in DFSStripedOutputStream based on failures
|
||||
in data streamers. (jing9)
|
||||
|
|
@ -170,6 +170,9 @@ Trunk (Unreleased)
|
|||
HDFS-5033. Bad error message for fs -put/copyFromLocal if user
|
||||
doesn't have permissions to read the source (Darrell Taylor via aw)
|
||||
|
||||
HDFS-9182. Cleanup the findbugs and other issues after HDFS EC merged to trunk.
|
||||
(umamahesh)
|
||||
|
||||
OPTIMIZATIONS
|
||||
|
||||
BUG FIXES
|
||||
|
|
@ -350,6 +353,467 @@ Trunk (Unreleased)
|
|||
|
||||
HDFS-8800. hdfs --daemon stop namenode corrupts logs (John Smith via aw)
|
||||
|
||||
BREAKDOWN OF HDFS-7285 SUBTASKS AND RELATED JIRAS
|
||||
|
||||
HDFS-7347. Configurable erasure coding policy for individual files and
|
||||
directories ( Zhe Zhang via vinayakumarb )
|
||||
|
||||
HDFS-7339. Representing striped block groups in NameNode with hierarchical
|
||||
naming protocol ( Zhe Zhang )
|
||||
|
||||
HDFS-7652. Process block reports for erasure coded blocks (Zhe Zhang)
|
||||
|
||||
HDFS-7716. Erasure Coding: extend BlockInfo to handle EC info (Jing Zhao)
|
||||
|
||||
HDFS-7749. Erasure Coding: Add striped block support in INodeFile (Jing Zhao)
|
||||
|
||||
HDFS-7837. Erasure Coding: allocate and persist striped blocks in NameNode
|
||||
(Jing Zhao via Zhe Zhang)
|
||||
|
||||
HDFS-7872. Erasure Coding: INodeFile.dumpTreeRecursively() supports to print
|
||||
striped blocks (Takuya Fukudome via jing9)
|
||||
|
||||
HDFS-7853. Erasure coding: extend LocatedBlocks to support reading from
|
||||
striped files (Jing Zhao)
|
||||
|
||||
HDFS-7826. Erasure Coding: Update INodeFile quota computation for striped
|
||||
blocks ( Kai Sasaki via jing9 )
|
||||
|
||||
HDFS-7912. Erasure Coding: track BlockInfo instead of Block in
|
||||
UnderReplicatedBlocks and PendingReplicationBlocks (Jing Zhao)
|
||||
|
||||
HDFS-7369. Erasure coding: distribute recovery work for striped blocks to
|
||||
DataNode (Zhe Zhang)
|
||||
|
||||
HDFS-7864. Erasure Coding: Update safemode calculation for striped blocks
|
||||
(GAO Rui via jing9)
|
||||
|
||||
HDFS-7827. Erasure Coding: support striped blocks in non-protobuf fsimage
|
||||
( Hui Zheng via jing9 )
|
||||
|
||||
HDFS-7616. Add a test for BlockGroup support in FSImage.
|
||||
(Takuya Fukudome via szetszwo)
|
||||
|
||||
HDFS-7907. Erasure Coding: track invalid, corrupt, and under-recovery striped
|
||||
blocks in NameNode (Jing Zhao)
|
||||
|
||||
HDFS-8005. Erasure Coding: simplify striped block recovery work computation
|
||||
and add tests (Jing Zhao)
|
||||
|
||||
HDFS-7617. Add unit tests for editlog transactions for EC
|
||||
(Hui Zheng via Zhe Zhang)
|
||||
|
||||
HDFS-7839. Erasure coding: implement facilities in NameNode to create and
|
||||
manage EC zones (Zhe Zhang)
|
||||
|
||||
HDFS-7969. Erasure coding: NameNode support for lease recovery of striped
|
||||
block groups. (Zhe Zhang)
|
||||
|
||||
HDFS-7782. Erasure coding: pread from files in striped layout.
|
||||
(Zhe Zhang and Jing Zhao via Zhe Zhang)
|
||||
|
||||
HDFS-8023. Erasure Coding: retrieve eraure coding schema for a file from
|
||||
NameNode (vinayakumarb)
|
||||
|
||||
HDFS-8074. Define a system-wide default EC schema. (Kai Zheng)
|
||||
|
||||
HDFS-8077. Erasure coding: fix bugs in EC zone and symlinks.
|
||||
(Jing Zhao and Zhe Zhang via Jing Zhao)
|
||||
|
||||
HDFS-8104. Make hard-coded values consistent with the system default schema first before remove them. (Kai Zheng)
|
||||
|
||||
HDFS-7889. Subclass DFSOutputStream to support writing striping layout files. (Li Bo via Kai Zheng)
|
||||
|
||||
HDFS-8090. Erasure Coding: Add RPC to client-namenode to list all
|
||||
ECSchemas loaded in Namenode. (vinayakumarb)
|
||||
|
||||
HDFS-8122. Erasure Coding: Support specifying ECSchema during creation of ECZone.
|
||||
(Vinayakumar B via Zhe Zhang)
|
||||
|
||||
HDFS-8114. Erasure coding: Add auditlog FSNamesystem#createErasureCodingZone if this
|
||||
operation fails. (Rakesh R via Zhe Zhang)
|
||||
|
||||
HDFS-8123. Erasure Coding: Better to move EC related proto messages to a
|
||||
separate erasurecoding proto file (Rakesh R via vinayakumarb)
|
||||
|
||||
HDFS-7349. Support DFS command for the EC encoding (vinayakumarb)
|
||||
|
||||
HDFS-8120. Erasure coding: created util class to analyze striped block groups.
|
||||
(Contributed by Zhe Zhang and Li Bo via Jing Zhao)
|
||||
|
||||
HDFS-7994. Detect if resevered EC Block ID is already used during namenode
|
||||
startup. (Hui Zheng via szetszwo)
|
||||
|
||||
HDFS-8167. BlockManager.addBlockCollectionWithCheck should check if the block is a striped block. (Hui Zheng via zhz).
|
||||
|
||||
HDFS-8166. DFSStripedOutputStream should not create empty blocks. (Jing Zhao)
|
||||
|
||||
HDFS-7937. Erasure Coding: INodeFile quota computation unit tests.
|
||||
(Kai Sasaki via Jing Zhao)
|
||||
|
||||
HDFS-8145. Fix the editlog corruption exposed by failed TestAddStripedBlocks.
|
||||
(Jing Zhao)
|
||||
|
||||
HDFS-8146. Protobuf changes for BlockECRecoveryCommand and its fields for
|
||||
making it ready for transfer to DN (Uma Maheswara Rao G via vinayakumarb)
|
||||
|
||||
HDFS-8181. createErasureCodingZone sets retryCache state as false always
|
||||
(Uma Maheswara Rao G via vinayakumarb)
|
||||
|
||||
HDFS-8190. StripedBlockUtil.getInternalBlockLength may have overflow error.
|
||||
(szetszwo)
|
||||
|
||||
HDFS-8216. TestDFSStripedOutputStream should use BlockReaderTestUtil to
|
||||
create BlockReader. (szetszwo via Zhe Zhang)
|
||||
|
||||
HDFS-8212. DistributedFileSystem.createErasureCodingZone should pass schema
|
||||
in FileSystemLinkResolver. (szetszwo via Zhe Zhang)
|
||||
|
||||
HDFS-8024. Erasure Coding: ECworker frame, basics, bootstraping and configuration.
|
||||
(umamahesh)
|
||||
|
||||
HDFS-8156. Add/implement necessary APIs even we just have the system default
|
||||
schema. (Kai Zheng via Zhe Zhang)
|
||||
|
||||
HDFS-8136. Client gets and uses EC schema when reads and writes a stripping
|
||||
file. (Kai Sasaki via Kai Zheng)
|
||||
|
||||
HDFS-8233. Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when the last
|
||||
stripe is at the block group boundary. (jing9)
|
||||
|
||||
HDFS-8223. Should calculate checksum for parity blocks in DFSStripedOutputStream.
|
||||
(Yi Liu via jing9)
|
||||
|
||||
HDFS-8228. Erasure Coding: SequentialBlockGroupIdGenerator#nextValue may cause
|
||||
block id conflicts (Jing Zhao via Zhe Zhang)
|
||||
|
||||
HDFS-8033. Erasure coding: stateful (non-positional) read from files in
|
||||
striped layout (Zhe Zhang)
|
||||
|
||||
HDFS-8230. Erasure Coding: Ignore DatanodeProtocol#DNA_ERASURE_CODING_RECOVERY
|
||||
commands from standbynode if any (vinayakumarb)
|
||||
|
||||
HDFS-8189. ClientProtocol#createErasureCodingZone API was wrongly annotated
|
||||
as Idempotent (vinayakumarb)
|
||||
|
||||
HDFS-8235. Erasure Coding: Create DFSStripedInputStream in DFSClient#open.
|
||||
(Kai Sasaki via jing9)
|
||||
|
||||
HDFS-8272. Erasure Coding: simplify the retry logic in DFSStripedInputStream
|
||||
(stateful read). (Jing Zhao via Zhe Zhang)
|
||||
|
||||
HDFS-8282. Erasure coding: move striped reading logic to StripedBlockUtil.
|
||||
(Zhe Zhang)
|
||||
|
||||
HDFS-8183. Erasure Coding: Improve DFSStripedOutputStream closing of
|
||||
datastreamer threads. (Rakesh R via Zhe Zhang)
|
||||
|
||||
HDFS-8308. Erasure Coding: NameNode may get blocked in waitForLoadingFSImage()
|
||||
when loading editlog. (jing9)
|
||||
|
||||
HDFS-7949. WebImageViewer need support file size calculation with striped
|
||||
blocks. (Rakesh R via Zhe Zhang)
|
||||
|
||||
HDFS-8316. Erasure coding: refactor EC constants to be consistent with HDFS-8249.
|
||||
(Zhe Zhang via jing9)
|
||||
|
||||
HDFS-8281. Erasure Coding: implement parallel stateful reading for striped layout.
|
||||
(jing9)
|
||||
|
||||
HDFS-8137. Send the EC schema to DataNode via EC encoding/recovering command(umamahesh)
|
||||
|
||||
HDFS-8242. Erasure Coding: XML based end-to-end test for ECCli commands
|
||||
(Rakesh R via vinayakumarb)
|
||||
|
||||
HDFS-8324. Add trace info to DFSClient#getErasureCodingZoneInfo(..) (vinayakumarb via
|
||||
umamahesh)
|
||||
|
||||
HDFS-7672. Handle write failure for stripping blocks and refactor the
|
||||
existing code in DFSStripedOutputStream and StripedDataStreamer. (szetszwo)
|
||||
|
||||
HDFS-7348. Erasure Coding: DataNode reconstruct striped blocks.
|
||||
(Yi Liu via Zhe Zhang)
|
||||
|
||||
HDFS-8334. Erasure coding: rename DFSStripedInputStream related test
|
||||
classes. (Zhe Zhang)
|
||||
|
||||
HDFS-8129. Erasure Coding: Maintain consistent naming for Erasure Coding related classes - EC/ErasureCoding
|
||||
(umamahesh)
|
||||
|
||||
HDFS-8203. Erasure Coding: Seek and other Ops in DFSStripedInputStream.
|
||||
(Yi Liu via jing9)
|
||||
|
||||
HDFS-8289. Erasure Coding: add ECSchema to HdfsFileStatus. (Yong Zhang via
|
||||
jing9)
|
||||
|
||||
HDFS-8355. Erasure Coding: Refactor BlockInfo and BlockInfoUnderConstruction.
|
||||
(Tsz Wo Nicholas Sze via jing9)
|
||||
|
||||
HDFS-7678. Erasure coding: DFSInputStream with decode functionality (pread).
|
||||
(Zhe Zhang)
|
||||
|
||||
HDFS-8372. Erasure coding: compute storage type quotas for striped files,
|
||||
to be consistent with HDFS-8327. (Zhe Zhang via jing9)
|
||||
|
||||
HDFS-8368. Erasure Coding: DFS opening a non-existent file need to be
|
||||
handled properly (Rakesh R via zhz)
|
||||
|
||||
HDFS-8363. Erasure Coding: DFSStripedInputStream#seekToNewSource. (yliu)
|
||||
|
||||
HDFS-8195. Erasure coding: Fix file quota change when we complete/commit
|
||||
the striped blocks. (Takuya Fukudome via zhz)
|
||||
|
||||
HDFS-8364. Erasure coding: fix some minor bugs in EC CLI
|
||||
(Walter Su via vinayakumarb)
|
||||
|
||||
HDFS-8391. NN should consider current EC tasks handling count from DN while
|
||||
assigning new tasks. (umamahesh)
|
||||
|
||||
HDFS-8367. BlockInfoStriped uses EC schema. (Kai Sasaki via Kai Zheng)
|
||||
|
||||
HDFS-8352. Erasure Coding: test webhdfs read write stripe file. (waltersu4549)
|
||||
|
||||
HDFS-8417. Erasure Coding: Pread failed to read data starting from not-first stripe.
|
||||
(Walter Su via jing9)
|
||||
|
||||
HDFS-8418. Fix the isNeededReplication calculation for Striped block in NN.
|
||||
(Yi Liu via jing9)
|
||||
|
||||
HDFS-8320. Erasure coding: consolidate striping-related terminologies. (zhz)
|
||||
|
||||
HDFS-8366. Erasure Coding: Make the timeout parameter of polling blocking queue
|
||||
configurable in DFSStripedOutputStream. (Li Bo)
|
||||
|
||||
HDFS-8378. Erasure Coding: Few improvements for the erasure coding worker.
|
||||
(Rakesh R via waltersu4549)
|
||||
|
||||
HDFS-8375. Add cellSize as an XAttr to ECZone. ( Vinayakumar B via zhz).
|
||||
|
||||
HDFS-8428. Erasure Coding: Fix the NullPointerException when deleting file.
|
||||
(Yi Liu via zhz).
|
||||
|
||||
HDFS-8323. Bump GenerationStamp for write faliure in DFSStripedOutputStream.
|
||||
(Tsz Wo Nicholas Sze via jing9)
|
||||
|
||||
HDFS-8427. Remove dataBlockNum and parityBlockNum from BlockInfoStriped.
|
||||
(Kai Sasaki via jing9)
|
||||
|
||||
HDFS-8186. Erasure coding: Make block placement policy for EC file configurable.
|
||||
(Walter Su via zhz)
|
||||
|
||||
HDFS-8294. Erasure Coding: Fix Findbug warnings present in erasure coding.
|
||||
(Rakesh R via zhz)
|
||||
|
||||
HDFS-8441. Erasure Coding: make condition check earlier for setReplication.
|
||||
(waltersu4549)
|
||||
|
||||
HDFS-7768. Change fsck to support EC files. (Takanobu Asanuma via szetszwo)
|
||||
|
||||
HDFS-8382. Remove chunkSize and initialize from erasure coder. (Kai Zheng)
|
||||
|
||||
HDFS-8408. Revisit and refactor ErasureCodingInfo (vinayakumarb)
|
||||
|
||||
HDFS-8479. Erasure coding: fix striping related logic in FSDirWriteFileOp to
|
||||
sync with HDFS-8421. (Zhe Zhang via jing9)
|
||||
|
||||
HDFS-8481. Erasure coding: remove workarounds in client side stripped blocks
|
||||
recovering. (zhz)
|
||||
|
||||
HDFS-8336. Expose some administrative erasure coding operations to HdfsAdmin
|
||||
(Uma Maheswara Rao G via vinayakumarb)
|
||||
|
||||
HDFS-8444. Erasure Coding: fix cannot rename a zone dir
|
||||
(Walter Su via vinayakumarb)
|
||||
|
||||
HDFS-8517. Fix a decoding issue in stripped block recovering in client side.
|
||||
(Kai Zheng via jing9)
|
||||
|
||||
HDFS-8453. Erasure coding: properly handle start offset for internal blocks
|
||||
in a block group. (Zhe Zhang via jing9)
|
||||
|
||||
HDFS-7621. Erasure Coding: update the Balancer/Mover data migration logic.
|
||||
(Walter Su via zhz)
|
||||
|
||||
HDFS-8328. Follow-on to update decode for DataNode striped blocks
|
||||
reconstruction. (yliu)
|
||||
|
||||
HDFS-8319. Erasure Coding: support decoding for stateful read.
|
||||
(Jing Zhao via zhz)
|
||||
|
||||
HDFS-8460. Erasure Coding: stateful read result doesn't match data
|
||||
occasionally because of flawed test. (Walter Su via zhz)
|
||||
|
||||
HDFS-8556. Erasure Coding: Fix usage of 'createZone' (vinayakumarb)
|
||||
|
||||
HDFS-8571. Fix TestErasureCodingCli test (Vinayakumar B via waltersu4549)
|
||||
|
||||
HDFS-8450. Erasure Coding: Consolidate erasure coding zone related
|
||||
implementation into a single class (Rakesh R via vinayakumarb)
|
||||
|
||||
HDFS-8585. Erasure Coding: Remove dataBlockNum and parityBlockNum from
|
||||
StripedBlockProto. (Yi Liu via jing9)
|
||||
|
||||
HDFS-8559. Erasure Coding: fix non-protobuf fsimage for striped blocks.
|
||||
(Jing Zhao via yliu)
|
||||
|
||||
HDFS-8580. Erasure coding: Persist cellSize in BlockInfoStriped and
|
||||
StripedBlocksFeature. (Walter Su via jing9)
|
||||
|
||||
HDFS-8466. Refactor BlockInfoContiguous and fix NPE in
|
||||
TestBlockInfo#testCopyConstructor() (vinayakumarb)
|
||||
|
||||
HDFS-8254. Avoid assigning a leading streamer in StripedDataStreamer to
|
||||
tolerate datanode failure. (Tsz Wo Nicholas Sze via jing9)
|
||||
|
||||
HDFS-8543. Erasure Coding: processOverReplicatedBlock() handles striped block.
|
||||
(Walter Su via jing9)
|
||||
|
||||
HDFS-8602. Erasure Coding: Client can't read(decode) the EC files which have
|
||||
corrupt blocks. (jing9 and Kai Sasaki)
|
||||
|
||||
HDFS-8567. Erasure Coding: SafeMode handles file smaller than a full stripe.
|
||||
(Walter Su via jing9)
|
||||
|
||||
HDFS-8253. DFSStripedOutputStream.closeThreads releases cellBuffers
|
||||
multiple times. (Kai Sasaki via szetszwo)
|
||||
|
||||
HDFS-8468. 2 RPC calls for every file read in DFSClient#open(..) resulting in
|
||||
double Audit log entries (vinayakumarb)
|
||||
|
||||
HDFS-8684. Erasure Coding: fix some block number calculation for striped
|
||||
block. (yliu)
|
||||
|
||||
HDFS-8461. Erasure coding: fix priority level of UnderReplicatedBlocks for
|
||||
striped block. (Walter Su via jing9)
|
||||
|
||||
HDFS-8719. Erasure Coding: client generates too many small packets when
|
||||
writing parity data. (Li Bo via waltersu4549)
|
||||
|
||||
HDFS-8563. Erasure Coding: fsck handles file smaller than a full stripe.
|
||||
(Walter Su via jing9)
|
||||
|
||||
HDFS-8484. Erasure coding: Two contiguous blocks occupy IDs belong to same
|
||||
striped group. (Walter Su via jing9)
|
||||
|
||||
HDFS-8744. Erasure Coding: the number of chunks in packet is not updated
|
||||
when writing parity data. (Li Bo)
|
||||
|
||||
HDFS-8669. Erasure Coding: handle missing internal block locations in
|
||||
DFSStripedInputStream. (jing9)
|
||||
|
||||
HDFS-8702. Erasure coding: update BlockManager.blockHasEnoughRacks(..) logic
|
||||
for striped block. (Kai Sasaki via jing9)
|
||||
|
||||
HDFS-8734. Erasure Coding: fix one cell need two packets. (Walter Su via
|
||||
jing9)
|
||||
|
||||
HDFS-8619. Erasure Coding: revisit replica counting for striped blocks.
|
||||
(Jing Zhao via yliu)
|
||||
|
||||
HDFS-8058. Erasure coding: use BlockInfo[] for both striped and contiguous
|
||||
blocks in INodeFile. (Zhe Zhang and Yi Liu via zhz)
|
||||
|
||||
HDFS-8787. Erasure coding: rename BlockInfoContiguousUC and BlockInfoStripedUC
|
||||
to be consistent with trunk. (zhz)
|
||||
|
||||
HDFS-8433. Erasure coding: set blockToken in LocatedStripedBlock.(waltersu4549)
|
||||
|
||||
HDFS-8760. Erasure Coding: reuse BlockReader when reading the same block in pread.
|
||||
(jing9)
|
||||
|
||||
HDFS-8781. Erasure Coding: Correctly handle BlockManager#InvalidateBlocks for
|
||||
striped block. (Yi Liu via jing9)
|
||||
|
||||
HDFS-8813. Erasure Coding: Client no need to decode missing parity blocks.
|
||||
(Walter Su via jing9)
|
||||
|
||||
HDFS-8798. Erasure Coding: fix DFSStripedInputStream/DFSStripedOutputStream
|
||||
re-fetch token when expired. (Walter Su via jing9)
|
||||
|
||||
HDFS-8769. Erasure coding: unit test for SequentialBlockGroupIdGenerator.
|
||||
(Rakesh R via waltersu4549)
|
||||
|
||||
HDFS-8202. Improve end to end stirpping file test to add erasure recovering
|
||||
test. (Xinwei Qin via zhz)
|
||||
|
||||
HDFS-8804. Erasure Coding: use DirectBufferPool in DFSStripedInputStream for
|
||||
buffer allocation. (jing9)
|
||||
|
||||
HDFS-8399. Erasure Coding: unit test the behaviour of BlockManager recovery
|
||||
work for the deleted blocks. (Rakesh R via zhz)
|
||||
|
||||
HDFS-8857. Erasure Coding: Fix ArrayIndexOutOfBoundsException in
|
||||
TestWriteStripedFileWithFailure. (Li Bo)
|
||||
|
||||
HDFS-8827. Erasure Coding: Fix NPE when NameNode processes over-replicated
|
||||
striped blocks. (Walter Su and Takuya Fukudome via jing9)
|
||||
|
||||
HDFS-8854. Erasure coding: add ECPolicy to replace schema+cellSize in
|
||||
hadoop-hdfs. (Walter Su via zhz)
|
||||
|
||||
HDFS-8220. Erasure Coding: StripedDataStreamer fails to handle the
|
||||
blocklocations which doesn't satisfy BlockGroupSize. (Rakesh R via zhz)
|
||||
|
||||
HDFS-8838. Erasure Coding: Tolerate datanode failures in DFSStripedOutputStream
|
||||
when the data length is small. (szetszwo via waltersu4549)
|
||||
|
||||
HDFS-8909. Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC
|
||||
to use BlockUnderConstructionFeature. (Jing Zhao via waltersu4549)
|
||||
|
||||
HDFS-8937. Erasure coding: do not throw exception when setting replication on
|
||||
EC file. (Gao Rui via jing9)
|
||||
|
||||
HDFS-8978. Erasure coding: fix 2 failed tests of DFSStripedOutputStream.
|
||||
(Walter Su via jing9)
|
||||
|
||||
HDFS-8833. Erasure coding: store EC schema and cell size in INodeFile and
|
||||
eliminate notion of EC zones. (zhz)
|
||||
|
||||
HDFS-8853. Erasure Coding: Provide ECSchema validation when setting EC
|
||||
policy. (andreina via zhz)
|
||||
|
||||
HDFS-8975. Erasure coding : Fix random failure in TestSafeModeWithStripedFile
|
||||
(J.Andreina via vinayakumarb)
|
||||
|
||||
HDFS-7351. Document the HDFS Erasure Coding feature.
|
||||
(umamahesh and Zhe Zhang via wang)
|
||||
|
||||
HDFS-8899. Erasure Coding: use threadpool for EC recovery tasks on DataNode.
|
||||
(Rakesh R via zhz)
|
||||
|
||||
HDFS-9088. Cleanup erasure coding documentation. (wang via zhz)
|
||||
|
||||
HDFS-9097. Erasure coding: update EC command "-s" flag to "-p" when
|
||||
specifying policy. (zhz)
|
||||
|
||||
HDFS-9086. Rename dfs.datanode.stripedread.threshold.millis to
|
||||
dfs.datanode.stripedread.timeout.millis. (wang via zhz)
|
||||
|
||||
HDFS-8550. Erasure Coding: Fix FindBugs Multithreaded correctness Warning.
|
||||
(Rakesh R via zhz)
|
||||
|
||||
HDFS-9113. ErasureCodingWorker#processErasureCodingTasks should not fail to process
|
||||
remaining tasks due to one invalid ECTask (umamahesh)
|
||||
|
||||
HDFS-9091. Erasure Coding: Provide DistributedFilesystem API to
|
||||
getAllErasureCodingPolicies. (Rakesh R via zhz)
|
||||
|
||||
HDFS-8920. Erasure Coding: when recovering lost blocks, logs can be too
|
||||
verbose and hurt performance. (Rui Li via Kai Zheng)
|
||||
|
||||
HDFS-8882. Erasure Coding: Use datablocks, parityblocks and cell size from
|
||||
ErasureCodingPolicy (Vinayakumar B via zhz)
|
||||
|
||||
HDFS-9040. Erasure coding: coordinate data streamers in
|
||||
DFSStripedOutputStream. (jing9 and Walter Su)
|
||||
|
||||
HDFS-9185. Fix null tracer in ErasureCodingWorker. (Rakesh R via jing9)
|
||||
|
||||
HDFS-9180. Update excluded DataNodes in DFSStripedOutputStream based on failures
|
||||
in data streamers. (jing9)
|
||||
|
||||
HDFS-8557. Allow to configure RS and XOR raw coders (Kai Zheng)
|
||||
|
||||
Release 2.8.0 - UNRELEASED
|
||||
|
||||
NEW FEATURES
|
||||
|
|
|
|||
Loading…
Reference in New Issue