Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

HDFS-11033. Add documents for native raw erasure coder in XOR codes. Contributed by SammiChen.

This commit is contained in:
Wei-Chiu Chuang 2016-10-21 16:20:07 -07:00
parent df857f0d10
commit 25438526ed
2 changed files with 24 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
View File

@ -647,7 +647,27 @@
<name>io.erasurecode.codec.rs-default.rawcoder</name> <name>io.erasurecode.codec.rs-default.rawcoder</name>
<value>org.apache.hadoop.io.erasurecode.rawcoder.RSRawErasureCoderFactory</value> <value>org.apache.hadoop.io.erasurecode.rawcoder.RSRawErasureCoderFactory</value>
<description> <description>
Raw coder implementation for the rs-default codec. Raw coder implementation for the rs-default codec. The default value is a
pure Java implementation. There is also a native implementation. Its value
is org.apache.hadoop.io.erasurecode.rawcoder.NativeRSRawErasureCoderFactory.
</description>
</property>
<property>
<name>io.erasurecode.codec.rs-legacy.rawcoder</name>
<value>org.apache.hadoop.io.erasurecode.rawcoder.RSRawErasureCoderFactoryLegacy</value>
<description>
Raw coder implementation for the rs-legacy codec.
</description>
</property>
<property>
<name>io.erasurecode.codec.xor.rawcoder</name>
<value>org.apache.hadoop.io.erasurecode.rawcoder.XORRawErasureCoderFactory</value>
<description>
Raw coder implementation for the xor codec. The default value is a pure Java
implementation. There is also a native implementation. Its value is
org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawErasureCoderFactory.
</description> </description>
</property> </property>

4
hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
View File

@ -102,7 +102,7 @@ Deployment
`io.erasurecode.codec.rs-default.rawcoder` for the default RS codec, `io.erasurecode.codec.rs-default.rawcoder` for the default RS codec,
`io.erasurecode.codec.rs-legacy.rawcoder` for the legacy RS codec, `io.erasurecode.codec.rs-legacy.rawcoder` for the legacy RS codec,
`io.erasurecode.codec.xor.rawcoder` for the XOR codec. `io.erasurecode.codec.xor.rawcoder` for the XOR codec.
The default implementations for all of these codecs are pure Java. For default RS codec, there is also a native implementation which leverages Intel ISA-L library to improve the encoding and decoding calculation. Please refer to section "Enable Intel ISA-L" for more detail information. The default implementations for all of these codecs are pure Java. For default RS codec, there is also a native implementation which leverages Intel ISA-L library to improve the performance of codec. For XOR codec, a native implementation which leverages Intel ISA-L library to improve the performance of codec is also supported. Please refer to section "Enable Intel ISA-L" for more detail information.
Erasure coding background recovery work on the DataNodes can also be tuned via the following configuration parameters: Erasure coding background recovery work on the DataNodes can also be tuned via the following configuration parameters:
@ -119,6 +119,8 @@ Deployment
To check ISA-L library enable state, try "Hadoop checknative" command. It will tell you if ISA-L library is enabled or not. To check ISA-L library enable state, try "Hadoop checknative" command. It will tell you if ISA-L library is enabled or not.
It also requires three steps to enable the native implementation of XOR codec. The first two steps are the same as the above step 1 and step 2. In step 3, configure the `io.erasurecode.codec.xor.rawcoder` key with `org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawErasureCoderFactory` on both HDFS client and DataNodes.
### Administrative commands ### Administrative commands
HDFS provides an `erasurecode` subcommand to perform administrative commands related to erasure coding. HDFS provides an `erasurecode` subcommand to perform administrative commands related to erasure coding.