Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

MAPREDUCE-6257. Document encrypted spills (Bibin A Chundatt via aw)

This commit is contained in:
Allen Wittenauer 2015-08-06 10:13:18 -07:00
parent e73a928a63
commit fb1be0b310
3 changed files with 29 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
hadoop-mapreduce-project/CHANGES.txt
View File

@ -206,6 +206,8 @@ Trunk (Unreleased)
MAPREDUCE-6406. Update FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION_DEFAULT
to match mapred-default.xml. (Ray Chiang via devaraj)
MAPREDUCE-6257. Document encrypted spills (Bibin A Chundatt via aw)
BREAKDOWN OF MAPREDUCE-2841 (NATIVE TASK) SUBTASKS
MAPREDUCE-5985. native-task: Fix build on macosx. Contributed by

20
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
View File

@ -1760,4 +1760,24 @@
</description>
</property>
<property>
<name>mapreduce.job.encrypted-intermediate-data</name>
<value>false</value>
<description>Encrypt intermediate MapReduce spill files or not
default is false</description>
</property>
<property>
<name>mapreduce.job.encrypted-intermediate-data-key-size-bits</name>
<value>128</value>
<description>Mapreduce encrypt data key size default is 128</description>
</property>
<property>
<name>mapreduce.job.encrypted-intermediate-data.buffer.kb</name>
<value>128</value>
<description>Buffer size for intermediate encrypt data in kb
default is 128</description>
</property>
</configuration>

6
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/EncryptedShuffle.md
View File

@ -260,4 +260,10 @@ Encrypted Intermediate Data Spill files
This capability allows encryption of the intermediate files generated during the merge and shuffle phases.
It can be enabled by setting the `mapreduce.job.encrypted-intermediate-data` job property to `true`.
| Name | Type | Description |
|:---- |:---- |:---- |
| mapreduce.job.encrypted-intermediate-data | boolean | Enable or disable encrypt intermediate mapreduce spill files.Default is false. |
| mapreduce.job.encrypted-intermediate-data-key-size-bits | int | The key length used by keygenerator to encrypt data spilled to disk. |
| mapreduce.job.encrypted-intermediate-data.buffer.kb | int | The buffer size in kb for stream written to disk after encryption. |
**NOTE:** Currently, enabling encrypted intermediate data spills would restrict the number of attempts of the job to 1.