From 719df99c05a4e0bc62a7cad6d22c8dbe5d92bde0 Mon Sep 17 00:00:00 2001 From: Yiqun Lin Date: Tue, 14 Feb 2017 18:48:08 +0800 Subject: [PATCH] HDFS-11407. Document the missing usages of OfflineImageViewer processors. Contributed by Yiqun Lin. --- .../src/site/markdown/HdfsImageViewer.md | 63 +++++++++++++++++++ 1 file changed, 63 insertions(+) diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsImageViewer.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsImageViewer.md index b677f6af480..10d98b23f27 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsImageViewer.md +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsImageViewer.md @@ -133,6 +133,69 @@ Applying the Offline Image Viewer with XML processor would result in the followi ...remaining output omitted... +### ReverseXML Processor + +ReverseXML processor is the opposite of the XML processor. Users can specify input XML file and output fsimage file via -i and -o command-line. + + bash$ bin/hdfs oiv -p ReverseXML -i fsimage.xml -o fsimage + +This will reconstruct an fsimage from an XML file. + +### FileDistribution Processor + +FileDistribution processor can analyze file sizes in the namespace image. Users can specify maxSize (128GB by default) and step (2MB by default) in bytes via -maxSize and -step command-line. + + bash$ bin/hdfs oiv -p FileDistribution -maxSize maxSize -step size -i fsimage -o output + +The processor will calculate how many files in the system fall into each segment. The output file is formatted as a tab separated two column table showed as the following output: + + Size NumFiles + 4 1 + 12 1 + 16 1 + 20 1 + totalFiles = 4 + totalDirectories = 2 + totalBlocks = 4 + totalSpace = 48 + maxFileSize = 21 + +To make the output result look more readable, users can specify -format option in addition. + + bash$ bin/hdfs oiv -p FileDistribution -maxSize maxSize -step size -format -i fsimage -o output + +This would result in the following output: + + Size Range NumFiles + (0 B, 4 B] 1 + (8 B, 12 B] 1 + (12 B, 16 B] 1 + (16 B, 21 B] 1 + totalFiles = 4 + totalDirectories = 2 + totalBlocks = 4 + totalSpace = 48 + maxFileSize = 21 + +### Delimited Processor + +Delimited processor generates a text representation of the fsimage, with each element separated by a delimiter string (\t by default). Users can specify a new delimiter string by -delimiter option. + + bash$ bin/hdfs oiv -p Delimited -delimiter delimiterString -i fsimage -o output + +In addition, users can specify a temporary dir to cache intermediate result by the following command: + + bash$ bin/hdfs oiv -p Delimited -delimiter delimiterString -t temporaryDir -i fsimage -o output + +If not set, Delimited processor will construct the namespace in memory before outputting text. The output result of this processor should be like the following output: + + Path Replication ModificationTime AccessTime PreferredBlockSize BlocksCount FileSize NSQUOTA DSQUOTA Permission UserName GroupName + / 0 2017-02-13 10:39 1970-01-01 08:00 0 0 0 9223372036854775807 -1 drwxr-xr-x root supergroup + /dir0 0 2017-02-13 10:39 1970-01-01 08:00 0 0 0 -1 -1 drwxr-xr-x root supergroup + /dir0/file0 1 2017-02-13 10:39 2017-02-13 10:39 134217728 1 1 0 0 -rw-r--r-- root supergroup + /dir0/file1 1 2017-02-13 10:39 2017-02-13 10:39 134217728 1 1 0 0 -rw-r--r-- root supergroup + /dir0/file2 1 2017-02-13 10:39 2017-02-13 10:39 134217728 1 1 0 0 -rw-r--r-- root supergroup + Options -------