There's some bugs in the FifoScheduler atm - doesn't distribute tasks across nodes and some headroom (available resource) issues. -That's not the best experience for users trying out the 2.0 branch. The CS with the default configuration of a single queue behaves the same as the FifoScheduler and doesn't have these issues. +There's some bugs in the FifoScheduler atm - doesn't distribute tasks across nodes and some headroom (available resource) issues. +That's not the best experience for users trying out the 2.0 branch. The CS with the default configuration of a single queue behaves the same as the FifoScheduler and doesn't have these issues.
The following exception is thrown -=========== -*org.apache.hadoop.HadoopIllegalArgumentException: Cannot close proxy - is not Closeable or does not provide closeable invocation handler class org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl* - *at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:624)* - *at org.hadoop.yarn.client.YarnClientImpl.stop(YarnClientImpl.java:102)* - at org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.run(UnmanagedAMLauncher.java:336) - at org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.testDSShell(TestUnmanagedAMLauncher.java:156) - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) - at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) - at java.lang.reflect.Method.invoke(Method.java:597) - at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) - at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) - at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) - at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) - at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) - at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) - at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49) - at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) - at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) - at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) - at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) - at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) - at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) - at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) - at org.junit.runners.ParentRunner.run(ParentRunner.java:236) - at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236) - at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134) - at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113) - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) - at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) - at java.lang.reflect.Method.invoke(Method.java:597) - at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) - at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) - at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) - at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103) - at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74) +The following exception is thrown +=========== +*org.apache.hadoop.HadoopIllegalArgumentException: Cannot close proxy - is not Closeable or does not provide closeable invocation handler class org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl* + *at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:624)* + *at org.hadoop.yarn.client.YarnClientImpl.stop(YarnClientImpl.java:102)* + at org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.run(UnmanagedAMLauncher.java:336) + at org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.testDSShell(TestUnmanagedAMLauncher.java:156) + at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) + at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) + at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) + at java.lang.reflect.Method.invoke(Method.java:597) + at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) + at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) + at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) + at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) + at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) + at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) + at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49) + at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) + at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) + at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) + at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) + at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) + at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) + at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) + at org.junit.runners.ParentRunner.run(ParentRunner.java:236) + at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236) + at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134) + at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113) + at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) + at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) + at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) + at java.lang.reflect.Method.invoke(Method.java:597) + at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) + at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) + at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) + at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103) + at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74) ===========
An AppMaster can send a container release at any point. Currently this results in an exception, if this is done while the RM considers the container to be RUNNING. +An AppMaster can send a container release at any point. Currently this results in an exception, if this is done while the RM considers the container to be RUNNING. The event not being processed correctly also implies that these containers do not show up in the Completed Container List seen by the AM (AMRMProtocol). MR-3902 depends on this set being complete.
The nodemanager is able to get into a state where containermanager.logaggregation.AppLogAggregatorImpl will apparently wait -indefinitely for log aggregation to complete for an application, even if that application has abnormally terminated and is no longer present. - -Observed behavior is that an attempt to stop the nodemanager daemon will return but have no effect, the nm log continually displays messages similar to this: - -[Thread-1]2012-08-21 17:44:07,581 INFO -org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: -Waiting for aggregation to complete for application_1345221477405_2733 - -The only recovery we found to work was to 'kill -9' the nm process. - -What exactly causes the NM to enter this state is unclear but we do see this behavior reliably when the NM has run a task which failed, for example when debugging oozie distcp actions and having a distcp map task fail, the NM that was running the container will now enter this state where a shutdown on said NM will never complete, 'never' in this case was waiting for 2 hours before killing the nodemanager process. +The nodemanager is able to get into a state where containermanager.logaggregation.AppLogAggregatorImpl will apparently wait +indefinitely for log aggregation to complete for an application, even if that application has abnormally terminated and is no longer present. + +Observed behavior is that an attempt to stop the nodemanager daemon will return but have no effect, the nm log continually displays messages similar to this: + +[Thread-1]2012-08-21 17:44:07,581 INFO +org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: +Waiting for aggregation to complete for application_1345221477405_2733 + +The only recovery we found to work was to 'kill -9' the nm process. + +What exactly causes the NM to enter this state is unclear but we do see this behavior reliably when the NM has run a task which failed, for example when debugging oozie distcp actions and having a distcp map task fail, the NM that was running the container will now enter this state where a shutdown on said NM will never complete, 'never' in this case was waiting for 2 hours before killing the nodemanager process.
If the default file permissions are set to something restrictive - like 700, application logs get aggregated and created with those restrictive file permissions which doesn't allow the history server to serve them up. - - -They need to be created with group readable similar to how log aggregation sets up the directory permissions. +If the default file permissions are set to something restrictive - like 700, application logs get aggregated and created with those restrictive file permissions which doesn't allow the history server to serve them up. + + +They need to be created with group readable similar to how log aggregation sets up the directory permissions.
NM throws NPE on startup if it doesn't have persmission's on nm local dir's - - -{code:xml} -2012-05-14 16:32:13,468 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager -org.apache.hadoop.yarn.YarnException: Failed to initialize LocalizationService - at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.init(ResourceLocalizationService.java:202) - at org.apache.hadoop.yarn.service.CompositeService.init(CompositeService.java:58) - at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.init(ContainerManagerImpl.java:183) - at org.apache.hadoop.yarn.service.CompositeService.init(CompositeService.java:58) - at org.apache.hadoop.yarn.server.nodemanager.NodeManager.init(NodeManager.java:166) - at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:268) - at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:284) -Caused by: java.io.IOException: mkdir of /mrv2/tmp/nm-local-dir/usercache failed - at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:907) - at org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:143) - at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:189) - at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:706) - at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:703) - at org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2325) - at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:703) - at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.init(ResourceLocalizationService.java:188) - ... 6 more -2012-05-14 16:32:13,472 INFO org.apache.hadoop.yarn.service.CompositeService: Error stopping org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler -java.lang.NullPointerException - at org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler.stop(NonAggregatingLogHandler.java:82) - at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:99) - at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89) - at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.stop(ContainerManagerImpl.java:266) - at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:99) - at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89) - at org.apache.hadoop.yarn.server.nodemanager.NodeManager.stop(NodeManager.java:182) - at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:122) - at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) -{code} +NM throws NPE on startup if it doesn't have persmission's on nm local dir's + + +{code:xml} +2012-05-14 16:32:13,468 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager +org.apache.hadoop.yarn.YarnException: Failed to initialize LocalizationService + at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.init(ResourceLocalizationService.java:202) + at org.apache.hadoop.yarn.service.CompositeService.init(CompositeService.java:58) + at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.init(ContainerManagerImpl.java:183) + at org.apache.hadoop.yarn.service.CompositeService.init(CompositeService.java:58) + at org.apache.hadoop.yarn.server.nodemanager.NodeManager.init(NodeManager.java:166) + at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:268) + at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:284) +Caused by: java.io.IOException: mkdir of /mrv2/tmp/nm-local-dir/usercache failed + at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:907) + at org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:143) + at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:189) + at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:706) + at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:703) + at org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2325) + at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:703) + at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.init(ResourceLocalizationService.java:188) + ... 6 more +2012-05-14 16:32:13,472 INFO org.apache.hadoop.yarn.service.CompositeService: Error stopping org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler +java.lang.NullPointerException + at org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler.stop(NonAggregatingLogHandler.java:82) + at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:99) + at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89) + at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.stop(ContainerManagerImpl.java:266) + at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:99) + at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89) + at org.apache.hadoop.yarn.server.nodemanager.NodeManager.stop(NodeManager.java:182) + at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:122) + at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) +{code}
- RM should generate the master-key randomly - - The master-key should roll every so often +- RM should generate the master-key randomly + - The master-key should roll every so often - NM should remember old expired keys so that already doled out container-requests can be satisfied.
branch-2.1.0-alpha doesn't build due to the following. Per YARN-1 I updated the mvn version to be 2.1.0-SNAPSHOT, before I hit this issue it didn't compile due to the bogus version. - -{noformat} -hadoop-branch-2.1.0-alpha $ mvn compile -[INFO] Scanning for projects... -[ERROR] The build could not read 1 project -> [Help 1] -[ERROR] -[ERROR] The project org.apache.hadoop:hadoop-yarn-project:2.1.0-SNAPSHOT (/home/eli/src/hadoop-branch-2.1.0-alpha/hadoop-yarn-project/pom.xml) has 1 error -[ERROR] 'dependencies.dependency.version' for org.hsqldb:hsqldb:jar is missing. @ line 160, column 17 +branch-2.1.0-alpha doesn't build due to the following. Per YARN-1 I updated the mvn version to be 2.1.0-SNAPSHOT, before I hit this issue it didn't compile due to the bogus version. + +{noformat} +hadoop-branch-2.1.0-alpha $ mvn compile +[INFO] Scanning for projects... +[ERROR] The build could not read 1 project -> [Help 1] +[ERROR] +[ERROR] The project org.apache.hadoop:hadoop-yarn-project:2.1.0-SNAPSHOT (/home/eli/src/hadoop-branch-2.1.0-alpha/hadoop-yarn-project/pom.xml) has 1 error +[ERROR] 'dependencies.dependency.version' for org.hsqldb:hsqldb:jar is missing. @ line 160, column 17 {noformat}
TestDelegationTokenRenewer fails when run with jdk7. - +TestDelegationTokenRenewer fails when run with jdk7. + With JDK7, test methods run in an undefined order. Here it is expecting that testDTRenewal runs first but it no longer is.
I see that we are duplicating (some) code for talking to RM via client API. In this light, a yarn-client module will be useful so that clients of all frameworks can use/extend it. - +I see that we are duplicating (some) code for talking to RM via client API. In this light, a yarn-client module will be useful so that clients of all frameworks can use/extend it. + And that same module can be the destination for all the YARN's command line tools.
Stumbled upon this problem while refreshing queues with incorrect configuration. The exact scenario was: -1. Added a new queue "newQueue" without defining its capacity. -2. "bin/mapred queue -refreshQueues" fails correctly with "Illegal capacity of -1 for queue root.newQueue" -3. However, after defining the capacity of "newQueue" followed by a second "bin/mapred queue -refreshQueues" throws "org.apache.hadoop.metrics2.MetricsException: Metrics source QueueMetrics,q0=root,q1=newQueue already exists!" Also see Hadoop:name=QueueMetrics,q0=root,q1=newQueue,service=ResourceManager metrics being available even though the queue was not added. - +Stumbled upon this problem while refreshing queues with incorrect configuration. The exact scenario was: +1. Added a new queue "newQueue" without defining its capacity. +2. "bin/mapred queue -refreshQueues" fails correctly with "Illegal capacity of -1 for queue root.newQueue" +3. However, after defining the capacity of "newQueue" followed by a second "bin/mapred queue -refreshQueues" throws "org.apache.hadoop.metrics2.MetricsException: Metrics source QueueMetrics,q0=root,q1=newQueue already exists!" Also see Hadoop:name=QueueMetrics,q0=root,q1=newQueue,service=ResourceManager metrics being available even though the queue was not added. + The expected behavior would be to refresh the queues correctly and allow addition of "newQueue".
Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period. - +Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period. + It might make sense for job history server to remove them.
-{code} - /** - * Default CLASSPATH for YARN applications. A comma-separated list of - * CLASSPATH entries - */ - public static final String[] DEFAULT_YARN_APPLICATION_CLASSPATH = { - "$HADOOP_CONF_DIR", "$HADOOP_COMMON_HOME/share/hadoop/common/*", - "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*", - "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*", - "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*", - "$YARN_HOME/share/hadoop/mapreduce/*", - "$YARN_HOME/share/hadoop/mapreduce/lib/*"}; -{code} - ++{code} + /** + * Default CLASSPATH for YARN applications. A comma-separated list of + * CLASSPATH entries + */ + public static final String[] DEFAULT_YARN_APPLICATION_CLASSPATH = { + "$HADOOP_CONF_DIR", "$HADOOP_COMMON_HOME/share/hadoop/common/*", + "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*", + "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*", + "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*", + "$YARN_HOME/share/hadoop/mapreduce/*", + "$YARN_HOME/share/hadoop/mapreduce/lib/*"}; +{code} + It should have {{share/yarn/}} and MR should add the {{share/mapreduce/}} (another JIRA?)
Trying to create a symlink to another file that is specified for the distributed cache will fail to create the link. For example: - -hadoop jar ... -files "x,y,x#z" - +Trying to create a symlink to another file that is specified for the distributed cache will fail to create the link. For example: + +hadoop jar ... -files "x,y,x#z" + will localize the files x and y as x and y, but the z symlink for x will not be created. This is a regression from 1.x behavior.
The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler: -org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception -Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time -Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time -Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time -The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE - +The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler: +org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception +Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time +Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time +Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time +The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE +
Removes two sets of previously available config properties: - -1. ( yarn.scheduler.fifo.minimum-allocation-mb and yarn.scheduler.fifo.maximum-allocation-mb ) and, -2. ( yarn.scheduler.capacity.minimum-allocation-mb and yarn.scheduler.capacity.maximum-allocation-mb ) - -In favor of two new, generically named properties: - -1. yarn.scheduler.minimum-allocation-mb - This acts as the floor value of memory resource requests for containers. -2. yarn.scheduler.maximum-allocation-mb - This acts as the ceiling value of memory resource requests for containers. - -Both these properties need to be set at the ResourceManager (RM) to take effect, as the RM is where the scheduler resides. - +Removes two sets of previously available config properties: + +1. ( yarn.scheduler.fifo.minimum-allocation-mb and yarn.scheduler.fifo.maximum-allocation-mb ) and, +2. ( yarn.scheduler.capacity.minimum-allocation-mb and yarn.scheduler.capacity.maximum-allocation-mb ) + +In favor of two new, generically named properties: + +1. yarn.scheduler.minimum-allocation-mb - This acts as the floor value of memory resource requests for containers. +2. yarn.scheduler.maximum-allocation-mb - This acts as the ceiling value of memory resource requests for containers. + +Both these properties need to be set at the ResourceManager (RM) to take effect, as the RM is where the scheduler resides. + Also changes the default minimum and maximums to 128 MB and 10 GB respectively.
Note that to apply this you should first run the script - ./MAPREDUCE-3543v3.sh svn, then apply the patch. - +Note that to apply this you should first run the script - ./MAPREDUCE-3543v3.sh svn, then apply the patch. + If this is merged to more then trunk, the version inside of hadoop-tools/hadoop-gridmix/pom.xml will need to be udpated accordingly.
This change adds two new configuration parameters. -# {{dfs.namenode.invalidate.work.pct.per.iteration}} for controlling deletion rate of blocks. -# {{dfs.namenode.replication.work.multiplier.per.iteration}} for controlling replication rate. This in turn allows controlling the time it takes for decommissioning. - +This change adds two new configuration parameters. +# {{dfs.namenode.invalidate.work.pct.per.iteration}} for controlling deletion rate of blocks. +# {{dfs.namenode.replication.work.multiplier.per.iteration}} for controlling replication rate. This in turn allows controlling the time it takes for decommissioning. + Please see hdfs-default.xml for detailed description.
Changed bin/mapred job -list to not print job-specific information not available at RM. - +Changed bin/mapred job -list to not print job-specific information not available at RM. + Very minor incompatibility in cmd-line output, inevitable due to MRv2 architecture.
New JMX Bean in ResourceManager to provide list of live node managers: - +New JMX Bean in ResourceManager to provide list of live node managers: + Hadoop:service=ResourceManager,name=RMNMInfo LiveNodeManagers
new files added: A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WebServicesIntro.apt.vm -A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/NodeManagerRest.apt.vm -A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm -A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/MapredAppMasterRest.apt.vm -A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm - +new files added: A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WebServicesIntro.apt.vm +A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/NodeManagerRest.apt.vm +A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm +A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/MapredAppMasterRest.apt.vm +A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm + The hadoop-project/src/site/site.xml is split into separate patch.
Moved log related components into yarn-common so that HistoryServer and clients can use them without depending on the yarn-server-nodemanager module. +Moved log related components into yarn-common so that HistoryServer and clients can use them without depending on the yarn-server-nodemanager module.
Reenabled and fixed bugs in the failing test TestDelegationToken. +Reenabled and fixed bugs in the failing test TestDelegationToken.
Reenabled and fixed bugs in the failing ant test TestAuditLogger. +Reenabled and fixed bugs in the failing ant test TestAuditLogger.
Reneabled and fixed bugs in the failing test TestNoJobSetupCleanup. +Reneabled and fixed bugs in the failing test TestNoJobSetupCleanup.
1. New configurations -a. dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read. -b. dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration. -c. dfs.client.read.shortcircuit.skip.checksum is the key to bypass checksum check at the client side. -2. By default none of the above are enabled and short circuit read will not kick in. -3. If security is on, the feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general. +1. New configurations +a. dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read. +b. dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration. +c. dfs.client.read.shortcircuit.skip.checksum is the key to bypass checksum check at the client side. +2. By default none of the above are enabled and short circuit read will not kick in. +3. If security is on, the feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general.
Here is a patch to enable this behavior +Here is a patch to enable this behavior
New Yarn configuration property: - -Name: yarn.app.mapreduce.am.scheduler.connection.retries +New Yarn configuration property: + +Name: yarn.app.mapreduce.am.scheduler.connection.retries Description: Number of times AM should retry to contact RM if connection is lost.
Removed inheritance of certain server environment variables (HADOOP_OPTS and HADOOP_ROOT_LOGGER) in task attempt process. +Removed inheritance of certain server environment variables (HADOOP_OPTS and HADOOP_ROOT_LOGGER) in task attempt process.
A new server has been added to yarn. It is a web proxy that sits in front of the AM web UI. The server is controlled by the yarn.web-proxy.address config. If that config is set, and it points to an address that is different then the RM web interface then a separate proxy server needs to be launched. - -This can be done by running - -yarn-daemon.sh start proxyserver - -If a separate proxy server is needed other configs also may need to be set, if security is enabled. -yarn.web-proxy.principal -yarn.web-proxy.keytab - +A new server has been added to yarn. It is a web proxy that sits in front of the AM web UI. The server is controlled by the yarn.web-proxy.address config. If that config is set, and it points to an address that is different then the RM web interface then a separate proxy server needs to be launched. + +This can be done by running + +yarn-daemon.sh start proxyserver + +If a separate proxy server is needed other configs also may need to be set, if security is enabled. +yarn.web-proxy.principal +yarn.web-proxy.keytab + The proxy server is stateless and should be able to support a VIP or other load balancing sitting in front of multiple instances of this server.
Capture intermediate task resource consumption information: -* Time taken so far -* CPU load [either at the time the data are taken, or exponentially smoothed] -* Memory load [also either at the time the data are taken, or exponentially smoothed] - -This would be taken at intervals that depend on the task progress plateaus. For example, reducers have three progress ranges - [0-1/3], (1/3-2/3], and (2/3-3/3] - where fundamentally different activities happen. Mappers have different boundaries that are not symmetrically placed [0-9/10], (9/10-1]. Data capture boundaries should coincide with activity boundaries. For the state information capture [CPU and memory] we should average over the covered interval. +Capture intermediate task resource consumption information: +* Time taken so far +* CPU load [either at the time the data are taken, or exponentially smoothed] +* Memory load [also either at the time the data are taken, or exponentially smoothed] + +This would be taken at intervals that depend on the task progress plateaus. For example, reducers have three progress ranges - [0-1/3], (1/3-2/3], and (2/3-3/3] - where fundamentally different activities happen. Mappers have different boundaries that are not symmetrically placed [0-9/10], (9/10-1]. Data capture boundaries should coincide with activity boundaries. For the state information capture [CPU and memory] we should average over the covered interval.
MapReduce has undergone a complete re-haul in hadoop-0.23 and we now have, what we call, MapReduce 2.0 (MRv2). - -The fundamental idea of MRv2 is to split up the two major functionalities of the JobTracker, resource management and job scheduling/monitoring, into separate daemons. The idea is to have a global ResourceManager (RM) and per-application ApplicationMaster (AM). An application is either a single job in the classical sense of Map-Reduce jobs or a DAG of jobs. The ResourceManager and per-node slave, the NodeManager (NM), form the data-computation framework. The ResourceManager is the ultimate authority that arbitrates resources among all the applications in the system. The per-application ApplicationMaster is, in effect, a framework specific library and is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the tasks. - -The ResourceManager has two main components: -* Scheduler (S) -* ApplicationsManager (ASM) - -The Scheduler is responsible for allocating resources to the various running applications subject to familiar constraints of capacities, queues etc. The Scheduler is pure scheduler in the sense that it performs no monitoring or tracking of status for the application. Also, it offers no guarantees on restarting failed tasks either due to application failure or hardware failures. The Scheduler performs its scheduling function based the resource requirements of the applications; it does so based on the abstract notion of a Resource Container which incorporates elements such as memory, cpu, disk, network etc. - -The Scheduler has a pluggable policy plug-in, which is responsible for partitioning the cluster resources among the various queues, applications etc. The current Map-Reduce schedulers such as the CapacityScheduler and the FairScheduler would be some examples of the plug-in. - -The CapacityScheduler supports hierarchical queues to allow for more predictable sharing of cluster resources. -The ApplicationsManager is responsible for accepting job-submissions, negotiating the first container for executing the application specific ApplicationMaster and provides the service for restarting the ApplicationMaster container on failure. - -The NodeManager is the per-machine framework agent who is responsible for launching the applications' containers, monitoring their resource usage (cpu, memory, disk, network) and reporting the same to the Scheduler. - -The per-application ApplicationMaster has the responsibility of negotiating appropriate resource containers from the Scheduler, tracking their status and monitoring for progress. +MapReduce has undergone a complete re-haul in hadoop-0.23 and we now have, what we call, MapReduce 2.0 (MRv2). + +The fundamental idea of MRv2 is to split up the two major functionalities of the JobTracker, resource management and job scheduling/monitoring, into separate daemons. The idea is to have a global ResourceManager (RM) and per-application ApplicationMaster (AM). An application is either a single job in the classical sense of Map-Reduce jobs or a DAG of jobs. The ResourceManager and per-node slave, the NodeManager (NM), form the data-computation framework. The ResourceManager is the ultimate authority that arbitrates resources among all the applications in the system. The per-application ApplicationMaster is, in effect, a framework specific library and is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the tasks. + +The ResourceManager has two main components: +* Scheduler (S) +* ApplicationsManager (ASM) + +The Scheduler is responsible for allocating resources to the various running applications subject to familiar constraints of capacities, queues etc. The Scheduler is pure scheduler in the sense that it performs no monitoring or tracking of status for the application. Also, it offers no guarantees on restarting failed tasks either due to application failure or hardware failures. The Scheduler performs its scheduling function based the resource requirements of the applications; it does so based on the abstract notion of a Resource Container which incorporates elements such as memory, cpu, disk, network etc. + +The Scheduler has a pluggable policy plug-in, which is responsible for partitioning the cluster resources among the various queues, applications etc. The current Map-Reduce schedulers such as the CapacityScheduler and the FairScheduler would be some examples of the plug-in. + +The CapacityScheduler supports hierarchical queues to allow for more predictable sharing of cluster resources. +The ApplicationsManager is responsible for accepting job-submissions, negotiating the first container for executing the application specific ApplicationMaster and provides the service for restarting the ApplicationMaster container on failure. + +The NodeManager is the per-machine framework agent who is responsible for launching the applications' containers, monitoring their resource usage (cpu, memory, disk, network) and reporting the same to the Scheduler. + +The per-application ApplicationMaster has the responsibility of negotiating appropriate resource containers from the Scheduler, tracking their status and monitoring for progress.
HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache. This support is currently considered experimental, and may be enabled by configuring the following keys: -dfs.datanode.drop.cache.behind.writes - set to true to drop data out of the buffer cache after writing -dfs.datanode.drop.cache.behind.reads - set to true to drop data out of the buffer cache when performing sequential reads -dfs.datanode.sync.behind.writes - set to true to trigger dirty page writeback immediately after writing data +HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache. This support is currently considered experimental, and may be enabled by configuring the following keys: +dfs.datanode.drop.cache.behind.writes - set to true to drop data out of the buffer cache after writing +dfs.datanode.drop.cache.behind.reads - set to true to drop data out of the buffer cache when performing sequential reads +dfs.datanode.sync.behind.writes - set to true to trigger dirty page writeback immediately after writing data dfs.datanode.readahead.bytes - set to a non-zero value to trigger readahead for sequential reads
Implemented a daemon thread to monitor the disk usage for periodically and if the disk usage reaches the threshold value, put the name node into Safe mode so that no modification to file system will occur. Once the disk usage reaches below the threshold, name node will be put out of the safe mode. Here threshold value and interval to check the disk usage are configurable. +Implemented a daemon thread to monitor the disk usage for periodically and if the disk usage reaches the threshold value, put the name node into Safe mode so that no modification to file system will occur. Once the disk usage reaches below the threshold, name node will be put out of the safe mode. Here threshold value and interval to check the disk usage are configurable.
Summary of changes to the decommissioning process: -# After nodes are decommissioned, they are not shutdown. The decommissioned nodes are not used for writes. For reads, the decommissioned nodes are given as the last location to read from. -# Number of live and dead decommissioned nodes are displayed in the namenode webUI. +Summary of changes to the decommissioning process: +# After nodes are decommissioned, they are not shutdown. The decommissioned nodes are not used for writes. For reads, the decommissioned nodes are given as the last location to read from. +# Number of live and dead decommissioned nodes are displayed in the namenode webUI. # Decommissioned nodes free capacity is not count towards the the cluster free capacity.
Offline edits viewer feature adds oev tool to hdfs script. Oev makes it possible to convert edits logs to/from native binary and XML formats. It uses the same framework as Offline image viewer. - -Example usage: - +Offline edits viewer feature adds oev tool to hdfs script. Oev makes it possible to convert edits logs to/from native binary and XML formats. It uses the same framework as Offline image viewer. + +Example usage: + $HADOOP_HOME/bin/hdfs oev -i edits -o output.xml
HADOOP-7681. Fixed security and hdfs audit log4j properties +HADOOP-7681. Fixed security and hdfs audit log4j properties (Arpit Gupta via Eric Yang)
1. Protocol version check is removed from proxy creation, instead version check is performed at server in every rpc call. -2. This change is backward incompatible because format of the rpc messages is changed to include client version, client method hash and rpc version. +1. Protocol version check is removed from proxy creation, instead version check is performed at server in every rpc call. +2. This change is backward incompatible because format of the rpc messages is changed to include client version, client method hash and rpc version. 3. rpc version is introduced which should change when the format of rpc messages is changed.
getFactory() method.
- */
-@InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce"})
-@InterfaceStability.Evolving
-public class ContextFactory {
-
- private static final String PROPERTIES_FILE =
- "/hadoop-metrics.properties";
- private static final String CONTEXT_CLASS_SUFFIX =
- ".class";
- private static final String DEFAULT_CONTEXT_CLASSNAME =
- "org.apache.hadoop.metrics.spi.NullContext";
-
- private static ContextFactory theFactory = null;
-
- private Maporg.apache.hadoop.metrics.spi.NullContext, which is a
- * dummy "no-op" context which will cause all metric data to be discarded.
- *
- * @param contextName the name of the context
- * @return the named MetricsContext
- */
- public synchronized MetricsContext getContext(String refName, String contextName)
- throws IOException, ClassNotFoundException,
- InstantiationException, IllegalAccessException {
- MetricsContext metricsContext = contextMap.get(refName);
- if (metricsContext == null) {
- String classNameAttribute = refName + CONTEXT_CLASS_SUFFIX;
- String className = (String) getAttribute(classNameAttribute);
- if (className == null) {
- className = DEFAULT_CONTEXT_CLASSNAME;
- }
- Class contextClass = Class.forName(className);
- metricsContext = (MetricsContext) contextClass.newInstance();
- metricsContext.init(contextName, this);
- contextMap.put(contextName, metricsContext);
- }
- return metricsContext;
- }
-
- public synchronized MetricsContext getContext(String contextName)
- throws IOException, ClassNotFoundException, InstantiationException,
- IllegalAccessException {
- return getContext(contextName, contextName);
- }
-
- /**
- * Returns all MetricsContexts built by this factory.
- */
- public synchronized Collectionhadoop-metrics.properties exists on the class path. If it
- * exists, it must be in the format defined by java.util.Properties, and all
- * the properties in the file are set as attributes on the newly created
- * ContextFactory instance.
- *
- * @return the singleton ContextFactory instance
- */
- public static synchronized ContextFactory getFactory() throws IOException {
- if (theFactory == null) {
- theFactory = new ContextFactory();
- theFactory.setAttributes();
- }
- return theFactory;
- }
-
- private void setAttributes() throws IOException {
- InputStream is = getClass().getResourceAsStream(PROPERTIES_FILE);
- if (is != null) {
- try {
- Properties properties = new Properties();
- properties.load(is);
- //for (Object propertyNameObj : properties.keySet()) {
- Iterator it = properties.keySet().iterator();
- while (it.hasNext()) {
- String propertyName = (String) it.next();
- String propertyValue = properties.getProperty(propertyName);
- setAttribute(propertyName, propertyValue);
- }
- } finally {
- is.close();
- }
- }
- }
-
-}
+/*
+ * ContextFactory.java
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.metrics;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.metrics.spi.NullContext;
+
+/**
+ * Factory class for creating MetricsContext objects. To obtain an instance
+ * of this class, use the static getFactory() method.
+ */
+@InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce"})
+@InterfaceStability.Evolving
+public class ContextFactory {
+
+ private static final String PROPERTIES_FILE =
+ "/hadoop-metrics.properties";
+ private static final String CONTEXT_CLASS_SUFFIX =
+ ".class";
+ private static final String DEFAULT_CONTEXT_CLASSNAME =
+ "org.apache.hadoop.metrics.spi.NullContext";
+
+ private static ContextFactory theFactory = null;
+
+ private Maporg.apache.hadoop.metrics.spi.NullContext, which is a
+ * dummy "no-op" context which will cause all metric data to be discarded.
+ *
+ * @param contextName the name of the context
+ * @return the named MetricsContext
+ */
+ public synchronized MetricsContext getContext(String refName, String contextName)
+ throws IOException, ClassNotFoundException,
+ InstantiationException, IllegalAccessException {
+ MetricsContext metricsContext = contextMap.get(refName);
+ if (metricsContext == null) {
+ String classNameAttribute = refName + CONTEXT_CLASS_SUFFIX;
+ String className = (String) getAttribute(classNameAttribute);
+ if (className == null) {
+ className = DEFAULT_CONTEXT_CLASSNAME;
+ }
+ Class contextClass = Class.forName(className);
+ metricsContext = (MetricsContext) contextClass.newInstance();
+ metricsContext.init(contextName, this);
+ contextMap.put(contextName, metricsContext);
+ }
+ return metricsContext;
+ }
+
+ public synchronized MetricsContext getContext(String contextName)
+ throws IOException, ClassNotFoundException, InstantiationException,
+ IllegalAccessException {
+ return getContext(contextName, contextName);
+ }
+
+ /**
+ * Returns all MetricsContexts built by this factory.
+ */
+ public synchronized Collectionhadoop-metrics.properties exists on the class path. If it
+ * exists, it must be in the format defined by java.util.Properties, and all
+ * the properties in the file are set as attributes on the newly created
+ * ContextFactory instance.
+ *
+ * @return the singleton ContextFactory instance
+ */
+ public static synchronized ContextFactory getFactory() throws IOException {
+ if (theFactory == null) {
+ theFactory = new ContextFactory();
+ theFactory.setAttributes();
+ }
+ return theFactory;
+ }
+
+ private void setAttributes() throws IOException {
+ InputStream is = getClass().getResourceAsStream(PROPERTIES_FILE);
+ if (is != null) {
+ try {
+ Properties properties = new Properties();
+ properties.load(is);
+ //for (Object propertyNameObj : properties.keySet()) {
+ Iterator it = properties.keySet().iterator();
+ while (it.hasNext()) {
+ String propertyName = (String) it.next();
+ String propertyValue = properties.getProperty(propertyName);
+ setAttribute(propertyName, propertyValue);
+ }
+ } finally {
+ is.close();
+ }
+ }
+ }
+
+}
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics/MetricsContext.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics/MetricsContext.java
index b84d5265036..e297e3738b1 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics/MetricsContext.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics/MetricsContext.java
@@ -1,122 +1,122 @@
-/*
- * MetricsContext.java
- *
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hadoop.metrics;
-
-import java.io.IOException;
-import java.util.Collection;
-import java.util.Map;
-
-import org.apache.hadoop.classification.InterfaceAudience;
-import org.apache.hadoop.classification.InterfaceStability;
-import org.apache.hadoop.metrics.spi.OutputRecord;
-
-/**
- * The main interface to the metrics package.
- */
-@InterfaceAudience.Private
-@InterfaceStability.Evolving
-public interface MetricsContext {
-
- /**
- * Default period in seconds at which data is sent to the metrics system.
- */
- public static final int DEFAULT_PERIOD = 5;
-
- /**
- * Initialize this context.
- * @param contextName The given name for this context
- * @param factory The creator of this context
- */
- public void init(String contextName, ContextFactory factory);
-
- /**
- * Returns the context name.
- *
- * @return the context name
- */
- public abstract String getContextName();
-
- /**
- * Starts or restarts monitoring, the emitting of metrics records as they are
- * updated.
- */
- public abstract void startMonitoring()
- throws IOException;
-
- /**
- * Stops monitoring. This does not free any data that the implementation
- * may have buffered for sending at the next timer event. It
- * is OK to call startMonitoring() again after calling
- * this.
- * @see #close()
- */
- public abstract void stopMonitoring();
-
- /**
- * Returns true if monitoring is currently in progress.
- */
- public abstract boolean isMonitoring();
-
- /**
- * Stops monitoring and also frees any buffered data, returning this
- * object to its initial state.
- */
- public abstract void close();
-
- /**
- * Creates a new MetricsRecord instance with the given recordName.
- * Throws an exception if the metrics implementation is configured with a fixed
- * set of record names and recordName is not in that set.
- *
- * @param recordName the name of the record
- * @throws MetricsException if recordName conflicts with configuration data
- */
- public abstract MetricsRecord createRecord(String recordName);
-
- /**
- * Registers a callback to be called at regular time intervals, as
- * determined by the implementation-class specific configuration.
- *
- * @param updater object to be run periodically; it should updated
- * some metrics records and then return
- */
- public abstract void registerUpdater(Updater updater);
-
- /**
- * Removes a callback, if it exists.
- *
- * @param updater object to be removed from the callback list
- */
- public abstract void unregisterUpdater(Updater updater);
-
- /**
- * Returns the timer period.
- */
- public abstract int getPeriod();
-
- /**
- * Retrieves all the records managed by this MetricsContext.
- * Useful for monitoring systems that are polling-based.
- *
- * @return A non-null map from all record names to the records managed.
- */
- MapstartMonitoring() again after calling
+ * this.
+ * @see #close()
+ */
+ public abstract void stopMonitoring();
+
+ /**
+ * Returns true if monitoring is currently in progress.
+ */
+ public abstract boolean isMonitoring();
+
+ /**
+ * Stops monitoring and also frees any buffered data, returning this
+ * object to its initial state.
+ */
+ public abstract void close();
+
+ /**
+ * Creates a new MetricsRecord instance with the given recordName.
+ * Throws an exception if the metrics implementation is configured with a fixed
+ * set of record names and recordName is not in that set.
+ *
+ * @param recordName the name of the record
+ * @throws MetricsException if recordName conflicts with configuration data
+ */
+ public abstract MetricsRecord createRecord(String recordName);
+
+ /**
+ * Registers a callback to be called at regular time intervals, as
+ * determined by the implementation-class specific configuration.
+ *
+ * @param updater object to be run periodically; it should updated
+ * some metrics records and then return
+ */
+ public abstract void registerUpdater(Updater updater);
+
+ /**
+ * Removes a callback, if it exists.
+ *
+ * @param updater object to be removed from the callback list
+ */
+ public abstract void unregisterUpdater(Updater updater);
+
+ /**
+ * Returns the timer period.
+ */
+ public abstract int getPeriod();
+
+ /**
+ * Retrieves all the records managed by this MetricsContext.
+ * Useful for monitoring systems that are polling-based.
+ *
+ * @return A non-null map from all record names to the records managed.
+ */
+ Mapupdate() to pass the record to the
- * client library.
- * Metric data is not immediately sent to the metrics system
- * each time that update() is called.
- * An internal table is maintained, identified by the record name. This
- * table has columns
- * corresponding to the tag and the metric names, and rows
- * corresponding to each unique set of tag values. An update
- * either modifies an existing row in the table, or adds a new row with a set of
- * tag values that are different from all the other rows. Note that if there
- * are no tags, then there can be at most one row in the table.
- *
- * Once a row is added to the table, its data will be sent to the metrics system
- * on every timer period, whether or not it has been updated since the previous
- * timer period. If this is inappropriate, for example if metrics were being
- * reported by some transient object in an application, the remove()
- * method can be used to remove the row and thus stop the data from being
- * sent.
- *
- * Note that the update() method is atomic. This means that it is
- * safe for different threads to be updating the same metric. More precisely,
- * it is OK for different threads to call update() on MetricsRecord instances
- * with the same set of tag names and tag values. Different threads should
- * not use the same MetricsRecord instance at the same time.
- */
-@InterfaceAudience.Private
-@InterfaceStability.Evolving
-public interface MetricsRecord {
-
- /**
- * Returns the record name.
- *
- * @return the record name
- */
- public abstract String getRecordName();
-
- /**
- * Sets the named tag to the specified value. The tagValue may be null,
- * which is treated the same as an empty String.
- *
- * @param tagName name of the tag
- * @param tagValue new value of the tag
- * @throws MetricsException if the tagName conflicts with the configuration
- */
- public abstract void setTag(String tagName, String tagValue);
-
- /**
- * Sets the named tag to the specified value.
- *
- * @param tagName name of the tag
- * @param tagValue new value of the tag
- * @throws MetricsException if the tagName conflicts with the configuration
- */
- public abstract void setTag(String tagName, int tagValue);
-
- /**
- * Sets the named tag to the specified value.
- *
- * @param tagName name of the tag
- * @param tagValue new value of the tag
- * @throws MetricsException if the tagName conflicts with the configuration
- */
- public abstract void setTag(String tagName, long tagValue);
-
- /**
- * Sets the named tag to the specified value.
- *
- * @param tagName name of the tag
- * @param tagValue new value of the tag
- * @throws MetricsException if the tagName conflicts with the configuration
- */
- public abstract void setTag(String tagName, short tagValue);
-
- /**
- * Sets the named tag to the specified value.
- *
- * @param tagName name of the tag
- * @param tagValue new value of the tag
- * @throws MetricsException if the tagName conflicts with the configuration
- */
- public abstract void setTag(String tagName, byte tagValue);
-
- /**
- * Removes any tag of the specified name.
- *
- * @param tagName name of a tag
- */
- public abstract void removeTag(String tagName);
-
- /**
- * Sets the named metric to the specified value.
- *
- * @param metricName name of the metric
- * @param metricValue new value of the metric
- * @throws MetricsException if the metricName or the type of the metricValue
- * conflicts with the configuration
- */
- public abstract void setMetric(String metricName, int metricValue);
-
- /**
- * Sets the named metric to the specified value.
- *
- * @param metricName name of the metric
- * @param metricValue new value of the metric
- * @throws MetricsException if the metricName or the type of the metricValue
- * conflicts with the configuration
- */
- public abstract void setMetric(String metricName, long metricValue);
-
- /**
- * Sets the named metric to the specified value.
- *
- * @param metricName name of the metric
- * @param metricValue new value of the metric
- * @throws MetricsException if the metricName or the type of the metricValue
- * conflicts with the configuration
- */
- public abstract void setMetric(String metricName, short metricValue);
-
- /**
- * Sets the named metric to the specified value.
- *
- * @param metricName name of the metric
- * @param metricValue new value of the metric
- * @throws MetricsException if the metricName or the type of the metricValue
- * conflicts with the configuration
- */
- public abstract void setMetric(String metricName, byte metricValue);
-
- /**
- * Sets the named metric to the specified value.
- *
- * @param metricName name of the metric
- * @param metricValue new value of the metric
- * @throws MetricsException if the metricName or the type of the metricValue
- * conflicts with the configuration
- */
- public abstract void setMetric(String metricName, float metricValue);
-
- /**
- * Increments the named metric by the specified value.
- *
- * @param metricName name of the metric
- * @param metricValue incremental value
- * @throws MetricsException if the metricName or the type of the metricValue
- * conflicts with the configuration
- */
- public abstract void incrMetric(String metricName, int metricValue);
-
- /**
- * Increments the named metric by the specified value.
- *
- * @param metricName name of the metric
- * @param metricValue incremental value
- * @throws MetricsException if the metricName or the type of the metricValue
- * conflicts with the configuration
- */
- public abstract void incrMetric(String metricName, long metricValue);
-
- /**
- * Increments the named metric by the specified value.
- *
- * @param metricName name of the metric
- * @param metricValue incremental value
- * @throws MetricsException if the metricName or the type of the metricValue
- * conflicts with the configuration
- */
- public abstract void incrMetric(String metricName, short metricValue);
-
- /**
- * Increments the named metric by the specified value.
- *
- * @param metricName name of the metric
- * @param metricValue incremental value
- * @throws MetricsException if the metricName or the type of the metricValue
- * conflicts with the configuration
- */
- public abstract void incrMetric(String metricName, byte metricValue);
-
- /**
- * Increments the named metric by the specified value.
- *
- * @param metricName name of the metric
- * @param metricValue incremental value
- * @throws MetricsException if the metricName or the type of the metricValue
- * conflicts with the configuration
- */
- public abstract void incrMetric(String metricName, float metricValue);
-
- /**
- * Updates the table of buffered data which is to be sent periodically.
- * If the tag values match an existing row, that row is updated;
- * otherwise, a new row is added.
- */
- public abstract void update();
-
- /**
- * Removes, from the buffered data table, all rows having tags
- * that equal the tags that have been set on this record. For example,
- * if there are no tags on this record, all rows for this record name
- * would be removed. Or, if there is a single tag on this record, then
- * just rows containing a tag with the same name and value would be removed.
- */
- public abstract void remove();
-
-}
+/*
+ * MetricsRecord.java
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.metrics;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+
+/**
+ * A named and optionally tagged set of records to be sent to the metrics
+ * system.
+ *
+ * A record name identifies the kind of data to be reported. For example, a
+ * program reporting statistics relating to the disks on a computer might use
+ * a record name "diskStats".
+ *
+ * A record has zero or more tags. A tag has a name and a value. To
+ * continue the example, the "diskStats" record might use a tag named
+ * "diskName" to identify a particular disk. Sometimes it is useful to have
+ * more than one tag, so there might also be a "diskType" with value "ide" or
+ * "scsi" or whatever.
+ *
+ * A record also has zero or more metrics. These are the named
+ * values that are to be reported to the metrics system. In the "diskStats"
+ * example, possible metric names would be "diskPercentFull", "diskPercentBusy",
+ * "kbReadPerSecond", etc.
+ *
+ * The general procedure for using a MetricsRecord is to fill in its tag and
+ * metric values, and then call update() to pass the record to the
+ * client library.
+ * Metric data is not immediately sent to the metrics system
+ * each time that update() is called.
+ * An internal table is maintained, identified by the record name. This
+ * table has columns
+ * corresponding to the tag and the metric names, and rows
+ * corresponding to each unique set of tag values. An update
+ * either modifies an existing row in the table, or adds a new row with a set of
+ * tag values that are different from all the other rows. Note that if there
+ * are no tags, then there can be at most one row in the table.
+ *
+ * Once a row is added to the table, its data will be sent to the metrics system
+ * on every timer period, whether or not it has been updated since the previous
+ * timer period. If this is inappropriate, for example if metrics were being
+ * reported by some transient object in an application, the remove()
+ * method can be used to remove the row and thus stop the data from being
+ * sent.
+ *
+ * Note that the update() method is atomic. This means that it is
+ * safe for different threads to be updating the same metric. More precisely,
+ * it is OK for different threads to call update() on MetricsRecord instances
+ * with the same set of tag names and tag values. Different threads should
+ * not use the same MetricsRecord instance at the same time.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Evolving
+public interface MetricsRecord {
+
+ /**
+ * Returns the record name.
+ *
+ * @return the record name
+ */
+ public abstract String getRecordName();
+
+ /**
+ * Sets the named tag to the specified value. The tagValue may be null,
+ * which is treated the same as an empty String.
+ *
+ * @param tagName name of the tag
+ * @param tagValue new value of the tag
+ * @throws MetricsException if the tagName conflicts with the configuration
+ */
+ public abstract void setTag(String tagName, String tagValue);
+
+ /**
+ * Sets the named tag to the specified value.
+ *
+ * @param tagName name of the tag
+ * @param tagValue new value of the tag
+ * @throws MetricsException if the tagName conflicts with the configuration
+ */
+ public abstract void setTag(String tagName, int tagValue);
+
+ /**
+ * Sets the named tag to the specified value.
+ *
+ * @param tagName name of the tag
+ * @param tagValue new value of the tag
+ * @throws MetricsException if the tagName conflicts with the configuration
+ */
+ public abstract void setTag(String tagName, long tagValue);
+
+ /**
+ * Sets the named tag to the specified value.
+ *
+ * @param tagName name of the tag
+ * @param tagValue new value of the tag
+ * @throws MetricsException if the tagName conflicts with the configuration
+ */
+ public abstract void setTag(String tagName, short tagValue);
+
+ /**
+ * Sets the named tag to the specified value.
+ *
+ * @param tagName name of the tag
+ * @param tagValue new value of the tag
+ * @throws MetricsException if the tagName conflicts with the configuration
+ */
+ public abstract void setTag(String tagName, byte tagValue);
+
+ /**
+ * Removes any tag of the specified name.
+ *
+ * @param tagName name of a tag
+ */
+ public abstract void removeTag(String tagName);
+
+ /**
+ * Sets the named metric to the specified value.
+ *
+ * @param metricName name of the metric
+ * @param metricValue new value of the metric
+ * @throws MetricsException if the metricName or the type of the metricValue
+ * conflicts with the configuration
+ */
+ public abstract void setMetric(String metricName, int metricValue);
+
+ /**
+ * Sets the named metric to the specified value.
+ *
+ * @param metricName name of the metric
+ * @param metricValue new value of the metric
+ * @throws MetricsException if the metricName or the type of the metricValue
+ * conflicts with the configuration
+ */
+ public abstract void setMetric(String metricName, long metricValue);
+
+ /**
+ * Sets the named metric to the specified value.
+ *
+ * @param metricName name of the metric
+ * @param metricValue new value of the metric
+ * @throws MetricsException if the metricName or the type of the metricValue
+ * conflicts with the configuration
+ */
+ public abstract void setMetric(String metricName, short metricValue);
+
+ /**
+ * Sets the named metric to the specified value.
+ *
+ * @param metricName name of the metric
+ * @param metricValue new value of the metric
+ * @throws MetricsException if the metricName or the type of the metricValue
+ * conflicts with the configuration
+ */
+ public abstract void setMetric(String metricName, byte metricValue);
+
+ /**
+ * Sets the named metric to the specified value.
+ *
+ * @param metricName name of the metric
+ * @param metricValue new value of the metric
+ * @throws MetricsException if the metricName or the type of the metricValue
+ * conflicts with the configuration
+ */
+ public abstract void setMetric(String metricName, float metricValue);
+
+ /**
+ * Increments the named metric by the specified value.
+ *
+ * @param metricName name of the metric
+ * @param metricValue incremental value
+ * @throws MetricsException if the metricName or the type of the metricValue
+ * conflicts with the configuration
+ */
+ public abstract void incrMetric(String metricName, int metricValue);
+
+ /**
+ * Increments the named metric by the specified value.
+ *
+ * @param metricName name of the metric
+ * @param metricValue incremental value
+ * @throws MetricsException if the metricName or the type of the metricValue
+ * conflicts with the configuration
+ */
+ public abstract void incrMetric(String metricName, long metricValue);
+
+ /**
+ * Increments the named metric by the specified value.
+ *
+ * @param metricName name of the metric
+ * @param metricValue incremental value
+ * @throws MetricsException if the metricName or the type of the metricValue
+ * conflicts with the configuration
+ */
+ public abstract void incrMetric(String metricName, short metricValue);
+
+ /**
+ * Increments the named metric by the specified value.
+ *
+ * @param metricName name of the metric
+ * @param metricValue incremental value
+ * @throws MetricsException if the metricName or the type of the metricValue
+ * conflicts with the configuration
+ */
+ public abstract void incrMetric(String metricName, byte metricValue);
+
+ /**
+ * Increments the named metric by the specified value.
+ *
+ * @param metricName name of the metric
+ * @param metricValue incremental value
+ * @throws MetricsException if the metricName or the type of the metricValue
+ * conflicts with the configuration
+ */
+ public abstract void incrMetric(String metricName, float metricValue);
+
+ /**
+ * Updates the table of buffered data which is to be sent periodically.
+ * If the tag values match an existing row, that row is updated;
+ * otherwise, a new row is added.
+ */
+ public abstract void update();
+
+ /**
+ * Removes, from the buffered data table, all rows having tags
+ * that equal the tags that have been set on this record. For example,
+ * if there are no tags on this record, all rows for this record name
+ * would be removed. Or, if there is a single tag on this record, then
+ * just rows containing a tag with the same name and value would be removed.
+ */
+ public abstract void remove();
+
+}
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics/file/FileContext.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics/file/FileContext.java
index f0aafa1c3a3..591b32c4914 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics/file/FileContext.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics/file/FileContext.java
@@ -1,154 +1,154 @@
-/*
- * FileContext.java
- *
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hadoop.metrics.file;
-
-import java.io.BufferedOutputStream;
-import java.io.File;
-import java.io.FileWriter;
-import java.io.IOException;
-import java.io.PrintWriter;
-
-import org.apache.hadoop.classification.InterfaceAudience;
-import org.apache.hadoop.classification.InterfaceStability;
-import org.apache.hadoop.metrics.ContextFactory;
-import org.apache.hadoop.metrics.spi.AbstractMetricsContext;
-import org.apache.hadoop.metrics.spi.OutputRecord;
-
-/**
- * Metrics context for writing metrics to a file.
- *
- * This class is configured by setting ContextFactory attributes which in turn
- * are usually configured through a properties file. All the attributes are
- * prefixed by the contextName. For example, the properties file might contain:
- * - * myContextName.fileName=/tmp/metrics.log - * myContextName.period=5 - *- * @see org.apache.hadoop.metrics2.sink.FileSink for metrics 2.0. - */ -@InterfaceAudience.Public -@InterfaceStability.Evolving -@Deprecated -public class FileContext extends AbstractMetricsContext { - - /* Configuration attribute names */ - @InterfaceAudience.Private - protected static final String FILE_NAME_PROPERTY = "fileName"; - @InterfaceAudience.Private - protected static final String PERIOD_PROPERTY = "period"; - - private File file = null; // file for metrics to be written to - private PrintWriter writer = null; - - /** Creates a new instance of FileContext */ - @InterfaceAudience.Private - public FileContext() {} - - @InterfaceAudience.Private - public void init(String contextName, ContextFactory factory) { - super.init(contextName, factory); - - String fileName = getAttribute(FILE_NAME_PROPERTY); - if (fileName != null) { - file = new File(fileName); - } - - parseAndSetPeriod(PERIOD_PROPERTY); - } - - /** - * Returns the configured file name, or null. - */ - @InterfaceAudience.Private - public String getFileName() { - if (file == null) { - return null; - } else { - return file.getName(); - } - } - - /** - * Starts or restarts monitoring, by opening in append-mode, the - * file specified by the
fileName attribute,
- * if specified. Otherwise the data will be written to standard
- * output.
- */
- @InterfaceAudience.Private
- public void startMonitoring()
- throws IOException
- {
- if (file == null) {
- writer = new PrintWriter(new BufferedOutputStream(System.out));
- } else {
- writer = new PrintWriter(new FileWriter(file, true));
- }
- super.startMonitoring();
- }
-
- /**
- * Stops monitoring, closing the file.
- * @see #close()
- */
- @InterfaceAudience.Private
- public void stopMonitoring() {
- super.stopMonitoring();
-
- if (writer != null) {
- writer.close();
- writer = null;
- }
- }
-
- /**
- * Emits a metrics record to a file.
- */
- @InterfaceAudience.Private
- public void emitRecord(String contextName, String recordName, OutputRecord outRec) {
- writer.print(contextName);
- writer.print(".");
- writer.print(recordName);
- String separator = ": ";
- for (String tagName : outRec.getTagNames()) {
- writer.print(separator);
- separator = ", ";
- writer.print(tagName);
- writer.print("=");
- writer.print(outRec.getTag(tagName));
- }
- for (String metricName : outRec.getMetricNames()) {
- writer.print(separator);
- separator = ", ";
- writer.print(metricName);
- writer.print("=");
- writer.print(outRec.getMetric(metricName));
- }
- writer.println();
- }
-
- /**
- * Flushes the output writer, forcing updates to disk.
- */
- @InterfaceAudience.Private
- public void flush() {
- writer.flush();
- }
-}
+/*
+ * FileContext.java
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.metrics.file;
+
+import java.io.BufferedOutputStream;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.PrintWriter;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.metrics.ContextFactory;
+import org.apache.hadoop.metrics.spi.AbstractMetricsContext;
+import org.apache.hadoop.metrics.spi.OutputRecord;
+
+/**
+ * Metrics context for writing metrics to a file.
+ *
+ * This class is configured by setting ContextFactory attributes which in turn
+ * are usually configured through a properties file. All the attributes are
+ * prefixed by the contextName. For example, the properties file might contain:
+ * + * myContextName.fileName=/tmp/metrics.log + * myContextName.period=5 + *+ * @see org.apache.hadoop.metrics2.sink.FileSink for metrics 2.0. + */ +@InterfaceAudience.Public +@InterfaceStability.Evolving +@Deprecated +public class FileContext extends AbstractMetricsContext { + + /* Configuration attribute names */ + @InterfaceAudience.Private + protected static final String FILE_NAME_PROPERTY = "fileName"; + @InterfaceAudience.Private + protected static final String PERIOD_PROPERTY = "period"; + + private File file = null; // file for metrics to be written to + private PrintWriter writer = null; + + /** Creates a new instance of FileContext */ + @InterfaceAudience.Private + public FileContext() {} + + @InterfaceAudience.Private + public void init(String contextName, ContextFactory factory) { + super.init(contextName, factory); + + String fileName = getAttribute(FILE_NAME_PROPERTY); + if (fileName != null) { + file = new File(fileName); + } + + parseAndSetPeriod(PERIOD_PROPERTY); + } + + /** + * Returns the configured file name, or null. + */ + @InterfaceAudience.Private + public String getFileName() { + if (file == null) { + return null; + } else { + return file.getName(); + } + } + + /** + * Starts or restarts monitoring, by opening in append-mode, the + * file specified by the
fileName attribute,
+ * if specified. Otherwise the data will be written to standard
+ * output.
+ */
+ @InterfaceAudience.Private
+ public void startMonitoring()
+ throws IOException
+ {
+ if (file == null) {
+ writer = new PrintWriter(new BufferedOutputStream(System.out));
+ } else {
+ writer = new PrintWriter(new FileWriter(file, true));
+ }
+ super.startMonitoring();
+ }
+
+ /**
+ * Stops monitoring, closing the file.
+ * @see #close()
+ */
+ @InterfaceAudience.Private
+ public void stopMonitoring() {
+ super.stopMonitoring();
+
+ if (writer != null) {
+ writer.close();
+ writer = null;
+ }
+ }
+
+ /**
+ * Emits a metrics record to a file.
+ */
+ @InterfaceAudience.Private
+ public void emitRecord(String contextName, String recordName, OutputRecord outRec) {
+ writer.print(contextName);
+ writer.print(".");
+ writer.print(recordName);
+ String separator = ": ";
+ for (String tagName : outRec.getTagNames()) {
+ writer.print(separator);
+ separator = ", ";
+ writer.print(tagName);
+ writer.print("=");
+ writer.print(outRec.getTag(tagName));
+ }
+ for (String metricName : outRec.getMetricNames()) {
+ writer.print(separator);
+ separator = ", ";
+ writer.print(metricName);
+ writer.print("=");
+ writer.print(outRec.getMetric(metricName));
+ }
+ writer.println();
+ }
+
+ /**
+ * Flushes the output writer, forcing updates to disk.
+ */
+ @InterfaceAudience.Private
+ public void flush() {
+ writer.flush();
+ }
+}
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics/spi/AbstractMetricsContext.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics/spi/AbstractMetricsContext.java
index 947b0a12958..6e1e210e670 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics/spi/AbstractMetricsContext.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics/spi/AbstractMetricsContext.java
@@ -1,481 +1,481 @@
-/*
- * AbstractMetricsContext.java
- *
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hadoop.metrics.spi;
-
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-import java.util.Timer;
-import java.util.TimerTask;
-import java.util.TreeMap;
-import java.util.Map.Entry;
-
-import org.apache.hadoop.classification.InterfaceAudience;
-import org.apache.hadoop.classification.InterfaceStability;
-import org.apache.hadoop.metrics.ContextFactory;
-import org.apache.hadoop.metrics.MetricsContext;
-import org.apache.hadoop.metrics.MetricsException;
-import org.apache.hadoop.metrics.MetricsRecord;
-import org.apache.hadoop.metrics.Updater;
-
-/**
- * The main class of the Service Provider Interface. This class should be
- * extended in order to integrate the Metrics API with a specific metrics
- * client library.
- *
- * This class implements the internal table of metric data, and the timer
- * on which data is to be sent to the metrics system. Subclasses must
- * override the abstract emitRecord method in order to transmit
- * the data.
- */
-@InterfaceAudience.Public
-@InterfaceStability.Evolving
-public abstract class AbstractMetricsContext implements MetricsContext {
-
- private int period = MetricsContext.DEFAULT_PERIOD;
- private Timer timer = null;
-
- private SetrecordName.
- * Throws an exception if the metrics implementation is configured with a fixed
- * set of record names and recordName is not in that set.
- *
- * @param recordName the name of the record
- * @throws MetricsException if recordName conflicts with configuration data
- */
- public final synchronized MetricsRecord createRecord(String recordName) {
- if (bufferedData.get(recordName) == null) {
- bufferedData.put(recordName, new RecordMap());
- }
- return newRecord(recordName);
- }
-
- /**
- * Subclasses should override this if they subclass MetricsRecordImpl.
- * @param recordName the name of the record
- * @return newly created instance of MetricsRecordImpl or subclass
- */
- protected MetricsRecord newRecord(String recordName) {
- return new MetricsRecordImpl(recordName, this);
- }
-
- /**
- * Registers a callback to be called at time intervals determined by
- * the configuration.
- *
- * @param updater object to be run periodically; it should update
- * some metrics records
- */
- public synchronized void registerUpdater(final Updater updater) {
- if (!updaters.contains(updater)) {
- updaters.add(updater);
- }
- }
-
- /**
- * Removes a callback, if it exists.
- *
- * @param updater object to be removed from the callback list
- */
- public synchronized void unregisterUpdater(Updater updater) {
- updaters.remove(updater);
- }
-
- private synchronized void clearUpdaters() {
- updaters.clear();
- }
-
- /**
- * Starts timer if it is not already started
- */
- private synchronized void startTimer() {
- if (timer == null) {
- timer = new Timer("Timer thread for monitoring " + getContextName(),
- true);
- TimerTask task = new TimerTask() {
- public void run() {
- try {
- timerEvent();
- }
- catch (IOException ioe) {
- ioe.printStackTrace();
- }
- }
- };
- long millis = period * 1000;
- timer.scheduleAtFixedRate(task, millis, millis);
- }
- }
-
- /**
- * Stops timer if it is running
- */
- private synchronized void stopTimer() {
- if (timer != null) {
- timer.cancel();
- timer = null;
- }
- }
-
- /**
- * Timer callback.
- */
- private void timerEvent() throws IOException {
- if (isMonitoring) {
- CollectionemitRecord method in order to transmit
+ * the data.
+ */
+@InterfaceAudience.Public
+@InterfaceStability.Evolving
+public abstract class AbstractMetricsContext implements MetricsContext {
+
+ private int period = MetricsContext.DEFAULT_PERIOD;
+ private Timer timer = null;
+
+ private SetrecordName.
+ * Throws an exception if the metrics implementation is configured with a fixed
+ * set of record names and recordName is not in that set.
+ *
+ * @param recordName the name of the record
+ * @throws MetricsException if recordName conflicts with configuration data
+ */
+ public final synchronized MetricsRecord createRecord(String recordName) {
+ if (bufferedData.get(recordName) == null) {
+ bufferedData.put(recordName, new RecordMap());
+ }
+ return newRecord(recordName);
+ }
+
+ /**
+ * Subclasses should override this if they subclass MetricsRecordImpl.
+ * @param recordName the name of the record
+ * @return newly created instance of MetricsRecordImpl or subclass
+ */
+ protected MetricsRecord newRecord(String recordName) {
+ return new MetricsRecordImpl(recordName, this);
+ }
+
+ /**
+ * Registers a callback to be called at time intervals determined by
+ * the configuration.
+ *
+ * @param updater object to be run periodically; it should update
+ * some metrics records
+ */
+ public synchronized void registerUpdater(final Updater updater) {
+ if (!updaters.contains(updater)) {
+ updaters.add(updater);
+ }
+ }
+
+ /**
+ * Removes a callback, if it exists.
+ *
+ * @param updater object to be removed from the callback list
+ */
+ public synchronized void unregisterUpdater(Updater updater) {
+ updaters.remove(updater);
+ }
+
+ private synchronized void clearUpdaters() {
+ updaters.clear();
+ }
+
+ /**
+ * Starts timer if it is not already started
+ */
+ private synchronized void startTimer() {
+ if (timer == null) {
+ timer = new Timer("Timer thread for monitoring " + getContextName(),
+ true);
+ TimerTask task = new TimerTask() {
+ public void run() {
+ try {
+ timerEvent();
+ }
+ catch (IOException ioe) {
+ ioe.printStackTrace();
+ }
+ }
+ };
+ long millis = period * 1000;
+ timer.scheduleAtFixedRate(task, millis, millis);
+ }
+ }
+
+ /**
+ * Stops timer if it is running
+ */
+ private synchronized void stopTimer() {
+ if (timer != null) {
+ timer.cancel();
+ timer = null;
+ }
+ }
+
+ /**
+ * Timer callback.
+ */
+ private void timerEvent() throws IOException {
+ if (isMonitoring) {
+ Collectionupdate
- * and remove().
- */
-@InterfaceAudience.Public
-@InterfaceStability.Evolving
-public class MetricsRecordImpl implements MetricsRecord {
-
- private TagMap tagTable = new TagMap();
- private Mapupdate
+ * and remove().
+ */
+@InterfaceAudience.Public
+@InterfaceStability.Evolving
+public class MetricsRecordImpl implements MetricsRecord {
+
+ private TagMap tagTable = new TagMap();
+ private Map-libhdfs is a JNI based C API for Hadoop's Distributed File System (HDFS). -It provides C APIs to a subset of the HDFS APIs to manipulate HDFS files and -the filesystem. libhdfs is part of the Hadoop distribution and comes -pre-compiled in ${HADOOP_PREFIX}/libhdfs/libhdfs.so . -
- --The libhdfs APIs are a subset of: hadoop fs APIs. -
--The header file for libhdfs describes each API in detail and is available in ${HADOOP_PREFIX}/src/c++/libhdfs/hdfs.h -
-
-See the Makefile for hdfs_test.c in the libhdfs source directory (${HADOOP_PREFIX}/src/c++/libhdfs/Makefile) or something like:
-gcc above_sample.c -I${HADOOP_PREFIX}/src/c++/libhdfs -L${HADOOP_PREFIX}/libhdfs -lhdfs -o above_sample
-
-The most common problem is the CLASSPATH is not set properly when calling a program that uses libhdfs. -Make sure you set it to all the Hadoop jars needed to run Hadoop itself. Currently, there is no way to -programmatically generate the classpath, but a good bet is to include all the jar files in ${HADOOP_PREFIX} -and ${HADOOP_PREFIX}/lib as well as the right configuration directory containing hdfs-site.xml -
-libdhfs is thread safe.
-+libhdfs is a JNI based C API for Hadoop's Distributed File System (HDFS). +It provides C APIs to a subset of the HDFS APIs to manipulate HDFS files and +the filesystem. libhdfs is part of the Hadoop distribution and comes +pre-compiled in ${HADOOP_PREFIX}/libhdfs/libhdfs.so . +
+ ++The libhdfs APIs are a subset of: hadoop fs APIs. +
++The header file for libhdfs describes each API in detail and is available in ${HADOOP_PREFIX}/src/c++/libhdfs/hdfs.h +
+
+See the Makefile for hdfs_test.c in the libhdfs source directory (${HADOOP_PREFIX}/src/c++/libhdfs/Makefile) or something like:
+gcc above_sample.c -I${HADOOP_PREFIX}/src/c++/libhdfs -L${HADOOP_PREFIX}/libhdfs -lhdfs -o above_sample
+
+The most common problem is the CLASSPATH is not set properly when calling a program that uses libhdfs. +Make sure you set it to all the Hadoop jars needed to run Hadoop itself. Currently, there is no way to +programmatically generate the classpath, but a good bet is to include all the jar files in ${HADOOP_PREFIX} +and ${HADOOP_PREFIX}/lib as well as the right configuration directory containing hdfs-site.xml +
+libdhfs is thread safe.
+