YARN-7323. Data structure update in service REST API. Contributed by Jian He
This commit is contained in:
Billie Rinaldi
Jian He
parent
3c30b1a97d
commit
68acd88dcb
|
|
@ -242,15 +242,6 @@ public Response updateService(@PathParam(SERVICE_NAME) String appName,
|
|||
return updateLifetime(appName, updateServiceData);
|
||||
}
|
||||
|
||||
// flex a single component app
|
||||
if (updateServiceData.getNumberOfContainers() != null && !ServiceApiUtil
|
||||
.hasComponent(updateServiceData)) {
|
||||
Component defaultComp = ServiceApiUtil
|
||||
.createDefaultComponent(updateServiceData);
|
||||
return updateComponent(updateServiceData.getName(), defaultComp.getName(),
|
||||
defaultComp);
|
||||
}
|
||||
|
||||
// If nothing happens consider it a no-op
|
||||
return Response.status(Status.NO_CONTENT).build();
|
||||
}
|
||||
|
|
|
|||
|
|
@ -19,28 +19,17 @@ swagger: '2.0'
|
|||
info:
|
||||
title: "YARN Simplified API layer for services"
|
||||
description: |
|
||||
Bringing a new service on YARN today is not a simple experience. The APIs of
|
||||
existing frameworks are either too low level (native YARN), require writing
|
||||
new code (for frameworks with programmatic APIs) or writing a complex spec
|
||||
(for declarative frameworks). In addition to building critical building blocks
|
||||
inside YARN (as part of other efforts at YARN-4692), there is a need for
|
||||
simplifying the user facing story for building services. Experience of projects
|
||||
like Apache Slider running real-life services like HBase, Storm, Accumulo,
|
||||
Solr etc, gives us some very good insights on how simplified APIs for services
|
||||
should look like.
|
||||
Bringing a new service on YARN today is not a simple experience. The APIs of existing
|
||||
frameworks are either too low level (native YARN), require writing new code (for frameworks with programmatic APIs)
|
||||
or writing a complex spec (for declarative frameworks).
|
||||
|
||||
To this end, we should look at a new simple-services API layer backed by REST
|
||||
interfaces. This API can be used to create and manage the lifecycle of YARN
|
||||
services. Services here can range from simple single-component service to
|
||||
complex multi-component assemblies needing orchestration. YARN-4793 tracks
|
||||
this effort.
|
||||
This simplified REST API can be used to create and manage the lifecycle of YARN services.
|
||||
In most cases, the application owner will not be forced to make any changes to their applications.
|
||||
This is primarily true if the application is packaged with containerization technologies like Docker.
|
||||
|
||||
This document spotlights on this specification. In most of the cases, the
|
||||
application owner will not be forced to make any changes to their applications.
|
||||
This is primarily true if the application is packaged with containerization
|
||||
technologies like docker. Irrespective of how complex the application is,
|
||||
there will be hooks provided at appropriate layers to allow pluggable and
|
||||
customizable application behavior.
|
||||
This document describes the API specifications (aka. YarnFile) for deploying/managing
|
||||
containerized services on YARN. The same JSON spec can be used for both REST API
|
||||
and CLI to manage the services.
|
||||
|
||||
version: "1.0.0"
|
||||
license:
|
||||
|
|
@ -216,22 +205,15 @@ definitions:
|
|||
type: string
|
||||
description: A unique service id.
|
||||
artifact:
|
||||
description: Artifact of single-component service.
|
||||
description: The default artifact for all components of the service except the components which has Artifact type set to SERVICE (optional).
|
||||
$ref: '#/definitions/Artifact'
|
||||
resource:
|
||||
description: Resource of single-component service or the global default for multi-component services. Mandatory if it is a single-component service and if cpus and memory are not specified at the Service level.
|
||||
description: The default resource for all components of the service (optional).
|
||||
$ref: '#/definitions/Resource'
|
||||
launch_command:
|
||||
type: string
|
||||
description: The custom launch command of a service component (optional). If not specified for services with docker images say, it will default to the default start command of the image. If there is a single component in this service, you can specify this without the need to have a 'components' section.
|
||||
launch_time:
|
||||
type: string
|
||||
format: date
|
||||
description: The time when the service was created, e.g. 2016-03-16T01:01:49.000Z.
|
||||
number_of_containers:
|
||||
type: integer
|
||||
format: int64
|
||||
description: Number of containers for each component in the service. Each component can further override this service-level global default.
|
||||
number_of_running_containers:
|
||||
type: integer
|
||||
format: int64
|
||||
|
|
@ -251,13 +233,8 @@ definitions:
|
|||
configuration:
|
||||
description: Config properties of a service. Configurations provided at the service/global level are available to all the components. Specific properties can be overridden at the component level.
|
||||
$ref: '#/definitions/Configuration'
|
||||
containers:
|
||||
description: Containers of a started service. Specifying a value for this attribute for the POST payload raises a validation error. This blob is available only in the GET response of a started service.
|
||||
type: array
|
||||
items:
|
||||
$ref: '#/definitions/Container'
|
||||
state:
|
||||
description: State of the service. Specifying a value for this attribute for the POST payload raises a validation error. This attribute is available only in the GET response of a started service.
|
||||
description: State of the service. Specifying a value for this attribute for the PUT payload means update the service to this desired state.
|
||||
$ref: '#/definitions/ServiceState'
|
||||
quicklinks:
|
||||
type: object
|
||||
|
|
@ -314,6 +291,9 @@ definitions:
|
|||
name:
|
||||
type: string
|
||||
description: Name of the service component (mandatory). If Registry DNS is enabled, the max length is 63 characters. If unique component support is enabled, the max length is lowered to 44 characters.
|
||||
state:
|
||||
description: The state of the component
|
||||
$ref: "#/definitions/ComponentState"
|
||||
dependencies:
|
||||
type: array
|
||||
items:
|
||||
|
|
@ -431,9 +411,11 @@ definitions:
|
|||
state:
|
||||
description: State of the container of a service.
|
||||
$ref: '#/definitions/ContainerState'
|
||||
component_name:
|
||||
component_instance_name:
|
||||
type: string
|
||||
description: Name of the component that this container instance belongs to.
|
||||
description: Name of the component instance that this container instance belongs to. Component instance name is named as $COMPONENT_NAME-i, where i is a
|
||||
monotonically increasing integer. E.g. A componet called nginx can have multiple component instances named as nginx-0, nginx-1 etc.
|
||||
Each component instance is backed by a container instance.
|
||||
resource:
|
||||
description: Resource used for this container.
|
||||
$ref: '#/definitions/Resource'
|
||||
|
|
@ -452,7 +434,7 @@ definitions:
|
|||
enum:
|
||||
- ACCEPTED
|
||||
- STARTED
|
||||
- READY
|
||||
- STABLE
|
||||
- STOPPED
|
||||
- FAILED
|
||||
ContainerState:
|
||||
|
|
@ -465,6 +447,15 @@ definitions:
|
|||
- INIT
|
||||
- STARTED
|
||||
- READY
|
||||
ComponentState:
|
||||
description: The state of the component
|
||||
properties:
|
||||
state:
|
||||
type: string
|
||||
description: enum of the state of the component
|
||||
enum:
|
||||
- FLEXING
|
||||
- STABLE
|
||||
ServiceStatus:
|
||||
description: The current status of a submitted service, returned as a response to the GET API.
|
||||
properties:
|
||||
|
|
|
|||
|
|
@ -604,6 +604,7 @@ public void onStartContainerError(ContainerId containerId, Throwable t) {
|
|||
LOG.error("No component instance exists for " + containerId);
|
||||
return;
|
||||
}
|
||||
LOG.error("Failed to start " + containerId, t);
|
||||
amRMClient.releaseAssignedContainer(containerId);
|
||||
// After container released, it'll get CONTAINER_COMPLETED event from RM
|
||||
// automatically which will trigger stopping COMPONENT INSTANCE
|
||||
|
|
|
|||
|
|
@ -59,6 +59,7 @@ public class Component implements Serializable {
|
|||
private Long numberOfContainers = null;
|
||||
private Boolean runPrivilegedContainer = false;
|
||||
private PlacementPolicy placementPolicy = null;
|
||||
private ComponentState state = ComponentState.FLEXING;
|
||||
private Configuration configuration = new Configuration();
|
||||
private List<String> quicklinks = new ArrayList<String>();
|
||||
private List<Container> containers =
|
||||
|
|
@ -305,6 +306,21 @@ public void setQuicklinks(List<String> quicklinks) {
|
|||
this.quicklinks = quicklinks;
|
||||
}
|
||||
|
||||
public Component state(ComponentState state) {
|
||||
this.state = state;
|
||||
return this;
|
||||
}
|
||||
|
||||
@ApiModelProperty(example = "null", value = "State of the component.")
|
||||
@JsonProperty("state")
|
||||
public ComponentState getState() {
|
||||
return state;
|
||||
}
|
||||
|
||||
public void setState(ComponentState state) {
|
||||
this.state = state;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean equals(java.lang.Object o) {
|
||||
if (this == o) {
|
||||
|
|
@ -325,14 +341,15 @@ public boolean equals(java.lang.Object o) {
|
|||
component.runPrivilegedContainer)
|
||||
&& Objects.equals(this.placementPolicy, component.placementPolicy)
|
||||
&& Objects.equals(this.configuration, component.configuration)
|
||||
&& Objects.equals(this.quicklinks, component.quicklinks);
|
||||
&& Objects.equals(this.quicklinks, component.quicklinks)
|
||||
&& Objects.equals(this.state, component.state);
|
||||
}
|
||||
|
||||
@Override
|
||||
public int hashCode() {
|
||||
return Objects.hash(name, dependencies, readinessCheck, artifact,
|
||||
launchCommand, resource, numberOfContainers,
|
||||
runPrivilegedContainer, placementPolicy, configuration, quicklinks);
|
||||
runPrivilegedContainer, placementPolicy, configuration, quicklinks, state);
|
||||
}
|
||||
|
||||
@Override
|
||||
|
|
@ -341,6 +358,7 @@ public String toString() {
|
|||
sb.append("class Component {\n");
|
||||
|
||||
sb.append(" name: ").append(toIndentedString(name)).append("\n");
|
||||
sb.append(" state: ").append(toIndentedString(state)).append("\n");
|
||||
sb.append(" dependencies: ").append(toIndentedString(dependencies))
|
||||
.append("\n");
|
||||
sb.append(" readinessCheck: ").append(toIndentedString(readinessCheck))
|
||||
|
|
|
|||
|
|
@ -0,0 +1,30 @@
|
|||
/*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
|
||||
package org.apache.hadoop.yarn.service.api.records;
|
||||
|
||||
import io.swagger.annotations.ApiModel;
|
||||
import org.apache.hadoop.classification.InterfaceAudience;
|
||||
import org.apache.hadoop.classification.InterfaceStability;
|
||||
|
||||
@InterfaceAudience.Public
|
||||
@InterfaceStability.Unstable
|
||||
@ApiModel(description = "The current state of a component.")
|
||||
public enum ComponentState {
|
||||
FLEXING, STABLE
|
||||
}
|
||||
|
|
@ -49,7 +49,7 @@ public class Container extends BaseResource {
|
|||
private String hostname = null;
|
||||
private String bareHost = null;
|
||||
private ContainerState state = null;
|
||||
private String componentName = null;
|
||||
private String componentInstanceName = null;
|
||||
private Resource resource = null;
|
||||
private Artifact artifact = null;
|
||||
private Boolean privilegedContainer = null;
|
||||
|
|
@ -176,19 +176,19 @@ public void setState(ContainerState state) {
|
|||
* Name of the component that this container instance belongs to.
|
||||
**/
|
||||
public Container componentName(String componentName) {
|
||||
this.componentName = componentName;
|
||||
this.componentInstanceName = componentName;
|
||||
return this;
|
||||
}
|
||||
|
||||
@ApiModelProperty(example = "null", value = "Name of the component that this container instance belongs to.")
|
||||
@JsonProperty("component_name")
|
||||
public String getComponentName() {
|
||||
return componentName;
|
||||
public String getComponentInstanceName() {
|
||||
return componentInstanceName;
|
||||
}
|
||||
|
||||
@XmlElement(name = "component_name")
|
||||
public void setComponentName(String componentName) {
|
||||
this.componentName = componentName;
|
||||
public void setComponentInstanceName(String componentInstanceName) {
|
||||
this.componentInstanceName = componentInstanceName;
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -274,7 +274,8 @@ public String toString() {
|
|||
sb.append(" hostname: ").append(toIndentedString(hostname)).append("\n");
|
||||
sb.append(" bareHost: ").append(toIndentedString(bareHost)).append("\n");
|
||||
sb.append(" state: ").append(toIndentedString(state)).append("\n");
|
||||
sb.append(" componentName: ").append(toIndentedString(componentName))
|
||||
sb.append(" componentInstanceName: ").append(toIndentedString(
|
||||
componentInstanceName))
|
||||
.append("\n");
|
||||
sb.append(" resource: ").append(toIndentedString(resource)).append("\n");
|
||||
sb.append(" artifact: ").append(toIndentedString(artifact)).append("\n");
|
||||
|
|
|
|||
|
|
@ -53,15 +53,12 @@ public class Service extends BaseResource {
|
|||
private String id = null;
|
||||
private Artifact artifact = null;
|
||||
private Resource resource = null;
|
||||
private String launchCommand = null;
|
||||
private Date launchTime = null;
|
||||
private Long numberOfContainers = null;
|
||||
private Long numberOfRunningContainers = null;
|
||||
private Long lifetime = null;
|
||||
private PlacementPolicy placementPolicy = null;
|
||||
private List<Component> components = new ArrayList<>();
|
||||
private Configuration configuration = new Configuration();
|
||||
private List<Container> containers = new ArrayList<>();
|
||||
private ServiceState state = null;
|
||||
private Map<String, String> quicklinks = new HashMap<>();
|
||||
private String queue = null;
|
||||
|
|
@ -142,29 +139,6 @@ public void setResource(Resource resource) {
|
|||
this.resource = resource;
|
||||
}
|
||||
|
||||
/**
|
||||
* The custom launch command of an service component (optional). If not
|
||||
* specified for services with docker images say, it will default to the
|
||||
* default start command of the image. If there is a single component in this
|
||||
* service, you can specify this without the need to have a 'components'
|
||||
* section.
|
||||
**/
|
||||
public Service launchCommand(String launchCommand) {
|
||||
this.launchCommand = launchCommand;
|
||||
return this;
|
||||
}
|
||||
|
||||
@ApiModelProperty(example = "null", value = "The custom launch command of an service component (optional). If not specified for services with docker images say, it will default to the default start command of the image. If there is a single component in this service, you can specify this without the need to have a 'components' section.")
|
||||
@JsonProperty("launch_command")
|
||||
public String getLaunchCommand() {
|
||||
return launchCommand;
|
||||
}
|
||||
|
||||
@XmlElement(name = "launch_command")
|
||||
public void setLaunchCommand(String launchCommand) {
|
||||
this.launchCommand = launchCommand;
|
||||
}
|
||||
|
||||
/**
|
||||
* The time when the service was created, e.g. 2016-03-16T01:01:49.000Z.
|
||||
**/
|
||||
|
|
@ -184,26 +158,6 @@ public void setLaunchTime(Date launchTime) {
|
|||
this.launchTime = launchTime == null ? null : (Date) launchTime.clone();
|
||||
}
|
||||
|
||||
/**
|
||||
* Number of containers for each component in the service. Each
|
||||
* component can further override this service-level global default.
|
||||
**/
|
||||
public Service numberOfContainers(Long numberOfContainers) {
|
||||
this.numberOfContainers = numberOfContainers;
|
||||
return this;
|
||||
}
|
||||
|
||||
@ApiModelProperty(example = "null", value = "Number of containers for each component in the service. Each component can further override this service-level global default.")
|
||||
@JsonProperty("number_of_containers")
|
||||
public Long getNumberOfContainers() {
|
||||
return numberOfContainers;
|
||||
}
|
||||
|
||||
@XmlElement(name = "number_of_containers")
|
||||
public void setNumberOfContainers(Long numberOfContainers) {
|
||||
this.numberOfContainers = numberOfContainers;
|
||||
}
|
||||
|
||||
/**
|
||||
* In get response this provides the total number of running containers for
|
||||
* this service (across all components) at the time of request. Note, a
|
||||
|
|
@ -322,30 +276,6 @@ public void setConfiguration(Configuration configuration) {
|
|||
this.configuration = configuration;
|
||||
}
|
||||
|
||||
/**
|
||||
* Containers of a started service. Specifying a value for this attribute
|
||||
* for the POST payload raises a validation error. This blob is available only
|
||||
* in the GET response of a started service.
|
||||
**/
|
||||
public Service containers(List<Container> containers) {
|
||||
this.containers = containers;
|
||||
return this;
|
||||
}
|
||||
|
||||
@ApiModelProperty(example = "null", value = "Containers of a started service. Specifying a value for this attribute for the POST payload raises a validation error. This blob is available only in the GET response of a started service.")
|
||||
@JsonProperty("containers")
|
||||
public List<Container> getContainers() {
|
||||
return containers;
|
||||
}
|
||||
|
||||
public void setContainers(List<Container> containers) {
|
||||
this.containers = containers;
|
||||
}
|
||||
|
||||
public void addContainer(Container container) {
|
||||
this.containers.add(container);
|
||||
}
|
||||
|
||||
/**
|
||||
* State of the service. Specifying a value for this attribute for the
|
||||
* POST payload raises a validation error. This attribute is available only in
|
||||
|
|
@ -428,12 +358,8 @@ public String toString() {
|
|||
sb.append(" id: ").append(toIndentedString(id)).append("\n");
|
||||
sb.append(" artifact: ").append(toIndentedString(artifact)).append("\n");
|
||||
sb.append(" resource: ").append(toIndentedString(resource)).append("\n");
|
||||
sb.append(" launchCommand: ").append(toIndentedString(launchCommand))
|
||||
.append("\n");
|
||||
sb.append(" launchTime: ").append(toIndentedString(launchTime))
|
||||
.append("\n");
|
||||
sb.append(" numberOfContainers: ")
|
||||
.append(toIndentedString(numberOfContainers)).append("\n");
|
||||
sb.append(" numberOfRunningContainers: ")
|
||||
.append(toIndentedString(numberOfRunningContainers)).append("\n");
|
||||
sb.append(" lifetime: ").append(toIndentedString(lifetime)).append("\n");
|
||||
|
|
@ -443,8 +369,6 @@ public String toString() {
|
|||
.append("\n");
|
||||
sb.append(" configuration: ").append(toIndentedString(configuration))
|
||||
.append("\n");
|
||||
sb.append(" containers: ").append(toIndentedString(containers))
|
||||
.append("\n");
|
||||
sb.append(" state: ").append(toIndentedString(state)).append("\n");
|
||||
sb.append(" quicklinks: ").append(toIndentedString(quicklinks))
|
||||
.append("\n");
|
||||
|
|
|
|||
|
|
@ -29,5 +29,5 @@
|
|||
@ApiModel(description = "The current state of an service.")
|
||||
@javax.annotation.Generated(value = "class io.swagger.codegen.languages.JavaClientCodegen", date = "2016-06-02T08:15:05.615-07:00")
|
||||
public enum ServiceState {
|
||||
ACCEPTED, STARTED, READY, STOPPED, FAILED;
|
||||
ACCEPTED, STARTED, STABLE, STOPPED, FAILED;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -62,8 +62,8 @@
|
|||
import org.apache.hadoop.yarn.proto.ClientAMProtocol.StopRequestProto;
|
||||
import org.apache.hadoop.yarn.service.ClientAMProtocol;
|
||||
import org.apache.hadoop.yarn.service.ServiceMaster;
|
||||
import org.apache.hadoop.yarn.service.api.records.Service;
|
||||
import org.apache.hadoop.yarn.service.api.records.Component;
|
||||
import org.apache.hadoop.yarn.service.api.records.Service;
|
||||
import org.apache.hadoop.yarn.service.api.records.ServiceState;
|
||||
import org.apache.hadoop.yarn.service.client.params.AbstractClusterBuildingActionArgs;
|
||||
import org.apache.hadoop.yarn.service.client.params.ActionCreateArgs;
|
||||
|
|
@ -73,23 +73,23 @@
|
|||
import org.apache.hadoop.yarn.service.client.params.ClientArgs;
|
||||
import org.apache.hadoop.yarn.service.client.params.CommonArgs;
|
||||
import org.apache.hadoop.yarn.service.conf.SliderExitCodes;
|
||||
import org.apache.hadoop.yarn.service.conf.YarnServiceConstants;
|
||||
import org.apache.hadoop.yarn.service.conf.YarnServiceConf;
|
||||
import org.apache.hadoop.yarn.service.conf.YarnServiceConstants;
|
||||
import org.apache.hadoop.yarn.service.containerlaunch.ClasspathConstructor;
|
||||
import org.apache.hadoop.yarn.service.containerlaunch.JavaCommandLineBuilder;
|
||||
import org.apache.hadoop.yarn.service.exceptions.BadClusterStateException;
|
||||
import org.apache.hadoop.yarn.service.exceptions.BadConfigException;
|
||||
import org.apache.hadoop.yarn.service.exceptions.SliderException;
|
||||
import org.apache.hadoop.yarn.service.exceptions.UsageException;
|
||||
import org.apache.hadoop.yarn.service.provider.AbstractClientProvider;
|
||||
import org.apache.hadoop.yarn.service.provider.ProviderUtils;
|
||||
import org.apache.hadoop.yarn.service.utils.ServiceApiUtil;
|
||||
import org.apache.hadoop.yarn.service.utils.ServiceRegistryUtils;
|
||||
import org.apache.hadoop.yarn.service.utils.SliderFileSystem;
|
||||
import org.apache.hadoop.yarn.service.utils.SliderUtils;
|
||||
import org.apache.hadoop.yarn.service.utils.ZookeeperUtils;
|
||||
import org.apache.hadoop.yarn.util.Records;
|
||||
import org.apache.hadoop.yarn.util.Times;
|
||||
import org.apache.hadoop.yarn.service.exceptions.BadClusterStateException;
|
||||
import org.apache.hadoop.yarn.service.exceptions.BadConfigException;
|
||||
import org.apache.hadoop.yarn.service.exceptions.SliderException;
|
||||
import org.apache.hadoop.yarn.service.exceptions.UsageException;
|
||||
import org.apache.hadoop.yarn.service.containerlaunch.ClasspathConstructor;
|
||||
import org.apache.hadoop.yarn.service.containerlaunch.JavaCommandLineBuilder;
|
||||
import org.apache.hadoop.yarn.service.utils.ZookeeperUtils;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
|
|
@ -196,11 +196,7 @@ public int actionCreate(ActionCreateArgs args)
|
|||
serviceDef = loadAppJsonFromLocalFS(args);
|
||||
} else if (!StringUtils.isEmpty(args.example)) {
|
||||
// create an example service
|
||||
String yarnHome = System
|
||||
.getenv(ApplicationConstants.Environment.HADOOP_YARN_HOME.key());
|
||||
args.file = new File(MessageFormat
|
||||
.format("{0}/share/hadoop/yarn/yarn-service-examples/{1}/{2}.json",
|
||||
yarnHome, args.example, args.example));
|
||||
args.file = findExampleService(args);
|
||||
serviceDef = loadAppJsonFromLocalFS(args);
|
||||
} else {
|
||||
throw new YarnException("No service definition provided!");
|
||||
|
|
@ -209,6 +205,27 @@ public int actionCreate(ActionCreateArgs args)
|
|||
return EXIT_SUCCESS;
|
||||
}
|
||||
|
||||
private File findExampleService(ActionCreateArgs args) throws YarnException {
|
||||
String yarnHome = System
|
||||
.getenv(ApplicationConstants.Environment.HADOOP_YARN_HOME.key());
|
||||
// First look for the standard location.
|
||||
File file = new File(MessageFormat
|
||||
.format("{0}/share/hadoop/yarn/yarn-service-examples/{1}/{2}.json",
|
||||
yarnHome, args.example, args.example));
|
||||
if (file.exists()) {
|
||||
return file;
|
||||
}
|
||||
// Then look for secondary location.
|
||||
file = new File(MessageFormat
|
||||
.format("{0}/yarn-service-examples/{1}/{2}.json", yarnHome,
|
||||
args.example, args.example));
|
||||
if (file.exists()) {
|
||||
return file;
|
||||
}
|
||||
throw new YarnException(
|
||||
"Example service " + args.example + " does not exist!");
|
||||
}
|
||||
|
||||
public ApplicationId actionCreate(Service service)
|
||||
throws IOException, YarnException {
|
||||
String serviceName = service.getName();
|
||||
|
|
|
|||
|
|
@ -202,6 +202,8 @@ public ComponentState transition(Component component,
|
|||
+ before + " to " + event.getDesired());
|
||||
component.requestContainers(delta);
|
||||
component.createNumCompInstances(delta);
|
||||
component.componentSpec.setState(
|
||||
org.apache.hadoop.yarn.service.api.records.ComponentState.FLEXING);
|
||||
return FLEXING;
|
||||
} else if (delta < 0){
|
||||
delta = 0 - delta;
|
||||
|
|
@ -224,10 +226,14 @@ public ComponentState transition(Component component,
|
|||
component.instanceIdCounter.decrementAndGet();
|
||||
instance.destroy();
|
||||
}
|
||||
component.componentSpec.setState(
|
||||
org.apache.hadoop.yarn.service.api.records.ComponentState.STABLE);
|
||||
return STABLE;
|
||||
} else {
|
||||
LOG.info("[FLEX COMPONENT " + component.getName() + "]: already has " +
|
||||
event.getDesired() + " instances, ignoring");
|
||||
component.componentSpec.setState(
|
||||
org.apache.hadoop.yarn.service.api.records.ComponentState.STABLE);
|
||||
return STABLE;
|
||||
}
|
||||
}
|
||||
|
|
@ -297,8 +303,12 @@ private static ComponentState checkIfStable(Component component) {
|
|||
// if desired == running
|
||||
if (component.componentMetrics.containersRunning.value() == component
|
||||
.getComponentSpec().getNumberOfContainers()) {
|
||||
component.componentSpec.setState(
|
||||
org.apache.hadoop.yarn.service.api.records.ComponentState.STABLE);
|
||||
return STABLE;
|
||||
} else {
|
||||
component.componentSpec.setState(
|
||||
org.apache.hadoop.yarn.service.api.records.ComponentState.FLEXING);
|
||||
return FLEXING;
|
||||
}
|
||||
}
|
||||
|
|
@ -317,6 +327,8 @@ public void transition(Component component, ComponentEvent event) {
|
|||
component.compInstanceDispatcher.getEventHandler().handle(
|
||||
new ComponentInstanceEvent(event.getStatus().getContainerId(),
|
||||
STOP).setStatus(event.getStatus()));
|
||||
component.componentSpec.setState(
|
||||
org.apache.hadoop.yarn.service.api.records.ComponentState.FLEXING);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -99,6 +99,11 @@ public class ComponentInstance implements EventHandler<ComponentInstanceEvent>,
|
|||
ComponentInstanceEventType, ComponentInstanceEvent>(INIT)
|
||||
.addTransition(INIT, STARTED, START,
|
||||
new ContainerStartedTransition())
|
||||
.addTransition(INIT, INIT, STOP,
|
||||
// container failed before launching, nothing to cleanup from registry
|
||||
// This could happen if NMClient#startContainerAsync failed, container
|
||||
// will be completed, but COMP_INSTANCE is still at INIT.
|
||||
new ContainerStoppedTransition(true))
|
||||
|
||||
//From Running
|
||||
.addTransition(STARTED, INIT, STOP,
|
||||
|
|
@ -159,7 +164,7 @@ private static class ContainerStartedTransition extends BaseTransition {
|
|||
container.setLaunchTime(new Date(containerStartTime));
|
||||
container.setState(ContainerState.RUNNING_BUT_UNREADY);
|
||||
container.setBareHost(compInstance.container.getNodeId().getHost());
|
||||
container.setComponentName(compInstance.getCompInstanceName());
|
||||
container.setComponentInstanceName(compInstance.getCompInstanceName());
|
||||
if (compInstance.containerSpec != null) {
|
||||
// remove the previous container.
|
||||
compInstance.getCompSpec().removeContainer(compInstance.containerSpec);
|
||||
|
|
@ -194,6 +199,16 @@ public void transition(ComponentInstance compInstance,
|
|||
}
|
||||
|
||||
private static class ContainerStoppedTransition extends BaseTransition {
|
||||
// whether the container failed before launched by AM or not.
|
||||
boolean failedBeforeLaunching = false;
|
||||
public ContainerStoppedTransition(boolean failedBeforeLaunching) {
|
||||
this.failedBeforeLaunching = failedBeforeLaunching;
|
||||
}
|
||||
|
||||
public ContainerStoppedTransition() {
|
||||
this(false);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void transition(ComponentInstance compInstance,
|
||||
ComponentInstanceEvent event) {
|
||||
|
|
@ -225,24 +240,27 @@ public void transition(ComponentInstance compInstance,
|
|||
shouldExit = true;
|
||||
}
|
||||
|
||||
// clean up registry
|
||||
// hdfs dir content will be overwritten when a new container gets started,
|
||||
// so no need remove.
|
||||
compInstance.scheduler.executorService
|
||||
.submit(compInstance::cleanupRegistry);
|
||||
if (!failedBeforeLaunching) {
|
||||
// clean up registry
|
||||
// If the container failed before launching, no need to cleanup registry,
|
||||
// because it was not registered before.
|
||||
// hdfs dir content will be overwritten when a new container gets started,
|
||||
// so no need remove.
|
||||
compInstance.scheduler.executorService
|
||||
.submit(compInstance::cleanupRegistry);
|
||||
if (compInstance.timelineServiceEnabled) {
|
||||
// record in ATS
|
||||
compInstance.serviceTimelinePublisher
|
||||
.componentInstanceFinished(compInstance,
|
||||
event.getStatus().getExitStatus(), event.getStatus().getState(),
|
||||
containerDiag);
|
||||
}
|
||||
compInstance.containerSpec.setState(ContainerState.STOPPED);
|
||||
}
|
||||
|
||||
// remove the failed ContainerId -> CompInstance mapping
|
||||
comp.getScheduler().removeLiveCompInstance(event.getContainerId());
|
||||
|
||||
if (compInstance.timelineServiceEnabled) {
|
||||
// record in ATS
|
||||
compInstance.serviceTimelinePublisher
|
||||
.componentInstanceFinished(compInstance,
|
||||
event.getStatus().getExitStatus(), event.getStatus().getState(),
|
||||
containerDiag);
|
||||
}
|
||||
|
||||
compInstance.containerSpec.setState(ContainerState.STOPPED);
|
||||
if (shouldExit) {
|
||||
// Sleep for 5 seconds in hope that the state can be recorded in ATS.
|
||||
// in case there's a client polling the comp state, it can be notified.
|
||||
|
|
|
|||
|
|
@ -30,10 +30,6 @@ public interface RestApiConstants {
|
|||
String SERVICE_NAME = "service_name";
|
||||
String COMPONENT_NAME = "component_name";
|
||||
|
||||
String DEFAULT_COMPONENT_NAME = "default";
|
||||
|
||||
String PROPERTY_REST_SERVICE_HOST = "REST_SERVICE_HOST";
|
||||
String PROPERTY_REST_SERVICE_PORT = "REST_SERVICE_PORT";
|
||||
Long DEFAULT_UNLIMITED_LIFETIME = -1l;
|
||||
|
||||
Integer ERROR_CODE_APP_DOES_NOT_EXIST = 404001;
|
||||
|
|
|
|||
|
|
@ -80,38 +80,14 @@ public static void validateAndResolveService(Service service,
|
|||
|
||||
validateNameFormat(service.getName(), conf);
|
||||
|
||||
// If the service has no components do top-level checks
|
||||
// If the service has no components, throw error
|
||||
if (!hasComponent(service)) {
|
||||
// If artifact is of type SERVICE, read other service components
|
||||
if (service.getArtifact() != null && service.getArtifact()
|
||||
.getType() == Artifact.TypeEnum.SERVICE) {
|
||||
if (StringUtils.isEmpty(service.getArtifact().getId())) {
|
||||
throw new IllegalArgumentException(
|
||||
RestApiErrorMessages.ERROR_ARTIFACT_ID_INVALID);
|
||||
}
|
||||
Service otherService = loadService(fs,
|
||||
service.getArtifact().getId());
|
||||
service.setComponents(otherService.getComponents());
|
||||
service.setArtifact(null);
|
||||
SliderUtils.mergeMapsIgnoreDuplicateKeys(service.getQuicklinks(),
|
||||
otherService.getQuicklinks());
|
||||
} else {
|
||||
// Since it is a simple service with no components, create a default
|
||||
// component
|
||||
Component comp = createDefaultComponent(service);
|
||||
validateComponent(comp, fs.getFileSystem(), conf);
|
||||
service.getComponents().add(comp);
|
||||
if (service.getLifetime() == null) {
|
||||
service.setLifetime(RestApiConstants.DEFAULT_UNLIMITED_LIFETIME);
|
||||
}
|
||||
return;
|
||||
}
|
||||
throw new IllegalArgumentException(
|
||||
"No component specified for " + service.getName());
|
||||
}
|
||||
|
||||
// Validate there are no component name collisions (collisions are not
|
||||
// currently supported) and add any components from external services
|
||||
// TODO allow name collisions? see AppState#roles
|
||||
// TODO or add prefix to external component names?
|
||||
Configuration globalConf = service.getConfiguration();
|
||||
Set<String> componentNames = new HashSet<>();
|
||||
List<Component> componentsToRemove = new ArrayList<>();
|
||||
|
|
@ -174,8 +150,6 @@ public static void validateAndResolveService(Service service,
|
|||
// values are not provided
|
||||
Artifact globalArtifact = service.getArtifact();
|
||||
Resource globalResource = service.getResource();
|
||||
Long globalNumberOfContainers = service.getNumberOfContainers();
|
||||
String globalLaunchCommand = service.getLaunchCommand();
|
||||
for (Component comp : service.getComponents()) {
|
||||
// fill in global artifact unless it is type SERVICE
|
||||
if (comp.getArtifact() == null && service.getArtifact() != null
|
||||
|
|
@ -187,14 +161,6 @@ public static void validateAndResolveService(Service service,
|
|||
if (comp.getResource() == null) {
|
||||
comp.setResource(globalResource);
|
||||
}
|
||||
// fill in global container count
|
||||
if (comp.getNumberOfContainers() == null) {
|
||||
comp.setNumberOfContainers(globalNumberOfContainers);
|
||||
}
|
||||
// fill in global launch command
|
||||
if (comp.getLaunchCommand() == null) {
|
||||
comp.setLaunchCommand(globalLaunchCommand);
|
||||
}
|
||||
// validate dependency existence
|
||||
if (comp.getDependencies() != null) {
|
||||
for (String dependency : comp.getDependencies()) {
|
||||
|
|
@ -360,7 +326,7 @@ public static void validateCompResourceSize(
|
|||
}
|
||||
}
|
||||
|
||||
public static boolean hasComponent(Service service) {
|
||||
private static boolean hasComponent(Service service) {
|
||||
if (service.getComponents() == null || service.getComponents()
|
||||
.isEmpty()) {
|
||||
return false;
|
||||
|
|
@ -368,17 +334,6 @@ public static boolean hasComponent(Service service) {
|
|||
return true;
|
||||
}
|
||||
|
||||
public static Component createDefaultComponent(Service service) {
|
||||
Component comp = new Component();
|
||||
comp.setName(RestApiConstants.DEFAULT_COMPONENT_NAME);
|
||||
comp.setArtifact(service.getArtifact());
|
||||
comp.setResource(service.getResource());
|
||||
comp.setNumberOfContainers(service.getNumberOfContainers());
|
||||
comp.setLaunchCommand(service.getLaunchCommand());
|
||||
comp.setConfiguration(service.getConfiguration());
|
||||
return comp;
|
||||
}
|
||||
|
||||
public static Collection<Component> sortByDependencies(List<Component>
|
||||
components) {
|
||||
Map<String, Component> sortedComponents =
|
||||
|
|
|
|||
|
|
@ -84,11 +84,11 @@ protected Service createExampleApplication() {
|
|||
return exampleApp;
|
||||
}
|
||||
|
||||
protected Component createComponent(String name) {
|
||||
public static Component createComponent(String name) {
|
||||
return createComponent(name, 2L, "sleep 1000");
|
||||
}
|
||||
|
||||
protected Component createComponent(String name, long numContainers,
|
||||
protected static Component createComponent(String name, long numContainers,
|
||||
String command) {
|
||||
Component comp1 = new Component();
|
||||
comp1.setNumberOfContainers(numContainers);
|
||||
|
|
|
|||
|
|
@ -20,11 +20,11 @@
|
|||
import org.apache.hadoop.conf.Configuration;
|
||||
import org.apache.hadoop.registry.client.api.RegistryConstants;
|
||||
import org.apache.hadoop.yarn.conf.YarnConfiguration;
|
||||
import org.apache.hadoop.yarn.service.api.records.Service;
|
||||
import org.apache.hadoop.yarn.service.exceptions.RestApiErrorMessages;
|
||||
import org.apache.hadoop.yarn.service.api.records.Artifact;
|
||||
import org.apache.hadoop.yarn.service.api.records.Component;
|
||||
import org.apache.hadoop.yarn.service.api.records.Resource;
|
||||
import org.apache.hadoop.yarn.service.api.records.Service;
|
||||
import org.apache.hadoop.yarn.service.exceptions.RestApiErrorMessages;
|
||||
import org.apache.hadoop.yarn.service.utils.ServiceApiUtil;
|
||||
import org.apache.hadoop.yarn.service.utils.SliderFileSystem;
|
||||
import org.junit.Assert;
|
||||
|
|
@ -36,9 +36,9 @@
|
|||
import java.io.IOException;
|
||||
import java.util.Arrays;
|
||||
import java.util.Collection;
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
|
||||
import static org.apache.hadoop.yarn.service.conf.RestApiConstants.DEFAULT_COMPONENT_NAME;
|
||||
import static org.apache.hadoop.yarn.service.conf.RestApiConstants.DEFAULT_UNLIMITED_LIFETIME;
|
||||
import static org.apache.hadoop.yarn.service.exceptions.RestApiErrorMessages.*;
|
||||
import static org.junit.Assert.assertEquals;
|
||||
|
|
@ -99,6 +99,8 @@ public void testResourceValidation() throws Exception {
|
|||
|
||||
// launch command not specified
|
||||
app.setName(LEN_64_STR);
|
||||
Component comp = new Component().name("comp1");
|
||||
app.addComponent(comp);
|
||||
try {
|
||||
ServiceApiUtil.validateAndResolveService(app, sfs, CONF_DEFAULT_DNS);
|
||||
Assert.fail(EXCEPTION_PREFIX + "service with no launch command");
|
||||
|
|
@ -118,18 +120,8 @@ public void testResourceValidation() throws Exception {
|
|||
e.getMessage());
|
||||
}
|
||||
|
||||
// resource not specified
|
||||
app.setLaunchCommand("sleep 3600");
|
||||
try {
|
||||
ServiceApiUtil.validateAndResolveService(app, sfs, CONF_DNS_ENABLED);
|
||||
Assert.fail(EXCEPTION_PREFIX + "service with no resource");
|
||||
} catch (IllegalArgumentException e) {
|
||||
assertEquals(String.format(
|
||||
RestApiErrorMessages.ERROR_RESOURCE_FOR_COMP_INVALID,
|
||||
DEFAULT_COMPONENT_NAME), e.getMessage());
|
||||
}
|
||||
|
||||
// memory not specified
|
||||
comp.setLaunchCommand("sleep 1");
|
||||
Resource res = new Resource();
|
||||
app.setResource(res);
|
||||
try {
|
||||
|
|
@ -138,7 +130,7 @@ public void testResourceValidation() throws Exception {
|
|||
} catch (IllegalArgumentException e) {
|
||||
assertEquals(String.format(
|
||||
RestApiErrorMessages.ERROR_RESOURCE_MEMORY_FOR_COMP_INVALID,
|
||||
DEFAULT_COMPONENT_NAME), e.getMessage());
|
||||
comp.getName()), e.getMessage());
|
||||
}
|
||||
|
||||
// invalid no of cpus
|
||||
|
|
@ -151,7 +143,7 @@ public void testResourceValidation() throws Exception {
|
|||
} catch (IllegalArgumentException e) {
|
||||
assertEquals(String.format(
|
||||
RestApiErrorMessages.ERROR_RESOURCE_CPUS_FOR_COMP_INVALID_RANGE,
|
||||
DEFAULT_COMPONENT_NAME), e.getMessage());
|
||||
comp.getName()), e.getMessage());
|
||||
}
|
||||
|
||||
// number of containers not specified
|
||||
|
|
@ -173,7 +165,7 @@ public void testResourceValidation() throws Exception {
|
|||
} catch (IllegalArgumentException e) {
|
||||
assertEquals(String.format(RestApiErrorMessages
|
||||
.ERROR_RESOURCE_PROFILE_MULTIPLE_VALUES_FOR_COMP_NOT_SUPPORTED,
|
||||
DEFAULT_COMPONENT_NAME),
|
||||
comp.getName()),
|
||||
e.getMessage());
|
||||
}
|
||||
|
||||
|
|
@ -202,25 +194,6 @@ public void testResourceValidation() throws Exception {
|
|||
Assert.assertTrue(e.getMessage()
|
||||
.startsWith(ERROR_CONTAINERS_COUNT_INVALID));
|
||||
}
|
||||
|
||||
// negative number of containers
|
||||
app.setNumberOfContainers(-1L);
|
||||
try {
|
||||
ServiceApiUtil.validateAndResolveService(app, sfs, CONF_DNS_ENABLED);
|
||||
Assert.fail(EXCEPTION_PREFIX + "negative number of containers");
|
||||
} catch (IllegalArgumentException e) {
|
||||
Assert.assertTrue(e.getMessage()
|
||||
.startsWith(ERROR_CONTAINERS_COUNT_INVALID));
|
||||
}
|
||||
|
||||
// everything valid here
|
||||
app.setNumberOfContainers(5L);
|
||||
try {
|
||||
ServiceApiUtil.validateAndResolveService(app, sfs, CONF_DNS_ENABLED);
|
||||
} catch (IllegalArgumentException e) {
|
||||
LOG.error("service attributes specified should be valid here", e);
|
||||
Assert.fail(NO_EXCEPTION_PREFIX + e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
|
|
@ -228,15 +201,17 @@ public void testArtifacts() throws IOException {
|
|||
SliderFileSystem sfs = ServiceTestUtils.initMockFs();
|
||||
|
||||
Service app = new Service();
|
||||
app.setName("name");
|
||||
app.setName("service1");
|
||||
Resource res = new Resource();
|
||||
app.setResource(res);
|
||||
res.setMemory("512M");
|
||||
app.setNumberOfContainers(3L);
|
||||
|
||||
// no artifact id fails with default type
|
||||
Artifact artifact = new Artifact();
|
||||
app.setArtifact(artifact);
|
||||
Component comp = ServiceTestUtils.createComponent("comp1");
|
||||
|
||||
app.setComponents(Collections.singletonList(comp));
|
||||
try {
|
||||
ServiceApiUtil.validateAndResolveService(app, sfs, CONF_DNS_ENABLED);
|
||||
Assert.fail(EXCEPTION_PREFIX + "service with no artifact id");
|
||||
|
|
@ -272,9 +247,6 @@ public void testArtifacts() throws IOException {
|
|||
Assert.fail(NO_EXCEPTION_PREFIX + e.getMessage());
|
||||
}
|
||||
|
||||
// defaults assigned
|
||||
assertEquals(app.getComponents().get(0).getName(),
|
||||
DEFAULT_COMPONENT_NAME);
|
||||
assertEquals(app.getLifetime(), DEFAULT_UNLIMITED_LIFETIME);
|
||||
}
|
||||
|
||||
|
|
@ -289,15 +261,14 @@ private static Component createValidComponent(String compName) {
|
|||
comp.setName(compName);
|
||||
comp.setResource(createValidResource());
|
||||
comp.setNumberOfContainers(1L);
|
||||
comp.setLaunchCommand("sleep 1");
|
||||
return comp;
|
||||
}
|
||||
|
||||
private static Service createValidApplication(String compName) {
|
||||
Service app = new Service();
|
||||
app.setLaunchCommand("sleep 3600");
|
||||
app.setName("name");
|
||||
app.setResource(createValidResource());
|
||||
app.setNumberOfContainers(1L);
|
||||
if (compName != null) {
|
||||
app.addComponent(createValidComponent(compName));
|
||||
}
|
||||
|
|
@ -315,7 +286,7 @@ public void testExternalApplication() throws IOException {
|
|||
artifact.setType(Artifact.TypeEnum.SERVICE);
|
||||
artifact.setId("id");
|
||||
app.setArtifact(artifact);
|
||||
|
||||
app.addComponent(ServiceTestUtils.createComponent("comp2"));
|
||||
try {
|
||||
ServiceApiUtil.validateAndResolveService(app, sfs, CONF_DNS_ENABLED);
|
||||
} catch (IllegalArgumentException e) {
|
||||
|
|
@ -323,7 +294,7 @@ public void testExternalApplication() throws IOException {
|
|||
}
|
||||
|
||||
assertEquals(1, app.getComponents().size());
|
||||
assertNotNull(app.getComponent("comp1"));
|
||||
assertNotNull(app.getComponent("comp2"));
|
||||
}
|
||||
|
||||
@Test
|
||||
|
|
@ -410,12 +381,13 @@ public static void verifyDependencySorting(List<Component> components,
|
|||
|
||||
@Test
|
||||
public void testDependencySorting() throws IOException {
|
||||
Component a = new Component().name("a");
|
||||
Component b = new Component().name("b");
|
||||
Component c = new Component().name("c");
|
||||
Component d = new Component().name("d").dependencies(Arrays.asList("c"));
|
||||
Component e = new Component().name("e").dependencies(Arrays.asList("b",
|
||||
"d"));
|
||||
Component a = ServiceTestUtils.createComponent("a");
|
||||
Component b = ServiceTestUtils.createComponent("b");
|
||||
Component c = ServiceTestUtils.createComponent("c");
|
||||
Component d =
|
||||
ServiceTestUtils.createComponent("d").dependencies(Arrays.asList("c"));
|
||||
Component e = ServiceTestUtils.createComponent("e")
|
||||
.dependencies(Arrays.asList("b", "d"));
|
||||
|
||||
verifyDependencySorting(Arrays.asList(a, b, c), a, b, c);
|
||||
verifyDependencySorting(Arrays.asList(c, a, b), c, a, b);
|
||||
|
|
|
|||
|
|
@ -176,7 +176,7 @@ private void checkContainerLaunchDependencies(ServiceClient client,
|
|||
for (String comp : compOrder) {
|
||||
long num = retrievedApp.getComponent(comp).getNumberOfContainers();
|
||||
for (int i = 0; i < num; i++) {
|
||||
String compInstanceName = containerList.get(index).getComponentName();
|
||||
String compInstanceName = containerList.get(index).getComponentInstanceName();
|
||||
String compName =
|
||||
compInstanceName.substring(0, compInstanceName.lastIndexOf('-'));
|
||||
Assert.assertEquals(comp, compName);
|
||||
|
|
@ -219,7 +219,7 @@ private void checkEachCompInstancesInOrder(Component component) {
|
|||
Assert.assertEquals(expectedNumInstances, component.getContainers().size());
|
||||
TreeSet<String> instances = new TreeSet<>();
|
||||
for (Container container : component.getContainers()) {
|
||||
instances.add(container.getComponentName());
|
||||
instances.add(container.getComponentInstanceName());
|
||||
}
|
||||
|
||||
int i = 0;
|
||||
|
|
|
|||
|
|
@ -1,7 +1,6 @@
|
|||
{
|
||||
"name": "app-1",
|
||||
"lifetime": "3600",
|
||||
"launch_command": "sleep 3600",
|
||||
"configuration": {
|
||||
"properties": {
|
||||
"g1": "a",
|
||||
|
|
@ -29,10 +28,11 @@
|
|||
"cpus": 1,
|
||||
"memory": "512"
|
||||
},
|
||||
"number_of_containers": 2,
|
||||
"components": [
|
||||
{
|
||||
"name": "simple",
|
||||
"launch_command": "sleep 3600",
|
||||
"number_of_containers": 2,
|
||||
"configuration": {
|
||||
"files": [
|
||||
{
|
||||
|
|
@ -47,6 +47,8 @@
|
|||
},
|
||||
{
|
||||
"name": "master",
|
||||
"launch_command": "sleep 3600",
|
||||
"number_of_containers": 2,
|
||||
"configuration": {
|
||||
"properties": {
|
||||
"name": "m",
|
||||
|
|
@ -56,6 +58,8 @@
|
|||
},
|
||||
{
|
||||
"name": "worker",
|
||||
"number_of_containers": 2,
|
||||
"launch_command": "sleep 3600",
|
||||
"resource": {
|
||||
"cpus": 1,
|
||||
"memory": "1024"
|
||||
|
|
|
|||
|
|
@ -2,7 +2,6 @@
|
|||
"name": "app-1",
|
||||
"id" : "application_1503358878042_0011",
|
||||
"lifetime": "3600",
|
||||
"launch_command": "sleep 3600",
|
||||
"configuration": {
|
||||
"properties": {
|
||||
"g1": "a",
|
||||
|
|
@ -14,14 +13,16 @@
|
|||
"cpus": 1,
|
||||
"memory": "512"
|
||||
},
|
||||
"number_of_containers": 2,
|
||||
"components": [
|
||||
{
|
||||
"name": "simple"
|
||||
"name": "simple",
|
||||
"number_of_containers": 2,
|
||||
"launch_command": "sleep 3600"
|
||||
},
|
||||
{
|
||||
"name": "master",
|
||||
"number_of_containers": 1,
|
||||
"launch_command": "sleep 3600",
|
||||
"configuration": {
|
||||
"properties": {
|
||||
"g1": "overridden",
|
||||
|
|
@ -33,6 +34,7 @@
|
|||
{
|
||||
"name": "worker",
|
||||
"number_of_containers": 5,
|
||||
"launch_command": "sleep 3600",
|
||||
"resource": {
|
||||
"cpus": 1,
|
||||
"memory": "1024"
|
||||
|
|
|
|||
|
|
@ -1,8 +1,15 @@
|
|||
{
|
||||
"name": "external-0",
|
||||
"lifetime": "3600",
|
||||
"artifact": {
|
||||
"type": "SERVICE",
|
||||
"id": "app-1"
|
||||
}
|
||||
|
||||
"components" : [
|
||||
{
|
||||
"name" : "comp1",
|
||||
"artifact": {
|
||||
"type": "SERVICE",
|
||||
"id": "app-1"
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
}
|
||||
|
|
|
|||
|
|
@ -201,7 +201,6 @@ Set a component's desired number of instanes
|
|||
|404|Service does not exist|No Content|
|
||||
|default|Unexpected error|ServiceStatus|
|
||||
|
||||
|
||||
## Definitions
|
||||
### Artifact
|
||||
|
||||
|
|
@ -221,6 +220,7 @@ One or more components of the service. If the service is HBase say, then the com
|
|||
|Name|Description|Required|Schema|Default|
|
||||
|----|----|----|----|----|
|
||||
|name|Name of the service component (mandatory). If Registry DNS is enabled, the max length is 63 characters. If unique component support is enabled, the max length is lowered to 44 characters.|true|string||
|
||||
|state|The state of the component|false|ComponentState||
|
||||
|dependencies|An array of service components which should be in READY state (as defined by readiness check), before this component can be started. The dependencies across all components of a service should be represented as a DAG.|false|string array||
|
||||
|readiness_check|Readiness check for this component.|false|ReadinessCheck||
|
||||
|artifact|Artifact of the component (optional). If not specified, the service level global artifact takes effect.|false|Artifact||
|
||||
|
|
@ -233,6 +233,15 @@ One or more components of the service. If the service is HBase say, then the com
|
|||
|quicklinks|A list of quicklink keys defined at the service level, and to be resolved by this component.|false|string array||
|
||||
|
||||
|
||||
### ComponentState
|
||||
|
||||
The state of the component
|
||||
|
||||
|Name|Description|Required|Schema|Default|
|
||||
|----|----|----|----|----|
|
||||
|state|enum of the state of the component|false|enum (FLEXING, STABLE)||
|
||||
|
||||
|
||||
### ConfigFile
|
||||
|
||||
A config file that needs to be created and made available as a volume in a service component container.
|
||||
|
|
@ -268,7 +277,7 @@ An instance of a running service container.
|
|||
|hostname|Fully qualified hostname of a running container, e.g. ctr-e3751-1458061340047-0008-01-000002.examplestg.site. The IP address and hostname attribute values are dependent on the cluster/docker network setup as per YARN-4007.|false|string||
|
||||
|bare_host|The bare node or host in which the container is running, e.g. cn008.example.com.|false|string||
|
||||
|state|State of the container of a service.|false|ContainerState||
|
||||
|component_name|Name of the component that this container instance belongs to.|false|string||
|
||||
|component_instance_name|Name of the component instance that this container instance belongs to. Component instance name is named as $COMPONENT_NAME-i, where i is a monotonically increasing integer. E.g. A componet called nginx can have multiple component instances named as nginx-0, nginx-1 etc. Each component instance is backed by a container instance.|false|string||
|
||||
|resource|Resource used for this container.|false|Resource||
|
||||
|artifact|Artifact used for this container.|false|Artifact||
|
||||
|privileged_container|Container running in privileged mode or not.|false|boolean||
|
||||
|
|
@ -322,18 +331,15 @@ a service resource has the following attributes.
|
|||
|----|----|----|----|----|
|
||||
|name|A unique service name. If Registry DNS is enabled, the max length is 63 characters.|true|string||
|
||||
|id|A unique service id.|false|string||
|
||||
|artifact|Artifact of single-component service.|false|Artifact||
|
||||
|resource|Resource of single-component service or the global default for multi-component services. Mandatory if it is a single-component service and if cpus and memory are not specified at the Service level.|false|Resource||
|
||||
|launch_command|The custom launch command of a service component (optional). If not specified for services with docker images say, it will default to the default start command of the image. If there is a single component in this service, you can specify this without the need to have a 'components' section.|false|string||
|
||||
|artifact|The default artifact for all components of the service except the components which has Artifact type set to SERVICE (optional).|false|Artifact||
|
||||
|resource|The default resource for all components of the service (optional).|false|Resource||
|
||||
|launch_time|The time when the service was created, e.g. 2016-03-16T01:01:49.000Z.|false|string (date)||
|
||||
|number_of_containers|Number of containers for each component in the service. Each component can further override this service-level global default.|false|integer (int64)||
|
||||
|number_of_running_containers|In get response this provides the total number of running containers for this service (across all components) at the time of request. Note, a subsequent request can return a different number as and when more containers get allocated until it reaches the total number of containers or if a flex request has been made between the two requests.|false|integer (int64)||
|
||||
|lifetime|Life time (in seconds) of the service from the time it reaches the STARTED state (after which it is automatically destroyed by YARN). For unlimited lifetime do not set a lifetime value.|false|integer (int64)||
|
||||
|placement_policy|(TBD) Advanced scheduling and placement policies. If not specified, it defaults to the default placement policy of the service owner. The design of placement policies are in the works. It is not very clear at this point, how policies in conjunction with labels be exposed to service owners. This is a placeholder for now. The advanced structure of this attribute will be determined by YARN-4902.|false|PlacementPolicy||
|
||||
|components|Components of a service.|false|Component array||
|
||||
|configuration|Config properties of a service. Configurations provided at the service/global level are available to all the components. Specific properties can be overridden at the component level.|false|Configuration||
|
||||
|containers|Containers of a started service. Specifying a value for this attribute for the POST payload raises a validation error. This blob is available only in the GET response of a started service.|false|Container array||
|
||||
|state|State of the service. Specifying a value for this attribute for the POST payload raises a validation error. This attribute is available only in the GET response of a started service.|false|ServiceState||
|
||||
|state|State of the service. Specifying a value for this attribute for the PUT payload means update the service to this desired state.|false|ServiceState||
|
||||
|quicklinks|A blob of key-value pairs of quicklinks to be exported for a service.|false|object||
|
||||
|queue|The YARN queue that this service should be submitted to.|false|string||
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue