--- title_tag: Deploy Kubernetes Cluster Services | Crosswalk meta_desc: This page will provide an overview on how to deploy Kubernetes Cluster Services. title: Cluster services h1: Kubernetes cluster services meta_image: /images/docs/meta-images/docs-clouds-kubernetes-meta-image.png menu: clouds: parent: kubernetes-guides identifier: kubernetes-guides-cluster-svcs weight: 7 aliases: - /docs/guides/crosswalk/kubernetes/cluster-services/ --- {{< chooser cloud "aws,azure,gcp" / >}} Cluster services are general services scoped at the Kubernetes cluster level. These services tend to include logging and monitoring at a minimum for the whole cluster, or a subset of apps and workloads. It could also include policy enforcement and service meshes. {{% choosable cloud aws %}} The full code for the AWS cluster services is on [GitHub][gh-repo-stack]. [gh-repo-stack]: https://github.com/pulumi/kubernetes-guides/tree/master/aws/04-cluster-services {{% /choosable %}} {{% choosable cloud azure %}} The full code for the Azure cluster services is on [GitHub][gh-repo-stack]. [gh-repo-stack]: https://github.com/pulumi/kubernetes-guides/tree/master/azure/04-cluster-services {{% /choosable %}} {{% choosable cloud gcp %}} GKE logging and monitoring is managed by Google Cloud through StackDriver. The repo for the Google Cloud cluster services is on [GitHub][gh-repo-stack], but it is empty since no extra steps are required after cluster and Node Pool creation in the [Cluster Configuration][crosswalk-k8s-cluster] stack. [crosswalk-k8s-cluster]: https://github.com/pulumi/kubernetes-guides/tree/master/gcp/03-cluster-configuration [gh-repo-stack]: https://github.com/pulumi/kubernetes-guides/tree/master/gcp/04-cluster-services {{% /choosable %}} The full code for the general cluster services is on [GitHub][gh-repo-stack]. [gh-repo-stack]: https://github.com/pulumi/kubernetes-guides/tree/master/general-cluster-services {{% choosable cloud aws %}} ## Overview We'll explore how to setup: * [AWS Logging](#aws-logging) * [AWS Monitoring](#aws-monitoring) * [Datadog](#datadog) See the [official AWS docs][k8s-logs-docs] for more details. ## Prerequisites Authenticate as the `admins` role from the [Identity][aws-admin-identity-stack] stack. ```bash $ aws sts assume-role --role-arn `pulumi stack output adminsIamRoleArn` --role-session-name k8s-admin $ export KUBECONFIG=`pwd`/kubeconfig-admin.json ``` [aws-admin-identity-stack]: /docs/clouds/kubernetes/guides/identity/#create-an-iam-role-for-admins ## AWS Logging ### Control Plane In the [Recommended Settings][crosswalk-control-plane-recommended] of Creating the Control Plane, we enabled cluster logging for the various controllers of the control plane. To view these logs, go to the [CloudWatch console][aws-cw-console], navigate to the logs in your region, and look for the following group. ``` /aws/eks/Cluster_Name/cluster ``` The cluster name can be retrieved from the [cluster stack][crosswalk-control-plane] output. ```bash $ pulumi stack output clusterName ``` ### Worker Nodes and Pods #### Configure Worker Node IAM Policy To work with [Cloudwatch Logs][aws-cw-logs], the identities created in [Identity][crosswalk-k8s-identity] for each worker node group must have the proper permissions in IAM. Attach the permissions to the IAM role for each nodegroup. ```typescript import * as aws from "@pulumi/aws"; // Parse out the role names e.g. `roleName-123456` from `arn:aws:iam::123456789012:role/roleName-123456` const stdNodegroupIamRoleName = config.stdNodegroupIamRoleArn.apply(s => s.split("/")).apply(s => s[1]) const perfNodegroupIamRoleName = config.perfNodegroupIamRoleArn.apply(s => s.split("/")).apply(s => s[1]) // Create a new IAM Policy for fluentd-cloudwatch to manage CloudWatch Logs. const name = "fluentd-cloudwatch"; const fluentdCloudWatchPolicy = new aws.iam.Policy(name, { description: "Allows fluentd-cloudwatch to work with CloudWatch Logs.", policy: JSON.stringify( { Version: "2012-10-17", Statement: [{Effect: "Allow", Action: ["logs:*"], Resource: ["arn:aws:logs:*:*:*"]}] } ) }, ); // Attach CloudWatch Logs policies to a role. function attachLogPolicies(name: string, arn: pulumi.Input) { new aws.iam.RolePolicyAttachment(name, { policyArn: fluentdCloudWatchPolicy.arn, role: arn}, ); } attachLogPolicies("stdRpa", stdNodegroupIamRoleName); attachLogPolicies("perfRpa", perfNodegroupIamRoleName); ``` [crosswalk-control-plane]: /docs/clouds/kubernetes/guides/control-plane/ [crosswalk-control-plane-recommended]: /docs/clouds/kubernetes/guides/control-plane/#recommended-settings [crosswalk-k8s-identity]: /docs/clouds/kubernetes/guides/identity/#create-iam-roles-for-eks-node-groups [aws-cw-console]: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/GettingSetup.html#ConsoleSignIn [aws-cw-logs]: https://aws.amazon.com/cloudwatch/features/ {{< chooser k8s-language "typescript,yaml" / >}} {{< choosable k8s-language yaml >}} Using the YAML manifests in the [AWS samples][k8s-logs-samples], we can provision `fluentd-cloudwatch` to run as a [DaemonSet][k8s-ds] and send worker and app logs to [CloudWatch Logs][aws-cw-logs]. #### Install fluentd Create a Namespace. ```bash $ kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/master/k8s-yaml-templates/cloudwatch-namespace.yaml ``` Create a ConfigMap. ```bash $ kubectl create configmap cluster-info --from-literal=cluster.name=`pulumi stack output clusterName` --from-literal=logs.region=`pulumi stack output region` -n amazon-cloudwatch ``` Deploy the DaemonSet. ```bash $ kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/master/k8s-yaml-templates/fluentd/fluentd.yaml ``` Validate the deployment. ```bash $ kubectl get pods -n amazon-cloudwatch ``` Verify the fluentd setup in the [CloudWatch console][aws-cw-console] by navigating to the logs in your region, and looking for the following groups. ``` /aws/containerinsights/Cluster_Name/application /aws/containerinsights/Cluster_Name/host /aws/containerinsights/Cluster_Name/dataplane ``` The cluster name can be retrieved from the [cluster stack][crosswalk-control-plane] output. ```bash $ pulumi stack output clusterName ``` Clean Up. ```bash $ kubectl delete ns amazon-cloudwatch ``` [crosswalk-control-plane]: /docs/clouds/kubernetes/guides/control-plane/ [k8s-ds]: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/ [k8s-logs-samples]: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs.html [aws-cw-console]: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/GettingSetup.html#ConsoleSignIn [crosswalk-k8s-identity]: /docs/clouds/kubernetes/guides/identity/#create-iam-roles-for-eks-node-groups [aws-cw-logs]: https://aws.amazon.com/cloudwatch/features/ {{< /choosable >}} {{< choosable k8s-language typescript >}} Using the [Helm chart][fluentd-helm], we can provision `fluentd-cloudwatch` in Pulumi to run as a [DaemonSet][k8s-ds] and send worker and app logs to [CloudWatch Logs][aws-cw-logs]. #### Install fluentd Deploy the Chart into the `cluster-svcs` namespace created in [Configure Cluster Defaults][crosswalk-k8s-defaults] . ```typescript import * as k8s from "@pulumi/kubernetes"; // Create a new provider to the cluster using the cluster's kubeconfig. const provider = new k8s.Provider("provider", {kubeconfig: config.kubeconfig}); // Create a new CloudWatch Log group for fluentd-cloudwatch. const fluentdCloudWatchLogGroup = new aws.cloudwatch.LogGroup(name); export let fluentdCloudWatchLogGroupName = fluentdCloudWatchLogGroup.name; // Deploy fluentd-cloudwatch using the Helm chart. const fluentdCloudwatch = new k8s.helm.v3.Chart(name, { namespace: config.clusterSvcsNamespaceName, chart: "fluentd-cloudwatch", version: "0.11.0", fetchOpts: { repo: "https://charts.helm.sh/incubator", }, values: { extraVars: [ "{ name: FLUENT_UID, value: '0' }" ], rbac: {create: true}, awsRegion: aws.config.region, logGroupName: fluentdCloudWatchLogGroup.name, }, transformations: [ (obj: any) => { // Do transformations on the YAML to set the namespace if (obj.metadata) { obj.metadata.namespace = config.clusterSvcsNamespaceName; } }, ], }, {providers: { kubernetes: provider }}, ); ``` Validate the deployment. ```bash $ kubectl get pods -n `pulumi stack output clusterSvcsNamespaceName` ``` Verify the fluentd setup in the [CloudWatch console][aws-cw-console] by navigating to the logs in your region, and looking for the following group. ```bash $ pulumi stack output fluentdCloudWatchLogGroupName ``` [crosswalk-k8s-identity]: /docs/clouds/kubernetes/guides/identity/#create-iam-roles-for-eks-node-groups [aws-cw-console]: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/GettingSetup.html#ConsoleSignIn [crosswalk-k8s-defaults]: /docs/clouds/kubernetes/guides/configure-defaults/#namespaces [aws-cw-logs]: https://aws.amazon.com/cloudwatch/features/ [k8s-ds]: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/ [fluentd-helm]: https://github.com/helm/charts/tree/master/incubator/fluentd-cloudwatch {{< /choosable >}} > Note: CloudWatch is rate limited and often times the size of the data being sent can cause `ThrottlingException error="Rate exceeded"`. This can cause a delay in logs showing up in CloudWatch. Request a limit increase, or alter the data being sent, if necessary. See the [CloudWatch limits][aws-cw-limits] for more details. [k8s-logs-docs]: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights.html [aws-cw-limits]: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_limits.html {{% /choosable %}} {{% choosable cloud azure %}} ## Overview We'll explore how to setup: * [Azure Logging and Monitoring](#azure-logging-and-monitoring) * [Datadog](#datadog) See the official [Azure Monitor][azure-monitor-overview] and [AKS][aks-logs] docs for more details. ## Azure Logging and Monitoring AKS monitoring is managed by Azure through [Log Analytics][azure-la]. Once enabled, in the Azure portal visit the cluster's Kubernetes service details, and [analyze its Azure Monitor][azure-monitor-analyze] information in the Monitoring section's: Insights, Logs, and Metrics. ### Enable Azure Monitor for the Cluster Enable the Log Analytics agent on the AKS cluster in the [Cluster Configuration][crosswalk-k8s-cluster] stack. ```ts import * as azure from "@pulumi/azure"; // Create the AKS cluster with LogAnalytics enabled in the given workspace. const cluster = new azure.containerservice.KubernetesCluster(`${name}`, { ... resourceGroupName: config.resourceGroupName, addonProfile: { omsAgent: { enabled: true, logAnalyticsWorkspaceId: config.logAnalyticsWorkspaceId, }, }, }); ``` Enable logging for the control plane, and monitoring of all metrics in the [Cluster Services][crosswalk-aks-cluster-svcs] stack. ```ts import * as azure from "@pulumi/azure"; // Enable the Monitoring Diagonostic control plane component logs and AllMetrics const azMonitoringDiagnostic = new azure.monitoring.DiagnosticSetting(name, { logAnalyticsWorkspaceId: config.logAnalyticsWorkspaceId, targetResourceId: config.clusterId, logs: ["kube-apiserver", "kube-controller-manager", "kube-scheduler", "kube-audit", "cluster-autoscaler"] .map(category => ({ category, enabled : true, retentionPolicy: { enabled: true }, })), metrics: [{ category: "AllMetrics", retentionPolicy: { enabled: true }, }], }); ``` ### Worker Nodes To get the Worker kubelet logs you need to SSH into the nodes. Use the node admin username and SSH key used in the [Cluster Configuration][crosswalk-k8s-cluster] stack. ```ts import * as azure from "@pulumi/azure"; // Create the AKS cluster with LogAnalytics enabled in the given workspace. const cluster = new azure.containerservice.KubernetesCluster(`${name}`, { ... resourceGroupName: config.resourceGroupName, linuxProfile: { adminUsername: "aksuser", sshKey: { keyData: sshPublicKey, }, }, }); ``` See the official [AKS docs][aks-kubelet-logs] for more details. [azure-monitor-analyze]: https://docs.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-analyze [azure-la]: https://docs.microsoft.com/en-us/azure/azure-monitor/log-query/get-started-portal [aks-kubelet-logs]: https://docs.microsoft.com/en-us/azure/aks/kubelet-logs [azure-monitor-overview]: https://docs.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-overview [aks-logs]: https://docs.microsoft.com/en-us/azure/aks/view-master-logs [crosswalk-k8s-cluster]: https://github.com/pulumi/kubernetes-guides/tree/master/azure/03-cluster-configuration [crosswalk-aks-cluster-svcs]: https://github.com/pulumi/kubernetes-guides/tree/master/azure/04-cluster-services {{% /choosable %}} {{% choosable cloud gcp %}} ## Overview We'll explore how to setup: * [Google Cloud Logging and Monitoring](#azure-logging-and-monitoring) * [Datadog](#datadog) See the official [GKE][gke-logs] and [StackDriver Observing][gke-sd-observe] docs for more details. ## Google Cloud Logging and Monitoring GKE monitoring is managed by Google Cloud through [StackDriver][gke-sd]. Stackdriver Kubernetes Engine Monitoring is the default logging option for GKE clusters, and it comes automatically enabled for all clusters starting with version 1.14. ### Enable the Node Pool Enable the cluster's Node Pool with the proper logging and monitoring permission in the [Cluster Configuration][crosswalk-k8s-cluster] stack. ```ts import * as gcp from "@pulumi/gcp"; // Create a GKE cluster. // Versions >= 1.14 have Stackdriver Monitoring enabled by default. const cluster = new gcp.container.Cluster(`${name}`, { ... minMasterVersion: "1.14.7-gke.17", } // Create the GKE Node Pool with OAuth scopes enabled for logging and monitoring. const standardNodes = new gcp.container.NodePool("standard-nodes", { ... cluster: cluster.name, version: "1.14.7-gke.17", nodeConfig: { machineType: "n1-standard-1", oauthScopes: [ "https://www.googleapis.com/auth/compute", "https://www.googleapis.com/auth/devstorage.read_only", "https://www.googleapis.com/auth/logging.write", "https://www.googleapis.com/auth/monitoring", ], }, }); ``` [gke-sd]: https://app.google.stackdriver.com/ [gke-logs]: https://cloud.google.com/monitoring/kubernetes-engine/ [gke-sd-observe]: https://cloud.google.com/monitoring/kubernetes-engine/observing [crosswalk-k8s-cluster]: https://github.com/pulumi/kubernetes-guides/tree/master/gcp/03-cluster-configuration {{% /choosable %}} {{% choosable cloud aws %}} ## AWS Monitoring Using the YAML manifests in the [AWS samples][aws-metrics-samples], we can provision the CloudWatch Agent to run as a [DaemonSet][k8s-ds] and send metrics to [CloudWatch][aws-cw]. [aws-cw]: https://aws.amazon.com/cloudwatch [k8s-ds]: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/ [aws-metrics-samples]: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-metrics.html ### Install CloudWatch Agent Create a Namespace. ```bash $ kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/master/k8s-yaml-templates/cloudwatch-namespace.yaml ``` Create a ServiceAccount. ```bash $ kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/master/k8s-yaml-templates/cwagent-kubernetes-monitoring/cwagent-serviceaccount.yaml ``` Create a ConfigMap. ```bash $ curl -s https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/master/k8s-yaml-templates/cwagent-kubernetes-monitoring/cwagent-configmap.yaml | sed -e "s#{{cluster_name}}#`pulumi stack output clusterName`#g" | kubectl apply -f - ``` Deploy the DaemonSet. ```bash $ kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/master/k8s-yaml-templates/cwagent-kubernetes-monitoring/cwagent-daemonset.yaml ``` Validate the deployment. ```bash $ kubectl get pods -n amazon-cloudwatch ``` Verify the metrics setup in the [CloudWatch console][aws-cw-console] by navigating to Logs in your region, and looking for the following group. ``` /aws/containerinsights/Cluster_Name/performance ``` The cluster name can be retrieved from the [cluster stack][crosswalk-control-plane] output. ```bash $ pulumi stack output clusterName ``` You can also examine the stats in the [CloudWatch console][aws-cw-console] by navigating to Metrics in your region, and looking for the ContainerInsights for your cluster by its name. Clean Up. ```bash $ kubectl delete ns amazon-cloudwatch ``` [crosswalk-control-plane]: /docs/clouds/kubernetes/guides/control-plane/ [aws-cw-console]: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/GettingSetup.html#ConsoleSignIn {{% /choosable %}} ## Datadog Deploy [Datadog][k8s-datadog] as a DaemonSet to aggregate Kubernetes, node, and container metrics and events, in addition to provider managed logging and monitoring. The full code for this app stack is on [GitHub][gh-dd-stack]. [gh-dd-stack]: https://github.com/pulumi/kubernetes-guides/tree/master/general-cluster-services/datadog-daemonset ```ts import * as k8s from "@pulumi/kubernetes"; const appName = "datadog"; const appLabels = { app: appName }; // Create a DataDog DaemonSet. const datadog = new k8s.apps.v1.DaemonSet(appName, { metadata: { labels: appLabels}, spec: { selector: { matchLabels: appLabels, }, template: { metadata: { labels: appLabels }, spec: { containers: [ { image: "datadog/agent:latest", name: "nginx", resources: {limits: {memory: "512Mi"}, requests: {memory: "512Mi"}}, env: [ { name: "DD_KUBERNETES_KUBELET_HOST", valueFrom: { fieldRef: { fieldPath: "status.hostIP", }, }, }, { name: "DD_API_KEY", valueFrom: { configMapKeyRef: { name: ddConfigMap.metadata.name, key: "DD_API_KEY", }, }, }, { name: "DD_PROCESS_AGENT_ENABLED", valueFrom: { configMapKeyRef: { name: ddConfigMap.metadata.name, key: "DD_PROCESS_AGENT_ENABLED", }, }, }, { name: "DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL", valueFrom: { configMapKeyRef: { name: ddConfigMap.metadata.name, key: "DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL", }, }, }, { name: "DD_COLLECT_KUBERNETES_EVENTS", valueFrom: { configMapKeyRef: { name: ddConfigMap.metadata.name, key: "DD_COLLECT_KUBERNETES_EVENTS", }, }, }, ... ], volumeMounts: [ {name: "dockersocket", mountPath: "/var/run/docker.sock"}, {name: "proc", mountPath: "/host/proc"}, {name: "cgroup", mountPath: "/host/sys/fs/cgroup"}, ], }, ], volumes: [ {name: "dockersocket", hostPath: {path: "/var/run/docker.sock"}}, {name: "proc", hostPath: {path: "/proc"}}, {name: "cgroup", hostPath: {path: "/sys/fs/cgroup"}}, ], }, }, }, }, { provider: provider }); ``` [k8s-datadog]: https://docs.datadoghq.com/integrations/kubernetes/