Sean Holung 026234bdbb
revert get started experience (#3076)
* add back getting started page

* decollapse getting started guide

* update style

* remove quickstart files

scss file

* lint

* assets

* update links

* update links

* update links

* update links

* update link
2023-06-16 14:10:53 -07:00

186 lines
9.2 KiB
Markdown
Raw Permalink Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

---
title: "Unified Logs with pulumi logs"
authors: ["luke-hoban"]
tags: ["AWS","features","Logging"]
date: "2019-04-02"
meta_desc: "The Pulumi CLI provides a seamless way to do logging for your applications without requiring the additional setup of cloud and third party logging solutions."
meta_image: "terminal-logs.png"
---
Pulumi makes developing and deploying
[rich serverless and container-based applications](/blog/easy-serverless-apps-and-infrastructure-real-events-real-code/)
a breeze. But how do you monitor and observe those applications while
they are being developed and once they are deployed? There are many
great answers: from the built-in capabilities of the underlying cloud
services (Lambda, ECS, Kubernetes, and more), to great 3rd party
solutions like [IOpipe](https://github.com/pulumi/pulumi-iopipe) and
[Epsagon](https://github.com/pulumi/pulumi-epsagon) which
[we highlighted recently](/blog/pulumi-and-epsagon-define-deploy-and-monitor-serverless-applications/)
on this blog.
The Pulumi CLI provides another way to do logging, without requiring the
additional setup of these existing solutions and seamlessly integrated
into your Pulumi development workflow. The `pulumi logs` command
provides a great first place to start for understanding your Pulumi
application's behavior. Especially during development, this command
provides direct insight into the behavior of your application, bringing
together logs across all of the different forms of compute you are using -
from code running in serverless functions to containers to VMs.
Let's take a quick look at `pulumi logs` and some of the ways it can be
used as part of the inner loop of your Pulumi development.
<!--more-->
> *Note: Currently `pulumi logs` supports only logs for applications
> running in AWS. In the near future it will be expanded to Kubernetes,
> Azure, GCP and additional cloud platforms.*
## Logging for Inner-Loop Development
When developing locally, it is common to use logging as an interactive
part of the development process. With `pulumi logs`, we can bring some
of these same approaches into the inner loop of the development process
even for applications being actively developed/deployed in the cloud.
Let's take a simple example:
```typescript
import * as aws from '@pulumi/aws'
const bucket = new aws.s3.Bucket("mybucket");
bucket.onObjectCreated("newObject", async (ev) => {
    console.log(JSON.stringify(ev));
});
export const bucketName = bucket.id;
```
We can deploy this app, copy an object into the newly provisioned
bucket, and then see the logs with the following:
$ pulumi up --skip-preview
 
$ aws s3 cp somefile.txt s3://$(pulumi stack output bucketName)
 
$ pulumi logs -f
Collecting logs for stack dev since
 
 2019-03-28T21:10:05.667-07:00[ newObject] {"Records":[{"eventVersion":"2.1","eventSource":"aws:s3","awsRegion":"us-west-2","eventTime":"2019-03-29T04:10:05.318Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"AWS:AIDAIB2I7NPKMQ2QAKXZ6"},"requestParameters":{"sourceIPAddress":"24.35.94.249"},"responseElements":{"x-amz-request-id":"F2D7DF26693AEFEE","x-amz-id-2":"o2leqNNwCVXTI8jfvUGk5KE7w6CiKsJYP/Q4nT0fHqZPrqwaVlSP0lNhspv5NiLAAwutCzJwlLc="},"s3":{"s3SchemaVersion":"1.0","configurationId":"tf-s3-lambda-20190328222809603300000002","bucket":{"name":"mybucket-97f10e0","ownerIdentity":{"principalId":"A3FZDZNSWOKQHV"},"arn":"arn:aws:s3:::mybucket-97f10e0"},"object":{"key":"out.txt","size":36000,"eTag":"818ac0cf0273f7e40e673d9892405e31","sequencer":"005C9D9A9D311D5C11"}}}]}
I can use this to quickly see what kind of values I get back on an S3
Bucket Object event, using live data running in the cloud. But I can do
this whole thing in just a couple of seconds - from deployment to logs.
This lets me iterate on my application live in the cloud with ease.
If there are errors in my code, I find out about them immediately:
```typescript
bucket.onObjectCreated("newObject", async (ev, ctx) => {
    throw new Error("oops");
});
```
Results in:
$ pulumi logs --since 1m
Collecting logs for stack dev since 2019-03-28T20:17:28.000-07:00.
 
 2019-03-28T21:17:01.269-07:00[ newObject] {"errorMessage":"oops","errorType":"Error","stackTrace":["/var/task/__index.js:26:11","Generator.next (<anonymous>)","/var/task/__index.js:12:71","new Promise (<anonymous>)","Object.<anonymous> (/var/task/__index.js:8:12)","Object.__f1 [as __awaiter] (/var/task/__index.js:17:34)","/var/task/__index.js:24:21","__f0 (/var/task/__index.js:30:34)"]}
## Aggregating Logs
As applications grow to span multiple forms of compute -- such as micro
services, lambdas, containers, and VMs -- it becomes important to be
able to correlate logs across services.
One great example of this is the [video thumbnailer](https://github.com/pulumi/examples/tree/master/aws-ts-thumbnailer)
example which implements this simple architecture involving two buckets,
a lambda event, and a long-running Fargate container task.
![unnamed](./unnamed.png)
The Pulumi program deploys several components, using different types of
cloud resources. But using `Pulumi logs` we can see all logs in a
single, unified view.
![terminal-logs](./terminal-logs.png)
From the image above, you can see the `onNewVideo` Lambda function, then
triggering the `ffmpegThumbTask` Fargate container, and then copying a
file back into S3 triggering the `onNewThumbtail` Lambda function. So
you don't need to jump around between different systems for viewing
logs, you can get a consistent view of what's going on in your cloud
application!
Underneath the hood, these logs live in different and uncorrelated
CloudWatch LogGroups, spread across an AWS account. But with
`pulumi logs` we can get the application-centric view we need to
understand the overall behavior of our application architecture.
## Controlling Log Rendering
The `pulumi logs` command supports a few options for controlling what
gets logged.
The `--since` flag retrieves logs only since a given relative or
absolute time. For example, `pulumi logs --since 1m` will get logs just
since 1 minute ago - useful for limiting logs to only what's happened
very recently - especially during inner-loop development. And to get
logs since a specific point in time (such as the timestamp of a known
event from a previous execution of `pulumi logs`) you can use
`pulumi logs --since 2019-03-28T21:17:01.269-07:00`.
A couple of the examples above used the `--follow` flag to follow the
log output until cancelled. This is useful when waiting for log output
to come in during debugging.
And for applications with many resources, it is often important to scope
results down to one or more specific resources, which can be done with
the `--resource` flag. For example, logs for just the Fargate task above
could be retrieved with `pulumi logs --resource ffmpegThumbTask`.
## Working Seamlessly with Your Cloud Provider
Like many other Pulumi features, `pulumi logs` works by building on top
of the great building blocks provided by the underlying cloud provider
services. Instead of adding new runtime logging infrastructure to your
application, `pulumi logs` lets you easily extract out logs directly
from your existing infrastructure. But because Pulumi understands the
natural application scope for what resources are part of a single
application, it can bring together the logs from all of those resources
into one place, instead of having to think about individual resources.
Behind the scenes, `pulumi logs` makes the same calls to CloudWatch Logs
or other backend log providers that you would make yourself via the
`aws` CLI or AWS Console. But you don't have to think about the physical
identities of the cloud resources themselves - Pulumi takes care of
finding the resources with supported logs, collecting those logs based
on the requested filters, and then presenting them in unified logical
view.
## What's Next?
We've seen Pulumi users working with Lambda, ECS and other AWS compute
offerings become heavy users of `pulumi logs` during development. But
supporting a broad set of cloud providers is key to our mission at
Pulumi - so we really want to bring these benefits to *all* Pulumi
users. We'll be expanding `pulumi logs` to Kubernetes, Azure and GCP in
the near future - with easy support for plugging in other cloud
providers and cloud resources as sources for log data.
Even better, because Pulumi allows you to work with multiple cloud
providers at once - like AWS and Kubernetes, or GCP and Azure, or even
two different AWS regions - `pulumi logs` will be able to aggregate logs
across multiple cloud providers and offer a seamless application-centric
view of your logs.
Users of the Pulumi Service at <https://app.pulumi.com> also would love
to have access to these logs at the click of a button - and this is
something we're also excited to enable in the near future.
You can take `pulumi` for a spin today by checking out the
[Getting Started Guide](/docs/get-started/) guide or some of the
[Pulumi Examples](https://github.com/pulumi/examples) on GitHub. Then join us in the
[Pulumi Community Slack](https://slack.pulumi.com) to chat about where
you want to see us go (and where you'd like to help out!) with
`pulumi logs` and other features of the Pulumi open source project in
the coming weeks and months.