Skip to content

Commit a877db2

Browse files
julianwoodjulianwood
authored
Merge pull request #52 from hariohmprasath/main
Centralized logging with Kinesis Firehose using Lambda Extensions
2 parents f4b218f + 35d9f01 commit a877db2

File tree

19 files changed

+1230
-1
lines changed

19 files changed

+1230
-1
lines changed

‎.gitignore

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,7 @@ out/
7878
# mpeltonen/sbt-idea plugin
7979
.idea_modules/
8080

81+
8182
# JIRA plugin
8283
atlassian-ide-plugin.xml
8384

@@ -227,4 +228,8 @@ csharp-*/*.zip
227228
*/.settings/
228229

229230
# SAM
230-
.aws-sam/
231+
.aws-sam/
232+
go-example-logs-api-extension/.idea/vcs.xml
233+
go-example-logs-api-extension/README.md
234+
kinesisfirehose-logs-extension-demo/run.sh
235+
kinesisfirehose-logs-extension-demo/bin/extensions/go-example-logs-api-extension

‎README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,8 @@ In this repository you will find a number of demos and sample projects from AWS
5555

5656
* [Demo: Logs to Amazon S3 extension: container image ](s3-logs-extension-demo-container-image/): Demo logs extension to receive logs directly from Lambda and send them to S3. This example packages the extension and function as separate container images. The demo is deployed using AWS SAM.
5757

58+
* [Demo: Logs to Kinesis firehose Logs API extension](kinesisfirehose-logs-extension-demo/): How to get a basic logs API extension for Amazon Kinesis Data Firehose written in Go. The extension explains the overall approach to streamline and centralize log collection using Amazon Kinesis Data firehose. The extension runs a local HTTP listener and subscribes to a stream of function and platform logs using the Logs API. It buffers them and sends them to Amazon Kinesis Data firehose periodically, which streams logs to Amazon S3. The demo gets deployed using AWS SAM.
59+
5860
### Other extension examples
5961

6062
* [Adaptive Batching extension in Go](go-example-adaptive-batching-extension/): How to use the Logs API extension written in Go to adaptively batch log data to a destination (S3). The logs are batched up and stored by the extension. Three environment variables with default values dictate when the logs are send to the destination. If any of the conditions are met, the logs queued will be shipped to S3.

‎kinesisfirehose-logs-extension-demo/Makefile

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
build:
2+
GOOS=linux GOARCH=amd64 go build -o bin/extensions/kinesisfirehose-logs-extension-demo main.go
3+
4+
build-KinesisFireHoseLogsApiExtensionLayer:
5+
GOOS=linux GOARCH=amd64 go build -o $(ARTIFACTS_DIR)/extensions/kinesisfirehose-logs-extension-demo main.go
6+
chmod +x $(ARTIFACTS_DIR)/extensions/kinesisfirehose-logs-extension-demo
7+
8+
run-KinesisFireHoseLogsApiExtensionLayer:
9+
go run kinesisfirehose-logs-extension-demo/main.go

‎kinesisfirehose-logs-extension-demo/README.md

Lines changed: 225 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,225 @@
1+
# Centralize log collection with Amazon Kinesis Data Firehose using Lambda Extensions
2+
3+
## Introduction
4+
5+
This example show how to centralize log collection for a Lambda function using Kinesis Data Firehose. The provided code sample uses Lambda extensions to receive logs from Lambda and send them directly to Kinesis Data Firehose without sending them to Amazon CloudWatch service.
6+
7+
> Note: This is a simple example extension to help you investigate an approach to centralize the log aggregation. This example code is not production ready. Use it with your own discretion after testing thoroughly.
8+
9+
This sample extension:
10+
11+
* Subscribes to receive `platform` and `function` logs.
12+
* Runs with a main, and a helper goroutine: The main goroutine registers to `ExtensionAPI` and process its `invoke` and `shutdown` events. The helper goroutine:
13+
* starts a local HTTP server at the provided port (default 1234, the port can be overridden with Lambda environment variable `HTTP_LOGS_LISTENER_PORT` ) that receives requests from Logs API with `NextEvent` method call
14+
* puts the logs in a synchronized queue (Producer) to be processed by the main goroutine (Consumer)
15+
* The main goroutine writes the received logs to Amazon Kinesis Data Firehose, which gets stored in Amazon S3
16+
17+
## Amazon Kinesis Data Firehose
18+
19+
Amazon Kinesis Data Firehose is the easiest way to reliably load streaming data into data lakes, data stores, and analytics services. It can capture, transform, and deliver streaming data to Amazon S3, Amazon Redshift, Amazon Elasticsearch Service, generic HTTP endpoints, and service providers like Datadog, New Relic, MongoDB, and Splunk, read more about it [here](https://aws.amazon.com/kinesis/data-firehose)
20+
21+
> Note: The code sample provided part of this pattern delivers logs from Kinesis Data Firehose to Amazon S3
22+
23+
## Lambda extensions
24+
25+
Lambda Extensions, a new way to easily integrate Lambda with your favorite monitoring, observability, security, and governance tools. Extensions are a new way for tools to integrate deeply into the Lambda environment. There is no complex installation or configuration, and this simplified experience makes it easier for you to use your preferred tools across your application portfolio today. You can use extensions for use-cases such as:
26+
27+
* capturing diagnostic information before, during, and after function invocation
28+
* automatically instrumenting your code without needing code changes
29+
* fetching configuration settings or secrets before the function invocation
30+
* detecting and alerting on function activity through hardened security agents, which can run as separate processes from the function
31+
32+
read more about it [here](https://aws.amazon.com/blogs/compute/introducing-aws-lambda-extensions-in-preview/)
33+
34+
> Note: The code sample provided part of this pattern uses **external** extension to listen to log events from the Lambda function
35+
36+
## Need to centralize log collection
37+
38+
Having a centralized log collecting mechanism using Kinesis Data Firehose provides the following benefits:
39+
40+
* Helps to collect logs from different sources in one place. Even though the sample provided sends logs from Lambda, log routers like `Fluentbit` and `Firelens` can send logs directly to Kinesis Data Firehose from container orchestrators like ["Amazon Elastic Kubernetes Service (EKS)"](https://aws.amazon.com/eks) and ["Amazon Elastic Container Service (ECS)"](https://aws.amazon.com/ecs)
41+
* Define and standardize the transformations before the log gets delivered to downstream systems like S3, elastic search, redshift, etc
42+
* Provides a secure storage area for log data, before it gets written out to the disk. In the event of machine/application failure, we still have access to the logs emitted from the source machine/application
43+
44+
## Architecture
45+
46+
### AWS Services
47+
48+
* AWS Lambda
49+
* AWS Lambda extension
50+
* Amazon Kinesis Data Firehose
51+
* Amazon S3
52+
53+
### High level architecture
54+
55+
Here is the high level view of all the components
56+
57+
![architecture](images/centralized-logging.svg)
58+
59+
Once deployed the overall flow looks like below:
60+
61+
* On start-up, the extension subscribes to receive logs for `Platform` and `Function` events.
62+
* A local HTTP server is started inside the external extension which receives the logs.
63+
* The extension also takes care of buffering the recieved log events in a synchronized queue and writing it to AWS Kinesis Firehose via direct `PUT` records
64+
65+
> Note: Kinesis Data Firehose stream name gets specified as an environment variable (`AWS_KINESIS_STREAM_NAME`)
66+
67+
* The Lambda function won't be able to send any logs events to Amazon CloudWatch service due to the following explicit `DENY` policy:
68+
69+
```yaml
70+
Sid: CloudWatchLogsDeny
71+
Effect: Deny
72+
Action:
73+
- logs:CreateLogGroup
74+
- logs:CreateLogStream
75+
- logs:PutLogEvents
76+
Resource: arn:aws:logs:*:*:*
77+
```
78+
79+
* The Kinesis Data Firehose stream configured part of this sample sends log directly to `AWS S3` (gzip compressed).
80+
81+
## Build and Deploy
82+
83+
AWS SAM template available part of the root directory can be used for deploying the sample Lambda function with this extension
84+
85+
### Pre-requistes
86+
87+
* AWS SAM CLI needs to get installed, follow the [link](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html) to learn how to install them
88+
89+
### Build
90+
91+
Check out the code by running the following command:
92+
93+
```bash
94+
git clone https://github.com/aws-samples/aws-lambda-extensions.git
95+
cd aws-lambda-extensions/kinesisfirehose-logs-extension-demo
96+
```
97+
98+
Run the following command from the root directory
99+
100+
```bash
101+
sam build
102+
```
103+
104+
**Output**
105+
106+
```bash
107+
Building codeuri: /Users/xxx/CodeBase/aws-lambda-extensions/kinesisfirehose-logs-extension-demo/hello-world runtime: nodejs12.x metadata: {} functions: ['HelloWorldFunction']
108+
Running NodejsNpmBuilder:NpmPack
109+
Running NodejsNpmBuilder:CopyNpmrc
110+
Running NodejsNpmBuilder:CopySource
111+
Running NodejsNpmBuilder:NpmInstall
112+
Running NodejsNpmBuilder:CleanUpNpmrc
113+
Building layer 'KinesisFireHoseLogsApiExtensionLayer'
114+
Running CustomMakeBuilder:CopySource
115+
Running CustomMakeBuilder:MakeBuild
116+
Current Artifacts Directory : /Users/xxx/CodeBase/aws-lambda-extensions/kinesisfirehose-logs-extension-demo/.aws-sam/build/KinesisFireHoseLogsApiExtensionLayer
117+
118+
Build Succeeded
119+
120+
Built Artifacts : .aws-sam/build
121+
Built Template : .aws-sam/build/template.yaml
122+
123+
Commands you can use next
124+
=========================
125+
[*] Invoke Function: sam local invoke
126+
[*] Deploy: sam deploy --guided
127+
```
128+
129+
### Deployment
130+
131+
Run the following command to deploy the sample Lambda function with the extension
132+
133+
```bash
134+
sam deploy --guided
135+
```
136+
137+
The following parameters can be customized part of the deployment
138+
139+
| Parameter | Description | Default |
140+
| ------------- | ------------- | ----|
141+
| FirehoseStreamName | Firehose stream name | lambda-logs-direct-s3-no-cloudwatch |
142+
| FirehoseS3Prefix | The S3 Key prefix for Kinesis Firehose | lambda-logs-direct-s3-no-cloudwatch |
143+
| FirehoseCompressionFormat | Compression format used by Kinesis Firehose, allowed value - `UNCOMPRESSED, GZIP, Snappy` | GZIP |
144+
| FirehoseBufferingInterval | How long Firehose will wait before writing a new batch into S3 | 60 |
145+
| FirehoseBufferingSize | Maximum batch size in MB | 10 |
146+
147+
> Note: We can either customize the parameters, or leave it as default to proceed with the deployment
148+
149+
**Output**
150+
151+
```bash
152+
CloudFormation outputs from deployed stack
153+
-------------------------------------------------------------------------------------------------------------------
154+
Outputs
155+
-------------------------------------------------------------------------------------------------------------------
156+
Key KinesisFireHoseLogsApiExtensionLayer
157+
Description Kinesis Log emiter Lambda Extension Layer Version ARN
158+
Value arn:aws:lambda:us-east-1:xxx:layer:kinesisfirehose-logs-extension-demo:5
159+
160+
Key BucketName
161+
Description The bucket where data will be stored
162+
Value sam-app-deliverybucket-xxxx
163+
164+
Key KinesisFireHoseIamRole
165+
Description Kinesis firehose IAM role
166+
Value arn:aws:firehose:us-east-1:xxx:deliverystream/lambda-logs-direct-s3-no-cloudwatch
167+
168+
Key HelloWorldFunction
169+
Description First Lambda Function ARN
170+
Value arn:aws:lambda:us-east-1:xxx:function:kinesisfirehose-logs-extension-demo-function
171+
-------------------------------------------------------------------------------------------------------------------
172+
```
173+
174+
## Testing
175+
176+
You can invoke the Lambda function using the [Lambda Console](https://console.aws.amazon.com/lambda/home), or the following CLI command
177+
178+
```bash
179+
aws lambda invoke \
180+
--function-name "<<function-name>>" \
181+
--payload '{"payload": "hello"}' /tmp/invoke-result \
182+
--cli-binary-format raw-in-base64-out \
183+
--log-type Tail
184+
```
185+
186+
>Note: Make sure to replace `function-name` with the actual Lambda function name
187+
188+
The function should return ```"StatusCode": 200```, with the below output
189+
190+
```bash
191+
{
192+
"StatusCode": 200,
193+
"LogResult": "<<Encoded>>",
194+
"ExecutedVersion": "$LATEST"
195+
}
196+
```
197+
198+
In a few minutes after the successful invocation of the Lambda function, you should see the log messages from the example extension sent to Amazon Kinesis Data Firehose which sends the messages to a Amazon S3 bucket.
199+
200+
* Login to AWS console:
201+
* Navigate to the S3 bucket mentioned under the parameter `BucketName` in the SAM output.
202+
* We can see the logs successfully written to the S3 bucket, partitioned based on date in `GZIP` format.
203+
![s3](images/S3.png)
204+
205+
* Navigate to `"/aws/lambda/${functionname}"` log group inside AWS CloudWatch service.
206+
* We shouldn't see any logs created under this log group as we have denied access to write any logs from the Lambda function.
207+
![cloudwatch](images/CloudWatch.png)
208+
209+
## Cleanup
210+
211+
Run the following command to delete the stack, use the correct stack names if you have changed them during sam deploy
212+
213+
```bash
214+
aws cloudformation delete-stack --stack-name sam-app
215+
```
216+
217+
## Resources
218+
219+
* [Using AWS Lambda extensions to send logs to custom destinations](https://aws.amazon.com/blogs/compute/using-aws-lambda-extensions-to-send-logs-to-custom-destinations/)
220+
* [Ingest streaming data into Amazon Elasticsearch Service within the privacy of your VPC with Amazon Kinesis Data Firehose](https://aws.amazon.com/blogs/big-data/ingest-streaming-data-into-amazon-elasticsearch-service-within-the-privacy-of-your-vpc-with-amazon-kinesis-data-firehose/)
221+
* [Example Logs API Extension in Go](https://github.com/aws-samples/aws-lambda-extensions/tree/main/go-example-logs-api-extension).
222+
223+
## Conclusion
224+
225+
This extension provides an approach to streamline and centralize log collection using Kinesis Data Firehose.

0 commit comments

Comments
 (0)