Skip to content

Commit bbd2097

Browse files
simran029simran029
authored
Merge pull request #102 from simran029/master
Added example to use EMF Firelens for ECS Fargate
2 parents 11369ac + 97739af commit bbd2097

File tree

6 files changed

+205
-7
lines changed

6 files changed

+205
-7
lines changed

‎examples/README.md

+47-2
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ Run the example:
2929
./examples/agent/bin/run.sh
3030
```
3131

32-
## FireLens on ECS
32+
## FireLens on ECS EC2
3333

3434
You can deploy the example by running the following:
3535

@@ -45,7 +45,7 @@ aws s3api create-bucket --bucket <bucket-name> --region <region>
4545
# create ECS service
4646

4747
# deploy
48-
./examples/ecs-firelens/publish.sh \
48+
./examples/ecs-firelens/bin/publish.sh \
4949
<account-id> \
5050
<region> \
5151
<image-name> \
@@ -54,3 +54,48 @@ aws s3api create-bucket --bucket <bucket-name> --region <region>
5454
<ecs-task-family> \
5555
<ecs-service-name>
5656
```
57+
58+
## FireLens on ECS Fargate
59+
60+
For running on Fargate, s3 file option is not supported for Fluent-bit config. Hence, we need to build the fluent-bit custom config image and then use its reference in our Firelens container definition.
61+
62+
For building the custom fluent-bit image, clone the [amazon-ecs-firelens-examples](https://github.com/aws-samples/amazon-ecs-firelens-examples) and modify the contents of [extra.conf](https://github.com/aws-samples/amazon-ecs-firelens-examples/blob/mainline/examples/fluent-bit/config-file-type-file/extra.conf) in the amazon-ecs-firelens-examples repository. Post this run the following commands to build the custom fluent-bit image:-
63+
64+
```sh
65+
# create an ECR repository for the Fluentbit-config image
66+
aws ecr create-repository --repository-name <config-image-name> --region <region>
67+
68+
# Navigate to the config file directory
69+
cd examples/fluent-bit/config-file-type-file
70+
71+
# Build the docker image from Dockerfile. Replace config-image-name with your image name
72+
docker build -t <config-image-name> .
73+
74+
# Tag the recently built docker image . Replace the config-image-name, account-id and region with your values.
75+
docker tag <config-image-name>:latest <account-id>.dkr.ecr.<region>.amazonaws.com/<config-image-name>:latest
76+
77+
# Push the docker image to ECR
78+
docker push <account-id>.dkr.ecr.<region>.amazonaws.com/<config-image-name>:latest
79+
```
80+
81+
For executing EMF application on Fargate, you need to execute the following commands :-
82+
83+
```sh
84+
# create an ECR repository for the example image
85+
aws ecr create-repository --repository-name <image-name> --region <region>
86+
87+
# create ECS cluster
88+
# create ECS task definition
89+
# create ECS service
90+
91+
# deploy
92+
./examples/ecs-firelens/bin/publish-fargate.sh
93+
<account-id> \
94+
<region> \
95+
<example-image> \
96+
<fargate-config-image> \
97+
<ecs-cluster> \
98+
<ecs-task> \
99+
<ecs-service>
100+
101+
```

‎examples/ecs-firelens/Dockerfile

+3-2
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
1-
FROM openjdk:8-jdk-slim
1+
#Make sure the jdk pulled in dockerfile matches your IDE compile version used for compiling Jar file
2+
FROM openjdk:11
23
RUN mkdir -p /app
34

45
# copy the source files over
56
COPY build/libs/*.jar /app/app.jar
67

78
ENV JAVA_OPTS=""
8-
ENTRYPOINT [ "sh", "-c", "java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar /app/app.jar" ]
9+
ENTRYPOINT [ "sh", "-c", "java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar /app/app.jar" ]

‎examples/ecs-firelens/bin/publish-fargate.sh

+82
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
#!/usr/bin/env bash
2+
# Usage:
3+
# ./examples/ecs-firelens/bin/publish-fargate.sh \
4+
# <account-id> \
5+
# <region> \
6+
# <image-name> \
7+
# <config-image-name> \
8+
# <ecs-cluster-name> \
9+
# <ecs-task-family> \
10+
# <ecs-service-name>
11+
12+
rootdir=$(git rev-parse --show-toplevel)
13+
14+
ACCOUNT_ID=$1
15+
REGION=$2
16+
IMAGE_NAME=$3 # emf-ecs-firelens
17+
FLUENT_BIT_CONFIG=$4
18+
CLUSTER_NAME=$5 # emf-example
19+
ECS_TASK_FAMILY=$6 # aws-emf-ecs-app-example
20+
ECS_SERVICE_NAME=$7 # aws-emf-ecs-firelens-ec2
21+
22+
LIB_PATH=$rootdir
23+
EXAMPLE_DIR=$rootdir/examples/ecs-firelens
24+
ECR_REMOTE=$ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com/$IMAGE_NAME
25+
26+
function check_exit() {
27+
last_exit_code=$?
28+
if [ $last_exit_code -ne 0 ];
29+
then
30+
echo "Last command failed with exit code: $last_exit_code."
31+
echo "Exiting."
32+
exit $last_exit_code;
33+
fi
34+
}
35+
36+
echo 'BUILDING THE LOCAL PROJECT'
37+
pushd $rootdir
38+
./gradlew :examples:ecs-firelens:build
39+
check_exit
40+
popd
41+
42+
pushd $EXAMPLE_DIR
43+
pwd
44+
45+
46+
echo 'UPDATING CONTAINER DEFINITIONS'
47+
sed "s/<account-id>/$ACCOUNT_ID/g" $EXAMPLE_DIR/container-definitions-fargate.template.json \
48+
| sed "s/<region>/$REGION/g" \
49+
| sed "s/<Firelens-custom-conf-image>/$FLUENT_BIT_CONFIG/g" \
50+
| sed "s/<image-name>/$IMAGE_NAME/g" \
51+
> $EXAMPLE_DIR/container-definitions.json
52+
check_exit
53+
54+
echo 'BUILDING THE EXAMPLE DOCKER IMAGE'
55+
`aws ecr get-login --no-include-email --region $REGION`
56+
docker build . -t $IMAGE_NAME:latest
57+
check_exit
58+
59+
echo 'PUSHING THE EXAMPLE DOCKER IMAGE TO ECR'
60+
imageid=$(docker images -q $IMAGE_NAME:latest)
61+
docker tag $imageid $ECR_REMOTE
62+
docker push $ECR_REMOTE
63+
check_exit
64+
65+
echo 'UPDATING THE ECS SERVICE'
66+
aws ecs update-service \
67+
--region $REGION \
68+
--cluster $CLUSTER_NAME \
69+
--service $ECS_SERVICE_NAME \
70+
--force-new-deployment \
71+
--task-definition $(aws ecs register-task-definition \
72+
--network-mode awsvpc \
73+
--task-role arn:aws:iam::$ACCOUNT_ID:role/ecsTaskExecutionRole \
74+
--execution-role-arn "arn:aws:iam::$ACCOUNT_ID:role/ecsTaskExecutionRole" \
75+
--region $REGION \
76+
--memory 512 \
77+
--cpu 256 \
78+
--family $ECS_TASK_FAMILY \
79+
--container-definitions "$(cat container-definitions.json)" \
80+
| jq --raw-output '.taskDefinition.taskDefinitionArn' | awk -F '/' '{ print $2 }')
81+
82+
popd

‎examples/ecs-firelens/container-definitions-fargate.template.json

+42
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
[
2+
{
3+
"name": "example",
4+
"image": "<account-id>.dkr.ecr.<region>.amazonaws.com/<image-name>:latest",
5+
"essential": true,
6+
"logConfiguration": {
7+
"logDriver": "awsfirelens",
8+
"options": {
9+
"Name": "cloudwatch",
10+
"region": "<region>",
11+
"log_key": "log",
12+
"log_group_name": "aws-emf-ecs-firelens-example-metrics",
13+
"auto_create_group": "true",
14+
"log_stream_prefix": "emf-",
15+
"retry_limit": "2",
16+
"log_format": "json/emf"
17+
}
18+
}
19+
},
20+
{
21+
"name": "fluent-bit",
22+
"image": "<account-id>.dkr.ecr.<region>.amazonaws.com/<Firelens-custom-conf-image>:latest",
23+
"essential": true,
24+
"firelensConfiguration": {
25+
"type": "fluentbit",
26+
"options": {
27+
"config-file-type": "file",
28+
"config-file-value": "/extra.conf",
29+
"enable-ecs-log-metadata": "false"
30+
}
31+
},
32+
"logConfiguration": {
33+
"logDriver": "awslogs",
34+
"options": {
35+
"awslogs-group": "firelens-container",
36+
"awslogs-region": "<region>",
37+
"awslogs-create-group": "true",
38+
"awslogs-stream-prefix": "firelens"
39+
}
40+
}
41+
}
42+
]

‎examples/ecs-firelens/extra.conf

+26
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# TCP input used for EMF payloads
2+
[INPUT]
3+
Name tcp
4+
Listen 0.0.0.0
5+
Port 25888
6+
Chunk_Size 32
7+
Buffer_Size 64
8+
Format none
9+
Tag emf-${HOSTNAME}
10+
# This tag is used by the output plugin to determine the LogStream
11+
# including the HOSTNAME is a way to increase the number of LogStreams.
12+
# The maximum throughput on a
13+
# single LogStream is 5 MB/s (max 1 MB at max 5 TPS).
14+
# In AWSVPC mode, the HOSTNAME is the ENI private IP
15+
# in bridge mode, the HOSTNAME is the Docker container ID
16+
17+
# Output for EMF over TCP -> CloudWatch
18+
[OUTPUT]
19+
Name cloudwatch
20+
Match emf-*
21+
region us-east-1
22+
log_key log
23+
log_group_name aws-emf-ecs-firelens-example-metrics
24+
log_stream_prefix from-fluent-bit-
25+
auto_create_group true
26+
log_format json/emf

‎examples/ecs-firelens/src/main/java/App.java

+5-3
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,6 @@
2424
import software.amazon.cloudwatchlogs.emf.logger.MetricsLogger;
2525
import software.amazon.cloudwatchlogs.emf.model.Unit;
2626
import sun.misc.Signal;
27-
2827
import java.io.IOException;
2928
import java.io.OutputStream;
3029
import java.net.InetSocketAddress;
@@ -36,9 +35,12 @@ public class App {
3635

3736
public static void main(String[] args) throws Exception {
3837
registerShutdownHook();
39-
40-
int portNumber = 8000;
38+
MetricsLogger logger = new MetricsLogger();
39+
logger.setNamespace("FargateEMF");
40+
logger.putMetric("Latency", 63, Unit.MILLISECONDS);
41+
logger.flush();
4142
HttpServer server = HttpServer.create(new InetSocketAddress(8000), 0);
43+
int portNumber = 8000;
4244
System.out.println("Server started. Listening on " + portNumber);
4345
server.createContext("/", new SimpleHandler());
4446
server.setExecutor(null);

0 commit comments

Comments
 (0)