Skip to content

Commit 65f14ad

Browse files
committed
Documentation content update
1 parent b08f722 commit 65f14ad

File tree

5 files changed

+88
-15
lines changed

5 files changed

+88
-15
lines changed
Loading
Loading
Binary file not shown.
Binary file not shown.

content/arduino-cloud/09.business/02.aws-s3-exporter/content.md

+88-15
Original file line numberDiff line numberDiff line change
@@ -27,14 +27,26 @@ The **Arduino AWS S3 CSV Exporter** is designed to extract time series data from
2727

2828
## Required Software
2929

30-
* [Arduino Cloud](https://cloud.arduino.cc/). If you do not have an account, you can create one for free inside [cloud.arduino.cc](https://cloud.arduino.cc/home/?get-started=true).
30+
* [Arduino Cloud](https://cloud.arduino.cc/). **If you do not have an account, you can create one for free inside [cloud.arduino.cc](https://cloud.arduino.cc/home/?get-started=true)**.
3131
* [AWS CLI](https://aws.amazon.com/cli/)
3232
* [Go Programming Language](https://go.dev/) (version 1.22 or higher)
3333
* [Official Arduino AWS S3 CSV Exporter Repository](https://github.com/arduino/aws-s3-integration)
3434

35+
## How It Works
36+
37+
The **Arduino AWS S3 CSV Exporter** extracts time series data from **Arduino Cloud** and publishes it to an **AWS S3** bucket. Data is extracted at a specified resolution using a **GO based AWS Lambda** function triggered periodically by **AWS EventBridge**.
38+
39+
Each function execution generates a CSV file containing samples from the selected **Arduino Things**, which are then uploaded to **S3** for data storage and management.
40+
41+
Data is extracted every hour by default, with samples aggregated at a 5 minute resolution. Both the extraction period and the aggregation rate are configurable.
42+
43+
Aggregation is performed as an average over the aggregation period, and non-numeric values, such as strings, are sampled at the specified resolution. Time series data is exported in **UTC** by default, and all Arduino Things in the account are exported unless filtered using [*tags*](#tag-filtering).
44+
45+
This setup allows you to easily manage and store time series data from connected devices, offering flexibility with configurable parameters like sampling intervals and data filtering.
46+
3547
## Deployment Using CloudFormation Template
3648

37-
The exporter setup involves deploying resources using a CloudFormation template. Ensure your AWS account has permissions for:
49+
The exporter setup involves deploying resources using a [**CloudFormation template**](https://github.com/arduino/aws-s3-integration/blob/0.3.0/deployment/cloud-formation-template/deployment.yaml). Ensure your AWS account has permissions for:
3850

3951
* CloudFormation stack creation (policy: `AWSCloudFormationFullAccess`)
4052
* S3 bucket management (policy: `AmazonS3FullAccess`)
@@ -43,17 +55,38 @@ The exporter setup involves deploying resources using a CloudFormation template.
4355
* EventBridge rule configuration (policy: `AmazonEventBridgeFullAccess`)
4456
* Parameter management in SSM (policy: `AmazonSSMFullAccess`)
4557

46-
### Setting Up
58+
## Pre-Requisite
59+
60+
Before continuing and creating the CloudFormation stack, two S3 buckets need to be created:
61+
62+
- **Temporary bucket**: This is where the Lambda binaries and **CloudFormation template (CFT)** will be uploaded.
63+
- **CSV destination bucket**: This is where all generated CSV files will be uploaded. Ensure this bucket is in the same AWS region where the stack will be created.
64+
65+
## Setting Up
66+
67+
Download the [Lambda binaries](https://github.com/arduino/aws-s3-integration/releases) (`.zip` file) and the [CloudFormation template](https://github.com/arduino/aws-s3-integration/releases) (`.yaml` file).
4768

48-
Get the [Lambda binaries](https://github.com/arduino/aws-s3-integration/releases) (`.zip` file) and the [CloudFormation template](https://github.com/arduino/aws-s3-integration/releases) (`.yaml` file).
69+
Upload the binaries and the template to an accessible S3 bucket. Note the following object URL for use in the stack creation process:
4970

50-
Upload the binaries and the template to an accessible S3 bucket. Note the object URL for use in the stack creation process.
71+
```bash
72+
https://arduino-s3-data-exporter-deployment.s3.amazonaws.com/deployment.yaml
73+
```
74+
75+
The **Object URL** is required for the **Amazon S3 URL** field within the stack creation.
76+
77+
Use the CloudFormation console to create a new stack.
78+
79+
![Stack creation](assets/cft-stack-1.png)
5180

52-
Use the CloudFormation console to create a new stack, entering the required parameters:
81+
### Stack Parameters
82+
83+
Enter the following parameters required for creating the stack:
5384

5485
- **Mandatory:** Arduino API key and secret, the S3 bucket for code, and the destination S3 bucket.
5586
- **Optional:** Tag filter, organization ID, and data resolution settings.
5687

88+
![Stack parameters](assets/cft-stack-2.png)
89+
5790
## Configuration Parameters
5891

5992
Below are the supported configuration parameters that are editable in the AWS Parameter Store. These parameters are pre-filled during stack creation but can be modified later:
@@ -72,45 +105,85 @@ Below are the supported configuration parameters that are editable in the AWS Pa
72105

73106
## Tag Filtering
74107

75-
You can filter devices in the Arduino Cloud using tags. Add tags in the Arduino Cloud UI under the Metadata section of each device.
108+
To export specific Arduino Things from the Arduino Cloud, you can apply **tag filtering**.
109+
110+
**Tags** can be added in the Arduino Cloud under the **Metadata** section of each device, referred to as **Things**.
111+
112+
![Things Metadata](assets/cloud_tag_2.png)
113+
114+
When you click on **ADD**, it will ask you to provide a **key** and its **value** for the tag.
115+
116+
![Things Metadata Tag](assets/cloud_tag_1.png)
76117

77118
During CloudFormation stack creation, configure tag filters using:
78119

79120
```bash
80121
/arduino/s3-exporter/{stack-name}/iot/filter/tags
81122
```
82123

124+
![Tag Filter from CFT Creation](assets/tag-filter.png)
125+
83126
## CSV File Structure
84127

85-
The exported CSV files have the following format:
128+
The CSV files generated by the exporter follow this structure:
86129

87-
```bash
130+
```csv
88131
timestamp,thing_id,thing_name,property_id,property_name,property_type,value,aggregation_statistic
89132
```
90133

91-
Files are organized by date, and the naming follows a structured format for easy identification:
134+
The following is an example of how the CSV files would look with data:
135+
136+
```
137+
2024-09-04T11:00:00Z,07846f3c-37ae-4722-a3f5-65d7b4449ad3,H7,137c02d0-b50f-47fb-a2eb-b6d23884ec51,m3,FLOAT,3,AVG
138+
2024-09-04T11:01:00Z,07846f3c-37ae-4722-a3f5-65d7b4449ad3,H7,137c02d0-b50f-47fb-a2eb-b6d23884ec51,m3,FLOAT,7,AVG
139+
```
140+
141+
Files are organized by date and time stamp, with a structured naming convention for easy identification:
92142

93143
```bash
94144
<bucket>:2024-09-04/2024-09-04-10-00.csv
95145
<bucket>:2024-09-04/2024-09-04-11-00.csv
146+
<bucket>:2024-09-04/2024-09-04-12-00.csv
96147
```
97148

98149
## Building the Code
99150

100-
Ensure [**Go (version 1.22)**](https://go.dev/) is installed to build the exporter locally. The core code can be built using:
151+
Ensure that at least [**Go version 1.22**](https://go.dev/) is installed to build the exporter locally. The core code can be built using the following command:
101152

102153
```bash
103154
./compile-lambda.sh
104155
```
105156

106-
This creates a **`arduino-s3-integration-lambda.zip`** file. Alternatively, run:
157+
This creates a **`arduino-s3-integration-lambda.zip`** file. Alternatively, you can run the following command to build the exporter:
107158

108159
```bash
109160
task go:build
110161
```
111162

163+
## Time Alignment
164+
165+
The data extraction is aligned with the function's execution time.
166+
167+
If required, the extraction can be configured to align with specific time windows by adjusting the following parameter:
168+
169+
```bash
170+
/arduino/s3-exporter/{stack-name}/iot/align_with_time_window
171+
```
172+
112173
## Additional Documentation
113174

114-
* [Arduino Cloud Documentation](https://docs.arduino.cc/cloud/iot-cloud)
115-
* [AWS Lambda Documentation](https://docs.aws.amazon.com/lambda/)
116-
* [Go Programming Language Documentation](https://go.dev/doc/)
175+
To help you get the most out of the exporter, the following documentation resources are recommended for your reference:
176+
177+
* [**Arduino Cloud Documentation**](https://docs.arduino.cc/cloud/iot-cloud): Here you can find information about setting up and managing **Arduino Cloud** projects, including device management, data collection, and integration with external services.
178+
179+
* [**Go Programming Language Documentation**](https://go.dev/doc/): The Go programming language is used to build the exporter. Visit the official Go documentation for guides, tutorials, and reference material to help you set up and build Go projects.
180+
181+
* [**AWS Lambda Documentation**](https://docs.aws.amazon.com/lambda/): You can explore the official AWS Lambda documentation here to learn more about building, deploying, and managing Lambda functions.
182+
183+
* [**AWS S3 Documentation**](https://docs.aws.amazon.com/s3/): Learn about **Amazon S3**, known as Simple Storage Service, where CSV files are stored. Here you can find information on S3 features, storage management, and security best practices.
184+
185+
* [**AWS CloudFormation Documentation**](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html): CloudFormation is used to deploy the exporter, and you can find more information about AWS CloudFormation here.
186+
187+
## Conclusion
188+
189+
In this tutorial, you have learned to use the **Arduino AWS S3 CSV Exporter** to capture time series data from **Arduino Cloud** and store it in **AWS S3** for management and analysis. With configurable options for aggregation intervals, tag filtering, and data compression, the exporter offers flexibility for various project needs. Following this tutorial, you can deploy and configure the exporter using a CloudFormation template, making it a useful tool for integrating cloud based data storage into your IoT projects.

0 commit comments

Comments
 (0)