You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/arduino-cloud/09.business/02.aws-s3-exporter/content.md
+88-15
Original file line number
Diff line number
Diff line change
@@ -27,14 +27,26 @@ The **Arduino AWS S3 CSV Exporter** is designed to extract time series data from
27
27
28
28
## Required Software
29
29
30
-
*[Arduino Cloud](https://cloud.arduino.cc/). If you do not have an account, you can create one for free inside [cloud.arduino.cc](https://cloud.arduino.cc/home/?get-started=true).
30
+
*[Arduino Cloud](https://cloud.arduino.cc/). **If you do not have an account, you can create one for free inside [cloud.arduino.cc](https://cloud.arduino.cc/home/?get-started=true)**.
31
31
*[AWS CLI](https://aws.amazon.com/cli/)
32
32
*[Go Programming Language](https://go.dev/) (version 1.22 or higher)
The **Arduino AWS S3 CSV Exporter** extracts time series data from **Arduino Cloud** and publishes it to an **AWS S3** bucket. Data is extracted at a specified resolution using a **GO based AWS Lambda** function triggered periodically by **AWS EventBridge**.
38
+
39
+
Each function execution generates a CSV file containing samples from the selected **Arduino Things**, which are then uploaded to **S3** for data storage and management.
40
+
41
+
Data is extracted every hour by default, with samples aggregated at a 5 minute resolution. Both the extraction period and the aggregation rate are configurable.
42
+
43
+
Aggregation is performed as an average over the aggregation period, and non-numeric values, such as strings, are sampled at the specified resolution. Time series data is exported in **UTC** by default, and all Arduino Things in the account are exported unless filtered using [*tags*](#tag-filtering).
44
+
45
+
This setup allows you to easily manage and store time series data from connected devices, offering flexibility with configurable parameters like sampling intervals and data filtering.
46
+
35
47
## Deployment Using CloudFormation Template
36
48
37
-
The exporter setup involves deploying resources using a CloudFormation template. Ensure your AWS account has permissions for:
49
+
The exporter setup involves deploying resources using a [**CloudFormation template**](https://github.com/arduino/aws-s3-integration/blob/0.3.0/deployment/cloud-formation-template/deployment.yaml). Ensure your AWS account has permissions for:
* Parameter management in SSM (policy: `AmazonSSMFullAccess`)
45
57
46
-
### Setting Up
58
+
## Pre-Requisite
59
+
60
+
Before continuing and creating the CloudFormation stack, two S3 buckets need to be created:
61
+
62
+
-**Temporary bucket**: This is where the Lambda binaries and **CloudFormation template (CFT)** will be uploaded.
63
+
-**CSV destination bucket**: This is where all generated CSV files will be uploaded. Ensure this bucket is in the same AWS region where the stack will be created.
64
+
65
+
## Setting Up
66
+
67
+
Download the [Lambda binaries](https://github.com/arduino/aws-s3-integration/releases) (`.zip` file) and the [CloudFormation template](https://github.com/arduino/aws-s3-integration/releases) (`.yaml` file).
47
68
48
-
Get the [Lambda binaries](https://github.com/arduino/aws-s3-integration/releases) (`.zip` file) and the [CloudFormation template](https://github.com/arduino/aws-s3-integration/releases) (`.yaml` file).
69
+
Upload the binariesand the template to an accessible S3 bucket. Note the following object URL for use in the stack creation process:
49
70
50
-
Upload the binaries and the template to an accessible S3 bucket. Note the object URL for use in the stack creation process.
The **Object URL** is required for the **Amazon S3 URL** field within the stack creation.
76
+
77
+
Use the CloudFormation console to create a new stack.
78
+
79
+

51
80
52
-
Use the CloudFormation console to create a new stack, entering the required parameters:
81
+
### Stack Parameters
82
+
83
+
Enter the following parameters required for creating the stack:
53
84
54
85
-**Mandatory:** Arduino API key and secret, the S3 bucket for code, and the destination S3 bucket.
55
86
-**Optional:** Tag filter, organization ID, and data resolution settings.
56
87
88
+

89
+
57
90
## Configuration Parameters
58
91
59
92
Below are the supported configuration parameters that are editable in the AWS Parameter Store. These parameters are pre-filled during stack creation but can be modified later:
@@ -72,45 +105,85 @@ Below are the supported configuration parameters that are editable in the AWS Pa
72
105
73
106
## Tag Filtering
74
107
75
-
You can filter devices in the Arduino Cloud using tags. Add tags in the Arduino Cloud UI under the Metadata section of each device.
108
+
To export specific Arduino Things from the Arduino Cloud, you can apply **tag filtering**.
109
+
110
+
**Tags** can be added in the Arduino Cloud under the **Metadata** section of each device, referred to as **Things**.
111
+
112
+

113
+
114
+
When you click on **ADD**, it will ask you to provide a **key** and its **value** for the tag.
115
+
116
+

76
117
77
118
During CloudFormation stack creation, configure tag filters using:
78
119
79
120
```bash
80
121
/arduino/s3-exporter/{stack-name}/iot/filter/tags
81
122
```
82
123
124
+

125
+
83
126
## CSV File Structure
84
127
85
-
The exported CSV files have the following format:
128
+
The CSV files generated by the exporter follow this structure:
Files are organized by date and time stamp, with a structured naming convention for easy identification:
92
142
93
143
```bash
94
144
<bucket>:2024-09-04/2024-09-04-10-00.csv
95
145
<bucket>:2024-09-04/2024-09-04-11-00.csv
146
+
<bucket>:2024-09-04/2024-09-04-12-00.csv
96
147
```
97
148
98
149
## Building the Code
99
150
100
-
Ensure [**Go (version 1.22)**](https://go.dev/) is installed to build the exporter locally. The core code can be built using:
151
+
Ensure that at least [**Go version 1.22**](https://go.dev/) is installed to build the exporter locally. The core code can be built using the following command:
101
152
102
153
```bash
103
154
./compile-lambda.sh
104
155
```
105
156
106
-
This creates a **`arduino-s3-integration-lambda.zip`** file. Alternatively, run:
157
+
This creates a **`arduino-s3-integration-lambda.zip`** file. Alternatively, you can run the following command to build the exporter:
107
158
108
159
```bash
109
160
task go:build
110
161
```
111
162
163
+
## Time Alignment
164
+
165
+
The data extraction is aligned with the function's execution time.
166
+
167
+
If required, the extraction can be configured to align with specific time windows by adjusting the following parameter:
*[Go Programming Language Documentation](https://go.dev/doc/)
175
+
To help you get the most out of the exporter, the following documentation resources are recommended for your reference:
176
+
177
+
*[**Arduino Cloud Documentation**](https://docs.arduino.cc/cloud/iot-cloud): Here you can find information about setting up and managing **Arduino Cloud** projects, including device management, data collection, and integration with external services.
178
+
179
+
*[**Go Programming Language Documentation**](https://go.dev/doc/): The Go programming language is used to build the exporter. Visit the official Go documentation for guides, tutorials, and reference material to help you set up and build Go projects.
180
+
181
+
*[**AWS Lambda Documentation**](https://docs.aws.amazon.com/lambda/): You can explore the official AWS Lambda documentation here to learn more about building, deploying, and managing Lambda functions.
182
+
183
+
*[**AWS S3 Documentation**](https://docs.aws.amazon.com/s3/): Learn about **Amazon S3**, known as Simple Storage Service, where CSV files are stored. Here you can find information on S3 features, storage management, and security best practices.
184
+
185
+
*[**AWS CloudFormation Documentation**](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html): CloudFormation is used to deploy the exporter, and you can find more information about AWS CloudFormation here.
186
+
187
+
## Conclusion
188
+
189
+
In this tutorial, you have learned to use the **Arduino AWS S3 CSV Exporter** to capture time series data from **Arduino Cloud** and store it in **AWS S3** for management and analysis. With configurable options for aggregation intervals, tag filtering, and data compression, the exporter offers flexibility for various project needs. Following this tutorial, you can deploy and configure the exporter using a CloudFormation template, making it a useful tool for integrating cloud based data storage into your IoT projects.
0 commit comments