Skip to content

Export Prometheus telemetry in daemon mode #573

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 39 commits into from
Mar 10, 2020
Merged

Export Prometheus telemetry in daemon mode #573

merged 39 commits into from
Mar 10, 2020

Conversation

rsora
Copy link
Contributor

@rsora rsora commented Jan 31, 2020

This PR implements the basis for a telemetry feature for the CLI in daemon mode.

Rationale:
Enhance the observability of the CLI internals at runtime, as we are proceeding internally to integrate the CLI as a gRPC server in all our services.

Module telemetry
Leverages https://github.com/segmentio/stats library to configure and expose a Prometheus endpoint when activated (in cli/daemon/daemon.go). It also provides helpers for transformation of commands input data in stats.Tags objects, using the installation id and secret generated by the inventory module.

The telemetry activation and endpoint config are available via CLI configuration.

Module repertory (feel free to ask for a name change 😄) inventory
This module leverages an additional Viper instance, in order to create a configuration repository that contains values that may create confusion in the user-facing arduino-cli.yaml file, polluting it.

This is the case for the installation values stored in the inventory.yaml file, that are used to generate server specific UUIDs that help the monitoring tools to perform statistics aggregation, when monitoring multiple CLI gRPC server instances.

The inventory file is created at the CLI first launch and it is stored in the directories.Data folder.

Instrumentation
Code instrumentation is done via standard stats apis for each command (only for the compile and board list in this PR):

  1. Create the set of tags for the command
tags := map[string]string{
		"fqbn":            req.Fqbn,
		"sketchPath":      telemetry.SanitizeSketchPath(req.SketchPath),
		"showProperties":  strconv.FormatBool(req.ShowProperties),
		"preprocess":      strconv.FormatBool(req.Preprocess),
		"buildProperties": strings.Join(req.BuildProperties, ","),
...
                "success":          "true",
  1. Modify the success tag according to each case
tags["success"] = "true"
  1. Push the data to the stats engine
stats.Incr("compile", stats.M(tags)...)

Tests

  • test_inventory_creation: python integ test
  • test_inventory_prometheus_endpoint: python integ test

Notes

@gvarisco gvarisco requested review from gvarisco and removed request for gvarisco February 11, 2020 16:44
@rsora rsora added this to the 0.10.0 milestone Mar 4, 2020
Copy link
Member

@cmaglie cmaglie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some change requests.

I see that prometeus does not keep stats when terminated (so metrics are reset on each run), is this expected?

Copy link
Contributor

@masci masci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left few comments but looks good overall

@rsora rsora requested review from masci and cmaglie March 9, 2020 18:32
)

# we block here until the test function using this fixture has returned
yield runner
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🍰

Copy link
Contributor

@masci masci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@rsora rsora requested a review from cmaglie March 10, 2020 17:47
@rsora rsora merged commit 2791756 into master Mar 10, 2020
@rsora rsora deleted the rsora/stat branch March 10, 2020 18:00
@per1234 per1234 added type: enhancement Proposed improvement topic: code Related to content of the project itself topic: gRPC Related to the gRPC interface labels Oct 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic: code Related to content of the project itself topic: gRPC Related to the gRPC interface type: enhancement Proposed improvement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants