docs(nx-cloud): improve nx agents feature page (#27606)

Author: juristr

docs/blog/2024-03-20-why-speed-matters.md

@@ -54,7 +54,7 @@ With Nx Replay, you can see significant speed improvements in your CI pipelines
 
 [Nx Agents](/ci/features/distribute-task-execution) represent the pinnacle of task distribution optimization, ensuring that tasks are executed as efficiently as possible based on the specific requirements of each change. Some features that make up this effort are:
 
-- [Easy integration with existing providers](/ci/features/distribute-task-execution#cicd-guides)
+- [Easy integration with existing providers](/ci/recipes/set-up)
   - Distribution is handled on the Nx Cloud infrastructure and all you need is a single line. What’s more, all results are played back to your original CI provider script which triggers the Nx Cloud distribution, so that you can make use of the resulting artifacts
 - [Efficient task distribution](/ci/features/dynamic-agents)
   - Save compute resources and reduce costs, minimizing idle time and compute waste

docs/shared/features/distribute-task-execution.md

@@ -3,54 +3,95 @@
 {% youtube
 src="https://youtu.be/XS-exYYP_Gg"
 title="Nx Agents Walkthrough"
- /%}
+/%}
 
-**Nx Agents** let you distribute your CI across many machines with minimal configuration. It also comes with the following features:
+Nx Agents is a **distributed task execution system that intelligently allocates tasks across multiple machines**, optimizing your CI pipeline for speed and efficiency. While using [Nx Affected](/ci/features/affected) and [remote caching (Nx Replay)](/ci/features/remote-cache) can significantly speed up your CI pipeline, you might still encounter bottlenecks as your codebase scales. Combining affected runs and remote caching with task distribution is key to maintaining low CI times. Nx Agents handles this distribution efficiently, avoiding the complexity and maintenance required if you were to set it up manually.
 
-- [Dynamically allocate the number and size of agents](/ci/features/dynamic-agents) based on the size of the PR
-- [Re-run flaky tasks](/ci/features/flaky-tasks) automatically whenever they fail in CI
-- Automatically [split large e2e tasks](/ci/features/split-e2e-tasks) into smaller tasks that can be distributed more efficiently
+![Nx Cloud visualization of how tasks are being distributed with Nx Agents](/shared/features/nx-agents-live-chart.avif)
 
-## Making a Distributed CI Pipeline Is Hard
+Nx Agents offer several key advantages:
 
-The only way to speed up your CI pipeline while still running all the necessary tasks is to distribute those tasks across multiple machines. Unfortunately, doing distribution right is hard to set up and hard to maintain. These are just some concerns you have to account for:
+- **Declarative Configuration:** No maintenance is required as your monorepo evolves, thanks to a declarative setup.
+- **Efficient Task Replay:** By leveraging [remote caching](/ci/features/remote-cache), tasks can be replayed efficiently across machines, enhancing distribution speed.
+- **Intelligent Task Distribution:** Tasks are distributed based on historical run times and dependencies, ensuring correct and optimal execution.
+- **Dynamic Resource Allocation:** Agents are [allocated dynamically based on the size of the PR](/ci/features/dynamic-agents), balancing cost and speed.
+- **Seamless CI Integration:** Easily adopt Nx Agents with your [existing CI provider](/ci/recipes/set-up), requiring minimal setup changes.
+- **Simple Activation:** Enable distribution with just a [single line of code](#enable-nx-agents) in your CI configuration.
 
-- Choose how many machines to set up
-- Set up each machine so that it is ready to execute tasks
-- Ensure that tasks are run in the correct order
-- Copy the output of certain tasks to the machines where those outputs are needed
-- Shut down machines when there are no more tasks to run
-- Shut down all the machines when the whole pipeline hits an error
-- Make sure sensitive information is being handled securely on all machines
+## Enable Nx Agents
 
-And each of these concerns will need to be reconsidered whenever the codebase changes. It would actually be best if they were reconsidered for every PR, because small PRs may not need as much distribution as large PRs.
+To enable task distribution with Nx Agents, make sure your Nx workspace is connected to Nx Cloud. If you haven't connected your workspace to Nx Cloud yet, run the following command:
 
-## Nx Agents Make Distributing Tasks Simple
+```shell
+npx nx connect
+```
 
-Nx Agents take care of all these concerns with a small initial configuration that does not need to be modified as your codebase changes. Your CI pipeline sends your tasks to be run on agent machines that Nx Cloud creates for you. All you need to do is specify how many agents and the type of agent. Then, when the pipeline is finished, your initial CI pipeline will contain all the logs and artifacts as if the tasks all ran on your main CI machine - but completed in a fraction of the time.
+Check out the [connect to Nx Cloud recipe](/ci/intro/connect-to-nx-cloud) for more details.
 
-![Distribute Task Execution with Nx Agents](/shared/images/dte/nx-agents-orchestration-diagram.svg)
+Then, adjust your CI pipeline configuration to **enable task distribution**. If you don't have a CI config yet, you can generate a new one using the following command:
 
-For a more thorough explanation of how Nx Agents optimizes your CI pipeline, read this [guide to parallelization and distribution in CI](/ci/concepts/parallelization-distribution).
+```shell
+npx nx g ci-workflow
+```
 
-## Enable Nx Agents
+The key line in your CI config is the `start-ci-run` command:
 
-To enable task distribution with Nx Agents, there are two requirements:
+```yaml {% fileName=".github/workflows/ci.yml" highlightLines=[13] %}
+name: CI
+...
 
-1. Enable version control system integration. The integrations currently available are [GitHub](/ci/recipes/source-control-integration/github), [GitLab](/ci/recipes/source-control-integration/gitlab), [Bitbucket](/ci/recipes/source-control-integration/bitbucket) and [Azure DevOps](/ci/recipes/source-control-integration/azure-devops). These integrations can be enabled from your [Nx Cloud dashboard](https://nx.app).
-2. Add a single line to your CI pipeline configuration.
+jobs:
+  main:
+    runs-on: ubuntu-latest
+    steps:
+      ...
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
 
-Add the `start-ci-run` command to your CI pipeline configuration after checking out the repository and before installing `node_modules`:
+      - run: pnpm dlx nx-cloud start-ci-run --distribute-on="3 linux-medium-js" --stop-agents-after="build"
 
-```yaml {% fileName=".github/workflows/main.yaml" %}
-# After checkout repository
-- name: Start CI run
-  run: 'npx nx-cloud start-ci-run --distribute-on="8 linux-medium-js"'
-# Before install node_modules
-# Run any nx commands as if running on a single machine
+      # Cache node_modules
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+          cache: 'pnpm'
+      ...
+
+      # Nx Affected runs only tasks affected by the changes in this PR/commit. Learn more: https://nx.dev/ci/features/affected
+      - run: pnpm exec nx affected -t lint test build
 ```
 
-The `--distribute-on` flag instructs Nx Cloud to distribute tasks across 8 agents of type `linux-medium-js`. `linux-medium-js` is the name of the launch template that will be used to provision the agent. Use on of the [default launch templates](https://github.com/nrwl/nx-cloud-workflows/blob/main/launch-templates/linux.yaml) or create your own [custom launch template](/ci/reference/launch-templates).
+This command tells Nx Cloud to:
+
+- Start a CI run (`npx nx-cloud start-ci-run`)
+- Collect all Nx commands that are being issued (e.g., `pnpm exec nx affected -t lint test build`) and
+- Distribute them across 3 agents (`3 linux-medium-js`) where `linux-medium-js` is a predefined agent [launch template](/ci/reference/launch-templates).
+
+### Configure Nx Agents on your CI Provider
+
+Every organization manages their CI/CD pipelines differently, so the guides don't cover org-specific aspects of CI/CD (e.g., deployment). They mainly focus on configuring Nx correctly using Nx Agents and [Nx Replay](/ci/features/remote-cache).
+
+- [Azure Pipelines](/ci/recipes/set-up/monorepo-ci-azure)
+- [Circle CI](/ci/recipes/set-up/monorepo-ci-circle-ci)
+- [GitHub Actions](/ci/recipes/set-up/monorepo-ci-github-actions)
+- [Jenkins](/ci/recipes/set-up/monorepo-ci-jenkins)
+- [GitLab](/ci/recipes/set-up/monorepo-ci-gitlab)
+- [Bitbucket Pipelines](/ci/recipes/set-up/monorepo-ci-bitbucket-pipelines)
+
+## How Nx Agents Work
+
+![Distribute Task Execution with Nx Agents](/shared/images/dte/nx-agents-orchestration-diagram.svg)
+
+_**Nx Agents are declarative**_ in that you only specify the number of agents and the type of agent you want to use. Nx Cloud then picks up the Nx commands that are being issued on your CI and distributes them automatically. This results in **low maintenance and a much more efficient distribution strategy**. A non-declarative approach would be one where you define which tasks or projects get executed on which machine, requiring you to adjust the configuration as your codebase changes.
+
+_**Nx Agents use a task-centric approach**_ to distribution. Current CI systems use VM-centric approaches, where tasks must be predefined for specific machines, often leading to inefficiencies as your codebase grows. Instead of defining which tasks run on which machine upfront, Nx Agents dynamically process tasks based on availability and task dependencies/ordering. Tasks are picked up by agents based on the task's required processing time (from historical data) and task dependency/ordering (from the Nx graph). This results in a faster and resource efficient processing, and is also more resilient to failures since any other agent can pick up work if one agent fails during bootup. Read more [on our blog post](/blog/reliable-ci-a-new-execution-model-fixing-both-flakiness-and-slowness).
+
+_**Nx Agents are cost and resource-efficient**_ as the tasks are automatically distributed - **optimizing for speed while also ensuring resource utilization is high** and idle time is low. You can also [dynamically adjust the number of agents](/ci/features/dynamic-agents) based on the size of the PR, and we're working on [some more AI-powered features](/ci/concepts/nx-cloud-ai) to optimize this even further. In addition, [remote caching](/ci/features/remote-cache) guarantees tasks are not run twice, and artifacts are shared efficiently among agents.
+
+_**Nx Agents are non-invasive**_ in that you don't need to completely overhaul your existing CI configuration or your Nx workspace to use them. You can start using Nx Agents with your existing CI provider by adding the `nx-cloud start-ci-run...` command mentioned previously. In addition, all artifacts and logs are played back to the main job so you can keep processing them as if they were run on the main job. Hence, your existing post-processing steps should still keep working as before.
+
+For a more thorough explanation of how Nx Agents optimize your CI pipeline, read this [guide to parallelization and distribution in CI](/ci/concepts/parallelization-distribution).
 
 ## Nx Agents Features
 
@@ -66,21 +107,10 @@ The `--distribute-on` flag instructs Nx Cloud to distribute tasks across 8 agent
 
 {% /cards %}
 
-## CI/CD Guides
-
-Every organization manages their CI/CD pipelines differently, so the guides don't cover org-specific aspects of
-CI/CD (e.g., deployment). They mainly focus on configuring Nx correctly using Nx Agents and [Nx Replay](/ci/features/remote-cache).
-
-- [Azure Pipelines](/ci/recipes/set-up/monorepo-ci-azure)
-- [Circle CI](/ci/recipes/set-up/monorepo-ci-circle-ci)
-- [GitHub Actions](/ci/recipes/set-up/monorepo-ci-github-actions)
-- [Jenkins](/ci/recipes/set-up/monorepo-ci-jenkins)
-- [GitLab](/ci/recipes/set-up/monorepo-ci-gitlab)
-- [Bitbucket Pipelines](/ci/recipes/set-up/monorepo-ci-bitbucket-pipelines)
-
-Note that only cacheable operations can be distributed because they have to be replayed on the main job.
-
 ## Relevant Repositories and Examples
 
+By integrating Nx Agents into your CI pipeline, you can significantly reduce build times, optimize resource use, and maintain a scalable, efficient development workflow.
+
+- [Reliable CI: A New Execution Model Fixing Both Flakiness and Slowness](/blog/reliable-ci-a-new-execution-model-fixing-both-flakiness-and-slowness)
 - [Nx: On how to make your CI 16 times faster with a small config change](https://github.com/vsavkin/interstellar)
 - ["Lerna & Distributed Task Execution" Example](https://github.com/vsavkin/lerna-dte)