chore: Update LRO help section (#9720)

* chore: Update LRO help section

* chore: Update wording

* chore: Apply suggestions from code review

Co-authored-by: Blake Li <[email protected]>

* chore: Resolve merge conflicts

* chore: Address PR comments

* chore: Add more info to README

* Update README.md

Co-authored-by: Tomo Suzuki <[email protected]>

* chore: Address PR comments

* chore: Address PR comments

* Apply suggestions from code review

Co-authored-by: Blake Li <[email protected]>

---------

Co-authored-by: Blake Li <[email protected]>
Co-authored-by: Tomo Suzuki <[email protected]>

Author: 3 people

README.md

@@ -384,20 +384,81 @@ public CloudTasksClient getService() throws IOException {
 ## Long Running Operations
 
 Long running operations (LROs) are often used for API calls that are expected to
-take a long time to complete (e.g. provisioning a GCE instance or a Dataflow pipeline).
-The initial API call creates an "operation" on the server and returns an operation ID
-to track its progress.
+take a long time to complete (i.e. provisioning a GCE instance or a Dataflow pipeline).
+The initial API call creates an "operation" on the server and returns an Operation ID
+to track its progress. LRO RPCs have the suffix `Async` appended to the call name
+(i.e. `clusterControllerClient.createClusterAsync()`)
 
-Our generated gRPC clients provide a nice interface for starting the operation and
+Our generated clients provide a nice interface for starting the operation and
 then waiting for the operation to complete. This is accomplished by returning an
 [`OperationFuture`](https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.longrunning.OperationFuture).
-When you call `get()` on the `OperationFuture` we poll the operation endpoint to
-check on the operation. These polling operations have a default timeout that
-varies from service to service and will throw a `java.util.concurrent.CancellationException`
-with the message: `Task was cancelled.` after that timeout has been reached.
+When calling `get()` on the `OperationFuture`, the client library will poll the operation to
+check the operation's status.
 
-### Configuring LRO Timeouts
+For example, take a sample `createCluster` Operation in google-cloud-dataproc v4.20.0:
+```java
+try (ClusterControllerClient clusterControllerClient = ClusterControllerClient.create()) {
+  CreateClusterRequest request =
+      CreateClusterRequest.newBuilder()
+          .setProjectId("{PROJECT_ID}")
+          .setRegion("{REGION}")
+          .setCluster(Cluster.newBuilder().build())
+          .setRequestId("{REQUEST_ID}")
+          .setActionOnFailedPrimaryWorkers(FailureAction.forNumber(0))
+          .build();
+  OperationFuture<Cluster, ClusterOperationMetadata> future =
+      clusterControllerClient.createClusterOperationCallable().futureCall(request);
+  // Do something.
+  Cluster response = future.get();
+} catch (CancellationException e) {
+  // Exceeded the default RPC timeout without the Operation completing.
+  // Library is no longer polling for the Operation status. Consider 
+  // increasing the timeout.
+}
+```
+
+### LRO Timeouts
+The polling operations have a default timeout that varies from service to service.
+The library will throw a `java.util.concurrent.CancellationException` with the message:
+`Task was cancelled.` if the timeout exceeds the operation. A `CancellationException`
+does not mean that the backend GCP Operation was cancelled. This exception is thrown from the
+client library when it has exceeded the total timeout without receiving a successful status from the operation.
+Our client libraries respect the configured values set in the OperationTimedPollAlgorithm for each RPC.
+
+Note: The client library handles the Operation's polling mechanism for you. By default, there is no need
+to manually poll the status yourself.
 
+### Default LRO Values
+Each LRO RPC has a pre-configured default values. You can find these values by 
+searching in each Client's `StubSettings`'s class. The default LRO settings are initialized
+inside the `initDefaults()` method in the nested Builder class.
+
+For example, in google-cloud-aiplatform v3.24.0, the default [OperationTimedPollAlgorithm](https://github.com/googleapis/google-cloud-java/blob/9ae786d1acdc7354adf86b78691570668caa293d/java-aiplatform/google-cloud-aiplatform/src/main/java/com/google/cloud/aiplatform/v1/stub/EndpointServiceStubSettings.java#L755-L765) 
+has these default values:
+```java
+OperationTimedPollAlgorithm.create(
+    RetrySettings.newBuilder()
+        .setInitialRetryDelay(Duration.ofMillis(5000L))
+        .setRetryDelayMultiplier(1.5)
+        .setMaxRetryDelay(Duration.ofMillis(45000L))
+        .setInitialRpcTimeout(Duration.ZERO)
+        .setRpcTimeoutMultiplier(1.0)
+        .setMaxRpcTimeout(Duration.ZERO)
+        .setTotalTimeout(Duration.ofMillis(300000L))
+        .build())
+```
+Both retries and LROs share the same RetrySettings class. Note the corresponding link:
+- Total Timeout (Max Time allowed for polling): 5 minutes
+- Initial Retry Delay (Initial delay before first poll): 5 seconds
+- Max Retry Delay (Maximum delay between each poll): 45 seconds
+- Retry Delay Multiplier (Multiplier value to increase the poll delay): 1.5
+
+The RPC Timeout values have no use in LROs and can be omitted or set to the default values
+(`Duration.ZERO` for Timeouts or `1.0` for the multiplier).
+
+### Configuring LRO Timeouts
+To configure the LRO values, create an OperationTimedPollAlgorithm object and update the
+RPC's polling algorithm. For example:
 ```java
 ClusterControllerSettings.Builder settingsBuilder = ClusterControllerSettings.newBuilder();
 TimedRetryAlgorithm timedRetryAlgorithm = OperationTimedPollAlgorithm.create(
@@ -415,6 +476,9 @@ settingsBuilder.createClusterOperationSettings()
 ClusterControllerClient clusterControllerClient = ClusterControllerClient.create(settingsBuilder.build());
 ```
 
+Note: The configuration above *only* modifies the LRO values for the `createClusterOperation` RPC.
+The other RPCs in the Client will still use each RPC's pre-configured LRO values. 
+
 ## Managing Dependencies
 
 If you are using more than one Google Cloud client library, we recommend you use one of