Document memcached configuration for caching. (#8373)

**What this PR does / why we need it**:
Memcached is our default options for caching. However, the options for Memcached are not well documented.

This change documents how Memcached should be configured and how Loki itself should be documented based on our ksonnet deployment.

**Checklist**
- [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**)
- [x] Documentation added
- [ ] Tests updated
- [ ] `CHANGELOG.md` updated
- [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`

Co-authored-by: J Stickler <[email protected]>

Author: jeschkies

docs/sources/installation/helm/install-scalable/index.md

@@ -23,6 +23,7 @@ It is not possible to run the scalable mode with the `filesystem` storage.
 - Helm 3 or above. See [Installing Helm](https://helm.sh/docs/intro/install/).
 - A running Kubernetes cluster.
 - A Prometheus operator installation in case meta-monitoring should be used.
+- Optionally a Memcached deployment for better performance. Consult the [caching section]({{<relref "../../../operations/caching">}}) on how to configure Memcached.
 
 **To deploy Loki in scalable mode:**
 

docs/sources/operations/caching.md

@@ -0,0 +1,94 @@
+---
+title: Caching 
+menuTitle: Caching 
+description: Enable and configure memcached for caching. 
+weight: 100
+keywords:
+  - memcached
+  - caching
+---
+
+# Caching
+
+Loki supports caching of index writes and lookups, chunks and query results to
+speed up query performance. This sections describes the recommended Memcached
+configuration to enable caching for chunks and query results. The index lookup
+cache is configured to be in-memory by default.
+
+## Before you begin
+
+- It is recommended to deploy four dedicated Memcached clusters.
+- As of 2023-02-01, the `memcached:1.6.17-alpine` version of the library is recommended.
+- Consult the Loki ksonnet [memcached](https://github.com/grafana/loki/blob/main/production/ksonnet/loki/memcached.libsonnet) deployment and the ksonnet [memcached library](https://github.com/grafana/jsonnet-libs/tree/master/memcached).
+
+## Steps
+
+To enable and configure Memcached:
+
+1. Deploy each Memcached service with at least three replicas and configure
+   each as follows:
+    1. Chunk cache 
+       ```
+       --memory-limit=4096 --max-item-size=2m --conn-limit=1024
+       ```
+    1. Query result cache
+       ```
+       --memory-limit=1024 --max-item-size=5m --conn-limit=1024
+       ```
+
+1. Configure Loki to use the cache.
+    1. If the Helm chart is used
+
+       Set `memcached.chunk_cache.host` to the Memecache address for the chunk cache, `memcached.results_cache.host` to the Memecache address for the query result cache, `memcached.chunk_cache.enabled=true` and `memcached.results_cache.enabled=true`. 
+       
+       Ensure that the connection limit of Memcached is at least `number_of_clients * max_idle_conns`.
+       
+       The options `host` and `service` depend on the type of installation. For example, using the `bitname/memcached` Helm Charts with the following commands, the `service` values are always `memcached`.
+       ```
+       helm upgrade --install chunk-cache -n loki bitnami/memcached -f memcached-overrides.yaml
+       helm upgrade --install write-cache -n loki bitnami/memcached -f memcached-overrides.yaml
+       helm upgrade --install results-cache -n loki bitnami/memcached -f memcached-overrides.yaml
+       helm upgrade --install index-cache -n loki bitnami/memcached -f memcached-overrides.yaml
+       ```
+       In this case, the Loki configuration would be
+       ```yaml
+       loki:
+         memcached:
+           chunk_cache:
+             enabled: true
+             host: chunk-cache-memcached.loki.svc
+             service: memcache
+             batch_size: 256
+             parallelism: 10
+           results_cache:
+             enabled: true
+             host: results-cache-memcached.loki.svc
+             service: memcache
+             default_validity: 12h
+       ```
+    1. If the Loki configuration is used
+        1. Configure the chunk cache
+           ```yaml
+           chunk_store_config:
+             chunk_cache_config:
+               memcached:
+                 batch_size: 256
+                 parallelism: 10
+               memcached_client:
+                 host: <memcached host>
+                 service: <port name of memcached service>
+           ```
+        1. Configure the query result cache
+           ```yaml
+           query_range:
+             cache_results: true
+             results_cache:
+               cache:
+                 memcached_client:
+                   consistent_hash: true
+                   host: <memcached host>
+                   service: <port name of memcached service>
+                   max_idle_conns: 16
+                   timeout: 500ms
+                   update_interval: 1m
+           ```