Using Datadog and Prometheus with Gluu Cloud Native edition#

Prerequisites#

Datadog Agent 6.5.0+

Instructions#

Install Gluu server using one of the following
Kustomize
Helm
Create a directory called monitoring for the manifests needed.
```
mkdir monitoring && cd monitoring
```

Get zip file of manifests and unzip

wget https://github.com/GluuFederation/cloud-native-edition/raw/4.1/monitoring.zip
unzip monitoring.zip

Create a namespace for Datadog and Prometheus

kubectl create ns datadog
kubectl create ns prometheus

Using Kubectl commands#

Configure RBAC permissions for the datadog agent in datadog namespace.

kubectl create -f "https://raw.githubusercontent.com/DataDog/datadog-agent/master/Dockerfiles/manifests/rbac/clusterrole.yaml" -ns datadog

kubectl create -f "https://raw.githubusercontent.com/DataDog/datadog-agent/master/Dockerfiles/manifests/rbac/serviceaccount.yaml" -ns datadog

kubectl create -f "https://raw.githubusercontent.com/DataDog/datadog-agent/master/Dockerfiles/manifests/rbac/serviceaccount.yaml" -ns datadog

Create a Secret containing datadog API key. It will be used in datadog agent Daemonset.

Warning

If the name of the secret below is to be changed , change it in the respective DaemonSet as well.
```
kubectl create secret generic datadog-secret --from-literal api-key="<API-KEY>"
```
Create a Datadog agent with custom metrics and APM logs collection enabled.
```
kubectl apply -f datadog/
```

Install Prometheus

kubectl apply -f prometheus/

Note

For datadog and prometheus integration, prometheus deployment has the annotations that allow datadog autodiscovery feature and gets all the metrics that prometheus collects from the cluster.

      annotations:
          ad.datadoghq.com/prometheus.check_names: |
            ["openmetrics"]
          ad.datadoghq.com/prometheus.init_configs: |
            [{}]
          ad.datadoghq.com/prometheus.instances: |
            [
              {
                "prometheus_url": "http://%%host%%:%%port%%/metrics",
                "namespace": "monitoring",
                "metrics": [ {"promhttp_metric_handler_requests_total": "prometheus.handler.requests.total"}]
              }
            ]

[Optional] Install kube state metrics which exposes all the metrics on /metrics URI. Prometheus can scrape all the metrics exposed by kube state metrics. This will be created in the kube-system namespace.
```
kubectl apply -f kube-state-metrics/
```
1. Add the following configuration as part of prometheus job configuration.
```
- job_name: 'kube-state-metrics'
  static_configs:
    - targets: ['kube-state-metrics.kube-system.svc.cluster.local:8080']
```
Note

This part has been included in by default in proth-cm. If it is not being used, it should be removed. If not it will cause health check error in prometheus targets.

Using helm#

Prerequisites#

Helm 2.10+
Tiller

Datadog#

Install Datadog

helm install --name datadog-agent-v1 \
  --set datadog.apiKey=<DataDog API Key> \
  --set datadog.apmEnabled=true \
  --set datadog.logsEnabled=true \
  stable/datadog
  --namespace datadog

Install prometheus

Note

The custom values.yaml file above has the annotations needed to integrate datadog with prometheus.

helm install --name prometheus --namespace prometheus -f helm/prometheus-values.yaml stable/prometheus

Access prometheus UI via port-forwarding or switch prometheus service upon installation to LoadBalancer

export PROMETHEUS_POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace prometheus port-forward PROMETHEUS_POD_NAME 9090

[Optional] Install kube state metrics which exposes all the metrics on /metrics URI. Prometheus can scrape all the metrics exposed by kube state metrics. This will be created in the kube-system namespace.
```
helm2 install --name kube-state stable/kube-state-metrics
```
1. Add the kube-state metrics job in prometheus serverFiles.
```
      - job_name: 'kube-state-metrics'
        static_configs:
          - targets: ['kube-state-kube-state-metrics.default.svc.cluster.local:8080']
```
Note

All the metrics will be exported to datadog metrics -> summary