Skip to main content
Version: Nightly

Self-Monitoring GreptimeDB Clusters

Before reading this document, ensure you understand how to deploy a GreptimeDB cluster on Kubernetes. This guide will walk you through configuring monitoring when deploying a GreptimeDB cluster.

Quick Start​

You can enable monitoring and Grafana by adding configurations to the values.yaml file when deploying the GreptimeDB cluster using Helm Chart. Here's a complete example of values.yaml for deploying a minimal GreptimeDB cluster with monitoring and Grafana:

image:
  # Image registry:
  # Use `docker.io` for OSS GreptimeDB,
  # consult staff for Enterprise GreptimeDB
  registry: <registry>
  # Image repository:
  # Use `greptime/greptimedb` for OSS GreptimeDB
  # Consult staff for GreptimeDB Enterprise
  repository: <repository>
  # Image tag:
  # Use database version `v1.3.0-alpha.1-nightly-20260907` for OSS GreptimeDB
  # Consult staff for GreptimeDB Enterprise
  tag: <tag>
  pullSecrets: []

initializer:
  registry: docker.io
  repository: greptime/greptimedb-initializer
  tag: "v0.6.1"

monitoring:
  # Enable monitoring
  enabled: true

grafana:
  # Enable Grafana deployment
  # Requires monitoring to be enabled first (monitoring.enabled: true)
  enabled: true

frontend:
  replicas: 1

meta:
  replicas: 1
  backendStorage:
    etcd:
      endpoints: ["etcd.etcd-cluster.svc.cluster.local:2379"]

datanode:
  replicas: 1

flownode:
  replicas: 1

When monitoring is enabled, GreptimeDB Operator launches an additional GreptimeDB Standalone instance to collect metrics and logs from the GreptimeDB cluster. To collect log data, GreptimeDB Operator starts a Vector sidecar container in each Pod.

When grafana is enabled, a Grafana instance is deployed that uses the GreptimeDB Standalone instance configured for cluster monitoring as its data source. This enables visualization of the GreptimeDB cluster's monitoring data out of the box using both Prometheus and MySQL protocols.

Then install the GreptimeDB cluster with the above values.yaml file:

helm upgrade --install mycluster \
  greptime/greptimedb-cluster \
  --values /path/to/values.yaml \
  -n default

After installation, you can check the Pod status of the GreptimeDB cluster:

kubectl -n default get pods
Expected Output
NAME                                 READY   STATUS    RESTARTS   AGE
mycluster-datanode-0                 2/2     Running   0          77s
mycluster-flownode-0                 2/2     Running   0          2m26s
mycluster-frontend-6ffdd549b-9s7gx   2/2     Running   0          66s
mycluster-grafana-675b64786-ktqps    1/1     Running   0          6m35s
mycluster-meta-58bc88b597-ppzvj      2/2     Running   0          86s
mycluster-monitor-standalone-0       1/1     Running   0          6m35s

You can then access the Grafana dashboard by port-forwarding the Grafana service to your local machine:

kubectl -n default port-forward svc/mycluster-grafana 18080:80

Then refer to the Access Grafana Dashboard section below for details on accessing Grafana.

Monitoring Configuration​

This section covers the details of monitoring configurations.

Enable Monitoring​

Add the following configuration to values.yaml to enable monitoring when deploying the GreptimeDB cluster:

monitoring:
  enabled: true

This deploys a GreptimeDBStandalone resource named ${cluster-name}-monitor to collect metrics and logs. You can verify the deployment with:

kubectl get greptimedbstandalones.greptime.io ${cluster-name}-monitor -n ${namespace}

The Service and StatefulSet it creates are named ${cluster-name}-monitor-standalone, and the Pods add the usual StatefulSet ordinal, for example ${cluster-name}-monitor-standalone-0. You can use the following addresses to access monitoring data:

  • Prometheus metrics: http://${cluster-name}-monitor-standalone.${namespace}.svc.cluster.local:4000/v1/prometheus
  • SQL logs: ${cluster-name}-monitor-standalone.${namespace}.svc.cluster.local:4002. By default, cluster logs are stored in the public._gt_logs table.

Customize Monitoring Storage​

By default, the GreptimeDB Standalone instance stores monitoring data using the Kubernetes default StorageClass in local storage. You can configure the GreptimeDB Standalone instance through the monitoring.standalone field in values.yaml. For example, the following configuration uses S3 object storage to store monitoring data:

monitoring:
  enabled: true
  ttl: 30d
  standalone:
    base:
     main:
       # Configure GreptimeDB Standalone instance image
       image: "greptime/greptimedb:v1.3.0-alpha.1-nightly-20260907"

       # Configure GreptimeDB Standalone instance resources
       resources:
         requests:
           cpu: "2"
           memory: "4Gi"
         limits:
           cpu: "2"
           memory: "4Gi"
    
    # Configure object storage for GreptimeDB Standalone instance
    objectStorage:
      s3:
        # Configure bucket
        bucket: "monitoring"
        # Configure region  
        region: "ap-southeast-1"
        # Configure secret name
        secretName: "s3-credentials"
        # Configure root path
        root: "standalone-with-s3-data"

Customize Vector Sidecar​

The Vector sidecar configuration for log collection can be customized via the monitoring.vector field. For example, you can adjust the Vector image and resource limits as follows:

monitoring:
  enabled: true
  ttl: 30d
  vector:
    # Configure Vector image registry
    registry: docker.io
    # Configure Vector image repository 
    repository: timberio/vector
    # Configure Vector image tag
    tag: "0.49.0-debian"

    # Configure Vector resources
    resources:
      requests:
        cpu: "50m"
        memory: "64Mi"
      limits:
        cpu: "250m" 
        memory: "256Mi"

YAML Configuration with kubectl Deployment​

If you're not using Helm Chart, you can also use the monitoring field to manually configure self-monitoring mode in the GreptimeDBCluster YAML:

monitoring:
  enabled: true

For detailed configuration options, refer to the GreptimeDBCluster API documentation.

Grafana Configuration​

Enable Grafana​

To enable Grafana deployment, add the following configuration to values.yaml. Note that monitoring must be enabled first (monitoring.enabled: true):

monitoring:
  enabled: true

grafana:
  enabled: true

Customize Grafana Data Sources​

By default, Grafana uses mycluster and default as the cluster name and namespace to create data sources. To monitor clusters with different names or namespaces, you need to create custom data source configurations based on the actual cluster names and namespaces. Here's an example values.yaml configuration:

monitoring:
  enabled: true

grafana:
  enabled: true
  datasources:
    datasources.yaml:
      datasources:
        # Query the cluster metrics.
        - name: greptimedb-metrics
          type: prometheus
          url: http://${cluster-name}-monitor-standalone.${namespace}.svc.cluster.local:4000/v1/prometheus
          access: proxy

        # Query the cluster traces.
        - name: greptimedb-traces
          type: jaeger
          url: http://${cluster-name}-monitor-standalone.${namespace}.svc.cluster.local:4000/v1/jaeger
          access: proxy

        # Query the cluster logs and slow queries.
        - name: greptimedb-logs
          type: mysql
          url: ${cluster-name}-monitor-standalone.${namespace}.svc.cluster.local:4002
          access: proxy
          database: public

        # Query the information schema from the cluster.
        - name: information_schema
          type: mysql
          url: ${cluster-name}-frontend.${namespace}.svc.cluster.local:4002
          access: proxy
          database: information_schema

This configuration creates the following data sources for GreptimeDB cluster monitoring in Grafana:

  • greptimedb-metrics: Cluster metrics stored in the standalone monitoring database, exposed via Prometheus protocol (type: prometheus)
  • greptimedb-logs: Cluster logs stored in the standalone monitoring database, exposed via MySQL protocol (type: mysql). Uses the public database by default
  • greptimedb-traces: Cluster traces stored in the standalone monitoring database, exposed via Jaeger protocol (type: jaeger). Uses the public database by default
  • information_schema: Cluster information stored in the frontend database, exposed via MySQL protocol (type: mysql). Uses the information_schema database by default

Access Grafana Dashboard​

You can access the Grafana dashboard by port-forwarding the Grafana service to your local machine:

kubectl -n ${namespace} port-forward svc/${cluster-name}-grafana 18080:80

Then open http://localhost:18080 to access the Grafana dashboard. The default login credentials for Grafana are:

  • Username: admin
  • Password: gt-operator

Navigate to the Dashboards section to explore the pre-configured dashboards for monitoring your GreptimeDB cluster.

Grafana Dashboard

Cleanup the PVCs​

danger

This removes the collected metrics and logs stored by the monitoring standalone instance. It does not touch the data of the GreptimeDB cluster itself, but the monitoring history is not recoverable afterwards.

To uninstall the GreptimeDB cluster, please refer to the Cleanup GreptimeDB Cluster documentation.

To clean up the Persistent Volume Claims (PVCs) used by the GreptimeDB standalone monitoring instance, delete the PVCs using the following command:

kubectl -n ${namespace} delete pvc -l app.greptime.io/component=${cluster-name}-monitor-standalone