Metrics Collection

Exposing Metrics in your Application

In your application, if you want to expose the metrics of your component (like webservice) to Prometheus, you just need to add the prometheus-scrape trait as follows.

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: my-app
spec:
  components:
    - name: my-app
      type: webservice
      properties:
        image: somefive/prometheus-client-example:new
      traits:
        - type: prometheus-scrape

You can also explicitly specify which port and which path to expose metrics.

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: my-app
spec:
  components:
    - name: my-app
      type: webservice
      properties:
        image: somefive/prometheus-client-example:new
      traits:
        - type: prometheus-scrape
          properties:
            port: 8080
            path: /metrics

This will let your application be scrapable by the prometheus server. If you want to see those metrics on Grafana, you need to create Grafana dashboard further. Go to Dashboard for learning the following steps.

Customized Prometheus Installation

If you want to make customization to your prometheus-server installation, you can put your configuration into an individual ConfigMap, like my-prom in namespace o11y-system. To distribute your custom config to all clusters, you can also use a KubeVela Application to do the job.

Recording Rules

For example, if you want to add some recording rules to all your prometheus server configurations in all clusters, you can firstly create an application to distribute your recording rules as below.

# my-prom.yaml
apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: my-prom
  namespace: o11y-system
spec:
  components:
    - type: k8s-objects
      name: my-prom
      properties:
        objects:
          - apiVersion: v1
            kind: ConfigMap
            metadata:
              name: my-prom
              namespace: o11y-system
            data:
              my-recording-rules.yml: |
                groups:
                  - name: example
                    rules:
                      - record: apiserver:requests:rate5m
                        expr: sum(rate(apiserver_request_total{job="kubernetes-nodes"}[5m]))
  policies:
    - type: topology
      name: topology
      properties:
        clusterLabelSelector: {}

Then you need to add customConfig parameter to the enabling process of the prometheus-server addon, like

vela addon enable prometheus-server thanos=true serviceType=LoadBalancer storage=1G customConfig=my-prom

Then you will be able to see the recording rules configuration being delivered into all prometheus instances.

Alerting rules & Other configurations

To make customization to other configurations like alerting rules, the process is same with the recording rules example shown above. You only need to change/add prometheus configurations in the application.

data:
  my-alerting-rules.yml: |
    groups:
    - name: example
      rules:
      - alert: HighApplicationQueueDepth
        expr: sum(workqueue_depth{app_kubernetes_io_name="vela-core",name="application"}) > 100
        for: 10m
        annotations:
          summary: High Application Queue Depth

Custom storage

If you want your prometheus-server to persist data in volumes, you can also specify storage parameter for your installation, like

vela addon enable prometheus-server storage=1G

This will create PersistentVolumeClaims and let the addon use the provided storage. The storage will not be automatically recycled even if the addon is disabled. You need to clean up the storage manually.