Skip to main content

Remove Prometheus monitoring

This guide is for those who want to remove the built-in Prometheus monitoring solution that comes pre-configured with Massdriver official Kubernetes bundles (AKS, EKS, GKE).

caution

Removing the built-in Prometheus monitoring solution will also disable Massdriver's package alarm feature for your Kubernetes deployment.

These changes will need to be made on both the Kubernetes cluster and also on all applications running on your Kubernetes cluster.

note

These changes can be made on an active cluster without impacting running applications.

Kubernetes Cluster​

  1. Open your massdriver.yaml and remove the entire monitoring block.
Monitoring block may look like:
monitoring:
  type: object
  title: Monitoring
  required:
    - prometheus
  properties:
    prometheus:
      type: object
      title: Prometheus Configuration
      description: Configuration settings for the Prometheus instances that are automatically installed into the cluster to provide monitoring capabilities
      required:
        - grafana_enabled
      properties:
        grafana_enabled:
          title: Enable Grafana
          type: boolean
          description: Install Grafana into the cluster to provide a metric visualizer
          default: false
      dependencies:
        grafana_enabled:
          oneOf:
          - properties:
              grafana_enabled:
                const: false
          - properties:
              grafana_enabled:
                const: true
              grafana_password:
                title: Grafana Admin Password
                description: Set the password for the `admin` user
                type: string
                format: password
            required:
            - grafana_password

  1. Remove monitoring from the required block in params, and also from the ui block.

  2. In core-services/core_services.tf, make the following changes:

AKS / EKS / GKE

Remove all instances of:

depends_on = [module.prometheus-observability]

Remove from module "ingress_nginx":

metrics = {
  enabled = true
  serviceMonitor = {
    enabled = true
  }
}
EKS Only

Remove from module "cluster-autoscaler":

serviceMonitor = {
  enabled   = true
  namespace = "kube-system"
  selector  = var.md_metadata.default_tags
}

  1. Delete core-services/monitoring.tf and core-services/observability.tf.

  2. Run mass bundle publish to publish the changes and redeploy your Kubernetes bundle.

  3. Access your Kubernetes cluster using your KUBECONFIG and kubectl.

  4. Ensure all of the following CRDs (Custom Resource Definition) are deleted:

alertmanagerconfigs.monitoring.coreos.com
alertmanagers.monitoring.coreos.com
podmonitors.monitoring.coreos.com
probes.monitoring.coreos.com
prometheuses.monitoring.coreos.com
prometheusrules.monitoring.coreos.com
scrapeconfigs.monitoring.coreos.com
servicemonitors.monitoring.coreos.com
thanosrulers.monitoring.coreos.com

List CRDs:

kubectl get crds

Delete CRD:

kubectl delete crd alertmanagerconfigs.monitoring.coreos.com

Kubernetes application​

You'll also need to remove the monitoring components of your Kubernetes applications. This should not incur any downtime of your application.

  1. Delete the src/monitoring.tf file.

  2. Run mass bundle publish to publish your changes, and redeploy your application.