{"id":12871,"date":"2026-08-19T10:10:12","date_gmt":"2026-08-19T08:10:12","guid":{"rendered":"https:\/\/mybox.com\/help\/?post_type=manual_kb&#038;p=12871"},"modified":"2026-08-19T10:10:17","modified_gmt":"2026-08-19T08:10:17","slug":"prometheus-monitoring-mistakes-10-causes-of-missing-misleading-or-expensive-metrics","status":"publish","type":"manual_kb","link":"https:\/\/mybox.com\/help\/en\/knowledgebase\/prometheus-monitoring-mistakes-10-causes-of-missing-misleading-or-expensive-metrics\/","title":{"rendered":"Prometheus Monitoring Mistakes: 10 Causes of Missing, Misleading, or Expensive Metrics"},"content":{"rendered":"\n<div class=\"translation-block translation-block-merged\">\n<p class=\"wp-block-paragraph\">Prometheus monitoring can produce incomplete data, noisy alerts, slow queries, or unexpectedly high storage costs when collection, labels, alert rules, or retention are not controlled. These problems can also affect Grafana dashboards because dashboards and alerts depend on the metrics and labels stored by Prometheus.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Website analytics show how measurement data is used to understand visitors, traffic sources, page visits, time on site, and completed actions. The same principle applies to infrastructure metrics: useful monitoring depends on collecting the right data and presenting it with clear meaning. <a href=\"https:\/\/mybox.com\/help\/knowledgebase\/key-metrics-to-watch-in-your-website-analytics\/\">Key metrics should be reviewed in context rather than treated as isolated numbers.<\/a><\/p>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_86 ez-toc-wrap-left counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/prometheus-monitoring-mistakes-10-causes-of-missing-misleading-or-expensive-metrics\/#1_Poor_label_design\" >1. Poor label design<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/prometheus-monitoring-mistakes-10-causes-of-missing-misleading-or-expensive-metrics\/#2_Unbounded_label_values\" >2. Unbounded label values<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/prometheus-monitoring-mistakes-10-causes-of-missing-misleading-or-expensive-metrics\/#3_Labels_that_make_metrics_misleading\" >3. Labels that make metrics misleading<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/prometheus-monitoring-mistakes-10-causes-of-missing-misleading-or-expensive-metrics\/#4_Incorrect_scrape_settings\" >4. Incorrect scrape settings<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/prometheus-monitoring-mistakes-10-causes-of-missing-misleading-or-expensive-metrics\/#5_Incomplete_target_coverage\" >5. Incomplete target coverage<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/prometheus-monitoring-mistakes-10-causes-of-missing-misleading-or-expensive-metrics\/#6_Scrape_settings_that_create_noisy_data\" >6. Scrape settings that create noisy data<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/prometheus-monitoring-mistakes-10-causes-of-missing-misleading-or-expensive-metrics\/#7_Alert_rules_that_are_too_broad\" >7. Alert rules that are too broad<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/prometheus-monitoring-mistakes-10-causes-of-missing-misleading-or-expensive-metrics\/#8_Alert_rules_that_do_not_match_the_metric\" >8. Alert rules that do not match the metric<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/prometheus-monitoring-mistakes-10-causes-of-missing-misleading-or-expensive-metrics\/#9_Queries_that_are_too_expensive\" >9. Queries that are too expensive<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/prometheus-monitoring-mistakes-10-causes-of-missing-misleading-or-expensive-metrics\/#10_Missing_retention_controls\" >10. Missing retention controls<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/prometheus-monitoring-mistakes-10-causes-of-missing-misleading-or-expensive-metrics\/#How_to_diagnose_the_symptoms\" >How to diagnose the symptoms<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"1_Poor_label_design\"><\/span>1. Poor label design<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<\/div>\n\n<div id=\"mybox-1491536222\" class=\"mybox-content mybox-entity-placement\"><div class=\"early-access-banner-inpost\">\r\n  <div class=\"banner-left-inpost\">\r\n    <div class=\"icon-box-inpost\">\r\n      <img decoding=\"async\" src=\"https:\/\/mybox.com\/help\/wp-content\/uploads\/2026\/02\/square-info-icon.svg\" alt=\"Info\">\r\n    <\/div>\r\n    <div class=\"text-box-inpost\">\r\n      <span class=\"label-inpost\"><span class=\"translation-block translation-block-banner-text\">Early access<\/span><\/span>\r\n      <h4><span class=\"translation-block translation-block-banner-text\">Still need help?<\/span><\/h4>\r\n      <p><span class=\"translation-block translation-block-banner-text\">Contact our customer service team.<\/span><\/p>\r\n    <\/div>\r\n  <\/div>\r\n\r\n  <div class=\"banner-right-inpost\">\r\n    <a href=\"https:\/\/panel.mybox.com\/helpdesk2\/v\/list\/\" class=\"banner-button-inpost\"><span class=\"translation-block translation-block-banner-text\">Message us<\/span><\/a>\r\n  <\/div>\r\n<\/div><\/div>\n\n<div class=\"translation-block translation-block-merged\"><p class=\"wp-block-paragraph\">Labels identify the dimensions of a metric. Poor label design makes similar time series difficult to compare and can make Grafana panels or alert rules return a confusing mix of results.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Review the labels used by each important metric. Keep labels that support a real dashboard view or alert decision. Remove labels that do not help users interpret the metric. Use the same label names and meanings across related metrics.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"2_Unbounded_label_values\"><\/span>2. Unbounded label values<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Unbounded cardinality occurs when a label can create a continuing stream of new values. This produces more time series over time and can increase query work and storage use.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Check which labels can grow without a fixed limit. Do not use a label design that allows the number of values to expand without control. Replace uncontrolled dimensions with a smaller, stable set of categories that still answers the monitoring need.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"3_Labels_that_make_metrics_misleading\"><\/span>3. Labels that make metrics misleading<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A metric may look precise while its labels change the meaning of each series. A Grafana panel can then combine values that should be viewed separately, or split one operational signal into many small results.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For every panel and alert, identify the exact metric and label dimensions it uses. Confirm that the displayed series represent the same operational concept. If a label changes the meaning of a result, group or filter the data consistently before using it for a decision.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"4_Incorrect_scrape_settings\"><\/span>4. Incorrect scrape settings<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Incorrect scrape settings can leave gaps in the collected data. Missing samples can make a dashboard look inactive and can prevent an alert rule from evaluating the condition it is intended to detect.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Review the scrape settings for the targets that supply the affected metrics. Confirm that the intended targets are included and that the settings match the data collection requirement. Check the resulting time range in Grafana after making a change.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"5_Incomplete_target_coverage\"><\/span>5. Incomplete target coverage<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A deployment can appear healthy while important systems are absent from the collected data. In that case, a dashboard may show only part of the environment and an alert may fail to represent the full service.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Map each important Grafana panel and alert to the targets that provide its metrics. Compare that list with the systems that should be monitored. Add collection for any required target that is not represented, then verify that its metrics appear in the relevant panels.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"6_Scrape_settings_that_create_noisy_data\"><\/span>6. Scrape settings that create noisy data<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Scrape settings also affect the amount and regularity of data available to dashboards and alerts. A poor fit can make charts noisy or make an alert react to short-lived changes instead of a useful operational signal.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Review the collection behaviour for metrics used by alerts. Compare the resulting chart with the condition the alert is meant to represent. Adjust the collection approach so that the data supports a stable, understandable signal.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"7_Alert_rules_that_are_too_broad\"><\/span>7. Alert rules that are too broad<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A broad alert rule can match many series and create more notifications than the team can act on. This makes alerts noisy and can hide the signals that need attention.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Inspect the metric and labels used by each broad alert. Narrow the rule to the service, resource, or condition that requires action. In Grafana, confirm that the alert result maps to a clear dashboard view and that each notification has a specific meaning.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"8_Alert_rules_that_do_not_match_the_metric\"><\/span>8. Alert rules that do not match the metric<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">An alert can be technically valid but still misleading when it uses the wrong metric or label grouping. The result may be an alert that does not represent the condition shown in the related dashboard.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Compare every alert rule with its Grafana panel. Use the same metric definition and label meaning in both places. Test the alert against the data shown by the panel and remove any grouping that changes the intended interpretation.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"9_Queries_that_are_too_expensive\"><\/span>9. Queries that are too expensive<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Slow queries can result from large or highly varied metric data, especially when labels create many time series. Slow panels delay investigation, and slow alert evaluations reduce the value of timely notifications.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Start with the panels and alerts that take the longest to return. Simplify their metric selection and label grouping. Avoid requesting more series than the dashboard or alert needs. Recheck the panel after each change so the result remains useful rather than merely smaller.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"10_Missing_retention_controls\"><\/span>10. Missing retention controls<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Without retention controls, stored metrics can grow beyond the expected level. This can lead to unexpectedly high storage costs and can also make queries work across more historical data than the user needs.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Set a retention approach that matches the monitoring purpose. Keep the historical period required for dashboards, alert review, and operational analysis. Review storage growth after applying the control and confirm that the selected history remains available for the intended use.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"How_to_diagnose_the_symptoms\"><\/span>How to diagnose the symptoms<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Use the symptom to choose the first area to inspect:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Missing metrics or gaps:<\/strong> check scrape settings and target coverage first.<\/li>\n\n\n\n<li><strong>Misleading dashboard results:<\/strong> review label design, label meaning, and grouping.<\/li>\n\n\n\n<li><strong>Noisy alerts:<\/strong> inspect broad rules and confirm that the alert matches the dashboard signal.<\/li>\n\n\n\n<li><strong>Slow queries:<\/strong> review high-cardinality data and simplify expensive panels or alerts.<\/li>\n\n\n\n<li><strong>High storage costs:<\/strong> review unbounded label values and retention controls.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Make one focused change at a time, then verify the result in the affected Grafana dashboard and alert. A useful metric must be complete enough to support the decision, labelled clearly enough to interpret, and limited enough to query and store predictably.<\/p>\n<\/div>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"template":"","format":"standard","manualknowledgebasecat":[42],"manual_kb_tag":[],"class_list":["post-12871","manual_kb","type-manual_kb","status-publish","format-standard","hentry","manualknowledgebasecat-miscellaneous"],"_links":{"self":[{"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manual_kb\/12871","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manual_kb"}],"about":[{"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/types\/manual_kb"}],"author":[{"embeddable":true,"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/users\/1"}],"version-history":[{"count":1,"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manual_kb\/12871\/revisions"}],"predecessor-version":[{"id":12885,"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manual_kb\/12871\/revisions\/12885"}],"wp:attachment":[{"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/media?parent=12871"}],"wp:term":[{"taxonomy":"manualknowledgebasecat","embeddable":true,"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manualknowledgebasecat?post=12871"},{"taxonomy":"manual_kb_tag","embeddable":true,"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manual_kb_tag?post=12871"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}