Dev48
Language
  • About
  • Services
  • Industries
  • Technologies
  • Articles
  • Contacts
Book a call
    Home/Articles/How to find at risk pubsub messages across google cloud projects
Dev48
info@dev48.org+375 29 320-40-96WhatsApp@isky_by

© 2026 · All rights reserved.

How to find at-risk Pub/Sub messages across Google Cloud projects

Фото: Shutter Speed (Unsplash) — https://unsplash.com/photos/a-close-up-of-a-cell-phone-with-the-google-logo-in-the-background-0uMU42uRw28?utm_source=dev48&utm_medium=referral

Source: Dynatrace news•November 19, 2026 at 05:20 PM

How to find at-risk Pub/Sub messages across Google Cloud projects

Google Cloud observability is now generally available Dynatrace Clouds. With this release, Dynatrace extends its Cloud Observability experience to Google Cloud, bringing resource configuration, metrics, and logs into the same investigation context. You can analyze that context with Dynatrace…

Google Cloud observability is now generally available Dynatrace Clouds. With this release, Dynatrace extends its Cloud Observability experience to Google Cloud, bringing resource configuration, metrics, and logs into the same investigation context. You can analyze that context with Dynatrace Query Language (DQL) and use existing cloud labels to route alerts to resource owners. To show what that looks like in practice, the rest of this blog follows a representative example incident in which failed Pub/Sub messages quietly approach their retention deadline. Along the way, you will learn how to combine configuration and metrics in a single DQL query, follow a relationship from one resource to another, and turn the result into an alert that covers every connected project.

Consider the following hypothetical scenario: An online fashion retailer runs a settlement job every Friday morning that approves all inspected returns in one batch. Last Friday’s batch, the first returns from the end-of-season sale, was three times the usual size, and for half an hour the service that pays refunds turned work away.

The incident review on Tuesday closed with three words, no data loss, and a ticket. Pub/Sub had moved every undelivered message to a separate dead-letter subscription, where the messages would remain available for seven days. Because the team could retry them later, it scheduled that work for the next sprint.

The dead-letter queue prevented immediate data loss while the consumer recovered but shifted the risk to retention. The messages could expire before the planned replay, making message age relative to the configured retention window the critical signal to watch. The rest of this post shows how to see that deadline before it arrives, starting with why dead-letter messages are so easy to overlook.

Follow the incident below for the full investigation. For the query pattern, start with “How to compare Pub/Sub message age with retention configuration using DQL.” To check other projects, go to “Find other at-risk Pub/Sub subscriptions across connected projects.” For replay and alerting, jump to “What to do before the retention deadline.”

Why Pub/Sub dead-letter messages can be easy to overlook

The settlement job publishes each approved refund to a Pub/Sub topic (refund-events). A push subscription (refunds-push) delivers it to refund-worker, a Cloud Run service that pays it out through the payment provider. A year ago, someone careful gave refunds-push a dead-letter policy: after five failed delivery attempts, a message moves to the topic refund-events-dlq, and from there into the subscription refund-events-dlq-sub.

Two properties of this setup matter:

  • Retention: Google Cloud Pub/Sub keeps an unacknowledged message for its subscription’s retention, seven days by default, after which the documentation says it “might discard the message independent of the acknowledgment state.” No Pub/Sub metric counts those discards. Google’s monitoring guide files an oldest message that “exceeds the subscription message retention duration” under Permanent data loss and recommends alerting before that point (Monitor Pub/Sub). The advice is sound, but there’s a catch. These specific alerts will only cover the subscriptions a team knows exist.
  • What a full worker does: A push endpoint acknowledges with a success code (e.g. 200), and “any other status code” sends the message back for redelivery. At its instance ceiling, Cloud Run answers 429, and its request metric doesn’t count those.

The pattern starts here: The protective mechanism takes the failure out of the number you’re watching.

How Dynatrace detects Cloud Run saturation

At 10:21 on Friday the settlement job published its batch, and within seconds refund-worker was at its ceiling of 20 instances and answering 429. Dynatrace opened a problem dated from 10:26: Cloud Run revision at max instances, on refund-worker.

This team had built an alert specifically for this in Dynatrace. The request metric can’t show the rejections, so the alert watches the other side: one DQL query joins the revision’s instance-time metric with the instance ceiling from its configuration in Smartscape, Dynatrace’s real-time topology model. Because Google Cloud metrics arrive every few minutes, it targets sustained saturation, not single spikes, and opened five minutes after the first 429 in this scenario. Asked to explain the problem, Dynatrace Intelligence reads the same topology, metrics and logs from Grail, Dynatrace’s data lakehouse.

As the explanation in the screenshot above suggests, the Cloud Run warning logs provide further insights and confirm that requests were aborted due to instance unavailability. As Google Cloud logs are linked to the revision at ingest, Dynatrace can analyze them directly in the context of the Cloud Run revision (e.g. in Clouds app):

The fix was quick and correct. The ceiling went from 20 to 60 at 10:43, and the last 429 was at 10:54. That left the question every incident with a queue in it ends on: Did we lose anything?

How to compare Pub/Sub message age with retention configuration using DQL

Once the backlog was identified, the important question was how much time remained before the messages became eligible for deletion. Compare each subscription’s oldest unacknowledged message age with its retention, as Google’s guidance suggests.

With Cloud Platform Monitoring for Google Cloud in Dynatrace, subscription configuration is available alongside its monitoring data in Smartscape. Retention isn’t a metric, but DQL can combine the configured retention period with the oldest unacknowledged message age to calculate how much of that window has already been consumed:

Because the metric is linked to the subscription, one DQL query can divide the oldest message’s age by that subscription’s retention:

On refunds-push, the ratio peaked at 0.0016, about sixteen minutes against seven days. The backlog never aged, because messages that couldn’t be delivered were retried and then forwarded to refund-events-dlq. Pub/Sub’s dead_letter_message_count showed close to 2,000 messages forwarded between 10:30 and 10:55.

On Tuesday the review wrote it up: No data loss. 1,932 refunds caught by the dead-letter queue, retained for seven days. Replay ticketed for next sprint, P3. It was a reasonable call. Seven days felt like plenty, and nothing was on fire.

The review’s conclusion, no data loss, was the right reading of correct evidence. The forwarding is why the backlog never aged: “After forwarding an undeliverable message, the Pub/Sub service removes the message from the subscription.”

The review’s conclusion, no data loss, was the right reading of correct evidence. The forwarding is why the backlog never aged: “After forwarding an undeliverable message, the Pub/Sub service removes the message from the subscription.”

The dead-letter policy caught the failures, but it took them out of the only subscription anyone was watching. The source backlog no longer counted the forwarded messages, just as request_count didn’t count the requests Cloud Run rejected: the failure had been removed from the number.

So where did they go? A dead-letter topic’s messages live in whatever subscription is attached to it. Pick a dead-letter topic with none, and the console offers to create one. That’s how refund-events-dlq-sub came to exist, with every property at its default: seven days of retention, no consumer and no alert.

An alert on its depth wouldn’t have changed Tuesday. num_undelivered_messages > 0 would have said what the review already knew. Depth says something is waiting, not until when, and until when was the only thing the review got wrong. It needed a date, and that date needs a metric and a configuration field side by side. In Dynatrace, they sit on the same entity, stored together in Grail.

Calculating the deadline requires three configuration fields and two metrics, starting from a field nobody planned to monitor. The source subscription’s deadLetterPolicy is queryable because the connection keeps the full Asset Inventory payload, not a curated subset:

The first of those messages becomes eligible for deletion on Friday morning, a week to the minute after it was forwarded. The replay ticket’s sprint starts the Monday after. That’s the condition Google’s guide files under Permanent data loss. Nobody outside the team will flag it first, refund confirmations said to allow five to ten business days, so the first “where’s my money?” will occur about when Pub/Sub may discard the record.

The dead-letter policy worked as documented and the review read the evidence correctly, but in three days the oldest messages would become eligible for deletion. And this ending can still be changed: replay refund-events-dlq-sub into refund-worker before Friday, then give it the alert it never had.

Find other at-risk Pub/Sub subscriptions across connected projects

Nothing in the age query is specific to refunds-push. Drop the filter and rank by ratio, and it becomes an estate sweep:

Alternatively, ask Dynatrace Intelligence, “Which Pub/Sub subscriptions are closest to their message retention?” Then ask, “Which of those are dead-letter subscriptions?” Dynatrace Intelligence can analyze the same supported Pub/Sub metrics, configuration, and resource relationships used by the DQL query.

Dead-letter subscriptions show up in that list without extra setup. refund-events-dlq-sub didn’t require individual onboarding. The inventory model makes the subscription available for investigation alongside other supported resources in connected projects.

The same configuration answers a second question, with no metrics: which dead-letter topics have no subscription attached? Google’s API reference is explicit: “messages published to a topic with no subscriptions are lost” (Pub/Sub API reference, DeadLetterPolicy). Unless topic retention is enabled, there’s no window to recover them.

One join finds them: every unattached safety net, listed before anyone needs it.

What to do before the retention deadline

Before the deadline: replay. Replay refund-events-dlq-sub into refund-worker, but for money, first check that the worker is idempotent on the refund ID. Here it’s safe: every failed attempt in this scenario was a 429 returned before reaching a container, with no partial processing indicated. Then rerun the deadline query until the dead-letter backlog reads zero. If the replay can’t happen in time, buy time: raising retention, up to 31 days, is non-destructive, but past 24 hours it costs extra. That cost is the point: holding a dead letter backlog should be a deliberate decision.

After recovery: create one rule for every dead-letter subscription. Alert on the share of retention used, not on depth. A fixed age threshold in Cloud Monitoring works while every subscription shares one retention; with mixed retentions it needs one threshold per value, kept in step by hand. In Dynatrace, the rule reads each subscription’s retention from its configuration: it’s the deadline query saved as an alert, scoped to every subscription a dead-letter policy points at, so it also covers the ones nobody remembered. An owner label on each subscription routes the problem to the team that owns it. Set at half the retention, it would have opened a problem on Monday evening, the night before the review.

What won’t prevent the retention risk?

  • Raising maxDeliveryAttempts: it sends the same messages to the same destination later
  • Fixing the consumer alone: It doesn’t resolve messages already sitting in the dead-letter subscription
  • Watching the source subscription more closely: Forwarded messages have moved elsewhere.
  • Creating a ticket without a deadline: The retention clock keeps running.

Why configuration and telemetry matter in the same investigation

The investigation is possible because Cloud Platform Monitoring for Google Cloud brings configuration and telemetry into the same Dynatrace context.

Configuration on the entity supplied the investigation’s three links: deadLetterPolicy identified the destination, a join exposed dead-letter topics without an attached subscription, and retention supplied both the age ratio’s divisor and the deletion deadline. Google recommends comparing a subscription’s oldest message with its retention. Here, that comparison runs as one query across every connected project, so no subscription has to be set up or remembered first.

Query configuration and metrics with DQL. With configuration and metrics together in Grail, DQL provides one query language to follow the dead-letter relationship, compare message age with retention, and calculate the deletion deadline.

One rule, routed to the right owner. The deadline query can become a single alert across dead-letter subscriptions, while existing Google Cloud owner labels route each problem to the appropriate team.

Keyless, and agentless where it counts. Service-account impersonation means there are no key files to store or rotate. Metrics and topology are collected through Google Cloud APIs, without deploying Dynatrace components in your projects. Logs are the exception: they use a Log Router sink into Pub/Sub with a Dataflow job, enriched by OpenPipeline at ingest.

Get started

Cloud Platform Monitoring for Google Cloud is now generally available. Connect a Google Cloud organization, open the Clouds app, and begin exploring connected resources, configuration, metrics, logs, and topology across your Google Cloud environment.

New to Dynatrace? Start a free trial.

← All articles

What this article says

More in Data & Analytics

All →
CMMC vs. NIS2: Two paths to the same secure data destination
Collibra

CMMC vs. NIS2: Two paths to the same secure data destination

Smarter Pipelines with Trusted Metadata
Matillion

Smarter Pipelines with Trusted Metadata

What’s New in the Data Productivity Cloud: July 2025
Matillion

What’s New in the Data Productivity Cloud: July 2025

How Chain, Our AI Workspace, Helps Investigators Fight Crime
Chainalysis

How Chain, Our AI Workspace, Helps Investigators Fight Crime

How Datadog resolves evolving OpenTelemetry semantics at query time
Datadog

How Datadog resolves evolving OpenTelemetry semantics at query time

Retain more logs for longer with Flex Logs BYOBucket
Datadog

Retain more logs for longer with Flex Logs BYOBucket

More from Dynatrace

State of Observability 2026: How to build an observability strategy for reliable AI
Dynatrace

State of Observability 2026: How to build an observability strategy for reliable AI

Evaluate, observe, act: A practical approach to operating AI agents with confidence
Dynatrace

Evaluate, observe, act: A practical approach to operating AI agents with confidence

What Dynatrace and Microsoft are bringing to the AI era at Microsoft Ignite 2026
Dynatrace

What Dynatrace and Microsoft are bringing to the AI era at Microsoft Ignite 2026

Get more from OpenTelemetry: From connected data to faster root cause analysis
Dynatrace

Get more from OpenTelemetry: From connected data to faster root cause analysis

From alerts to answers: The new Dynatrace Event Connector for ServiceNow ITOM
Dynatrace

From alerts to answers: The new Dynatrace Event Connector for ServiceNow ITOM

The SRE Agent is generally available: investigation and triage start the moment a problem is detected
Dynatrace

The SRE Agent is generally available: investigation and triage start the moment a problem is detected