Creating custom metrics from a trace in Datadog

Recently, I wanted to create a monitor from a trace in Datadog. When creating an APM monitor, you can specify the service, and the resource, but nothing else. This didn’t cater for what I wanted to do. So this post is going to show you how you can create a custom metric, and then use that in your monitor.

Once logged into Datadog, you want to navigate to APM and then Setup & Configuration. On the top bar, you should be able to see Generate Metrics. Go ahead and click on that link.

This is where we can create new metrics. So click the New Metric button, and then you can define your metric. This should feel familiar if you’ve created Datadog dashboards before.

Define:

• Metric name - I named my metric outbound.request.failures.
• The query you want to use
• What to group by

In my case, I wanted to see some error codes from a service called requests but ignore anything with ngrok as the host. I always wanted to be able to group by @http.host, so I added that to the group by field.

So my query was defined as:

service:requests @http.status_code:(4* OR 5*) [email protected]:*.ngrok.io

Group by: @http.host

Give the metric a description if you wish and save it.

Now, you can create a Monitor via the Metric definition. In the query box you can use the metric name you defined above. So for me this was outbound.request.failures. You should now be able to follow the guide on the Datadog website to configure a monitor