📘

Availability

This workflow action is available on our Business and Enterprise for Incident Management pricing plans. Please contact our Sales Team to upgrade to an account with this action.

Description

Search and retrieve logs from Grafana Loki datasources using LogQL queries. This action queries your Grafana Loki logs within a specified time range and returns matching log entries along with their timestamps and labels.

During an incident, quick access to relevant logs is critical for identifying the root cause and understanding system behavior. The Grafana: Search Logs action enables your incident response team to automatically retrieve diagnostic data from your Grafana Loki logs as part of your incident workflow.

Common scenarios include:

• Error Investigation: Automatically pull error logs from affected services when an incident is triggered to understand what went wrong
• Performance Analysis: Retrieve logs showing slow queries, timeout errors, or resource constraints during performance incidents
• Security Events: Pull authentication logs, access logs, or security-related events for security incidents
• Correlation: Gather logs from multiple services to correlate events and identify cascading failures
• Documentation: Automatically capture relevant logs and attach them to the incident for post-incident review and analysis

By automating log retrieval, your team can focus on analysis and resolution rather than manually searching through log systems, reducing mean time to resolution (MTTR).

Instructions

1. If you have not done so already, please follow our instructions to Create an Incident Workflow.
2. When the instructions prompt you to add actions , select this action.
3. Enter the following Inputs and then click Save. Continue following instructions to Publish the Workflow. When the action runs, you will see the Outputs listed below.

Inputs

📘

Field References

Fields with the {+} icon accept Field References, which can be useful for referencing incident data or outputs created in prior workflow steps. To add Field References, click {+}, or enter {{, and select relevant fields. Please see our Field References article for more information.

Outputs

Examples

Here are some common LogQL query patterns you can use:

• Service-specific logs: {service="user-service"} or {app="checkout"}
• Filter by log level: {service="api"} |= "ERROR" or {job="app"} |= "WARN"
• Exclude patterns: {service="web"} != "health-check"
• Multiple filters: {environment="prod", service="payments"} |= "failed" |= "transaction"
• Namespace queries: {namespace="kube-system"} or {pod=~"nginx-.*"}
• Regex matching: {service="api"} |~ "error|exception|fatal"

📘

Tips

Start with a narrow time range: Use shorter time windows (1m-5m) for faster results, especially during active incidents.
Test your queries: Verify your LogQL queries in Grafana's Explore view before using them in workflows.
Adjust Max Lines: If you're not seeing expected results, increase the Max Lines parameter, but be aware this may slow response times.
Use specific labels: The more specific your label selectors, the faster and more relevant your results will be.
Combine with other actions: Use the Logs output with subsequent actions like "Add Note to Incident" to automatically document findings, or "Send Notification" to alert team members of specific log patterns.