GCP Cloud Logging Integration

Forward logs and metrics from Google Cloud Platform into Nirofy for AI-powered incident detection and diagnosis.

[!NOTE] The GCP connector is currently in Beta. Some features like automated deployment via Terraform are being finalized.

Architecture

Nirofy ingests logs from GCP using a Log Sink that forwards entries to a Cloud Pub/Sub topic. A Cloud Function then triggers on each message, maps the raw log data to Nirofy's RawLogInput format, and sends it to the Ingestion API.

GCP Log Sink (Filters)
        │
        ▼
   Cloud Pub/Sub
        │
        ▼
  Cloud Function ──► POST /events/logs/batch ──► Nirofy Ingestion

Step 1: Create a Pub/Sub Topic

1. Go to the Pub/Sub section in your GCP Console.
2. Create a new topic named operyn-logs-ingest.
3. (Optional) Set a message retention period (default 7 days is fine).

Step 2: Create a Log Sink

1. Go to Logging → Log Router.
2. Create absolute "Sink".
3. Sink Name: operyn-forwarder
4. Sink Destination: Cloud Pub/Sub topic operyn-logs-ingest.
5. Inclusion Filter: Define which logs to forward. For example:

   resource.type="gce_instance" OR resource.type="k8s_container"
   severity >= WARNING
   

Step 3: Deploy the Cloud Function

Deploy a Cloud Function (Node.js 20) to process the messages and forward them to Nirofy.

Function Code (index.js)

const { OPERYN_INGEST_URL, OPERYN_API_KEY, OPERYN_ORG_ID } = process.env;

exports.handler = async (pubSubEvent, context) => {
  const data = JSON.parse(Buffer.from(pubSubEvent.data, "base64").toString());

  const body = [
    {
      service: data.resource.labels.service_name || data.resource.type,
      level: data.severity.toLowerCase(),
      message: data.textPayload || JSON.stringify(data.jsonPayload),
      timestamp: data.timestamp,
      metadata: {
        source: "gcp",
        projectId: data.resource.labels.project_id,
        resourceType: data.resource.type,
      },
    },
  ];

  const response = await fetch(`${OPERYN_INGEST_URL}/events/logs/batch`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "x-api-key": OPERYN_API_KEY,
      "x-organization-id": OPERYN_ORG_ID,
    },
    body: JSON.stringify(body),
  });

  if (!response.ok) {
    throw new Error(`Ingestion failed: ${await response.text()}`);
  }
};

Step 4: Configure Environment Variables

Set the following variables on your Cloud Function:

Next Steps

• AWS CloudWatch Integration — for multi-cloud monitoring.
• Ingestion API Reference — for custom log formats.
• Concepts: Incidents — learn how log events trigger incidents.