GCP Cloud Logging Integration

Forward logs and metrics from Google Cloud Platform into Operyn for AI-powered incident detection and diagnosis.

[!NOTE] The GCP connector is currently in Beta. Some features like automated deployment via Terraform are being finalized.

Architecture

Operyn ingests logs from GCP using a Log Sink that forwards entries to a Cloud Pub/Sub topic. A Cloud Function then triggers on each message, maps the raw log data to Operyn's RawLogInput format, and sends it to the Ingestion API.

GCP Log Sink (Filters)
        │
        ▼
   Cloud Pub/Sub
        │
        ▼
  Cloud Function ──► POST /events/logs/batch ──► Operyn Ingestion

Step 1: Create a Pub/Sub Topic

1. Go to the Pub/Sub section in your GCP Console.
2. Create a new topic named operyn-logs-ingest.
3. (Optional) Set a message retention period (default 7 days is fine).

Step 2: Create a Log Sink

1. Go to Logging → Log Router.
2. Create absolute "Sink".
3. Sink Name: operyn-forwarder
4. Sink Destination: Cloud Pub/Sub topic operyn-logs-ingest.
5. Inclusion Filter: Define which logs to forward. For example:

   resource.type="gce_instance" OR resource.type="k8s_container"
   severity >= WARNING
   

Step 3: Deploy the Cloud Function

Deploy a Cloud Function (Node.js 20) to process the messages and forward them to Operyn.

Function Code (index.js)

const { OPERYN_INGEST_URL, OPERYN_API_KEY, OPERYN_ORG_ID } = process.env;

exports.handler = async (pubSubEvent, context) => {
  const data = JSON.parse(Buffer.from(pubSubEvent.data, 'base64').toString());
  
  const body = [{
    service: data.resource.labels.service_name || data.resource.type,
    level: data.severity.toLowerCase(),
    message: data.textPayload || JSON.stringify(data.jsonPayload),
    timestamp: data.timestamp,
    metadata: {
      source: 'gcp',
      projectId: data.resource.labels.project_id,
      resourceType: data.resource.type,
    }
  }];

  const response = await fetch(`${OPERYN_INGEST_URL}/events/logs/batch`, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'x-api-key': OPERYN_API_KEY,
      'x-organization-id': OPERYN_ORG_ID,
    },
    body: JSON.stringify(body),
  });

  if (!response.ok) {
    throw new Error(`Ingestion failed: ${await response.text()}`);
  }
};

Step 4: Configure Environment Variables

Set the following variables on your Cloud Function:

Next Steps

• AWS CloudWatch Integration — for multi-cloud monitoring.
• Ingestion API Reference — for custom log formats.
• Concepts: Incidents — learn how log events trigger incidents.