Building a Cloud-Native SIEM: Integrating AWS CloudWatch with Elastic Stack at Scale

The Challenge

When I led the security engineering team at a rapidly growing fintech company, we faced a critical challenge: our existing SIEM solution couldn't handle our explosive log growth (>10TB daily) while maintaining sub-minute alert latency. With regulatory requirements demanding 12-month log retention and real-time threat detection, we needed a scalable, cloud-native solution.

Technical Background

SIEM Architecture Requirements

• Ingest and process >100,000 events per second

• Support complex correlation rules across multiple data sources

• Maintain sub-minute alert latency

• Enable ML-based anomaly detection

• Ensure compliance with SOC 2 and PCI DSS requirements

Solution Design

High-Level Architecture

Key Components

1. Log Collection Layer

   • CloudWatch Logs as primary collector

   • Kinesis Firehose for buffering and batching

   • Lambda for real-time enrichment

2. Processing Layer

   • Elasticsearch cluster for storage and analysis

   • Custom ML models for anomaly detection

   • Alert correlation engine

3. Presentation Layer

   • Kibana dashboards

   • Custom alert management interface

   • Automated response workflows

Implementation Journey

1. Setting Up the Log Pipeline

First, let's configure the CloudWatch to Elasticsearch pipeline:

# CloudFormation template snippet for Kinesis Firehose
Resources:
  LogFirehose:
    Type: 'AWS::KinesisFirehose::DeliveryStream'
    Properties:
      DeliveryStreamName: 'siem-log-stream'
      ElasticsearchDestinationConfiguration:
        DomainARN: !Ref ElasticsearchDomainArn
        IndexName: 'logs-#{timestamp:yyyy.MM.dd}'
        BufferingHints:
          IntervalInSeconds: 60
          SizeInMBs: 50
        RetryOptions:
          DurationInSeconds: 300

2. Implementing Log Enrichment

Created a Lambda function for real-time log enrichment:

import json
import boto3
import requests

def enrich_log(event, context):
    # Decode and enrich CloudWatch log data
    enriched_records = []
    for record in event['records']:
        # Decode and parse the log data
        data = json.loads(record['data'])

        # Enrich with threat intelligence
        if 'sourceIPAddress' in data:
            threat_intel = query_threat_intel(data['sourceIPAddress'])
            data['threatIntel'] = threat_intel

        # Add geo-location data
        if 'sourceIPAddress' in data:
            geo_data = get_geo_location(data['sourceIPAddress'])
            data['geoLocation'] = geo_data

        enriched_records.append({
            'recordId': record['recordId'],
            'result': 'Ok',
            'data': json.dumps(data)
        })

    return {'records': enriched_records}

3. Elasticsearch Configuration

Optimized Elasticsearch for high-throughput ingestion:

{
  "index.refresh_interval": "10s",
  "index.number_of_shards": 5,
  "index.number_of_replicas": 1,
  "index.routing.allocation.total_shards_per_node": 3,
  "index.mapping.total_fields.limit": 2000
}

4. Setting Up ML-Based Detection

Implemented anomaly detection using Elasticsearch ML:

{
  "job_id": "unusual_auth_activity",
  "description": "Detect unusual authentication patterns",
  "analysis_config": {
    "bucket_span": "15m",
    "detectors": [
      {
        "function": "rare",
        "by_field_name": "user.name",
        "over_field_name": "source.ip"
      }
    ],
    "influencers": [
      "source.ip",
      "user.name",
      "auth.type"
    ]
  },
  "data_description": {
    "time_field": "@timestamp"
  }
}

Challenges Encountered

1. Performance Bottlenecks

Initially faced indexing bottlenecks. Solved by:

• Implementing hot-warm architecture

• Optimizing index templates

• Fine-tuning JVM settings

# Elasticsearch hot node configuration
cluster.routing.allocation.awareness.attributes: temp
node.attr.temp: hot

2. Alert Noise Reduction

Implemented correlation rules to reduce false positives:

def correlate_alerts(alerts):
    correlated = []
    for alert in alerts:
        if alert['severity'] >= 7:
            related = find_related_events(alert, timewindow='5m')
            if len(related) >= 3:
                create_incident(alert, related)

Validation and Monitoring

Performance Metrics

• Achieved 150,000 events per second ingestion

• Alert latency reduced to <30 seconds

• False positive rate decreased by 76%

Monitoring Dashboard

Created a custom Kibana dashboard for SIEM health monitoring:

{
  "visualization": {
    "title": "SIEM Health Metrics",
    "type": "metrics",
    "params": {
      "index_pattern": "siem-metrics-*",
      "interval": "1m",
      "time_field": "@timestamp",
      "metrics": [
        {"field": "ingestion_rate", "aggregation": "avg"},
        {"field": "processing_latency", "aggregation": "max"},
        {"field": "alert_count", "aggregation": "sum"}
      ]
    }
  }
}

Business Impact

Security Improvements

• 99.99% log collection reliability

• 76% reduction in false positives

• 85% faster incident response time

Cost Optimization

• 40% reduction in storage costs through optimized retention

• Automated response reduced manual investigation time by 60%

Resources and References

1. AWS CloudWatch Logs Documentation

2. Elasticsearch Performance Tuning Guide

3. Security Information and Event Management (SIEM) Implementation Guide

This implementation has been running in production for over a year, processing over 3 petabytes of security logs while maintaining sub-minute alert latency. The key to success was focusing on performance optimization at every layer of the stack and implementing intelligent correlation to reduce alert noise.

Remember to regularly review and update your SIEM rules and ML models as threat landscapes evolve. Security is a journey, not a destination.