Multi-Cloud Networking Costs: Transit Gateway, VPC Peering, and Cross-Cloud Egress

The Networking Cost Taxonomy

Cloud networking costs have layers that compound on top of each other. Understanding the cost hierarchy is the first step to controlling it.

AWS Networking Cost Hierarchy (per GB):

FREE:
  Same EC2 instance, any protocol
  Same AZ within a VPC (EC2 to EC2)
  S3/DynamoDB via Gateway VPC Endpoints
  Public IP to Internet (inbound)

LOW COST:
  $0.01/GB  Cross-AZ traffic within same Region
  $0.01/GB  EC2 to VPC Endpoint (Interface) within same AZ

MEDIUM COST:
  $0.02/GB  VPC Peering cross-AZ (same region)
  $0.045/GB NAT Gateway data processing
  $0.045/GB Internet egress (first 10TB, then decreases)

HIGH COST:
  $0.05/GB  AWS Transit Gateway data processing
  $0.02-0.09/GB Cross-Region data transfer (varies by region pair)

VERY HIGH COST:
  $0.08/GB  AWS Direct Connect data transfer (dedicated 10Gbps)
  Variable  Multi-cloud egress: AWS to GCP/Azure ($0.08-0.09/GB)
  
  Note: Cloudflare exits this cost by charging flat rate regardless of egress

Cross-AZ Traffic: The Hidden Cost in EKS/ECS

import boto3
from collections import defaultdict

ec2 = boto3.client('ec2', region_name='us-east-1')

def analyze_cross_az_cost(cluster_name: str) -> dict:
    """
    Calculate estimated cross-AZ networking cost for an EKS cluster.
    Key insight: microservices in different AZs pay $0.01/GB each direction.
    """
    
    # Find all subnets in the VPC
    vpcs = ec2.describe_vpcs(
        Filters=[{'Name': 'tag:Name', 'Values': [f'{cluster_name}-vpc']}]
    )['Vpcs']
    
    if not vpcs:
        return {}
    
    vpc_id = vpcs[0]['VpcId']
    
    subnets = ec2.describe_subnets(
        Filters=[{'Name': 'vpc-id', 'Values': [vpc_id]}]
    )['Subnets']
    
    az_map = {s['SubnetId']: s['AvailabilityZone'] for s in subnets}
    
    print(f"VPC: {vpc_id}")
    print(f"Subnets by AZ: {defaultdict(list, {v: [] for v in set(az_map.values())})}")
    print()
    print("Cross-AZ cost reduction strategies:")
    print("  1. Use topology-aware routing in Kubernetes (topologySpreadConstraints)")
    print("  2. Pin stateful services to a single AZ")
    print("  3. Use AWS Load Balancer Controller with zone-affinity=strict")
    print("  4. For EKS: use TopologyAwareRouting in Services")
    
    return az_map

# Topology-aware routing in Kubernetes (added in K8s 1.27)
TOPOLOGY_ROUTING_SERVICE = """
apiVersion: v1
kind: Service
metadata:
  name: api
  annotations:
    service.kubernetes.io/topology-mode: Auto  # Route to same-AZ pods first
spec:
  selector:
    app: api
  ports:
    - port: 80
      targetPort: 8080
"""

print("Topology-aware routing reduces cross-AZ traffic by 60-80% for most workloads")

# Kubernetes: spread pods across AZs but reduce cross-AZ calls
# Use topologySpreadConstraints to distribute pods
# Use topology-aware routing to prefer same-AZ pods

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 6
  template:
    spec:
      topologySpreadConstraints:
        # Spread evenly across AZs (max 2 pods difference between AZs)
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: api
        # Spread across nodes within AZ
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: api
      containers:
        - name: api
          image: api:latest

Transit Gateway vs VPC Peering

Transit Gateway vs VPC Peering Cost Comparison:

VPC PEERING:
  Cost: $0.01/GB (cross-AZ) or $0.02/GB (cross-region)
  No per-attachment or per-hour cost
  Limitation: Does not support transitive routing
    (VPC-A can talk to VPC-B, VPC-B to VPC-C, but NOT VPC-A to VPC-C via B)
  Best for: Simple 1-to-1 VPC connections

TRANSIT GATEWAY:
  Cost: $0.05/GB processed + $0.05/hr per attachment
  Per attachment: $0.05 × 720hr = $36/month per VPC attached
  Supports transitive routing (hub-and-spoke)
  Best for: Many VPCs needing any-to-any connectivity

Cost Example: 5 VPCs, 100GB/month cross-VPC traffic each
  
  VPC Peering (full mesh: 10 connections):
    Setup: 10 peering connections (free)
    Data: 10 connections × 100GB × $0.01 = $10/month
    Total: $10/month
  
  Transit Gateway:
    Attachments: 5 VPCs × $36/month = $180/month
    Data: 5 × 100GB × $0.05 = $25/month
    Total: $205/month
  
  VPC Peering saves $195/month for this small setup.
  
Break-even: TGW pays off when VPC count is large enough
that full-mesh peering complexity exceeds operational savings.
Rule: <10 VPCs = peering, 10+ VPCs = consider Transit Gateway.

Cross-Region Cost Reduction

# Strategy: Use CloudFront to route cross-region traffic
# CloudFront charges $0.0075-0.02/GB vs $0.02-0.09/GB for direct cross-region

resource "aws_cloudfront_distribution" "api" {
  origin {
    domain_name = "api.us-east-1.example.com"
    origin_id   = "primary-region"
    
    custom_origin_config {
      http_port              = 80
      https_port             = 443
      origin_protocol_policy = "https-only"
    }
  }

  # Failover origin in eu-west-1
  origin {
    domain_name = "api.eu-west-1.example.com"
    origin_id   = "secondary-region"
    
    custom_origin_config {
      http_port              = 80
      https_port             = 443
      origin_protocol_policy = "https-only"
    }
  }

  origin_group {
    origin_id = "api-with-failover"
    
    failover_criteria {
      status_codes = [500, 502, 503, 504]
    }
    
    member { origin_id = "primary-region" }
    member { origin_id = "secondary-region" }
  }

  default_cache_behavior {
    target_origin_id       = "api-with-failover"
    viewer_protocol_policy = "redirect-to-https"
    cache_policy_id        = "658327ea-f89d-4fab-a63d-7e88639e58f6"  # CachingOptimized
    
    allowed_methods = ["DELETE", "GET", "HEAD", "OPTIONS", "PATCH", "POST", "PUT"]
    cached_methods  = ["GET", "HEAD"]
  }

  price_class = "PriceClass_100"  # Use only US/Europe/Japan PoPs
  enabled     = true
}

Cloudflare as a Multi-Cloud Networking Layer

Cloudflare Magic Transit and Workers provide a neutral networking plane
that eliminates cloud provider egress charges between clouds.

Traditional multi-cloud networking (AWS to GCP):
  1GB data transfer AWS to GCP:
    AWS egress: $0.09/GB
    GCP ingress: Free
    Total: $0.09/GB per 1TB = $92.16

With Cloudflare R2 (no egress fees) as intermediary:
  Store data in R2: $0.015/GB/month
  Serve from R2 to anywhere: $0 egress
  
  For multi-cloud ML training or data pipelines:
    Traditional: $0.09 × 10TB = $921.60/month egress
    Cloudflare R2: $0.015 × 10TB storage + $0 egress = $153.60/month
    Saving: $768/month (83%)

Cloudflare Workers (edge compute):
  Run code at 300+ PoPs without paying cloud egress
  $0.50 per 1M requests + $0.02 per 1M GB-seconds
  
  For API responses that go cross-region:
    Instead of: API in us-east-1 → user in Singapore ($0.09/GB)
    Workers edge function in Singapore handles response transformation
    Only fetch from origin once per cache TTL
    Near-zero egress charges

Conclusion

Multi-cloud and cross-region networking costs are highly manageable once you understand the cost hierarchy. Free intra-AZ traffic is the cheapest — keep latency-sensitive microservices in the same AZ using topology-aware routing. Use VPC Peering instead of Transit Gateway for small numbers of VPCs. Use CloudFront and Cloudflare to eliminate cross-region egress by caching at the edge.

Model your networking costs before you build the architecture, not after. A distributed system design that ignores data movement costs can easily add $5,000-50,000/month to the cloud bill for large organizations.