Log Service

Overview

The Log Service (gc_log_service) is a centralized logging service that collects, stores, and provides access to logs from various applications and services in the NextGen Gateway ecosystem. It receives log entries over TCP, stores them in CSV files, and allows clients to query logs based on different criteria.

Features

Core Functionality

• Centralized Logging: Aggregates logs from multiple services and runtimes
• Persistent Storage: Stores logs in configurable directory as CSV files
• Log Querying: Allows searching logs by service name, time range, severity, and more
• TCP Interface: Simple TCP-based protocol for receiving and querying logs

Syslog Forwarding (v0.7.0+)

Starting with version 0.7.0, the Log Service supports syslog forwarding to external syslog servers. This feature enables integration with existing enterprise syslog/SIEM solutions for centralized log management.

Syslog Forwarding

Overview

The syslog forwarding feature allows the Log Service to forward received log entries to external syslog servers using UDP, TCP, or TLS protocols. This is completely optional and disabled by default, ensuring backward compatibility with existing deployments.

Use Cases

Protocols Supported

Message Formats

Configuration

Storage Modes

The Log Service supports three storage modes:

Note: When using syslog mode, logs are NOT stored locally. Ensure your syslog server is reliable or consider using both mode for redundancy.

Quick Start

Enable Syslog Forwarding via Environment Variables

gc_log_service \
  --storage-mode both \
  --syslog-address logserver.company.com \
  --syslog-port 514 \
  --syslog-protocol udp \
  --syslog-facility 1

Enable via Configuration File

Add to your service configuration:

{
  "storage_mode": "both",
  "syslog_enabled": true,
  "syslog_address": "logserver.company.com",
  "syslog_port": 514,
  "syslog_protocol": "udp",
  "syslog_facility": 1,
  "syslog_format": "rfc5424",
  "syslog_app_name": "nextgen_gateway",
  "syslog_hostname": "gc-gateway-01"
}

Full Configuration Options

Core Settings

Syslog Destination

Syslog Formatting

TLS Configuration (TLS protocol only)

Buffer Configuration (TCP/TLS protocols only)

Environment Variables

All configuration options can also be set via environment variables:

# Core
export GC_LOG_STORAGE_MODE=both
export GC_LOG_SYSLOG_ENABLED=true

# Destination
export GC_LOG_SYSLOG_ADDRESS=logserver.company.com
export GC_LOG_SYSLOG_PORT=514
export GC_LOG_SYSLOG_PROTOCOL=udp

# Formatting
export GC_LOG_SYSLOG_FACILITY=1
export GC_LOG_SYSLOG_APP_NAME=nextgen_gateway
export GC_LOG_SYSLOG_HOSTNAME=gc-gateway-01
export GC_LOG_SYSLOG_FORMAT=rfc5424

# TLS
export GC_LOG_SYSLOG_TLS_CA_CERT=/etc/ssl/certs/syslog_ca.pem
export GC_LOG_SYSLOG_TLS_CLIENT_CERT=/etc/ssl/certs/client.pem
export GC_LOG_SYSLOG_TLS_CLIENT_KEY=/etc/ssl/private/client.key
export GC_LOG_SYSLOG_TLS_VERIFY_SERVER=true

# Buffer Configuration
export GC_LOG_SYSLOG_BUFFER_BYTES=8000000
export GC_LOG_SYSLOG_RECONNECT_DELAY_MS=100
export GC_LOG_SYSLOG_MAX_RECONNECT_ATTEMPTS=5

Examples

Local Storage Only (Default)

# Store logs in CSV files only (backward compatible)
gc_log_service \
  --writer-port 3003 \
  --reader-port 3004 \
  --log-dir /var/log/gc_gateway

Forward to UDP Syslog Server

gc_log_service \
  --storage-mode syslog \
  --syslog-enabled true \
  --syslog-address 192.168.1.100 \
  --syslog-port 514 \
  --syslog-protocol udp \
  --syslog-facility 1 \
  --log-dir /var/log/gc_gateway

Local + UDP Syslog (Redundancy)

gc_log_service \
  --storage-mode both \
  --syslog-address logserver.local \
  --syslog-port 514 \
  --syslog-protocol udp \
  --log-dir /var/log/gc_gateway

TCP Syslog with RFC 5424 Format

gc_log_service \
  --storage-mode syslog \
  --syslog-address logserver.company.com \
  --syslog-port 6514 \
  --syslog-protocol tcp \
  --syslog-format rfc5424 \
  --syslog-app-name nextgen_gateway \
  --log-dir /var/log/gc_gateway

TLS with Server Certificate Verification

gc_log_service \
  --storage-mode both \
  --syslog-address logserver.company.com \
  --syslog-port 6514 \
  --syslog-protocol tls \
  --syslog-format rfc5424 \
  --syslog-tls-ca-cert /etc/ssl/certs/syslog_ca.pem \
  --syslog-tls-verify-server true \
  --log-dir /var/log/gc_gateway

Mutual TLS (mTLS)

gc_log_service \
  --storage-mode both \
  --syslog-address logserver.company.com \
  --syslog-port 6514 \
  --syslog-protocol tls \
  --syslog-tls-ca-cert /etc/ssl/certs/syslog_ca.pem \
  --syslog-tls-client-cert /etc/ssl/certs/client.pem \
  --syslog-tls-client-key /etc/ssl/private/client.key \
  --syslog-tls-verify-server true \
  --log-dir /var/log/gc_gateway

TCP with Custom Buffer and Reconnection Settings

gc_log_service \
  --storage-mode both \
  --syslog-address logserver.company.com \
  --syslog-port 6514 \
  --syslog-protocol tcp \
  --syslog-format rfc5424 \
  --syslog-buffer-bytes 16000000 \
  --syslog-reconnect-delay-ms 200 \
  --syslog-max-reconnect-attempts 10 \
  --log-dir /var/log/gc_gateway

This configuration uses a 16MB buffer, starts with a 200ms reconnection delay, and attempts up to 10 reconnections before logging an error.

Message Format Examples

RFC 3164 (BSD Syslog)

<14>Jun 17 14:30:45 gc-gateway gc_gateway:[runtime_1] Starting plugin

• <14> is the PRI (Priority): facility * 8 + severity
• Facility 1 (user-level) + Severity 6 (info) = 1*8 + 6 = 14

RFC 5424 (Structured Syslog)

<14>1 2026-08-17T14:30:45.123Z gc-gateway gc_gateway - - - [gc_gateway@12345 plugin="runtime_1"] Starting plugin

• <14>1 is the PRI and version
• ISO 8601 timestamp
• Structured data includes plugin information

Syslog Facilities

Syslog Severities

Architecture

Data Flow

The syslog forwarding feature integrates seamlessly with the existing log service architecture:

Client → [TCP Writer] → Log Processing
                                    ├──→ Log Store (CSV Files)  [if storage_mode ≠ syslog]
                                    └──→ Syslog Forwarder → Syslog Server  [if storage_mode ≠ local]

For UDP protocol, messages are sent directly to the syslog server immediately.

For TCP and TLS protocols, messages are:

1. Queued in the message buffer
2. Sent by a background worker task
3. Automatically retried with exponential backoff if the connection fails

Background Worker (TCP/TLS)

The syslog forwarder spawns a dedicated background worker task for TCP and TLS protocols that:

• Continuously processes messages from the buffer
• Establishes and maintains the connection to the syslog server
• Automatically reconnects when the connection is lost
• Verifies connection health after each write operation
• Re-queues messages at the front of the buffer if sending fails

This ensures non-blocking operation - the log service can continue receiving and processing logs even when the syslog server is temporarily unavailable.

Severity Mapping

NextGen Gateway log levels are mapped to syslog severities as follows:

Security Considerations

TLS Configuration

• Use TLS 1.2 or higher (default)
• Verify server certificates in production environments
• Protect private keys with appropriate file permissions (0600)
• Feature flag: TLS support requires the tls feature to be enabled when building the service

Note: The gc_log_service binary must be built with the --features tls flag to enable TLS protocol support. Without this feature, selecting the TLS protocol will result in a warning message and no forwarding.

Message Sanitization

• Newlines (\n), carriage returns (\r), and null bytes (\0) are automatically removed from messages
• This prevents syslog injection attacks

Network Security

• Use firewall rules to restrict access to the syslog port
• Consider IP allowlisting for syslog servers
• Use dedicated VLANs for logging traffic in secure environments

Performance Considerations

Error Handling

Connection Failures

The Log Service implements robust error handling for syslog forwarding:

• TCP and TLS connections use exponential backoff retry with configurable parameters
• Base delay: 100ms by default, doubles on each retry (100ms, 200ms, 400ms, 800ms, 1600ms)
• Maximum attempts: 5 retries by default before logging an error
• Failed connections are logged with warning messages
• Connection health verification: After each write, the service verifies the connection is still alive by attempting a small read with timeout

Message Buffering (TCP/TLS only)

For TCP and TLS protocols, the Log Service implements a message buffer to handle temporary network issues:

• Messages are queued in memory when the syslog server is unavailable
• Buffer size: Configurable, default is 8MB (approximately 80,000 messages at 100 bytes each)
• When the buffer is full, oldest messages are dropped to make room for new ones
• The background worker continuously attempts to send buffered messages with automatic reconnection
• Once the connection is restored, all buffered messages are sent in order

Behavior by Storage Mode

Troubleshooting

Syslog Messages Not Appearing

1. Check syslog server is running: systemctl status rsyslog (or your syslog daemon)
2. Check network connectivity: telnet logserver 514 or nc -zv logserver 514
3. Check firewall rules: Ensure the gateway can reach the syslog server
4. Check server accepts connections: Verify the syslog server is configured to accept connections from the gateway’s IP
5. Test manually: echo "<14>Test" | nc -u logserver 514

TLS Connection Failures

1. Check TLS is enabled: Ensure the --syslog-protocol tls option is set
2. Check certificates: Verify CA certificate, client cert, and key paths are correct
3. Check certificate validity: Ensure certificates are not expired
4. Check server certificate: Verify the server certificate is trusted by the CA
5. Test with openssl: openssl s_client -connect logserver:6514 -CAfile /path/to/ca.pem

Connection Refused

1. Check syslog server is running: Verify the syslog daemon is active
2. Check port is listening: netstat -tulnp | grep 514 or ss -tulnp | grep 514
3. Check server accepts remote connections: Some syslog servers only listen on localhost by default

Invalid Configuration

1. Check facility value: Must be between 0 and 23
2. Check port value: Must be between 1 and 65535
3. Check protocol value: Must be udp, tcp, or tls
4. Check format value: Must be rfc3164 or rfc5424
5. TLS protocol requires certificates: If using TLS, ensure certificate paths are provided

Buffer Full Warnings

1. Check buffer size: If you see “Buffer full” messages, increase syslog_buffer_bytes
2. Check syslog server availability: The syslog server may be down or overloaded
3. Check network connectivity: Ensure the gateway can reach the syslog server
4. Increase buffer size: For high-volume environments, consider increasing to 16MB or 32MB
5. Use both mode: To prevent log loss, use storage_mode=both so logs are stored locally even if syslog forwarding fails

Migration Guide

For Existing Deployments

No changes required for existing deployments using local storage only. The syslog forwarding feature is completely optional and disabled by default.

Enabling Syslog Forwarding

To enable syslog forwarding for an existing deployment:

1. Add syslog configuration to your service startup:

   gc_log_service \
     --storage-mode both \
     --syslog-address logserver.local \
     --syslog-port 514

2. Or use environment variables:

   export GC_LOG_STORAGE_MODE=both
   export GC_LOG_SYSLOG_ADDRESS=logserver.local
   export GC_LOG_SYSLOG_PORT=514

3. Restart the service:

   systemctl restart gc_log_service

Rollback

If you need to rollback to a version without syslog support:

1. Stop the new service
2. Reinstall the previous version
3. Start the service

Note: If you were using storage_mode = syslog, logs were NOT stored locally. Rolling back may result in lost logs that were sent during the new version’s operation.

References

• RFC 3164 - BSD syslog Protocol
• RFC 5424 - The Syslog Protocol
• RFC 5425 - TLS Transport for Syslog