Crate gc_gateway

The main binary crate that can be considered as the “glue” between all the components of the Plugin runtime.

Architecture

Below is a diagram that shows how the plugin runtime is structured and how the components interact with each other.

%% Runtime Architecture (detailed)
flowchart TD
    subgraph Plugins["Plugins (Same Process)"]
        P1["Plugin 1"]
        P2["Plugin 2"]
        Pn["..."]
    end
    subgraph Runtime["Runtime"]
        RT["Runtime Core"]
        Plugins
        ABI["C ABI"]
    end
    %% External services
    SH_EXT["Service Hub"]
    LS_EXT["Log Service"]
    RT_EXT["Remote Runtime"]

    P1 -- "C SDK / Rust SDK" --- ABI
    P2 -- "C SDK / Rust SDK" --- ABI
    Pn -- "C SDK / Rust SDK" --- ABI
    ABI --- RT
    RT -- "HTTP (Config/Control)" --> SH_EXT
    RT -- "TCP (Log)" --> LS_EXT
    RT -- "TCP (Remote Runtime)" --> RT_EXT
    SH_EXT --- SH_NOTE2["Note: Service Hub sends config, receives status"]
    LS_EXT --- LS_NOTE2["Note: Log Service receives logs from Runtime"]
    Plugins --- PLUG_NOTE2["Note: Plugins extend runtime with data sources, sinks, processing. All plugins connect via the C ABI and use C or Rust SDKs."]

Data flow

Below is a diagram that shows how the data flows through the plugin runtime. Specifically this shows the processing of a single datapoint from the moment that is published until it is received by some subscriber.

sequenceDiagram
    participant Publisher as Publisher
    participant MemoryAllocator as MemoryAllocator
    participant MessageBroker as MessageBroker
    participant RealtimeDatabase as Realtime Database
    participant Subscriber1 as Subscriber N

    Publisher->>MemoryAllocator: 1. Store DatapointValue
    MemoryAllocator-->>Publisher: 2. Get reference
    Publisher->>MessageBroker: 3. Sends DatapointValue reference
    MessageBroker->>RealtimeDatabase: 4. Store reference
    RealtimeDatabase-->>MessageBroker: 


    loop For each subscriber
        MessageBroker->>Subscriber1: 5. Sends datapoint reference
    end

The publisher and subscribers are usually plugins but can also be other components of the system. The message broker also has the ability to dispatch the data in the same context as the publisher (synchronous) or in a different thread (asynchronous).

Communication between runtimes

Each runtime can communicate with other runtimes using TCP, this allows for a distributed system where multiple runtimes can be running on different machines and communicate with each other, all the communication is handled by the [gc_runtime_tcp] crate.
The diagram below shows how the communication between runtimes works.

sequenceDiagram
    participant RuntimeA as Runtime A (Initiator)
    participant RuntimeB as Runtime B (Receiver)

    Note over RuntimeA,RuntimeB: TCP connection established

    Note over RuntimeA,RuntimeB: Subscription Phase (must happen first)
    RuntimeA->>RuntimeB: TCPAnnounceSubscribed
    RuntimeB->>RuntimeA: TCPAnnounceSubscribed

    Note over RuntimeA,RuntimeB: Data Exchange Phase
Packets can flow in any order
    par Bidirectional Exchange
        loop Continuous
            RuntimeA->>RuntimeB: TCPDatapointValue
        end
        loop Continuous
            RuntimeB->>RuntimeA: TCPDatapointValue
        end
    end

The connection can be initiated by either side, and this is managed by the Service Hub which will command (via HTTP) a runtime to establish connection with a remote one. The port of the tcp connection can be configured via CLI using the --runtime-tcp-port flag.

How does the runtime get configuration

Once the runtime starts it starts an REST API server and announces itself to the Service Hub, also trough HTTP. Once the Service Hub receives the announcement, it will push configurations to the runtime using HTTP. So both the runtime and the Service Hub act as servers, the runtime being the one that receives the configuration and the Service Hub being the one that sends it.

Examples

Check all the available options:

gc_gateway --help

Start the plugin runtime, using custom async queue capacity and threads:

gc_gateway --async-queue-capacity 2000 --threads 4 --port 8080

Start the plugin runtime specifying a custom host and runtime id for the config hub:

gc_gateway --config-hub localhost:3002 --runtime-id testing

Modules

config_hub_notifier 🔒

Structs

Args

Command line arguments for the Gateway

Functions

run