Gthulhu API Server

Gthulhu API Server is a Golang-based API server designed to integrate the Linux kernel scheduler (sched_ext) with the Kubernetes orchestration system. The core objective of this project is to collect scheduling metrics from eBPF programs and provide intelligent scheduling decisions to the kernel scheduler based on Kubernetes Pod labels and user-defined strategies.

DEMO

Click the image below to watch our DEMO on YouTube!

System Architecture

This API Server adopts a dual-mode architecture, consisting of two independent services:

┌─────────────────────────────────────────────────────────────────────────────────┐
│                              Gthulhu Architecture                               │
├─────────────────────────────────────────────────────────────────────────────────┤
│                                                                                 │
│   ┌─────────────┐         ┌─────────────────────┐         ┌─────────────────┐   │
│   │    User     │ ──────▶ │      Manager        │ ──────▶ │    MongoDB      │   │
│   │  (Web UI)   │         │ (Central Management)│         │  (Persistence)  │   │
│   └─────────────┘         └──────────┬──────────┘         └─────────────────┘   │
│                                      │                                          │
│                                      │ Query Pods via K8s API                   │
│                                      ▼                                          │
│                           ┌─────────────────────┐                               │
│                           │   Kubernetes API    │                               │
│                           │   (Pod Informer)    │                               │
│                           └─────────────────────┘                               │
│                                      │                                          │
│              ┌───────────────────────┼───────────────────────┐                  │
│              │                       │                       │                  │
│              ▼                       ▼                       ▼                  │
│   ┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐           │
│   │ Decision Maker  │     │ Decision Maker  │     │ Decision Maker  │           │
│   │   (Node 1)      │     │   (Node 2)      │     │   (Node N)      │           │
│   └────────┬────────┘     └────────┬────────┘     └────────┬────────┘           │
│            │                       │                       │                    │
│            ▼                       ▼                       ▼                    │
│   ┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐           │
│   │  sched_ext      │     │  sched_ext      │     │  sched_ext      │           │
│   │ (eBPF Scheduler)│     │ (eBPF Scheduler)│     │ (eBPF Scheduler)│           │
│   └─────────────────┘     └─────────────────┘     └─────────────────┘           │
│                                                                                 │
└─────────────────────────────────────────────────────────────────────────────────┘

Manager Mode

Purpose: Serves as the central management service, responsible for handling user requests, managing scheduling strategies, user authentication, and access control.

Key Features:

User authentication and authorization (JWT Token)
Role and permission management (RBAC)
CRUD operations for scheduling strategies
Monitor Pod status via Kubernetes Informer
Distribute scheduling intents to Decision Makers on each node
Data persistence to MongoDB

Default Port: :8081

Decision Maker Mode

Purpose: Deployed on each Kubernetes node (as DaemonSet), responsible for receiving scheduling intents and interacting with the local sched_ext scheduler.

Key Features:

Receive scheduling intents from Manager
Scan /proc filesystem to discover Pod processes
Convert scheduling strategies into concrete PID-based scheduling decisions
Collect eBPF scheduler metrics
Expose metrics via Prometheus endpoint

Default Port: :8080

Core Features

Manager Service Features

User Management: Create, query users, password reset
Role & Permission Management: RBAC role management, permission assignment
Scheduling Strategy Management: Create Pod label-based scheduling strategies
Scheduling Intent Tracking: Track strategy execution status
Kubernetes Integration: Real-time Pod monitoring via Pod Informer
JWT Authentication: RSA asymmetric encryption Token authentication

Decision Maker Service Features

Intent Processing: Receive and process scheduling intents from Manager
Process Discovery: Parse cgroup information to map PIDs to Pods
Scheduling Strategy Provider: Provide concrete PID scheduling strategies to sched_ext
Metrics Collection: Collect and expose eBPF scheduler metrics to Prometheus
Token Authentication: Validate requests from Manager

API Endpoints

Manager Endpoints

System Endpoints

Endpoint	Method	Description
`/health`	GET	Health check
`/version`	GET	Version information
`/swagger/*`	GET	Swagger documentation

Authentication Endpoints

Endpoint	Method	Description
`/api/v1/auth/login`	POST	User login

User Management Endpoints

Endpoint	Method	Description
`/api/v1/users`	POST	Create user
`/api/v1/users`	GET	List users
`/api/v1/users/password`	PUT	Reset password
`/api/v1/users/permissions`	PUT	Update permissions
`/api/v1/users/self/password`	PUT	Change own password
`/api/v1/users/self`	GET	Get own information

Role Management Endpoints

Endpoint	Method	Description
`/api/v1/roles`	POST	Create role
`/api/v1/roles`	GET	List roles
`/api/v1/roles`	PUT	Update role
`/api/v1/roles`	DELETE	Delete role
`/api/v1/permissions`	GET	List permissions

Scheduling Strategy Endpoints

Endpoint	Method	Description
`/api/v1/strategies`	POST	Create scheduling strategy
`/api/v1/strategies/self`	GET	List own strategies
`/api/v1/intents/self`	GET	List own scheduling intents

Decision Maker Endpoints

Endpoint	Method	Description
`/health`	GET	Health check
`/version`	GET	Version information
`/metrics`	GET	Prometheus metrics
`/api/v1/auth/token`	POST	Get authentication token
`/api/v1/intents`	POST	Receive scheduling intents
`/api/v1/scheduling/strategies`	GET	Get scheduling strategies
`/api/v1/metrics`	POST	Update metrics data

Data Structures

ScheduleStrategy

Field	Type	Description
`strategyNamespace`	string	Strategy namespace
`labelSelectors`	[]LabelSelector	Pod label selectors
`k8sNamespace`	[]string	Kubernetes namespaces
`commandRegex`	string	Process command regex
`priority`	int	Priority level
`executionTime`	int64	Execution time (nanoseconds)

ScheduleIntent

Field	Type	Description
`podID`	string	Pod UID
`podName`	string	Pod name
`nodeID`	string	Node name
`k8sNamespace`	string	Kubernetes namespace
`commandRegex`	string	Process command regex
`priority`	int	Priority level
`executionTime`	int64	Execution time (nanoseconds)
`podLabels`	map[string]string	Pod labels
`state`	string	Intent state

MetricSet

Field	Type	Description
`usersched_last_run_at`	uint64	User scheduler last run timestamp
`nr_queued`	uint64	Number of tasks in scheduling queue
`nr_scheduled`	uint64	Number of scheduled tasks
`nr_running`	uint64	Number of running tasks
`nr_online_cpus`	uint64	Number of online CPUs
`nr_user_dispatches`	uint64	Number of user-space dispatches
`nr_kernel_dispatches`	uint64	Number of kernel-space dispatches
`nr_cancel_dispatches`	uint64	Number of cancelled dispatches
`nr_bounce_dispatches`	uint64	Number of bounce dispatches
`nr_failed_dispatches`	uint64	Number of failed dispatches
`nr_sched_congested`	uint64	Number of scheduler congestion events

Quick Start

0. Test Environment Setup

For local development, you need to set up a MongoDB instance. You can use Docker to start the infrastructure:

# Start MongoDB using Docker
$ docker run --rm --network host mongo:6.0 mongosh "mongodb://test:test@localhost:27017/?authSource=admin&authMechanism=SCRAM-SHA-256" --eval 'db.runCommand({ ping: 1 })'

After starting MongoDB, create the required user and verify the connection:

# Create a test user with root privileges in MongoDB
$ docker exec mongodb mongosh --eval 'db.getSiblingDB("admin").createUser({user:"test",pwd:"test",roles:[{role:"root",db:"admin"}]})'

# Verify the connection with the created user credentials
$ docker exec mongodb mongosh "mongodb://test:test@localhost:27017/?authSource=admin&authMechanism=SCRAM-SHA-256" --eval 'db.runCommand({ ping: 1 })'

1. Install Dependencies

go mod tidy

2. Configuration

Manager Configuration (`config/manager_config.toml`)

[server]
host = ":8081"

[logging]
level = "info"

[mongodb]
host = "localhost"
port = "27017"
user = "test"
password = "test"
database = "manager"

[k8s]
kube_config_path = "/path/to/.kube/config"
in_cluster = false

[key]
rsa_private_key_pem = "..."
dm_public_key_pem = "..."
client_id = "your-client-id"

[account]
admin_email = "admin@example.com"
admin_password = "your-password"

# mTLS for Manager → Decision Maker communication (optional, default: disabled)
[mtls]
enable = false
cert_pem = "..."   # Manager's client certificate (signed by private CA)
key_pem  = "..."   # Manager's client private key
ca_pem   = "..."   # Private CA certificate (to verify Decision Maker's server cert)

Decision Maker Configuration (`config/dm_config.toml`)

[server]
host = ":8080"

[logging]
level = "info"

[token]
rsa_private_key_pem = "..."
token_duration_hr = 24

# mTLS server for Manager → Decision Maker communication (optional, default: disabled)
[mtls]
enable = false
cert_pem = "..."   # Decision Maker's server certificate (signed by private CA)
key_pem  = "..."   # Decision Maker's server private key
ca_pem   = "..."   # Private CA certificate (to verify Manager's client cert)

3. Start Services

Start Manager

go run main.go manager

# With custom configuration
go run main.go manager -c manager_config -d /path/to/config

Start Decision Maker

go run main.go decisionmaker

# With custom configuration
go run main.go decisionmaker -c dm_config -d /path/to/config

4. Test API

Please refer to https://github.com/Gthulhu/chart?tab=readme-ov-file#testing for testing the API endpoints using curl.

mTLS Setup: Manager ↔ Decision Maker

The Manager communicates with every Decision Maker node using mutual TLS (mTLS). Both sides authenticate each other with certificates signed by a shared private CA, so neither plain-text traffic nor untrusted connections are accepted.

Note: The Manager's external HTTP API (web GUI, /api/v1/…) intentionally remains plain HTTP. In a production cluster this endpoint is typically exposed through a Kubernetes Ingress with TLS termination.

Why mTLS?

Scheduling decisions affect the Linux kernel scheduler on every node. A compromised connection between the Manager and a Decision Maker could allow an attacker to manipulate per-process CPU priorities. mTLS provides:

Server authentication – the Manager verifies it is talking to a genuine Decision Maker.
Client authentication – the Decision Maker verifies only the authorised Manager can push intents.
Encrypted channel – all scheduling intents are protected in transit.

Step-by-step: Generate certificates with a private CA

The commands below use only the OpenSSL CLI. Replace <DM_IP> with the actual IP or hostname of each Decision Maker node.

1. Create the private CA

# Generate CA private key (EC P-256 recommended; RSA-4096 also works)
openssl ecparam -name prime256v1 -genkey -noout -out ca.key

# Self-signed CA certificate (10-year validity)
openssl req -new -x509 -days 3650 \
  -key ca.key \
  -out ca.crt \
  -subj "/CN=Gthulhu-Private-CA"

2. Generate the Manager client certificate

# Manager private key
openssl ecparam -name prime256v1 -genkey -noout -out manager.key

# Certificate signing request
openssl req -new \
  -key manager.key \
  -out manager.csr \
  -subj "/CN=gthulhu-manager"

# Sign with the private CA (2-year validity, client-auth EKU)
openssl x509 -req -days 730 \
  -in manager.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -extfile <(printf "extendedKeyUsage=clientAuth") \
  -out manager.crt

3. Generate a Decision Maker server certificate

Repeat for each DM node, setting the correct IP/DNS in subjectAltName.

# Decision Maker private key
openssl ecparam -name prime256v1 -genkey -noout -out dm.key

# CSR
openssl req -new \
  -key dm.key \
  -out dm.csr \
  -subj "/CN=gthulhu-decisionmaker"

# Sign with the private CA (2-year validity, server-auth + client-auth EKUs + SAN)
openssl x509 -req -days 730 \
  -in dm.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -extfile <(printf "subjectAltName=IP:<DM_IP>\nextendedKeyUsage=serverAuth,clientAuth") \
  -out dm.crt

4. Embed certificates in configuration

Paste the PEM file contents into the respective config files.

config/manager_config.toml

[mtls]
enable = true
cert_pem = """
-----BEGIN CERTIFICATE-----
<contents of manager.crt>
-----END CERTIFICATE-----
"""
key_pem = """
-----BEGIN EC PRIVATE KEY-----
<contents of manager.key>
-----END EC PRIVATE KEY-----
"""
ca_pem = """
-----BEGIN CERTIFICATE-----
<contents of ca.crt>
-----END CERTIFICATE-----
"""

config/dm_config.toml

[mtls]
enable = true
cert_pem = """
-----BEGIN CERTIFICATE-----
<contents of dm.crt>
-----END CERTIFICATE-----
"""
key_pem = """
-----BEGIN EC PRIVATE KEY-----
<contents of dm.key>
-----END EC PRIVATE KEY-----
"""
ca_pem = """
-----BEGIN CERTIFICATE-----
<contents of ca.crt>
-----END CERTIFICATE-----
"""

5. Verify

Start both services and confirm the Decision Maker log contains:

starting dm server with mTLS on port :8080

And the Manager log shows successful intent reconciliation without TLS errors.

Kubernetes: mounting certificates as Secrets

In a production deployment, store PEM content in Kubernetes Secrets and mount them as environment variables or files, then reference them in the TOML config.

kubectl create secret generic gthulhu-mtls-certs \
  --from-file=ca.crt \
  --from-file=manager.crt \
  --from-file=manager.key

Kubernetes Deployment

Deployment Architecture

Manager: Deployed as a Deployment, typically single replica
Decision Maker: Deployed as a DaemonSet on every node

Deployment Manifest Locations

deployment/
├── kind/                    # Kind local test environment
│   ├── local_setup.sh
│   ├── decisonmaker/
│   │   ├── daemonset.yaml
│   │   └── service.yaml
│   ├── manager/
│   │   ├── deployment.yaml
│   │   └── service.yaml
│   └── mongo/
│       ├── secret.yaml
│       ├── service.yaml
│       └── statefulset.yaml
└── local/
    └── docker-compose.infra.yaml  # Docker Compose for local development

Quick Deployment

# Local testing with Kind
cd deployment/kind
./local_setup.sh

# Or manual deployment
kubectl apply -f deployment/kind/mongo/
kubectl apply -f deployment/kind/manager/
kubectl apply -f deployment/kind/decisonmaker/

RBAC Permission Requirements

Manager requires the following Kubernetes RBAC permissions:

pods: list, watch, get
namespaces: list, get

Development Guide

Project Structure

.
├── main.go                 # Entry point, uses Cobra for subcommands
├── config/                 # Configuration definition and parsing
├── manager/               # Manager service
│   ├── app/              # Application initialization
│   ├── cmd/              # Cobra commands
│   ├── domain/           # Domain models and interfaces
│   ├── k8s_adapter/      # Kubernetes client
│   ├── client/           # Decision Maker client
│   ├── repository/       # MongoDB data access
│   ├── rest/             # HTTP handlers
│   ├── service/          # Business logic
│   └── migration/        # Database migrations
├── decisionmaker/        # Decision Maker service
│   ├── app/             # Application initialization
│   ├── cmd/             # Cobra commands
│   ├── domain/          # Domain models
│   ├── rest/            # HTTP handlers
│   └── service/         # Business logic and process discovery
├── pkg/                  # Shared packages
│   ├── logger/          # Logging utilities
│   ├── middleware/      # HTTP middleware
│   └── util/            # Common utility functions
└── deployment/          # Deployment configurations

Tech Stack

Web Framework: Echo v4
Dependency Injection: Uber fx
CLI: Cobra
Logging: zerolog
Database: MongoDB (mongo-driver v2)
Kubernetes Client: client-go
Metrics Collection: Prometheus client_golang
Configuration Management: Viper

Build and Test

# Build
make build

# Run tests
make test

# Build Docker image
make image

Container Images

This project uses GitHub Actions to automatically build and publish to GitHub Container Registry.

# Use latest version
docker pull ghcr.io/gthulhu/api:main

# Use specific version
docker pull ghcr.io/gthulhu/api:v1.0.0

License

This project is open source.

Name		Name	Last commit message	Last commit date
Latest commit History 113 Commits
.github		.github
config		config
decisionmaker		decisionmaker
deployment		deployment
docs/manager		docs/manager
manager		manager
pkg		pkg
web		web
.gitignore		.gitignore
.mockery.yaml		.mockery.yaml
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
go.mod		go.mod
go.sum		go.sum
main.go		main.go
notice.txt		notice.txt

License

Gthulhu/api

Folders and files

Latest commit

History

Repository files navigation

Gthulhu API Server

DEMO

System Architecture

Manager Mode

Decision Maker Mode

Core Features

Manager Service Features

Decision Maker Service Features

API Endpoints

Manager Endpoints

System Endpoints

Authentication Endpoints

User Management Endpoints

Role Management Endpoints

Scheduling Strategy Endpoints

Decision Maker Endpoints

Data Structures

ScheduleStrategy

ScheduleIntent

MetricSet

Quick Start

0. Test Environment Setup

1. Install Dependencies

2. Configuration

Manager Configuration (config/manager_config.toml)

Decision Maker Configuration (config/dm_config.toml)

3. Start Services

Start Manager

Start Decision Maker

4. Test API

mTLS Setup: Manager ↔ Decision Maker

Why mTLS?

Step-by-step: Generate certificates with a private CA

1. Create the private CA

2. Generate the Manager client certificate

3. Generate a Decision Maker server certificate

4. Embed certificates in configuration

5. Verify

Kubernetes: mounting certificates as Secrets

Kubernetes Deployment

Deployment Architecture

Deployment Manifest Locations

Quick Deployment

RBAC Permission Requirements

Development Guide

Project Structure

Tech Stack

Build and Test

Container Images

License

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 9

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Manager Configuration (`config/manager_config.toml`)

Decision Maker Configuration (`config/dm_config.toml`)

Packages