Case study

Threat.gg

A threat intelligence platform built for security teams that need clean, correlated, queryable IOC data — not another firehose to triage.

Visit Threat.gg →

The problem

Most threat intelligence feeds are noisy. Indicators of compromise — IPs, domains, URLs, file hashes — flow in from dozens of sources at high volume, much of it stale, much of it duplicated, and almost none of it correlated to the broader infrastructure each indicator belongs to. Security teams end up with a feed that is technically rich but operationally unusable, drowning in alerts they cannot act on.

The Threat.gg opportunity was to build a platform that ingested feeds from many sources, correlated indicators across them, enriched each one with infrastructure context (ASN, geolocation, related domains, hosting history), and served the result via APIs that security teams could actually integrate into their SIEMs and SOAR platforms. The hard part was building the data infrastructure that could keep up with the ingestion rate while staying queryable in milliseconds.

Our approach

We designed Threat.gg as a streaming ingestion platform on top of Postgres and ClickHouse. Indicators flow in from feeds via dedicated workers, get normalized into a common schema, deduplicated, enriched with infrastructure context, and persisted. The query layer runs against Postgres for the canonical record and ClickHouse for the high-cardinality analytical workloads (counts, time-series, related-IOC lookups).

The API surface is small but composable: lookup by IOC, lookup by infrastructure (ASN, IP block, domain registrar), lookup by feed, and a streaming endpoint for new IOCs in a given category. We resisted feature creep — every new endpoint costs in maintenance, in documentation, and in the cognitive load of users figuring out which one to use. The result is an API that feels small and is easy to integrate.

Enrichment is the part that took the longest to get right. An IP address by itself is not useful — what matters is its ASN, its geolocation, its hosting provider, the historical domains pointed at it, and its reputation across other feeds. We built enrichment as a pipeline of cheap-first/expensive-later stages, with caching at every step, so the cost per IOC stays predictable as ingestion scales.

Stack

Backend

  • Go
  • PostgreSQL
  • ClickHouse
  • Redis

Streaming

  • NATS JetStream
  • Worker pools in Go
  • Backpressure-aware ingestion

Enrichment

  • MaxMind GeoIP
  • ASN data
  • Passive DNS
  • Custom feed adapters

API

  • REST + OpenAPI
  • Streaming endpoints
  • API key auth + rate limiting

Infrastructure

  • Kubernetes
  • Cloudflare Workers (edge cache)
  • Object storage for archives

Observability

  • OpenTelemetry
  • Prometheus + Grafana
  • Loki for logs

Outcome

Threat.gg ingests, normalizes, enriches, and serves cybersecurity threat intelligence at production scale. The platform supports both web-dashboard usage by security analysts and programmatic API integration by SIEMs and SOAR platforms.

The streaming ingestion architecture has scaled across multiple feed sources without re-architecting; new feeds are added by writing an adapter, not by changing the core pipeline. Query latency stays in single-digit milliseconds for the common lookup patterns even as the indicator dataset grows.

The enrichment pipeline has held up well as a design choice. Caching at every stage keeps marginal enrichment cost low, and the staged structure (cheap data first, expensive lookups only when justified) means we never pay for enrichment we will not use.

What we learned

  • In data platforms, ingestion architecture matters more than query architecture. If your ingestion cannot scale, no query layer saves you.
  • Postgres + ClickHouse is a powerful combination for OLTP-plus-analytics workloads. Use Postgres for the canonical record; use ClickHouse for the queries Postgres was not built for.
  • Small composable APIs beat large omnibus APIs. Every endpoint is a maintenance commitment.
  • Enrichment is a cost center, not a feature. Cache aggressively; structure the pipeline to skip expensive lookups when cheap ones suffice.
  • For security infrastructure, time-zone alignment with customers matters. Most SOC teams want vendors who can pick up the phone during business hours in the same hemisphere.

Have a project that needs the same standards?

Email us a paragraph about what you are building. We respond within one business day.

[email protected]