patternjavascriptModerate
ELK stack basics: indexing structured logs for search and analysis
Viewed 0 times
Elasticsearch ^8.x
ELK stackElasticsearchKibanaFluent BitFilebeatlog shippingKQLindex lifecycle managementdata streams
Problem
Logs are scattered across pod stdout, server files, and various services. There is no central place to search logs across services or correlate events across a distributed system.
Solution
The ELK stack (Elasticsearch, Logstash, Kibana) or its modern replacement OpenSearch provides centralized log aggregation and search.
Architecture:
Key Kibana queries (KQL):
Tip: Use data streams instead of time-based indices for log data — they handle rollover and ILM automatically.
Architecture:
- Filebeat/Fluent Bit — lightweight log shipper running as a DaemonSet, reads container stdout
- Logstash (optional) — enrichment, transformation, routing
- Elasticsearch/OpenSearch — storage and indexing
- Kibana/OpenSearch Dashboards — search UI and dashboards
Key Kibana queries (KQL):
# Find errors for a specific service
service: "order-service" AND level: "error"
# Find logs for a specific trace
traceId: "4bf92f3577b34da6a3ce929d0e0e4736"
# Count errors by service in last 1h
# Use Lens or Data Visualizer for aggregationsTip: Use data streams instead of time-based indices for log data — they handle rollover and ILM automatically.
Why
Elasticsearch inverted indexes allow full-text and structured field search across billions of log records in milliseconds, which is impossible with grep on files.
Gotchas
- High cardinality fields (UUIDs, IPs) should be mapped as keyword, not text, to avoid bloating the index
- Index Lifecycle Management (ILM) must be configured to delete or rollover old indices — without it, disk fills up
- Logstash is resource-heavy — Fluent Bit or Vector are much lighter alternatives for log shipping
- Never store raw bytes or binary data in Elasticsearch — it will explode index size
Context
Setting up centralized log aggregation for a multi-service application
Revisions (0)
No revisions yet.