Splunk
Splunk is the most widely deployed SIEM in the world. It collects logs from every system you own, makes them searchable in seconds, and turns raw events into detections, alerts and dashboards.
What it is
A SIEM — Security Information and Event Management — solves one problem: logs are scattered across dozens of systems and nobody can look at them all. Splunk pulls those logs into one searchable index so you can ask questions across everything at once.
Its query language, SPL (Search Processing Language), is the reason it dominates. A single SPL query can filter, extract fields, calculate statistics, correlate across sources and render a chart. That is how a SOC analyst goes from "something happened" to "here is exactly what happened, on which host, from which IP, and how often" in one line.
Splunk Enterprise is free to download with a daily ingestion limit (500 MB/day) — which is more than enough to learn on, and enough for a small home lab. Splunk Cloud and enterprise licensing are paid. For learning, the free tier plus public datasets (like Boss of the SOC) is the standard path.
How to use Splunk
Follow it exactly as you would in a real session — every command, every line of output, and what each result actually tells you. This is a interface walkthrough.
Pick the index and the time range first
index= and the time picker are what keep a search at under a second instead of timing out. Searching index=* over a year is the classic beginner mistake.Ask a question with stats
Ask the follow-up question: did any succeed?
Turn it into a saved alert with throttling
Flags & options
| FLAG | WHAT IT DOES |
|---|---|
| index= | Which data store to search. Always specify it. |
| sourcetype= | How Splunk parses the log. Wrong sourcetype = no fields. |
| | stats count by field | Aggregate — the most-used command in the language. |
| | eval | Create or transform a field. |
| | where | Filter on a calculated field (stats output). |
| | rex field=_raw "…" | Extract a field the parser missed. |
| | timechart span=1h | Time-series aggregation for graphs. |
| | table a, b, c | Choose and order the columns. |
| | sort - count | Sort descending by count. |
| | top limit=10 | Most frequent values. |
| tstats | Very fast search over indexed fields. |
| Save As → Alert | Schedule the search and fire when results appear. |
Why it matters
A SOC analyst's entire value is asking the right question of the right data fast. Splunk is that interface. The skill is not memorising commands — it is knowing that "failed logins" alone mean nothing, while "failed logins followed by one success from the same IP" is a compromise.
How it works
The architecture has three roles, and understanding them explains most performance problems:
- Universal Forwarder — a small agent on each machine that ships logs to the indexer. It does not parse; it just forwards.
- Indexer — receives data, breaks it into events, extracts the timestamp, and writes it to disk in buckets. This is where storage and CPU are consumed.
- Search Head — where you type SPL. It distributes the search to indexers and merges the results.
In a small setup all three run on one machine. The concept that matters when you write queries is sourcetype: it tells Splunk how to parse a log format (Linux syslog, Windows WinEventLog:Security, JSON from an application, and so on). If the sourcetype is wrong, fields will not extract, and every query after that becomes painful.
SPL is a pipeline. Each | passes results to the next command:
- search / index — narrow the data first. Always.
- stats — aggregate:
count by src_ip. - eval — create or transform fields.
- rex — extract fields with a regular expression when parsing did not.
- timechart — time-series aggregation for graphs.
- table / sort / head — shape the output.
Aggregates results into a table: counts, sums, averages, distinct values.
TYPICAL USE| stats count by src_ip — who is doing the thing most often.
Always narrow the search before stats. Stats over a huge time range is how dashboards time out.
Creates or transforms fields using expressions — the logic layer.
TYPICAL USE| eval severity=if(count>100,"high","low") — turn numbers into decisions.
Eval does not filter. Pair it with where to act on the new field.
Extracts fields with a regular expression when the parser did not.
TYPICAL USE| rex field=_raw "user=(?<username>\w+)"
If you are writing many rex commands, the sourcetype is probably wrong — fix parsing instead.
Time-series aggregation — the basis of almost every dashboard panel.
TYPICAL USE| timechart span=1h count by host — spot the host that behaves differently at 03:00.
Choose the span deliberately: 1m over 30 days is unreadable, 1d over an hour is empty.
Commands & syntax
# narrow first — always specify index and time range index=main sourcetype=linux_secure # failed SSH logins grouped by source IP index=main sourcetype=linux_secure "Failed password" | stats count by src_ip | sort - count
# brute force: more than 10 failures from one IP in 5 minutes index=main sourcetype=linux_secure "Failed password" | bin _time span=5m | stats count by src_ip, _time | where count > 10 # Windows: successful logon type 3 (network) outside business hours index=win EventCode=4624 Logon_Type=3 | eval hour=strftime(_time,"%H") | where hour < 7 OR hour > 20 | stats count by Account_Name, src_ip
# extract a field the parser missed index=main sourcetype=app_log | rex field=_raw "user=(?[a-zA-Z0-9._-]+)" | stats count by username # create a severity field and use it index=main | eval severity=if(count>100,"high",if(count>10,"medium","low")) | table severity, src_ip, count
# events per hour by host — the basis of a panel index=main | timechart span=1h count by host # top talkers for a table panel index=main | top limit=10 src_ip
# Save As → Alert → run every 5 minutes → trigger when results > 0 # Add throttling, or you will drown in duplicate alerts index=main sourcetype=linux_secure "Failed password" | stats count by src_ip | where count > 10
splunk@errordex:~$ search "index=main sourcetype=linux_secure \"Failed password\" | stats count by src_ip, user | sort - count | head 5" index=main sourcetype=linux_secure "Failed password" | stats count by src_ip, user | sort - count | head 5 src_ip user count --------------- ----------- ----- 203.0.113.44 root 412 203.0.113.44 admin 87 203.0.113.44 oracle 41 198.51.100.9 deploy 6 10.0.0.14 root 2 # 412 failures against root from one external IP in 15 minutes. # Next question: did ANY of them succeed? index=main sourcetype=linux_secure "Accepted password" src=203.0.113.44 | table _time, user, src_ip _time user src_ip -------------------- ------- ------------- 2026-11-11 03:42:19 root 203.0.113.44 # Failures then a success = compromised. That is the finding.
Common mistakes
Searching everything, always
index=* across a year of data will time out. Narrow by index and time first — it is the difference between two seconds and two minutes.
Wrong sourcetype
Fields do not extract, so every query becomes a regex fight.
Alert fatigue
A detection that fires 200 times a day gets ignored within a week — which is exactly how real incidents are missed.
No throttling on alerts
The same event generates fifty emails.
Ignoring the free-tier limit
Exceed 500 MB/day and indexing stops; your dashboards silently go empty.
Treating Splunk as a log archive
It is a detection engine. Data nobody queries is just disk usage.
Practice safely
Free forever: Splunk Enterprise has a free licence with a 500 MB/day limit — enough for a home lab with a couple of VMs and a firewall sending syslog.
Public datasets: Boss of the SOC (BOTS) is a set of real-world attack datasets released by Splunk specifically for practice, with published answers. Working through BOTS v1 is the single fastest way to learn SOC investigation in SPL.