Splunk Introduction
COMP90073 Security Analytics
Dr. , CIS Semester 2, 2021
COMP90073 Security Analytics © University of Melbourne 2021
Copyright By PowCoder代写 加微信 powcoder
• WhatisSplunk&WhySplunk
• SplunkSoftware
• SearchProcessingLanguage(SPL)
COMP90073 Security Analytics © University of Melbourne 2021
What is Splunk & Why Splunk
A software for searching, monitoring, and analysing machine generated big data using a web-style interface
A typical web server log
Challenging to analyse multiple logs in real-time to detect security events!
COMP90073 Security Analytics © University of Melbourne 2021
What is Splunk & Why Splunk
Gartner 2020 Magic Quadrant for Security Information and Event Management (SIEM)
• Advancedthreatdetectionand response solution
– Userandentitybehavior analytics (UEBA)
– Endpointdetectionand response (EDR)
– Automatedthreatintelligence
– Real-timedashboardsand
– Andmore…
COMP90073 Security Analytics © University of Melbourne 2021
Splunk Software
• SplunkCapabilities
• SplunkArchitecture
• WhatCanbeIndexed
• WebInterfaceOverview
• Search&Reporting
• Events&Fields
• DefaultFields
• DataType&CommonOperators
COMP90073 Security Analytics © University of Melbourne 2021
Splunk Capabilities
• Collect,index,andcorrelatemachinedatainreal-time
– Indexing:transformingdataintoaseriesofeventsthatcontainsearchable
fields (e.g. IP addresses of source and destination in a network packet) • Index:ArepositoryforSplunkdata
• Generategraphs,reports,alerts,dashboardsandvisualizations
COMP90073 Security Analytics © University of Melbourne 2021
Splunk Architecture
• Datasources:logs,filesystems,Netflow,etc.
• Splunkforwarders:forwardsthedatafromdifferent
data input sources to the indexers
• Splunkindexers:createsandmanagesindexesfor the incoming data
Search Tier
Indexers Tier
Forwarders Tier
Data Sources
Forwarders with load balancing
Forwarders
• Splunksearchtier:includessearchheadsthat process the search queries from users on the indexed data
COMP90073 Security Analytics © University of Melbourne 2021
What Can be Indexed
COMP90073 Security Analytics © University of Melbourne 2021
Web Interface Overview
Splunk bar
Manage and run applications
Add forwarders or import data from file
Add custom dashboards for data visualisation
COMP90073 Security Analytics © University of Melbourne 2021
Search & Reporting
Search bar
Time range picker
Summary of indexed data
Rerun past searches
COMP90073 Security Analytics © University of Melbourne 2021
Event & Fields
Search command
Data Source: https://live.splunk.com/splunk-security-dataset-project
COMP90073 Security Analytics © University of Melbourne 2021
Default Fields
• Shellscripts,pythonscripts,Windowsbatchfiles,PowerShell,etc.,canbe used to customise the data indexing and generate useful fields
Type of field
List of fields
Internal fields:
Contain general information about events
Original raw data of an event
An event’s timestamp expressed in Unix time The time that an event was indexed
An address for an event within the index
The bucket that an event is stored in
_indextime
Description
COMP90073 Security Analytics © University of Melbourne 2021
Default Fields
Type of field
List of fields
Default fields:
Contain information about where an event originated
Hostname/IP address of the device that generated the event (e.g., cisco_router)
The name of the index in which a given event is indexed (e.g., default is “main”)
The number of lines an event contains
The punctuation pattern that is extracted from an event
The file, stream, or other input from which an event originates (e.g., stream:http)
The format of the data input from which the event originates (e.g. syslog)
The Splunk server containing the event An event’s timestamp value
sourcetype
splunk_server
Description
COMP90073 Security Analytics © University of Melbourne 2021
Default Fields
Type of field
Default datetime fields:
Contain additional searchable granularity to event timestamps
The hour in which an event occurred
The day of the month on which an event occurred
The minute in which an event occurred The month in which an event occurred
The seconds portion of an event’s timestamp
The day of the week on which an event occurred
The year in which an event occurred
The value of time for the local time-zone of an event
List of fields
date_minute
date_month
date_second
Description
COMP90073 Security Analytics © University of Melbourne 2021
Data Types & Common Operators
• Datatypes:bool,int,float,string
• Comparison operators: = != < <= > >= • Logicaloperators:AND,OR,NOT
– Clause“src_port!=80”isdifferentfrom“NOTsrc_port=80”
• Records with missing value of “src_port” field are returned in the
second clause but are not returned in the first one
– Ifnologicaloperatorisusedbetweenclauses,thedefaultoperatorisAND
• “src_port !=80 host=server01” is equivalent to “src_port !=80 AND host=server01”
COMP90073 Security Analytics © University of Melbourne 2021
Search Processing Language (SPL)
• FilteringResults
• Sorting&GroupingResults • Filtering&ModifyingFields
COMP90073 Security Analytics © University of Melbourne 2021
Common SPL Commands – Pipe
• Common search string in SPL: command1 | command2 | … | commandk
• Results after the pipe character “|” are used as input for its following command
• The pipe character is always followed by an SPL command
Command1 Command2 . . . Commandk
COMP90073 Security Analytics © University of Melbourne 2021
Common SPL Commands
• “search”commandisimplicitlyappliedinthebeginningofthesearchpipeline and you should not use it explicitly in this location
– Example:“src_port=80|topdest_ip”
“search” command is implicitly applied here
Category Description Commands
Filtering Taking a set of results and filtering search, where, dedup, Results them into a smaller set of results head, tail
Sorting Results
Reporting Results
Ordering (and optionally limiting the number of) results
Generating a summary of results for reporting
top/rare, table, stats,
chart, timechart
Grouping Grouping events for identifying transaction Results patterns
Filtering, Modifying, and Adding Fields
Filtering out some fields to focus on fields, replace, rename, most related ones, modifying or eval, rex, lookup
adding fields to enrich results
Source: https://docs.splunk.com/
COMP90073 Security Analytics © University of Melbourne 2021
Common SPL Commands – Syntax Tips
• Requiredargumentsareshowninanglebrackets<>
• Optionalargumentsareenclosedinsquarebrackets[] • Groupargumentsareshowninparenthesis()
• Repeatingargumentsareshownbyellipsis…
– Syntax:replace(
– Example:replace200WITHOK404WITH“NotFound”INstatus
HTTP status field in indexed data
COMP90073 Security Analytics © University of Melbourne 2021
Filtering the Results
COMP90073 Security Analytics © University of Melbourne 2021
Search command
• FilterseventsfromSplunkindexesgivenasetofqueriedconditions
• Syntax:search
• logical-expression
– comparison-expression
– index-expression
– time-opts Youcanalsousethetimerangepickerfortimeoptions
• Precedenceoflogicaloperatorsinsearchcommand:expressionswith parenthesis, then NOT then OR then AND
COMP90073 Security Analytics © University of Melbourne 2021
Search command: comparison-expression
•
– Examples: src_port < 100, src_ip=192.168.10.1
•
– Example: dest_port IN (21,80,8080)
– IN operator checks if a value is a member of a group of values
• Search command examples for the toy HTTP data: – search status >= 400
• Returns events with error in HTTP requests – search status IN (401,403)
• Returns events with unauthorized or Forbidden HTTP requests
COMP90073 Security Analytics © University of Melbourne 2021
Search command: index-expression
• “
– Keywords or quoted phrases to match, Examples: fail*, login, “http://”
• Wildcard: asterisk wildcard (*) character is used to match an unrestricted number of characters in a string
•
–
– Example: sourcetype=syslog
• Searchexample:
– search sourcetype=stream:http fail* password
• Thisisequivalentto“searchsourcetype=stream:httpANDfail*ANDpassword”
COMP90073 Security Analytics © University of Melbourne 2021
Valid unit abbreviations
s, sec, secs, second, seconds
Search command: time-opts
• [
– timeformat=…
– Example:timeformat=%d/%m/%Y:%H:%M:%S – Defaulttimeformatis%m/%d/%Y:%H:%M:%S
•
– earliest,latest,_index_earliest,_index_latest,now(),time() –
• Hint:youcanusethewebinterfaceforsettingthetimeoptions
m, min, minute, minutes
h, hr, hrs, hour, hours
d, day, days
w, week, weeks
mon, month, months
q, qtr, qtrs, quarter, quarters
y, yr, yrs, year, years
COMP90073 Security Analytics © University of Melbourne 2021
Tips for search command
• Fieldnamesarebydefaultcase-sensitive
• Literalsarenotcasesensitivebydefault
– Example:searchingforlogin,Login,or”Login”allreturnsameresults – UseCASE(
• CASE(Login) only returns events that include Login (not login)
• Splunksearchesforwholeword
– Searchresultsfor“fail”and“failure”☛useasteriskwildcard(*)☛fail*
• Forphrasesorfieldvaluescontainingbreakingcharacters,e.g.,whitespace,
commas, pipes, square brackets and equal sign use quotation marks
– Examples:host=“server1”
– Usebackslash(\)toscapequoteinthefiledvalue,e.g.,host=“server\”1” → looking for records with host name equal to
COMP90073 Security Analytics © University of Melbourne 2021
Where command
• Quotedstringsareinterpretedasliterals
• Unquoted strings are treated as a field name Compare two different fields
• CanalsobeusedwithINoperatorandavalue-list – Example:…|wheredest_portIN(80,8080)
• Precedenceoflogicaloperatorsinwhere:expressionswithparenthesis,then NOT then AND then OR
• Examples
– … | where src_port=dst_port
– … | where bytes_in>2*bytes_out
COMP90073 Security Analytics © University of Melbourne 2021
Head and tail commands
• Headreturnsthemostrecentresultsofasearch – … | head 25
• Tailreturnstheearliestresultsofasearch – …|tail15
• Iftheintegerargumentisnotgiven,bothcommandsreturn10resultsbydefault status>400 | tail 20
status>400 | head 20
BOTS: https://live.splunk.com/splunk-security-dataset-project
COMP90073 Security Analytics © University of Melbourne 2021
Sorting & Grouping Results
COMP90073 Security Analytics © University of Melbourne 2021
Sort command
• Tochangetheordering/numberoftheresults
• Syntax:sort[
• Defaultvalueoftheoptionalfieldcountis10,000;pass0toreturnalltheresults
• sort-by-clause:[±]
– The value of sort-filed can be a field (such as “src_port”) or
• auto(
• ip(
• Defaultsortingorderisascending
– Use minus sign for descending order, e.g., sort –src_port, +ip(src_ip)
• Examples:
– … | sort lastname, -firstname
– … | sort 100 -num(size), +str(source)
COMP90073 Security Analytics © University of Melbourne 2021
Transaction command
• Groupofconceptually-relatedeventsthatspanstime – Examples
• Different events from the same source and the same host
• Different events from different sources but from the same host • Similar events from different hosts and different sources
• A set of events related to a firewall intrusion incident
• Syntax:transaction[
• Thiscommandaddstwofieldstotherawevents:durationandeventcount
• Theargumentfield-listspecifiesonefieldormorefieldnamestogroupevents
into transactions based on the values of the field(s)
– Therelationshipamongthefieldscanbeconjunction,disjunction, transitive, …
COMP90073 Security Analytics © University of Melbourne 2021
Transaction command: transaction definition options
• transaction-definition-options
– endswith=
• To start or end a transaction if the filter-string is satisfied by an event
– maxspan=
• Events in the transaction must span less than integer specified for maxspan. Events that exceed the maxspan limit are treated as part of a separate transaction
– maxpause=
• To specify the maximum length of time for the pause between the
events in a transaction
– maxevents=
• To specify the maximum number of events in a transaction. The default value is1000.
– Anegativevalueforeachoftheseconstraintsmeansthatthereisnolimit on the its value
COMP90073 Security Analytics © University of Melbourne 2021
Transaction command: example
status>400 | transaction maxpause=1m src_ip,dest_ip | sort -eventcount
The source 40.80.148.42 is scanning the destination 192.168.250.70??
Acunetix is a vulnerability scanner
BOTS: https://live.splunk.com/splunk-security-dataset-project
COMP90073 Security Analytics © University of Melbourne 2021
Reporting Results
COMP90073 Security Analytics © University of Melbourne 2021
Commands for statistical calculations
• Calculateaggregatestatistics(average,count,sum,…)overaresultsset • Commands
– stats:returnsatableofresultswhereeachrowrepresentsasingleunique combination of the values grouped by a set of chosen fields
• See others: eventstats, streamstats, geostats
– chart:similartostatsbutcreatestabulardataoutputsuitableforcharting
– timechart:createsachartforastatisticalaggregationappliedtoafield against time as the x-axis
COMP90073 Security Analytics © University of Melbourne 2021
Stats command
Syntax: stats [partitions=
(
Lower-case “or” in these slides is used to show alternative available options
stats-agg-term:
– Choices of stats-func → next slide
– Input field argument can be an existing field-name (e.g., src_port) or evaled- field created using eval command inside stats
• stats count(eval(src_port=80)) → evaled-field is “eval(src_port=80)” – Wildcard field names can be used: this option returns separate results
applying stats-func on each field: stats count(eval(*_port=80))
– The optional argument [AS
and can be wildcard field names:
• Example 1: “stats count(eval(*_port=80)) AS *_port80”
COMP90073 Security Analytics © University of Melbourne 2021
Options for stats-func
Type of function
Supported functions and syntax
Aggregate functions
count() distinct_count() estdc() estdc_error()
exactperc
perc
sumsq() upperperc
Event order functions
Multi-value stats and chart functions
Time functions
earliest() earliest_time()
latest() latest_time()
More detail on the functions:
https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/CommonStatsFunctions
COMP90073 Security Analytics © University of Melbourne 2021
Stats command (example)
Execution per src_ip:
1. eval(if(status>=400,1,0))
2. stats command sums over the output of eval splitting by source IP address 3. sort command sorts the results
… | stats sum(eval(if(status>=400,1,0))) AS statusError BY src_ip | sort – statusError
Status Error for this source IP is much higher than others
BOTS: https://live.splunk.com/splunk-security-dataset-projectCOMP90073 Security Analytics © University of Melbourne 2021
Stats command (example)
COMP90073 Security Analytics © University of Melbourne 2021
Stats command: sparkline-agg-term
• Sparkline:aninlinechartthatappearswithintablecellsinsearchresultsto display time-based trends associated with the primary key of each row
• Syntax:sparkline(
– sparkline-funcoptions:count(),mean(),avg(),stdev(),min(),max(),etc. – span-lengthexamples:1d,10min,1mon
Example: index=* | stats sparkline(avg(bytes_*),1m) AS avg_bytes_* BY src_ip,dest_ip
These lines change as the search proceeds
BOTS: https://live.splunk.com/splunk-security-dataset-project
COMP90073 Security Analytics © University of Melbourne 2021
Stats command: other arguments
• partitions=
• allnum=
all of the values of that field are numerical
• delim=
COMP90073 Security Analytics © University of Melbourne 2021
Chart command
Syntax: chart (
[( BY
• row-split
–
– bin-options:bins,span,…
• Examples: bins=5, span=1min, … • column-split
–
– tc-options:
COMP90073 Security Analytics © University of Melbourne 2021
Compare stats and chart commands
chart count(eval(src_port=80)) AS port80 OVER dest_port bins=10 BY dest_ip
10.120.137.110
10.120.251.250
10.186.60.244
10.85.245.109
10000-20000
60000-70000
stats count(eval(src_port=80)) AS port80 BY dest_port, dest_ip
10.168.80.39
10.122.27.216
10.122.68.227
10.120.137.110
COMP90073 Security Analytics © University of Melbourne 2021
Top and rare commands
• top [
– Mostcommon(optionallyN)valuesforthefields – Example:“topsrc_ipdest_ip”
• rare [
– Leastcommon(optionallyN)valuesforthefields
• Twofieldsareaddedtoeventswhenusingtopandrare:countandpercentage • Optionalby_clauseisforgroupingandorderingtheresultsusingotherfields
top src_ip dest_ip dest_port top src_ip dest_ip by dest_port
40.80.148.42
192.168.250.70
23.22.63.114
192.168.250.70
40.80.148.42
192.168.250.40
40.80.148.42
192.168.250.70
23.22.63.114
192.168.250.70
40.80.148.42
192.168.250.40
COMP90073 Security Analytics © University of Melbourne 2021
Top and rare commands: options
• showcount=
• countfield=
• showperc=
• percentfield=
• limit=
程序代写 CS代考 加微信: powcoder QQ: 1823890830 Email: powcoder@163.com