TOP SECFI TO LISA, FVEY -51fur-'- {fijxkeyscore@nsa I. TOP 5. TO USA. TOP SECRETHCOMINTHREL TO USA. FVEY sic Syntax is similar to C: functionCname', level, {optional inf-3::- 'search terms and patterns'; - Two main functions -- appid and ?ngerprint: 8.5, wireshark='icq', chatproc='lCO') and ?ag: 'user-agent: nokia' or 'pro?le: TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USA. FVEY nventions I. Appids are named using a pseudo directory conven?on: /applica tion_type/sub_type/name TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TD USA. FVEY Levels are 1.0 9.9 with lower numbers meaning higher priority. This allows multiple signatures to match a piece of traf?c, but only the most speci?c appiol will be applied. For example: appid('chat', 9.9) 9.8) 9.7 If a session matches all three signatures, the appicl will be 'chat/yahoo/incoming' since that has the best priority. TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USA. FVEY Iiti I. Third parameter is the application type; if missing, we use the appid name up to the ?rst slash as the type 9.2, 'web') 9.1) TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USA. FVEY I Patterns XKEYSCORE supports Boolean operations and regular expressions Raw text must be encapsulated between single quotes 'search term' Terms can be combined with Boolean logic - 'search term' and 'another term' and not 'olefeat term' - 'search term' or 'another term' TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USA. FVEY :7 -2 appidCVin/Sip/IMS', 6.0, wireshark='sip') ('Via: sip' or sip') and 'cseqz' and 'p-access-network-infoz' or 'p-called-party-idz' or 'p-charging-vectorz' or or 'p-media?authorizationz' or 'security?verifyz' or 'proxy-authorization:' and or 'path:' and or 'path:' and TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USABinary patterns can be represented by putting a \x in front of each value: '\xff\xff\x00\x0 2' Or use the hex function: Use slashes to enclose regular expressions: TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USA. FVEY iivit I. Keywords and regular expressions are NOT case sensitive by default. Append a to request case-sensitive evaluation: 'keyword'c /regex/c TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TD USA. FVEY I Len 0 th Keywords must be at least 3 characters or they will never hit. This minimum is increased to 4 at some sites for performance reasons. Regular expressions must include a ?xed "anchor" meeting the minimum keyword length. Bad: OK: TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USA. _:Oiro in 0 er rint Each session gets one appid -- lowest level wins. It gets databased in the 'application' ?eld. All matching ?ngerprints are stored in the '?ngerprint' ?eld. Level is ignored and can be omitted from ?ngerprint de?nitions. ?gplioationTyoe*: ioationInfo*: . . . I Winning appid ?oolioetion: I - I E/all fingerprints ?fir-DID E+Finoernrint?i* [tum?Bet]: [Field Builder] TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL To USA. Flevi HP 9.0) 'Hest: mail.yahee'; 3.0) 'Hest: mail.yahee' and '/1egin'; 'mail' and 'Hest: mail.yahee' and GET flegin.htm1 Referer: Accept-Language: ar Accept-Encoding: gzip, deflate User-Agent: Mozilla/4.9 (compatible; MSIE 6.0; Windews NT 5.1; SV1) Hest: mail.yahee.eem Connection: Keep-Alive Cookie: B=fn59ehd2612e2&b=3&s=rp; Application: mail/yahee/Iegin Fingerprint: mail/yahee/Iegin mail/arabic mail/yahoe/ymbm TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TD use. FVEY orl exa mar tewee tsweb 1m.le I mcp?m JAM ?34m? smea- Jus- -- ere-Isl Begin ASRAR El Mejshedeen will Message Hi! Lu IME Ejg'l Mj Y1 ?Em?e EUwa Jwal MGIEMTAEED jElegE Hj?'l DEG-11 MTASH DD Rh?fjeSr-J mEINDFk?Dky?n?th?n?j?fEMD?1 dh?n?m IjliM? NTID Mje3 MIEIDWEWIW E1 ij??fj 3.9 I eEE?SRUIdUEWupz?hhgd-??ef MI I .l I k1 Urdu ?lm ?3'14? mg? End ASRAR El Mejshedeen Message Displs'xinglitemsl I Hidden Fields I TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USA. FVEY - Appids and ?ngerprints are distributed across the XKEYSCORE network every hour Changes will take effect within 2 hours of check-in Current de?nitions are available on the website: TOP SECRETHCOMINTHREL TO USA. TOP SECRI: TO USA, FVEY ragga- l-'Ina.? .1 I TOP 3. .RETHCUMINTHREL TO USA. FVEY TOP SECRETHCOMINTHREL TO USA. H0 ti on You can append derived metadata ?elds onto the end of an appid: 7.7, and net $http; This will result in an appid like 'p2p/kazaa/image/jpeg'. TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USA. FVEY cti 0 ns . VJ- ibi exbr) Matches against an IP Address looks in to address and from address in the session headere ip( '10.10.10.1' toportl expr Matches against the Destinationfl'o port. Note this must be a numeric representation of a port. fromportl exbr Matches against the SourceIFrom port. Note this must be a numeric representation of a port. - fromport( 80 l; bort( exbr) Matches against the either port. Note this must be a numeric representation of a port. -pon(566?l; next_protoco { exbr Matches against the integer version of the next protocol. next_protocol( 250 ('text') Will only work for IP next protocol names as de?ned in the IANA next protocol numbers document TOP SECRETHCOMINTHREL TO USA. - rriac_address(addr) smaeiaddr) dmaciaddr) ipiaddr) fram_ip(addr) te_ip(addr) TOP SECRETHCOMINTHREL TO USA. fuctions permutes just like strang_se ectar (just like DECODEORDAIN Tasks a mac address tasks this IP address {either ta er frerri) tasks this IP address anly when it is the ariginatar tasks this IP address anly when it is the destinatiari TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TD USA. FVEY fl in functions . rst(exp r) lposlexpr) posiexpr) betweenl expr ?term'c ?term'u Matches against a pattern at the beginning of the session Matches against a pattern at the beginning of each line (in) expression occurs at offset in the session - pos('Hello') - posUGood.*Grief? 10 - between('Hello', 'Worlcl', 10, 100) Separation between ?Hello? and ?World? is greater than or equal to 10 bytes and less than or equal to 100 bytes This is the same as using the following regular expression: I iHeIIo. Does a case sensitive match of the term Treats the term as UTF-16 TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TD USA. FVEY Ie 9.9, wireshark='skinny') port(2000); 3.0, wireshark='skinny') toport(2000) and 3.0, wireshark='skinny') from port(2000) and TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USA. FVEY 7 Ie . I'll-J 8.5, wireshark='smtp') toport(25) and or ?rst('ehlo') or ?rst('data') or (Ipos('To: and or pos('QUIT'c) or pos('mai from:') or pos('rcpt TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USA. RDs You can assign a pattern to a variable (CHAINWORD) and reuse the variable in many patterns. $sip 'via: sip' and 'Cseq:' and Now we can use this variable in future de?nitions: 7.2) $sip; 6.9) $sip and TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USA. Cha i nword There are a number of chainwerds prede?ned for convenience: $tcp - $http_clelete - $udp - $http_trace - $icmp - $http_head - $sctp - $http_eptiens $rpc - $http_partia $arp - $vbulletin - $ssl - $mime_type $http_cmcl - $user_agent $http $http_get - $http_put - $http_pest TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USA. $icq and $http and not (port(80) or $html_body or $http_cmd); 8.5, wireshark='icq', chatproc='lCQ') and $icq; 9.0, wireshark='icq', chatproc='lCQ') ?rst('icq') and not port(25); TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TD USA. FVEY sensitivit Expressions are evaluated only with a certain context instead of across the session as a whole. Mail' or 'Yahoo! Address Book') only hits if those keywords are seen within the title of a web page only hits within the "Host:" header TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USA. FVEY ensitivit I ?Conte Why use context-sensitive scanning? More intuitive - you can say what you mean More accurate - if 'maps.google.com' is mentioned in a blog post. you don't want to try processing it as a Google Maps session Better performance for XKEYSCORE TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USA. FVEY ontsensitivit Sample contexts: htm _tit e mename url ?le_ext http_host d0c_tit e http_referer http_c00kie gogauthor 0c org p'sewetr d0c_hash user_agen doc?body web search emall_b0dy chat_b0dy fr0m_cc TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USA. i 8.0) and ('exchange' or 'conver')) or or 3,0) or or TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USA. FVEY . - it appitl aptians: --help this help message -- ist-a l list all the applicatianf?ngerprint names and levels -- ist-appids list all the applicatien names (ne ?ngerprints) -- ist-?ngerprints list all the applicatian names (na appitls) -- ist-types list all the applicatian types -- ist-leuels list all the applicatian levels --unit-test perfarm unit tests with data in the heirachy 'tlatatlir', with ?les matching '?lespec' --quiet dan't print any laatl messages --appid_fname arg lacatian pf appid.cfg --input-?le arg input ?le ta test --t:latadir arg The test data directary. Defaults ta (XSCO IR Ha i sis --?lespee arg A regular expressian ta match against ?les to cheek --neexit arg do not step on the ?rst errer TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TD use. FVEY Vlidation appid sample.u124 Leading appida Leading Leading -::-Leading Leading -::-Leading -::-Leading Leading -::-Leading Finished leading appids Filename: sample.u124 Appid: Tetal Size: 19.35Kbit5 Tetal Time: 0.015ee5 Rate: 1.935Mbitai'a Overall perfermance: Tetal Time: 0.015325 Tetal Bits: 0.01936Mbit5 I{Zlilrerall Rate: 1.936Mbit5l?5 TOP SECRETHCOMINTHREL TO USA. TOP SECRI: TO USA, FVEY .1 .. I - -- Lb hm. #33 gr?: . Sl TO USA. .. . If" T0 TOP SECRETHCOMINTHREL TO USA. FVEY "Codeed a . ids Keywords and regular expressions don't work for everything - Looking down columns in packet data Checksums - Decoding (urlencoding, base64, gzip, etc.) TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USA. ased - ids Basic idea: 1. Preliminary "trigger" using standard keywords and regular expressions 2. Secondary test using a snippet of code TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USA. FVEY I a ids Example -- verifying a length ?eld: 2, wireshark='ospf') protocol('ospf') if (size() 4) return false; const uint3_t *data begin(); return TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USA. - ids Example -- packet data: 'Next Pretneel 25?' paeket_t pkt; int euunt a; while [(pkt eeunt 26) ++eeunt; if (pkt.eize 16) return false; if (pkt.data[4] axcc pkt.data[5] ?x45 pkt.data[15] 6x72) return false; return (enunt TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USA. FVEY tsensitive code?" Contex Example -- code-based check on certain extracted ?les: or 'x15' or 'ppt' return std::string::npus; TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USA. FVEY Simplified regex-based metadata extraction and ka [Keep-Alive: accept[] main if(ka ka[B] "333") fur(size_t i i "Encnding" accept[i][1] "gzip") return true; return false; TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TD USA. FVEY Support for ?ex-based pattern matching and flex "User-Agent: std::string agent(yytext); std::string::npus) return true; TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TD USA. FVEY I ?o in The next step: giving code-based appids (limited) access to the XKS core - Accessing top-level session metadata - Throwing common events Contributing metadata for databasing The goal: higher level of agility with lower learning curve TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USA. 0 0 ins . Example: accessing session metadata and main return TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TD USA. FVEY ins Example: throwing a document_metadata event and main xks::doc_meta_t dm; dm.filename "google.txt"; dm.author "Google, Inc."; TOP SECRETHCOMINTHREL TO USA. TOP SECRETHCOMINTHREL TO USA. FVEY ins Example: contributing metadata to Activity and extractors CI main 11?01) return true; TOP SECRETHCOMINTHREL TO USA.