TDP TD USA, AUS, CAN, GER, HZL I Introduction to Context Sensitive Scanning with X-KEYSCRE Fingerprints May 2010 TOP SECRETHCOMINTHREL USA, AUS, CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL uestion: How do you find your target?s activity in DNI traffic? TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GERpening uestion: What if you don?t know your targets E?rnail address? Or you?re trying to find new ones they may be using? What if the traffic you?re interested in doesn?t even contain an E-maii address? What do you do then? TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GERpening uestion: You may try to look for keywords or patterns to help find your target. But how do we scan for keywords in the large volumes of data we see in DNI collection? Won?t we get too many false hits? TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL - 7 ?FContext Sensitive Scanning Context sensitive scanning gives a powerful way to surgically target the traffic you?re interested in, by only applying the keywords in the manner in which the analyst intended them to be applied TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL Context SenSItIve For example, think about these scenarios: I want to look for documents from Iran that mention a banned item? - want to look for people doing web searches on Jihad from Kabul? i want to look for people using Mojahedeen Secrets From anIPhone? I want to look for documents containing this regular expression? It want to look for E-mails that mention words from various categories of interest to 1* How would you go about targeting those in passive TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL . J. ff Fingerprints can help! Fingerprints are an extremely flexible way to target DNI traf?c without the foreknowledge of a strong selector They take advantage of context sensitive scanning engine that has over 70 unique contexts that can be targeted. An XKS Fingerprint is simply a meta-data tag that gets applied to a session when a certain criteria is met Think of fingerprints as analyst-defined ?attributes? of a session TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GERfr,- ?There?s an App for that!? There are currently almost 10,000 Apple and Fingerprints in the full list is available from the NSA XKS Home Page Odds are there may already be a fingerprint for the traffic you?re interested in. If not you can easily create your own! TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GEREd: For example I?m an analyst in CT I want to find anytime Mojahadeen Secrets 2 is seen in DNI Traf?c. I?m an analyst in CP I want to ?nd E?mails or Documents relating to the Iranian Nuclear Procurement network I?m an analyst in I want to find traffic from a known botnet TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL Use Fingerprints! Field Builder Appl? Fin ge rprinte] mpiefwmd?iran?iriell v? mpiey'wmdg'irang'irielfediz Field Builder nquD [+Find er urinte] me je he rd rrie jeh ede r12 jahedenEfenEddedheeder id deri jahedenth id denE Fl," rne jah ede r1th id den 44 neryptienfme Ie _enI:I:Ided I Field Eluilder Appl? Fingerprints] tic-metre remande ie I bemeeiznleckenergrbetfce mma?dfil: mp bemetrbleckenergrbetfce mma?d?eyn bemetrhleckenergrbetfce rrirneridfweit TD USA, AUS. CAN, GER, NZL - . lrhu. a . mr 3cm? Em, nrz.IaEagle a mm a mm I 34 Imam TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL - etti Sta rted What are the basics of XKS Fingerprints? Simple XKS fingerprints are keyword or regular expression based signatures that are evaluated across the data collected and processed by KEYSCO RE TD USA, AUS. CAN, GER, NZL TDP TD USA, AUS, CAN, GER, NZL Getting Started *1 Int-1er . . Lilli-J F'r--1 [En ED33- 3432 a 3.2m?Dl?l?l?la :Uj? (mt: El ltwmu ?E-Jju? t?W?h qnlw' wim-HNI I3 L-I: I Begin El Mujahedeen Message E1Ejg'1 Mi EEldl MEIAENTIJIEITIUI MWMEID DEDWFIMIQ MD Iy? Tg?l g?ij DD I'u'1j CHM EBIDWEWIW E1 ij??f'i UWU I - ou?l?" mu? 3'9 I Egg Gul? u'rru rlE H1 ail-mm ?awJI . . I 91 lm Limb Liam m3 ubd.d End ASHER El Mujahedeen 1:212! Message 1 ltElTE I Hdzlen I TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL Boolean Equations Basic fingerprints can also use Boolean equatlons 6-9: Wireah?rkg?ip') ('Via: sip' or sip') and 'cseq:' and nr 'p?called?party?idf 'p?charging-vectorf or or 'p?media?authorization:' or or 'proXy-authorizationf and or 'path:' and or 'path and TD USA, AUS. CAN, GER, NZL - 9" TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, Ina-3. . MN pressions ..-: -. Regular Ex And Regular Expressions or or (?:Begin End) or Regular expressions must include a ?xed "anchor" meeting the minimum keyword length. Bad: OK: TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL . . Binary Patterns And Binary Patterns $http and and $http and hex(?5353480000000000?) and TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL . . . - Positional Logic 4 and 24; $http and 14.4 and 134) and 164 and ti: 204); TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL What?s not 0 For example, take the first scenario: want to look for documents from Iran that mention a banned item? 0? Just using keywords with Boolean equations, how could we restrict the term to only a document body and only coming from Iran? TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GERntext Sensitive Scanning context sensitive scanning engine allows you to explicitly say where you want a term to hit. As an early example, the Tech Strings in Documents capability allowed to restrict terms to only Email, Chat or Documents Bodies The full XKS Context Sensitive Scanning engine allows for over 70 unique contexts to be used as part of an fingerprint TD USA, AUS. CAN, GER, NZL TDP TD USA, AUS, CAN, GER, NZL Context Sensitive Scanning For example, take the first scenario: want to look for documents from Iran that mention a banned item? 0 Using the XKS context for Country Code (based on NKB information) and the XKS context for Document Bodies, this easily becomes: cc(?ir?) and item?) TD USA, AUS. CAN, GER, NZL TDP TD USA, AUS, CAN, GER, NZL Context Sensitive Scanning 0 As another example, let?s say we want to tag all lphone usage 0 Using the XKS context for User Agent this easily becomes TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL a: - i?c?u? . . - of RA Considerations XKS Fingerprints may not be USSID18 or HRA compliant if they are queried on by themselves For example, we may want to fingerprint the use of mobile web devices like the lPhone, so that attribute could be used as part of a more complex query. But querying for the lPhone ?ngerprint itself would be a USSID18 and I-IRA Violation. TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL x? Considerations But if you want to look for an IPhone user from an Iranian Proxy accessing his Mail.ru account: IP address: Either ?Fingeri?iint?jl Field Builder 11me (+Fingerprint5] Field Builder i mailfweljmailfmailru mailfweljmailfmailru AFFID browser iphor'lel I mailfweljrnailfmailrufpost Acidic: Field Close TD USA, AUS. CAN, GER, NZL TDP TD USA, AUS, CAN, GER, NZL .. .. Sensitive Sca ning What contexts are available for use in XKS Fingerprints? TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL HTTP Activity Contexts (1 of 2) html_title(eapr) The normalized extracted text web page titles to? and ?bomb?) http_host(expr) The ?Host? name given in the header. http_url(eapr) Every URL from HTTP GET and. POST commands. http_url_args(expr) All arguments given as part of a URL (ie. all text following the in a URL string) http_u http_referer(expr) The ?Refererz? URL given in the HTTP header http_language(expr) The normalized two letter iso?6393 language code as inferred from any and or header info or TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL 2 HTTPActivity Conte;ts (2 0T http_c00kie(expr) The ?Cookie:? ?eld given in the header. http_server(expr) The "Server:? type name in the header. 1? 0r ?Apaehe? http_user_agent(eXpr) The ?User-Agent" ?eld given in the header. 0r ?Chmme?) web_search(expr) The normalized extracted text from web searches 0r ?plague?) x_f0marded_f0r(expr) The X?Forwarded For IP address from the HTTP Header TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL med Contexts 1 of 2 The source or destination IP address of the session from_ip(expr) The source IP address of the session to_ip(expr) Every URL From HTTP GET and POST commands. IP subnet in CIDR notation. The source or destination TCP or UDP port nLunber. from_port(expr) The source TCP or UDP port number. from _port(?22?) The destination TCP or UDP port number. TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL rotocol Contexts 1 of 2 cc(expr) The country (either to OR from) based on IP address cc(?ir? or from_cc(expr) The source country based on IP address or ?pl? to_cc(expr) The destination country based on IP address or protocol(expr) The textual form of the 1P next protocol. next_protocol(expr) The textual form of the next protocol. ip_next_protocol(? mac_address(expr) The MAC address of the target network device. 'l TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL ommunlcatlon Based Contexts email_body(expr) The normalized text of all email bodies. to? and ?b nild? and (?bomh? or ?weapon?? chat_body(expr) The UTF-8 normalized text of all chat bodies. to? and ?build? and (?bomb? or Weapon?) document_body(expr) The normalized text of the Omce document. O?ice documents include (but are not limited to) Microsoft O?ice, Open Office, Google Docs and Spreadsheets. to? and ?build? and (?bomb? or ?Weapon?D calendar_body(expr) The UTF-8 normalized text of all calendars. An example is Google Calendar. archive_files(expr) Matches a list of files from within an archive. For example is a ZIP ?le is transmitted, all names of?les within are passed to this context. or ?virus.doc') http_post_body(expr) The UTF-8 normalized text HTTP nrl-encoded POSTS. and ?badguy@yahoo?) TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL a Communication Based Contexts Aliases dec_email_bedy(expr) This eevers the email_bedy and decument_bedy contexts te? and ?build? and (?bemb? er ?weapen?D cemmunieationjwed?expr) This eevere the email_bedy, deeument_bedy and chat_bedy contexts te? and ?build? and (?bemb? er ?Weapen?D TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL ?ied-"F "f Context sensitivity Why use context?sensitive scanning? More intuitive you can say what you mean More accurate - if 'maps.google.com' is mentioned in a biog post, you don't want to try processing it as a Google Maps session Better performance for XKEYSCORE TD USA, AUS. CAN, GER, NZL TDP TD USA, AUS, CAN, GER-..-.Examples want to look for people doing web searches on Jihad from Kabul? Using the from_city() and web_search() context this becomes and TD USA, AUS. CAN, GER, NZL TDP TD USA, AUS, CAN, GER-..-.Examples 0 want to look for people using Mojahedeen Secrets from an IPhone? You can even use existing fingerprints in a fingerprint de?nition! So this becomes: and fingerprinthrowser/ cellphone /iphone?) TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL Examples 0 want to 100k for documents containing this regular expression? Using d0c_body this becomes: doc_b0dy( blah 5}something TD USA, AUS. CAN, GER, NZL . TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GERExample 4 0 want to 100k for E?mails that mention words from various categories of interest to You can use multiple variables in an equation like this: and sachositions and ($acwc0untries 0r $acwbr0kers or $acwp0rts)); TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL 4 $acwitems ?machine gun? or ?grenade? 01* 47? $acwpositi0ns ?minister of defence? or ?defense minister? C- $acwc0untries ?somalia? or ?liberia? or ?sudan? $acwbr0kers ?south africa? 01' ?serbia? 0r ?bulgaria? $acwp0rts ?I'angood? or ?albasra? or ?dar es salam? and $acwljositions and ($acwc0untries 0r $acwbr0kers 0r $acwp0rts)); TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL hr . . - Hf!- i s. Advanced Code-Based What happens when there are no keywords or regular expressions that will help identify the traf?c of interest to you? As enough example, many of the CT Targets are now smart enough to not leave the Mojahedeen Secrets header in the E?mails they send. How can we detect that the E?mail (which looks like junk) is in fact Mojahedeen Secrets text A code ?ngerprint can help evaluate that data TD USA, AUS. CAN, GER, NZL TDP TD USA, AUS, CAN, GER, NZL Code Based Fingerprint Chili-J I I3 L-I: I F-r--?Dl?l?l?la :Uj? (mt: El ltwmu ?E-Jju? t?W?h qnlw' wim-HNI IEI I'?i II E1Ejg'1 Mi m3 Erma ijA: MWMEID kwn4M2Mx?Rir~EFil~le3MjE1 gamm DD .2. cf .. -- szTcanm ij??r?j gx?u?E?u'iHTm UWU . I . I Gul? u'rru rlE H1 ail-mm ?awJI . . I 91 lm Limb Liam m3 ubd.d 1 ltElTE Hdzlen TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL dvanced Code-Based Fingerprints $mn1_ciuker_fir3t_tEEt msg; ng3 main std::string mag; if [macli msgl[D]; if ?mjg?j msga[n]; ELSE return false; char bufZlE]: chm": Field Builder char char if?zruejf E?ur intf [chunk 1? "2nsg[1D] Dxff, n:g[11] IE DHILJ msng?j snprintf?chun?I nsqu?1 mHg[1E] ?wff, nsg[15] m:g[1T] lfzu.r msg[l?] msg[19] mHg[E?] Diff, nsg[31] chunk?j -- D) chunk?; II chunij std::string msg_decnded furisize_t i T: i EE 'a msg_decnded[ij 3 rcturn f?l?t; Fingerprinli] snprintf?hui, 15J Dxff, DIff. DHff; std::string kEyid_hEH huf; TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL a? Advanced Code-Based Fingerprints As another example, some of the activity from the Conficker botnet simply can?t be detected with keywords or regular expressions In cases like this, code can be used inside a fingerprint to test the data further TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL Advanced Code-Based Finer Sun++li if ?l?qqifin?r1nn: H?h. If HUT EeleaEEkle tn thir??partiEE uin:3_t kEyE; 1:23.15; uin23 type; uin:D_E uiuJ33_L LunLiLg_hu:h - uln:33_: H_h1ct; uln:33_: u1n132_t th?Fn_h?HhP?L4j rn'i App": [+Fingerprint5] uin:32 wax Len: 111112333: pkt; ?FiEld Builder Uhilc?phj; - 9:1; Packtt?ll Field 1 L: kuH key? [uintE_thkeFE{{2 :f A DEED: r::1tn falcc; If He: Cjnfickcr, an abut: Li A 3x03] LEJJLH Lultt; If hi; JJL :uL ILL UDP puu1:Lj L: E??ll?uE: Li (($299 A canzinue; :yp: ?kcy? A _L?yig_Lyp: 5 [110] :JJumLy uu:1ditu?:; ff mat a uacke: nn??innP; min pk: len max pk: len ?uint?? :kat.size; high - uint?E uint?? uint?E HLnt?? R_Luu - runni:1:r_hash Cl fnr?1=1; 1++j if?tb=EJ decryp: ?ata TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL . J. - . Meta-data Extracting Fingerprints What happens when you find data and want some pieces of meta?data extracted? XKS Fingerprints can be used to extract meta?data to select XKS database tables. Or if no existing database is applicable, you can define your own database schema for the meta?data TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL . - - 12?7- I. - Free File Upload Sites As a real life example, think of all of various Free File Upload (FFU) sites of interest When a user uploads a document they get a response page that looks like this: TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL Free File Upload Sites W'elmme te ISHARE With free eah uplead ?les, images, redees, audle and ?ash fer ?ee. Simply use the uplead ferrh helew and start sharing! Tee earl alse use ESME as Fear persenal ?le sterage: hael-tup year data and preteet ?les. First Time? Read eur .133 ill UElead r1ew Lir?' Create Free ?re-seem I Premium File Upleaded The ?le ltlti pierseip was upleaded! Teu're new read}: te share it with pe eple er l-teep it as a leaeltup. Dewrrle ad Lll?-L 3T 1 1T4 Lirrlt ?er ?erurrts: Drreet Lirrl-t: Mis?t,? Delete Lirrl-t: E?rhajl Me This Infe Te receive all the infe en the ?le yet; upleaded, sueh as remeval instructiena and dewulearl link, enter freer e?rhajl address en the ?eld E: elew: Tear TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL i ?ue Upload Sites 0 Look at all the great information on that page: File Uploaded The ?lae uploaded! You're now ready to share it with unlimited people or keep it as a backup. Download Link aaharenetl'downlo adlo 3? 1 3?9 5701:: Elf?legit Linl-t for fonana: [Ll FtL=http1fowvweeharenetEd E'ril Direet Link: F?iloElff Delete Link: TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL - Ida-1H?- Free File Upload Sites How can we quickly get that information extracted as Meta-data and be agile enough to respond to each FFU site which may have its own format XKS Fingerprints allow you to use the XKS Fingerprint Language to extract meta?data into the XKS database Fingerprints are deployed within an hour of being accepted meaning you no longer need to wait for all 13o+ XKS sites to be upgraded to have the latest and greatest capabilities. TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL File Upload Sites 5.Uj= and '35h?tE.?Etf?ElEtE.hLml' HfE_filE_?amE wft_?elete_url wft_uplaad_id H?fnnt Mft_ur1 fifnnt mft_up1na?er_username fismall?Lngge? in as: main if [mft_dElEtE_uEl "uplaad"; if wat_file_name] if [mft_url] if Emft_upluader_uaernameh DE.apply?j: Hust ragexs didn't mat:h?j; return true; H: TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GERMeta-data Extracting Fingerprints All you do is tell XKS when to start extracting meta?data appid?'filetrans .Huplaadfrespn?ae'; http_tit vahare.nEEIdElEtE.html' I: TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER1-15 II I . i i Meta-data Extracting Use Regular Expressions to tell it what to extract: filE name mft_delEtE_url uplaad id Hifn?t url fifmnt mft_uplmader_username in as: TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER.H-- A .r?F Meta-data Extracting Finally tell it which database tables you want to store the information: 1't'I El. 1 1'1. if [mfe delet DE[Fmeh f' upload id[D]; "upload"; File URL Filenerne Hhi Treneter Type Upload ID Delete ID Site Heme ll tlot tl zelmredte?t TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL eta?data Extracting Fingerprints 4 What if the meta?data you want to extract doesn?t fit nicely into any of the existing XKS meta?data tables? a Claeain: E. a Clag?il: ht: ,5 anti Metatigtg FE Ham?: LDQE F: ?ller?! Metatiata 5 ENE 5 Prime Number Extractnr 5 Ca" L093 5 Came-3W UNI HEGISTFW "El Cellular mp Giant: Paeawnrge HE.de L993 5 Datumerrt Metade Healiuielziia Dneument Tagging li Email Lug E?ra?t?d FHEE Tech Stringe in Dneurnerlta Full [mu Leer mull-fly 5 HTTP Activity MAN IRE Cafe Genlneatinn wet. pram Lngina anti Wiregham TD USA, AUS. CAN, GER, NZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL a Meta?data Extracting Fingerprints Define your own with the ?Microplugin? query forms: ij l'uii-art-plugins Elat- Flt-ad 5 Cane Ellindmarlaarnen Elea-aan '21] Cane Byzantine Hat-tar Trajan3 Cane Traffic Can-a 1Jia?cirn ateg Jaraa Exif Metadata g?j lpu? Adar-eases g3] Mailer?ccaun? Mai Extra-:1 Heyida Mung-ad Traffic: E?j HetStringa El DUANTUMEIEIT g1] Saudi l'u'lfa Udp Hetrunlitnarne External Elk-?Fun UEEFE Elam-at- Geu Cell Tmmera Wet- Gea Heaulta ;3 Wet: Gel:- 'IIlI'i1?i Tami-era Dieti-Jnanr Cad-a Snippet atur-u: Iur-uzL IU Lian, Hun. LrHl?il, HZL TDP SECRETHEDMINTHREL TD USA, AUS, CAN, GER, NZL Meta?data Extracting Fingerprints '13 Example M82 I