Tribune Technology Post Mortem Report LATimes Web Defacement Incident Executive Summary On Dec 14, 2010, at 3:49 pm. CST it appears that a page in the LATimes.com website was a victim of a digital defacement that affected and targeted speci?c articles, by-lines and other journalistic content The content that was changed was substituted with the chippy1337 moniker. The malicious person(s] of this crime created or garnished an ID for the Assembler content management system They (he/she) then actively viewed and edited content in an unauthorized manner with the Oxygen framework which utilizes the Assembler (CMS) application to facilitate changes. This application is used to update content of new stories, bylines, etc. by Tribunes'/LATimes authorized users. The logs we collected from the web server and Assembler application log illustrate and identify the IP address and User 1D the offenders used to gain access and perpetrate the web site defacement. The change in the article was noticed and reported to Mr. Daniel Gaines-Managing Editor, Operations, latimes.com who facilitated its restora]. The incident was then reported by email at Tuesday, December 14, 2010 5:02 PM to the Market Services and Jason Jedlinski. The article was restored to its original format by LAT Editorial at 4:29 pm. CST the same day [December 14, 2010]. To help rectify, diagnose and troubleshoot these issue twenty three individuals responded to this emergency situation. These individuals included personnel from editorial department all the way to systems people. The team worked in identifying are remediated this issue as much as possible. This document contains security recommendations to help prevent this type of web site defacement in the future. The staff of individuals should be recognized for not only their dedication to duty to resolve this issue in crisis mode but for follow-up guidance that is represented here later in this document. In conclusion we are cooperating with federal law enforcement authorities to pursue criminal charges against this perpetrator. Overview of Change The picture below represents what the article should look like in its unaltered form. Pressure builds in House to pass tax-cut package House Democratic leader Steny Hoyer sees 'very good things' in the tax-cut deal, which many representatives oppose. But with the bill set to clear the Senate, reluctant House Democrats are feeling the heat to pass it. Share 34 Meets 0?99 Tax plan imperfect Obama says Tax-cut deal now taking fire from both sides as (?51?7 nI1+ but still a ?good deal; By Lisa Mascaro, Tribune Washington Bureau December 14, 2010 2:28p.m.. 8 E-mail Print Share AL Text Size ?uke [i Up to see what your friends like. Reporting from Washington After the Senate overwhelmingly voted to advance the tax?cuts package, House Majority Leader Steny Hoyer acknowledged Tuesday? the urgency in passing the legislation to avoid a tax hike on Jan. 1. The picture below represents the various defacement alterations that were introduced to this same article. Pressure builds in House to elect CHIPPY 1337 House Democratic leader Steny Hoyer sees 'very good things' in the deal out which will see uber skid Chippy 1337 take his rightful place, as head of the Senate, reluctant House Democrats told to SUCK IT UP By CHIPPY NO 1 FAN, Tribune Washington Bureau 0 . 14 Meet 0 December 24, 2020 zo.o4 cm. Digg 8 E-mail Print Share A Text Size JOE: 1? people lie th?s. Be the first of your Friends. RELATED Tax cuts will pass despite Democratic Obama . . advisor says g? Reporting from Washington After the Senate overwhelming voted to advance the tax-cuts package, House Tax plan imperfect . . - .- but Still a .good deal; Majority Leader mung Pic-ye: acknowledged Tuesday the Obama says urgency in passing the legislation to avoid a tax hike on Jan. 1. I The main difference in the article story is that various plugs, keywords and by-Iines were altered to reflect the CHIPPY, CHIPPY 1337 or other non-Tribune edited references. While other references are present hackers such as Chippy1337 who want to generate a high degree of visibility of their exploits, typically will alter content of websites they deface to re?ect the misguided pride of their illegal conquest. Web sites like My archives defaced websites for educational purposes. Zone-H also catalogs the cybercrime archive?s mirror of the hacked website as well as a mirror of the vandalisms perpetrated, which like Chippy1337, typically has the hacker?s moniker that was responsible for the defacement. While conducting further investigation, we discovered that the Chippy1337 hacker has in ?hacker? forums for the most part claimed or known to have claimed credit for the Dec 14, 2010 web defacement that occurred against the LATimes website. Detailed Description The original story is presented here in this link vote-20101215.0.4770429.st0rv. The timeline of events is given below in the section marked "Timeline of Events? Here is where we found who updated the content and what ID they garnered to make the unauthorized changes, through the Assembler User Interface Application. We matched the time of the previous and last change of content with the Assembler User Interface. The Content Item Update Log showed four used to make changed to the affected storyline and another used to make corrections post incident. From the time of the last authorized change to the story to the use of a garnered user ID [ngarcia] From the content item update log entries below we can surmise several clues. The content item update log displays the last known update to the web content through the Assembler application which is from where this specific type of content is updated. From viewing thelog we surmised the content items history number is 58266801. This number is signi?cant because it is the referenced number used in the web servers logs and can be tied to a specific IP address. ht assembler.tila.trb content contentitem view-contentitemhisto .ui?contentitem=58266801 :1 k. vau- and (1 any; 0f the [D's that made changes to the web content on the web servers through the Assembler user interface, we found that speci?c were legitimate and one was not. The "feeds" line is the export from our editorial system. The "tgarrison" (Tim Garrison] and ?mfari? (Frank Farrar) lD's are two producers working on the story in the morning. The "ngarcia" ID is where we see the curseword headline and byline was changed. We could not tract this ID to a known user who would have access to update this type of web content The [Brian Hanrahan) ID is a copy editor who looked at and eventually ?xed the defacement article. Tom Commings the system admin gathered the information displayed below while looking into the issue. In Assembler, logs, he grepped for the time stamp and found this: grep Once we ?gured out what content item time and login was used due to the time the change was discovered we looked for the speci?c content item ID "58266801". This content item from the U1 (user interface] used this number to track changes made though the web session from the user to the web server. Tom then and verified this content item ID was associated with the change though the web and discovered the IP Address that was responsible for the unauthorized changes that were made to the article. Here is a sample of the output of the session that clearly identi?es the web posts to the assembler ui with an exact content item id of 58266801 originating from this speci?c IP address 188.165.6.178. It even gives speci?c detail about the browser used to complete these web requests (Mozilla/5.0 (en- rv:1.9.1.2) Gecko/20090729 188.165.6.178 - - [14/Dec/2010:13:49:15 -0800] 200 8192 "htt s: content contentitem "Mozilla/5.0 (en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2" 188.165.6.178 - - [14/Dec/2010:13:49:19 -0800] 200 1876 "Mozilla/5.0 [en-US; rv:1.9.1.2] Gecko/20090729 Firefox/3.5.2" We attempted to trace the offenders IP address (188.165.6.178) and it geolocates to the below information according to RIPE: organisation: ORG-OHS-RIPE org-name: OVH Hosting Limited org-type: OTHER address: 5 Fitzwiliiam Place address: Dublin 2 address: Ireland abuse-mailbox: e-mail: mnt-ref: OVH-MNT mnt-by: OVH-MNT changed: 20090916 source: RIPE Here is a sample of the assembler log matches for this IP from the web server logs on these two assets, $15 and 516 SI 5: grep '188.165.6' 188.165.6.178 - - [14/Dec/2010:11:37:03 -0800] 302 0 ?Mozilla/5.0 (en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2" 188.165.6.178 - - [14/Dec/2010:11:37:31 -0800] /images/ti.jpeg 200 32965 ?Mozilla/5.0 [en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2" 188.165.6.178 - - [14/Dec/2010:11:37:32 -0800] 200 334 ?Mozilla/5.0 [en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2" 188.165.6.178 - - [14/Dec/2010:11:37:38 -0800] /favic0n.ic0 200 3574 ?Mozilla/5.0 (en- rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2" 188.165.6.178 - - [14/Dec/2010:11:37:42 -0800] /access/loginmodule.ldap 200 1859 ?Mozilla/5.0 [en-US; rv:1.9.1.2) Gecko/20090729 irefox/3.5.2" 188.165.6.178 - - [14/Dec/2010:11:38:04 -0800] /access/loginmodule.ldap 200 1859 ?Mozilla/5.0 [en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2" 188.165.6.178 - [14/Dec/2010:11:38:08 ?0800] /stylesheets/ui.css 304 0 ?Mozilla/5.0 (en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2" 188.165.6.178 - - [14/Dec/2010:11:38:10 -0800] 304 0 [access [loginmodule.ldap" ?Mozilla/5.0 (en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2" 188.165.6.178 - - [14/Dec/2010:11:38:13 -0800] /favicon.ic0 200 3574 ?Mozilla/5.0 (en- rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2" 188.165.6.178 - - [14/Dec/2010:11:38:23 -0800] /access/loginmodule.ldap 200 1859 ?Mozilla/5.0 [en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2" 188.165.6.178 - - [14/Dec/2010211238t39 0800] /aecess/loginmodule.ldap 200 1859 ?Mozilla/5.0 [en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2" 188.165.6.178 - - [14/Dec/2010:11:38:42 -0800] /favicon.ico 200 3574 ?Mozilla/5.0 (en- rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2" The complete logs and other information gathered during the course of this investigation are contained here: Logs captured: I Web Server 1. 2. 1 0 1 06.1 1 .48.gz. LDAP-Server 3. 4. 5. 6. 7. investigation Documentation: \\163.192.80.103\Infosec Timeline of Events The list of events in a timeline fashion is listed below. A brief description of the events that transpired is also included. TIMELINE December 14th 2010 On December 14th 2010 at approximately 3:49 PM-CST an unauthorized addition was implemented to a speci?c article in the LATimes.com website. The story was edited by "ngarcia" at 3:49 pm. CST, and was restored by LAT Editorial at 4:29 PM. CST. It was reported by Daniel Gaines, Managing Editor, Operations, latimes.com. It was later discovered that the ngatrcia ld used belonged to a user that had been previously terminated. The following investigative and restorative actions were taken by our Information technology staff. - Tad Lin conducted or examined PZP/Assembler Secuirty Action - LDAP Change (SuperUser). Dwayne Butler conducted or examined P2 P/Assembler Security Action - LDAP Change (SuperUser). 0 Tom Comings conducted or examined P2P/Assembler. 0 Jason Potkanski conducted or examined P2P/Assembler Security Action - LDAP Change (SuperUser). A group discussion was commenced in order to decide on course of action- The attendee's consisted of Chris Phillips, Kyle McClusky, Lucy Jacobson and Diane Yamazaki. December 15th 2010 Planning, implementation the remediation and code changes including preventative measures were performed by the following personnel. 0 Tad Lin conducted or examined PZP/Assembler Security - Maintenance, Release Planning. 0 Dwayne Butler conducted or implemented P2P Assembler Security. 0 Tom Comings conducted or examined Assembler/P2P Web Log Analysis. 0 Brandon conducted or examined Assembler SuperUser LDAP Report. 0 Greg Noth conducted or implemented Assembler Change. 0 Jason Potkanski conducted or examined Assembler SuperUser LDAP Review. A discussion with software engineering was commenced in order to discuss potential assembler and P2P changes with Tad Lin and Chris Phillips's group. December 16th 2010 Planning, implementation the remediation and code changes including preventative measures were performed by the following personnel. 0 Tad Lin examined and conducted PZP/Assembler Security. - Dwayne Butler conducted Assembler LDAP URL Review. - Tom Comings conducted Assembler and P2P Web Log Analysis. 0 Richard Benjamin conducted Assembler LDAP Review. 0 Jason Potkanski conducted Assembler LDAP Review. 0 Brandon conducted Assembler LDAP Review. --December 17th Chris Phillips documented P2P code changes for this recent change. December 18th 2010 Nothing to report. January 4th 2011 Tim Rodriguez was assigned to gather evidence, case information and submit that information to FBI - Special Agent John Cauthen. This includes the information gathered in the packet. January 7th 2011 Tim Rodriguez initiated a follow up call with area headquarters in Sacramento, California- Special Agent John Cauthen. Tim Rodriguez then explained the progress we were making on case number 0001 (LATimes hack and defacement). I sent two emails which included the progress and logs and data that we were able to gather. Most importantly we were able to identify the P2P User ID and corresponding IP Address that made the change that caused the defacement back. The following Items are outstanding and need to be garnered and submitted to - Special Agent John Cauthen: List of all employees, assets, consultants and personnel that worked this case- Provided. List of all employees, assets, consultants and personnel's hourly payment schedule that worked this case. This is to help provide exact costs the Tribune incurred during this investigation Outstanding. January 25th 2011 Sent followup email to FBI - Special Agent John Cauthen with details of LDAP logs. Created and named incident report, will submit to Amanda Caro by EOB today. One item is outstanding: List of all employees, assets, consultants and personnel's hourly payment schedule that worked this case. This is to help provide exact costs the Tribune incurred during this investigation Outstanding. Resources Involved in Investigation The list of employees, contractors, Law enforcement personnel is included below as well as the time spent troubleshooting, examining, remediation, development and investigation of this issue. Their functional title, name, assignment tasks in hours and total hours spent is also included. ancuonal Title Preventative Total Name Hours Hours Hours Hours System Engineer HeusinkVeld, Brian 20 8 28 System Engineer Downard, Sabrina 20 12 32 System Engineer Casey, Conor 5 10 15 System Engineer Hancock, Craig 12 15 27 System Engineer Sahasrabudhe, Satish 3 6 9 Unix Bezouska, Joe 20 12 32 System Engineer Jones, Ken 2 3 5 System Engineer Chung, Holly 3 4 7 Server MGR Russ, Robin 3 10 13 Software Engineering Tad Lin 16 4 20 Software Engineering Dwayne Butler 5 2 2 9 System Engineer Tom Comings 2 10 12 Applications Developer Richard Benjamin 2 2 Applications Developer Jason Potkanski 5 4 9 Applications Developer Brandon 5 Applications Developer Greg Noth 1 1 Software Engineering Chris Phillips 0.5 4.5 Market Liaison Diane Yamazaki 1 1 Project Management Of?ce Lucy Jacobson 2 2 Software Engineer Kyle McClusky 2 Software Development Matthew Pulley 1 1 Managing Dir, Tech Architect Armando Caro 36 36 Information Security Tim Rodriguez 50 50 Total Hours 323 Systems Affected/Assets Lost Two systems that were affected by the web defacement are listed below and have had web content restored and are up and running as of the data of this publication. No other loss to systems or services was reported or experience issues at that time. Financial Loss Financial loss did occur, however it occurred in the resulting clean up, diagnosing and preventative measures taken to remediate this issue. The total man hours devoted to the rectification of this web site defacement has exceeded three hundred and twenty three hours, which translated into costs may well exceed ten to ?fteen thousand dollars. The costs of implementing two factor authentication, enhanced logging and auditing as well as the other protective measures in the ?Solutions? section could translate into the hundreds of thousands of dollars. Overall the impact severity was medium; although it had unknown initial impact, is still a topic for search on other media such as Google. The severity of the site defacement incident may have had a detrimental impact on the reputation of the news site involved that ties back to tribune itself. The integrity of the content as well as the content itself that this news media we publish is of high value. This demonstrate? able defacement of the article in question could in the public tarnish the reputation of our publishing services. For over 163 years the Tribune and its holdings have built a solid reputation in the news media, the repercussions of this attack and its brand tarnishing could have detrimental effects on our sterling reputation. Our reputation is the one commodity that is earned, hard to obtain and cannot be purchased. Contact list The following list of technical and non technical resources that had been assigned the various tasks detailed above, have been listed and added to this contact list for reference. Heusinkveld, Brian Downard, Sabrina Casey, Conor Hancock, Craig Sahasrabudhe, Satish Bezouska, Joe Jones, Ken Chung, Holly Russ, Robin Tad Lin Dwayne Butler Tom Comings Richard Benjamin Jason Potkanski Brandon Greg Noth Chris Phillips Diane Yamazaki Lucy Jacobson Kyle McClusky Matthew Pulley Armando Caro Tim Rodriguez Solution bheusinkveld@tribune.com sdownard@tribune.com ccaseyQtribunecom chancock@tribune.com ssahasrabudhe@tribune.com keniones@tribune.com hollv.chung@tribune.com rorusthribunecom tlin@tribune.com dbutler@tribune.com tcomings@tribune.com rcbeniamin@tribune.com ipotkanski@tribune.com gnotthribunecom ccphillips@tribune.com DYamazaki@tribune.com liacobson@tribune.com mpulley@tribune.com acaro@tribune.com tprodriguez@tribune.com 312-222-7809 312-222-3073 312-222-2368 312-222-3709 312-222-6690 312-527-8704 213-237-5858 312-222-5565 312-222-2441 312-222-4304 312-222-6158 312-222-5435 312-222-4210 312-222-4687 312-222-3565 317-379-3171 312-222-2106 312-222-4392 312-222-5467 312-222-7879 312-222-2708 312-222-3938 The ease in which the hacker was able to login to the Assembler Web Content Management System from the intemet and make changes to a speci?c storyline is illustrative to the ease of access to this system. Typically strong security mechanisms should exist to prevent unauthorized persons from making changes to important assets such as our news media sites. In this case by disrupting the article in question with the apparent ease that this exploit illustrated the hacker brings to light several deficiencies that exist in protecting our digital news media services. The next few sections will give our recommendations for the implementation of additional security protections, processes and procedures. These recommendations will provide a higher degree of protection to help combat the hordes of malicious individuals that seem to plague high value organizations such as ours, looking to damage and mar our news content that we bring online for the whole world to view and become informed. The protective mechanisms will also help strengthen the presentation of our valued product and help protect our "brands" already highly esteemed reputation. In order to help facilitate protection ofyour news media and mission critical applications all forward facing web, application servers and devices should be protected in the following manner. Technology controls. Web /Applications must have the ability to generate logs that support the following features: They must have the ability to generate logs that support the following features: 1. A ZFA- two factor authentication mechanism should be put in place to help guarantee the authenticity of the user requesting access to Tribune resources. This is a critical path, especially for users who sit beyond our company boundaries and that have access to mission critical resources like our news feeds. 2. Logs must be remotely consolidated to a centralized logging device and then compressed, hashed, categorized into speci?c venues, annotated and backed up to a SAN or other robust media. Each entry must have a reliable and universal timestamp that was synced from a trusted timeserver as well as a username or unique authentication server identification tag. Originating lP (External) Address or a traceable log balancer sequence ID that can be traced and used to correspond to a Globally Unique IP address. 3. Optimally an options performed log entry such as a post for web server content or an action performed such as updating content for a web authoring or content management system must be logged and transaction data be forensically available. 4. A correlation mechanism should be implemented to data mine information from application, web and system logs such as matching usemames, IP Addresses and access times to various and critical resources that require a high level of trapping and tracing access attempts successful or otherwise. A data mining mechanism should be implemented to harvest audit and forensic able data to create reports of access and unauthorized events. 5. Applications and services must be root jailed to help prevent a vulnerable or compromised application from gaining high level access to the operating system and other services. 6. Deploy a Web Application Firewall (WAF) to help protect critical assets against common attacks such as Cross-site Scripting (XSS), SQL Injection, brute force and other types of application layer attacks. Process controls. 1. Accounts, lD's, roles and permissions values that have access to high value assets should be checked at least every month for termination, unauthorized addition or deletion and role changes. Creation of a formalized process that audits termination lists from HR and verify that the terminated user had all accounts disabled or removed within 24 hours. Communication needs to exist of terminated accounts and process that tracks if the account has been re-enabled. 2. Security awareness training will be conducted to provide and illustrate the proper procedures and security related processes that need to be followed and discuss common social engineering techniques and speci?c defense techniques used to combat them. Social Engineering exercises will be conducted to check account activation and deactivation processes and procedures are intact. 3. Applications that are available externally that cannot support the correct data protection; user or session authentication must be protected with a VPN. 4. Either a technical or non-technical approved and monitored process for maintaining the integrity of important and or critical news sites must be maintained. It must have noti?cations mechanisms for altering the content of these sites. Repeatable security control processes. 1. The formal creation of a (Computer Security Incident Response Team) to facilitate incident handling duties should be established. Creation and delegation of duties, roles, responsibilities and processes to support the program. 2. Create a process to report security related incidents through the Service Manager ticketing system. This can help escalate events and create metrics to help measure effectiveness in responding to and measuring related incidents. 3. A post incident wrap-up, root cause analysis and incident summaries should be created to be reviewed and historically saved after any major Information Security Incident. 4. An audit of mission critical systems from their established base should be performed twice a year. 5. A penetration test of mission critical systems and there applications, services and operating systems should be performed twice a year. 6. Active vulnerability testing should be ongoing and run to check for vulnerabilities, misconfigurations or unapproved services and con?gurations. Follow-up Actions This incident has been reported to the appropriate law enforcement authorities and an ongoing investigation with the full cooperation of the information Security group, is underway. The "Solutions" partion of this document will be discussed and reviewed in order to create a more secure mechanism and process for assets that host our media services. These solutions will be intelligently implemented and giVen the upmost priority to protect our mission critical assets. These lessons learned in respects to the development of customized software for Tribune services will glean a more beneficial practice for the ongoing security development process. Revision History Authors Date Version Comment Tim Rodriguez-IT 01-14-2011 None Started initial Draft. Security Tim Rodriguez-IT 01-25-2011 .1 Security Created and revised initial draft, submitted for case approval.