Vulnerability Assessment and Forensics Services for a hosting company with 2500+ applications

Code Review, Application Security Assessment For a US Based Intelligence Agency
November 5, 2007
Code Review and Web Application Security Assessment For a US State Department
July 29, 2008

Our customer owns and manages more than 2500 websites and portals, having over 5000 domain names in their Domain Bank. They serve a variety of verticals like verticals GIS, travel, software development, greeting cards, celebrations, education, media and photography.

Their core expertise lies in offshore product development, content development, software development, maps and geospatial services, web services and management of captive units.

The Challenge

Our customer was facing considerable downtime, loss of revenues and loss of credibility due to recurring and constant breaches where malicious scripts and exploits were inserted in the webpages of their hosted applications, which were then used to compromise innocuous surfers.

This put them in a very tricky situation where they not only had to bear the brunt of ailing customers, but also the individuals who visited these legitimate websites. Apart from this, there was an observable loss in productivity due to the amount of time they had to spend on fixing and dealing with such issues.

After conducting a preliminary investigation and interaction with various departments of the customer, Torrid Threat Research Team quickly realized the reasons why the mitigations strategies of the customer were not working properly. It was simply the case of providing too much leverage to the developers, arising due to lack awareness on internal security.

For the customer, it always seemed as if the compromises were happening on the server-side, which was not the case. This investigation also highlights the fact that even if you adopt stringent perimeter security measures, internal compromises and lack of awareness can your bring you back to square one.

Figure 1 – The Development Update Process

The above figure highlights the process of pushing developer updates to the various hosted websites. To make it up to the deadlines, the developers used to make these updations from home or cyber-cafes. Due to the lack of a clearly defined policy on updates, the developers didn’t feel the impetus of following security best practices. Cyber-cafes and insecure home computer were found be outdated in terms of the latest security updates, thus making them susceptible to a variety of attacks. Cyber-cafes are especially notorious for being badly infected with malware, some of which like W32.Fujacks, Trojan.Popuper and Trojan.Realor can spread via HTML/RealPlayer files and insecure network shares. Moreover, the customer didn’t really have any policy to log and track development updates and versioning.  So it was really difficult to pinpoint the exact point of compromise.

Figure 2 – Malicious script insertion

However, upon investigating malicious script insertions, it was clear that this was the job of malware like W32.Fujacks, Trojan.Popuper and Trojan.Realor, which infect various local HTML files (Figure 2). Upon infection, the malware used to embed malicious iframe links, which when accessed found to host a plethora of obfuscated exploits leveraging known client-side and browser-based vulnerabilities.  In such a scenario, it is really difficult to mitigate these infections as every machine on the development premises had to be scanned for malware and all the infected had to be manually cleaned.

The Solution

Torrid suggested a thorough, end-to-end security assessment to the customer.  This involved doing an external penetration test of the hosted servers, an application security audit of the internal web applications used for developer updates, to conducting a forensics of the compromise.

The external penetration test would strengthen the customer’s network security posture whereas an application security audit would make their development life-cycle more robust and making the developers aware about the common security mistakes while coding web applications. Due to the seriousness of the compromise, an elaborate forensics would help in tracking the source of malicious activity.

The modus operandi being followed was as below:

Undertake detailed rounds of interaction with the development, IT, infrastructure management and security teams to understand the various loopholes in their security policies.

  1. Perform an elaborate external vulnerability assessment of their server farm to bulletproof their network security posture.
  2. Conduct web application audit on their internal web applications to strengthen their SDLC.
  3. Try to trace back the source of malicious activity by conducting a detailed forensics and incident analysis.
  4. Provide a multi-facted, three-pronged report having step-by-step tutorials on security best practices which is easily understandable by developers and IT professionals.
  5. Impart training on secure coding and incident analysis to help them gear for future incidents.
  6. Set forth a policy of pushing developers updated in a more accountable manner by introducing versioning and detailed logging systems.

The Results

Torrid helped them in assessing their overall security posture, strengthening their SDLC and mitigating the malicious compromise.

The detailed, three-staged reporting processes, a heavy stress was laid on best practices and step-by-step tutorials on various security issues which would help them bear the brunt of such attacks in the future. Some brief trainings and sessions on secure coding and incident analysis were also imparted. We effectively and non-intrusively aligned their security posture with the business and operational processes to provide them with an end-to-end and sustainable security robustness which would help them gear for the future.