Nitesh Dhanjani

Amazon's Elastic Compute Cloud [EC2]: Initial Thoughts on Security Implications

The Cloud Computing buzz is everywhere.  The concept of grid computing on the Internet to provide elasticity and virtualization of resources is quite appealing, and hence there has been a lot of academic brain-storming going on recently that has given rise to abstract ideas on how cloud computing is destined to change the way technology resources are deployed and used.

Amazon's EC2 isn't abstract - it's real. And it's very impressive.

Q: What can developers now do that they could not before?

Until now, small developers did not have the capital to acquire massive compute resources and insure they had the capacity they needed to handle unexpected spikes in load. Amazon EC2 enables any developer to leverage Amazon's own benefits of massive scale with no up-front investment or performance compromises. Developers are now free to innovate knowing that no matter how successful their businesses become, it will be inexpensive and simple to ensure they have the compute capacity they need to meet their business requirements.

The "Elastic" nature of the service allows developers to instantly scale to meet spikes in traffic or demand. When computing requirements unexpectedly change (up or down), Amazon EC2 can instantly respond, meaning that developers have the ability to control how many resources are in use at any given point in time. In contrast, traditional hosting services generally provide a fixed number of resources for a fixed amount of time, meaning that users have a limited ability to easily respond when their usage is rapidly changing, unpredictable, or is known to experience large peaks at various intervals.

I was able to go through the Getting Started Guide and I had myself a Linux environment in the Amazon cloud in no time:

Amazon EC2 presents a true virtual computing environment, allowing you to use web service interfaces to requisition machines for use, load them with your custom application environment, manage your network's access permissions, and run your image using as many or few systems as you desire.

To use Amazon EC2, you simply:
* Create an Amazon Machine Image (AMI) containing your applications, libraries, data and associated configuration settings. Or use pre-configured, templated images to get up and running immediately.
* Upload the AMI into Amazon S3. Amazon EC2 provides tools that make storing the AMI simple. Amazon S3 provides a safe, reliable and fast repository to store your images.
* Use Amazon EC2 web service to configure security and network access.
* Start, terminate, and monitor as many instances of your AMI as needed, using the web service APIs.
* Pay only for the resources that you actually consume, like instance-hours or data transfer.

Based on my recent experience, here are some initial thoughts (with bias on security):

Pre-pwned Images. Amazon lets you select from a list of AMIs (Amazon Machine Image) to kick start your virtual environment. They even let you submit your own AMIs to share with other EC2 users:

To add your AMI:

   1. Make sure you're logged in to the site.
   2. Click the "Add a Document" link in the Tools box to the right.
   3. Enter as much information as possible in the form, then click Preview. (Tip: You can use HTML in the listing body.)
   4. If everything looks good, click Submit. Otherwise, click "Go Back/Edit" and make your corrections.

Important: Your listing will show up on the site after a quick review by AWS.

Interesting. I wonder what the "quick review" entails. What if someone submits an AMI with a back-door installed? Does the Amazon team have the resources and processes to identify malicious AMIs before sharing them with their customers?

Keys to the Cloud. The Amazon services are based upon a key-based approach (you get your own key-pair to authenticate to Amazon and to sign your own AMIs) - this is good. There is the burden of key management, but it is still a better approach than implementing a static password based system.

Firewall. Instances initially boot in a firewalled environment where you have to explicitly open up ports to allow inbound access. This is a good approach as well.

Dude, Where's My Data? Data in the virtual instance only persists as long as the instance is running. From Amazon's FAQ:

Q: What happens to my data when a system terminates?
The data stored on a specific instance persists only as long as that instance is alive. You have several options to persist your data:

1. Prior to terminating an instance, backup the data to persistent storage, either over the Internet, or to Amazon S3.
2. Run a redundant set of systems with replication of the data between them.

We recommend you should not rely on a single instance to provide reliability for your data.

This means that existing applications and systems need to be re-engineered to persist data in the cloud (outside of the virtual environment). Makes sense, in order to take advantage of the elasticity of cloud computing, your data has got to be 'in the cloud' and not tied to a single virtual instance. This may have some legal implications, and it may make some organizations (initially) uncomfortable to come to terms with the idea that their live data does not persist on their own hardware.

I'm afraid we are likely to see cases of security issues arising from badly re-engineered application code as developers attempt to code their applications to persist data using services like S3 instead of local data stores.

Risk Based on Data. Most often, organizations fail to risk inventory their assets based on the type of data the system reads and writes. The cloud computing paradigm will force organizations to think of data foremost when building a risk inventory. This is a good thing.

Security Principles. Obvious and well known security principles apply to cloud based services like EC2.  You've got to ensure that your VMs are configured securely, that your applications are developed securely, and that you communicate securely - think about authentication, authorization, access control, cryptography, and monitoring on all layers and tiers of the system.

The Threat of Mono-culture. I'm reminded of Dan Geer's words on the threat of mono-culture. If you start up hundreds of instances of a virtual image, a vulnerability in one instance will apply to all other instances of the same image. Imagine a situation where a remotely exploitable vulnerability is found in the generic kick start image Amazon recommends to its customers - suddenly, the security of a considerable amount of resources and data within the cloud will be at stake.

Cloud Insecurity. Security issues within the Amazon web services will have an extremely high impact on EC2 customers. For example, suppose a malicious user is able to invoke the services behind ec2-terminate-instances to terminate instances outside of his or her role. Such a vulnerability could be abused to black-out the Amazon cloud.

Perimeter? What Perimeter? The concept of relying on a network based parameter has been losing steady ground. Cloud computing services like EC2 will be a catalyst  to this recommendation - data and resources will be distributed in a shared cloud space. The concept of network based perimeter will no longer apply. Instead, security controls will need to be assured on all layers and tiers of the architecture. However, there are bound to be cases where organizations will try to build trust within the cloud to construct a virtual perimeter to imitate legacy designs.

Service Provider Liability. As the concept of cloud computing gains ground, it is likely that the service providers will seek to implement technical solutions that will allow  them to provide resources in the cloud without the legal liability of hosting and computing secret or illegal data. For example, a consumer or legal requirement may warrant the customer of the cloud to have the ability to compute or store data in the cloud without exposing the computation  result or data to the provider.  This may facilitate tangible products to arise from academic concepts of zero knowledge based solutions.

Single Point of Failure. Amazon provides the concept of Zones and Regions (currently limited to 1 region):

Amazon EC2 now provides the ability to place instances in multiple locations. Amazon EC2 locations are composed of regions and availability zones. Regions are geographically dispersed and will be in separate geographic areas or countries. Currently, Amazon EC2 exposes only a single region. Availability zones are distinct locations that are engineered to be insulated from failures in other availability zones and provide inexpensive, low latency network connectivity to other availability zones in the same region. Regions consist of one or more availability zones. By launching instances in separate availability zones, you can protect your applications from failure of a single location.

This is good - Amazon allows for instances to be booted in different zones to prevent impact from the failure of a particular location. But what about Amazon as a whole as the single point of failure? The concept of resources being distributed geographically makes this scenario less probable. As cloud offerings from other companies emerge, it may make sense for larger organizations to host on other cloud service offerings to further decrease the single point of failure scenario. Doing so could be a little difficult since competing services may require adherence to specific programming languages and environments. For example, the Google App Engine SDK is currently limited to Python and is not based on the concept of allowing users to configure full blown virtual environments. Perhaps I'll write my thoughts on the Google App Engine in the near future.

I'm excited about the concept of cloud based computing. It's the future, and Amazon has done a good job of turning the hype into reality. I'll be interested to see how Google's offerings mature, and what Microsoft and IBM have up their sleeves.

These are just my initial thoughts on security implications of the emerging cloud computing paradigm. I'll continue to post updates as I have time to think about it some more. If you'd like to share some ideas, I'd be interested to hear them.

Interview With [IN]Secure Magazine

Issue 16 of [IN]Secure Magazine is available. Mirko Zorz interviewed me in this edition (Page 41). If you decide to read it, I'd be delighted to hear your thoughts and feedback. The magazine edition of the interview is much better looking and highly recommended (as are the other articles), but for the sake of convenience, the interview session is below.

Enterprises need to formulate high-level goals for application security efforts before implementing specific service lines. What are the key areas they have to cover in order to make their endeavor successful?
I agree. You've got to strategize high-level goals before deciding on specifics. Most businesses are not in the business of being secure. They are in the business of generating revenue, protecting their brand, and their intellectual property. Application security goals must derive from and support these business goals to promise risk reduction across the enterprise cheaply and effectively. Such promises in turn require specific implementations and processes such as hiring the right talent, laying the right framework, hooking security into the development lifecycles, training, metrics, and executive support.

Despite owning a plethora of software and hardware solutions, the critical asset to an organization is still the security professional who works with those acquisitions. How exactly important is the security team?
Talent is key. What good is an application scanner or code analyzer if you don't have professionals in your team who actually understand the results? The fastest way to lose credibility with the business is to employ individuals who cannot go beyond running assessment tools and exporting reports. The job of a security team in any organization is not to hire people who can point and click their way into running assessment tools, but to establish a world-class effort that serves the needs of the business. You do that by hiring subject matter experts. You do that by hiring talent that can impress the business and demonstrate tangible value and progress.

With the threat landscape constantly evolving and old issues still not resolved, the organization has to battle problems such as a lack of security awareness that bring in a myriad of complications. What is the right approach to take in order to battle difficulties one can't completely protect against?
That's a two-part question: how to deal with known issues, and how to keep up with the latest attack vectors. First, you've got to establish a process that aims to remove security gaps at the root. Training and awareness offers the best ROI in this regard: bugs that don't get created in the first place - imagine that! It is also vital to embed security into the development lifecycles of applications. However many organizations have trouble deciding where to begin. The solution is to assign efforts based on risk. Start by understanding what applications you own and what their business impact is. What type of data do these applications read and write? What is the business criticality of these applications? Once you have a good understanding of your application portfolio, it will be much easier to assign effort so you can focus clearly.

As for the second part of the question, the solution is to invest effort into research and development so you continue to understand how the latest attack vectors may target your software. Yet again, training and awareness wins in this regard. Set aside a budget to send your team to information security conferences and training programs so they can soak up new knowledge. Allow analysts to take some time out to investigate the latest attack techniques. Most hands on security professionals are scientists at heart - understand what makes them thrive and support their talent. Support their desire to learn new ways to break security controls. Finally, capture and communicate this knowledge to the business. For example, ensure your threat modeling attack libraries are up to date and reflect the latest attack vectors, that your code review and assessment methodologies are bleeding edge, and that you take time out to brief the architects and developers on what they need to know to keep up.

Although applications have the largest attack vector today, CSOs don't take this into account when strategizing security spending. What kind of issues can this bring in the near future?
I overwhelmingly agree - the security spend of many organizations is out of whack with the real threat landscape. I feel there are multiple reasons behind this situation, the most common reason being, in my personal opinion, that many individuals who have been hands on in the past remember and hold on to the notion of the network layer being the only big thing to worry about. I can sympathize with that view - if you rewind a couple of years, majority of the high impact attacks could be identified and blocked via network controls because the attack surface of applications was low compared to today's scenario where a typical enterprise level web application is comprised of millions of lines of custom code. Perhaps another reasoning for this situation is that solutions around application security do not provide the instant gratification of throwing in a few appliances to solve problems. Well, perhaps, I should take that back, there are a few web application firewall solutions in the form of appliances that are starting to be marketed that way, but that's the nature of marketing and I'll save my rant for another forum. Also, quite unfortunately, there are going to be more situations in the coming future where many security efforts will align with reality after learning the hard way - i.e. after an application related exposure has already taken place. That said, the onus is still on the security professionals and researchers - we need to do an even better job of demonstrating impact and educating decision makers on why a solid application security strategy is vital to any organization's overall security effort.

While having consultants come in and perform black box penetration audits of applications every year is more costly than investing in a solid SDLC process, many organizations still believe it to be the proper strategy. What should they take into consideration when making this decision?
Black-box penetration tests are useful, yet they are extremely expensive and ineffective when relied upon as an exclusive solution. Paraphrasing a colleague of mine, "Companies that solely on black-box assessments to guide their security efforts do something similar to having consultants come in, throw a grenade at them, and have the consultants close the door on their way out". Gary McGraw likens this situation to what he calls a "Badnessometer". Black-box assessments correlate to symptoms that reflect the level of trouble you may be in. The response shouldn't be to just fix the black-box assessment results, but to respond to the situation strategically, and ensure you are responding to and eliminating the root cause of your problems to make sure they do not re-occur.

The solution is to "push left". The ingredients of a typical application development cycle, from left to right, includes the requirements phase, followed by design, then implementation, test, and production. The more effort you put into implementing the right security controls at an earlier tollgate, the lesser it will cost you. For example, assume that a review of security controls during the design phase results in an architect having to re-engineer the authentication mechanism. Now, imagine if this issue was not caught during the design phase, but uncovered during an attack & penetration assessment after the application is in production. It is not trivial to re-engineer a product that is already in production. And it is extremely costly - multiple times costlier.

Black-box assessments, coupled with the right strategy, do have their place. Going back to Gary McGraw's point, they help uncover symptoms. These assessments can be used to further augment and enhance an existing security SDLC process. For example, if you are finding too many issues via black-box assessments during pre-production that you missed to uncover during design and test, then it is time to re-evaluate your SDLC process and approach.

Besides having an excellent technical background, the CSO has to be good at demonstrating a tangible impact of his actions to the management in order to justify security spending. This ability is becoming increasingly important and can take the focus off the main areas of security he should be working on. How can the CSO lighten this load?
Justifying security spend for application security is not difficult. In addition, application security efforts can have positive political side effects for the CSO and the security team. I'll tackle these two points separately. First, application security must tie into the overall IT risk strategy. Start with asking what the company's business goals are, and how you want to demonstrate value. Map these goals to efforts based on risk that will flow into specific tactics that can demonstrate ROI. For example, if your organization currently relies on yearly black-box assessments, calculate the cost of performing the assessments in addition to the cost of remediation. Compare the cost with progress you've made by evaluating the last two assessment results for the same application. Most likely, you will find that you have made little tangible progress in the form of risk reduction and that the remediation cost has been high. Now calculate the cost of investing in a "push left" scenario. Put these two scenarios side by side, and you'll have a solid case for ROI. The returns for a good application strategy are tremendous. The important thing is to continue to measure returns by formulating a good metrics program. Keep track of how security is helping business and technology improve, measure the drop of defects per lines of code, measure the amount of risk reduction. Show value. Demonstrate how your program is embedding security into the organization's DNA.

To my second point, a well thought out application security strategy can help the CSO politically. If you hire the right talent, and approach business and technology with the right attitude, i.e. to enable and not disable, you'll make a lot of new friends. Security departments often complain that the revenue generating business units view them with disapproval, yet the quest for application security is a fantastic opportunity for the security team to work closely with business. Do it right, and the business will love you for it. You will win their credibility and gratitude.

How important is threat modeling?
If you want to do application security right, you've got to invest in threat modeling. The goal of a threat model is to enumerate the malicious entity's goals even if the threats being enumerated have been mitigated. This helps the business, developers, architects, and security analysts understand the real world threats to their applications. Threat modeling should be initiated during the design phase of the application and it should be treated as a living document. As the application development process progresses, the threat model can be further enhanced so it is increasingly valuable. For example, a threat model created during the design phase can be further augmented to map to actual code review results to help developers and architects understand areas they need to improve on and areas where they are doing a good job. Threat modeling is a core component of the push-left strategy, so you eliminate defects as early as possible.

What recommendations would you give to a new organization that is just starting to build an application security strategy?
Once you've derived from your overall business and security goals, it's time to list specifics.

1. Talent and Framework. Hire the right talent and lay the framework: policies, requirements, best practices, and methodologies.

2. Kick start efforts on critical applications.  Kick start your efforts on your critical applications: work with business and technology to help them understand the risks to their applications and what they can do to eliminate them as early as possible. Help them invest their by offering advise on their architecture level security controls and threat modeling. Give the development teams guidance on secure coding policies. Assess the code for security defects, followed by a penetration test before the application is turned over to production. Ensure proper application logging mechanisms are built in and monitored. 

3. Application portfolio. Come up with a formula to calculate business impact of an application based on key questions. Rank the applications by impact, and assign effort. At this point, you may want to take regulatory requirements into account, most often based on the type of data handled.

4. Invest in training and awareness. The security team, business, and technology must have access to continuous security training. Calculate metrics from code review results to target security training to certain business units. After a code review assessment, get a few of the developers into a room and show them the impact of the vulnerabilities found. Work together to enhance the threat model and possibly fixing the defects. The goal is training, awareness, and knowledge transfer.

5. Metrics. Demonstrate value, for example, a graph showing defects per lines of code decrease within the span of the last few months. Demonstrate risk reduction per business unit which often leads to some healthy competition, and that's a good thing. Overall, you've got to show application risk reduction across the enterprise.

6. Stay cutting edge. Retain talent. Treat your team well. Understand the latest attack vectors. Invest in research. Communicate and support the business - they are your clients and they need your help.

Be Secure, and You'll be Compliant

There's been some recent chatter and speculation on the upcoming enhancement to the PCI standard. Among the discussions, I'd like to publicize my opinion on one argument I've heard multiple times during the last few days. The argument goes something like this: The cost of performing security code reviews is too high, but the cost of performing black box reviews and/or implementing web application firewalls is lower. Therefore, the solution is to recommend that organizations rely on penetration assessments and/or web application firewalls.

Three points:

First, no company should ever strategize their overall security efforts based on a 3rd party requirement. A company's strategy should be based on its specific business goals that should be used to drive the security strategy. Tony Spinelli, CSO of Equifax, has articulated this point very well. He says: Most companies and [their] security leaders are getting lost because of [having to be] compliant -- regulations saying you have to do X or Y.... A lot of people are letting compliance drive security, and that's as wrong as you can get....You have to become secure to be compliant; otherwise, you respond and react and reinvest without leverage.

Second, it is not true that security code reviews are overwhelmingly more expensive than black box reviews. The entire purpose of a security code review is to combine it with a solid security SDLC process, with the aim to push left, and the goal to find and remediate security vulnerabilities earlier on in the development cycle - the overall costs of which is likely to be lower than the cost of relying upon black-box penetration assessments. Run, don't walk, from any vendor that tells you to base your application security strategy on black-box penetration assessments because anything else is too expensive - you'll end up paying through your nose while failing to fix the root cause of what's ailing your development efforts.

Third, web application firewalls can be useful, yet the most terrible band-aids when applied for the wrong reasons. Just because a 3rd party standard may require it doesn't mean it's the only thing you need to do.

In summary, please do not let a requirement like PCI drive your overall strategy. Understand your goals and needs, aim to be secure, and you will be compliant. Try the formula the other way around, and your strategy will be flawed, your security budget won't be big enough, you will struggle to keep up with requirements & regulations, and you will fail to demonstrate risk reduction to your organization.