Security Case Study

Table of Contents

Project Objectives

In this group project, you will get acquainted with the security challenges of a real, complex, messy software product. In class, you learn about security ideals, common mistakes, and best practices. In this case study, you will see how those ideals are applied (or not applied!) to actual systems. In the process, you'll investigate software projects from the outside in, reading bug reports, documentation, mailing lists, commit logs, and any other primary source material you can get your hands on.

Proposal

You must find your own team for the case study. (2-3 members per team). As a team, choose a software project for your case study carefully. The minimum requirements for a good case study project are:

  • Domain: The project must have significant security risks
  • Scale: The project must be of non-trivial size. It should be large enough that one single person cannot know everything about it. A rule of thumb is 10k+ lines of code or 20 developers. Be careful not to pick too large of a project either, however. For example, an entire operating system is not recommended. You will not need to compile the project in order to write this case study.
  • Deployed: The project must actually be used in production
  • Source code: Source code must be available
  • Vulnerabilities: The project must have reported vulnerabilities in recent history. The more detailed the records, the better. For example, do the vulnerability reports trace to bugs and source code patches? Do they have defect tracking of their vulnerabilities? Note that third-party sources such as the CVE or CVEDetails.com are NOT sufficient for this project. The project themselves need to acknowledge vulnerabilities and you must look into the primary project history yourself.
  • Unique: Your choice must not be the same as other groups in the class

Your proposal should be written in paragraph form and must provide a convincing narrative about how you will be able to study your project empirically. In your proposal, include the following information:

  • The names of each team member
  • Project overview
    • What is the product used for?
    • What is the development team like? (size? composition?)
    • Are there any corporate sponsorships or connections? Or is it primarily volunteer-driven?
    • How often does the project release?
  • Security Risks
    • What are some example security risks of this project?
    • What kinds of data does this project protect?
    • What are the ramifications if the software is compromised?
  • Links
    • Project website
    • Source code repository
    • Issue tracker
    • Reports of past vulnerabilities
    • Other relevant information

Chapter 1: Security Assessment

The purpose of this chapter is to provide an assessment of the security risks posed by your selected software project. Your chapter must include:

  • Project Overview. A description of the project itself and an overview of the findings of this chapter. Discuss users, purpose, business objectives, development team, development process, and any other information that might be relevant to understanding the project. This section must be more in-depth than your case study proposal.
  • Assets. Provide a listing (i.e. a table or figure) and discussion of the project resources, functionality, and intangible properties that an attacker would want to exploit. Be as comprehensive as possible. For example:
    • What are the pieces of data that this system is supposed to protect? Be specific - use names of files, use case-study specific names. "User data" is too broad.
    • Where are the places that unexpected code execution could occur? Think both within your program, and using your program as a vector into its host system.
    • Is the overall availability of your program an asset to protect?
    • What are the "things" that this system is supposed to protect?
  • Example Attacks. Provide some examples scenarios of how someone could attack this system. Highlight what elements of CIA (Confidentiality, Integrity, Availability) could be compromised.
    • Think about malicious actors for this system. What would they want to do to it? What would they want to do with it?
    • Think about the assets protected by your program. Is this list comprehensive? What would an attack on each of those assets look like?
    • For all example attacks, be more specific than a misuse or abuse case. Include both the "how" and the "what" of each attack.
  • Vulnerability History. Vulnerabilities are engineering failures that should be understood in detail. Each member of your team must research one vulnerability for your case study and provide an analysis. For each vulnerability, you must provide the following:
    • A unique name, such as the CVE number
    • A link to the development team's acknowledgment of this vulnerability. This is NOT the CVE entry (which is a third-party report), but the original team acknowledging that they fixed this vulnerability.
    • Link to the source code commit(s) (e.g. Git, SVN) where the vulnerability was fixed.
    • Link to the source code commit(s) where the vulnerable code was originally introduced. (Use git blame and other annotation tools to traverse the history of the code). This is going to be an informed judgment call on your part - people don't usually track this data. Make this a part of the discussion.
      • Note that we are not asking for when a vulnerability was reported (typically these dates are just a few days prior to the fix and are uninteresting). From an historical perspective, I don't care about how long it took them to fix the vulnerability, only how we can further prevent vulnerabilities like this one to happen in the future.
    • Explain the vulnerability in your own words, answering the following questions:
      • What does the vulnerability allow?
      • From a coding perspective, what was the specific mistake that was made?
      • What software engineering practices could have helped find this problem prior to being released to the public?
      • What category of CIA does this vulnerability fall into?
      • How long was the vulnerability in the system from the time it was introduced to fixed?
      • How many different developers worked on this source code file(s) from when it was introduced to fixed? (study git log and similar commands depending on their version control system)

Remember: you are now a software archaeologist. This is a research project as much as it is a writing project. Plan on taking a lot of time tracking down source control commits, release histories, bug reports, mailing lists, or whatever software development artifacts you need to track down. Feel free to also share your sleuthing strategies with your team and your class - there are many clever ways of mining software repositories for this information.

Remember to keep the writing cohesive. I recommend setting a team writing deadline ahead of time, and giving a few days to revise. Even though the research is individual, the team should be collaborating heavily on the writing.

Chapter 2: Design Analysis

The purpose of this chapter is to assess the high-level design of the system. Your chapter must include:

  • Architecture Overview. Describe the architecture of the system. Please answer the following questions:
    • What are the subsystems? What does each subsystem do?
    • Are there subsystems that are expressly security features? (e.g. encryption, authentication) Describe these in more depth.
    • How critical is this subsystem? Consider the cost of developer mistakes in terms of subsystems. For example "if a developer makes a mistake in this subsystem, what happens?"
    • Do some subsystems appear to be more susceptible to code-level vulnerabilities?
    • Do you see security built into this system at this architectural level? (e.g. "distrustful decomposition" of overall project into separate components)
    • How might compromising one subsystem affect the security (integrity, reliability, etc) of the others or of the system as a whole?
    • How does the system's interaction with external systems put it at risk?
    • How might the system architecture change over time?
  • Threat Model. Build a threat model diagram of the system.
    • What are the machine boundaries of the system? Does this vary from deployment to deployment?
    • What are the trust boundaries of the system? Are there any trust boundaries more important than others?
    • Please make this diagram legible - I would rather you show the riskiest parts of the threat model than enumerate every possibility. As a guide, threat models get pretty confusing after about 12 primitives per diagram. If you feel your diagram must be more complex, feel free to break it up into multiple diagrams.
  • Assets to Threat Model Tracing. Provide a mapping from your assets in Chapter 1 to the elements in your threat model. There should be a figure or table that maps primitives in your threat model to the assets listed in Chapter 1. Discuss this mapping:
    • Does the location in the architecture affect the potential or confirmed exploitation of an asset? Why or why not?
    • Are there locations in the architecture where no assets reside? (If so, are you sure you didn't miss any assets there?)
    • In your opinion, are there locations in the architecture where too many assets reside?

Note: Inferring the architecture for the system can be a real challenge if it's not documented. Here are a few places you can look for hints on what the subsystems are:

  • Bugs. Any custom fields, or what is mentioned in discussions.
  • File system layout.
  • Javadoc or other documentation.
  • API documentation for third-party developers can imply the subsystem layout.
  • You can always ask the actual developers on their mailing list. If you do, be sure to have the whole group review the email before it goes out.

Also due with Chapter 2 are revisions to your earlier Chapter 1, based on prior feedback received from the instructor.

Presentation

Prepare a presentation of your case study. You do not need to cover every paragraph of your paper. Instead, choose the most interesting part(s) of your case study and present it in 20 minutes, with 5 minutes of questions. PowerPoint is not required, but is recommended, at least for technical diagrams. Each group member should participate in part of the overall presentation.

Grading

Proposal (10 pts)

  • 3pts - Overview: Proposal provides a high-level overview of the project
  • 5pt - Security Risks: Proposal demonstrates a surface-level understanding of the project's security risks
  • 2pts - Links: Proposal includes relevant technical links

Chapter 1 - Security Assessment (40pts)

  • 5pts - Overview section
  • 10pts - Assets section
  • 10pts - Example Attacks section
  • 15pts - Vulnerability History section

Chapter 2 - Design Analysis (40 pts)

  • 20pts - Overview and subsystems
  • 15pts - Threat Model diagram and description
  • 5pts - Mapping to Assets

Presentation (10 pts)

  • 4pts - Demonstrating thorough research of the project
  • 4pts - Engaging choice of content from the case study
  • 2pts - Engaging style

Submission

Submit all files to the Canvas CMS site. Only one submission is needed for the group.

There will be separate Canvas assignments for:

  1. Proposal
  2. Case Study (Chapter 1)
  3. Case Study (Chapters 1 and 2 combined)
  4. Presentation

With thanks to the original Case Study project in SSE 331 "Engineering Secure Software" at RIT