Project Requirements and Grading

Introduction

One of the most important aspects of 5271 is a research-oriented group project. The goal of the project is to introduce you to current research topics and methods in computer security. Based on the experience of past offerings of the course, the most reliable style of project is one that engages with a recently published research paper in the area. So our plan is that most projects will be based on a paper published in one of the major security conferences since 2013.

Here's a table with links to the programs and papers from the four major general computer security conferences (in alphabetical order) since 2013:

There are many ways that a paper could be used as the basis for a class project, but the most likely possibilities are:

• Replicate and Extend: The original paper describes an attack or defense mechanism; your project independently reproduces the paper's results, and extends them to a new scenario, such as a different device, operating system, or attack model.
• Replicate and Smash!: The original paper describes a defense mechanism; your project independently reproduces the evaluation (or describes why the evaluation is flawed) and then describes and evaluates an attack on this mechanism.

Other forms of engagement with a previous paper, or projects that do not engage a previously published paper may also be suitable as topics for a class project, but should be discussed with the instructor at a very early stage (in particular before the pre-proposal due date).

The required components of a project are:

1. A "pre-proposal" listing your group members and a tentative topic.
2. Three individual "progress reports" that summarize a discussion of your progress with the instructor.
3. An oral presentation on your project by one group member, to share what you have learned with the class.
4. A paper reporting the results of your project.

Due dates

• Preproposal: by 11:55pm on Wednesday, September 20th
• Progress meetings: as scheduled, about once a month in between progress reports
• Progress reports: by 11:55pm on Wednesday, October 11th; Wednesday, November 8th; and Monday, December 4th.
• Final report: Wednesday December 13th (last day of instruction, so no extensions)

Group size

Groups should have a minimum of 3 students and a maximum of 6. The preferred group size is 4-5. Groups of 4 or fewer students risk being merged with another group if there are too many groups.

Meeting times

Your group will meet with the instructor to discuss your progress three times during the semester, approximately in late September, late October, and late November/early December, in between the progress reports.

To facilitate scheduling these meetings, you should pick half-hour time slots from among the following when your group members would be available to meet. To help ensure the scheduling problem is satisfiable, each slot is listed with a points value of (2), (4), or (5) in inverse proportion to how popular I expect it will be, among other factors. You should give an ordered list of slots that would work for your group, in your order of preference, but the slots in the list should add up to at least 10 points (more popular times will have more contention, so you need to give more choices so that at least one will be available). Also to encourage you to choose more diverse slots, deduct one point for any two slots that are adjacent. So for instance you could give just two slots if they're both 5 points and not adjacent, but if you only choose from the more popular 2-point slots you need to give five or more.

• Monday 8:00a- 8:30a (5)
• Monday 8:30a- 9:00a (5)
• Monday 9:00a- 9:30a (4)
• Monday 9:30a-10:00a (4)
• Monday 11:00a-11:30a (2)
• Monday 11:30a-12:00n (2)
• Monday 12:00n-12:30p (2)
• Monday 2:30p- 3:00p (2)
• Monday 3:00p- 3:30p (2)
• Monday 3:30p- 4:00p (3)
• Monday 4:00p- 4:30p (3)
• Monday 4:30p- 5:00p (3)
• Tuesday 9:00a- 9:30a (4)
• Tuesday 9:30a-10:00a (4)
• Tuesday 11:30a-12:00n (2)
• Tuesday 12:00n-12:30p (2)
• Tuesday 12:30p- 1:00p (2)
• Tuesday 1:00p- 1:30p (2)
• Tuesday 1:30p- 2:00p (3)
• Tuesday 4:00p- 4:30p (4)
• Tuesday 4:30p- 5:00p (4)
• Tuesday 5:00p- 5:30p (5)
• Tuesday 5:30p- 6:00p (5)
• Wednesday 11:00a-11:30a (2)
• Wednesday 2:30p- 3:00p (2)
• Wednesday 3:00p- 3:30p (2)
• Wednesday 3:30p- 4:00p (3)
• Wednesday 4:00p- 4:30p (3)
• Wednesday 4:30p- 5:00p (3)
• Wednesday 5:00p- 5:30p (4)
• Wednesday 5:30p- 6:00p (4)
• Wednesday 6:00p- 6:30p (5)
• Wednesday 6:30p- 7:00p (5)
• Thursday 9:00a- 9:30a (3)
• Thursday 9:30a-10:00a (3)
• Thursday 11:00a-11:30a (2)
• Thursday 11:30a-12:00n (2)
• Thursday 2:00p- 2:30p (2)
• Thursday 2:30p- 3:00p (2)
• Thursday 3:00p- 3:30p (2)
• Thursday 3:30p- 4:00p (2)
• Thursday 4:00p- 4:30p (2)
• Thursday 4:30p- 5:00p (2)
• Thursday 5:00p- 5:30p (3)
• Thursday 5:30p- 6:00p (3)
• Thursday 6:00p- 6:30p (4)
• Thursday 6:30p- 7:00p (4)
• Thursday 7:00p- 7:30p (5)
• Thursday 7:30p- 8:00p (5)
• Friday 10:00a-10:30a (2)
• Friday 10:30a-11:00a (2)
• Friday 11:00a-11:30a (2)
• Friday 11:30a-12:00n (2)
• Friday 12:00n-12:30p (2)
• Friday 12:30p- 1:00p (2)
• Friday 1:00p- 1:30p (3)
• Friday 1:30p- 2:00p (3)
• Friday 2:00p- 2:30p (3)
• Friday 2:30p- 3:00p (3)
• Friday 3:00p- 3:30p (2)
• Friday 3:30p- 4:00p (2)
• Friday 4:00p- 4:30p (3)
• Friday 4:30p- 5:00p (3)
• Friday 5:00p- 5:30p (4)
• Friday 5:30p- 6:00p (4)
• Friday 6:00p- 6:30p (4)
• Friday 6:30p- 7:00p (4)
• Friday 7:00p- 7:30p (5)
• Friday 7:30p- 8:00p (5)

In tabular form:

Detailed schedule

• Pre-proposal. (5% but required)
  By 11:55pm on Wednesday, September 20th, one member of each group should submit a one-page pre-proposal (in PDF format) with the following information:
• Who: All members of the proposed group, and a group or project name.
• What: The paper(s) your project will be based on (listing one or two related choices for discussion would be acceptable).
• Why: Why is this a good paper for your group to base a project around? What skills and resources can you access that will allow you to succesfully complete the project?
• How: How do you intend to reproduce the paper's results? What extensions or attacks do you have in mind, and how will you evaluate them?
• When: A list of at half-hour time slots during a typical week in which your group members are all available for a progress meeting with the instructor. A list of the instructor's available slots are shown above.
• Progress meetings and reports. (5% each but required)
  After submitting the pre-proposal, each group will receive by email a schedule of progress meeting times. Students are expected to attend as many of these progress meetings as possible. At a minimum, each student should attend at least two meetings and each meeting should be attended by at least two team members. After each progress meeting, every student must submit an individual progress report which includes the following information:
• What contributions you, indvidually, made to the project since the previous meeting. (Do not list items that will apply generically, such as learning tools that are not specific to the topic of your project)
• Why and how this is different from the agreed-on course from the previous meeting. (E.g. "approach X failed because...") There is no need to justify time management difficulties, but this should not be a frequently given reason!
• What tasks you plan to accomplish by the next meeting.
• Any "deliverables" agreed on in the previous meeting (e.g. a multi-paragraph summary of some piece of related work; a draft of some portion of the report)

In addition, along with the third and final progress report, each group should also submit an example of the formatting they will be using for the final report, which should follow the ACM proceedings format. This could be a very early draft of the report, or just the usual progress information, but the main point is to give you some practice working with the formatting conventions. 8-12 pages in this format is more text than 10-page reports you might have written in other classes.

• Presentation (10%). Each group will give an in-class presentation of their work during the last few lectures. This could be a demo, and/or a presentation with slides. The time available will depend on the number of groups--likely around 10-15 minutes. The entire group has responsibility to ensure the best possible presentation, but given the time constraints we expect that only one team member will speak. (Presenting will be taken into account along with that student's other individual contributions as described below.)
• Final Report (70%). By 11:55pm on Wednesday, December 13th, one member of each group should submit an archive which includes any "deliverables" and a project report in PDF format. The report should be in the style of a computer science conference paper, and it should present the project's motivation, approach, results, contribution, related work, and conclusions. The report should be between 8 and 12 pages plus references, using the ACM proceedings format. A separate file ("contrib.pdf") should summarize the hours spent, and contributions of, each group member.

Seeing that the pre-proposal and project reports each contribute only 5% to your project grade, you should not conclude that they are unimportant. Even though you don't get much credit just for completing them, these reports and meetings are an important part of choosing a good topic and making progress on it during the semester. (That's also why the instructor allocates quite a bit of his time to them.) Neglecting these milestones is a danger sign for your project, and the instructor reserves the right to deduct additional penalties, up to including assigning a grade of F for the entire project, if you blow them off entirely.

Report grading

Report grades will be based on the following criteria:

• Originality (15%) To get full credit, your project should contain new ideas beyond the paper it engages (new design, new analysis, new implementation, new evaluation), of the sort that could be published in an academic conference (not necessarily top-notch). A project that is well done and the result of lots of hard work but has no original ideas or results is worth a maximum of 85%.
• Scholarship (15%)

  A good scholar knows the state of the art in their project area. Think about questions like: Does the paper include the important related work? How is the paper contrasted with other work? How well do the authors distinguish between what has been written by others and their own work? A sign of poor scholarship is a lack of well-explained comparisons to previous papers from top-notch venues; in particular, any good project should be able to cite, and explain its relationship to, at least 5-10 such papers.

  What are top-notch venues? In security, the "highest-quality" conferences are (in alphabetical order) CCS, NDSS, Oakland, and USENIX Security. Other good conferences with slightly lower quality or a more narrow scope include RAID, ACSAC, ESORICS, CSF, PET(S), WEIS and SOUPS. The top cryptography conferences are CRYPTO, Eurocrypt, Asiacrypt and TCC. Good conferences in some other fields related to security include STOC, FOCS, SODA and PODC for theory; SIGCOMM, INFOCOM and NSDI for networks; OSDI and SOSP for operating systems; and CHI and UIST for usability. This is not an exhaustive list, and of course other venues can publish important and interesting papers, but if you have no papers from venues on this list you are probably missing some of the definitive work related to your topic. (Or working on a topic that is not of much interest to security researchers)

• Evaluation (30%) To what extent do the authors evaluate their approach and establish or support their claims? For example:
  • If you propose a new extension to defense mechanism, what are some potential ways to defeat or avoid it? How feasible are they? How did your system perform on standard benchmarks?
  • If you propose a new attack, how realistic is it? Can you carry it out empirically? Are there simple countermeasures that can reduce the effectiveness of your attack? Is your attack trivially less effective than known previously-known ones?
  • When reporting on experiments, do you analyze their statistical significance? Do you give enough information that another group could replicate your results? If your claims are analytical, do you give enough details of the proofs? An important part of the evaluation is simply making the description of your project clear enough for others to understand what you have done!
• Individual Contribution (40%). Each student should have made some identifiable contribution to the project. This component will be based on the individual student's contribution: did you attend progress meetings? Did you contribute (at least) as much to your project as others? Did you incorporate feedback received at meetings throughout the semester? A graduate class major project like this one is expected to require at least 50 hours of work per group member over the course of the semester (and that's for an average project, not an "A" project). Of course the amount of time spent isn't a sufficient condition for a good project. But our experience is that being unable to devote enough time to a project is the most common cause of poor results. That's why we ask you to keep track of the time you spend and think about how to use your time effectively.

(This project description is based on descriptions used in previous editions of 5271 and written by Prof. Nick Hopper.)