Project Requirements and Grading

Introduction

One of the most important aspects of 5271 is a research-oriented group project. The goal of the project is to introduce you to current research topics and methods in computer security. Based on the experience of past offerings of the course, the most reliable style of project is one that engages with a recently published research paper in the area. So our plan is that most projects will be based on a paper published in one of the major security conferences since 2017.

Here's a table with links to the programs and papers from the four major general computer security conferences (in alphabetical order) since 2020:

There are many ways that a paper could be used as the basis for a class project, but the most likely possibilities are:

• Replicate and Extend: The original paper describes an attack or defense mechanism; your project independently reproduces the paper's results, and extends them to a new scenario, such as a different device, operating system, or attack model.
• Replicate and Smash!: The original paper describes a defense mechanism; your project independently reproduces the evaluation (or describes why the evaluation is flawed) and then describes and evaluates an attack on this mechanism.

Other forms of engagement with a previous paper, or projects that do not engage a previously published paper may also be suitable as topics for a class project, but should be discussed with the instructor at a very early stage (in particular before the pre-proposal due date).

The required components of a project are:

1. A "pre-proposal" listing your group members and a tentative topic.
2. Three individual "progress reports" that summarize a discussion of your progress with the instructor.
3. An oral presentation on your project by one group member, to share what you have learned with the class.
4. A paper reporting the results of your project.

Due dates

• Preproposal: by 11:59pm on Wednesday, September 18th
• Progress meetings: as scheduled, about once a month in between progress reports
• Progress reports: by 11:59pm on Wednesday, October 9th; Wednesday, October 30th; and Wednesday, November 27th
• Final report: by 11:59pm on Wednesday, December 11th (last day of instruction, so no extensions)

Group size

Groups should have a minimum of 4 students and a maximum of 6. The preferred group size is 5-6. Groups of 4 students risk being merged with another group if there are too many groups.

Meeting times

Your group will meet with the instructor to discuss your progress three times during the semester, approximately in late September, mid-October, and mid-November, in between the progress reports.

To facilitate scheduling these meetings, you should pick half-hour time slots from among the following when your group members would be available to meet. To help ensure the scheduling problem is satisfiable, each slot is listed with a points value of (2), (3) (4), or (5) in inverse proportion to how popular I expect it will be, among other factors. You should give an ordered list of slots that would work for your group, in your order of preference, but the slots in the list should add up to at least 10 points (more popular times will have more contention, so you need to give more choices so that at least one will be available). Also to encourage you to choose more diverse slots, deduct one point for any two slots that are adjacent. So for instance you could give just two slots if they're both 5 points and not adjacent, but if you only choose from the more popular 2-point slots you need to give five or more.

• Monday 8:00a- 8:30a (4)
• Monday 8:30a- 9:00a (4)
• Monday 9:00a- 9:30a (3)
• Monday 9:30a-10:00a (3)
• Monday 10:00a-10:30a (4)
• Monday 10:30a-11:00a (4)
• Monday 2:30p- 3:00p (2)
• Monday 3:00p- 3:30p (2)
• Monday 3:30p- 4:00p (2)
• Monday 5:00p- 5:30p (2)
• Monday 5:30p- 6:00p (2)
• Tuesday 8:00a- 8:30a (4)
• Tuesday 8:30a- 9:00a (4)
• Tuesday 9:00a- 9:30a (3)
• Tuesday 9:30a-10:00a (3)
• Tuesday 10:00a-10:30a (3)
• Tuesday 10:30a-11:00a (3)
• Tuesday 11:00a-11:30a (3)
• Tuesday 11:30a-12:00n (2)
• Tuesday 12:00n-12:30p (2)
• Tuesday 12:30p- 1:00p (2)
• Tuesday 1:00p- 1:30p (2)
• Tuesday 1:30p- 2:00p (2)
• Tuesday 2:00p- 2:30p (2)
• Tuesday 2:30p- 3:00p (2)
• Tuesday 3:00p- 3:30p (2)
• Tuesday 3:30p- 4:00p (2)
• Tuesday 4:00p- 4:30p (2)
• Tuesday 4:30p- 5:00p (2)
• Tuesday 5:00p- 5:30p (3)
• Wednesday 9:30a-10:00a (2)
• Wednesday 10:00a-10:30a (2)
• Wednesday 10:30a-11:00a (2)
• Wednesday 11:00a-11:30a (2)
• Wednesday 4:00p- 4:30p (2)
• Wednesday 4:30p- 5:00p (2)
• Wednesday 5:00p- 5:30p (3)
• Wednesday 5:30p- 6:00p (3)
• Wednesday 6:00p- 6:30p (4)
• Wednesday 6:30p- 7:00p (4)
• Thursday 10:00a-10:30a (3)
• Thursday 10:30a-11:00a (3)
• Thursday 11:00a-11:30a (3)
• Thursday 11:30a-12:00n (3)
• Thursday 12:00n-12:30p (2)
• Thursday 12:30p- 1:00p (2)
• Thursday 1:00p- 1:30p (2)
• Thursday 1:30p- 2:00p (2)
• Thursday 2:00p- 2:30p (2)
• Thursday 2:30p- 3:00p (2)
• Thursday 3:00p- 3:30p (2)
• Thursday 3:30p- 4:00p (2)
• Thursday 4:00p- 4:30p (2)
• Thursday 4:30p- 5:00p (2)
• Thursday 5:00p- 5:30p (3)
• Thursday 5:30p- 6:00p (3)
• Thursday 6:00p- 6:30p (4)
• Thursday 6:30p- 7:00p (4)
• Thursday 7:00p- 7:30p (5)
• Thursday 7:30p- 8:00p (5)
• Friday 12:00n-12:30p (3)
• Friday 12:30p- 1:00p (3)
• Friday 1:00p- 1:30p (2)
• Friday 1:30p- 2:00p (2)
• Friday 2:00p- 2:30p (2)
• Friday 2:30p- 3:00p (2)
• Friday 4:30p- 5:00p (2)
• Friday 5:00p- 5:30p (3)
• Friday 5:30p- 6:00p (3)
• Friday 6:00p- 6:30p (4)
• Friday 6:30p- 7:00p (4)
• Friday 7:00p- 7:30p (5)
• Friday 7:30p- 8:00p (5)

In tabular form:

Also mention whether your would prefer the meeting to be in-person (in the professor's office), online (Zoom), or if you have no preference.

Detailed schedule

• Pre-proposal. (5% but required)
  By 11:59pm on Wednesday, September 18th, one member of each group should submit a one-page pre-proposal (in PDF format) with the following information:
• Who: All members of the proposed group, and a group or project name.
• What: The paper(s) your project will be based on (listing one or two related choices for discussion would be acceptable).
• Why: Why is this a good paper for your group to base a project around? What skills and resources can you access that will allow you to successfully complete the project?
• How: How do you intend to reproduce the paper's results? What extensions or attacks do you have in mind, and how will you evaluate them?
• When: A list of at half-hour time slots during a typical week in which your group members are all available for a progress meeting with the instructor. A list of the instructor's available slots is available above.
• Progress meetings and reports. (5% each but required)
  After submitting the pre-proposal, each group will receive by email a schedule of progress meeting times. Students are expected to attend as many of these progress meetings as possible. At a minimum, each student should attend at least two meetings and each meeting should be attended by at least two team members. Before the next progress meeting, every student must submit an individual progress report which includes the following information:
• What contributions you, individually, made to the project since the previous meeting. (Do not list items that will apply generically, such as learning tools that are not specific to the topic of your project)
• How much time did you spend on this project up until now? Given an estimate in terms of the number of hours you spent on various tasks.
• Why and how this is different from the agreed-on course from the previous meeting. (E.g. "approach X failed because...")
• What tasks you plan to accomplish by the next meeting.
• Any "deliverables" agreed on in the previous meeting (e.g. a multi-paragraph summary of some piece of related work; a draft of some portion of the report)

In addition, along with the third and final progress report, each group should also submit an example of the formatting they will be using for the final report, which should follow the ACM proceedings format. This could be a very early draft of the report, or just the usual progress information, but the main point is to give you some practice working with the formatting conventions. Note that 8-12 pages in this format is more text than 10-page reports you might have written in other classes.

• Presentation (10%). Each group will give an in-class presentation of their work during the last few lectures. This could be a demo, and/or a presentation with slides. The time available will depend on the number of groups--likely around 10-15 minutes. The entire group has responsibility to ensure the best possible presentation, but given the time constraints we expect that only one team member will speak. (Presenting will be taken into account along with that student's other individual contributions as described below.)
• Final Report (70%). By 11:59pm on Wednesday, December 15th, one member of each group should submit an archive which includes any "deliverables" and a project report in PDF format. The report should be in the style of a computer science conference paper, and it should present the project's motivation, approach, results, contribution, related work, and conclusions. The report should be between 8 and 12 pages plus references, using the ACM proceedings format. Specifically, the report should look similar to other conference papers we have read, most specifically ACM CCS. If you're using the LaTeX version of the ACM template, the setting should be sigconf; there is also a Microsoft Word template. A separate file ("contrib.pdf") should summarize the hours spent, and contributions of, each group member.

Seeing that the pre-proposal and project reports each contribute only 5% to your project grade, you should not conclude that they are unimportant. Even though you don't get much credit just for completing them, these reports and meetings are an important part of choosing a good topic and making progress on it during the semester. (That's also why the instructor allocates quite a bit of his time to them.) Neglecting these milestones is a danger sign for your project, and the instructor reserves the right to deduct additional penalties, up to including assigning a grade of F for the entire project, if you blow them off entirely.

Report grading

Report grades will be based on the following criteria:

• Originality (15%) To get full credit, your project should contain new ideas beyond the paper it engages (new design, new analysis, new implementation, new evaluation), of the sort that could be published in an academic conference (not necessarily top-notch). A project that is well done and the result of lots of hard work but has no original ideas or results is worth a maximum of 85%.
• Scholarship (15%)

  A good scholar knows the state of the art in their project area. Think about questions like: Does the paper include the important related work? How is the paper contrasted with other work? How well do the authors distinguish between what has been written by others and their own work? A sign of poor scholarship is a lack of well-explained comparisons to previous papers from top-notch venues; in particular, any good project should be able to cite, and explain its relationship to, at least 5-10 such papers.

  What are top-notch venues? In security, the "highest-quality" conferences are (in alphabetical order) CCS, NDSS, Oakland, and USENIX Security. Other good conferences with slightly lower quality or a more narrow scope include RAID, ACSAC, ESORICS, CSF, PET(S), WEIS and SOUPS. The top cryptography conferences are CRYPTO, Eurocrypt, Asiacrypt and TCC. Good conferences in some other fields related to security include STOC, FOCS, SODA and PODC for theory; SIGCOMM, INFOCOM and NSDI for networks; OSDI and SOSP for operating systems; and CHI and UIST for usability. This is not an exhaustive list, and of course other venues can publish important and interesting papers, but if you have no papers from venues on this list you are probably missing some of the definitive work related to your topic. (Or working on a topic that is not of much interest to security researchers)

• Evaluation (30%) To what extent do the authors evaluate their approach and establish or support their claims? For example:
  • If you propose a new extension to defense mechanism, what are some potential ways to defeat or avoid it? How feasible are they? How did your system perform on standard benchmarks?
  • If you propose a new attack, how realistic is it? Can you carry it out empirically? Are there simple countermeasures that can reduce the effectiveness of your attack? Is your attack trivially less effective than known previously-known ones?
  • When reporting on experiments, do you analyze their statistical significance? Do you give enough information that another group could replicate your results? If your claims are analytical, do you give enough details of the proofs? An important part of the evaluation is simply making the description of your project clear enough for others to understand what you have done!
• Individual Contribution (40%). Each student should have made some identifiable contribution to the project. This component will be based on the individual student's contribution: did you attend progress meetings? Did you contribute (at least) as much to your project as others? Did you incorporate feedback received at meetings throughout the semester? A graduate class major project like this one is expected to require at least 50 hours of work per group member over the course of the semester (and that's for an average project, not an "A" project). Of course the amount of time spent isn't a sufficient condition for a good project. But our experience is that being unable to devote enough time to a project is the most common cause of poor results. That's why we ask you to keep track of the time you spend and think about how to use your time effectively.

(This project description is based on descriptions used in previous editions of 5271 and written by Prof. Nick Hopper.)