Project Requirements and Grading
One of the most important aspects of 5271 is a research-oriented group
project. The goal of the project is to introduce you to current
research topics and methods in computer security. Based on the
experience of past offerings of the course, the most reliable style of
project is one that engages with a recently published research paper
in the area. So our plan is that most projects will be based on a
paper published in one of the major security conferences since 2013.
Here's a table with links to the programs and papers from the four
major general computer security conferences (in alphabetical order)
There are many ways that a paper could be used as the basis for a
class project, but the most likely possibilities are:
- Replicate and Extend: The original paper describes an
attack or defense mechanism; your project independently reproduces the
paper's results, and extends them to a new scenario, such as a different
device, operating system, or attack model.
- Replicate and Smash!: The original paper describes a
defense mechanism; your project independently reproduces the evaluation
(or describes why the evaluation is flawed) and then describes and
evaluates an attack on this mechanism.
Other forms of engagement with a previous paper, or projects that
do not engage a previously published paper may also be suitable as
topics for a class project, but should be discussed with the
instructor at a very early stage (in particular before the
pre-proposal due date).
The required components of a project are:
- A "pre-proposal" listing your group members and a tentative topic.
- Three individual "progress reports" that summarize a discussion of your progress with the instructor.
- An oral presentation on your project by one group member, to share what you have learned with the class.
- A paper reporting the results of your project.
- Preproposal: by 11:55pm on Wednesday, September 20th
- Progress meetings: as scheduled, about once a month in between progress reports
- Progress reports: by 11:55pm on Wednesday, October 11th; Wednesday, November 8th; and Monday, December 4th.
- Final report: Wednesday December 13th (last day of instruction, so no extensions)
Groups should have a minimum of 3 students and a maximum of 6. The
preferred group size is 4-5. Groups of 4 or fewer students risk being
merged with another group if there are too many groups.
Your group will meet with the instructor to discuss your progress
three times during the semester, approximately in late September, late
October, and late November/early December, in between the progress
To facilitate scheduling these meetings, you should pick half-hour
time slots from among the following when your group members would be
available to meet. To help ensure the scheduling problem is
satisfiable, each slot is listed with a points value of (2), (4), or
(5) in inverse proportion to how popular I expect it will be, among
other factors. You should give an ordered list of slots that would
work for your group, in your order of preference, but the slots in the
list should add up to at least 10 points (more popular times will have
more contention, so you need to give more choices so that at least one
will be available). Also to encourage you to choose more diverse
slots, deduct one point for any two slots that are adjacent.
So for instance you could give just two slots if they're both 5 points
and not adjacent, but if you only choose from the more popular 2-point
slots you need to give five or more.
- Monday 8:00a- 8:30a (5)
- Monday 8:30a- 9:00a (5)
- Monday 9:00a- 9:30a (4)
- Monday 9:30a-10:00a (4)
- Monday 11:00a-11:30a (2)
- Monday 11:30a-12:00n (2)
- Monday 12:00n-12:30p (2)
- Monday 2:30p- 3:00p (2)
- Monday 3:00p- 3:30p (2)
- Monday 3:30p- 4:00p (3)
- Monday 4:00p- 4:30p (3)
- Monday 4:30p- 5:00p (3)
- Tuesday 9:00a- 9:30a (4)
- Tuesday 9:30a-10:00a (4)
- Tuesday 11:30a-12:00n (2)
- Tuesday 12:00n-12:30p (2)
- Tuesday 12:30p- 1:00p (2)
- Tuesday 1:00p- 1:30p (2)
- Tuesday 1:30p- 2:00p (3)
- Tuesday 4:00p- 4:30p (4)
- Tuesday 4:30p- 5:00p (4)
- Tuesday 5:00p- 5:30p (5)
- Tuesday 5:30p- 6:00p (5)
- Wednesday 11:00a-11:30a (2)
- Wednesday 2:30p- 3:00p (2)
- Wednesday 3:00p- 3:30p (2)
- Wednesday 3:30p- 4:00p (3)
- Wednesday 4:00p- 4:30p (3)
- Wednesday 4:30p- 5:00p (3)
- Wednesday 5:00p- 5:30p (4)
- Wednesday 5:30p- 6:00p (4)
- Wednesday 6:00p- 6:30p (5)
- Wednesday 6:30p- 7:00p (5)
- Thursday 9:00a- 9:30a (3)
- Thursday 9:30a-10:00a (3)
- Thursday 11:00a-11:30a (2)
- Thursday 11:30a-12:00n (2)
- Thursday 2:00p- 2:30p (2)
- Thursday 2:30p- 3:00p (2)
- Thursday 3:00p- 3:30p (2)
- Thursday 3:30p- 4:00p (2)
- Thursday 4:00p- 4:30p (2)
- Thursday 4:30p- 5:00p (2)
- Thursday 5:00p- 5:30p (3)
- Thursday 5:30p- 6:00p (3)
- Thursday 6:00p- 6:30p (4)
- Thursday 6:30p- 7:00p (4)
- Thursday 7:00p- 7:30p (5)
- Thursday 7:30p- 8:00p (5)
- Friday 10:00a-10:30a (2)
- Friday 10:30a-11:00a (2)
- Friday 11:00a-11:30a (2)
- Friday 11:30a-12:00n (2)
- Friday 12:00n-12:30p (2)
- Friday 12:30p- 1:00p (2)
- Friday 1:00p- 1:30p (3)
- Friday 1:30p- 2:00p (3)
- Friday 2:00p- 2:30p (3)
- Friday 2:30p- 3:00p (3)
- Friday 3:00p- 3:30p (2)
- Friday 3:30p- 4:00p (2)
- Friday 4:00p- 4:30p (3)
- Friday 4:30p- 5:00p (3)
- Friday 5:00p- 5:30p (4)
- Friday 5:30p- 6:00p (4)
- Friday 6:00p- 6:30p (4)
- Friday 6:30p- 7:00p (4)
- Friday 7:00p- 7:30p (5)
- Friday 7:30p- 8:00p (5)
In tabular form:
|8:00-8:30am ||5 || || || || |
|8:30-9:00am ||5 || || || || |
|9:00-9:30am ||4 ||4 || ||3 || |
|9:30-10:00am ||4 ||4 || ||3 || |
|10:00-10:30am|| || || || ||2 |
|10:30-11:00am|| || || || ||2 |
|11:00-11:30am||2 || ||2 ||2 ||2 |
|11:30am-noon ||2 ||2 || ||2 ||2 |
|noon-12:30pm ||2 ||2 || || ||2 |
|12:30-1:00pm || ||2 || || ||2 |
|1:00-1:30pm || ||2 || || ||3 |
|1:30-2:00pm || ||3 || || ||3 |
|2:00-2:30pm || || || ||2 ||3 |
|2:30-3:00pm ||2 || ||2 ||2 ||3 |
|3:00-3:30pm ||2 || ||2 ||2 ||2 |
|3:30-4:00pm || || ||3 ||2 ||2 |
|4:00-4:30pm || ||4 ||3 ||2 ||3 |
|4:30-5:00pm || ||4 ||3 ||2 ||3 |
|5:00-5:30pm || ||5 ||4 ||3 ||4 |
|5:30-6:00pm || ||5 ||4 ||3 ||4 |
|6:00-6:30pm || || ||5 ||4 ||4 |
|6:30-7:00pm || || ||5 ||4 ||4 |
|7:00-7:30pm || || || ||5 ||5 |
|7:30-8:00pm || || || ||5 ||5 |
- Pre-proposal. (5% but required)
By 11:55pm on Wednesday, September 20th, one member of each group should submit a one-page pre-proposal (in PDF format) with the following information:
- Who: All members of the proposed group, and a group or project name.
- What: The paper(s) your project will be based on (listing one or two related choices for discussion would be acceptable).
- Why: Why is this a good
paper for your group to base a project around? What skills and resources
can you access that will allow you to succesfully complete the project?
- How: How do you intend to reproduce the paper's
results? What extensions or attacks do you have in mind, and how
will you evaluate them?
- When: A list of at half-hour time slots during
a typical week in which your group members are all available for a
progress meeting with the instructor. A list of the instructor's
available slots are shown above.
- Progress meetings and reports. (5% each but required)
After submitting the pre-proposal, each group will receive by
email a schedule of progress meeting times. Students are
expected to attend as many of these progress meetings as possible. At a minimum, each student should attend at least two meetings and each meeting should be attended by at least two team members. After each progress meeting, every student must submit an individual progress report which includes the following information:
- What contributions you, indvidually, made to the project since the
previous meeting. (Do not list items that will apply generically,
such as learning tools that are not specific to the topic of your
- Why and how this is different from the agreed-on course from the
previous meeting. (E.g. "approach X failed because...") There is
no need to justify time management difficulties, but this should not be a
frequently given reason!
- What tasks you plan to accomplish by the next meeting.
- Any "deliverables" agreed on in the previous meeting (e.g. a
multi-paragraph summary of some piece of related work; a draft of some
portion of the report)
In addition, along with the third and final
progress report, each group should also submit an example of the
formatting they will be using for the final report, which should
follow the ACM proceedings format.
This could be a very early draft of the report, or just the usual
progress information, but the main point is to give you some practice
working with the formatting conventions. 8-12 pages in this format is
more text than 10-page reports you might have written in other
- Presentation (10%). Each group will give an
in-class presentation of their work during the last few
lectures. This could be a demo, and/or a presentation with slides. The
time available will depend on the number of groups--likely around 10-15
minutes. The entire group has responsibility to ensure the best
possible presentation, but given the time constraints we expect that
only one team member will speak. (Presenting will be taken into
account along with that student's other individual contributions as
- Final Report (70%).
By 11:55pm on Wednesday, December 13th, one member of each group
should submit an archive which includes any "deliverables" and a
project report in PDF format. The report should be in the style of a
computer science conference paper, and it should present the project's
motivation, approach, results, contribution, related work, and
The report should be between 8 and 12 pages plus references, using the
ACM proceedings format.
A separate file ("contrib.pdf")
should summarize the hours spent, and contributions of, each group
Seeing that the pre-proposal and project reports each contribute only
5% to your project grade, you should not conclude that they are
Even though you don't get much credit just for completing them, these
reports and meetings are an important part of choosing a good topic
and making progress on it during the semester. (That's also why the
instructor allocates quite a bit of his time to them.)
Neglecting these milestones is a danger sign for your project, and the
instructor reserves the right to deduct additional penalties, up to
including assigning a grade of F for the entire project, if you blow
them off entirely.
Report grades will be based on the following criteria:
- Originality (15%) To get
full credit, your project should contain new ideas beyond the paper it
engages (new design, new analysis, new implementation, new evaluation),
of the sort that could be published in an academic conference (not necessarily top-notch). A project that is well done and the result of lots of hard
work but has no original ideas or results is worth a maximum of 85%.
- Scholarship (15%)
A good scholar knows the state of the art in their project
area. Think about questions like: Does the paper include the important related
work? How is the paper contrasted with other work? How well
do the authors distinguish between what has been written by others and
their own work? A sign of poor scholarship is a lack of
well-explained comparisons to previous papers from top-notch venues; in
particular, any good project should be able to cite, and explain its relationship to, at least 5-10 such papers.
What are top-notch venues?
In security, the "highest-quality" conferences are (in alphabetical
order) CCS, NDSS, Oakland, and USENIX Security. Other good
conferences with slightly lower quality or a more narrow scope include
RAID, ACSAC, ESORICS, CSF, PET(S), WEIS and SOUPS.
The top cryptography conferences are CRYPTO, Eurocrypt, Asiacrypt and
TCC. Good conferences in some other fields related to security
include STOC, FOCS, SODA and PODC for theory; SIGCOMM, INFOCOM and NSDI
for networks; OSDI and SOSP for operating systems; and CHI and UIST for
usability. This is not an exhaustive list, and of
course other venues can publish important and interesting papers, but
if you have no papers from venues on this list you are probably missing
some of the definitive work related to your topic. (Or working on a
topic that is not of much interest to security researchers)
- Evaluation (30%)
To what extent do the authors evaluate their approach and establish or support their claims? For example:
- If you propose a new extension to defense mechanism, what are some
potential ways to defeat or avoid it? How feasible are they?
How did your system perform on standard benchmarks?
- If you propose a new attack, how realistic is it? Can you
carry it out empirically? Are there simple countermeasures that
can reduce the effectiveness of your attack? Is your attack
trivially less effective than known previously-known ones?
- When reporting on experiments, do you analyze their statistical
significance? Do you give enough information that another group
could replicate your results? If your claims are analytical, do
you give enough details of the proofs? An important part of the
evaluation is simply making the description of your project clear enough
for others to understand what you have done!
- Individual Contribution (40%).
Each student should have made some identifiable contribution to the
project. This component will be based on the individual student's
contribution: did you attend progress meetings? Did you
contribute (at least) as much to your project as others? Did you
incorporate feedback received at meetings throughout the
A graduate class major project like this one is expected to require
at least 50 hours of work per group member over the course of the
semester (and that's for an average project, not an "A" project).
Of course the amount of time spent isn't a sufficient condition for a
But our experience is that being unable to devote enough time to a
project is the most common cause of poor results.
That's why we ask you to keep track of the time you spend and think
about how to use your time effectively.
(This project description is based on descriptions used in previous
editions of 5271 and written by Prof. Nick Hopper.)