The project has an ultimate due date of Wed, April 21st. In the past, students/mobs have requested extra time on these assignments. I have preemptively granted those requests. I gave you the rest of that week and the weekend to put on finishing touches on your document; you can submit this late-without-penalty up to 10pm on the 26th. Presentations will have to be due on the 21st also. Likewise you have additional leeway to have met with me on or before the evening of the 28th. DRAFT "Progress Report" - 10% overall grade I'll ask to receive a progress status report from you all on Wed the 14th. This can be in a rougher shape, and I won't grade as harshly on grammar. But it should have the content of what you've done (try to be getting close!) and what you're prepared to do next and how you're going to do it. Submit it as a PDF with links to a repository I can access. This should include enough detail to convince the reader that you've got a good problem you understand, that you've already gotten pretty far in attacking it, and you have a pretty firm idea what you have left to do to finish it off. This last part should not be vague or gauzy. It should not approach the [Feynman problem solving algorithm](https://wiki.c2.com/?FeynmanAlgorithm). If automating, strongly consider adding a by-hand, "like-a-professional" style proof of the remainder of what you have yet to automate. Use this to make sure you are sufficiently far along, not over your heads, and that you have or are working to get the resources you need to proceed. Read the friendly manual, and co. Also, do not hesitate to reach out; you can update me or one of our other staff members before this first due date. We can talk if you get stuck. I'm not super scary, nor do I want to be, and they aren't either. 25%: Clarity and scope of setup and explanation of work thus far. 25%: Description of current concerns and issues, and outstanding to-do. 25%: Overall first draft report content/status. 25%: Code status, thus far (if ACL2 code, I should be able to load and run in ACL2s.) FINAL PROJECT+PRESENTATION 15% overall grade Along with the project/code itself, each pair/mob will write and submit a report. Assuming decent, standard font and margin sizes, I imagine that an adequate accompanying report will be about six pages (excluding references and appendix). These can be short because they get directly to the point, and because you are supplementing them with some thoroughly legible and well-documented source code. Try to cut out fluff and/or filler words/sentences; *omit anything for which the converse is obviously false*. I would worry if it seems like you can set everything forth in under four pages, and I would also worry if you start going beyond eight. These are rough, approximate guidelines, please do not feel compelled to play the font family/size/margin game to hit an upper or lower limit. Submit this as a PDF with links to a repository I can access. # Project Grading Rubric This is what I earnestly expect to use as my rubric to grade your projects. This is of course not an iron-clad legally binding contract, but my best effort, and so might require some adjustments. As with other parts of the course, "pull requests" are welcome. Like I assess your work and offer suggestions, corrections, and improvements, your feedback is valuable and you have unique perspectives to offer. ## Preliminary Criteria I will look further at and grade closer only something meeting basic, prerequisite, base-line minimum-specification expectations. These include: - Be on topic. - Your group will schedule and present your question, research, and solution - You offer a demonstrable, excutable solution. - You provide an accompanying report/document - Clear articulates and addresses the research question to which we agreed - Prose in English. - Your report provides me a working link to your source code ## Actual Rubric I have sub-divided this assessment into parts. Further, each of these components is sub-divided into the following sub-components, or criteria by which I will assess them. I am measuring each of those sub-areas via their self-evident description and the criteria outlined in the following sections. Pay special attention to the style and grammar section as outlined in the below description. | REPORT CONTENT | Missing | Beginning / Developing | Meets Expectations | Excellent / Advanced | |----------------------------+---------+------------------------+--------------------+----------------------| | General problem area | | | | | | Approach | | | | | | Methodology | | | | | | Metrics | | | | | | Summary | | | | | |----------------------------+---------+------------------------+--------------------+----------------------| | SOFTWARE | Missing | Beginning / Developing | Meets Expectations | Excellent / Advanced | |----------------------------+---------+------------------------+--------------------+----------------------| | Well-written documentation | | | | | | Source code | | | | | | Tests | | | | | | Incidental code quality | | | | | |----------------------------+---------+------------------------+--------------------+----------------------| | REPORT STYLE/WRITING | Missing | Beginning / Developing | Meets Expectations | Excellent / Advanced | |----------------------------+---------+------------------------+--------------------+----------------------| | Organization | | | | | | Depth/Emphasis | | | | | | Language | | | | | | Mechanics | | | | | | Minor Errors | | | | | |----------------------------+---------+------------------------+--------------------+----------------------| | PRESENTATION/DEMONSTRATION | Missing | Beginning / Developing | Meets Expectations | Excellent / Advanced | |----------------------------+---------+------------------------+--------------------+----------------------| | Organization | | | | | | Depth/Emphasis | | | | | | Program Demonstration | | | | | | Style | | | | | | Teamwork | | | | | |----------------------------+---------+------------------------+--------------------+----------------------| | RECEIPT | |-----------------------------------------------------------------------------------------------------------| I outline below the criteria I intend to use to assess each of the above-mentioned aspects of your presentation/demonstration. ## Report Content (25%) ### General problem area - Clear, engaging introduction that narrows in focus as it proceeds - Contextualizes the problem within the related (narrowly construed) research area ### Approach - Discusses and compares (related/the same) solutions to (the same/related) problems. - Unambiguously demarcate the novel part of your work. - Restate here the predefined metric for success (either from your proposal or meeting w/me) ### Methodology - What specific steps did you take? - Which of these steps proved particularly hard? - Which of these steps were unexpectedly hard or intriguing? ### Metrics - Did you succeed by your own metric for success? - Provide supporting evidence and clear reasoning for the preceding.* - How well/how far do your results generalize? - What lingering, niggling doubts make you qualify your achievement? If you did not achieve the intended results, see me. It's better than weasel-wording around it. ### Summary - What did we (civilization) learn from you accomplishing this project? - What remaining, related, still-open research problems/issues or corollaries are now within easier reach? ## Software Criteria (25%) I outline below the criteria I intend to use to assess each of the above-mentioned aspects of your software. ### Source - Repo, directories, files, modules, classes, etc are well-organized and logically structured - Transparently states required packages/dependencies and how to install them - Source directory includes test data - Source code is reasonably well-written - Comprehensible error messages correctly report errors or bad data. ### Well-written documentation, including: - Classes, methods, and functions have signatures and purpose statements - Data descriptions that express well-formed and valid inputs - For lemmas, give an English statement of what each lemma shows. - Comments sufficient to clarify difficult/unique/opaque portions - A repo-level README w/setup instructions - Diagrams of program's overall organization, structure or data flow ### Tests - Tests help methods Comprehensive tests exhibit program's - completeness viz. research question Tests show program's full - input/output behavior and exhibits use-case. Some independent - verification exhibits program's correctness. For some style of projects, proofs themselves will be a part of this testing. ### Incidental code quality This is deliberately vague, and intended to catch some issues not otherwise mentioned. These can include non-critical bugs or inefficiencies, missing edge cases, and inadequate/inappropriate use of libraries. ## Report Style/Writing (25%) I ask that you follow these general guidelines of a small proposal. Beyond merely content, we will also judge your prose itself as scientific writing. This will be a significant part of your grade. Do not under-emphasize this. Failing to lint for these common errors or to read the relevant, related pages describing each piece was a common flaw in under-performing submissions. I'll be more persnickety on final submission than on your drafts. Some of you might consider a tool like http://www.hemingwayapp.com/. Treat the [following guidelines and references](http://writing.engr.psu.edu/checklists/proposal.html) as the criteria by which we will assess your writing. For a copy of the reference text, [The Craft of Scientific Writing](https://onesearch.library.northeastern.edu/permalink/f/365rt0/NEU_ALMA51327615270001401) check out the e-book from the NEU library. ## Presentation/Demonstration (10%) Show me that your project works. Let me see that everyone has an equal understanding of the problem and the result, with everyone able to equally explain each part. Be able to answer my interested questions. You needn't put together a slide deck, Consider a slide deck if it would help you walk me through your document. ### Organization - Clear, engaging introduction body and conclusion - Recapitulate the problem area - Logical progression of ideas - Smooth transitions to different parts ### Depth/Emphasis - Situates your precise problem in context - Maintains focus on your work and contribution - Explanations have appropriate level of detail - Devotes ample time for demonstration ### Program Demonstration - Demonstrate an executable solution - Explain *how* this demonstrates the solution - Walk an audience through how your solution works internally - Answer audience questions about research and result ### Style - Professional demeanor - Clear and articulate delivery - Adequate vocabulary and nomenclature ### Teamwork - Group coordinates to suggest and schedule a 15-20m block w/course staff. - Group members can trade off with one another - All members of the group can demonstrate mastery of the results ## Receipt (5%) Submit to me a "receipt" for either having checked out from the library one of (several well-known and trusted style guides)[https://subjectguides.lib.neu.edu/c.php?g=713111&p=5073143#s-lg-box-wrapper-18808483] or for having had (an appointment at the writing center)[https://cssh.northeastern.edu/writingcenter/tutoring/online-appointments/] where you have copy-edited your draft. They have a whole bevy of resources for you. Please note I am asking you to do more than an elementary grammar check and appropriate citations not just for egregious mistakes, but I want you to thoroughly copy-edit your prose. You will submit this separately to the separately listed submission drop box. ### Overall structure Below, you can find a sample basic layout and structure. Some students have different or difficult-to-characterize projects. If this format is a poor fit to your project, please let me know and we can discuss how to adjust to your situation. You needn't follow this order precisely, but these are the components I'll look for. You can also look to some of the exemplary pieces to which we've linked, either publications or prior semesters' students' submissions. Here is that template you should try to follow: ## Sample Layout ### Intro Remind me, specifically and in English, the aim of your project, either what you proved, or the object of your encoding. Briefly and broadly describe here the overall structure of the work itself including any auxiliary infrastructure (lemmas, tools, etc) that you had to write, or If you needed to use any auxiliary libraries (e.g. sets) or programs you had to find, install, or use (e.g. miniSAT), mention those details here. If there were any similar papers, solutions to analogous problems, proofs or proof structures that you referenced or found insightful (or that would benefit the reader to know about) reference them here. ### Background See the "Approach" criteria above. Give an overview of the logical structure of the main thrust of your overall work. Signpost the steps you'll be discussing in future sections. Describe and recall the overall context: For proofs, these would be the data type(s) you're discussing, if they're non-trivial or interesting choices and the function(s) you're working with, both the purpose and any interesting "twists" we made along the way. For reductions, these would likely be the novel pieces that made what you were doing difficult, or beyond what had already been done. You should include here also a link to the runnable source code for your solution. For ACL2 folk, if you needed to use or to find any additional complicated ACL2 machinery beyond defthm and implications (additional guard information, hints non-rewrite rules) mention those details here. For SAT folk, if you used something besides strictly SAT, remind me of that piece. ### Walkthrough (You may divide this over several sections) See the "Methodology" metrics above. Break the overall structure of your work down into bite-sized chunks. Bring the reader through your result. E.g, to prove a clearly stated theorem, often times you have to also prove more opaque lemmas along the way; if so, interpret your less clear lemmas, and what they say in English, and why they are necessary for the ultimate claim. * Did any of your lemmas require sub-lemmata? ** Which and what? * For which subsequent claims were each of the lemmata you show necessary? Mutatis mutandis for problems with encodings. ### Results See the "Metrics" criteria above. ### Personal Progress Whereas you write the other parts from the perspective of someone who has now learned the answer presenting it in the best light and easiest form for a reader, here you can discuss, as a narrative how it *actually* came about. The rest of this document should focus on the problem and solution as its best explained, but this part can be the problem as you historically solved it. - Where did you get stuck and need to backtrack or regroup? - Where did you need to step back and take a different approach to prove your ultimate claim? - Did you try to prove any claims that were in fact invalid? - Were there any simpler claims you proved first, as a "trial run"? ### Summary See the "Summary" criteria above. ### References & Appendix - Formatted - Easy to follow back to original sources - Referenced in context.