Reduce the size of feedback files

[lgpage]

Is there any way to reduce the size of feedback files? This is what I am sitting with for example: Autograded directory: 47 MB Feedback directory: 912 MB

[jhamrick]

Oof, wow.

I am not really sure what is taking up so much space, but I think it is including a bunch of javascript and css in the header of the HTML files, which might be what is taking up so much space? It think it is possible to generate the HTML files without that (i.e. just link it in rather than including it verbosely), so that's one option which would divide the needed space by N, where N is the number of notebooks per assignment.

[lgpage]

Hmm, yeah, roughly 11800 of 12400 lines (95%) is css

[jhamrick]

One way that we might be able to address this is to include the CSS via CDN rather than including it in the file itself: http://getbootstrap.com/getting-started/#download-cdn

The downside to doing this is that students would have to be online for the CSS to load properly, but this probably isn't a huge issue.

[jhamrick]

Hmm, no, that doesn't really work--I think the notebook must be using a customized version of bootstrap. :(

[lgpage]

@jhamrick I see two options for this:

1. As you said link to CDN's in the template files instead of in-lining resources.
   • I am not a fan of this at all, because of the need for the students to be online to review feedback (in my case I have a number of students that don't have internet access from home)
   • If this option is implemented I think there should be a way for the user to select either in-lining or linking
2. Zip up the feedback files
   • This is the method I opted for in the end. On my side, for all assignment feedback files:
     • Un-zipped feedback - 5.9 GB
     • Zipped feedback - 1 GB
   • This overlaps a bit with #760
   • From a manual uploading (distribution) point of view I found it easier to zip the files into the following structure:

   - upload-feedback
     - assignment_id
       - student_id.zip
   

   • Here is the Python script I used for zipping the feedback

import os
import sys
import zipfile

from zipfile import ZIP_DEFLATED

def makedirs_if_missing(path):
    if not os.path.isdir(path):
        os.makedirs(path)

def make_zip(root, dest, assignment_id):
    with zipfile.ZipFile(dest, 'w', ZIP_DEFLATED) as zfh:
        for path, _, files in os.walk(root):
            for f in files:
                filename = os.path.join(path, f)
                arcname = os.path.join(
                    assignment_id, os.path.relpath(filename, root))
                zfh.write(filename, arcname)

def main(assignment_id):
    root = os.getcwd()
    feedback_dir = os.path.join(root, 'feedback')
    dest_dir = os.path.join(root, 'uploaded', 'feedback', assignment_id)
    makedirs_if_missing(dest_dir)

    for student in sorted(os.listdir(feedback_dir)):
        assignment_dir = os.path.join(feedback_dir, student, assignment_id)
        if os.path.exists(assignment_dir):
            print("Zipping student feedback: feedback/{}/{}".format(
                student, assignment_id))
            dest = os.path.join(dest_dir, "{}.zip".format(student))
            make_zip(assignment_dir, dest, assignment_id)

if __name__ == "__main__":
    extra_args = sys.argv[1:]
    if len(extra_args) > 1:
        print("Only one argument (the assignment id) may be specified")
        sys.exit(1)
    elif len(extra_args) == 0:
        print("An assignment id must be specified")
        sys.exit(1)
    else:
        main(extra_args[0])

[jhamrick]

Nice!

It also maybe overlaps a bit with #436 . It's good to have the feedback be regular files initially, I'd think, so instructors can inspect them. But maybe in actually releasing feedback we could do that by first (1) zipping it up and (2) then releasing it.

Something else that will help, as I mentioned in an earlier comment, is to generate a standalone CSS file (one per student per assignment) that gets included in the HTML notebooks, so that will at least divide the needed amount of space by N.