Copy in

[KesterTan]

Attempts to fix the copy-in issue that we're still facing even after retries.

1. Captures invalid inputfiles structure
2. Adds -p flag and permissions when creating autolab directory. This ensures that the specified directory and any necessary parent directories are created. If the directory already exists, the command does not raise an error, it would originally throw an error.
3. Added thread pool to limit scp commands so that instance is not overloaded
4. Added log statements with job id for more in-depth logging.

Ran a thousand jobs on https://dev.autolabproject.com/courses/test-course/jobs?id=500 without failure.