OpenUserJS.org Migrate the scripts in S3 to use `script.

Currently we are unable to upgrade the aws-sdk dependency because "something" changed that makes getObject to fail.

This has been explained in detail at aws/aws-sdk-js#430 . With all the test deploys done trying to figure out what the steps to reproduce are it appears that normalizing the casing when we save to the DB is an option.

Looking for more options related to advancing the goal of updating the dep... e.g. perhaps we could validate the current aReq.params.username against the real username (name) that is typically in the Raw JSON Data for an account. e.g. a lookup to correct the casing around here... but I suspect this is going to need to be a DB migration with a BLOCKING label until it is finished.

See also:

https://github.com/aws/aws-sdk-js/issues/397#issuecomment-60807108 error trap suggestion for loosely related
https://github.com/OpenUserJs/OpenUserJS.org/blob/71ef808762ccf14a048c8a639d6d620e4e4e91e4/controllers/script.js#L350 Probably related and should be fixed

Test points:

https://openuserjs.org/scripts/Marti/oujs_-Meta_View/source vs https://openuserjs.org/scripts/marti/oujs-_Meta_View/source e.g. manual url entry in the addy bar
https://openuserjs.org/scripts/JRI/Geocaching_Map_Enhancements/source Since jri is currently in the DB not JRI
https://openuserjs.org/users/jri (http://localhost:8080/users/jri with local pro)

Dec 06 '14 15:12 Martii

With #487 we have another option of inline live migration... that patch currently fixes the net timeout in dev when selecting some other script source with fakeS3 and it usually just spins the browser spinner for an incredibly long time since our local dev DB doesn't usually have that source.

We can either correct the casing to what the user entered, if applicable, (e.g. a quick user search in the DB and if it is a hit compare it to the username portion generated from installName and migrate) from what we store in the Raw JSON Data or we can just lowercase migrate everything username wise... It is probably not recommended that we should alter the scriptname value though in casing with the Key.

Updated the above for some test points and other linkage.

Dec 06 '14 22:12 Martii

Normally you wouldn't base the filename/keys based on the username/scriptname. Create an id for each source code and store it in the Script model. If we wanted to future proof ourselves, we wouldn't use the Script._id either as that means only 1 source code per Script. But since we're not versioning it atm, we could just use that.

We have to fetch the Script model first before we doing any AWS stuff anyways.

Edit: Also noticed this when I tried to refactor the scriptEditSourceCodePage that never got finished. Most of the callbacks need to follow the callback(err, ...) pattern so we can chain error up to the route handlers.

Dec 06 '14 22:12 Zren

I agree with @Zren. We should use Script._id instead since we have to do a lookup anyway.

Dec 06 '14 23:12 sizzlemctwizzle

Create an id for each source code and store it in the Script model.

Yah I'm not entirely sure how lower casing a Unicode character would affect things so a unique id would probably be best. EDIT: Searching will need to be retested for this as well.

Dec 07 '14 00:12 Martii

Just a FYI here's a short snippet from recent production log with the isPro console message I added in to check for this on pro e.g. the version of aws-sdk we are currently using hides/mishandles the error and possibly returns an empty payload as one of the maintainers stated... jri's script does show source and does install.

[12/08 23:36:21 EST][err] S3 Key Not Found jri/Geocaching_Map_Enhancements.user.js
[12/08 23:40:33 EST][out] Group(imgur) Rating Updated
[12/08 23:45:24 EST][out] Group(deviantART) Rating Updated
[12/08 23:48:10 EST][err] S3 Key Not Found mienaikage/OkChoices.user.js
[12/08 23:48:53 EST][out] Group(Subtitles) Rating Updated
[12/08 23:49:42 EST][out] Group(Steam) Rating Updated
[12/08 23:49:42 EST][out] Group(dota2lounge) Rating Updated
[12/08 23:49:47 EST][out] Group(download) Rating Updated
[12/08 23:51:02 EST][out] GitHub client authenticated
[12/08 23:51:10 EST][out] Group(bookmarks) Rating Updated

Hmmm their clock appears to be a bit skewed too.

Dec 09 '14 04:12 Martii

Hmmm... retested this randomly with [email protected] and doesn't appear to crash dev nor local pro anymore... not sure on actual production. All prior test points appear to be okay (without crashing local pro). :\

@sizzlemctwizzle Any objections to trying it on production? I'll give this a day or so for any responses.

Apr 27 '15 07:04 Martii

I have no problem with a temporary fix, but we definitely need to migrate the scripts in S3 to use script._id instead of the path value somewhere down the line. It's a simple migration but I'm always afraid I'm going to break something when I mess with S3. I spent hours trying to get everything right when I removed namespaces.

Apr 27 '15 07:04 sizzlemctwizzle

Of course the issue needs to stay open... just thought I could get the dep updated a little bit. I am on a slightly newer node.js though which is why it would need actual production testing... just got done with local pro and seems okay not to crash here.

Migrate the scripts in S3 to use `script._id` instead of the path value