Is it possible to use --s3.path or --outputFolder while keeping the timestamps subfolder?

[markmssd]

Use case would be when testing the url www.mydomain.com and another url, www.mydomain.com/some/path/with/same/domain.html. So both reports will be merging and overriding each other. e.g.

www.mydomain.com/2017-01-17-13-00-00/<the report>
www.mydomain.com/2017-02-17-13-00-00/<the report>
...

So I'd like to give them each a custom name, while also keeping the timestamp:

custom-name-one/2017-01-17-13-00-00/<the report>
custom-name-two/2017-02-17-13-00-00/<the report>
...

From what I can see here: https://github.com/sitespeedio/sitespeed.io/blob/master/lib/core/resultsStorage/index.js#L24 it doesn't seem possible.

Am I missing something? Thanks!

[soulgalore]

Hi @markmssd thanks for the report, 'm not sure, can you show me how you run, so I understand how they are overriding each other? Best Peter

[markmssd]

Sure,

I am passing in those arguments for a sitespeed report:

https://www.mydomain.com
-b
chrome
--speedIndex
--html.showAllWaterfallSummary

and another sitespeed report with those ones:

https://www.mydomain.com/some/path/with/same/domain.html
-b
chrome
--speedIndex
--html.showAllWaterfallSummary

At the end, they are both being saved in www.mydomain.com/2017-01-17-13-00-00, because it is based on the hostname and not the full url (https://github.com/sitespeedio/sitespeed.io/blob/master/lib/core/resultsStorage/index.js#L11)

What I would need is to be able to specify a --s3.path option, while also keeping the timestamp subfolder: CUSTOM_FOLDER_FOR_FIRST_URL/2017-01-17-13-00-00 CUSTOM_FOLDER_FOR_SECOND_URL/2017-01-17-13-00-00

[markmssd]

My suggestion would be to recognize the placeholder %timestamp%, so we could pass e.g. --s3.path CUSTOM_FOLDER/%timestamp%, and inside of this IF block on line 24, have it as

if (outputFolder) {
    resultsSubFolders.push(path.basename(outputFolder.replace('%timestamp%', ''));

    // check if user wants timestamp subfolders
    if (outputFolder.includes('%timestamp%')) {
        resultsSubFolders.push(timestamp.format('YYYY-MM-DD-HH-mm-ss'))
    }

    storageBasePath = path.resolve('sitespeed-result', ...resultsSubFolders);
} else { ... }

[soulgalore]

So you mean they collide on S3? Or are you running them on the same time on the same server? For s3 I usually setup different buckets for different servers. I never run multiple tests on the same machine at the same time so that they don't disturb each other.

@tobli do you have any input?

[markmssd]

Ah! we have them all in the same bucket sitespeed-reports, to keep things organized. Yes they collide on S3 when getting saved...

[lbod]

I have the same issue but I did appear to have it working at one point (where it was creating host/date folders for each report) but that was using v5, so I presumed it's just my configuration with v6. We start with something like:

  /start.sh,urls.txt,--config,config.json,--plugins.disable,screenshot,--firstParty,www.somedomain.com,--html.topListSize,120,--s3.key,***********,--s3.secret,*******,--s3.bucketname,*******,--s3.removeLocalResult,true,--resultBaseURL,https://someurl

However the reports are all being written to:

bucketname/sitespeed-results

i.e. no date or host, so every report no matter host or date is going to the same place. (We're on the very latest stable docker images, not the beta ones)

[lbod]

Hi @soulgalore and @tobli - just looking at the code now and I'm a bit confused by the expected usage.

So for S3 runs, the baseDir is uploaded to the S3 bucket i.e. /some/directory/sitespeed-result/www.bbc.co.uk/2018-02-06-11-15-01 which is just the contents of that particular run. We have been using the date and domain folders as some basic search/calendar view to navigate reports but not on s3 yet. The only way I can achieve the same kind of folder structure is by passing in a date to the s3.path.

However it strikes me this wasn't your intended usage?

[chosroes]

An option like this would be better:

if (outputFolder) {
  if (outputFolderTimestamp) {
      resultsSubFolders.push(path.basename(outputFolder), timestamp.format(customTimestampFormatConfig));
  } else {
      resultsSubFolders.push(path.basename(outputFolder));
  }
  
  storageBasePath = path.resolve(outputFolder);
} else {
  resultsSubFolders.push(
    options.slug || getDomainOrFileName(input),
    timestamp.format(customTimestampFormatConfig)
  );

  storageBasePath = path.resolve('sitespeed-result', ...resultsSubFolders);
}