Develop new solution for usage monitoring

[ljwh2]

Following the recent round of reporting, document any limitations of current usage monitoring and propose improvements that can be implemented before the next reporting period beginning 1st July.

[ljwh2]

we require a script to automate the process that is done at the moment -pulling info from log files in different places. To be run monthly to collate information per month.

[sprintell]

There might be need to query ebi meter via Kibana Rest API feature, if thats going to take more time than expected, that step can be kept manual at the moment.

[ala-ebi]

• [x] Change all bsub to sbatch/srun
• [x] Copy (this month’s) logs from 4 prod VMs into one work directory (fetch_logs.sh), needs updating as currently retrieves all files
• [x] Call parse_logs.sh (internally calls analyse_logs.sh) —> generates stats files for the following:
  • [x] Downloads
  • [x] Rest API
  • [x] Search terms
  • [x] Number of searches? (Website sheet column G) (search_DATE_stats file)
• [x] Remove date filter in parse_logs.sh/analyse_logs.sh as not needed since we only retrieve the files for the specified time period, no need to filter
• [ ] Investigate possibility of getting FTP logs
• [x] Merge stats files into one - deprioritised
• [x] Email file to gwas-dev/gwas-curator

[ljwh2]

@ala-ebi to fin location on ftp to store files ,and set up cron job to run monthly

[sajo-ebi]

@ala to work in this sprint

[ala-ebi]

added disk storage and entry in scrontab to run on the first of each month, will give stats for the previous month.

updated documentation https://www.ebi.ac.uk/seqdb/confluence/display/GOCI/GWAS+Catalog+monthly+usage+stats

[ljwh2]

To be verified after 1st Oct