How to speed up processing / identify bottlenecks

[ericriff]

Hi All

Our CDash instance started considerably lagging behind recently and we're not sure how to debug this.

We use CDash version v3.0.3-14-gef36cf0d with a remote mysql db over aws rds.

Our ctest logs are big-ish (~1GB) but this hasn't been a problem before.

The monitor page reports

[projectName], 33.76 hours behind
...
Pending Submissions
project.name, project.id, count of pending queued submissions

[projectName], 2, 2364

The instances that host CDash and the remote db show very little resources utilization, so we're not sure why CDash is lagging behind so much.

Thanks,

Eric.