pulsar-manager icon indicating copy to clipboard operation
pulsar-manager copied to clipboard

Create Index on topics_stats to fix tenants tab latency

Open sourabhaggrawal opened this issue 4 years ago • 7 comments

Fixes #419

(or if this PR is one task of a github issue, please add Master Issue: #<xyz> to link to the master issue.)

Master Issue: #

Motivation

Tenants tab in Pulsar Manager is slow and takes forever to load sometime.

Modifications

We are using Postgres for data storage and noticed that while loading tenants page it is stuck at one query and takes forever to load. Tenants page request load data from topics_stats table and load only one topic status with latest timestamp. The query used here https://github.com/sourabhaggrawal/pulsar-manager/blob/5343f09f381729ef6798ddf1bffccd55aaff04b8/src/main/java/com/manager/pulsar/mapper/TopicsStatsMapper.java#L37 fetch only single record by ordering on the timestamp column everytime, with too many records this query takes too much time .

Added the index to get result faster from the query.

Verifying this change

  • [x] Make sure that the change passes the ./gradlew build checks.

sourabhaggrawal avatar Sep 10 '21 07:09 sourabhaggrawal

Hello @tuteng Please take a look.

sourabhaggrawal avatar Sep 15 '21 11:09 sourabhaggrawal

Can you handle the ci exception? @sourabhaggrawal

tuteng avatar Nov 25 '21 03:11 tuteng

Hi @tuteng the change is in sql file to create an index and should not impact any existing test case. Below is the test case which is failing org.apache.pulsar.manager.service.TopicsServiceImplTest > getTopicsStatsImplTest FAILED java.lang.AssertionError at TopicsServiceImplTest.java:125

can you please help me understand how can I fix this or rerun the build?

sourabhaggrawal avatar Dec 08 '21 14:12 sourabhaggrawal

I don't seem to be able to re-run the build either, I think you could do an empty commit and push to this branch again and that might re-trigger the build. @sourabhaggrawal

git commit --allow-empty
git push origin sourabhaggrawal:master

tuteng avatar Dec 08 '21 15:12 tuteng

Hi @tuteng Thanks for the suggestion and looks like the build is triggered. I see 1 workflow is waiting for an approval. Can you please help me get it resolved too ?

sourabhaggrawal avatar Dec 08 '21 16:12 sourabhaggrawal

@tuteng @eolivelli Can you please approve the test workflow run?

shiv4289 avatar Dec 09 '21 04:12 shiv4289

Hello @eolivelli @tuteng, please approve the test workflow run.

sourabhaggrawal avatar Dec 10 '21 06:12 sourabhaggrawal