promscale icon indicating copy to clipboard operation
promscale copied to clipboard

Ease locking to speedup ingestion on startup

Open niksajakovljevic opened this issue 3 years ago • 1 comments

Due to a lot of locking while inserting metadata on series we tend to have a slow start. It can take up to 15-20 minutes to get to full speed when ingesting heavily (eg. 500K/s). Trying to ease some locking.

SQL function get_or_create_series_id_for_label_array was taking unnecessary ACCESS SHARE lock on metric data table. The label keys are actually created in another function so no need to take this lock.

SQL function get_new_pos_for_key was taking SHARE UPDATE EXCLUSIVE lock on metric data table. Using advisory lock here should enable writes to metric data table while we safely get new key position.

NOTE: This is work in progress as I believe there are few more places where we can improve locking.

niksajakovljevic avatar Jan 06 '22 12:01 niksajakovljevic

@niksajakovljevic can this PR be closed?

cevian avatar Mar 30 '22 01:03 cevian

Since SQL logic was moved to extension repository this PR can be closed.

niksajakovljevic avatar Nov 29 '22 09:11 niksajakovljevic