Big jsons uploaded to WMArchive

[todor-ivanov]

Impact of the bug WMAgent, WMArchive

Describe the bug While debugging missing monitoring information from ES the Monit team has discovered some quite big json files uploaded to WMarchive and ES [1]. The document's size has been of the order of 2MBs. The documents themselves seem to be sane. Their origin are few workflows which tend to create huge LFNArrays for the input files for the job itself. This is a behavior on which we need more feedback from other groups like P&R and PDMv in order to estimate the expected future frequency of those.

In the meantime there was a temporary solution applied by Monit side so that we may continue to use the monitoring properly (all is well explained in the relevant ticket [1]), but we cannot rely on that in the long term. So this ticket is expected to be a follow up on the decisions taken on our side to revisit the information which we upload to WMArchive and ES.

[1] https://cern.service-now.com/service-portal?id=ticket&table=incident&n=INC2961279

How to reproduce it N/A In order to reproduce those huge documents we need some feedback so we can relate to a speciffic type of workflows/campaigns.

Expected behavior Documents uploaded to WMArchive and ES to be in the order of kilobytes rather than megabytes.

Additional context and error message N/A