HamQTH upload is doing too much work

[oliversturm]

I have uploads configured for HamQTH and the option checked to upload immediately. I also upload and download information from eQSL and LoTW every now and then (no automatic "immediate" option for these, is there?).

I have noticed that the HamQTH upload goes crazy as soon as there are any updates received from eQSL or LoTW (theoretically it may just be one of the two, not both - I haven't checked that). For instance, I may receive ~10 new confirmations from eQSL and LoTW, but the next time I log a contact I can see in the "Status of log upload" window (and the terminal I started cqrlog from) that the HamQTH upload seems to go through every one of my log entries and re-upload it. Of course I can't tell exactly that it's every one of my entries, but it's certainly a very long list - much longer than the ~10 items I would expect - and it goes on a very long time as well.

Is this somehow on purpose? I can't be the only person using these services in combination?

[phl0]

That is working as designed. As you download eQSL or LotW data all the entries in your cqrlog DB get updated and the QSL received columns are set. This triggers a mySQL trigger that marks the corresponding QSOs as to be re-uploaded as now the QSOs are confirmed. The re-update of HamQTH is not triggered after eQSL or LotW download but is then carried out as soon as you trigger the upload or log a QSO with auto-upload enabled.

[oliversturm]

Yes... understood. I just checked the exported backup files and came to the same conclusion. I also use the function "mark all as uploaded to eQSL/LotW" because I have found that at least LotW got confused about items that were previously included in an upload (if I did not "mark as uploaded" in between). I'm not sure if this is the correct way of handling this - it's quite fiddly to do it all manually anyway. But this "mark" function updates timestamps in the records, as far as I can tell from the export.

However, here's what I don't understand. I have logged in cqrlog for a couple of weeks or so, I have ~160 QSOs in the log, and this HamQTH upload is already wildly inefficient - when a new QSO is logged, it takes ages for the upload process to run through all the updates and finally get round to logging the new QSO. I'm using a reasonably fast 4G internet connection. Surely some people must have many thousands or tens of thousands of log entries. How is this supposed to work?

[oliversturm]

Re-reading your message, I'm not sure I understood this part correctly:

As you download eQSL or LotW data all the entries in your cqrlog DB get updated and the QSL received columns are set. This triggers a mySQL trigger that marks the corresponding QSOs as to be re-uploaded as now the QSOs are confirmed.

All the entries get updated? Shouldn't only those entries get updated that actually received updates? As far as I can see, the number of entries getting re-uploaded is much larger than the number of entries that actually had updates recently.

I suspect that some other field(s) in all records are being updated and thereby cause the re-upload. Perhaps my workflow is at fault somehow. Here's what I normally do:

• Log a few items, which are uploaded to HamQTH live
• Done for the day, I click the "upload to eQSL" and "upload to LotW" buttons
• I also click the "download from eQSL" and "download from LotW" buttons
• I "mark all as uploaded" for eQSL and LotW
• Close cqrlog
• Next day, I open it again - the next time I log an item, HamQTH upload re-uploads (apparently) everything

[phl0]

Sorry no. Not all entries but only the QSOs that you downloaded QSL info for.

[oliversturm]

I see, okay. In that case I guess something else must trigger the re-upload for me. Perhaps I'm not supposed to "mark all as uploaded" on a regular basis? I would have thought that this function would only update timestamps for those items that hadn't been uploaded previously. I didn't start out using it either, but after a few iterations I kept getting warnings from the LotW upload about items that were included a second time, and I found that "mark all as uploaded" fixed these issues.

[phl0]

Are really all your QSOs marked to be re-uploaded? I guess the re-upload fails and due to that these QSO get re-uploaded again and again?

[oliversturm]

I can't be sure that it's all QSOs - hard to tell how long the log really is. But it certainly looks like it could be ~160 entries scrolling by. But no, nothing fails - I can see the detailed output on the terminal and there are never any errors.

...
SyncUpdate:
SyncMsg   :HamQTH: Deleting original LB8OH
SyncUpdate:OK
SyncMsg   :HamQTH: 
Item:HamQTH: Deleting original LB8OH ... OK
SyncUpdate:
SyncMsg   :HamQTH: Uploading updated LB8OH
SyncUpdate:OK
SyncMsg   :HamQTH: 
Item:HamQTH: Uploading updated LB8OH ... OK
SyncUpdate:
SyncMsg   :HamQTH: Deleting original RA7KV
SyncUpdate:OK
SyncMsg   :HamQTH: 
Item:HamQTH: Deleting original RA7KV ... OK
SyncUpdate:
SyncMsg   :HamQTH: Uploading updated RA7KV
SyncUpdate:OK
SyncMsg   :HamQTH: 
Item:HamQTH: Uploading updated RA7KV ... OK
SyncUpdate:
SyncMsg   :HamQTH: Deleting original JA3FYC
SyncUpdate:OK
SyncMsg   :HamQTH: 
Item:HamQTH: Deleting original JA3FYC ... OK
SyncUpdate:
SyncMsg   :HamQTH: Uploading updated JA3FYC
SyncUpdate:OK
SyncMsg   :HamQTH: 
Item:HamQTH: Uploading updated JA3FYC ... OK
...

[dl7oap]

Hi, the hamqth api also takes the status of send/received LOTW and EQSL QSL confirmations via ADIF-fields (https://www.hamqth.com/developers.php#log_upload).

Thats why all qsos with a change ( LOTW_QSL_SENT, LOTW_QSL_RCVD, EQSL_QSL_SENT, EQSL_QSL_RCVD) get another upload. i personally, too, don't like these many updates (i have this observation, too).

maybe we should add in preferences an option, that the user can choose to upload only initial qso, but QSL-Informationchanges should not lead to an update. this todo is on my personal whish list, but something which i would programm during the long cold dark period in winter and not in summer pause :-)

[oliversturm]

Well, this would not be a big problem if only the QSO records that receive updates from eQSL or LotW were updated in HamQTH. But it seems clear to me that other records are also uploaded - so I have to assume they are changed locally for some reason. Now all I need is to figure out that reason and then I can consider what to do about it :)

[dl7oap]

Hi, just found that the option is already implemented: File > Preferences > Online log upload and down there check the last option "Ignore changes caused by LoTW/eQSL upload or download".

overlooked for years now (._.);

[oliversturm]

Sorry, missed this comment. Just to clarify one more time - as far as I can see, cqrlog uploads more changes to HamQTH than those that can be explained by actual changes due to LotW/eQSL downloads. I counted one time and saw about 80 uploads to HamQTH, after recent downloads from LotW and eQSL brought ~15 new confirmations. At that time I had ~200 QSO records, so the selection of 80 for re-upload seemed random.

This problem seems to be connected to the menu function "mark all as uploaded". I had developed the habit of using this function, as described before, because I had trouble before when the state somehow got out of sync with LotW. I have since stopped using this function, and now cqrlog's upload behavior seems more predictable.

The "mark all as uploaded" function seems to do something strange here. When I used it, I was really expecting it to "mark all as uploaded that had not been marked as uploaded before". I would have also accepted if it did "mark ALL as uploaded, never mind whether it's been uploaded 20 times before" - although this would be quite useless in my eyes. But as far as I can tell, what the function really does is "mark recently changed QSOs as uploaded, plus a random selection of 65 others".

I understand now that this function does not need to be part of my daily workflow with cqrlog, and as a result the upload behavior is not getting incrementally worse for me anymore. At the same time I still believe that the function is misbehaving somehow, and I wonder if I'll need it one day when I have thousands of QSOs in my log and then I'll wait a couple days for the uploads to run through.