Requests getting lost in workbox queue

[adaaaam7777]

Library Affected: workbox-background-sync

Browser & Platform: Google Chrome latest for Android (only one used)

Issue or Feature Request Description: I have disappearing requests from workbxox queue and can't for the life of me figure out what is happening. Here is the relevant code:

const queue = new workbox.backgroundSync.Queue('orderQueue', {
	maxRetentionTime: 10 * 24 * 60,
	onSync: async ({ queue }) => {
		try {
		        await mutex.lock();

			let entry;
			const queueSize = await queue.size();
			if (queueSize !== 0) {
				try {
					const queueContents = await queue.getAll();
					const queueStateAtSend = await Promise.all(queueContents.map(req => parseReadableStreamToJson(req.request.clone().body)));
					channel.postMessage({
						type: 'logQueueResendAttempt',
						queueStateAtAttempt: JSON.stringify(queueStateAtSend)
					});
				} catch (e) {
					console.log(e);
				}
			}

			while (entry = await queue.shiftRequest()) {
				let requestBody;
				try {
					const fetchResponse = await fetch(entry.request.clone());
					if (!fetchResponse.ok) {
						channel.postMessage({ type: 'errorReturnedFromServer' , requestBody: JSON.stringify(fetchResponse) });
						throw new Error();
					}

					const queueRemainingSize = await queue.size();
					if (queueRemainingSize === 0) {
						channel.postMessage({ type: 'successfullyFinishedReplaying' });
					}
				} catch (error) {
					await queue.unshiftRequest(entry);
					channel.postMessage({ type: 'failedToFinishReplaying' });
					throw error;
				}
			}
		} catch (error) {
			console.error('An error occurred during sync:', error);
			throw error;
		} finally {
			mutex.unlock();
		}
	},
});

What's happening is, a user places orders throughout the day with no internet. They get placed in the 'orderQueue'. When they finally acquire wifi, Sync event is received and all the orders get sent one by one. However the first 1-2 requests are sometimes disappearing. Always the first 1-2 and not every day. I added a mutex, because for some reason there were multiple sync events in rapid succession when I inspected the logs. Did not solve the issue unfortunately. I suspect it either has to do with weak network connection or some underlying workbox mechanic I am not aware of. Or a capital mistake in my code? There is absolutely zero trace of the missing requests apart from that they are logged in 'logQueueResendAttempt', so they are tried to be resent. It is not logged as network or other error and the server claims it did not receive any of these missing ones. An added difficulty is that I can not reproduce it on my own tablet or desktop Chrome, can only inspect logs I get from the users' devices. What is happening here? Thanks for the help!

[parkercornbrooks]

This was happening to us to. We found that multiple sync events can occur - a bad connection may come on and offline multiple times, for example, each one firing a sync event. When this happens, the onSync function will fire each time. In this case, the first call would pull the first request off the queue (queue.shiftRequest()). The second call would also pull the "first" request off the queue, so two requests total would be pulled off the queue. In cases of a bad connection, when the fetch attempts fail, the attempt to add requests back to the queue is the source of the error. The queue.unshiftRequest() function does a rudimentary check of the queue to find the lowest id, and then attempts to add the entry at id = lowestId-1 If the timing is just right, both onSync calls are calculating the same id, both attempt to unshiftRequest() to the same id, and the second will throw a DOMException: Key already exists in the object store. - essentially deleting the request from the queue. The way you have your try/catch set up, you are re-throwing the original fetch failed error and this one would get hidden.

One solution may be to use a pushRequest() instead of unshiftRequest(). It uses the primary key generator which should not cause the same id collisions. It adds the request to the end of the queue, so it is only a viable solution if you do not care about the ordering of your re-tried requests.

Another option is to use a flag that allows only one call to onSync at a time. We are in the process of implementing this solution.

[adaaaam7777]

Thanks for the reply. For now I just disabled automatic resend and allowing manual only. Howevever I have other simple queues with the same issue and also duplication happens as well, probably caused by the same problem.

[tomayac]

Hi there,

Workbox is moving to a new engineering team within Google. As part of this move, we're declaring a partial bug bankruptcy to allow the new team to start fresh. We realize this isn't optimal, but realistically, this is the only way we see it working. For transparency, here're the criteria we applied:

• For PRs, we closed everything that doesn't target the current v7 branch.
• For Issues, we closed everything that was created before 2024.

Thanks, and we hope for your understanding! The Workbox team