How do I prevent Listen in Background from recording the Speech of the Computer itself

[msalmonw]

I've developed a Voice Assistant with active background listening at all times. But, whenever the assistant responds to any user query, the speech recognition picks up the Assistant's Speech as well, which is ruining my conversation flow for the assistant.

PS: I don't want to stop background listening for the time when the Assistant is talking and turn it back on afterwards.

Any help would really be appreciated. Thanks