Google -> My Account -> Go to Activity Controls -> Voice & Audio Activity -> Manage Activity
If you have it turned off (PAUSE), then you should have nothing listed (I hope). If you turn it on, you'll start accumulating data there and you can delete stuff. What happens is it normally wants to note activity of you using the "OK Google" service, but it will log a lot of false positives.
I find bits of conversations with wife, meetings at work, other people at the bar, and more. Theoretically I had to agree to turn this on, but I don't remember doing it, I thought it worked on my phone out-of-the-box, but maybe I did turn it on and didn't realize the implication?
At this point I have to trust Google that when I delete things that they actually delete all copies of it.