Oh this happens to me too! So far i've noticed when i'm on youtube and when i'm using a headset, one of the earpiece loses the sound and then it comes back. I tried the headset on my note 5 and it works fine, so i know for sure it's not the headset that's having a problem. I haven't attempted yet to return this to the store because they will not outrightly offer a replacement and a repair could only worsen the phone's condition! Not like with apple, they'll replace it right away!
@user1DyYMHnJ3G does this happen in safe mode?
Give this link a try.