Thank you for reaching out. I would recommend moving your contacts app to the secure folder. Give this link a try for more information about secure folders: https://www.samsung.com/us/support/answer/ANS00081123/
Be sure to click " ✓ Accept as Solution" when you find an answer that works for you.
Please note, notification emails are a DO NOT REPLY address, you must log-in on the community page in order to respond.
Um. Contacts stored locally on the phone are technically only your contacts. If you're referring to the Google side of it where everything backs it up there is a way to deny the backup option. I do believe it is in the Google app and under account. That's as much as I know.
Not sure if this helps, but have you tried using the secure folder feature and store your contacts there. You will always need to log in to view your contacts but it would be visible only to you. Cheers