I just had this fixed by my carrier. It's not an app issue or the need to download a new app. You have to request the carrier to allow you to use visual voicemail. I asked my carrier to figure it out and they fixed it in 5 min. The preinstalled samsung voicemail app has visual voicemail in it. So when they unlocked it on my phone suddenly it populated with voicemail and I never downloaded a new app.
If your carrier doesn't support visual though, then you may have to download an app, I'm unsure. This is how I fixed it on my phone.
My memory may be failing me, but I thought past Samsung devices had a built in visual voicemail but I can't seem to enable it on my Note 10 plus.
I have an unlocked S10 and couldn't get the icon. I chatted with AT&T technical support and I think they helped me resolve the issue. I don't know if it was them, because i was messing with a lot of things on my phone, but a notification came on my screen that a new download for the Samsung visual voicemail was available, so I downloaded it. Now i have the app and moved it to my home screen. Works great.