I am not sure if anyone else had the same problem like me. The face recognition feature on s10 is highly unsecure. My brother was able to unlock the phone with his face. We both share a few similar facial features but that doesn't mean he should be able to unlock my phone. I was shocked when he did it. Luckily I was right by his side when this happened. I tried to delete the face data and added new face image and retried with his face. To my surprise, the phone unlocked again. One of my friends was able to unlock the phone with his face as well. This is highly unsecure and I want Samsung to have a look at this immediately.
Actually Samsung mentioned about the security issue. As the face unlock function is just using normal cameras you can't expect it has the same security level as iPhone's face ID. There is no way to improve besides adding more sensor.
So the only way to be secure, is not using it.