One thing on some people's minds when you talk about shoes-off in homes is stubbing of toes. Obviously, if you are worried about stubbing your toes, you can wear slippers and if you let guests know about your policy, they can bring their own.
Believe it or not, your feet do get tougher if you let them. As I am barefoot at home and nearly always in flip flops when outside, my toes get stubbed a lot. But it happens so often that it barely hurts that much any more.