I doubt that would be where the noise is coming from. My thinking was if that the bad part, it would be some sort of grinding noise rather than the knocking noise because it rotates. Your noise is from going over bumps cause by impact pressure.
Not sure if there should have a little plays (not a lot) in them? All four on my two FXes has a little play in them and I don't hear any kind of noise on mine. You have a good point that your driver side doesn't have any play in it though, idk. Not sure my friend.
As for hours to replace it, it should be 2.0 to 2.5 hours. I think it still safe to drive.