Scripture calls them men of renown, men who did great feats, this is what made them giants among men. They were not literally giants, they were seen as great among men and had distracted man's attention from God.
God loves those who believe and those who do not and the same goes for me, you have no choice in this matter. That puts the matter of total free will to rest.