It of course depends on what you mean by "agnostic" and what you mean by "atheist". A baby has no belief in God, since it does not have the mental capacity to hold beliefs yet. It is therefore as much an atheist as a rock. The same goes for being born agnostic, only I'd hold that agnosticism is a positive belief, whilst atheism is more of a lack or "passive" non-belief.
In simple terms, it is not possible for a baby to hold opinions on agnosticism, whilst it is possible for a baby to lack belief in a God.
In simple terms, it is not possible for a baby to hold opinions on agnosticism, whilst it is possible for a baby to lack belief in a God.