You can grow potatoes there, because it originated there. The Andeans were the ones that domesticated potatoes. Until the Spaniards spread it, it was the only place in the world was it was eaten.
As for the bison, it's not the cold that's the problem; it's the altitude, and the lack of appropriate grasslands.
As for the pampas, not many people lived there, just some semi-nomads, and no high culture whatsoever. If you look at a map of the Inca Empire at it's greatest extension, you'll be seeing all the regions inhabited by civilized peoples, with the sole exception of the Chibchas in Colombia. The Incas only stopped growing when they run out of people worth conquering, and made very little attempts to settle beyond that, because it was so different form what they were accustomed to.