Forgetting the plausibility of it for a moment, if this were the case colonization would be drastically different. There would be no empires like the Aztec or Inca to accumulate large sums of wealth to draw in Spaniards and other Europeans, nor are there empires like them providing a native bureaucracy where Europeans could simply swap out the leadership.
If there is no widespread agriculture, then there are no recognizable maize, potatoes, squashes, sunflowers, cocoa, coca, peppers, and most importantly tobacco. Also no domesticated llamas, alpacas, or turkeys, but that's a smaller issue in the big picture of things. Without this huge package of domesticated plants to bring back to the Old World, European populations might have a hard time growing.
The native peoples will still die en masse to Old World diseases, but it would take longer for these diseases to spread. OTL, places like the Andes, Mesoamerica, and the Mississippi basin were densely populated, perfect conditions for new diseases to spread far and wide and fast. Without agriculture, the whole continent will only be able to support a fraction of its OTL population, and that smaller population will be more thinly spread out. This could mean Europeans would rely on imported labor even earlier and on a larger scale.
On the other hand, without mountains of precious metals and gems already dug up and ready for the taking, without native governments to more easily supplant, without a host of food and cash crops available, and with an even tinier native population to trade with or enslave, I think a lot of colonization in the West is stillborn. It will gradually happen, but not nearly as quickly as OTL. Europe in general is going to be poorer for longer, since it won't be able to exploit the massive amounts of capital generated from the New World, which will stunt its population growth, economic growth, and probably even industrialization. Demographically, there would be even lower numbers of Indian and mestizo populations, but also a significantly higher African population from imported slaves/indentured servants.
Oh I almost forgot, but quinine is also derived from a native plant, though I'm not sure how much humans have influenced its natural evolution. It's possible this plant is never harvested or its properties never discovered, and if that's the case the combination of lack of capital and anti-malarial drugs means that large scale African colonialism just isn't going to happen.