Complex, stratified societies can definitely exist without agriculture. The PNW is a shining example of this with class separation, economic and social complexity, elaborate architecture and an interesting political system. The Jomon, too, are an example of this, as are many other cultures.
But...