To be fair there were also plenty of poor whites who saw no point in fighting for slavery and called the whole conflict a rich man's war. Problem for them was that the Confederate government wasn't shy about drafting people (notably exempting the elite of society from conscription) and hunting down those who refused.
True enough, in any case more than just the slaveowners had a stake in slavery.