Kentucky really wasn't neutral. It just had a governor who was pro-Confederacy, and a legislature which favored the Union. In 1861 though, it favored trying to reconcile the two sides. As that hope faded, everyone knew the time for neutrality was over. It was an expedient measure that temporarily satisfied all sides, not a permanent doctrine.
Public opinion was turning in favor of the Union, and in June 1861, the special congressional election saw 8 of 9 seats go to Unionists. At this point in time, Kentuckians of all stripes were leaving the state to enlist in the army of their choice.
This is not an issue of either the Union or Confederacy violating Kentucky's neutrality, or that Kentuckians didn't care about the issues and would join a side based on whoever provoked it. It's that Kentucky began 1861 in a very divided state, and neither the Unionists or Confederates could control events. So both opted for neutrality to give themselves more time to take control of the state. As 1861 progressed, the state steadily became more Unionist as the undecideds made up their mind.
The Confederate invasion of Kentucky was just a useful pretext for Kentucky to abandon the status quo. If that hadn't happened, Kentucky was going to go for the Union anyway. There is a reason when after both Union and Confederate armies entered the state, Kentucky passed a resolution only demanding the Confederate army leave.
Absent a Confederate invasion, Kentucky was going to slowly align itself with the Union. It's just a matter of finessing it.
So the issue isn't an outside force. It's the internal balance within Kentucky which could not long sustain a policy of neutrality.