Why by 1861 was the compromise between the North and South no longer possible?

There are a number of reasons why this was the
case.

First, all sorts of compromises had already been
tried and had failed to end sectional tensions.  This meant that there were not really
any more ideas out there for how to create another
compromise.

Second, this first problem was partly due to
changes in law.  The Dred Scott decision had made it so Congress
could no longer make laws about slavery in the
territiories.

Third, the North and South had completely run
out of good will towards one another.  As the years went by and various fights over
policy had occurred, the North and the South came to trust each other less and less. 
Compromise is more possible when some level of trust exists between the two sides.  By
1861, there was no trust left, only animosity.

For these
reasons, compromise was no longer possible by 1861.