Before I answer that question, here is some of the background to the Milo Yiannopoulos case.
Late last year, a Simon & Schuster imprint, Threshold, offered Milo Yiannopoulos a $250,000 advance for his book Dangerous. Yiannapoulos had gained a reputation as a notorious right-wing "troll" and was editor of Breitbart Tech. His racist, misogynist, and generally over-the-top statements were outrageous enough that he became "cool." (Twitter did not think Milo was "cool." Yiannopoulos was banned for hate speech after his racist tirade against Ghostbusters actress, Leslie Jones.)
Yiannopoulos' bad behavior, if anything, made him more attractive to S&S, which was willing to pay a quarter of a million dollars for the opportunity to publish his book. There was some pushback however, as one reviewer announced he would not review the book once it was published, and bookstores said they would not carry it. (That, by the way, is not a violation of the First Amendment; it is the operation of the free market. Nobody is obligated to review or sell a book.) What made S&S think twice was Yiannopoulos' public endorsement of pedophilia between underage boys and men. Conservatives immediately dropped Yiannopoulos, and his contract with S&S was canceled shortly thereafter.
Not to be outdone, Yiannopoulos, sued S&S for breach of contract six months later. In the interim, he self-published his book, which sold 18,000 copies the first week. (Yiannopoulos' publicist said the book had sold 100,000 copies the first week, but all independent sales tracking sources disagree.)
This brings us back to the question: When can a publisher terminate a contract? The answer is: It depends on what is in the contract.
All publishing contracts contain a clause that specifies the circumstances under which a contract may be terminated by the publisher. These may include Acts of God (a hurricane wipes out your publisher's headquarters), buy-outs (a larger company purchases your publisher), failure to deliver a manuscript as promised, legal liability (plagiarism, possibility of lawsuit), and any other reason that a publisher might decide at the publisher's discretion. Publishing houses also claim the right of discretionary termination for an "unsatisfactory" manuscript, the definition of "unsatisfactory" being left entirely to the publisher. In that case, the publisher may terminate and demand all or part of the advance.
Can Yiannopolous win his lawsuit? Probably not. S&S returned all rights to Yiannopoulos and let him keep $80,000 of his advance. (An advance is not paid out all at once, but in stages. The second of those stages would have been acceptance of the manuscript, which did not happen.) A further problem for Yiannopoulos is that one of those discretionary termination clauses was included in his contract. Once you sign on the dotted line, especially if you keep your advance and accept a reversion of rights, it's hard to have your day in court, especially if that day is, as S&S put it, “a meritless publicity stunt.”
You can read more about the suit here: S&S Asks Court to Dismiss Yiannopoulos Suit