But his response to the video was deemed "insufficient" by Matt Schlapp, the chairman of the American Conservative Union, which hosts CPAC.
Mr Yiannopoulos was permanently banned from Twitter in 2016 soon after posting racist about Ghostbusters actor Leslie Jones, although the social media giant refused to say which of his tweets prompted the ban.
Organizers of the Conservative Political Action Conference have disinvited professional provocateur Milo Yiannopoulos after a video interview emerged of him appearing to condone pedophilia emerged. "I am a gay man, and a child abuse victim", he wrote.
On Sunday, a video surfaced in which Yiannopoulos said some relationships between adults and 13-year-olds are consensual.
At a press conference in NY following his resignation, Yiannopoulos offered a quasi-apology, saying, "I am certainly guilty of imprecise language, which I regret".
However at another point in the video he says the USA age of consent, which is 16 to 18 years old depending on location, is "about right".
In a statement from the publisher, Simon & Schuster wrote, "After careful consideration, Simon & Schuster and its Threshold Editions imprint have cancelled publication of "Dangerous" by Milo Yiannopoulos".
Yiannopoulos has vigorously defended his comments via Facebook, claiming they had been "selectively edited".
The rightwing commentator's latest trouble stems from his appearance on an internet livestream a year ago, during which he said that relationships between adult men and teenage boys can be beneficial.
Yiannopoulos dismissed the notion the controversy over the video tapes will derail his career.
"I certainly consider myself to be one of them - people who were sexually active younger".
Denying that he had condoned pedophilia, he said: "I find those crimes to be absolutely disgusting".
On Facebook on Monday, he expressed regret for using the word "boys" instead of young men while discussing the benefits of relationships between men with large age differences. "And I've repeatedly expressed disgust at pedophilia in my feature and opinion writing".
The University of California at Berkeley cancelled a planned speech by Yiannopoulos this month after protests against him turned violent.