Neurosurgeon Ben Carson says the White House wanted him to apologize for “offending” President Obama after he famously delivered a conservative message at the National Prayer Breakfast last year.

Carson, the former director of pediatric neurosurgery at Johns Hopkins Hospital, recalls the events surrounding his 2013 speech in his new book, One Nation: What We Can All Do To Save America’s Future. The Daily Caller obtained an advance copy of the book, which is set for release May 20.

“He did not appear to be hostile or angry,” Carson writes of Obama, “but within a matter of minutes after the conclusion of the program, I received a call from some of the prayer breakfast organizers saying that the White House was upset and requesting that I call the president and apologize for offending him. I said that I did not think that he was offended and that I didn’t think that such a call was warranted.”

Conservatives rallied around Carson last year after his remarks, made from a podium as Obama sat just feet away. In his speech, he railed against “political correctness” and offered specific ideas for health care reform.