Halevy was born in London to an established Orthodox Jewish family. He emigrated to Israel in 1948. He attended Ma'aleh, a religious high school in Jerusalem, and later, graduated (with commendation) in law from the Hebrew University of Jerusalem. Between 1957–1961 he was the editor of the journal Monthly Survey (סקירה חודשית), published by the Chief Education Officer. In 1961, he began his work in the Mossad. In 1967, he was selected to the Chief Branches Forum.
Halevy remained in the Mossad for the next 28 years, heading three different branches throughout. Between 1990-1995, under the directorship of Shabtai Shavit, he served as deputy director and as head of the headquarters branch. In 1996, he became the Israeliambassador to the European Union in Brussels. In March 1998, he became the director of Mossad following the resignation ofDanny Yatom.
Halevy served as the envoy and confidant of five Prime Ministers: Yitzhak Shamir, Yitzhak Rabin, Benjamin Netanyahu, Ehud Barakand Ariel Sharon. He took an active part in a special mission by Rabin in forging the Israel–Jordan Treaty of Peace. After the failure of the Mossad operation to assassinate Hamas leader Khaled Mashal in 1997, he took an active part in Benjamin Netanyahu's mission to return the Mossad men captured in Jordan, and to settle the crisis with the King of Jordan.
On October 2002, he was appointed the second head of the National Security Council and an advisor to Prime Minister Ariel Sharon. In August 2003, he resigned from this position after Dov Weissglass, the bureau chief of Prime Minister Ariel Sharon, got too much power and he believed the Roadmap for peace was not favourable for Israel, and Prime Minister Sharon refrained from accepting his recommendations on a host of issues, and went to teach at Hebrew University of Jerusalem. He is the author of the book The role of the intelligence community in the age of strategic alternatives for Israel.
Halevy is known as a hard-headed pragmatist on issues involving the Israeli-Palestinian conflict, willing to ruffle feathers on the right and the left, unlike many others in the intelligence establishment who are known to take more extreme ideological positions on these issues. He believes that Israel should take up Hamas's offer of a long-term truce and try negotiating, because the Islamic movement is respected by Palestinians and generally keeps its word, he said. He pointed to the cease-fire in attacks on Israel that Hamas declared two years ago and has largely honored. "They're not very pleasant people, but they are very, very credible," Halevy said. He believes in an aggressive approach, but at the same time in recognizing the moderate leaders of Hamas as a partner in the Palestinian government.
In 2006 he published the book Man in the Shadows, covering Middle Eastern history since the late 1980s. Halevy was interviewed about his book on The Daily Show with Jon Stewart on April 24, 2006, and by guest host Brian Ross on the Charlie Rose Show
On January 27, 2007 an interview of his was published in Portugal, in which he stated "We are in the midst of a Third World War" with radical Islam, and predicted that it will take at least 25 years for the West to win.
In November 2011 Halevy said Iran should be prevented from becoming a nuclear power but expressed opposition to an attack which he said "could affect not only Israel, but the entire region for 100 years." He added "The growing haredi radicalization poses a bigger risk than Ahmadinejad".
Halevy has written extensively on Israel's relationship with the United States, generally taking a moderate, pragmatic view of the Washington-Jerusalem alliance. He wrote, for example: "Never, but NEVER surprise the president of the United States is a dictum I learned very quickly when entering the Mossad in 1961."
For October 23, 2012, during the United States presidential campaign between Barack Obama and Mitt Romney, Halevy published an op-ed piece "Who Threw Israel Under the Bus?" in the New York Times countering Mitt Romney's assertions about Obama's stance toward Israel. In it Halevy cited several key instances over the years in which the "Republican White House acted in a cold and determined manner, with no regard for Israel’s national pride, strategic interests or sensitivities" but that "no Democratic president has ever strong-armed Israel on any key national security issue." He concluded: "That's food for thought in October 2012."
In 2013, Halevy became chairman of the Shorashim program, a program that helps immigrants to Israel from the former Soviet Union verify their Jewish ancestry.
In December 2014 Halevy was interviewed by The Times of Israel. He claimed that Israel would never have peace unless Palestinians were treated with dignity as equals. He was critical of Benjamin Netanyahu and Naftali Bennett and their policies, especially regarding Jerusalem. He underscored the importance of the upcoming March 17, 2015 elections, which from his perspective "constitute an unprecedented opportunity to determine Israel’s policy vis-à-vis the peace process."