"We are delighted to announce Bill's appointment as CEO," RFU Chairman Andy Cosslett said in a statement. "He brings a rare combination of experience from the worlds of blue chip business and elite sport and has a tremendous track record.
"His passion for the game is deep rooted and this will stand him in very good stead."
During his time at the BOA, Sweeney oversaw Team Great Britain's performances at the Winter Olympics in Sochi in 2014 and in PyeongChang in 2018, as well as at the Summer Games in Rio in 2016.
Before joining the BOA, he held senior roles in sports businesses including Adidas and Puma, and gained management experience with Shell, Mars and Unilever.
"I leave behind a very strong British Olympic Association, both financially and in readiness for Tokyo 2020," Sweeney said. "This is the only opportunity that I would have left the BOA for.
"From the grassroots game to our England teams, rugby's values and opportunities are very special. I am passionate about the game and it is an honour to be joining the RFU team."
The official date of Sweeney's switch to the RFU is yet to be agreed, with interim CEO Nigel Melville set to continue in the role until Sweeney starts at Twickenham.
(Reporting by Simon Jennings in Bengaluru; Editing by Alexander Smith)
