Simon Cowell has always been "torn" between Cheryl Cole and Nicole Scherzinger.

The music mogul had spent months debating who should join him on the judging panel for the US version of 'The X Factor' and while he initially went with fellow Brit Cheryl, it has now been claimed she has been replaced by Nicole.

Insiders claim Simon - who is also on the panel with Paula Abdul and L.A. Reid and worked with Cheryl on the UK version for three series - knew he had to change things because the chemistry between the original judges wasn't right and felt the former Pussycat Dolls star - who was originally meant to co-host alongside Steve Jones - would fit in better than Cheryl.

A source told UsMagazine.com: "Throughout the whole decision making process, Simon was really torn between Cheryl and Nicole. He absolutely loved Nicole, really felt like she was the right one, but also knew his chemistry with Cheryl had been great before in the UK. He finally decided on the more comfortable option and chose Cheryl.

"With the pressure building for this show to be a success, Cheryl was a much bigger risk and instinctively, as time went on, Nicole felt more and more like the right choice. Nicole was so disappointed before when she wasn't picked, but now, thankfully, she is in the role she was always meant for. And Simon knows that."

Simon knows he's under huge pressure for the show to be a hit and accepted there would be hard decisions to make along the way.

The source added: "It's his name on the line all the way through this -- it literally can't fail. The chemistry has to be right; it's the vital element that holds this whole thing together. The judges' connection with the audience could make or break the entire project."

It has yet to be confirmed whether Cheryl has been dropped but insiders claim she has flown back to the UK and is now in hiding.