As ratings plummet worryingly for a second year, Simon Cowell and his close associates are starting to consider the unthinkable: Is it time to rest The X Factor?