The launch of The X Factor proved a turn-off for TV viewers on Saturday night - ratings plummeted by more than three million compared to last year.