A former Sunderland AFC director who claimed he was wrongly forced out of his job has had his £1m-plus damages claim thrown out by a top judge.