Whisk the egg yolks and caster sugar together in a bowl until pale and thick. Warm the milk in a saucepan over low heat until just hot, then slowly whisk into the egg yolk mixture in a slow, steady stream, whisking continuously.5
Pour the egg yolk mixture into the pan. Return the pan to a very low heat and cook stirring continuously with a wooden spoon, for 3 minutes or until the mixture coast the back of the spoon. Do not let it simmer or it may curdle. (Be careful not to overcook the egg or it will scramble; have a bowl of ice alongside the stove in case you need to quickly cool the mixture down.) Immediately pour the eggnog through a coarse-mesh strainer into a heatproof bowl. Stir in the cream, and then stir in the brandy. Pour into a serving jug and serve warm or chilled. Add nutmeg to taste.