"The children will look up and shout 'trick or treat' and I will look down and whisper 'no'."