if its round his neck and its really small bits of "dirt" it could actually be mite eggs, i thot it was dirt on my ferret but when it didnt go away after a few days i knew something was up and when i checked his coat he was riddled with mites and fleas.
spray it with some duramitex i think its called which kills mite ticks and fleas and see what happens, i think its around 7quid a bottle in a local petshop.
although your not supposed to clean ferrets to much as it reduces the amount of natural oils on there skin which can make them actually smell worse as their bodies are trying to ren