You cant reliably determine sex from physical characteristics until the tort is several years old. At 5 years old you should be able to have a good idea.
The one exception to the rule is that males sometimes flash their penis, if you see that of course you know what you have, but if not then you have to wait a little longer for the tort to develop.