If the foal is a male, it is a colt. Colt is only applicable if the foal is male. Like filly, colt applies until the horse turns four. However, if you castrate a colt, it is then a gelding.
Hopefully, you will easily understand the word filly every time you hear it now. The easiest way to remember is that a filly is always a female horse. Then, if you can remember, that it is always a young female horse.