Back in high school I used to freelance for the Associated Press, mainly at the Saratoga Race Course. This means I was fortunate enough to have a press pass that gave me access to the majority of the track most of the day.
Once of my favorite times to visit the track was early in the morning to watch the horses train and warmup for the day. The whole place is eerily empty at that hour and you can really appreciate the horses for the amazing animals they are without the distractions of the crowd.
As with many kinds of photography, some of the best light occurs just after sunrise. If the weather conditions are just right (a warm humid day followed by a chilly night), fog completely covers the racecourse during pre-dawn and slowly burns off as the sun comes up.
Even if you're not headed there for photography, I highly recommend you visit at least once in the early morning hours (with a good cup of coffee in tow) to witness the track in a completely different light.