hey man new to the forum. i had this problem with my logic too and did some research and its not a computer problem or anything...

prefrences....audio....general tab.... you will see a box u can check that says low latency mode check that... it turns off a few things but it makes it work how u want it. good luck

if you lowered the buffer to the minimum, then there might be one or more plugins that introduce the latency.
do you have compressors, eqs and reverbs on your recording track?
do you have a brickwall limiter in the master track?

if one or more of these plugins have an activated lookahead function, then this is the problem.

either you deactivate those plugins manually or you can activate the low latency mode, as line5gigety1 said. you can activate/deactivate it with the appropriate button on the right side of the transporter bar.

With Logic Pro a buffer setting of 128 is about as low as you typically can go.

You may want to do some tests to verify your system can handle a low buffer setting.

I have half the latency with the higer rates: 88.2+96K,

YMMV.