The process of playing in reverse involves decoding a bit, reversing it, and then seeking back to do the next block. Using a smaller buffer obviously increases the number of times it has to stop/seek/start - that's more CPU used on the overhead, and not the actual decoding/reversing of the stream data.
So, in conclusion, use a larger buffer

... 2 seconds should be ok.
Note that the decoding/reversing buffer is not the same thing as the playback buffer... you can use a large buffer for decoding/reversing the data, and still use a small buffer length for playback.