Minim Library: Cannot change buffer size

I have just started learning about and using the minim library. However, if I use anything other than the default value (1024) for buffer length in the 'loadFile()" method, the audio becomes garbled. Values close to 1024 (like 1000, 900 etc.. ) seem to work but values like say 512 cannot be used.. is this normal? I don't understand why changing the buffer length should affect playback.