for example if you just want to check if the same identical MP3 files are lying at two places in HDD to be compared than it requires to have just a binary file comparison like diff - this is as good as comparing any file. 

However, if you want to see if two different files has the same song letsay - then you have to decode both files using any standard decoder (in your case consider looking at JMF - Java media framework). This will give you samples of each of the files -a long of array of distinct values. after this you can find the difference between the two. this way you can compare between a MP3 with any other format as well. You must ensure that Sampling rate and typical sound level of files are same. 

Sometimes one can just compare few minutes of audio initially to make a quick check.

Dipan