Prefetch is designed to be a list of binaries that individual applications will be requesting (immediately, or very shortly) after they are launched - by using these tiny files the OS can order its disk reads and have the files in memory before they are requested, removing latency from the application launch.
(Superfetch is a further improvement that looks at what applications you typically launch, and prefetches the necessary files
before the icon is even double-clicked.)
So I would doubt removing prefetch files or disabling Superfetch could
improve performance, but possibly the opposite.
Superfetch only uses idle time to do its lower priority I/O, so even if it reads parts of files into cache and does not use them, it's not impacting user experience or delaying system services that want to use the disk.
As for a community source (hence independent from Microsoft) regarding the prefetcher:
http://en.wikipedia.org/wiki/PrefetcherQUOTE
A second myth is that the user should delete the prefetch folder contents to speed up the computer. If this is done, Windows will need to re-create all the prefetch files again, thereby slowing down Windows during boot and program starts until the prefetch files are created.