I was working on a strange problem the other day.
One of my customers had a VM that was unusually slow partitioning a disk. When the partition table was being build the performance tab in vCenter showed spikes in Datastore latency up to 70.000 ms, thats only about 11 minutes between writes…
Also any attempts to move machines of the datastore, or any datastores that was on the path with broken GBIC, resulted in errors and crazy latency spikes.
The ESXi logs showed no apperent errors, the storage subsystem had no errors and nothing was “visually” broken. Our storage specialist finally found out that is was a faulty GBIC, not broken, but just not functioning correctly.
Update: The machine that had the high latency had some ghost snapshots, that did not appear in the Snapshot Manager, and it seems as they had not been deleted because the deletion of the snapshots timed out. After the GBIC had been replaced I created a new snapshot and then deleted it and all the snapshots (7 total, 110 gb data) were merged into the vmdk.