To improve the performance of the column sets, remove as much information as possible from the column set. Removing any unnecessary information minimizes the work performed by the server.
For example, the workflow name in the SPFDocumentVersion column set must expand an edge to get from the version to the current workflow to display the workflow name. By removing the column item, the server sends less data to the client (Desktop Client or custom interface) and is taxed less, as it no longer needs to expand the edge to get to the workflow.
Similarly, when file names are exposed in list view columns, the server must expand the relationship to find the related files and their data, even if it is displaying only the file name.
Minimize the use of related items in the column sets, as this brings back the complete related object to the client.