I have an SSRS performance bottleneck on my production server that we have diagnosed as being related to the use of subreports.
Background facts:
* Our Production and Development servers are identically configured
* We've tried the basic restart/reboot activities, didn't change anything about the performance.
* The Development server was "cloned" from the Production server about a month ago, so all application settings (memory usage, logging, etc.) are identical between the two
* For the bottlenecked report the underlying stored procedure executes in 3 seconds, returning 901 rows, in both environments with the same parameters. The execution plan is identical between the two servers, and the underlying tables and indexing
is identical. Stats run regularly on both servers.
* In the development environment the report runs in 12 seconds. But on Production the report takes well over a minute to return, ranging from 1:10 up to 1:40.
* If I point the Development SSRS report to the PROD datasource I get a return time of 14 seconds (the additional two seconds due to the transfer of data over the network).
* If I point the Production SSRS report to the DEV datasource I get a return time of well over a minute.
* I have tried deleting the Production report definition and uploading it as new to see if there was a corruption issue, this didn't change the runtimes.
* Out of the hundreds of Production SSRS reports that we have, the only two that exhibit dramatically different performance between Dev and Prod are the ones that contain subreports.
* Queries against the ReportServerTempDB also confirm that these two reports are the major contributors to TempDB utilization.
* We have verified that the ReportServerTempDB is being backed up and shrunk on a regular basis.
These factors tell me that the issue is not with the database or the SQL. The tests on the Development server also prove that the reports and subreports are not an issue in themselves - it is possible to get acceptable performance from them in the
Development environment, or when they are pointed from the Dev reportserver over to the Prod database.
Based on these details, what should we check on our Prod server to resolve the performance issue with subreports on this particular server?