In your shoes, I'd open a support case. Something doesn't seem right.
Occasionally extracts which use "incremental updates" will slow a little (while extracting, not while being "read" by workbooks) after many updates, but you shouldn't be seeing anything/something like this otherwise.
did you ever find out the reason for your slow down? I have similar experience. Some unknown and strange action/reason sometimes triggers a severe slowdown in my workbook, forcing me to roll back to previous version of the workbook to get the speed back.