Very Large Excel - Some Columns Are Slow to Copy and Paste

ShawnC84

New Member
Joined
Mar 27, 2020
Messages
4
Office Version
  1. 365
  2. 2016
Platform
  1. Windows
All,

I created a very large Excel file (i.e. hundreds of MBs or 100,000 rows by 369 columns) for calculating a sample of about 100,000 insurance policies across five or so different companies. I started off by building out the first row with all the needed formulas and then have copied them down to row 100,000 of the worksheet. I have been the columns in sections of about 50 and as expected it can take the computer up to 10 seconds to complete the copy and paste. On rare occasions the copy and paste can take 30 to 60 minutes to complete and it does seem to be repeatable for the same columns. To be clear this is only the copy and paste. If I turn on the "Automatic" calculations, I usually stay in Manual mode once I build out the full spreadsheet, the entire completed worksheet still calculates relatively quick (i.e. a few seconds). Are there certain formulas that Excel struggles with copying and pasting more than others? I have not been able figure out the reason.

Computer Details:
Windows 10 64bit
Excel 365 64bit
Intel i7-7700K
32 GB Ram

Thanks for any thoughts.
 

Excel Facts

Best way to learn Power Query?
Read M is for (Data) Monkey book by Ken Puls and Miguel Escobar. It is the complete guide to Power Query.
Welcome to the forum.

Which Excel functions have you included in the formulas? Some volatile functions slow things down considerably. Other culprits include formulas that use arrays, heavy use of conditional formatting, user-defined functions and formulas that require error-handling.

Have you considered turning the data into an official Excel Table? Putting the formula in any cell of a Table will populate the entire field (i.e., the column of the Table) thus speeding calculation considerably.

Can you upload some sample data for us to mess around with? Include the various formulas. Use xl2bb to upload it.

There are dozens of websites with advice on speeding Excel files. Here is one.
 
Last edited:
Upvote 0
Hi DRSteele,

Attached you will see snapshots of the columns (OA through QZ in the original spreadsheet)/formulas that are giving me the trouble. As you can see most of the formulas are VLOOKUPs with some INDEX, IF, PRODUCT, SUM, and ROUNDs thrown in for good measure. The formulas used in columns OA through Qz are very similar to the formulas used in earlier columns, which do not have the copy paste time delay. Is it possible that Excel gets "overwhelmed" despite having plenty of memory available? I have not yet tried to make it an official Excel Table, but I will give it a try. Also below is the output from the xl2bb. Please let me know your thoughts. Thanks

100,000 character limit hit (Part 1 or 2)

Excel Copy Paste Example - Most Ros Removed.xlsb
ABCDEFGHIJKLMNOPQRSTUVWXYZAAABACADAEAFAGAHAIAJAKALAMANAOAPAQARASATAUAVAWAXAYAZBABBBCBDBEBFBGBHBIBJBKBLBMBNBOBPBQBRBSBTBUBVBWBXBYBZCACBCCCDCECFCGCHCICJCKCLCMCNCOCPCQCRCSCTCUCVCWCXCYCZDADBDCDDDEDFDGDHDIDJDKDLDMDNDODPDQDRDSDTDUDVDWDXDYDZEAEBECEDEEEFEGEHEIEJEKELEMENEOEPEQERESETEUEVEWEXEYEZFAFBFCFDFEFFFGFHFIFJFKFLFMFNFOFPFQFRFSFTFUFVFWFXFYFZGAGBGCGDGEGFGGGHGIGJGKGLGMGNGOGPGQGRGSGTGUGVGWGXGYGZHAHBHCHDHEHFHGHHHIHJHKHLHMHNHOHPHQHRHSHTHUHVHWHXHYHZIAIBICIDIEIFIGIHIIIJIKILIMINIOIPIQIRISITIUIVIWIXIYIZJAJBJCJDJEJFJGJHJIJJJKJLJMJNJOJPJQJRJSJTJUJVJWJXJYJZKAKBKCKDKEKFKGKHKIKJKKKLKMKNKOKPKQKRKSKTKUKVKWKXKYKZLALBLCLDLELFLGLHLILJLKLLLMLNLOLPLQLRLSLTLULVLWLXLYLZMAMBMCMDMEMFMGMHMIMJMKMLMMMNMOMPMQMRMSMTMUMVMWMXMYMZNANBNCNDNENFNGNHNINJNKNLNMNNNONPNQNRNSNTNUNVNWNXNYNZOAOBOCODOEOFOGOHOIOJOKOLOMONOOOPOQOROSOTOUOVOWOXOYOZPAPBPCPDPEPFPGPHPIPJPKPLPMPNPOPPPQPRPSPTPUPVPWPXPYPZQAQBQCQDQEQFQGQHQIQJQKQLQMQNQOQPQQQRQSQTQUQVQWQXQYQZ
3KeyGridIDCountyCounty TierZip CodePO NameTHS TerritoryTerritory TypeDtC_THSDtc_THS_BandedCov A LimitPPCTypeOccupancyCov B %Cov B Limit (Buyup)Cov C %Cov C LimitCov D %Cov D Limit (Buyup)Age of HomeAge of Home BandedInsurance ScoreClaimsConstruction TypeResidence TypeAge of InsuredAge of Insured BandedDeductibleDed AOP PercentW/H DeductRoof MaterialRoof ShapeAge of RoofNumber of StoriesSq FeetWind_ExclusionPrior_LiabiltyFull WaterAAL + Reinsurance - AllAAL + Reinsurance - Cov AAAL + Reinsurance - Cov CCountWinAverage - AllForemost - AllNCRB_2018 - AllNCRB_2011 - AllTHS_NC_Normal - AllTHS_SC - AllAMS - AllAMPC - AllAuto_Owners - AllASI - AllAverage - AForemost - ANCRB_2018 - ANCRB_2011 - ATHS_NC_Normal - ATHS_SC - AAMS - AAMPC - AAuto_Owners - AASI - AAverage - BForemost - BNCRB_2018 - BNCRB_2011 - BTHS_NC_Normal - BTHS_SC - BAMS - BAMPC - BAuto_Owners - BASI - BAverage - CForemost - CNCRB_2018 - CNCRB_2011 - CTHS_NC_Normal - CTHS_SC - CAMS - CAMPC - CAuto_Owners - CASI - CAverage - DForemost - DNCRB_2018 - DNCRB_2011 - DTHS_NC_Normal - DTHS_SC - DAMS - DAMPC - DAuto_Owners - DASI - DTerritoryBase - FireBase - ECBaseCalendar Year ModifierDeductibleDed_Max CreditAge of HomeInsurance ScoreClaimsConstructionResidence TypeAge of InsuredOccupancyWH_ExclusionFirelineSettlementWH_DedComp VacantCov ACov BCov C - FireCov C - ECCov DTerritoryBase DWBase OSBase PPAge of InsuredConstruction TypeResidence TypeRoof_ShapeRoof_MaterialTerritory FactorAge of HomeDeductibleClaimsInsurance ScorePPCDtCCovA UVRCCovB UVRCCovC UVRCWildfire ScoreWater ScalingWater Damage LimitPolicy FeeCov ACov BCov CCov DWH ExclWaterTerritoryTerritory TypeAOP and WH Ded SameKey - Fire - AKey Factor/UVRC - Fire - ABase Premium - Fire - ADeductible_Amt (406) - Fire - AKey - EC - ASeasonalModified Key Premium (Product) - AWind Mit - AWH_Exclusion AModified Key Premium (Sum) - AKey Factor/UVRC - EC - ABase Premium - EC - ADeductible_Amt (406) - EC - AKey - Fire - CKey Factor/UVRC - Fire - CBase Premium - Fire - CDeductible_Amt (406) - Fire - CKey - EC - CSeasonalModified Key Premium (Product) - CWind Mit - CWH_Exclusion CModified Key Premium (Sum) - CKey Factor/UVRC - EC - CBase Premium - EC - CDeductible_Amt (406) - EC - CWH_Exclu_Credit (A3) - EC - ADed - Step 1 - EC - ADed - Step 2 (Adjusted Ded Credit) - EC - ADed - Step 3 - EC - ADed - Step 4 - EC -ADed Step 5 - EC - AWH_Exclu_Credit (A3) - EC - CDed - Step 1 - EC - CDed - Step 2 (Adjusted Ded Credit) - EC - ADed - Step 3 - EC - CDed - Step 4 - EC -CDed Step 5 - EC - CCov ACov BCov CCov DUnprotectedTerritoryTerritory TypeAOP and WH Ded SameKey - Fire - AKey Factor/UVRC - Fire - ABase Premium - Fire - ADeductible_Amt (406) - Fire - AKey - EC - ASeasonalModified Key Premium (Product) - AWind Mit - AWH_Exclusion AModified Key Premium (Sum) - AKey Factor/UVRC - EC - ABase Premium - EC - ADeductible_Amt (406) - EC - AKey - Fire - CKey Factor/UVRC - Fire - CBase Premium - Fire - CDeductible_Amt (406) - Fire - CKey - EC - CSeasonalModified Key Premium (Product) - CWind Mit - CWH_Exclusion CModified Key Premium (Sum) - CKey Factor/UVRC - EC - CBase Premium - EC - CDeductible_Amt (406) - EC - CWH_Exclu_Credit (A3) - EC - ADed - Step 1 - EC - ADed - Step 2 (Adjusted Ded Credit) - EC - ADed - Step 3 - EC - ADed - Step 4 - EC -ADed Step 5 - EC - AWH_Exclu_Credit (A3) - EC - CDed - Step 1 - EC - CDed - Step 2 (Adjusted Ded Credit) - EC - ADed - Step 3 - EC - CDed - Step 4 - EC -CDed Step 5 - EC - CCov ACov BCov CCov DUnprotectedNCRB_2008_TerritoryASI_TerritoryTerritory TypeAOP and WH Ded SameKey - Fire - ADeviation - Fire - AKey Factor/UVRC - Fire - ABase Premium - Fire - ADeductible_Amt (406) - Fire - AKey - EC - ADeviation - EC - ASeasonalModified Key Premium (Product) - AWind Mit - AWH_Exclusion AModified Key Premium (Sum) - AKey Factor/UVRC - EC - ABase Premium - EC - ADeductible_Amt (406) - EC - AKey - Fire - CDeviation - Fire - CKey Factor/UVRC - Fire - CBase Premium - Fire - CDeductible_Amt (406) - Fire - CKey - EC - CDeviation - EC - CSeasonalModified Key Premium (Product) - CWind Mit - CWH_Exclusion CModified Key Premium (Sum) - CKey Factor/UVRC - EC - CBase Premium - EC - CDeductible_Amt (406) - EC - CWH_Exclu_Credit (A3) - EC - AZone_Dev EC - ADed - Step 1 - EC - ADed - Step 2 (Adjusted Ded Credit) - EC - ADed - Step 3 - EC - ADed - Step 4 - EC -ADed Step 5 - EC - AWH_Exclu_Credit (A3) - EC - CZone_Dev EC - CDed - Step 1 - EC - CDed - Step 2 (Adjusted Ded Credit) - EC - ADed - Step 3 - EC - CDed - Step 4 - EC -CDed Step 5 - EC - CCov ACov BCov CCov DUnprotectedTerritoryTierPPCConstructionTerritoryStep C (Product)Step D (Form)Step E (Settlement)Step F (Base Rate)Cov A Step G (Base Premium)CovC (Base Premium)Age of Home - AResidence Type - AOccupancy - ADeductible - AAge of Home - CResidence Type - COccupancy - CDeductible - CCov A - Step VCov C - Step VClaimsRoof AgeInsurance ScoreNonPayAge of InsuredAdvanced Quote - AAdvanced Quote - CCov ACov BCov CCov DTerritory NCRBTerritory AMPCTerritory TypeAOP and WH Ded SameKey - Fire - AAge of InsuredAge of HomeClaimsTerritoryInsurance ScoreKey Factor/UVRC - Fire - ABase Premium - Fire - ADeductible_Amt (406) - Fire - AKey - EC - ASeasonalModified Key Premium (Product) - AWind Mit - AWH_Exclusion AModified Key Premium (Sum) - AAge of InsuredAge of HomeClaimsTerritoryInsurance ScoreKey Factor/UVRC - EC - ABase Premium - EC - ADeductible_Amt (406) - EC - AKey - Fire - CAge of InsuredAge of HomeKey Factor/UVRC - Fire - CBase Premium - Fire - CDeductible_Amt (406) - Fire - CKey - EC - CSeasonalModified Key Premium (Product) - CWind Mit - CWH_Exclusion CModified Key Premium (Sum) - CAge of InsuredAge of HomeKey Factor/UVRC - EC - CBase Premium - EC - CDeductible_Amt (406) - EC - CWH_Exclu_Credit (A3) - EC - ADed - Step 1 - EC - ADed - Step 2 (Adjusted Ded Credit) - EC - ADed - Step 3 - EC - ADed - Step 4 - EC -ADed Step 5 - EC - AWH_Exclu_Credit (A3) - EC - CDed - Step 1 - EC - CDed - Step 2 (Adjusted Ded Credit) - EC - ADed - Step 3 - EC - CDed - Step 4 - EC -CDed Step 5 - EC - CCov ACov BCov CCov DUnprotectedTerritoryTerritory TypeAOP and WH Ded SameKey - Fire - AKey Factor/UVRC - Fire - ABase Premium - Fire - ADeductible_Amt (406) - Fire - AAOH - Fire - AAOI - Fire - AClaims - Fire - ADtC - Fire - AInsurance Score - Fire - ARoof Age - Fire - ARoof_Material - Fire - AYear Built - Fire - AKey - EC - ASeasonalModified Key Premium (Product) - AWind Mit - AWH_Exclusion AModified Key Premium (Sum) - AKey Factor/UVRC - EC - ABase Premium - EC - ADeductible_Amt (406) - EC - AAOH - EC - AAOI - EC - AClaims - EC - ADtC - EC - AInsurance Score - EC - ARoof Age - EC - ARoof_Material - EC - AYear Built - EC - AKey - Fire - CKey Factor/UVRC - Fire - CBase Premium - Fire - CDeductible_Amt (406) - Fire - CAOH - Fire - CAOI - Fire - CClaims - Fire - CDtC - Fire - CInsurance Score - Fire - CRoof Age - Fire - CRoof_Material - Fire - CYear Built - Fire - CKey - EC - CSeasonalModified Key Premium (Product) - CWind Mit - CWH_Exclusion CModified Key Premium (Sum) - CKey Factor/UVRC - EC - CBase Premium - EC - CDeductible_Amt (406) - EC - CAOH - EC - CAOI - EC - CClaims - EC - CDtC - EC - CInsurance Score - EC - CRoof Age - EC - CRoof_Material - EC - CYear Built - EC - CWH_Exclu_Credit (A3) - EC - ADed - Step 1 - EC - ADed - Step 2 (Adjusted Ded Credit) - EC - ADed - Step 3 - EC - ADed - Step 4 - EC -ADed Step 5 - EC - AWH_Exclu_Credit (A3) - EC - CDed - Step 1 - EC - CDed - Step 2 (Adjusted Ded Credit) - EC - ADed - Step 3 - EC - CDed - Step 4 - EC -CDed Step 5 - EC - CCov ACov BCov CCov DUnprotected
436.4391014535134, -79.78009596851640364307978RockinghamRemainder of State27288Eden310Inland2002002500001DP-3Seasonal10%00%010%022204500Masonry1 Family41405000.2%1000TileGable1211894No>$300,000Yes111047 867721433.52012867115111841142954.741 8045723011986.93804108311849030 0000000075.125 63149190636802390 0000000025000050007500017500075000A00010.060000-0.15000001000100000310684.194.2550.920.85110.831.1811.0330.922.11111.890900100251986.9300000310InlandNo2410.42500291.5544.950044.9512.79575-21.275700011.551.55001.5542.4266-2.7060000.03721.275-21.2750000.0412.706-2.7068040630060InlandNo2210.4229-11.45201.6533003312.79422-67.521000022.154.3004.342.42182-32.760000.1667.52-67.520000.1832.76-32.765720149006086InlandNo221.662910.4380-19201.528351.6550.43560050.435612.79645-103.2101.704300021.59852.156.87355006.8735542.42292-52.5601.329000.16103.2-103.201.39000.1852.56-52.5690302390064Premier0.9450.9030.980.8360.8360.836564.0213190800-66-6900001264010.9311.1250.9441-6601184000310310InlandNo240.881.2710.94110.42620291.5544.950044.950.961.130.851.61112.79853-31.56170.881.2700011.551.55001.550.961.1342.4271-2.9110000.03731.561-31.5610000.0412.911-2.911108306800310InlandNo2410.425000.769210.713210.729610.81291.5544.950044.9512.79575-21.275110.71320.93550.729610.90.9170000.769210.713210.729610.8111.551.55001.5542.4266-2.7060.769210.71320.93550.729610.90.910000.03721.275-21.2750000.0412.706-2.70630101900
Calculator
Cell Formulas
RangeFormula
A4A4=VLOOKUP(RANDBETWEEN(1,MAX(Shared!$W$3:$W$98967)),Shared!$W$3:$X$98967,2,TRUE)
B4B4=CONCATENATE("0",LEFT(A4,2),MID(A4,4,2),"0",MID(A4,FIND("-",A4,1)+1,2),MID(A4,FIND(".",A4,5)+1,2))
C4C4=IF(VLOOKUP($A4,Territory_Assignment_Shared,1,TRUE)=$A4,VLOOKUP($A4,Territory_Assignment_Shared,3,TRUE),NA())
D4D4=IF(VLOOKUP($A4,Territory_Assignment_Shared,1,TRUE)=$A4,VLOOKUP($A4,Territory_Assignment_Shared,16,TRUE),NA())
E4E4=IF(VLOOKUP($A4,Territory_Assignment_Shared,1,TRUE)=$A4,VLOOKUP($A4,Territory_Assignment_Shared,2,TRUE),NA())
F4F4=IF(VLOOKUP($A4,Territory_Assignment_Shared,1,TRUE)=$A4,VLOOKUP(A4,Territory_Assignment_Shared,4,TRUE),NA())
G4G4=$DT4
H4,OB4H4=VLOOKUP($EW4,Territory_Type_THS_Proposed,2,FALSE)
I4I4=IF(VLOOKUP($A4,Territory_Assignment_Shared,1,TRUE)=$A4,VLOOKUP(A4,Territory_Assignment_Shared,15,TRUE),NA())
J4J4=IF(I4<=1,MROUND(I4,0.25),IF(I4<=10,MROUND(I4,1),IF(I4<=30,MROUND(I4,5),IF(I4<=100,MROUND(I4,10),MROUND(I4,25)))))
K4K4=IF($M4="DP-1",$CQ4,IF(CQ4>=75000,CQ4,IF(CR4>=75000,CR4,IF(CS4>=75000,CS4,IF(CT4>=75000,CT4,IF(CU4>=75000,CU4,75000))))))
L4L4=VLOOKUP($A4,Territory_Assignment_Shared,5,FALSE)
M4M4=VLOOKUP(RANDBETWEEN(1,2),Weight_Type,2,FALSE)
N4N4=IF($M4="DP-1",VLOOKUP(RANDBETWEEN(1,10),Weight_DP_1_Occupancy,2,FALSE),VLOOKUP(RANDBETWEEN(1,10),Weight_DP_3_Occupancy,2,FALSE))
O4O4=ROUND(P4/K4,2)+VLOOKUP($M4,Included_CovB_NCRB2018,2,FALSE)
P4P4=VLOOKUP(VLOOKUP(RANDBETWEEN(1,10),Weight_CovB_Percent,2,FALSE)*K4,Weight_Possible_CovB_Limits,1,TRUE)
Q4Q4=ROUND(R4/K4,2)
R4R4=VLOOKUP(VLOOKUP(RANDBETWEEN(1,10),Weight_CovC_Percent,2,FALSE)*K4,Weight_Possible_CovC_Limits,1,TRUE)
S4S4=ROUND(T4/K4,2)+VLOOKUP($M4,Included_CovD_NCRB2018,2,FALSE)
T4T4=IF($N4="Vacant",0,VLOOKUP(VLOOKUP(RANDBETWEEN(1,10),Weights_CovDPercent,2,FALSE)*K4,Weights_Possible_CovD_Limits,1,TRUE))
U4U4=MIN(100,IF($M4="DP-3",0,RANDBETWEEN(0,50))+VLOOKUP(RANDBETWEEN(1,INDEX(Weight_Age_of_Home,103,MATCH($E4,Weight_Age_of_Home_HeaderRow,0))),OFFSET(Weight_Age_of_Home,0,MATCH($E4,Weight_Age_of_Home_HeaderRow,0)-1),Weight_Age_of_Home_ColumnCount-MATCH($E4,Weight_Age_of_Home_HeaderRow,0)+1,TRUE))
V4V4=VLOOKUP($U4,Age_Banding,1,TRUE)
W4W4=VLOOKUP(RANDBETWEEN(1,ISWeights!$A$16),ISWeights!$A$2:$B$15,2,TRUE)
Y4Y4=VLOOKUP(RANDBETWEEN(1,10),Weight_Construction_Type,2,FALSE)
Z4Z4=VLOOKUP(RANDBETWEEN(1,100),Weight_Residence_Type,2,FALSE)
AA4AA4=RANDBETWEEN(25,80)
AB4AB4=VLOOKUP(AA4,Age_Banding,1,TRUE)
AC4AC4=VLOOKUP(RANDBETWEEN(1,10),Weight_Deductible,2,FALSE)
AD4AD4=ROUND(AC4/K4,3)
AE4AE4=IF($AK4="Yes",AC4,IF(AC4=500,1000,AC4))
AF4AF4=VLOOKUP(RANDBETWEEN(1,10),Weight_Roof_Material,2,FALSE)
AG4AG4=VLOOKUP(RANDBETWEEN(1,10),Weight_Roof_Shape,2,FALSE)
AH4AH4=IF($U4<=20,$U4,12)
AI4AI4=VLOOKUP(RANDBETWEEN(1,100),Weights_Number_of_Stories,2,FALSE)
AJ4AJ4=MAX(200,ROUND($K4/$AJ$2,0))
AK4AK4=IF($AK$
 

Attachments

  • Formulas 01-38.PNG
    Formulas 01-38.PNG
    113.1 KB · Views: 16
  • Formulas 39-76.PNG
    Formulas 39-76.PNG
    106 KB · Views: 15
Upvote 0
How long does it take if you turn off workbook calculation before doing the copy and paste ? (don't forget to turn it on again after)
 
Upvote 0
Hi rlv01,

Usually it only takes Excel a second or so to move from Automatic calculation to Manual calculation. Though today and yesterday after the final column copy and paste it only took Excel 5ish seconds to go from Manual to Automatic (including the formula calculation). Moving back to Manual afterward took about 15 minutes. This was a new development. I have never had that problem before.

Thanks
 
Upvote 0
I've never had an issue switching to manual calculation. My guess is that something else was going on and the switch to manual was coincidental. With 100k rows of cells containing vlookup formulas, you should always turn off calculation for copy/paste. Or before sorting.
 
Upvote 0
All,

I have determined the cause of the slow copy and paste. It is the "+5" in the below formula:
=INDEX(Year_Built_Table_THS_Proposed,MATCH($C4,Year_Built_RowHeaders_THS_Proposed,0),MATCH($G$1-$U4,Year_Built_ColHeaders_THS_Proposed,1)+5)

No idea why that would be the issue but removing the "+5" solves the problem.
 
Upvote 0

Forum statistics

Threads
1,224,813
Messages
6,181,109
Members
453,021
Latest member
Justyna P

We've detected that you are using an adblocker.

We have a great community of people providing Excel help here, but the hosting costs are enormous. You can help keep this site running by allowing ads on MrExcel.com.
Allow Ads at MrExcel

Which adblocker are you using?

Disable AdBlock

Follow these easy steps to disable AdBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the icon in the browser’s toolbar.
2)Click on the "Pause on this site" option.
Go back

Disable AdBlock Plus

Follow these easy steps to disable AdBlock Plus

1)Click on the icon in the browser’s toolbar.
2)Click on the toggle to disable it for "mrexcel.com".
Go back

Disable uBlock Origin

Follow these easy steps to disable uBlock Origin

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back

Disable uBlock

Follow these easy steps to disable uBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back
Back
Top