Skip to content

Latest commit

 

History

History

experiments

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 

Computational experiments using exact SCIP and VIPR

This document contains supplementary information on the empirical experiments conducted for the article

Kevin K.H. Cheung, Ambros Gleixner, and Daniel E. Steffy: Verifying Integer Programming Results, ZIB-Report 16-58, November 2016, urn:nbn:de:0297-zib-61044.

Experimental setup

We created a modified version of the exact rational MIP solver SCIP described by Cook et al. 2013 and used it to compute certificates for several MIP instances from the literature. Exact SCIP uses a hybrid of floating-point and exact rational arithmetic to efficiently compute exact solutions using a pure branch-and-bound algorithm. In our experiments, the rational MIP solver uses CPLEX 12.6.0.0 as its underlying floating-point LP solver and a patched version of QSopt_ex 2.5.10 as its underlying exact LP solver. The exact MIP solver supports several methods for computing valid dual bounds. Our certificate printing functionality is currently supported by the project-and-shift method (for dual feasible LP relaxations) and the exact LP method (for dual feasible LP relaxations and Farkas proofs). For details on these methods see Cook et al. 2013. Future plans are to include certificate printing functionality in all dual bounding methods and release this in subsequent versions of exact SCIP. Our development version is currently available from the authors by request.

We ran both exact SCIP with default settings and our modified SCIP version with a time limit of 3,600 seconds each and a memory limit of 10GB on the certificate file size. On the certifcate files generated by our modified version of SCIP, we called viprttn -t to tighten the certificates and viprchk to verify their correctness. Note that, although we cannot verify the precise optimal value of the MIP instance when SCIP hits the time or memory limit, we produce and verify a consistent certificate file for the primal and dual bounds available at this point of the solution process.

All experiments were conducted on a cluster of Intel(R) Xeon(R) E5-2660 v3 CPUs running at 2.60GHz with 30GB RAM. Jobs were run exclusively to ensure accurate time measurement.

Computational results

As a test bed, we considered the easy and numerically difficult (below referred to as hard) test sets from Cook et al. 2013. These test sets consist of 107 publically available instances from well-known libraries including MIPLIB 2003, MIPLIB 3, MIPLIB 2010, COR@L, and Hans Mittelmann. We had to exclude only one instance (nw04) because viprttn ran out of memory.

Aggregate results

Each certificate file could be verified as correct: viprchk did not exhibit any case in which exact SCIP pruned an optimal solution or incorrectly accepted an infeasible solution in exact rational arithmetic. Aggregate statistics of the computational performance are given in ZIB-Report 16-58 and the following table:

Test set inst SCIP solved SCIP time (s) SCIP+C solved SCIP+C time (s) ttn time (s) chk time (s) raw size (MB) ttn size (MB) gz size (MB)
easy-all 56 53 62.0 39 180.8 25.8 28.9 214 72 22
easy-solved 39 39 23.2 39 48.0 9.6 13.4 77 34 10
easy-memout 5 4 600.6 0 1769.4 377.5 169.8 10286 513 159
easy-timeout 12 10 357.4 0 3600.0 83.7 97.5 1151 368 108
hard-all 50 23 725.1 14 976.6 31.2 15.1 372 38 11
hard-solved 13 13 22.9 13 40.8 7.1 6.3 49 15 5
hard-memout 10 0 3600.0 0 1833.9 275.7 53.8 10269 146 39
hard-timeout 27 10 1811.1 1 3255.1 20.7 11.9 286 35 9

Here, the SCIP columns report on the performance of the exact version of SCIP, using its default dual bounding strategy. The SCIP+C columns state the performance for the version of exact SCIP that generates certificates during the solution process. Since certificate printing is not supported for all dual bounding methods it uses only a subset of the dual bounding methods, contributing to its slower average speed. In an independent experiment, we estimated the average overhead for collecting dual LP solutions and file input and output to be about 7% over these test sets, which is included in the time reported for SCIP+C.

Columns inst and solved refer to the number of instances in the subset and the number of instances solved to optimality or proven infeasibility by SCIP and SCIP+C, respectively. The last five columns report time for tightening and checking certificate files and the sizes of the certifcate files: before tightening, after tightening, and after running them through gzip compression.

The rows labeled all contain all instances of the easy and hard set, respectively. The solved rows comprise the instances solved by both SCIP versions. The memout rows give averages over the instances where the certificate produced by SCIP-C hit the file size limit of 10GB. The timeout row contains the remaining instances. Average running times and certificate file sizes were computed as shifted geometric means with a shift of 10 seconds and 1MB, respectively.

Finally, in the following tables, we report instance-wise results on the time, nodes, and and memory required to produce and verify certificates. Note that file sizes are stated in kB in these tables. Additionally, the final column states the range of primal and dual bound verified for the instance, or infeasible for the three unsatisfiable MIPs alu10_5, alu16_2, and opti_157_0. The instance names link to gzipped certificates for the proven primal and dual bounds stated in the last column.

Results for 56 numerically easy MIPs from Cook et al. 2013

Instance SCIP nodes SCIP time (s) SCIP+C nodes SCIP+C time (s) ttn time (s) chk time (s) raw size (kB) tightened size (kB) gzipped size (kB) verified range [dualbound, primalbound]
30:70:4_5:0_95:100 549 188.3 321 116.7 0.5 0.4 8488 3888 500 [3, 3]
acc-0 63 10.7 63 11.1 0.1 0.1 464 476 68 [0, 0]
acc-1 50 16.6 48 8.2 0.1 0.1 640 664 88 [0, 0]
acc-2 65 28.1 80 15.9 0.1 0.1 824 744 96 [0, 0]
air03 11 1.1 21 2.1 0.3 1.5 4552 3672 1128 [340160, 340160]
air05 653 26.2 26964 3600.0 333.7 426.1 3293068 880364 316200 [7217371784160637/274877906944, 26536]
bc1 44528 3600.0 4667 3600.0 7.1 81.5 1127016 468912 127264 [7677622021646343/4503599627370496, 509779105561676040950537410251567595449186579772827333490018700226416986215080747642965436704233842514333106675453237057/152703338324330694722974844929002695674318892373641886227384705977426250447047855421387906252680844296971834035564360269]
bell3a 65134 51.4 57522 60.3 20.0 21.0 535820 262324 25460 [219607579/250, 219607579/250]
bell5 413280 222.3 41798 33.3 10.1 7.6 301620 94384 12088 [28020020286/3125, 28020020286/3125]
bienst1 12730 35.5 13308 53.2 11.5 14.8 396564 223788 35936 [187/4, 187/4]
bienst2 130318 494.6 130286 648.0 183.7 173.2 4579632 2586104 499740 [273/5, 273/5]
blend2 8827 14.6 5023 14.4 3.3 3.6 174748 58192 18272 [1519797/200000, 1519797/200000]
dano3_3 47 254.2 15 3600.0 0.4 1.5 77136 17296 4536 [5068857164776087/8796093022208, 6589775704487905050110770026286969379461979446598769419744790255440005736228788540991415781158356452754204670278380019058141209966434704266446177356135997272060256249092620464129512648317495443417360287/11432318575327552324477631004975495533432641145022022129256961419612508472812576776037186454949311211259399712218543935603219466111928952029438441229110711213217229923602353376452873487155258705448000]
dano3_4 27 189.2 180 3600.0 4.0 29.5 900060 390344 103960 [316893624371107/549755813888, 1148657247713192226440847917316426522651303694883397150229092774351699319089184144898848926431555267350949163345318733454397895738349771615655166740670704384599773207385027081234662909968064627/1991842246937951258182079729901639848023338281929864053027065669347921319647701735968702723794379481044749138761996504856056032972580515466309503291105966248265239629331492728275071099653250]
dano3_5 323 3231.6 13 3600.0 0.4 1.5 67132 16880 4432 [5068857164776087/8796093022208, inf)
dcmulti 2192 6.0 2115 7.5 1.9 2.4 63360 23052 6480 [188182, 188182]
egout 8832 4.0 8337 8.6 2.0 1.8 65532 19944 3808 [5681007/10000, 5681007/10000]
eilD76 14341 95.5 75714 3600.0 267.2 425.7 4852640 1538060 496964 [441702135556829/549755813888, 456246431/500000]
enigma 566 0.2 566 0.4 0.1 0.1 1008 12 4 [0, 0]
flugpl 4302 0.5 4204 1.0 0.2 0.1 3376 676 168 [1201500, 1201500]
gen 1005 2.4 491 3.8 0.7 0.6 32336 6400 1100 [56156681359/500000, 56156681359/500000]
gesa3 5633 54.0 6050 70.0 15.7 15.2 1260436 240188 52716 [125881983070952346961091799641922753/4497223795921220170000000000, 125881983070952346961091799641922753/4497223795921220170000000000]
gesa3_o 5765 51.8 5615 62.4 14.2 14.0 1162740 226528 45328 [125881983070952346961091799641922753/4497223795921220170000000000, 125881983070952346961091799641922753/4497223795921220170000000000]
irp 18730 164.5 11520 3600.0 716.2 633.6 5103920 2143688 714112 [3337159806685465/274877906944, 2432196571/200000]
khb05250 2816 0.8 2816 8.2 3.6 5.9 34516 15748 4084 [106940226, 106940226]
l152lav 1140 3.7 1140 19.5 3.1 9.0 155552 55724 15552 [4722, 4722]
lseu 35116 3.0 33224 20.6 5.1 8.6 156168 64624 19124 [1120, 1120]
markshare1_1 22633808 3600.0 6099519 2306.1 433.4 0.1 10597168 8 4 [0, 8387/398953428]
markshare4_0 1297895 137.1 1297895 334.1 47.2 127.0 1489740 1377988 424928 [1, 1]
mas284 29342 10.1 43371 841.8 15.0 418.8 2949580 1217628 332396 [457028618411/5000000, 457028618411/5000000]
mas76 730148 98.9 374461 2017.3 132.1 368.8 10510396 1991808 558652 [5448098594849289/137438953472, 10320245343/250000]
misc03 810 0.5 810 1.5 0.2 0.4 5924 2684 724 [3360, 3360]
misc07 25144 16.8 26080 98.8 13.9 26.6 361292 175844 47236 [2810, 2810]
mod008 14743 1.9 10422 30.0 4.9 14.6 235704 114780 35848 [307, 307]
mod010 231 1.9 90045 1755.4 399.9 813.4 2427720 1275876 460824 [6548, 6548]
mod011 33077 3600.0 26892 3600.0 497.8 360.5 8442864 2483232 702640 [-3838009375069719/67108864, -81482099889824797479/1705625000000]
neos11 16704 971.5 17429 1157.6 35.0 31.8 2468120 542452 83992 [9, 9]
neos21 10538 76.3 188737 3600.0 131.5 73.1 903064 336512 94948 [6582478788394115/1125899906842624, 7]
neos5 2332860 1004.2 1864282 3054.7 420.5 678.8 10548348 2615112 616192 [8303511812463569/562949953421312, 15]
neos8 1180 233.5 2875 3600.0 4.2 1.8 52608 18952 2540 [-7521/2, inf)
neos897005 12 139.2 92 214.8 2.6 1.0 28876 7916 1676 [14, 14]
nug08 15 14.8 13 14.3 0.1 0.2 1660 1084 284 [214, 214]
p0033 1376 0.1 1381 0.4 0.1 0.1 1332 412 68 [3089, 3089]
p0201 379 0.5 394 1.0 0.2 0.5 5172 2496 560 [7615, 7615]
pk1 721285 998.8 763306 1321.5 194.9 281.8 8105196 1833072 490880 [11, 11]
qap10 9 49.6 248 855.1 1.1 2.4 34396 12936 4044 [340, 340]
qnet1_o 1531 2.8 1640 18.3 3.6 4.2 160072 51976 15252 [16029692681/1000000, 16029692681/1000000]
ran13x13 797429 1312.8 517808 1076.9 560.2 263.6 10501992 2635972 701488 [7063355329680749/2199023255552, 3311]
rentacar 61 12.3 171 28.4 0.5 0.7 16128 3632 1108 [61302410414087064221219/2019398922240000, 61302410414087064221219/2019398922240000]
rgn 5161 11.8 4181 15.2 1.3 1.7 45076 16312 2964 [2054999981/25000000, 2054999981/25000000]
stein27 4345 0.6 4345 2.0 0.3 0.5 7892 3424 540 [18, 18]
stein45 50206 13.9 50206 49.9 9.0 14.3 259776 128268 17484 [30, 30]
swath1 15737 352.2 46957 3600.0 497.6 379.5 6376408 1984748 666204 [6203932979709361/17592186044416, 7685346609/20000000]
swath2 20269 525.3 46508 3600.0 489.1 280.8 6306004 1471424 494116 [6176554855184325/17592186044416, 7815448301/20000000]
vpm1 307794 48.3 307794 583.9 298.3 277.3 6132956 2997324 227164 [20, 20]
vpm2 780423 150.9 466524 1130.6 554.7 250.7 10505052 2913832 477432 [7437176122954717/562949953421312, 55/4]

Results for 50 numerically difficult MIPs from Cook et al. 2013

Instance SCIP nodes SCIP time (s) SCIP+C nodes SCIP+C time (s) tightening time (s) checking time (s) raw size (kB) tightened size (kB) gzipped size (kB) verified range [dualbound, primalbound]
alu10_1 1630142 3600.0 884351 2682.4 852.2 119.6 10499492 894736 65152 [2544271988888683/2199023255552, inf)
alu10_5 3661 2.1 4075 13.4 2.1 0.2 111884 1376 128 infeasible
alu10_7 5951049 3600.0 697972 2279.9 797.6 69.7 10495864 648644 76204 [7703588447558093/70368744177664, inf)
alu10_8 5626153 3600.0 347382 1333.9 384.3 40.0 10492720 351032 53268 [1894667407139053/17592186044416, inf)
alu10_9 6182364 3600.0 430245 1489.7 444.4 50.7 10495812 542072 89236 [7600158590118817/70368744177664, inf)
alu16_1 2227840 3600.0 712994 2613.6 584.3 34.1 10494324 284984 50596 [91442538121529/2147483648, inf)
alu16_2 63 3.3 63 0.6 0.1 0.1 1224 200 24 infeasible
alu16_5 419052 3600.0 362067 3600.0 369.8 29.7 7155180 25860 3864 [6155442237676161/16777216, inf)
alu16_7 106094 3600.0 447490 1734.0 380.6 0.1 10498452 364 52 [439803675788291/8796093022208, inf)
alu16_8 5293381 3600.0 217446 807.5 265.8 0.1 10490860 304 44 [-496378337539802944, inf)
alu16_9 1333351 3600.0 403935 2561.6 454.7 0.1 10493788 304 44 [-293196311638809/536870912, inf)
bernd2 2242 3600.0 26 3600.0 0.5 2.5 179824 45972 14156 [7086424560532419/137438953472, inf)
cnr_dual_mip1 41355 3064.8 6231 3600.0 190.2 123.1 2589424 417812 82028 [7917105942922069/134217728, 29918065150113917/500000000]
cnr_heur_mip1 22708 1551.4 147 3600.0 2.4 2.4 76344 19372 4304 [7897528697417871/134217728, inf)
dfn6_load 19645 3600.0 11721 3600.0 3.2 4.4 432628 82288 22708 [7625009310040395/2251799813685248, inf)
dfn6f_cost 47163 3600.0 145 3600.0 0.1 0.1 9304 1264 176 [800, inf)
dfn6fp_cost 50273 3600.0 181 3600.0 0.4 0.4 25672 4704 1256 [7036972962610833/8796093022208, inf)
dfn6fp_load 21406 3600.0 90 3600.0 0.3 1.3 83056 23220 9240 [1788417086098823/281474976710656, inf)
ilp_sh5 12021 3600.0 440 2512.5 21.8 902.4 10614908 9206896 3156672 [5783875749308023/4398046511104, inf)
ilp_sh6 10515 3600.0 401 1496.9 20.7 846.2 10582972 8601020 3022588 [5849140831198403/4398046511104, inf)
neos-1053591 243537 3030.0 222824 2525.0 420.6 162.4 10489696 2259976 338972 [-4027416974360919/1099511627776, -4578643/1250]
neos-1062641 806 62.2 123 8.0 0.1 0.1 472 348 64 [0, 0]
neos-1367061 876 3600.0 1331 3600.0 55.9 1.3 1088060 27636 2432 [8407520959488351/268435456, inf)
neos-1603965 52 3600.0 39 3600.0 0.5 0.6 12988 8312 848 [5194598256532745/8388608, inf)
neos-522351 31472 229.3 29914 281.5 110.7 54.1 3598564 757064 106952 [4472769279/250000, 4472769279/250000]
neos-619167 41573 3600.0 270 3600.0 0.4 0.2 33724 2980 328 [3342063234358541/2251799813685248, inf)
neos-799716 662 3600.0 2705 3600.0 2.6 0.5 41484 6636 1116 [5296414793476493/1073741824, 348369865482/70625]
neos-839838 53218 2722.5 14551 3600.0 196.4 29.1 3871272 252008 85996 [3630582153038007/268435456, 216154648669/2000]
neumaiershcherbina 5 0.1 5 0.1 0.1 0.1 12 8 4 [-2, -2]
normalized-aim-200-1_6-yes1-3 13271 26.1 13795 73.8 6.7 1.3 325452 10992 1528 [200, 200]
npmv07 20 3600.0 2 3600.0 2.0 5.5 66420 63376 14288 [3434403169950365/32768, inf)
ns1629327 40193 1491.5 5091 3600.0 20.2 10.4 973328 71656 21848 [-1582671328730437/140737488355328, inf)
ns1770598 9001 38.9 4841 593.2 23.9 321.2 10499104 3113440 756040 [3241814004977687/137438953472, 89834353929974953382923610590747242935159173444391256802327665676316204815539763065760508542311794737181077558299227732351929735234406318219580335312630112135112834127664065830445059526404979965208963482226734722452432847263457002478183396711660658623847481278452527348213655658438044433468661993638518105568090351661835885769730044066113913158513874849135968248655381190429834311692704341489870113679068570471340793946694720426548997505133020895986294235469578597302314870180978311152054113868383892871889291430282083384972182392282800955416024898881798840342404506991103026319584359201785114855735141968909416466402913845247410157956085184276012449116579667484276143004298513396703306238042628011167294091326596944161325831198051010532298825922441801019577096143193949495154777436424621213630190657907558014619753764602511056621703405246227737829971801164951581182490885308624847792961577888932676258996667732517892019761818717196788874031766211756355294319100587231867124885716329615143636553299490438185336467581755709922430092975524618744393431202707607875364476611938937301923730572577071153942267024578221/3194161461050852704307502106391729430472763027318339131230473113410199213995688417624274320334180040072604686198494803919807307749535485168880498458771658080453522816508393610766056099212203609043478063738566604237910046315125165002373275256325064584772511348671803693358961156758335269966847078607611945124053145392280607367957523780425969306541828046604915192932150958146414814614413300571309009293932930914092300083963547165652785481808562307502710207175132794694920323971328968051919139441766101658693433983540758019074009411782956820685080520790119285502873753055200335575839504129464533903054076074508282119762345100929212285323833752916857712903989368798284908561003651862813379956142883569002813457643252841356061740351050779954765860039064193111701481710707026489021488761153826695974721208846687378979946563889508432192891950006366834486731973726433522234484203441294758974332258312511663359407549704983196821429242584727458778058579001134641216832353890745953002987110739163093245516357022930118771766825092401115417441838322217268594393508608026273582431962716864188177976509106072401881700000000
ns1859355 193802 285.3 21074 3600.0 1.0 0.1 29132 1436 192 [0, inf)
ns1866531 168 11.1 183 10.5 0.1 0.1 2564 740 152 [10, 10]
ns1900685 34498 6.5 29816 31.8 8.6 5.3 191872 48920 7540 [3453, 3453]
ns1925218 319880 3600.0 128811 3600.0 238.1 0.4 7023868 4244 688 [6931675893999861/34359738368, inf)
ns2017839 2188502 3600.0 1 3600.0 0.6 1.6 19964 20784 3244 [4929640932321283/64, inf)
ns2080781 72617 3600.0 27922 3600.0 6.4 0.1 393360 172 20 [0, inf)
opti_157_0 119 2.9 3 1.2 0.2 0.2 2704 2780 440 infeasible
p4 1 184.8 938 3600.0 66.7 18.0 1193552 73520 16308 [4014352503518715/8, inf)
prodplan1 1 3600.0 2 3600.0 2.7 7.4 96728 91208 16412 [-7325416861334513/134217728, inf)
prodplan2 4 2.5 27 1401.1 0.9 0.7 28984 9532 1624 [-185497557187228867655290921686042318951/774845425502612687685652190000000, -185497557187228867655290921686042318951/774845425502612687685652190000000]
ran14x18 2188502 3600.0 4620 3600.0 3.5 3.8 264592 35316 11064 [1907094538583883/549755813888, 4864]
sp98ir 24636 298.4 28091 3600.0 77.9 61.5 1033008 164252 59884 [7318013340697871/33554432, 225571452]
tkat3K 119250 266.9 6118 60.1 12.5 14.3 574632 217744 52440 [47728181/10, 47728181/10]
tkat3T 8074 29.6 7984 115.2 20.8 36.4 1087572 547192 143372 [22259567/4, 22259567/4]
tkat3TV 15533 73.2 6370 104.0 18.7 28.3 891956 417988 109240 [167767973/20, 167767973/20]
tkatTV5 2432005 3600.0 40038 2036.3 148.5 156.3 1287316 526508 181960 [1124705769/40, 1124705769/40]
x01 1 58.6 10 3600.0 1.5 2.3 35196 19936 4580 [5522129193649787/262144, inf)