From 7 hour "perf record" on 2104.06.24: doug@s15:~/temp$ ../c/perf_extract2 s316rc2.10 s316rc2.10e perf_extract: Begin... perf_extract: mperf too large at 161626.630259 mperf: 110225998 tsc: 491200004 c0: 22.44 delta t: 0.144 perf_extract: mperf too large at 177159.683954 mperf: 148375416 tsc: 5307709260 c0: 2.80 delta t: 1.555 perf_extract: mperf too large at 177255.082255 mperf: 106670240 tsc: 4502661740 c0: 2.37 delta t: 1.319 perf_extract: mperf too large at 177286.717805 mperf: 104660524 tsc: 559454950 c0: 18.71 delta t: 0.164 perf_extract: Number of samples: 50594 Delta time: Min: 0.019915 Max: 4.009416 perf_extract: dumping to output file 's316rc2.10e' perf_extract: Done. Exiting... doug@s15:~/temp$ ../c/perf_extract2 s316rc2.11 s316rc2.11e perf_extract: Begin... perf_extract: mperf too large at 177139.333118 mperf: 90359464 tsc: 95514460 c0: 94.60 delta t: 0.028 perf_extract: mperf too large at 177218.541895 mperf: 97227496 tsc: 382061974 c0: 25.45 delta t: 0.112 perf_extract: Number of samples: 34169 Delta time: Min: 0.019953 Max: 2.018700 perf_extract: dumping to output file 's316rc2.11e' perf_extract: Done. Exiting... doug@s15:~/temp$ ../c/perf_extract2 s316rc2.12 s316rc2.12e perf_extract: Begin... perf_extract: Number of samples: 24935 Delta time: Min: 0.019904 Max: 4.013434 perf_extract: dumping to output file 's316rc2.12e' perf_extract: Done. Exiting... doug@s15:~/temp$ ../c/perf_extract2 s316rc2.13 s316rc2.13e perf_extract: Begin... perf_extract: mperf too large at 163195.747403 mperf: 102782904 tsc: 791363314 c0: 12.99 delta t: 0.232 perf_extract: mperf too large at 163210.126104 mperf: 126214801 tsc: 723161417 c0: 17.45 delta t: 0.212 perf_extract: mperf too large at 163218.744531 mperf: 128608360 tsc: 641261087 c0: 20.06 delta t: 0.188 perf_extract: Number of samples: 30582 Delta time: Min: 0.019919 Max: 2.010725 perf_extract: dumping to output file 's316rc2.13e' perf_extract: Done. Exiting... doug@s15:~/temp$ ../c/perf_extract2 s316rc2.14 s316rc2.14e perf_extract: Begin... perf_extract: mperf too large at 162427.580258 mperf: 90411940 tsc: 1473697586 c0: 6.14 delta t: 0.432 perf_extract: mperf too large at 162429.055304 mperf: 199947310 tsc: 3670374586 c0: 5.45 delta t: 1.075 perf_extract: mperf too large at 176541.927532 mperf: 112487222 tsc: 6822251844 c0: 1.65 delta t: 1.999 perf_extract: mperf too large at 176545.924947 mperf: 112421656 tsc: 6822255160 c0: 1.65 delta t: 1.999 perf_extract: mperf too large at 177284.974920 mperf: 102473572 tsc: 354756116 c0: 28.89 delta t: 0.104 perf_extract: mperf too large at 177537.583543 mperf: 101303948 tsc: 463913336 c0: 21.84 delta t: 0.136 perf_extract: Number of samples: 30686 Delta time: Min: 0.019950 Max: 2.014724 perf_extract: dumping to output file 's316rc2.14e' perf_extract: Done. Exiting... doug@s15:~/temp$ ../c/perf_extract2 s316rc2.15 s316rc2.15e perf_extract: Begin... perf_extract: mperf too large at 177379.181990 mperf: 104157430 tsc: 313819808 c0: 33.19 delta t: 0.092 perf_extract: Number of samples: 35434 Delta time: Min: 0.019950 Max: 2.014697 perf_extract: dumping to output file 's316rc2.15e' perf_extract: Done. Exiting... doug@s15:~/temp$ ../c/perf_extract2 s316rc2.16 s316rc2.16e perf_extract: Begin... perf_extract: Number of samples: 39473 Delta time: Min: 0.019936 Max: 1.327147 perf_extract: dumping to output file 's316rc2.16e' perf_extract: Done. Exiting... Note: a lot of these, below, are because I was running "consume" under conditions that have a higher probability of manifesting the issue. doug@s15:~/temp$ ../c/perf_extract2 s316rc2.17 s316rc2.17e perf_extract: Begin... perf_extract: mperf too large at 174285.410980 mperf: 90058902 tsc: 13644511432 c0: 0.66 delta t: 3.997 perf_extract: mperf too large at 174289.408393 mperf: 96452104 tsc: 13644506126 c0: 0.71 delta t: 3.997 perf_extract: mperf too large at 174293.405809 mperf: 136610428 tsc: 13644509910 c0: 1.00 delta t: 3.997 perf_extract: mperf too large at 174297.403223 mperf: 163568354 tsc: 13644506332 c0: 1.20 delta t: 3.997 perf_extract: mperf too large at 174300.864957 mperf: 139649426 tsc: 11816058280 c0: 1.18 delta t: 3.462 perf_extract: mperf too large at 174321.387710 mperf: 228001243 tsc: 10069743582 c0: 2.26 delta t: 2.950 perf_extract: mperf too large at 174325.385126 mperf: 362304172 tsc: 13644510352 c0: 2.66 delta t: 3.997 perf_extract: mperf too large at 174329.382507 mperf: 286000376 tsc: 10765497602 c0: 2.66 delta t: 3.154 perf_extract: mperf too large at 174333.379953 mperf: 408178054 tsc: 13644614640 c0: 2.99 delta t: 3.997 perf_extract: mperf too large at 174337.377367 mperf: 430640120 tsc: 13644506152 c0: 3.16 delta t: 3.997 perf_extract: mperf too large at 174341.374782 mperf: 442101096 tsc: 13644508992 c0: 3.24 delta t: 3.997 perf_extract: mperf too large at 174345.372198 mperf: 498963004 tsc: 13644507590 c0: 3.66 delta t: 3.997 perf_extract: mperf too large at 174349.369612 mperf: 498232173 tsc: 13644511652 c0: 3.65 delta t: 3.997 perf_extract: mperf too large at 174353.367027 mperf: 543975269 tsc: 13644504682 c0: 3.99 delta t: 3.997 perf_extract: mperf too large at 174357.364443 mperf: 566277781 tsc: 13644515826 c0: 4.15 delta t: 3.997 perf_extract: mperf too large at 174361.361858 mperf: 577230698 tsc: 13644508228 c0: 4.23 delta t: 3.997 perf_extract: mperf too large at 174365.359271 mperf: 633964654 tsc: 13644504682 c0: 4.65 delta t: 3.997 perf_extract: mperf too large at 174369.356684 mperf: 633293460 tsc: 13644506782 c0: 4.64 delta t: 3.997 perf_extract: mperf too large at 174370.831701 mperf: 233660828 tsc: 5034726672 c0: 4.64 delta t: 1.475 perf_extract: mperf too large at 174381.348927 mperf: 108904269 tsc: 1855756553 c0: 5.87 delta t: 0.544 perf_extract: mperf too large at 174382.360246 mperf: 192651854 tsc: 3451968928 c0: 5.58 delta t: 1.011 perf_extract: mperf too large at 174385.346342 mperf: 577491457 tsc: 10192539886 c0: 5.67 delta t: 2.986 perf_extract: mperf too large at 174389.343756 mperf: 703947533 tsc: 12430241254 c0: 5.66 delta t: 3.642 perf_extract: mperf too large at 174390.822771 mperf: 285715195 tsc: 5048372748 c0: 5.66 delta t: 1.479 perf_extract: mperf too large at 174433.315319 mperf: 144555066 tsc: 1801175916 c0: 8.03 delta t: 0.528 perf_extract: mperf too large at 174437.312732 mperf: 1109977388 tsc: 13644507336 c0: 8.13 delta t: 3.997 perf_extract: mperf too large at 174440.798448 mperf: 969543444 tsc: 11897912273 c0: 8.15 delta t: 3.486 perf_extract: mperf too large at 174453.302391 mperf: 786272231 tsc: 8568843306 c0: 9.18 delta t: 2.510 perf_extract: mperf too large at 174457.299806 mperf: 1245841844 tsc: 13644510914 c0: 9.13 delta t: 3.997 perf_extract: mperf too large at 174460.789517 mperf: 1091012421 tsc: 11911552095 c0: 9.16 delta t: 3.490 perf_extract: mperf too large at 174481.284292 mperf: 195984384 tsc: 1801182730 c0: 10.88 delta t: 0.528 perf_extract: mperf too large at 174485.281707 mperf: 1450267286 tsc: 13644509714 c0: 10.63 delta t: 3.997 perf_extract: mperf too large at 174488.215809 mperf: 1064940570 tsc: 10015067956 c0: 10.63 delta t: 2.934 perf_extract: mperf too large at 174489.279122 mperf: 384131680 tsc: 3629442090 c0: 10.58 delta t: 1.063 perf_extract: mperf too large at 174490.774123 mperf: 541873672 tsc: 5102941370 c0: 10.62 delta t: 1.495 perf_extract: mperf too large at 174505.268780 mperf: 468371568 tsc: 3984192166 c0: 11.76 delta t: 1.167 perf_extract: mperf too large at 174509.266195 mperf: 1585484636 tsc: 13644510182 c0: 11.62 delta t: 3.997 perf_extract: mperf too large at 174510.765194 mperf: 595293086 tsc: 5116586466 c0: 11.63 delta t: 1.499 perf_extract: mperf too large at 174533.250682 mperf: 385522264 tsc: 2960855524 c0: 13.02 delta t: 0.867 perf_extract: mperf too large at 174537.248100 mperf: 1790713310 tsc: 13644519422 c0: 13.12 delta t: 3.997 perf_extract: mperf too large at 174537.463958 mperf: 98628632 tsc: 736794748 c0: 13.39 delta t: 0.216 perf_extract: mperf too large at 174540.749801 mperf: 1474738712 tsc: 11215681326 c0: 13.15 delta t: 3.286 perf_extract: mperf too large at 177577.209912 mperf: 103922664 tsc: 382039872 c0: 27.20 delta t: 0.112 perf_extract: mperf too large at 182096.359096 mperf: 226241686 tsc: 13644507452 c0: 1.66 delta t: 3.997 perf_extract: mperf too large at 182099.776862 mperf: 196411238 tsc: 11665976198 c0: 1.68 delta t: 3.418 perf_extract: mperf too large at 182124.340999 mperf: 149164136 tsc: 4912123566 c0: 3.04 delta t: 1.439 perf_extract: mperf too large at 182125.404282 mperf: 111743213 tsc: 3629337722 c0: 3.08 delta t: 1.063 perf_extract: mperf too large at 182128.338413 mperf: 302235698 tsc: 10015168966 c0: 3.02 delta t: 2.934 perf_extract: mperf too large at 182132.335829 mperf: 458607976 tsc: 13644512680 c0: 3.36 delta t: 3.997 perf_extract: mperf too large at 182136.333243 mperf: 479275952 tsc: 13644505100 c0: 3.51 delta t: 3.997 perf_extract: mperf too large at 182140.330658 mperf: 491255416 tsc: 13644509052 c0: 3.60 delta t: 3.997 perf_extract: mperf too large at 182144.328073 mperf: 546004264 tsc: 13644508278 c0: 4.00 delta t: 3.997 perf_extract: mperf too large at 182148.325488 mperf: 516645970 tsc: 13644508914 c0: 3.79 delta t: 3.997 perf_extract: mperf too large at 182152.322904 mperf: 530451946 tsc: 13644510922 c0: 3.89 delta t: 3.997 perf_extract: mperf too large at 182156.320318 mperf: 523335400 tsc: 13644509018 c0: 3.84 delta t: 3.997 perf_extract: mperf too large at 182160.317732 mperf: 509742708 tsc: 13644509416 c0: 3.74 delta t: 3.997 perf_extract: mperf too large at 182164.315146 mperf: 553425994 tsc: 13644502406 c0: 4.06 delta t: 3.997 perf_extract: mperf too large at 182168.312560 mperf: 533044094 tsc: 13644510190 c0: 3.91 delta t: 3.997 perf_extract: mperf too large at 182169.743604 mperf: 189674102 tsc: 4884634110 c0: 3.88 delta t: 1.431 perf_extract: mperf too large at 182172.309974 mperf: 375392764 tsc: 8759873114 c0: 4.29 delta t: 2.566 perf_extract: mperf too large at 182176.307389 mperf: 581093968 tsc: 13644509416 c0: 4.26 delta t: 3.997 perf_extract: mperf too large at 182178.102201 mperf: 261502046 tsc: 6126290970 c0: 4.27 delta t: 1.795 perf_extract: mperf too large at 182180.304804 mperf: 328999680 tsc: 7518218830 c0: 4.38 delta t: 2.203 perf_extract: mperf too large at 182184.302218 mperf: 632037732 tsc: 13644508184 c0: 4.63 delta t: 3.997 perf_extract: mperf too large at 182188.299632 mperf: 632023008 tsc: 13644506936 c0: 4.63 delta t: 3.997 perf_extract: mperf too large at 182189.734675 mperf: 227699398 tsc: 4898279918 c0: 4.65 delta t: 1.435 perf_extract: mperf too large at 182208.286708 mperf: 590870990 tsc: 10942894458 c0: 5.40 delta t: 3.206 perf_extract: mperf too large at 182209.745731 mperf: 259086273 tsc: 4980139304 c0: 5.20 delta t: 1.459 perf_extract: mperf too large at 182212.104204 mperf: 368829914 tsc: 6917640129 c0: 5.33 delta t: 2.027 perf_extract: mperf too large at 182213.443337 mperf: 209464441 tsc: 3956906648 c0: 5.29 delta t: 1.159 perf_extract: mperf too large at 182216.281502 mperf: 513216967 tsc: 9687601781 c0: 5.30 delta t: 2.838 perf_extract: mperf too large at 182218.584017 mperf: 415333409 tsc: 7859249168 c0: 5.28 delta t: 2.303 perf_extract: mperf too large at 182220.278950 mperf: 301034209 tsc: 5594358437 c0: 5.38 delta t: 1.639 perf_extract: mperf too large at 182224.276363 mperf: 769661375 tsc: 13644503866 c0: 5.64 delta t: 3.997 perf_extract: mperf too large at 182228.273779 mperf: 772529057 tsc: 13644514126 c0: 5.66 delta t: 3.997 perf_extract: mperf too large at 182232.271194 mperf: 800855104 tsc: 13644506706 c0: 5.87 delta t: 3.997 perf_extract: mperf too large at 182233.290537 mperf: 208189273 tsc: 3479357690 c0: 5.98 delta t: 1.019 perf_extract: mperf too large at 182236.268608 mperf: 607749371 tsc: 10165151744 c0: 5.98 delta t: 2.978 perf_extract: mperf too large at 182239.710351 mperf: 703870275 tsc: 11747818957 c0: 5.99 delta t: 3.442 perf_extract: mperf too large at 182252.258267 mperf: 565392787 tsc: 8787171236 c0: 6.43 delta t: 2.574 perf_extract: mperf too large at 182256.255681 mperf: 875016468 tsc: 13644506594 c0: 6.41 delta t: 3.997 perf_extract: mperf too large at 182259.701422 mperf: 757313805 tsc: 11761466453 c0: 6.44 delta t: 3.446 perf_extract: mperf too large at 182288.234999 mperf: 862745258 tsc: 11174858378 c0: 7.72 delta t: 3.274 perf_extract: mperf too large at 182289.686028 mperf: 366380152 tsc: 4952855002 c0: 7.40 delta t: 1.451 perf_extract: mperf too large at 182292.232414 mperf: 676476696 tsc: 8691652478 c0: 7.78 delta t: 2.546 perf_extract: mperf too large at 182296.229826 mperf: 1018744110 tsc: 13644506356 c0: 7.47 delta t: 3.997 perf_extract: mperf too large at 182300.227241 mperf: 1026408977 tsc: 13644510566 c0: 7.52 delta t: 3.997 perf_extract: mperf too large at 182304.224655 mperf: 1062773590 tsc: 13644505038 c0: 7.79 delta t: 3.997 perf_extract: mperf too large at 182308.222069 mperf: 1062837170 tsc: 13644507846 c0: 7.79 delta t: 3.997 perf_extract: mperf too large at 182309.677098 mperf: 387256391 tsc: 4966502190 c0: 7.80 delta t: 1.455 perf_extract: mperf too large at 182336.203939 mperf: 778976620 tsc: 9551150417 c0: 8.16 delta t: 2.798 perf_extract: mperf too large at 182339.661705 mperf: 958985268 tsc: 11802507652 c0: 8.13 delta t: 3.458 perf_extract: mperf too large at 182356.191045 mperf: 1107254035 tsc: 11843531322 c0: 9.35 delta t: 3.470 perf_extract: mperf too large at 182359.652776 mperf: 1112024452 tsc: 11816044062 c0: 9.41 delta t: 3.462 perf_extract: mperf too large at 182372.180704 mperf: 837775099 tsc: 8650709658 c0: 9.68 delta t: 2.534 perf_extract: mperf too large at 182376.178119 mperf: 1317147647 tsc: 13644508184 c0: 9.65 delta t: 3.997 perf_extract: mperf too large at 182379.643847 mperf: 1146874540 tsc: 11829687870 c0: 9.69 delta t: 3.466 perf_extract: mperf too large at 182400.162607 mperf: 189152948 tsc: 1814829590 c0: 10.42 delta t: 0.532 perf_extract: mperf too large at 182400.502357 mperf: 118418790 tsc: 1159681562 c0: 10.21 delta t: 0.340 perf_extract: mperf too large at 182404.160020 mperf: 1273547186 tsc: 12484822942 c0: 10.20 delta t: 3.658 perf_extract: mperf too large at 182408.157436 mperf: 1391767486 tsc: 13644511804 c0: 10.20 delta t: 3.997 perf_extract: mperf too large at 182409.628453 mperf: 513090036 tsc: 5021074092 c0: 10.22 delta t: 1.471 perf_extract: mperf too large at 182424.147094 mperf: 1470218674 tsc: 13644499668 c0: 10.78 delta t: 3.997 perf_extract: mperf too large at 182428.144509 mperf: 1470514346 tsc: 13644508040 c0: 10.78 delta t: 3.997 perf_extract: mperf too large at 182429.619524 mperf: 544205938 tsc: 5034721598 c0: 10.81 delta t: 1.475 perf_extract: mperf too large at 182456.126381 mperf: 892217796 tsc: 7900170980 c0: 11.29 delta t: 2.315 perf_extract: mperf too large at 182459.604129 mperf: 1335034336 tsc: 11870718380 c0: 11.25 delta t: 3.478 perf_extract: mperf too large at 182472.116068 mperf: 153791368 tsc: 1323614712 c0: 11.62 delta t: 0.388 perf_extract: mperf too large at 182476.113485 mperf: 1616554922 tsc: 13644511270 c0: 11.85 delta t: 3.997 perf_extract: mperf too large at 182479.595200 mperf: 1369316222 tsc: 11884264178 c0: 11.52 delta t: 3.482 perf_extract: mperf too large at 182504.095386 mperf: 300757304 tsc: 2524335704 c0: 11.91 delta t: 0.740 perf_extract: mperf too large at 182508.092803 mperf: 1624042742 tsc: 13644514856 c0: 11.90 delta t: 3.997 perf_extract: mperf too large at 182509.579807 mperf: 606560452 tsc: 5075643498 c0: 11.95 delta t: 1.487 perf_extract: mperf too large at 182520.085045 mperf: 226629084 tsc: 1814829602 c0: 12.49 delta t: 0.532 perf_extract: mperf too large at 182524.082460 mperf: 1696712128 tsc: 13644512120 c0: 12.44 delta t: 3.997 perf_extract: mperf too large at 182528.079875 mperf: 1699047278 tsc: 13644508508 c0: 12.45 delta t: 3.997 perf_extract: mperf too large at 182529.570877 mperf: 636173212 tsc: 5089292982 c0: 12.50 delta t: 1.491 perf_extract: mperf too large at 182552.064363 mperf: 1145091522 tsc: 8623445036 c0: 13.28 delta t: 2.526 perf_extract: mperf too large at 182556.061775 mperf: 1802918262 tsc: 13644500960 c0: 13.21 delta t: 3.997 perf_extract: mperf too large at 182559.555484 mperf: 1574535000 tsc: 11925196412 c0: 13.20 delta t: 3.494 perf_extract: mperf too large at 182579.546555 mperf: 908469590 tsc: 6576650742 c0: 13.81 delta t: 1.927 perf_extract: mperf too large at 182609.531161 mperf: 184022514 tsc: 1309867516 c0: 14.05 delta t: 0.384 perf_extract: mperf too large at 182649.945054 mperf: 224995416 tsc: 1487346574 c0: 15.13 delta t: 0.436 perf_extract: mperf too large at 182651.999726 mperf: 1048262296 tsc: 7013278380 c0: 14.95 delta t: 2.055 perf_extract: mperf too large at 182654.941823 mperf: 1495931314 tsc: 10042357530 c0: 14.90 delta t: 2.942 perf_extract: mperf too large at 182655.997140 mperf: 539457586 tsc: 3602149490 c0: 14.98 delta t: 1.055 perf_extract: mperf too large at 182659.506839 mperf: 1787884964 tsc: 11979775658 c0: 14.92 delta t: 3.510 perf_extract: mperf too large at 182699.968671 mperf: 284182936 tsc: 1719215710 c0: 16.53 delta t: 0.504 perf_extract: mperf too large at 182703.966085 mperf: 2202093836 tsc: 13644508676 c0: 16.14 delta t: 3.997 perf_extract: mperf too large at 182707.963500 mperf: 2202171736 tsc: 13644509052 c0: 16.14 delta t: 3.997 perf_extract: mperf too large at 182709.482516 mperf: 834011264 tsc: 5184910650 c0: 16.09 delta t: 1.519 perf_extract: Number of samples: 179208 Delta time: Min: 0.019952 Max: 4.005433 perf_extract: dumping to output file 's316rc2.17e' perf_extract: Done. Exiting... doug@s15:~/temp$