Trace number 3250648

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.45 1202.64

General information on the benchmark

Namecrafted/kullmann/VanDerWaerden/
VanderWaerden3k/VanDerWaerden_2-3-18_312.cnf
MD5SUM1144bbd60c180aa5e72d35b74e9676cb
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables312
Number of clauses26889
Sum of the clauses size121302
Maximum clause length18
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 324180
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 52709

Solver Data 

0.01/0.03	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250648-1304093011.cnf 
0.01/0.03	c Using SATTIMEOUT of 4800
0.01/0.03	c Using SATRAM of 15500
0.01/0.03	c Size of subset node = 32
0.01/0.03	c Variables = 312
0.01/0.03	c Clauses = 26889
0.01/0.03	c Literals = 121302
0.01/0.03	c Reduced size:
0.01/0.03	c Variables = 312
0.01/0.03	c Clauses = 26889
0.01/0.03	c Literals = 121302
0.01/0.03	c a2
0.09/0.16	c a1
0.29/0.37	c Independent sub-problems = 1
0.29/0.37	c Sub-problem with 312 variables and 26889 clauses
0.29/0.37	c using permutation method 4
0.29/0.37	c count = 1
0.29/0.37	c Ordering finished 0.360 seconds
0.29/0.37	c mapped 0.370 seconds
0.29/0.37	c permuted 0.370 seconds
0.29/0.37	c about to sort
0.29/0.38	c sorted
0.39/0.48	c clauses now 26889
0.39/0.48	c init finished 0.380 seconds
0.39/0.48	c tree built 0.480 seconds
0.39/0.48	c 
0.39/0.48	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.39/0.48	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.39/0.48	c ................................ | ..........                    |                                  0.5         0.0             
0.39/0.48	c ................................ | ..........                  1 |                                  0.5         2.1             
0.39/0.48	c ................................ | ..........                  2 |                                  0.5         4.2             
0.39/0.48	c ................................ | ..........                  3 |                                  0.5         6.2             
0.39/0.48	c ................................ | ..........                  4 |                                  0.5         8.3             
0.39/0.48	c ................................ | ..........                  5 |                                  0.5        10.4             
0.51/0.51	c ................................ | ..........               1000 | 58           292                 0.6      1818.2        27417
95.98/24.41	c ................................ | ..........             438000 | 56872        99794              96.0      4560.1     28225018
191.86/48.45	c ................................ | ..........             697000 | 91591        149107            192.1      3629.3     46222297
288.13/72.71	c ................................ | ..........             873000 | 118113       189823            288.2      3029.1     60081479
383.97/96.79	c ................................ | ..........            1027000 | 139461       227101            384.3      2672.2     72953449
480.22/120.83	c ................................ | *.........            1176000 | 158940       262788            480.4      2448.1     85581787
576.48/144.92	c ................................ | *.........            1303000 | 176839       295918            576.6      2259.8     96914589
671.95/168.82	c ................................ | *.........            1400000 | 192814       321087            672.0      2083.2    105471900
768.21/192.96	c ................................ | *.........            1519000 | 208231       343925            768.5      1976.7    114566053
864.09/216.91	c ................................ | *.........            1639000 | 224861       368797            864.1      1896.7    122763312
960.35/241.09	c ................................ | **........            1764000 | 242299       391652            960.7      1836.1    132331987
1056.64/265.19	c ................................ | **........            1882000 | 258679       417207           1057.0      1780.5    140951487
1152.47/289.11	c ................................ | **........            1958000 | 270831       436252           1152.5      1698.8    148683229
1248.73/313.20	c ................................ | **........            2054000 | 285082       454961           1248.8      1644.8    154634036
1344.59/337.22	c ................................ | **........            2150000 | 298253       473306           1344.7      1598.9    156178639
1440.06/361.15	c ................................ | ***.......            2242000 | 312433       490271           1440.3      1556.6    160250309
1535.93/385.17	c ................................ | ***.......            2331000 | 326296       509194           1536.2      1517.4    168314780
1632.59/409.30	c ................................ | ***.......            2431000 | 340940       529549           1632.6      1489.0    175922572
1728.88/433.41	c ................................ | ***.......            2526000 | 355743       547609           1728.9      1461.0    182042848
1824.33/457.34	c ................................ | ***.......            2617000 | 369700       568448           1824.5      1434.4    189551981
1920.21/481.30	c ................................ | ****......            2704000 | 382851       587295           1920.2      1408.2    195901678
2016.47/505.45	c ................................ | ****......            2776000 | 394741       607640           2016.7      1376.5    200637829
2112.74/529.52	c ................................ | ****......            2841000 | 405881       626981           2112.8      1344.6    202468592
2208.59/553.57	c ................................ | ****......            2902000 | 416382       643841           2208.9      1313.8    204034056
2304.87/577.65	c ................................ | ****......            2968000 | 426954       660450           2305.1      1287.6    205478548
2400.33/601.51	c ................................ | *****.....            3055000 | 440322       676437           2400.4      1272.7    212326377
2496.61/625.62	c ................................ | *****.....            3149000 | 454774       694898           2496.7      1261.3    220130900
2592.46/649.67	c ................................ | *****.....            3229000 | 466969       711958           2592.7      1245.4    222197401
2688.29/673.61	c ................................ | *****.....            3315000 | 479555       728083           2688.4      1233.1    228239166
2784.16/697.68	c ................................ | *****.....            3399000 | 492131       742618           2784.5      1220.7    235495098
2880.43/721.75	c ................................ | ******....            3477000 | 504260       757955           2880.6      1207.0    241700381
2976.32/745.76	c ................................ | ******....            3549000 | 515892       773621           2976.5      1192.3    242876729
3072.54/769.81	c ................................ | ******....            3622000 | 527006       790586           3072.6      1178.8    245674617
3168.81/793.98	c ................................ | ******....            3687000 | 536961       804753           3169.1      1163.4    246997398
3264.28/817.80	c ................................ | ******....            3756000 | 546897       821596           3264.3      1150.6    249522348
3360.55/841.94	c ................................ | *******...            3825000 | 558125       838880           3360.7      1138.1    252490673
3456.41/865.95	c ................................ | *******...            3895000 | 569980       854400           3456.6      1126.8    254381068
3552.30/889.96	c ................................ | *******...            3957000 | 578028       871549           3552.5      1113.9    255245630
3648.56/914.10	c ................................ | *******...            4035000 | 588392       889410           3648.9      1105.8    256616331
3745.93/938.46	c ................................ | *******...            4100000 | 597594       904385           3746.2      1094.4    256711892
3840.20/962.02	c ................................ | ********..            4182000 | 609339       921098           3840.3      1089.0    256711892
3936.06/986.09	c ................................ | ********..            4257000 | 618658       936506           3936.4      1081.4    256711892
4031.92/1010.06	c ................................ | ********..            4335000 | 628038       952518           4032.2      1075.1    256711892
4128.59/1034.29	c ................................ | ********..            4418000 | 638044       967166           4129.0      1070.0    256711892
4224.86/1058.32	c ................................ | ********..            4494000 | 647446       981494           4225.0      1063.7    256711892
4320.33/1082.21	c ................................ | *********.            4571000 | 657394       995669           4320.4      1058.0    256711892
4415.80/1106.16	c ................................ | *********.            4638000 | 665735       1008255          4416.1      1050.3    256711892
4512.06/1130.27	c ................................ | *********.            4702000 | 674278       1020507          4512.4      1042.0    256711892
4609.14/1154.53	c ................................ | *********.            4767000 | 683175       1033734          4609.3      1034.2    257917863
4704.99/1178.58	c ................................ | *********.            4827000 | 691355       1048117          4705.3      1025.9    260692088

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3250648-1304093011/watcher-3250648-1304093011 -o /tmp/evaluation-result-3250648-1304093011/solver-3250648-1304093011 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250648-1304093011.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 4.64 7.01 7.74 2/181 10251
/proc/meminfo: memFree=25017392/32951124 swapFree=67111352/67111528
[pid=10251] ppid=10249 vsize=9540 CPUtime=0 cores=1,3,5,7
/proc/10251/stat : 10251 (ttsth-5-4) R 10249 10251 9884 0 -1 4202496 439 0 0 0 0 0 0 0 23 0 1 0 520899251 9768960 382 33554432000 4194304 4338100 140734363195904 18446744073709551615 246516495045 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/10251/statm: 2385 383 146 36 0 290 0

[startup+0.0114809 s]
/proc/loadavg: 4.64 7.01 7.74 2/181 10251
/proc/meminfo: memFree=25017392/32951124 swapFree=67111352/67111528
[pid=10251] ppid=10249 vsize=10428 CPUtime=0.01 cores=1,3,5,7
/proc/10251/stat : 10251 (ttsth-5-4) R 10249 10251 9884 0 -1 4202496 662 0 0 0 1 0 0 0 23 0 1 0 520899251 10678272 605 33554432000 4194304 4338100 140734363195904 18446744073709551615 246516495045 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/10251/statm: 2607 606 146 36 0 512 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 10428

[startup+0.100469 s]
/proc/loadavg: 4.64 7.01 7.74 2/181 10251
/proc/meminfo: memFree=25017392/32951124 swapFree=67111352/67111528
[pid=10251] ppid=10249 vsize=27856 CPUtime=0.09 cores=1,3,5,7
/proc/10251/stat : 10251 (ttsth-5-4) R 10249 10251 9884 0 -1 4202496 4189 0 0 0 9 0 0 0 23 0 1 0 520899251 28524544 4046 33554432000 4194304 4338100 140734363195904 18446744073709551615 246516229322 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/10251/statm: 6964 4046 410 36 0 4869 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 27856

[startup+0.300428 s]
/proc/loadavg: 4.64 7.01 7.74 2/181 10251
/proc/meminfo: memFree=25017392/32951124 swapFree=67111352/67111528
[pid=10251] ppid=10249 vsize=15652 CPUtime=0.29 cores=1,3,5,7
/proc/10251/stat : 10251 (ttsth-5-4) R 10249 10251 9884 0 -1 4202496 4602 0 0 0 29 0 0 0 25 0 1 0 520899251 16027648 1924 33554432000 4194304 4338100 140734363195904 18446744073709551615 4245205 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/10251/statm: 3913 1924 155 36 0 1818 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 15652

[startup+0.700325 s]
/proc/loadavg: 4.64 7.01 7.74 2/181 10251
/proc/meminfo: memFree=25017392/32951124 swapFree=67111352/67111528
[pid=10251] ppid=10249 vsize=172728 CPUtime=1.32 cores=1,3,5,7
/proc/10251/stat : 10251 (ttsth-5-4) S 10249 10251 9884 0 -1 4202496 15763 0 0 0 130 2 0 0 25 0 5 0 520899251 176873472 12913 33554432000 4194304 4338100 140734363195904 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/10251/statm: 43182 12916 176 36 0 41087 0
Current children cumulated CPU time (s) 1.32
Current children cumulated vsize (KiB) 172728

[startup+1.50118 s]
/proc/loadavg: 4.64 7.01 7.74 11/190 10260
/proc/meminfo: memFree=24869080/32951124 swapFree=67111352/67111528
[pid=10251] ppid=10249 vsize=217708 CPUtime=4.51 cores=1,3,5,7
/proc/10251/stat : 10251 (ttsth-5-4) S 10249 10251 9884 0 -1 4202496 27352 0 0 0 446 5 0 0 25 0 5 0 520899251 222932992 24502 33554432000 4194304 4338100 140734363195904 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/10251/statm: 54427 24502 176 36 0 52332 0
[pid=10251/tid=10257] ppid=10249 vsize=217708 CPUtime=1.01 cores=1,3,5,7
/proc/10251/task/10257/stat : 10257 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 4249 0 0 0 100 1 0 0 19 0 5 0 520899300 222932992 24502 33554432000 4194304 4338100 140734363195904 18446744073709551615 246528646862 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=10251/tid=10258] ppid=10249 vsize=217708 CPUtime=0.99 cores=1,3,5,7
/proc/10251/task/10258/stat : 10258 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 3198 0 0 0 99 0 0 0 25 0 5 0 520899300 222932992 24502 33554432000 4194304 4338100 140734363195904 18446744073709551615 246516229185 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=10251/tid=10259] ppid=10249 vsize=217708 CPUtime=1 cores=1,3,5,7
/proc/10251/task/10259/stat : 10259 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 3767 0 0 0 99 1 0 0 19 0 5 0 520899300 222932992 24502 33554432000 4194304 4338100 140734363195904 18446744073709551615 4255953 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=10251/tid=10260] ppid=10249 vsize=217708 CPUtime=1 cores=1,3,5,7
/proc/10251/task/10260/stat : 10260 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 4101 0 0 0 100 0 0 0 19 0 5 0 520899300 222932992 24502 33554432000 4194304 4338100 140734363195904 18446744073709551615 4256403 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4.51
Current children cumulated vsize (KiB) 217708

[startup+3.10089 s]
/proc/loadavg: 4.64 7.01 7.74 10/190 10260
/proc/meminfo: memFree=24745452/32951124 swapFree=67111352/67111528
[pid=10251] ppid=10249 vsize=306084 CPUtime=10.9 cores=1,3,5,7
/proc/10251/stat : 10251 (ttsth-5-4) S 10249 10251 9884 0 -1 4202496 48534 0 0 0 1081 9 0 0 25 0 5 0 520899251 313430016 45684 33554432000 4194304 4338100 140734363195904 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/10251/statm: 76521 45684 176 36 0 74426 0
[pid=10251/tid=10257] ppid=10249 vsize=306084 CPUtime=2.61 cores=1,3,5,7
/proc/10251/task/10257/stat : 10257 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 9859 0 0 0 259 2 0 0 23 0 5 0 520899300 313430016 45684 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=10251/tid=10258] ppid=10249 vsize=306084 CPUtime=2.59 cores=1,3,5,7
/proc/10251/task/10258/stat : 10258 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 8256 0 0 0 258 1 0 0 25 0 5 0 520899300 313430016 45684 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=10251/tid=10259] ppid=10249 vsize=306084 CPUtime=2.6 cores=1,3,5,7
/proc/10251/task/10259/stat : 10259 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 9023 0 0 0 258 2 0 0 22 0 5 0 520899300 313430016 45684 33554432000 4194304 4338100 140734363195904 18446744073709551615 246528648451 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=10251/tid=10260] ppid=10249 vsize=306084 CPUtime=2.6 cores=1,3,5,7
/proc/10251/task/10260/stat : 10260 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 9359 0 0 0 258 2 0 0 22 0 5 0 520899300 313430016 45684 33554432000 4194304 4338100 140734363195904 18446744073709551615 4256403 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 10.9
Current children cumulated vsize (KiB) 306084

[startup+6.3003 s]
/proc/loadavg: 4.91 7.02 7.74 9/190 10260
/proc/meminfo: memFree=24446584/32951124 swapFree=67111352/67111528
[pid=10251] ppid=10249 vsize=444776 CPUtime=23.68 cores=1,3,5,7
/proc/10251/stat : 10251 (ttsth-5-4) S 10249 10251 9884 0 -1 4202496 85127 0 0 0 2353 15 0 0 25 0 5 0 520899251 455450624 82277 33554432000 4194304 4338100 140734363195904 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/10251/statm: 111194 82277 176 36 0 109099 0
[pid=10251/tid=10257] ppid=10249 vsize=444776 CPUtime=5.81 cores=1,3,5,7
/proc/10251/task/10257/stat : 10257 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 19092 0 0 0 577 4 0 0 25 0 5 0 520899300 455450624 82277 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=10251/tid=10258] ppid=10249 vsize=444776 CPUtime=5.76 cores=1,3,5,7
/proc/10251/task/10258/stat : 10258 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 17610 0 0 0 574 2 0 0 25 0 5 0 520899300 455450624 82277 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=10251/tid=10259] ppid=10249 vsize=444776 CPUtime=5.79 cores=1,3,5,7
/proc/10251/task/10259/stat : 10259 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 17433 0 0 0 576 3 0 0 25 0 5 0 520899300 455450624 82278 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=10251/tid=10260] ppid=10249 vsize=444776 CPUtime=5.8 cores=1,3,5,7
/proc/10251/task/10260/stat : 10260 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 18955 0 0 0 577 3 0 0 25 0 5 0 520899300 455450624 82278 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 23.68
Current children cumulated vsize (KiB) 444776

[startup+12.7011 s]
/proc/loadavg: 5.15 7.04 7.75 10/190 10260
/proc/meminfo: memFree=24003816/32951124 swapFree=67111352/67111528
[pid=10251] ppid=10249 vsize=677304 CPUtime=49.24 cores=1,3,5,7
/proc/10251/stat : 10251 (ttsth-5-4) S 10249 10251 9884 0 -1 4202496 142780 0 0 0 4897 27 0 0 25 0 5 0 520899251 693559296 139930 33554432000 4194304 4338100 140734363195904 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/10251/statm: 169326 139930 176 36 0 167231 0
[pid=10251/tid=10257] ppid=10249 vsize=677304 CPUtime=12.2 cores=1,3,5,7
/proc/10251/task/10257/stat : 10257 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 31527 0 0 0 1213 7 0 0 25 0 5 0 520899300 693559296 139930 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=10251/tid=10258] ppid=10249 vsize=677304 CPUtime=12.14 cores=1,3,5,7
/proc/10251/task/10258/stat : 10258 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 32410 0 0 0 1208 6 0 0 25 0 5 0 520899300 693559296 139930 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=10251/tid=10259] ppid=10249 vsize=677304 CPUtime=12.19 cores=1,3,5,7
/proc/10251/task/10259/stat : 10259 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 32903 0 0 0 1213 6 0 0 25 0 5 0 520899300 693559296 139930 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=10251/tid=10260] ppid=10249 vsize=677304 CPUtime=12.2 cores=1,3,5,7
/proc/10251/task/10260/stat : 10260 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 33903 0 0 0 1214 6 0 0 25 0 5 0 520899300 693559296 139930 33554432000 4194304 4338100 140734363195904 18446744073709551615 4239539 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 49.24
Current children cumulated vsize (KiB) 677304

[startup+25.5008 s]
/proc/loadavg: 5.78 7.09 7.75 10/190 10261
/proc/meminfo: memFree=23341892/32951124 swapFree=67111352/67111528
[pid=10251] ppid=10249 vsize=1085424 CPUtime=100.38 cores=1,3,5,7
/proc/10251/stat : 10251 (ttsth-5-4) S 10249 10251 9884 0 -1 4202496 243930 0 0 0 9992 46 0 0 25 0 5 0 520899251 1111474176 241080 33554432000 4194304 4338100 140734363195904 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/10251/statm: 271356 241080 176 36 0 269261 0
[pid=10251/tid=10257] ppid=10249 vsize=1085424 CPUtime=24.93 cores=1,3,5,7
/proc/10251/task/10257/stat : 10257 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 54613 0 0 0 2481 12 0 0 25 0 5 0 520899300 1111474176 241081 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=10251/tid=10258] ppid=10249 vsize=1085424 CPUtime=24.93 cores=1,3,5,7
/proc/10251/task/10258/stat : 10258 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 58147 0 0 0 2483 10 0 0 25 0 5 0 520899300 1111474176 241081 33554432000 4194304 4338100 140734363195904 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=10251/tid=10259] ppid=10249 vsize=1085424 CPUtime=24.99 cores=1,3,5,7
/proc/10251/task/10259/stat : 10259 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 56380 0 0 0 2489 10 0 0 25 0 5 0 520899300 1111474176 241081 33554432000 4194304 4338100 140734363195904 18446744073709551615 4256139 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=10251/tid=10260] ppid=10249 vsize=1085424 CPUtime=25 cores=1,3,5,7
/proc/10251/task/10260/stat : 10260 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 62754 0 0 0 2490 10 0 0 25 0 5 0 520899300 1111474176 241081 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 100.38
Current children cumulated vsize (KiB) 1085424

[startup+51.1011 s]
/proc/loadavg: 6.61 7.18 7.76 10/190 10263
/proc/meminfo: memFree=22404032/32951124 swapFree=67111352/67111528
[pid=10251] ppid=10249 vsize=1687432 CPUtime=202.63 cores=1,3,5,7
/proc/10251/stat : 10251 (ttsth-5-4) S 10249 10251 9884 0 -1 4202496 394265 0 0 0 20185 78 0 0 25 0 5 0 520899251 1727930368 391415 33554432000 4194304 4338100 140734363195904 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/10251/statm: 421858 391416 176 36 0 419763 0
[pid=10251/tid=10257] ppid=10249 vsize=1687432 CPUtime=50.39 cores=1,3,5,7
/proc/10251/task/10257/stat : 10257 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 91380 0 0 0 5019 20 0 0 25 0 5 0 520899300 1727930368 391416 33554432000 4194304 4338100 140734363195904 18446744073709551615 246516229363 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=10251/tid=10258] ppid=10249 vsize=1687432 CPUtime=50.54 cores=1,3,5,7
/proc/10251/task/10258/stat : 10258 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 96008 0 0 0 5036 18 0 0 25 0 5 0 520899300 1727930368 391416 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=10251/tid=10259] ppid=10249 vsize=1687432 CPUtime=50.58 cores=1,3,5,7
/proc/10251/task/10259/stat : 10259 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 92716 0 0 0 5040 18 0 0 25 0 5 0 520899300 1727930368 391416 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254291 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=10251/tid=10260] ppid=10249 vsize=1687432 CPUtime=50.6 cores=1,3,5,7
/proc/10251/task/10260/stat : 10260 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 102125 0 0 0 5041 19 0 0 25 0 5 0 520899300 1727930368 391416 33554432000 4194304 4338100 140734363195904 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 202.63
Current children cumulated vsize (KiB) 1687432

[startup+102.307 s]
/proc/loadavg: 7.61 7.37 7.80 11/190 10266
/proc/meminfo: memFree=20990844/32951124 swapFree=67111352/67111528
[pid=10251] ppid=10249 vsize=2558732 CPUtime=406.34 cores=1,3,5,7
/proc/10251/stat : 10251 (ttsth-5-4) S 10249 10251 9884 0 -1 4202496 612451 0 0 0 40511 123 0 0 25 0 5 0 520899251 2620141568 609601 33554432000 4194304 4338100 140734363195904 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/10251/statm: 639683 609601 176 36 0 637588 0
[pid=10251/tid=10257] ppid=10249 vsize=2558732 CPUtime=101.31 cores=1,3,5,7
/proc/10251/task/10257/stat : 10257 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 128639 0 0 0 10103 28 0 0 25 0 5 0 520899300 2620141568 609601 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=10251/tid=10258] ppid=10249 vsize=2558732 CPUtime=100.92 cores=1,3,5,7
/proc/10251/task/10258/stat : 10258 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 149730 0 0 0 10063 29 0 0 25 0 5 0 520899300 2620141568 609601 33554432000 4194304 4338100 140734363195904 18446744073709551615 4284177 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=10251/tid=10259] ppid=10249 vsize=2558732 CPUtime=101.78 cores=1,3,5,7
/proc/10251/task/10259/stat : 10259 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 152683 0 0 0 10149 29 0 0 25 0 5 0 520899300 2620141568 609601 33554432000 4194304 4338100 140734363195904 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=10251/tid=10260] ppid=10249 vsize=2558732 CPUtime=101.8 cores=1,3,5,7
/proc/10251/task/10260/stat : 10260 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 169362 0 0 0 10147 33 0 0 25 0 5 0 520899300 2620141568 609601 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 406.34
Current children cumulated vsize (KiB) 2558732

[startup+162.301 s]
/proc/loadavg: 7.85 7.48 7.81 9/190 10267
/proc/meminfo: memFree=19738512/32951124 swapFree=67111352/67111528
[pid=10251] ppid=10249 vsize=3441256 CPUtime=645.99 cores=1,3,5,7
/proc/10251/stat : 10251 (ttsth-5-4) S 10249 10251 9884 0 -1 4202496 832112 0 0 0 64428 171 0 0 25 0 5 0 520899251 3523846144 829262 33554432000 4194304 4338100 140734363195904 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/10251/statm: 860314 829262 176 36 0 858219 0
[pid=10251/tid=10257] ppid=10249 vsize=3441256 CPUtime=160.98 cores=1,3,5,7
/proc/10251/task/10257/stat : 10257 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 169952 0 0 0 16060 38 0 0 25 0 5 0 520899300 3523846144 829262 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254291 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=10251/tid=10258] ppid=10249 vsize=3441256 CPUtime=160.91 cores=1,3,5,7
/proc/10251/task/10258/stat : 10258 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 210322 0 0 0 16048 43 0 0 25 0 5 0 520899300 3523846144 829262 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=10251/tid=10259] ppid=10249 vsize=3441256 CPUtime=161.77 cores=1,3,5,7
/proc/10251/task/10259/stat : 10259 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 208080 0 0 0 16138 39 0 0 25 0 5 0 520899300 3523846144 829262 33554432000 4194304 4338100 140734363195904 18446744073709551615 4240545 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=10251/tid=10260] ppid=10249 vsize=3441256 CPUtime=161.8 cores=1,3,5,7
/proc/10251/task/10260/stat : 10260 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 231721 0 0 0 16133 47 0 0 25 0 5 0 520899300 3523846144 829262 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 645.99
Current children cumulated vsize (KiB) 3441256


################
# More data... #
################

/proc/meminfo: memFree=12819748/32951124 swapFree=67111352/67111528
[pid=10251] ppid=10249 vsize=8084696 CPUtime=3042.63 cores=1,3,5,7
/proc/10251/stat : 10251 (ttsth-5-4) S 10249 10251 9884 0 -1 4202496 1965933 0 0 0 303839 424 0 0 25 0 5 0 520899251 8278728704 1963083 33554432000 4194304 4338100 140734363195904 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/10251/statm: 2021174 1963083 176 36 0 2019079 0
[pid=10251/tid=10257] ppid=10249 vsize=8084696 CPUtime=760.09 cores=1,3,5,7
/proc/10251/task/10257/stat : 10257 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 418542 0 0 0 75916 93 0 0 25 0 5 0 520899300 8278728704 1963083 33554432000 4194304 4338100 140734363195904 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=10251/tid=10258] ppid=10249 vsize=8084696 CPUtime=760.6 cores=1,3,5,7
/proc/10251/task/10258/stat : 10258 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 464894 0 0 0 75959 101 0 0 25 0 5 0 520899300 8278728704 1963083 33554432000 4194304 4338100 140734363195904 18446744073709551615 4235651 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=10251/tid=10259] ppid=10249 vsize=8084696 CPUtime=761.09 cores=1,3,5,7
/proc/10251/task/10259/stat : 10259 (ttsth-5-4) S 10249 10251 9884 0 -1 4202560 489385 0 0 0 76004 105 0 0 25 0 5 0 520899300 8278728704 1963083 33554432000 4194304 4338100 140734363195904 18446744073709551615 246528646918 0 0 4096 0 18446604471118157888 0 0 -1 1 0 0 0
[pid=10251/tid=10260] ppid=10249 vsize=8084696 CPUtime=760.33 cores=1,3,5,7
/proc/10251/task/10260/stat : 10260 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 581075 0 0 0 75911 122 0 0 25 0 5 0 520899300 8278728704 1963083 33554432000 4194304 4338100 140734363195904 18446744073709551615 4235756 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3042.63
Current children cumulated vsize (KiB) 8084696

[startup+822.301 s]
/proc/loadavg: 8.23 8.07 7.95 10/190 10285
/proc/meminfo: memFree=12501408/32951124 swapFree=67111352/67111528
[pid=10251] ppid=10249 vsize=8247880 CPUtime=3282.26 cores=1,3,5,7
/proc/10251/stat : 10251 (ttsth-5-4) S 10249 10251 9884 0 -1 4202496 2007295 0 0 0 327792 434 0 0 25 0 5 0 520899251 8445829120 2004445 33554432000 4194304 4338100 140734363195904 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/10251/statm: 2061970 2004446 176 36 0 2059875 0
[pid=10251/tid=10257] ppid=10249 vsize=8247880 CPUtime=819.77 cores=1,3,5,7
/proc/10251/task/10257/stat : 10257 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 422425 0 0 0 81882 95 0 0 25 0 5 0 520899300 8445829120 2004447 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=10251/tid=10258] ppid=10249 vsize=8247880 CPUtime=820.6 cores=1,3,5,7
/proc/10251/task/10258/stat : 10258 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 487661 0 0 0 81954 106 0 0 25 0 5 0 520899300 8445829120 2004447 33554432000 4194304 4338100 140734363195904 18446744073709551615 4287681 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=10251/tid=10259] ppid=10249 vsize=8247880 CPUtime=821.08 cores=1,3,5,7
/proc/10251/task/10259/stat : 10259 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 500236 0 0 0 82001 107 0 0 25 0 5 0 520899300 8445829120 2004448 33554432000 4194304 4338100 140734363195904 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=10251/tid=10260] ppid=10249 vsize=8247880 CPUtime=820.29 cores=1,3,5,7
/proc/10251/task/10260/stat : 10260 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 584939 0 0 0 81907 122 0 0 25 0 5 0 520899300 8445829120 2004448 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3282.26
Current children cumulated vsize (KiB) 8247880

[startup+882.3 s]
/proc/loadavg: 8.19 8.08 7.96 9/190 10287
/proc/meminfo: memFree=12239224/32951124 swapFree=67111352/67111528
[pid=10251] ppid=10249 vsize=8384624 CPUtime=3521.92 cores=1,3,5,7
/proc/10251/stat : 10251 (ttsth-5-4) S 10249 10251 9884 0 -1 4202496 2037924 0 0 0 351750 442 0 0 25 0 5 0 520899251 8585854976 2035074 33554432000 4194304 4338100 140734363195904 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/10251/statm: 2096156 2035074 176 36 0 2094061 0
[pid=10251/tid=10257] ppid=10249 vsize=8384624 CPUtime=879.43 cores=1,3,5,7
/proc/10251/task/10257/stat : 10257 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 423676 0 0 0 87848 95 0 0 25 0 5 0 520899300 8585854976 2035074 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=10251/tid=10258] ppid=10249 vsize=8384624 CPUtime=880.6 cores=1,3,5,7
/proc/10251/task/10258/stat : 10258 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 490749 0 0 0 87953 107 0 0 25 0 5 0 520899300 8585854976 2035074 33554432000 4194304 4338100 140734363195904 18446744073709551615 4206899 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=10251/tid=10259] ppid=10249 vsize=8384624 CPUtime=881.09 cores=1,3,5,7
/proc/10251/task/10259/stat : 10259 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 511001 0 0 0 87998 111 0 0 25 0 5 0 520899300 8585854976 2035074 33554432000 4194304 4338100 140734363195904 18446744073709551615 4239554 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=10251/tid=10260] ppid=10249 vsize=8384624 CPUtime=880.28 cores=1,3,5,7
/proc/10251/task/10260/stat : 10260 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 600461 0 0 0 87902 126 0 0 25 0 5 0 520899300 8585854976 2035074 33554432000 4194304 4338100 140734363195904 18446744073709551615 4265608 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3521.92
Current children cumulated vsize (KiB) 8384624

[startup+942.3 s]
/proc/loadavg: 8.22 8.11 7.98 10/190 10288
/proc/meminfo: memFree=11712292/32951124 swapFree=67111352/67111528
[pid=10251] ppid=10249 vsize=8781652 CPUtime=3761.5 cores=1,3,5,7
/proc/10251/stat : 10251 (ttsth-5-4) S 10249 10251 9884 0 -1 4202496 2115439 0 0 0 375690 460 0 0 25 0 5 0 520899251 8992411648 2112589 33554432000 4194304 4338100 140734363195904 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/10251/statm: 2195413 2112589 176 36 0 2193318 0
[pid=10251/tid=10257] ppid=10249 vsize=8781652 CPUtime=939.07 cores=1,3,5,7
/proc/10251/task/10257/stat : 10257 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 446548 0 0 0 93806 101 0 0 25 0 5 0 520899300 8992411648 2112589 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=10251/tid=10258] ppid=10249 vsize=8781652 CPUtime=940.58 cores=1,3,5,7
/proc/10251/task/10258/stat : 10258 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 500607 0 0 0 93949 109 0 0 25 0 5 0 520899300 8992411648 2112589 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254294 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=10251/tid=10259] ppid=10249 vsize=8781652 CPUtime=941.08 cores=1,3,5,7
/proc/10251/task/10259/stat : 10259 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 529481 0 0 0 93992 116 0 0 25 0 5 0 520899300 8992411648 2112589 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=10251/tid=10260] ppid=10249 vsize=8781652 CPUtime=940.25 cores=1,3,5,7
/proc/10251/task/10260/stat : 10260 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 626766 0 0 0 93894 131 0 0 25 0 5 0 520899300 8992411648 2112589 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3761.5
Current children cumulated vsize (KiB) 8781652

[startup+1002.3 s]
/proc/loadavg: 8.12 8.10 7.98 10/190 10290
/proc/meminfo: memFree=11504788/32951124 swapFree=67111352/67111528
[pid=10251] ppid=10249 vsize=8782444 CPUtime=4001.16 cores=1,3,5,7
/proc/10251/stat : 10251 (ttsth-5-4) S 10249 10251 9884 0 -1 4202496 2115774 0 0 0 399655 461 0 0 25 0 5 0 520899251 8993222656 2112924 33554432000 4194304 4338100 140734363195904 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/10251/statm: 2195611 2112924 176 36 0 2193516 0
[pid=10251/tid=10257] ppid=10249 vsize=8782444 CPUtime=998.73 cores=1,3,5,7
/proc/10251/task/10257/stat : 10257 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 446649 0 0 0 99772 101 0 0 25 0 5 0 520899300 8993222656 2112924 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=10251/tid=10258] ppid=10249 vsize=8782444 CPUtime=1000.59 cores=1,3,5,7
/proc/10251/task/10258/stat : 10258 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 500672 0 0 0 99950 109 0 0 25 0 5 0 520899300 8993222656 2112924 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=10251/tid=10259] ppid=10249 vsize=8782444 CPUtime=1001.08 cores=1,3,5,7
/proc/10251/task/10259/stat : 10259 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 529576 0 0 0 99992 116 0 0 25 0 5 0 520899300 8993222656 2112924 33554432000 4194304 4338100 140734363195904 18446744073709551615 246528648448 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=10251/tid=10260] ppid=10249 vsize=8782444 CPUtime=1000.25 cores=1,3,5,7
/proc/10251/task/10260/stat : 10260 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 626840 0 0 0 99893 132 0 0 25 0 5 0 520899300 8993222656 2112924 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4001.16
Current children cumulated vsize (KiB) 8782444

[startup+1062.3 s]
/proc/loadavg: 8.13 8.11 8.00 10/190 10291
/proc/meminfo: memFree=11369588/32951124 swapFree=67111352/67111528
[pid=10251] ppid=10249 vsize=8783104 CPUtime=4240.83 cores=1,3,5,7
/proc/10251/stat : 10251 (ttsth-5-4) S 10249 10251 9884 0 -1 4202496 2116067 0 0 0 423621 462 0 0 25 0 5 0 520899251 8993898496 2113217 33554432000 4194304 4338100 140734363195904 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/10251/statm: 2195776 2113217 176 36 0 2193681 0
[pid=10251/tid=10257] ppid=10249 vsize=8783104 CPUtime=1058.42 cores=1,3,5,7
/proc/10251/task/10257/stat : 10257 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 446731 0 0 0 105740 102 0 0 25 0 5 0 520899300 8993898496 2113217 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=10251/tid=10258] ppid=10249 vsize=8783104 CPUtime=1060.58 cores=1,3,5,7
/proc/10251/task/10258/stat : 10258 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 500740 0 0 0 105949 109 0 0 25 0 5 0 520899300 8993898496 2113217 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254294 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=10251/tid=10259] ppid=10249 vsize=8783104 CPUtime=1061.08 cores=1,3,5,7
/proc/10251/task/10259/stat : 10259 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 529654 0 0 0 105992 116 0 0 25 0 5 0 520899300 8993898496 2113217 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=10251/tid=10260] ppid=10249 vsize=8783104 CPUtime=1060.24 cores=1,3,5,7
/proc/10251/task/10260/stat : 10260 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 626905 0 0 0 105892 132 0 0 25 0 5 0 520899300 8993898496 2113217 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4240.83
Current children cumulated vsize (KiB) 8783104

[startup+1122.3 s]
/proc/loadavg: 8.08 8.10 8.00 10/190 10293
/proc/meminfo: memFree=11286216/32951124 swapFree=67111352/67111528
[pid=10251] ppid=10249 vsize=8783500 CPUtime=4480.51 cores=1,3,5,7
/proc/10251/stat : 10251 (ttsth-5-4) S 10249 10251 9884 0 -1 4202496 2116322 0 0 0 447588 463 0 0 25 0 5 0 520899251 8994304000 2113472 33554432000 4194304 4338100 140734363195904 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/10251/statm: 2195875 2113472 176 36 0 2193780 0
[pid=10251/tid=10257] ppid=10249 vsize=8783500 CPUtime=1118.08 cores=1,3,5,7
/proc/10251/task/10257/stat : 10257 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 446797 0 0 0 111706 102 0 0 25 0 5 0 520899300 8994304000 2113472 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=10251/tid=10258] ppid=10249 vsize=8783500 CPUtime=1120.59 cores=1,3,5,7
/proc/10251/task/10258/stat : 10258 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 500796 0 0 0 111950 109 0 0 25 0 5 0 520899300 8994304000 2113472 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=10251/tid=10259] ppid=10249 vsize=8783500 CPUtime=1121.08 cores=1,3,5,7
/proc/10251/task/10259/stat : 10259 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 529735 0 0 0 111992 116 0 0 25 0 5 0 520899300 8994304000 2113472 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=10251/tid=10260] ppid=10249 vsize=8783500 CPUtime=1120.23 cores=1,3,5,7
/proc/10251/task/10260/stat : 10260 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 626957 0 0 0 111891 132 0 0 25 0 5 0 520899300 8994304000 2113472 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4480.51
Current children cumulated vsize (KiB) 8783500

[startup+1182.3 s]
/proc/loadavg: 8.03 8.08 8.00 10/190 10295
/proc/meminfo: memFree=11147036/32951124 swapFree=67111352/67111528
[pid=10251] ppid=10249 vsize=8921660 CPUtime=4720.17 cores=1,3,5,7
/proc/10251/stat : 10251 (ttsth-5-4) S 10249 10251 9884 0 -1 4202496 2151012 0 0 0 471544 473 0 0 25 0 5 0 520899251 9135779840 2148162 33554432000 4194304 4338100 140734363195904 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/10251/statm: 2230415 2148162 176 36 0 2228320 0
[pid=10251/tid=10257] ppid=10249 vsize=8921660 CPUtime=1177.77 cores=1,3,5,7
/proc/10251/task/10257/stat : 10257 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 446863 0 0 0 117674 103 0 0 25 0 5 0 520899300 9135779840 2148162 33554432000 4194304 4338100 140734363195904 18446744073709551615 4257007 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=10251/tid=10258] ppid=10249 vsize=8921660 CPUtime=1180.59 cores=1,3,5,7
/proc/10251/task/10258/stat : 10258 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 522327 0 0 0 117943 116 0 0 25 0 5 0 520899300 9135779840 2148162 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=10251/tid=10259] ppid=10249 vsize=8921660 CPUtime=1181.07 cores=1,3,5,7
/proc/10251/task/10259/stat : 10259 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 542745 0 0 0 117988 119 0 0 25 0 5 0 520899300 9135779840 2148162 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=10251/tid=10260] ppid=10249 vsize=8921660 CPUtime=1180.22 cores=1,3,5,7
/proc/10251/task/10260/stat : 10260 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 627040 0 0 0 117890 132 0 0 25 0 5 0 520899300 9135779840 2148162 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4720.17
Current children cumulated vsize (KiB) 8921660



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.31 s]
/proc/loadavg: 8.02 8.08 8.00 6/184 10296
/proc/meminfo: memFree=16312156/32951124 swapFree=67111352/67111528
[pid=10251] ppid=10249 vsize=9018760 CPUtime=4800.1 cores=1,3,5,7
/proc/10251/stat : 10251 (ttsth-5-4) S 10249 10251 9884 0 -1 4202496 2175241 0 0 0 479531 479 0 0 25 0 5 0 520899251 9235210240 2172391 33554432000 4194304 4338100 140734363195904 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/10251/statm: 2254690 2172392 176 36 0 2252595 0
[pid=10251/tid=10257] ppid=10249 vsize=9018760 CPUtime=1197.66 cores=1,3,5,7
/proc/10251/task/10257/stat : 10257 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 446893 0 0 0 119663 103 0 0 25 0 5 0 520899300 9235210240 2172392 33554432000 4194304 4338100 140734363195904 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=10251/tid=10258] ppid=10249 vsize=9018760 CPUtime=1200.61 cores=1,3,5,7
/proc/10251/task/10258/stat : 10258 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 531732 0 0 0 119943 118 0 0 25 0 5 0 520899300 9235210240 2172392 33554432000 4194304 4338100 140734363195904 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=10251/tid=10259] ppid=10249 vsize=9018760 CPUtime=1201.09 cores=1,3,5,7
/proc/10251/task/10259/stat : 10259 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 552275 0 0 0 119987 122 0 0 25 0 5 0 520899300 9235210240 2172392 33554432000 4194304 4338100 140734363195904 18446744073709551615 4229542 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=10251/tid=10260] ppid=10249 vsize=9018760 CPUtime=1200.23 cores=1,3,5,7
/proc/10251/task/10260/stat : 10260 (ttsth-5-4) R 10249 10251 9884 0 -1 4202560 632305 0 0 0 119890 133 0 0 25 0 5 0 520899300 9235210240 2172392 33554432000 4194304 4338100 140734363195904 18446744073709551615 246528646862 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4800.1
Current children cumulated vsize (KiB) 9018760

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.6 s]
/proc/loadavg: 8.02 8.08 8.00 6/184 10296
/proc/meminfo: memFree=16312156/32951124 swapFree=67111352/67111528
[pid=10251] ppid=10249 vsize=0 CPUtime=4800.41 cores=1,3,5,7
/proc/10251/stat : 10251 (ttsth-5-4) Z 10249 10251 9884 0 -1 4203548 2175254 0 0 0 479533 508 0 0 18 0 2 0 520899251 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/10251/statm: 0 0 0 0 0 0 0
[pid=10251/tid=10257] ppid=10249 vsize=0 CPUtime=1197.94 cores=1,3,5,7
/proc/10251/task/10257/stat : 10257 (ttsth-5-4) R 10249 10251 9884 0 -1 4203588 446893 0 0 0 119663 131 0 0 25 0 2 0 520899300 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.41
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.64
CPU time (s): 4800.45
CPU user time (s): 4795.34
CPU system time (s): 5.11622
CPU usage (%): 399.16
Max. virtual memory (cumulated for all children) (KiB): 9018760

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4795.33
system time used= 5.11622
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2175254
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 922
involuntary context switches= 35118

runsolver used 1.75773 second user time and 4.81227 second system time

The end

Launcher Data 

Begin job on node134 at 2011-04-29 18:03:31
IDJOB=3250648
IDBENCH=83335
IDSOLVER=1654
FILE ID=node134/3250648-1304093011
RUNJOBID= node134-1304093010-10232
PBS_JOBID= 13170571
Free space on /tmp= 71584 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/kullmann/VanDerWaerden/VanderWaerden3k/VanDerWaerden_2-3-18_312.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3250648-1304093011/watcher-3250648-1304093011 -o /tmp/evaluation-result-3250648-1304093011/solver-3250648-1304093011 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250648-1304093011.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 1144bbd60c180aa5e72d35b74e9676cb
RANDOM SEED=1628799452

node134.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 7186.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      25017672 kB
Buffers:       1489708 kB
Cached:        5854252 kB
SwapCached:         64 kB
Active:        1465904 kB
Inactive:      5946492 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      25017672 kB
SwapTotal:    67111528 kB
SwapFree:     67111352 kB
Dirty:             836 kB
Writeback:           0 kB
AnonPages:       67776 kB
Mapped:          15040 kB
Slab:           459352 kB
PageTables:       4464 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   214652 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71584 MiB
End job on node134 at 2011-04-29 18:23:35