Trace number 3264290

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.37 1202.03

General information on the benchmark

NameSAT04/handmade/anton/l5k3/
lksat-n1100-m4103-k3-l5-s558351420.used-as.sat04-934.cnf
MD5SUMecdbc194541c1a4c742d6834592d26be
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.019996
Satisfiable
(Un)Satisfiability was proved
Number of variables1090
Number of clauses4103
Sum of the clauses size12309
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 34103
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264290-1304099406.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 1090
0.00/0.00	c Clauses = 4103
0.00/0.00	c Literals = 12309
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 1090
0.00/0.00	c Clauses = 3994
0.00/0.00	c Literals = 11982
0.00/0.00	c a2
0.00/0.00	c a1
0.09/0.13	c Independent sub-problems = 1
0.09/0.13	c Sub-problem with 1055 variables and 3994 clauses
0.09/0.13	c using permutation method 4
0.09/0.13	c count = 2
0.09/0.13	c Ordering finished 0.120 seconds
0.09/0.13	c mapped 0.120 seconds
0.09/0.13	c permuted 0.120 seconds
0.09/0.13	c about to sort
0.09/0.13	c sorted
0.09/0.14	c clauses now 3994
0.09/0.14	c init finished 0.130 seconds
0.09/0.14	c tree built 0.140 seconds
0.09/0.14	c 
0.09/0.14	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.09/0.14	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.09/0.14	c ................................ | ..........                    |                                  0.1         0.0             
0.09/0.14	c ................................ | ..........                  1 |                                  0.1         7.1             
0.09/0.14	c ................................ | ..........                  2 |                                  0.1        14.3             
0.09/0.14	c ................................ | ..........                  3 |                                  0.1        21.4             
0.09/0.14	c ................................ | ..........                  4 |                                  0.1        28.6             
0.09/0.14	c ................................ | ..........                  5 |                                  0.1        35.7             
0.09/0.14	c ................................ | ..........               1000 | 3            236                 0.1      7142.9          127
95.82/24.17	c ................................ | ..........            1483000 | 130467       758820             96.1     15436.7      7727749
191.66/48.18	c ................................ | ..........            2010000 | 195167       1061647           192.0     10468.2     11907247
287.92/72.25	c ................................ | ..........            2605000 | 258389       1312626           288.1      9040.7     14843074
383.72/96.28	c ................................ | ..........            3381000 | 337462       1601095           384.1      8803.3     17971579
479.93/120.33	c ................................ | *.........            4060000 | 411475       1853844           480.1      8456.7     21112882
576.18/144.44	c ................................ | *.........            4412000 | 453109       2029729           576.3      7655.2     23563631
672.02/168.41	c ................................ | *.........            4647000 | 487205       2156920           672.1      6914.3     25638482
767.87/192.44	c ................................ | *.........            5052000 | 536879       2355993           768.0      6577.9     28305696
863.70/216.49	c ................................ | *.........            5549000 | 600497       2543450           864.1      6421.9     30685606
959.97/240.54	c ................................ | **........            5721000 | 627226       2642530           960.2      5958.4     32141456
1055.83/264.54	c ................................ | **........            5890000 | 653315       2740942          1056.0      5577.5     33376146
1152.11/288.69	c ................................ | **........            6046000 | 679901       2843593          1152.5      5246.1     35391990
1247.97/312.62	c ................................ | **........            6184000 | 703785       2931120          1248.1      4954.9     36737722
1343.86/336.65	c ................................ | **........            6316000 | 728497       3035952          1344.0      4699.2     38563836
1440.12/360.73	c ................................ | ***.......            6562000 | 765804       3175644          1440.2      4556.2     40346379
1536.37/384.88	c ................................ | ***.......            6734000 | 793187       3282986          1536.7      4382.1     42411453
1632.24/408.87	c ................................ | ***.......            6867000 | 815432       3378443          1632.5      4206.3     43700340
1727.70/432.79	c ................................ | ***.......            7025000 | 840501       3477499          1728.1      4065.3     44932458
1823.96/456.84	c ................................ | ***.......            7233000 | 874117       3583603          1824.1      3965.2     46769591
1919.83/480.84	c ................................ | ****......            7425000 | 906506       3679593          1920.0      3867.2     48232364
2016.12/504.93	c ................................ | ****......            7646000 | 941433       3792722          2016.2      3792.2     49702630
2111.97/528.96	c ................................ | ****......            7825000 | 963779       3894888          2112.2      3704.6     50941184
2207.84/552.99	c ................................ | ****......            8012000 | 990216       3997151          2208.2      3628.2     52186703
2304.10/577.04	c ................................ | ****......            8199000 | 1017733      4090515          2304.3      3558.1     53164284
2400.37/601.18	c ................................ | *****.....            8360000 | 1043013      4167348          2400.7      3482.3     54236865
2495.83/625.06	c ................................ | *****.....            8502000 | 1065592      4244998          2496.1      3406.2     55073898
2593.29/649.44	c ................................ | *****.....            8649000 | 1084430      4319451          2593.5      3334.9     55833920
2688.75/673.34	c ................................ | *****.....            8734000 | 1098185      4379064          2688.9      3248.1     56903356
2784.65/697.32	c ................................ | *****.....            8815000 | 1111196      4441236          2784.7      3165.5     57775599
2880.49/721.34	c ................................ | ******....            8893000 | 1123591      4496818          2880.7      3087.1     58503512
2975.97/745.29	c ................................ | ******....            8976000 | 1136939      4552929          2976.4      3015.8     58898471
3072.63/769.47	c ................................ | ******....            9065000 | 1149545      4617436          3072.9      2950.0     59758248
3168.90/793.56	c ................................ | ******....            9155000 | 1164943      4681830          3169.2      2888.8     60372699
3264.77/817.54	c ................................ | ******....            9229000 | 1176759      4746816          3264.9      2826.7     61098274
3359.84/841.38	c ................................ | *******...            9324000 | 1190799      4804300          3360.2      2774.9     61967165
3456.51/865.54	c ................................ | *******...            9448000 | 1208664      4863406          3456.7      2733.3     62788494
3551.98/889.47	c ................................ | *******...            9557000 | 1225436      4924286          3552.3      2690.4     63237429
3648.24/913.52	c ................................ | *******...            9666000 | 1241825      4986797          3648.3      2649.4     64138479
3744.45/937.63	c ................................ | *******...            9790000 | 1261224      5045790          3744.6      2614.4     65024802
3840.72/961.78	c ................................ | ********..            9869000 | 1274555      5091201          3841.0      2569.4     65738952
3936.59/985.76	c ................................ | ********..            9929000 | 1283749      5130486          3936.9      2522.1     66482374
4032.48/1009.71	c ................................ | ********..            9998000 | 1293975      5170120          4032.5      2479.3     67099421
4128.73/1033.87	c ................................ | ********..           10068000 | 1304336      5211697          4129.0      2438.3     67495766
4224.20/1057.73	c ................................ | ********..           10132000 | 1315146      5259218          4224.4      2398.5     68227960
4320.46/1081.81	c ................................ | *********.           10193000 | 1324095      5307624          4320.5      2359.2     69086587
4416.74/1105.90	c ................................ | *********.           10275000 | 1336331      5355986          4416.8      2326.4     69674339
4512.19/1129.82	c ................................ | *********.           10424000 | 1356301      5422605          4512.3      2310.1     70889591
4608.49/1153.91	c ................................ | *********.           10545000 | 1373997      5488228          4608.5      2288.2     71829127
4704.34/1177.98	c ................................ | *********.           10638000 | 1387651      5538832          4704.7      2261.1     72736844

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3264290-1304099406/watcher-3264290-1304099406 -o /tmp/evaluation-result-3264290-1304099406/solver-3264290-1304099406 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264290-1304099406.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.53 7.97 7.98 2/172 3418
/proc/meminfo: memFree=27651848/32951124 swapFree=67111356/67111528
[pid=3418] ppid=3416 vsize=9376 CPUtime=0 cores=0,2,4,6
/proc/3418/stat : 3418 (ttsth-5-4) R 3416 3418 2524 0 -1 4202496 421 0 0 0 0 0 0 0 23 0 1 0 521538283 9601024 364 33554432000 4194304 4338100 140737100531408 18446744073709551615 218905391813 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/3418/statm: 2344 366 150 36 0 249 0

[startup+0.028261 s]
/proc/loadavg: 7.53 7.97 7.98 2/172 3418
/proc/meminfo: memFree=27651848/32951124 swapFree=67111356/67111528
[pid=3418] ppid=3416 vsize=9904 CPUtime=0.02 cores=0,2,4,6
/proc/3418/stat : 3418 (ttsth-5-4) R 3416 3418 2524 0 -1 4202496 561 0 0 0 2 0 0 0 23 0 1 0 521538283 10141696 504 33554432000 4194304 4338100 140737100531408 18446744073709551615 4245205 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/3418/statm: 2476 504 153 36 0 381 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 9904

[startup+0.100291 s]
/proc/loadavg: 7.53 7.97 7.98 2/172 3418
/proc/meminfo: memFree=27651848/32951124 swapFree=67111356/67111528
[pid=3418] ppid=3416 vsize=9904 CPUtime=0.09 cores=0,2,4,6
/proc/3418/stat : 3418 (ttsth-5-4) R 3416 3418 2524 0 -1 4202496 567 0 0 0 9 0 0 0 23 0 1 0 521538283 10141696 510 33554432000 4194304 4338100 140737100531408 18446744073709551615 4298020 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/3418/statm: 2476 510 153 36 0 381 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 9904

[startup+0.300181 s]
/proc/loadavg: 7.53 7.97 7.98 2/172 3418
/proc/meminfo: memFree=27651848/32951124 swapFree=67111356/67111528
[pid=3418] ppid=3416 vsize=208488 CPUtime=0.74 cores=0,2,4,6
/proc/3418/stat : 3418 (ttsth-5-4) S 3416 3418 2524 0 -1 4202496 4822 0 0 0 74 0 0 0 24 0 5 0 521538283 213491712 4764 33554432000 4194304 4338100 140737100531408 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3418/statm: 52122 4765 175 36 0 50027 0
Current children cumulated CPU time (s) 0.74
Current children cumulated vsize (KiB) 208488

[startup+0.701111 s]
/proc/loadavg: 7.53 7.97 7.98 2/172 3418
/proc/meminfo: memFree=27651848/32951124 swapFree=67111356/67111528
[pid=3418] ppid=3416 vsize=215532 CPUtime=2.35 cores=0,2,4,6
/proc/3418/stat : 3418 (ttsth-5-4) S 3416 3418 2524 0 -1 4202496 7271 0 0 0 234 1 0 0 24 0 5 0 521538283 220704768 7213 33554432000 4194304 4338100 140737100531408 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3418/statm: 53883 7213 175 36 0 51788 0
Current children cumulated CPU time (s) 2.35
Current children cumulated vsize (KiB) 215532

[startup+1.50096 s]
/proc/loadavg: 7.53 7.97 7.98 6/182 3429
/proc/meminfo: memFree=27615336/32951124 swapFree=67111356/67111528
[pid=3418] ppid=3416 vsize=229084 CPUtime=5.54 cores=0,2,4,6
/proc/3418/stat : 3418 (ttsth-5-4) S 3416 3418 2524 0 -1 4202496 11527 0 0 0 551 3 0 0 24 0 5 0 521538283 234582016 11469 33554432000 4194304 4338100 140737100531408 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3418/statm: 57271 11469 175 36 0 55176 0
[pid=3418/tid=3426] ppid=3416 vsize=229084 CPUtime=1.34 cores=0,2,4,6
/proc/3418/task/3426/stat : 3426 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 807 0 0 0 134 0 0 0 20 0 5 0 521538298 234582016 11469 33554432000 4194304 4338100 140737100531408 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3418/tid=3427] ppid=3416 vsize=229084 CPUtime=1.34 cores=0,2,4,6
/proc/3418/task/3427/stat : 3427 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 1884 0 0 0 134 0 0 0 19 0 5 0 521538298 234582016 11469 33554432000 4194304 4338100 140737100531408 18446744073709551615 218905125976 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3418/tid=3428] ppid=3416 vsize=229084 CPUtime=1.34 cores=0,2,4,6
/proc/3418/task/3428/stat : 3428 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 2453 0 0 0 133 1 0 0 25 0 5 0 521538298 234582016 11469 33554432000 4194304 4338100 140737100531408 18446744073709551615 218905427401 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3418/tid=3429] ppid=3416 vsize=229084 CPUtime=1.34 cores=0,2,4,6
/proc/3418/task/3429/stat : 3429 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 2566 0 0 0 134 0 0 0 19 0 5 0 521538298 234582016 11469 33554432000 4194304 4338100 140737100531408 18446744073709551615 4256374 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 5.54
Current children cumulated vsize (KiB) 229084

[startup+3.10068 s]
/proc/loadavg: 7.33 7.92 7.96 6/182 3429
/proc/meminfo: memFree=27593512/32951124 swapFree=67111356/67111528
[pid=3418] ppid=3416 vsize=255396 CPUtime=11.93 cores=0,2,4,6
/proc/3418/stat : 3418 (ttsth-5-4) S 3416 3418 2524 0 -1 4202496 18039 0 0 0 1187 6 0 0 24 0 5 0 521538283 261525504 17981 33554432000 4194304 4338100 140737100531408 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3418/statm: 63849 17981 175 36 0 61754 0
[pid=3418/tid=3426] ppid=3416 vsize=255396 CPUtime=2.95 cores=0,2,4,6
/proc/3418/task/3426/stat : 3426 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 1484 0 0 0 294 1 0 0 19 0 5 0 521538298 261525504 17981 33554432000 4194304 4338100 140737100531408 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3418/tid=3427] ppid=3416 vsize=255396 CPUtime=2.94 cores=0,2,4,6
/proc/3418/task/3427/stat : 3427 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 3445 0 0 0 293 1 0 0 18 0 5 0 521538298 261525504 17981 33554432000 4194304 4338100 140737100531408 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3418/tid=3428] ppid=3416 vsize=255396 CPUtime=2.92 cores=0,2,4,6
/proc/3418/task/3428/stat : 3428 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 4660 0 0 0 291 1 0 0 19 0 5 0 521538298 261525504 17982 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3418/tid=3429] ppid=3416 vsize=255396 CPUtime=2.95 cores=0,2,4,6
/proc/3418/task/3429/stat : 3429 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 4633 0 0 0 294 1 0 0 24 0 5 0 521538298 261525504 17982 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 11.93
Current children cumulated vsize (KiB) 255396

[startup+6.3011 s]
/proc/loadavg: 7.33 7.92 7.96 6/182 3429
/proc/meminfo: memFree=27544060/32951124 swapFree=67111356/67111528
[pid=3418] ppid=3416 vsize=297616 CPUtime=24.7 cores=0,2,4,6
/proc/3418/stat : 3418 (ttsth-5-4) S 3416 3418 2524 0 -1 4202496 29117 0 0 0 2457 13 0 0 24 0 5 0 521538283 304758784 29059 33554432000 4194304 4338100 140737100531408 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3418/statm: 74404 29059 175 36 0 72309 0
[pid=3418/tid=3426] ppid=3416 vsize=297616 CPUtime=6.14 cores=0,2,4,6
/proc/3418/task/3426/stat : 3426 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 2638 0 0 0 611 3 0 0 18 0 5 0 521538298 304758784 29059 33554432000 4194304 4338100 140737100531408 18446744073709551615 218917556917 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3418/tid=3427] ppid=3416 vsize=297616 CPUtime=6.14 cores=0,2,4,6
/proc/3418/task/3427/stat : 3427 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 5516 0 0 0 610 4 0 0 25 0 5 0 521538298 304758784 29059 33554432000 4194304 4338100 140737100531408 18446744073709551615 218917556917 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3418/tid=3428] ppid=3416 vsize=297616 CPUtime=6.11 cores=0,2,4,6
/proc/3418/task/3428/stat : 3428 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 9176 0 0 0 608 3 0 0 22 0 5 0 521538298 304758784 29059 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3418/tid=3429] ppid=3416 vsize=297616 CPUtime=6.14 cores=0,2,4,6
/proc/3418/task/3429/stat : 3429 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 7970 0 0 0 612 2 0 0 25 0 5 0 521538298 304758784 29059 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 24.7
Current children cumulated vsize (KiB) 297616

[startup+12.7009 s]
/proc/loadavg: 6.97 7.82 7.93 6/182 3429
/proc/meminfo: memFree=27466744/32951124 swapFree=67111356/67111528
[pid=3418] ppid=3416 vsize=365104 CPUtime=50.26 cores=0,2,4,6
/proc/3418/stat : 3418 (ttsth-5-4) S 3416 3418 2524 0 -1 4202496 47281 0 0 0 5003 23 0 0 24 0 5 0 521538283 373866496 47223 33554432000 4194304 4338100 140737100531408 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3418/statm: 91276 47223 175 36 0 89181 0
[pid=3418/tid=3426] ppid=3416 vsize=365104 CPUtime=12.53 cores=0,2,4,6
/proc/3418/task/3426/stat : 3426 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 5347 0 0 0 1247 6 0 0 25 0 5 0 521538298 373866496 47223 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3418/tid=3427] ppid=3416 vsize=365104 CPUtime=12.53 cores=0,2,4,6
/proc/3418/task/3427/stat : 3427 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 8581 0 0 0 1246 7 0 0 25 0 5 0 521538298 373866496 47223 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3418/tid=3428] ppid=3416 vsize=365104 CPUtime=12.47 cores=0,2,4,6
/proc/3418/task/3428/stat : 3428 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 16472 0 0 0 1243 4 0 0 25 0 5 0 521538298 373866496 47223 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3418/tid=3429] ppid=3416 vsize=365104 CPUtime=12.54 cores=0,2,4,6
/proc/3418/task/3429/stat : 3429 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 13064 0 0 0 1250 4 0 0 25 0 5 0 521538298 373866496 47223 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 50.26
Current children cumulated vsize (KiB) 365104

[startup+25.5006 s]
/proc/loadavg: 7.13 7.83 7.93 9/186 3434
/proc/meminfo: memFree=27136496/32951124 swapFree=67111356/67111528
[pid=3418] ppid=3416 vsize=461524 CPUtime=101.38 cores=0,2,4,6
/proc/3418/stat : 3418 (ttsth-5-4) S 3416 3418 2524 0 -1 4202496 72570 0 0 0 10099 39 0 0 24 0 5 0 521538283 472600576 72512 33554432000 4194304 4338100 140737100531408 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3418/statm: 115381 72512 175 36 0 113286 0
[pid=3418/tid=3426] ppid=3416 vsize=461524 CPUtime=25.34 cores=0,2,4,6
/proc/3418/task/3426/stat : 3426 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 10282 0 0 0 2520 14 0 0 25 0 5 0 521538298 472600576 72512 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3418/tid=3427] ppid=3416 vsize=461524 CPUtime=25.26 cores=0,2,4,6
/proc/3418/task/3427/stat : 3427 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 13808 0 0 0 2514 12 0 0 25 0 5 0 521538298 472600576 72512 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3418/tid=3428] ppid=3416 vsize=461524 CPUtime=25.28 cores=0,2,4,6
/proc/3418/task/3428/stat : 3428 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 24950 0 0 0 2521 7 0 0 25 0 5 0 521538298 472600576 72512 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3418/tid=3429] ppid=3416 vsize=461524 CPUtime=25.34 cores=0,2,4,6
/proc/3418/task/3429/stat : 3429 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 19713 0 0 0 2529 5 0 0 25 0 5 0 521538298 472600576 72512 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 101.38
Current children cumulated vsize (KiB) 461524

[startup+51.1009 s]
/proc/loadavg: 7.56 7.87 7.94 9/186 3435
/proc/meminfo: memFree=26793744/32951124 swapFree=67111356/67111528
[pid=3418] ppid=3416 vsize=599620 CPUtime=203.65 cores=0,2,4,6
/proc/3418/stat : 3418 (ttsth-5-4) S 3416 3418 2524 0 -1 4202496 107889 0 0 0 20300 65 0 0 24 0 5 0 521538283 614010880 107831 33554432000 4194304 4338100 140737100531408 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3418/statm: 149905 107831 175 36 0 147810 0
[pid=3418/tid=3426] ppid=3416 vsize=599620 CPUtime=50.93 cores=0,2,4,6
/proc/3418/task/3426/stat : 3426 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 16276 0 0 0 5067 26 0 0 25 0 5 0 521538298 614010880 107831 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3418/tid=3427] ppid=3416 vsize=599620 CPUtime=50.72 cores=0,2,4,6
/proc/3418/task/3427/stat : 3427 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 20084 0 0 0 5052 20 0 0 25 0 5 0 521538298 614010880 107831 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3418/tid=3428] ppid=3416 vsize=599620 CPUtime=50.88 cores=0,2,4,6
/proc/3418/task/3428/stat : 3428 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 37913 0 0 0 5078 10 0 0 25 0 5 0 521538298 614010880 107831 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3418/tid=3429] ppid=3416 vsize=599620 CPUtime=50.94 cores=0,2,4,6
/proc/3418/task/3429/stat : 3429 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 29799 0 0 0 5086 8 0 0 25 0 5 0 521538298 614010880 107831 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 203.65
Current children cumulated vsize (KiB) 599620

[startup+102.307 s]
/proc/loadavg: 7.96 7.92 7.95 9/186 3436
/proc/meminfo: memFree=26279616/32951124 swapFree=67111356/67111528
[pid=3418] ppid=3416 vsize=806228 CPUtime=408.08 cores=0,2,4,6
/proc/3418/stat : 3418 (ttsth-5-4) S 3416 3418 2524 0 -1 4202496 162502 0 0 0 40699 109 0 0 24 0 5 0 521538283 825577472 162444 33554432000 4194304 4338100 140737100531408 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3418/statm: 201557 162444 175 36 0 199462 0
[pid=3418/tid=3426] ppid=3416 vsize=806228 CPUtime=102.04 cores=0,2,4,6
/proc/3418/task/3426/stat : 3426 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 27293 0 0 0 10159 45 0 0 18 0 5 0 521538298 825577472 162444 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3418/tid=3427] ppid=3416 vsize=806228 CPUtime=101.84 cores=0,2,4,6
/proc/3418/task/3427/stat : 3427 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 31821 0 0 0 10148 36 0 0 25 0 5 0 521538298 825577472 162444 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3418/tid=3428] ppid=3416 vsize=806228 CPUtime=102.08 cores=0,2,4,6
/proc/3418/task/3428/stat : 3428 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 56252 0 0 0 10193 15 0 0 25 0 5 0 521538298 825577472 162444 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3418/tid=3429] ppid=3416 vsize=806228 CPUtime=101.94 cores=0,2,4,6
/proc/3418/task/3429/stat : 3429 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 43319 0 0 0 10183 11 0 0 25 0 5 0 521538298 825577472 162444 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 408.08
Current children cumulated vsize (KiB) 806228

[startup+162.301 s]
/proc/loadavg: 8.11 7.97 7.96 9/186 3437
/proc/meminfo: memFree=25836772/32951124 swapFree=67111356/67111528
[pid=3418] ppid=3416 vsize=1012716 CPUtime=647.66 cores=0,2,4,6
/proc/3418/stat : 3418 (ttsth-5-4) S 3416 3418 2524 0 -1 4202496 215612 0 0 0 64625 141 0 0 24 0 5 0 521538283 1037021184 215546 33554432000 4194304 4338100 140737100531408 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3418/statm: 253179 215546 176 36 0 251084 0
[pid=3418/tid=3426] ppid=3416 vsize=1012716 CPUtime=161.96 cores=0,2,4,6
/proc/3418/task/3426/stat : 3426 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 37510 0 0 0 16140 56 0 0 18 0 5 0 521538298 1037021184 215546 33554432000 4194304 4338100 140737100531408 18446744073709551615 218917556917 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3418/tid=3427] ppid=3416 vsize=1012716 CPUtime=161.83 cores=0,2,4,6
/proc/3418/task/3427/stat : 3427 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 43323 0 0 0 16137 46 0 0 25 0 5 0 521538298 1037021184 215546 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3418/tid=3428] ppid=3416 vsize=1012716 CPUtime=162.08 cores=0,2,4,6
/proc/3418/task/3428/stat : 3428 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 73067 0 0 0 16187 21 0 0 25 0 5 0 521538298 1037021184 215546 33554432000 4194304 4338100 140737100531408 18446744073709551615 4279536 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3418/tid=3429] ppid=3416 vsize=1012716 CPUtime=161.62 cores=0,2,4,6
/proc/3418/task/3429/stat : 3429 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 57895 0 0 0 16145 17 0 0 25 0 5 0 521538298 1037021184 215546 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 647.66
Current children cumulated vsize (KiB) 1012716


################
# More data... #
################

/proc/meminfo: memFree=23172288/32951124 swapFree=67111356/67111528
[pid=3418] ppid=3416 vsize=2192276 CPUtime=3044.29 cores=0,2,4,6
/proc/3418/stat : 3418 (ttsth-5-4) S 3416 3418 2524 0 -1 4202496 505767 0 0 0 304075 354 0 0 24 0 5 0 521538283 2244890624 505701 33554432000 4194304 4338100 140737100531408 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3418/statm: 548069 505701 176 36 0 545974 0
[pid=3418/tid=3426] ppid=3416 vsize=2192276 CPUtime=760.04 cores=0,2,4,6
/proc/3418/task/3426/stat : 3426 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 90156 0 0 0 75866 138 0 0 25 0 5 0 521538298 2244890624 505701 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3418/tid=3427] ppid=3416 vsize=2192276 CPUtime=761.71 cores=0,2,4,6
/proc/3418/task/3427/stat : 3427 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 109042 0 0 0 76052 119 0 0 25 0 5 0 521538298 2244890624 505701 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3418/tid=3428] ppid=3416 vsize=2192276 CPUtime=760.95 cores=0,2,4,6
/proc/3418/task/3428/stat : 3428 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 163787 0 0 0 76044 51 0 0 25 0 5 0 521538298 2244890624 505701 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3418/tid=3429] ppid=3416 vsize=2192276 CPUtime=761.41 cores=0,2,4,6
/proc/3418/task/3429/stat : 3429 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 138965 0 0 0 76097 44 0 0 25 0 5 0 521538298 2244890624 505701 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3044.29
Current children cumulated vsize (KiB) 2192276

[startup+822.3 s]
/proc/loadavg: 8.12 8.10 8.02 9/186 3465
/proc/meminfo: memFree=23012300/32951124 swapFree=67111356/67111528
[pid=3418] ppid=3416 vsize=2249864 CPUtime=3283.94 cores=0,2,4,6
/proc/3418/stat : 3418 (ttsth-5-4) S 3416 3418 2524 0 -1 4202496 521190 0 0 0 328030 364 0 0 24 0 5 0 521538283 2303860736 521124 33554432000 4194304 4338100 140737100531408 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3418/statm: 562466 521124 176 36 0 560371 0
[pid=3418/tid=3426] ppid=3416 vsize=2249864 CPUtime=819.72 cores=0,2,4,6
/proc/3418/task/3426/stat : 3426 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 92850 0 0 0 81830 142 0 0 25 0 5 0 521538298 2303860736 521124 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3418/tid=3427] ppid=3416 vsize=2249864 CPUtime=821.71 cores=0,2,4,6
/proc/3418/task/3427/stat : 3427 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 113097 0 0 0 82048 123 0 0 25 0 5 0 521538298 2303860736 521124 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3418/tid=3428] ppid=3416 vsize=2249864 CPUtime=820.94 cores=0,2,4,6
/proc/3418/task/3428/stat : 3428 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 167366 0 0 0 82042 52 0 0 25 0 5 0 521538298 2303860736 521124 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254304 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3418/tid=3429] ppid=3416 vsize=2249864 CPUtime=821.4 cores=0,2,4,6
/proc/3418/task/3429/stat : 3429 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 144060 0 0 0 82095 45 0 0 25 0 5 0 521538298 2303860736 521124 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3283.94
Current children cumulated vsize (KiB) 2249864

[startup+882.3 s]
/proc/loadavg: 8.44 8.17 8.04 9/186 3470
/proc/meminfo: memFree=22839984/32951124 swapFree=67111356/67111528
[pid=3418] ppid=3416 vsize=2309880 CPUtime=3523.61 cores=0,2,4,6
/proc/3418/stat : 3418 (ttsth-5-4) S 3416 3418 2524 0 -1 4202496 537123 0 0 0 351986 375 0 0 24 0 5 0 521538283 2365317120 537057 33554432000 4194304 4338100 140737100531408 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3418/statm: 577470 537057 176 36 0 575375 0
[pid=3418/tid=3426] ppid=3416 vsize=2309880 CPUtime=879.38 cores=0,2,4,6
/proc/3418/task/3426/stat : 3426 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 95919 0 0 0 87792 146 0 0 25 0 5 0 521538298 2365317120 537057 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3418/tid=3427] ppid=3416 vsize=2309880 CPUtime=881.7 cores=0,2,4,6
/proc/3418/task/3427/stat : 3427 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 116547 0 0 0 88044 126 0 0 25 0 5 0 521538298 2365317120 537057 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254278 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3418/tid=3428] ppid=3416 vsize=2309880 CPUtime=880.93 cores=0,2,4,6
/proc/3418/task/3428/stat : 3428 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 172505 0 0 0 88039 54 0 0 25 0 5 0 521538298 2365317120 537057 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3418/tid=3429] ppid=3416 vsize=2309880 CPUtime=881.4 cores=0,2,4,6
/proc/3418/task/3429/stat : 3429 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 148335 0 0 0 88094 46 0 0 25 0 5 0 521538298 2365317120 537057 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3523.61
Current children cumulated vsize (KiB) 2309880

[startup+942.3 s]
/proc/loadavg: 8.22 8.15 8.04 9/186 3471
/proc/meminfo: memFree=22705620/32951124 swapFree=67111356/67111528
[pid=3418] ppid=3416 vsize=2379976 CPUtime=3763.22 cores=0,2,4,6
/proc/3418/stat : 3418 (ttsth-5-4) S 3416 3418 2524 0 -1 4202496 554373 0 0 0 375933 389 0 0 24 0 5 0 521538283 2437095424 554307 33554432000 4194304 4338100 140737100531408 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3418/statm: 594994 554307 176 36 0 592899 0
[pid=3418/tid=3426] ppid=3416 vsize=2379976 CPUtime=939.05 cores=0,2,4,6
/proc/3418/task/3426/stat : 3426 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 97086 0 0 0 93753 152 0 0 25 0 5 0 521538298 2437095424 554307 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3418/tid=3427] ppid=3416 vsize=2379976 CPUtime=941.69 cores=0,2,4,6
/proc/3418/task/3427/stat : 3427 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 121839 0 0 0 94038 131 0 0 25 0 5 0 521538298 2437095424 554307 33554432000 4194304 4338100 140737100531408 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3418/tid=3428] ppid=3416 vsize=2379976 CPUtime=940.93 cores=0,2,4,6
/proc/3418/task/3428/stat : 3428 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 176387 0 0 0 94037 56 0 0 25 0 5 0 521538298 2437095424 554308 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3418/tid=3429] ppid=3416 vsize=2379976 CPUtime=941.39 cores=0,2,4,6
/proc/3418/task/3429/stat : 3429 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 155244 0 0 0 94090 49 0 0 25 0 5 0 521538298 2437095424 554308 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3763.22
Current children cumulated vsize (KiB) 2379976

[startup+1002.3 s]
/proc/loadavg: 8.19 8.15 8.05 9/186 3473
/proc/meminfo: memFree=22563348/32951124 swapFree=67111356/67111528
[pid=3418] ppid=3416 vsize=2436920 CPUtime=4002.9 cores=0,2,4,6
/proc/3418/stat : 3418 (ttsth-5-4) S 3416 3418 2524 0 -1 4202496 569819 0 0 0 399891 399 0 0 24 0 5 0 521538283 2495406080 569753 33554432000 4194304 4338100 140737100531408 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3418/statm: 609230 569753 176 36 0 607135 0
[pid=3418/tid=3426] ppid=3416 vsize=2436920 CPUtime=998.71 cores=0,2,4,6
/proc/3418/task/3426/stat : 3426 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 99446 0 0 0 99716 155 0 0 25 0 5 0 521538298 2495406080 569753 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3418/tid=3427] ppid=3416 vsize=2436920 CPUtime=1001.69 cores=0,2,4,6
/proc/3418/task/3427/stat : 3427 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 125603 0 0 0 100037 132 0 0 25 0 5 0 521538298 2495406080 569753 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3418/tid=3428] ppid=3416 vsize=2436920 CPUtime=1000.93 cores=0,2,4,6
/proc/3418/task/3428/stat : 3428 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 181247 0 0 0 100034 59 0 0 25 0 5 0 521538298 2495406080 569753 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3418/tid=3429] ppid=3416 vsize=2436920 CPUtime=1001.39 cores=0,2,4,6
/proc/3418/task/3429/stat : 3429 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 159706 0 0 0 100088 51 0 0 25 0 5 0 521538298 2495406080 569753 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4002.9
Current children cumulated vsize (KiB) 2436920

[startup+1062.3 s]
/proc/loadavg: 8.07 8.12 8.04 9/186 3474
/proc/meminfo: memFree=22448240/32951124 swapFree=67111356/67111528
[pid=3418] ppid=3416 vsize=2475484 CPUtime=4242.57 cores=0,2,4,6
/proc/3418/stat : 3418 (ttsth-5-4) S 3416 3418 2524 0 -1 4202496 580568 0 0 0 423851 406 0 0 24 0 5 0 521538283 2534895616 580502 33554432000 4194304 4338100 140737100531408 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3418/statm: 618871 580502 176 36 0 616776 0
[pid=3418/tid=3426] ppid=3416 vsize=2475484 CPUtime=1058.39 cores=0,2,4,6
/proc/3418/task/3426/stat : 3426 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 100839 0 0 0 105681 158 0 0 25 0 5 0 521538298 2534895616 580502 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3418/tid=3427] ppid=3416 vsize=2475484 CPUtime=1061.69 cores=0,2,4,6
/proc/3418/task/3427/stat : 3427 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 128712 0 0 0 106034 135 0 0 25 0 5 0 521538298 2534895616 580502 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3418/tid=3428] ppid=3416 vsize=2475484 CPUtime=1060.93 cores=0,2,4,6
/proc/3418/task/3428/stat : 3428 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 185751 0 0 0 106033 60 0 0 25 0 5 0 521538298 2534895616 580502 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3418/tid=3429] ppid=3416 vsize=2475484 CPUtime=1061.39 cores=0,2,4,6
/proc/3418/task/3429/stat : 3429 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 161449 0 0 0 106087 52 0 0 25 0 5 0 521538298 2534895616 580502 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254213 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4242.57
Current children cumulated vsize (KiB) 2475484

[startup+1122.3 s]
/proc/loadavg: 8.02 8.10 8.04 9/186 3476
/proc/meminfo: memFree=22328920/32951124 swapFree=67111356/67111528
[pid=3418] ppid=3416 vsize=2545452 CPUtime=4482.25 cores=0,2,4,6
/proc/3418/stat : 3418 (ttsth-5-4) S 3416 3418 2524 0 -1 4202496 598666 0 0 0 447808 417 0 0 24 0 5 0 521538283 2606542848 598600 33554432000 4194304 4338100 140737100531408 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3418/statm: 636363 598600 176 36 0 634268 0
[pid=3418/tid=3426] ppid=3416 vsize=2545452 CPUtime=1118.06 cores=0,2,4,6
/proc/3418/task/3426/stat : 3426 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 103584 0 0 0 111645 161 0 0 25 0 5 0 521538298 2606542848 598600 33554432000 4194304 4338100 140737100531408 18446744073709551615 4283332 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3418/tid=3427] ppid=3416 vsize=2545452 CPUtime=1121.7 cores=0,2,4,6
/proc/3418/task/3427/stat : 3427 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 131399 0 0 0 112032 138 0 0 25 0 5 0 521538298 2606542848 598600 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3418/tid=3428] ppid=3416 vsize=2545452 CPUtime=1120.92 cores=0,2,4,6
/proc/3418/task/3428/stat : 3428 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 191076 0 0 0 112031 61 0 0 25 0 5 0 521538298 2606542848 598600 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3418/tid=3429] ppid=3416 vsize=2545452 CPUtime=1121.39 cores=0,2,4,6
/proc/3418/task/3429/stat : 3429 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 168790 0 0 0 112084 55 0 0 25 0 5 0 521538298 2606542848 598600 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4482.25
Current children cumulated vsize (KiB) 2545452

[startup+1182.3 s]
/proc/loadavg: 8.01 8.08 8.03 9/186 3478
/proc/meminfo: memFree=22202660/32951124 swapFree=67111356/67111528
[pid=3418] ppid=3416 vsize=2621676 CPUtime=4721.91 cores=0,2,4,6
/proc/3418/stat : 3418 (ttsth-5-4) S 3416 3418 2524 0 -1 4202496 618536 0 0 0 471762 429 0 0 24 0 5 0 521538283 2684596224 618470 33554432000 4194304 4338100 140737100531408 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3418/statm: 655419 618470 176 36 0 653324 0
[pid=3418/tid=3426] ppid=3416 vsize=2621676 CPUtime=1177.74 cores=0,2,4,6
/proc/3418/task/3426/stat : 3426 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 106722 0 0 0 117608 166 0 0 25 0 5 0 521538298 2684596224 618470 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3418/tid=3427] ppid=3416 vsize=2621676 CPUtime=1181.69 cores=0,2,4,6
/proc/3418/task/3427/stat : 3427 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 136505 0 0 0 118028 141 0 0 25 0 5 0 521538298 2684596224 618470 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3418/tid=3428] ppid=3416 vsize=2621676 CPUtime=1180.91 cores=0,2,4,6
/proc/3418/task/3428/stat : 3428 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 197962 0 0 0 118028 63 0 0 25 0 5 0 521538298 2684596224 618470 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3418/tid=3429] ppid=3416 vsize=2621676 CPUtime=1181.38 cores=0,2,4,6
/proc/3418/task/3429/stat : 3429 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 173530 0 0 0 118082 56 0 0 25 0 5 0 521538298 2684596224 618470 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4721.91
Current children cumulated vsize (KiB) 2621676



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1201.9 s]
/proc/loadavg: 8.00 8.07 8.03 9/186 3478
/proc/meminfo: memFree=22151304/32951124 swapFree=67111356/67111528
[pid=3418] ppid=3416 vsize=2653468 CPUtime=4800.22 cores=0,2,4,6
/proc/3418/stat : 3418 (ttsth-5-4) S 3416 3418 2524 0 -1 4202496 625517 0 0 0 479589 433 0 0 24 0 5 0 521538283 2717151232 625451 33554432000 4194304 4338100 140737100531408 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3418/statm: 663367 625451 176 36 0 661272 0
[pid=3418/tid=3426] ppid=3416 vsize=2653468 CPUtime=1197.23 cores=0,2,4,6
/proc/3418/task/3426/stat : 3426 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 108014 0 0 0 119556 167 0 0 25 0 5 0 521538298 2717151232 625451 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3418/tid=3427] ppid=3416 vsize=2653468 CPUtime=1201.31 cores=0,2,4,6
/proc/3418/task/3427/stat : 3427 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 137588 0 0 0 119988 143 0 0 25 0 5 0 521538298 2717151232 625451 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3418/tid=3428] ppid=3416 vsize=2653468 CPUtime=1200.52 cores=0,2,4,6
/proc/3418/task/3428/stat : 3428 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 200547 0 0 0 119987 65 0 0 25 0 5 0 521538298 2717151232 625451 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3418/tid=3429] ppid=3416 vsize=2653468 CPUtime=1200.99 cores=0,2,4,6
/proc/3418/task/3429/stat : 3429 (ttsth-5-4) R 3416 3418 2524 0 -1 4202560 175551 0 0 0 120042 57 0 0 25 0 5 0 521538298 2717151232 625451 33554432000 4194304 4338100 140737100531408 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.22
Current children cumulated vsize (KiB) 2653468

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.01 s]
/proc/loadavg: 8.00 8.07 8.03 9/186 3478
/proc/meminfo: memFree=22151304/32951124 swapFree=67111356/67111528
[pid=3418] ppid=3416 vsize=0 CPUtime=4800.34 cores=0,2,4,6
/proc/3418/stat : 3418 (ttsth-5-4) Z 3416 3418 2524 0 -1 4203548 625517 0 0 0 479591 443 0 0 18 0 2 0 521538283 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3418/statm: 0 0 0 0 0 0 0
[pid=3418/tid=3426] ppid=3416 vsize=0 CPUtime=1197.34 cores=0,2,4,6
/proc/3418/task/3426/stat : 3426 (ttsth-5-4) R 3416 3418 2524 0 -1 4203588 108014 0 0 0 119557 177 0 0 25 0 2 0 521538298 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.34
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.03
CPU time (s): 4800.37
CPU user time (s): 4795.92
CPU system time (s): 4.45132
CPU usage (%): 399.356
Max. virtual memory (cumulated for all children) (KiB): 2653468

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4795.92
system time used= 4.45132
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 625517
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1993
involuntary context switches= 22847

runsolver used 1.78173 second user time and 4.77027 second system time

The end

Launcher Data 

Begin job on node137 at 2011-04-29 19:50:06
IDJOB=3264290
IDBENCH=84569
IDSOLVER=1654
FILE ID=node137/3264290-1304099406
RUNJOBID= node137-1304099406-3402
PBS_JOBID= 13170581
Free space on /tmp= 71620 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT04/handmade/anton/l5k3/lksat-n1100-m4103-k3-l5-s558351420.used-as.sat04-934.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3264290-1304099406/watcher-3264290-1304099406 -o /tmp/evaluation-result-3264290-1304099406/solver-3264290-1304099406 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264290-1304099406.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= ecdbc194541c1a4c742d6834592d26be
RANDOM SEED=2046423411

node137.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      27652012 kB
Buffers:        921356 kB
Cached:        3849744 kB
SwapCached:         56 kB
Active:         334020 kB
Inactive:      4499664 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      27652012 kB
SwapTotal:    67111528 kB
SwapFree:     67111356 kB
Dirty:              12 kB
Writeback:           0 kB
AnonPages:       63160 kB
Mapped:          14480 kB
Slab:           401968 kB
PageTables:       4464 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   187064 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71620 MiB
End job on node137 at 2011-04-29 20:10:10