Trace number 3251660

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.5 1211.17

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m52-q918.75-n50-p15-s98191766.cnf
MD5SUM71638ad618721ad48c6bd2d48b828020
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark3.2785
Satisfiable
(Un)Satisfiability was proved
Number of variables2377
Number of clauses566749
Sum of the clauses size1135775
Maximum clause length52
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2566699
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 550

Solver Data 

0.39/0.41	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251660-1304108586.cnf 
0.39/0.41	c Using SATTIMEOUT of 4800
0.39/0.41	c Using SATRAM of 15500
0.39/0.41	c Size of subset node = 32
0.39/0.41	c Variables = 2377
0.39/0.41	c Clauses = 566749
0.39/0.41	c Literals = 1135775
0.39/0.41	c Reduced size:
0.39/0.41	c Variables = 2377
0.39/0.41	c Clauses = 566749
0.39/0.41	c Literals = 1135775
0.39/0.41	c a2
0.69/0.72	c a1
1.79/1.86	c Independent sub-problems = 1
1.79/1.86	c Sub-problem with 2377 variables and 566749 clauses
1.79/1.86	c using permutation method 4
1.79/1.86	c count = 1
1.79/1.86	c Ordering finished 1.790 seconds
1.79/1.86	c mapped 1.860 seconds
1.79/1.86	c permuted 1.860 seconds
1.79/1.86	c about to sort
1.89/1.91	c sorted
4.09/4.14	c clauses now 566749
4.09/4.14	c init finished 2.060 seconds
4.09/4.14	c tree built 4.130 seconds
4.09/4.14	c 
4.09/4.14	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
4.09/4.14	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
4.09/4.14	c ................................ | ..........                    |                                  4.1         0.0             
4.09/4.14	c ................................ | ..........                  1 |                                  4.1         0.2             
4.09/4.14	c ................................ | ..........                  2 |                                  4.1         0.5             
4.09/4.14	c ................................ | ..........                  3 |                                  4.1         0.7             
4.09/4.14	c ................................ | ..........                  4 |                                  4.1         1.0             
4.09/4.14	c ................................ | ..........                  5 |                                  4.1         1.2             
4.09/4.14	c ................................ | ..........               1000 |              62                  4.1       241.5         2433
95.82/27.34	c ................................ | ..........            4668000 | 905          430305             96.0     48625.0     20173697
191.67/51.58	c ................................ | ..........            9312000 | 1959         864972            192.0     48497.5     39241867
287.94/75.82	c ................................ | ..........           13682000 | 2430         1274027           288.0     47506.9     51666129
383.79/100.06	c ................................ | ..........           17937000 | 3505         1686056           384.0     46710.9     65288986
479.62/124.30	c ................................ | *.........           22011000 | 4314         2084426           480.0     45855.3     75780506
575.90/148.53	c ................................ | *.........           25925000 | 5129         2475380           576.0     45007.1     82942018
671.82/172.78	c ................................ | *.........           30100000 | 5804         2859344           672.1     44783.0     93273195
767.70/196.98	c ................................ | *.........           34389000 | 6438         3250942           768.0     44777.3    103985719
863.94/221.21	c ................................ | *.........           38360000 | 6941         3619774           864.0     44398.1    110364687
959.84/245.44	c ................................ | **........           42376000 | 7614         3997294           960.0     44141.2    121420071
1055.74/269.66	c ................................ | **........           46353000 | 8478         4367220          1056.0     43894.5    134114288
1152.00/293.90	c ................................ | **........           50333000 | 9455         4731684          1152.0     43691.5    140285581
1247.93/318.11	c ................................ | **........           54307000 | 10366        5095094          1248.0     43515.2    147891310
1343.85/342.37	c ................................ | **........           58270000 | 11314        5459265          1344.1     43351.5    163267024
1439.82/366.44	c ................................ | ***.......           61848000 | 12237        5811992          1440.0     42950.0    163991646
1535.69/390.48	c ................................ | ***.......           65325000 | 13275        6163031          1536.0     42528.7    167360292
1631.91/414.52	c ................................ | ***.......           68990000 | 14501        6516884          1632.0     42273.3    177024066
1727.70/438.57	c ................................ | ***.......           73012000 | 15575        6875645          1728.0     42252.3    186237293
1823.94/462.61	c ................................ | ***.......           76787000 | 16130        7214112          1824.0     42098.1    192646870
1919.76/486.65	c ................................ | ****......           80601000 | 17187        7562711          1920.0     41979.7    198992997
2015.96/510.71	c ................................ | ****......           84153000 | 18316        7899777          2016.0     41742.1    203100765
2111.78/534.76	c ................................ | ****......           87875000 | 18810        8238302          2112.0     41607.3    207542329
2207.63/558.80	c ................................ | ****......           91660000 | 19393        8583542          2208.0     41512.3    215097787
2303.84/582.84	c ................................ | ****......           95349000 | 20150        8915286          2304.0     41383.8    218981687
2399.68/606.88	c ................................ | *****.....           99057000 | 20858        9261601          2400.0     41273.4    227606735
2495.92/630.92	c ................................ | *****.....          102605000 | 21474        9599738          2496.0     41107.8    233991550
2591.72/654.96	c ................................ | *****.....          106157000 | 22410        9940026          2592.0     40955.6    245191949
2687.94/679.01	c ................................ | *****.....          109747000 | 22972        10265841         2688.0     40828.5    246240040
2783.75/703.06	c ................................ | *****.....          113481000 | 23601        10600448         2784.0     40761.9    253396041
2880.00/727.10	c ................................ | ******....          117029000 | 23973        10926044         2880.0     40634.9    262134857
2975.82/751.14	c ................................ | ******....          120521000 | 24754        11246667         2976.0     40497.4    266127759
3071.66/775.18	c ................................ | ******....          123944000 | 25701        11571157         3072.0     40346.4    274955826
3167.89/799.23	c ................................ | ******....          127700000 | 25828        11900037         3168.0     40309.1    282516003
3263.74/823.26	c ................................ | ******....          131249000 | 26063        12215186         3264.0     40210.8    287107861
3359.68/847.37	c ................................ | *******...          134894000 | 26513        12546551         3360.0     40147.0    294024102
3455.61/871.60	c ................................ | *******...          138414000 | 27283        12883028         3456.0     40050.3    300087177
3551.95/895.81	c ................................ | *******...          141942000 | 28380        13213328         3552.0     39961.1    307624395
3647.83/920.04	c ................................ | *******...          145400000 | 30034        13533666         3648.0     39857.2    315746621
3743.70/944.27	c ................................ | *******...          148878000 | 31358        13856988         3744.0     39764.3    316118380
3839.97/968.51	c ................................ | ********..          152142000 | 32280        14175143         3840.0     39620.2    317387784
3935.92/992.72	c ................................ | ********..          155402000 | 33271        14487378         3936.0     39482.2    320642633
4031.86/1016.94	c ................................ | ********..          158906000 | 33968        14817954         4032.0     39411.0    327534326
4127.79/1041.15	c ................................ | ********..          162302000 | 34837        15135623         4128.0     39317.2    331560278
4223.74/1065.36	c ................................ | ********..          165577000 | 35182        15443518         4224.0     39199.0    340828660
4319.67/1089.58	c ................................ | *********.          168926000 | 35948        15763950         4320.0     39103.2    343275187
4415.61/1113.80	c ................................ | *********.          172198000 | 36555        16076225         4416.0     38994.1    347953095
4511.92/1138.02	c ................................ | *********.          175207000 | 37226        16374063         4512.0     38831.3    351583348
4607.85/1162.23	c ................................ | *********.          178570000 | 37677        16700022         4608.0     38752.2    355416011
4703.83/1186.44	c ................................ | *********.          182021000 | 38293        17028634         4704.0     38694.8    366176247
4799.73/1210.66	c CPU limit (4800) reached
4799.73/1210.66	c Time = 4800.00 seconds
4799.73/1210.66	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3251660-1304108586/watcher-3251660-1304108586 -o /tmp/evaluation-result-3251660-1304108586/solver-3251660-1304108586 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251660-1304108586.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.15 8.15 8.10 5/185 11681
/proc/meminfo: memFree=22728640/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=9772 CPUtime=0 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) R 11679 11681 11448 0 -1 4202496 491 0 0 0 0 0 0 0 25 0 1 0 522456833 10006528 433 33554432000 4194304 4338100 140734955866896 18446744073709551615 246516531296 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/11681/statm: 2443 434 147 36 0 348 0

[startup+0.0254069 s]
/proc/loadavg: 8.15 8.15 8.10 5/185 11681
/proc/meminfo: memFree=22728640/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=13688 CPUtime=0.02 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) R 11679 11681 11448 0 -1 4202496 1463 0 0 0 2 0 0 0 25 0 1 0 522456833 14016512 1405 33554432000 4194304 4338100 140734955866896 18446744073709551615 246516495045 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/11681/statm: 3422 1407 148 36 0 1327 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 13688

[startup+0.100386 s]
/proc/loadavg: 8.15 8.15 8.10 5/185 11681
/proc/meminfo: memFree=22728640/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=28292 CPUtime=0.09 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) R 11679 11681 11448 0 -1 4202496 5066 0 0 0 9 0 0 0 25 0 1 0 522456833 28971008 5008 33554432000 4194304 4338100 140734955866896 18446744073709551615 246516495045 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/11681/statm: 7073 5009 148 36 0 4978 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 28292

[startup+0.300348 s]
/proc/loadavg: 8.15 8.15 8.10 5/185 11681
/proc/meminfo: memFree=22728640/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=66988 CPUtime=0.29 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) R 11679 11681 11448 0 -1 4202496 14412 0 0 0 28 1 0 0 25 0 1 0 522456833 68595712 14354 33554432000 4194304 4338100 140734955866896 18446744073709551615 246516495045 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/11681/statm: 16780 14356 148 36 0 14685 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 66988

[startup+0.700277 s]
/proc/loadavg: 8.15 8.15 8.10 5/185 11681
/proc/meminfo: memFree=22728640/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=116212 CPUtime=0.69 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) R 11679 11681 11448 0 -1 4202496 25425 0 0 0 66 3 0 0 25 0 1 0 522456833 119001088 25365 33554432000 4194304 4338100 140734955866896 18446744073709551615 246516229314 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/11681/statm: 29053 25365 173 36 0 26958 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 116212

[startup+1.50013 s]
/proc/loadavg: 8.15 8.15 8.10 6/186 11682
/proc/meminfo: memFree=22622168/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=132096 CPUtime=1.49 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) R 11679 11681 11448 0 -1 4202496 39886 0 0 0 144 5 0 0 25 0 1 0 522456833 135266304 30330 33554432000 4194304 4338100 140734955866896 18446744073709551615 246516495045 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/11681/statm: 33024 30330 155 36 0 30929 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 132096

[startup+3.10084 s]
/proc/loadavg: 7.89 8.10 8.08 6/186 11682
/proc/meminfo: memFree=22574764/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=215552 CPUtime=3.09 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) R 11679 11681 11448 0 -1 4202496 62411 0 0 0 302 7 0 0 25 0 1 0 522456833 220725248 51748 33554432000 4194304 4338100 140734955866896 18446744073709551615 246516501408 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/11681/statm: 53888 51748 164 36 0 51793 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 215552

[startup+6.30022 s]
/proc/loadavg: 7.89 8.10 8.08 9/190 11687
/proc/meminfo: memFree=22325488/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=487656 CPUtime=12.67 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) S 11679 11681 11448 0 -1 4202496 102617 0 0 0 1247 20 0 0 25 0 5 0 522456833 499359744 91954 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11681/statm: 121914 91954 176 36 0 119819 0
[pid=11681/tid=11683] ppid=11679 vsize=487656 CPUtime=2.15 cores=0,2,4,6
/proc/11681/task/11683/stat : 11683 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 5427 0 0 0 214 1 0 0 25 0 5 0 522457247 499359744 91954 33554432000 4194304 4338100 140734955866896 18446744073709551615 4234400 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11681/tid=11684] ppid=11679 vsize=487656 CPUtime=2.14 cores=0,2,4,6
/proc/11681/task/11684/stat : 11684 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 2946 0 0 0 209 5 0 0 20 0 5 0 522457247 499359744 91954 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=11681/tid=11685] ppid=11679 vsize=487656 CPUtime=2.13 cores=0,2,4,6
/proc/11681/task/11685/stat : 11685 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 5075 0 0 0 212 1 0 0 21 0 5 0 522457247 499359744 91954 33554432000 4194304 4338100 140734955866896 18446744073709551615 246516229401 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=11681/tid=11686] ppid=11679 vsize=487656 CPUtime=2.07 cores=0,2,4,6
/proc/11681/task/11686/stat : 11686 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 3297 0 0 0 206 1 0 0 18 0 5 0 522457247 499359744 91954 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 12.67
Current children cumulated vsize (KiB) 487656

[startup+12.701 s]
/proc/loadavg: 7.91 8.10 8.08 9/190 11687
/proc/meminfo: memFree=22104024/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=677652 CPUtime=38.01 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) S 11679 11681 11448 0 -1 4202496 151679 0 0 0 3750 51 0 0 25 0 5 0 522456833 693915648 141016 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11681/statm: 169413 141016 176 36 0 167318 0
[pid=11681/tid=11683] ppid=11679 vsize=677652 CPUtime=8.55 cores=0,2,4,6
/proc/11681/task/11683/stat : 11683 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 23558 0 0 0 848 7 0 0 25 0 5 0 522457247 693915648 141016 33554432000 4194304 4338100 140734955866896 18446744073709551615 4256674 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11681/tid=11684] ppid=11679 vsize=677652 CPUtime=8.54 cores=0,2,4,6
/proc/11681/task/11684/stat : 11684 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 9989 0 0 0 832 22 0 0 25 0 5 0 522457247 693915648 141016 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=11681/tid=11685] ppid=11679 vsize=677652 CPUtime=8.49 cores=0,2,4,6
/proc/11681/task/11685/stat : 11685 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 20454 0 0 0 845 4 0 0 25 0 5 0 522457247 693915648 141016 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528648360 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=11681/tid=11686] ppid=11679 vsize=677652 CPUtime=8.24 cores=0,2,4,6
/proc/11681/task/11686/stat : 11686 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 11806 0 0 0 818 6 0 0 18 0 5 0 522457247 693915648 141016 33554432000 4194304 4338100 140734955866896 18446744073709551615 246516230011 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 38.01
Current children cumulated vsize (KiB) 677652

[startup+25.5007 s]
/proc/loadavg: 7.92 8.09 8.08 9/190 11687
/proc/meminfo: memFree=21681316/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=1003372 CPUtime=88.69 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) S 11679 11681 11448 0 -1 4202496 235527 0 0 0 8757 112 0 0 25 0 5 0 522456833 1027452928 224864 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11681/statm: 250843 224864 176 36 0 248748 0
[pid=11681/tid=11683] ppid=11679 vsize=1003372 CPUtime=21.33 cores=0,2,4,6
/proc/11681/task/11683/stat : 11683 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 56086 0 0 0 2117 16 0 0 25 0 5 0 522457247 1027452928 224864 33554432000 4194304 4338100 140734955866896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=11681/tid=11684] ppid=11679 vsize=1003372 CPUtime=21.34 cores=0,2,4,6
/proc/11681/task/11684/stat : 11684 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 20323 0 0 0 2075 59 0 0 25 0 5 0 522457247 1027452928 224864 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=11681/tid=11685] ppid=11679 vsize=1003372 CPUtime=21.24 cores=0,2,4,6
/proc/11681/task/11685/stat : 11685 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 48375 0 0 0 2113 11 0 0 25 0 5 0 522457247 1027452928 224864 33554432000 4194304 4338100 140734955866896 18446744073709551615 246516530688 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11681/tid=11686] ppid=11679 vsize=1003372 CPUtime=20.6 cores=0,2,4,6
/proc/11681/task/11686/stat : 11686 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 24871 0 0 0 2045 15 0 0 18 0 5 0 522457247 1027452928 224864 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 88.69
Current children cumulated vsize (KiB) 1003372

[startup+51.101 s]
/proc/loadavg: 8.09 8.12 8.09 9/190 11688
/proc/meminfo: memFree=20953684/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=1664900 CPUtime=190.09 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) S 11679 11681 11448 0 -1 4202496 394166 0 0 0 18777 232 0 0 25 0 5 0 522456833 1704857600 383503 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11681/statm: 416225 383503 176 36 0 414130 0
[pid=11681/tid=11683] ppid=11679 vsize=1664900 CPUtime=46.79 cores=0,2,4,6
/proc/11681/task/11683/stat : 11683 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 118040 0 0 0 4646 33 0 0 25 0 5 0 522457247 1704857600 383503 33554432000 4194304 4338100 140734955866896 18446744073709551615 4256770 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=11681/tid=11684] ppid=11679 vsize=1664900 CPUtime=46.95 cores=0,2,4,6
/proc/11681/task/11684/stat : 11684 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 39103 0 0 0 4559 136 0 0 25 0 5 0 522457247 1704857600 383503 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=11681/tid=11685] ppid=11679 vsize=1664900 CPUtime=46.83 cores=0,2,4,6
/proc/11681/task/11685/stat : 11685 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 103596 0 0 0 4662 21 0 0 25 0 5 0 522457247 1704857600 383503 33554432000 4194304 4338100 140734955866896 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11681/tid=11686] ppid=11679 vsize=1664900 CPUtime=45.34 cores=0,2,4,6
/proc/11681/task/11686/stat : 11686 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 47555 0 0 0 4504 30 0 0 18 0 5 0 522457247 1704857600 383503 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 190.09
Current children cumulated vsize (KiB) 1664900

[startup+102.307 s]
/proc/loadavg: 8.19 8.13 8.09 9/190 11689
/proc/meminfo: memFree=19970832/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=2641656 CPUtime=392.9 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) S 11679 11681 11448 0 -1 4202496 622229 0 0 0 38852 438 0 0 25 0 5 0 522456833 2705055744 611566 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11681/statm: 660414 611566 176 36 0 658319 0
[pid=11681/tid=11683] ppid=11679 vsize=2641656 CPUtime=97.72 cores=0,2,4,6
/proc/11681/task/11683/stat : 11683 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 230959 0 0 0 9707 65 0 0 25 0 5 0 522457247 2705055744 611566 33554432000 4194304 4338100 140734955866896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=11681/tid=11684] ppid=11679 vsize=2641656 CPUtime=98.15 cores=0,2,4,6
/proc/11681/task/11684/stat : 11684 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 66614 0 0 0 9545 270 0 0 25 0 5 0 522457247 2705055744 611566 33554432000 4194304 4338100 140734955866896 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=11681/tid=11685] ppid=11679 vsize=2641656 CPUtime=98.04 cores=0,2,4,6
/proc/11681/task/11685/stat : 11685 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 169079 0 0 0 9768 36 0 0 25 0 5 0 522457247 2705055744 611566 33554432000 4194304 4338100 140734955866896 18446744073709551615 4265616 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11681/tid=11686] ppid=11679 vsize=2641656 CPUtime=94.82 cores=0,2,4,6
/proc/11681/task/11686/stat : 11686 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 69705 0 0 0 9426 56 0 0 18 0 5 0 522457247 2705055744 611566 33554432000 4194304 4338100 140734955866896 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 392.9
Current children cumulated vsize (KiB) 2641656

[startup+162.301 s]
/proc/loadavg: 8.02 8.09 8.08 9/190 11691
/proc/meminfo: memFree=18908492/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=3362120 CPUtime=630.57 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) S 11679 11681 11448 0 -1 4202496 797427 0 0 0 62396 661 0 0 25 0 5 0 522456833 3442810880 786764 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11681/statm: 840530 786765 176 36 0 838435 0
[pid=11681/tid=11683] ppid=11679 vsize=3362120 CPUtime=157.37 cores=0,2,4,6
/proc/11681/task/11683/stat : 11683 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 303777 0 0 0 15648 89 0 0 25 0 5 0 522457247 3442810880 786765 33554432000 4194304 4338100 140734955866896 18446744073709551615 4239456 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=11681/tid=11684] ppid=11679 vsize=3362120 CPUtime=158.14 cores=0,2,4,6
/proc/11681/task/11684/stat : 11684 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 74553 0 0 0 15388 426 0 0 25 0 5 0 522457247 3442810880 786765 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=11681/tid=11685] ppid=11679 vsize=3362120 CPUtime=158.03 cores=0,2,4,6
/proc/11681/task/11685/stat : 11685 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 247419 0 0 0 15751 52 0 0 25 0 5 0 522457247 3442810880 786765 33554432000 4194304 4338100 140734955866896 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11681/tid=11686] ppid=11679 vsize=3362120 CPUtime=152.87 cores=0,2,4,6
/proc/11681/task/11686/stat : 11686 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 85807 0 0 0 15204 83 0 0 18 0 5 0 522457247 3442810880 786765 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 630.57
Current children cumulated vsize (KiB) 3362120

[startup+222.301 s]
/proc/loadavg: 8.18 8.11 8.09 10/190 11692
/proc/meminfo: memFree=17634216/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=4111416 CPUtime=868.3 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) S 11679 11681 11448 0 -1 4202496 976785 0 0 0 85944 886 0 0 25 0 5 0 522456833 4210089984 966122 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11681/statm: 1027854 966122 176 36 0 1025759 0
[pid=11681/tid=11683] ppid=11679 vsize=4111416 CPUtime=217.02 cores=0,2,4,6
/proc/11681/task/11683/stat : 11683 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 388226 0 0 0 21586 116 0 0 25 0 5 0 522457247 4210089984 966122 33554432000 4194304 4338100 140734955866896 18446744073709551615 4283598 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=11681/tid=11684] ppid=11679 vsize=4111416 CPUtime=218.09 cores=0,2,4,6
/proc/11681/task/11684/stat : 11684 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 92469 0 0 0 21227 582 0 0 25 0 5 0 522457247 4210089984 966122 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=11681/tid=11685] ppid=11679 vsize=4111416 CPUtime=218.02 cores=0,2,4,6
/proc/11681/task/11685/stat : 11685 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 306616 0 0 0 21740 62 0 0 25 0 5 0 522457247 4210089984 966122 33554432000 4194304 4338100 140734955866896 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11681/tid=11686] ppid=11679 vsize=4111416 CPUtime=211 cores=0,2,4,6
/proc/11681/task/11686/stat : 11686 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 103602 0 0 0 20986 114 0 0 18 0 5 0 522457247 4210089984 966122 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 868.3
Current children cumulated vsize (KiB) 4111416

################
# More data... #
################

/proc/11681/task/11683/stat : 11683 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 1123177 0 0 0 75249 359 0 0 25 0 5 0 522457247 9809985536 2245537 33554432000 4194304 4338100 140734955866896 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=11681/tid=11684] ppid=11679 vsize=9580064 CPUtime=756.55 cores=0,2,4,6
/proc/11681/task/11684/stat : 11684 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 126026 0 0 0 73660 1995 0 0 25 0 5 0 522457247 9809985536 2245537 33554432000 4194304 4338100 140734955866896 18446744073709551615 246516229369 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=11681/tid=11685] ppid=11679 vsize=9580064 CPUtime=757.69 cores=0,2,4,6
/proc/11681/task/11685/stat : 11685 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 776892 0 0 0 75606 163 0 0 25 0 5 0 522457247 9809985536 2245538 33554432000 4194304 4338100 140734955866896 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=11681/tid=11686] ppid=11679 vsize=9580064 CPUtime=746.06 cores=0,2,4,6
/proc/11681/task/11686/stat : 11686 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 144234 0 0 0 73348 1258 0 0 25 0 5 0 522457247 9809985536 2245538 33554432000 4194304 4338100 140734955866896 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3020.56
Current children cumulated vsize (KiB) 9580064

[startup+822.301 s]
/proc/loadavg: 4.09 5.07 6.61 5/180 11726
/proc/meminfo: memFree=20602932/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=10101024 CPUtime=3260.15 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) S 11679 11681 11448 0 -1 4202496 2389359 0 0 0 321945 4070 0 0 25 0 5 0 522456833 10343448576 2378696 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11681/statm: 2525256 2378696 176 36 0 2523161 0
[pid=11681/tid=11683] ppid=11679 vsize=10101024 CPUtime=816.05 cores=0,2,4,6
/proc/11681/task/11683/stat : 11683 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 1207991 0 0 0 81218 387 0 0 25 0 5 0 522457247 10343448576 2378696 33554432000 4194304 4338100 140734955866896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=11681/tid=11684] ppid=11679 vsize=10101024 CPUtime=816.45 cores=0,2,4,6
/proc/11681/task/11684/stat : 11684 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 126029 0 0 0 79524 2121 0 0 25 0 5 0 522457247 10343448576 2378696 33554432000 4194304 4338100 140734955866896 18446744073709551615 4256275 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11681/tid=11685] ppid=11679 vsize=10101024 CPUtime=817.68 cores=0,2,4,6
/proc/11681/task/11685/stat : 11685 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 825230 0 0 0 81595 173 0 0 25 0 5 0 522457247 10343448576 2378696 33554432000 4194304 4338100 140734955866896 18446744073709551615 246516230665 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=11681/tid=11686] ppid=11679 vsize=10101024 CPUtime=805.8 cores=0,2,4,6
/proc/11681/task/11686/stat : 11686 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 144237 0 0 0 79202 1378 0 0 25 0 5 0 522457247 10343448576 2378696 33554432000 4194304 4338100 140734955866896 18446744073709551615 4256358 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3260.15
Current children cumulated vsize (KiB) 10101024

[startup+882.301 s]
/proc/loadavg: 4.20 4.92 6.46 5/180 11728
/proc/meminfo: memFree=20103196/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=10593096 CPUtime=3498.44 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) S 11679 11681 11448 0 -1 4202496 2514888 0 0 0 345566 4278 0 0 25 0 5 0 522456833 10847330304 2504225 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11681/statm: 2648274 2504226 176 36 0 2646179 0
[pid=11681/tid=11683] ppid=11679 vsize=10593096 CPUtime=875.87 cores=0,2,4,6
/proc/11681/task/11683/stat : 11683 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 1284598 0 0 0 87173 414 0 0 25 0 5 0 522457247 10847330304 2504226 33554432000 4194304 4338100 140734955866896 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=11681/tid=11684] ppid=11679 vsize=10593096 CPUtime=875.05 cores=0,2,4,6
/proc/11681/task/11684/stat : 11684 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 126031 0 0 0 85340 2165 0 0 18 0 5 0 522457247 10847330304 2504226 33554432000 4194304 4338100 140734955866896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=11681/tid=11685] ppid=11679 vsize=10593096 CPUtime=877.64 cores=0,2,4,6
/proc/11681/task/11685/stat : 11685 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 874150 0 0 0 87580 184 0 0 25 0 5 0 522457247 10847330304 2504226 33554432000 4194304 4338100 140734955866896 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11681/tid=11686] ppid=11679 vsize=10593096 CPUtime=865.7 cores=0,2,4,6
/proc/11681/task/11686/stat : 11686 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 144238 0 0 0 85067 1503 0 0 25 0 5 0 522457247 10847330304 2504226 33554432000 4194304 4338100 140734955866896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3498.44
Current children cumulated vsize (KiB) 10593096

[startup+942.301 s]
/proc/loadavg: 4.10 4.76 6.31 5/180 11729
/proc/meminfo: memFree=19674012/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=11042840 CPUtime=3736.19 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) S 11679 11681 11448 0 -1 4202496 2621076 0 0 0 369175 4444 0 0 25 0 5 0 522456833 11307868160 2610413 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11681/statm: 2760710 2610413 176 36 0 2758615 0
[pid=11681/tid=11683] ppid=11679 vsize=11042840 CPUtime=935.79 cores=0,2,4,6
/proc/11681/task/11683/stat : 11683 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 1337475 0 0 0 93146 433 0 0 25 0 5 0 522457247 11307868160 2610413 33554432000 4194304 4338100 140734955866896 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11681/tid=11684] ppid=11679 vsize=11042840 CPUtime=933.15 cores=0,2,4,6
/proc/11681/task/11684/stat : 11684 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 136897 0 0 0 91137 2178 0 0 18 0 5 0 522457247 11307868160 2610413 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=11681/tid=11685] ppid=11679 vsize=11042840 CPUtime=937.39 cores=0,2,4,6
/proc/11681/task/11685/stat : 11685 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 909561 0 0 0 93548 191 0 0 25 0 5 0 522457247 11307868160 2610413 33554432000 4194304 4338100 140734955866896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=11681/tid=11686] ppid=11679 vsize=11042840 CPUtime=925.69 cores=0,2,4,6
/proc/11681/task/11686/stat : 11686 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 151271 0 0 0 90938 1631 0 0 25 0 5 0 522457247 11307868160 2610413 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3736.19
Current children cumulated vsize (KiB) 11042840

[startup+1002.3 s]
/proc/loadavg: 4.15 4.65 6.17 5/180 11731
/proc/meminfo: memFree=19480048/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=11290816 CPUtime=3973.98 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) S 11679 11681 11448 0 -1 4202496 2670135 0 0 0 392791 4607 0 0 25 0 5 0 522456833 11561795584 2659472 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11681/statm: 2822704 2659472 176 36 0 2820609 0
[pid=11681/tid=11683] ppid=11679 vsize=11290816 CPUtime=995.78 cores=0,2,4,6
/proc/11681/task/11683/stat : 11683 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 1364334 0 0 0 99131 447 0 0 21 0 5 0 522457247 11561795584 2659472 33554432000 4194304 4338100 140734955866896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11681/tid=11684] ppid=11679 vsize=11290816 CPUtime=991.89 cores=0,2,4,6
/proc/11681/task/11684/stat : 11684 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 138656 0 0 0 96951 2238 0 0 25 0 5 0 522457247 11561795584 2659472 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528648451 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=11681/tid=11685] ppid=11679 vsize=11290816 CPUtime=997.3 cores=0,2,4,6
/proc/11681/task/11685/stat : 11685 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 922273 0 0 0 99535 195 0 0 25 0 5 0 522457247 11561795584 2659473 33554432000 4194304 4338100 140734955866896 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=11681/tid=11686] ppid=11679 vsize=11290816 CPUtime=984.84 cores=0,2,4,6
/proc/11681/task/11686/stat : 11686 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 159000 0 0 0 96769 1715 0 0 18 0 5 0 522457247 11561795584 2659473 33554432000 4194304 4338100 140734955866896 18446744073709551615 246516229858 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3973.98
Current children cumulated vsize (KiB) 11290816

[startup+1062.3 s]
/proc/loadavg: 4.11 4.55 6.04 5/180 11733
/proc/meminfo: memFree=18849228/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=11933080 CPUtime=4211.83 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) S 11679 11681 11448 0 -1 4202496 2828842 0 0 0 416396 4787 0 0 25 0 5 0 522456833 12219473920 2818179 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11681/statm: 2983270 2818179 176 36 0 2981175 0
[pid=11681/tid=11683] ppid=11679 vsize=11933080 CPUtime=1055.76 cores=0,2,4,6
/proc/11681/task/11683/stat : 11683 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 1462951 0 0 0 105100 476 0 0 25 0 5 0 522457247 12219473920 2818179 33554432000 4194304 4338100 140734955866896 18446744073709551615 4235651 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11681/tid=11684] ppid=11679 vsize=11933080 CPUtime=1051.83 cores=0,2,4,6
/proc/11681/task/11684/stat : 11684 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 140275 0 0 0 102820 2363 0 0 18 0 5 0 522457247 12219473920 2818179 33554432000 4194304 4338100 140734955866896 18446744073709551615 4235651 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=11681/tid=11685] ppid=11679 vsize=11933080 CPUtime=1057.29 cores=0,2,4,6
/proc/11681/task/11685/stat : 11685 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 980743 0 0 0 105519 210 0 0 25 0 5 0 522457247 12219473920 2818179 33554432000 4194304 4338100 140734955866896 18446744073709551615 4235902 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=11681/tid=11686] ppid=11679 vsize=11933080 CPUtime=1042.78 cores=0,2,4,6
/proc/11681/task/11686/stat : 11686 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 159001 0 0 0 102551 1727 0 0 21 0 5 0 522457247 12219473920 2818179 33554432000 4194304 4338100 140734955866896 18446744073709551615 4235746 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4211.83
Current children cumulated vsize (KiB) 11933080

[startup+1122.3 s]
/proc/loadavg: 4.00 4.43 5.90 5/180 11734
/proc/meminfo: memFree=18537840/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=12232632 CPUtime=4449.7 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) S 11679 11681 11448 0 -1 4202496 2904634 0 0 0 440017 4953 0 0 25 0 5 0 522456833 12526215168 2893971 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11681/statm: 3058158 2893971 176 36 0 3056063 0
[pid=11681/tid=11683] ppid=11679 vsize=12232632 CPUtime=1115.74 cores=0,2,4,6
/proc/11681/task/11683/stat : 11683 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 1518480 0 0 0 111075 499 0 0 25 0 5 0 522457247 12526215168 2893971 33554432000 4194304 4338100 140734955866896 18446744073709551615 4279651 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11681/tid=11684] ppid=11679 vsize=12232632 CPUtime=1111.39 cores=0,2,4,6
/proc/11681/task/11684/stat : 11684 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 143724 0 0 0 108670 2469 0 0 25 0 5 0 522457247 12526215168 2893971 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=11681/tid=11685] ppid=11679 vsize=12232632 CPUtime=1117.27 cores=0,2,4,6
/proc/11681/task/11685/stat : 11685 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 997549 0 0 0 111513 214 0 0 25 0 5 0 522457247 12526215168 2893971 33554432000 4194304 4338100 140734955866896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=11681/tid=11686] ppid=11679 vsize=12232632 CPUtime=1101.13 cores=0,2,4,6
/proc/11681/task/11686/stat : 11686 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 159009 0 0 0 108353 1760 0 0 18 0 5 0 522457247 12526215168 2893971 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528659828 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4449.7
Current children cumulated vsize (KiB) 12232632

[startup+1182.3 s]
/proc/loadavg: 4.00 4.35 5.78 5/180 11736
/proc/meminfo: memFree=18062044/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=12763880 CPUtime=4687.55 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) S 11679 11681 11448 0 -1 4202496 3027422 0 0 0 463628 5127 0 0 25 0 5 0 522456833 13070213120 3016759 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11681/statm: 3190970 3016759 176 36 0 3188875 0
[pid=11681/tid=11683] ppid=11679 vsize=12763880 CPUtime=1175.65 cores=0,2,4,6
/proc/11681/task/11683/stat : 11683 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 1585132 0 0 0 117045 520 0 0 25 0 5 0 522457247 13070213120 3016759 33554432000 4194304 4338100 140734955866896 18446744073709551615 246516530633 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=11681/tid=11684] ppid=11679 vsize=12763880 CPUtime=1170.5 cores=0,2,4,6
/proc/11681/task/11684/stat : 11684 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 148975 0 0 0 114507 2543 0 0 18 0 5 0 522457247 13070213120 3016759 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11681/tid=11685] ppid=11679 vsize=12763880 CPUtime=1177.07 cores=0,2,4,6
/proc/11681/task/11685/stat : 11685 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 1048434 0 0 0 117482 225 0 0 24 0 5 0 522457247 13070213120 3016760 33554432000 4194304 4338100 140734955866896 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=11681/tid=11686] ppid=11679 vsize=12763880 CPUtime=1160.17 cores=0,2,4,6
/proc/11681/task/11686/stat : 11686 (ttsth-5-4) R 11679 11681 11448 0 -1 4202560 159009 0 0 0 114189 1828 0 0 20 0 5 0 522457247 13070213120 3016760 33554432000 4194304 4338100 140734955866896 18446744073709551615 246528648347 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4687.55
Current children cumulated vsize (KiB) 12763880



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1210.7 s]
/proc/loadavg: 4.00 4.32 5.73 2/177 11737
/proc/meminfo: memFree=18443960/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=0 CPUtime=4800.04 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) Z 11679 11681 11448 0 -1 4203548 3089282 0 0 0 474794 5210 0 0 18 0 2 0 522456833 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11681/statm: 0 0 0 0 0 0 0
[pid=11681/tid=11686] ppid=11679 vsize=0 CPUtime=1188.57 cores=0,2,4,6
/proc/11681/task/11686/stat : 11686 (ttsth-5-4) R 11679 11681 11448 0 -1 4203588 159010 0 0 0 116966 1891 0 0 25 0 2 0 522457247 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.04
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1211 s]
/proc/loadavg: 3.76 4.26 5.71 2/178 11738
/proc/meminfo: memFree=25696868/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=0 CPUtime=4800.34 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) Z 11679 11681 11448 0 -1 4203548 3089282 0 0 0 474794 5240 0 0 18 0 2 0 522456833 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11681/statm: 0 0 0 0 0 0 0
[pid=11681/tid=11686] ppid=11679 vsize=0 CPUtime=1188.87 cores=0,2,4,6
/proc/11681/task/11686/stat : 11686 (ttsth-5-4) R 11679 11681 11448 0 -1 4203588 159010 0 0 0 116966 1921 0 0 25 0 2 0 522457247 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.34
Current children cumulated vsize (KiB) 0

[startup+1211.11 s]
/proc/loadavg: 3.76 4.26 5.71 2/178 11738
/proc/meminfo: memFree=25696868/32951124 swapFree=67102692/67111528
[pid=11681] ppid=11679 vsize=0 CPUtime=4800.44 cores=0,2,4,6
/proc/11681/stat : 11681 (ttsth-5-4) Z 11679 11681 11448 0 -1 4203548 3089282 0 0 0 474794 5250 0 0 18 0 2 0 522456833 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11681/statm: 0 0 0 0 0 0 0
[pid=11681/tid=11686] ppid=11679 vsize=0 CPUtime=1188.97 cores=0,2,4,6
/proc/11681/task/11686/stat : 11686 (ttsth-5-4) R 11679 11681 11448 0 -1 4203588 159010 0 0 0 116966 1931 0 0 25 0 2 0 522457247 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.44
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1211.17
CPU time (s): 4800.5
CPU user time (s): 4747.94
CPU system time (s): 52.564
CPU usage (%): 396.354
Max. virtual memory (cumulated for all children) (KiB): 13076872

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4747.94
system time used= 52.564
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 3089282
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 31773
involuntary context switches= 69087

runsolver used 2.06868 second user time and 4.36434 second system time

The end

Launcher Data 

Begin job on node134 at 2011-04-29 22:23:06
IDJOB=3251660
IDBENCH=88228
IDSOLVER=1654
FILE ID=node134/3251660-1304108586
RUNJOBID= node134-1304106566-11466
PBS_JOBID= 13170663
Free space on /tmp= 71572 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m52-q918.75-n50-p15-s98191766.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3251660-1304108586/watcher-3251660-1304108586 -o /tmp/evaluation-result-3251660-1304108586/solver-3251660-1304108586 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251660-1304108586.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 71638ad618721ad48c6bd2d48b828020
RANDOM SEED=1209417029

node134.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 7186.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      22728928 kB
Buffers:        990728 kB
Cached:        1371040 kB
SwapCached:       4400 kB
Active:        7507584 kB
Inactive:      2302176 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      22728928 kB
SwapTotal:    67111528 kB
SwapFree:     67102692 kB
Dirty:            7376 kB
Writeback:           4 kB
AnonPages:     7445088 kB
Mapped:          15252 kB
Slab:           334548 kB
PageTables:      19776 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  7950468 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71576 MiB
End job on node134 at 2011-04-29 22:43:19