Trace number 3263146

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.33 1207.92

General information on the benchmark

NameSAT03/handmade/anton/UNSAT/
SGI_30_50_30_20_5-log.shuffled-as.sat03-111.cnf
MD5SUMab60d1d6c6130bc12a1aa4855d06424a
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark4306.81
Satisfiable
(Un)Satisfiability was proved
Number of variables150
Number of clauses55222
Sum of the clauses size551920
Maximum clause length10
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 560
Number of clauses of size over 555162

Solver Data 

0.09/0.13	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263146-1304095775.cnf 
0.09/0.13	c Using SATTIMEOUT of 4800
0.09/0.13	c Using SATRAM of 15500
0.09/0.13	c Size of subset node = 32
0.09/0.13	c Variables = 150
0.09/0.13	c Clauses = 55222
0.09/0.13	c Literals = 551920
0.09/0.13	c Reduced size:
0.09/0.13	c Variables = 150
0.09/0.13	c Clauses = 55222
0.09/0.13	c Literals = 551920
0.09/0.13	c a2
0.89/0.97	c a1
2.59/2.64	c Independent sub-problems = 1
2.59/2.64	c Sub-problem with 150 variables and 55222 clauses
2.59/2.64	c using permutation method 4
2.59/2.64	c count = 1
2.59/2.64	c Ordering finished 2.600 seconds
2.59/2.64	c mapped 2.630 seconds
2.59/2.64	c permuted 2.630 seconds
2.59/2.64	c about to sort
2.59/2.66	c sorted
2.69/2.78	c clauses now 55222
2.69/2.78	c init finished 2.670 seconds
2.69/2.78	c tree built 2.780 seconds
2.69/2.78	c 
2.69/2.78	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
2.69/2.78	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
2.69/2.78	c ................................ | ..........                    |                                  2.8         0.0             
2.69/2.78	c ................................ | ..........                  1 |                                  2.8         0.4             
2.69/2.78	c ................................ | ..........                  2 |                                  2.8         0.7             
2.69/2.78	c ................................ | ..........                  3 |                                  2.8         1.1             
2.69/2.78	c ................................ | ..........                  4 |                                  2.8         1.4             
2.69/2.78	c ................................ | ..........                  5 |                                  2.8         1.8             
2.69/2.79	c ................................ | ..........               1000 | 70           1660                2.8       359.7         8976
95.78/26.15	c ................................ | ..........            1678000 | 423344       2208565            96.0     17479.2     19512615
191.86/50.24	c ................................ | ..........            2818000 | 720225       3717376           192.0     14674.8     31066075
287.88/74.35	c ................................ | ..........            3792000 | 972327       4985249           288.1     13163.0     40353481
383.90/98.44	c ................................ | ..........            4673000 | 1202906      6119972           384.1     12167.1     47293342
479.95/122.52	c ................................ | *.........            5498000 | 1419026      7187779           480.1     11452.5     54909157
575.66/146.58	c ................................ | *.........            6268000 | 1622989      8172059           576.0     10881.6     61079892
671.76/170.69	c ................................ | *.........            7036000 | 1827692      9096795           672.1     10468.5     67596352
767.96/194.81	c ................................ | *.........            7813000 | 2032832      9989405           768.0     10173.2     71645964
864.00/218.91	c ................................ | *.........            8587000 | 2235210      10872481          864.0      9938.3     77142992
960.02/243.01	c ................................ | **........            9299000 | 2422780      11720692          960.0      9686.2     79519249
1055.90/267.13	c ................................ | **........            9978000 | 2601910      12542511         1056.0      9448.6     82793641
1151.89/291.24	c ................................ | **........           10622000 | 2774315      13346643         1152.1      9219.8     87734952
1247.92/315.32	c ................................ | **........           11228000 | 2935784      14112243         1248.0      8996.7     90781273
1344.00/339.41	c ................................ | **........           11900000 | 3116631      14906320         1344.1      8853.7     94241886
1439.62/363.49	c ................................ | ***.......           12576000 | 3296064      15705236         1440.0      8733.2     97909598
1535.95/387.61	c ................................ | ***.......           13190000 | 3454995      16470316         1536.0      8587.2    101342491
1631.90/411.73	c ................................ | ***.......           13799000 | 3615028      17205408         1632.0      8455.2    104529812
1727.97/435.81	c ................................ | ***.......           14376000 | 3763634      17935519         1728.0      8319.4    109872097
1823.82/459.98	c ................................ | ***.......           14952000 | 3912761      18647197         1824.1      8196.7    113213946
1920.09/484.11	c ................................ | ****......           15521000 | 4062618      19355573         1920.1      8083.3    114713228
2015.83/508.15	c ................................ | ****......           16086000 | 4208806      20072420         2016.0      7979.1    118884547
2111.81/532.26	c ................................ | ****......           16661000 | 4358699      20778284         2112.1      7888.4    121221909
2208.14/556.40	c ................................ | ****......           17197000 | 4500499      21468282         2208.2      7788.0    122858900
2303.96/580.51	c ................................ | ****......           17737000 | 4642236      22143945         2304.0      7698.3    123681990
2399.91/604.64	c ................................ | *****.....           18280000 | 4785223      22812968         2400.1      7616.3    124735133
2496.02/628.82	c ................................ | *****.....           18803000 | 4924043      23471056         2496.1      7532.9    126457015
2592.03/652.91	c ................................ | *****.....           19323000 | 5064306      24150619         2592.1      7454.6    129287538
2687.99/677.00	c ................................ | *****.....           19852000 | 5206723      24811960         2688.0      7385.4    131794002
2784.02/701.10	c ................................ | *****.....           20370000 | 5343561      25458107         2784.1      7316.7    131976684
2879.95/725.23	c ................................ | ******....           20875000 | 5475999      26096482         2880.1      7248.1    133285874
2975.98/749.33	c ................................ | ******....           21384000 | 5610803      26712080         2976.1      7185.2    134692708
3071.91/773.43	c ................................ | ******....           21896000 | 5748044      27328030         3072.1      7127.5    134725059
3168.21/797.62	c ................................ | ******....           22410000 | 5881369      27957597         3168.3      7073.1    135159073
3263.97/821.61	c ................................ | ******....           22920000 | 6015756      28592903         3264.0      7022.1    135475897
3360.05/845.72	c ................................ | *******...           23427000 | 6148796      29235862         3360.1      6972.0    136602122
3455.79/869.89	c ................................ | *******...           23901000 | 6274194      29852872         3456.2      6915.5    137111577
3551.82/893.95	c ................................ | *******...           24388000 | 6402995      30486485         3552.1      6865.9    137635732
3647.62/918.19	c ................................ | *******...           24913000 | 6542558      31141276         3648.0      6829.2    139400507
3744.13/942.40	c ................................ | *******...           25408000 | 6676324      31760511         3744.1      6786.1    140649103
3839.91/966.45	c ................................ | ********..           25939000 | 6816035      32392874         3840.1      6754.7    140784640
3935.99/990.63	c ................................ | ********..           26494000 | 6964326      33019159         3936.1      6731.0    141904019
4031.87/1014.75	c ................................ | ********..           27020000 | 7100694      33634876         4032.1      6701.3    142448340
4127.79/1038.85	c ................................ | ********..           27542000 | 7239235      34258551         4128.0      6672.0    143219221
4224.12/1063.00	c ................................ | ********..           28093000 | 7390032      34892249         4224.1      6650.6    144515582
4319.80/1087.06	c ................................ | *********.           28592000 | 7522182      35498121         4320.1      6618.4    146172159
4415.71/1111.17	c ................................ | *********.           29103000 | 7660990      36102457         4416.0      6590.3    147189079
4512.02/1135.33	c ................................ | *********.           29593000 | 7790348      36697572         4512.2      6558.5    147689790
4608.03/1159.43	c ................................ | *********.           30095000 | 7923502      37294622         4608.1      6530.8    147893272
4703.94/1183.54	c ................................ | *********.           30632000 | 8068595      37904127         4704.1      6511.8    148765962
4799.80/1207.66	c CPU limit (4800) reached
4799.80/1207.66	c Time = 4800.06 seconds
4799.80/1207.66	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3263146-1304095775/watcher-3263146-1304095775 -o /tmp/evaluation-result-3263146-1304095775/solver-3263146-1304095775 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263146-1304095775.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.57 7.77 7.92 3/177 3095
/proc/meminfo: memFree=27644536/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=8840 CPUtime=0 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) R 3093 3095 2524 0 -1 4202496 291 0 0 0 0 0 0 0 21 0 1 0 521175191 9052160 233 33554432000 4194304 4338100 140734582137184 18446744073709551615 218905391813 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/3095/statm: 2210 233 146 36 0 115 0

[startup+0.013878 s]
/proc/loadavg: 6.57 7.77 7.92 3/177 3095
/proc/meminfo: memFree=27644536/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=9500 CPUtime=0.01 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) R 3093 3095 2524 0 -1 4202496 451 0 0 0 1 0 0 0 21 0 1 0 521175191 9728000 393 33554432000 4194304 4338100 140734582137184 18446744073709551615 218905391813 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/3095/statm: 2375 393 147 36 0 280 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 9500

[startup+0.100854 s]
/proc/loadavg: 6.57 7.77 7.92 3/177 3095
/proc/meminfo: memFree=27644536/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=15660 CPUtime=0.09 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) R 3093 3095 2524 0 -1 4202496 1962 0 0 0 9 0 0 0 21 0 1 0 521175191 16035840 1904 33554432000 4194304 4338100 140734582137184 18446744073709551615 218905730816 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/3095/statm: 3915 1906 148 36 0 1820 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 15660

[startup+0.300821 s]
/proc/loadavg: 6.57 7.77 7.92 3/177 3095
/proc/meminfo: memFree=27644536/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=89972 CPUtime=0.29 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) R 3093 3095 2524 0 -1 4202496 10829 0 0 0 29 0 0 0 22 0 1 0 521175191 92131328 10355 33554432000 4194304 4338100 140734582137184 18446744073709551615 4206987 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/3095/statm: 22493 10355 348 36 0 20398 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 89972

[startup+0.700752 s]
/proc/loadavg: 6.57 7.77 7.92 3/177 3095
/proc/meminfo: memFree=27644536/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=89972 CPUtime=0.69 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) R 3093 3095 2524 0 -1 4202496 10829 0 0 0 69 0 0 0 24 0 1 0 521175191 92131328 10355 33554432000 4194304 4338100 140734582137184 18446744073709551615 4206993 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/3095/statm: 22493 10355 348 36 0 20398 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 89972

[startup+1.5006 s]
/proc/loadavg: 6.57 7.77 7.92 7/182 3101
/proc/meminfo: memFree=27548456/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=23072 CPUtime=1.49 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) R 3093 3095 2524 0 -1 4202496 12067 0 0 0 148 1 0 0 25 0 1 0 521175191 23625728 3745 33554432000 4194304 4338100 140734582137184 18446744073709551615 4245205 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/3095/statm: 5768 3745 155 36 0 3673 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 23072

[startup+3.10024 s]
/proc/loadavg: 6.57 7.77 7.92 7/182 3101
/proc/meminfo: memFree=27550068/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=303852 CPUtime=4.02 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) S 3093 3095 2524 0 -1 4202496 21104 0 0 0 400 2 0 0 25 0 5 0 521175191 311144448 12674 33554432000 4194304 4338100 140734582137184 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3095/statm: 75963 12674 176 36 0 73868 0
Current children cumulated CPU time (s) 4.02
Current children cumulated vsize (KiB) 303852

[startup+6.30067 s]
/proc/loadavg: 6.68 7.77 7.92 10/186 3106
/proc/meminfo: memFree=27346948/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=442604 CPUtime=16.79 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) S 3093 3095 2524 0 -1 4202496 59470 0 0 0 1667 12 0 0 25 0 5 0 521175191 453226496 51040 33554432000 4194304 4338100 140734582137184 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3095/statm: 110651 51040 176 36 0 108556 0
[pid=3095/tid=3102] ppid=3093 vsize=442604 CPUtime=3.5 cores=0,2,4,6
/proc/3095/task/3102/stat : 3102 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 11527 0 0 0 348 2 0 0 25 0 5 0 521175470 453226496 51040 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3095/tid=3103] ppid=3093 vsize=442604 CPUtime=3.51 cores=0,2,4,6
/proc/3095/task/3103/stat : 3103 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 11448 0 0 0 348 3 0 0 25 0 5 0 521175470 453226496 51040 33554432000 4194304 4338100 140734582137184 18446744073709551615 218917545115 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3095/tid=3104] ppid=3093 vsize=442604 CPUtime=3.5 cores=0,2,4,6
/proc/3095/task/3104/stat : 3104 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 11837 0 0 0 348 2 0 0 25 0 5 0 521175470 453226496 51040 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3095/tid=3105] ppid=3093 vsize=442604 CPUtime=3.48 cores=0,2,4,6
/proc/3095/task/3105/stat : 3105 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 11053 0 0 0 346 2 0 0 25 0 5 0 521175470 453226496 51040 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 16.79
Current children cumulated vsize (KiB) 442604

[startup+12.7005 s]
/proc/loadavg: 6.79 7.78 7.92 10/186 3106
/proc/meminfo: memFree=27035800/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=627148 CPUtime=42.34 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) S 3093 3095 2524 0 -1 4202496 108271 0 0 0 4208 26 0 0 25 0 5 0 521175191 642199552 99841 33554432000 4194304 4338100 140734582137184 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3095/statm: 156787 99841 176 36 0 154692 0
[pid=3095/tid=3102] ppid=3093 vsize=627148 CPUtime=9.89 cores=0,2,4,6
/proc/3095/task/3102/stat : 3102 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 23307 0 0 0 982 7 0 0 25 0 5 0 521175470 642199552 99841 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3095/tid=3103] ppid=3093 vsize=627148 CPUtime=9.89 cores=0,2,4,6
/proc/3095/task/3103/stat : 3103 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 22866 0 0 0 983 6 0 0 25 0 5 0 521175470 642199552 99841 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3095/tid=3104] ppid=3093 vsize=627148 CPUtime=9.9 cores=0,2,4,6
/proc/3095/task/3104/stat : 3104 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 25287 0 0 0 985 5 0 0 25 0 5 0 521175470 642199552 99842 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3095/tid=3105] ppid=3093 vsize=627148 CPUtime=9.84 cores=0,2,4,6
/proc/3095/task/3105/stat : 3105 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 23206 0 0 0 979 5 0 0 25 0 5 0 521175470 642199552 99842 33554432000 4194304 4338100 140734582137184 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 42.34
Current children cumulated vsize (KiB) 627148

[startup+25.5012 s]
/proc/loadavg: 7.05 7.79 7.92 10/186 3106
/proc/meminfo: memFree=26616072/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=889736 CPUtime=93.38 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) S 3093 3095 2524 0 -1 4202496 180085 0 0 0 9294 44 0 0 25 0 5 0 521175191 911089664 171655 33554432000 4194304 4338100 140734582137184 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3095/statm: 222434 171655 176 36 0 220339 0
[pid=3095/tid=3102] ppid=3093 vsize=889736 CPUtime=22.66 cores=0,2,4,6
/proc/3095/task/3102/stat : 3102 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 42633 0 0 0 2253 13 0 0 25 0 5 0 521175470 911089664 171656 33554432000 4194304 4338100 140734582137184 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3095/tid=3103] ppid=3093 vsize=889736 CPUtime=22.68 cores=0,2,4,6
/proc/3095/task/3103/stat : 3103 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 39643 0 0 0 2257 11 0 0 25 0 5 0 521175470 911089664 171656 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3095/tid=3104] ppid=3093 vsize=889736 CPUtime=22.67 cores=0,2,4,6
/proc/3095/task/3104/stat : 3104 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 44287 0 0 0 2258 9 0 0 25 0 5 0 521175470 911089664 171656 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3095/tid=3105] ppid=3093 vsize=889736 CPUtime=22.54 cores=0,2,4,6
/proc/3095/task/3105/stat : 3105 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 39918 0 0 0 2246 8 0 0 25 0 5 0 521175470 911089664 171656 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 93.38
Current children cumulated vsize (KiB) 889736

[startup+51.1006 s]
/proc/loadavg: 7.53 7.84 7.93 10/186 3107
/proc/meminfo: memFree=25980504/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=1292052 CPUtime=195.45 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) S 3093 3095 2524 0 -1 4202496 288553 0 0 0 19467 78 0 0 25 0 5 0 521175191 1323061248 280123 33554432000 4194304 4338100 140734582137184 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3095/statm: 323013 280123 176 36 0 320918 0
[pid=3095/tid=3102] ppid=3093 vsize=1292052 CPUtime=48.05 cores=0,2,4,6
/proc/3095/task/3102/stat : 3102 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 70376 0 0 0 4784 21 0 0 25 0 5 0 521175470 1323061248 280123 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3095/tid=3103] ppid=3093 vsize=1292052 CPUtime=48.25 cores=0,2,4,6
/proc/3095/task/3103/stat : 3103 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 58909 0 0 0 4808 17 0 0 25 0 5 0 521175470 1323061248 280123 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3095/tid=3104] ppid=3093 vsize=1292052 CPUtime=48.24 cores=0,2,4,6
/proc/3095/task/3104/stat : 3104 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 71913 0 0 0 4806 18 0 0 19 0 5 0 521175470 1323061248 280123 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3095/tid=3105] ppid=3093 vsize=1292052 CPUtime=48.09 cores=0,2,4,6
/proc/3095/task/3105/stat : 3105 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 73750 0 0 0 4791 18 0 0 19 0 5 0 521175470 1323061248 280123 33554432000 4194304 4338100 140734582137184 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 195.45
Current children cumulated vsize (KiB) 1292052

[startup+102.306 s]
/proc/loadavg: 7.87 7.88 7.94 10/186 3108
/proc/meminfo: memFree=25096740/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=1840504 CPUtime=399.43 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) S 3093 3095 2524 0 -1 4202496 440400 0 0 0 39815 128 0 0 25 0 5 0 521175191 1884676096 431970 33554432000 4194304 4338100 140734582137184 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3095/statm: 460126 431970 176 36 0 458031 0
[pid=3095/tid=3102] ppid=3093 vsize=1840504 CPUtime=98.91 cores=0,2,4,6
/proc/3095/task/3102/stat : 3102 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 107561 0 0 0 9856 35 0 0 20 0 5 0 521175470 1884676096 431970 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3095/tid=3103] ppid=3093 vsize=1840504 CPUtime=99.21 cores=0,2,4,6
/proc/3095/task/3103/stat : 3103 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 89434 0 0 0 9893 28 0 0 25 0 5 0 521175470 1884676096 431970 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3095/tid=3104] ppid=3093 vsize=1840504 CPUtime=99.31 cores=0,2,4,6
/proc/3095/task/3104/stat : 3104 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 119645 0 0 0 9899 32 0 0 25 0 5 0 521175470 1884676096 431970 33554432000 4194304 4338100 140734582137184 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3095/tid=3105] ppid=3093 vsize=1840504 CPUtime=99.2 cores=0,2,4,6
/proc/3095/task/3105/stat : 3105 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 110155 0 0 0 9889 31 0 0 25 0 5 0 521175470 1884676096 431970 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 399.43
Current children cumulated vsize (KiB) 1840504

[startup+162.3 s]
/proc/loadavg: 8.10 7.93 7.95 10/186 3110
/proc/meminfo: memFree=24266408/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=2531464 CPUtime=638.67 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) S 3093 3095 2524 0 -1 4202496 594897 0 0 0 63692 175 0 0 25 0 5 0 521175191 2592219136 586467 33554432000 4194304 4338100 140734582137184 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3095/statm: 632866 586467 176 36 0 630771 0
[pid=3095/tid=3102] ppid=3093 vsize=2531464 CPUtime=158.75 cores=0,2,4,6
/proc/3095/task/3102/stat : 3102 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 146076 0 0 0 15828 47 0 0 25 0 5 0 521175470 2592219136 586467 33554432000 4194304 4338100 140734582137184 18446744073709551615 4256601 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3095/tid=3103] ppid=3093 vsize=2531464 CPUtime=159.09 cores=0,2,4,6
/proc/3095/task/3103/stat : 3103 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 125777 0 0 0 15872 37 0 0 25 0 5 0 521175470 2592219136 586467 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3095/tid=3104] ppid=3093 vsize=2531464 CPUtime=159.23 cores=0,2,4,6
/proc/3095/task/3104/stat : 3104 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 159596 0 0 0 15878 45 0 0 25 0 5 0 521175470 2592219136 586467 33554432000 4194304 4338100 140734582137184 18446744073709551615 4284177 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3095/tid=3105] ppid=3093 vsize=2531464 CPUtime=158.78 cores=0,2,4,6
/proc/3095/task/3105/stat : 3105 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 149843 0 0 0 15835 43 0 0 25 0 5 0 521175470 2592219136 586467 33554432000 4194304 4338100 140734582137184 18446744073709551615 4256139 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 638.67
Current children cumulated vsize (KiB) 2531464

[startup+222.301 s]
/proc/loadavg: 8.10 7.95 7.96 9/186 3111
/proc/meminfo: memFree=23424160/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=2992660 CPUtime=877.5 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) S 3093 3095 2524 0 -1 4202496 706153 0 0 0 87533 217 0 0 25 0 5 0 521175191 3064483840 697723 33554432000 4194304 4338100 140734582137184 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3095/statm: 748165 697723 176 36 0 746070 0
[pid=3095/tid=3102] ppid=3093 vsize=2992660 CPUtime=218.4 cores=0,2,4,6
/proc/3095/task/3102/stat : 3102 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 182549 0 0 0 21779 61 0 0 25 0 5 0 521175470 3064483840 697724 33554432000 4194304 4338100 140734582137184 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3095/tid=3103] ppid=3093 vsize=2992660 CPUtime=218.74 cores=0,2,4,6
/proc/3095/task/3103/stat : 3103 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 146972 0 0 0 21830 44 0 0 25 0 5 0 521175470 3064483840 697724 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3095/tid=3104] ppid=3093 vsize=2992660 CPUtime=219.06 cores=0,2,4,6
/proc/3095/task/3104/stat : 3104 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 190347 0 0 0 21850 56 0 0 25 0 5 0 521175470 3064483840 697724 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3095/tid=3105] ppid=3093 vsize=2992660 CPUtime=218.47 cores=0,2,4,6
/proc/3095/task/3105/stat : 3105 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 172681 0 0 0 21795 52 0 0 25 0 5 0 521175470 3064483840 697724 33554432000 4194304 4338100 140734582137184 18446744073709551615 4263842 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 877.5
Current children cumulated vsize (KiB) 2992660

################
# More data... #
################

/proc/3095/task/3102/stat : 3102 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 383722 0 0 0 75363 147 0 0 25 0 5 0 521175470 5457633280 1275109 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3095/tid=3103] ppid=3093 vsize=5329720 CPUtime=757.31 cores=0,2,4,6
/proc/3095/task/3103/stat : 3103 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 255472 0 0 0 75632 99 0 0 25 0 5 0 521175470 5457633280 1275109 33554432000 4194304 4338100 140734582137184 18446744073709551615 4284177 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3095/tid=3104] ppid=3093 vsize=5329720 CPUtime=756.58 cores=0,2,4,6
/proc/3095/task/3104/stat : 3104 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 324727 0 0 0 75549 109 0 0 25 0 5 0 521175470 5457633280 1275109 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3095/tid=3105] ppid=3093 vsize=5329720 CPUtime=755.99 cores=0,2,4,6
/proc/3095/task/3105/stat : 3105 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 306013 0 0 0 75497 102 0 0 25 0 5 0 521175470 5457633280 1275109 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3027.8
Current children cumulated vsize (KiB) 5329720

[startup+822.301 s]
/proc/loadavg: 8.09 8.06 8.00 10/186 3141
/proc/meminfo: memFree=20125676/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=5416228 CPUtime=3266.76 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) S 3093 3095 2524 0 -1 4202496 1307880 0 0 0 326193 483 0 0 25 0 5 0 521175191 5546217472 1299450 33554432000 4194304 4338100 140734582137184 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3095/statm: 1354057 1299450 176 36 0 1351962 0
[pid=3095/tid=3102] ppid=3093 vsize=5416228 CPUtime=814.82 cores=0,2,4,6
/proc/3095/task/3102/stat : 3102 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 393792 0 0 0 81327 155 0 0 25 0 5 0 521175470 5546217472 1299450 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3095/tid=3103] ppid=3093 vsize=5416228 CPUtime=816.88 cores=0,2,4,6
/proc/3095/task/3103/stat : 3103 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 260040 0 0 0 81585 103 0 0 25 0 5 0 521175470 5546217472 1299450 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3095/tid=3104] ppid=3093 vsize=5416228 CPUtime=816.53 cores=0,2,4,6
/proc/3095/task/3104/stat : 3104 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 329947 0 0 0 81539 114 0 0 25 0 5 0 521175470 5546217472 1299450 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3095/tid=3105] ppid=3093 vsize=5416228 CPUtime=815.71 cores=0,2,4,6
/proc/3095/task/3105/stat : 3105 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 310496 0 0 0 81464 107 0 0 25 0 5 0 521175470 5546217472 1299450 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3266.76
Current children cumulated vsize (KiB) 5416228

[startup+882.301 s]
/proc/loadavg: 8.18 8.09 8.02 9/186 3147
/proc/meminfo: memFree=19937800/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=5478288 CPUtime=3505.62 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) S 3093 3095 2524 0 -1 4202496 1332275 0 0 0 350058 504 0 0 25 0 5 0 521175191 5609766912 1323845 33554432000 4194304 4338100 140734582137184 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3095/statm: 1369572 1323845 176 36 0 1367477 0
[pid=3095/tid=3102] ppid=3093 vsize=5478288 CPUtime=874.57 cores=0,2,4,6
/proc/3095/task/3102/stat : 3102 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 409271 0 0 0 87293 164 0 0 25 0 5 0 521175470 5609766912 1323845 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3095/tid=3103] ppid=3093 vsize=5478288 CPUtime=876.48 cores=0,2,4,6
/proc/3095/task/3103/stat : 3103 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 263044 0 0 0 87541 107 0 0 25 0 5 0 521175470 5609766912 1323845 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3095/tid=3104] ppid=3093 vsize=5478288 CPUtime=876.19 cores=0,2,4,6
/proc/3095/task/3104/stat : 3104 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 333085 0 0 0 87500 119 0 0 25 0 5 0 521175470 5609766912 1323845 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3095/tid=3105] ppid=3093 vsize=5478288 CPUtime=875.57 cores=0,2,4,6
/proc/3095/task/3105/stat : 3105 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 313270 0 0 0 87446 111 0 0 25 0 5 0 521175470 5609766912 1323845 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3505.62
Current children cumulated vsize (KiB) 5478288

[startup+942.301 s]
/proc/loadavg: 7.96 8.07 8.01 10/186 3150
/proc/meminfo: memFree=19659500/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=5606432 CPUtime=3743.73 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) S 3093 3095 2524 0 -1 4202496 1371954 0 0 0 373846 527 0 0 25 0 5 0 521175191 5740986368 1363524 33554432000 4194304 4338100 140734582137184 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3095/statm: 1401608 1363524 176 36 0 1399513 0
[pid=3095/tid=3102] ppid=3093 vsize=5606432 CPUtime=933.92 cores=0,2,4,6
/proc/3095/task/3102/stat : 3102 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 427974 0 0 0 93220 172 0 0 25 0 5 0 521175470 5740986368 1363524 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3095/tid=3103] ppid=3093 vsize=5606432 CPUtime=936.09 cores=0,2,4,6
/proc/3095/task/3103/stat : 3103 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 266092 0 0 0 93498 111 0 0 25 0 5 0 521175470 5740986368 1363524 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3095/tid=3104] ppid=3093 vsize=5606432 CPUtime=935.69 cores=0,2,4,6
/proc/3095/task/3104/stat : 3104 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 339566 0 0 0 93445 124 0 0 25 0 5 0 521175470 5740986368 1363524 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3095/tid=3105] ppid=3093 vsize=5606432 CPUtime=935.21 cores=0,2,4,6
/proc/3095/task/3105/stat : 3105 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 324717 0 0 0 93404 117 0 0 25 0 5 0 521175470 5740986368 1363524 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3743.73
Current children cumulated vsize (KiB) 5606432

[startup+1002.3 s]
/proc/loadavg: 8.06 8.07 8.01 10/186 3152
/proc/meminfo: memFree=19464908/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=5666684 CPUtime=3982.6 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) S 3093 3095 2524 0 -1 4202496 1396192 0 0 0 397714 546 0 0 25 0 5 0 521175191 5802684416 1387762 33554432000 4194304 4338100 140734582137184 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3095/statm: 1416671 1387762 176 36 0 1414576 0
[pid=3095/tid=3102] ppid=3093 vsize=5666684 CPUtime=993.61 cores=0,2,4,6
/proc/3095/task/3102/stat : 3102 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 437179 0 0 0 99184 177 0 0 25 0 5 0 521175470 5802684416 1387762 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3095/tid=3103] ppid=3093 vsize=5666684 CPUtime=995.88 cores=0,2,4,6
/proc/3095/task/3103/stat : 3103 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 269135 0 0 0 99473 115 0 0 25 0 5 0 521175470 5802684416 1387762 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3095/tid=3104] ppid=3093 vsize=5666684 CPUtime=995.57 cores=0,2,4,6
/proc/3095/task/3104/stat : 3104 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 347475 0 0 0 99427 130 0 0 25 0 5 0 521175470 5802684416 1387762 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3095/tid=3105] ppid=3093 vsize=5666684 CPUtime=994.71 cores=0,2,4,6
/proc/3095/task/3105/stat : 3105 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 328798 0 0 0 99350 121 0 0 25 0 5 0 521175470 5802684416 1387762 33554432000 4194304 4338100 140734582137184 18446744073709551615 4243536 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3982.6
Current children cumulated vsize (KiB) 5666684

[startup+1062.3 s]
/proc/loadavg: 8.02 8.05 8.00 10/186 3154
/proc/meminfo: memFree=19206576/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=5882768 CPUtime=4221.32 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) S 3093 3095 2524 0 -1 4202496 1425854 0 0 0 421564 568 0 0 25 0 5 0 521175191 6023954432 1417424 33554432000 4194304 4338100 140734582137184 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3095/statm: 1470692 1417424 176 36 0 1468597 0
[pid=3095/tid=3102] ppid=3093 vsize=5882768 CPUtime=1053.05 cores=0,2,4,6
/proc/3095/task/3102/stat : 3102 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 453587 0 0 0 105119 186 0 0 25 0 5 0 521175470 6023954432 1417424 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3095/tid=3103] ppid=3093 vsize=5882768 CPUtime=1055.7 cores=0,2,4,6
/proc/3095/task/3103/stat : 3103 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 272179 0 0 0 105450 120 0 0 25 0 5 0 521175470 6023954432 1417424 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3095/tid=3104] ppid=3093 vsize=5882768 CPUtime=1055.23 cores=0,2,4,6
/proc/3095/task/3104/stat : 3104 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 355093 0 0 0 105387 136 0 0 25 0 5 0 521175470 6023954432 1417424 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3095/tid=3105] ppid=3093 vsize=5882768 CPUtime=1054.54 cores=0,2,4,6
/proc/3095/task/3105/stat : 3105 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 331390 0 0 0 105329 125 0 0 25 0 5 0 521175470 6023954432 1417424 33554432000 4194304 4338100 140734582137184 18446744073709551615 218905126137 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4221.32
Current children cumulated vsize (KiB) 5882768

[startup+1122.3 s]
/proc/loadavg: 8.07 8.06 8.00 10/186 3155
/proc/meminfo: memFree=19009348/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=6048164 CPUtime=4460.29 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) S 3093 3095 2524 0 -1 4202496 1459191 0 0 0 445440 589 0 0 25 0 5 0 521175191 6193319936 1450761 33554432000 4194304 4338100 140734582137184 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3095/statm: 1512041 1450761 176 36 0 1509946 0
[pid=3095/tid=3102] ppid=3093 vsize=6048164 CPUtime=1112.62 cores=0,2,4,6
/proc/3095/task/3102/stat : 3102 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 467588 0 0 0 111069 193 0 0 25 0 5 0 521175470 6193319936 1450761 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3095/tid=3103] ppid=3093 vsize=6048164 CPUtime=1115.5 cores=0,2,4,6
/proc/3095/task/3103/stat : 3103 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 274964 0 0 0 111426 124 0 0 25 0 5 0 521175470 6193319936 1450761 33554432000 4194304 4338100 140734582137184 18446744073709551615 4257007 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3095/tid=3104] ppid=3093 vsize=6048164 CPUtime=1114.92 cores=0,2,4,6
/proc/3095/task/3104/stat : 3104 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 365885 0 0 0 111353 139 0 0 25 0 5 0 521175470 6193319936 1450761 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3095/tid=3105] ppid=3093 vsize=6048164 CPUtime=1114.41 cores=0,2,4,6
/proc/3095/task/3105/stat : 3105 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 337149 0 0 0 111312 129 0 0 25 0 5 0 521175470 6193319936 1450761 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4460.29
Current children cumulated vsize (KiB) 6048164

[startup+1182.3 s]
/proc/loadavg: 8.15 8.08 8.01 10/186 3157
/proc/meminfo: memFree=18915788/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=6100708 CPUtime=4699.13 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) S 3093 3095 2524 0 -1 4202496 1481093 0 0 0 469304 609 0 0 25 0 5 0 521175191 6247124992 1472663 33554432000 4194304 4338100 140734582137184 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3095/statm: 1525177 1472663 176 36 0 1523082 0
[pid=3095/tid=3102] ppid=3093 vsize=6100708 CPUtime=1172.21 cores=0,2,4,6
/proc/3095/task/3102/stat : 3102 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 480408 0 0 0 117018 203 0 0 25 0 5 0 521175470 6247124992 1472663 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3095/tid=3103] ppid=3093 vsize=6100708 CPUtime=1175.27 cores=0,2,4,6
/proc/3095/task/3103/stat : 3103 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 277759 0 0 0 117399 128 0 0 25 0 5 0 521175470 6247124992 1472663 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3095/tid=3104] ppid=3093 vsize=6100708 CPUtime=1174.6 cores=0,2,4,6
/proc/3095/task/3104/stat : 3104 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 368862 0 0 0 117319 141 0 0 25 0 5 0 521175470 6247124992 1472663 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3095/tid=3105] ppid=3093 vsize=6100708 CPUtime=1174.22 cores=0,2,4,6
/proc/3095/task/3105/stat : 3105 (ttsth-5-4) R 3093 3095 2524 0 -1 4202560 340459 0 0 0 117289 133 0 0 25 0 5 0 521175470 6247124992 1472663 33554432000 4194304 4338100 140734582137184 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4699.13
Current children cumulated vsize (KiB) 6100708



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1207.7 s]
/proc/loadavg: 7.79 8.00 7.99 6/183 3191
/proc/meminfo: memFree=22417900/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=0 CPUtime=4800.11 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) Z 3093 3095 2524 0 -1 4203548 1501298 0 0 0 479384 627 0 0 18 0 2 0 521175191 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3095/statm: 0 0 0 0 0 0 0
[pid=3095/tid=3105] ppid=3093 vsize=0 CPUtime=1199.53 cores=0,2,4,6
/proc/3095/task/3105/stat : 3105 (ttsth-5-4) R 3093 3095 2524 0 -1 4203588 346938 0 0 0 119811 142 0 0 25 0 2 0 521175470 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.11
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1207.82 s]
/proc/loadavg: 7.79 8.00 7.99 6/183 3191
/proc/meminfo: memFree=22417900/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=0 CPUtime=4800.21 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) Z 3093 3095 2524 0 -1 4203548 1501298 0 0 0 479384 637 0 0 18 0 2 0 521175191 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3095/statm: 0 0 0 0 0 0 0
[pid=3095/tid=3105] ppid=3093 vsize=0 CPUtime=1199.64 cores=0,2,4,6
/proc/3095/task/3105/stat : 3105 (ttsth-5-4) R 3093 3095 2524 0 -1 4203588 346938 0 0 0 119811 153 0 0 25 0 2 0 521175470 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.21
Current children cumulated vsize (KiB) 0

[startup+1207.9 s]
/proc/loadavg: 7.79 8.00 7.99 6/183 3191
/proc/meminfo: memFree=22417900/32951124 swapFree=67111356/67111528
[pid=3095] ppid=3093 vsize=0 CPUtime=4800.3 cores=0,2,4,6
/proc/3095/stat : 3095 (ttsth-5-4) Z 3093 3095 2524 0 -1 4203548 1501298 0 0 0 479384 646 0 0 18 0 2 0 521175191 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3095/statm: 0 0 0 0 0 0 0
[pid=3095/tid=3105] ppid=3093 vsize=0 CPUtime=1199.72 cores=0,2,4,6
/proc/3095/task/3105/stat : 3105 (ttsth-5-4) R 3093 3095 2524 0 -1 4203588 346938 0 0 0 119811 161 0 0 25 0 2 0 521175470 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.3
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1207.92
CPU time (s): 4800.33
CPU user time (s): 4793.84
CPU system time (s): 6.48201
CPU usage (%): 397.403
Max. virtual memory (cumulated for all children) (KiB): 6171676

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4793.84
system time used= 6.48201
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1501298
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 17956
involuntary context switches= 32854

runsolver used 1.9927 second user time and 4.52031 second system time

The end

Launcher Data 

Begin job on node137 at 2011-04-29 18:49:35
IDJOB=3263146
IDBENCH=83930
IDSOLVER=1654
FILE ID=node137/3263146-1304095775
RUNJOBID= node137-1304095775-3074
PBS_JOBID= 13170581
Free space on /tmp= 71616 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT03/handmade/anton/UNSAT/SGI_30_50_30_20_5-log.shuffled-as.sat03-111.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3263146-1304095775/watcher-3263146-1304095775 -o /tmp/evaluation-result-3263146-1304095775/solver-3263146-1304095775 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263146-1304095775.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= ab60d1d6c6130bc12a1aa4855d06424a
RANDOM SEED=378359717

node137.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      27645064 kB
Buffers:        919020 kB
Cached:        3851808 kB
SwapCached:         56 kB
Active:         342280 kB
Inactive:      4498680 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      27645064 kB
SwapTotal:    67111528 kB
SwapFree:     67111356 kB
Dirty:            3288 kB
Writeback:           0 kB
AnonPages:       69512 kB
Mapped:          14872 kB
Slab:           401836 kB
PageTables:       4700 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   213408 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71616 MiB
End job on node137 at 2011-04-29 19:09:45