Trace number 3265654

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.19 1203.44

General information on the benchmark

NameSAT05/crafted/sorge05/QG7a/
gensys-ukn009.shuffled-as.sat05-3849.cnf
MD5SUMf2a1eb7937a7cda114ce344a30ca1145
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark824.93
Satisfiable
(Un)Satisfiability was proved
Number of variables2969
Number of clauses18718
Sum of the clauses size134151
Maximum clause length13
Minimum clause length1
Number of clauses of size 142
Number of clauses of size 2435
Number of clauses of size 32341
Number of clauses of size 4498
Number of clauses of size 5677
Number of clauses of size over 514725

Solver Data 

0.01/0.04	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265654-1304102391.cnf 
0.01/0.04	c Using SATTIMEOUT of 4800
0.01/0.04	c Using SATRAM of 15500
0.01/0.04	c Size of subset node = 32
0.01/0.04	c Variables = 2969
0.01/0.04	c Clauses = 18718
0.01/0.04	c Literals = 134151
0.01/0.04	c Reduced size:
0.01/0.04	c Variables = 2969
0.01/0.04	c Clauses = 18177
0.01/0.04	c Literals = 130516
0.01/0.04	c a2
0.09/0.19	c a1
0.39/0.40	c Independent sub-problems = 1
0.39/0.40	c Sub-problem with 2913 variables and 18177 clauses
0.39/0.40	c using permutation method 4
0.39/0.40	c count = 1
0.39/0.40	c Ordering finished 0.380 seconds
0.39/0.40	c mapped 0.390 seconds
0.39/0.40	c permuted 0.390 seconds
0.39/0.40	c about to sort
0.39/0.40	c sorted
0.49/0.50	c clauses now 18177
0.49/0.50	c init finished 0.420 seconds
0.49/0.50	c tree built 0.500 seconds
0.49/0.50	c 
0.49/0.50	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.49/0.50	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.49/0.50	c ................................ | ..........                    |                                  0.5         0.0             
0.49/0.50	c ................................ | ..........                  1 |                                  0.5         2.0             
0.49/0.50	c ................................ | ..........                  2 |                                  0.5         4.0             
0.49/0.50	c ................................ | ..........                  3 |                                  0.5         6.0             
0.49/0.50	c ................................ | ..........                  4 |                                  0.5         8.0             
0.49/0.50	c ................................ | ..........                  5 |                                  0.5        10.0             
0.49/0.51	c ................................ | ..........               1000 | 13           652                 0.5      1886.8         2388
95.96/24.41	c ................................ | ..........            3412000 | 111394       1682041            96.0     35530.6      4580745
191.86/48.45	c ................................ | ..........            5685000 | 218216       2918048           192.0     29604.7      8216621
287.71/72.47	c ................................ | ..........            7768000 | 310558       3998296           288.0     26971.3     11165407
383.96/96.51	c ................................ | ..........            9906000 | 394495       4992558           384.0     25795.5     14131785
479.83/120.55	c ................................ | *.........           11671000 | 479421       5881499           480.1     24312.1     16741962
575.71/144.58	c ................................ | *.........           13044000 | 572094       6796916           576.0     22644.3     19481294
671.97/168.62	c ................................ | *.........           14247000 | 659315       7722935           672.1     21198.4     21154344
767.83/192.66	c ................................ | *.........           15322000 | 740963       8377467           768.1     19948.7     23545637
863.74/216.67	c ................................ | *.........           16484000 | 811226       8955082           864.0     19078.7     25848042
959.60/240.70	c ................................ | **........           17618000 | 887273       9573337           960.0     18352.1     27262736
1055.88/264.74	c ................................ | **........           18498000 | 953586       10080258         1056.1     17515.9     29053712
1151.75/288.76	c ................................ | **........           19711000 | 1023515      10678328         1152.0     17109.8     30274779
1248.01/312.82	c ................................ | **........           20797000 | 1093822      11206811         1248.1     16663.1     31677212
1343.87/336.86	c ................................ | **........           21758000 | 1165353      11809060         1344.1     16187.7     33153038
1439.74/360.88	c ................................ | ***.......           22504000 | 1232156      12354220         1440.0     15627.5     34398923
1536.00/384.92	c ................................ | ***.......           23479000 | 1292329      12942438         1536.1     15284.9     36019917
1631.85/408.96	c ................................ | ***.......           24526000 | 1363715      13656854         1632.1     15027.4     37712285
1727.89/433.07	c ................................ | ***.......           25421000 | 1423965      14212987         1728.0     14711.1     39049962
1823.95/457.16	c ................................ | ***.......           26381000 | 1482979      14671390         1824.1     14462.5     40002765
1920.00/481.27	c ................................ | ****......           27170000 | 1538457      15069711         1920.0     14151.0     41083098
2016.08/505.37	c ................................ | ****......           27745000 | 1591902      15481369         2016.0     13762.2     42745352
2111.72/529.39	c ................................ | ****......           28692000 | 1640060      15916014         2112.0     13585.1     43582401
2208.17/553.51	c ................................ | ****......           29380000 | 1697427      16362421         2208.2     13305.2     43951727
2303.83/577.59	c ................................ | ****......           30098000 | 1767351      16824359         2304.0     13063.4     44790253
2399.90/601.66	c ................................ | *****.....           30744000 | 1837831      17236286         2400.1     12809.3     45708444
2496.00/625.80	c ................................ | *****.....           31265000 | 1882372      17580619         2496.2     12525.0     46710796
2592.12/649.82	c ................................ | *****.....           31800000 | 1938438      17958716         2592.2     12267.8     47212992
2688.19/673.93	c ................................ | *****.....           32432000 | 1991510      18313348         2688.2     12064.5     48073006
2784.27/698.01	c ................................ | *****.....           32972000 | 2039687      18645954         2784.1     11843.1     49290563
2880.32/722.12	c ................................ | ******....           33521000 | 2090828      18952230         2880.1     11638.7     49644430
2975.99/746.14	c ................................ | ******....           34121000 | 2147445      19228673         2976.1     11465.0     50139883
3072.04/770.25	c ................................ | ******....           34716000 | 2188931      19514501         3072.1     11300.3     51361692
3168.09/794.30	c ................................ | ******....           35553000 | 2232911      19873996         3168.0     11222.4     52051097
3263.77/818.37	c ................................ | ******....           36328000 | 2272576      20205921         3264.0     11129.9     52726411
3359.83/842.47	c ................................ | *******...           37003000 | 2325254      20600720         3360.0     11012.7     53420787
3456.31/866.61	c ................................ | *******...           37593000 | 2379843      20946163         3456.3     10876.6     54556633
3552.02/890.61	c ................................ | *******...           38072000 | 2421383      21225557         3552.0     10718.4     55622217
3648.28/914.72	c ................................ | *******...           38476000 | 2462590      21485549         3648.4     10546.1     56773209
3744.15/938.70	c ................................ | *******...           39004000 | 2504596      21755037         3744.2     10417.3     58025930
3840.01/962.72	c ................................ | ********..           39846000 | 2558588      22064451         3840.1     10376.3     58721952
3935.90/986.74	c ................................ | ********..           40516000 | 2605727      22373905         3936.1     10293.5     59007054
4031.75/1010.77	c ................................ | ********..           41190000 | 2652272      22695135         4032.0     10215.7     59107124
4128.01/1034.81	c ................................ | ********..           41639000 | 2691676      22930721         4128.1     10086.8     59671037
4224.19/1058.98	c ................................ | ********..           42174000 | 2746805      23245784         4224.1      9984.1     59896641
4320.28/1083.07	c ................................ | *********.           42842000 | 2817478      23639134         4320.2      9916.7     60609854
4416.37/1107.14	c ................................ | *********.           43300000 | 2878528      23902875         4416.3      9804.5     61281801
4512.46/1131.20	c ................................ | *********.           43756000 | 2929449      24153386         4512.3      9697.0     61838779
4608.18/1155.27	c ................................ | *********.           44160000 | 2970768      24378555         4608.2      9582.9     62217791
4704.29/1179.32	c ................................ | *********.           44626000 | 3025333      24641733         4704.2      9486.5     62373961

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3265654-1304102391/watcher-3265654-1304102391 -o /tmp/evaluation-result-3265654-1304102391/solver-3265654-1304102391 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265654-1304102391.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 4.35 6.79 7.61 4/177 20407
/proc/meminfo: memFree=17905952/32951124 swapFree=67111528/67111528
[pid=20406] ppid=20404 vsize=9104 CPUtime=0 cores=0,2,4,6
/proc/20406/stat : 20406 (ttsth-5-4) R 20404 20406 19929 0 -1 4202496 333 0 0 0 0 0 0 0 23 0 1 0 521841715 9322496 275 33554432000 4194304 4338100 140737323047840 18446744073709551615 216634176197 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/20406/statm: 2276 275 147 36 0 181 0

[startup+0.0114359 s]
/proc/loadavg: 4.35 6.79 7.61 4/177 20407
/proc/meminfo: memFree=17905952/32951124 swapFree=67111528/67111528
[pid=20406] ppid=20404 vsize=9512 CPUtime=0.01 cores=0,2,4,6
/proc/20406/stat : 20406 (ttsth-5-4) R 20404 20406 19929 0 -1 4202496 445 0 0 0 1 0 0 0 23 0 1 0 521841715 9740288 387 33554432000 4194304 4338100 140737323047840 18446744073709551615 216634515200 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/20406/statm: 2378 387 147 36 0 283 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 9512

[startup+0.100429 s]
/proc/loadavg: 4.35 6.79 7.61 4/177 20407
/proc/meminfo: memFree=17905952/32951124 swapFree=67111528/67111528
[pid=20406] ppid=20404 vsize=26928 CPUtime=0.09 cores=0,2,4,6
/proc/20406/stat : 20406 (ttsth-5-4) R 20404 20406 19929 0 -1 4202496 4353 0 0 0 9 0 0 0 23 0 1 0 521841715 27574272 4293 33554432000 4194304 4338100 140737323047840 18446744073709551615 4206990 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/20406/statm: 6732 4293 156 36 0 4637 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 26928

[startup+0.300388 s]
/proc/loadavg: 4.35 6.79 7.61 4/177 20407
/proc/meminfo: memFree=17905952/32951124 swapFree=67111528/67111528
[pid=20406] ppid=20404 vsize=26996 CPUtime=0.29 cores=0,2,4,6
/proc/20406/stat : 20406 (ttsth-5-4) R 20404 20406 19929 0 -1 4202496 4440 0 0 0 29 0 0 0 25 0 1 0 521841715 27643904 4307 33554432000 4194304 4338100 140737323047840 18446744073709551615 4245192 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/20406/statm: 6749 4307 159 36 0 4654 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 26996

[startup+0.700228 s]
/proc/loadavg: 4.35 6.79 7.61 4/177 20407
/proc/meminfo: memFree=17905952/32951124 swapFree=67111528/67111528
[pid=20406] ppid=20404 vsize=249396 CPUtime=1.27 cores=0,2,4,6
/proc/20406/stat : 20406 (ttsth-5-4) S 20404 20406 19929 0 -1 4202496 15133 0 0 0 126 1 0 0 25 0 5 0 521841715 255381504 14895 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20406/statm: 62349 14895 176 36 0 60254 0
Current children cumulated CPU time (s) 1.27
Current children cumulated vsize (KiB) 249396

[startup+1.50109 s]
/proc/loadavg: 4.35 6.79 7.61 10/186 20416
/proc/meminfo: memFree=17791004/32951124 swapFree=67111528/67111528
[pid=20406] ppid=20404 vsize=253712 CPUtime=4.48 cores=0,2,4,6
/proc/20406/stat : 20406 (ttsth-5-4) S 20404 20406 19929 0 -1 4202496 17211 0 0 0 446 2 0 0 25 0 5 0 521841715 259801088 16973 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20406/statm: 63428 16973 176 36 0 61333 0
[pid=20406/tid=20413] ppid=20404 vsize=253712 CPUtime=0.99 cores=0,2,4,6
/proc/20406/task/20413/stat : 20413 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 459 0 0 0 99 0 0 0 19 0 5 0 521841766 259801088 16973 33554432000 4194304 4338100 140737323047840 18446744073709551615 216633910740 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20406/tid=20414] ppid=20404 vsize=253712 CPUtime=0.99 cores=0,2,4,6
/proc/20406/task/20414/stat : 20414 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 782 0 0 0 99 0 0 0 19 0 5 0 521841766 259801088 16973 33554432000 4194304 4338100 140737323047840 18446744073709551615 4256056 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20406/tid=20415] ppid=20404 vsize=253712 CPUtime=0.98 cores=0,2,4,6
/proc/20406/task/20415/stat : 20415 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 646 0 0 0 98 0 0 0 25 0 5 0 521841766 259801088 16973 33554432000 4194304 4338100 140737323047840 18446744073709551615 4239571 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20406/tid=20416] ppid=20404 vsize=253712 CPUtime=0.99 cores=0,2,4,6
/proc/20406/task/20416/stat : 20416 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 1232 0 0 0 99 0 0 0 19 0 5 0 521841766 259801088 16973 33554432000 4194304 4338100 140737323047840 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4.48
Current children cumulated vsize (KiB) 253712

[startup+3.10081 s]
/proc/loadavg: 4.64 6.81 7.61 9/186 20416
/proc/meminfo: memFree=17766584/32951124 swapFree=67111528/67111528
[pid=20406] ppid=20404 vsize=265076 CPUtime=10.86 cores=0,2,4,6
/proc/20406/stat : 20406 (ttsth-5-4) S 20404 20406 19929 0 -1 4202496 20632 0 0 0 1083 3 0 0 25 0 5 0 521841715 271437824 20394 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20406/statm: 66269 20394 176 36 0 64174 0
[pid=20406/tid=20413] ppid=20404 vsize=265076 CPUtime=2.59 cores=0,2,4,6
/proc/20406/task/20413/stat : 20413 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 998 0 0 0 259 0 0 0 23 0 5 0 521841766 271437824 20394 33554432000 4194304 4338100 140737323047840 18446744073709551615 216633910740 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20406/tid=20414] ppid=20404 vsize=265076 CPUtime=2.59 cores=0,2,4,6
/proc/20406/task/20414/stat : 20414 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 1702 0 0 0 259 0 0 0 23 0 5 0 521841766 271437824 20394 33554432000 4194304 4338100 140737323047840 18446744073709551615 216634176197 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20406/tid=20415] ppid=20404 vsize=265076 CPUtime=2.57 cores=0,2,4,6
/proc/20406/task/20415/stat : 20415 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 1326 0 0 0 257 0 0 0 25 0 5 0 521841766 271437824 20394 33554432000 4194304 4338100 140737323047840 18446744073709551615 216633910502 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20406/tid=20416] ppid=20404 vsize=265076 CPUtime=2.58 cores=0,2,4,6
/proc/20406/task/20416/stat : 20416 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 2514 0 0 0 258 0 0 0 22 0 5 0 521841766 271437824 20394 33554432000 4194304 4338100 140737323047840 18446744073709551615 216634211833 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 10.86
Current children cumulated vsize (KiB) 265076

[startup+6.30024 s]
/proc/loadavg: 4.64 6.81 7.61 8/186 20416
/proc/meminfo: memFree=17705700/32951124 swapFree=67111528/67111528
[pid=20406] ppid=20404 vsize=287012 CPUtime=23.64 cores=0,2,4,6
/proc/20406/stat : 20406 (ttsth-5-4) S 20404 20406 19929 0 -1 4202496 28044 0 0 0 2355 9 0 0 25 0 5 0 521841715 293900288 27806 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20406/statm: 71753 27806 176 36 0 69658 0
[pid=20406/tid=20413] ppid=20404 vsize=287012 CPUtime=5.78 cores=0,2,4,6
/proc/20406/task/20413/stat : 20413 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 2209 0 0 0 576 2 0 0 25 0 5 0 521841766 293900288 27806 33554432000 4194304 4338100 140737323047840 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20406/tid=20414] ppid=20404 vsize=287012 CPUtime=5.78 cores=0,2,4,6
/proc/20406/task/20414/stat : 20414 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 4190 0 0 0 578 0 0 0 25 0 5 0 521841766 293900288 27806 33554432000 4194304 4338100 140737323047840 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20406/tid=20415] ppid=20404 vsize=287012 CPUtime=5.76 cores=0,2,4,6
/proc/20406/task/20415/stat : 20415 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 2847 0 0 0 574 2 0 0 25 0 5 0 521841766 293900288 27806 33554432000 4194304 4338100 140737323047840 18446744073709551615 216633911294 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20406/tid=20416] ppid=20404 vsize=287012 CPUtime=5.78 cores=0,2,4,6
/proc/20406/task/20416/stat : 20416 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 4706 0 0 0 577 1 0 0 25 0 5 0 521841766 293900288 27806 33554432000 4194304 4338100 140737323047840 18446744073709551615 4265585 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 23.64
Current children cumulated vsize (KiB) 287012

[startup+12.7011 s]
/proc/loadavg: 5.16 6.85 7.62 9/186 20417
/proc/meminfo: memFree=17603520/32951124 swapFree=67111528/67111528
[pid=20406] ppid=20404 vsize=335972 CPUtime=49.22 cores=0,2,4,6
/proc/20406/stat : 20406 (ttsth-5-4) S 20404 20406 19929 0 -1 4202496 41635 0 0 0 4898 24 0 0 25 0 5 0 521841715 344035328 41397 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20406/statm: 83993 41397 176 36 0 81898 0
[pid=20406/tid=20413] ppid=20404 vsize=335972 CPUtime=12.15 cores=0,2,4,6
/proc/20406/task/20413/stat : 20413 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 4686 0 0 0 1206 9 0 0 25 0 5 0 521841766 344035328 41397 33554432000 4194304 4338100 140737323047840 18446744073709551615 4240550 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20406/tid=20414] ppid=20404 vsize=335972 CPUtime=12.19 cores=0,2,4,6
/proc/20406/task/20414/stat : 20414 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 7871 0 0 0 1217 2 0 0 25 0 5 0 521841766 344035328 41397 33554432000 4194304 4338100 140737323047840 18446744073709551615 4255956 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20406/tid=20415] ppid=20404 vsize=335972 CPUtime=12.16 cores=0,2,4,6
/proc/20406/task/20415/stat : 20415 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 5664 0 0 0 1209 7 0 0 25 0 5 0 521841766 344035328 41397 33554432000 4194304 4338100 140737323047840 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20406/tid=20416] ppid=20404 vsize=335972 CPUtime=12.19 cores=0,2,4,6
/proc/20406/task/20416/stat : 20416 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 9322 0 0 0 1216 3 0 0 25 0 5 0 521841766 344035328 41397 33554432000 4194304 4338100 140737323047840 18446744073709551615 4256111 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 49.22
Current children cumulated vsize (KiB) 335972

[startup+25.5009 s]
/proc/loadavg: 5.59 6.89 7.62 9/186 20417
/proc/meminfo: memFree=17439932/32951124 swapFree=67111528/67111528
[pid=20406] ppid=20404 vsize=477548 CPUtime=100.35 cores=0,2,4,6
/proc/20406/stat : 20406 (ttsth-5-4) S 20404 20406 19929 0 -1 4202496 64095 0 0 0 9974 61 0 0 25 0 5 0 521841715 489009152 63857 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20406/statm: 119387 63858 176 36 0 117292 0
[pid=20406/tid=20413] ppid=20404 vsize=477548 CPUtime=24.88 cores=0,2,4,6
/proc/20406/task/20413/stat : 20413 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 8189 0 0 0 2462 26 0 0 25 0 5 0 521841766 489009152 63858 33554432000 4194304 4338100 140737323047840 18446744073709551615 4254788 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20406/tid=20414] ppid=20404 vsize=477548 CPUtime=24.99 cores=0,2,4,6
/proc/20406/task/20414/stat : 20414 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 15284 0 0 0 2495 4 0 0 25 0 5 0 521841766 489009152 63858 33554432000 4194304 4338100 140737323047840 18446744073709551615 216634176197 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20406/tid=20415] ppid=20404 vsize=477548 CPUtime=24.96 cores=0,2,4,6
/proc/20406/task/20415/stat : 20415 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 10173 0 0 0 2472 24 0 0 25 0 5 0 521841766 489009152 63858 33554432000 4194304 4338100 140737323047840 18446744073709551615 4258020 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20406/tid=20416] ppid=20404 vsize=477548 CPUtime=24.99 cores=0,2,4,6
/proc/20406/task/20416/stat : 20416 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 16358 0 0 0 2494 5 0 0 25 0 5 0 521841766 489009152 63858 33554432000 4194304 4338100 140737323047840 18446744073709551615 216634182560 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 100.35
Current children cumulated vsize (KiB) 477548

[startup+51.1004 s]
/proc/loadavg: 6.41 6.98 7.63 9/186 20418
/proc/meminfo: memFree=17181012/32951124 swapFree=67111528/67111528
[pid=20406] ppid=20404 vsize=603492 CPUtime=202.62 cores=0,2,4,6
/proc/20406/stat : 20406 (ttsth-5-4) S 20404 20406 19929 0 -1 4202496 102908 0 0 0 20112 150 0 0 25 0 5 0 521841715 617975808 102670 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20406/statm: 150873 102670 176 36 0 148778 0
[pid=20406/tid=20413] ppid=20404 vsize=603492 CPUtime=50.35 cores=0,2,4,6
/proc/20406/task/20413/stat : 20413 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 14512 0 0 0 4967 68 0 0 25 0 5 0 521841766 617975808 102670 33554432000 4194304 4338100 140737323047840 18446744073709551615 4285628 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20406/tid=20414] ppid=20404 vsize=603492 CPUtime=50.59 cores=0,2,4,6
/proc/20406/task/20414/stat : 20414 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 27507 0 0 0 5051 8 0 0 25 0 5 0 521841766 617975808 102670 33554432000 4194304 4338100 140737323047840 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20406/tid=20415] ppid=20404 vsize=603492 CPUtime=50.57 cores=0,2,4,6
/proc/20406/task/20415/stat : 20415 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 17610 0 0 0 4994 63 0 0 25 0 5 0 521841766 617975808 102670 33554432000 4194304 4338100 140737323047840 18446744073709551615 4288153 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20406/tid=20416] ppid=20404 vsize=603492 CPUtime=50.58 cores=0,2,4,6
/proc/20406/task/20416/stat : 20416 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 29187 0 0 0 5050 8 0 0 25 0 5 0 521841766 617975808 102670 33554432000 4194304 4338100 140737323047840 18446744073709551615 4286764 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 202.62
Current children cumulated vsize (KiB) 603492

[startup+102.306 s]
/proc/loadavg: 7.42 7.16 7.65 9/186 20419
/proc/meminfo: memFree=16786028/32951124 swapFree=67111528/67111528
[pid=20406] ppid=20404 vsize=827120 CPUtime=407.15 cores=0,2,4,6
/proc/20406/stat : 20406 (ttsth-5-4) S 20404 20406 19929 0 -1 4202496 170106 0 0 0 40340 375 0 0 25 0 5 0 521841715 846970880 169868 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20406/statm: 206780 169868 176 36 0 204685 0
[pid=20406/tid=20413] ppid=20404 vsize=827120 CPUtime=101.34 cores=0,2,4,6
/proc/20406/task/20413/stat : 20413 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 28786 0 0 0 9955 179 0 0 25 0 5 0 521841766 846970880 169868 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20406/tid=20414] ppid=20404 vsize=827120 CPUtime=101.75 cores=0,2,4,6
/proc/20406/task/20414/stat : 20414 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 47793 0 0 0 10162 13 0 0 25 0 5 0 521841766 846970880 169868 33554432000 4194304 4338100 140737323047840 18446744073709551615 216633910515 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20406/tid=20415] ppid=20404 vsize=827120 CPUtime=101.72 cores=0,2,4,6
/proc/20406/task/20415/stat : 20415 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 31955 0 0 0 10006 166 0 0 25 0 5 0 521841766 846970880 169868 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20406/tid=20416] ppid=20404 vsize=827120 CPUtime=101.79 cores=0,2,4,6
/proc/20406/task/20416/stat : 20416 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 47480 0 0 0 10166 13 0 0 25 0 5 0 521841766 846970880 169868 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646329603 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 407.15
Current children cumulated vsize (KiB) 827120

[startup+162.301 s]
/proc/loadavg: 7.79 7.31 7.67 9/186 20421
/proc/meminfo: memFree=16413016/32951124 swapFree=67111528/67111528
[pid=20406] ppid=20404 vsize=1034688 CPUtime=646.81 cores=0,2,4,6
/proc/20406/stat : 20406 (ttsth-5-4) S 20404 20406 19929 0 -1 4202496 234280 0 0 0 64040 641 0 0 25 0 5 0 521841715 1059520512 234042 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20406/statm: 258672 234042 176 36 0 256577 0
[pid=20406/tid=20413] ppid=20404 vsize=1034688 CPUtime=161.33 cores=0,2,4,6
/proc/20406/task/20413/stat : 20413 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 41749 0 0 0 15826 307 0 0 25 0 5 0 521841766 1059520512 234042 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20406/tid=20414] ppid=20404 vsize=1034688 CPUtime=161.44 cores=0,2,4,6
/proc/20406/task/20414/stat : 20414 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 72993 0 0 0 16121 23 0 0 25 0 5 0 521841766 1059520512 234042 33554432000 4194304 4338100 140737323047840 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20406/tid=20415] ppid=20404 vsize=1034688 CPUtime=161.7 cores=0,2,4,6
/proc/20406/task/20415/stat : 20415 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 44661 0 0 0 15881 289 0 0 25 0 5 0 521841766 1059520512 234042 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20406/tid=20416] ppid=20404 vsize=1034688 CPUtime=161.79 cores=0,2,4,6
/proc/20406/task/20416/stat : 20416 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 60785 0 0 0 16160 19 0 0 25 0 5 0 521841766 1059520512 234042 33554432000 4194304 4338100 140737323047840 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 646.81
Current children cumulated vsize (KiB) 1034688


################
# More data... #
################

/proc/meminfo: memFree=14193872/32951124 swapFree=67111528/67111528
[pid=20406] ppid=20404 vsize=2481876 CPUtime=3040.57 cores=0,2,4,6
/proc/20406/stat : 20406 (ttsth-5-4) S 20404 20406 19929 0 -1 4202496 560759 0 0 0 301903 2154 0 0 25 0 5 0 521841715 2541441024 560521 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20406/statm: 620469 560521 176 36 0 618374 0
[pid=20406/tid=20413] ppid=20404 vsize=2481876 CPUtime=761 cores=0,2,4,6
/proc/20406/task/20413/stat : 20413 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 106912 0 0 0 74950 1150 0 0 25 0 5 0 521841766 2541441024 560521 33554432000 4194304 4338100 140737323047840 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20406/tid=20414] ppid=20404 vsize=2481876 CPUtime=760.38 cores=0,2,4,6
/proc/20406/task/20414/stat : 20414 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 164640 0 0 0 75973 65 0 0 25 0 5 0 521841766 2541441024 560521 33554432000 4194304 4338100 140737323047840 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20406/tid=20415] ppid=20404 vsize=2481876 CPUtime=758.41 cores=0,2,4,6
/proc/20406/task/20415/stat : 20415 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 104499 0 0 0 74969 872 0 0 18 0 5 0 521841766 2541441024 560521 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20406/tid=20416] ppid=20404 vsize=2481876 CPUtime=760.25 cores=0,2,4,6
/proc/20406/task/20416/stat : 20416 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 170616 0 0 0 75960 65 0 0 25 0 5 0 521841766 2541441024 560521 33554432000 4194304 4338100 140737323047840 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3040.57
Current children cumulated vsize (KiB) 2481876

[startup+822.301 s]
/proc/loadavg: 8.16 7.99 7.86 9/186 20438
/proc/meminfo: memFree=14038348/32951124 swapFree=67111528/67111528
[pid=20406] ppid=20404 vsize=2543592 CPUtime=3279.71 cores=0,2,4,6
/proc/20406/stat : 20406 (ttsth-5-4) S 20404 20406 19929 0 -1 4202496 581173 0 0 0 325753 2218 0 0 25 0 5 0 521841715 2604638208 580935 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20406/statm: 635898 580935 176 36 0 633803 0
[pid=20406/tid=20413] ppid=20404 vsize=2543592 CPUtime=821 cores=0,2,4,6
/proc/20406/task/20413/stat : 20413 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 111387 0 0 0 80900 1200 0 0 25 0 5 0 521841766 2604638208 580935 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20406/tid=20414] ppid=20404 vsize=2543592 CPUtime=820.36 cores=0,2,4,6
/proc/20406/task/20414/stat : 20414 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 171757 0 0 0 81969 67 0 0 25 0 5 0 521841766 2604638208 580935 33554432000 4194304 4338100 140737323047840 18446744073709551615 4256655 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20406/tid=20415] ppid=20404 vsize=2543592 CPUtime=817.87 cores=0,2,4,6
/proc/20406/task/20415/stat : 20415 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 106578 0 0 0 80906 881 0 0 18 0 5 0 521841766 2604638208 580935 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20406/tid=20416] ppid=20404 vsize=2543592 CPUtime=819.94 cores=0,2,4,6
/proc/20406/task/20416/stat : 20416 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 177359 0 0 0 81927 67 0 0 25 0 5 0 521841766 2604638208 580935 33554432000 4194304 4338100 140737323047840 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3279.71
Current children cumulated vsize (KiB) 2543592

[startup+882.301 s]
/proc/loadavg: 8.18 8.02 7.87 9/186 20440
/proc/meminfo: memFree=13876016/32951124 swapFree=67111528/67111528
[pid=20406] ppid=20404 vsize=2631652 CPUtime=3518.93 cores=0,2,4,6
/proc/20406/stat : 20406 (ttsth-5-4) S 20404 20406 19929 0 -1 4202496 608428 0 0 0 349609 2284 0 0 25 0 5 0 521841715 2694811648 608190 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20406/statm: 657913 608190 176 36 0 655818 0
[pid=20406/tid=20413] ppid=20404 vsize=2631652 CPUtime=880.99 cores=0,2,4,6
/proc/20406/task/20413/stat : 20413 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 116214 0 0 0 86852 1247 0 0 25 0 5 0 521841766 2694811648 608190 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20406/tid=20414] ppid=20404 vsize=2631652 CPUtime=880.36 cores=0,2,4,6
/proc/20406/task/20414/stat : 20414 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 180631 0 0 0 87965 71 0 0 25 0 5 0 521841766 2694811648 608190 33554432000 4194304 4338100 140737323047840 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20406/tid=20415] ppid=20404 vsize=2631652 CPUtime=877.41 cores=0,2,4,6
/proc/20406/task/20415/stat : 20415 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 109199 0 0 0 86850 891 0 0 18 0 5 0 521841766 2694811648 608190 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20406/tid=20416] ppid=20404 vsize=2631652 CPUtime=879.63 cores=0,2,4,6
/proc/20406/task/20416/stat : 20416 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 188292 0 0 0 87891 72 0 0 25 0 5 0 521841766 2694811648 608190 33554432000 4194304 4338100 140737323047840 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3518.93
Current children cumulated vsize (KiB) 2631652

[startup+942.301 s]
/proc/loadavg: 8.24 8.06 7.89 9/186 20441
/proc/meminfo: memFree=13721620/32951124 swapFree=67111528/67111528
[pid=20406] ppid=20404 vsize=2724120 CPUtime=3758.53 cores=0,2,4,6
/proc/20406/stat : 20406 (ttsth-5-4) S 20404 20406 19929 0 -1 4202496 634245 0 0 0 373416 2437 0 0 25 0 5 0 521841715 2789498880 634007 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20406/statm: 681030 634007 176 36 0 678935 0
[pid=20406/tid=20413] ppid=20404 vsize=2724120 CPUtime=940.71 cores=0,2,4,6
/proc/20406/task/20413/stat : 20413 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 120405 0 0 0 92747 1324 0 0 25 0 5 0 521841766 2789498880 634007 33554432000 4194304 4338100 140737323047840 18446744073709551615 4229598 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20406/tid=20414] ppid=20404 vsize=2724120 CPUtime=940.36 cores=0,2,4,6
/proc/20406/task/20414/stat : 20414 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 189226 0 0 0 93962 74 0 0 25 0 5 0 521841766 2789498880 634007 33554432000 4194304 4338100 140737323047840 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20406/tid=20415] ppid=20404 vsize=2724120 CPUtime=937.34 cores=0,2,4,6
/proc/20406/task/20415/stat : 20415 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 113810 0 0 0 92772 962 0 0 25 0 5 0 521841766 2789498880 634007 33554432000 4194304 4338100 140737323047840 18446744073709551615 4240496 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20406/tid=20416] ppid=20404 vsize=2724120 CPUtime=939.58 cores=0,2,4,6
/proc/20406/task/20416/stat : 20416 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 196712 0 0 0 93884 74 0 0 25 0 5 0 521841766 2789498880 634007 33554432000 4194304 4338100 140737323047840 18446744073709551615 4254294 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3758.53
Current children cumulated vsize (KiB) 2724120

[startup+1002.3 s]
/proc/loadavg: 8.14 8.06 7.91 9/186 20443
/proc/meminfo: memFree=13614584/32951124 swapFree=67111528/67111528
[pid=20406] ppid=20404 vsize=2757684 CPUtime=3998.19 cores=0,2,4,6
/proc/20406/stat : 20406 (ttsth-5-4) S 20404 20406 19929 0 -1 4202496 648465 0 0 0 397155 2664 0 0 25 0 5 0 521841715 2823868416 648227 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20406/statm: 689421 648227 176 36 0 687326 0
[pid=20406/tid=20413] ppid=20404 vsize=2757684 CPUtime=1000.66 cores=0,2,4,6
/proc/20406/task/20413/stat : 20413 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 124856 0 0 0 98627 1439 0 0 25 0 5 0 521841766 2823868416 648227 33554432000 4194304 4338100 140737323047840 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20406/tid=20414] ppid=20404 vsize=2757684 CPUtime=1000.35 cores=0,2,4,6
/proc/20406/task/20414/stat : 20414 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 190133 0 0 0 99961 74 0 0 25 0 5 0 521841766 2823868416 648227 33554432000 4194304 4338100 140737323047840 18446744073709551615 4254291 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20406/tid=20415] ppid=20404 vsize=2757684 CPUtime=997.06 cores=0,2,4,6
/proc/20406/task/20415/stat : 20415 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 117972 0 0 0 98634 1072 0 0 25 0 5 0 521841766 2823868416 648227 33554432000 4194304 4338100 140737323047840 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20406/tid=20416] ppid=20404 vsize=2757684 CPUtime=999.59 cores=0,2,4,6
/proc/20406/task/20416/stat : 20416 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 201412 0 0 0 99883 76 0 0 25 0 5 0 521841766 2823868416 648227 33554432000 4194304 4338100 140737323047840 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3998.19
Current children cumulated vsize (KiB) 2757684

[startup+1062.3 s]
/proc/loadavg: 8.11 8.06 7.91 9/186 20445
/proc/meminfo: memFree=13525028/32951124 swapFree=67111528/67111528
[pid=20406] ppid=20404 vsize=2795828 CPUtime=4237.75 cores=0,2,4,6
/proc/20406/stat : 20406 (ttsth-5-4) S 20404 20406 19929 0 -1 4202496 660618 0 0 0 420956 2819 0 0 25 0 5 0 521841715 2862927872 660380 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20406/statm: 698957 660380 176 36 0 696862 0
[pid=20406/tid=20413] ppid=20404 vsize=2795828 CPUtime=1060.53 cores=0,2,4,6
/proc/20406/task/20413/stat : 20413 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 127589 0 0 0 104535 1518 0 0 25 0 5 0 521841766 2862927872 660380 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20406/tid=20414] ppid=20404 vsize=2795828 CPUtime=1060.34 cores=0,2,4,6
/proc/20406/task/20414/stat : 20414 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 190850 0 0 0 105959 75 0 0 25 0 5 0 521841766 2862927872 660380 33554432000 4194304 4338100 140737323047840 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20406/tid=20415] ppid=20404 vsize=2795828 CPUtime=1056.83 cores=0,2,4,6
/proc/20406/task/20415/stat : 20415 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 119357 0 0 0 104540 1143 0 0 18 0 5 0 521841766 2862927872 660380 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20406/tid=20416] ppid=20404 vsize=2795828 CPUtime=1059.5 cores=0,2,4,6
/proc/20406/task/20416/stat : 20416 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 208730 0 0 0 105871 79 0 0 25 0 5 0 521841766 2862927872 660380 33554432000 4194304 4338100 140737323047840 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4237.75
Current children cumulated vsize (KiB) 2795828

[startup+1122.3 s]
/proc/loadavg: 8.14 8.08 7.93 9/186 20446
/proc/meminfo: memFree=13385496/32951124 swapFree=67111528/67111528
[pid=20406] ppid=20404 vsize=2858072 CPUtime=4476.97 cores=0,2,4,6
/proc/20406/stat : 20406 (ttsth-5-4) S 20404 20406 19929 0 -1 4202496 680732 0 0 0 444816 2881 0 0 25 0 5 0 521841715 2926665728 680494 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20406/statm: 714518 680494 176 36 0 712423 0
[pid=20406/tid=20413] ppid=20404 vsize=2858072 CPUtime=1120.53 cores=0,2,4,6
/proc/20406/task/20413/stat : 20413 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 131064 0 0 0 110488 1565 0 0 25 0 5 0 521841766 2926665728 680494 33554432000 4194304 4338100 140737323047840 18446744073709551615 216634513467 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20406/tid=20414] ppid=20404 vsize=2858072 CPUtime=1120.35 cores=0,2,4,6
/proc/20406/task/20414/stat : 20414 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 194015 0 0 0 111958 77 0 0 25 0 5 0 521841766 2926665728 680494 33554432000 4194304 4338100 140737323047840 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20406/tid=20415] ppid=20404 vsize=2858072 CPUtime=1116.38 cores=0,2,4,6
/proc/20406/task/20415/stat : 20415 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 120977 0 0 0 110486 1152 0 0 18 0 5 0 521841766 2926665728 680494 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20406/tid=20416] ppid=20404 vsize=2858072 CPUtime=1119.18 cores=0,2,4,6
/proc/20406/task/20416/stat : 20416 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 220584 0 0 0 111833 85 0 0 25 0 5 0 521841766 2926665728 680494 33554432000 4194304 4338100 140737323047840 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4476.97
Current children cumulated vsize (KiB) 2858072

[startup+1182.3 s]
/proc/loadavg: 8.14 8.09 7.94 9/186 20448
/proc/meminfo: memFree=13269284/32951124 swapFree=67111528/67111528
[pid=20406] ppid=20404 vsize=2881724 CPUtime=4716.25 cores=0,2,4,6
/proc/20406/stat : 20406 (ttsth-5-4) S 20404 20406 19929 0 -1 4202496 690609 0 0 0 468693 2932 0 0 25 0 5 0 521841715 2950885376 690371 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20406/statm: 720431 690371 176 36 0 718336 0
[pid=20406/tid=20413] ppid=20404 vsize=2881724 CPUtime=1180.53 cores=0,2,4,6
/proc/20406/task/20413/stat : 20413 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 132530 0 0 0 116451 1602 0 0 25 0 5 0 521841766 2950885376 690371 33554432000 4194304 4338100 140737323047840 18446744073709551615 4230438 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20406/tid=20414] ppid=20404 vsize=2881724 CPUtime=1180.34 cores=0,2,4,6
/proc/20406/task/20414/stat : 20414 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 194651 0 0 0 117957 77 0 0 25 0 5 0 521841766 2950885376 690371 33554432000 4194304 4338100 140737323047840 18446744073709551615 216633910515 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20406/tid=20415] ppid=20404 vsize=2881724 CPUtime=1175.97 cores=0,2,4,6
/proc/20406/task/20415/stat : 20415 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 122217 0 0 0 116438 1159 0 0 18 0 5 0 521841766 2950885376 690371 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20406/tid=20416] ppid=20404 vsize=2881724 CPUtime=1178.88 cores=0,2,4,6
/proc/20406/task/20416/stat : 20416 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 227119 0 0 0 117797 91 0 0 25 0 5 0 521841766 2950885376 690371 33554432000 4194304 4338100 140737323047840 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4716.25
Current children cumulated vsize (KiB) 2881724



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.3 s]
/proc/loadavg: 8.10 8.08 7.94 5/180 20449
/proc/meminfo: memFree=15123612/32951124 swapFree=67111528/67111528
[pid=20406] ppid=20404 vsize=2892428 CPUtime=4800.03 cores=0,2,4,6
/proc/20406/stat : 20406 (ttsth-5-4) S 20404 20406 19929 0 -1 4202496 694135 0 0 0 477055 2948 0 0 25 0 5 0 521841715 2961846272 693897 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20406/statm: 723107 693897 176 36 0 721012 0
[pid=20406/tid=20413] ppid=20404 vsize=2892428 CPUtime=1201.54 cores=0,2,4,6
/proc/20406/task/20413/stat : 20413 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 132868 0 0 0 118539 1615 0 0 25 0 5 0 521841766 2961846272 693897 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20406/tid=20414] ppid=20404 vsize=2892428 CPUtime=1201.36 cores=0,2,4,6
/proc/20406/task/20414/stat : 20414 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 194853 0 0 0 120058 78 0 0 25 0 5 0 521841766 2961846272 693897 33554432000 4194304 4338100 140737323047840 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20406/tid=20415] ppid=20404 vsize=2892428 CPUtime=1196.84 cores=0,2,4,6
/proc/20406/task/20415/stat : 20415 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 122555 0 0 0 118524 1160 0 0 19 0 5 0 521841766 2961846272 693897 33554432000 4194304 4338100 140737323047840 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20406/tid=20416] ppid=20404 vsize=2892428 CPUtime=1199.76 cores=0,2,4,6
/proc/20406/task/20416/stat : 20416 (ttsth-5-4) R 20404 20406 19929 0 -1 4202560 229767 0 0 0 119884 92 0 0 25 0 5 0 521841766 2961846272 693897 33554432000 4194304 4338100 140737323047840 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4800.03
Current children cumulated vsize (KiB) 2892428

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.42 s]
/proc/loadavg: 8.10 8.08 7.94 5/180 20449
/proc/meminfo: memFree=15123612/32951124 swapFree=67111528/67111528
[pid=20406] ppid=20404 vsize=0 CPUtime=4800.15 cores=0,2,4,6
/proc/20406/stat : 20406 (ttsth-5-4) Z 20404 20406 19929 0 -1 4203548 694135 0 0 0 477057 2958 0 0 18 0 2 0 521841715 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20406/statm: 0 0 0 0 0 0 0
[pid=20406/tid=20416] ppid=20404 vsize=0 CPUtime=1199.87 cores=0,2,4,6
/proc/20406/task/20416/stat : 20416 (ttsth-5-4) R 20404 20406 19929 0 -1 4203588 229767 0 0 0 119885 102 0 0 25 0 2 0 521841766 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.15
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1203.44
CPU time (s): 4800.19
CPU user time (s): 4770.58
CPU system time (s): 29.6065
CPU usage (%): 398.871
Max. virtual memory (cumulated for all children) (KiB): 2892428

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4770.58
system time used= 29.6065
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 694135
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 13031
involuntary context switches= 21769

runsolver used 2.00869 second user time and 4.20436 second system time

The end

Launcher Data 

Begin job on node103 at 2011-04-29 20:39:51
IDJOB=3265654
IDBENCH=85793
IDSOLVER=1654
FILE ID=node103/3265654-1304102391
RUNJOBID= node103-1304102391-20387
PBS_JOBID= 13170621
Free space on /tmp= 71516 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT05/crafted/sorge05/QG7a/gensys-ukn009.shuffled-as.sat05-3849.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3265654-1304102391/watcher-3265654-1304102391 -o /tmp/evaluation-result-3265654-1304102391/solver-3265654-1304102391 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265654-1304102391.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= f2a1eb7937a7cda114ce344a30ca1145
RANDOM SEED=1540586023

node103.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      17906480 kB
Buffers:       2148992 kB
Cached:       12248128 kB
SwapCached:          0 kB
Active:        5586732 kB
Inactive:      8875032 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      17906480 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:            1256 kB
Writeback:           0 kB
AnonPages:       62980 kB
Mapped:          14832 kB
Slab:           519844 kB
PageTables:       4620 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   197300 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71516 MiB
End job on node103 at 2011-04-29 20:59:56