Trace number 3257954

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.3 1204.08

General information on the benchmark

NameCRAFTED/rbsat/random/unforced/
rbsat-v760c43649g5.cnf
MD5SUMfac733a1c5caf879cadf25b989055e5e
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.421935
Satisfiable
(Un)Satisfiability was proved
Number of variables760
Number of clauses43649
Sum of the clauses size87978
Maximum clause length19
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 243609
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 540

Solver Data 

0.01/0.02	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3257954-1304048460.cnf 
0.01/0.02	c Using SATTIMEOUT of 4800
0.01/0.02	c Using SATRAM of 15500
0.01/0.02	c Size of subset node = 32
0.01/0.02	c Variables = 760
0.01/0.02	c Clauses = 43649
0.01/0.02	c Literals = 87978
0.01/0.02	c Reduced size:
0.01/0.02	c Variables = 760
0.01/0.02	c Clauses = 43649
0.01/0.02	c Literals = 87978
0.01/0.02	c a2
0.01/0.05	c a1
0.09/0.10	c Independent sub-problems = 1
0.09/0.10	c Sub-problem with 760 variables and 43649 clauses
0.09/0.10	c using permutation method 4
0.09/0.10	c count = 1
0.09/0.10	c Ordering finished 0.080 seconds
0.09/0.10	c mapped 0.090 seconds
0.09/0.10	c permuted 0.090 seconds
0.09/0.10	c about to sort
0.09/0.10	c sorted
0.09/0.18	c clauses now 43649
0.09/0.18	c init finished 0.100 seconds
0.09/0.18	c tree built 0.180 seconds
0.09/0.18	c 
0.09/0.18	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.09/0.18	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.09/0.18	c ................................ | ..........                    |                                  0.2         0.0             
0.09/0.18	c ................................ | ..........                  1 |                                  0.2         5.6             
0.09/0.18	c ................................ | ..........                  2 |                                  0.2        11.1             
0.09/0.18	c ................................ | ..........                  3 |                                  0.2        16.7             
0.09/0.18	c ................................ | ..........                  4 |                                  0.2        22.2             
0.09/0.18	c ................................ | ..........                  5 |                                  0.2        27.8             
0.09/0.19	c ................................ | ..........               1000 | 13           180                 0.2      5263.2         7872
95.67/24.18	c ................................ | ..........            2752000 | 129251       539309             96.0     28663.7     16545362
191.83/48.24	c ................................ | ..........            4696000 | 234935       933333            192.0     24454.5     27225946
287.97/72.31	c ................................ | ..........            6255000 | 327207       1252088           288.0     21718.8     36912919
383.72/96.39	c ................................ | ..........            7939000 | 418311       1565520           384.1     20671.2     45257511
479.89/120.44	c ................................ | *.........            9382000 | 491656       1845741           480.1     19543.4     49193320
575.70/144.48	c ................................ | *.........           10671000 | 561344       2094534           576.0     18525.4     56268734
671.88/168.53	c ................................ | *.........           11839000 | 622704       2311891           672.0     17617.6     61584101
767.65/192.59	c ................................ | *.........           12946000 | 680513       2552781           768.0     16856.1     66975481
863.74/216.68	c ................................ | *.........           14255000 | 754092       2815125           864.1     16497.3     69605625
959.93/240.72	c ................................ | **........           15523000 | 823340       3044386           960.0     16169.1     76951828
1055.74/264.77	c ................................ | **........           16600000 | 891468       3245930          1056.0     15719.7     84447566
1151.78/288.86	c ................................ | **........           17708000 | 954603       3454980          1152.0     15371.5     88062947
1248.20/313.03	c ................................ | **........           18898000 | 1014027      3665892          1248.1     15141.7     93935048
1343.81/337.05	c ................................ | **........           19918000 | 1066865      3865855          1344.0     14819.8     95984570
1439.85/361.15	c ................................ | ***.......           20803000 | 1113633      4027122          1440.0     14446.0     97899858
1535.80/385.25	c ................................ | ***.......           21659000 | 1163835      4232234          1536.0     14100.9    100440838
1631.79/409.36	c ................................ | ***.......           22525000 | 1211566      4393544          1632.0     13801.8    105618109
1727.86/433.46	c ................................ | ***.......           23319000 | 1259878      4563379          1728.1     13494.1    109328425
1823.81/457.56	c ................................ | ***.......           24146000 | 1306459      4743202          1824.0     13237.6    114555731
1919.86/481.64	c ................................ | ****......           24924000 | 1353894      4907261          1920.0     12981.0    117515393
2015.88/505.74	c ................................ | ****......           25798000 | 1405198      5063635          2016.1     12796.2    122321327
2111.92/529.83	c ................................ | ****......           26934000 | 1462696      5239527          2112.0     12752.7    127159142
2207.97/553.91	c ................................ | ****......           28152000 | 1515625      5397777          2208.0     12749.9    130066855
2303.93/578.03	c ................................ | ****......           29412000 | 1576602      5588146          2304.1     12765.3    134231712
2399.99/602.12	c ................................ | *****.....           30622000 | 1631185      5753321          2400.1     12758.7    140690515
2495.96/626.22	c ................................ | *****.....           31484000 | 1677440      5889752          2496.1     12613.5    142482034
2591.93/650.34	c ................................ | *****.....           32309000 | 1722426      6044149          2592.1     12464.4    142482034
2687.91/674.43	c ................................ | *****.....           33144000 | 1769605      6183944          2688.1     12330.1    143369274
2784.05/698.51	c ................................ | *****.....           33871000 | 1809726      6315069          2784.1     12165.9    144274878
2879.83/722.54	c ................................ | ******....           34674000 | 1851388      6440505          2880.0     12039.6    145773854
2975.70/746.58	c ................................ | ******....           35454000 | 1893692      6576825          2976.0     11913.2    149199113
3071.86/770.65	c ................................ | ******....           36441000 | 1949947      6721417          3072.0     11862.2    152144004
3168.03/794.70	c ................................ | ******....           37566000 | 2004510      6886935          3168.1     11857.8    155572697
3263.70/818.79	c ................................ | ******....           38502000 | 2055511      7057325          3264.1     11795.7    156175424
3359.90/842.83	c ................................ | *******...           39420000 | 2104027      7222155          3360.0     11732.0    157702733
3455.97/866.91	c ................................ | *******...           40314000 | 2150710      7366348          3456.0     11664.9    159510179
3551.69/890.98	c ................................ | *******...           41076000 | 2190365      7500909          3552.0     11564.1    161392464
3648.20/915.19	c ................................ | *******...           41800000 | 2231571      7637802          3648.6     11456.5    161392464
3744.01/939.12	c ................................ | *******...           42520000 | 2271311      7766395          3744.1     11356.5    161392464
3839.80/963.15	c ................................ | ********..           43249000 | 2314018      7886381          3840.0     11262.8    162090069
3935.89/987.26	c ................................ | ********..           44140000 | 2363450      8088323          3936.1     11214.1    163584261
4031.65/1011.28	c ................................ | ********..           44997000 | 2407234      8256217          4032.0     11160.0    166596828
4127.76/1035.37	c ................................ | ********..           45824000 | 2448843      8425856          4128.1     11100.6    169601713
4223.90/1059.42	c ................................ | ********..           46731000 | 2490705      8592352          4224.0     11063.2    174116100
4319.72/1083.48	c ................................ | *********.           47551000 | 2536464      8729280          4320.0     11007.1    176304969
4415.84/1107.55	c ................................ | *********.           48412000 | 2579045      8878829          4416.1     10962.7    178325436
4511.88/1131.64	c ................................ | *********.           49416000 | 2633040      9018037          4512.1     10952.0    179352337
4607.88/1155.75	c ................................ | *********.           50418000 | 2687042      9154584          4608.1     10941.2    179880684
4703.65/1179.79	c ................................ | *********.           51342000 | 2737843      9287532          4704.0     10914.5    180048036
4799.89/1203.84	c CPU limit (4800) reached
4799.89/1203.84	c Time = 4800.06 seconds
4799.89/1203.84	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3257954-1304048460/watcher-3257954-1304048460 -o /tmp/evaluation-result-3257954-1304048460/solver-3257954-1304048460 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3257954-1304048460.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.78 8.08 7.86 5/181 28126
/proc/meminfo: memFree=19590640/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=9760 CPUtime=0 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) R 28124 28126 27953 0 -1 4202496 494 0 0 0 0 0 0 0 25 0 1 0 595076660 9994240 436 33554432000 4194304 4338100 140733250814656 18446744073709551615 251367207621 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/28126/statm: 2440 437 147 36 0 345 0

[startup+0.024446 s]
/proc/loadavg: 7.78 8.08 7.86 5/181 28126
/proc/meminfo: memFree=19590640/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=13676 CPUtime=0.01 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) R 28124 28126 27953 0 -1 4202496 1446 0 0 0 1 0 0 0 25 0 1 0 595076660 14004224 1388 33554432000 4194304 4338100 140733250814656 18446744073709551615 251367207621 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/28126/statm: 3419 1389 148 36 0 1324 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 13676

[startup+0.100434 s]
/proc/loadavg: 7.78 8.08 7.86 5/181 28126
/proc/meminfo: memFree=19590640/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=18536 CPUtime=0.09 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) R 28124 28126 27953 0 -1 4202496 3533 0 0 0 9 0 0 0 25 0 1 0 595076660 18980864 2672 33554432000 4194304 4338100 140733250814656 18446744073709551615 251367204592 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/28126/statm: 4634 2672 161 36 0 2539 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 18536

[startup+1.05018 s]
/proc/loadavg: 7.78 8.08 7.86 5/181 28126
/proc/meminfo: memFree=19590640/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=259172 CPUtime=3.77 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) S 28124 28126 27953 0 -1 4202496 19484 0 0 0 370 7 0 0 25 0 5 0 595076660 265392128 18450 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28126/statm: 64793 18451 176 36 0 62698 0
Current children cumulated CPU time (s) 3.77
Current children cumulated vsize (KiB) 259172

[startup+1.1516 s]
/proc/loadavg: 7.78 8.08 7.86 5/181 28126
/proc/meminfo: memFree=19590640/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=259172 CPUtime=4.19 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) S 28124 28126 27953 0 -1 4202496 20566 0 0 0 411 8 0 0 25 0 5 0 595076660 265392128 19532 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28126/statm: 64793 19532 176 36 0 62698 0
Current children cumulated CPU time (s) 4.19
Current children cumulated vsize (KiB) 259172

[startup+1.53034 s]
/proc/loadavg: 7.78 8.08 7.86 5/181 28126
/proc/meminfo: memFree=19590640/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=278988 CPUtime=5.53 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) S 28124 28126 27953 0 -1 4202496 24311 0 0 0 543 10 0 0 25 0 5 0 595076660 285683712 23277 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28126/statm: 69747 23475 176 36 0 67652 0
Current children cumulated CPU time (s) 5.53
Current children cumulated vsize (KiB) 278988

[startup+3.10081 s]
/proc/loadavg: 7.80 8.08 7.86 9/186 28131
/proc/meminfo: memFree=19334612/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=332164 CPUtime=11.8 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) S 28124 28126 27953 0 -1 4202496 38001 0 0 0 1158 22 0 0 25 0 5 0 595076660 340135936 36967 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28126/statm: 83041 36967 176 36 0 80946 0
[pid=28126/tid=28128] ppid=28124 vsize=332164 CPUtime=2.9 cores=0,2,4,6
/proc/28126/task/28128/stat : 28128 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 9506 0 0 0 289 1 0 0 24 0 5 0 595076679 340135936 36967 33554432000 4194304 4338100 140733250814656 18446744073709551615 4284509 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28126/tid=28129] ppid=28124 vsize=332164 CPUtime=2.9 cores=0,2,4,6
/proc/28126/task/28129/stat : 28129 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 5702 0 0 0 282 8 0 0 21 0 5 0 595076679 340135936 36967 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379361027 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28126/tid=28130] ppid=28124 vsize=332164 CPUtime=2.91 cores=0,2,4,6
/proc/28126/task/28130/stat : 28130 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 10033 0 0 0 289 2 0 0 24 0 5 0 595076679 340135936 36967 33554432000 4194304 4338100 140733250814656 18446744073709551615 251367243268 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28126/tid=28131] ppid=28124 vsize=332164 CPUtime=2.88 cores=0,2,4,6
/proc/28126/task/28131/stat : 28131 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 4029 0 0 0 280 8 0 0 24 0 5 0 595076679 340135936 36968 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 11.8
Current children cumulated vsize (KiB) 332164

[startup+6.30122 s]
/proc/loadavg: 7.80 8.08 7.86 9/186 28131
/proc/meminfo: memFree=19141632/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=414612 CPUtime=24.59 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) S 28124 28126 27953 0 -1 4202496 57834 0 0 0 2413 46 0 0 25 0 5 0 595076660 424562688 56800 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28126/statm: 103653 56800 176 36 0 101558 0
[pid=28126/tid=28128] ppid=28124 vsize=414612 CPUtime=6.11 cores=0,2,4,6
/proc/28126/task/28128/stat : 28128 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 15791 0 0 0 608 3 0 0 25 0 5 0 595076679 424562688 56801 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28126/tid=28129] ppid=28124 vsize=414612 CPUtime=6.1 cores=0,2,4,6
/proc/28126/task/28129/stat : 28129 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 9084 0 0 0 592 18 0 0 25 0 5 0 595076679 424562688 56801 33554432000 4194304 4338100 140733250814656 18446744073709551615 4256770 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28126/tid=28130] ppid=28124 vsize=414612 CPUtime=6.1 cores=0,2,4,6
/proc/28126/task/28130/stat : 28130 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 17906 0 0 0 607 3 0 0 25 0 5 0 595076679 424562688 56801 33554432000 4194304 4338100 140733250814656 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28126/tid=28131] ppid=28124 vsize=414612 CPUtime=6.07 cores=0,2,4,6
/proc/28126/task/28131/stat : 28131 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 6323 0 0 0 588 19 0 0 25 0 5 0 595076679 424562688 56801 33554432000 4194304 4338100 140733250814656 18446744073709551615 4256745 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 24.59
Current children cumulated vsize (KiB) 414612

[startup+12.701 s]
/proc/loadavg: 7.83 8.07 7.86 9/186 28131
/proc/meminfo: memFree=18829932/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=547988 CPUtime=50.16 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) S 28124 28126 27953 0 -1 4202496 90998 0 0 0 4928 88 0 0 25 0 5 0 595076660 561139712 89964 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28126/statm: 136997 89964 176 36 0 134902 0
[pid=28126/tid=28128] ppid=28124 vsize=547988 CPUtime=12.51 cores=0,2,4,6
/proc/28126/task/28128/stat : 28128 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 26581 0 0 0 1246 5 0 0 25 0 5 0 595076679 561139712 89964 33554432000 4194304 4338100 140733250814656 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28126/tid=28129] ppid=28124 vsize=547988 CPUtime=12.5 cores=0,2,4,6
/proc/28126/task/28129/stat : 28129 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 15082 0 0 0 1214 36 0 0 25 0 5 0 595076679 561139712 89964 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28126/tid=28130] ppid=28124 vsize=547988 CPUtime=12.51 cores=0,2,4,6
/proc/28126/task/28130/stat : 28130 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 29649 0 0 0 1246 5 0 0 25 0 5 0 595076679 561139712 89964 33554432000 4194304 4338100 140733250814656 18446744073709551615 4285738 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28126/tid=28131] ppid=28124 vsize=547988 CPUtime=12.44 cores=0,2,4,6
/proc/28126/task/28131/stat : 28131 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 10955 0 0 0 1205 39 0 0 25 0 5 0 595076679 561139712 89964 33554432000 4194304 4338100 140733250814656 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 50.16
Current children cumulated vsize (KiB) 547988

[startup+25.5007 s]
/proc/loadavg: 7.86 8.07 7.86 9/186 28131
/proc/meminfo: memFree=18333812/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=776980 CPUtime=101.26 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) S 28124 28126 27953 0 -1 4202496 149505 0 0 0 9959 167 0 0 25 0 5 0 595076660 795627520 147991 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28126/statm: 194245 147991 176 36 0 192150 0
[pid=28126/tid=28128] ppid=28124 vsize=776980 CPUtime=25.29 cores=0,2,4,6
/proc/28126/task/28128/stat : 28128 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 46994 0 0 0 2519 10 0 0 25 0 5 0 595076679 795627520 147991 33554432000 4194304 4338100 140733250814656 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28126/tid=28129] ppid=28124 vsize=776980 CPUtime=25.29 cores=0,2,4,6
/proc/28126/task/28129/stat : 28129 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 25771 0 0 0 2457 72 0 0 25 0 5 0 595076679 795627520 147991 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28126/tid=28130] ppid=28124 vsize=776980 CPUtime=25.29 cores=0,2,4,6
/proc/28126/task/28130/stat : 28130 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 50097 0 0 0 2519 10 0 0 25 0 5 0 595076679 795627520 147991 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28126/tid=28131] ppid=28124 vsize=776980 CPUtime=25.16 cores=0,2,4,6
/proc/28126/task/28131/stat : 28131 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 17912 0 0 0 2444 72 0 0 25 0 5 0 595076679 795627520 147991 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 101.26
Current children cumulated vsize (KiB) 776980

[startup+51.1009 s]
/proc/loadavg: 8.02 8.09 7.87 9/186 28132
/proc/meminfo: memFree=17594228/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=1143840 CPUtime=203.41 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) S 28124 28126 27953 0 -1 4202496 240842 0 0 0 20002 339 0 0 25 0 5 0 595076660 1171292160 239328 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28126/statm: 285960 239328 176 36 0 283865 0
[pid=28126/tid=28128] ppid=28124 vsize=1143840 CPUtime=50.86 cores=0,2,4,6
/proc/28126/task/28128/stat : 28128 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 76215 0 0 0 5069 17 0 0 25 0 5 0 595076679 1171292160 239328 33554432000 4194304 4338100 140733250814656 18446744073709551615 4239184 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28126/tid=28129] ppid=28124 vsize=1143840 CPUtime=50.86 cores=0,2,4,6
/proc/28126/task/28129/stat : 28129 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 41599 0 0 0 4933 153 0 0 25 0 5 0 595076679 1171292160 239328 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28126/tid=28130] ppid=28124 vsize=1143840 CPUtime=50.87 cores=0,2,4,6
/proc/28126/task/28130/stat : 28130 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 83102 0 0 0 5071 16 0 0 25 0 5 0 595076679 1171292160 239328 33554432000 4194304 4338100 140733250814656 18446744073709551615 4283577 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28126/tid=28131] ppid=28124 vsize=1143840 CPUtime=50.6 cores=0,2,4,6
/proc/28126/task/28131/stat : 28131 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 31195 0 0 0 4910 150 0 0 25 0 5 0 595076679 1171292160 239328 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 203.41
Current children cumulated vsize (KiB) 1143840

[startup+102.3 s]
/proc/loadavg: 8.14 8.11 7.89 9/186 28133
/proc/meminfo: memFree=16296928/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=1722792 CPUtime=407.63 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) S 28124 28126 27953 0 -1 4202496 384166 0 0 0 40101 662 0 0 25 0 5 0 595076660 1764139008 382652 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28126/statm: 430698 382652 176 36 0 428603 0
[pid=28126/tid=28128] ppid=28124 vsize=1722792 CPUtime=101.96 cores=0,2,4,6
/proc/28126/task/28128/stat : 28128 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 123492 0 0 0 10165 31 0 0 25 0 5 0 595076679 1764139008 382652 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379361027 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28126/tid=28129] ppid=28124 vsize=1722792 CPUtime=101.87 cores=0,2,4,6
/proc/28126/task/28129/stat : 28129 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 60539 0 0 0 9884 303 0 0 25 0 5 0 595076679 1764139008 382652 33554432000 4194304 4338100 140733250814656 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28126/tid=28130] ppid=28124 vsize=1722792 CPUtime=102.01 cores=0,2,4,6
/proc/28126/task/28130/stat : 28130 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 142965 0 0 0 10172 29 0 0 25 0 5 0 595076679 1764139008 382652 33554432000 4194304 4338100 140733250814656 18446744073709551615 4286315 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28126/tid=28131] ppid=28124 vsize=1722792 CPUtime=101.57 cores=0,2,4,6
/proc/28126/task/28131/stat : 28131 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 48439 0 0 0 9860 297 0 0 25 0 5 0 595076679 1764139008 382652 33554432000 4194304 4338100 140733250814656 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 407.63
Current children cumulated vsize (KiB) 1722792

[startup+162.3 s]
/proc/loadavg: 8.05 8.08 7.90 9/186 28135
/proc/meminfo: memFree=15360964/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=2156040 CPUtime=647.14 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) S 28124 28126 27953 0 -1 4202496 492926 0 0 0 63738 976 0 0 25 0 5 0 595076660 2207784960 491412 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28126/statm: 539010 491412 176 36 0 536915 0
[pid=28126/tid=28128] ppid=28124 vsize=2156040 CPUtime=161.89 cores=0,2,4,6
/proc/28126/task/28128/stat : 28128 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 161577 0 0 0 16145 44 0 0 25 0 5 0 595076679 2207784960 491412 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28126/tid=28129] ppid=28124 vsize=2156040 CPUtime=161.52 cores=0,2,4,6
/proc/28126/task/28129/stat : 28129 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 76251 0 0 0 15710 442 0 0 25 0 5 0 595076679 2207784960 491412 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28126/tid=28130] ppid=28124 vsize=2156040 CPUtime=161.95 cores=0,2,4,6
/proc/28126/task/28130/stat : 28130 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 183504 0 0 0 16158 37 0 0 25 0 5 0 595076679 2207784960 491412 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28126/tid=28131] ppid=28124 vsize=2156040 CPUtime=161.55 cores=0,2,4,6
/proc/28126/task/28131/stat : 28131 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 62863 0 0 0 15706 449 0 0 25 0 5 0 595076679 2207784960 491412 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 647.14
Current children cumulated vsize (KiB) 2156040

[startup+222.3 s]
/proc/loadavg: 8.01 8.07 7.90 9/186 28136
/proc/meminfo: memFree=14580856/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=2498708 CPUtime=886.46 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) S 28124 28126 27953 0 -1 4202496 579067 0 0 0 87371 1275 0 0 25 0 5 0 595076660 2558676992 577553 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28126/statm: 624677 577553 176 36 0 622582 0
[pid=28126/tid=28128] ppid=28124 vsize=2498708 CPUtime=221.68 cores=0,2,4,6
/proc/28126/task/28128/stat : 28128 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 193265 0 0 0 22112 56 0 0 25 0 5 0 595076679 2558676992 577553 33554432000 4194304 4338100 140733250814656 18446744073709551615 251367544891 0 0 4096 0 0 0 0 -1 4 0 0 0

################
# More data... #
################

/proc/28126/task/28131/stat : 28131 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 128714 0 0 0 69037 1029 0 0 25 0 5 0 595076679 4987838464 1171391 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 2799.2
Current children cumulated vsize (KiB) 4870936

[startup+762.301 s]
/proc/loadavg: 8.21 8.10 7.96 9/186 28151
/proc/meminfo: memFree=9066240/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=5111036 CPUtime=3038.71 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) S 28124 28126 27953 0 -1 4202496 1228490 0 0 0 301638 2233 0 0 25 0 5 0 595076660 5233700864 1226976 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28126/statm: 1277759 1226976 176 36 0 1275664 0
[pid=28126/tid=28128] ppid=28124 vsize=5111036 CPUtime=759.74 cores=0,2,4,6
/proc/28126/task/28128/stat : 28128 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 441632 0 0 0 75835 139 0 0 25 0 5 0 595076679 5233700864 1226976 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28126/tid=28129] ppid=28124 vsize=5111036 CPUtime=757.82 cores=0,2,4,6
/proc/28126/task/28129/stat : 28129 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 142307 0 0 0 74927 855 0 0 25 0 5 0 595076679 5233700864 1226976 33554432000 4194304 4338100 140733250814656 18446744073709551615 251366941828 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28126/tid=28130] ppid=28124 vsize=5111036 CPUtime=760.29 cores=0,2,4,6
/proc/28126/task/28130/stat : 28130 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 506429 0 0 0 75916 113 0 0 25 0 5 0 595076679 5233700864 1226976 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28126/tid=28131] ppid=28124 vsize=5111036 CPUtime=760.64 cores=0,2,4,6
/proc/28126/task/28131/stat : 28131 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 129391 0 0 0 74941 1123 0 0 25 0 5 0 595076679 5233700864 1226976 33554432000 4194304 4338100 140733250814656 18446744073709551615 251366941977 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3038.71
Current children cumulated vsize (KiB) 5111036

[startup+822.3 s]
/proc/loadavg: 8.20 8.11 7.97 9/186 28152
/proc/meminfo: memFree=8742592/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=5259108 CPUtime=3278.09 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) S 28124 28126 27953 0 -1 4202496 1265705 0 0 0 325357 2452 0 0 25 0 5 0 595076660 5385326592 1264191 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28126/statm: 1314777 1264191 176 36 0 1312682 0
[pid=28126/tid=28128] ppid=28124 vsize=5259108 CPUtime=819.68 cores=0,2,4,6
/proc/28126/task/28128/stat : 28128 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 464189 0 0 0 81819 149 0 0 25 0 5 0 595076679 5385326592 1264191 33554432000 4194304 4338100 140733250814656 18446744073709551615 4256021 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28126/tid=28129] ppid=28124 vsize=5259108 CPUtime=817.4 cores=0,2,4,6
/proc/28126/task/28129/stat : 28129 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 142438 0 0 0 80781 959 0 0 25 0 5 0 595076679 5385326592 1264191 33554432000 4194304 4338100 140733250814656 18446744073709551615 4256318 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28126/tid=28130] ppid=28124 vsize=5259108 CPUtime=820.25 cores=0,2,4,6
/proc/28126/task/28130/stat : 28130 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 520518 0 0 0 81909 116 0 0 25 0 5 0 595076679 5385326592 1264191 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28126/tid=28131] ppid=28124 vsize=5259108 CPUtime=820.56 cores=0,2,4,6
/proc/28126/task/28131/stat : 28131 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 129829 0 0 0 80830 1226 0 0 25 0 5 0 595076679 5385326592 1264191 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3278.09
Current children cumulated vsize (KiB) 5259108

[startup+882.3 s]
/proc/loadavg: 8.16 8.11 7.98 9/186 28154
/proc/meminfo: memFree=8272872/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=5439908 CPUtime=3517.4 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) S 28124 28126 27953 0 -1 4202496 1308977 0 0 0 349083 2657 0 0 25 0 5 0 595076660 5570465792 1307463 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28126/statm: 1359977 1307463 176 36 0 1357882 0
[pid=28126/tid=28128] ppid=28124 vsize=5439908 CPUtime=879.57 cores=0,2,4,6
/proc/28126/task/28128/stat : 28128 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 487937 0 0 0 87800 157 0 0 25 0 5 0 595076679 5570465792 1307463 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28126/tid=28129] ppid=28124 vsize=5439908 CPUtime=877.15 cores=0,2,4,6
/proc/28126/task/28129/stat : 28129 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 142828 0 0 0 86659 1056 0 0 25 0 5 0 595076679 5570465792 1307463 33554432000 4194304 4338100 140733250814656 18446744073709551615 4206909 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28126/tid=28130] ppid=28124 vsize=5439908 CPUtime=880.17 cores=0,2,4,6
/proc/28126/task/28130/stat : 28130 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 538223 0 0 0 87896 121 0 0 25 0 5 0 595076679 5570465792 1307463 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28126/tid=28131] ppid=28124 vsize=5439908 CPUtime=880.29 cores=0,2,4,6
/proc/28126/task/28131/stat : 28131 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 131258 0 0 0 86709 1320 0 0 25 0 5 0 595076679 5570465792 1307463 33554432000 4194304 4338100 140733250814656 18446744073709551615 4206902 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3517.4
Current children cumulated vsize (KiB) 5439908

[startup+942.306 s]
/proc/loadavg: 8.10 8.10 7.99 9/186 28155
/proc/meminfo: memFree=8005764/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=5440700 CPUtime=3756.79 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) S 28124 28126 27953 0 -1 4202496 1313529 0 0 0 372822 2857 0 0 25 0 5 0 595076660 5571276800 1312015 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28126/statm: 1360175 1312015 176 36 0 1358080 0
[pid=28126/tid=28128] ppid=28124 vsize=5440700 CPUtime=939.52 cores=0,2,4,6
/proc/28126/task/28128/stat : 28128 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 488937 0 0 0 93793 159 0 0 25 0 5 0 595076679 5571276800 1312015 33554432000 4194304 4338100 140733250814656 18446744073709551615 251366941712 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28126/tid=28129] ppid=28124 vsize=5440700 CPUtime=937.05 cores=0,2,4,6
/proc/28126/task/28129/stat : 28129 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 144500 0 0 0 92549 1156 0 0 25 0 5 0 595076679 5571276800 1312015 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28126/tid=28130] ppid=28124 vsize=5440700 CPUtime=939.94 cores=0,2,4,6
/proc/28126/task/28130/stat : 28130 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 538363 0 0 0 93873 121 0 0 25 0 5 0 595076679 5571276800 1312015 33554432000 4194304 4338100 140733250814656 18446744073709551615 4239490 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28126/tid=28131] ppid=28124 vsize=5440700 CPUtime=940.07 cores=0,2,4,6
/proc/28126/task/28131/stat : 28131 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 132998 0 0 0 92588 1419 0 0 25 0 5 0 595076679 5571276800 1312015 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3756.79
Current children cumulated vsize (KiB) 5440700

[startup+1002.3 s]
/proc/loadavg: 8.16 8.11 8.00 9/186 28157
/proc/meminfo: memFree=7664876/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=5563748 CPUtime=3996.17 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) S 28124 28126 27953 0 -1 4202496 1344113 0 0 0 396529 3088 0 0 25 0 5 0 595076660 5697277952 1342599 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28126/statm: 1390937 1342599 176 36 0 1388842 0
[pid=28126/tid=28128] ppid=28124 vsize=5563748 CPUtime=999.43 cores=0,2,4,6
/proc/28126/task/28128/stat : 28128 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 494231 0 0 0 99780 163 0 0 25 0 5 0 595076679 5697277952 1342599 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28126/tid=28129] ppid=28124 vsize=5563748 CPUtime=996.98 cores=0,2,4,6
/proc/28126/task/28129/stat : 28129 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 144683 0 0 0 98433 1265 0 0 25 0 5 0 595076679 5697277952 1342600 33554432000 4194304 4338100 140733250814656 18446744073709551615 4229520 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28126/tid=28130] ppid=28124 vsize=5563748 CPUtime=999.78 cores=0,2,4,6
/proc/28126/task/28130/stat : 28130 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 563288 0 0 0 99850 128 0 0 25 0 5 0 595076679 5697277952 1342600 33554432000 4194304 4338100 140733250814656 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28126/tid=28131] ppid=28124 vsize=5563748 CPUtime=999.76 cores=0,2,4,6
/proc/28126/task/28131/stat : 28131 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 133181 0 0 0 98447 1529 0 0 25 0 5 0 595076679 5697277952 1342600 33554432000 4194304 4338100 140733250814656 18446744073709551615 251366942121 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3996.17
Current children cumulated vsize (KiB) 5563748

[startup+1062.3 s]
/proc/loadavg: 8.05 8.09 8.00 9/186 28159
/proc/meminfo: memFree=7128808/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=5858704 CPUtime=4235.48 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) S 28124 28126 27953 0 -1 4202496 1418654 0 0 0 420206 3342 0 0 25 0 5 0 595076660 5999312896 1417140 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28126/statm: 1464676 1417140 176 36 0 1462581 0
[pid=28126/tid=28128] ppid=28124 vsize=5858704 CPUtime=1059.31 cores=0,2,4,6
/proc/28126/task/28128/stat : 28128 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 524506 0 0 0 105758 173 0 0 25 0 5 0 595076679 5999312896 1417140 33554432000 4194304 4338100 140733250814656 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28126/tid=28129] ppid=28124 vsize=5858704 CPUtime=1056.63 cores=0,2,4,6
/proc/28126/task/28129/stat : 28129 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 152924 0 0 0 104280 1383 0 0 25 0 5 0 595076679 5999312896 1417140 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28126/tid=28130] ppid=28124 vsize=5858704 CPUtime=1059.68 cores=0,2,4,6
/proc/28126/task/28130/stat : 28130 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 599140 0 0 0 105832 136 0 0 25 0 5 0 595076679 5999312896 1417140 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28126/tid=28131] ppid=28124 vsize=5858704 CPUtime=1059.64 cores=0,2,4,6
/proc/28126/task/28131/stat : 28131 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 133353 0 0 0 104317 1647 0 0 25 0 5 0 595076679 5999312896 1417140 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4235.48
Current children cumulated vsize (KiB) 5858704

[startup+1122.3 s]
/proc/loadavg: 8.13 8.10 8.00 9/186 28160
/proc/meminfo: memFree=6773164/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=6000256 CPUtime=4474.86 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) S 28124 28126 27953 0 -1 4202496 1454157 0 0 0 443933 3553 0 0 25 0 5 0 595076660 6144262144 1452643 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28126/statm: 1500064 1452643 176 36 0 1497969 0
[pid=28126/tid=28128] ppid=28124 vsize=6000256 CPUtime=1119.21 cores=0,2,4,6
/proc/28126/task/28128/stat : 28128 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 544355 0 0 0 111739 182 0 0 25 0 5 0 595076679 6144262144 1452643 33554432000 4194304 4338100 140733250814656 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28126/tid=28129] ppid=28124 vsize=6000256 CPUtime=1116.21 cores=0,2,4,6
/proc/28126/task/28129/stat : 28129 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 155469 0 0 0 110139 1482 0 0 25 0 5 0 595076679 6144262144 1452643 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28126/tid=28130] ppid=28124 vsize=6000256 CPUtime=1119.66 cores=0,2,4,6
/proc/28126/task/28130/stat : 28130 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 612112 0 0 0 111826 140 0 0 25 0 5 0 595076679 6144262144 1452643 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28126/tid=28131] ppid=28124 vsize=6000256 CPUtime=1119.55 cores=0,2,4,6
/proc/28126/task/28131/stat : 28131 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 133490 0 0 0 110210 1745 0 0 25 0 5 0 595076679 6144262144 1452643 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4474.86
Current children cumulated vsize (KiB) 6000256

[startup+1182.3 s]
/proc/loadavg: 8.09 8.09 8.00 9/186 28162
/proc/meminfo: memFree=6582188/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=6035324 CPUtime=4714.03 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) S 28124 28126 27953 0 -1 4202496 1464412 0 0 0 467669 3734 0 0 25 0 5 0 595076660 6180171776 1462898 33554432000 4194304 4338100 140733250814656 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28126/statm: 1508831 1462898 176 36 0 1506736 0
[pid=28126/tid=28128] ppid=28124 vsize=6035324 CPUtime=1179.06 cores=0,2,4,6
/proc/28126/task/28128/stat : 28128 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 553490 0 0 0 117716 190 0 0 25 0 5 0 595076679 6180171776 1462898 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28126/tid=28129] ppid=28124 vsize=6035324 CPUtime=1175.71 cores=0,2,4,6
/proc/28126/task/28129/stat : 28129 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 155588 0 0 0 116002 1569 0 0 25 0 5 0 595076679 6180171776 1462898 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28126/tid=28130] ppid=28124 vsize=6035324 CPUtime=1179.57 cores=0,2,4,6
/proc/28126/task/28130/stat : 28130 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 612240 0 0 0 117817 140 0 0 25 0 5 0 595076679 6180171776 1462898 33554432000 4194304 4338100 140733250814656 18446744073709551615 251366942289 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28126/tid=28131] ppid=28124 vsize=6035324 CPUtime=1179.47 cores=0,2,4,6
/proc/28126/task/28131/stat : 28131 (ttsth-5-4) R 28124 28126 27953 0 -1 4202560 134363 0 0 0 116116 1831 0 0 25 0 5 0 595076679 6180171776 1462898 33554432000 4194304 4338100 140733250814656 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4714.03
Current children cumulated vsize (KiB) 6035324



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.9 s]
/proc/loadavg: 7.74 8.02 7.98 6/183 28204
/proc/meminfo: memFree=14927280/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=0 CPUtime=4800.13 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) Z 28124 28126 27953 0 -1 4203548 1476799 0 0 0 476196 3817 0 0 18 0 2 0 595076660 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28126/statm: 0 0 0 0 0 0 0
[pid=28126/tid=28130] ppid=28124 vsize=0 CPUtime=1201.18 cores=0,2,4,6
/proc/28126/task/28130/stat : 28130 (ttsth-5-4) R 28124 28126 27953 0 -1 4203588 615570 0 0 0 119971 147 0 0 25 0 2 0 595076679 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.13
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1204.01 s]
/proc/loadavg: 7.74 8.02 7.98 6/184 28205
/proc/meminfo: memFree=17831976/32950928 swapFree=67111364/67111528
[pid=28126] ppid=28124 vsize=0 CPUtime=4800.24 cores=0,2,4,6
/proc/28126/stat : 28126 (ttsth-5-4) Z 28124 28126 27953 0 -1 4203548 1476799 0 0 0 476196 3828 0 0 18 0 2 0 595076660 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28126/statm: 0 0 0 0 0 0 0
[pid=28126/tid=28130] ppid=28124 vsize=0 CPUtime=1201.29 cores=0,2,4,6
/proc/28126/task/28130/stat : 28130 (ttsth-5-4) R 28124 28126 27953 0 -1 4203588 615570 0 0 0 119971 158 0 0 25 0 2 0 595076679 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.24
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1204.08
CPU time (s): 4800.3
CPU user time (s): 4761.96
CPU system time (s): 38.3402
CPU usage (%): 398.671
Max. virtual memory (cumulated for all children) (KiB): 6082640

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4761.96
system time used= 38.3402
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1476799
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 14440
involuntary context switches= 22736

runsolver used 2.07868 second user time and 4.48432 second system time

The end

Launcher Data 

Begin job on node113 at 2011-04-29 05:41:00
IDJOB=3257954
IDBENCH=71286
IDSOLVER=1654
FILE ID=node113/3257954-1304048460
RUNJOBID= node113-1304047252-27971
PBS_JOBID= 13170405
Free space on /tmp= 71164 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/rbsat/random/unforced/rbsat-v760c43649g5.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3257954-1304048460/watcher-3257954-1304048460 -o /tmp/evaluation-result-3257954-1304048460/solver-3257954-1304048460 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3257954-1304048460.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= fac733a1c5caf879cadf25b989055e5e
RANDOM SEED=1927172134

node113.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      19590796 kB
Buffers:       1742848 kB
Cached:       10820216 kB
SwapCached:        164 kB
Active:        6447940 kB
Inactive:      6322644 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      19590796 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:            1696 kB
Writeback:           0 kB
AnonPages:      206252 kB
Mapped:          15208 kB
Slab:           526684 kB
PageTables:       5244 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   399596 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71164 MiB
End job on node113 at 2011-04-29 06:01:06