Trace number 3264158

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.3 1204.35

General information on the benchmark

NameSAT04/handmade/kexu/frb-0.8/
frb45-21-1.used-as.sat04-883.cnf
MD5SUMd30b88c897dd41570f0430a76eb70b55
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark1.36879
Satisfiable
(Un)Satisfiability was proved
Number of variables945
Number of clauses61855
Sum of the clauses size124565
Maximum clause length21
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 261810
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 545

Solver Data 

0.01/0.04	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264158-1304099035.cnf 
0.01/0.04	c Using SATTIMEOUT of 4800
0.01/0.04	c Using SATRAM of 15500
0.01/0.04	c Size of subset node = 32
0.01/0.04	c Variables = 945
0.01/0.04	c Clauses = 61855
0.01/0.04	c Literals = 124565
0.01/0.04	c Reduced size:
0.01/0.04	c Variables = 945
0.01/0.04	c Clauses = 61855
0.01/0.04	c Literals = 124565
0.01/0.04	c a2
0.01/0.07	c a1
0.09/0.14	c Independent sub-problems = 1
0.09/0.14	c Sub-problem with 945 variables and 61855 clauses
0.09/0.14	c using permutation method 4
0.09/0.14	c count = 1
0.09/0.14	c Ordering finished 0.120 seconds
0.09/0.14	c mapped 0.130 seconds
0.09/0.14	c permuted 0.130 seconds
0.09/0.14	c about to sort
0.09/0.15	c sorted
0.18/0.27	c clauses now 61855
0.18/0.27	c init finished 0.150 seconds
0.18/0.27	c tree built 0.260 seconds
0.18/0.27	c 
0.18/0.27	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.18/0.27	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.18/0.27	c ................................ | ..........                    |                                  0.3         0.0             
0.18/0.27	c ................................ | ..........                  1 |                                  0.3         3.8             
0.18/0.27	c ................................ | ..........                  2 |                                  0.3         7.7             
0.18/0.27	c ................................ | ..........                  3 |                                  0.3        11.5             
0.18/0.27	c ................................ | ..........                  4 |                                  0.3        15.4             
0.18/0.27	c ................................ | ..........                  5 |                                  0.3        19.2             
0.18/0.28	c ................................ | ..........               1000 | 7            186                 0.3      3703.7         7778
95.87/24.36	c ................................ | ..........            3325000 | 92151        576584             96.0     34628.2     20454666
192.24/48.52	c ................................ | ..........            5899000 | 162244       1000344           192.0     30720.8     34190973
287.84/72.58	c ................................ | ..........            8004000 | 221863       1327814           288.0     27791.7     42566150
384.25/96.74	c ................................ | ..........            9925000 | 277351       1626736           384.0     25844.3     50523791
479.89/120.74	c ................................ | *.........           11489000 | 323241       1909611           480.0     23935.4     61043918
575.92/144.87	c ................................ | *.........           12899000 | 363530       2164949           576.0     22392.5     66711227
671.96/168.96	c ................................ | *.........           14202000 | 402135       2398083           672.1     21132.0     73807257
767.99/193.05	c ................................ | *.........           15343000 | 434658       2630924           768.1     19976.3     78548017
864.02/217.12	c ................................ | *.........           17080000 | 486188       2876833           864.0     19768.5     86989799
960.08/241.28	c ................................ | **........           18700000 | 535651       3120369           960.0     19478.4     96952940
1055.75/265.27	c ................................ | **........           20218000 | 579782       3366154          1056.0     19145.3    104488226
1151.84/289.37	c ................................ | **........           21529000 | 619141       3585339          1152.0     18688.2    109531065
1247.83/313.48	c ................................ | **........           22992000 | 665000       3814616          1248.0     18422.9    111383711
1343.88/337.58	c ................................ | **........           24455000 | 709575       4036950          1344.0     18195.3    118699283
1439.96/361.62	c ................................ | ***.......           25751000 | 751465       4243397          1440.0     17882.6    123863464
1536.03/385.70	c ................................ | ***.......           26999000 | 792621       4433356          1536.0     17577.0    128146944
1632.10/409.88	c ................................ | ***.......           28217000 | 831311       4623907          1632.0     17289.5    128950007
1728.23/433.92	c ................................ | ***.......           29398000 | 869258       4817677          1728.1     17012.1    133133571
1823.95/457.96	c ................................ | ***.......           30456000 | 904994       4986974          1824.1     16696.8    137198777
1920.06/482.01	c ................................ | ****......           31512000 | 939113       5142985          1920.0     16412.4    143282662
2015.80/506.07	c ................................ | ****......           32520000 | 969842       5305517          2016.1     16130.2    147845432
2111.93/530.18	c ................................ | ****......           33539000 | 1001647      5459414          2112.1     15879.5    153734669
2207.97/554.23	c ................................ | ****......           34510000 | 1034325      5607215          2208.0     15629.5    154604696
2304.07/578.30	c ................................ | ****......           35437000 | 1063625      5764351          2304.1     15380.3    157631195
2400.21/602.42	c ................................ | *****.....           36359000 | 1093381      5908649          2400.0     15149.5    161634162
2495.92/626.44	c ................................ | *****.....           37729000 | 1134754      6074730          2496.0     15115.6    167004105
2592.05/650.58	c ................................ | *****.....           38909000 | 1174176      6226486          2592.1     15010.8    170765285
2688.17/674.61	c ................................ | *****.....           40096000 | 1208380      6376614          2688.0     14916.7    172526688
2783.85/698.66	c ................................ | *****.....           41149000 | 1241378      6526883          2784.0     14780.4    173493243
2880.01/722.77	c ................................ | ******....           42107000 | 1274667      6671291          2880.0     14620.4    174155046
2975.73/746.79	c ................................ | ******....           43186000 | 1308172      6819091          2976.0     14511.4    177148484
3072.28/770.90	c ................................ | ******....           44181000 | 1339921      6959723          3072.1     14381.6    180364532
3167.97/794.92	c ................................ | ******....           45128000 | 1370559      7097018          3168.1     14244.6    183599616
3263.71/818.98	c ................................ | ******....           45990000 | 1398041      7234381          3264.1     14089.8    186193050
3359.84/843.04	c ................................ | *******...           46812000 | 1424476      7359491          3360.0     13932.0    186975756
3455.98/867.11	c ................................ | *******...           47677000 | 1450965      7484067          3456.0     13795.3    187604043
3551.71/891.18	c ................................ | *******...           48528000 | 1476714      7626515          3552.0     13662.0    190577604
3647.85/915.26	c ................................ | *******...           49384000 | 1502162      7768929          3648.1     13537.1    192312723
3743.95/939.32	c ................................ | *******...           50145000 | 1525193      7900265          3744.0     13393.3    192531226
3840.07/963.40	c ................................ | ********..           50907000 | 1548040      8033266          3840.1     13256.7    193521702
3935.70/987.49	c ................................ | ********..           51830000 | 1579377      8165583          3936.1     13168.0    194797680
4031.77/1011.56	c ................................ | ********..           52834000 | 1610356      8299670          4032.0     13103.5    196419189
4127.90/1035.64	c ................................ | ********..           53831000 | 1642671      8435507          4128.1     13040.3    196419189
4223.64/1059.69	c ................................ | ********..           54737000 | 1672589      8562099          4224.0     12958.5    196623992
4319.82/1083.75	c ................................ | *********.           55678000 | 1702753      8694012          4320.0     12888.4    198524574
4415.91/1107.84	c ................................ | *********.           56583000 | 1732485      8841647          4416.1     12812.9    201217612
4512.02/1131.93	c ................................ | *********.           57446000 | 1764039      8982288          4512.1     12731.5    201217612
4607.68/1155.99	c ................................ | *********.           58333000 | 1793716      9131114          4608.1     12658.9    201249737
4704.63/1180.23	c ................................ | *********.           59200000 | 1823237      9287626          4704.7     12583.1    203854858
4799.99/1204.11	c CPU limit (4800) reached
4799.99/1204.11	c Time = 4800.06 seconds
4799.99/1204.11	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3264158-1304099035/watcher-3264158-1304099035 -o /tmp/evaluation-result-3264158-1304099035/solver-3264158-1304099035 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264158-1304099035.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.43 7.97 7.99 3/177 5501
/proc/meminfo: memFree=23619764/32950928 swapFree=67071984/67111528
[pid=5501] ppid=5499 vsize=9412 CPUtime=0 cores=0,2,4,6
/proc/5501/stat : 5501 (ttsth-5-4) R 5499 5501 5016 0 -1 4202496 409 0 0 0 0 0 0 0 22 0 1 0 600134156 9637888 351 33554432000 4194304 4338100 140736395387376 18446744073709551615 260905054917 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/5501/statm: 2353 352 147 36 0 258 0

[startup+0.0216911 s]
/proc/loadavg: 7.43 7.97 7.99 3/177 5501
/proc/meminfo: memFree=23619764/32950928 swapFree=67071984/67111528
[pid=5501] ppid=5499 vsize=12724 CPUtime=0.01 cores=0,2,4,6
/proc/5501/stat : 5501 (ttsth-5-4) R 5499 5501 5016 0 -1 4202496 1229 0 0 0 1 0 0 0 22 0 1 0 600134156 13029376 1171 33554432000 4194304 4338100 140736395387376 18446744073709551615 260905054917 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/5501/statm: 3181 1172 148 36 0 1086 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 12724

[startup+0.100739 s]
/proc/loadavg: 7.43 7.97 7.99 3/177 5501
/proc/meminfo: memFree=23619764/32950928 swapFree=67071984/67111528
[pid=5501] ppid=5499 vsize=20528 CPUtime=0.09 cores=0,2,4,6
/proc/5501/stat : 5501 (ttsth-5-4) R 5499 5501 5016 0 -1 4202496 3717 0 0 0 9 0 0 0 22 0 1 0 600134156 21020672 3160 33554432000 4194304 4338100 140736395387376 18446744073709551615 260905054917 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/5501/statm: 5132 3164 155 36 0 3037 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 20528

[startup+0.300592 s]
/proc/loadavg: 7.43 7.97 7.99 3/177 5501
/proc/meminfo: memFree=23619764/32950928 swapFree=67071984/67111528
[pid=5501] ppid=5499 vsize=170660 CPUtime=0.35 cores=0,2,4,6
/proc/5501/stat : 5501 (ttsth-5-4) S 5499 5501 5016 0 -1 4202496 12222 0 0 0 35 0 0 0 23 0 5 0 600134156 174755840 11543 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/5501/statm: 42665 11543 176 36 0 40570 0
Current children cumulated CPU time (s) 0.35
Current children cumulated vsize (KiB) 170660

[startup+0.700521 s]
/proc/loadavg: 7.43 7.97 7.99 3/177 5501
/proc/meminfo: memFree=23619764/32950928 swapFree=67071984/67111528
[pid=5501] ppid=5499 vsize=255756 CPUtime=1.95 cores=0,2,4,6
/proc/5501/stat : 5501 (ttsth-5-4) S 5499 5501 5016 0 -1 4202496 17275 0 0 0 193 2 0 0 23 0 5 0 600134156 261894144 16596 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/5501/statm: 63939 16597 176 36 0 61844 0
Current children cumulated CPU time (s) 1.95
Current children cumulated vsize (KiB) 255756

[startup+1.50037 s]
/proc/loadavg: 7.47 7.97 7.99 9/186 5510
/proc/meminfo: memFree=23483928/32950928 swapFree=67071984/67111528
[pid=5501] ppid=5499 vsize=281176 CPUtime=5.13 cores=0,2,4,6
/proc/5501/stat : 5501 (ttsth-5-4) S 5499 5501 5016 0 -1 4202496 25329 0 0 0 507 6 0 0 23 0 5 0 600134156 287924224 24650 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/5501/statm: 70294 24650 176 36 0 68199 0
[pid=5501/tid=5507] ppid=5499 vsize=281176 CPUtime=1.21 cores=0,2,4,6
/proc/5501/task/5507/stat : 5507 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 4679 0 0 0 121 0 0 0 19 0 5 0 600134184 287924224 24650 33554432000 4194304 4338100 140736395387376 18446744073709551615 260904789119 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5501/tid=5508] ppid=5499 vsize=281176 CPUtime=1.19 cores=0,2,4,6
/proc/5501/task/5508/stat : 5508 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 2512 0 0 0 119 0 0 0 18 0 5 0 600134184 287924224 24650 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=5501/tid=5509] ppid=5499 vsize=281176 CPUtime=1.21 cores=0,2,4,6
/proc/5501/task/5509/stat : 5509 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 4234 0 0 0 120 1 0 0 25 0 5 0 600134184 287924224 24651 33554432000 4194304 4338100 140736395387376 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5501/tid=5510] ppid=5499 vsize=281176 CPUtime=1.21 cores=0,2,4,6
/proc/5501/task/5510/stat : 5510 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 2435 0 0 0 119 2 0 0 19 0 5 0 600134184 287924224 24651 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 5.13
Current children cumulated vsize (KiB) 281176

[startup+3.10108 s]
/proc/loadavg: 7.47 7.97 7.99 9/186 5510
/proc/meminfo: memFree=23412636/32950928 swapFree=67071984/67111528
[pid=5501] ppid=5499 vsize=343868 CPUtime=11.5 cores=0,2,4,6
/proc/5501/stat : 5501 (ttsth-5-4) S 5499 5501 5016 0 -1 4202496 40051 0 0 0 1138 12 0 0 23 0 5 0 600134156 352120832 39372 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/5501/statm: 85967 39372 176 36 0 83872 0
[pid=5501/tid=5507] ppid=5499 vsize=343868 CPUtime=2.81 cores=0,2,4,6
/proc/5501/task/5507/stat : 5507 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 9444 0 0 0 280 1 0 0 24 0 5 0 600134184 352120832 39372 33554432000 4194304 4338100 140736395387376 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5501/tid=5508] ppid=5499 vsize=343868 CPUtime=2.77 cores=0,2,4,6
/proc/5501/task/5508/stat : 5508 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 5031 0 0 0 275 2 0 0 18 0 5 0 600134184 352120832 39372 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=5501/tid=5509] ppid=5499 vsize=343868 CPUtime=2.8 cores=0,2,4,6
/proc/5501/task/5509/stat : 5509 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 9149 0 0 0 279 1 0 0 25 0 5 0 600134184 352120832 39372 33554432000 4194304 4338100 140736395387376 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5501/tid=5510] ppid=5499 vsize=343868 CPUtime=2.81 cores=0,2,4,6
/proc/5501/task/5510/stat : 5510 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 4958 0 0 0 276 5 0 0 19 0 5 0 600134184 352120832 39373 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 11.5
Current children cumulated vsize (KiB) 343868

[startup+6.30049 s]
/proc/loadavg: 7.47 7.97 7.99 9/186 5510
/proc/meminfo: memFree=23226524/32950928 swapFree=67071984/67111528
[pid=5501] ppid=5499 vsize=453744 CPUtime=24.22 cores=0,2,4,6
/proc/5501/stat : 5501 (ttsth-5-4) S 5499 5501 5016 0 -1 4202496 67370 0 0 0 2398 24 0 0 23 0 5 0 600134156 464633856 66691 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/5501/statm: 113436 66691 176 36 0 111341 0
[pid=5501/tid=5507] ppid=5499 vsize=453744 CPUtime=6.02 cores=0,2,4,6
/proc/5501/task/5507/stat : 5507 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 19201 0 0 0 598 4 0 0 25 0 5 0 600134184 464633856 66691 33554432000 4194304 4338100 140736395387376 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5501/tid=5508] ppid=5499 vsize=453744 CPUtime=5.91 cores=0,2,4,6
/proc/5501/task/5508/stat : 5508 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 9728 0 0 0 586 5 0 0 18 0 5 0 600134184 464633856 66691 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=5501/tid=5509] ppid=5499 vsize=453744 CPUtime=5.98 cores=0,2,4,6
/proc/5501/task/5509/stat : 5509 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 17240 0 0 0 596 2 0 0 25 0 5 0 600134184 464633856 66691 33554432000 4194304 4338100 140736395387376 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5501/tid=5510] ppid=5499 vsize=453744 CPUtime=6.01 cores=0,2,4,6
/proc/5501/task/5510/stat : 5510 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 9732 0 0 0 590 11 0 0 25 0 5 0 600134184 464633856 66691 33554432000 4194304 4338100 140736395387376 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 24.22
Current children cumulated vsize (KiB) 453744

[startup+12.7003 s]
/proc/loadavg: 7.55 7.97 7.99 9/186 5510
/proc/meminfo: memFree=22928452/32950928 swapFree=67071984/67111528
[pid=5501] ppid=5499 vsize=639656 CPUtime=49.69 cores=0,2,4,6
/proc/5501/stat : 5501 (ttsth-5-4) S 5499 5501 5016 0 -1 4202496 114186 0 0 0 4924 45 0 0 23 0 5 0 600134156 655007744 113507 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/5501/statm: 159914 113507 176 36 0 157819 0
[pid=5501/tid=5507] ppid=5499 vsize=639656 CPUtime=12.41 cores=0,2,4,6
/proc/5501/task/5507/stat : 5507 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 35923 0 0 0 1235 6 0 0 25 0 5 0 600134184 655007744 113507 33554432000 4194304 4338100 140736395387376 18446744073709551615 260904789124 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5501/tid=5508] ppid=5499 vsize=639656 CPUtime=12.22 cores=0,2,4,6
/proc/5501/task/5508/stat : 5508 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 17965 0 0 0 1214 8 0 0 18 0 5 0 600134184 655007744 113507 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=5501/tid=5509] ppid=5499 vsize=639656 CPUtime=12.34 cores=0,2,4,6
/proc/5501/task/5509/stat : 5509 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 31897 0 0 0 1230 4 0 0 25 0 5 0 600134184 655007744 113507 33554432000 4194304 4338100 140736395387376 18446744073709551615 260904789473 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5501/tid=5510] ppid=5499 vsize=639656 CPUtime=12.41 cores=0,2,4,6
/proc/5501/task/5510/stat : 5510 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 16932 0 0 0 1217 24 0 0 25 0 5 0 600134184 655007744 113507 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 49.69
Current children cumulated vsize (KiB) 639656

[startup+25.501 s]
/proc/loadavg: 7.62 7.97 7.99 9/186 5511
/proc/meminfo: memFree=22498672/32950928 swapFree=67071984/67111528
[pid=5501] ppid=5499 vsize=911608 CPUtime=100.65 cores=0,2,4,6
/proc/5501/stat : 5501 (ttsth-5-4) S 5499 5501 5016 0 -1 4202496 182034 0 0 0 9983 82 0 0 23 0 5 0 600134156 933486592 181010 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/5501/statm: 227902 181011 176 36 0 225807 0
[pid=5501/tid=5507] ppid=5499 vsize=911608 CPUtime=25.21 cores=0,2,4,6
/proc/5501/task/5507/stat : 5507 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 60384 0 0 0 2508 13 0 0 25 0 5 0 600134184 933486592 181011 33554432000 4194304 4338100 140736395387376 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5501/tid=5508] ppid=5499 vsize=911608 CPUtime=24.85 cores=0,2,4,6
/proc/5501/task/5508/stat : 5508 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 29816 0 0 0 2471 14 0 0 18 0 5 0 600134184 933486592 181011 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=5501/tid=5509] ppid=5499 vsize=911608 CPUtime=25.07 cores=0,2,4,6
/proc/5501/task/5509/stat : 5509 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 53606 0 0 0 2499 8 0 0 25 0 5 0 600134184 933486592 181011 33554432000 4194304 4338100 140736395387376 18446744073709551615 260904789031 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5501/tid=5510] ppid=5499 vsize=911608 CPUtime=25.22 cores=0,2,4,6
/proc/5501/task/5510/stat : 5510 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 26760 0 0 0 2477 45 0 0 25 0 5 0 600134184 933486592 181011 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 100.65
Current children cumulated vsize (KiB) 911608

[startup+51.1003 s]
/proc/loadavg: 7.75 7.97 7.99 9/186 5511
/proc/meminfo: memFree=21893256/32950928 swapFree=67071984/67111528
[pid=5501] ppid=5499 vsize=1363940 CPUtime=202.59 cores=0,2,4,6
/proc/5501/stat : 5501 (ttsth-5-4) S 5499 5501 5016 0 -1 4202496 294548 0 0 0 20112 147 0 0 23 0 5 0 600134156 1396674560 293524 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/5501/statm: 340985 293524 176 36 0 338890 0
[pid=5501/tid=5507] ppid=5499 vsize=1363940 CPUtime=50.81 cores=0,2,4,6
/proc/5501/task/5507/stat : 5507 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 110861 0 0 0 5056 25 0 0 25 0 5 0 600134184 1396674560 293524 33554432000 4194304 4338100 140736395387376 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5501/tid=5508] ppid=5499 vsize=1363940 CPUtime=50.12 cores=0,2,4,6
/proc/5501/task/5508/stat : 5508 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 47605 0 0 0 4989 23 0 0 18 0 5 0 600134184 1396674560 293524 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=5501/tid=5509] ppid=5499 vsize=1363940 CPUtime=50.54 cores=0,2,4,6
/proc/5501/task/5509/stat : 5509 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 80884 0 0 0 5042 12 0 0 25 0 5 0 600134184 1396674560 293524 33554432000 4194304 4338100 140736395387376 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5501/tid=5510] ppid=5499 vsize=1363940 CPUtime=50.81 cores=0,2,4,6
/proc/5501/task/5510/stat : 5510 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 43729 0 0 0 4997 84 0 0 25 0 5 0 600134184 1396674560 293524 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 202.59
Current children cumulated vsize (KiB) 1363940

[startup+102.307 s]
/proc/loadavg: 7.90 7.97 7.99 9/186 5513
/proc/meminfo: memFree=20976268/32950928 swapFree=67071984/67111528
[pid=5501] ppid=5499 vsize=1953992 CPUtime=406.57 cores=0,2,4,6
/proc/5501/stat : 5501 (ttsth-5-4) S 5499 5501 5016 0 -1 4202496 440578 0 0 0 40406 251 0 0 23 0 5 0 600134156 2000887808 439554 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/5501/statm: 488498 439554 176 36 0 486403 0
[pid=5501/tid=5507] ppid=5499 vsize=1953992 CPUtime=102 cores=0,2,4,6
/proc/5501/task/5507/stat : 5507 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 179575 0 0 0 10157 43 0 0 25 0 5 0 600134184 2000887808 439554 33554432000 4194304 4338100 140736395387376 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5501/tid=5508] ppid=5499 vsize=1953992 CPUtime=100.8 cores=0,2,4,6
/proc/5501/task/5508/stat : 5508 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 65188 0 0 0 10044 36 0 0 18 0 5 0 600134184 2000887808 439554 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=5501/tid=5509] ppid=5499 vsize=1953992 CPUtime=101.46 cores=0,2,4,6
/proc/5501/task/5509/stat : 5509 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 123803 0 0 0 10125 21 0 0 25 0 5 0 600134184 2000887808 439554 33554432000 4194304 4338100 140736395387376 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5501/tid=5510] ppid=5499 vsize=1953992 CPUtime=102.01 cores=0,2,4,6
/proc/5501/task/5510/stat : 5510 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 60543 0 0 0 10053 148 0 0 25 0 5 0 600134184 2000887808 439554 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 406.57
Current children cumulated vsize (KiB) 1953992

[startup+162.301 s]
/proc/loadavg: 8.04 8.00 8.00 9/186 5514
/proc/meminfo: memFree=20239172/32950928 swapFree=67071984/67111528
[pid=5501] ppid=5499 vsize=2518616 CPUtime=645.68 cores=0,2,4,6
/proc/5501/stat : 5501 (ttsth-5-4) S 5499 5501 5016 0 -1 4202496 581550 0 0 0 64211 357 0 0 23 0 5 0 600134156 2579062784 580526 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/5501/statm: 629654 580526 176 36 0 627559 0
[pid=5501/tid=5507] ppid=5499 vsize=2518616 CPUtime=161.98 cores=0,2,4,6
/proc/5501/task/5507/stat : 5507 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 239703 0 0 0 16137 61 0 0 25 0 5 0 600134184 2579062784 580526 33554432000 4194304 4338100 140736395387376 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5501/tid=5508] ppid=5499 vsize=2518616 CPUtime=160.27 cores=0,2,4,6
/proc/5501/task/5508/stat : 5508 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 87601 0 0 0 15977 50 0 0 18 0 5 0 600134184 2579062784 580526 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=5501/tid=5509] ppid=5499 vsize=2518616 CPUtime=161.11 cores=0,2,4,6
/proc/5501/task/5509/stat : 5509 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 163937 0 0 0 16085 26 0 0 25 0 5 0 600134184 2579062784 580526 33554432000 4194304 4338100 140736395387376 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5501/tid=5510] ppid=5499 vsize=2518616 CPUtime=162 cores=0,2,4,6
/proc/5501/task/5510/stat : 5510 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 78840 0 0 0 15983 217 0 0 25 0 5 0 600134184 2579062784 580526 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 645.68
Current children cumulated vsize (KiB) 2518616


################
# More data... #
################

/proc/5501/task/5510/stat : 5510 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 163945 0 0 0 69504 627 0 0 18 0 5 0 600134184 5949153280 1400638 33554432000 4194304 4338100 140736395387376 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 2798.63
Current children cumulated vsize (KiB) 5809720

[startup+762.307 s]
/proc/loadavg: 8.08 8.09 8.02 9/186 5530
/proc/meminfo: memFree=14679096/32950928 swapFree=67071984/67111528
[pid=5501] ppid=5499 vsize=5988676 CPUtime=3038 cores=0,2,4,6
/proc/5501/stat : 5501 (ttsth-5-4) S 5499 5501 5016 0 -1 4202496 1450386 0 0 0 302680 1120 0 0 23 0 5 0 600134156 6132404224 1449362 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/5501/statm: 1497169 1449362 176 36 0 1495074 0
[pid=5501/tid=5507] ppid=5499 vsize=5988676 CPUtime=759.87 cores=0,2,4,6
/proc/5501/task/5507/stat : 5507 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 657309 0 0 0 75808 179 0 0 25 0 5 0 600134184 6132404224 1449362 33554432000 4194304 4338100 140736395387376 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5501/tid=5508] ppid=5499 vsize=5988676 CPUtime=757.03 cores=0,2,4,6
/proc/5501/task/5508/stat : 5508 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 176729 0 0 0 75481 222 0 0 25 0 5 0 600134184 6132404224 1449362 33554432000 4194304 4338100 140736395387376 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5501/tid=5509] ppid=5499 vsize=5988676 CPUtime=759.78 cores=0,2,4,6
/proc/5501/task/5509/stat : 5509 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 438993 0 0 0 75893 85 0 0 25 0 5 0 600134184 6132404224 1449362 33554432000 4194304 4338100 140736395387376 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5501/tid=5510] ppid=5499 vsize=5988676 CPUtime=761.03 cores=0,2,4,6
/proc/5501/task/5510/stat : 5510 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 165886 0 0 0 75472 631 0 0 18 0 5 0 600134184 6132404224 1449362 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3038
Current children cumulated vsize (KiB) 5988676

[startup+822.301 s]
/proc/loadavg: 8.03 8.07 8.02 9/186 5532
/proc/meminfo: memFree=14336348/32950928 swapFree=67071984/67111528
[pid=5501] ppid=5499 vsize=6217680 CPUtime=3277.28 cores=0,2,4,6
/proc/5501/stat : 5501 (ttsth-5-4) S 5499 5501 5016 0 -1 4202496 1507443 0 0 0 326547 1181 0 0 23 0 5 0 600134156 6366904320 1506419 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/5501/statm: 1554420 1506419 176 36 0 1552325 0
[pid=5501/tid=5507] ppid=5499 vsize=6217680 CPUtime=819.51 cores=0,2,4,6
/proc/5501/task/5507/stat : 5507 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 692840 0 0 0 81758 193 0 0 25 0 5 0 600134184 6366904320 1506419 33554432000 4194304 4338100 140736395387376 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5501/tid=5508] ppid=5499 vsize=6217680 CPUtime=817 cores=0,2,4,6
/proc/5501/task/5508/stat : 5508 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 176826 0 0 0 81440 260 0 0 25 0 5 0 600134184 6366904320 1506419 33554432000 4194304 4338100 140736395387376 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5501/tid=5509] ppid=5499 vsize=6217680 CPUtime=819.74 cores=0,2,4,6
/proc/5501/task/5509/stat : 5509 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 460324 0 0 0 81884 90 0 0 25 0 5 0 600134184 6366904320 1506419 33554432000 4194304 4338100 140736395387376 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=5501/tid=5510] ppid=5499 vsize=6217680 CPUtime=820.72 cores=0,2,4,6
/proc/5501/task/5510/stat : 5510 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 165984 0 0 0 81437 635 0 0 18 0 5 0 600134184 6366904320 1506419 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3277.28
Current children cumulated vsize (KiB) 6217680

[startup+882.301 s]
/proc/loadavg: 8.05 8.07 8.02 9/186 5533
/proc/meminfo: memFree=14124156/32950928 swapFree=67071984/67111528
[pid=5501] ppid=5499 vsize=6290416 CPUtime=3516.61 cores=0,2,4,6
/proc/5501/stat : 5501 (ttsth-5-4) S 5499 5501 5016 0 -1 4202496 1526308 0 0 0 350430 1231 0 0 23 0 5 0 600134156 6441385984 1525284 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/5501/statm: 1572604 1525284 176 36 0 1570509 0
[pid=5501/tid=5507] ppid=5499 vsize=6290416 CPUtime=879.18 cores=0,2,4,6
/proc/5501/task/5507/stat : 5507 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 705000 0 0 0 87719 199 0 0 25 0 5 0 600134184 6441385984 1525284 33554432000 4194304 4338100 140736395387376 18446744073709551615 4229565 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5501/tid=5508] ppid=5499 vsize=6290416 CPUtime=877 cores=0,2,4,6
/proc/5501/task/5508/stat : 5508 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 176908 0 0 0 87401 299 0 0 25 0 5 0 600134184 6441385984 1525284 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5501/tid=5509] ppid=5499 vsize=6290416 CPUtime=879.73 cores=0,2,4,6
/proc/5501/task/5509/stat : 5509 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 466871 0 0 0 87881 92 0 0 25 0 5 0 600134184 6441385984 1525284 33554432000 4194304 4338100 140736395387376 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=5501/tid=5510] ppid=5499 vsize=6290416 CPUtime=880.4 cores=0,2,4,6
/proc/5501/task/5510/stat : 5510 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 166060 0 0 0 87401 639 0 0 18 0 5 0 600134184 6441385984 1525284 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3516.61
Current children cumulated vsize (KiB) 6290416

[startup+942.301 s]
/proc/loadavg: 8.05 8.07 8.02 9/186 5535
/proc/meminfo: memFree=13897964/32950928 swapFree=67071984/67111528
[pid=5501] ppid=5499 vsize=6406944 CPUtime=3755.9 cores=0,2,4,6
/proc/5501/stat : 5501 (ttsth-5-4) S 5499 5501 5016 0 -1 4202496 1559913 0 0 0 374301 1289 0 0 23 0 5 0 600134156 6560710656 1558889 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/5501/statm: 1601736 1558889 176 36 0 1599641 0
[pid=5501/tid=5507] ppid=5499 vsize=6406944 CPUtime=938.81 cores=0,2,4,6
/proc/5501/task/5507/stat : 5507 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 717984 0 0 0 93677 204 0 0 25 0 5 0 600134184 6560710656 1558889 33554432000 4194304 4338100 140736395387376 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5501/tid=5508] ppid=5499 vsize=6406944 CPUtime=936.96 cores=0,2,4,6
/proc/5501/task/5508/stat : 5508 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 177771 0 0 0 93355 341 0 0 25 0 5 0 600134184 6560710656 1558889 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5501/tid=5509] ppid=5499 vsize=6406944 CPUtime=939.69 cores=0,2,4,6
/proc/5501/task/5509/stat : 5509 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 484442 0 0 0 93873 96 0 0 25 0 5 0 600134184 6560710656 1558889 33554432000 4194304 4338100 140736395387376 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=5501/tid=5510] ppid=5499 vsize=6406944 CPUtime=940.12 cores=0,2,4,6
/proc/5501/task/5510/stat : 5510 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 168247 0 0 0 93367 645 0 0 18 0 5 0 600134184 6560710656 1558889 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3755.9
Current children cumulated vsize (KiB) 6406944

[startup+1002.3 s]
/proc/loadavg: 8.05 8.07 8.01 9/186 5537
/proc/meminfo: memFree=13711680/32950928 swapFree=67071984/67111528
[pid=5501] ppid=5499 vsize=6611516 CPUtime=3995.12 cores=0,2,4,6
/proc/5501/stat : 5501 (ttsth-5-4) S 5499 5501 5016 0 -1 4202496 1593616 0 0 0 398171 1341 0 0 23 0 5 0 600134156 6770192384 1592592 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/5501/statm: 1652879 1592592 176 36 0 1650784 0
[pid=5501/tid=5507] ppid=5499 vsize=6611516 CPUtime=998.45 cores=0,2,4,6
/proc/5501/task/5507/stat : 5507 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 727585 0 0 0 99636 209 0 0 25 0 5 0 600134184 6770192384 1592592 33554432000 4194304 4338100 140736395387376 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5501/tid=5508] ppid=5499 vsize=6611516 CPUtime=996.93 cores=0,2,4,6
/proc/5501/task/5508/stat : 5508 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 179213 0 0 0 99316 377 0 0 25 0 5 0 600134184 6770192384 1592592 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5501/tid=5509] ppid=5499 vsize=6611516 CPUtime=999.65 cores=0,2,4,6
/proc/5501/task/5509/stat : 5509 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 505508 0 0 0 99862 103 0 0 25 0 5 0 600134184 6770192384 1592592 33554432000 4194304 4338100 140736395387376 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=5501/tid=5510] ppid=5499 vsize=6611516 CPUtime=999.8 cores=0,2,4,6
/proc/5501/task/5510/stat : 5510 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 169841 0 0 0 99330 650 0 0 18 0 5 0 600134184 6770192384 1592592 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3995.12
Current children cumulated vsize (KiB) 6611516

[startup+1062.3 s]
/proc/loadavg: 8.14 8.08 8.02 9/186 5547
/proc/meminfo: memFree=13487116/32950928 swapFree=67071984/67111528
[pid=5501] ppid=5499 vsize=6668776 CPUtime=4234.42 cores=0,2,4,6
/proc/5501/stat : 5501 (ttsth-5-4) S 5499 5501 5016 0 -1 4202496 1611777 0 0 0 422055 1387 0 0 23 0 5 0 600134156 6828826624 1610753 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/5501/statm: 1667194 1610753 176 36 0 1665099 0
[pid=5501/tid=5507] ppid=5499 vsize=6668776 CPUtime=1058.08 cores=0,2,4,6
/proc/5501/task/5507/stat : 5507 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 733950 0 0 0 105595 213 0 0 25 0 5 0 600134184 6828826624 1610753 33554432000 4194304 4338100 140736395387376 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5501/tid=5508] ppid=5499 vsize=6668776 CPUtime=1056.9 cores=0,2,4,6
/proc/5501/task/5508/stat : 5508 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 184181 0 0 0 105277 413 0 0 25 0 5 0 600134184 6828826624 1610753 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5501/tid=5509] ppid=5499 vsize=6668776 CPUtime=1059.61 cores=0,2,4,6
/proc/5501/task/5509/stat : 5509 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 508455 0 0 0 105857 104 0 0 25 0 5 0 600134184 6828826624 1610753 33554432000 4194304 4338100 140736395387376 18446744073709551615 260904789438 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=5501/tid=5510] ppid=5499 vsize=6668776 CPUtime=1059.53 cores=0,2,4,6
/proc/5501/task/5510/stat : 5510 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 173722 0 0 0 105299 654 0 0 18 0 5 0 600134184 6828826624 1610753 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4234.42
Current children cumulated vsize (KiB) 6668776

[startup+1122.3 s]
/proc/loadavg: 8.16 8.10 8.02 9/186 5549
/proc/meminfo: memFree=13302932/32950928 swapFree=67071984/67111528
[pid=5501] ppid=5499 vsize=6807200 CPUtime=4473.75 cores=0,2,4,6
/proc/5501/stat : 5501 (ttsth-5-4) S 5499 5501 5016 0 -1 4202496 1649403 0 0 0 445928 1447 0 0 23 0 5 0 600134156 6970572800 1648379 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/5501/statm: 1701800 1648379 176 36 0 1699705 0
[pid=5501/tid=5507] ppid=5499 vsize=6807200 CPUtime=1117.74 cores=0,2,4,6
/proc/5501/task/5507/stat : 5507 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 757486 0 0 0 111553 221 0 0 25 0 5 0 600134184 6970572800 1648379 33554432000 4194304 4338100 140736395387376 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5501/tid=5508] ppid=5499 vsize=6807200 CPUtime=1116.89 cores=0,2,4,6
/proc/5501/task/5508/stat : 5508 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 186380 0 0 0 111233 456 0 0 25 0 5 0 600134184 6970572800 1648379 33554432000 4194304 4338100 140736395387376 18446744073709551615 4254218 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5501/tid=5509] ppid=5499 vsize=6807200 CPUtime=1119.6 cores=0,2,4,6
/proc/5501/task/5509/stat : 5509 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 519072 0 0 0 111852 108 0 0 25 0 5 0 600134184 6970572800 1648379 33554432000 4194304 4338100 140736395387376 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=5501/tid=5510] ppid=5499 vsize=6807200 CPUtime=1119.22 cores=0,2,4,6
/proc/5501/task/5510/stat : 5510 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 174996 0 0 0 111262 660 0 0 18 0 5 0 600134184 6970572800 1648379 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4473.75
Current children cumulated vsize (KiB) 6807200

[startup+1182.3 s]
/proc/loadavg: 8.23 8.12 8.03 9/186 5550
/proc/meminfo: memFree=13102788/32950928 swapFree=67071984/67111528
[pid=5501] ppid=5499 vsize=6968112 CPUtime=4712.97 cores=0,2,4,6
/proc/5501/stat : 5501 (ttsth-5-4) S 5499 5501 5016 0 -1 4202496 1676449 0 0 0 469794 1503 0 0 23 0 5 0 600134156 7135346688 1675425 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/5501/statm: 1742028 1675425 176 36 0 1739933 0
[pid=5501/tid=5507] ppid=5499 vsize=6968112 CPUtime=1177.39 cores=0,2,4,6
/proc/5501/task/5507/stat : 5507 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 765920 0 0 0 117513 226 0 0 25 0 5 0 600134184 7135346688 1675425 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917208093 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5501/tid=5508] ppid=5499 vsize=6968112 CPUtime=1176.85 cores=0,2,4,6
/proc/5501/task/5508/stat : 5508 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 189855 0 0 0 117187 498 0 0 25 0 5 0 600134184 7135346688 1675425 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5501/tid=5509] ppid=5499 vsize=6968112 CPUtime=1179.55 cores=0,2,4,6
/proc/5501/task/5509/stat : 5509 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 531584 0 0 0 117843 112 0 0 25 0 5 0 600134184 7135346688 1675425 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917206734 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=5501/tid=5510] ppid=5499 vsize=6968112 CPUtime=1178.87 cores=0,2,4,6
/proc/5501/task/5510/stat : 5510 (ttsth-5-4) R 5499 5501 5016 0 -1 4202560 177621 0 0 0 117222 665 0 0 18 0 5 0 600134184 7135346688 1675425 33554432000 4194304 4338100 140736395387376 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4712.97
Current children cumulated vsize (KiB) 6968112



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1204.2 s]
/proc/loadavg: 8.17 8.11 8.03 2/177 5552
/proc/meminfo: memFree=19085072/32950928 swapFree=67071984/67111528
[pid=5501] ppid=5499 vsize=0 CPUtime=4800.15 cores=0,2,4,6
/proc/5501/stat : 5501 (ttsth-5-4) Z 5499 5501 5016 0 -1 4203548 1684493 0 0 0 478457 1558 0 0 18 0 2 0 600134156 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/5501/statm: 0 0 0 0 0 0 0
[pid=5501/tid=5509] ppid=5499 vsize=0 CPUtime=1201.46 cores=0,2,4,6
/proc/5501/task/5509/stat : 5509 (ttsth-5-4) R 5499 5501 5016 0 -1 4203588 538157 0 0 0 120023 123 0 0 25 0 2 0 600134184 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.15
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1204.31 s]
/proc/loadavg: 8.17 8.11 8.03 2/177 5552
/proc/meminfo: memFree=19085072/32950928 swapFree=67071984/67111528
[pid=5501] ppid=5499 vsize=0 CPUtime=4800.25 cores=0,2,4,6
/proc/5501/stat : 5501 (ttsth-5-4) Z 5499 5501 5016 0 -1 4203548 1684493 0 0 0 478457 1568 0 0 18 0 2 0 600134156 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/5501/statm: 0 0 0 0 0 0 0
[pid=5501/tid=5509] ppid=5499 vsize=0 CPUtime=1201.56 cores=0,2,4,6
/proc/5501/task/5509/stat : 5509 (ttsth-5-4) R 5499 5501 5016 0 -1 4203588 538157 0 0 0 120023 133 0 0 25 0 2 0 600134184 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.25
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1204.35
CPU time (s): 4800.3
CPU user time (s): 4784.58
CPU system time (s): 15.7256
CPU usage (%): 398.58
Max. virtual memory (cumulated for all children) (KiB): 6999656

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4784.58
system time used= 15.7256
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1684493
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 12661
involuntary context switches= 30930

runsolver used 2.13167 second user time and 4.38533 second system time

The end

Launcher Data 

Begin job on node116 at 2011-04-29 19:43:55
IDJOB=3264158
IDBENCH=84559
IDSOLVER=1654
FILE ID=node116/3264158-1304099035
RUNJOBID= node116-1304099035-5481
PBS_JOBID= 13170597
Free space on /tmp= 72516 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT04/handmade/kexu/frb-0.8/frb45-21-1.used-as.sat04-883.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3264158-1304099035/watcher-3264158-1304099035 -o /tmp/evaluation-result-3264158-1304099035/solver-3264158-1304099035 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264158-1304099035.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= d30b88c897dd41570f0430a76eb70b55
RANDOM SEED=1054548790

node116.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      23620292 kB
Buffers:       1904872 kB
Cached:        6953468 kB
SwapCached:       9732 kB
Active:        4403216 kB
Inactive:      4498180 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      23620292 kB
SwapTotal:    67111528 kB
SwapFree:     67071984 kB
Dirty:            1308 kB
Writeback:           0 kB
AnonPages:       36380 kB
Mapped:          13880 kB
Slab:           365820 kB
PageTables:       5128 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   209540 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72512 MiB
End job on node116 at 2011-04-29 20:04:01