Trace number 3251308

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.57 1207.89

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m37-q505.75-n35-p30-s59841049.cnf
MD5SUMc86ce640ddb0138bf41ccc37bc6fcf28
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.84587
Satisfiable
(Un)Satisfiability was proved
Number of variables1295
Number of clauses148400
Sum of the clauses size298025
Maximum clause length37
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2148365
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 535

Solver Data 

0.09/0.10	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251308-1304107377.cnf 
0.09/0.10	c Using SATTIMEOUT of 4800
0.09/0.10	c Using SATRAM of 15500
0.09/0.10	c Size of subset node = 32
0.09/0.10	c Variables = 1295
0.09/0.10	c Clauses = 148400
0.09/0.10	c Literals = 298025
0.09/0.10	c Reduced size:
0.09/0.10	c Variables = 1295
0.09/0.10	c Clauses = 148400
0.09/0.10	c Literals = 298025
0.09/0.10	c a2
0.09/0.18	c a1
0.39/0.40	c Independent sub-problems = 1
0.39/0.40	c Sub-problem with 1295 variables and 148400 clauses
0.39/0.40	c using permutation method 4
0.39/0.40	c count = 1
0.39/0.40	c Ordering finished 0.370 seconds
0.39/0.40	c mapped 0.390 seconds
0.39/0.40	c permuted 0.390 seconds
0.39/0.40	c about to sort
0.39/0.41	c sorted
0.79/0.80	c clauses now 148400
0.79/0.80	c init finished 0.430 seconds
0.79/0.80	c tree built 0.800 seconds
0.79/0.80	c 
0.79/0.80	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.79/0.80	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.79/0.80	c ................................ | ..........                    |                                  0.8         0.0             
0.79/0.80	c ................................ | ..........                  1 |                                  0.8         1.2             
0.79/0.80	c ................................ | ..........                  2 |                                  0.8         2.5             
0.79/0.80	c ................................ | ..........                  3 |                                  0.8         3.8             
0.79/0.80	c ................................ | ..........                  4 |                                  0.8         5.0             
0.79/0.80	c ................................ | ..........                  5 |                                  0.8         6.2             
0.79/0.81	c ................................ | ..........               1000 |              105                 0.8      1234.6         5700
95.78/24.75	c ................................ | ..........            5870000 | 2012         693753             96.0     61145.8     28933083
191.95/48.91	c ................................ | ..........           11365000 | 4351         1335949           192.0     59192.7     47655245
287.76/73.06	c ................................ | ..........           16388000 | 6882         1922708           288.0     56902.8     59974822
383.98/97.20	c ................................ | ..........           21515000 | 7937         2518145           384.0     56027.2     69883635
479.82/121.34	c ................................ | *.........           26594000 | 9666         3102158           480.0     55404.2     83944491
575.64/145.48	c ................................ | *.........           31783000 | 10603        3677379           576.0     55178.8     99198693
671.83/169.64	c ................................ | *.........           36805000 | 12488        4263706           672.0     54769.3    113511484
767.64/193.79	c ................................ | *.........           41444000 | 14405        4808290           768.0     53963.5    119405458
863.77/217.95	c ................................ | *.........           46371000 | 16957        5391700           864.0     53670.1    132521055
959.90/242.12	c ................................ | **........           51214000 | 20295        5956116           960.0     53347.9    146717777
1055.71/266.28	c ................................ | **........           55585000 | 21534        6484218          1056.0     52637.3    148375028
1151.90/290.43	c ................................ | **........           60002000 | 23832        7008133          1152.0     52085.1    160843287
1247.98/314.61	c ................................ | **........           64715000 | 25259        7559919          1248.0     51854.6    166508919
1343.75/338.77	c ................................ | **........           69492000 | 27623        8101122          1344.0     51705.0    180945779
1439.90/362.82	c ................................ | ***.......           74219000 | 29353        8642838          1440.0     51541.0    196358551
1536.00/387.00	c ................................ | ***.......           78587000 | 31355        9173660          1536.0     51163.1    207210562
1631.68/411.18	c ................................ | ***.......           83027000 | 34760        9712928          1632.0     50874.4    221043176
1727.77/435.36	c ................................ | ***.......           87177000 | 35985        10233580         1728.0     50449.4    227423457
1823.93/459.52	c ................................ | ***.......           91284000 | 38116        10733807         1824.0     50045.8    234537538
1919.73/483.57	c ................................ | ****......           95173000 | 39759        11213531         1920.0     49569.0    239129144
2015.94/507.71	c ................................ | ****......           99426000 | 40777        11744610         2016.0     49318.5    245906374
2111.73/531.86	c ................................ | ****......          103899000 | 42590        12270266         2112.0     49194.4    252706620
2207.92/556.02	c ................................ | ****......          108352000 | 46072        12780550         2208.0     49072.0    256915983
2303.67/580.18	c ................................ | ****......          112774000 | 49226        13283971         2304.0     48946.8    261752650
2399.80/604.35	c ................................ | *****.....          117231000 | 52596        13800249         2400.0     48846.0    274233903
2495.93/628.51	c ................................ | *****.....          121865000 | 55242        14328572         2496.0     48824.1    284894690
2591.65/652.68	c ................................ | *****.....          126700000 | 59047        14884073         2592.0     48881.2    301335961
2687.64/676.89	c ................................ | *****.....          130766000 | 62259        15364406         2688.0     48647.9    305068398
2783.82/701.04	c ................................ | *****.....          134848000 | 65212        15869133         2784.0     48436.8    305068398
2879.93/725.22	c ................................ | ******....          139166000 | 66336        16368542         2880.0     48321.4    307750435
2975.84/749.34	c ................................ | ******....          143642000 | 69086        16876950         2976.0     48266.6    317882367
3071.87/773.43	c ................................ | ******....          147677000 | 72164        17349655         3072.0     48071.9    319111715
3167.64/797.49	c ................................ | ******....          151877000 | 74607        17821109         3168.0     47941.0    325025175
3263.83/821.54	c ................................ | ******....          156578000 | 78868        18337810         3264.0     47971.1    340492505
3359.65/845.59	c ................................ | *******...          160739000 | 81410        18809443         3360.0     47838.8    347692951
3455.86/869.64	c ................................ | *******...          164912000 | 83977        19307651         3456.0     47717.5    355205568
3551.79/893.75	c ................................ | *******...          169208000 | 86351        19813717         3552.0     47637.3    360157227
3647.92/917.92	c ................................ | *******...          173473000 | 89680        20307149         3648.0     47552.9    370389341
3744.00/942.10	c ................................ | *******...          177631000 | 91570        20787030         3744.0     47444.2    376273427
3839.75/966.26	c ................................ | ********..          181452000 | 93781        21244339         3840.0     47253.1    377729722
3935.97/990.40	c ................................ | ********..          185480000 | 97098        21700984         3936.0     47124.0    384978804
4031.76/1014.56	c ................................ | ********..          189443000 | 99732        22159546         4032.0     46984.8    390356427
4127.92/1038.72	c ................................ | ********..          193505000 | 101264       22649755         4128.0     46876.2    403880443
4223.72/1062.87	c ................................ | ********..          197636000 | 103339       23144469         4224.0     46788.7    413719052
4319.68/1086.98	c ................................ | *********.          201364000 | 104803       23591860         4320.0     46611.9    419857344
4415.83/1111.04	c ................................ | *********.          205534000 | 108233       24057507         4416.0     46543.0    421090391
4511.98/1135.11	c ................................ | *********.          209546000 | 113985       24498719         4512.0     46441.9    424436362
4607.76/1159.16	c ................................ | *********.          213645000 | 119005       24949871         4608.0     46363.9    427071603
4703.94/1183.21	c ................................ | *********.          217669000 | 123322       25397347         4704.0     46273.0    430798781
4799.90/1207.32	c CPU limit (4800) reached
4799.90/1207.32	c Time = 4800.00 seconds
4799.90/1207.32	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3251308-1304107377/watcher-3251308-1304107377 -o /tmp/evaluation-result-3251308-1304107377/solver-3251308-1304107377 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251308-1304107377.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.08 8.10 8.07 5/185 11582
/proc/meminfo: memFree=20303272/32951124 swapFree=67102692/67111528
[pid=11582] ppid=11580 vsize=9764 CPUtime=0 cores=0,2,4,6
/proc/11582/stat : 11582 (ttsth-5-4) R 11580 11582 11448 0 -1 4202496 488 0 0 0 0 0 0 0 25 0 1 0 522335857 9998336 431 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516834048 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/11582/statm: 2441 432 147 36 0 346 0

[startup+0.067579 s]
/proc/loadavg: 8.08 8.10 8.07 5/185 11582
/proc/meminfo: memFree=20303272/32951124 swapFree=67102692/67111528
[pid=11582] ppid=11580 vsize=21804 CPUtime=0.06 cores=0,2,4,6
/proc/11582/stat : 11582 (ttsth-5-4) R 11580 11582 11448 0 -1 4202496 3482 0 0 0 6 0 0 0 25 0 1 0 522335857 22327296 3425 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516495045 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/11582/statm: 5451 3427 148 36 0 3356 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 21804

[startup+0.100571 s]
/proc/loadavg: 8.08 8.10 8.07 5/185 11582
/proc/meminfo: memFree=20303272/32951124 swapFree=67102692/67111528
[pid=11582] ppid=11580 vsize=26568 CPUtime=0.09 cores=0,2,4,6
/proc/11582/stat : 11582 (ttsth-5-4) R 11580 11582 11448 0 -1 4202496 4597 0 0 0 9 0 0 0 25 0 1 0 522335857 27205632 4540 33554432000 4194304 4338100 140736598218752 18446744073709551615 4221392 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/11582/statm: 6642 4540 150 36 0 4547 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 26568

[startup+0.300546 s]
/proc/loadavg: 8.08 8.10 8.07 5/185 11582
/proc/meminfo: memFree=20303272/32951124 swapFree=67102692/67111528
[pid=11582] ppid=11580 vsize=39840 CPUtime=0.29 cores=0,2,4,6
/proc/11582/stat : 11582 (ttsth-5-4) R 11580 11582 11448 0 -1 4202496 10231 0 0 0 28 1 0 0 25 0 1 0 522335857 40796160 7997 33554432000 4194304 4338100 140736598218752 18446744073709551615 4245205 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/11582/statm: 9960 7998 155 36 0 7865 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 39840

[startup+0.700473 s]
/proc/loadavg: 8.08 8.10 8.07 5/185 11582
/proc/meminfo: memFree=20303272/32951124 swapFree=67102692/67111528
[pid=11582] ppid=11580 vsize=83592 CPUtime=0.69 cores=0,2,4,6
/proc/11582/stat : 11582 (ttsth-5-4) R 11580 11582 11448 0 -1 4202496 21426 0 0 0 67 2 0 0 25 0 1 0 522335857 85598208 18902 33554432000 4194304 4338100 140736598218752 18446744073709551615 4278223 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/11582/statm: 20898 18902 164 36 0 18803 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 83592

[startup+1.50028 s]
/proc/loadavg: 8.08 8.10 8.07 9/190 11587
/proc/meminfo: memFree=20191752/32951124 swapFree=67102692/67111528
[pid=11582] ppid=11580 vsize=249856 CPUtime=3.54 cores=0,2,4,6
/proc/11582/stat : 11582 (ttsth-5-4) S 11580 11582 11448 0 -1 4202496 35310 0 0 0 349 5 0 0 25 0 5 0 522335857 255852544 32786 33554432000 4194304 4338100 140736598218752 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11582/statm: 62464 32786 176 36 0 60369 0
[pid=11582/tid=11584] ppid=11580 vsize=249856 CPUtime=0.68 cores=0,2,4,6
/proc/11582/task/11584/stat : 11584 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 3984 0 0 0 68 0 0 0 18 0 5 0 522335938 255852544 32786 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516229557 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=11582/tid=11585] ppid=11580 vsize=249856 CPUtime=0.68 cores=0,2,4,6
/proc/11582/task/11585/stat : 11585 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 1814 0 0 0 68 0 0 0 18 0 5 0 522335938 255852544 32786 33554432000 4194304 4338100 140736598218752 18446744073709551615 4255920 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=11582/tid=11586] ppid=11580 vsize=249856 CPUtime=0.68 cores=0,2,4,6
/proc/11582/task/11586/stat : 11586 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 3670 0 0 0 68 0 0 0 18 0 5 0 522335938 255852544 32788 33554432000 4194304 4338100 140736598218752 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11582/tid=11587] ppid=11580 vsize=249856 CPUtime=0.65 cores=0,2,4,6
/proc/11582/task/11587/stat : 11587 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 1426 0 0 0 65 0 0 0 18 0 5 0 522335938 255852544 32789 33554432000 4194304 4338100 140736598218752 18446744073709551615 4283732 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3.54
Current children cumulated vsize (KiB) 249856

[startup+3.10099 s]
/proc/loadavg: 8.08 8.10 8.07 9/190 11587
/proc/meminfo: memFree=20110780/32951124 swapFree=67102692/67111528
[pid=11582] ppid=11580 vsize=334228 CPUtime=9.89 cores=0,2,4,6
/proc/11582/stat : 11582 (ttsth-5-4) S 11580 11582 11448 0 -1 4202496 56740 0 0 0 977 12 0 0 25 0 5 0 522335857 342249472 54216 33554432000 4194304 4338100 140736598218752 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11582/statm: 83557 54216 176 36 0 81462 0
[pid=11582/tid=11584] ppid=11580 vsize=334228 CPUtime=2.28 cores=0,2,4,6
/proc/11582/task/11584/stat : 11584 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 12563 0 0 0 227 1 0 0 22 0 5 0 522335938 342249472 54216 33554432000 4194304 4338100 140736598218752 18446744073709551615 246528648347 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=11582/tid=11585] ppid=11580 vsize=334228 CPUtime=2.28 cores=0,2,4,6
/proc/11582/task/11585/stat : 11585 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 5012 0 0 0 225 3 0 0 20 0 5 0 522335938 342249472 54216 33554432000 4194304 4338100 140736598218752 18446744073709551615 4240496 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=11582/tid=11586] ppid=11580 vsize=334228 CPUtime=2.28 cores=0,2,4,6
/proc/11582/task/11586/stat : 11586 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 10724 0 0 0 227 1 0 0 21 0 5 0 522335938 342249472 54216 33554432000 4194304 4338100 140736598218752 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11582/tid=11587] ppid=11580 vsize=334228 CPUtime=2.21 cores=0,2,4,6
/proc/11582/task/11587/stat : 11587 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 4024 0 0 0 220 1 0 0 18 0 5 0 522335938 342249472 54216 33554432000 4194304 4338100 140736598218752 18446744073709551615 4245205 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 9.89
Current children cumulated vsize (KiB) 334228

[startup+6.3004 s]
/proc/loadavg: 8.08 8.10 8.07 9/190 11587
/proc/meminfo: memFree=19947964/32951124 swapFree=67102692/67111528
[pid=11582] ppid=11580 vsize=484596 CPUtime=22.59 cores=0,2,4,6
/proc/11582/stat : 11582 (ttsth-5-4) S 11580 11582 11448 0 -1 4202496 94118 0 0 0 2235 24 0 0 25 0 5 0 522335857 496226304 91594 33554432000 4194304 4338100 140736598218752 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11582/statm: 121149 91594 176 36 0 119054 0
[pid=11582/tid=11584] ppid=11580 vsize=484596 CPUtime=5.48 cores=0,2,4,6
/proc/11582/task/11584/stat : 11584 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 27961 0 0 0 544 4 0 0 25 0 5 0 522335938 496226304 91594 33554432000 4194304 4338100 140736598218752 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=11582/tid=11585] ppid=11580 vsize=484596 CPUtime=5.48 cores=0,2,4,6
/proc/11582/task/11585/stat : 11585 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 10209 0 0 0 540 8 0 0 25 0 5 0 522335938 496226304 91594 33554432000 4194304 4338100 140736598218752 18446744073709551615 4254041 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=11582/tid=11586] ppid=11580 vsize=484596 CPUtime=5.48 cores=0,2,4,6
/proc/11582/task/11586/stat : 11586 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 23510 0 0 0 544 4 0 0 25 0 5 0 522335938 496226304 91594 33554432000 4194304 4338100 140736598218752 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11582/tid=11587] ppid=11580 vsize=484596 CPUtime=5.31 cores=0,2,4,6
/proc/11582/task/11587/stat : 11587 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 8021 0 0 0 527 4 0 0 18 0 5 0 522335938 496226304 91596 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516229588 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 22.59
Current children cumulated vsize (KiB) 484596

[startup+12.7002 s]
/proc/loadavg: 8.06 8.10 8.07 9/190 11587
/proc/meminfo: memFree=19660896/32951124 swapFree=67102692/67111528
[pid=11582] ppid=11580 vsize=754188 CPUtime=48.05 cores=0,2,4,6
/proc/11582/stat : 11582 (ttsth-5-4) S 11580 11582 11448 0 -1 4202496 161153 0 0 0 4757 48 0 0 25 0 5 0 522335857 772288512 157810 33554432000 4194304 4338100 140736598218752 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11582/statm: 188547 157811 176 36 0 186452 0
[pid=11582/tid=11584] ppid=11580 vsize=754188 CPUtime=11.88 cores=0,2,4,6
/proc/11582/task/11584/stat : 11584 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 56562 0 0 0 1179 9 0 0 25 0 5 0 522335938 772288512 157811 33554432000 4194304 4338100 140736598218752 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=11582/tid=11585] ppid=11580 vsize=754188 CPUtime=11.75 cores=0,2,4,6
/proc/11582/task/11585/stat : 11585 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 18690 0 0 0 1163 12 0 0 18 0 5 0 522335938 772288512 157811 33554432000 4194304 4338100 140736598218752 18446744073709551615 4263768 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=11582/tid=11586] ppid=11580 vsize=754188 CPUtime=11.88 cores=0,2,4,6
/proc/11582/task/11586/stat : 11586 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 44814 0 0 0 1181 7 0 0 25 0 5 0 522335938 772288512 157812 33554432000 4194304 4338100 140736598218752 18446744073709551615 4254112 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11582/tid=11587] ppid=11580 vsize=754188 CPUtime=11.71 cores=0,2,4,6
/proc/11582/task/11587/stat : 11587 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 16671 0 0 0 1155 16 0 0 25 0 5 0 522335938 772288512 157812 33554432000 4194304 4338100 140736598218752 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 48.05
Current children cumulated vsize (KiB) 754188

[startup+25.5009 s]
/proc/loadavg: 8.13 8.11 8.07 9/190 11588
/proc/meminfo: memFree=19055024/32951124 swapFree=67102692/67111528
[pid=11582] ppid=11580 vsize=1182448 CPUtime=98.96 cores=0,2,4,6
/proc/11582/stat : 11582 (ttsth-5-4) S 11580 11582 11448 0 -1 4202496 266711 0 0 0 9806 90 0 0 25 0 5 0 522335857 1210826752 263368 33554432000 4194304 4338100 140736598218752 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11582/statm: 295612 263368 176 36 0 293517 0
[pid=11582/tid=11584] ppid=11580 vsize=1182448 CPUtime=24.68 cores=0,2,4,6
/proc/11582/task/11584/stat : 11584 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 108059 0 0 0 2450 18 0 0 25 0 5 0 522335938 1210826752 263368 33554432000 4194304 4338100 140736598218752 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=11582/tid=11585] ppid=11580 vsize=1182448 CPUtime=24.39 cores=0,2,4,6
/proc/11582/task/11585/stat : 11585 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 26677 0 0 0 2414 25 0 0 25 0 5 0 522335938 1210826752 263368 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516229566 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=11582/tid=11586] ppid=11580 vsize=1182448 CPUtime=24.67 cores=0,2,4,6
/proc/11582/task/11586/stat : 11586 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 80744 0 0 0 2455 12 0 0 25 0 5 0 522335938 1210826752 263368 33554432000 4194304 4338100 140736598218752 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11582/tid=11587] ppid=11580 vsize=1182448 CPUtime=24.37 cores=0,2,4,6
/proc/11582/task/11587/stat : 11587 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 26814 0 0 0 2408 29 0 0 18 0 5 0 522335938 1210826752 263368 33554432000 4194304 4338100 140736598218752 18446744073709551615 4263571 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 98.96
Current children cumulated vsize (KiB) 1182448

[startup+51.1012 s]
/proc/loadavg: 8.22 8.13 8.08 10/190 11588
/proc/meminfo: memFree=18030600/32951124 swapFree=67102692/67111528
[pid=11582] ppid=11580 vsize=1887400 CPUtime=200.7 cores=0,2,4,6
/proc/11582/stat : 11582 (ttsth-5-4) S 11580 11582 11448 0 -1 4202496 439331 0 0 0 19889 181 0 0 25 0 5 0 522335857 1932697600 435988 33554432000 4194304 4338100 140736598218752 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11582/statm: 471850 435988 176 36 0 469755 0
[pid=11582/tid=11584] ppid=11580 vsize=1887400 CPUtime=50.27 cores=0,2,4,6
/proc/11582/task/11584/stat : 11584 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 195598 0 0 0 4988 39 0 0 25 0 5 0 522335938 1932697600 435988 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516530731 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=11582/tid=11585] ppid=11580 vsize=1887400 CPUtime=49.43 cores=0,2,4,6
/proc/11582/task/11585/stat : 11585 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 37678 0 0 0 4898 45 0 0 18 0 5 0 522335938 1932697600 435988 33554432000 4194304 4338100 140736598218752 18446744073709551615 4245205 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=11582/tid=11586] ppid=11580 vsize=1887400 CPUtime=50.28 cores=0,2,4,6
/proc/11582/task/11586/stat : 11586 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 140855 0 0 0 5005 23 0 0 25 0 5 0 522335938 1932697600 435989 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516230664 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11582/tid=11587] ppid=11580 vsize=1887400 CPUtime=49.89 cores=0,2,4,6
/proc/11582/task/11587/stat : 11587 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 40784 0 0 0 4920 69 0 0 25 0 5 0 522335938 1932697600 435989 33554432000 4194304 4338100 140736598218752 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 200.7
Current children cumulated vsize (KiB) 1887400

[startup+102.307 s]
/proc/loadavg: 8.32 8.17 8.10 9/190 11593
/proc/meminfo: memFree=17062368/32951124 swapFree=67102692/67111528
[pid=11582] ppid=11580 vsize=2708792 CPUtime=404.28 cores=0,2,4,6
/proc/11582/stat : 11582 (ttsth-5-4) S 11580 11582 11448 0 -1 4202496 627629 0 0 0 40109 319 0 0 25 0 5 0 522335857 2773803008 624286 33554432000 4194304 4338100 140736598218752 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11582/statm: 677198 624286 176 36 0 675103 0
[pid=11582/tid=11584] ppid=11580 vsize=2708792 CPUtime=101.43 cores=0,2,4,6
/proc/11582/task/11584/stat : 11584 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 316566 0 0 0 10076 67 0 0 25 0 5 0 522335938 2773803008 624287 33554432000 4194304 4338100 140736598218752 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=11582/tid=11585] ppid=11580 vsize=2708792 CPUtime=99.47 cores=0,2,4,6
/proc/11582/task/11585/stat : 11585 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 46005 0 0 0 9872 75 0 0 18 0 5 0 522335938 2773803008 624288 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516229545 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=11582/tid=11586] ppid=11580 vsize=2708792 CPUtime=101.48 cores=0,2,4,6
/proc/11582/task/11586/stat : 11586 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 193475 0 0 0 10117 31 0 0 25 0 5 0 522335938 2773803008 624288 33554432000 4194304 4338100 140736598218752 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11582/tid=11587] ppid=11580 vsize=2708792 CPUtime=101.07 cores=0,2,4,6
/proc/11582/task/11587/stat : 11587 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 47168 0 0 0 9965 142 0 0 25 0 5 0 522335938 2773803008 624288 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516229146 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 404.28
Current children cumulated vsize (KiB) 2708792

[startup+162.301 s]
/proc/loadavg: 8.21 8.17 8.10 9/190 11594
/proc/meminfo: memFree=15809348/32951124 swapFree=67102692/67111528
[pid=11582] ppid=11580 vsize=3903688 CPUtime=642.81 cores=0,2,4,6
/proc/11582/stat : 11582 (ttsth-5-4) S 11580 11582 11448 0 -1 4202496 926178 0 0 0 63790 491 0 0 25 0 5 0 522335857 3997376512 922835 33554432000 4194304 4338100 140736598218752 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11582/statm: 975922 922835 176 36 0 973827 0
[pid=11582/tid=11584] ppid=11580 vsize=3903688 CPUtime=161.28 cores=0,2,4,6
/proc/11582/task/11584/stat : 11584 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 508459 0 0 0 16021 107 0 0 25 0 5 0 522335938 3997376512 922835 33554432000 4194304 4338100 140736598218752 18446744073709551615 4235847 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=11582/tid=11585] ppid=11580 vsize=3903688 CPUtime=158.18 cores=0,2,4,6
/proc/11582/task/11585/stat : 11585 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 46612 0 0 0 15716 102 0 0 18 0 5 0 522335938 3997376512 922835 33554432000 4194304 4338100 140736598218752 18446744073709551615 4235661 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=11582/tid=11586] ppid=11580 vsize=3903688 CPUtime=161.46 cores=0,2,4,6
/proc/11582/task/11586/stat : 11586 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 292613 0 0 0 16099 47 0 0 25 0 5 0 522335938 3997376512 922835 33554432000 4194304 4338100 140736598218752 18446744073709551615 4235770 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11582/tid=11587] ppid=11580 vsize=3903688 CPUtime=161.04 cores=0,2,4,6
/proc/11582/task/11587/stat : 11587 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 54077 0 0 0 15874 230 0 0 25 0 5 0 522335938 3997376512 922835 33554432000 4194304 4338100 140736598218752 18446744073709551615 4235651 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 642.81
Current children cumulated vsize (KiB) 3903688


################
# More data... #
################

/proc/11582/task/11587/stat : 11587 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 101786 0 0 0 68566 1003 0 0 25 0 5 0 522335938 10929020928 2572386 33554432000 4194304 4338100 140736598218752 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2788.99
Current children cumulated vsize (KiB) 10672872

[startup+762.306 s]
/proc/loadavg: 8.12 8.15 8.10 10/190 11642
/proc/meminfo: memFree=15188276/32951124 swapFree=67102692/67111528
[pid=11582] ppid=11580 vsize=11413448 CPUtime=3027.61 cores=0,2,4,6
/proc/11582/stat : 11582 (ttsth-5-4) S 11580 11582 11448 0 -1 4202496 2708740 0 0 0 300167 2594 0 0 25 0 5 0 522335857 11687370752 2705397 33554432000 4194304 4338100 140736598218752 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11582/statm: 2853362 2705397 176 36 0 2851267 0
[pid=11582/tid=11584] ppid=11580 vsize=11413448 CPUtime=759.46 cores=0,2,4,6
/proc/11582/task/11584/stat : 11584 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 1621727 0 0 0 75558 388 0 0 25 0 5 0 522335938 11687370752 2705397 33554432000 4194304 4338100 140736598218752 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=11582/tid=11585] ppid=11580 vsize=11413448 CPUtime=751.76 cores=0,2,4,6
/proc/11582/task/11585/stat : 11585 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 121528 0 0 0 74264 912 0 0 25 0 5 0 522335938 11687370752 2705397 33554432000 4194304 4338100 140736598218752 18446744073709551615 4229521 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=11582/tid=11586] ppid=11580 vsize=11413448 CPUtime=760.03 cores=0,2,4,6
/proc/11582/task/11586/stat : 11586 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 823566 0 0 0 75869 134 0 0 25 0 5 0 522335938 11687370752 2705397 33554432000 4194304 4338100 140736598218752 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=11582/tid=11587] ppid=11580 vsize=11413448 CPUtime=755.52 cores=0,2,4,6
/proc/11582/task/11587/stat : 11587 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 117502 0 0 0 74397 1155 0 0 25 0 5 0 522335938 11687370752 2705397 33554432000 4194304 4338100 140736598218752 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3027.61
Current children cumulated vsize (KiB) 11413448

[startup+822.3 s]
/proc/loadavg: 8.19 8.17 8.10 9/190 11644
/proc/meminfo: memFree=13976264/32951124 swapFree=67102692/67111528
[pid=11582] ppid=11580 vsize=12126632 CPUtime=3267.02 cores=0,2,4,6
/proc/11582/stat : 11582 (ttsth-5-4) S 11580 11582 11448 0 -1 4202496 2896633 0 0 0 323573 3129 0 0 25 0 5 0 522335857 12417671168 2893290 33554432000 4194304 4338100 140736598218752 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11582/statm: 3031658 2893290 176 36 0 3029563 0
[pid=11582/tid=11584] ppid=11580 vsize=12126632 CPUtime=819.33 cores=0,2,4,6
/proc/11582/task/11584/stat : 11584 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 1747372 0 0 0 81513 420 0 0 25 0 5 0 522335938 12417671168 2893290 33554432000 4194304 4338100 140736598218752 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11582/tid=11585] ppid=11580 vsize=12126632 CPUtime=811.61 cores=0,2,4,6
/proc/11582/task/11585/stat : 11585 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 126408 0 0 0 80003 1158 0 0 25 0 5 0 522335938 12417671168 2893290 33554432000 4194304 4338100 140736598218752 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=11582/tid=11586] ppid=11580 vsize=12126632 CPUtime=819.98 cores=0,2,4,6
/proc/11582/task/11586/stat : 11586 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 876741 0 0 0 81854 144 0 0 25 0 5 0 522335938 12417671168 2893291 33554432000 4194304 4338100 140736598218752 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=11582/tid=11587] ppid=11580 vsize=12126632 CPUtime=815.26 cores=0,2,4,6
/proc/11582/task/11587/stat : 11587 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 121695 0 0 0 80123 1403 0 0 25 0 5 0 522335938 12417671168 2893291 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516229562 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3267.02
Current children cumulated vsize (KiB) 12126632

[startup+882.3 s]
/proc/loadavg: 8.17 8.16 8.10 9/190 11646
/proc/meminfo: memFree=13136624/32951124 swapFree=67102692/67111528
[pid=11582] ppid=11580 vsize=12577064 CPUtime=3506.48 cores=0,2,4,6
/proc/11582/stat : 11582 (ttsth-5-4) S 11580 11582 11448 0 -1 4202496 3021731 0 0 0 347036 3612 0 0 25 0 5 0 522335857 12878913536 3018388 33554432000 4194304 4338100 140736598218752 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11582/statm: 3144266 3018388 176 36 0 3142171 0
[pid=11582/tid=11584] ppid=11580 vsize=12577064 CPUtime=879.26 cores=0,2,4,6
/proc/11582/task/11584/stat : 11584 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 1827730 0 0 0 87483 443 0 0 25 0 5 0 522335938 12878913536 3018388 33554432000 4194304 4338100 140736598218752 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=11582/tid=11585] ppid=11580 vsize=12577064 CPUtime=871.5 cores=0,2,4,6
/proc/11582/task/11585/stat : 11585 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 132724 0 0 0 85774 1376 0 0 18 0 5 0 522335938 12878913536 3018388 33554432000 4194304 4338100 140736598218752 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=11582/tid=11586] ppid=11580 vsize=12577064 CPUtime=879.96 cores=0,2,4,6
/proc/11582/task/11586/stat : 11586 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 909238 0 0 0 87847 149 0 0 25 0 5 0 522335938 12878913536 3018388 33554432000 4194304 4338100 140736598218752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11582/tid=11587] ppid=11580 vsize=12577064 CPUtime=874.93 cores=0,2,4,6
/proc/11582/task/11587/stat : 11587 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 127622 0 0 0 85853 1640 0 0 25 0 5 0 522335938 12878913536 3018388 33554432000 4194304 4338100 140736598218752 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3506.48
Current children cumulated vsize (KiB) 12577064

[startup+942.3 s]
/proc/loadavg: 8.27 8.19 8.11 9/190 11647
/proc/meminfo: memFree=11967872/32951124 swapFree=67102692/67111528
[pid=11582] ppid=11580 vsize=13567880 CPUtime=3744.8 cores=0,2,4,6
/proc/11582/stat : 11582 (ttsth-5-4) S 11580 11582 11448 0 -1 4202496 3190435 0 0 0 370713 3767 0 0 25 0 5 0 522335857 13893509120 3187092 33554432000 4194304 4338100 140736598218752 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11582/statm: 3391970 3187093 176 36 0 3389875 0
[pid=11582/tid=11584] ppid=11580 vsize=13567880 CPUtime=938.97 cores=0,2,4,6
/proc/11582/task/11584/stat : 11584 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 1943678 0 0 0 93424 473 0 0 25 0 5 0 522335938 13893509120 3187093 33554432000 4194304 4338100 140736598218752 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11582/tid=11585] ppid=11580 vsize=13567880 CPUtime=930.31 cores=0,2,4,6
/proc/11582/task/11585/stat : 11585 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 133319 0 0 0 91633 1398 0 0 18 0 5 0 522335938 13893509120 3187093 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516230003 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=11582/tid=11586] ppid=11580 vsize=13567880 CPUtime=939.77 cores=0,2,4,6
/proc/11582/task/11586/stat : 11586 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 958439 0 0 0 93820 157 0 0 20 0 5 0 522335938 13893509120 3187093 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516229229 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=11582/tid=11587] ppid=11580 vsize=13567880 CPUtime=934.89 cores=0,2,4,6
/proc/11582/task/11587/stat : 11587 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 130583 0 0 0 91755 1734 0 0 25 0 5 0 522335938 13893509120 3187093 33554432000 4194304 4338100 140736598218752 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3744.8
Current children cumulated vsize (KiB) 13567880

[startup+1002.3 s]
/proc/loadavg: 8.17 8.17 8.11 9/190 11649
/proc/meminfo: memFree=11092772/32951124 swapFree=67102692/67111528
[pid=11582] ppid=11580 vsize=13936984 CPUtime=3983.27 cores=0,2,4,6
/proc/11582/stat : 11582 (ttsth-5-4) S 11580 11582 11448 0 -1 4202496 3305888 0 0 0 394412 3915 0 0 25 0 5 0 522335857 14271471616 3302545 33554432000 4194304 4338100 140736598218752 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11582/statm: 3484246 3302545 176 36 0 3482151 0
[pid=11582/tid=11584] ppid=11580 vsize=13936984 CPUtime=998.93 cores=0,2,4,6
/proc/11582/task/11584/stat : 11584 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 2013826 0 0 0 99399 494 0 0 25 0 5 0 522335938 14271471616 3302545 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516229262 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11582/tid=11585] ppid=11580 vsize=13936984 CPUtime=989.25 cores=0,2,4,6
/proc/11582/task/11585/stat : 11585 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 144452 0 0 0 97499 1426 0 0 18 0 5 0 522335938 14271471616 3302545 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516530633 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=11582/tid=11586] ppid=11580 vsize=13936984 CPUtime=999.4 cores=0,2,4,6
/proc/11582/task/11586/stat : 11586 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 980912 0 0 0 99780 160 0 0 25 0 5 0 522335938 14271471616 3302545 33554432000 4194304 4338100 140736598218752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=11582/tid=11587] ppid=11580 vsize=13936984 CPUtime=994.84 cores=0,2,4,6
/proc/11582/task/11587/stat : 11587 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 142281 0 0 0 97655 1829 0 0 25 0 5 0 522335938 14271471616 3302545 33554432000 4194304 4338100 140736598218752 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3983.27
Current children cumulated vsize (KiB) 13936984

[startup+1062.3 s]
/proc/loadavg: 8.21 8.17 8.10 9/190 11650
/proc/meminfo: memFree=9923788/32951124 swapFree=67102692/67111528
[pid=11582] ppid=11580 vsize=14731496 CPUtime=4221.73 cores=0,2,4,6
/proc/11582/stat : 11582 (ttsth-5-4) S 11580 11582 11448 0 -1 4202496 3508471 0 0 0 418094 4079 0 0 25 0 5 0 522335857 15085051904 3505128 33554432000 4194304 4338100 140736598218752 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11582/statm: 3682874 3505128 176 36 0 3680779 0
[pid=11582/tid=11584] ppid=11580 vsize=14731496 CPUtime=1058.78 cores=0,2,4,6
/proc/11582/task/11584/stat : 11584 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 2144067 0 0 0 105352 526 0 0 25 0 5 0 522335938 15085051904 3505128 33554432000 4194304 4338100 140736598218752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=11582/tid=11585] ppid=11580 vsize=14731496 CPUtime=1048.19 cores=0,2,4,6
/proc/11582/task/11585/stat : 11585 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 145771 0 0 0 103366 1453 0 0 18 0 5 0 522335938 15085051904 3505128 33554432000 4194304 4338100 140736598218752 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=11582/tid=11586] ppid=11580 vsize=14731496 CPUtime=1059.12 cores=0,2,4,6
/proc/11582/task/11586/stat : 11586 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 1048467 0 0 0 105741 171 0 0 25 0 5 0 522335938 15085051904 3505128 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516229870 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11582/tid=11587] ppid=11580 vsize=14731496 CPUtime=1054.81 cores=0,2,4,6
/proc/11582/task/11587/stat : 11587 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 145749 0 0 0 103556 1925 0 0 25 0 5 0 522335938 15085051904 3505128 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516492017 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4221.73
Current children cumulated vsize (KiB) 14731496

[startup+1122.3 s]
/proc/loadavg: 8.11 8.15 8.10 9/190 11652
/proc/meminfo: memFree=9038712/32951124 swapFree=67102692/67111528
[pid=11582] ppid=11580 vsize=14997376 CPUtime=4460.89 cores=0,2,4,6
/proc/11582/stat : 11582 (ttsth-5-4) S 11580 11582 11448 0 -1 4202496 3603828 0 0 0 441597 4492 0 0 25 0 5 0 522335857 15357313024 3600485 33554432000 4194304 4338100 140736598218752 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11582/statm: 3749344 3600485 176 36 0 3747249 0
[pid=11582/tid=11584] ppid=11580 vsize=14997376 CPUtime=1118.41 cores=0,2,4,6
/proc/11582/task/11584/stat : 11584 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 2190956 0 0 0 111296 545 0 0 25 0 5 0 522335938 15357313024 3600485 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516229136 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11582/tid=11585] ppid=11580 vsize=14997376 CPUtime=1107.93 cores=0,2,4,6
/proc/11582/task/11585/stat : 11585 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 159515 0 0 0 109149 1644 0 0 25 0 5 0 522335938 15357313024 3600485 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516229154 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=11582/tid=11586] ppid=11580 vsize=14997376 CPUtime=1119.08 cores=0,2,4,6
/proc/11582/task/11586/stat : 11586 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 1068194 0 0 0 111733 175 0 0 25 0 5 0 522335938 15357313024 3600485 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516530656 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=11582/tid=11587] ppid=11580 vsize=14997376 CPUtime=1114.64 cores=0,2,4,6
/proc/11582/task/11587/stat : 11587 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 160746 0 0 0 109341 2123 0 0 25 0 5 0 522335938 15357313024 3600485 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516229557 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4460.89
Current children cumulated vsize (KiB) 14997376

[startup+1182.3 s]
/proc/loadavg: 8.22 8.17 8.10 9/190 11654
/proc/meminfo: memFree=8232868/32951124 swapFree=67102692/67111528
[pid=11582] ppid=11580 vsize=15272640 CPUtime=4700.35 cores=0,2,4,6
/proc/11582/stat : 11582 (ttsth-5-4) S 11580 11582 11448 0 -1 4202496 3690783 0 0 0 465082 4953 0 0 25 0 5 0 522335857 15639183360 3687440 33554432000 4194304 4338100 140736598218752 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11582/statm: 3818160 3687440 176 36 0 3816065 0
[pid=11582/tid=11584] ppid=11580 vsize=15272640 CPUtime=1178.33 cores=0,2,4,6
/proc/11582/task/11584/stat : 11584 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 2259929 0 0 0 117263 570 0 0 25 0 5 0 522335938 15639183360 3687440 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516229154 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=11582/tid=11585] ppid=11580 vsize=15272640 CPUtime=1167.65 cores=0,2,4,6
/proc/11582/task/11585/stat : 11585 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 167876 0 0 0 114904 1861 0 0 25 0 5 0 522335938 15639183360 3687440 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516229257 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=11582/tid=11586] ppid=11580 vsize=15272640 CPUtime=1179.05 cores=0,2,4,6
/proc/11582/task/11586/stat : 11586 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 1069286 0 0 0 117729 176 0 0 25 0 5 0 522335938 15639183360 3687440 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516229167 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=11582/tid=11587] ppid=11580 vsize=15272640 CPUtime=1174.48 cores=0,2,4,6
/proc/11582/task/11587/stat : 11587 (ttsth-5-4) R 11580 11582 11448 0 -1 4202560 169275 0 0 0 115106 2342 0 0 25 0 5 0 522335938 15639183360 3687440 33554432000 4194304 4338100 140736598218752 18446744073709551615 246516530617 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4700.35
Current children cumulated vsize (KiB) 15272640



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1207.4 s]
/proc/loadavg: 8.15 8.15 8.10 6/187 11654
/proc/meminfo: memFree=9333012/32951124 swapFree=67102692/67111528
[pid=11582] ppid=11580 vsize=0 CPUtime=4800.08 cores=0,2,4,6
/proc/11582/stat : 11582 (ttsth-5-4) Z 11580 11582 11448 0 -1 4203548 3750994 0 0 0 474888 5120 0 0 18 0 2 0 522335857 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11582/statm: 0 0 0 0 0 0 0
[pid=11582/tid=11585] ppid=11580 vsize=0 CPUtime=1192.65 cores=0,2,4,6
/proc/11582/task/11585/stat : 11585 (ttsth-5-4) R 11580 11582 11448 0 -1 4203588 167876 0 0 0 117319 1946 0 0 25 0 2 0 522335938 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.08
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1207.81 s]
/proc/loadavg: 8.15 8.15 8.10 6/188 11655
/proc/meminfo: memFree=17196440/32951124 swapFree=67102692/67111528
[pid=11582] ppid=11580 vsize=0 CPUtime=4800.48 cores=0,2,4,6
/proc/11582/stat : 11582 (ttsth-5-4) Z 11580 11582 11448 0 -1 4203548 3750994 0 0 0 474888 5160 0 0 18 0 2 0 522335857 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/11582/statm: 0 0 0 0 0 0 0
[pid=11582/tid=11585] ppid=11580 vsize=0 CPUtime=1193.05 cores=0,2,4,6
/proc/11582/task/11585/stat : 11585 (ttsth-5-4) R 11580 11582 11448 0 -1 4203588 167876 0 0 0 117319 1986 0 0 25 0 2 0 522335938 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.48
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1207.89
CPU time (s): 4800.57
CPU user time (s): 4748.89
CPU system time (s): 51.6851
CPU usage (%): 397.434
Max. virtual memory (cumulated for all children) (KiB): 15513496

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4748.89
system time used= 51.6851
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 3750994
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 27479
involuntary context switches= 76026

runsolver used 2.05069 second user time and 4.55031 second system time

The end

Launcher Data 

Begin job on node134 at 2011-04-29 22:02:57
IDJOB=3251308
IDBENCH=88218
IDSOLVER=1654
FILE ID=node134/3251308-1304107377
RUNJOBID= node134-1304106566-11466
PBS_JOBID= 13170663
Free space on /tmp= 71580 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m37-q505.75-n35-p30-s59841049.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3251308-1304107377/watcher-3251308-1304107377 -o /tmp/evaluation-result-3251308-1304107377/solver-3251308-1304107377 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251308-1304107377.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= c86ce640ddb0138bf41ccc37bc6fcf28
RANDOM SEED=1405155707

node134.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 7186.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      20303552 kB
Buffers:        989424 kB
Cached:        1360736 kB
SwapCached:       4400 kB
Active:        9931176 kB
Inactive:      2299436 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      20303552 kB
SwapTotal:    67111528 kB
SwapFree:     67102692 kB
Dirty:            1988 kB
Writeback:           4 kB
AnonPages:     9877792 kB
Mapped:          15240 kB
Slab:           334484 kB
PageTables:      23904 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 10118112 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71580 MiB
End job on node134 at 2011-04-29 22:23:06