Trace number 3251352

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.29 1207.71

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m62-q1504.5-n60-p30-s88600538.cnf
MD5SUM76c30bc0971bc865c2b988ddaca89385
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark5.29319
Satisfiable
(Un)Satisfiability was proved
Number of variables3651
Number of clauses926696
Sum of the clauses size1856923
Maximum clause length62
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2926636
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 560

Solver Data 

0.59/0.64	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251352-1304107444.cnf 
0.59/0.64	c Using SATTIMEOUT of 4800
0.59/0.64	c Using SATRAM of 15500
0.59/0.64	c Size of subset node = 32
0.59/0.64	c Variables = 3651
0.59/0.64	c Clauses = 926696
0.59/0.64	c Literals = 1856923
0.59/0.64	c Reduced size:
0.59/0.64	c Variables = 3651
0.59/0.64	c Clauses = 926696
0.59/0.64	c Literals = 1856923
0.59/0.64	c a2
1.10/1.14	c a1
3.09/3.13	c Independent sub-problems = 1
3.09/3.13	c Sub-problem with 3651 variables and 926696 clauses
3.09/3.13	c using permutation method 4
3.09/3.13	c count = 1
3.09/3.13	c Ordering finished 3.010 seconds
3.09/3.13	c mapped 3.120 seconds
3.09/3.13	c permuted 3.120 seconds
3.09/3.13	c about to sort
3.19/3.21	c sorted
7.08/7.15	c clauses now 926696
7.08/7.15	c init finished 3.470 seconds
7.08/7.15	c tree built 7.130 seconds
7.08/7.15	c 
7.08/7.15	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
7.08/7.15	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
7.08/7.15	c ................................ | ..........                    |                                  7.1         0.0             
7.08/7.15	c ................................ | ..........                  1 |                                  7.1         0.1             
7.08/7.15	c ................................ | ..........                  2 |                                  7.1         0.3             
7.08/7.15	c ................................ | ..........                  3 |                                  7.1         0.4             
7.08/7.15	c ................................ | ..........                  4 |                                  7.1         0.6             
7.08/7.15	c ................................ | ..........                  5 |                                  7.1         0.7             
7.08/7.15	c ................................ | ..........               1000 |              39                  7.1       140.1             
95.66/29.48	c ................................ | ..........            4203000 | 3703         285355             96.0     43781.2     11412880
191.92/53.52	c ................................ | ..........            8508000 | 7623         584651            192.0     44307.9     22425828
287.79/77.55	c ................................ | ..........           12853000 | 10863        878828            288.0     44626.9     30260171
383.65/101.59	c ................................ | ..........           17090000 | 14298        1166649           384.0     44502.9     37832293
479.91/125.62	c ................................ | *.........           21277000 | 18159        1446414           480.0     44327.1     42973628
575.79/149.66	c ................................ | *.........           25389000 | 22311        1725010           576.0     44077.4     48259192
671.64/173.69	c ................................ | *.........           29483000 | 26007        2000684           672.0     43873.5     57458103
767.87/197.73	c ................................ | *.........           33402000 | 29345        2273113           768.0     43491.6     64539880
863.73/221.77	c ................................ | *.........           37367000 | 34259        2540712           864.0     43247.8     64544759
959.99/245.80	c ................................ | **........           41286000 | 38648        2802489           960.0     43005.4     68444202
1055.84/269.84	c ................................ | **........           45257000 | 43459        3069379          1056.0     42857.0     76468009
1151.69/293.87	c ................................ | **........           48907000 | 46567        3320796          1152.0     42453.6     79036956
1247.97/317.91	c ................................ | **........           52720000 | 49658        3582427          1248.0     42243.6     80130672
1343.81/341.95	c ................................ | **........           56734000 | 53111        3847940          1344.0     42212.5     85936440
1439.67/365.98	c ................................ | ***.......           60556000 | 56387        4102237          1440.0     42052.2     88637938
1535.93/390.01	c ................................ | ***.......           64468000 | 59834        4367817          1536.0     41971.4     96263549
1631.78/414.05	c ................................ | ***.......           68296000 | 63483        4626587          1632.0     41848.0    104101704
1727.61/438.09	c ................................ | ***.......           71788000 | 65761        4871656          1728.0     41544.0    104179263
1823.87/462.13	c ................................ | ***.......           75702000 | 69168        5124295          1824.0     41503.1    108035488
1919.74/486.17	c ................................ | ****......           79443000 | 72983        5374569          1920.0     41376.3    111220078
2015.96/510.21	c ................................ | ****......           83171000 | 77906        5623424          2016.0     41255.3    113676501
2111.80/534.25	c ................................ | ****......           87055000 | 82656        5874540          2112.0     41219.2    120018812
2207.64/558.29	c ................................ | ****......           90818000 | 86840        6128502          2208.0     41131.2    127069905
2303.87/582.33	c ................................ | ****......           94411000 | 90834        6377682          2304.0     40976.8    131577932
2399.70/606.38	c ................................ | *****.....           98092000 | 94512        6629816          2400.0     40871.3    136510623
2495.95/630.41	c ................................ | *****.....          102057000 | 98895        6879585          2496.0     40888.1    140058191
2591.78/654.46	c ................................ | *****.....          105976000 | 102808       7129751          2592.0     40885.3    143998039
2687.99/678.50	c ................................ | *****.....          109809000 | 107944       7374412          2688.0     40851.4    146977258
2783.83/702.54	c ................................ | *****.....          113505000 | 111774       7618080          2784.0     40770.5    149518217
2879.65/726.58	c ................................ | ******....          117440000 | 116830       7867125          2880.0     40777.8    154383562
2975.87/750.63	c ................................ | ******....          121272000 | 121977       8117181          2976.0     40749.9    162615987
3071.69/774.68	c ................................ | ******....          125124000 | 127436       8360311          3072.0     40730.3    168045089
3167.92/798.72	c ................................ | ******....          128848000 | 133021       8601953          3168.0     40671.7    168843566
3263.78/822.76	c ................................ | ******....          132400000 | 136090       8834750          3264.0     40563.6    168887180
3359.99/846.80	c ................................ | *******...          136217000 | 141997       9075705          3360.0     40540.7    172633615
3455.84/870.84	c ................................ | *******...          139737000 | 146869       9305640          3456.0     40433.2    173759263
3551.69/894.88	c ................................ | *******...          143346000 | 150851       9542723          3552.0     40356.3    175324572
3647.94/918.91	c ................................ | *******...          147191000 | 156007       9790412          3648.0     40348.4    182851032
3743.77/942.95	c ................................ | *******...          150778000 | 160641       10027276         3744.0     40271.8    186374090
3839.99/967.00	c ................................ | ********..          154135000 | 164482       10255921         3840.0     40139.2    187299341
3935.86/991.04	c ................................ | ********..          157679000 | 168611       10501297         3936.0     40060.6    189450437
4031.67/1015.08	c ................................ | ********..          161368000 | 172663       10741035         4032.0     40021.7    192678533
4127.90/1039.13	c ................................ | ********..          165056000 | 178149       10971919         4128.0     39984.1    196989502
4223.71/1063.17	c ................................ | ********..          168685000 | 183769       11205858         4224.0     39934.9    199992838
4319.95/1087.21	c ................................ | *********.          172214000 | 189020       11437349         4320.0     39864.4    202491115
4415.80/1111.25	c ................................ | *********.          175654000 | 193802       11681826         4416.0     39776.7    204829537
4512.00/1135.30	c ................................ | *********.          179189000 | 197588       11921477         4512.0     39713.9    210571556
4607.84/1159.35	c ................................ | *********.          182873000 | 203166       12163101         4608.0     39685.8    213889316
4703.66/1183.38	c ................................ | *********.          186361000 | 207204       12399305         4704.0     39617.6    216378155
4799.92/1207.41	c CPU limit (4800) reached
4799.92/1207.41	c Time = 4800.01 seconds
4799.92/1207.41	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3251352-1304107444/watcher-3251352-1304107444 -o /tmp/evaluation-result-3251352-1304107444/solver-3251352-1304107444 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251352-1304107444.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.80 8.00 7.98 6/181 17705
/proc/meminfo: memFree=22636896/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=9632 CPUtime=0 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) R 17703 17705 17262 0 -1 4202496 468 0 0 0 0 0 0 0 25 0 1 0 522343138 9863168 410 33554432000 4194304 4338100 140733571547136 18446744073709551615 251813900997 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17705/statm: 2408 411 147 36 0 313 0

[startup+0.024961 s]
/proc/loadavg: 7.80 8.00 7.98 6/181 17705
/proc/meminfo: memFree=22636896/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=13404 CPUtime=0.02 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) R 17703 17705 17262 0 -1 4202496 1435 0 0 0 2 0 0 0 25 0 1 0 522343138 13725696 1377 33554432000 4194304 4338100 140733571547136 18446744073709551615 251813900997 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17705/statm: 3351 1379 148 36 0 1256 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 13404

[startup+0.100949 s]
/proc/loadavg: 7.80 8.00 7.98 6/181 17705
/proc/meminfo: memFree=22636896/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=28828 CPUtime=0.09 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) R 17703 17705 17262 0 -1 4202496 5198 0 0 0 9 0 0 0 25 0 1 0 522343138 29519872 5140 33554432000 4194304 4338100 140733571547136 18446744073709551615 251813900997 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17705/statm: 7207 5142 148 36 0 5112 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 28828

[startup+0.300916 s]
/proc/loadavg: 7.80 8.00 7.98 6/181 17705
/proc/meminfo: memFree=22636896/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=69240 CPUtime=0.29 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) R 17703 17705 17262 0 -1 4202496 15003 0 0 0 28 1 0 0 25 0 1 0 522343138 70901760 14945 33554432000 4194304 4338100 140733571547136 18446744073709551615 251813900997 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17705/statm: 17310 14947 148 36 0 15215 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 69240

[startup+0.700853 s]
/proc/loadavg: 7.80 8.00 7.98 6/181 17705
/proc/meminfo: memFree=22636896/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=156856 CPUtime=0.69 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) R 17703 17705 17262 0 -1 4202496 36328 0 0 0 65 4 0 0 25 0 1 0 522343138 160620544 36239 33554432000 4194304 4338100 140733571547136 18446744073709551615 251813936907 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17705/statm: 39214 36244 213 36 0 37119 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 156856

[startup+1.50072 s]
/proc/loadavg: 7.80 8.00 7.98 6/182 17706
/proc/meminfo: memFree=22463800/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=162796 CPUtime=1.49 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) R 17703 17705 17262 0 -1 4202496 52729 0 0 0 143 6 0 0 25 0 1 0 522343138 166703104 36017 33554432000 4194304 4338100 140733571547136 18446744073709551615 251813900997 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17705/statm: 40699 36017 155 36 0 38604 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 162796

[startup+3.10045 s]
/proc/loadavg: 7.80 8.00 7.98 6/182 17706
/proc/meminfo: memFree=22445696/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=216128 CPUtime=3.09 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) R 17703 17705 17262 0 -1 4202496 68774 0 0 0 301 8 0 0 25 0 1 0 522343138 221315072 52062 33554432000 4194304 4338100 140733571547136 18446744073709551615 251813898482 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17705/statm: 54032 52062 161 36 0 51937 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 216128

[startup+6.30097 s]
/proc/loadavg: 7.26 7.89 7.94 6/182 17740
/proc/meminfo: memFree=30535252/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=416076 CPUtime=6.29 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) R 17703 17705 17262 0 -1 4202496 120590 0 0 0 614 15 0 0 25 0 1 0 522343138 426061824 102068 33554432000 4194304 4338100 140733571547136 18446744073709551615 4278560 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17705/statm: 104019 102072 164 36 0 101924 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 416076

[startup+12.7008 s]
/proc/loadavg: 7.32 7.89 7.94 9/186 17744
/proc/meminfo: memFree=30021428/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=758336 CPUtime=29.3 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) S 17703 17705 17262 0 -1 4202496 166936 0 0 0 2860 70 0 0 25 0 5 0 522343138 776536064 148414 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17705/statm: 189584 148415 176 36 0 187489 0
[pid=17705/tid=17741] ppid=17703 vsize=758336 CPUtime=5.54 cores=1,3,5,7
/proc/17705/task/17741/stat : 17741 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 7872 0 0 0 551 3 0 0 25 0 5 0 522343853 776536064 148415 33554432000 4194304 4338100 140733571547136 18446744073709551615 4234400 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17705/tid=17742] ppid=17703 vsize=758336 CPUtime=5.51 cores=1,3,5,7
/proc/17705/task/17742/stat : 17742 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 4334 0 0 0 528 23 0 0 25 0 5 0 522343853 776536064 148415 33554432000 4194304 4338100 140733571547136 18446744073709551615 4285802 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17705/tid=17743] ppid=17703 vsize=758336 CPUtime=5.54 cores=1,3,5,7
/proc/17705/task/17743/stat : 17743 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 10606 0 0 0 553 1 0 0 25 0 5 0 522343853 776536064 148415 33554432000 4194304 4338100 140733571547136 18446744073709551615 4229521 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17705/tid=17744] ppid=17703 vsize=758336 CPUtime=5.53 cores=1,3,5,7
/proc/17705/task/17744/stat : 17744 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 4223 0 0 0 530 23 0 0 25 0 5 0 522343853 776536064 148415 33554432000 4194304 4338100 140733571547136 18446744073709551615 4285817 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 29.3
Current children cumulated vsize (KiB) 758336

[startup+25.5007 s]
/proc/loadavg: 7.69 7.94 7.96 10/187 17745
/proc/meminfo: memFree=29392484/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=931792 CPUtime=80.07 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) S 17703 17705 17262 0 -1 4202496 217259 0 0 0 7819 188 0 0 25 0 5 0 522343138 954155008 198737 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17705/statm: 232948 198737 176 36 0 230853 0
[pid=17705/tid=17741] ppid=17703 vsize=931792 CPUtime=18.18 cores=1,3,5,7
/proc/17705/task/17741/stat : 17741 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 25975 0 0 0 1805 13 0 0 25 0 5 0 522343853 954155008 198737 33554432000 4194304 4338100 140733571547136 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17705/tid=17742] ppid=17703 vsize=931792 CPUtime=18.07 cores=1,3,5,7
/proc/17705/task/17742/stat : 17742 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 10549 0 0 0 1730 77 0 0 25 0 5 0 522343853 954155008 198737 33554432000 4194304 4338100 140733571547136 18446744073709551615 251813635111 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17705/tid=17743] ppid=17703 vsize=931792 CPUtime=18.34 cores=1,3,5,7
/proc/17705/task/17743/stat : 17743 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 28632 0 0 0 1830 4 0 0 25 0 5 0 522343853 954155008 198737 33554432000 4194304 4338100 140733571547136 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17705/tid=17744] ppid=17703 vsize=931792 CPUtime=18.31 cores=1,3,5,7
/proc/17705/task/17744/stat : 17744 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 12201 0 0 0 1756 75 0 0 25 0 5 0 522343853 954155008 198737 33554432000 4194304 4338100 140733571547136 18446744073709551615 251813635209 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 80.07
Current children cumulated vsize (KiB) 931792

[startup+51.1004 s]
/proc/loadavg: 7.85 7.96 7.96 9/186 17746
/proc/meminfo: memFree=28414072/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=1316064 CPUtime=182.33 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) S 17703 17705 17262 0 -1 4202496 315370 0 0 0 17794 439 0 0 25 0 5 0 522343138 1347649536 296848 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17705/statm: 329016 296848 176 36 0 326921 0
[pid=17705/tid=17741] ppid=17703 vsize=1316064 CPUtime=43.64 cores=1,3,5,7
/proc/17705/task/17741/stat : 17741 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 64894 0 0 0 4337 27 0 0 25 0 5 0 522343853 1347649536 296848 33554432000 4194304 4338100 140733571547136 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17705/tid=17742] ppid=17703 vsize=1316064 CPUtime=43.66 cores=1,3,5,7
/proc/17705/task/17742/stat : 17742 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 20011 0 0 0 4173 193 0 0 25 0 5 0 522343853 1347649536 296848 33554432000 4194304 4338100 140733571547136 18446744073709551615 251813635321 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17705/tid=17743] ppid=17703 vsize=1316064 CPUtime=43.94 cores=1,3,5,7
/proc/17705/task/17743/stat : 17743 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 63145 0 0 0 4383 11 0 0 25 0 5 0 522343853 1347649536 296848 33554432000 4194304 4338100 140733571547136 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17705/tid=17744] ppid=17703 vsize=1316064 CPUtime=43.91 cores=1,3,5,7
/proc/17705/task/17744/stat : 17744 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 27418 0 0 0 4203 188 0 0 25 0 5 0 522343853 1347649536 296848 33554432000 4194304 4338100 140733571547136 18446744073709551615 251813635124 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 182.33
Current children cumulated vsize (KiB) 1316064

[startup+102.307 s]
/proc/loadavg: 8.02 8.00 7.98 9/186 17747
/proc/meminfo: memFree=26899340/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=1865176 CPUtime=386.87 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) S 17703 17705 17262 0 -1 4202496 446987 0 0 0 37743 944 0 0 25 0 5 0 522343138 1909940224 428465 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17705/statm: 466294 428465 176 36 0 464199 0
[pid=17705/tid=17741] ppid=17703 vsize=1865176 CPUtime=94.57 cores=1,3,5,7
/proc/17705/task/17741/stat : 17741 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 128991 0 0 0 9401 56 0 0 25 0 5 0 522343853 1909940224 428465 33554432000 4194304 4338100 140733571547136 18446744073709551615 251813635540 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17705/tid=17742] ppid=17703 vsize=1865176 CPUtime=94.87 cores=1,3,5,7
/proc/17705/task/17742/stat : 17742 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 31960 0 0 0 9064 423 0 0 25 0 5 0 522343853 1909940224 428465 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17705/tid=17743] ppid=17703 vsize=1865176 CPUtime=95.14 cores=1,3,5,7
/proc/17705/task/17743/stat : 17743 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 99806 0 0 0 9495 19 0 0 25 0 5 0 522343853 1909940224 428465 33554432000 4194304 4338100 140733571547136 18446744073709551615 4285620 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17705/tid=17744] ppid=17703 vsize=1865176 CPUtime=95.12 cores=1,3,5,7
/proc/17705/task/17744/stat : 17744 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 46328 0 0 0 9085 427 0 0 25 0 5 0 522343853 1909940224 428466 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 386.87
Current children cumulated vsize (KiB) 1865176

[startup+162.301 s]
/proc/loadavg: 8.08 8.02 7.99 9/186 17749
/proc/meminfo: memFree=25559588/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=2409344 CPUtime=626.49 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) S 17703 17705 17262 0 -1 4202496 569859 0 0 0 61086 1563 0 0 25 0 5 0 522343138 2467168256 551337 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17705/statm: 602336 551337 176 36 0 600241 0
[pid=17705/tid=17741] ppid=17703 vsize=2409344 CPUtime=154.22 cores=1,3,5,7
/proc/17705/task/17741/stat : 17741 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 200333 0 0 0 15336 86 0 0 25 0 5 0 522343853 2467168256 551337 33554432000 4194304 4338100 140733571547136 18446744073709551615 4256352 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17705/tid=17742] ppid=17703 vsize=2409344 CPUtime=154.86 cores=1,3,5,7
/proc/17705/task/17742/stat : 17742 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 38760 0 0 0 14772 714 0 0 25 0 5 0 522343853 2467168256 551337 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17705/tid=17743] ppid=17703 vsize=2409344 CPUtime=155.13 cores=1,3,5,7
/proc/17705/task/17743/stat : 17743 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 132603 0 0 0 15487 26 0 0 25 0 5 0 522343853 2467168256 551337 33554432000 4194304 4338100 140733571547136 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17705/tid=17744] ppid=17703 vsize=2409344 CPUtime=155.1 cores=1,3,5,7
/proc/17705/task/17744/stat : 17744 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 58261 0 0 0 14793 717 0 0 25 0 5 0 522343853 2467168256 551337 33554432000 4194304 4338100 140733571547136 18446744073709551615 251813635509 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 626.49
Current children cumulated vsize (KiB) 2409344

[startup+222.301 s]
/proc/loadavg: 8.07 8.03 7.99 9/186 17751
/proc/meminfo: memFree=24382540/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=2853096 CPUtime=866.13 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) S 17703 17705 17262 0 -1 4202496 677717 0 0 0 84455 2158 0 0 25 0 5 0 522343138 2921570304 659195 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17705/statm: 713274 659196 176 36 0 711179 0
[pid=17705/tid=17741] ppid=17703 vsize=2853096 CPUtime=214.11 cores=1,3,5,7
/proc/17705/task/17741/stat : 17741 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 246211 0 0 0 21303 108 0 0 25 0 5 0 522343853 2921570304 659196 33554432000 4194304 4338100 140733571547136 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17705/tid=17742] ppid=17703 vsize=2853096 CPUtime=214.71 cores=1,3,5,7
/proc/17705/task/17742/stat : 17742 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 52028 0 0 0 20474 997 0 0 25 0 5 0 522343853 2921570304 659196 33554432000 4194304 4338100 140733571547136 18446744073709551615 251813635111 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17705/tid=17743] ppid=17703 vsize=2853096 CPUtime=215.13 cores=1,3,5,7
/proc/17705/task/17743/stat : 17743 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 167037 0 0 0 21479 34 0 0 25 0 5 0 522343853 2921570304 659196 33554432000 4194304 4338100 140733571547136 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17705/tid=17744] ppid=17703 vsize=2853096 CPUtime=214.99 cores=1,3,5,7
/proc/17705/task/17744/stat : 17744 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 72540 0 0 0 20500 999 0 0 25 0 5 0 522343853 2921570304 659196 33554432000 4194304 4338100 140733571547136 18446744073709551615 4258020 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 866.13
Current children cumulated vsize (KiB) 2853096

[startup+282.301 s]
/proc/loadavg: 8.06 8.04 8.00 9/186 17752
/proc/meminfo: memFree=23307656/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=3293072 CPUtime=1105.76 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) S 17703 17705 17262 0 -1 4202496 787655 0 0 0 107830 2746 0 0 25 0 5 0 522343138 3372105728 769133 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17705/statm: 823268 769133 176 36 0 821173 0
[pid=17705/tid=17741] ppid=17703 vsize=3293072 CPUtime=274.1 cores=1,3,5,7

################
# More data... #
################

/proc/17705/statm: 1552232 1460419 176 36 0 1550137 0
[pid=17705/tid=17741] ppid=17703 vsize=6208928 CPUtime=753.84 cores=1,3,5,7
/proc/17705/task/17741/stat : 17741 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 776161 0 0 0 75032 352 0 0 25 0 5 0 522343853 6357942272 1460419 33554432000 4194304 4338100 140733571547136 18446744073709551615 251813635088 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17705/tid=17742] ppid=17703 vsize=6208928 CPUtime=754.15 cores=1,3,5,7
/proc/17705/task/17742/stat : 17742 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 63384 0 0 0 71754 3661 0 0 25 0 5 0 522343853 6357942272 1460419 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826065780 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17705/tid=17743] ppid=17703 vsize=6208928 CPUtime=753.32 cores=1,3,5,7
/proc/17705/task/17743/stat : 17743 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 410735 0 0 0 75244 88 0 0 25 0 5 0 522343853 6357942272 1460419 33554432000 4194304 4338100 140733571547136 18446744073709551615 4241503 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17705/tid=17744] ppid=17703 vsize=6208928 CPUtime=754.12 cores=1,3,5,7
/proc/17705/task/17744/stat : 17744 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 88759 0 0 0 71771 3641 0 0 25 0 5 0 522343853 6357942272 1460419 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3022.6
Current children cumulated vsize (KiB) 6208928

[startup+822.301 s]
/proc/loadavg: 8.19 8.13 8.04 9/186 17767
/proc/meminfo: memFree=15627908/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=6398712 CPUtime=3262.16 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) S 17703 17705 17262 0 -1 4202496 1506249 0 0 0 317882 8334 0 0 25 0 5 0 522343138 6552281088 1487727 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17705/statm: 1599678 1487727 176 36 0 1597583 0
[pid=17705/tid=17741] ppid=17703 vsize=6398712 CPUtime=813.82 cores=1,3,5,7
/proc/17705/task/17741/stat : 17741 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 791992 0 0 0 81013 369 0 0 25 0 5 0 522343853 6552281088 1487727 33554432000 4194304 4338100 140733571547136 18446744073709551615 251813635321 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17705/tid=17742] ppid=17703 vsize=6398712 CPUtime=813.8 cores=1,3,5,7
/proc/17705/task/17742/stat : 17742 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 66668 0 0 0 77439 3941 0 0 25 0 5 0 522343853 6552281088 1487727 33554432000 4194304 4338100 140733571547136 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17705/tid=17743] ppid=17703 vsize=6398712 CPUtime=813.3 cores=1,3,5,7
/proc/17705/task/17743/stat : 17743 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 416498 0 0 0 81240 90 0 0 25 0 5 0 522343853 6552281088 1487727 33554432000 4194304 4338100 140733571547136 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17705/tid=17744] ppid=17703 vsize=6398712 CPUtime=814.08 cores=1,3,5,7
/proc/17705/task/17744/stat : 17744 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 91189 0 0 0 77492 3916 0 0 25 0 5 0 522343853 6552281088 1487727 33554432000 4194304 4338100 140733571547136 18446744073709551615 251813936644 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3262.16
Current children cumulated vsize (KiB) 6398712

[startup+882.301 s]
/proc/loadavg: 8.11 8.12 8.04 9/186 17768
/proc/meminfo: memFree=14981404/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=6526112 CPUtime=3501.76 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) S 17703 17705 17262 0 -1 4202496 1550309 0 0 0 341251 8925 0 0 25 0 5 0 522343138 6682738688 1531787 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17705/statm: 1631528 1531787 176 36 0 1629433 0
[pid=17705/tid=17741] ppid=17703 vsize=6526112 CPUtime=873.8 cores=1,3,5,7
/proc/17705/task/17741/stat : 17741 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 815768 0 0 0 86994 386 0 0 25 0 5 0 522343853 6682738688 1531787 33554432000 4194304 4338100 140733571547136 18446744073709551615 251813635497 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17705/tid=17742] ppid=17703 vsize=6526112 CPUtime=873.66 cores=1,3,5,7
/proc/17705/task/17742/stat : 17742 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 70568 0 0 0 83140 4226 0 0 25 0 5 0 522343853 6682738688 1531787 33554432000 4194304 4338100 140733571547136 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17705/tid=17743] ppid=17703 vsize=6526112 CPUtime=873.28 cores=1,3,5,7
/proc/17705/task/17743/stat : 17743 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 430871 0 0 0 87234 94 0 0 25 0 5 0 522343853 6682738688 1531787 33554432000 4194304 4338100 140733571547136 18446744073709551615 4253966 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17705/tid=17744] ppid=17703 vsize=6526112 CPUtime=873.85 cores=1,3,5,7
/proc/17705/task/17744/stat : 17744 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 93200 0 0 0 83185 4200 0 0 25 0 5 0 522343853 6682738688 1531787 33554432000 4194304 4338100 140733571547136 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3501.76
Current children cumulated vsize (KiB) 6526112

[startup+942.301 s]
/proc/loadavg: 8.08 8.11 8.04 9/186 17770
/proc/meminfo: memFree=14118032/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=6950224 CPUtime=3741.38 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) S 17703 17705 17262 0 -1 4202496 1646998 0 0 0 364606 9532 0 0 25 0 5 0 522343138 7117029376 1628476 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17705/statm: 1737556 1628476 176 36 0 1735461 0
[pid=17705/tid=17741] ppid=17703 vsize=6950224 CPUtime=933.62 cores=1,3,5,7
/proc/17705/task/17741/stat : 17741 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 882313 0 0 0 92947 415 0 0 25 0 5 0 522343853 7117029376 1628476 33554432000 4194304 4338100 140733571547136 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17705/tid=17742] ppid=17703 vsize=6950224 CPUtime=933.64 cores=1,3,5,7
/proc/17705/task/17742/stat : 17742 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 70570 0 0 0 88850 4514 0 0 25 0 5 0 522343853 7117029376 1628476 33554432000 4194304 4338100 140733571547136 18446744073709551615 251813635500 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17705/tid=17743] ppid=17703 vsize=6950224 CPUtime=933.28 cores=1,3,5,7
/proc/17705/task/17743/stat : 17743 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 461010 0 0 0 93227 101 0 0 25 0 5 0 522343853 7117029376 1628476 33554432000 4194304 4338100 140733571547136 18446744073709551615 251813635231 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17705/tid=17744] ppid=17703 vsize=6950224 CPUtime=933.66 cores=1,3,5,7
/proc/17705/task/17744/stat : 17744 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 93203 0 0 0 88884 4482 0 0 25 0 5 0 522343853 7117029376 1628476 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3741.38
Current children cumulated vsize (KiB) 6950224

[startup+1002.3 s]
/proc/loadavg: 8.07 8.10 8.03 9/186 17771
/proc/meminfo: memFree=13694716/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=7061344 CPUtime=3980.97 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) S 17703 17705 17262 0 -1 4202496 1686922 0 0 0 387988 10109 0 0 25 0 5 0 522343138 7230816256 1668400 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17705/statm: 1765336 1668400 176 36 0 1763241 0
[pid=17705/tid=17741] ppid=17703 vsize=7061344 CPUtime=993.6 cores=1,3,5,7
/proc/17705/task/17741/stat : 17741 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 911327 0 0 0 98924 436 0 0 25 0 5 0 522343853 7230816256 1668400 33554432000 4194304 4338100 140733571547136 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17705/tid=17742] ppid=17703 vsize=7061344 CPUtime=993.63 cores=1,3,5,7
/proc/17705/task/17742/stat : 17742 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 74889 0 0 0 94571 4792 0 0 25 0 5 0 522343853 7230816256 1668400 33554432000 4194304 4338100 140733571547136 18446744073709551615 4253950 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17705/tid=17743] ppid=17703 vsize=7061344 CPUtime=992.94 cores=1,3,5,7
/proc/17705/task/17743/stat : 17743 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 464883 0 0 0 99193 101 0 0 25 0 5 0 522343853 7230816256 1668400 33554432000 4194304 4338100 140733571547136 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17705/tid=17744] ppid=17703 vsize=7061344 CPUtime=993.63 cores=1,3,5,7
/proc/17705/task/17744/stat : 17744 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 95921 0 0 0 94602 4761 0 0 25 0 5 0 522343853 7230816256 1668400 33554432000 4194304 4338100 140733571547136 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3980.97
Current children cumulated vsize (KiB) 7061344

[startup+1062.3 s]
/proc/loadavg: 8.06 8.10 8.03 9/186 17773
/proc/meminfo: memFree=13179908/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=7400384 CPUtime=4220.52 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) S 17703 17705 17262 0 -1 4202496 1761424 0 0 0 411310 10742 0 0 25 0 5 0 522343138 7577993216 1742902 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17705/statm: 1850096 1742902 176 36 0 1848001 0
[pid=17705/tid=17741] ppid=17703 vsize=7400384 CPUtime=1053.55 cores=1,3,5,7
/proc/17705/task/17741/stat : 17741 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 966917 0 0 0 104890 465 0 0 25 0 5 0 522343853 7577993216 1742902 33554432000 4194304 4338100 140733571547136 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17705/tid=17742] ppid=17703 vsize=7400384 CPUtime=1053.59 cores=1,3,5,7
/proc/17705/task/17742/stat : 17742 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 74891 0 0 0 100266 5093 0 0 25 0 5 0 522343853 7577993216 1742902 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17705/tid=17743] ppid=17703 vsize=7400384 CPUtime=1052.6 cores=1,3,5,7
/proc/17705/task/17743/stat : 17743 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 483792 0 0 0 105154 106 0 0 25 0 5 0 522343853 7577993216 1742903 33554432000 4194304 4338100 140733571547136 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17705/tid=17744] ppid=17703 vsize=7400384 CPUtime=1053.6 cores=1,3,5,7
/proc/17705/task/17744/stat : 17744 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 95922 0 0 0 100301 5059 0 0 25 0 5 0 522343853 7577993216 1742903 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4220.52
Current children cumulated vsize (KiB) 7400384

[startup+1122.3 s]
/proc/loadavg: 8.05 8.09 8.03 9/186 17775
/proc/meminfo: memFree=12528932/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=7684072 CPUtime=4460.1 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) S 17703 17705 17262 0 -1 4202496 1828757 0 0 0 434640 11370 0 0 25 0 5 0 522343138 7868489728 1810235 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17705/statm: 1921018 1810235 176 36 0 1918923 0
[pid=17705/tid=17741] ppid=17703 vsize=7684072 CPUtime=1113.53 cores=1,3,5,7
/proc/17705/task/17741/stat : 17741 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 1016797 0 0 0 110863 490 0 0 25 0 5 0 522343853 7868489728 1810235 33554432000 4194304 4338100 140733571547136 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17705/tid=17742] ppid=17703 vsize=7684072 CPUtime=1113.56 cores=1,3,5,7
/proc/17705/task/17742/stat : 17742 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 74893 0 0 0 105964 5392 0 0 25 0 5 0 522343853 7868489728 1810235 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17705/tid=17743] ppid=17703 vsize=7684072 CPUtime=1112.26 cores=1,3,5,7
/proc/17705/task/17743/stat : 17743 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 501232 0 0 0 111116 110 0 0 25 0 5 0 522343853 7868489728 1810235 33554432000 4194304 4338100 140733571547136 18446744073709551615 4235770 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17705/tid=17744] ppid=17703 vsize=7684072 CPUtime=1113.57 cores=1,3,5,7
/proc/17705/task/17744/stat : 17744 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 95933 0 0 0 105999 5358 0 0 25 0 5 0 522343853 7868489728 1810235 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4460.1
Current children cumulated vsize (KiB) 7684072

[startup+1182.3 s]
/proc/loadavg: 8.07 8.09 8.03 9/186 17776
/proc/meminfo: memFree=12037316/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=7915664 CPUtime=4699.67 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) S 17703 17705 17262 0 -1 4202496 1892072 0 0 0 457956 12011 0 0 25 0 5 0 522343138 8105639936 1873550 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17705/statm: 1978916 1873550 176 36 0 1976821 0
[pid=17705/tid=17741] ppid=17703 vsize=7915664 CPUtime=1173.48 cores=1,3,5,7
/proc/17705/task/17741/stat : 17741 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 1062229 0 0 0 116835 513 0 0 25 0 5 0 522343853 8105639936 1873550 33554432000 4194304 4338100 140733571547136 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17705/tid=17742] ppid=17703 vsize=7915664 CPUtime=1173.54 cores=1,3,5,7
/proc/17705/task/17742/stat : 17742 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 74925 0 0 0 111651 5703 0 0 25 0 5 0 522343853 8105639936 1873550 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17705/tid=17743] ppid=17703 vsize=7915664 CPUtime=1171.91 cores=1,3,5,7
/proc/17705/task/17743/stat : 17743 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 519054 0 0 0 117077 114 0 0 25 0 5 0 522343853 8105639936 1873550 33554432000 4194304 4338100 140733571547136 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17705/tid=17744] ppid=17703 vsize=7915664 CPUtime=1173.55 cores=1,3,5,7
/proc/17705/task/17744/stat : 17744 (ttsth-5-4) R 17703 17705 17262 0 -1 4202560 95962 0 0 0 111694 5661 0 0 25 0 5 0 522343853 8105639936 1873550 33554432000 4194304 4338100 140733571547136 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4699.67
Current children cumulated vsize (KiB) 7915664



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1207.5 s]
/proc/loadavg: 8.05 8.08 8.03 6/183 17777
/proc/meminfo: memFree=13753348/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=0 CPUtime=4800.09 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) Z 17703 17705 17262 0 -1 4203548 1917550 0 0 0 467716 12293 0 0 18 0 2 0 522343138 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17705/statm: 0 0 0 0 0 0 0
[pid=17705/tid=17743] ppid=17703 vsize=0 CPUtime=1196.97 cores=1,3,5,7
/proc/17705/task/17743/stat : 17743 (ttsth-5-4) R 17703 17705 17262 0 -1 4203588 531844 0 0 0 119571 126 0 0 25 0 2 0 522343853 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.09
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1207.61 s]
/proc/loadavg: 8.05 8.08 8.03 6/183 17777
/proc/meminfo: memFree=13753348/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=0 CPUtime=4800.19 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) Z 17703 17705 17262 0 -1 4203548 1917550 0 0 0 467716 12303 0 0 18 0 2 0 522343138 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17705/statm: 0 0 0 0 0 0 0
[pid=17705/tid=17743] ppid=17703 vsize=0 CPUtime=1197.08 cores=1,3,5,7
/proc/17705/task/17743/stat : 17743 (ttsth-5-4) R 17703 17705 17262 0 -1 4203588 531844 0 0 0 119571 137 0 0 25 0 2 0 522343853 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.19
Current children cumulated vsize (KiB) 0

[startup+1207.7 s]
/proc/loadavg: 8.05 8.08 8.03 6/184 17778
/proc/meminfo: memFree=19395840/32950928 swapFree=67111448/67111528
[pid=17705] ppid=17703 vsize=0 CPUtime=4800.28 cores=1,3,5,7
/proc/17705/stat : 17705 (ttsth-5-4) Z 17703 17705 17262 0 -1 4203548 1917550 0 0 0 467716 12312 0 0 18 0 1 0 522343138 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17705/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 4800.28
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1207.71
CPU time (s): 4800.29
CPU user time (s): 4677.16
CPU system time (s): 123.128
CPU usage (%): 397.472
Max. virtual memory (cumulated for all children) (KiB): 8043064

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4677.16
system time used= 123.128
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1917550
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 8875
involuntary context switches= 33658

runsolver used 1.9737 second user time and 4.6273 second system time

The end

Launcher Data 

Begin job on node126 at 2011-04-29 22:04:04
IDJOB=3251352
IDBENCH=88219
IDSOLVER=1654
FILE ID=node126/3251352-1304107444
RUNJOBID= node126-1304106237-17609
PBS_JOBID= 13170635
Free space on /tmp= 71576 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m62-q1504.5-n60-p30-s88600538.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3251352-1304107444/watcher-3251352-1304107444 -o /tmp/evaluation-result-3251352-1304107444/solver-3251352-1304107444 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251352-1304107444.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 76c30bc0971bc865c2b988ddaca89385
RANDOM SEED=454794093

node126.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.56
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      22637052 kB
Buffers:        253816 kB
Cached:        1428000 kB
SwapCached:         76 kB
Active:        8390708 kB
Inactive:      1595348 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      22637052 kB
SwapTotal:    67111528 kB
SwapFree:     67111448 kB
Dirty:           12176 kB
Writeback:          36 kB
AnonPages:     8303916 kB
Mapped:          15276 kB
Slab:           247292 kB
PageTables:      21336 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:  8852868 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71588 MiB
End job on node126 at 2011-04-29 22:24:14