Trace number 3257382

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.24 1203.09

General information on the benchmark

NameCRAFTED/edgematching/
compact/em_11_3_4_cmp.cnf
MD5SUM3b5e80540ef2fcb09cfe50ac7e3686c1
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark135.155
Satisfiable
(Un)Satisfiability was proved
Number of variables8713
Number of clauses99699
Sum of the clauses size335345
Maximum clause length81
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 210866
Number of clauses of size 367270
Number of clauses of size 410556
Number of clauses of size 510773
Number of clauses of size over 5234

Solver Data 

0.00/0.10	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3257382-1304046015.cnf 
0.00/0.10	c Using SATTIMEOUT of 4800
0.00/0.10	c Using SATRAM of 15500
0.00/0.10	c Size of subset node = 32
0.00/0.10	c Variables = 8713
0.00/0.10	c Clauses = 99699
0.00/0.10	c Literals = 335345
0.00/0.10	c Reduced size:
0.00/0.10	c Variables = 8713
0.00/0.10	c Clauses = 99699
0.00/0.10	c Literals = 335345
0.00/0.10	c a2
0.29/0.32	c a1
1.29/1.31	c Independent sub-problems = 1
1.29/1.31	c Sub-problem with 8713 variables and 99699 clauses
1.29/1.31	c using permutation method 4
1.29/1.31	c count = 1
1.29/1.31	c Ordering finished 1.290 seconds
1.29/1.31	c mapped 1.310 seconds
1.29/1.31	c permuted 1.310 seconds
1.29/1.31	c about to sort
1.29/1.32	c sorted
1.49/1.58	c clauses now 99699
1.49/1.58	c init finished 1.400 seconds
1.49/1.58	c tree built 1.570 seconds
1.49/1.58	c 
1.49/1.58	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
1.49/1.58	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
1.49/1.58	c ................................ | ..........                    |                                  1.6         0.0             
1.49/1.58	c ................................ | ..........                  1 |                                  1.6         0.6             
1.49/1.58	c ................................ | ..........                  2 |                                  1.6         1.3             
1.49/1.58	c ................................ | ..........                  3 |                                  1.6         1.9             
1.49/1.58	c ................................ | ..........                  4 |                                  1.6         2.5             
1.49/1.58	c ................................ | ..........                  5 |                                  1.6         3.2             
1.49/1.58	c ................................ | ..........               1000 |              123                 1.6       628.9         1433
95.91/25.27	c ................................ | ..........            1201000 | 25989        109772             96.2     12483.1     12969111
191.75/49.28	c ................................ | ..........            1658000 | 48846        171582            192.1      8630.5     24071426
288.01/73.31	c ................................ | ..........            2057000 | 67930        227724            288.1      7141.1     31026770
383.88/97.35	c ................................ | ..........            2377000 | 84755        279492            384.1      6188.7     38192155
480.15/121.44	c ................................ | *.........            2709000 | 101874       330702            480.3      5640.1     46790351
576.02/145.47	c ................................ | *.........            2928000 | 113757       377560            576.3      5080.6     52339602
672.30/169.51	c ................................ | *.........            3168000 | 130416       423474            672.3      4712.1     59817457
768.16/193.51	c ................................ | *.........            3436000 | 145251       461712            768.2      4472.6     67988528
864.02/217.50	c ................................ | *.........            3673000 | 158415       495239            864.0      4251.0     75237706
959.89/241.54	c ................................ | **........            3868000 | 168748       530160            960.0      4029.0     80665150
1056.16/265.63	c ................................ | **........            4106000 | 181018       567940           1056.3      3887.2     88372124
1152.01/289.61	c ................................ | **........            4306000 | 191127       600462           1152.1      3737.7     92902546
1248.28/313.77	c ................................ | **........            4494000 | 201156       629529           1248.6      3599.3     96928268
1343.76/337.68	c ................................ | **........            4653000 | 209436       659186           1344.1      3461.8    100695371
1440.03/361.76	c ................................ | ***.......            4823000 | 218670       686079           1440.3      3348.7    104185145
1536.30/385.81	c ................................ | ***.......            4999000 | 227656       714981           1536.3      3253.8    107907120
1632.16/409.87	c ................................ | ***.......            5201000 | 238289       744628           1632.4      3186.0    111417458
1728.41/433.91	c ................................ | ***.......            5378000 | 248126       773916           1728.5      3111.4    114543803
1824.68/458.03	c ................................ | ***.......            5539000 | 256670       802106           1824.8      3035.4    119150550
1920.17/481.99	c ................................ | ****......            5723000 | 266541       824525           1920.5      2980.0    122792740
2016.02/505.98	c ................................ | ****......            5878000 | 274580       847131           2016.4      2915.2    126903750
2112.29/530.07	c ................................ | ****......            6027000 | 282686       874446           2112.6      2852.9    131417529
2208.55/554.10	c ................................ | ****......            6166000 | 290190       901368           2208.6      2791.9    134394224
2304.02/578.02	c ................................ | ****......            6319000 | 297657       922619           2304.1      2742.5    138051621
2399.88/602.03	c ................................ | *****.....            6459000 | 304832       942249           2400.0      2691.2    140183937
2495.78/626.07	c ................................ | *****.....            6594000 | 312177       962842           2496.1      2641.8    142665150
2592.04/650.16	c ................................ | *****.....            6727000 | 319418       984080           2592.3      2595.0    144934480
2687.89/674.13	c ................................ | *****.....            6862000 | 326617       1010010          2688.0      2552.8    148513822
2784.55/698.32	c ................................ | *****.....            7015000 | 334206       1031227          2784.6      2519.2    149932890
2880.42/722.36	c ................................ | ******....            7150000 | 342087       1048902          2880.7      2482.1    153746141
2976.29/746.39	c ................................ | ******....            7295000 | 349920       1070632          2976.7      2450.7    156409543
3072.17/770.32	c ................................ | ******....            7448000 | 357448       1087339          3072.3      2424.3    158350282
3168.06/794.36	c ................................ | ******....            7588000 | 364691       1106371          3168.3      2395.0    158986240
3264.32/818.43	c ................................ | ******....            7714000 | 371604       1125262          3264.4      2363.1    160267532
3360.17/842.40	c ................................ | *******...            7853000 | 379475       1143100          3360.2      2337.1    163294805
3456.03/866.49	c ................................ | *******...            8013000 | 388169       1162751          3456.4      2318.3    165937122
3552.30/890.51	c ................................ | *******...            8130000 | 394843       1184985          3552.3      2288.6    168035952
3648.56/914.65	c ................................ | *******...            8248000 | 401449       1202687          3648.8      2260.5    170146752
3744.41/938.68	c ................................ | *******...            8371000 | 410675       1219573          3744.7      2235.4    172619604
3840.30/962.68	c ................................ | ********..            8490000 | 419362       1238626          3840.6      2210.6    173958539
3935.74/986.58	c ................................ | ********..            8621000 | 425529       1261345          3936.1      2190.3    175013812
4032.01/1010.67	c ................................ | ********..            8751000 | 433184       1279279          4032.3      2170.2    175804042
4127.87/1034.69	c ................................ | ********..            8876000 | 440624       1297897          4128.2      2150.1    176858358
4224.13/1058.74	c ................................ | ********..            9024000 | 448532       1321037          4224.3      2136.2    177825490
4319.99/1082.73	c ................................ | *********.            9164000 | 456170       1336645          4320.1      2121.2    178983931
4416.65/1106.91	c ................................ | *********.            9309000 | 464402       1353609          4416.7      2107.7    179782279
4512.55/1130.92	c ................................ | *********.            9422000 | 470100       1371909          4512.6      2087.9    179922071
4608.40/1154.92	c ................................ | *********.            9541000 | 476515       1389203          4608.5      2070.3    180851694
4703.86/1178.88	c ................................ | *********.            9665000 | 483517       1407375          4704.2      2054.6    182826039
4799.72/1202.87	c CPU limit (4800) reached
4799.72/1202.87	c Time = 4800.02 seconds
4799.72/1202.87	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3257382-1304046015/watcher-3257382-1304046015 -o /tmp/evaluation-result-3257382-1304046015/solver-3257382-1304046015 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3257382-1304046015.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.75 5.39 6.70 2/174 19195
/proc/meminfo: memFree=11753496/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=9104 CPUtime=0 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) R 19193 19195 18020 0 -1 4202496 357 0 0 0 0 0 0 0 23 0 1 0 594832115 9322496 299 33554432000 4194304 4338100 140734631482624 18446744073709551615 216734839493 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/19195/statm: 2276 300 147 36 0 181 0

[startup+0.105299 s]
/proc/loadavg: 3.75 5.39 6.70 2/174 19195
/proc/meminfo: memFree=11753496/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=22344 CPUtime=0.09 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) R 19193 19195 18020 0 -1 4202496 3565 0 0 0 9 0 0 0 23 0 1 0 594832115 22880256 3508 33554432000 4194304 4338100 140734631482624 18446744073709551615 216734839493 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/19195/statm: 5586 3516 151 36 0 3491 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 22344

[startup+0.200283 s]
/proc/loadavg: 3.75 5.39 6.70 2/174 19195
/proc/meminfo: memFree=11753496/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=47400 CPUtime=0.19 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) R 19193 19195 18020 0 -1 4202496 9491 0 0 0 18 1 0 0 24 0 1 0 594832115 48537600 9433 33554432000 4194304 4338100 140734631482624 18446744073709551615 216734573753 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/19195/statm: 11850 9433 151 36 0 9755 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 47400

[startup+0.300265 s]
/proc/loadavg: 3.75 5.39 6.70 2/174 19195
/proc/meminfo: memFree=11753496/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=47400 CPUtime=0.29 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) R 19193 19195 18020 0 -1 4202496 9496 0 0 0 28 1 0 0 25 0 1 0 594832115 48537600 9437 33554432000 4194304 4338100 140734631482624 18446744073709551615 216734573753 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/19195/statm: 11850 9437 153 36 0 9755 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 47400

[startup+0.700194 s]
/proc/loadavg: 3.75 5.39 6.70 2/174 19195
/proc/meminfo: memFree=11753496/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=47232 CPUtime=0.69 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) R 19193 19195 18020 0 -1 4202496 9932 0 0 0 68 1 0 0 25 0 1 0 594832115 48365568 9483 33554432000 4194304 4338100 140734631482624 18446744073709551615 4245192 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/19195/statm: 11808 9483 156 36 0 9713 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 47232

[startup+1.50106 s]
/proc/loadavg: 3.61 5.33 6.67 7/182 19206
/proc/meminfo: memFree=11647544/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=141764 CPUtime=1.49 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) R 19193 19195 18020 0 -1 4202496 33814 0 0 0 145 4 0 0 25 0 1 0 594832115 145166336 33170 33554432000 4194304 4338100 140734631482624 18446744073709551615 216734839493 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/19195/statm: 35441 33171 164 36 0 33346 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 141764

[startup+3.10073 s]
/proc/loadavg: 3.61 5.33 6.67 9/186 19210
/proc/meminfo: memFree=11501192/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=426148 CPUtime=7.64 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) S 19193 19195 18020 0 -1 4202496 45876 0 0 0 750 14 0 0 25 0 5 0 594832115 436375552 45036 33554432000 4194304 4338100 140734631482624 18446744073709551615 216746981349 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/19195/statm: 106537 45036 176 36 0 104442 0
[pid=19195/tid=19207] ppid=19193 vsize=426148 CPUtime=1.5 cores=1,3,5,7
/proc/19195/task/19207/stat : 19207 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 2387 0 0 0 150 0 0 0 20 0 5 0 594832273 436375552 45036 33554432000 4194304 4338100 140734631482624 18446744073709551615 216734573798 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=19195/tid=19208] ppid=19193 vsize=426148 CPUtime=1.5 cores=1,3,5,7
/proc/19195/task/19208/stat : 19208 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 2141 0 0 0 147 3 0 0 19 0 5 0 594832273 436375552 45036 33554432000 4194304 4338100 140734631482624 18446744073709551615 216747004597 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=19195/tid=19209] ppid=19193 vsize=426148 CPUtime=1.5 cores=1,3,5,7
/proc/19195/task/19209/stat : 19209 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 1728 0 0 0 150 0 0 0 25 0 5 0 594832273 436375552 45036 33554432000 4194304 4338100 140734631482624 18446744073709551615 216734573584 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=19195/tid=19210] ppid=19193 vsize=426148 CPUtime=1.51 cores=1,3,5,7
/proc/19195/task/19210/stat : 19210 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 2155 0 0 0 148 3 0 0 20 0 5 0 594832273 436375552 45036 33554432000 4194304 4338100 140734631482624 18446744073709551615 216747004597 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 7.64
Current children cumulated vsize (KiB) 426148

[startup+6.30117 s]
/proc/loadavg: 3.96 5.38 6.68 9/186 19210
/proc/meminfo: memFree=11410796/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=463592 CPUtime=20.41 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) S 19193 19195 18020 0 -1 4202496 56036 0 0 0 2015 26 0 0 25 0 5 0 594832115 474718208 55196 33554432000 4194304 4338100 140734631482624 18446744073709551615 216746981349 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/19195/statm: 115898 55198 176 36 0 113803 0
[pid=19195/tid=19207] ppid=19193 vsize=463592 CPUtime=4.71 cores=1,3,5,7
/proc/19195/task/19207/stat : 19207 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 3925 0 0 0 470 1 0 0 25 0 5 0 594832273 474718208 55198 33554432000 4194304 4338100 140734631482624 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=19195/tid=19208] ppid=19193 vsize=463592 CPUtime=4.71 cores=1,3,5,7
/proc/19195/task/19208/stat : 19208 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 4805 0 0 0 461 10 0 0 25 0 5 0 594832273 474718208 55198 33554432000 4194304 4338100 140734631482624 18446744073709551615 216734573633 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=19195/tid=19209] ppid=19193 vsize=463592 CPUtime=4.68 cores=1,3,5,7
/proc/19195/task/19209/stat : 19209 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 4774 0 0 0 467 1 0 0 20 0 5 0 594832273 474718208 55198 33554432000 4194304 4338100 140734631482624 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=19195/tid=19210] ppid=19193 vsize=463592 CPUtime=4.71 cores=1,3,5,7
/proc/19195/task/19210/stat : 19210 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 5069 0 0 0 463 8 0 0 22 0 5 0 594832273 474718208 55198 33554432000 4194304 4338100 140734631482624 18446744073709551615 216734574318 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 20.41
Current children cumulated vsize (KiB) 463592

[startup+12.7011 s]
/proc/loadavg: 4.29 5.42 6.69 9/186 19211
/proc/meminfo: memFree=11251212/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=570844 CPUtime=45.98 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) S 19193 19195 18020 0 -1 4202496 80997 0 0 0 4548 50 0 0 25 0 5 0 594832115 584544256 80157 33554432000 4194304 4338100 140734631482624 18446744073709551615 216746981349 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/19195/statm: 142711 80157 176 36 0 140616 0
[pid=19195/tid=19207] ppid=19193 vsize=570844 CPUtime=11.11 cores=1,3,5,7
/proc/19195/task/19207/stat : 19207 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 13072 0 0 0 1108 3 0 0 25 0 5 0 594832273 584544256 80157 33554432000 4194304 4338100 140734631482624 18446744073709551615 4230411 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=19195/tid=19208] ppid=19193 vsize=570844 CPUtime=11.1 cores=1,3,5,7
/proc/19195/task/19208/stat : 19208 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 9753 0 0 0 1090 20 0 0 25 0 5 0 594832273 584544256 80157 33554432000 4194304 4338100 140734631482624 18446744073709551615 216747004597 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=19195/tid=19209] ppid=19193 vsize=570844 CPUtime=11.05 cores=1,3,5,7
/proc/19195/task/19209/stat : 19209 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 10509 0 0 0 1103 2 0 0 25 0 5 0 594832273 584544256 80157 33554432000 4194304 4338100 140734631482624 18446744073709551615 4285386 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=19195/tid=19210] ppid=19193 vsize=570844 CPUtime=11.1 cores=1,3,5,7
/proc/19195/task/19210/stat : 19210 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 10198 0 0 0 1092 18 0 0 25 0 5 0 594832273 584544256 80157 33554432000 4194304 4338100 140734631482624 18446744073709551615 216747004597 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 45.98
Current children cumulated vsize (KiB) 570844

[startup+25.5008 s]
/proc/loadavg: 5.11 5.55 6.71 9/186 19211
/proc/meminfo: memFree=10772328/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=822144 CPUtime=97.11 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) S 19193 19195 18020 0 -1 4202496 145166 0 0 0 9609 102 0 0 25 0 5 0 594832115 841875456 144326 33554432000 4194304 4338100 140734631482624 18446744073709551615 216746981349 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/19195/statm: 205536 144326 176 36 0 203441 0
[pid=19195/tid=19207] ppid=19193 vsize=822144 CPUtime=23.91 cores=1,3,5,7
/proc/19195/task/19207/stat : 19207 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 41339 0 0 0 2382 9 0 0 25 0 5 0 594832273 841875456 144326 33554432000 4194304 4338100 140734631482624 18446744073709551615 4256032 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=19195/tid=19208] ppid=19193 vsize=822144 CPUtime=23.91 cores=1,3,5,7
/proc/19195/task/19208/stat : 19208 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 19496 0 0 0 2349 42 0 0 25 0 5 0 594832273 841875456 144326 33554432000 4194304 4338100 140734631482624 18446744073709551615 216747004597 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=19195/tid=19209] ppid=19193 vsize=822144 CPUtime=23.77 cores=1,3,5,7
/proc/19195/task/19209/stat : 19209 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 26577 0 0 0 2372 5 0 0 25 0 5 0 594832273 841875456 144326 33554432000 4194304 4338100 140734631482624 18446744073709551615 4283664 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=19195/tid=19210] ppid=19193 vsize=822144 CPUtime=23.9 cores=1,3,5,7
/proc/19195/task/19210/stat : 19210 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 20289 0 0 0 2351 39 0 0 25 0 5 0 594832273 841875456 144326 33554432000 4194304 4338100 140734631482624 18446744073709551615 216747004597 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 97.11
Current children cumulated vsize (KiB) 822144

[startup+51.1004 s]
/proc/loadavg: 6.09 5.74 6.74 9/186 19221
/proc/meminfo: memFree=9944212/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=1193608 CPUtime=199.34 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) S 19193 19195 18020 0 -1 4202496 237014 0 0 0 19731 203 0 0 25 0 5 0 594832115 1222254592 236174 33554432000 4194304 4338100 140734631482624 18446744073709551615 216746981349 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/19195/statm: 298402 236174 176 36 0 296307 0
[pid=19195/tid=19207] ppid=19193 vsize=1193608 CPUtime=49.48 cores=1,3,5,7
/proc/19195/task/19207/stat : 19207 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 81362 0 0 0 4934 14 0 0 25 0 5 0 594832273 1222254592 236174 33554432000 4194304 4338100 140734631482624 18446744073709551615 216734574525 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=19195/tid=19208] ppid=19193 vsize=1193608 CPUtime=49.51 cores=1,3,5,7
/proc/19195/task/19208/stat : 19208 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 28721 0 0 0 4863 88 0 0 25 0 5 0 594832273 1222254592 236174 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=19195/tid=19209] ppid=19193 vsize=1193608 CPUtime=49.24 cores=1,3,5,7
/proc/19195/task/19209/stat : 19209 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 60276 0 0 0 4913 11 0 0 25 0 5 0 594832273 1222254592 236174 33554432000 4194304 4338100 140734631482624 18446744073709551615 216734573793 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=19195/tid=19210] ppid=19193 vsize=1193608 CPUtime=49.51 cores=1,3,5,7
/proc/19195/task/19210/stat : 19210 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 29190 0 0 0 4867 84 0 0 25 0 5 0 594832273 1222254592 236174 33554432000 4194304 4338100 140734631482624 18446744073709551615 216747004597 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 199.34
Current children cumulated vsize (KiB) 1193608

[startup+102.3 s]
/proc/loadavg: 7.22 6.10 6.81 9/186 19222
/proc/meminfo: memFree=8689628/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=1673916 CPUtime=403.87 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) S 19193 19195 18020 0 -1 4202496 358881 0 0 0 39983 404 0 0 25 0 5 0 594832115 1714089984 358041 33554432000 4194304 4338100 140734631482624 18446744073709551615 216746981349 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/19195/statm: 418479 358041 176 36 0 416384 0
[pid=19195/tid=19207] ppid=19193 vsize=1673916 CPUtime=100.69 cores=1,3,5,7
/proc/19195/task/19207/stat : 19207 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 133264 0 0 0 10045 24 0 0 25 0 5 0 594832273 1714089984 358041 33554432000 4194304 4338100 140734631482624 18446744073709551615 216734574010 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=19195/tid=19208] ppid=19193 vsize=1673916 CPUtime=100.71 cores=1,3,5,7
/proc/19195/task/19208/stat : 19208 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 34003 0 0 0 9894 177 0 0 25 0 5 0 594832273 1714089984 358041 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=19195/tid=19209] ppid=19193 vsize=1673916 CPUtime=100.16 cores=1,3,5,7
/proc/19195/task/19209/stat : 19209 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 119524 0 0 0 9994 22 0 0 25 0 5 0 594832273 1714089984 358041 33554432000 4194304 4338100 140734631482624 18446744073709551615 4253933 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=19195/tid=19210] ppid=19193 vsize=1673916 CPUtime=100.71 cores=1,3,5,7
/proc/19195/task/19210/stat : 19210 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 34625 0 0 0 9897 174 0 0 25 0 5 0 594832273 1714089984 358041 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 403.87
Current children cumulated vsize (KiB) 1673916

[startup+162.301 s]
/proc/loadavg: 7.79 6.47 6.89 9/186 19224
/proc/meminfo: memFree=7500096/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=2244028 CPUtime=643.52 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) S 19193 19195 18020 0 -1 4202496 501322 0 0 0 63723 629 0 0 25 0 5 0 594832115 2297884672 500482 33554432000 4194304 4338100 140734631482624 18446744073709551615 216746981349 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/19195/statm: 561007 500482 176 36 0 558912 0
[pid=19195/tid=19207] ppid=19193 vsize=2244028 CPUtime=160.67 cores=1,3,5,7
/proc/19195/task/19207/stat : 19207 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 168818 0 0 0 16036 31 0 0 25 0 5 0 594832273 2297884672 500482 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=19195/tid=19208] ppid=19193 vsize=2244028 CPUtime=160.7 cores=1,3,5,7
/proc/19195/task/19208/stat : 19208 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 50942 0 0 0 15789 281 0 0 25 0 5 0 594832273 2297884672 500482 33554432000 4194304 4338100 140734631482624 18446744073709551615 216734875121 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=19195/tid=19209] ppid=19193 vsize=2244028 CPUtime=159.84 cores=1,3,5,7
/proc/19195/task/19209/stat : 19209 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 193295 0 0 0 15947 37 0 0 25 0 5 0 594832273 2297884672 500482 33554432000 4194304 4338100 140734631482624 18446744073709551615 216734573798 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=19195/tid=19210] ppid=19193 vsize=2244028 CPUtime=160.71 cores=1,3,5,7
/proc/19195/task/19210/stat : 19210 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 50802 0 0 0 15797 274 0 0 25 0 5 0 594832273 2297884672 500482 33554432000 4194304 4338100 140734631482624 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 643.52
Current children cumulated vsize (KiB) 2244028

[startup+222.307 s]
/proc/loadavg: 8.06 6.79 6.97 9/186 19225
/proc/meminfo: memFree=6371316/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=2818036 CPUtime=883.22 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) S 19193 19195 18020 0 -1 4202496 645888 0 0 0 87515 807 0 0 25 0 5 0 594832115 2885668864 645048 33554432000 4194304 4338100 140734631482624 18446744073709551615 216746981349 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/19195/statm: 704509 645048 176 36 0 702414 0
[pid=19195/tid=19207] ppid=19193 vsize=2818036 CPUtime=220.67 cores=1,3,5,7
/proc/19195/task/19207/stat : 19207 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 215532 0 0 0 22026 41 0 0 25 0 5 0 594832273 2885668864 645048 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0

################
# More data... #
################

/proc/19195/task/19210/stat : 19210 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 123103 0 0 0 69177 893 0 0 25 0 5 0 594832273 5262675968 1227239 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2800.54
Current children cumulated vsize (KiB) 5139332

[startup+762.3 s]
/proc/loadavg: 8.12 7.90 7.47 9/186 19240
/proc/meminfo: memFree=897116/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=5378128 CPUtime=3040.23 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) S 19193 19195 18020 0 -1 4202496 1286914 0 0 0 301917 2106 0 0 25 0 5 0 594832115 5507203072 1286074 33554432000 4194304 4338100 140734631482624 18446744073709551615 216746981349 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/19195/statm: 1344532 1286074 176 36 0 1342437 0
[pid=19195/tid=19207] ppid=19193 vsize=5378128 CPUtime=760.62 cores=1,3,5,7
/proc/19195/task/19207/stat : 19207 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 343108 0 0 0 75995 67 0 0 25 0 5 0 594832273 5507203072 1286074 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=19195/tid=19208] ppid=19193 vsize=5378128 CPUtime=760.71 cores=1,3,5,7
/proc/19195/task/19208/stat : 19208 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 132355 0 0 0 75103 968 0 0 25 0 5 0 594832273 5507203072 1286074 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=19195/tid=19209] ppid=19193 vsize=5378128 CPUtime=756.57 cores=1,3,5,7
/proc/19195/task/19209/stat : 19209 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 642738 0 0 0 75540 117 0 0 25 0 5 0 594832273 5507203072 1286074 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=19195/tid=19210] ppid=19193 vsize=5378128 CPUtime=760.71 cores=1,3,5,7
/proc/19195/task/19210/stat : 19210 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 131248 0 0 0 75125 946 0 0 25 0 5 0 594832273 5507203072 1286074 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3040.23
Current children cumulated vsize (KiB) 5378128

[startup+822.301 s]
/proc/loadavg: 8.06 7.93 7.50 9/186 19241
/proc/meminfo: memFree=593360/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=5456724 CPUtime=3279.88 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) S 19193 19195 18020 0 -1 4202496 1307078 0 0 0 325779 2209 0 0 25 0 5 0 594832115 5587685376 1306238 33554432000 4194304 4338100 140734631482624 18446744073709551615 216746981349 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/19195/statm: 1364181 1306238 176 36 0 1362086 0
[pid=19195/tid=19207] ppid=19193 vsize=5456724 CPUtime=820.61 cores=1,3,5,7
/proc/19195/task/19207/stat : 19207 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 355709 0 0 0 81991 70 0 0 25 0 5 0 594832273 5587685376 1306238 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=19195/tid=19208] ppid=19193 vsize=5456724 CPUtime=820.7 cores=1,3,5,7
/proc/19195/task/19208/stat : 19208 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 134771 0 0 0 81052 1018 0 0 25 0 5 0 594832273 5587685376 1306238 33554432000 4194304 4338100 140734631482624 18446744073709551615 216747004597 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=19195/tid=19209] ppid=19193 vsize=5456724 CPUtime=816.24 cores=1,3,5,7
/proc/19195/task/19209/stat : 19209 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 645709 0 0 0 81506 118 0 0 25 0 5 0 594832273 5587685376 1306238 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=19195/tid=19210] ppid=19193 vsize=5456724 CPUtime=820.71 cores=1,3,5,7
/proc/19195/task/19210/stat : 19210 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 133424 0 0 0 81076 995 0 0 25 0 5 0 594832273 5587685376 1306238 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3279.88
Current children cumulated vsize (KiB) 5456724

[startup+882.307 s]
/proc/loadavg: 8.13 7.98 7.55 9/186 19243
/proc/meminfo: memFree=254808/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=5687352 CPUtime=3519.56 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) S 19193 19195 18020 0 -1 4202496 1364362 0 0 0 349714 2242 0 0 25 0 5 0 594832115 5823848448 1363522 33554432000 4194304 4338100 140734631482624 18446744073709551615 216746981349 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/19195/statm: 1421838 1363522 176 36 0 1419743 0
[pid=19195/tid=19207] ppid=19193 vsize=5687352 CPUtime=880.61 cores=1,3,5,7
/proc/19195/task/19207/stat : 19207 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 368400 0 0 0 87986 75 0 0 25 0 5 0 594832273 5823848448 1363522 33554432000 4194304 4338100 140734631482624 18446744073709551615 4258020 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=19195/tid=19208] ppid=19193 vsize=5687352 CPUtime=880.71 cores=1,3,5,7
/proc/19195/task/19208/stat : 19208 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 137628 0 0 0 87042 1029 0 0 25 0 5 0 594832273 5823848448 1363522 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=19195/tid=19209] ppid=19193 vsize=5687352 CPUtime=875.92 cores=1,3,5,7
/proc/19195/task/19209/stat : 19209 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 683360 0 0 0 87465 127 0 0 25 0 5 0 594832273 5823848448 1363522 33554432000 4194304 4338100 140734631482624 18446744073709551615 4263856 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=19195/tid=19210] ppid=19193 vsize=5687352 CPUtime=880.71 cores=1,3,5,7
/proc/19195/task/19210/stat : 19210 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 137509 0 0 0 87066 1005 0 0 25 0 5 0 594832273 5823848448 1363522 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3519.56
Current children cumulated vsize (KiB) 5687352

[startup+942.301 s]
/proc/loadavg: 8.16 8.03 7.59 9/186 19244
/proc/meminfo: memFree=94148/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=5859392 CPUtime=3759.2 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) S 19193 19195 18020 0 -1 4202496 1407932 0 0 0 373664 2256 0 0 25 0 5 0 594832115 6000017408 1407092 33554432000 4194304 4338100 140734631482624 18446744073709551615 216746981349 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/19195/statm: 1464848 1407092 176 36 0 1462753 0
[pid=19195/tid=19207] ppid=19193 vsize=5859392 CPUtime=940.59 cores=1,3,5,7
/proc/19195/task/19207/stat : 19207 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 377852 0 0 0 93982 77 0 0 25 0 5 0 594832273 6000017408 1407092 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=19195/tid=19208] ppid=19193 vsize=5859392 CPUtime=940.71 cores=1,3,5,7
/proc/19195/task/19208/stat : 19208 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 137793 0 0 0 93039 1032 0 0 25 0 5 0 594832273 6000017408 1407092 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=19195/tid=19209] ppid=19193 vsize=5859392 CPUtime=935.6 cores=1,3,5,7
/proc/19195/task/19209/stat : 19209 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 717262 0 0 0 93426 134 0 0 25 0 5 0 594832273 6000017408 1407092 33554432000 4194304 4338100 140734631482624 18446744073709551615 216734573648 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=19195/tid=19210] ppid=19193 vsize=5859392 CPUtime=940.7 cores=1,3,5,7
/proc/19195/task/19210/stat : 19210 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 137560 0 0 0 93063 1007 0 0 25 0 5 0 594832273 6000017408 1407092 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3759.2
Current children cumulated vsize (KiB) 5859392

[startup+1002.3 s]
/proc/loadavg: 8.21 8.07 7.63 9/186 19246
/proc/meminfo: memFree=96672/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=5960344 CPUtime=3998.85 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) S 19193 19195 18020 0 -1 4202496 1431203 0 0 0 397617 2268 0 0 25 0 5 0 594832115 6103392256 1430363 33554432000 4194304 4338100 140734631482624 18446744073709551615 216746981349 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/19195/statm: 1490086 1430363 176 36 0 1487991 0
[pid=19195/tid=19207] ppid=19193 vsize=5960344 CPUtime=1000.56 cores=1,3,5,7
/proc/19195/task/19207/stat : 19207 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 390626 0 0 0 99975 81 0 0 25 0 5 0 594832273 6103392256 1430363 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=19195/tid=19208] ppid=19193 vsize=5960344 CPUtime=1000.71 cores=1,3,5,7
/proc/19195/task/19208/stat : 19208 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 138112 0 0 0 99036 1035 0 0 25 0 5 0 594832273 6103392256 1430363 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=19195/tid=19209] ppid=19193 vsize=5960344 CPUtime=995.27 cores=1,3,5,7
/proc/19195/task/19209/stat : 19209 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 726254 0 0 0 99391 136 0 0 25 0 5 0 594832273 6103392256 1430363 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=19195/tid=19210] ppid=19193 vsize=5960344 CPUtime=1000.71 cores=1,3,5,7
/proc/19195/task/19210/stat : 19210 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 138746 0 0 0 99061 1010 0 0 25 0 5 0 594832273 6103392256 1430363 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3998.85
Current children cumulated vsize (KiB) 5960344

[startup+1062.3 s]
/proc/loadavg: 8.15 8.07 7.65 9/186 19248
/proc/meminfo: memFree=95356/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=6045656 CPUtime=4238.51 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) S 19193 19195 18020 0 -1 4202496 1451051 0 0 0 421572 2279 0 0 25 0 5 0 594832115 6190751744 1450211 33554432000 4194304 4338100 140734631482624 18446744073709551615 216746981349 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/19195/statm: 1511414 1450211 176 36 0 1509319 0
[pid=19195/tid=19207] ppid=19193 vsize=6045656 CPUtime=1060.54 cores=1,3,5,7
/proc/19195/task/19207/stat : 19207 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 396361 0 0 0 105970 84 0 0 25 0 5 0 594832273 6190751744 1450211 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=19195/tid=19208] ppid=19193 vsize=6045656 CPUtime=1060.71 cores=1,3,5,7
/proc/19195/task/19208/stat : 19208 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 138190 0 0 0 105034 1037 0 0 25 0 5 0 594832273 6190751744 1450211 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=19195/tid=19209] ppid=19193 vsize=6045656 CPUtime=1054.94 cores=1,3,5,7
/proc/19195/task/19209/stat : 19209 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 739603 0 0 0 105355 139 0 0 25 0 5 0 594832273 6190751744 1450211 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=19195/tid=19210] ppid=19193 vsize=6045656 CPUtime=1060.71 cores=1,3,5,7
/proc/19195/task/19210/stat : 19210 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 139432 0 0 0 105059 1012 0 0 25 0 5 0 594832273 6190751744 1450211 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4238.51
Current children cumulated vsize (KiB) 6045656

[startup+1122.3 s]
/proc/loadavg: 8.12 8.07 7.67 9/186 19249
/proc/meminfo: memFree=99820/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=6105484 CPUtime=4478.17 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) S 19193 19195 18020 0 -1 4202496 1469984 0 0 0 445530 2287 0 0 25 0 5 0 594832115 6252015616 1469144 33554432000 4194304 4338100 140734631482624 18446744073709551615 216746981349 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/19195/statm: 1526371 1469144 176 36 0 1524276 0
[pid=19195/tid=19207] ppid=19193 vsize=6105484 CPUtime=1120.53 cores=1,3,5,7
/proc/19195/task/19207/stat : 19207 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 406834 0 0 0 111966 87 0 0 25 0 5 0 594832273 6252015616 1469144 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=19195/tid=19208] ppid=19193 vsize=6105484 CPUtime=1120.71 cores=1,3,5,7
/proc/19195/task/19208/stat : 19208 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 140065 0 0 0 111032 1039 0 0 25 0 5 0 594832273 6252015616 1469144 33554432000 4194304 4338100 140734631482624 18446744073709551615 4245202 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=19195/tid=19209] ppid=19193 vsize=6105484 CPUtime=1114.61 cores=1,3,5,7
/proc/19195/task/19209/stat : 19209 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 744584 0 0 0 111321 140 0 0 25 0 5 0 594832273 6252015616 1469144 33554432000 4194304 4338100 140734631482624 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=19195/tid=19210] ppid=19193 vsize=6105484 CPUtime=1120.71 cores=1,3,5,7
/proc/19195/task/19210/stat : 19210 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 141036 0 0 0 111057 1014 0 0 25 0 5 0 594832273 6252015616 1469144 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4478.17
Current children cumulated vsize (KiB) 6105484

[startup+1182.3 s]
/proc/loadavg: 8.14 8.08 7.70 9/186 19251
/proc/meminfo: memFree=95864/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=6208972 CPUtime=4717.83 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) S 19193 19195 18020 0 -1 4202496 1495792 0 0 0 469418 2365 0 0 25 0 5 0 594832115 6357987328 1494952 33554432000 4194304 4338100 140734631482624 18446744073709551615 216746981349 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/19195/statm: 1552243 1494952 176 36 0 1550148 0
[pid=19195/tid=19207] ppid=19193 vsize=6208972 CPUtime=1180.52 cores=1,3,5,7
/proc/19195/task/19207/stat : 19207 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 414430 0 0 0 117962 90 0 0 25 0 5 0 594832273 6357987328 1494952 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=19195/tid=19208] ppid=19193 vsize=6208972 CPUtime=1180.71 cores=1,3,5,7
/proc/19195/task/19208/stat : 19208 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 140126 0 0 0 116995 1076 0 0 25 0 5 0 594832273 6357987328 1494952 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=19195/tid=19209] ppid=19193 vsize=6208972 CPUtime=1174.29 cores=1,3,5,7
/proc/19195/task/19209/stat : 19209 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 762673 0 0 0 117285 144 0 0 25 0 5 0 594832273 6357987328 1494952 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=19195/tid=19210] ppid=19193 vsize=6208972 CPUtime=1180.71 cores=1,3,5,7
/proc/19195/task/19210/stat : 19210 (ttsth-5-4) R 19193 19195 18020 0 -1 4202560 141098 0 0 0 117022 1049 0 0 25 0 5 0 594832273 6357987328 1494952 33554432000 4194304 4338100 140734631482624 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4717.83
Current children cumulated vsize (KiB) 6208972



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.9 s]
/proc/loadavg: 8.10 8.08 7.71 6/183 19251
/proc/meminfo: memFree=676072/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=0 CPUtime=4800.05 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) Z 19193 19195 18020 0 -1 4203548 1509719 0 0 0 477591 2414 0 0 18 0 2 0 594832115 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/19195/statm: 0 0 0 0 0 0 0
[pid=19195/tid=19208] ppid=19193 vsize=0 CPUtime=1201.31 cores=1,3,5,7
/proc/19195/task/19208/stat : 19208 (ttsth-5-4) R 19193 19195 18020 0 -1 4203588 140154 0 0 0 119031 1100 0 0 25 0 2 0 594832273 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.05
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.01 s]
/proc/loadavg: 8.10 8.08 7.71 6/183 19251
/proc/meminfo: memFree=676072/32951124 swapFree=67102140/67111528
[pid=19195] ppid=19193 vsize=0 CPUtime=4800.15 cores=1,3,5,7
/proc/19195/stat : 19195 (ttsth-5-4) Z 19193 19195 18020 0 -1 4203548 1509719 0 0 0 477591 2424 0 0 18 0 2 0 594832115 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/19195/statm: 0 0 0 0 0 0 0
[pid=19195/tid=19208] ppid=19193 vsize=0 CPUtime=1201.42 cores=1,3,5,7
/proc/19195/task/19208/stat : 19208 (ttsth-5-4) R 19193 19195 18020 0 -1 4203588 140154 0 0 0 119031 1111 0 0 25 0 2 0 594832273 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.15
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1203.09
CPU time (s): 4800.24
CPU user time (s): 4775.91
CPU system time (s): 24.3233
CPU usage (%): 398.992
Max. virtual memory (cumulated for all children) (KiB): 6262280

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4775.91
system time used= 24.3233
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1509719
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 4444
involuntary context switches= 33947

runsolver used 2.05569 second user time and 4.51531 second system time

The end

Launcher Data 

Begin job on node119 at 2011-04-29 05:00:15
IDJOB=3257382
IDBENCH=70910
IDSOLVER=1654
FILE ID=node119/3257382-1304046015
RUNJOBID= node119-1304046014-19182
PBS_JOBID= 13170378
Free space on /tmp= 71816 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/edgematching/compact/em_11_3_4_cmp.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3257382-1304046015/watcher-3257382-1304046015 -o /tmp/evaluation-result-3257382-1304046015/solver-3257382-1304046015 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3257382-1304046015.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 3b5e80540ef2fcb09cfe50ac7e3686c1
RANDOM SEED=1813970973

node119.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.63
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      11754024 kB
Buffers:       2243984 kB
Cached:       18305056 kB
SwapCached:       5504 kB
Active:        9532720 kB
Inactive:     11080852 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      11754024 kB
SwapTotal:    67111528 kB
SwapFree:     67102140 kB
Dirty:            2012 kB
Writeback:           0 kB
AnonPages:       61648 kB
Mapped:          14264 kB
Slab:           520328 kB
PageTables:       4728 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   191532 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71812 MiB
End job on node119 at 2011-04-29 05:20:20