Trace number 3251088

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.39 1204.61

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m47-q742.5-n45-p15-s28972035.cnf
MD5SUMbfa662ff57088c54e0febd091ce11e5c
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark2.39763
Satisfiable
(Un)Satisfiability was proved
Number of variables1985
Number of clauses409171
Sum of the clauses size820237
Maximum clause length47
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2409126
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 545

Solver Data 

0.19/0.27	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251088-1304106237.cnf 
0.19/0.27	c Using SATTIMEOUT of 4800
0.19/0.27	c Using SATRAM of 15500
0.19/0.27	c Size of subset node = 32
0.19/0.27	c Variables = 1985
0.19/0.27	c Clauses = 409171
0.19/0.27	c Literals = 820237
0.19/0.27	c Reduced size:
0.19/0.27	c Variables = 1985
0.19/0.27	c Clauses = 409171
0.19/0.27	c Literals = 820237
0.19/0.27	c a2
0.39/0.49	c a1
1.19/1.22	c Independent sub-problems = 1
1.19/1.22	c Sub-problem with 1985 variables and 409171 clauses
1.19/1.22	c using permutation method 4
1.19/1.22	c count = 1
1.19/1.22	c Ordering finished 1.170 seconds
1.19/1.22	c mapped 1.220 seconds
1.19/1.22	c permuted 1.220 seconds
1.19/1.22	c about to sort
1.19/1.26	c sorted
2.59/2.66	c clauses now 409171
2.59/2.66	c init finished 1.350 seconds
2.59/2.66	c tree built 2.650 seconds
2.59/2.66	c 
2.59/2.66	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
2.59/2.66	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
2.59/2.66	c ................................ | ..........                    |                                  2.6         0.0             
2.59/2.66	c ................................ | ..........                  1 |                                  2.6         0.4             
2.59/2.66	c ................................ | ..........                  2 |                                  2.6         0.8             
2.59/2.66	c ................................ | ..........                  3 |                                  2.6         1.1             
2.59/2.66	c ................................ | ..........                  4 |                                  2.6         1.5             
2.59/2.66	c ................................ | ..........                  5 |                                  2.6         1.9             
2.59/2.66	c ................................ | ..........               1000 |              56                  2.7       375.9         2613
95.87/26.03	c ................................ | ..........            5262000 | 2508         516246             96.0     54812.5     15295834
191.72/50.06	c ................................ | ..........           10227000 | 5414         1012652           192.0     53265.6     25816055
287.97/74.10	c ................................ | ..........           15072000 | 8897         1498240           288.0     52333.3     37463957
383.81/98.15	c ................................ | ..........           19632000 | 10910        1972456           384.0     51125.0     49654606
479.65/122.19	c ................................ | *.........           24438000 | 12905        2435661           480.0     50911.4     53712945
575.90/146.23	c ................................ | *.........           29209000 | 14129        2889511           576.0     50709.2     60591991
671.74/170.26	c ................................ | *.........           34003000 | 15935        3343272           672.0     50599.7     72277974
767.95/194.31	c ................................ | *.........           38692000 | 20101        3780892           768.0     50380.2     78139801
863.82/218.34	c ................................ | *.........           43395000 | 22116        4234469           864.0     50225.7     85268654
959.63/242.39	c ................................ | **........           47753000 | 25009        4662403           960.0     49742.7     88958694
1055.89/266.43	c ................................ | **........           52383000 | 28843        5104778          1056.0     49604.2     95985174
1151.75/290.46	c ................................ | **........           56906000 | 31373        5546257          1152.0     49397.6    101686463
1247.98/314.50	c ................................ | **........           61447000 | 33157        5989508          1248.0     49236.4    110294564
1343.77/338.55	c ................................ | **........           65938000 | 34975        6428019          1344.0     49061.0    121380967
1440.01/362.60	c ................................ | ***.......           70491000 | 38413        6860688          1440.0     48951.7    130797789
1535.80/386.65	c ................................ | ***.......           74888000 | 40391        7279394          1536.0     48755.2    134778478
1631.61/410.69	c ................................ | ***.......           79385000 | 42795        7711150          1632.0     48642.5    147089705
1727.86/434.74	c ................................ | ***.......           83669000 | 44997        8135486          1728.0     48419.3    152507934
1823.69/458.78	c ................................ | ***.......           88027000 | 46541        8568349          1824.0     48260.4    159055020
1919.88/482.84	c ................................ | ****......           92104000 | 48539        8980804          1920.0     47970.6    160840603
2016.48/507.09	c ................................ | ****......           96210000 | 51742        9397143          2016.8     47704.0    164277410
2111.87/530.93	c ................................ | ****......          100145000 | 53273        9801260          2112.0     47416.7    167547167
2207.68/554.98	c ................................ | ****......          104164000 | 54518        10204852         2208.0     47175.3    173602710
2303.94/579.01	c ................................ | ****......          108348000 | 59304        10626645         2304.0     47025.8    179819771
2399.77/603.05	c ................................ | *****.....          112581000 | 61472        11033969         2400.0     46908.8    186397158
2495.99/627.10	c ................................ | *****.....          116842000 | 64212        11448331         2496.0     46811.5    194330193
2591.84/651.14	c ................................ | *****.....          120948000 | 66517        11859179         2592.0     46661.9    202873122
2687.63/675.19	c ................................ | *****.....          124893000 | 69424        12259185         2688.0     46463.2    206244774
2783.86/699.23	c ................................ | *****.....          128934000 | 73251        12670701         2784.0     46312.3    210829169
2879.65/723.28	c ................................ | ******....          132863000 | 78819        13071444         2880.0     46133.0    215408660
2975.86/747.33	c ................................ | ******....          136797000 | 80141        13472473         2976.0     45966.7    218659016
3072.11/771.44	c ................................ | ******....          140840000 | 81718        13877373         3072.3     45842.0    227253403
3167.94/795.41	c ................................ | ******....          144775000 | 83808        14274166         3168.0     45699.2    234289755
3263.70/819.48	c ................................ | ******....          148397000 | 85217        14657155         3264.0     45464.8    234289755
3359.90/843.52	c ................................ | *******...          152263000 | 86820        15045338         3360.0     45316.2    237568049
3455.75/867.56	c ................................ | *******...          156128000 | 88323        15440474         3456.0     45175.8    246132689
3551.97/891.61	c ................................ | *******...          159818000 | 89616        15835341         3552.0     44993.7    250802968
3647.80/915.65	c ................................ | *******...          163630000 | 90583        16232060         3648.0     44854.7    254931404
3743.98/939.70	c ................................ | *******...          167420000 | 91493        16628551         3744.0     44716.8    261541535
3839.78/963.76	c ................................ | ********..          171099000 | 92576        17019221         3840.0     44556.8    268090528
3935.63/987.80	c ................................ | ********..          174857000 | 94814        17427379         3936.0     44424.9    271467555
4031.84/1011.84	c ................................ | ********..          179463000 | 97268        17852583         4032.0     44509.7    279003879
4127.66/1035.88	c ................................ | ********..          183910000 | 101201       18272116         4128.0     44551.8    284914453
4223.89/1059.93	c ................................ | ********..          188104000 | 104759       18676240         4224.0     44532.2    289998994
4319.65/1083.98	c ................................ | *********.          192004000 | 107548       19074538         4320.0     44445.4    290263217
4415.85/1108.04	c ................................ | *********.          196342000 | 109481       19481186         4416.0     44461.3    294841570
4511.68/1132.08	c ................................ | *********.          200625000 | 112524       19885057         4512.0     44464.7    303436965
4607.94/1156.12	c ................................ | *********.          204447000 | 115797       20269571         4608.0     44367.7    304409748
4703.69/1180.17	c ................................ | *********.          208741000 | 117780       20660111         4704.0     44375.2    309535978
4799.90/1204.22	c CPU limit (4800) reached
4799.90/1204.22	c Time = 4800.00 seconds
4799.90/1204.22	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3251088-1304106237/watcher-3251088-1304106237 -o /tmp/evaluation-result-3251088-1304106237/solver-3251088-1304106237 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251088-1304106237.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.12 7.84 7.94 2/174 17622
/proc/meminfo: memFree=30904152/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=9768 CPUtime=0 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) R 17620 17622 17262 0 -1 4202496 487 0 0 0 0 0 0 0 22 0 1 0 522222460 10002432 429 33554432000 4194304 4338100 140736873902832 18446744073709551615 251813937360 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17622/statm: 2442 432 147 36 0 347 0

[startup+0.043214 s]
/proc/loadavg: 7.12 7.84 7.94 2/174 17622
/proc/meminfo: memFree=30904152/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=17224 CPUtime=0.04 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) R 17620 17622 17262 0 -1 4202496 2350 0 0 0 4 0 0 0 22 0 1 0 522222460 17637376 2292 33554432000 4194304 4338100 140736873902832 18446744073709551615 251813900997 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17622/statm: 4306 2293 148 36 0 2211 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 17224

[startup+0.10028 s]
/proc/loadavg: 7.12 7.84 7.94 2/174 17622
/proc/meminfo: memFree=30904152/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=28816 CPUtime=0.09 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) R 17620 17622 17262 0 -1 4202496 5191 0 0 0 9 0 0 0 22 0 1 0 522222460 29507584 5133 33554432000 4194304 4338100 140736873902832 18446744073709551615 251813900997 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17622/statm: 7204 5135 148 36 0 5109 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 28816

[startup+0.300233 s]
/proc/loadavg: 7.12 7.84 7.94 2/174 17622
/proc/meminfo: memFree=30904152/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=71424 CPUtime=0.29 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) R 17620 17622 17262 0 -1 4202496 15585 0 0 0 28 1 0 0 23 0 1 0 522222460 73138176 15527 33554432000 4194304 4338100 140736873902832 18446744073709551615 251813936907 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17622/statm: 17856 15531 151 36 0 15761 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 71424

[startup+0.700167 s]
/proc/loadavg: 7.12 7.84 7.94 2/174 17622
/proc/meminfo: memFree=30904152/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=81292 CPUtime=0.69 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) R 17620 17622 17262 0 -1 4202496 23661 0 0 0 67 2 0 0 25 0 1 0 522222460 83243008 18305 33554432000 4194304 4338100 140736873902832 18446744073709551615 251813900997 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17622/statm: 20323 18306 155 36 0 18228 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 81292

[startup+1.50097 s]
/proc/loadavg: 7.12 7.84 7.94 3/178 17629
/proc/meminfo: memFree=30638024/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=114196 CPUtime=1.5 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) R 17620 17622 17262 0 -1 4202496 32260 0 0 0 147 3 0 0 25 0 1 0 522222460 116936704 26104 33554432000 4194304 4338100 140736873902832 18446744073709551615 251813900997 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17622/statm: 28549 26111 164 36 0 26454 0
Current children cumulated CPU time (s) 1.5
Current children cumulated vsize (KiB) 114196

[startup+3.10071 s]
/proc/loadavg: 6.71 7.74 7.91 3/178 17629
/proc/meminfo: memFree=30567964/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=351348 CPUtime=4.4 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) S 17620 17622 17262 0 -1 4202496 64919 0 0 0 428 12 0 0 25 0 5 0 522222460 359780352 57962 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17622/statm: 87837 57964 176 36 0 85742 0
Current children cumulated CPU time (s) 4.4
Current children cumulated vsize (KiB) 351348

[startup+6.30117 s]
/proc/loadavg: 6.71 7.74 7.91 6/182 17633
/proc/meminfo: memFree=30290256/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=499972 CPUtime=17.17 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) S 17620 17622 17262 0 -1 4202496 87452 0 0 0 1672 45 0 0 25 0 5 0 522222460 511971328 80495 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17622/statm: 124993 80495 176 36 0 122898 0
[pid=17622/tid=17630] ppid=17620 vsize=499972 CPUtime=3.63 cores=1,3,5,7
/proc/17622/task/17630/stat : 17630 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 9213 0 0 0 360 3 0 0 25 0 5 0 522222726 511971328 80495 33554432000 4194304 4338100 140736873902832 18446744073709551615 251813635506 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17622/tid=17631] ppid=17620 vsize=499972 CPUtime=3.62 cores=1,3,5,7
/proc/17622/task/17631/stat : 17631 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 4087 0 0 0 346 16 0 0 25 0 5 0 522222726 511971328 80495 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17622/tid=17632] ppid=17620 vsize=499972 CPUtime=3.61 cores=1,3,5,7
/proc/17622/task/17632/stat : 17632 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 8136 0 0 0 360 1 0 0 25 0 5 0 522222726 511971328 80495 33554432000 4194304 4338100 140736873902832 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17622/tid=17633] ppid=17620 vsize=499972 CPUtime=3.62 cores=1,3,5,7
/proc/17622/task/17633/stat : 17633 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 5056 0 0 0 345 17 0 0 25 0 5 0 522222726 511971328 80495 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 17.17
Current children cumulated vsize (KiB) 499972

[startup+12.7011 s]
/proc/loadavg: 6.68 7.70 7.89 9/186 17637
/proc/meminfo: memFree=29849620/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=636584 CPUtime=42.74 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) S 17620 17622 17262 0 -1 4202496 129778 0 0 0 4129 145 0 0 25 0 5 0 522222460 651862016 122821 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17622/statm: 159146 122821 176 36 0 157051 0
[pid=17622/tid=17630] ppid=17620 vsize=636584 CPUtime=10.03 cores=1,3,5,7
/proc/17622/task/17630/stat : 17630 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 24850 0 0 0 995 8 0 0 25 0 5 0 522222726 651862016 122821 33554432000 4194304 4338100 140736873902832 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17622/tid=17631] ppid=17620 vsize=636584 CPUtime=10.03 cores=1,3,5,7
/proc/17622/task/17631/stat : 17631 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 9957 0 0 0 939 64 0 0 21 0 5 0 522222726 651862016 122821 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17622/tid=17632] ppid=17620 vsize=636584 CPUtime=9.97 cores=1,3,5,7
/proc/17622/task/17632/stat : 17632 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 20862 0 0 0 994 3 0 0 25 0 5 0 522222726 651862016 122821 33554432000 4194304 4338100 140736873902832 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17622/tid=17633] ppid=17620 vsize=636584 CPUtime=10.02 cores=1,3,5,7
/proc/17622/task/17633/stat : 17633 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 13149 0 0 0 940 62 0 0 25 0 5 0 522222726 651862016 122821 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 42.74
Current children cumulated vsize (KiB) 636584

[startup+25.5009 s]
/proc/loadavg: 6.89 7.71 7.89 9/186 17637
/proc/meminfo: memFree=29395884/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=830368 CPUtime=93.87 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) S 17620 17622 17262 0 -1 4202496 183298 0 0 0 9034 353 0 0 25 0 5 0 522222460 850296832 176341 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17622/statm: 207592 176341 176 36 0 205497 0
[pid=17622/tid=17630] ppid=17620 vsize=830368 CPUtime=22.83 cores=1,3,5,7
/proc/17622/task/17630/stat : 17630 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 47946 0 0 0 2266 17 0 0 25 0 5 0 522222726 850296832 176341 33554432000 4194304 4338100 140736873902832 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17622/tid=17631] ppid=17620 vsize=830368 CPUtime=22.83 cores=1,3,5,7
/proc/17622/task/17631/stat : 17631 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 12666 0 0 0 2120 163 0 0 25 0 5 0 522222726 850296832 176341 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17622/tid=17632] ppid=17620 vsize=830368 CPUtime=22.7 cores=1,3,5,7
/proc/17622/task/17632/stat : 17632 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 41785 0 0 0 2262 8 0 0 25 0 5 0 522222726 850296832 176341 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826052814 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17622/tid=17633] ppid=17620 vsize=830368 CPUtime=22.82 cores=1,3,5,7
/proc/17622/task/17633/stat : 17633 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 19941 0 0 0 2125 157 0 0 25 0 5 0 522222726 850296832 176341 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 93.87
Current children cumulated vsize (KiB) 830368

[startup+51.1007 s]
/proc/loadavg: 7.33 7.75 7.90 9/186 17638
/proc/meminfo: memFree=28589060/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=1252972 CPUtime=196.12 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) S 17620 17622 17262 0 -1 4202496 274166 0 0 0 18843 769 0 0 25 0 5 0 522222460 1283043328 267209 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17622/statm: 313243 267209 176 36 0 311148 0
[pid=17622/tid=17630] ppid=17620 vsize=1252972 CPUtime=48.29 cores=1,3,5,7
/proc/17622/task/17630/stat : 17630 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 90946 0 0 0 4799 30 0 0 25 0 5 0 522222726 1283043328 267209 33554432000 4194304 4338100 140736873902832 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17622/tid=17631] ppid=17620 vsize=1252972 CPUtime=48.42 cores=1,3,5,7
/proc/17622/task/17631/stat : 17631 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 17222 0 0 0 4479 363 0 0 25 0 5 0 522222726 1283043328 267209 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17622/tid=17632] ppid=17620 vsize=1252972 CPUtime=48.29 cores=1,3,5,7
/proc/17622/task/17632/stat : 17632 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 76173 0 0 0 4815 14 0 0 25 0 5 0 522222726 1283043328 267209 33554432000 4194304 4338100 140736873902832 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17622/tid=17633] ppid=17620 vsize=1252972 CPUtime=48.43 cores=1,3,5,7
/proc/17622/task/17633/stat : 17633 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 28865 0 0 0 4489 354 0 0 25 0 5 0 522222726 1283043328 267209 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 196.12
Current children cumulated vsize (KiB) 1252972

[startup+102.306 s]
/proc/loadavg: 7.80 7.80 7.91 9/186 17639
/proc/meminfo: memFree=27137372/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=2130664 CPUtime=400.58 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) S 17620 17622 17262 0 -1 4202496 476922 0 0 0 38434 1624 0 0 25 0 5 0 522222460 2181799936 469965 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17622/statm: 532666 469965 176 36 0 530571 0
[pid=17622/tid=17630] ppid=17620 vsize=2130664 CPUtime=99.23 cores=1,3,5,7
/proc/17622/task/17630/stat : 17630 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 183774 0 0 0 9859 64 0 0 25 0 5 0 522222726 2181799936 469965 33554432000 4194304 4338100 140736873902832 18446744073709551615 4241252 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17622/tid=17631] ppid=17620 vsize=2130664 CPUtime=99.6 cores=1,3,5,7
/proc/17622/task/17631/stat : 17631 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 30139 0 0 0 9186 774 0 0 25 0 5 0 522222726 2181799936 469965 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17622/tid=17632] ppid=17620 vsize=2130664 CPUtime=99.45 cores=1,3,5,7
/proc/17622/task/17632/stat : 17632 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 145652 0 0 0 9916 29 0 0 25 0 5 0 522222726 2181799936 469965 33554432000 4194304 4338100 140736873902832 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17622/tid=17633] ppid=17620 vsize=2130664 CPUtime=99.61 cores=1,3,5,7
/proc/17622/task/17633/stat : 17633 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 56397 0 0 0 9212 749 0 0 25 0 5 0 522222726 2181799936 469965 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 400.58
Current children cumulated vsize (KiB) 2130664

[startup+162.301 s]
/proc/loadavg: 7.99 7.85 7.92 9/186 17641
/proc/meminfo: memFree=25988724/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=2728704 CPUtime=640.18 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) S 17620 17622 17262 0 -1 4202496 620802 0 0 0 61413 2605 0 0 25 0 5 0 522222460 2794192896 613845 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17622/statm: 682176 613845 176 36 0 680081 0
[pid=17622/tid=17630] ppid=17620 vsize=2728704 CPUtime=159.21 cores=1,3,5,7
/proc/17622/task/17630/stat : 17630 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 271140 0 0 0 15826 95 0 0 25 0 5 0 522222726 2794192896 613845 33554432000 4194304 4338100 140736873902832 18446744073709551615 4256674 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17622/tid=17631] ppid=17620 vsize=2728704 CPUtime=159.59 cores=1,3,5,7
/proc/17622/task/17631/stat : 17631 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 31223 0 0 0 14711 1248 0 0 25 0 5 0 522222726 2794192896 613845 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17622/tid=17632] ppid=17620 vsize=2728704 CPUtime=159.11 cores=1,3,5,7
/proc/17622/task/17632/stat : 17632 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 201067 0 0 0 15870 41 0 0 25 0 5 0 522222726 2794192896 613845 33554432000 4194304 4338100 140736873902832 18446744073709551615 4240496 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17622/tid=17633] ppid=17620 vsize=2728704 CPUtime=159.58 cores=1,3,5,7
/proc/17622/task/17633/stat : 17633 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 56412 0 0 0 14745 1213 0 0 25 0 5 0 522222726 2794192896 613845 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 640.18
Current children cumulated vsize (KiB) 2728704

[startup+222.301 s]
/proc/loadavg: 8.08 7.91 7.93 9/186 17643
/proc/meminfo: memFree=24989264/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=3324968 CPUtime=879.78 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) S 17620 17622 17262 0 -1 4202496 768063 0 0 0 84403 3575 0 0 25 0 5 0 522222460 3404767232 761106 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17622/statm: 831242 761106 176 36 0 829147 0
[pid=17622/tid=17630] ppid=17620 vsize=3324968 CPUtime=219.19 cores=1,3,5,7
/proc/17622/task/17630/stat : 17630 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 354396 0 0 0 21791 128 0 0 25 0 5 0 522222726 3404767232 761106 33554432000 4194304 4338100 140736873902832 18446744073709551615 251813635540 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17622/tid=17631] ppid=17620 vsize=3324968 CPUtime=219.56 cores=1,3,5,7
/proc/17622/task/17631/stat : 17631 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 31265 0 0 0 20240 1716 0 0 25 0 5 0 522222726 3404767232 761106 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17622/tid=17632] ppid=17620 vsize=3324968 CPUtime=218.77 cores=1,3,5,7
/proc/17622/task/17632/stat : 17632 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 256811 0 0 0 21827 50 0 0 25 0 5 0 522222726 3404767232 761107 33554432000 4194304 4338100 140736873902832 18446744073709551615 251813635088 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17622/tid=17633] ppid=17620 vsize=3324968 CPUtime=219.56 cores=1,3,5,7
/proc/17622/task/17633/stat : 17633 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 64631 0 0 0 20284 1672 0 0 25 0 5 0 522222726 3404767232 761107 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 879.78
Current children cumulated vsize (KiB) 3324968

################
# More data... #
################

/proc/17622/task/17633/stat : 17633 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 102881 0 0 0 64750 5120 0 0 25 0 5 0 522222726 7918149632 1793464 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 2796.23
Current children cumulated vsize (KiB) 7732568

[startup+762.306 s]
/proc/loadavg: 8.08 8.04 7.97 9/186 17657
/proc/meminfo: memFree=17060068/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=8068352 CPUtime=3035.79 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) S 17620 17622 17262 0 -1 4202496 1910551 0 0 0 291775 11804 0 0 25 0 5 0 522222460 8261992448 1903594 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17622/statm: 2017088 1903595 176 36 0 2014993 0
[pid=17622/tid=17630] ppid=17620 vsize=8068352 CPUtime=758.4 cores=1,3,5,7
/proc/17622/task/17630/stat : 17630 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 1020543 0 0 0 75440 400 0 0 25 0 5 0 522222726 8261992448 1903595 33554432000 4194304 4338100 140736873902832 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17622/tid=17631] ppid=17620 vsize=8068352 CPUtime=758.94 cores=1,3,5,7
/proc/17622/task/17631/stat : 17631 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 62505 0 0 0 70185 5709 0 0 25 0 5 0 522222726 8261992448 1903595 33554432000 4194304 4338100 140736873902832 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17622/tid=17632] ppid=17620 vsize=8068352 CPUtime=757.08 cores=1,3,5,7
/proc/17622/task/17632/stat : 17632 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 656628 0 0 0 75574 134 0 0 25 0 5 0 522222726 8261992448 1903595 33554432000 4194304 4338100 140736873902832 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17622/tid=17633] ppid=17620 vsize=8068352 CPUtime=758.68 cores=1,3,5,7
/proc/17622/task/17633/stat : 17633 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 109916 0 0 0 70316 5552 0 0 25 0 5 0 522222726 8261992448 1903595 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3035.79
Current children cumulated vsize (KiB) 8068352

[startup+822.301 s]
/proc/loadavg: 8.14 8.06 7.98 9/186 17659
/proc/meminfo: memFree=16251800/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=8558208 CPUtime=3275.25 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) S 17620 17622 17262 0 -1 4202496 1999964 0 0 0 314809 12716 0 0 25 0 5 0 522222460 8763604992 1993007 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17622/statm: 2139552 1993007 176 36 0 2137457 0
[pid=17622/tid=17630] ppid=17620 vsize=8558208 CPUtime=818.37 cores=1,3,5,7
/proc/17622/task/17630/stat : 17630 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 1069991 0 0 0 81413 424 0 0 25 0 5 0 522222726 8763604992 1993007 33554432000 4194304 4338100 140736873902832 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17622/tid=17631] ppid=17620 vsize=8558208 CPUtime=818.86 cores=1,3,5,7
/proc/17622/task/17631/stat : 17631 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 71690 0 0 0 75716 6170 0 0 25 0 5 0 522222726 8763604992 1993007 33554432000 4194304 4338100 140736873902832 18446744073709551615 4229573 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17622/tid=17632] ppid=17620 vsize=8558208 CPUtime=816.83 cores=1,3,5,7
/proc/17622/task/17632/stat : 17632 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 682406 0 0 0 81543 140 0 0 25 0 5 0 522222726 8763604992 1993007 33554432000 4194304 4338100 140736873902832 18446744073709551615 4235847 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17622/tid=17633] ppid=17620 vsize=8558208 CPUtime=818.52 cores=1,3,5,7
/proc/17622/task/17633/stat : 17633 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 114917 0 0 0 75877 5975 0 0 25 0 5 0 522222726 8763604992 1993007 33554432000 4194304 4338100 140736873902832 18446744073709551615 4229521 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3275.25
Current children cumulated vsize (KiB) 8558208

[startup+882.301 s]
/proc/loadavg: 8.05 8.05 7.98 9/186 17660
/proc/meminfo: memFree=15173844/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=9100368 CPUtime=3514.84 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) S 17620 17622 17262 0 -1 4202496 2140938 0 0 0 337886 13598 0 0 25 0 5 0 522222460 9318776832 2133981 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17622/statm: 2275092 2133981 176 36 0 2272997 0
[pid=17622/tid=17630] ppid=17620 vsize=9100368 CPUtime=878.34 cores=1,3,5,7
/proc/17622/task/17630/stat : 17630 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 1151262 0 0 0 87382 452 0 0 25 0 5 0 522222726 9318776832 2133981 33554432000 4194304 4338100 140736873902832 18446744073709551615 4253990 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17622/tid=17631] ppid=17620 vsize=9100368 CPUtime=878.8 cores=1,3,5,7
/proc/17622/task/17631/stat : 17631 (ttsth-5-4) S 17620 17622 17262 0 -1 4202560 78180 0 0 0 81287 6593 0 0 25 0 5 0 522222726 9318776832 2133981 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826065780 0 0 4096 0 18446604468691298240 0 0 -1 1 0 0 0
[pid=17622/tid=17632] ppid=17620 vsize=9100368 CPUtime=876.81 cores=1,3,5,7
/proc/17622/task/17632/stat : 17632 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 730538 0 0 0 87530 151 0 0 25 0 5 0 522222726 9318776832 2133981 33554432000 4194304 4338100 140736873902832 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17622/tid=17633] ppid=17620 vsize=9100368 CPUtime=878.19 cores=1,3,5,7
/proc/17622/task/17633/stat : 17633 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 119998 0 0 0 81426 6393 0 0 25 0 5 0 522222726 9318776832 2133981 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826052814 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3514.84
Current children cumulated vsize (KiB) 9100368

[startup+942.301 s]
/proc/loadavg: 8.02 8.04 7.98 9/186 17662
/proc/meminfo: memFree=14533608/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=9511456 CPUtime=3754.36 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) S 17620 17622 17262 0 -1 4202496 2232324 0 0 0 360974 14462 0 0 25 0 5 0 522222460 9739730944 2225367 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17622/statm: 2377864 2225368 176 36 0 2375769 0
[pid=17622/tid=17630] ppid=17620 vsize=9511456 CPUtime=938.3 cores=1,3,5,7
/proc/17622/task/17630/stat : 17630 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 1207531 0 0 0 93356 474 0 0 25 0 5 0 522222726 9739730944 2225368 33554432000 4194304 4338100 140736873902832 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17622/tid=17631] ppid=17620 vsize=9511456 CPUtime=938.77 cores=1,3,5,7
/proc/17622/task/17631/stat : 17631 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 80474 0 0 0 86857 7020 0 0 25 0 5 0 522222726 9739730944 2225368 33554432000 4194304 4338100 140736873902832 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17622/tid=17632] ppid=17620 vsize=9511456 CPUtime=936.79 cores=1,3,5,7
/proc/17622/task/17632/stat : 17632 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 761219 0 0 0 93522 157 0 0 25 0 5 0 522222726 9739730944 2225368 33554432000 4194304 4338100 140736873902832 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17622/tid=17633] ppid=17620 vsize=9511456 CPUtime=937.82 cores=1,3,5,7
/proc/17622/task/17633/stat : 17633 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 122141 0 0 0 86979 6803 0 0 25 0 5 0 522222726 9739730944 2225368 33554432000 4194304 4338100 140736873902832 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3754.36
Current children cumulated vsize (KiB) 9511456

[startup+1002.3 s]
/proc/loadavg: 8.00 8.03 7.98 9/186 17663
/proc/meminfo: memFree=13918272/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=10147832 CPUtime=3993.89 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) S 17620 17622 17262 0 -1 4202496 2358199 0 0 0 384042 15347 0 0 25 0 5 0 522222460 10391379968 2351242 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17622/statm: 2536958 2351243 176 36 0 2534863 0
[pid=17622/tid=17630] ppid=17620 vsize=10147832 CPUtime=998.28 cores=1,3,5,7
/proc/17622/task/17630/stat : 17630 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 1279893 0 0 0 99324 504 0 0 25 0 5 0 522222726 10391379968 2351243 33554432000 4194304 4338100 140736873902832 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17622/tid=17631] ppid=17620 vsize=10147832 CPUtime=998.61 cores=1,3,5,7
/proc/17622/task/17631/stat : 17631 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 84658 0 0 0 92419 7442 0 0 25 0 5 0 522222726 10391379968 2351243 33554432000 4194304 4338100 140736873902832 18446744073709551615 251813635111 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17622/tid=17632] ppid=17620 vsize=10147832 CPUtime=996.77 cores=1,3,5,7
/proc/17622/task/17632/stat : 17632 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 808083 0 0 0 99511 166 0 0 25 0 5 0 522222726 10391379968 2351243 33554432000 4194304 4338100 140736873902832 18446744073709551615 251813635297 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17622/tid=17633] ppid=17620 vsize=10147832 CPUtime=997.54 cores=1,3,5,7
/proc/17622/task/17633/stat : 17633 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 124606 0 0 0 92527 7227 0 0 25 0 5 0 522222726 10391379968 2351243 33554432000 4194304 4338100 140736873902832 18446744073709551615 4285802 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3993.89
Current children cumulated vsize (KiB) 10147832

[startup+1062.3 s]
/proc/loadavg: 8.00 8.02 7.98 9/186 17674
/proc/meminfo: memFree=13182108/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=10558088 CPUtime=4233.47 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) S 17620 17622 17262 0 -1 4202496 2472450 0 0 0 407105 16242 0 0 25 0 5 0 522222460 10811482112 2465493 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17622/statm: 2639522 2465493 176 36 0 2637427 0
[pid=17622/tid=17630] ppid=17620 vsize=10558088 CPUtime=1058.25 cores=1,3,5,7
/proc/17622/task/17630/stat : 17630 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 1346170 0 0 0 105295 530 0 0 25 0 5 0 522222726 10811482112 2465493 33554432000 4194304 4338100 140736873902832 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17622/tid=17631] ppid=17620 vsize=10558088 CPUtime=1058.57 cores=1,3,5,7
/proc/17622/task/17631/stat : 17631 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 91199 0 0 0 97982 7875 0 0 25 0 5 0 522222726 10811482112 2465493 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17622/tid=17632] ppid=17620 vsize=10558088 CPUtime=1056.52 cores=1,3,5,7
/proc/17622/task/17632/stat : 17632 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 845077 0 0 0 105478 174 0 0 25 0 5 0 522222726 10811482112 2465493 33554432000 4194304 4338100 140736873902832 18446744073709551615 4256051 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17622/tid=17633] ppid=17620 vsize=10558088 CPUtime=1057.43 cores=1,3,5,7
/proc/17622/task/17633/stat : 17633 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 129044 0 0 0 98089 7654 0 0 25 0 5 0 522222726 10811482112 2465493 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4233.47
Current children cumulated vsize (KiB) 10558088

[startup+1122.3 s]
/proc/loadavg: 8.17 8.06 8.00 9/186 17676
/proc/meminfo: memFree=12511480/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=10913408 CPUtime=4472.96 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) S 17620 17622 17262 0 -1 4202496 2567683 0 0 0 430152 17144 0 0 25 0 5 0 522222460 11175329792 2560726 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17622/statm: 2728352 2560726 176 36 0 2726257 0
[pid=17622/tid=17630] ppid=17620 vsize=10913408 CPUtime=1118.14 cores=1,3,5,7
/proc/17622/task/17630/stat : 17630 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 1404855 0 0 0 111260 554 0 0 25 0 5 0 522222726 11175329792 2560726 33554432000 4194304 4338100 140736873902832 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17622/tid=17631] ppid=17620 vsize=10913408 CPUtime=1118.42 cores=1,3,5,7
/proc/17622/task/17631/stat : 17631 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 98195 0 0 0 103522 8320 0 0 25 0 5 0 522222726 11175329792 2560726 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17622/tid=17632] ppid=17620 vsize=10913408 CPUtime=1116.36 cores=1,3,5,7
/proc/17622/task/17632/stat : 17632 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 869360 0 0 0 111456 180 0 0 25 0 5 0 522222726 11175329792 2560726 33554432000 4194304 4338100 140736873902832 18446744073709551615 251813635111 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17622/tid=17633] ppid=17620 vsize=10913408 CPUtime=1117.36 cores=1,3,5,7
/proc/17622/task/17633/stat : 17633 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 134313 0 0 0 103654 8082 0 0 25 0 5 0 522222726 11175329792 2560726 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4472.96
Current children cumulated vsize (KiB) 10913408

[startup+1182.3 s]
/proc/loadavg: 8.19 8.08 8.00 9/186 17677
/proc/meminfo: memFree=12156932/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=11173888 CPUtime=4712.46 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) S 17620 17622 17262 0 -1 4202496 2647367 0 0 0 453228 18018 0 0 25 0 5 0 522222460 11442061312 2640410 33554432000 4194304 4338100 140736873902832 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17622/statm: 2793472 2640410 176 36 0 2791377 0
[pid=17622/tid=17630] ppid=17620 vsize=11173888 CPUtime=1177.91 cores=1,3,5,7
/proc/17622/task/17630/stat : 17630 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 1458743 0 0 0 117208 583 0 0 25 0 5 0 522222726 11442061312 2640410 33554432000 4194304 4338100 140736873902832 18446744073709551615 251813635137 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17622/tid=17631] ppid=17620 vsize=11173888 CPUtime=1178.24 cores=1,3,5,7
/proc/17622/task/17631/stat : 17631 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 105689 0 0 0 109077 8747 0 0 25 0 5 0 522222726 11442061312 2640410 33554432000 4194304 4338100 140736873902832 18446744073709551615 251813635349 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17622/tid=17632] ppid=17620 vsize=11173888 CPUtime=1176.31 cores=1,3,5,7
/proc/17622/task/17632/stat : 17632 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 883585 0 0 0 117448 183 0 0 25 0 5 0 522222726 11442061312 2640410 33554432000 4194304 4338100 140736873902832 18446744073709551615 4245205 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17622/tid=17633] ppid=17620 vsize=11173888 CPUtime=1177.31 cores=1,3,5,7
/proc/17622/task/17633/stat : 17633 (ttsth-5-4) R 17620 17622 17262 0 -1 4202560 138390 0 0 0 109234 8497 0 0 25 0 5 0 522222726 11442061312 2640410 33554432000 4194304 4338100 140736873902832 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4712.46
Current children cumulated vsize (KiB) 11173888



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1204.3 s]
/proc/loadavg: 8.13 8.07 8.00 6/183 17678
/proc/meminfo: memFree=13728000/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=0 CPUtime=4800.08 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) Z 17620 17622 17262 0 -1 4203548 2690226 0 0 0 461642 18366 0 0 18 0 2 0 522222460 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17622/statm: 0 0 0 0 0 0 0
[pid=17622/tid=17632] ppid=17620 vsize=0 CPUtime=1198.31 cores=1,3,5,7
/proc/17622/task/17632/stat : 17632 (ttsth-5-4) R 17620 17622 17262 0 -1 4203588 895010 0 0 0 119638 193 0 0 25 0 2 0 522222726 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.08
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1204.6 s]
/proc/loadavg: 8.13 8.07 8.00 6/184 17679
/proc/meminfo: memFree=16795280/32950928 swapFree=67111448/67111528
[pid=17622] ppid=17620 vsize=0 CPUtime=4800.38 cores=1,3,5,7
/proc/17622/stat : 17622 (ttsth-5-4) Z 17620 17622 17262 0 -1 4203548 2690226 0 0 0 461642 18396 0 0 18 0 2 0 522222460 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17622/statm: 0 0 0 0 0 0 0
[pid=17622/tid=17632] ppid=17620 vsize=0 CPUtime=1198.6 cores=1,3,5,7
/proc/17622/task/17632/stat : 17632 (ttsth-5-4) R 17620 17622 17262 0 -1 4203588 895010 0 0 0 119638 222 0 0 25 0 2 0 522222726 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.38
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1204.61
CPU time (s): 4800.39
CPU user time (s): 4616.43
CPU system time (s): 183.967
CPU usage (%): 398.502
Max. virtual memory (cumulated for all children) (KiB): 11310640

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4616.43
system time used= 183.967
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2690226
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 17082
involuntary context switches= 33437

runsolver used 1.9417 second user time and 4.6273 second system time

The end

Launcher Data 

Begin job on node126 at 2011-04-29 21:43:58
IDJOB=3251088
IDBENCH=88209
IDSOLVER=1654
FILE ID=node126/3251088-1304106237
RUNJOBID= node126-1304106237-17609
PBS_JOBID= 13170635
Free space on /tmp= 71588 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m47-q742.5-n45-p15-s28972035.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3251088-1304106237/watcher-3251088-1304106237 -o /tmp/evaluation-result-3251088-1304106237/solver-3251088-1304106237 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251088-1304106237.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= bfa662ff57088c54e0febd091ce11e5c
RANDOM SEED=1161527426

node126.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.56
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      30904308 kB
Buffers:        253000 kB
Cached:        1413052 kB
SwapCached:         76 kB
Active:         134080 kB
Inactive:      1600372 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      30904308 kB
SwapTotal:    67111528 kB
SwapFree:     67111448 kB
Dirty:           12180 kB
Writeback:           0 kB
AnonPages:       67952 kB
Mapped:          14496 kB
Slab:           247316 kB
PageTables:       4688 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   190612 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71580 MiB
End job on node126 at 2011-04-29 22:04:04