Trace number 3251000

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.4 1209.03

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m52-q1041.25-n50-p30-s30907550.cnf
MD5SUM002a4cb53ea02f66e0c3418dc8834f7c
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark2.82957
Satisfiable
(Un)Satisfiability was proved
Number of variables2561
Number of clauses490309
Sum of the clauses size983079
Maximum clause length52
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2490259
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 550

Solver Data 

0.29/0.33	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251000-1304105845.cnf 
0.29/0.33	c Using SATTIMEOUT of 4800
0.29/0.33	c Using SATRAM of 15500
0.29/0.33	c Size of subset node = 32
0.29/0.33	c Variables = 2561
0.29/0.33	c Clauses = 490309
0.29/0.33	c Literals = 983079
0.29/0.33	c Reduced size:
0.29/0.33	c Variables = 2561
0.29/0.33	c Clauses = 490309
0.29/0.33	c Literals = 983079
0.29/0.33	c a2
0.49/0.59	c a1
1.39/1.48	c Independent sub-problems = 1
1.39/1.48	c Sub-problem with 2561 variables and 490309 clauses
1.39/1.48	c using permutation method 4
1.39/1.48	c count = 1
1.39/1.48	c Ordering finished 1.410 seconds
1.39/1.48	c mapped 1.470 seconds
1.39/1.48	c permuted 1.470 seconds
1.39/1.48	c about to sort
1.49/1.52	c sorted
3.20/3.21	c clauses now 490309
3.20/3.21	c init finished 1.630 seconds
3.20/3.21	c tree built 3.210 seconds
3.20/3.21	c 
3.20/3.21	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
3.20/3.21	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
3.20/3.21	c ................................ | ..........                    |                                  3.2         0.0             
3.20/3.21	c ................................ | ..........                  1 |                                  3.2         0.3             
3.20/3.21	c ................................ | ..........                  2 |                                  3.2         0.6             
3.20/3.21	c ................................ | ..........                  3 |                                  3.2         0.9             
3.20/3.21	c ................................ | ..........                  4 |                                  3.2         1.2             
3.20/3.21	c ................................ | ..........                  5 |                                  3.2         1.6             
3.20/3.22	c ................................ | ..........               1000 |              20                  3.2       311.5          986
96.01/26.60	c ................................ | ..........            4787000 | 2680         394642             96.0     49859.4     15553605
191.71/50.77	c ................................ | ..........            9406000 | 6120         775992            192.0     48989.6     25440940
287.84/74.94	c ................................ | ..........           13744000 | 8020         1144401           288.0     47720.6     35427757
383.97/99.11	c ................................ | ..........           18239000 | 9579         1503209           384.0     47496.2     45586097
479.71/123.28	c ................................ | *.........           22608000 | 11199        1862715           480.0     47100.0     55769273
575.77/147.46	c ................................ | *.........           26928000 | 12662        2213042           576.0     46750.0     63907266
671.88/171.63	c ................................ | *.........           31161000 | 14303        2565504           672.0     46369.8     71170605
767.60/195.80	c ................................ | *.........           35239000 | 15674        2904061           768.0     45884.1     75950348
863.79/219.96	c ................................ | *.........           39635000 | 16402        3252662           864.0     45873.8     86455113
959.96/244.11	c ................................ | **........           44021000 | 17623        3597175           960.0     45854.7     97280195
1055.64/268.29	c ................................ | **........           48166000 | 18463        3939135          1056.0     45611.7    104494523
1151.81/292.34	c ................................ | **........           52182000 | 20397        4268966          1152.0     45296.1    109988015
1247.65/316.39	c ................................ | **........           56209000 | 22488        4600113          1248.0     45038.9    113826682
1343.83/340.44	c ................................ | **........           60271000 | 24798        4926651          1344.0     44844.5    118676878
1439.91/364.52	c ................................ | ***.......           64308000 | 26631        5264045          1440.0     44658.3    127020518
1535.70/388.57	c ................................ | ***.......           68086000 | 28596        5585705          1536.0     44326.8    130729660
1631.95/412.62	c ................................ | ***.......           72084000 | 30114        5919781          1632.0     44168.6    139166602
1727.79/436.65	c ................................ | ***.......           76200000 | 33153        6256274          1728.0     44097.2    150567095
1824.01/460.81	c ................................ | ***.......           80145000 | 35888        6582529          1824.0     43938.2    159146106
1919.76/484.96	c ................................ | ****......           83653000 | 37768        6882539          1920.0     43569.0    161362799
2015.93/509.12	c ................................ | ****......           87274000 | 39433        7196944          2016.0     43290.5    161362799
2111.68/533.28	c ................................ | ****......           90899000 | 41117        7515090          2112.0     43038.9    163837321
2207.84/557.44	c ................................ | ****......           94765000 | 42821        7848271          2208.0     42918.9    171829732
2303.99/581.60	c ................................ | ****......           98495000 | 44276        8162048          2304.0     42749.6    175982065
2399.79/605.75	c ................................ | *****.....          102638000 | 44936        8486627          2400.0     42765.7    183248574
2495.70/629.87	c ................................ | *****.....          106657000 | 45825        8800458          2496.0     42731.2    187066906
2591.90/653.92	c ................................ | *****.....          110491000 | 47085        9107138          2592.0     42627.7    193902117
2687.67/677.98	c ................................ | *****.....          114319000 | 47712        9432507          2688.0     42529.4    199463604
2783.91/702.02	c ................................ | *****.....          117987000 | 48747        9737828          2784.0     42380.4    204122930
2879.72/726.07	c ................................ | ******....          121974000 | 49668        10062868         2880.0     42352.1    212691121
2975.91/750.13	c ................................ | ******....          125620000 | 50813        10362597         2976.0     42210.5    215462470
3071.67/774.17	c ................................ | ******....          129492000 | 52251        10676362         3072.0     42152.3    221127084
3168.68/798.53	c ................................ | ******....          133260000 | 53189        10987948         3168.8     42054.2    225214951
3263.65/822.49	c ................................ | ******....          136987000 | 54070        11297824         3264.0     41968.9    232633866
3359.83/846.64	c ................................ | *******...          140648000 | 55005        11599109         3360.0     41859.3    239828323
3455.95/870.81	c ................................ | *******...          144161000 | 56175        11902608         3456.0     41713.0    241456107
3551.74/894.96	c ................................ | *******...          147841000 | 57543        12213159         3552.0     41621.9    250097797
3647.92/919.12	c ................................ | *******...          151355000 | 58069        12497678         3648.0     41489.7    251792086
3743.66/943.28	c ................................ | *******...          155170000 | 58528        12806814         3744.0     41444.9    255996317
3839.83/967.44	c ................................ | ********..          158646000 | 59193        13095721         3840.0     41314.1    257776090
3935.96/991.61	c ................................ | ********..          162411000 | 60132        13400838         3936.0     41263.0    265707748
4031.71/1015.78	c ................................ | ********..          166155000 | 60955        13707268         4032.0     41208.9    275838055
4127.86/1039.93	c ................................ | ********..          169656000 | 61955        14002673         4128.0     41098.7    283308279
4223.64/1064.09	c ................................ | ********..          173038000 | 62808        14286690         4224.0     40965.2    285899609
4319.75/1088.26	c ................................ | *********.          176361000 | 63663        14566821         4320.0     40824.3    286516090
4416.62/1112.63	c ................................ | *********.          179810000 | 64760        14862610         4416.8     40710.9    292558025
4511.66/1136.48	c ................................ | *********.          183267000 | 65993        15157958         4512.0     40617.6    295308403
4607.92/1160.52	c ................................ | *********.          186765000 | 67544        15450183         4608.0     40530.6    301414502
4703.66/1184.58	c ................................ | *********.          190145000 | 69515        15735477         4704.0     40422.0    303621322
4799.86/1208.63	c CPU limit (4800) reached
4799.86/1208.63	c Time = 4800.00 seconds
4799.86/1208.63	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3251000-1304105845/watcher-3251000-1304105845 -o /tmp/evaluation-result-3251000-1304105845/solver-3251000-1304105845 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251000-1304105845.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.56 7.98 7.98 3/177 3891
/proc/meminfo: memFree=19192728/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=9556 CPUtime=0 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) R 3889 3891 3320 0 -1 4202496 453 0 0 0 0 0 0 0 25 0 1 0 522184194 9785344 395 33554432000 4194304 4338100 140735754812096 18446744073709551615 274473628357 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/3891/statm: 2389 395 147 36 0 294 0

[startup+0.060746 s]
/proc/loadavg: 7.56 7.98 7.98 3/177 3891
/proc/meminfo: memFree=19192728/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=20756 CPUtime=0.05 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) R 3889 3891 3320 0 -1 4202496 3200 0 0 0 5 0 0 0 25 0 1 0 522184194 21254144 3142 33554432000 4194304 4338100 140735754812096 18446744073709551615 274473628357 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/3891/statm: 5189 3144 148 36 0 3094 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 20756

[startup+0.10069 s]
/proc/loadavg: 7.56 7.98 7.98 3/177 3891
/proc/meminfo: memFree=19192728/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=28820 CPUtime=0.08 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) R 3889 3891 3320 0 -1 4202496 5185 0 0 0 8 0 0 0 25 0 1 0 522184194 29511680 5127 33554432000 4194304 4338100 140735754812096 18446744073709551615 274473628705 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/3891/statm: 7205 5129 148 36 0 5110 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 28820

[startup+0.300656 s]
/proc/loadavg: 7.56 7.98 7.98 3/177 3891
/proc/meminfo: memFree=19192728/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=67920 CPUtime=0.29 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) R 3889 3891 3320 0 -1 4202496 14620 0 0 0 27 2 0 0 25 0 1 0 522184194 69550080 14562 33554432000 4194304 4338100 140735754812096 18446744073709551615 4220544 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/3891/statm: 16980 14562 150 36 0 14885 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 67920

[startup+0.700587 s]
/proc/loadavg: 7.56 7.98 7.98 3/177 3891
/proc/meminfo: memFree=19192728/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=82656 CPUtime=0.69 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) R 3889 3891 3320 0 -1 4202496 26005 0 0 0 66 3 0 0 25 0 1 0 522184194 84639744 16598 33554432000 4194304 4338100 140735754812096 18446744073709551615 274473628357 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/3891/statm: 20664 16604 155 36 0 18569 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 82656

[startup+1.50045 s]
/proc/loadavg: 7.56 7.98 7.98 6/182 3896
/proc/meminfo: memFree=19029140/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=118296 CPUtime=1.49 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) R 3889 3891 3320 0 -1 4202496 37951 0 0 0 145 4 0 0 25 0 1 0 522184194 121135104 27586 33554432000 4194304 4338100 140735754812096 18446744073709551615 4211004 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/3891/statm: 29574 27586 161 36 0 27479 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 118296

[startup+3.1001 s]
/proc/loadavg: 7.56 7.98 7.98 6/182 3896
/proc/meminfo: memFree=18905648/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=257080 CPUtime=3.09 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) R 3889 3891 3320 0 -1 4202496 72678 0 0 0 301 8 0 0 25 0 1 0 522184194 263249920 62313 33554432000 4194304 4338100 140735754812096 18446744073709551615 274473622612 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/3891/statm: 64270 62313 164 36 0 62175 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 257080

[startup+6.30054 s]
/proc/loadavg: 7.59 7.98 7.98 9/186 3900
/proc/meminfo: memFree=18591232/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=464620 CPUtime=15.43 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) S 3889 3891 3320 0 -1 4202496 98766 0 0 0 1526 17 0 0 25 0 5 0 522184194 475770880 88401 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3891/statm: 116155 88401 176 36 0 114060 0
[pid=3891/tid=3897] ppid=3889 vsize=464620 CPUtime=3.07 cores=0,2,4,6
/proc/3891/task/3897/stat : 3897 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 7948 0 0 0 306 1 0 0 25 0 5 0 522184516 475770880 88401 33554432000 4194304 4338100 140735754812096 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3891/tid=3898] ppid=3889 vsize=464620 CPUtime=2.99 cores=0,2,4,6
/proc/3891/task/3898/stat : 3898 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 4021 0 0 0 297 2 0 0 18 0 5 0 522184516 475770880 88401 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3891/tid=3899] ppid=3889 vsize=464620 CPUtime=3.06 cores=0,2,4,6
/proc/3891/task/3899/stat : 3899 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 8170 0 0 0 305 1 0 0 25 0 5 0 522184516 475770880 88401 33554432000 4194304 4338100 140735754812096 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3891/tid=3900] ppid=3889 vsize=464620 CPUtime=3.07 cores=0,2,4,6
/proc/3891/task/3900/stat : 3900 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 3148 0 0 0 303 4 0 0 23 0 5 0 522184516 475770880 88401 33554432000 4194304 4338100 140735754812096 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 15.43
Current children cumulated vsize (KiB) 464620

[startup+12.7004 s]
/proc/loadavg: 7.62 7.98 7.98 9/186 3900
/proc/meminfo: memFree=18190616/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=601772 CPUtime=40.83 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) S 3889 3891 3320 0 -1 4202496 134676 0 0 0 4043 40 0 0 25 0 5 0 522184194 616214528 124311 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3891/statm: 150443 124311 176 36 0 148348 0
[pid=3891/tid=3897] ppid=3889 vsize=601772 CPUtime=9.47 cores=0,2,4,6
/proc/3891/task/3897/stat : 3897 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 23896 0 0 0 941 6 0 0 25 0 5 0 522184516 616214528 124312 33554432000 4194304 4338100 140735754812096 18446744073709551615 4234400 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3891/tid=3898] ppid=3889 vsize=601772 CPUtime=9.22 cores=0,2,4,6
/proc/3891/task/3898/stat : 3898 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 9012 0 0 0 916 6 0 0 18 0 5 0 522184516 616214528 124312 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3891/tid=3899] ppid=3889 vsize=601772 CPUtime=9.42 cores=0,2,4,6
/proc/3891/task/3899/stat : 3899 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 19320 0 0 0 939 3 0 0 25 0 5 0 522184516 616214528 124312 33554432000 4194304 4338100 140735754812096 18446744073709551615 4286319 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3891/tid=3900] ppid=3889 vsize=601772 CPUtime=9.47 cores=0,2,4,6
/proc/3891/task/3900/stat : 3900 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 6970 0 0 0 932 15 0 0 25 0 5 0 522184516 616214528 124312 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 40.83
Current children cumulated vsize (KiB) 601772

[startup+25.5012 s]
/proc/loadavg: 7.71 7.98 7.98 9/186 3901
/proc/meminfo: memFree=17490412/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=933280 CPUtime=91.64 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) S 3889 3891 3320 0 -1 4202496 206287 0 0 0 9082 82 0 0 25 0 5 0 522184194 955678720 195922 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3891/statm: 233320 195922 176 36 0 231225 0
[pid=3891/tid=3897] ppid=3889 vsize=933280 CPUtime=22.27 cores=0,2,4,6
/proc/3891/task/3897/stat : 3897 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 55713 0 0 0 2212 15 0 0 25 0 5 0 522184516 955678720 195922 33554432000 4194304 4338100 140735754812096 18446744073709551615 4286331 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3891/tid=3898] ppid=3889 vsize=933280 CPUtime=21.7 cores=0,2,4,6
/proc/3891/task/3898/stat : 3898 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 17966 0 0 0 2158 12 0 0 18 0 5 0 522184516 955678720 195922 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3891/tid=3899] ppid=3889 vsize=933280 CPUtime=22.15 cores=0,2,4,6
/proc/3891/task/3899/stat : 3899 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 41739 0 0 0 2208 7 0 0 25 0 5 0 522184516 955678720 195922 33554432000 4194304 4338100 140735754812096 18446744073709551615 4241503 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3891/tid=3900] ppid=3889 vsize=933280 CPUtime=22.27 cores=0,2,4,6
/proc/3891/task/3900/stat : 3900 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 15390 0 0 0 2189 38 0 0 18 0 5 0 522184516 955678720 195922 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 91.64
Current children cumulated vsize (KiB) 933280

[startup+51.1007 s]
/proc/loadavg: 7.81 7.98 7.98 9/186 3901
/proc/meminfo: memFree=16777560/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=1251556 CPUtime=193.29 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) S 3889 3891 3320 0 -1 4202496 289434 0 0 0 19175 154 0 0 25 0 5 0 522184194 1281593344 279069 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3891/statm: 312889 279069 176 36 0 310794 0
[pid=3891/tid=3897] ppid=3889 vsize=1251556 CPUtime=47.87 cores=0,2,4,6
/proc/3891/task/3897/stat : 3897 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 101809 0 0 0 4759 28 0 0 25 0 5 0 522184516 1281593344 279069 33554432000 4194304 4338100 140735754812096 18446744073709551615 274473363327 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3891/tid=3898] ppid=3889 vsize=1251556 CPUtime=46.7 cores=0,2,4,6
/proc/3891/task/3898/stat : 3898 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 22785 0 0 0 4648 22 0 0 18 0 5 0 522184516 1281593344 279069 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3891/tid=3899] ppid=3889 vsize=1251556 CPUtime=47.6 cores=0,2,4,6
/proc/3891/task/3899/stat : 3899 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 68776 0 0 0 4749 11 0 0 25 0 5 0 522184516 1281593344 279069 33554432000 4194304 4338100 140735754812096 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3891/tid=3900] ppid=3889 vsize=1251556 CPUtime=47.86 cores=0,2,4,6
/proc/3891/task/3900/stat : 3900 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 20585 0 0 0 4703 83 0 0 25 0 5 0 522184516 1281593344 279069 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 193.29
Current children cumulated vsize (KiB) 1251556

[startup+102.307 s]
/proc/loadavg: 8.12 8.04 8.00 9/186 3903
/proc/meminfo: memFree=15723920/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=1948812 CPUtime=396.7 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) S 3889 3891 3320 0 -1 4202496 464824 0 0 0 39360 310 0 0 25 0 5 0 522184194 1995583488 454459 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3891/statm: 487203 454459 176 36 0 485108 0
[pid=3891/tid=3897] ppid=3889 vsize=1948812 CPUtime=99.08 cores=0,2,4,6
/proc/3891/task/3897/stat : 3897 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 202562 0 0 0 9852 56 0 0 25 0 5 0 522184516 1995583488 454459 33554432000 4194304 4338100 140735754812096 18446744073709551615 274473362448 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3891/tid=3898] ppid=3889 vsize=1948812 CPUtime=96.77 cores=0,2,4,6
/proc/3891/task/3898/stat : 3898 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 27493 0 0 0 9631 46 0 0 18 0 5 0 522184516 1995583488 454459 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3891/tid=3899] ppid=3889 vsize=1948812 CPUtime=98.54 cores=0,2,4,6
/proc/3891/task/3899/stat : 3899 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 137503 0 0 0 9830 24 0 0 25 0 5 0 522184516 1995583488 454460 33554432000 4194304 4338100 140735754812096 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3891/tid=3900] ppid=3889 vsize=1948812 CPUtime=99.07 cores=0,2,4,6
/proc/3891/task/3900/stat : 3900 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 21787 0 0 0 9733 174 0 0 25 0 5 0 522184516 1995583488 454460 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 396.7
Current children cumulated vsize (KiB) 1948812

[startup+162.301 s]
/proc/loadavg: 8.09 8.04 8.00 9/186 3904
/proc/meminfo: memFree=14546772/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=2695736 CPUtime=634.93 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) S 3889 3891 3320 0 -1 4202496 625733 0 0 0 62993 500 0 0 25 0 5 0 522184194 2760433664 615368 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3891/statm: 673934 615368 176 36 0 671839 0
[pid=3891/tid=3897] ppid=3889 vsize=2695736 CPUtime=159.06 cores=0,2,4,6
/proc/3891/task/3897/stat : 3897 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 303454 0 0 0 15819 87 0 0 25 0 5 0 522184516 2760433664 615368 33554432000 4194304 4338100 140735754812096 18446744073709551615 274473362662 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3891/tid=3898] ppid=3889 vsize=2695736 CPUtime=155.35 cores=0,2,4,6
/proc/3891/task/3898/stat : 3898 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 30829 0 0 0 15461 74 0 0 18 0 5 0 522184516 2760433664 615368 33554432000 4194304 4338100 140735754812096 18446744073709551615 274473362828 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3891/tid=3899] ppid=3889 vsize=2695736 CPUtime=158.22 cores=0,2,4,6
/proc/3891/task/3899/stat : 3899 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 191795 0 0 0 15788 34 0 0 25 0 5 0 522184516 2760433664 615368 33554432000 4194304 4338100 140735754812096 18446744073709551615 274473362555 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3891/tid=3900] ppid=3889 vsize=2695736 CPUtime=159.06 cores=0,2,4,6
/proc/3891/task/3900/stat : 3900 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 24176 0 0 0 15610 296 0 0 25 0 5 0 522184516 2760433664 615368 33554432000 4194304 4338100 140735754812096 18446744073709551615 4241503 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 634.93
Current children cumulated vsize (KiB) 2695736

[startup+222.301 s]
/proc/loadavg: 8.08 8.05 8.00 9/186 3906
/proc/meminfo: memFree=13691156/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=3355336 CPUtime=873.31 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) S 3889 3891 3320 0 -1 4202496 785108 0 0 0 86658 673 0 0 25 0 5 0 522184194 3435864064 774743 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3891/statm: 838834 774743 176 36 0 836739 0
[pid=3891/tid=3897] ppid=3889 vsize=3355336 CPUtime=218.9 cores=0,2,4,6
/proc/3891/task/3897/stat : 3897 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 405677 0 0 0 21776 114 0 0 25 0 5 0 522184516 3435864064 774743 33554432000 4194304 4338100 140735754812096 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3891/tid=3898] ppid=3889 vsize=3355336 CPUtime=214.09 cores=0,2,4,6
/proc/3891/task/3898/stat : 3898 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 32586 0 0 0 21311 98 0 0 18 0 5 0 522184516 3435864064 774743 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3891/tid=3899] ppid=3889 vsize=3355336 CPUtime=218.03 cores=0,2,4,6
/proc/3891/task/3899/stat : 3899 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 244005 0 0 0 21759 44 0 0 25 0 5 0 522184516 3435864064 774743 33554432000 4194304 4338100 140735754812096 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3891/tid=3900] ppid=3889 vsize=3355336 CPUtime=219.04 cores=0,2,4,6
/proc/3891/task/3900/stat : 3900 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 27361 0 0 0 21497 407 0 0 25 0 5 0 522184516 3435864064 774743 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 873.31
Current children cumulated vsize (KiB) 3355336

################
# More data... #
################

/proc/3891/task/3900/stat : 3900 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 65295 0 0 0 67666 1798 0 0 25 0 5 0 522184516 7870210048 1782080 33554432000 4194304 4338100 140735754812096 18446744073709551615 4235746 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 2785.12
Current children cumulated vsize (KiB) 7685752

[startup+762.307 s]
/proc/loadavg: 8.12 8.11 8.02 9/186 3920
/proc/meminfo: memFree=7833208/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=8338824 CPUtime=3024.61 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) S 3889 3891 3320 0 -1 4202496 1907252 0 0 0 297876 4585 0 0 25 0 5 0 522184194 8538955776 1896887 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3891/statm: 2084706 1896887 176 36 0 2082611 0
[pid=3891/tid=3897] ppid=3889 vsize=8338824 CPUtime=757.26 cores=0,2,4,6
/proc/3891/task/3897/stat : 3897 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 1132288 0 0 0 75392 334 0 0 25 0 5 0 522184516 8538955776 1896887 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485781667 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3891/tid=3898] ppid=3889 vsize=8338824 CPUtime=752.16 cores=0,2,4,6
/proc/3891/task/3898/stat : 3898 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 69198 0 0 0 73178 2038 0 0 25 0 5 0 522184516 8538955776 1896887 33554432000 4194304 4338100 140735754812096 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3891/tid=3899] ppid=3889 vsize=8338824 CPUtime=757.62 cores=0,2,4,6
/proc/3891/task/3899/stat : 3899 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 559798 0 0 0 75662 100 0 0 25 0 5 0 522184516 8538955776 1896887 33554432000 4194304 4338100 140735754812096 18446744073709551615 274473362541 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3891/tid=3900] ppid=3889 vsize=8338824 CPUtime=754.32 cores=0,2,4,6
/proc/3891/task/3900/stat : 3900 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 70489 0 0 0 73329 2103 0 0 25 0 5 0 522184516 8538955776 1896887 33554432000 4194304 4338100 140735754812096 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3024.61
Current children cumulated vsize (KiB) 8338824

[startup+822.301 s]
/proc/loadavg: 8.10 8.10 8.02 9/186 3922
/proc/meminfo: memFree=7258612/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=8801592 CPUtime=3263.25 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) S 3889 3891 3320 0 -1 4202496 2033425 0 0 0 321470 4855 0 0 25 0 5 0 522184194 9012830208 2023060 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3891/statm: 2200398 2023060 176 36 0 2198303 0
[pid=3891/tid=3897] ppid=3889 vsize=8801592 CPUtime=816.94 cores=0,2,4,6
/proc/3891/task/3897/stat : 3897 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 1204210 0 0 0 81335 359 0 0 25 0 5 0 522184516 9012830208 2023060 33554432000 4194304 4338100 140735754812096 18446744073709551615 4239191 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3891/tid=3898] ppid=3889 vsize=8801592 CPUtime=811.24 cores=0,2,4,6
/proc/3891/task/3898/stat : 3898 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 73413 0 0 0 79003 2121 0 0 18 0 5 0 522184516 9012830208 2023060 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3891/tid=3899] ppid=3889 vsize=8801592 CPUtime=817.58 cores=0,2,4,6
/proc/3891/task/3899/stat : 3899 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 605487 0 0 0 81649 109 0 0 25 0 5 0 522184516 9012830208 2023060 33554432000 4194304 4338100 140735754812096 18446744073709551615 4256131 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3891/tid=3900] ppid=3889 vsize=8801592 CPUtime=814.23 cores=0,2,4,6
/proc/3891/task/3900/stat : 3900 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 74836 0 0 0 79168 2255 0 0 25 0 5 0 522184516 9012830208 2023060 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3263.25
Current children cumulated vsize (KiB) 8801592

[startup+882.3 s]
/proc/loadavg: 7.97 8.06 8.01 9/186 3923
/proc/meminfo: memFree=6793480/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=9173608 CPUtime=3501.64 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) S 3889 3891 3320 0 -1 4202496 2139616 0 0 0 345148 5016 0 0 25 0 5 0 522184194 9393774592 2129251 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3891/statm: 2293402 2129251 176 36 0 2291307 0
[pid=3891/tid=3897] ppid=3889 vsize=9173608 CPUtime=876.57 cores=0,2,4,6
/proc/3891/task/3897/stat : 3897 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 1281360 0 0 0 87270 387 0 0 25 0 5 0 522184516 9393774592 2129251 33554432000 4194304 4338100 140735754812096 18446744073709551615 4254024 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3891/tid=3898] ppid=3889 vsize=9173608 CPUtime=870.05 cores=0,2,4,6
/proc/3891/task/3898/stat : 3898 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 79911 0 0 0 84862 2143 0 0 18 0 5 0 522184516 9393774592 2129251 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3891/tid=3899] ppid=3889 vsize=9173608 CPUtime=877.55 cores=0,2,4,6
/proc/3891/task/3899/stat : 3899 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 622437 0 0 0 87643 112 0 0 25 0 5 0 522184516 9393774592 2129251 33554432000 4194304 4338100 140735754812096 18446744073709551615 274473362461 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3891/tid=3900] ppid=3889 vsize=9173608 CPUtime=874.22 cores=0,2,4,6
/proc/3891/task/3900/stat : 3900 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 80429 0 0 0 85058 2364 0 0 25 0 5 0 522184516 9393774592 2129251 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3501.64
Current children cumulated vsize (KiB) 9173608

[startup+942.301 s]
/proc/loadavg: 8.06 8.07 8.01 9/186 3925
/proc/meminfo: memFree=6170432/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=9526088 CPUtime=3740.09 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) S 3889 3891 3320 0 -1 4202496 2238195 0 0 0 368837 5172 0 0 25 0 5 0 522184194 9754714112 2227830 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3891/statm: 2381522 2227830 176 36 0 2379427 0
[pid=3891/tid=3897] ppid=3889 vsize=9526088 CPUtime=936.21 cores=0,2,4,6
/proc/3891/task/3897/stat : 3897 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 1346149 0 0 0 93209 412 0 0 25 0 5 0 522184516 9754714112 2227830 33554432000 4194304 4338100 140735754812096 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3891/tid=3898] ppid=3889 vsize=9526088 CPUtime=928.91 cores=0,2,4,6
/proc/3891/task/3898/stat : 3898 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 86042 0 0 0 90726 2165 0 0 18 0 5 0 522184516 9754714112 2227830 33554432000 4194304 4338100 140735754812096 18446744073709551615 274473363194 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3891/tid=3899] ppid=3889 vsize=9526088 CPUtime=937.53 cores=0,2,4,6
/proc/3891/task/3899/stat : 3899 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 643391 0 0 0 93635 118 0 0 25 0 5 0 522184516 9754714112 2227830 33554432000 4194304 4338100 140735754812096 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3891/tid=3900] ppid=3889 vsize=9526088 CPUtime=934.19 cores=0,2,4,6
/proc/3891/task/3900/stat : 3900 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 87134 0 0 0 90952 2467 0 0 25 0 5 0 522184516 9754714112 2227830 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3740.09
Current children cumulated vsize (KiB) 9526088

[startup+1002.3 s]
/proc/loadavg: 7.95 8.04 8.00 9/186 3927
/proc/meminfo: memFree=5550148/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=9936760 CPUtime=3978.46 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) S 3889 3891 3320 0 -1 4202496 2364930 0 0 0 392511 5335 0 0 25 0 5 0 522184194 10175242240 2354565 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3891/statm: 2484190 2354565 176 36 0 2482095 0
[pid=3891/tid=3897] ppid=3889 vsize=9936760 CPUtime=995.85 cores=0,2,4,6
/proc/3891/task/3897/stat : 3897 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 1427435 0 0 0 99146 439 0 0 25 0 5 0 522184516 10175242240 2354565 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485781672 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3891/tid=3898] ppid=3889 vsize=9936760 CPUtime=987.67 cores=0,2,4,6
/proc/3891/task/3898/stat : 3898 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 92089 0 0 0 96579 2188 0 0 18 0 5 0 522184516 10175242240 2354565 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3891/tid=3899] ppid=3889 vsize=9936760 CPUtime=997.52 cores=0,2,4,6
/proc/3891/task/3899/stat : 3899 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 676702 0 0 0 99626 126 0 0 25 0 5 0 522184516 10175242240 2354565 33554432000 4194304 4338100 140735754812096 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3891/tid=3900] ppid=3889 vsize=9936760 CPUtime=994.18 cores=0,2,4,6
/proc/3891/task/3900/stat : 3900 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 93225 0 0 0 96846 2572 0 0 25 0 5 0 522184516 10175242240 2354565 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3978.46
Current children cumulated vsize (KiB) 9936760

[startup+1062.3 s]
/proc/loadavg: 8.10 8.06 8.01 9/186 3928
/proc/meminfo: memFree=5007064/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=10498040 CPUtime=4216.88 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) S 3889 3891 3320 0 -1 4202496 2489679 0 0 0 416205 5483 0 0 25 0 5 0 522184194 10749992960 2479314 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3891/statm: 2624510 2479314 176 36 0 2622415 0
[pid=3891/tid=3897] ppid=3889 vsize=10498040 CPUtime=1055.49 cores=0,2,4,6
/proc/3891/task/3897/stat : 3897 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 1509943 0 0 0 105085 464 0 0 25 0 5 0 522184516 10749992960 2479314 33554432000 4194304 4338100 140735754812096 18446744073709551615 4256080 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3891/tid=3898] ppid=3889 vsize=10498040 CPUtime=1046.51 cores=0,2,4,6
/proc/3891/task/3898/stat : 3898 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 93048 0 0 0 102443 2208 0 0 18 0 5 0 522184516 10749992960 2479314 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3891/tid=3899] ppid=3889 vsize=10498040 CPUtime=1057.48 cores=0,2,4,6
/proc/3891/task/3899/stat : 3899 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 716994 0 0 0 105614 134 0 0 25 0 5 0 522184516 10749992960 2479314 33554432000 4194304 4338100 140735754812096 18446744073709551615 274473362818 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3891/tid=3900] ppid=3889 vsize=10498040 CPUtime=1054.16 cores=0,2,4,6
/proc/3891/task/3900/stat : 3900 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 94215 0 0 0 102749 2667 0 0 25 0 5 0 522184516 10749992960 2479314 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4216.88
Current children cumulated vsize (KiB) 10498040

[startup+1122.3 s]
/proc/loadavg: 7.98 8.03 8.00 9/186 3930
/proc/meminfo: memFree=4596000/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=11021080 CPUtime=4455.36 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) S 3889 3891 3320 0 -1 4202496 2555372 0 0 0 439815 5721 0 0 25 0 5 0 522184194 11285585920 2545007 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3891/statm: 2755270 2545007 176 36 0 2753175 0
[pid=3891/tid=3897] ppid=3889 vsize=11021080 CPUtime=1115.17 cores=0,2,4,6
/proc/3891/task/3897/stat : 3897 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 1551828 0 0 0 111036 481 0 0 25 0 5 0 522184516 11285585920 2545007 33554432000 4194304 4338100 140735754812096 18446744073709551615 4256294 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3891/tid=3898] ppid=3889 vsize=11021080 CPUtime=1105.39 cores=0,2,4,6
/proc/3891/task/3898/stat : 3898 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 99577 0 0 0 108257 2282 0 0 25 0 5 0 522184516 11285585920 2545007 33554432000 4194304 4338100 140735754812096 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3891/tid=3899] ppid=3889 vsize=11021080 CPUtime=1117.48 cores=0,2,4,6
/proc/3891/task/3899/stat : 3899 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 727276 0 0 0 111611 137 0 0 25 0 5 0 522184516 11285585920 2545007 33554432000 4194304 4338100 140735754812096 18446744073709551615 4263568 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3891/tid=3900] ppid=3889 vsize=11021080 CPUtime=1114.07 cores=0,2,4,6
/proc/3891/task/3900/stat : 3900 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 101212 0 0 0 108596 2811 0 0 25 0 5 0 522184516 11285585920 2545007 33554432000 4194304 4338100 140735754812096 18446744073709551615 274473362471 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4455.36
Current children cumulated vsize (KiB) 11021080

[startup+1182.3 s]
/proc/loadavg: 8.06 8.05 8.00 9/186 3931
/proc/meminfo: memFree=3961460/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=11700824 CPUtime=4694.87 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) S 3889 3891 3320 0 -1 4202496 2655620 0 0 0 463122 6365 0 0 25 0 5 0 522184194 11981643776 2645256 33554432000 4194304 4338100 140735754812096 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3891/statm: 2925206 2645256 176 36 0 2923111 0
[pid=3891/tid=3897] ppid=3889 vsize=11700824 CPUtime=1175.14 cores=0,2,4,6
/proc/3891/task/3897/stat : 3897 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 1604416 0 0 0 117014 500 0 0 25 0 5 0 522184516 11981643776 2645256 33554432000 4194304 4338100 140735754812096 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3891/tid=3898] ppid=3889 vsize=11700824 CPUtime=1165.17 cores=0,2,4,6
/proc/3891/task/3898/stat : 3898 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 107943 0 0 0 113915 2602 0 0 25 0 5 0 522184516 11981643776 2645256 33554432000 4194304 4338100 140735754812096 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3891/tid=3899] ppid=3889 vsize=11700824 CPUtime=1177.47 cores=0,2,4,6
/proc/3891/task/3899/stat : 3899 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 760456 0 0 0 117604 143 0 0 25 0 5 0 522184516 11981643776 2645256 33554432000 4194304 4338100 140735754812096 18446744073709551615 274473362833 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3891/tid=3900] ppid=3889 vsize=11700824 CPUtime=1173.85 cores=0,2,4,6
/proc/3891/task/3900/stat : 3900 (ttsth-5-4) R 3889 3891 3320 0 -1 4202560 107327 0 0 0 114275 3110 0 0 25 0 5 0 522184516 11981643776 2645256 33554432000 4194304 4338100 140735754812096 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4694.87
Current children cumulated vsize (KiB) 11700824



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1208.7 s]
/proc/loadavg: 7.48 7.93 7.96 2/173 3950
/proc/meminfo: memFree=9955824/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=0 CPUtime=4800.07 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) Z 3889 3891 3320 0 -1 4203548 2687941 0 0 0 473362 6645 0 0 18 0 2 0 522184194 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3891/statm: 0 0 0 0 0 0 0
[pid=3891/tid=3898] ppid=3889 vsize=0 CPUtime=1191.46 cores=0,2,4,6
/proc/3891/task/3898/stat : 3898 (ttsth-5-4) R 3889 3891 3320 0 -1 4203588 108566 0 0 0 116404 2742 0 0 25 0 2 0 522184516 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.07
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1209 s]
/proc/loadavg: 7.48 7.93 7.96 2/174 3951
/proc/meminfo: memFree=13058772/32950928 swapFree=67111528/67111528
[pid=3891] ppid=3889 vsize=0 CPUtime=4800.37 cores=0,2,4,6
/proc/3891/stat : 3891 (ttsth-5-4) Z 3889 3891 3320 0 -1 4203548 2687941 0 0 0 473362 6675 0 0 18 0 2 0 522184194 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3891/statm: 0 0 0 0 0 0 0
[pid=3891/tid=3898] ppid=3889 vsize=0 CPUtime=1191.76 cores=0,2,4,6
/proc/3891/task/3898/stat : 3898 (ttsth-5-4) R 3889 3891 3320 0 -1 4203588 108566 0 0 0 116404 2772 0 0 25 0 2 0 522184516 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.37
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1209.03
CPU time (s): 4800.4
CPU user time (s): 4733.63
CPU system time (s): 66.7759
CPU usage (%): 397.047
Max. virtual memory (cumulated for all children) (KiB): 11831064

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4733.63
system time used= 66.7758
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2687941
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 23555
involuntary context switches= 23423

runsolver used 2.03469 second user time and 4.44932 second system time

The end

Launcher Data 

Begin job on node125 at 2011-04-29 21:37:25
IDJOB=3251000
IDBENCH=88207
IDSOLVER=1654
FILE ID=node125/3251000-1304105845
RUNJOBID= node125-1304104641-3775
PBS_JOBID= 13170627
Free space on /tmp= 72756 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m52-q1041.25-n50-p30-s30907550.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3251000-1304105845/watcher-3251000-1304105845 -o /tmp/evaluation-result-3251000-1304105845/solver-3251000-1304105845 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251000-1304105845.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 002a4cb53ea02f66e0c3418dc8834f7c
RANDOM SEED=949565187

node125.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5266.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      19193380 kB
Buffers:       2218852 kB
Cached:       10926208 kB
SwapCached:          0 kB
Active:        8453664 kB
Inactive:      4772876 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      19193380 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:            7012 kB
Writeback:           0 kB
AnonPages:       80700 kB
Mapped:          15076 kB
Slab:           467072 kB
PageTables:       3864 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   222348 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72756 MiB
End job on node125 at 2011-04-29 21:57:36