Trace number 3258130

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.47 1205.24

General information on the benchmark

NameCRAFTED/rbsat/random/unforced/
rbsat-v760c43649g7.cnf
MD5SUM57c91698da60cd77ab278c961fa35654
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.594909
Satisfiable
(Un)Satisfiability was proved
Number of variables760
Number of clauses43649
Sum of the clauses size87978
Maximum clause length19
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 243609
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 540

Solver Data 

0.00/0.02	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258130-1304049666.cnf 
0.00/0.02	c Using SATTIMEOUT of 4800
0.00/0.02	c Using SATRAM of 15500
0.00/0.02	c Size of subset node = 32
0.00/0.02	c Variables = 760
0.00/0.02	c Clauses = 43649
0.00/0.02	c Literals = 87978
0.00/0.02	c Reduced size:
0.00/0.02	c Variables = 760
0.00/0.02	c Clauses = 43649
0.00/0.02	c Literals = 87978
0.00/0.02	c a2
0.00/0.05	c a1
0.09/0.10	c Independent sub-problems = 1
0.09/0.10	c Sub-problem with 760 variables and 43649 clauses
0.09/0.10	c using permutation method 4
0.09/0.10	c count = 1
0.09/0.10	c Ordering finished 0.090 seconds
0.09/0.10	c mapped 0.090 seconds
0.09/0.10	c permuted 0.090 seconds
0.09/0.10	c about to sort
0.09/0.10	c sorted
0.09/0.18	c clauses now 43649
0.09/0.18	c init finished 0.100 seconds
0.09/0.18	c tree built 0.170 seconds
0.09/0.18	c 
0.09/0.18	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.09/0.18	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.09/0.18	c ................................ | ..........                    |                                  0.2         0.0             
0.09/0.18	c ................................ | ..........                  1 |                                  0.2         5.9             
0.09/0.18	c ................................ | ..........                  2 |                                  0.2        11.8             
0.09/0.18	c ................................ | ..........                  3 |                                  0.2        17.6             
0.09/0.18	c ................................ | ..........                  4 |                                  0.2        23.5             
0.09/0.18	c ................................ | ..........                  5 |                                  0.2        29.4             
0.09/0.19	c ................................ | ..........               1000 | 1            250                 0.2      5263.2         8419
95.70/24.18	c ................................ | ..........            3759000 | 104838       837834             96.0     39156.2     26252952
191.91/48.22	c ................................ | ..........            6991000 | 198293       1459841           192.0     36407.7     37942548
287.74/72.26	c ................................ | ..........           10179000 | 280123       2098917           288.0     35343.8     49171495
383.97/96.31	c ................................ | ..........           12917000 | 356118       2721431           384.0     33636.3     62793515
479.76/120.36	c ................................ | *.........           15996000 | 438199       3271588           480.0     33325.0     72974062
575.94/144.42	c ................................ | *.........           18802000 | 512989       3772754           576.0     32640.7     79198551
672.00/168.50	c ................................ | *.........           21431000 | 578251       4391572           672.0     31890.9     94785426
767.91/192.63	c ................................ | *.........           23726000 | 635259       4985649           768.0     30892.0    102708030
863.89/216.73	c ................................ | *.........           25781000 | 693842       5520270           864.0     29838.4    109343771
959.76/240.87	c ................................ | **........           27909000 | 748991       6145051           960.0     29070.7    123256021
1055.79/264.96	c ................................ | **........           30030000 | 810194       6644924          1056.0     28436.4    126173854
1151.92/289.02	c ................................ | **........           32584000 | 886500       7200221          1152.0     28284.7    135801492
1247.67/313.08	c ................................ | **........           34937000 | 951562       7668910          1248.0     27993.9    146950974
1343.78/337.15	c ................................ | **........           37443000 | 1008848      8104086          1344.0     27859.4    154838427
1439.80/361.25	c ................................ | ***.......           39492000 | 1068270      8490024          1440.0     27424.4    164642539
1535.69/385.38	c ................................ | ***.......           41571000 | 1124961      8960727          1536.0     27064.1    176307568
1631.92/409.52	c ................................ | ***.......           43454000 | 1174937      9416414          1632.0     26626.2    179707910
1727.84/433.64	c ................................ | ***.......           45099000 | 1224912      9823391          1728.0     26099.0    182252133
1823.83/457.74	c ................................ | ***.......           47126000 | 1280786      10239238         1824.0     25836.6    188833731
1919.89/481.84	c ................................ | ****......           49411000 | 1337894      10647111         1920.0     25734.5    197817092
2015.86/505.94	c ................................ | ****......           51405000 | 1395144      11012846         2016.0     25498.3    207901944
2111.78/530.06	c ................................ | ****......           53826000 | 1454680      11396367         2112.0     25485.4    218575942
2207.88/554.13	c ................................ | ****......           56008000 | 1511252      11754399         2208.0     25365.7    223065383
2303.61/578.20	c ................................ | ****......           58049000 | 1563021      12107276         2304.0     25194.8    227884295
2399.73/602.27	c ................................ | *****.....           59771000 | 1610977      12434094         2400.0     24904.3    231341659
2495.82/626.35	c ................................ | *****.....           61698000 | 1662168      12792301         2496.0     24718.6    233371639
2592.00/650.40	c ................................ | *****.....           63160000 | 1700432      13140635         2592.0     24367.0    236859018
2688.02/674.50	c ................................ | *****.....           64470000 | 1741303      13437032         2688.0     23984.0    242778212
2784.00/698.60	c ................................ | *****.....           65858000 | 1780576      13772306         2784.0     23655.6    249776421
2880.00/722.72	c ................................ | ******....           67016000 | 1814500      14048915         2880.1     23269.0    252258674
2975.60/746.80	c ................................ | ******....           68320000 | 1854259      14442450         2976.0     22957.0    259615760
3071.78/770.97	c ................................ | ******....           69614000 | 1892263      14781903         3072.1     22660.3    259682864
3167.76/795.06	c ................................ | ******....           70830000 | 1926463      15101363         3168.0     22357.9    263999553
3263.79/819.15	c ................................ | ******....           72241000 | 1964244      15425063         3264.0     22132.6    267050118
3359.79/843.26	c ................................ | *******...           73923000 | 2008456      15765354         3360.0     22000.7    270978452
3455.66/867.38	c ................................ | *******...           75381000 | 2047190      16123754         3456.0     21811.6    270978452
3551.97/891.51	c ................................ | *******...           76852000 | 2085200      16456373         3552.0     21636.2    273550090
3647.64/915.59	c ................................ | *******...           78125000 | 2119947      16771681         3648.0     21415.8    275970948
3743.70/939.69	c ................................ | *******...           79298000 | 2153712      17057283         3744.1     21179.6    278105688
3839.77/963.76	c ................................ | ********..           80401000 | 2188981      17340145         3840.0     20937.6    283962737
3935.97/987.82	c ................................ | ********..           81436000 | 2222331      17595086         3936.0     20689.9    288893596
4032.03/1011.91	c ................................ | ********..           82427000 | 2258035      17840167         4032.1     20442.9    291542377
4127.74/1035.97	c ................................ | ********..           83616000 | 2290328      18095924         4128.0     20255.6    297160110
4223.80/1060.06	c ................................ | ********..           84766000 | 2325244      18325231         4224.0     20067.5    299570984
4319.98/1084.22	c ................................ | *********.           85879000 | 2362558      18627581         4320.1     19879.0    303967041
4415.92/1108.32	c ................................ | *********.           86981000 | 2399153      18959311         4416.0     19696.7    311154498
4511.72/1132.47	c ................................ | *********.           88000000 | 2434002      19269808         4512.0     19503.4    316400233
4607.74/1156.57	c ................................ | *********.           89183000 | 2464182      19568681         4608.0     19353.8    318592326
4703.92/1180.72	c ................................ | *********.           90272000 | 2493858      19902876         4704.0     19190.5    325931286
4799.95/1204.82	c CPU limit (4800) reached
4799.95/1204.82	c Time = 4800.05 seconds
4799.95/1204.82	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3258130-1304049666/watcher-3258130-1304049666 -o /tmp/evaluation-result-3258130-1304049666/solver-3258130-1304049666 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258130-1304049666.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.74 8.02 7.98 5/181 28231
/proc/meminfo: memFree=19547836/32950928 swapFree=67111364/67111528
[pid=28231] ppid=28229 vsize=9760 CPUtime=0 cores=0,2,4,6
/proc/28231/stat : 28231 (ttsth-5-4) R 28229 28231 27953 0 -1 4202496 495 0 0 0 0 0 0 0 25 0 1 0 595197267 9994240 437 33554432000 4194304 4338100 140734813377504 18446744073709551615 251367207621 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/28231/statm: 2440 438 147 36 0 345 0

[startup+0.0932769 s]
/proc/loadavg: 7.74 8.02 7.98 5/181 28231
/proc/meminfo: memFree=19547836/32950928 swapFree=67111364/67111528
[pid=28231] ppid=28229 vsize=18612 CPUtime=0.09 cores=0,2,4,6
/proc/28231/stat : 28231 (ttsth-5-4) R 28229 28231 27953 0 -1 4202496 3500 0 0 0 9 0 0 0 25 0 1 0 595197267 19058688 2667 33554432000 4194304 4338100 140734813377504 18446744073709551615 251367201792 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/28231/statm: 4653 2667 156 36 0 2558 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 18612

[startup+0.100273 s]
/proc/loadavg: 7.74 8.02 7.98 5/181 28231
/proc/meminfo: memFree=19547836/32950928 swapFree=67111364/67111528
[pid=28231] ppid=28229 vsize=18612 CPUtime=0.09 cores=0,2,4,6
/proc/28231/stat : 28231 (ttsth-5-4) R 28229 28231 27953 0 -1 4202496 3505 0 0 0 9 0 0 0 25 0 1 0 595197267 19058688 2672 33554432000 4194304 4338100 140734813377504 18446744073709551615 4228256 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/28231/statm: 4653 2672 161 36 0 2558 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 18612

[startup+0.300171 s]
/proc/loadavg: 7.74 8.02 7.98 5/181 28231
/proc/meminfo: memFree=19547836/32950928 swapFree=67111364/67111528
[pid=28231] ppid=28229 vsize=224252 CPUtime=0.63 cores=0,2,4,6
/proc/28231/stat : 28231 (ttsth-5-4) S 28229 28231 27953 0 -1 4202496 11038 0 0 0 62 1 0 0 25 0 5 0 595197267 229634048 10033 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/28231/statm: 56063 10033 176 36 0 53968 0
Current children cumulated CPU time (s) 0.63
Current children cumulated vsize (KiB) 224252

[startup+0.700096 s]
/proc/loadavg: 7.74 8.02 7.98 5/181 28231
/proc/meminfo: memFree=19547836/32950928 swapFree=67111364/67111528
[pid=28231] ppid=28229 vsize=249804 CPUtime=2.22 cores=0,2,4,6
/proc/28231/stat : 28231 (ttsth-5-4) S 28229 28231 27953 0 -1 4202496 17006 0 0 0 219 3 0 0 25 0 5 0 595197267 255799296 16001 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/28231/statm: 62451 16001 176 36 0 60356 0
Current children cumulated CPU time (s) 2.22
Current children cumulated vsize (KiB) 249804

[startup+1.50095 s]
/proc/loadavg: 7.84 8.03 7.99 9/186 28236
/proc/meminfo: memFree=19438064/32950928 swapFree=67111364/67111528
[pid=28231] ppid=28229 vsize=290600 CPUtime=5.43 cores=0,2,4,6
/proc/28231/stat : 28231 (ttsth-5-4) S 28229 28231 27953 0 -1 4202496 27241 0 0 0 534 9 0 0 25 0 5 0 595197267 297574400 26236 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/28231/statm: 72650 26237 176 36 0 70555 0
[pid=28231/tid=28233] ppid=28229 vsize=290600 CPUtime=1.31 cores=0,2,4,6
/proc/28231/task/28233/stat : 28233 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 5927 0 0 0 130 1 0 0 20 0 5 0 595197286 297574400 26237 33554432000 4194304 4338100 140734813377504 18446744073709551615 4254407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28231/tid=28234] ppid=28229 vsize=290600 CPUtime=1.3 cores=0,2,4,6
/proc/28231/task/28234/stat : 28234 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 3650 0 0 0 128 2 0 0 25 0 5 0 595197286 297574400 26237 33554432000 4194304 4338100 140734813377504 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28231/tid=28235] ppid=28229 vsize=290600 CPUtime=1.3 cores=0,2,4,6
/proc/28231/task/28235/stat : 28235 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 6538 0 0 0 129 1 0 0 19 0 5 0 595197286 297574400 26237 33554432000 4194304 4338100 140734813377504 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28231/tid=28236] ppid=28229 vsize=290600 CPUtime=1.3 cores=0,2,4,6
/proc/28231/task/28236/stat : 28236 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 2393 0 0 0 128 2 0 0 19 0 5 0 595197286 297574400 26237 33554432000 4194304 4338100 140734813377504 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 5.43
Current children cumulated vsize (KiB) 290600

[startup+3.10065 s]
/proc/loadavg: 7.84 8.03 7.99 9/186 28236
/proc/meminfo: memFree=19352456/32950928 swapFree=67111364/67111528
[pid=28231] ppid=28229 vsize=363556 CPUtime=11.81 cores=0,2,4,6
/proc/28231/stat : 28231 (ttsth-5-4) S 28229 28231 27953 0 -1 4202496 45733 0 0 0 1162 19 0 0 25 0 5 0 595197267 372281344 44728 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/28231/statm: 90889 44729 176 36 0 88794 0
[pid=28231/tid=28233] ppid=28229 vsize=363556 CPUtime=2.9 cores=0,2,4,6
/proc/28231/task/28233/stat : 28233 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 12161 0 0 0 288 2 0 0 24 0 5 0 595197286 372281344 44729 33554432000 4194304 4338100 140734813377504 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28231/tid=28234] ppid=28229 vsize=363556 CPUtime=2.89 cores=0,2,4,6
/proc/28231/task/28234/stat : 28234 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 6978 0 0 0 283 6 0 0 25 0 5 0 595197286 372281344 44729 33554432000 4194304 4338100 140734813377504 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28231/tid=28235] ppid=28229 vsize=363556 CPUtime=2.9 cores=0,2,4,6
/proc/28231/task/28235/stat : 28235 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 13009 0 0 0 288 2 0 0 24 0 5 0 595197286 372281344 44729 33554432000 4194304 4338100 140734813377504 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28231/tid=28236] ppid=28229 vsize=363556 CPUtime=2.9 cores=0,2,4,6
/proc/28231/task/28236/stat : 28236 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 4852 0 0 0 284 6 0 0 24 0 5 0 595197286 372281344 44729 33554432000 4194304 4338100 140734813377504 18446744073709551615 251366941819 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 11.81
Current children cumulated vsize (KiB) 363556

[startup+6.30106 s]
/proc/loadavg: 7.84 8.03 7.99 9/186 28236
/proc/meminfo: memFree=19109808/32950928 swapFree=67111364/67111528
[pid=28231] ppid=28229 vsize=567476 CPUtime=24.6 cores=0,2,4,6
/proc/28231/stat : 28231 (ttsth-5-4) S 28229 28231 27953 0 -1 4202496 80023 0 0 0 2419 41 0 0 25 0 5 0 595197267 581095424 79018 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/28231/statm: 141869 79019 176 36 0 139774 0
[pid=28231/tid=28233] ppid=28229 vsize=567476 CPUtime=6.1 cores=0,2,4,6
/proc/28231/task/28233/stat : 28233 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 23636 0 0 0 605 5 0 0 25 0 5 0 595197286 581095424 79019 33554432000 4194304 4338100 140734813377504 18446744073709551615 251366941730 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28231/tid=28234] ppid=28229 vsize=567476 CPUtime=6.08 cores=0,2,4,6
/proc/28231/task/28234/stat : 28234 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 12941 0 0 0 593 15 0 0 25 0 5 0 595197286 581095424 79021 33554432000 4194304 4338100 140734813377504 18446744073709551615 4284509 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28231/tid=28235] ppid=28229 vsize=567476 CPUtime=6.1 cores=0,2,4,6
/proc/28231/task/28235/stat : 28235 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 24744 0 0 0 606 4 0 0 25 0 5 0 595197286 581095424 79022 33554432000 4194304 4338100 140734813377504 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28231/tid=28236] ppid=28229 vsize=567476 CPUtime=6.11 cores=0,2,4,6
/proc/28231/task/28236/stat : 28236 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 9971 0 0 0 596 15 0 0 25 0 5 0 595197286 581095424 79022 33554432000 4194304 4338100 140734813377504 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 24.6
Current children cumulated vsize (KiB) 567476

[startup+12.7009 s]
/proc/loadavg: 7.87 8.03 7.99 9/186 28236
/proc/meminfo: memFree=18707800/32950928 swapFree=67111364/67111528
[pid=28231] ppid=28229 vsize=794488 CPUtime=50.16 cores=0,2,4,6
/proc/28231/stat : 28231 (ttsth-5-4) S 28229 28231 27953 0 -1 4202496 137145 0 0 0 4932 84 0 0 25 0 5 0 595197267 813555712 136140 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/28231/statm: 198622 136140 176 36 0 196527 0
[pid=28231/tid=28233] ppid=28229 vsize=794488 CPUtime=12.5 cores=0,2,4,6
/proc/28231/task/28233/stat : 28233 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 44970 0 0 0 1241 9 0 0 25 0 5 0 595197286 813555712 136140 33554432000 4194304 4338100 140734813377504 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28231/tid=28234] ppid=28229 vsize=794488 CPUtime=12.43 cores=0,2,4,6
/proc/28231/task/28234/stat : 28234 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 21777 0 0 0 1211 32 0 0 25 0 5 0 595197286 813555712 136140 33554432000 4194304 4338100 140734813377504 18446744073709551615 251366942133 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28231/tid=28235] ppid=28229 vsize=794488 CPUtime=12.5 cores=0,2,4,6
/proc/28231/task/28235/stat : 28235 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 43558 0 0 0 1243 7 0 0 25 0 5 0 595197286 813555712 136140 33554432000 4194304 4338100 140734813377504 18446744073709551615 251366942072 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28231/tid=28236] ppid=28229 vsize=794488 CPUtime=12.51 cores=0,2,4,6
/proc/28231/task/28236/stat : 28236 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 18106 0 0 0 1218 33 0 0 25 0 5 0 595197286 813555712 136140 33554432000 4194304 4338100 140734813377504 18446744073709551615 251366942121 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 50.16
Current children cumulated vsize (KiB) 794488

[startup+25.5005 s]
/proc/loadavg: 7.97 8.05 7.99 9/186 28236
/proc/meminfo: memFree=18112484/32950928 swapFree=67111364/67111528
[pid=28231] ppid=28229 vsize=1149688 CPUtime=101.28 cores=0,2,4,6
/proc/28231/stat : 28231 (ttsth-5-4) S 28229 28231 27953 0 -1 4202496 225254 0 0 0 9951 177 0 0 25 0 5 0 595197267 1177280512 224249 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/28231/statm: 287422 224249 176 36 0 285327 0
[pid=28231/tid=28233] ppid=28229 vsize=1149688 CPUtime=25.3 cores=0,2,4,6
/proc/28231/task/28233/stat : 28233 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 77614 0 0 0 2512 18 0 0 25 0 5 0 595197286 1177280512 224249 33554432000 4194304 4338100 140734813377504 18446744073709551615 4256731 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28231/tid=28234] ppid=28229 vsize=1149688 CPUtime=25.16 cores=0,2,4,6
/proc/28231/task/28234/stat : 28234 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 30915 0 0 0 2445 71 0 0 25 0 5 0 595197286 1177280512 224249 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379359526 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28231/tid=28235] ppid=28229 vsize=1149688 CPUtime=25.3 cores=0,2,4,6
/proc/28231/task/28235/stat : 28235 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 81408 0 0 0 2516 14 0 0 25 0 5 0 595197286 1177280512 224249 33554432000 4194304 4338100 140734813377504 18446744073709551615 251366941712 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28231/tid=28236] ppid=28229 vsize=1149688 CPUtime=25.3 cores=0,2,4,6
/proc/28231/task/28236/stat : 28236 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 26583 0 0 0 2459 71 0 0 25 0 5 0 595197286 1177280512 224249 33554432000 4194304 4338100 140734813377504 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 101.28
Current children cumulated vsize (KiB) 1149688

[startup+51.1008 s]
/proc/loadavg: 7.98 8.04 7.99 9/186 28237
/proc/meminfo: memFree=17377860/32950928 swapFree=67111364/67111528
[pid=28231] ppid=28229 vsize=1554008 CPUtime=203.49 cores=0,2,4,6
/proc/28231/stat : 28231 (ttsth-5-4) S 28229 28231 27953 0 -1 4202496 321580 0 0 0 20005 344 0 0 25 0 5 0 595197267 1591304192 320575 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/28231/statm: 388502 320575 176 36 0 386407 0
[pid=28231/tid=28233] ppid=28229 vsize=1554008 CPUtime=50.88 cores=0,2,4,6
/proc/28231/task/28233/stat : 28233 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 125837 0 0 0 5059 29 0 0 25 0 5 0 595197286 1591304192 320575 33554432000 4194304 4338100 140734813377504 18446744073709551615 251366942124 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28231/tid=28234] ppid=28229 vsize=1554008 CPUtime=50.61 cores=0,2,4,6
/proc/28231/task/28234/stat : 28234 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 36553 0 0 0 4917 144 0 0 25 0 5 0 595197286 1591304192 320575 33554432000 4194304 4338100 140734813377504 18446744073709551615 251367243268 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28231/tid=28235] ppid=28229 vsize=1554008 CPUtime=50.89 cores=0,2,4,6
/proc/28231/task/28235/stat : 28235 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 118689 0 0 0 5067 22 0 0 25 0 5 0 595197286 1591304192 320575 33554432000 4194304 4338100 140734813377504 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28231/tid=28236] ppid=28229 vsize=1554008 CPUtime=50.89 cores=0,2,4,6
/proc/28231/task/28236/stat : 28236 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 31767 0 0 0 4943 146 0 0 25 0 5 0 595197286 1591304192 320575 33554432000 4194304 4338100 140734813377504 18446744073709551615 4286350 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 203.49
Current children cumulated vsize (KiB) 1554008

[startup+102.306 s]
/proc/loadavg: 8.05 8.05 8.00 9/186 28238
/proc/meminfo: memFree=15786440/32950928 swapFree=67111364/67111528
[pid=28231] ppid=28229 vsize=2399928 CPUtime=407.94 cores=0,2,4,6
/proc/28231/stat : 28231 (ttsth-5-4) S 28229 28231 27953 0 -1 4202496 534848 0 0 0 40124 670 0 0 25 0 5 0 595197267 2457526272 533843 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/28231/statm: 599982 533845 176 36 0 597887 0
[pid=28231/tid=28233] ppid=28229 vsize=2399928 CPUtime=102.06 cores=0,2,4,6
/proc/28231/task/28233/stat : 28233 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 219405 0 0 0 10155 51 0 0 25 0 5 0 595197286 2457526272 533845 33554432000 4194304 4338100 140734813377504 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28231/tid=28234] ppid=28229 vsize=2399928 CPUtime=101.51 cores=0,2,4,6
/proc/28231/task/28234/stat : 28234 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 55304 0 0 0 9862 289 0 0 25 0 5 0 595197286 2457526272 533845 33554432000 4194304 4338100 140734813377504 18446744073709551615 251366942297 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28231/tid=28235] ppid=28229 vsize=2399928 CPUtime=102.08 cores=0,2,4,6
/proc/28231/task/28235/stat : 28235 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 208571 0 0 0 10170 38 0 0 25 0 5 0 595197286 2457526272 533845 33554432000 4194304 4338100 140734813377504 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28231/tid=28236] ppid=28229 vsize=2399928 CPUtime=102.08 cores=0,2,4,6
/proc/28231/task/28236/stat : 28236 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 42836 0 0 0 9918 290 0 0 25 0 5 0 595197286 2457526272 533845 33554432000 4194304 4338100 140734813377504 18446744073709551615 4263548 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 407.94
Current children cumulated vsize (KiB) 2399928

[startup+162.3 s]
/proc/loadavg: 8.07 8.05 8.00 9/186 28240
/proc/meminfo: memFree=14665096/32950928 swapFree=67111364/67111528
[pid=28231] ppid=28229 vsize=3218684 CPUtime=647.33 cores=0,2,4,6
/proc/28231/stat : 28231 (ttsth-5-4) S 28229 28231 27953 0 -1 4202496 742764 0 0 0 63710 1023 0 0 25 0 5 0 595197267 3295932416 741759 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/28231/statm: 804671 741759 176 36 0 802576 0
[pid=28231/tid=28233] ppid=28229 vsize=3218684 CPUtime=161.96 cores=0,2,4,6
/proc/28231/task/28233/stat : 28233 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 340841 0 0 0 16117 79 0 0 25 0 5 0 595197286 3295932416 741759 33554432000 4194304 4338100 140734813377504 18446744073709551615 4254141 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28231/tid=28234] ppid=28229 vsize=3218684 CPUtime=161.13 cores=0,2,4,6
/proc/28231/task/28234/stat : 28234 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 68878 0 0 0 15671 442 0 0 18 0 5 0 595197286 3295932416 741759 33554432000 4194304 4338100 140734813377504 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28231/tid=28235] ppid=28229 vsize=3218684 CPUtime=162.03 cores=0,2,4,6
/proc/28231/task/28235/stat : 28235 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 264352 0 0 0 16156 47 0 0 25 0 5 0 595197286 3295932416 741759 33554432000 4194304 4338100 140734813377504 18446744073709551615 4256753 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28231/tid=28236] ppid=28229 vsize=3218684 CPUtime=161.98 cores=0,2,4,6
/proc/28231/task/28236/stat : 28236 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 59959 0 0 0 15746 452 0 0 25 0 5 0 595197286 3295932416 741759 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 647.33
Current children cumulated vsize (KiB) 3218684


################
# More data... #
################

/proc/28231/task/28233/stat : 28233 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 1005394 0 0 0 75689 272 0 0 25 0 5 0 595197286 8846790656 2107141 33554432000 4194304 4338100 140734813377504 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28231/tid=28234] ppid=28229 vsize=8639444 CPUtime=756.58 cores=0,2,4,6
/proc/28231/task/28234/stat : 28234 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 156235 0 0 0 74547 1111 0 0 25 0 5 0 595197286 8846790656 2107141 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28231/tid=28235] ppid=28229 vsize=8639444 CPUtime=760.85 cores=0,2,4,6
/proc/28231/task/28235/stat : 28235 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 795490 0 0 0 75927 158 0 0 25 0 5 0 595197286 8846790656 2107141 33554432000 4194304 4338100 140734813377504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28231/tid=28236] ppid=28229 vsize=8639444 CPUtime=760.29 cores=0,2,4,6
/proc/28231/task/28236/stat : 28236 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 142293 0 0 0 74774 1255 0 0 18 0 5 0 595197286 8846790656 2107141 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3037.55
Current children cumulated vsize (KiB) 8639444

[startup+822.3 s]
/proc/loadavg: 8.16 8.11 8.03 9/186 28258
/proc/meminfo: memFree=5179780/32950928 swapFree=67111364/67111528
[pid=28231] ppid=28229 vsize=8873028 CPUtime=3276.55 cores=0,2,4,6
/proc/28231/stat : 28231 (ttsth-5-4) S 28229 28231 27953 0 -1 4202496 2167486 0 0 0 324761 2894 0 0 25 0 5 0 595197267 9085980672 2166481 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/28231/statm: 2218257 2166481 176 36 0 2216162 0
[pid=28231/tid=28233] ppid=28229 vsize=8873028 CPUtime=819.27 cores=0,2,4,6
/proc/28231/task/28233/stat : 28233 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 1025183 0 0 0 81646 281 0 0 25 0 5 0 595197286 9085980672 2166481 33554432000 4194304 4338100 140734813377504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28231/tid=28234] ppid=28229 vsize=8873028 CPUtime=816.44 cores=0,2,4,6
/proc/28231/task/28234/stat : 28234 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 156710 0 0 0 80478 1166 0 0 25 0 5 0 595197286 9085980672 2166481 33554432000 4194304 4338100 140734813377504 18446744073709551615 251366941819 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28231/tid=28235] ppid=28229 vsize=8873028 CPUtime=820.8 cores=0,2,4,6
/proc/28231/task/28235/stat : 28235 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 834092 0 0 0 81911 169 0 0 25 0 5 0 595197286 9085980672 2166481 33554432000 4194304 4338100 140734813377504 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28231/tid=28236] ppid=28229 vsize=8873028 CPUtime=819.82 cores=0,2,4,6
/proc/28231/task/28236/stat : 28236 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 142767 0 0 0 80706 1276 0 0 25 0 5 0 595197286 9085980672 2166481 33554432000 4194304 4338100 140734813377504 18446744073709551615 251367243232 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3276.55
Current children cumulated vsize (KiB) 8873028

[startup+882.3 s]
/proc/loadavg: 8.06 8.09 8.03 9/186 28259
/proc/meminfo: memFree=4946024/32950928 swapFree=67111364/67111528
[pid=28231] ppid=28229 vsize=9132228 CPUtime=3515.33 cores=0,2,4,6
/proc/28231/stat : 28231 (ttsth-5-4) S 28229 28231 27953 0 -1 4202496 2215992 0 0 0 348526 3007 0 0 25 0 5 0 595197267 9351401472 2214987 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/28231/statm: 2283057 2214987 176 36 0 2280962 0
[pid=28231/tid=28233] ppid=28229 vsize=9132228 CPUtime=878.83 cores=0,2,4,6
/proc/28231/task/28233/stat : 28233 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 1054740 0 0 0 87590 293 0 0 25 0 5 0 595197286 9351401472 2214987 33554432000 4194304 4338100 140734813377504 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28231/tid=28234] ppid=28229 vsize=9132228 CPUtime=875.89 cores=0,2,4,6
/proc/28231/task/28234/stat : 28234 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 157891 0 0 0 86387 1202 0 0 18 0 5 0 595197286 9351401472 2214987 33554432000 4194304 4338100 140734813377504 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28231/tid=28235] ppid=28229 vsize=9132228 CPUtime=880.72 cores=0,2,4,6
/proc/28231/task/28235/stat : 28235 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 848859 0 0 0 87899 173 0 0 25 0 5 0 595197286 9351401472 2214987 33554432000 4194304 4338100 140734813377504 18446744073709551615 4256331 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28231/tid=28236] ppid=28229 vsize=9132228 CPUtime=879.66 cores=0,2,4,6
/proc/28231/task/28236/stat : 28236 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 145768 0 0 0 86629 1337 0 0 25 0 5 0 595197286 9351401472 2214987 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3515.33
Current children cumulated vsize (KiB) 9132228

[startup+942.3 s]
/proc/loadavg: 8.19 8.12 8.04 9/186 28261
/proc/meminfo: memFree=4517752/32950928 swapFree=67111364/67111528
[pid=28231] ppid=28229 vsize=9309112 CPUtime=3754.47 cores=0,2,4,6
/proc/28231/stat : 28231 (ttsth-5-4) S 28229 28231 27953 0 -1 4202496 2264193 0 0 0 372375 3072 0 0 25 0 5 0 595197267 9532530688 2263188 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/28231/statm: 2327278 2263188 176 36 0 2325183 0
[pid=28231/tid=28233] ppid=28229 vsize=9309112 CPUtime=938.48 cores=0,2,4,6
/proc/28231/task/28233/stat : 28233 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 1072717 0 0 0 93547 301 0 0 25 0 5 0 595197286 9532530688 2263188 33554432000 4194304 4338100 140734813377504 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28231/tid=28234] ppid=28229 vsize=9309112 CPUtime=935.46 cores=0,2,4,6
/proc/28231/task/28234/stat : 28234 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 158965 0 0 0 92339 1207 0 0 18 0 5 0 595197286 9532530688 2263188 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28231/tid=28235] ppid=28229 vsize=9309112 CPUtime=940.71 cores=0,2,4,6
/proc/28231/task/28235/stat : 28235 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 876106 0 0 0 93891 180 0 0 25 0 5 0 595197286 9532530688 2263188 33554432000 4194304 4338100 140734813377504 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28231/tid=28236] ppid=28229 vsize=9309112 CPUtime=939.6 cores=0,2,4,6
/proc/28231/task/28236/stat : 28236 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 147671 0 0 0 92578 1382 0 0 25 0 5 0 595197286 9532530688 2263188 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3754.47
Current children cumulated vsize (KiB) 9309112

[startup+1002.3 s]
/proc/loadavg: 8.12 8.11 8.04 9/186 28262
/proc/meminfo: memFree=4027520/32950928 swapFree=67111364/67111528
[pid=28231] ppid=28229 vsize=9707952 CPUtime=3993.73 cores=0,2,4,6
/proc/28231/stat : 28231 (ttsth-5-4) S 28229 28231 27953 0 -1 4202496 2364577 0 0 0 396172 3201 0 0 25 0 5 0 595197267 9940942848 2363572 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/28231/statm: 2426988 2363572 176 36 0 2424893 0
[pid=28231/tid=28233] ppid=28229 vsize=9707952 CPUtime=998.19 cores=0,2,4,6
/proc/28231/task/28233/stat : 28233 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 1115987 0 0 0 99506 313 0 0 25 0 5 0 595197286 9940942848 2363572 33554432000 4194304 4338100 140734813377504 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28231/tid=28234] ppid=28229 vsize=9707952 CPUtime=995.21 cores=0,2,4,6
/proc/28231/task/28234/stat : 28234 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 159300 0 0 0 98274 1247 0 0 18 0 5 0 595197286 9940942848 2363572 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28231/tid=28235] ppid=28229 vsize=9707952 CPUtime=1000.7 cores=0,2,4,6
/proc/28231/task/28235/stat : 28235 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 932540 0 0 0 99875 195 0 0 25 0 5 0 595197286 9940942848 2363572 33554432000 4194304 4338100 140734813377504 18446744073709551615 251366942446 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28231/tid=28236] ppid=28229 vsize=9707952 CPUtime=999.42 cores=0,2,4,6
/proc/28231/task/28236/stat : 28236 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 148016 0 0 0 98498 1444 0 0 25 0 5 0 595197286 9940942848 2363572 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3993.73
Current children cumulated vsize (KiB) 9707952

[startup+1062.3 s]
/proc/loadavg: 8.09 8.10 8.04 9/186 28264
/proc/meminfo: memFree=3668684/32950928 swapFree=67111364/67111528
[pid=28231] ppid=28229 vsize=9991188 CPUtime=4232.94 cores=0,2,4,6
/proc/28231/stat : 28231 (ttsth-5-4) S 28229 28231 27953 0 -1 4202496 2436615 0 0 0 420027 3267 0 0 25 0 5 0 595197267 10230976512 2435610 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/28231/statm: 2497797 2435610 176 36 0 2495702 0
[pid=28231/tid=28233] ppid=28229 vsize=9991188 CPUtime=1057.81 cores=0,2,4,6
/proc/28231/task/28233/stat : 28233 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 1152378 0 0 0 105453 328 0 0 25 0 5 0 595197286 10230976512 2435610 33554432000 4194304 4338100 140734813377504 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28231/tid=28234] ppid=28229 vsize=9991188 CPUtime=1054.86 cores=0,2,4,6
/proc/28231/task/28234/stat : 28234 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 159629 0 0 0 104234 1252 0 0 18 0 5 0 595197286 10230976512 2435610 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28231/tid=28235] ppid=28229 vsize=9991188 CPUtime=1060.67 cores=0,2,4,6
/proc/28231/task/28235/stat : 28235 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 967528 0 0 0 105864 203 0 0 25 0 5 0 595197286 10230976512 2435610 33554432000 4194304 4338100 140734813377504 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28231/tid=28236] ppid=28229 vsize=9991188 CPUtime=1059.37 cores=0,2,4,6
/proc/28231/task/28236/stat : 28236 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 148346 0 0 0 104456 1481 0 0 25 0 5 0 595197286 10230976512 2435610 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4232.94
Current children cumulated vsize (KiB) 9991188

[startup+1122.3 s]
/proc/loadavg: 8.12 8.11 8.04 9/186 28266
/proc/meminfo: memFree=3138604/32950928 swapFree=67111364/67111528
[pid=28231] ppid=28229 vsize=10456244 CPUtime=4471.55 cores=0,2,4,6
/proc/28231/stat : 28231 (ttsth-5-4) S 28229 28231 27953 0 -1 4202496 2554177 0 0 0 443795 3360 0 0 25 0 5 0 595197267 10707193856 2553172 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/28231/statm: 2614061 2553172 176 36 0 2611966 0
[pid=28231/tid=28233] ppid=28229 vsize=10456244 CPUtime=1117.31 cores=0,2,4,6
/proc/28231/task/28233/stat : 28233 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 1203087 0 0 0 111387 344 0 0 25 0 5 0 595197286 10707193856 2553172 33554432000 4194304 4338100 140734813377504 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28231/tid=28234] ppid=28229 vsize=10456244 CPUtime=1114.24 cores=0,2,4,6
/proc/28231/task/28234/stat : 28234 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 179472 0 0 0 110160 1264 0 0 18 0 5 0 595197286 10707193856 2553172 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28231/tid=28235] ppid=28229 vsize=10456244 CPUtime=1120.57 cores=0,2,4,6
/proc/28231/task/28235/stat : 28235 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 1014025 0 0 0 111841 216 0 0 25 0 5 0 595197286 10707193856 2553172 33554432000 4194304 4338100 140734813377504 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28231/tid=28236] ppid=28229 vsize=10456244 CPUtime=1119.22 cores=0,2,4,6
/proc/28231/task/28236/stat : 28236 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 148859 0 0 0 110388 1534 0 0 25 0 5 0 595197286 10707193856 2553173 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4471.55
Current children cumulated vsize (KiB) 10456244

[startup+1182.3 s]
/proc/loadavg: 8.09 8.10 8.04 9/186 28267
/proc/meminfo: memFree=2692556/32950928 swapFree=67111364/67111528
[pid=28231] ppid=28229 vsize=10827460 CPUtime=4710.3 cores=0,2,4,6
/proc/28231/stat : 28231 (ttsth-5-4) S 28229 28231 27953 0 -1 4202496 2648643 0 0 0 467590 3440 0 0 25 0 5 0 595197267 11087319040 2647638 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/28231/statm: 2706865 2647638 176 36 0 2704770 0
[pid=28231/tid=28233] ppid=28229 vsize=10827460 CPUtime=1176.86 cores=0,2,4,6
/proc/28231/task/28233/stat : 28233 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 1238470 0 0 0 117330 356 0 0 25 0 5 0 595197286 11087319040 2647638 33554432000 4194304 4338100 140734813377504 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28231/tid=28234] ppid=28229 vsize=10827460 CPUtime=1173.68 cores=0,2,4,6
/proc/28231/task/28234/stat : 28234 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 194481 0 0 0 116092 1276 0 0 18 0 5 0 595197286 11087319040 2647638 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28231/tid=28235] ppid=28229 vsize=10827460 CPUtime=1180.45 cores=0,2,4,6
/proc/28231/task/28235/stat : 28235 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 1047068 0 0 0 117821 224 0 0 25 0 5 0 595197286 11087319040 2647638 33554432000 4194304 4338100 140734813377504 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28231/tid=28236] ppid=28229 vsize=10827460 CPUtime=1179.1 cores=0,2,4,6
/proc/28231/task/28236/stat : 28236 (ttsth-5-4) R 28229 28231 27953 0 -1 4202560 159890 0 0 0 116327 1583 0 0 25 0 5 0 595197286 11087319040 2647638 33554432000 4194304 4338100 140734813377504 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4710.3
Current children cumulated vsize (KiB) 10827460



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1204.9 s]
/proc/loadavg: 7.74 8.03 8.02 2/173 28286
/proc/meminfo: memFree=10577176/32950928 swapFree=67111364/67111528
[pid=28231] ppid=28229 vsize=0 CPUtime=4800.13 cores=0,2,4,6
/proc/28231/stat : 28231 (ttsth-5-4) Z 28229 28231 27953 0 -1 4203548 2680850 0 0 0 476526 3487 0 0 18 0 2 0 595197267 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/28231/statm: 0 0 0 0 0 0 0
[pid=28231/tid=28234] ppid=28229 vsize=0 CPUtime=1196.09 cores=0,2,4,6
/proc/28231/task/28234/stat : 28234 (ttsth-5-4) R 28229 28231 27953 0 -1 4203588 200847 0 0 0 118322 1287 0 0 18 0 2 0 595197286 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.13
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1205.01 s]
/proc/loadavg: 7.74 8.03 8.02 2/173 28286
/proc/meminfo: memFree=10577176/32950928 swapFree=67111364/67111528
[pid=28231] ppid=28229 vsize=0 CPUtime=4800.23 cores=0,2,4,6
/proc/28231/stat : 28231 (ttsth-5-4) Z 28229 28231 27953 0 -1 4203548 2680850 0 0 0 476526 3497 0 0 18 0 2 0 595197267 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/28231/statm: 0 0 0 0 0 0 0
[pid=28231/tid=28234] ppid=28229 vsize=0 CPUtime=1196.2 cores=0,2,4,6
/proc/28231/task/28234/stat : 28234 (ttsth-5-4) R 28229 28231 27953 0 -1 4203588 200847 0 0 0 118322 1298 0 0 18 0 2 0 595197286 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.23
Current children cumulated vsize (KiB) 0

[startup+1205.2 s]
/proc/loadavg: 7.74 8.03 8.02 2/173 28286
/proc/meminfo: memFree=10577176/32950928 swapFree=67111364/67111528
[pid=28231] ppid=28229 vsize=0 CPUtime=4800.42 cores=0,2,4,6
/proc/28231/stat : 28231 (ttsth-5-4) Z 28229 28231 27953 0 -1 4203548 2680850 0 0 0 476526 3516 0 0 18 0 2 0 595197267 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/28231/statm: 0 0 0 0 0 0 0
[pid=28231/tid=28234] ppid=28229 vsize=0 CPUtime=1196.39 cores=0,2,4,6
/proc/28231/task/28234/stat : 28234 (ttsth-5-4) R 28229 28231 27953 0 -1 4203588 200847 0 0 0 118322 1317 0 0 19 0 2 0 595197286 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.42
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1205.24
CPU time (s): 4800.47
CPU user time (s): 4765.27
CPU system time (s): 35.2056
CPU usage (%): 398.301
Max. virtual memory (cumulated for all children) (KiB): 10956632

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4765.27
system time used= 35.2056
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2680850
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 18118
involuntary context switches= 22681

runsolver used 2.01069 second user time and 4.55731 second system time

The end

Launcher Data 

Begin job on node113 at 2011-04-29 06:01:06
IDJOB=3258130
IDBENCH=71307
IDSOLVER=1654
FILE ID=node113/3258130-1304049666
RUNJOBID= node113-1304047252-27971
PBS_JOBID= 13170405
Free space on /tmp= 71164 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/rbsat/random/unforced/rbsat-v760c43649g7.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3258130-1304049666/watcher-3258130-1304049666 -o /tmp/evaluation-result-3258130-1304049666/solver-3258130-1304049666 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258130-1304049666.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 57c91698da60cd77ab278c961fa35654
RANDOM SEED=372908664

node113.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      19547992 kB
Buffers:       1742904 kB
Cached:       10820452 kB
SwapCached:        164 kB
Active:        6491512 kB
Inactive:      6321936 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      19547992 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:            1888 kB
Writeback:           0 kB
AnonPages:      248836 kB
Mapped:          15208 kB
Slab:           526928 kB
PageTables:       4652 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   443692 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71168 MiB
End job on node113 at 2011-04-29 06:21:13