Trace number 3263762

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.34 1245.43

General information on the benchmark

NameSAT03/handmade/stanion/hwb/
hwb-n28-02-S818962541.shuffled-as.sat03-1628.cnf
MD5SUMd705043763df769aab1e385bbbc2b11e
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables184
Number of clauses896
Sum of the clauses size3042
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 214
Number of clauses of size 3514
Number of clauses of size 4368
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263762-1304097856.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 184
0.00/0.00	c Clauses = 896
0.00/0.00	c Literals = 3042
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 184
0.00/0.00	c Clauses = 896
0.00/0.00	c Literals = 3042
0.00/0.00	c a2
0.00/0.00	c a1
0.19/0.22	c Independent sub-problems = 1
0.19/0.22	c Sub-problem with 184 variables and 896 clauses
0.19/0.22	c using permutation method 4
0.19/0.22	c count = 3
0.19/0.22	c Ordering finished 0.220 seconds
0.19/0.22	c mapped 0.220 seconds
0.19/0.22	c permuted 0.220 seconds
0.19/0.22	c about to sort
0.19/0.22	c sorted
0.19/0.23	c clauses now 896
0.19/0.23	c init finished 0.220 seconds
0.19/0.23	c tree built 0.220 seconds
0.19/0.23	c 
0.19/0.23	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.19/0.23	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.19/0.23	c ................................ | ..........                    |                                  0.2         0.0             
0.19/0.23	c ................................ | ..........                  1 |                                  0.2         4.5             
0.19/0.23	c ................................ | ..........                  2 |                                  0.2         9.1             
0.19/0.23	c ................................ | ..........                  3 |                                  0.2        13.6             
0.19/0.23	c ................................ | ..........                  4 |                                  0.2        18.2             
0.19/0.23	c ................................ | ..........                  5 |                                  0.2        22.7             
0.19/0.26	c ................................ | ..........               1000 | 16           626                 0.2      4166.7        24851
96.10/24.62	c ................................ | ..........            2084000 | 106009       2508647            96.0     21699.3     18609551
191.92/49.21	c ................................ | ..........            3756000 | 195625       4569553           192.0     19562.5     23568671
287.94/73.92	c ................................ | ..........            5255000 | 277207       6478943           288.1     18243.4     26136196
383.76/98.58	c ................................ | ..........            6805000 | 356865       8306117           384.1     17719.0     31481655
480.02/123.41	c ................................ | *.........            8260000 | 431558       10114085          480.1     17205.8     34751395
575.98/148.20	c ................................ | *.........            9660000 | 505684       11864928          576.0     16770.3     37419223
671.83/173.25	c ................................ | *.........           11089000 | 584082       13554407          672.0     16500.8     42507514
767.76/198.26	c ................................ | *.........           12480000 | 661451       15278234          768.0     16249.4     44598309
863.70/223.08	c ................................ | *.........           13793000 | 734960       16966488          864.0     15963.2     46279072
959.73/247.87	c ................................ | **........           15114000 | 809468       18631960          960.0     15743.6     49504279
1057.00/273.16	c ................................ | **........           16520000 | 883716       20293309         1057.2     15625.4     53069902
1151.99/297.81	c ................................ | **........           17835000 | 951837       21875506         1152.0     15481.2     55698706
1248.02/322.81	c ................................ | **........           19118000 | 1020129      23473623         1248.0     15318.3     58653336
1343.85/347.75	c ................................ | **........           20386000 | 1087897      24992851         1344.1     15167.4     62566056
1439.82/372.85	c ................................ | ***.......           22006000 | 1168595      26562451         1440.0     15281.9     68702469
1536.58/397.84	c ................................ | ***.......           23430000 | 1241917      28185421         1536.8     15246.4     70070294
1631.84/422.45	c ................................ | ***.......           24823000 | 1314381      29739632         1632.0     15209.9     72244256
1727.99/447.21	c ................................ | ***.......           26263000 | 1391697      31352754         1728.0     15198.1     75903700
1823.69/472.08	c ................................ | ***.......           27789000 | 1470500      32982080         1824.0     15234.9     80693751
1919.99/497.13	c ................................ | ****......           29131000 | 1541734      34539838         1920.0     15172.3     84588728
2015.99/522.20	c ................................ | ****......           30568000 | 1619561      36102501         2016.0     15162.6     90437692
2111.87/547.36	c ................................ | ****......           31896000 | 1685463      37586841         2112.0     15102.0     92308648
2207.98/572.54	c ................................ | ****......           33298000 | 1752821      39122259         2208.1     15080.2     95456964
2304.18/597.65	c ................................ | ****......           34650000 | 1821410      40587641         2304.4     15036.6     98219941
2399.90/622.32	c ................................ | *****.....           35898000 | 1884609      42130581         2400.0     14957.5    100017877
2496.25/647.29	c ................................ | *****.....           37140000 | 1949920      43634198         2496.6     14876.1    101080504
2592.86/672.20	c ................................ | *****.....           38540000 | 2020111      45173406         2592.9     14863.7    105078918
2687.90/696.93	c ................................ | *****.....           39959000 | 2090578      46771113         2688.0     14865.6    109843656
2783.96/722.11	c ................................ | *****.....           41299000 | 2158101      48260728         2784.0     14834.4    113503038
2879.69/746.98	c ................................ | ******....           42498000 | 2221240      49785264         2880.0     14756.2    113596158
2976.02/772.20	c ................................ | ******....           43856000 | 2291613      51307175         2976.0     14736.4    117680864
3071.83/797.25	c ................................ | ******....           45109000 | 2359100      52785318         3072.0     14683.7    119965596
3167.91/822.03	c ................................ | ******....           46238000 | 2420171      54219325         3168.1     14595.1    120491002
3263.88/846.78	c ................................ | ******....           47348000 | 2480594      55685767         3264.0     14506.1    121403165
3360.44/871.97	c ................................ | *******...           48450000 | 2540912      57078494         3360.8     14416.4    124101148
3455.94/896.53	c ................................ | *******...           49594000 | 2603356      58471862         3456.1     14349.9    126259860
3551.76/921.57	c ................................ | *******...           50671000 | 2661827      59782189         3552.0     14265.4    129163676
3647.94/946.35	c ................................ | *******...           51711000 | 2718515      61111973         3648.1     14174.9    129163676
3743.77/971.16	c ................................ | *******...           52739000 | 2773699      62384005         3744.0     14086.2    132130714
3839.88/996.04	c ................................ | ********..           53870000 | 2831729      63685168         3840.1     14028.5    133930918
3936.12/1020.92	c ................................ | ********..           54905000 | 2887654      64993230         3936.1     13949.1    135115723
4031.76/1045.96	c ................................ | ********..           55936000 | 2942114      66246187         4032.0     13872.9    137691247
4127.72/1070.99	c ................................ | ********..           56912000 | 2994252      67524636         4128.0     13786.7    138280014
4224.00/1095.81	c ................................ | ********..           57937000 | 3049913      68832951         4224.1     13716.0    140505505
4319.67/1120.69	c ................................ | *********.           58989000 | 3105184      70081173         4320.0     13654.7    142322636
4415.78/1145.36	c ................................ | *********.           59992000 | 3159732      71402725         4416.1     13585.0    142322636
4511.80/1170.16	c ................................ | *********.           60991000 | 3213733      72662157         4512.1     13517.4    143819828
4607.91/1194.84	c ................................ | *********.           61983000 | 3268103      73945946         4608.1     13450.9    144452183
4703.73/1220.08	c ................................ | *********.           63023000 | 3322676      75220180         4704.0     13397.7    146718151
4799.96/1245.12	c CPU limit (4800) reached
4799.96/1245.12	c Time = 4800.02 seconds
4799.96/1245.12	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3263762-1304097856/watcher-3263762-1304097856 -o /tmp/evaluation-result-3263762-1304097856/solver-3263762-1304097856 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263762-1304097856.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.32 7.77 8.01 3/177 10673
/proc/meminfo: memFree=25017156/32951124 swapFree=67111352/67111528
[pid=10673] ppid=10670 vsize=8972 CPUtime=0 cores=0,2,4,6
/proc/10673/stat : 10673 (ttsth-5-4) R 10670 10673 10639 0 -1 4202496 312 0 0 0 0 0 0 0 23 0 1 0 521383800 9187328 254 33554432000 4194304 4338100 140734972272528 18446744073709551615 4296000 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/10673/statm: 2243 254 151 36 0 148 0

[startup+0.094719 s]
/proc/loadavg: 6.32 7.77 8.01 3/177 10673
/proc/meminfo: memFree=25017156/32951124 swapFree=67111352/67111528
[pid=10673] ppid=10670 vsize=8972 CPUtime=0.09 cores=0,2,4,6
/proc/10673/stat : 10673 (ttsth-5-4) R 10670 10673 10639 0 -1 4202496 313 0 0 0 9 0 0 0 24 0 1 0 521383800 9187328 255 33554432000 4194304 4338100 140734972272528 18446744073709551615 4295924 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/10673/statm: 2243 255 152 36 0 148 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 8972

[startup+0.100701 s]
/proc/loadavg: 6.32 7.77 8.01 3/177 10673
/proc/meminfo: memFree=25017156/32951124 swapFree=67111352/67111528
[pid=10673] ppid=10670 vsize=8972 CPUtime=0.09 cores=0,2,4,6
/proc/10673/stat : 10673 (ttsth-5-4) R 10670 10673 10639 0 -1 4202496 313 0 0 0 9 0 0 0 24 0 1 0 521383800 9187328 255 33554432000 4194304 4338100 140734972272528 18446744073709551615 4295890 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/10673/statm: 2243 255 152 36 0 148 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 8972

[startup+0.300602 s]
/proc/loadavg: 6.32 7.77 8.01 3/177 10673
/proc/meminfo: memFree=25017156/32951124 swapFree=67111352/67111528
[pid=10673] ppid=10670 vsize=196040 CPUtime=0.49 cores=0,2,4,6
/proc/10673/stat : 10673 (ttsth-5-4) S 10670 10673 10639 0 -1 4202496 3336 0 0 0 49 0 0 0 25 0 5 0 521383800 200744960 3278 33554432000 4194304 4338100 140734972272528 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/10673/statm: 49010 3279 173 36 0 46915 0
Current children cumulated CPU time (s) 0.49
Current children cumulated vsize (KiB) 196040

[startup+0.70053 s]
/proc/loadavg: 6.32 7.77 8.01 3/177 10673
/proc/meminfo: memFree=25017156/32951124 swapFree=67111352/67111528
[pid=10673] ppid=10670 vsize=296628 CPUtime=2.09 cores=0,2,4,6
/proc/10673/stat : 10673 (ttsth-5-4) S 10670 10673 10639 0 -1 4202496 12058 0 0 0 208 1 0 0 25 0 5 0 521383800 303747072 12000 33554432000 4194304 4338100 140734972272528 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/10673/statm: 74157 12000 173 36 0 72062 0
Current children cumulated CPU time (s) 2.09
Current children cumulated vsize (KiB) 296628

[startup+1.50038 s]
/proc/loadavg: 6.32 7.77 8.01 9/190 10686
/proc/meminfo: memFree=24924416/32951124 swapFree=67111352/67111528
[pid=10673] ppid=10670 vsize=347576 CPUtime=5.27 cores=0,2,4,6
/proc/10673/stat : 10673 (ttsth-5-4) S 10670 10673 10639 0 -1 4202496 24562 0 0 0 523 4 0 0 25 0 5 0 521383800 355917824 24504 33554432000 4194304 4338100 140734972272528 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/10673/statm: 86894 24504 173 36 0 84799 0
[pid=10673/tid=10679] ppid=10670 vsize=347576 CPUtime=1.26 cores=0,2,4,6
/proc/10673/task/10679/stat : 10679 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 6309 0 0 0 125 1 0 0 19 0 5 0 521383823 355917824 24504 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=10673/tid=10680] ppid=10670 vsize=347576 CPUtime=1.24 cores=0,2,4,6
/proc/10673/task/10680/stat : 10680 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 6229 0 0 0 124 0 0 0 25 0 5 0 521383823 355917824 24504 33554432000 4194304 4338100 140734972272528 18446744073709551615 4255963 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=10673/tid=10681] ppid=10670 vsize=347576 CPUtime=1.25 cores=0,2,4,6
/proc/10673/task/10681/stat : 10681 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 6135 0 0 0 124 1 0 0 19 0 5 0 521383823 355917824 24505 33554432000 4194304 4338100 140734972272528 18446744073709551615 246516530656 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=10673/tid=10682] ppid=10670 vsize=347576 CPUtime=1.25 cores=0,2,4,6
/proc/10673/task/10682/stat : 10682 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 5015 0 0 0 125 0 0 0 19 0 5 0 521383823 355917824 24505 33554432000 4194304 4338100 140734972272528 18446744073709551615 246516229496 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 5.27
Current children cumulated vsize (KiB) 347576

[startup+3.10109 s]
/proc/loadavg: 6.32 7.77 8.01 9/190 10686
/proc/meminfo: memFree=24835640/32951124 swapFree=67111352/67111528
[pid=10673] ppid=10670 vsize=442284 CPUtime=11.63 cores=0,2,4,6
/proc/10673/stat : 10673 (ttsth-5-4) S 10670 10673 10639 0 -1 4202496 48537 0 0 0 1155 8 0 0 25 0 5 0 521383800 452898816 48431 33554432000 4194304 4338100 140734972272528 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/10673/statm: 110571 48431 175 36 0 108476 0
[pid=10673/tid=10679] ppid=10670 vsize=442284 CPUtime=2.84 cores=0,2,4,6
/proc/10673/task/10679/stat : 10679 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 12868 0 0 0 282 2 0 0 24 0 5 0 521383823 452898816 48431 33554432000 4194304 4338100 140734972272528 18446744073709551615 246516229506 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=10673/tid=10680] ppid=10670 vsize=442284 CPUtime=2.83 cores=0,2,4,6
/proc/10673/task/10680/stat : 10680 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 13260 0 0 0 282 1 0 0 20 0 5 0 521383823 452898816 48431 33554432000 4194304 4338100 140734972272528 18446744073709551615 4245241 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=10673/tid=10681] ppid=10670 vsize=442284 CPUtime=2.86 cores=0,2,4,6
/proc/10673/task/10681/stat : 10681 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 11694 0 0 0 283 3 0 0 18 0 5 0 521383823 452898816 48431 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=10673/tid=10682] ppid=10670 vsize=442284 CPUtime=2.84 cores=0,2,4,6
/proc/10673/task/10682/stat : 10682 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 9841 0 0 0 283 1 0 0 24 0 5 0 521383823 452898816 48431 33554432000 4194304 4338100 140734972272528 18446744073709551615 246516229588 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 11.63
Current children cumulated vsize (KiB) 442284

[startup+6.30051 s]
/proc/loadavg: 6.46 7.77 8.01 9/190 10686
/proc/meminfo: memFree=24663984/32951124 swapFree=67111352/67111528
[pid=10673] ppid=10670 vsize=521172 CPUtime=24.24 cores=0,2,4,6
/proc/10673/stat : 10673 (ttsth-5-4) S 10670 10673 10639 0 -1 4202496 69285 0 0 0 2411 13 0 0 25 0 5 0 521383800 533680128 69179 33554432000 4194304 4338100 140734972272528 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/10673/statm: 130293 69179 175 36 0 128198 0
[pid=10673/tid=10679] ppid=10670 vsize=521172 CPUtime=5.98 cores=0,2,4,6
/proc/10673/task/10679/stat : 10679 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 20620 0 0 0 594 4 0 0 18 0 5 0 521383823 533680128 69179 33554432000 4194304 4338100 140734972272528 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=10673/tid=10680] ppid=10670 vsize=521172 CPUtime=5.97 cores=0,2,4,6
/proc/10673/task/10680/stat : 10680 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 21202 0 0 0 595 2 0 0 18 0 5 0 521383823 533680128 69179 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=10673/tid=10681] ppid=10670 vsize=521172 CPUtime=6.02 cores=0,2,4,6
/proc/10673/task/10681/stat : 10681 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 14995 0 0 0 598 4 0 0 18 0 5 0 521383823 533680128 69179 33554432000 4194304 4338100 140734972272528 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=10673/tid=10682] ppid=10670 vsize=521172 CPUtime=6.01 cores=0,2,4,6
/proc/10673/task/10682/stat : 10682 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 11594 0 0 0 599 2 0 0 24 0 5 0 521383823 533680128 69180 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 24.24
Current children cumulated vsize (KiB) 521172

[startup+12.7003 s]
/proc/loadavg: 6.58 7.77 8.01 10/190 10687
/proc/meminfo: memFree=24328836/32951124 swapFree=67111352/67111528
[pid=10673] ppid=10670 vsize=613160 CPUtime=49.45 cores=0,2,4,6
/proc/10673/stat : 10673 (ttsth-5-4) S 10670 10673 10639 0 -1 4202496 93133 0 0 0 4925 20 0 0 25 0 5 0 521383800 627875840 93027 33554432000 4194304 4338100 140734972272528 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/10673/statm: 153290 93027 175 36 0 151195 0
[pid=10673/tid=10679] ppid=10670 vsize=613160 CPUtime=12.26 cores=0,2,4,6
/proc/10673/task/10679/stat : 10679 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 30441 0 0 0 1220 6 0 0 25 0 5 0 521383823 627875840 93027 33554432000 4194304 4338100 140734972272528 18446744073709551615 246516229506 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=10673/tid=10680] ppid=10670 vsize=613160 CPUtime=12.26 cores=0,2,4,6
/proc/10673/task/10680/stat : 10680 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 29220 0 0 0 1222 4 0 0 25 0 5 0 521383823 627875840 93027 33554432000 4194304 4338100 140734972272528 18446744073709551615 246516530860 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=10673/tid=10681] ppid=10670 vsize=613160 CPUtime=12.34 cores=0,2,4,6
/proc/10673/task/10681/stat : 10681 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 20008 0 0 0 1228 6 0 0 25 0 5 0 521383823 627875840 93027 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=10673/tid=10682] ppid=10670 vsize=613160 CPUtime=12.33 cores=0,2,4,6
/proc/10673/task/10682/stat : 10682 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 12590 0 0 0 1231 2 0 0 25 0 5 0 521383823 627875840 93027 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254259 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 49.45
Current children cumulated vsize (KiB) 613160

[startup+25.501 s]
/proc/loadavg: 6.89 7.78 8.01 9/190 10687
/proc/meminfo: memFree=23688292/32951124 swapFree=67111352/67111528
[pid=10673] ppid=10670 vsize=912160 CPUtime=99.6 cores=0,2,4,6
/proc/10673/stat : 10673 (ttsth-5-4) S 10670 10673 10639 0 -1 4202496 174178 0 0 0 9918 42 0 0 25 0 5 0 521383800 934051840 174072 33554432000 4194304 4338100 140734972272528 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/10673/statm: 228040 174073 175 36 0 225945 0
[pid=10673/tid=10679] ppid=10670 vsize=912160 CPUtime=24.69 cores=0,2,4,6
/proc/10673/task/10679/stat : 10679 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 63225 0 0 0 2453 16 0 0 19 0 5 0 521383823 934051840 174073 33554432000 4194304 4338100 140734972272528 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=10673/tid=10680] ppid=10670 vsize=912160 CPUtime=24.76 cores=0,2,4,6
/proc/10673/task/10680/stat : 10680 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 55357 0 0 0 2468 8 0 0 19 0 5 0 521383823 934051840 174073 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=10673/tid=10681] ppid=10670 vsize=912160 CPUtime=24.97 cores=0,2,4,6
/proc/10673/task/10681/stat : 10681 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 30363 0 0 0 2488 9 0 0 25 0 5 0 521383823 934051840 174073 33554432000 4194304 4338100 140734972272528 18446744073709551615 246516229882 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=10673/tid=10682] ppid=10670 vsize=912160 CPUtime=24.92 cores=0,2,4,6
/proc/10673/task/10682/stat : 10682 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 24360 0 0 0 2485 7 0 0 25 0 5 0 521383823 934051840 174073 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 99.6
Current children cumulated vsize (KiB) 912160

[startup+51.1003 s]
/proc/loadavg: 7.00 7.74 7.99 9/190 10688
/proc/meminfo: memFree=23160872/32951124 swapFree=67111352/67111528
[pid=10673] ppid=10670 vsize=1140280 CPUtime=199.32 cores=0,2,4,6
/proc/10673/stat : 10673 (ttsth-5-4) S 10670 10673 10639 0 -1 4202496 234789 0 0 0 19863 69 0 0 25 0 5 0 521383800 1167646720 234683 33554432000 4194304 4338100 140734972272528 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/10673/statm: 285070 234683 175 36 0 282975 0
[pid=10673/tid=10679] ppid=10670 vsize=1140280 CPUtime=49.29 cores=0,2,4,6
/proc/10673/task/10679/stat : 10679 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 102826 0 0 0 4900 29 0 0 25 0 5 0 521383823 1167646720 234683 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=10673/tid=10680] ppid=10670 vsize=1140280 CPUtime=49.8 cores=0,2,4,6
/proc/10673/task/10680/stat : 10680 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 64022 0 0 0 4967 13 0 0 25 0 5 0 521383823 1167646720 234683 33554432000 4194304 4338100 140734972272528 18446744073709551615 4256374 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=10673/tid=10681] ppid=10670 vsize=1140280 CPUtime=50.06 cores=0,2,4,6
/proc/10673/task/10681/stat : 10681 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 35612 0 0 0 4992 14 0 0 25 0 5 0 521383823 1167646720 234683 33554432000 4194304 4338100 140734972272528 18446744073709551615 246528646950 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=10673/tid=10682] ppid=10670 vsize=1140280 CPUtime=49.91 cores=0,2,4,6
/proc/10673/task/10682/stat : 10682 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 31455 0 0 0 4980 11 0 0 18 0 5 0 521383823 1167646720 234683 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 199.32
Current children cumulated vsize (KiB) 1140280

[startup+102.307 s]
/proc/loadavg: 7.63 7.79 7.99 6/190 10689
/proc/meminfo: memFree=22421692/32951124 swapFree=67111352/67111528
[pid=10673] ppid=10670 vsize=1470600 CPUtime=398.62 cores=0,2,4,6
/proc/10673/stat : 10673 (ttsth-5-4) S 10670 10673 10639 0 -1 4202496 332737 0 0 0 39745 117 0 0 25 0 5 0 521383800 1505894400 332631 33554432000 4194304 4338100 140734972272528 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/10673/statm: 367650 332631 175 36 0 365555 0
[pid=10673/tid=10679] ppid=10670 vsize=1470600 CPUtime=98.52 cores=0,2,4,6
/proc/10673/task/10679/stat : 10679 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 175871 0 0 0 9798 54 0 0 20 0 5 0 521383823 1505894400 332631 33554432000 4194304 4338100 140734972272528 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=10673/tid=10680] ppid=10670 vsize=1470600 CPUtime=100 cores=0,2,4,6
/proc/10673/task/10680/stat : 10680 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 73288 0 0 0 9979 21 0 0 25 0 5 0 521383823 1505894400 332631 33554432000 4194304 4338100 140734972272528 18446744073709551615 246516230657 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=10673/tid=10681] ppid=10670 vsize=1470600 CPUtime=100.07 cores=0,2,4,6
/proc/10673/task/10681/stat : 10681 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 45462 0 0 0 9985 22 0 0 25 0 5 0 521383823 1505894400 332632 33554432000 4194304 4338100 140734972272528 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=10673/tid=10682] ppid=10670 vsize=1470600 CPUtime=99.76 cores=0,2,4,6
/proc/10673/task/10682/stat : 10682 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 37242 0 0 0 9958 18 0 0 18 0 5 0 521383823 1505894400 332632 33554432000 4194304 4338100 140734972272528 18446744073709551615 246516530608 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 398.62
Current children cumulated vsize (KiB) 1470600

[startup+162.301 s]
/proc/loadavg: 7.60 7.74 7.96 9/190 10691
/proc/meminfo: memFree=21519452/32951124 swapFree=67111352/67111528
[pid=10673] ppid=10670 vsize=1999736 CPUtime=630.15 cores=0,2,4,6
/proc/10673/stat : 10673 (ttsth-5-4) S 10670 10673 10639 0 -1 4202496 433287 0 0 0 62845 170 0 0 25 0 5 0 521383800 2047729664 433181 33554432000 4194304 4338100 140734972272528 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/10673/statm: 499934 433181 175 36 0 497839 0
[pid=10673/tid=10679] ppid=10670 vsize=1999736 CPUtime=155.53 cores=0,2,4,6
/proc/10673/task/10679/stat : 10679 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 247318 0 0 0 15469 84 0 0 25 0 5 0 521383823 2047729664 433181 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=10673/tid=10680] ppid=10670 vsize=1999736 CPUtime=157.96 cores=0,2,4,6
/proc/10673/task/10680/stat : 10680 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 82454 0 0 0 15767 29 0 0 24 0 5 0 521383823 2047729664 433181 33554432000 4194304 4338100 140734972272528 18446744073709551615 246516530816 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=10673/tid=10681] ppid=10670 vsize=1999736 CPUtime=158.42 cores=0,2,4,6
/proc/10673/task/10681/stat : 10681 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 56855 0 0 0 15812 30 0 0 25 0 5 0 521383823 2047729664 433181 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=10673/tid=10682] ppid=10670 vsize=1999736 CPUtime=157.97 cores=0,2,4,6
/proc/10673/task/10682/stat : 10682 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 45786 0 0 0 15772 25 0 0 23 0 5 0 521383823 2047729664 433181 33554432000 4194304 4338100 140734972272528 18446744073709551615 246516229229 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 630.15
Current children cumulated vsize (KiB) 1999736


################
# More data... #
################

/proc/10673/task/10682/stat : 10682 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 150090 0 0 0 73656 115 0 0 25 0 5 0 521383823 6079569920 1398856 33554432000 4194304 4338100 140734972272528 18446744073709551615 4283831 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 2938.97
Current children cumulated vsize (KiB) 5937080

[startup+822.3 s]
/proc/loadavg: 7.73 7.77 7.87 7/190 10708
/proc/meminfo: memFree=14532592/32951124 swapFree=67111352/67111528
[pid=10673] ppid=10670 vsize=6439580 CPUtime=3169.11 cores=0,2,4,6
/proc/10673/stat : 10673 (ttsth-5-4) S 10670 10673 10639 0 -1 4202496 1475001 0 0 0 316199 712 0 0 25 0 5 0 521383800 6594129920 1474895 33554432000 4194304 4338100 140734972272528 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/10673/statm: 1609895 1474896 175 36 0 1607800 0
[pid=10673/tid=10679] ppid=10670 vsize=6439580 CPUtime=778.92 cores=0,2,4,6
/proc/10673/task/10679/stat : 10679 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 959238 0 0 0 77546 346 0 0 19 0 5 0 521383823 6594129920 1474896 33554432000 4194304 4338100 140734972272528 18446744073709551615 4235651 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=10673/tid=10680] ppid=10670 vsize=6439580 CPUtime=796.87 cores=0,2,4,6
/proc/10673/task/10680/stat : 10680 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 171212 0 0 0 79582 105 0 0 25 0 5 0 521383823 6594129920 1474897 33554432000 4194304 4338100 140734972272528 18446744073709551615 4235847 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=10673/tid=10681] ppid=10670 vsize=6439580 CPUtime=797.37 cores=0,2,4,6
/proc/10673/task/10681/stat : 10681 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 185125 0 0 0 79600 137 0 0 25 0 5 0 521383823 6594129920 1474897 33554432000 4194304 4338100 140734972272528 18446744073709551615 4235847 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=10673/tid=10682] ppid=10670 vsize=6439580 CPUtime=795.69 cores=0,2,4,6
/proc/10673/task/10682/stat : 10682 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 158554 0 0 0 79446 123 0 0 25 0 5 0 521383823 6594129920 1474897 33554432000 4194304 4338100 140734972272528 18446744073709551615 4240533 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3169.11
Current children cumulated vsize (KiB) 6439580

[startup+882.3 s]
/proc/loadavg: 7.52 7.73 7.85 10/190 10710
/proc/meminfo: memFree=14017720/32951124 swapFree=67111352/67111528
[pid=10673] ppid=10670 vsize=6705784 CPUtime=3400.57 cores=0,2,4,6
/proc/10673/stat : 10673 (ttsth-5-4) S 10670 10673 10639 0 -1 4202496 1537675 0 0 0 339302 755 0 0 25 0 5 0 521383800 6866722816 1537569 33554432000 4194304 4338100 140734972272528 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/10673/statm: 1676446 1537569 175 36 0 1674351 0
[pid=10673/tid=10679] ppid=10670 vsize=6705784 CPUtime=835.87 cores=0,2,4,6
/proc/10673/task/10679/stat : 10679 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 994264 0 0 0 83222 365 0 0 25 0 5 0 521383823 6866722816 1537569 33554432000 4194304 4338100 140734972272528 18446744073709551615 246516229350 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=10673/tid=10680] ppid=10670 vsize=6705784 CPUtime=854.71 cores=0,2,4,6
/proc/10673/task/10680/stat : 10680 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 180371 0 0 0 85357 114 0 0 20 0 5 0 521383823 6866722816 1537569 33554432000 4194304 4338100 140734972272528 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=10673/tid=10681] ppid=10670 vsize=6705784 CPUtime=855.66 cores=0,2,4,6
/proc/10673/task/10681/stat : 10681 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 194410 0 0 0 85424 142 0 0 25 0 5 0 521383823 6866722816 1537569 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=10673/tid=10682] ppid=10670 vsize=6705784 CPUtime=854.09 cores=0,2,4,6
/proc/10673/task/10682/stat : 10682 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 167756 0 0 0 85276 133 0 0 18 0 5 0 521383823 6866722816 1537569 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3400.57
Current children cumulated vsize (KiB) 6705784

[startup+942.3 s]
/proc/loadavg: 7.56 7.72 7.84 9/190 10711
/proc/meminfo: memFree=13549840/32951124 swapFree=67111352/67111528
[pid=10673] ppid=10670 vsize=6944792 CPUtime=3632.19 cores=0,2,4,6
/proc/10673/stat : 10673 (ttsth-5-4) S 10670 10673 10639 0 -1 4202496 1600073 0 0 0 362424 795 0 0 25 0 5 0 521383800 7111467008 1599967 33554432000 4194304 4338100 140734972272528 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/10673/statm: 1736198 1599967 175 36 0 1734103 0
[pid=10673/tid=10679] ppid=10670 vsize=6944792 CPUtime=892.99 cores=0,2,4,6
/proc/10673/task/10679/stat : 10679 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 1039634 0 0 0 88914 385 0 0 20 0 5 0 521383823 7111467008 1599967 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=10673/tid=10680] ppid=10670 vsize=6944792 CPUtime=913.03 cores=0,2,4,6
/proc/10673/task/10680/stat : 10680 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 188899 0 0 0 91180 123 0 0 20 0 5 0 521383823 7111467008 1599967 33554432000 4194304 4338100 140734972272528 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=10673/tid=10681] ppid=10670 vsize=6944792 CPUtime=913.73 cores=0,2,4,6
/proc/10673/task/10681/stat : 10681 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 198670 0 0 0 91227 146 0 0 25 0 5 0 521383823 7111467008 1599967 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=10673/tid=10682] ppid=10670 vsize=6944792 CPUtime=912.17 cores=0,2,4,6
/proc/10673/task/10682/stat : 10682 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 171996 0 0 0 91078 139 0 0 19 0 5 0 521383823 7111467008 1599967 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3632.19
Current children cumulated vsize (KiB) 6944792

[startup+1002.3 s]
/proc/loadavg: 7.85 7.77 7.85 6/190 10713
/proc/meminfo: memFree=13000120/32951124 swapFree=67111352/67111528
[pid=10673] ppid=10670 vsize=7108896 CPUtime=3864.28 cores=0,2,4,6
/proc/10673/stat : 10673 (ttsth-5-4) S 10670 10673 10639 0 -1 4202496 1661316 0 0 0 385589 839 0 0 25 0 5 0 521383800 7279509504 1661210 33554432000 4194304 4338100 140734972272528 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/10673/statm: 1777224 1661210 175 36 0 1775129 0
[pid=10673/tid=10679] ppid=10670 vsize=7108896 CPUtime=950.07 cores=0,2,4,6
/proc/10673/task/10679/stat : 10679 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 1083028 0 0 0 94601 406 0 0 18 0 5 0 521383823 7279509504 1661210 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=10673/tid=10680] ppid=10670 vsize=7108896 CPUtime=971.39 cores=0,2,4,6
/proc/10673/task/10680/stat : 10680 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 196333 0 0 0 97005 134 0 0 25 0 5 0 521383823 7279509504 1661210 33554432000 4194304 4338100 140734972272528 18446744073709551615 4258042 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=10673/tid=10681] ppid=10670 vsize=7108896 CPUtime=972.16 cores=0,2,4,6
/proc/10673/task/10681/stat : 10681 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 202944 0 0 0 97066 150 0 0 25 0 5 0 521383823 7279509504 1661210 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=10673/tid=10682] ppid=10670 vsize=7108896 CPUtime=970.4 cores=0,2,4,6
/proc/10673/task/10682/stat : 10682 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 178137 0 0 0 96893 147 0 0 25 0 5 0 521383823 7279509504 1661210 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3864.28
Current children cumulated vsize (KiB) 7108896

[startup+1062.3 s]
/proc/loadavg: 7.88 7.77 7.84 9/190 10715
/proc/meminfo: memFree=12603116/32951124 swapFree=67111352/67111528
[pid=10673] ppid=10670 vsize=7393932 CPUtime=4094.1 cores=0,2,4,6
/proc/10673/stat : 10673 (ttsth-5-4) S 10670 10673 10639 0 -1 4202496 1717325 0 0 0 408533 877 0 0 25 0 5 0 521383800 7571386368 1717219 33554432000 4194304 4338100 140734972272528 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/10673/statm: 1848483 1717219 175 36 0 1846388 0
[pid=10673/tid=10679] ppid=10670 vsize=7393932 CPUtime=1006.47 cores=0,2,4,6
/proc/10673/task/10679/stat : 10679 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 1123092 0 0 0 100221 426 0 0 25 0 5 0 521383823 7571386368 1717219 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=10673/tid=10680] ppid=10670 vsize=7393932 CPUtime=1029.23 cores=0,2,4,6
/proc/10673/task/10680/stat : 10680 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 202846 0 0 0 102780 143 0 0 25 0 5 0 521383823 7571386368 1717219 33554432000 4194304 4338100 140734972272528 18446744073709551615 4256993 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=10673/tid=10681] ppid=10670 vsize=7393932 CPUtime=1030.04 cores=0,2,4,6
/proc/10673/task/10681/stat : 10681 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 207618 0 0 0 102851 153 0 0 25 0 5 0 521383823 7571386368 1717219 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=10673/tid=10682] ppid=10670 vsize=7393932 CPUtime=1028.1 cores=0,2,4,6
/proc/10673/task/10682/stat : 10682 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 182895 0 0 0 102657 153 0 0 25 0 5 0 521383823 7571386368 1717219 33554432000 4194304 4338100 140734972272528 18446744073709551615 246516530731 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4094.1
Current children cumulated vsize (KiB) 7393932

[startup+1122.3 s]
/proc/loadavg: 8.02 7.84 7.85 9/190 10716
/proc/meminfo: memFree=12256640/32951124 swapFree=67111352/67111528
[pid=10673] ppid=10670 vsize=7601304 CPUtime=4326.24 cores=0,2,4,6
/proc/10673/stat : 10673 (ttsth-5-4) S 10670 10673 10639 0 -1 4202496 1776534 0 0 0 431700 924 0 0 25 0 5 0 521383800 7783735296 1776428 33554432000 4194304 4338100 140734972272528 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/10673/statm: 1900326 1776428 175 36 0 1898231 0
[pid=10673/tid=10679] ppid=10670 vsize=7601304 CPUtime=1063.72 cores=0,2,4,6
/proc/10673/task/10679/stat : 10679 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 1160586 0 0 0 105924 448 0 0 21 0 5 0 521383823 7783735296 1776428 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=10673/tid=10680] ppid=10670 vsize=7601304 CPUtime=1087.29 cores=0,2,4,6
/proc/10673/task/10680/stat : 10680 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 209312 0 0 0 108573 156 0 0 19 0 5 0 521383823 7783735296 1776428 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=10673/tid=10681] ppid=10670 vsize=7601304 CPUtime=1088.62 cores=0,2,4,6
/proc/10673/task/10681/stat : 10681 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 215105 0 0 0 108704 158 0 0 25 0 5 0 521383823 7783735296 1776428 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=10673/tid=10682] ppid=10670 vsize=7601304 CPUtime=1086.35 cores=0,2,4,6
/proc/10673/task/10682/stat : 10682 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 190657 0 0 0 108474 161 0 0 19 0 5 0 521383823 7783735296 1776428 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4326.24
Current children cumulated vsize (KiB) 7601304

[startup+1182.3 s]
/proc/loadavg: 7.87 7.83 7.85 9/190 10718
/proc/meminfo: memFree=11915464/32951124 swapFree=67111352/67111528
[pid=10673] ppid=10670 vsize=7993364 CPUtime=4559.41 cores=0,2,4,6
/proc/10673/stat : 10673 (ttsth-5-4) S 10670 10673 10639 0 -1 4202496 1819782 0 0 0 454972 969 0 0 25 0 5 0 521383800 8185204736 1819676 33554432000 4194304 4338100 140734972272528 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/10673/statm: 1998341 1819676 175 36 0 1996246 0
[pid=10673/tid=10679] ppid=10670 vsize=7993364 CPUtime=1121.28 cores=0,2,4,6
/proc/10673/task/10679/stat : 10679 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 1178083 0 0 0 111666 462 0 0 25 0 5 0 521383823 8185204736 1819676 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=10673/tid=10680] ppid=10670 vsize=7993364 CPUtime=1145.78 cores=0,2,4,6
/proc/10673/task/10680/stat : 10680 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 217611 0 0 0 114411 167 0 0 25 0 5 0 521383823 8185204736 1819676 33554432000 4194304 4338100 140734972272528 18446744073709551615 246516229136 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=10673/tid=10681] ppid=10670 vsize=7993364 CPUtime=1147.13 cores=0,2,4,6
/proc/10673/task/10681/stat : 10681 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 222253 0 0 0 114551 162 0 0 25 0 5 0 521383823 8185204736 1819676 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=10673/tid=10682] ppid=10670 vsize=7993364 CPUtime=1144.96 cores=0,2,4,6
/proc/10673/task/10682/stat : 10682 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 200961 0 0 0 114319 177 0 0 25 0 5 0 521383823 8185204736 1819676 33554432000 4194304 4338100 140734972272528 18446744073709551615 246516229574 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4559.41
Current children cumulated vsize (KiB) 7993364

[startup+1242.3 s]
/proc/loadavg: 7.85 7.82 7.84 10/190 10719
/proc/meminfo: memFree=11326552/32951124 swapFree=67111352/67111528
[pid=10673] ppid=10670 vsize=8304780 CPUtime=4789.19 cores=0,2,4,6
/proc/10673/stat : 10673 (ttsth-5-4) S 10670 10673 10639 0 -1 4202496 1882820 0 0 0 477908 1011 0 0 25 0 5 0 521383800 8504094720 1882714 33554432000 4194304 4338100 140734972272528 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/10673/statm: 2076195 1882714 175 36 0 2074100 0
[pid=10673/tid=10679] ppid=10670 vsize=8304780 CPUtime=1177.49 cores=0,2,4,6
/proc/10673/task/10679/stat : 10679 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 1223856 0 0 0 117268 481 0 0 25 0 5 0 521383823 8504094720 1882714 33554432000 4194304 4338100 140734972272528 18446744073709551615 246528648225 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=10673/tid=10680] ppid=10670 vsize=8304780 CPUtime=1203.61 cores=0,2,4,6
/proc/10673/task/10680/stat : 10680 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 222832 0 0 0 120183 178 0 0 18 0 5 0 521383823 8504094720 1882714 33554432000 4194304 4338100 140734972272528 18446744073709551615 246516229172 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=10673/tid=10681] ppid=10670 vsize=8304780 CPUtime=1205.1 cores=0,2,4,6
/proc/10673/task/10681/stat : 10681 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 226511 0 0 0 120343 167 0 0 25 0 5 0 521383823 8504094720 1882714 33554432000 4194304 4338100 140734972272528 18446744073709551615 4284214 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=10673/tid=10682] ppid=10670 vsize=8304780 CPUtime=1202.74 cores=0,2,4,6
/proc/10673/task/10682/stat : 10682 (ttsth-5-4) R 10670 10673 10639 0 -1 4202560 208747 0 0 0 120090 184 0 0 25 0 5 0 521383823 8504094720 1882715 33554432000 4194304 4338100 140734972272528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4789.19
Current children cumulated vsize (KiB) 8304780



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1245.21 s]
/proc/loadavg: 7.87 7.82 7.84 6/187 10719
/proc/meminfo: memFree=13058708/32951124 swapFree=67111352/67111528
[pid=10673] ppid=10670 vsize=0 CPUtime=4800.11 cores=0,2,4,6
/proc/10673/stat : 10673 (ttsth-5-4) Z 10670 10673 10639 0 -1 4203548 1888674 0 0 0 478989 1022 0 0 18 0 2 0 521383800 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/10673/statm: 0 0 0 0 0 0 0
[pid=10673/tid=10682] ppid=10670 vsize=0 CPUtime=1205.53 cores=0,2,4,6
/proc/10673/task/10682/stat : 10682 (ttsth-5-4) R 10670 10673 10639 0 -1 4203588 208920 0 0 0 120360 193 0 0 25 0 2 0 521383823 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.11
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1245.4 s]
/proc/loadavg: 7.87 7.82 7.84 6/187 10719
/proc/meminfo: memFree=13058708/32951124 swapFree=67111352/67111528
[pid=10673] ppid=10670 vsize=0 CPUtime=4800.3 cores=0,2,4,6
/proc/10673/stat : 10673 (ttsth-5-4) Z 10670 10673 10639 0 -1 4203548 1888674 0 0 0 478989 1041 0 0 18 0 2 0 521383800 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/10673/statm: 0 0 0 0 0 0 0
[pid=10673/tid=10682] ppid=10670 vsize=0 CPUtime=1205.72 cores=0,2,4,6
/proc/10673/task/10682/stat : 10682 (ttsth-5-4) R 10670 10673 10639 0 -1 4203588 208920 0 0 0 120360 212 0 0 25 0 2 0 521383823 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.3
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1245.43
CPU time (s): 4800.34
CPU user time (s): 4789.9
CPU system time (s): 10.4434
CPU usage (%): 385.436
Max. virtual memory (cumulated for all children) (KiB): 8324828

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4789.9
system time used= 10.4434
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1888674
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 240085
involuntary context switches= 66174

runsolver used 1.90571 second user time and 4.89126 second system time

The end

Launcher Data 

Begin job on node134 at 2011-04-29 19:24:16
IDJOB=3263762
IDBENCH=84018
IDSOLVER=1654
FILE ID=node134/3263762-1304097856
RUNJOBID= node134-1304097856-10657
PBS_JOBID= 13170616
Free space on /tmp= 71584 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT03/handmade/stanion/hwb/hwb-n28-02-S818962541.shuffled-as.sat03-1628.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3263762-1304097856/watcher-3263762-1304097856 -o /tmp/evaluation-result-3263762-1304097856/solver-3263762-1304097856 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263762-1304097856.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= d705043763df769aab1e385bbbc2b11e
RANDOM SEED=995650289

node134.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 7186.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      25017668 kB
Buffers:       1491456 kB
Cached:        5854736 kB
SwapCached:         64 kB
Active:        1464380 kB
Inactive:      5945760 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      25017668 kB
SwapTotal:    67111528 kB
SwapFree:     67111352 kB
Dirty:             108 kB
Writeback:          48 kB
AnonPages:       63520 kB
Mapped:          14404 kB
Slab:           459360 kB
PageTables:       4916 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   188060 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71584 MiB
End job on node134 at 2011-04-29 19:45:03