Trace number 3265786

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.14 1202.31

General information on the benchmark

NameSAT05/crafted/sorge05/QG7a/
gensys-ukn004.shuffled-as.sat05-3844.cnf
MD5SUM7d96ba103689bec761c9e7df7b158323
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark3199.02
Satisfiable
(Un)Satisfiability was proved
Number of variables2731
Number of clauses17983
Sum of the clauses size126997
Maximum clause length13
Minimum clause length1
Number of clauses of size 142
Number of clauses of size 2442
Number of clauses of size 32370
Number of clauses of size 4477
Number of clauses of size 5642
Number of clauses of size over 514010

Solver Data 

0.01/0.04	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265786-1304103010.cnf 
0.01/0.04	c Using SATTIMEOUT of 4800
0.01/0.04	c Using SATRAM of 15500
0.01/0.04	c Size of subset node = 32
0.01/0.04	c Variables = 2731
0.01/0.04	c Clauses = 17983
0.01/0.04	c Literals = 126997
0.01/0.04	c Reduced size:
0.01/0.04	c Variables = 2731
0.01/0.04	c Clauses = 17435
0.01/0.04	c Literals = 123360
0.01/0.04	c a2
0.09/0.17	c a1
0.29/0.35	c Independent sub-problems = 1
0.29/0.35	c Sub-problem with 2675 variables and 17435 clauses
0.29/0.35	c using permutation method 4
0.29/0.35	c count = 1
0.29/0.35	c Ordering finished 0.330 seconds
0.29/0.35	c mapped 0.340 seconds
0.29/0.35	c permuted 0.340 seconds
0.29/0.35	c about to sort
0.29/0.35	c sorted
0.39/0.44	c clauses now 17435
0.39/0.44	c init finished 0.370 seconds
0.39/0.44	c tree built 0.440 seconds
0.39/0.44	c 
0.39/0.44	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.39/0.44	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.39/0.44	c ................................ | ..........                    |                                  0.4         0.0             
0.39/0.44	c ................................ | ..........                  1 |                                  0.4         2.3             
0.39/0.44	c ................................ | ..........                  2 |                                  0.4         4.5             
0.39/0.44	c ................................ | ..........                  3 |                                  0.4         6.8             
0.39/0.44	c ................................ | ..........                  4 |                                  0.4         9.1             
0.39/0.44	c ................................ | ..........                  5 |                                  0.4        11.4             
0.39/0.45	c ................................ | ..........               1000 | 3            584                 0.5      2173.9         1272
95.73/24.37	c ................................ | ..........            2031000 | 121694       1911084            96.0     21145.2      1540304
191.99/48.40	c ................................ | ..........            6496000 | 243021       3934552           192.0     33829.8      3085764
287.84/72.44	c ................................ | ..........           10402000 | 340253       5469779           288.0     36115.5      4857879
383.67/96.48	c ................................ | ..........           13331000 | 415401       6719773           384.0     34716.1      5890836
479.91/120.53	c ................................ | *.........           15723000 | 483675       7777765           480.0     32753.5      7366911
575.76/144.55	c ................................ | *.........           18177000 | 551994       8752434           576.0     31557.3      8690421
671.64/168.60	c ................................ | *.........           20327000 | 618400       9706912           672.0     30247.6      9702001
767.86/192.63	c ................................ | *.........           22512000 | 679949       10600101          768.0     29312.1     10690469
863.69/216.68	c ................................ | *.........           24859000 | 737808       11565310          864.0     28771.3     11577095
959.93/240.72	c ................................ | **........           27060000 | 801169       12445745          960.0     28187.2     12153457
1055.77/264.76	c ................................ | **........           28428000 | 847132       13139233         1056.0     26919.4     12701884
1152.01/288.80	c ................................ | **........           29686000 | 891706       13805779         1152.0     25768.6     13094488
1247.86/312.85	c ................................ | **........           31684000 | 936513       14714542         1248.1     25386.2     13524052
1343.72/336.89	c ................................ | **........           33399000 | 977928       15436346         1344.1     24848.8     14360873
1439.96/360.94	c ................................ | ***.......           35379000 | 1018371      16170613         1440.1     24566.5     15932824
1535.82/384.99	c ................................ | ***.......           37076000 | 1054110      16870953         1536.2     24135.2     18051440
1632.07/409.00	c ................................ | ***.......           38029000 | 1089102      17321095         1632.1     23300.8     19552273
1727.93/433.02	c ................................ | ***.......           39088000 | 1120647      17716818         1728.0     22620.2     20951838
1823.78/457.06	c ................................ | ***.......           40165000 | 1151991      18084620         1824.0     22020.0     22078404
1920.04/481.10	c ................................ | ****......           41243000 | 1187164      18447646         1920.0     21480.2     23147545
2015.92/505.12	c ................................ | ****......           42265000 | 1225963      18797102         2016.0     20964.8     24405433
2112.15/529.20	c ................................ | ****......           43197000 | 1270229      19116962         2112.2     20451.6     25324012
2207.99/553.21	c ................................ | ****......           43891000 | 1309931      19456939         2208.0     19877.9     25934329
2303.85/577.27	c ................................ | ****......           44602000 | 1341914      19778413         2304.2     19357.2     26425066
2399.71/601.27	c ................................ | *****.....           45381000 | 1373244      20108144         2400.0     18908.7     26484650
2495.97/625.31	c ................................ | *****.....           46259000 | 1406474      20492498         2496.0     18533.1     27135985
2591.83/649.35	c ................................ | *****.....           47363000 | 1450319      20906625         2592.0     18272.6     27631586
2687.65/673.40	c ................................ | *****.....           48499000 | 1496225      21296360         2688.1     18042.4     28432736
2783.90/697.44	c ................................ | *****.....           49242000 | 1540470      21579832         2784.1     17687.1     29271892
2880.15/721.51	c ................................ | ******....           49683000 | 1589895      21790368         2880.2     17249.7     30168449
2976.01/745.58	c ................................ | ******....           50118000 | 1637776      22013046         2976.4     16838.7     30983293
3071.86/769.56	c ................................ | ******....           50732000 | 1676796      22287416         3072.1     16513.8     31580076
3167.72/793.58	c ................................ | ******....           51252000 | 1712041      22555017         3168.1     16177.7     32219041
3263.99/817.62	c ................................ | ******....           51693000 | 1741304      22777638         3264.0     15837.1     32800764
3359.82/841.65	c ................................ | *******...           52272000 | 1788837      23011606         3360.0     15557.1     33509686
3456.06/865.73	c ................................ | *******...           53127000 | 1821559      23314864         3456.2     15371.5     33810269
3551.92/889.72	c ................................ | *******...           53888000 | 1848725      23569925         3552.0     15171.0     34362411
3647.77/913.79	c ................................ | *******...           54543000 | 1874549      23800021         3648.2     14950.8     34882114
3744.02/937.80	c ................................ | *******...           55492000 | 1903102      24103814         3744.0     14821.4     35611789
3839.89/961.83	c ................................ | ********..           56366000 | 1928454      24373587         3840.0     14678.6     36483694
3936.17/985.93	c ................................ | ********..           57111000 | 1955971      24601188         3936.3     14509.0     37040851
4032.00/1009.90	c ................................ | ********..           57967000 | 1988515      24879403         4032.0     14376.7     37807055
4127.85/1033.94	c ................................ | ********..           58988000 | 2022664      25210307         4128.0     14289.6     38310534
4224.10/1058.01	c ................................ | ********..           60051000 | 2055428      25528271         4224.2     14216.1     38766101
4319.95/1082.01	c ................................ | *********.           60932000 | 2084738      25827015         4320.0     14104.6     39526985
4415.82/1106.07	c ................................ | *********.           61804000 | 2112517      26096631         4416.1     13995.1     40246419
4512.03/1130.10	c ................................ | *********.           62639000 | 2142361      26398431         4512.0     13882.7     40675877
4607.91/1154.14	c ................................ | *********.           63471000 | 2171872      26698674         4608.1     13773.9     41450901
4703.76/1178.17	c ................................ | *********.           64184000 | 2199086      26972235         4704.1     13644.4     42094159

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3265786-1304103010/watcher-3265786-1304103010 -o /tmp/evaluation-result-3265786-1304103010/solver-3265786-1304103010 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265786-1304103010.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.07 1.18 2.02 2/177 21754
/proc/meminfo: memFree=18414624/32951124 swapFree=67110396/67111528
[pid=21754] ppid=21752 vsize=9104 CPUtime=0 cores=1,3,5,7
/proc/21754/stat : 21754 (ttsth-5-4) R 21752 21754 20607 0 -1 4202496 334 0 0 0 0 0 0 0 25 0 1 0 600531693 9322496 276 33554432000 4194304 4338100 140733244184880 18446744073709551615 224284586693 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21754/statm: 2276 276 147 36 0 181 0

[startup+0.018323 s]
/proc/loadavg: 1.07 1.18 2.02 2/177 21754
/proc/meminfo: memFree=18414624/32951124 swapFree=67110396/67111528
[pid=21754] ppid=21752 vsize=10044 CPUtime=0.01 cores=1,3,5,7
/proc/21754/stat : 21754 (ttsth-5-4) R 21752 21754 20607 0 -1 4202496 581 0 0 0 1 0 0 0 25 0 1 0 600531693 10285056 523 33554432000 4194304 4338100 140733244184880 18446744073709551615 224284925696 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21754/statm: 2511 524 147 36 0 416 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 10044

[startup+0.100311 s]
/proc/loadavg: 1.07 1.18 2.02 2/177 21754
/proc/meminfo: memFree=18414624/32951124 swapFree=67110396/67111528
[pid=21754] ppid=21752 vsize=26112 CPUtime=0.09 cores=1,3,5,7
/proc/21754/stat : 21754 (ttsth-5-4) R 21752 21754 20607 0 -1 4202496 4154 0 0 0 9 0 0 0 25 0 1 0 600531693 26738688 4092 33554432000 4194304 4338100 140733244184880 18446744073709551615 4206976 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21754/statm: 6528 4092 178 36 0 4433 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 26112

[startup+0.300274 s]
/proc/loadavg: 1.07 1.18 2.02 2/177 21754
/proc/meminfo: memFree=18414624/32951124 swapFree=67110396/67111528
[pid=21754] ppid=21752 vsize=26108 CPUtime=0.29 cores=1,3,5,7
/proc/21754/stat : 21754 (ttsth-5-4) R 21752 21754 20607 0 -1 4202496 4247 0 0 0 29 0 0 0 25 0 1 0 600531693 26734592 4098 33554432000 4194304 4338100 140733244184880 18446744073709551615 4295648 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21754/statm: 6527 4098 165 36 0 4432 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 26108

[startup+0.845965 s]
/proc/loadavg: 1.07 1.18 2.02 2/177 21754
/proc/meminfo: memFree=18414624/32951124 swapFree=67110396/67111528
[pid=21754] ppid=21752 vsize=244688 CPUtime=2.03 cores=1,3,5,7
/proc/21754/stat : 21754 (ttsth-5-4) S 21752 21754 20607 0 -1 4202496 13841 0 0 0 202 1 0 0 25 0 5 0 600531693 250560512 13547 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296728549 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21754/statm: 61172 13547 175 36 0 59077 0
Current children cumulated CPU time (s) 2.03
Current children cumulated vsize (KiB) 244688

[startup+1.50104 s]
/proc/loadavg: 1.07 1.18 2.02 9/186 21763
/proc/meminfo: memFree=18298372/32951124 swapFree=67110396/67111528
[pid=21754] ppid=21752 vsize=245612 CPUtime=4.65 cores=1,3,5,7
/proc/21754/stat : 21754 (ttsth-5-4) S 21752 21754 20607 0 -1 4202496 15121 0 0 0 464 1 0 0 25 0 5 0 600531693 251506688 14827 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296728549 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21754/statm: 61403 14827 175 36 0 59308 0
[pid=21754/tid=21756] ppid=21752 vsize=245612 CPUtime=1.05 cores=1,3,5,7
/proc/21754/task/21756/stat : 21756 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 392 0 0 0 105 0 0 0 19 0 5 0 600531737 251506688 14827 33554432000 4194304 4338100 140733244184880 18446744073709551615 224284321249 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21754/tid=21757] ppid=21752 vsize=245612 CPUtime=1.04 cores=1,3,5,7
/proc/21754/task/21757/stat : 21757 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 607 0 0 0 104 0 0 0 19 0 5 0 600531737 251506688 14827 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21754/tid=21758] ppid=21752 vsize=245612 CPUtime=1.05 cores=1,3,5,7
/proc/21754/task/21758/stat : 21758 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 896 0 0 0 105 0 0 0 19 0 5 0 600531737 251506688 14827 33554432000 4194304 4338100 140733244184880 18446744073709551615 224284622492 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21754/tid=21759] ppid=21752 vsize=245612 CPUtime=1.05 cores=1,3,5,7
/proc/21754/task/21759/stat : 21759 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 272 0 0 0 105 0 0 0 25 0 5 0 600531737 251506688 14827 33554432000 4194304 4338100 140733244184880 18446744073709551615 224284321214 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4.65
Current children cumulated vsize (KiB) 245612

[startup+3.10077 s]
/proc/loadavg: 1.07 1.18 2.02 9/186 21763
/proc/meminfo: memFree=18282300/32951124 swapFree=67110396/67111528
[pid=21754] ppid=21752 vsize=253848 CPUtime=11.05 cores=1,3,5,7
/proc/21754/stat : 21754 (ttsth-5-4) S 21752 21754 20607 0 -1 4202496 17619 0 0 0 1102 3 0 0 25 0 5 0 600531693 259940352 17325 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296728549 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21754/statm: 63462 17325 175 36 0 61367 0
[pid=21754/tid=21756] ppid=21752 vsize=253848 CPUtime=2.64 cores=1,3,5,7
/proc/21754/task/21756/stat : 21756 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 719 0 0 0 264 0 0 0 23 0 5 0 600531737 259940352 17325 33554432000 4194304 4338100 140733244184880 18446744073709551615 4239490 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21754/tid=21757] ppid=21752 vsize=253848 CPUtime=2.63 cores=1,3,5,7
/proc/21754/task/21757/stat : 21757 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 1435 0 0 0 263 0 0 0 23 0 5 0 600531737 259940352 17325 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21754/tid=21758] ppid=21752 vsize=253848 CPUtime=2.65 cores=1,3,5,7
/proc/21754/task/21758/stat : 21758 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 1962 0 0 0 265 0 0 0 23 0 5 0 600531737 259940352 17325 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21754/tid=21759] ppid=21752 vsize=253848 CPUtime=2.65 cores=1,3,5,7
/proc/21754/task/21759/stat : 21759 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 549 0 0 0 265 0 0 0 25 0 5 0 600531737 259940352 17325 33554432000 4194304 4338100 140733244184880 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 11.05
Current children cumulated vsize (KiB) 253848

[startup+6.30024 s]
/proc/loadavg: 1.62 1.29 2.06 9/186 21763
/proc/meminfo: memFree=18245840/32951124 swapFree=67110396/67111528
[pid=21754] ppid=21752 vsize=262876 CPUtime=23.83 cores=1,3,5,7
/proc/21754/stat : 21754 (ttsth-5-4) S 21752 21754 20607 0 -1 4202496 21184 0 0 0 2378 5 0 0 25 0 5 0 600531693 269185024 20890 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296728549 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21754/statm: 65719 20890 175 36 0 63624 0
[pid=21754/tid=21756] ppid=21752 vsize=262876 CPUtime=5.85 cores=1,3,5,7
/proc/21754/task/21756/stat : 21756 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 1262 0 0 0 584 1 0 0 25 0 5 0 600531737 269185024 20890 33554432000 4194304 4338100 140733244184880 18446744073709551615 4239500 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21754/tid=21757] ppid=21752 vsize=262876 CPUtime=5.82 cores=1,3,5,7
/proc/21754/task/21757/stat : 21757 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 2580 0 0 0 581 1 0 0 25 0 5 0 600531737 269185024 20890 33554432000 4194304 4338100 140733244184880 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21754/tid=21758] ppid=21752 vsize=262876 CPUtime=5.84 cores=1,3,5,7
/proc/21754/task/21758/stat : 21758 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 3372 0 0 0 584 0 0 0 25 0 5 0 600531737 269185024 20890 33554432000 4194304 4338100 140733244184880 18446744073709551615 224284586693 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21754/tid=21759] ppid=21752 vsize=262876 CPUtime=5.84 cores=1,3,5,7
/proc/21754/task/21759/stat : 21759 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 1016 0 0 0 584 0 0 0 25 0 5 0 600531737 269185024 20890 33554432000 4194304 4338100 140733244184880 18446744073709551615 224284321506 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 23.83
Current children cumulated vsize (KiB) 262876

[startup+12.7012 s]
/proc/loadavg: 2.13 1.40 2.09 9/186 21763
/proc/meminfo: memFree=18181020/32951124 swapFree=67110396/67111528
[pid=21754] ppid=21752 vsize=278184 CPUtime=49.39 cores=1,3,5,7
/proc/21754/stat : 21754 (ttsth-5-4) S 21752 21754 20607 0 -1 4202496 27774 0 0 0 4932 7 0 0 25 0 5 0 600531693 284860416 27480 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296728549 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21754/statm: 69546 27480 175 36 0 67451 0
[pid=21754/tid=21756] ppid=21752 vsize=278184 CPUtime=12.25 cores=1,3,5,7
/proc/21754/task/21756/stat : 21756 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 2475 0 0 0 1223 2 0 0 25 0 5 0 600531737 284860416 27480 33554432000 4194304 4338100 140733244184880 18446744073709551615 224284320820 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21754/tid=21757] ppid=21752 vsize=278184 CPUtime=12.18 cores=1,3,5,7
/proc/21754/task/21757/stat : 21757 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 5052 0 0 0 1217 1 0 0 25 0 5 0 600531737 284860416 27480 33554432000 4194304 4338100 140733244184880 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21754/tid=21758] ppid=21752 vsize=278184 CPUtime=12.25 cores=1,3,5,7
/proc/21754/task/21758/stat : 21758 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 5277 0 0 0 1224 1 0 0 25 0 5 0 600531737 284860416 27480 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21754/tid=21759] ppid=21752 vsize=278184 CPUtime=12.24 cores=1,3,5,7
/proc/21754/task/21759/stat : 21759 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 2016 0 0 0 1224 0 0 0 25 0 5 0 600531737 284860416 27480 33554432000 4194304 4338100 140733244184880 18446744073709551615 224284320815 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 49.39
Current children cumulated vsize (KiB) 278184

[startup+25.501 s]
/proc/loadavg: 3.58 1.76 2.19 9/186 21763
/proc/meminfo: memFree=18078320/32951124 swapFree=67110396/67111528
[pid=21754] ppid=21752 vsize=376820 CPUtime=100.54 cores=1,3,5,7
/proc/21754/stat : 21754 (ttsth-5-4) S 21752 21754 20607 0 -1 4202496 39112 0 0 0 10040 14 0 0 25 0 5 0 600531693 385863680 38818 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296728549 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21754/statm: 94205 38818 175 36 0 92110 0
[pid=21754/tid=21756] ppid=21752 vsize=376820 CPUtime=25.05 cores=1,3,5,7
/proc/21754/task/21756/stat : 21756 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 3955 0 0 0 2501 4 0 0 25 0 5 0 600531737 385863680 38818 33554432000 4194304 4338100 140733244184880 18446744073709551615 4229648 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21754/tid=21757] ppid=21752 vsize=376820 CPUtime=24.91 cores=1,3,5,7
/proc/21754/task/21757/stat : 21757 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 9834 0 0 0 2489 2 0 0 25 0 5 0 600531737 385863680 38818 33554432000 4194304 4338100 140733244184880 18446744073709551615 224284622574 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21754/tid=21758] ppid=21752 vsize=376820 CPUtime=25.06 cores=1,3,5,7
/proc/21754/task/21758/stat : 21758 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 8897 0 0 0 2503 3 0 0 25 0 5 0 600531737 385863680 38818 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21754/tid=21759] ppid=21752 vsize=376820 CPUtime=25.05 cores=1,3,5,7
/proc/21754/task/21759/stat : 21759 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 3472 0 0 0 2503 2 0 0 25 0 5 0 600531737 385863680 38818 33554432000 4194304 4338100 140733244184880 18446744073709551615 4256770 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 100.54
Current children cumulated vsize (KiB) 376820

[startup+51.1008 s]
/proc/loadavg: 5.22 2.29 2.36 9/186 21765
/proc/meminfo: memFree=17886944/32951124 swapFree=67110396/67111528
[pid=21754] ppid=21752 vsize=443048 CPUtime=202.77 cores=1,3,5,7
/proc/21754/stat : 21754 (ttsth-5-4) S 21752 21754 20607 0 -1 4202496 66114 0 0 0 20141 136 0 0 25 0 5 0 600531693 453681152 65820 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296728549 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21754/statm: 110762 65820 175 36 0 108667 0
[pid=21754/tid=21756] ppid=21752 vsize=443048 CPUtime=50.65 cores=1,3,5,7
/proc/21754/task/21756/stat : 21756 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 9263 0 0 0 5018 47 0 0 25 0 5 0 600531737 453681152 65820 33554432000 4194304 4338100 140733244184880 18446744073709551615 224284321667 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21754/tid=21757] ppid=21752 vsize=443048 CPUtime=50.37 cores=1,3,5,7
/proc/21754/task/21757/stat : 21757 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 17879 0 0 0 5032 5 0 0 25 0 5 0 600531737 453681152 65821 33554432000 4194304 4338100 140733244184880 18446744073709551615 224284320993 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21754/tid=21758] ppid=21752 vsize=443048 CPUtime=50.65 cores=1,3,5,7
/proc/21754/task/21758/stat : 21758 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 17607 0 0 0 5025 40 0 0 25 0 5 0 600531737 453681152 65821 33554432000 4194304 4338100 140733244184880 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21754/tid=21759] ppid=21752 vsize=443048 CPUtime=50.64 cores=1,3,5,7
/proc/21754/task/21759/stat : 21759 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 8412 0 0 0 5022 42 0 0 25 0 5 0 600531737 453681152 65821 33554432000 4194304 4338100 140733244184880 18446744073709551615 224284320820 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 202.77
Current children cumulated vsize (KiB) 443048

[startup+102.3 s]
/proc/loadavg: 7.06 3.23 2.67 9/186 21767
/proc/meminfo: memFree=17560564/32951124 swapFree=67110396/67111528
[pid=21754] ppid=21752 vsize=557932 CPUtime=407.23 cores=1,3,5,7
/proc/21754/stat : 21754 (ttsth-5-4) S 21752 21754 20607 0 -1 4202496 107803 0 0 0 40132 591 0 0 25 0 5 0 600531693 571322368 107509 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296728549 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21754/statm: 139483 107509 176 36 0 137388 0
[pid=21754/tid=21756] ppid=21752 vsize=557932 CPUtime=101.83 cores=1,3,5,7
/proc/21754/task/21756/stat : 21756 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 18997 0 0 0 9979 204 0 0 25 0 5 0 600531737 571322368 107509 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296751797 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21754/tid=21757] ppid=21752 vsize=557932 CPUtime=101.29 cores=1,3,5,7
/proc/21754/task/21757/stat : 21757 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 29373 0 0 0 10120 9 0 0 25 0 5 0 600531737 571322368 107509 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21754/tid=21758] ppid=21752 vsize=557932 CPUtime=101.82 cores=1,3,5,7
/proc/21754/task/21758/stat : 21758 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 27718 0 0 0 9996 186 0 0 25 0 5 0 600531737 571322368 107509 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21754/tid=21759] ppid=21752 vsize=557932 CPUtime=101.82 cores=1,3,5,7
/proc/21754/task/21759/stat : 21759 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 18761 0 0 0 9993 189 0 0 25 0 5 0 600531737 571322368 107509 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296751797 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 407.23
Current children cumulated vsize (KiB) 557932

[startup+162.3 s]
/proc/loadavg: 7.61 4.08 3.00 9/186 21767
/proc/meminfo: memFree=17210496/32951124 swapFree=67110396/67111528
[pid=21754] ppid=21752 vsize=677616 CPUtime=646.85 cores=1,3,5,7
/proc/21754/stat : 21754 (ttsth-5-4) S 21752 21754 20607 0 -1 4202496 149355 0 0 0 63621 1064 0 0 25 0 5 0 600531693 693878784 149061 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296728549 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21754/statm: 169404 149061 176 36 0 167309 0
[pid=21754/tid=21756] ppid=21752 vsize=677616 CPUtime=161.81 cores=1,3,5,7
/proc/21754/task/21756/stat : 21756 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 27907 0 0 0 15809 372 0 0 25 0 5 0 600531737 693878784 149061 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296751797 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21754/tid=21757] ppid=21752 vsize=677616 CPUtime=160.96 cores=1,3,5,7
/proc/21754/task/21757/stat : 21757 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 40294 0 0 0 16079 17 0 0 25 0 5 0 600531737 693878784 149061 33554432000 4194304 4338100 140733244184880 18446744073709551615 224284321236 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21754/tid=21758] ppid=21752 vsize=677616 CPUtime=161.8 cores=1,3,5,7
/proc/21754/task/21758/stat : 21758 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 37905 0 0 0 15846 334 0 0 25 0 5 0 600531737 693878784 149061 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296751797 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21754/tid=21759] ppid=21752 vsize=677616 CPUtime=161.81 cores=1,3,5,7
/proc/21754/task/21759/stat : 21759 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 30295 0 0 0 15842 339 0 0 25 0 5 0 600531737 693878784 149061 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296751797 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 646.85
Current children cumulated vsize (KiB) 677616


################
# More data... #
################

[pid=21754/tid=21757] ppid=21752 vsize=1736536 CPUtime=758.33 cores=1,3,5,7
/proc/21754/task/21757/stat : 21757 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 162836 0 0 0 75762 71 0 0 25 0 5 0 600531737 1778212864 405960 33554432000 4194304 4338100 140733244184880 18446744073709551615 4240513 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21754/tid=21758] ppid=21752 vsize=1736536 CPUtime=761.66 cores=1,3,5,7
/proc/21754/task/21758/stat : 21758 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 82017 0 0 0 74733 1433 0 0 25 0 5 0 600531737 1778212864 405960 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296751797 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21754/tid=21759] ppid=21752 vsize=1736536 CPUtime=760.97 cores=1,3,5,7
/proc/21754/task/21759/stat : 21759 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 75180 0 0 0 74646 1451 0 0 25 0 5 0 600531737 1778212864 405960 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3043.11
Current children cumulated vsize (KiB) 1736536

[startup+822.301 s]
/proc/loadavg: 8.00 7.58 5.53 9/186 21794
/proc/meminfo: memFree=15051724/32951124 swapFree=67110396/67111528
[pid=21754] ppid=21752 vsize=1863404 CPUtime=3282.74 cores=1,3,5,7
/proc/21754/stat : 21754 (ttsth-5-4) S 21752 21754 20607 0 -1 4202496 423336 0 0 0 323541 4733 0 0 25 0 5 0 600531693 1908125696 423042 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296728549 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21754/statm: 465851 423043 176 36 0 463756 0
[pid=21754/tid=21756] ppid=21752 vsize=1863404 CPUtime=821.66 cores=1,3,5,7
/proc/21754/task/21756/stat : 21756 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 77095 0 0 0 80542 1624 0 0 25 0 5 0 600531737 1908125696 423043 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21754/tid=21757] ppid=21752 vsize=1863404 CPUtime=818.34 cores=1,3,5,7
/proc/21754/task/21757/stat : 21757 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 174170 0 0 0 81758 76 0 0 25 0 5 0 600531737 1908125696 423043 33554432000 4194304 4338100 140733244184880 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21754/tid=21758] ppid=21752 vsize=1863404 CPUtime=821.64 cores=1,3,5,7
/proc/21754/task/21758/stat : 21758 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 83041 0 0 0 80657 1507 0 0 25 0 5 0 600531737 1908125696 423043 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296751797 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21754/tid=21759] ppid=21752 vsize=1863404 CPUtime=820.63 cores=1,3,5,7
/proc/21754/task/21759/stat : 21759 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 76077 0 0 0 80540 1523 0 0 25 0 5 0 600531737 1908125696 423044 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3282.74
Current children cumulated vsize (KiB) 1863404

[startup+882.301 s]
/proc/loadavg: 8.08 7.67 5.69 9/186 21796
/proc/meminfo: memFree=14941108/32951124 swapFree=67110396/67111528
[pid=21754] ppid=21752 vsize=1974712 CPUtime=3522.36 cores=1,3,5,7
/proc/21754/stat : 21754 (ttsth-5-4) S 21752 21754 20607 0 -1 4202496 437974 0 0 0 347223 5013 0 0 25 0 5 0 600531693 2022105088 437680 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296728549 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21754/statm: 493678 437680 176 36 0 491583 0
[pid=21754/tid=21756] ppid=21752 vsize=1974712 CPUtime=881.62 cores=1,3,5,7
/proc/21754/task/21756/stat : 21756 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 78745 0 0 0 86445 1717 0 0 25 0 5 0 600531737 2022105088 437680 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21754/tid=21757] ppid=21752 vsize=1974712 CPUtime=878.34 cores=1,3,5,7
/proc/21754/task/21757/stat : 21757 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 184900 0 0 0 87754 80 0 0 25 0 5 0 600531737 2022105088 437680 33554432000 4194304 4338100 140733244184880 18446744073709551615 224284321202 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21754/tid=21758] ppid=21752 vsize=1974712 CPUtime=881.63 cores=1,3,5,7
/proc/21754/task/21758/stat : 21758 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 84115 0 0 0 86564 1599 0 0 25 0 5 0 600531737 2022105088 437680 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21754/tid=21759] ppid=21752 vsize=1974712 CPUtime=880.31 cores=1,3,5,7
/proc/21754/task/21759/stat : 21759 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 77260 0 0 0 86416 1615 0 0 25 0 5 0 600531737 2022105088 437680 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3522.36
Current children cumulated vsize (KiB) 1974712

[startup+942.301 s]
/proc/loadavg: 8.06 7.74 5.84 9/186 21798
/proc/meminfo: memFree=14824900/32951124 swapFree=67110396/67111528
[pid=21754] ppid=21752 vsize=2028984 CPUtime=3762 cores=1,3,5,7
/proc/21754/stat : 21754 (ttsth-5-4) S 21752 21754 20607 0 -1 4202496 455506 0 0 0 370944 5256 0 0 25 0 5 0 600531693 2077679616 455212 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296728549 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21754/statm: 507246 455212 176 36 0 505151 0
[pid=21754/tid=21756] ppid=21752 vsize=2028984 CPUtime=941.27 cores=1,3,5,7
/proc/21754/task/21756/stat : 21756 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 79773 0 0 0 92330 1797 0 0 25 0 5 0 600531737 2077679616 455212 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21754/tid=21757] ppid=21752 vsize=2028984 CPUtime=938.34 cores=1,3,5,7
/proc/21754/task/21757/stat : 21757 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 197006 0 0 0 93749 85 0 0 25 0 5 0 600531737 2077679616 455212 33554432000 4194304 4338100 140733244184880 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21754/tid=21758] ppid=21752 vsize=2028984 CPUtime=941.62 cores=1,3,5,7
/proc/21754/task/21758/stat : 21758 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 86382 0 0 0 92483 1679 0 0 25 0 5 0 600531737 2077679616 455212 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296751797 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21754/tid=21759] ppid=21752 vsize=2028984 CPUtime=940.3 cores=1,3,5,7
/proc/21754/task/21759/stat : 21759 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 79391 0 0 0 92337 1693 0 0 25 0 5 0 600531737 2077679616 455212 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254400 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3762
Current children cumulated vsize (KiB) 2028984

[startup+1002.3 s]
/proc/loadavg: 8.09 7.80 5.97 9/186 21800
/proc/meminfo: memFree=14732868/32951124 swapFree=67110396/67111528
[pid=21754] ppid=21752 vsize=2086384 CPUtime=4001.66 cores=1,3,5,7
/proc/21754/stat : 21754 (ttsth-5-4) S 21752 21754 20607 0 -1 4202496 472736 0 0 0 394666 5500 0 0 25 0 5 0 600531693 2136457216 472442 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296728549 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21754/statm: 521596 472442 176 36 0 519501 0
[pid=21754/tid=21756] ppid=21752 vsize=2086384 CPUtime=1000.94 cores=1,3,5,7
/proc/21754/task/21756/stat : 21756 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 80809 0 0 0 98219 1875 0 0 25 0 5 0 600531737 2136457216 472442 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21754/tid=21757] ppid=21752 vsize=2086384 CPUtime=998.34 cores=1,3,5,7
/proc/21754/task/21757/stat : 21757 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 206766 0 0 0 99745 89 0 0 25 0 5 0 600531737 2136457216 472442 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21754/tid=21758] ppid=21752 vsize=2086384 CPUtime=1001.61 cores=1,3,5,7
/proc/21754/task/21758/stat : 21758 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 89837 0 0 0 98401 1760 0 0 25 0 5 0 600531737 2136457216 472442 33554432000 4194304 4338100 140733244184880 18446744073709551615 224284321214 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21754/tid=21759] ppid=21752 vsize=2086384 CPUtime=1000.29 cores=1,3,5,7
/proc/21754/task/21759/stat : 21759 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 82370 0 0 0 98256 1773 0 0 25 0 5 0 600531737 2136457216 472442 33554432000 4194304 4338100 140733244184880 18446744073709551615 224284320900 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4001.66
Current children cumulated vsize (KiB) 2086384

[startup+1062.3 s]
/proc/loadavg: 8.17 7.87 6.11 9/186 21800
/proc/meminfo: memFree=14599648/32951124 swapFree=67110396/67111528
[pid=21754] ppid=21752 vsize=2136512 CPUtime=4241.27 cores=1,3,5,7
/proc/21754/stat : 21754 (ttsth-5-4) S 21752 21754 20607 0 -1 4202496 489324 0 0 0 418296 5831 0 0 25 0 5 0 600531693 2187788288 489030 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296728549 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21754/statm: 534128 489030 176 36 0 532033 0
[pid=21754/tid=21756] ppid=21752 vsize=2136512 CPUtime=1060.82 cores=1,3,5,7
/proc/21754/task/21756/stat : 21756 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 83918 0 0 0 104096 1986 0 0 25 0 5 0 600531737 2187788288 489030 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21754/tid=21757] ppid=21752 vsize=2136512 CPUtime=1058.34 cores=1,3,5,7
/proc/21754/task/21757/stat : 21757 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 211258 0 0 0 105742 92 0 0 25 0 5 0 600531737 2187788288 489030 33554432000 4194304 4338100 140733244184880 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21754/tid=21758] ppid=21752 vsize=2136512 CPUtime=1061.59 cores=1,3,5,7
/proc/21754/task/21758/stat : 21758 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 94518 0 0 0 104290 1869 0 0 25 0 5 0 600531737 2187788288 489030 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21754/tid=21759] ppid=21752 vsize=2136512 CPUtime=1060.05 cores=1,3,5,7
/proc/21754/task/21759/stat : 21759 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 86676 0 0 0 104124 1881 0 0 25 0 5 0 600531737 2187788288 489030 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4241.27
Current children cumulated vsize (KiB) 2136512

[startup+1122.31 s]
/proc/loadavg: 8.13 7.91 6.23 9/186 21802
/proc/meminfo: memFree=14489640/32951124 swapFree=67110396/67111528
[pid=21754] ppid=21752 vsize=2194572 CPUtime=4480.91 cores=1,3,5,7
/proc/21754/stat : 21754 (ttsth-5-4) S 21752 21754 20607 0 -1 4202496 507070 0 0 0 441995 6096 0 0 25 0 5 0 600531693 2247241728 506776 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296728549 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21754/statm: 548643 506776 176 36 0 546548 0
[pid=21754/tid=21756] ppid=21752 vsize=2194572 CPUtime=1120.81 cores=1,3,5,7
/proc/21754/task/21756/stat : 21756 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 87847 0 0 0 110001 2080 0 0 25 0 5 0 600531737 2247241728 506776 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21754/tid=21757] ppid=21752 vsize=2194572 CPUtime=1118.35 cores=1,3,5,7
/proc/21754/task/21757/stat : 21757 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 218566 0 0 0 111741 94 0 0 25 0 5 0 600531737 2247241728 506776 33554432000 4194304 4338100 140733244184880 18446744073709551615 224284321032 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21754/tid=21758] ppid=21752 vsize=2194572 CPUtime=1121.58 cores=1,3,5,7
/proc/21754/task/21758/stat : 21758 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 97490 0 0 0 110205 1953 0 0 25 0 5 0 600531737 2247241728 506776 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21754/tid=21759] ppid=21752 vsize=2194572 CPUtime=1119.7 cores=1,3,5,7
/proc/21754/task/21759/stat : 21759 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 90213 0 0 0 110004 1966 0 0 25 0 5 0 600531737 2247241728 506776 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4480.91
Current children cumulated vsize (KiB) 2194572

[startup+1182.3 s]
/proc/loadavg: 8.17 7.95 6.35 9/186 21804
/proc/meminfo: memFree=14380268/32951124 swapFree=67110396/67111528
[pid=21754] ppid=21752 vsize=2249768 CPUtime=4720.53 cores=1,3,5,7
/proc/21754/stat : 21754 (ttsth-5-4) S 21752 21754 20607 0 -1 4202496 524850 0 0 0 465703 6350 0 0 25 0 5 0 600531693 2303762432 524556 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296728549 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21754/statm: 562442 524556 176 36 0 560347 0
[pid=21754/tid=21756] ppid=21752 vsize=2249768 CPUtime=1180.8 cores=1,3,5,7
/proc/21754/task/21756/stat : 21756 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 89084 0 0 0 115912 2168 0 0 25 0 5 0 600531737 2303762432 524556 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21754/tid=21757] ppid=21752 vsize=2249768 CPUtime=1178.34 cores=1,3,5,7
/proc/21754/task/21757/stat : 21757 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 228777 0 0 0 117735 99 0 0 25 0 5 0 600531737 2303762432 524556 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21754/tid=21758] ppid=21752 vsize=2249768 CPUtime=1181.56 cores=1,3,5,7
/proc/21754/task/21758/stat : 21758 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 100437 0 0 0 116125 2031 0 0 25 0 5 0 600531737 2303762432 524556 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21754/tid=21759] ppid=21752 vsize=2249768 CPUtime=1179.35 cores=1,3,5,7
/proc/21754/task/21759/stat : 21759 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 93598 0 0 0 115886 2049 0 0 25 0 5 0 600531737 2303762432 524556 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4720.53
Current children cumulated vsize (KiB) 2249768



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.2 s]
/proc/loadavg: 8.12 7.95 6.38 9/186 21804
/proc/meminfo: memFree=14318376/32951124 swapFree=67110396/67111528
[pid=21754] ppid=21752 vsize=2276376 CPUtime=4800.02 cores=1,3,5,7
/proc/21754/stat : 21754 (ttsth-5-4) S 21752 21754 20607 0 -1 4202496 532773 0 0 0 473563 6439 0 0 25 0 5 0 600531693 2331009024 532479 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296728549 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21754/statm: 569094 532479 176 36 0 566999 0
[pid=21754/tid=21756] ppid=21752 vsize=2276376 CPUtime=1200.61 cores=1,3,5,7
/proc/21754/task/21756/stat : 21756 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 90567 0 0 0 117862 2199 0 0 25 0 5 0 600531737 2331009024 532479 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21754/tid=21757] ppid=21752 vsize=2276376 CPUtime=1198.25 cores=1,3,5,7
/proc/21754/task/21757/stat : 21757 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 232921 0 0 0 119723 102 0 0 25 0 5 0 600531737 2331009024 532479 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21754/tid=21758] ppid=21752 vsize=2276376 CPUtime=1201.45 cores=1,3,5,7
/proc/21754/task/21758/stat : 21758 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 101177 0 0 0 118087 2058 0 0 25 0 5 0 600531737 2331009024 532479 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21754/tid=21759] ppid=21752 vsize=2276376 CPUtime=1199.24 cores=1,3,5,7
/proc/21754/task/21759/stat : 21759 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 95154 0 0 0 117847 2077 0 0 25 0 5 0 600531737 2331009024 532479 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4800.02
Current children cumulated vsize (KiB) 2276376

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.2 s]
/proc/loadavg: 8.12 7.95 6.38 9/186 21804
/proc/meminfo: memFree=14318376/32951124 swapFree=67110396/67111528
[pid=21754] ppid=21752 vsize=2276376 CPUtime=4800.02 cores=1,3,5,7
/proc/21754/stat : 21754 (ttsth-5-4) S 21752 21754 20607 0 -1 4202496 532773 0 0 0 473563 6439 0 0 25 0 5 0 600531693 2331009024 532479 33554432000 4194304 4338100 140733244184880 18446744073709551615 224296728549 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21754/statm: 569094 532479 176 36 0 566999 0
[pid=21754/tid=21756] ppid=21752 vsize=2276376 CPUtime=1200.61 cores=1,3,5,7
/proc/21754/task/21756/stat : 21756 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 90567 0 0 0 117862 2199 0 0 25 0 5 0 600531737 2331009024 532479 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21754/tid=21757] ppid=21752 vsize=2276376 CPUtime=1198.25 cores=1,3,5,7
/proc/21754/task/21757/stat : 21757 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 232921 0 0 0 119723 102 0 0 25 0 5 0 600531737 2331009024 532479 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21754/tid=21758] ppid=21752 vsize=2276376 CPUtime=1201.45 cores=1,3,5,7
/proc/21754/task/21758/stat : 21758 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 101177 0 0 0 118087 2058 0 0 25 0 5 0 600531737 2331009024 532479 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21754/tid=21759] ppid=21752 vsize=2276376 CPUtime=1199.24 cores=1,3,5,7
/proc/21754/task/21759/stat : 21759 (ttsth-5-4) R 21752 21754 20607 0 -1 4202560 95154 0 0 0 117847 2077 0 0 25 0 5 0 600531737 2331009024 532479 33554432000 4194304 4338100 140733244184880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4800.02
Current children cumulated vsize (KiB) 2276376

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.31
CPU time (s): 4800.14
CPU user time (s): 4735.66
CPU system time (s): 64.4852
CPU usage (%): 399.245
Max. virtual memory (cumulated for all children) (KiB): 2276376

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4735.66
system time used= 64.4852
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 532775
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 108564
involuntary context switches= 17558

runsolver used 1.9897 second user time and 4.5743 second system time

The end

Launcher Data 

Begin job on node118 at 2011-04-29 20:50:10
IDJOB=3265786
IDBENCH=85805
IDSOLVER=1654
FILE ID=node118/3265786-1304103010
RUNJOBID= node118-1304103010-21735
PBS_JOBID= 13170577
Free space on /tmp= 71616 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT05/crafted/sorge05/QG7a/gensys-ukn004.shuffled-as.sat05-3844.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3265786-1304103010/watcher-3265786-1304103010 -o /tmp/evaluation-result-3265786-1304103010/solver-3265786-1304103010 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265786-1304103010.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 7d96ba103689bec761c9e7df7b158323
RANDOM SEED=110173812

node118.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5374.59
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      18415028 kB
Buffers:       1902912 kB
Cached:       12029612 kB
SwapCached:        600 kB
Active:        9491460 kB
Inactive:      4508072 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      18415028 kB
SwapTotal:    67111528 kB
SwapFree:     67110396 kB
Dirty:            1664 kB
Writeback:           0 kB
AnonPages:       65852 kB
Mapped:          14964 kB
Slab:           472464 kB
PageTables:       4736 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   208104 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71616 MiB
End job on node118 at 2011-04-29 21:10:15