Trace number 3260418

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.14 1205.92

General information on the benchmark

Namecrafted/mosoi/289/
289-sat-6x8.cnf
MD5SUM0836620198442c187bac3443fd060059
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.001998
Satisfiable
(Un)Satisfiability was proved
Number of variables192
Number of clauses2016
Sum of the clauses size7488
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2288
Number of clauses of size 30
Number of clauses of size 41728
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3260418-1304065877.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 192
0.00/0.00	c Clauses = 2016
0.00/0.00	c Literals = 7488
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 192
0.00/0.00	c Clauses = 2016
0.00/0.00	c Literals = 7488
0.00/0.00	c a2
0.00/0.00	c a1
0.03/0.04	c Independent sub-problems = 1
0.03/0.04	c Sub-problem with 192 variables and 2016 clauses
0.03/0.04	c using permutation method 4
0.03/0.04	c count = 2
0.03/0.04	c Ordering finished 0.040 seconds
0.03/0.04	c mapped 0.040 seconds
0.03/0.04	c permuted 0.040 seconds
0.03/0.04	c about to sort
0.03/0.04	c sorted
0.03/0.04	c clauses now 2016
0.03/0.04	c init finished 0.040 seconds
0.03/0.04	c tree built 0.040 seconds
0.03/0.04	c 
0.03/0.04	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.03/0.04	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.03/0.04	c ................................ | ..........                    |                                  0.1         0.0             
0.03/0.04	c ................................ | ..........                  1 |                                  0.1        10.0             
0.03/0.04	c ................................ | ..........                  2 |                                  0.1        20.0             
0.03/0.04	c ................................ | ..........                  3 |                                  0.1        30.0             
0.03/0.04	c ................................ | ..........                  4 |                                  0.1        40.0             
0.03/0.04	c ................................ | ..........                  5 |                                  0.1        50.0             
0.03/0.05	c ................................ | ..........               1000 | 14           1682                0.1     10000.0        11012
95.93/24.15	c ................................ | ..........             913000 | 211643       1587805            96.1      9496.6      8455066
192.05/48.22	c ................................ | ..........            1525000 | 365597       2548202           192.1      7936.9     12160554
288.08/72.31	c ................................ | ..........            2091000 | 524167       3309237           288.1      7257.1     16000461
383.75/96.40	c ................................ | ..........            2514000 | 647024       3881806           384.1      6544.5     19246095
480.07/120.51	c ................................ | *.........            2954000 | 775274       4435668           480.1      6152.8     21321316
576.04/144.62	c ................................ | *.........            3415000 | 914097       4946847           576.1      5927.4     22879369
671.74/168.66	c ................................ | *.........            3889000 | 1054551      5458228           672.0      5787.0     25840293
768.05/192.82	c ................................ | *.........            4418000 | 1213056      5963252           768.1      5751.6     29534518
863.77/216.87	c ................................ | *.........            4827000 | 1326669      6431864           864.1      5586.4     32903471
959.77/240.98	c ................................ | **........            5193000 | 1429057      6877446           960.1      5408.8     35221129
1055.93/265.12	c ................................ | **........            5514000 | 1519348      7230863          1056.0      5221.5     35666294
1152.18/289.30	c ................................ | **........            5853000 | 1615180      7627564          1152.2      5079.9     35855088
1248.03/313.43	c ................................ | **........            6179000 | 1706107      8015035          1248.2      4950.5     36328248
1343.71/337.49	c ................................ | **........            6552000 | 1812645      8472202          1344.1      4874.7     36909644
1439.94/361.68	c ................................ | ***.......            6885000 | 1908723      8820804          1440.3      4780.4     37641726
1535.90/385.75	c ................................ | ***.......            7185000 | 1995377      9149111          1536.1      4677.4     38586792
1631.97/409.86	c ................................ | ***.......            7501000 | 2088931      9543703          1632.2      4595.6     40290884
1728.04/433.98	c ................................ | ***.......            7779000 | 2169362      9835527          1728.4      4500.7     40688067
1824.18/458.10	c ................................ | ***.......            8097000 | 2263593      10172332         1824.2      4438.7     40982154
1920.13/482.21	c ................................ | ****......            8410000 | 2358012      10548119         1920.2      4379.8     41020672
2016.16/506.32	c ................................ | ****......            8753000 | 2463970      10862354         2016.3      4341.2     41937081
2112.00/530.41	c ................................ | ****......            9099000 | 2568282      11180481         2112.0      4308.2     42278334
2207.71/554.47	c ................................ | ****......            9508000 | 2693154      11541696         2208.0      4306.1     43246940
2303.82/578.67	c ................................ | ****......            9975000 | 2836427      11908189         2304.1      4329.3     44684345
2399.94/602.73	c ................................ | *****.....           10307000 | 2932979      12246269         2400.1      4294.4     46759580
2495.93/626.85	c ................................ | *****.....           10627000 | 3026302      12578336         2496.2      4257.3     48091972
2591.87/650.95	c ................................ | *****.....           10915000 | 3108393      12872154         2592.1      4210.9     48103663
2687.80/675.08	c ................................ | *****.....           11215000 | 3194132      13180164         2688.2      4172.0     48103663
2783.85/699.13	c ................................ | *****.....           11497000 | 3272741      13487633         2784.0      4129.7     48128785
2879.87/723.26	c ................................ | ******....           11798000 | 3358082      13819404         2880.2      4096.3     49321366
2975.92/747.38	c ................................ | ******....           12090000 | 3443237      14113468         2976.2      4062.2     49898011
3072.10/771.50	c ................................ | ******....           12352000 | 3520489      14369131         3072.1      4020.7     50521890
3167.98/795.62	c ................................ | ******....           12625000 | 3599447      14664643         3168.1      3985.1     50988660
3264.14/819.80	c ................................ | ******....           12898000 | 3679322      14922598         3264.2      3951.4     51628066
3359.79/843.90	c ................................ | *******...           13198000 | 3766965      15213857         3360.2      3927.8     52031100
3455.96/868.04	c ................................ | *******...           13472000 | 3847831      15492513         3456.1      3898.0     52354306
3551.94/892.18	c ................................ | *******...           13791000 | 3946402      15769124         3552.3      3882.3     52538601
3647.92/916.24	c ................................ | *******...           14115000 | 4044244      16031517         3648.1      3869.1     53188610
3744.05/940.40	c ................................ | *******...           14482000 | 4155785      16314827         3744.1      3868.0     53640351
3839.83/964.58	c ................................ | ********..           14912000 | 4287825      16615279         3840.2      3883.2     54648790
3935.93/988.67	c ................................ | ********..           15206000 | 4370513      16896206         3936.2      3863.1     55739224
4031.75/1012.77	c ................................ | ********..           15511000 | 4457681      17168938         4032.1      3846.9     57129209
4127.92/1036.92	c ................................ | ********..           15792000 | 4535093      17462112         4128.0      3825.6     57496559
4224.09/1061.12	c ................................ | ********..           16116000 | 4628848      17766826         4224.2      3815.2     57894441
4320.19/1085.21	c ................................ | *********.           16380000 | 4701917      18014356         4320.2      3791.4     58596548
4415.86/1109.33	c ................................ | *********.           16658000 | 4778548      18300266         4416.0      3772.2     58898806
4511.72/1133.49	c ................................ | *********.           16962000 | 4865188      18591550         4512.1      3759.2     58901973
4608.00/1157.62	c ................................ | *********.           17231000 | 4938737      18886772         4608.1      3739.3     59109142
4703.86/1181.75	c ................................ | *********.           17509000 | 5015286      19193955         4704.1      3722.1     59479872
4799.96/1205.82	c CPU limit (4800) reached
4799.96/1205.82	c Time = 4800.03 seconds
4799.96/1205.82	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3260418-1304065877/watcher-3260418-1304065877 -o /tmp/evaluation-result-3260418-1304065877/solver-3260418-1304065877 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3260418-1304065877.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.84 8.04 7.68 2/172 11725
/proc/meminfo: memFree=15583292/32951124 swapFree=67111528/67111528
[pid=11725] ppid=11723 vsize=9376 CPUtime=0 cores=1,3,5,7
/proc/11725/stat : 11725 (ttsth-5-4) R 11723 11725 9766 0 -1 4202496 405 0 0 0 0 0 0 0 25 0 1 0 518188131 9601024 347 33554432000 4194304 4338100 140734933938368 18446744073709551615 244446340268 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/11725/statm: 2344 347 149 36 0 249 0

[startup+0.038944 s]
/proc/loadavg: 7.84 8.04 7.68 2/172 11725
/proc/meminfo: memFree=15583292/32951124 swapFree=67111528/67111528
[pid=11725] ppid=11723 vsize=9376 CPUtime=0.03 cores=1,3,5,7
/proc/11725/stat : 11725 (ttsth-5-4) R 11723 11725 9766 0 -1 4202496 416 0 0 0 3 0 0 0 25 0 1 0 518188131 9601024 358 33554432000 4194304 4338100 140734933938368 18446744073709551615 4296052 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/11725/statm: 2344 358 152 36 0 249 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 9376

[startup+0.111573 s]
/proc/loadavg: 7.84 8.04 7.68 2/172 11725
/proc/meminfo: memFree=15583292/32951124 swapFree=67111528/67111528
[pid=11725] ppid=11723 vsize=206340 CPUtime=1.19 cores=1,3,5,7
/proc/11725/stat : 11725 (ttsth-5-4) S 11723 11725 9766 0 -1 4202496 4771 0 0 0 119 0 0 0 25 0 5 0 518188131 211292160 4713 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458747877 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/11725/statm: 51585 4713 173 36 0 49490 0
Current children cumulated CPU time (s) 1.19
Current children cumulated vsize (KiB) 206340

[startup+0.438027 s]
/proc/loadavg: 7.84 8.04 7.68 2/172 11725
/proc/meminfo: memFree=15583292/32951124 swapFree=67111528/67111528
[pid=11725] ppid=11723 vsize=209596 CPUtime=1.59 cores=1,3,5,7
/proc/11725/stat : 11725 (ttsth-5-4) S 11723 11725 9766 0 -1 4202496 5793 0 0 0 158 1 0 0 25 0 5 0 518188131 214626304 5735 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458747877 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/11725/statm: 52399 5735 173 36 0 50304 0
Current children cumulated CPU time (s) 1.59
Current children cumulated vsize (KiB) 209596

[startup+0.700785 s]
/proc/loadavg: 7.84 8.04 7.68 2/172 11725
/proc/meminfo: memFree=15583292/32951124 swapFree=67111528/67111528
[pid=11725] ppid=11723 vsize=212980 CPUtime=2.63 cores=1,3,5,7
/proc/11725/stat : 11725 (ttsth-5-4) S 11723 11725 9766 0 -1 4202496 7821 0 0 0 262 1 0 0 25 0 5 0 518188131 218091520 7763 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458747877 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/11725/statm: 53245 7764 173 36 0 51150 0
Current children cumulated CPU time (s) 2.63
Current children cumulated vsize (KiB) 212980

[startup+1.50064 s]
/proc/loadavg: 7.84 8.04 7.68 6/182 11740
/proc/meminfo: memFree=15457120/32951124 swapFree=67111528/67111528
[pid=11725] ppid=11723 vsize=297796 CPUtime=5.82 cores=1,3,5,7
/proc/11725/stat : 11725 (ttsth-5-4) S 11723 11725 9766 0 -1 4202496 12266 0 0 0 578 4 0 0 25 0 5 0 518188131 304943104 12208 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458747877 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/11725/statm: 74449 12208 173 36 0 72354 0
[pid=11725/tid=11727] ppid=11723 vsize=297796 CPUtime=1.44 cores=1,3,5,7
/proc/11725/task/11727/stat : 11727 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 2541 0 0 0 143 1 0 0 20 0 5 0 518188136 304943104 12209 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11725/tid=11728] ppid=11723 vsize=297796 CPUtime=1.43 cores=1,3,5,7
/proc/11725/task/11728/stat : 11728 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 2754 0 0 0 142 1 0 0 20 0 5 0 518188136 304943104 12210 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11725/tid=11729] ppid=11723 vsize=297796 CPUtime=1.43 cores=1,3,5,7
/proc/11725/task/11729/stat : 11729 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 2834 0 0 0 143 0 0 0 25 0 5 0 518188136 304943104 12210 33554432000 4194304 4338100 140734933938368 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11725/tid=11730] ppid=11723 vsize=297796 CPUtime=1.43 cores=1,3,5,7
/proc/11725/task/11730/stat : 11730 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 3137 0 0 0 143 0 0 0 25 0 5 0 518188136 304943104 12210 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254304 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 5.82
Current children cumulated vsize (KiB) 297796

[startup+3.10034 s]
/proc/loadavg: 7.84 8.04 7.68 6/182 11740
/proc/meminfo: memFree=15348260/32951124 swapFree=67111528/67111528
[pid=11725] ppid=11723 vsize=327272 CPUtime=12.2 cores=1,3,5,7
/proc/11725/stat : 11725 (ttsth-5-4) S 11723 11725 9766 0 -1 4202496 19997 0 0 0 1208 12 0 0 25 0 5 0 518188131 335126528 19931 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458747877 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/11725/statm: 81818 19931 174 36 0 79723 0
[pid=11725/tid=11727] ppid=11723 vsize=327272 CPUtime=3.03 cores=1,3,5,7
/proc/11725/task/11727/stat : 11727 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 4291 0 0 0 300 3 0 0 18 0 5 0 518188136 335126528 19931 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458771125 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11725/tid=11728] ppid=11723 vsize=327272 CPUtime=3.02 cores=1,3,5,7
/proc/11725/task/11728/stat : 11728 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 4469 0 0 0 299 3 0 0 25 0 5 0 518188136 335126528 19931 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11725/tid=11729] ppid=11723 vsize=327272 CPUtime=3.03 cores=1,3,5,7
/proc/11725/task/11729/stat : 11729 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 4866 0 0 0 301 2 0 0 18 0 5 0 518188136 335126528 19931 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458771125 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11725/tid=11730] ppid=11723 vsize=327272 CPUtime=3.04 cores=1,3,5,7
/proc/11725/task/11730/stat : 11730 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 5370 0 0 0 302 2 0 0 25 0 5 0 518188136 335126528 19931 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458771125 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 12.2
Current children cumulated vsize (KiB) 327272

[startup+6.30074 s]
/proc/loadavg: 7.61 7.99 7.67 6/182 11740
/proc/meminfo: memFree=15310856/32951124 swapFree=67111528/67111528
[pid=11725] ppid=11723 vsize=369592 CPUtime=24.96 cores=1,3,5,7
/proc/11725/stat : 11725 (ttsth-5-4) S 11723 11725 9766 0 -1 4202496 31122 0 0 0 2462 34 0 0 25 0 5 0 518188131 378462208 31048 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458747877 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/11725/statm: 92398 31048 175 36 0 90303 0
[pid=11725/tid=11727] ppid=11723 vsize=369592 CPUtime=6.22 cores=1,3,5,7
/proc/11725/task/11727/stat : 11727 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 6643 0 0 0 613 9 0 0 25 0 5 0 518188136 378462208 31048 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458771125 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11725/tid=11728] ppid=11723 vsize=369592 CPUtime=6.2 cores=1,3,5,7
/proc/11725/task/11728/stat : 11728 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 6840 0 0 0 611 9 0 0 25 0 5 0 518188136 378462208 31048 33554432000 4194304 4338100 140734933938368 18446744073709551615 244446641836 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11725/tid=11729] ppid=11723 vsize=369592 CPUtime=6.22 cores=1,3,5,7
/proc/11725/task/11729/stat : 11729 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 8003 0 0 0 615 7 0 0 25 0 5 0 518188136 378462208 31048 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458771125 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11725/tid=11730] ppid=11723 vsize=369592 CPUtime=6.23 cores=1,3,5,7
/proc/11725/task/11730/stat : 11730 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 8635 0 0 0 616 7 0 0 25 0 5 0 518188136 378462208 31048 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458771125 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 24.96
Current children cumulated vsize (KiB) 369592

[startup+12.7005 s]
/proc/loadavg: 7.64 7.99 7.67 9/186 11745
/proc/meminfo: memFree=15166100/32951124 swapFree=67111528/67111528
[pid=11725] ppid=11723 vsize=439692 CPUtime=50.46 cores=1,3,5,7
/proc/11725/stat : 11725 (ttsth-5-4) S 11723 11725 9766 0 -1 4202496 50361 0 0 0 4979 67 0 0 25 0 5 0 518188131 450244608 50287 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458747877 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/11725/statm: 109923 50287 175 36 0 107828 0
[pid=11725/tid=11727] ppid=11723 vsize=439692 CPUtime=12.6 cores=1,3,5,7
/proc/11725/task/11727/stat : 11727 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 11357 0 0 0 1243 17 0 0 25 0 5 0 518188136 450244608 50287 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11725/tid=11728] ppid=11723 vsize=439692 CPUtime=12.55 cores=1,3,5,7
/proc/11725/task/11728/stat : 11728 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 11193 0 0 0 1237 18 0 0 25 0 5 0 518188136 450244608 50287 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11725/tid=11729] ppid=11723 vsize=439692 CPUtime=12.61 cores=1,3,5,7
/proc/11725/task/11729/stat : 11729 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 13165 0 0 0 1246 15 0 0 25 0 5 0 518188136 450244608 50287 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11725/tid=11730] ppid=11723 vsize=439692 CPUtime=12.62 cores=1,3,5,7
/proc/11725/task/11730/stat : 11730 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 13645 0 0 0 1247 15 0 0 25 0 5 0 518188136 450244608 50287 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 50.46
Current children cumulated vsize (KiB) 439692

[startup+25.5062 s]
/proc/loadavg: 7.79 8.00 7.68 9/186 11745
/proc/meminfo: memFree=14991292/32951124 swapFree=67111528/67111528
[pid=11725] ppid=11723 vsize=539092 CPUtime=101.54 cores=1,3,5,7
/proc/11725/stat : 11725 (ttsth-5-4) S 11723 11725 9766 0 -1 4202496 75766 0 0 0 10038 116 0 0 25 0 5 0 518188131 552030208 75692 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458747877 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/11725/statm: 134773 75692 175 36 0 132678 0
[pid=11725/tid=11727] ppid=11723 vsize=539092 CPUtime=25.39 cores=1,3,5,7
/proc/11725/task/11727/stat : 11727 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 17400 0 0 0 2509 30 0 0 25 0 5 0 518188136 552030208 75692 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11725/tid=11728] ppid=11723 vsize=539092 CPUtime=25.26 cores=1,3,5,7
/proc/11725/task/11728/stat : 11728 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 16631 0 0 0 2494 32 0 0 25 0 5 0 518188136 552030208 75692 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11725/tid=11729] ppid=11723 vsize=539092 CPUtime=25.4 cores=1,3,5,7
/proc/11725/task/11729/stat : 11729 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 19771 0 0 0 2514 26 0 0 25 0 5 0 518188136 552030208 75692 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11725/tid=11730] ppid=11723 vsize=539092 CPUtime=25.41 cores=1,3,5,7
/proc/11725/task/11730/stat : 11730 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 20963 0 0 0 2515 26 0 0 25 0 5 0 518188136 552030208 75692 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 101.54
Current children cumulated vsize (KiB) 539092

[startup+51.1004 s]
/proc/loadavg: 7.86 8.00 7.69 9/186 11746
/proc/meminfo: memFree=14711944/32951124 swapFree=67111528/67111528
[pid=11725] ppid=11723 vsize=671040 CPUtime=203.6 cores=1,3,5,7
/proc/11725/stat : 11725 (ttsth-5-4) S 11723 11725 9766 0 -1 4202496 111619 0 0 0 20124 236 0 0 25 0 5 0 518188131 687144960 111545 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458747877 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/11725/statm: 167760 111545 175 36 0 165665 0
[pid=11725/tid=11727] ppid=11723 vsize=671040 CPUtime=50.93 cores=1,3,5,7
/proc/11725/task/11727/stat : 11727 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 24684 0 0 0 5031 62 0 0 25 0 5 0 518188136 687144960 111545 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458771125 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11725/tid=11728] ppid=11723 vsize=671040 CPUtime=50.68 cores=1,3,5,7
/proc/11725/task/11728/stat : 11728 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 24682 0 0 0 5003 65 0 0 25 0 5 0 518188136 687144960 111545 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11725/tid=11729] ppid=11723 vsize=671040 CPUtime=50.95 cores=1,3,5,7
/proc/11725/task/11729/stat : 11729 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 29031 0 0 0 5043 52 0 0 25 0 5 0 518188136 687144960 111546 33554432000 4194304 4338100 140734933938368 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11725/tid=11730] ppid=11723 vsize=671040 CPUtime=50.97 cores=1,3,5,7
/proc/11725/task/11730/stat : 11730 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 32222 0 0 0 5042 55 0 0 25 0 5 0 518188136 687144960 111546 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458771125 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 203.6
Current children cumulated vsize (KiB) 671040

[startup+102.301 s]
/proc/loadavg: 8.03 8.03 7.71 9/186 11747
/proc/meminfo: memFree=14272980/32951124 swapFree=67111528/67111528
[pid=11725] ppid=11723 vsize=892072 CPUtime=407.63 cores=1,3,5,7
/proc/11725/stat : 11725 (ttsth-5-4) S 11723 11725 9766 0 -1 4202496 169843 0 0 0 40355 408 0 0 25 0 5 0 518188131 913481728 169769 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458747877 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/11725/statm: 223018 169769 175 36 0 220923 0
[pid=11725/tid=11727] ppid=11723 vsize=892072 CPUtime=101.93 cores=1,3,5,7
/proc/11725/task/11727/stat : 11727 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 44907 0 0 0 10091 102 0 0 25 0 5 0 518188136 913481728 169769 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458771125 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11725/tid=11728] ppid=11723 vsize=892072 CPUtime=101.49 cores=1,3,5,7
/proc/11725/task/11728/stat : 11728 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 35283 0 0 0 10035 114 0 0 25 0 5 0 518188136 913481728 169769 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11725/tid=11729] ppid=11723 vsize=892072 CPUtime=102.05 cores=1,3,5,7
/proc/11725/task/11729/stat : 11729 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 41442 0 0 0 10112 93 0 0 25 0 5 0 518188136 913481728 169769 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458771125 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11725/tid=11730] ppid=11723 vsize=892072 CPUtime=102.08 cores=1,3,5,7
/proc/11725/task/11730/stat : 11730 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 47210 0 0 0 10111 97 0 0 25 0 5 0 518188136 913481728 169769 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458771125 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 407.63
Current children cumulated vsize (KiB) 892072

[startup+162.301 s]
/proc/loadavg: 7.99 8.02 7.73 9/186 11749
/proc/meminfo: memFree=13861536/32951124 swapFree=67111528/67111528
[pid=11725] ppid=11723 vsize=1062544 CPUtime=646.64 cores=1,3,5,7
/proc/11725/stat : 11725 (ttsth-5-4) S 11723 11725 9766 0 -1 4202496 214655 0 0 0 64058 606 0 0 25 0 5 0 518188131 1088045056 214581 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458747877 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/11725/statm: 265636 214583 175 36 0 263541 0
[pid=11725/tid=11727] ppid=11723 vsize=1062544 CPUtime=161.72 cores=1,3,5,7
/proc/11725/task/11727/stat : 11727 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 61348 0 0 0 16021 151 0 0 25 0 5 0 518188136 1088045056 214583 33554432000 4194304 4338100 140734933938368 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11725/tid=11728] ppid=11723 vsize=1062544 CPUtime=161.15 cores=1,3,5,7
/proc/11725/task/11728/stat : 11728 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 42773 0 0 0 15943 172 0 0 25 0 5 0 518188136 1088045056 214584 33554432000 4194304 4338100 140734933938368 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11725/tid=11729] ppid=11723 vsize=1062544 CPUtime=161.88 cores=1,3,5,7
/proc/11725/task/11729/stat : 11729 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 50193 0 0 0 16051 137 0 0 25 0 5 0 518188136 1088045056 214584 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254010 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11725/tid=11730] ppid=11723 vsize=1062544 CPUtime=161.82 cores=1,3,5,7
/proc/11725/task/11730/stat : 11730 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 59343 0 0 0 16037 145 0 0 25 0 5 0 518188136 1088045056 214585 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 646.64
Current children cumulated vsize (KiB) 1062544


################
# More data... #
################

/proc/11725/task/11730/stat : 11730 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 111385 0 0 0 69397 536 0 0 25 0 5 0 518188136 1894789120 427154 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458771125 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 2796.63
Current children cumulated vsize (KiB) 1850380

[startup+762.3 s]
/proc/loadavg: 8.02 8.01 7.84 9/186 11765
/proc/meminfo: memFree=11645628/32951124 swapFree=67111528/67111528
[pid=11725] ppid=11723 vsize=1915248 CPUtime=3035.59 cores=1,3,5,7
/proc/11725/stat : 11725 (ttsth-5-4) S 11723 11725 9766 0 -1 4202496 444539 0 0 0 301091 2468 0 0 25 0 5 0 518188131 1961213952 444465 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458747877 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/11725/statm: 478812 444465 175 36 0 476717 0
[pid=11725/tid=11727] ppid=11723 vsize=1915248 CPUtime=758.67 cores=1,3,5,7
/proc/11725/task/11727/stat : 11727 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 149010 0 0 0 75226 641 0 0 25 0 5 0 518188136 1961213952 444465 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11725/tid=11728] ppid=11723 vsize=1915248 CPUtime=758.37 cores=1,3,5,7
/proc/11725/task/11728/stat : 11728 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 71604 0 0 0 75164 673 0 0 25 0 5 0 518188136 1961213952 444465 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11725/tid=11729] ppid=11723 vsize=1915248 CPUtime=759.26 cores=1,3,5,7
/proc/11725/task/11729/stat : 11729 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 109319 0 0 0 75348 578 0 0 25 0 5 0 518188136 1961213952 444465 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11725/tid=11730] ppid=11723 vsize=1915248 CPUtime=759.21 cores=1,3,5,7
/proc/11725/task/11730/stat : 11730 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 113605 0 0 0 75347 574 0 0 25 0 5 0 518188136 1961213952 444465 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3035.59
Current children cumulated vsize (KiB) 1915248

[startup+822.3 s]
/proc/loadavg: 8.06 8.02 7.85 9/186 11766
/proc/meminfo: memFree=11458148/32951124 swapFree=67111528/67111528
[pid=11725] ppid=11723 vsize=1961220 CPUtime=3274.03 cores=1,3,5,7
/proc/11725/stat : 11725 (ttsth-5-4) S 11723 11725 9766 0 -1 4202496 457726 0 0 0 324806 2597 0 0 25 0 5 0 518188131 2008289280 457652 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458747877 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/11725/statm: 490305 457652 175 36 0 488210 0
[pid=11725/tid=11727] ppid=11723 vsize=1961220 CPUtime=817.95 cores=1,3,5,7
/proc/11725/task/11727/stat : 11727 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 155035 0 0 0 81127 668 0 0 25 0 5 0 518188136 2008289280 457652 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11725/tid=11728] ppid=11723 vsize=1961220 CPUtime=818.02 cores=1,3,5,7
/proc/11725/task/11728/stat : 11728 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 72543 0 0 0 81089 713 0 0 25 0 5 0 518188136 2008289280 457652 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11725/tid=11729] ppid=11723 vsize=1961220 CPUtime=818.95 cores=1,3,5,7
/proc/11725/task/11729/stat : 11729 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 112458 0 0 0 81284 611 0 0 25 0 5 0 518188136 2008289280 457652 33554432000 4194304 4338100 140734933938368 18446744073709551615 244446340170 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11725/tid=11730] ppid=11723 vsize=1961220 CPUtime=819.04 cores=1,3,5,7
/proc/11725/task/11730/stat : 11730 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 116689 0 0 0 81300 604 0 0 25 0 5 0 518188136 2008289280 457652 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3274.03
Current children cumulated vsize (KiB) 1961220

[startup+882.301 s]
/proc/loadavg: 8.02 8.01 7.86 9/186 11768
/proc/meminfo: memFree=11307740/32951124 swapFree=67111528/67111528
[pid=11725] ppid=11723 vsize=1998532 CPUtime=3512.94 cores=1,3,5,7
/proc/11725/stat : 11725 (ttsth-5-4) S 11723 11725 9766 0 -1 4202496 467386 0 0 0 348548 2746 0 0 25 0 5 0 518188131 2046496768 467312 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458747877 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/11725/statm: 499633 467312 175 36 0 497538 0
[pid=11725/tid=11727] ppid=11723 vsize=1998532 CPUtime=877.5 cores=1,3,5,7
/proc/11725/task/11727/stat : 11727 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 159108 0 0 0 87054 696 0 0 25 0 5 0 518188136 2046496768 467312 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458771125 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11725/tid=11728] ppid=11723 vsize=1998532 CPUtime=877.78 cores=1,3,5,7
/proc/11725/task/11728/stat : 11728 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 74435 0 0 0 87020 758 0 0 25 0 5 0 518188136 2046496768 467312 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11725/tid=11729] ppid=11723 vsize=1998532 CPUtime=878.75 cores=1,3,5,7
/proc/11725/task/11729/stat : 11729 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 114175 0 0 0 87227 648 0 0 25 0 5 0 518188136 2046496768 467312 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458771125 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11725/tid=11730] ppid=11723 vsize=1998532 CPUtime=878.83 cores=1,3,5,7
/proc/11725/task/11730/stat : 11730 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 118667 0 0 0 87241 642 0 0 25 0 5 0 518188136 2046496768 467312 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458771125 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3512.94
Current children cumulated vsize (KiB) 1998532

[startup+942.301 s]
/proc/loadavg: 8.01 8.01 7.87 9/186 11769
/proc/meminfo: memFree=11145784/32951124 swapFree=67111528/67111528
[pid=11725] ppid=11723 vsize=2037608 CPUtime=3751.62 cores=1,3,5,7
/proc/11725/stat : 11725 (ttsth-5-4) S 11723 11725 9766 0 -1 4202496 478567 0 0 0 372245 2917 0 0 25 0 5 0 518188131 2086510592 478493 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458747877 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/11725/statm: 509402 478493 175 36 0 507307 0
[pid=11725/tid=11727] ppid=11723 vsize=2037608 CPUtime=937.19 cores=1,3,5,7
/proc/11725/task/11727/stat : 11727 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 165358 0 0 0 92973 746 0 0 25 0 5 0 518188136 2086510592 478493 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11725/tid=11728] ppid=11723 vsize=2037608 CPUtime=937.2 cores=1,3,5,7
/proc/11725/task/11728/stat : 11728 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 74836 0 0 0 92919 801 0 0 25 0 5 0 518188136 2086510592 478493 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11725/tid=11729] ppid=11723 vsize=2037608 CPUtime=938.55 cores=1,3,5,7
/proc/11725/task/11729/stat : 11729 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 118210 0 0 0 93166 689 0 0 25 0 5 0 518188136 2086510592 478493 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11725/tid=11730] ppid=11723 vsize=2037608 CPUtime=938.62 cores=1,3,5,7
/proc/11725/task/11730/stat : 11730 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 119162 0 0 0 93182 680 0 0 25 0 5 0 518188136 2086510592 478493 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458771125 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3751.62
Current children cumulated vsize (KiB) 2037608

[startup+1002.31 s]
/proc/loadavg: 8.08 8.02 7.88 9/186 11771
/proc/meminfo: memFree=10967456/32951124 swapFree=67111528/67111528
[pid=11725] ppid=11723 vsize=2127248 CPUtime=3990.48 cores=1,3,5,7
/proc/11725/stat : 11725 (ttsth-5-4) S 11723 11725 9766 0 -1 4202496 502383 0 0 0 395944 3104 0 0 25 0 5 0 518188131 2178301952 502309 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458747877 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/11725/statm: 531812 502309 175 36 0 529717 0
[pid=11725/tid=11727] ppid=11723 vsize=2127248 CPUtime=996.85 cores=1,3,5,7
/proc/11725/task/11727/stat : 11727 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 178704 0 0 0 98887 798 0 0 25 0 5 0 518188136 2178301952 502309 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11725/tid=11728] ppid=11723 vsize=2127248 CPUtime=996.85 cores=1,3,5,7
/proc/11725/task/11728/stat : 11728 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 82773 0 0 0 98832 853 0 0 20 0 5 0 518188136 2178301952 502309 33554432000 4194304 4338100 140734933938368 18446744073709551615 4285730 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11725/tid=11729] ppid=11723 vsize=2127248 CPUtime=998.22 cores=1,3,5,7
/proc/11725/task/11729/stat : 11729 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 120176 0 0 0 99090 732 0 0 25 0 5 0 518188136 2178301952 502309 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11725/tid=11730] ppid=11723 vsize=2127248 CPUtime=998.49 cores=1,3,5,7
/proc/11725/task/11730/stat : 11730 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 119729 0 0 0 99129 720 0 0 20 0 5 0 518188136 2178301952 502309 33554432000 4194304 4338100 140734933938368 18446744073709551615 244446340334 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3990.48
Current children cumulated vsize (KiB) 2127248

[startup+1062.3 s]
/proc/loadavg: 7.84 7.98 7.87 9/186 11773
/proc/meminfo: memFree=10859708/32951124 swapFree=67111528/67111528
[pid=11725] ppid=11723 vsize=2176348 CPUtime=4228.89 cores=1,3,5,7
/proc/11725/stat : 11725 (ttsth-5-4) S 11723 11725 9766 0 -1 4202496 515595 0 0 0 419635 3254 0 0 25 0 5 0 518188131 2228580352 515521 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458747877 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/11725/statm: 544087 515521 175 36 0 541992 0
[pid=11725/tid=11727] ppid=11723 vsize=2176348 CPUtime=1056.34 cores=1,3,5,7
/proc/11725/task/11727/stat : 11727 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 185582 0 0 0 104793 841 0 0 25 0 5 0 518188136 2228580352 515521 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11725/tid=11728] ppid=11723 vsize=2176348 CPUtime=1056.21 cores=1,3,5,7
/proc/11725/task/11728/stat : 11728 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 85753 0 0 0 104729 892 0 0 25 0 5 0 518188136 2228580352 515521 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11725/tid=11729] ppid=11723 vsize=2176348 CPUtime=1057.94 cores=1,3,5,7
/proc/11725/task/11729/stat : 11729 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 122971 0 0 0 105029 765 0 0 25 0 5 0 518188136 2228580352 515521 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11725/tid=11730] ppid=11723 vsize=2176348 CPUtime=1058.32 cores=1,3,5,7
/proc/11725/task/11730/stat : 11730 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 120288 0 0 0 105078 754 0 0 25 0 5 0 518188136 2228580352 515521 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4228.89
Current children cumulated vsize (KiB) 2176348

[startup+1122.3 s]
/proc/loadavg: 7.98 7.99 7.88 9/186 11774
/proc/meminfo: memFree=10723188/32951124 swapFree=67111528/67111528
[pid=11725] ppid=11723 vsize=2209680 CPUtime=4467.65 cores=1,3,5,7
/proc/11725/stat : 11725 (ttsth-5-4) S 11723 11725 9766 0 -1 4202496 525079 0 0 0 443330 3435 0 0 25 0 5 0 518188131 2262712320 525005 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458747877 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/11725/statm: 552420 525005 175 36 0 550325 0
[pid=11725/tid=11727] ppid=11723 vsize=2209680 CPUtime=1115.68 cores=1,3,5,7
/proc/11725/task/11727/stat : 11727 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 189573 0 0 0 110678 890 0 0 20 0 5 0 518188136 2262712320 525005 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11725/tid=11728] ppid=11723 vsize=2209680 CPUtime=1116.02 cores=1,3,5,7
/proc/11725/task/11728/stat : 11728 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 87358 0 0 0 110661 941 0 0 25 0 5 0 518188136 2262712320 525005 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11725/tid=11729] ppid=11723 vsize=2209680 CPUtime=1117.73 cores=1,3,5,7
/proc/11725/task/11729/stat : 11729 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 126348 0 0 0 110964 809 0 0 25 0 5 0 518188136 2262712320 525005 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11725/tid=11730] ppid=11723 vsize=2209680 CPUtime=1118.16 cores=1,3,5,7
/proc/11725/task/11730/stat : 11730 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 120799 0 0 0 111022 794 0 0 25 0 5 0 518188136 2262712320 525005 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458771125 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4467.65
Current children cumulated vsize (KiB) 2209680

[startup+1182.3 s]
/proc/loadavg: 8.07 8.01 7.89 9/186 11776
/proc/meminfo: memFree=10559256/32951124 swapFree=67111528/67111528
[pid=11725] ppid=11723 vsize=2230496 CPUtime=4706.25 cores=1,3,5,7
/proc/11725/stat : 11725 (ttsth-5-4) S 11723 11725 9766 0 -1 4202496 531673 0 0 0 467013 3612 0 0 25 0 5 0 518188131 2284027904 531599 33554432000 4194304 4338100 140734933938368 18446744073709551615 244458747877 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/11725/statm: 557624 531599 175 36 0 555529 0
[pid=11725/tid=11727] ppid=11723 vsize=2230496 CPUtime=1175.24 cores=1,3,5,7
/proc/11725/task/11727/stat : 11727 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 193494 0 0 0 116583 941 0 0 25 0 5 0 518188136 2284027904 531599 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11725/tid=11728] ppid=11723 vsize=2230496 CPUtime=1175.77 cores=1,3,5,7
/proc/11725/task/11728/stat : 11728 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 88988 0 0 0 116588 989 0 0 25 0 5 0 518188136 2284027904 531599 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11725/tid=11729] ppid=11723 vsize=2230496 CPUtime=1177.23 cores=1,3,5,7
/proc/11725/task/11729/stat : 11729 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 126801 0 0 0 116876 847 0 0 25 0 5 0 518188136 2284027904 531599 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11725/tid=11730] ppid=11723 vsize=2230496 CPUtime=1177.94 cores=1,3,5,7
/proc/11725/task/11730/stat : 11730 (ttsth-5-4) R 11723 11725 9766 0 -1 4202560 121389 0 0 0 116960 834 0 0 25 0 5 0 518188136 2284027904 531599 33554432000 4194304 4338100 140734933938368 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4706.25
Current children cumulated vsize (KiB) 2230496



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1205.9 s]
/proc/loadavg: 8.11 8.02 7.90 6/183 11776
/proc/meminfo: memFree=12238540/32951124 swapFree=67111528/67111528
[pid=11725] ppid=11723 vsize=0 CPUtime=4800.11 cores=1,3,5,7
/proc/11725/stat : 11725 (ttsth-5-4) Z 11723 11725 9766 0 -1 4203548 539831 0 0 0 476321 3690 0 0 18 0 2 0 518188131 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/11725/statm: 0 0 0 0 0 0 0
[pid=11725/tid=11730] ppid=11723 vsize=0 CPUtime=1201.51 cores=1,3,5,7
/proc/11725/task/11730/stat : 11730 (ttsth-5-4) R 11723 11725 9766 0 -1 4203588 122425 0 0 0 119292 859 0 0 25 0 2 0 518188136 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4800.11
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1205.9 s]
/proc/loadavg: 8.11 8.02 7.90 6/183 11776
/proc/meminfo: memFree=12238540/32951124 swapFree=67111528/67111528
[pid=11725] ppid=11723 vsize=0 CPUtime=4800.11 cores=1,3,5,7
/proc/11725/stat : 11725 (ttsth-5-4) Z 11723 11725 9766 0 -1 4203548 539831 0 0 0 476321 3690 0 0 18 0 2 0 518188131 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/11725/statm: 0 0 0 0 0 0 0
[pid=11725/tid=11730] ppid=11723 vsize=0 CPUtime=1201.51 cores=1,3,5,7
/proc/11725/task/11730/stat : 11730 (ttsth-5-4) R 11723 11725 9766 0 -1 4203588 122425 0 0 0 119292 859 0 0 25 0 2 0 518188136 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4800.11
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1205.92
CPU time (s): 4800.14
CPU user time (s): 4763.22
CPU system time (s): 36.9184
CPU usage (%): 398.049
Max. virtual memory (cumulated for all children) (KiB): 2259420

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4763.22
system time used= 36.9184
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 539831
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 144971
involuntary context switches= 32426

runsolver used 2.16167 second user time and 4.41733 second system time

The end

Launcher Data 

Begin job on node105 at 2011-04-29 10:31:17
IDJOB=3260418
IDBENCH=83183
IDSOLVER=1654
FILE ID=node105/3260418-1304065877
RUNJOBID= node105-1304064673-11621
PBS_JOBID= 13170404
Free space on /tmp= 71940 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/mosoi/289/289-sat-6x8.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3260418-1304065877/watcher-3260418-1304065877 -o /tmp/evaluation-result-3260418-1304065877/solver-3260418-1304065877 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3260418-1304065877.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 0836620198442c187bac3443fd060059
RANDOM SEED=303031272

node105.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5410.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      15583324 kB
Buffers:       2301264 kB
Cached:       14431536 kB
SwapCached:          0 kB
Active:        8941560 kB
Inactive:      7855056 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      15583324 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:              24 kB
Writeback:           0 kB
AnonPages:       62836 kB
Mapped:          14772 kB
Slab:           507172 kB
PageTables:       4636 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   183992 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71924 MiB
End job on node105 at 2011-04-29 10:51:25