Trace number 3261738

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.31 1218.03

General information on the benchmark

Namecrafted/skvortsov/battleship/
battleship-11-21-sat.cnf
MD5SUMb96c20b4b7951c39c5802e0553b42cad
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.005999
Satisfiable
(Un)Satisfiability was proved
Number of variables231
Number of clauses1276
Sum of the clauses size4851
Maximum clause length21
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 21155
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 5121

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3261738-1304089037.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 231
0.00/0.00	c Clauses = 1276
0.00/0.00	c Literals = 4851
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 231
0.00/0.00	c Clauses = 1276
0.00/0.00	c Literals = 4851
0.00/0.00	c a2
0.00/0.00	c a1
19.80/19.82	c Independent sub-problems = 1
19.80/19.82	c Sub-problem with 231 variables and 1276 clauses
19.80/19.82	c using permutation method 4
19.80/19.82	c count = 3
19.80/19.82	c Ordering finished 19.820 seconds
19.80/19.82	c mapped 19.820 seconds
19.80/19.82	c permuted 19.820 seconds
19.80/19.82	c about to sort
19.80/19.82	c sorted
19.80/19.83	c clauses now 1276
19.80/19.83	c init finished 19.820 seconds
19.80/19.83	c tree built 19.830 seconds
19.80/19.83	c 
19.80/19.83	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
19.80/19.83	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
19.80/19.83	c ................................ | ..........                    |                                 19.8         0.0             
19.80/19.83	c ................................ | ..........                  1 |                                 19.8         0.1             
19.80/19.83	c ................................ | ..........                  2 |                                 19.8         0.1             
19.80/19.83	c ................................ | ..........                  3 |                                 19.8         0.2             
19.80/19.83	c ................................ | ..........                  4 |                                 19.8         0.2             
19.80/19.83	c ................................ | ..........                  5 |                                 19.8         0.3             
19.80/19.83	c ................................ | ..........               1000 | 130          165                19.8        50.4         3535
96.31/39.00	c ................................ | ..........             459000 | 136452       5402               96.3      4765.9     11777458
191.68/62.99	c ................................ | ..........             810000 | 245421       6087              192.1      4217.4     16876724
287.90/87.02	c ................................ | ..........            1088000 | 331178       6827              288.0      3777.6     21177088
384.11/111.17	c ................................ | ..........            1304000 | 398162       7037              384.4      3392.4     23861616
480.32/135.22	c ................................ | *.........            1517000 | 463979       7395              480.4      3157.8     28345991
576.01/159.26	c ................................ | *.........            1705000 | 523123       7634              576.3      2958.7     28920836
673.66/183.74	c ................................ | *.........            1910000 | 587551       8455              673.8      2834.7     30909403
768.23/207.42	c ................................ | *.........            2047000 | 631073       9115              768.3      2664.3     34429433
864.18/231.55	c ................................ | *.........            2205000 | 680947       9731              864.4      2551.0     35378053
960.28/255.68	c ................................ | **........            2439000 | 755286       9863              960.6      2539.0     36381707
1056.77/279.82	c ................................ | **........            2692000 | 834851       10208            1056.9      2547.1     37252783
1151.79/303.66	c ................................ | **........            2874000 | 892337       10316            1152.0      2494.8     39091141
1248.00/327.73	c ................................ | **........            3055000 | 948078       10642            1248.1      2447.7     42263245
1344.56/351.92	c ................................ | **........            3200000 | 991576       10877            1344.6      2379.8     44257346
1439.94/375.84	c ................................ | ***.......            3393000 | 1049485      11009            1440.1      2356.1     47774955
1536.68/400.15	c ................................ | ***.......            3630000 | 1123489      11114            1536.9      2361.9     50411032
1632.45/424.15	c ................................ | ***.......            3803000 | 1176136      11276            1632.7      2329.3     51433501
1728.91/448.30	c ................................ | ***.......            3960000 | 1224365      11405            1728.9      2290.4     52384445
1824.69/472.36	c ................................ | ***.......            4083000 | 1262704      11432            1824.9      2237.3     53515540
1920.17/496.28	c ................................ | ****......            4237000 | 1310951      11447            1920.5      2206.2     55607886
2017.08/520.57	c ................................ | ****......            4387000 | 1356741      11462            2017.4      2174.6     56527199
2112.53/544.43	c ................................ | ****......            4537000 | 1402230      11542            2112.7      2147.5     57029676
2209.20/568.61	c ................................ | ****......            4636000 | 1433429      11569            2209.2      2098.5     59555335
2305.45/592.70	c ................................ | ****......            4737000 | 1465773      11656            2305.4      2054.7     63747418
2400.36/616.59	c ................................ | *****.....            4830000 | 1495794      11703            2400.8      2011.9     65282112
2496.55/640.61	c ................................ | *****.....            4956000 | 1535713      11730            2496.6      1985.1     65468236
2592.39/664.67	c ................................ | *****.....            5091000 | 1578642      11778            2592.7      1963.6     67366963
2688.12/688.61	c ................................ | *****.....            5267000 | 1633397      11838            2688.1      1959.3     68755089
2785.08/712.99	c ................................ | *****.....            5384000 | 1669146      11856            2785.5      1932.9     70247395
2880.01/736.74	c ................................ | ******....            5484000 | 1700437      11871            2880.2      1904.0     71504929
2976.46/760.90	c ................................ | ******....            5575000 | 1727417      11895            2976.5      1873.0     72412351
3072.33/784.92	c ................................ | ******....            5677000 | 1757918      11910            3072.4      1847.7     74371844
3168.01/808.97	c ................................ | ******....            5826000 | 1803326      11955            3168.3      1838.8     75235934
3264.55/833.11	c ................................ | ******....            5958000 | 1845571      11964            3264.6      1825.0     75235934
3359.92/857.09	c ................................ | *******...            6082000 | 1883480      12030            3360.3      1810.0     75677218
3456.01/881.15	c ................................ | *******...            6162000 | 1908875      12045            3456.2      1782.9     76050893
3552.68/905.35	c ................................ | *******...            6258000 | 1938775      12069            3552.9      1761.4     77821860
3649.18/929.53	c ................................ | *******...            6342000 | 1964719      12069            3649.3      1737.9     78775619
3743.94/953.36	c ................................ | *******...            6417000 | 1988521      12072            3744.2      1713.9     78969015
3840.18/977.48	c ................................ | ********..            6525000 | 2022172      12081            3840.5      1699.0     78969015
3936.04/1001.44	c ................................ | ********..            6633000 | 2054437      12105            3936.2      1685.1     78969015
4033.09/1025.73	c ................................ | ********..            6740000 | 2087426      12120            4033.2      1671.1     80796573
4130.09/1050.08	c ................................ | ********..            6809000 | 2109230      12129            4130.4      1648.5     82139804
4224.38/1073.65	c ................................ | ********..            6860000 | 2125255      12129            4224.6      1623.8     83247539
4320.65/1097.71	c ................................ | *********.            6930000 | 2147945      12132            4320.7      1603.9     84429103
4416.83/1121.87	c ................................ | *********.            7005000 | 2171867      12132            4417.1      1585.9     85624803
4512.70/1145.81	c ................................ | *********.            7079000 | 2195154      12150            4512.8      1568.7     87039728
4608.03/1169.71	c ................................ | *********.            7155000 | 2219647      12153            4608.1      1552.7     87562336
4704.51/1193.91	c ................................ | *********.            7270000 | 2255619      12174            4704.6      1545.3     87835319

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3261738-1304089037/watcher-3261738-1304089037 -o /tmp/evaluation-result-3261738-1304089037/solver-3261738-1304089037 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3261738-1304089037.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.10 8.11 7.85 5/181 25406
/proc/meminfo: memFree=15462692/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=9904 CPUtime=0 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) R 25404 25406 25048 0 -1 4202496 542 0 0 0 0 0 0 0 25 0 1 0 520504614 10141696 483 33554432000 4194304 4338100 140733802659744 18446744073709551615 270612505794 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/25406/statm: 2476 483 152 36 0 381 0

[startup+0.0728591 s]
/proc/loadavg: 8.10 8.11 7.85 5/181 25406
/proc/meminfo: memFree=15462692/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=9904 CPUtime=0.07 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) R 25404 25406 25048 0 -1 4202496 549 0 0 0 7 0 0 0 25 0 1 0 520504614 10141696 490 33554432000 4194304 4338100 140733802659744 18446744073709551615 4295924 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/25406/statm: 2476 490 154 36 0 381 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 9904

[startup+0.100853 s]
/proc/loadavg: 8.10 8.11 7.85 5/181 25406
/proc/meminfo: memFree=15462692/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=9904 CPUtime=0.1 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) R 25404 25406 25048 0 -1 4202496 549 0 0 0 10 0 0 0 25 0 1 0 520504614 10141696 490 33554432000 4194304 4338100 140733802659744 18446744073709551615 4295924 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/25406/statm: 2476 490 154 36 0 381 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 9904

[startup+0.30082 s]
/proc/loadavg: 8.10 8.11 7.85 5/181 25406
/proc/meminfo: memFree=15462692/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=9904 CPUtime=0.3 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) R 25404 25406 25048 0 -1 4202496 549 0 0 0 30 0 0 0 25 0 1 0 520504614 10141696 490 33554432000 4194304 4338100 140733802659744 18446744073709551615 4295924 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/25406/statm: 2476 490 154 36 0 381 0
Current children cumulated CPU time (s) 0.3
Current children cumulated vsize (KiB) 9904

[startup+0.700747 s]
/proc/loadavg: 8.10 8.11 7.85 5/181 25406
/proc/meminfo: memFree=15462692/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=9904 CPUtime=0.7 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) R 25404 25406 25048 0 -1 4202496 550 0 0 0 70 0 0 0 25 0 1 0 520504614 10141696 491 33554432000 4194304 4338100 140733802659744 18446744073709551615 4296000 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/25406/statm: 2476 491 155 36 0 381 0
Current children cumulated CPU time (s) 0.7
Current children cumulated vsize (KiB) 9904

[startup+1.50061 s]
/proc/loadavg: 8.10 8.11 7.85 6/182 25407
/proc/meminfo: memFree=15461236/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=9904 CPUtime=1.5 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) R 25404 25406 25048 0 -1 4202496 550 0 0 0 150 0 0 0 25 0 1 0 520504614 10141696 491 33554432000 4194304 4338100 140733802659744 18446744073709551615 4295920 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/25406/statm: 2476 491 155 36 0 381 0
Current children cumulated CPU time (s) 1.5
Current children cumulated vsize (KiB) 9904

[startup+3.10032 s]
/proc/loadavg: 7.85 8.06 7.84 6/182 25407
/proc/meminfo: memFree=15460492/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=9904 CPUtime=3.1 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) R 25404 25406 25048 0 -1 4202496 550 0 0 0 310 0 0 0 25 0 1 0 520504614 10141696 491 33554432000 4194304 4338100 140733802659744 18446744073709551615 4296000 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/25406/statm: 2476 491 155 36 0 381 0
Current children cumulated CPU time (s) 3.1
Current children cumulated vsize (KiB) 9904

[startup+6.30076 s]
/proc/loadavg: 7.85 8.06 7.84 6/182 25407
/proc/meminfo: memFree=15455904/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=9904 CPUtime=6.3 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) R 25404 25406 25048 0 -1 4202496 550 0 0 0 630 0 0 0 25 0 1 0 520504614 10141696 491 33554432000 4194304 4338100 140733802659744 18446744073709551615 4295973 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/25406/statm: 2476 491 155 36 0 381 0
Current children cumulated CPU time (s) 6.3
Current children cumulated vsize (KiB) 9904

[startup+12.7006 s]
/proc/loadavg: 7.41 7.96 7.81 6/182 25407
/proc/meminfo: memFree=15440772/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=9904 CPUtime=12.7 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) R 25404 25406 25048 0 -1 4202496 550 0 0 0 1270 0 0 0 25 0 1 0 520504614 10141696 491 33554432000 4194304 4338100 140733802659744 18446744073709551615 4296022 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/25406/statm: 2476 491 155 36 0 381 0
Current children cumulated CPU time (s) 12.7
Current children cumulated vsize (KiB) 9904

[startup+25.5003 s]
/proc/loadavg: 7.28 7.91 7.79 9/186 25412
/proc/meminfo: memFree=15217680/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=345328 CPUtime=42.43 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) S 25404 25406 25048 0 -1 4202496 40097 0 0 0 4225 18 0 0 25 0 5 0 520504614 353615872 39925 33554432000 4194304 4338100 140733802659744 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/25406/statm: 86332 39925 176 36 0 84237 0
[pid=25406/tid=25409] ppid=25404 vsize=345328 CPUtime=5.65 cores=1,3,5,7
/proc/25406/task/25409/stat : 25409 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 8226 0 0 0 564 1 0 0 25 0 5 0 520506598 353615872 39925 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=25406/tid=25410] ppid=25404 vsize=345328 CPUtime=5.64 cores=1,3,5,7
/proc/25406/task/25410/stat : 25410 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 6394 0 0 0 559 5 0 0 25 0 5 0 520506598 353615872 39925 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=25406/tid=25411] ppid=25404 vsize=345328 CPUtime=5.66 cores=1,3,5,7
/proc/25406/task/25411/stat : 25411 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 18566 0 0 0 561 5 0 0 25 0 5 0 520506598 353615872 39926 33554432000 4194304 4338100 140733802659744 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=25406/tid=25412] ppid=25404 vsize=345328 CPUtime=5.62 cores=1,3,5,7
/proc/25406/task/25412/stat : 25412 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 5541 0 0 0 557 5 0 0 25 0 5 0 520506598 353615872 39926 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 42.43
Current children cumulated vsize (KiB) 345328

[startup+51.1008 s]
/proc/loadavg: 7.67 7.95 7.81 9/186 25412
/proc/meminfo: memFree=14791580/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=724208 CPUtime=144.58 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) S 25404 25406 25048 0 -1 4202496 118300 0 0 0 14395 63 0 0 25 0 5 0 520504614 741588992 118120 33554432000 4194304 4338100 140733802659744 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/25406/statm: 181052 118120 176 36 0 178957 0
[pid=25406/tid=25409] ppid=25404 vsize=724208 CPUtime=31.23 cores=1,3,5,7
/proc/25406/task/25409/stat : 25409 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 25079 0 0 0 3117 6 0 0 25 0 5 0 520506598 741588992 118120 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=25406/tid=25410] ppid=25404 vsize=724208 CPUtime=31.22 cores=1,3,5,7
/proc/25406/task/25410/stat : 25410 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 16146 0 0 0 3101 21 0 0 25 0 5 0 520506598 741588992 118120 33554432000 4194304 4338100 140733802659744 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=25406/tid=25411] ppid=25404 vsize=724208 CPUtime=31.22 cores=1,3,5,7
/proc/25406/task/25411/stat : 25411 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 59944 0 0 0 3108 14 0 0 25 0 5 0 520506598 741588992 118120 33554432000 4194304 4338100 140733802659744 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=25406/tid=25412] ppid=25404 vsize=724208 CPUtime=31.05 cores=1,3,5,7
/proc/25406/task/25412/stat : 25412 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 15760 0 0 0 3085 20 0 0 25 0 5 0 520506598 741588992 118120 33554432000 4194304 4338100 140733802659744 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 144.58
Current children cumulated vsize (KiB) 724208

[startup+102.307 s]
/proc/loadavg: 8.00 7.99 7.83 9/186 25414
/proc/meminfo: memFree=14444816/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=1006904 CPUtime=348.99 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) S 25404 25406 25048 0 -1 4202496 188016 0 0 0 34779 120 0 0 25 0 5 0 520504614 1031069696 187836 33554432000 4194304 4338100 140733802659744 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/25406/statm: 251726 187837 176 36 0 249631 0
[pid=25406/tid=25409] ppid=25404 vsize=1006904 CPUtime=82.41 cores=1,3,5,7
/proc/25406/task/25409/stat : 25409 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 46401 0 0 0 8228 13 0 0 25 0 5 0 520506598 1031069696 187837 33554432000 4194304 4338100 140733802659744 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=25406/tid=25410] ppid=25404 vsize=1006904 CPUtime=82.39 cores=1,3,5,7
/proc/25406/task/25410/stat : 25410 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 21294 0 0 0 8199 40 0 0 25 0 5 0 520506598 1031069696 187837 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=25406/tid=25411] ppid=25404 vsize=1006904 CPUtime=82.42 cores=1,3,5,7
/proc/25406/task/25411/stat : 25411 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 95365 0 0 0 8215 27 0 0 25 0 5 0 520506598 1031069696 187837 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=25406/tid=25412] ppid=25404 vsize=1006904 CPUtime=81.91 cores=1,3,5,7
/proc/25406/task/25412/stat : 25412 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 23586 0 0 0 8152 39 0 0 25 0 5 0 520506598 1031069696 187837 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 348.99
Current children cumulated vsize (KiB) 1006904

[startup+162.301 s]
/proc/loadavg: 8.21 8.05 7.85 9/186 25415
/proc/meminfo: memFree=14161296/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=1259440 CPUtime=588.27 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) S 25404 25406 25048 0 -1 4202496 245945 0 0 0 58648 179 0 0 25 0 5 0 520504614 1289666560 245765 33554432000 4194304 4338100 140733802659744 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/25406/statm: 314860 245765 176 36 0 312765 0
[pid=25406/tid=25409] ppid=25404 vsize=1259440 CPUtime=142.33 cores=1,3,5,7
/proc/25406/task/25409/stat : 25409 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 61517 0 0 0 14215 18 0 0 25 0 5 0 520506598 1289666560 245765 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=25406/tid=25410] ppid=25404 vsize=1259440 CPUtime=142.24 cores=1,3,5,7
/proc/25406/task/25410/stat : 25410 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 28024 0 0 0 14160 64 0 0 25 0 5 0 520506598 1289666560 245765 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=25406/tid=25411] ppid=25404 vsize=1259440 CPUtime=142.15 cores=1,3,5,7
/proc/25406/task/25411/stat : 25411 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 121039 0 0 0 14180 35 0 0 25 0 5 0 520506598 1289666560 245765 33554432000 4194304 4338100 140733802659744 18446744073709551615 4265634 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=25406/tid=25412] ppid=25404 vsize=1259440 CPUtime=141.67 cores=1,3,5,7
/proc/25406/task/25412/stat : 25412 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 33994 0 0 0 14107 60 0 0 25 0 5 0 520506598 1289666560 245765 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254317 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 588.27
Current children cumulated vsize (KiB) 1259440

[startup+222.301 s]
/proc/loadavg: 8.21 8.07 7.87 9/186 25417
/proc/meminfo: memFree=13939888/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=1450248 CPUtime=827.46 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) S 25404 25406 25048 0 -1 4202496 293876 0 0 0 82502 244 0 0 25 0 5 0 520504614 1485053952 293696 33554432000 4194304 4338100 140733802659744 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/25406/statm: 362562 293696 176 36 0 360467 0
[pid=25406/tid=25409] ppid=25404 vsize=1450248 CPUtime=202.28 cores=1,3,5,7
/proc/25406/task/25409/stat : 25409 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 61518 0 0 0 20208 20 0 0 25 0 5 0 520506598 1485053952 293696 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=25406/tid=25410] ppid=25404 vsize=1450248 CPUtime=201.89 cores=1,3,5,7
/proc/25406/task/25410/stat : 25410 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 28358 0 0 0 20101 88 0 0 25 0 5 0 520506598 1485053952 293696 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=25406/tid=25411] ppid=25404 vsize=1450248 CPUtime=201.92 cores=1,3,5,7
/proc/25406/task/25411/stat : 25411 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 168635 0 0 0 20143 49 0 0 25 0 5 0 520506598 1485053952 293696 33554432000 4194304 4338100 140733802659744 18446744073709551615 270612505986 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=25406/tid=25412] ppid=25404 vsize=1450248 CPUtime=201.5 cores=1,3,5,7
/proc/25406/task/25412/stat : 25412 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 33994 0 0 0 20065 85 0 0 25 0 5 0 520506598 1485053952 293696 33554432000 4194304 4338100 140733802659744 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 827.46
Current children cumulated vsize (KiB) 1450248

[startup+282.301 s]
/proc/loadavg: 8.13 8.07 7.88 9/186 25428
/proc/meminfo: memFree=13822504/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=1542672 CPUtime=1066.76 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) S 25404 25406 25048 0 -1 4202496 315779 0 0 0 106385 291 0 0 25 0 5 0 520504614 1579696128 315599 33554432000 4194304 4338100 140733802659744 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/25406/statm: 385668 315599 176 36 0 383573 0
[pid=25406/tid=25409] ppid=25404 vsize=1542672 CPUtime=262.17 cores=1,3,5,7
/proc/25406/task/25409/stat : 25409 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 61518 0 0 0 26197 20 0 0 25 0 5 0 520506598 1579696128 315599 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=25406/tid=25410] ppid=25404 vsize=1542672 CPUtime=261.82 cores=1,3,5,7
/proc/25406/task/25410/stat : 25410 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 30356 0 0 0 26076 106 0 0 25 0 5 0 520506598 1579696128 315599 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=25406/tid=25411] ppid=25404 vsize=1542672 CPUtime=261.75 cores=1,3,5,7
/proc/25406/task/25411/stat : 25411 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 186704 0 0 0 26118 57 0 0 25 0 5 0 520506598 1579696128 315599 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=25406/tid=25412] ppid=25404 vsize=1542672 CPUtime=261.14 cores=1,3,5,7
/proc/25406/task/25412/stat : 25412 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 35830 0 0 0 26009 105 0 0 25 0 5 0 520506598 1579696128 315599 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 1066.76

################
# More data... #
################

/proc/meminfo: memFree=8474784/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=2709284 CPUtime=2982.08 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) S 25404 25406 25048 0 -1 4202496 594620 0 0 0 297520 688 0 0 25 0 5 0 520504614 2774306816 594440 33554432000 4194304 4338100 140733802659744 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/25406/statm: 677321 594440 176 36 0 675226 0
[pid=25406/tid=25409] ppid=25404 vsize=2709284 CPUtime=741.71 cores=1,3,5,7
/proc/25406/task/25409/stat : 25409 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 143175 0 0 0 74119 52 0 0 25 0 5 0 520506598 2774306816 594440 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=25406/tid=25410] ppid=25404 vsize=2709284 CPUtime=741.15 cores=1,3,5,7
/proc/25406/task/25410/stat : 25410 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 30417 0 0 0 73860 255 0 0 25 0 5 0 520506598 2774306816 594440 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=25406/tid=25411] ppid=25404 vsize=2709284 CPUtime=740.79 cores=1,3,5,7
/proc/25406/task/25411/stat : 25411 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 380149 0 0 0 73955 124 0 0 25 0 5 0 520506598 2774306816 594440 33554432000 4194304 4338100 140733802659744 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=25406/tid=25412] ppid=25404 vsize=2709284 CPUtime=738.56 cores=1,3,5,7
/proc/25406/task/25412/stat : 25412 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 39508 0 0 0 73601 255 0 0 25 0 5 0 520506598 2774306816 594440 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 2982.08
Current children cumulated vsize (KiB) 2709284

[startup+822.3 s]
/proc/loadavg: 8.12 8.11 7.97 9/186 25477
/proc/meminfo: memFree=7724544/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=2796868 CPUtime=3221.44 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) S 25404 25406 25048 0 -1 4202496 616700 0 0 0 321414 730 0 0 25 0 5 0 520504614 2863992832 616520 33554432000 4194304 4338100 140733802659744 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/25406/statm: 699217 616520 176 36 0 697122 0
[pid=25406/tid=25409] ppid=25404 vsize=2796868 CPUtime=801.61 cores=1,3,5,7
/proc/25406/task/25409/stat : 25409 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 143175 0 0 0 80109 52 0 0 25 0 5 0 520506598 2863992832 616520 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=25406/tid=25410] ppid=25404 vsize=2796868 CPUtime=801.07 cores=1,3,5,7
/proc/25406/task/25410/stat : 25410 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 30417 0 0 0 79837 270 0 0 25 0 5 0 520506598 2863992832 616520 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=25406/tid=25411] ppid=25404 vsize=2796868 CPUtime=800.54 cores=1,3,5,7
/proc/25406/task/25411/stat : 25411 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 402229 0 0 0 79921 133 0 0 25 0 5 0 520506598 2863992832 616520 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=25406/tid=25412] ppid=25404 vsize=2796868 CPUtime=798.34 cores=1,3,5,7
/proc/25406/task/25412/stat : 25412 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 39508 0 0 0 79562 272 0 0 25 0 5 0 520506598 2863992832 616520 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3221.44
Current children cumulated vsize (KiB) 2796868

[startup+882.301 s]
/proc/loadavg: 8.15 8.12 7.98 9/186 25479
/proc/meminfo: memFree=6988532/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=2825020 CPUtime=3460.78 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) S 25404 25406 25048 0 -1 4202496 623076 0 0 0 345316 762 0 0 25 0 5 0 520504614 2892820480 622896 33554432000 4194304 4338100 140733802659744 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/25406/statm: 706255 622896 176 36 0 704160 0
[pid=25406/tid=25409] ppid=25404 vsize=2825020 CPUtime=861.51 cores=1,3,5,7
/proc/25406/task/25409/stat : 25409 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 145136 0 0 0 86098 53 0 0 25 0 5 0 520506598 2892820480 622896 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=25406/tid=25410] ppid=25404 vsize=2825020 CPUtime=860.96 cores=1,3,5,7
/proc/25406/task/25410/stat : 25410 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 30417 0 0 0 85810 286 0 0 25 0 5 0 520506598 2892820480 622896 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=25406/tid=25411] ppid=25404 vsize=2825020 CPUtime=860.52 cores=1,3,5,7
/proc/25406/task/25411/stat : 25411 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 406644 0 0 0 85916 136 0 0 25 0 5 0 520506598 2892820480 622896 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=25406/tid=25412] ppid=25404 vsize=2825020 CPUtime=857.92 cores=1,3,5,7
/proc/25406/task/25412/stat : 25412 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 39508 0 0 0 85508 284 0 0 25 0 5 0 520506598 2892820480 622896 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3460.78
Current children cumulated vsize (KiB) 2825020

[startup+942.301 s]
/proc/loadavg: 8.10 8.11 7.98 9/186 25480
/proc/meminfo: memFree=6265900/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=2915732 CPUtime=3700.08 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) S 25404 25406 25048 0 -1 4202496 646622 0 0 0 369215 793 0 0 25 0 5 0 520504614 2985709568 646442 33554432000 4194304 4338100 140733802659744 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/25406/statm: 728933 646442 176 36 0 726838 0
[pid=25406/tid=25409] ppid=25404 vsize=2915732 CPUtime=921.41 cores=1,3,5,7
/proc/25406/task/25409/stat : 25409 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 152620 0 0 0 92085 56 0 0 25 0 5 0 520506598 2985709568 646442 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=25406/tid=25410] ppid=25404 vsize=2915732 CPUtime=920.83 cores=1,3,5,7
/proc/25406/task/25410/stat : 25410 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 30417 0 0 0 91786 297 0 0 25 0 5 0 520506598 2985709568 646442 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=25406/tid=25411] ppid=25404 vsize=2915732 CPUtime=920.48 cores=1,3,5,7
/proc/25406/task/25411/stat : 25411 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 422706 0 0 0 91906 142 0 0 25 0 5 0 520506598 2985709568 646442 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=25406/tid=25412] ppid=25404 vsize=2915732 CPUtime=917.48 cores=1,3,5,7
/proc/25406/task/25412/stat : 25412 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 39508 0 0 0 91453 295 0 0 25 0 5 0 520506598 2985709568 646442 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3700.08
Current children cumulated vsize (KiB) 2915732

[startup+1002.3 s]
/proc/loadavg: 8.14 8.12 8.00 9/186 25482
/proc/meminfo: memFree=5689296/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=2915732 CPUtime=3939.63 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) S 25404 25406 25048 0 -1 4202496 650981 0 0 0 393153 810 0 0 25 0 5 0 520504614 2985709568 650801 33554432000 4194304 4338100 140733802659744 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/25406/statm: 728933 650801 176 36 0 726838 0
[pid=25406/tid=25409] ppid=25404 vsize=2915732 CPUtime=981.41 cores=1,3,5,7
/proc/25406/task/25409/stat : 25409 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 152620 0 0 0 98084 57 0 0 25 0 5 0 520506598 2985709568 650801 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=25406/tid=25410] ppid=25404 vsize=2915732 CPUtime=980.78 cores=1,3,5,7
/proc/25406/task/25410/stat : 25410 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 32658 0 0 0 97772 306 0 0 25 0 5 0 520506598 2985709568 650801 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=25406/tid=25411] ppid=25404 vsize=2915732 CPUtime=980.47 cores=1,3,5,7
/proc/25406/task/25411/stat : 25411 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 424230 0 0 0 97904 143 0 0 25 0 5 0 520506598 2985709568 650801 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=25406/tid=25412] ppid=25404 vsize=2915732 CPUtime=977.12 cores=1,3,5,7
/proc/25406/task/25412/stat : 25412 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 40102 0 0 0 97409 303 0 0 25 0 5 0 520506598 2985709568 650801 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3939.63
Current children cumulated vsize (KiB) 2915732

[startup+1062.3 s]
/proc/loadavg: 8.20 8.13 8.01 9/186 25483
/proc/meminfo: memFree=5080176/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=3034596 CPUtime=4179.22 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) S 25404 25406 25048 0 -1 4202496 680875 0 0 0 417076 846 0 0 25 0 5 0 520504614 3107426304 680695 33554432000 4194304 4338100 140733802659744 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/25406/statm: 758649 680695 176 36 0 756554 0
[pid=25406/tid=25409] ppid=25404 vsize=3034596 CPUtime=1041.38 cores=1,3,5,7
/proc/25406/task/25409/stat : 25409 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 162749 0 0 0 104077 61 0 0 25 0 5 0 520506598 3107426304 680695 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254317 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=25406/tid=25410] ppid=25404 vsize=3034596 CPUtime=1040.74 cores=1,3,5,7
/proc/25406/task/25410/stat : 25410 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 32658 0 0 0 103756 318 0 0 25 0 5 0 520506598 3107426304 680695 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=25406/tid=25411] ppid=25404 vsize=3034596 CPUtime=1040.47 cores=1,3,5,7
/proc/25406/task/25411/stat : 25411 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 443995 0 0 0 103897 150 0 0 25 0 5 0 520506598 3107426304 680695 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=25406/tid=25412] ppid=25404 vsize=3034596 CPUtime=1036.76 cores=1,3,5,7
/proc/25406/task/25412/stat : 25412 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 40102 0 0 0 103361 315 0 0 25 0 5 0 520506598 3107426304 680695 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4179.22
Current children cumulated vsize (KiB) 3034596

[startup+1122.3 s]
/proc/loadavg: 8.18 8.13 8.01 9/186 25485
/proc/meminfo: memFree=4548012/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=3125308 CPUtime=4418.81 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) S 25404 25406 25048 0 -1 4202496 703182 0 0 0 441007 874 0 0 25 0 5 0 520504614 3200315392 703002 33554432000 4194304 4338100 140733802659744 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/25406/statm: 781327 703002 176 36 0 779232 0
[pid=25406/tid=25409] ppid=25404 vsize=3125308 CPUtime=1101.36 cores=1,3,5,7
/proc/25406/task/25409/stat : 25409 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 172564 0 0 0 110069 67 0 0 25 0 5 0 520506598 3200315392 703002 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=25406/tid=25410] ppid=25404 vsize=3125308 CPUtime=1100.72 cores=1,3,5,7
/proc/25406/task/25410/stat : 25410 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 32658 0 0 0 109745 327 0 0 25 0 5 0 520506598 3200315392 703002 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=25406/tid=25411] ppid=25404 vsize=3125308 CPUtime=1100.32 cores=1,3,5,7
/proc/25406/task/25411/stat : 25411 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 456487 0 0 0 109877 155 0 0 25 0 5 0 520506598 3200315392 703002 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=25406/tid=25412] ppid=25404 vsize=3125308 CPUtime=1096.56 cores=1,3,5,7
/proc/25406/task/25412/stat : 25412 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 40102 0 0 0 109332 324 0 0 25 0 5 0 520506598 3200315392 703002 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4418.81
Current children cumulated vsize (KiB) 3125308

[startup+1182.3 s]
/proc/loadavg: 8.19 8.14 8.02 9/186 25487
/proc/meminfo: memFree=4084792/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=3194124 CPUtime=4658.17 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) S 25404 25406 25048 0 -1 4202496 721028 0 0 0 464917 900 0 0 25 0 5 0 520504614 3270782976 720848 33554432000 4194304 4338100 140733802659744 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/25406/statm: 798531 720848 176 36 0 796436 0
[pid=25406/tid=25409] ppid=25404 vsize=3194124 CPUtime=1161.3 cores=1,3,5,7
/proc/25406/task/25409/stat : 25409 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 181305 0 0 0 116059 71 0 0 25 0 5 0 520506598 3270782976 720848 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=25406/tid=25410] ppid=25404 vsize=3194124 CPUtime=1160.62 cores=1,3,5,7
/proc/25406/task/25410/stat : 25410 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 32658 0 0 0 115726 336 0 0 25 0 5 0 520506598 3270782976 720848 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=25406/tid=25411] ppid=25404 vsize=3194124 CPUtime=1160.01 cores=1,3,5,7
/proc/25406/task/25411/stat : 25411 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 464896 0 0 0 115842 159 0 0 25 0 5 0 520506598 3270782976 720848 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=25406/tid=25412] ppid=25404 vsize=3194124 CPUtime=1156.37 cores=1,3,5,7
/proc/25406/task/25412/stat : 25412 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 40798 0 0 0 115304 333 0 0 25 0 5 0 520506598 3270782976 720848 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4658.17
Current children cumulated vsize (KiB) 3194124



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1217.9 s]
/proc/loadavg: 8.16 8.13 8.02 9/186 25488
/proc/meminfo: memFree=3796588/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=3194124 CPUtime=4800.15 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) S 25404 25406 25048 0 -1 4202496 721028 0 0 0 479097 918 0 0 25 0 5 0 520504614 3270782976 720848 33554432000 4194304 4338100 140733802659744 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/25406/statm: 798531 720848 176 36 0 796436 0
[pid=25406/tid=25409] ppid=25404 vsize=3194124 CPUtime=1196.84 cores=1,3,5,7
/proc/25406/task/25409/stat : 25409 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 181305 0 0 0 119613 71 0 0 25 0 5 0 520506598 3270782976 720848 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=25406/tid=25410] ppid=25404 vsize=3194124 CPUtime=1196.21 cores=1,3,5,7
/proc/25406/task/25410/stat : 25410 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 32658 0 0 0 119276 345 0 0 25 0 5 0 520506598 3270782976 720848 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=25406/tid=25411] ppid=25404 vsize=3194124 CPUtime=1195.52 cores=1,3,5,7
/proc/25406/task/25411/stat : 25411 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 464896 0 0 0 119393 159 0 0 25 0 5 0 520506598 3270782976 720848 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=25406/tid=25412] ppid=25404 vsize=3194124 CPUtime=1191.73 cores=1,3,5,7
/proc/25406/task/25412/stat : 25412 (ttsth-5-4) R 25404 25406 25048 0 -1 4202560 40798 0 0 0 118832 341 0 0 25 0 5 0 520506598 3270782976 720848 33554432000 4194304 4338100 140733802659744 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.15
Current children cumulated vsize (KiB) 3194124

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1218.01 s]
/proc/loadavg: 8.16 8.13 8.02 9/186 25488
/proc/meminfo: memFree=3796588/32951124 swapFree=67111528/67111528
[pid=25406] ppid=25404 vsize=0 CPUtime=4800.28 cores=1,3,5,7
/proc/25406/stat : 25406 (ttsth-5-4) Z 25404 25406 25048 0 -1 4203548 721028 0 0 0 479100 928 0 0 18 0 2 0 520504614 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/25406/statm: 0 0 0 0 0 0 0
[pid=25406/tid=25411] ppid=25404 vsize=0 CPUtime=1195.62 cores=1,3,5,7
/proc/25406/task/25411/stat : 25411 (ttsth-5-4) R 25404 25406 25048 0 -1 4203588 464896 0 0 0 119393 169 0 0 25 0 2 0 520506598 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.28
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1218.03
CPU time (s): 4800.31
CPU user time (s): 4791
CPU system time (s): 9.30458
CPU usage (%): 394.105
Max. virtual memory (cumulated for all children) (KiB): 3194124

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4791
system time used= 9.30458
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 721028
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 9261
involuntary context switches= 21430

runsolver used 1.92671 second user time and 4.71328 second system time

The end

Launcher Data 

Begin job on node108 at 2011-04-29 16:57:17
IDJOB=3261738
IDBENCH=83239
IDSOLVER=1654
FILE ID=node108/3261738-1304089037
RUNJOBID= node108-1304086608-25067
PBS_JOBID= 13170551
Free space on /tmp= 72120 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/skvortsov/battleship/battleship-11-21-sat.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3261738-1304089037/watcher-3261738-1304089037 -o /tmp/evaluation-result-3261738-1304089037/solver-3261738-1304089037 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3261738-1304089037.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= b96c20b4b7951c39c5802e0553b42cad
RANDOM SEED=1809601752

node108.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.64
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      15462972 kB
Buffers:       2164620 kB
Cached:       12756236 kB
SwapCached:          0 kB
Active:        6123132 kB
Inactive:     10780768 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      15462972 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:              52 kB
Writeback:           0 kB
AnonPages:     1982128 kB
Mapped:          15220 kB
Slab:           517792 kB
PageTables:       8560 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  2221148 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72120 MiB
End job on node108 at 2011-04-29 17:17:37