Trace number 3261694

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.14 1202.98

General information on the benchmark

Namecrafted/skvortsov/battleship/
battleship-15-29-sat.cnf
MD5SUM89115b5a973735d51018fb3202154f32
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.021995
Satisfiable
(Un)Satisfiability was proved
Number of variables435
Number of clauses3270
Sum of the clauses size12615
Maximum clause length29
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 23045
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 5225

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3261694-1304088804.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 435
0.00/0.00	c Clauses = 3270
0.00/0.00	c Literals = 12615
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 435
0.00/0.00	c Clauses = 3270
0.00/0.00	c Literals = 12615
0.00/0.00	c a2
0.01/0.02	c a1
1.00/1.04	c Independent sub-problems = 1
1.00/1.04	c Sub-problem with 435 variables and 3270 clauses
1.00/1.04	c using permutation method 4
1.00/1.04	c count = 2
1.00/1.04	c Ordering finished 1.040 seconds
1.00/1.04	c mapped 1.040 seconds
1.00/1.04	c permuted 1.040 seconds
1.00/1.04	c about to sort
1.00/1.04	c sorted
1.00/1.05	c clauses now 3270
1.00/1.05	c init finished 1.040 seconds
1.00/1.05	c tree built 1.050 seconds
1.00/1.05	c 
1.00/1.05	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
1.00/1.05	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
1.00/1.05	c ................................ | ..........                    |                                  1.1         0.0             
1.00/1.05	c ................................ | ..........                  1 |                                  1.1         1.0             
1.00/1.05	c ................................ | ..........                  2 |                                  1.1         1.9             
1.00/1.05	c ................................ | ..........                  3 |                                  1.1         2.9             
1.00/1.05	c ................................ | ..........                  4 |                                  1.1         3.8             
1.00/1.05	c ................................ | ..........                  5 |                                  1.1         4.8             
1.00/1.05	c ................................ | ..........               1000 | 133          27                  1.1       952.4         1808
95.87/24.89	c ................................ | ..........             585000 | 162928       5776               96.3      6077.3      8590520
192.13/48.90	c ................................ | ..........             762000 | 212477       5900              192.2      3965.4     11096391
287.99/72.94	c ................................ | ..........             901000 | 250352       8614              288.1      3126.8     12685251
384.19/97.00	c ................................ | ..........            1098000 | 308627       10339             384.2      2857.8     15624046
479.67/121.00	c ................................ | *.........            1325000 | 375988       10357             480.0      2760.2     18582808
576.73/145.23	c ................................ | *.........            1470000 | 416810       10720             576.9      2548.3     20208649
672.16/169.13	c ................................ | *.........            1641000 | 469232       10825             672.3      2440.9     23299982
768.39/193.22	c ................................ | *.........            1784000 | 515450       10825             768.5      2321.5     24641979
864.64/217.34	c ................................ | *.........            1912000 | 555825       10828             864.8      2210.8     24641979
960.09/241.25	c ................................ | **........            2008000 | 584874       10973             960.3      2091.0     25360279
1055.93/265.26	c ................................ | **........            2130000 | 619844       12537            1056.2      2016.7     25750967
1152.14/289.39	c ................................ | **........            2298000 | 670815       12800            1152.5      1993.9     25918032
1247.97/313.39	c ................................ | **........            2432000 | 708802       13287            1248.4      1948.1     26677452
1344.23/337.47	c ................................ | **........            2531000 | 736989       14359            1344.5      1882.4     27519164
1440.09/361.40	c ................................ | ***.......            2665000 | 778266       14745            1440.1      1850.6     29107536
1535.98/385.46	c ................................ | ***.......            2782000 | 814552       15165            1536.2      1811.0     31010073
1632.24/409.53	c ................................ | ***.......            2917000 | 855592       15165            1632.4      1787.0     33431375
1728.06/433.58	c ................................ | ***.......            3016000 | 885410       15271            1728.4      1745.0     33775771
1824.35/457.62	c ................................ | ***.......            3116000 | 914834       15469            1824.4      1707.9     34881024
1920.17/481.70	c ................................ | ****......            3191000 | 936988       15515            1920.6      1661.5     35262691
2017.18/505.92	c ................................ | ****......            3261000 | 957089       15737            2017.3      1616.5     35303875
2112.63/529.80	c ................................ | ****......            3372000 | 989994       16826            2112.6      1596.1     35303875
2208.48/553.86	c ................................ | ****......            3497000 | 1028768      16826            2208.7      1583.3     35689442
2304.73/577.95	c ................................ | ****......            3576000 | 1053096      16826            2304.9      1551.5     36297211
2400.99/602.09	c ................................ | *****.....            3661000 | 1077725      16898            2401.4      1524.6     36697478
2497.67/626.40	c ................................ | *****.....            3720000 | 1095361      16919            2497.7      1489.4     36957063
2593.84/650.51	c ................................ | *****.....            3780000 | 1112893      17620            2593.9      1457.3     36998525
2688.52/674.22	c ................................ | *****.....            3840000 | 1131956      17841            2688.6      1428.3     36998525
2785.92/698.63	c ................................ | *****.....            3916000 | 1156550      17841            2786.0      1405.6     36998525
2881.39/722.57	c ................................ | ******....            3963000 | 1171676      17903            2881.7      1375.2     36998525
2976.46/746.37	c ................................ | ******....            4015000 | 1188042      18133            2976.7      1348.8     36998525
3072.33/770.33	c ................................ | ******....            4106000 | 1216684      18136            3072.5      1336.4     37893084
3167.74/794.27	c ................................ | ******....            4229000 | 1254451      18136            3168.0      1334.9     37948400
3264.76/818.51	c ................................ | ******....            4294000 | 1274877      18162            3264.8      1315.2     37948400
3361.05/842.61	c ................................ | *******...            4377000 | 1299849      18291            3361.1      1302.3     37948400
3456.50/866.54	c ................................ | *******...            4433000 | 1317388      18362            3456.7      1282.4     37948400
3552.76/890.63	c ................................ | *******...            4472000 | 1329540      18469            3552.9      1258.7     37948400
3648.21/914.50	c ................................ | *******...            4601000 | 1369525      18487            3648.2      1261.2     39281722
3744.08/938.57	c ................................ | *******...            4713000 | 1403975      18487            3744.4      1258.7     40548559
3839.96/962.55	c ................................ | ********..            4798000 | 1429284      18547            3840.1      1249.4     41118687
3937.37/986.94	c ................................ | ********..            4852000 | 1446213      18547            3937.6      1232.2     41118687
4032.04/1010.65	c ................................ | ********..            4940000 | 1474282      18547            4032.2      1225.1     41558362
4128.27/1034.71	c ................................ | ********..            5092000 | 1520877      18563            4128.3      1233.4     43340995
4224.53/1058.81	c ................................ | ********..            5193000 | 1551669      18588            4224.6      1229.2     44351473
4320.00/1082.71	c ................................ | *********.            5292000 | 1578826      18819            4320.1      1225.0     45112050
4416.28/1106.82	c ................................ | *********.            5360000 | 1599024      19088            4416.4      1213.7     45674521
4512.57/1131.00	c ................................ | *********.            5413000 | 1615317      19189            4513.0      1199.4     46091427
4608.46/1154.93	c ................................ | *********.            5479000 | 1635421      19223            4608.6      1188.9     46977464
4703.90/1178.90	c ................................ | *********.            5590000 | 1670072      19223            4704.3      1188.3     48506093

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3261694-1304088804/watcher-3261694-1304088804 -o /tmp/evaluation-result-3261694-1304088804/solver-3261694-1304088804 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3261694-1304088804.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.57 7.96 7.97 6/181 24413
/proc/meminfo: memFree=25134880/32951124 swapFree=67051468/67111528
[pid=24413] ppid=24411 vsize=9904 CPUtime=0 cores=0,2,4,6
/proc/24413/stat : 24413 (ttsth-5-4) R 24411 24413 24182 0 -1 4202496 534 0 0 0 0 0 0 0 25 0 1 0 520483152 10141696 477 33554432000 4194304 4338100 140735199350864 18446744073709551615 218370618053 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/24413/statm: 2476 485 150 36 0 381 0

[startup+0.0154101 s]
/proc/loadavg: 7.57 7.96 7.97 6/181 24413
/proc/meminfo: memFree=25134880/32951124 swapFree=67051468/67111528
[pid=24413] ppid=24411 vsize=11092 CPUtime=0.01 cores=0,2,4,6
/proc/24413/stat : 24413 (ttsth-5-4) R 24411 24413 24182 0 -1 4202496 843 0 0 0 1 0 0 0 25 0 1 0 520483152 11358208 785 33554432000 4194304 4338100 140735199350864 18446744073709551615 218370352358 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/24413/statm: 2773 785 152 36 0 678 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 11092

[startup+0.100392 s]
/proc/loadavg: 7.57 7.96 7.97 6/181 24413
/proc/meminfo: memFree=25134880/32951124 swapFree=67051468/67111528
[pid=24413] ppid=24411 vsize=11092 CPUtime=0.09 cores=0,2,4,6
/proc/24413/stat : 24413 (ttsth-5-4) R 24411 24413 24182 0 -1 4202496 858 0 0 0 9 0 0 0 25 0 1 0 520483152 11358208 800 33554432000 4194304 4338100 140735199350864 18446744073709551615 4295973 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/24413/statm: 2773 800 155 36 0 678 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 11092

[startup+0.30036 s]
/proc/loadavg: 7.57 7.96 7.97 6/181 24413
/proc/meminfo: memFree=25134880/32951124 swapFree=67051468/67111528
[pid=24413] ppid=24411 vsize=11092 CPUtime=0.29 cores=0,2,4,6
/proc/24413/stat : 24413 (ttsth-5-4) R 24411 24413 24182 0 -1 4202496 858 0 0 0 29 0 0 0 25 0 1 0 520483152 11358208 800 33554432000 4194304 4338100 140735199350864 18446744073709551615 4296000 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/24413/statm: 2773 800 155 36 0 678 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 11092

[startup+0.700296 s]
/proc/loadavg: 7.57 7.96 7.97 6/181 24413
/proc/meminfo: memFree=25134880/32951124 swapFree=67051468/67111528
[pid=24413] ppid=24411 vsize=11092 CPUtime=0.69 cores=0,2,4,6
/proc/24413/stat : 24413 (ttsth-5-4) R 24411 24413 24182 0 -1 4202496 858 0 0 0 69 0 0 0 25 0 1 0 520483152 11358208 800 33554432000 4194304 4338100 140735199350864 18446744073709551615 4296000 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/24413/statm: 2773 800 155 36 0 678 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 11092

[startup+1.50011 s]
/proc/loadavg: 7.57 7.96 7.97 6/182 24414
/proc/meminfo: memFree=25113700/32951124 swapFree=67051468/67111528
[pid=24413] ppid=24411 vsize=273228 CPUtime=2.82 cores=0,2,4,6
/proc/24413/stat : 24413 (ttsth-5-4) S 24411 24413 24182 0 -1 4202496 5574 0 0 0 280 2 0 0 25 0 5 0 520483152 279785472 5422 33554432000 4194304 4338100 140735199350864 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/24413/statm: 68307 5422 175 36 0 66212 0
Current children cumulated CPU time (s) 2.82
Current children cumulated vsize (KiB) 273228

[startup+3.10083 s]
/proc/loadavg: 7.60 7.96 7.97 9/186 24418
/proc/meminfo: memFree=25066300/32951124 swapFree=67051468/67111528
[pid=24413] ppid=24411 vsize=310896 CPUtime=9.21 cores=0,2,4,6
/proc/24413/stat : 24413 (ttsth-5-4) S 24411 24413 24182 0 -1 4202496 14692 0 0 0 914 7 0 0 25 0 5 0 520483152 318357504 14540 33554432000 4194304 4338100 140735199350864 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/24413/statm: 77724 14540 175 36 0 75629 0
[pid=24413/tid=24415] ppid=24411 vsize=310896 CPUtime=2.04 cores=0,2,4,6
/proc/24413/task/24415/stat : 24415 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 4806 0 0 0 203 1 0 0 21 0 5 0 520483257 318357504 14540 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=24413/tid=24416] ppid=24411 vsize=310896 CPUtime=2.03 cores=0,2,4,6
/proc/24413/task/24416/stat : 24416 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 1592 0 0 0 201 2 0 0 21 0 5 0 520483257 318357504 14540 33554432000 4194304 4338100 140735199350864 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24413/tid=24417] ppid=24411 vsize=310896 CPUtime=2.03 cores=0,2,4,6
/proc/24413/task/24417/stat : 24417 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 4042 0 0 0 201 2 0 0 25 0 5 0 520483257 318357504 14541 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=24413/tid=24418] ppid=24411 vsize=310896 CPUtime=2.04 cores=0,2,4,6
/proc/24413/task/24418/stat : 24418 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 1962 0 0 0 202 2 0 0 21 0 5 0 520483257 318357504 14541 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 9.21
Current children cumulated vsize (KiB) 310896

[startup+6.30026 s]
/proc/loadavg: 7.60 7.96 7.97 9/186 24418
/proc/meminfo: memFree=24955204/32951124 swapFree=67051468/67111528
[pid=24413] ppid=24411 vsize=376848 CPUtime=21.99 cores=0,2,4,6
/proc/24413/stat : 24413 (ttsth-5-4) S 24411 24413 24182 0 -1 4202496 32170 0 0 0 2176 23 0 0 25 0 5 0 520483152 385892352 32018 33554432000 4194304 4338100 140735199350864 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/24413/statm: 94212 32019 175 36 0 92117 0
[pid=24413/tid=24415] ppid=24411 vsize=376848 CPUtime=5.24 cores=0,2,4,6
/proc/24413/task/24415/stat : 24415 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 11545 0 0 0 522 2 0 0 25 0 5 0 520483257 385892352 32019 33554432000 4194304 4338100 140735199350864 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=24413/tid=24416] ppid=24411 vsize=376848 CPUtime=5.23 cores=0,2,4,6
/proc/24413/task/24416/stat : 24416 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 4652 0 0 0 516 7 0 0 25 0 5 0 520483257 385892352 32019 33554432000 4194304 4338100 140735199350864 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24413/tid=24417] ppid=24411 vsize=376848 CPUtime=5.21 cores=0,2,4,6
/proc/24413/task/24417/stat : 24417 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 8451 0 0 0 515 6 0 0 25 0 5 0 520483257 385892352 32019 33554432000 4194304 4338100 140735199350864 18446744073709551615 218370352534 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=24413/tid=24418] ppid=24411 vsize=376848 CPUtime=5.23 cores=0,2,4,6
/proc/24413/task/24418/stat : 24418 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 5233 0 0 0 516 7 0 0 25 0 5 0 520483257 385892352 32019 33554432000 4194304 4338100 140735199350864 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 21.99
Current children cumulated vsize (KiB) 376848

[startup+12.7011 s]
/proc/loadavg: 7.66 7.96 7.97 9/186 24418
/proc/meminfo: memFree=24834316/32951124 swapFree=67051468/67111528
[pid=24413] ppid=24411 vsize=451920 CPUtime=47.56 cores=0,2,4,6
/proc/24413/stat : 24413 (ttsth-5-4) S 24411 24413 24182 0 -1 4202496 50234 0 0 0 4712 44 0 0 25 0 5 0 520483152 462766080 50082 33554432000 4194304 4338100 140735199350864 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/24413/statm: 112980 50082 175 36 0 110885 0
[pid=24413/tid=24415] ppid=24411 vsize=451920 CPUtime=11.64 cores=0,2,4,6
/proc/24413/task/24415/stat : 24415 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 18883 0 0 0 1160 4 0 0 25 0 5 0 520483257 462766080 50082 33554432000 4194304 4338100 140735199350864 18446744073709551615 218370955323 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=24413/tid=24416] ppid=24411 vsize=451920 CPUtime=11.61 cores=0,2,4,6
/proc/24413/task/24416/stat : 24416 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 7895 0 0 0 1147 14 0 0 25 0 5 0 520483257 462766080 50082 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=24413/tid=24417] ppid=24411 vsize=451920 CPUtime=11.6 cores=0,2,4,6
/proc/24413/task/24417/stat : 24417 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 12428 0 0 0 1148 12 0 0 25 0 5 0 520483257 462766080 50082 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24413/tid=24418] ppid=24411 vsize=451920 CPUtime=11.63 cores=0,2,4,6
/proc/24413/task/24418/stat : 24418 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 8738 0 0 0 1150 13 0 0 25 0 5 0 520483257 462766080 50082 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 47.56
Current children cumulated vsize (KiB) 451920

[startup+25.5008 s]
/proc/loadavg: 7.71 7.96 7.97 9/186 24419
/proc/meminfo: memFree=24654896/32951124 swapFree=67051468/67111528
[pid=24413] ppid=24411 vsize=536088 CPUtime=98.67 cores=0,2,4,6
/proc/24413/stat : 24413 (ttsth-5-4) S 24411 24413 24182 0 -1 4202496 71404 0 0 0 9793 74 0 0 25 0 5 0 520483152 548954112 71252 33554432000 4194304 4338100 140735199350864 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/24413/statm: 134022 71252 176 36 0 131927 0
[pid=24413/tid=24415] ppid=24411 vsize=536088 CPUtime=24.43 cores=0,2,4,6
/proc/24413/task/24415/stat : 24415 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 28518 0 0 0 2437 6 0 0 25 0 5 0 520483257 548954112 71252 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=24413/tid=24416] ppid=24411 vsize=536088 CPUtime=24.33 cores=0,2,4,6
/proc/24413/task/24416/stat : 24416 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 10344 0 0 0 2410 23 0 0 25 0 5 0 520483257 548954112 71252 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=24413/tid=24417] ppid=24411 vsize=536088 CPUtime=24.4 cores=0,2,4,6
/proc/24413/task/24417/stat : 24417 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 17271 0 0 0 2420 20 0 0 25 0 5 0 520483257 548954112 71252 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24413/tid=24418] ppid=24411 vsize=536088 CPUtime=24.42 cores=0,2,4,6
/proc/24413/task/24418/stat : 24418 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 12981 0 0 0 2419 23 0 0 25 0 5 0 520483257 548954112 71252 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 98.67
Current children cumulated vsize (KiB) 536088

[startup+51.1003 s]
/proc/loadavg: 7.81 7.96 7.97 9/186 24420
/proc/meminfo: memFree=24473068/32951124 swapFree=67051468/67111528
[pid=24413] ppid=24411 vsize=617416 CPUtime=200.91 cores=0,2,4,6
/proc/24413/stat : 24413 (ttsth-5-4) S 24411 24413 24182 0 -1 4202496 91256 0 0 0 19974 117 0 0 25 0 5 0 520483152 632233984 91104 33554432000 4194304 4338100 140735199350864 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/24413/statm: 154354 91104 176 36 0 152259 0
[pid=24413/tid=24415] ppid=24411 vsize=617416 CPUtime=50.02 cores=0,2,4,6
/proc/24413/task/24415/stat : 24415 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 36250 0 0 0 4991 11 0 0 25 0 5 0 520483257 632233984 91104 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=24413/tid=24416] ppid=24411 vsize=617416 CPUtime=49.8 cores=0,2,4,6
/proc/24413/task/24416/stat : 24416 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 14130 0 0 0 4942 38 0 0 20 0 5 0 520483257 632233984 91104 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=24413/tid=24417] ppid=24411 vsize=617416 CPUtime=49.99 cores=0,2,4,6
/proc/24413/task/24417/stat : 24417 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 21582 0 0 0 4968 31 0 0 25 0 5 0 520483257 632233984 91104 33554432000 4194304 4338100 140735199350864 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24413/tid=24418] ppid=24411 vsize=617416 CPUtime=50.02 cores=0,2,4,6
/proc/24413/task/24418/stat : 24418 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 17004 0 0 0 4966 36 0 0 25 0 5 0 520483257 632233984 91104 33554432000 4194304 4338100 140735199350864 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 200.91
Current children cumulated vsize (KiB) 617416

[startup+102.306 s]
/proc/loadavg: 7.96 7.98 7.98 9/186 24421
/proc/meminfo: memFree=24068204/32951124 swapFree=67051468/67111528
[pid=24413] ppid=24411 vsize=790200 CPUtime=405.38 cores=0,2,4,6
/proc/24413/stat : 24413 (ttsth-5-4) S 24411 24413 24182 0 -1 4202496 133748 0 0 0 40344 194 0 0 25 0 5 0 520483152 809164800 133596 33554432000 4194304 4338100 140735199350864 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/24413/statm: 197550 133596 176 36 0 195455 0
[pid=24413/tid=24415] ppid=24411 vsize=790200 CPUtime=101.21 cores=0,2,4,6
/proc/24413/task/24415/stat : 24415 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 53778 0 0 0 10102 19 0 0 25 0 5 0 520483257 809164800 133596 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=24413/tid=24416] ppid=24411 vsize=790200 CPUtime=100.71 cores=0,2,4,6
/proc/24413/task/24416/stat : 24416 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 21072 0 0 0 10009 62 0 0 25 0 5 0 520483257 809164800 133596 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=24413/tid=24417] ppid=24411 vsize=790200 CPUtime=101.17 cores=0,2,4,6
/proc/24413/task/24417/stat : 24417 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 31564 0 0 0 10065 52 0 0 25 0 5 0 520483257 809164800 133596 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24413/tid=24418] ppid=24411 vsize=790200 CPUtime=101.19 cores=0,2,4,6
/proc/24413/task/24418/stat : 24418 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 25044 0 0 0 10061 58 0 0 25 0 5 0 520483257 809164800 133596 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 405.38
Current children cumulated vsize (KiB) 790200

[startup+162.3 s]
/proc/loadavg: 8.07 8.01 7.99 9/186 24422
/proc/meminfo: memFree=23798736/32951124 swapFree=67051468/67111528
[pid=24413] ppid=24411 vsize=968568 CPUtime=645.01 cores=0,2,4,6
/proc/24413/stat : 24413 (ttsth-5-4) S 24411 24413 24182 0 -1 4202496 178466 0 0 0 64222 279 0 0 25 0 5 0 520483152 991813632 178314 33554432000 4194304 4338100 140735199350864 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/24413/statm: 242142 178314 176 36 0 240047 0
[pid=24413/tid=24415] ppid=24411 vsize=968568 CPUtime=161.2 cores=0,2,4,6
/proc/24413/task/24415/stat : 24415 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 73228 0 0 0 16095 25 0 0 25 0 5 0 520483257 991813632 178314 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=24413/tid=24416] ppid=24411 vsize=968568 CPUtime=160.39 cores=0,2,4,6
/proc/24413/task/24416/stat : 24416 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 29503 0 0 0 15948 91 0 0 25 0 5 0 520483257 991813632 178314 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=24413/tid=24417] ppid=24411 vsize=968568 CPUtime=161.16 cores=0,2,4,6
/proc/24413/task/24417/stat : 24417 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 40506 0 0 0 16042 74 0 0 25 0 5 0 520483257 991813632 178314 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24413/tid=24418] ppid=24411 vsize=968568 CPUtime=161.18 cores=0,2,4,6
/proc/24413/task/24418/stat : 24418 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 32939 0 0 0 16030 88 0 0 25 0 5 0 520483257 991813632 178314 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 645.01
Current children cumulated vsize (KiB) 968568

[startup+222.301 s]
/proc/loadavg: 8.07 8.02 7.99 9/186 24424
/proc/meminfo: memFree=23472484/32951124 swapFree=67051468/67111528
[pid=24413] ppid=24411 vsize=1098536 CPUtime=884.59 cores=0,2,4,6
/proc/24413/stat : 24413 (ttsth-5-4) S 24411 24413 24182 0 -1 4202496 209970 0 0 0 88109 350 0 0 25 0 5 0 520483152 1124900864 209818 33554432000 4194304 4338100 140735199350864 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/24413/statm: 274634 209818 176 36 0 272539 0
[pid=24413/tid=24415] ppid=24411 vsize=1098536 CPUtime=221.19 cores=0,2,4,6
/proc/24413/task/24415/stat : 24415 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 85857 0 0 0 22090 29 0 0 25 0 5 0 520483257 1124900864 209818 33554432000 4194304 4338100 140735199350864 18446744073709551615 4263505 0 0 4096 0 0 0 0 -1 0 0 0 0

################
# More data... #
################

[pid=24413/tid=24416] ppid=24411 vsize=1524240 CPUtime=759.64 cores=0,2,4,6
/proc/24413/task/24416/stat : 24416 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 37759 0 0 0 75727 237 0 0 25 0 5 0 520483257 1560821760 309995 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24413/tid=24417] ppid=24411 vsize=1524240 CPUtime=760.24 cores=0,2,4,6
/proc/24413/task/24417/stat : 24417 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 64705 0 0 0 75854 170 0 0 25 0 5 0 520483257 1560821760 309995 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=24413/tid=24418] ppid=24411 vsize=1524240 CPUtime=758.57 cores=0,2,4,6
/proc/24413/task/24418/stat : 24418 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 54500 0 0 0 75620 237 0 0 25 0 5 0 520483257 1560821760 309995 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3040.39
Current children cumulated vsize (KiB) 1524240

[startup+822.301 s]
/proc/loadavg: 8.15 8.10 8.02 9/186 24452
/proc/meminfo: memFree=21887108/32951124 swapFree=67051468/67111528
[pid=24413] ppid=24411 vsize=1552392 CPUtime=3279.94 cores=0,2,4,6
/proc/24413/stat : 24413 (ttsth-5-4) S 24411 24413 24182 0 -1 4202496 319679 0 0 0 327272 722 0 0 25 0 5 0 520483152 1589649408 319527 33554432000 4194304 4338100 140735199350864 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/24413/statm: 388098 319527 176 36 0 386003 0
[pid=24413/tid=24415] ppid=24411 vsize=1552392 CPUtime=820.82 cores=0,2,4,6
/proc/24413/task/24415/stat : 24415 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 154095 0 0 0 82022 60 0 0 25 0 5 0 520483257 1589649408 319527 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=24413/tid=24416] ppid=24411 vsize=1552392 CPUtime=819.61 cores=0,2,4,6
/proc/24413/task/24416/stat : 24416 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 39361 0 0 0 81717 244 0 0 25 0 5 0 520483257 1589649408 319527 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24413/tid=24417] ppid=24411 vsize=1552392 CPUtime=820.22 cores=0,2,4,6
/proc/24413/task/24417/stat : 24417 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 68765 0 0 0 81851 171 0 0 25 0 5 0 520483257 1589649408 319527 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=24413/tid=24418] ppid=24411 vsize=1552392 CPUtime=818.2 cores=0,2,4,6
/proc/24413/task/24418/stat : 24418 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 55168 0 0 0 81575 245 0 0 25 0 5 0 520483257 1589649408 319527 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3279.94
Current children cumulated vsize (KiB) 1552392

[startup+882.301 s]
/proc/loadavg: 8.15 8.11 8.03 9/186 24454
/proc/meminfo: memFree=21757508/32951124 swapFree=67051468/67111528
[pid=24413] ppid=24411 vsize=1552392 CPUtime=3519.62 cores=0,2,4,6
/proc/24413/stat : 24413 (ttsth-5-4) S 24411 24413 24182 0 -1 4202496 319679 0 0 0 351227 735 0 0 25 0 5 0 520483152 1589649408 319527 33554432000 4194304 4338100 140735199350864 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/24413/statm: 388098 319527 176 36 0 386003 0
[pid=24413/tid=24415] ppid=24411 vsize=1552392 CPUtime=880.81 cores=0,2,4,6
/proc/24413/task/24415/stat : 24415 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 154095 0 0 0 88021 60 0 0 25 0 5 0 520483257 1589649408 319527 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=24413/tid=24416] ppid=24411 vsize=1552392 CPUtime=879.62 cores=0,2,4,6
/proc/24413/task/24416/stat : 24416 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 39361 0 0 0 87711 251 0 0 25 0 5 0 520483257 1589649408 319527 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24413/tid=24417] ppid=24411 vsize=1552392 CPUtime=880.23 cores=0,2,4,6
/proc/24413/task/24417/stat : 24417 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 68765 0 0 0 87852 171 0 0 25 0 5 0 520483257 1589649408 319527 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=24413/tid=24418] ppid=24411 vsize=1552392 CPUtime=877.87 cores=0,2,4,6
/proc/24413/task/24418/stat : 24418 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 55168 0 0 0 87536 251 0 0 25 0 5 0 520483257 1589649408 319527 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3519.62
Current children cumulated vsize (KiB) 1552392

[startup+942.3 s]
/proc/loadavg: 8.14 8.12 8.03 9/186 24455
/proc/meminfo: memFree=21585880/32951124 swapFree=67051468/67111528
[pid=24413] ppid=24411 vsize=1636848 CPUtime=3759.25 cores=0,2,4,6
/proc/24413/stat : 24413 (ttsth-5-4) S 24411 24413 24182 0 -1 4202496 341109 0 0 0 375158 767 0 0 25 0 5 0 520483152 1676132352 340957 33554432000 4194304 4338100 140735199350864 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/24413/statm: 409212 340957 176 36 0 407117 0
[pid=24413/tid=24415] ppid=24411 vsize=1636848 CPUtime=940.8 cores=0,2,4,6
/proc/24413/task/24415/stat : 24415 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 166091 0 0 0 94015 65 0 0 25 0 5 0 520483257 1676132352 340957 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=24413/tid=24416] ppid=24411 vsize=1636848 CPUtime=939.61 cores=0,2,4,6
/proc/24413/task/24416/stat : 24416 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 39361 0 0 0 93700 261 0 0 25 0 5 0 520483257 1676132352 340957 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24413/tid=24417] ppid=24411 vsize=1636848 CPUtime=940.23 cores=0,2,4,6
/proc/24413/task/24417/stat : 24417 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 78199 0 0 0 93844 179 0 0 25 0 5 0 520483257 1676132352 340957 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=24413/tid=24418] ppid=24411 vsize=1636848 CPUtime=937.54 cores=0,2,4,6
/proc/24413/task/24418/stat : 24418 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 55168 0 0 0 93493 261 0 0 25 0 5 0 520483257 1676132352 340957 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3759.25
Current children cumulated vsize (KiB) 1636848

[startup+1002.3 s]
/proc/loadavg: 8.16 8.12 8.04 9/186 24457
/proc/meminfo: memFree=21460376/32951124 swapFree=67051468/67111528
[pid=24413] ppid=24411 vsize=1649360 CPUtime=3998.89 cores=0,2,4,6
/proc/24413/stat : 24413 (ttsth-5-4) S 24411 24413 24182 0 -1 4202496 344476 0 0 0 399085 804 0 0 25 0 5 0 520483152 1688944640 344324 33554432000 4194304 4338100 140735199350864 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/24413/statm: 412340 344324 176 36 0 410245 0
[pid=24413/tid=24415] ppid=24411 vsize=1649360 CPUtime=1000.77 cores=0,2,4,6
/proc/24413/task/24415/stat : 24415 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 168455 0 0 0 100012 65 0 0 25 0 5 0 520483257 1688944640 344324 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=24413/tid=24416] ppid=24411 vsize=1649360 CPUtime=999.61 cores=0,2,4,6
/proc/24413/task/24416/stat : 24416 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 39361 0 0 0 99686 275 0 0 25 0 5 0 520483257 1688944640 344324 33554432000 4194304 4338100 140735199350864 18446744073709551615 4263768 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24413/tid=24417] ppid=24411 vsize=1649360 CPUtime=1000.22 cores=0,2,4,6
/proc/24413/task/24417/stat : 24417 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 79202 0 0 0 99834 188 0 0 25 0 5 0 520483257 1688944640 344324 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=24413/tid=24418] ppid=24411 vsize=1649360 CPUtime=997.21 cores=0,2,4,6
/proc/24413/task/24418/stat : 24418 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 55168 0 0 0 99446 275 0 0 25 0 5 0 520483257 1688944640 344324 33554432000 4194304 4338100 140735199350864 18446744073709551615 4279576 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3998.89
Current children cumulated vsize (KiB) 1649360

[startup+1062.3 s]
/proc/loadavg: 8.18 8.13 8.04 9/186 24458
/proc/meminfo: memFree=21257864/32951124 swapFree=67051468/67111528
[pid=24413] ppid=24411 vsize=1752584 CPUtime=4238.5 cores=0,2,4,6
/proc/24413/stat : 24413 (ttsth-5-4) S 24411 24413 24182 0 -1 4202496 370517 0 0 0 422997 853 0 0 25 0 5 0 520483152 1794646016 370365 33554432000 4194304 4338100 140735199350864 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/24413/statm: 438146 370365 176 36 0 436051 0
[pid=24413/tid=24415] ppid=24411 vsize=1752584 CPUtime=1060.72 cores=0,2,4,6
/proc/24413/task/24415/stat : 24415 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 184166 0 0 0 106001 71 0 0 25 0 5 0 520483257 1794646016 370365 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=24413/tid=24416] ppid=24411 vsize=1752584 CPUtime=1059.6 cores=0,2,4,6
/proc/24413/task/24416/stat : 24416 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 39361 0 0 0 105670 290 0 0 25 0 5 0 520483257 1794646016 370365 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254304 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24413/tid=24417] ppid=24411 vsize=1752584 CPUtime=1060.2 cores=0,2,4,6
/proc/24413/task/24417/stat : 24417 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 89532 0 0 0 105820 200 0 0 25 0 5 0 520483257 1794646016 370365 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=24413/tid=24418] ppid=24411 vsize=1752584 CPUtime=1056.91 cores=0,2,4,6
/proc/24413/task/24418/stat : 24418 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 55168 0 0 0 105400 291 0 0 25 0 5 0 520483257 1794646016 370365 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4238.5
Current children cumulated vsize (KiB) 1752584

[startup+1122.3 s]
/proc/loadavg: 8.13 8.12 8.04 9/186 24460
/proc/meminfo: memFree=21098888/32951124 swapFree=67051468/67111528
[pid=24413] ppid=24411 vsize=1802632 CPUtime=4478.21 cores=0,2,4,6
/proc/24413/stat : 24413 (ttsth-5-4) S 24411 24413 24182 0 -1 4202496 382141 0 0 0 446938 883 0 0 25 0 5 0 520483152 1845895168 381989 33554432000 4194304 4338100 140735199350864 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/24413/statm: 450658 381989 176 36 0 448563 0
[pid=24413/tid=24415] ppid=24411 vsize=1802632 CPUtime=1120.71 cores=0,2,4,6
/proc/24413/task/24415/stat : 24415 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 192919 0 0 0 111997 74 0 0 25 0 5 0 520483257 1845895168 381989 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=24413/tid=24416] ppid=24411 vsize=1802632 CPUtime=1119.6 cores=0,2,4,6
/proc/24413/task/24416/stat : 24416 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 39361 0 0 0 111661 299 0 0 25 0 5 0 520483257 1845895168 381989 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24413/tid=24417] ppid=24411 vsize=1802632 CPUtime=1120.19 cores=0,2,4,6
/proc/24413/task/24417/stat : 24417 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 92402 0 0 0 111811 208 0 0 25 0 5 0 520483257 1845895168 381989 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=24413/tid=24418] ppid=24411 vsize=1802632 CPUtime=1116.62 cores=0,2,4,6
/proc/24413/task/24418/stat : 24418 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 55169 0 0 0 111362 300 0 0 25 0 5 0 520483257 1845895168 381989 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4478.21
Current children cumulated vsize (KiB) 1802632

[startup+1182.3 s]
/proc/loadavg: 8.05 8.10 8.04 9/186 24462
/proc/meminfo: memFree=20916348/32951124 swapFree=67051468/67111528
[pid=24413] ppid=24411 vsize=1890216 CPUtime=4717.88 cores=0,2,4,6
/proc/24413/stat : 24413 (ttsth-5-4) S 24411 24413 24182 0 -1 4202496 403710 0 0 0 470865 923 0 0 25 0 5 0 520483152 1935581184 403558 33554432000 4194304 4338100 140735199350864 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/24413/statm: 472554 403558 176 36 0 470459 0
[pid=24413/tid=24415] ppid=24411 vsize=1890216 CPUtime=1180.68 cores=0,2,4,6
/proc/24413/task/24415/stat : 24415 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 206234 0 0 0 117990 78 0 0 25 0 5 0 520483257 1935581184 403558 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=24413/tid=24416] ppid=24411 vsize=1890216 CPUtime=1179.6 cores=0,2,4,6
/proc/24413/task/24416/stat : 24416 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 39363 0 0 0 117649 311 0 0 25 0 5 0 520483257 1935581184 403558 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24413/tid=24417] ppid=24411 vsize=1890216 CPUtime=1180.19 cores=0,2,4,6
/proc/24413/task/24417/stat : 24417 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 99449 0 0 0 117801 218 0 0 25 0 5 0 520483257 1935581184 403558 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=24413/tid=24418] ppid=24411 vsize=1890216 CPUtime=1176.31 cores=0,2,4,6
/proc/24413/task/24418/stat : 24418 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 56374 0 0 0 117318 313 0 0 25 0 5 0 520483257 1935581184 403558 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4717.88
Current children cumulated vsize (KiB) 1890216



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.9 s]
/proc/loadavg: 8.11 8.10 8.04 9/186 24494
/proc/meminfo: memFree=23330792/32951124 swapFree=67051468/67111528
[pid=24413] ppid=24411 vsize=1893344 CPUtime=4800.04 cores=0,2,4,6
/proc/24413/stat : 24413 (ttsth-5-4) S 24411 24413 24182 0 -1 4202496 405511 0 0 0 479069 935 0 0 25 0 5 0 520483152 1938784256 405359 33554432000 4194304 4338100 140735199350864 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/24413/statm: 473336 405359 176 36 0 471241 0
[pid=24413/tid=24415] ppid=24411 vsize=1893344 CPUtime=1201.24 cores=0,2,4,6
/proc/24413/task/24415/stat : 24415 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 207580 0 0 0 120045 79 0 0 25 0 5 0 520483257 1938784256 405359 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=24413/tid=24416] ppid=24411 vsize=1893344 CPUtime=1200.16 cores=0,2,4,6
/proc/24413/task/24416/stat : 24416 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 39363 0 0 0 119700 316 0 0 25 0 5 0 520483257 1938784256 405359 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24413/tid=24417] ppid=24411 vsize=1893344 CPUtime=1200.8 cores=0,2,4,6
/proc/24413/task/24417/stat : 24417 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 99597 0 0 0 119859 221 0 0 25 0 5 0 520483257 1938784256 405359 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254347 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=24413/tid=24418] ppid=24411 vsize=1893344 CPUtime=1196.77 cores=0,2,4,6
/proc/24413/task/24418/stat : 24418 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 56681 0 0 0 119359 318 0 0 25 0 5 0 520483257 1938784256 405359 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4800.04
Current children cumulated vsize (KiB) 1893344

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.9 s]
/proc/loadavg: 8.11 8.10 8.04 9/186 24494
/proc/meminfo: memFree=23330792/32951124 swapFree=67051468/67111528
[pid=24413] ppid=24411 vsize=1893344 CPUtime=4800.04 cores=0,2,4,6
/proc/24413/stat : 24413 (ttsth-5-4) S 24411 24413 24182 0 -1 4202496 405511 0 0 0 479069 935 0 0 25 0 5 0 520483152 1938784256 405359 33554432000 4194304 4338100 140735199350864 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/24413/statm: 473336 405359 176 36 0 471241 0
[pid=24413/tid=24415] ppid=24411 vsize=1893344 CPUtime=1201.24 cores=0,2,4,6
/proc/24413/task/24415/stat : 24415 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 207580 0 0 0 120045 79 0 0 25 0 5 0 520483257 1938784256 405359 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=24413/tid=24416] ppid=24411 vsize=1893344 CPUtime=1200.16 cores=0,2,4,6
/proc/24413/task/24416/stat : 24416 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 39363 0 0 0 119700 316 0 0 25 0 5 0 520483257 1938784256 405359 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24413/tid=24417] ppid=24411 vsize=1893344 CPUtime=1200.8 cores=0,2,4,6
/proc/24413/task/24417/stat : 24417 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 99597 0 0 0 119859 221 0 0 25 0 5 0 520483257 1938784256 405359 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254347 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=24413/tid=24418] ppid=24411 vsize=1893344 CPUtime=1196.77 cores=0,2,4,6
/proc/24413/task/24418/stat : 24418 (ttsth-5-4) R 24411 24413 24182 0 -1 4202560 56681 0 0 0 119359 318 0 0 25 0 5 0 520483257 1938784256 405359 33554432000 4194304 4338100 140735199350864 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4800.04
Current children cumulated vsize (KiB) 1893344

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.98
CPU time (s): 4800.14
CPU user time (s): 4790.72
CPU system time (s): 9.42157
CPU usage (%): 399.021
Max. virtual memory (cumulated for all children) (KiB): 1893344

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4790.72
system time used= 9.42157
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 405511
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 9127
involuntary context switches= 23049

runsolver used 1.88071 second user time and 4.39133 second system time

The end

Launcher Data 

Begin job on node101 at 2011-04-29 16:53:24
IDJOB=3261694
IDBENCH=83238
IDSOLVER=1654
FILE ID=node101/3261694-1304088804
RUNJOBID= node101-1304087588-24200
PBS_JOBID= 13170559
Free space on /tmp= 72024 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/skvortsov/battleship/battleship-15-29-sat.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3261694-1304088804/watcher-3261694-1304088804 -o /tmp/evaluation-result-3261694-1304088804/solver-3261694-1304088804 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3261694-1304088804.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 89115b5a973735d51018fb3202154f32
RANDOM SEED=123434512

node101.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      25135160 kB
Buffers:       2025280 kB
Cached:        5126100 kB
SwapCached:      30452 kB
Active:        3245564 kB
Inactive:      4126604 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      25135160 kB
SwapTotal:    67111528 kB
SwapFree:     67051468 kB
Dirty:             776 kB
Writeback:           0 kB
AnonPages:      211708 kB
Mapped:           9404 kB
Slab:           380780 kB
PageTables:       4588 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   443616 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72024 MiB
End job on node101 at 2011-04-29 17:13:29