Trace number 3274109

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.41 1205.13

General information on the benchmark

Namecrafted/spence/sgen/
sgen3-n260-s62321009-sat.cnf
MD5SUMf2b430bdd9bc1c472e9e437567706bb7
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark3.48747
Satisfiable
(Un)Satisfiability was proved
Number of variables260
Number of clauses624
Sum of the clauses size1560
Maximum clause length5
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2520
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 5104
Number of clauses of size over 50

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3274109-1304059873.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 260
0.00/0.00	c Clauses = 624
0.00/0.00	c Literals = 1560
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 260
0.00/0.00	c Clauses = 624
0.00/0.00	c Literals = 1560
0.00/0.00	c a2
0.00/0.00	c a1
3.20/3.28	c Independent sub-problems = 1
3.20/3.28	c Sub-problem with 260 variables and 624 clauses
3.20/3.28	c using permutation method 3
3.20/3.28	c Ordering finished 3.280 seconds
3.20/3.28	c mapped 3.280 seconds
3.20/3.28	c permuted 3.280 seconds
3.20/3.28	c about to sort
3.20/3.28	c sorted
3.20/3.29	c clauses now 624
3.20/3.29	c init finished 3.280 seconds
3.20/3.29	c tree built 3.280 seconds
3.20/3.29	c 
3.20/3.29	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
3.20/3.29	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
3.20/3.29	c ................................ | ..........                    |                                  3.3         0.0             
3.20/3.29	c ................................ | ..........                  1 |                                  3.3         0.3             
3.20/3.29	c ................................ | ..........                  2 |              19                  3.3         0.6             
3.20/3.29	c ................................ | ..........                  3 |              20                  3.3         0.9             
3.20/3.29	c ................................ | ..........                  4 |              20                  3.3         1.2             
3.20/3.29	c ................................ | ..........                  5 |              20                  3.3         1.5             
3.20/3.29	c ................................ | ..........               1000 | 147          795                 3.3       302.1        21462
95.96/26.55	c ................................ | ..........             456000 | 101540       160272             96.2      4742.1      9934790
191.79/50.56	c ................................ | ..........             856000 | 185723       254833            192.0      4457.6     17495770
288.80/74.81	c ................................ | ..........            1005000 | 221810       294394            288.9      3479.3     20131563
385.85/99.18	c ................................ | ..........            1112000 | 247065       333466            386.2      2879.6     24630347
479.73/122.69	c ................................ | *.........            1213000 | 274242       369362            480.1      2526.6     28563102
576.34/146.86	c ................................ | *.........            1359000 | 308866       409390            576.6      2357.0     33630136
672.79/171.05	c ................................ | *.........            1536000 | 350279       436242            673.0      2282.3     36623658
769.39/195.20	c ................................ | *.........            1602000 | 366510       461032            769.4      2082.1     37032737
864.07/218.99	c ................................ | *.........            1659000 | 382022       470915            864.4      1919.2     38362501
959.89/242.95	c ................................ | **........            1762000 | 406391       487714            960.1      1835.2     40823226
1057.72/267.40	c ................................ | **........            1874000 | 432245       504886           1057.7      1771.7     43089701
1152.38/291.13	c ................................ | **........            1965000 | 455590       517594           1152.5      1705.0     45146764
1248.25/315.19	c ................................ | **........            2088000 | 486390       532421           1248.6      1672.3     47056983
1344.09/339.18	c ................................ | **........            2204000 | 510874       555390           1344.4      1639.4     47430618
1440.68/363.37	c ................................ | ***.......            2353000 | 545016       581585           1441.0      1632.9     49002927
1536.19/387.22	c ................................ | ***.......            2390000 | 554728       595558           1536.3      1555.7     50731471
1633.99/411.70	c ................................ | ***.......            2458000 | 570566       612088           1634.0      1504.3     52045428
1729.02/435.54	c ................................ | ***.......            2536000 | 588690       627373           1729.2      1466.6     52872648
1823.77/459.38	c ................................ | ***.......            2738000 | 633806       666893           1824.1      1501.0     53751621
1921.18/483.75	c ................................ | ****......            2900000 | 668492       703795           1921.4      1509.3     54724931
2015.87/507.46	c ................................ | ****......            3069000 | 708779       724982           2016.1      1522.2     56392963
2112.28/531.66	c ................................ | ****......            3288000 | 754471       762893           2112.5      1556.4     57198480
2207.67/555.60	c ................................ | ****......            3369000 | 774377       790516           2208.0      1525.8     57810851
2304.72/579.80	c ................................ | ****......            3421000 | 786754       806064           2304.7      1484.3     58730473
2400.18/603.71	c ................................ | *****.....            3462000 | 797320       816184           2400.2      1442.4     60357168
2495.91/627.77	c ................................ | *****.....            3567000 | 821047       837135           2496.2      1429.0     62307898
2592.12/651.80	c ................................ | *****.....            3701000 | 850026       856272           2592.1      1427.8     62783577
2690.71/676.55	c ................................ | *****.....            3757000 | 862900       878378           2690.9      1396.2     62783577
2784.46/700.04	c ................................ | *****.....            3830000 | 879670       908464           2784.6      1375.4     63269457
2879.71/723.99	c ................................ | ******....            3922000 | 900922       929403           2880.1      1361.8     63488687
2976.35/748.11	c ................................ | ******....            4020000 | 924514       939495           2976.4      1350.6     63488687
3072.60/772.22	c ................................ | ******....            4138000 | 950056       955367           3072.7      1346.7     63488687
3168.37/796.22	c ................................ | ******....            4269000 | 977606       967138           3168.5      1347.3     63488687
3264.12/820.20	c ................................ | ******....            4418000 | 1012420      993022           3264.1      1353.5     63488687
3360.31/844.39	c ................................ | *******...            4595000 | 1046063      1022788          3360.7      1367.3     63488687
3457.70/868.72	c ................................ | *******...            4814000 | 1090336      1049791          3457.8      1392.2     63488687
3552.73/892.55	c ................................ | *******...            4906000 | 1110759      1089510          3552.9      1380.8     64187651
3648.21/916.49	c ................................ | *******...            4960000 | 1122644      1137256          3648.6      1359.4     66199447
3743.67/940.39	c ................................ | *******...            4999000 | 1131851      1161544          3744.0      1335.2     67234394
3841.07/964.71	c ................................ | ********..            5048000 | 1144240      1177961          3841.1      1314.2     67982014
3936.33/988.68	c ................................ | ********..            5116000 | 1158469      1198507          3936.7      1299.6     68622611
4032.19/1012.61	c ................................ | ********..            5170000 | 1168665      1221649          4032.3      1282.2     69854106
4128.05/1036.69	c ................................ | ********..            5208000 | 1176389      1243531          4128.4      1261.5     71740020
4224.60/1060.85	c ................................ | ********..            5322000 | 1200393      1260917          4224.8      1259.7     72279820
4322.45/1085.30	c ................................ | *********.            5372000 | 1210947      1275035          4322.5      1242.8     73162922
4416.15/1108.80	c ................................ | *********.            5415000 | 1221646      1296595          4416.1      1226.2     73900307
4513.53/1133.26	c ................................ | *********.            5473000 | 1232922      1311600          4513.8      1212.5     73900307
4607.76/1156.86	c ................................ | *********.            5543000 | 1247324      1319939          4608.0      1202.9     74347110
4703.98/1180.91	c ................................ | *********.            5604000 | 1259509      1333214          4704.0      1191.3     74771064

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3274109-1304059873/watcher-3274109-1304059873 -o /tmp/evaluation-result-3274109-1304059873/solver-3274109-1304059873 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3274109-1304059873.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.26 8.07 8.01 5/181 9599
/proc/meminfo: memFree=19320204/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=8840 CPUtime=0 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) R 9597 9599 9362 0 -1 4202496 313 0 0 0 0 0 0 0 25 0 1 0 498403420 9052160 255 33554432000 4194304 4338100 140736625587312 18446744073709551615 255106436078 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/9599/statm: 2210 255 163 36 0 115 0

[startup+0.066677 s]
/proc/loadavg: 8.26 8.07 8.01 5/181 9599
/proc/meminfo: memFree=19320204/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=8840 CPUtime=0.06 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) R 9597 9599 9362 0 -1 4202496 313 0 0 0 6 0 0 0 25 0 1 0 498403420 9052160 255 33554432000 4194304 4338100 140736625587312 18446744073709551615 4290835 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/9599/statm: 2210 255 163 36 0 115 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 8840

[startup+0.10067 s]
/proc/loadavg: 8.26 8.07 8.01 5/181 9599
/proc/meminfo: memFree=19320204/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=8840 CPUtime=0.09 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) R 9597 9599 9362 0 -1 4202496 313 0 0 0 9 0 0 0 25 0 1 0 498403420 9052160 255 33554432000 4194304 4338100 140736625587312 18446744073709551615 255106424723 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/9599/statm: 2210 255 163 36 0 115 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 8840

[startup+0.300636 s]
/proc/loadavg: 8.26 8.07 8.01 5/181 9599
/proc/meminfo: memFree=19320204/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=8840 CPUtime=0.29 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) R 9597 9599 9362 0 -1 4202496 313 0 0 0 29 0 0 0 25 0 1 0 498403420 9052160 255 33554432000 4194304 4338100 140736625587312 18446744073709551615 4301283 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/9599/statm: 2210 255 163 36 0 115 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 8840

[startup+0.700564 s]
/proc/loadavg: 8.26 8.07 8.01 5/181 9599
/proc/meminfo: memFree=19320204/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=8840 CPUtime=0.69 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) R 9597 9599 9362 0 -1 4202496 324 0 0 0 69 0 0 0 25 0 1 0 498403420 9052160 266 33554432000 4194304 4338100 140736625587312 18446744073709551615 255106427017 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/9599/statm: 2210 266 173 36 0 115 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 8840

[startup+1.50042 s]
/proc/loadavg: 8.26 8.07 8.01 6/182 9600
/proc/meminfo: memFree=19310424/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=8840 CPUtime=1.49 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) R 9597 9599 9362 0 -1 4202496 324 0 0 0 149 0 0 0 25 0 1 0 498403420 9052160 266 33554432000 4194304 4338100 140736625587312 18446744073709551615 4246488 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/9599/statm: 2210 266 173 36 0 115 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 8840

[startup+3.10017 s]
/proc/loadavg: 8.00 8.02 8.00 6/182 9600
/proc/meminfo: memFree=19298152/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=8840 CPUtime=3.09 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) R 9597 9599 9362 0 -1 4202496 326 0 0 0 309 0 0 0 25 0 1 0 498403420 9052160 268 33554432000 4194304 4338100 140736625587312 18446744073709551615 4290891 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/9599/statm: 2210 268 175 36 0 115 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 8840

[startup+6.30054 s]
/proc/loadavg: 8.00 8.02 8.00 9/186 9604
/proc/meminfo: memFree=19178944/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=285060 CPUtime=15.3 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) S 9597 9599 9362 0 -1 4202496 25498 0 0 0 1524 6 0 0 25 0 5 0 498403420 291901440 25440 33554432000 4194304 4338100 140736625587312 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9599/statm: 71265 25440 196 36 0 69170 0
[pid=9599/tid=9601] ppid=9597 vsize=285060 CPUtime=3 cores=0,2,4,6
/proc/9599/task/9601/stat : 9601 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 7462 0 0 0 299 1 0 0 25 0 5 0 498403749 291901440 25440 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9599/tid=9602] ppid=9597 vsize=285060 CPUtime=3 cores=0,2,4,6
/proc/9599/task/9602/stat : 9602 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 4639 0 0 0 299 1 0 0 25 0 5 0 498403749 291901440 25441 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9599/tid=9603] ppid=9597 vsize=285060 CPUtime=3 cores=0,2,4,6
/proc/9599/task/9603/stat : 9603 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 8661 0 0 0 299 1 0 0 25 0 5 0 498403749 291901440 25441 33554432000 4194304 4338100 140736625587312 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9599/tid=9604] ppid=9597 vsize=285060 CPUtime=2.99 cores=0,2,4,6
/proc/9599/task/9604/stat : 9604 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 3669 0 0 0 297 2 0 0 25 0 5 0 498403749 291901440 25441 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 15.3
Current children cumulated vsize (KiB) 285060

[startup+12.7004 s]
/proc/loadavg: 8.00 8.02 8.00 9/186 9604
/proc/meminfo: memFree=19045892/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=354424 CPUtime=40.86 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) S 9597 9599 9362 0 -1 4202496 42018 0 0 0 4073 13 0 0 25 0 5 0 498403420 362930176 41960 33554432000 4194304 4338100 140736625587312 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9599/statm: 88606 41960 196 36 0 86511 0
[pid=9599/tid=9601] ppid=9597 vsize=354424 CPUtime=9.4 cores=0,2,4,6
/proc/9599/task/9601/stat : 9601 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 11939 0 0 0 938 2 0 0 25 0 5 0 498403749 362930176 41960 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9599/tid=9602] ppid=9597 vsize=354424 CPUtime=9.4 cores=0,2,4,6
/proc/9599/task/9602/stat : 9602 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 7457 0 0 0 937 3 0 0 25 0 5 0 498403749 362930176 41960 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9599/tid=9603] ppid=9597 vsize=354424 CPUtime=9.39 cores=0,2,4,6
/proc/9599/task/9603/stat : 9603 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 15189 0 0 0 937 2 0 0 25 0 5 0 498403749 362930176 41960 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9599/tid=9604] ppid=9597 vsize=354424 CPUtime=9.34 cores=0,2,4,6
/proc/9599/task/9604/stat : 9604 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 6365 0 0 0 930 4 0 0 25 0 5 0 498403749 362930176 41960 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 40.86
Current children cumulated vsize (KiB) 354424

[startup+25.5012 s]
/proc/loadavg: 8.00 8.02 8.00 9/186 9605
/proc/meminfo: memFree=18805844/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=497660 CPUtime=91.97 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) S 9597 9599 9362 0 -1 4202496 77873 0 0 0 9168 29 0 0 25 0 5 0 498403420 509603840 77815 33554432000 4194304 4338100 140736625587312 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9599/statm: 124415 77815 198 36 0 122320 0
[pid=9599/tid=9601] ppid=9597 vsize=497660 CPUtime=22.19 cores=0,2,4,6
/proc/9599/task/9601/stat : 9601 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 22182 0 0 0 2215 4 0 0 25 0 5 0 498403749 509603840 77815 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9599/tid=9602] ppid=9597 vsize=497660 CPUtime=22.2 cores=0,2,4,6
/proc/9599/task/9602/stat : 9602 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 13699 0 0 0 2211 9 0 0 25 0 5 0 498403749 509603840 77815 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9599/tid=9603] ppid=9597 vsize=497660 CPUtime=22.2 cores=0,2,4,6
/proc/9599/task/9603/stat : 9603 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 28907 0 0 0 2214 6 0 0 25 0 5 0 498403749 509603840 77815 33554432000 4194304 4338100 140736625587312 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9599/tid=9604] ppid=9597 vsize=497660 CPUtime=22.07 cores=0,2,4,6
/proc/9599/task/9604/stat : 9604 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 12017 0 0 0 2198 9 0 0 25 0 5 0 498403749 509603840 77815 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 91.97
Current children cumulated vsize (KiB) 497660

[startup+51.1007 s]
/proc/loadavg: 8.00 8.02 8.00 9/186 9606
/proc/meminfo: memFree=18343068/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=759056 CPUtime=194.13 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) S 9597 9599 9362 0 -1 4202496 142900 0 0 0 19349 64 0 0 25 0 5 0 498403420 777273344 142834 33554432000 4194304 4338100 140736625587312 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9599/statm: 189764 142834 198 36 0 187669 0
[pid=9599/tid=9601] ppid=9597 vsize=759056 CPUtime=47.75 cores=0,2,4,6
/proc/9599/task/9601/stat : 9601 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 44250 0 0 0 4765 10 0 0 25 0 5 0 498403749 777273344 142834 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9599/tid=9602] ppid=9597 vsize=759056 CPUtime=47.77 cores=0,2,4,6
/proc/9599/task/9602/stat : 9602 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 26922 0 0 0 4756 21 0 0 25 0 5 0 498403749 777273344 142834 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9599/tid=9603] ppid=9597 vsize=759056 CPUtime=47.78 cores=0,2,4,6
/proc/9599/task/9603/stat : 9603 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 46140 0 0 0 4768 10 0 0 25 0 5 0 498403749 777273344 142834 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9599/tid=9604] ppid=9597 vsize=759056 CPUtime=47.52 cores=0,2,4,6
/proc/9599/task/9604/stat : 9604 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 24520 0 0 0 4730 22 0 0 25 0 5 0 498403749 777273344 142834 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 194.13
Current children cumulated vsize (KiB) 759056

[startup+102.307 s]
/proc/loadavg: 8.03 8.03 8.00 9/186 9607
/proc/meminfo: memFree=17877184/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=996992 CPUtime=398.65 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) S 9597 9599 9362 0 -1 4202496 202871 0 0 0 39766 99 0 0 25 0 5 0 498403420 1020919808 202805 33554432000 4194304 4338100 140736625587312 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9599/statm: 249248 202805 198 36 0 247153 0
[pid=9599/tid=9601] ppid=9597 vsize=996992 CPUtime=98.94 cores=0,2,4,6
/proc/9599/task/9601/stat : 9601 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 58711 0 0 0 9880 14 0 0 25 0 5 0 498403749 1020919808 202805 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9599/tid=9602] ppid=9597 vsize=996992 CPUtime=98.97 cores=0,2,4,6
/proc/9599/task/9602/stat : 9602 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 34773 0 0 0 9864 33 0 0 25 0 5 0 498403749 1020919808 202805 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9599/tid=9603] ppid=9597 vsize=996992 CPUtime=98.97 cores=0,2,4,6
/proc/9599/task/9603/stat : 9603 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 76289 0 0 0 9880 17 0 0 25 0 5 0 498403749 1020919808 202805 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254259 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9599/tid=9604] ppid=9597 vsize=996992 CPUtime=98.45 cores=0,2,4,6
/proc/9599/task/9604/stat : 9604 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 32030 0 0 0 9812 33 0 0 25 0 5 0 498403749 1020919808 202805 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 398.65
Current children cumulated vsize (KiB) 996992

[startup+162.301 s]
/proc/loadavg: 8.01 8.02 8.00 9/186 9608
/proc/meminfo: memFree=17327980/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=1341840 CPUtime=638.1 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) S 9597 9599 9362 0 -1 4202496 289159 0 0 0 63670 140 0 0 25 0 5 0 498403420 1374044160 289093 33554432000 4194304 4338100 140736625587312 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9599/statm: 335460 289093 198 36 0 333365 0
[pid=9599/tid=9601] ppid=9597 vsize=1341840 CPUtime=158.85 cores=0,2,4,6
/proc/9599/task/9601/stat : 9601 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 81167 0 0 0 15865 20 0 0 25 0 5 0 498403749 1374044160 289093 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254400 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9599/tid=9602] ppid=9597 vsize=1341840 CPUtime=158.92 cores=0,2,4,6
/proc/9599/task/9602/stat : 9602 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 46976 0 0 0 15846 46 0 0 25 0 5 0 498403749 1374044160 289093 33554432000 4194304 4338100 140736625587312 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9599/tid=9603] ppid=9597 vsize=1341840 CPUtime=158.93 cores=0,2,4,6
/proc/9599/task/9603/stat : 9603 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 117382 0 0 0 15866 27 0 0 25 0 5 0 498403749 1374044160 289093 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9599/tid=9604] ppid=9597 vsize=1341840 CPUtime=158.09 cores=0,2,4,6
/proc/9599/task/9604/stat : 9604 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 42566 0 0 0 15763 46 0 0 25 0 5 0 498403749 1374044160 289093 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 638.1
Current children cumulated vsize (KiB) 1341840

[startup+222.301 s]
/proc/loadavg: 8.07 8.03 8.00 9/186 9610
/proc/meminfo: memFree=17168748/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=1453836 CPUtime=877.63 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) S 9597 9599 9362 0 -1 4202496 314794 0 0 0 87607 156 0 0 25 0 5 0 498403420 1488728064 314728 33554432000 4194304 4338100 140736625587312 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9599/statm: 363459 314729 198 36 0 361364 0
[pid=9599/tid=9601] ppid=9597 vsize=1453836 CPUtime=218.53 cores=0,2,4,6
/proc/9599/task/9601/stat : 9601 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 87608 0 0 0 21830 23 0 0 25 0 5 0 498403749 1488728064 314729 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9599/tid=9602] ppid=9597 vsize=1453836 CPUtime=218.83 cores=0,2,4,6
/proc/9599/task/9602/stat : 9602 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 49465 0 0 0 21835 48 0 0 25 0 5 0 498403749 1488728064 314729 33554432000 4194304 4338100 140736625587312 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9599/tid=9603] ppid=9597 vsize=1453836 CPUtime=218.86 cores=0,2,4,6
/proc/9599/task/9603/stat : 9603 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 131602 0 0 0 21855 31 0 0 25 0 5 0 498403749 1488728064 314730 33554432000 4194304 4338100 140736625587312 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9599/tid=9604] ppid=9597 vsize=1453836 CPUtime=218.08 cores=0,2,4,6
/proc/9599/task/9604/stat : 9604 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 45053 0 0 0 21756 52 0 0 25 0 5 0 498403749 1488728064 314730 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 877.63
Current children cumulated vsize (KiB) 1453836

################
# More data... #
################

/proc/meminfo: memFree=16496544/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=2328324 CPUtime=3033.04 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) S 9597 9599 9362 0 -1 4202496 526902 0 0 0 302945 359 0 0 25 0 5 0 498403420 2384203776 526836 33554432000 4194304 4338100 140736625587312 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9599/statm: 582081 526836 198 36 0 579986 0
[pid=9599/tid=9601] ppid=9597 vsize=2328324 CPUtime=757.13 cores=0,2,4,6
/proc/9599/task/9601/stat : 9601 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 178922 0 0 0 75657 56 0 0 25 0 5 0 498403749 2384203776 526836 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9599/tid=9602] ppid=9597 vsize=2328324 CPUtime=758.16 cores=0,2,4,6
/proc/9599/task/9602/stat : 9602 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 68991 0 0 0 75702 114 0 0 25 0 5 0 498403749 2384203776 526836 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9599/tid=9603] ppid=9597 vsize=2328324 CPUtime=758.63 cores=0,2,4,6
/proc/9599/task/9603/stat : 9603 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 222537 0 0 0 75805 58 0 0 25 0 5 0 498403749 2384203776 526836 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9599/tid=9604] ppid=9597 vsize=2328324 CPUtime=755.8 cores=0,2,4,6
/proc/9599/task/9604/stat : 9604 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 55384 0 0 0 75451 129 0 0 25 0 5 0 498403749 2384203776 526836 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3033.04
Current children cumulated vsize (KiB) 2328324

[startup+822.301 s]
/proc/loadavg: 8.09 8.04 8.01 9/186 9666
/proc/meminfo: memFree=16402912/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=2390020 CPUtime=3272.51 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) S 9597 9599 9362 0 -1 4202496 534436 0 0 0 326871 380 0 0 25 0 5 0 498403420 2447380480 534370 33554432000 4194304 4338100 140736625587312 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9599/statm: 597505 534370 198 36 0 595410 0
[pid=9599/tid=9601] ppid=9597 vsize=2390020 CPUtime=817.09 cores=0,2,4,6
/proc/9599/task/9601/stat : 9601 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 179097 0 0 0 81652 57 0 0 25 0 5 0 498403749 2447380480 534370 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9599/tid=9602] ppid=9597 vsize=2390020 CPUtime=817.84 cores=0,2,4,6
/proc/9599/task/9602/stat : 9602 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 73170 0 0 0 81660 124 0 0 25 0 5 0 498403749 2447380480 534370 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9599/tid=9603] ppid=9597 vsize=2390020 CPUtime=818.58 cores=0,2,4,6
/proc/9599/task/9603/stat : 9603 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 224492 0 0 0 81799 59 0 0 25 0 5 0 498403749 2447380480 534370 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9599/tid=9604] ppid=9597 vsize=2390020 CPUtime=815.69 cores=0,2,4,6
/proc/9599/task/9604/stat : 9604 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 56609 0 0 0 81430 139 0 0 25 0 5 0 498403749 2447380480 534370 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3272.51
Current children cumulated vsize (KiB) 2390020

[startup+882.3 s]
/proc/loadavg: 8.26 8.09 8.02 9/186 9668
/proc/meminfo: memFree=16288200/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=2431620 CPUtime=3512 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) S 9597 9599 9362 0 -1 4202496 544950 0 0 0 350798 402 0 0 25 0 5 0 498403420 2489978880 544884 33554432000 4194304 4338100 140736625587312 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9599/statm: 607905 544884 198 36 0 605810 0
[pid=9599/tid=9601] ppid=9597 vsize=2431620 CPUtime=877.04 cores=0,2,4,6
/proc/9599/task/9601/stat : 9601 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 180797 0 0 0 87646 58 0 0 25 0 5 0 498403749 2489978880 544884 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9599/tid=9602] ppid=9597 vsize=2431620 CPUtime=877.68 cores=0,2,4,6
/proc/9599/task/9602/stat : 9602 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 75946 0 0 0 87633 135 0 0 25 0 5 0 498403749 2489978880 544884 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9599/tid=9603] ppid=9597 vsize=2431620 CPUtime=878.52 cores=0,2,4,6
/proc/9599/task/9603/stat : 9603 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 224745 0 0 0 87793 59 0 0 25 0 5 0 498403749 2489978880 544884 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9599/tid=9604] ppid=9597 vsize=2431620 CPUtime=875.45 cores=0,2,4,6
/proc/9599/task/9604/stat : 9604 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 62394 0 0 0 87396 149 0 0 25 0 5 0 498403749 2489978880 544884 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3512
Current children cumulated vsize (KiB) 2431620

[startup+942.301 s]
/proc/loadavg: 8.24 8.10 8.03 9/186 9669
/proc/meminfo: memFree=16152032/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=2550996 CPUtime=3751.64 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) S 9597 9599 9362 0 -1 4202496 574588 0 0 0 374732 432 0 0 25 0 5 0 498403420 2612219904 574522 33554432000 4194304 4338100 140736625587312 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9599/statm: 637749 574522 198 36 0 635654 0
[pid=9599/tid=9601] ppid=9597 vsize=2550996 CPUtime=937.03 cores=0,2,4,6
/proc/9599/task/9601/stat : 9601 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 180827 0 0 0 93645 58 0 0 25 0 5 0 498403749 2612219904 574522 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9599/tid=9602] ppid=9597 vsize=2550996 CPUtime=937.34 cores=0,2,4,6
/proc/9599/task/9602/stat : 9602 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 76082 0 0 0 93588 146 0 0 25 0 5 0 498403749 2612219904 574522 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9599/tid=9603] ppid=9597 vsize=2550996 CPUtime=938.49 cores=0,2,4,6
/proc/9599/task/9603/stat : 9603 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 254077 0 0 0 93783 66 0 0 25 0 5 0 498403749 2612219904 574522 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9599/tid=9604] ppid=9597 vsize=2550996 CPUtime=935.45 cores=0,2,4,6
/proc/9599/task/9604/stat : 9604 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 62534 0 0 0 93386 159 0 0 25 0 5 0 498403749 2612219904 574522 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3751.64
Current children cumulated vsize (KiB) 2550996

[startup+1002.3 s]
/proc/loadavg: 8.04 8.07 8.01 9/186 9671
/proc/meminfo: memFree=16041532/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=2610684 CPUtime=3991.04 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) S 9597 9599 9362 0 -1 4202496 590799 0 0 0 398651 453 0 0 25 0 5 0 498403420 2673340416 590733 33554432000 4194304 4338100 140736625587312 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9599/statm: 652671 590733 198 36 0 650576 0
[pid=9599/tid=9601] ppid=9597 vsize=2610684 CPUtime=996.98 cores=0,2,4,6
/proc/9599/task/9601/stat : 9601 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 180855 0 0 0 99640 58 0 0 25 0 5 0 498403749 2673340416 590733 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9599/tid=9602] ppid=9597 vsize=2610684 CPUtime=996.94 cores=0,2,4,6
/proc/9599/task/9602/stat : 9602 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 76777 0 0 0 99540 154 0 0 25 0 5 0 498403749 2673340416 590733 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9599/tid=9603] ppid=9597 vsize=2610684 CPUtime=998.46 cores=0,2,4,6
/proc/9599/task/9603/stat : 9603 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 269520 0 0 0 99773 73 0 0 25 0 5 0 498403749 2673340416 590733 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9599/tid=9604] ppid=9597 vsize=2610684 CPUtime=995.35 cores=0,2,4,6
/proc/9599/task/9604/stat : 9604 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 62579 0 0 0 99368 167 0 0 25 0 5 0 498403749 2673340416 590733 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3991.04
Current children cumulated vsize (KiB) 2610684

[startup+1062.3 s]
/proc/loadavg: 8.21 8.10 8.02 9/186 9672
/proc/meminfo: memFree=15870772/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=2711036 CPUtime=4230.58 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) S 9597 9599 9362 0 -1 4202496 615532 0 0 0 422588 470 0 0 25 0 5 0 498403420 2776100864 615466 33554432000 4194304 4338100 140736625587312 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9599/statm: 677759 615466 198 36 0 675664 0
[pid=9599/tid=9601] ppid=9597 vsize=2711036 CPUtime=1056.96 cores=0,2,4,6
/proc/9599/task/9601/stat : 9601 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 180900 0 0 0 105638 58 0 0 25 0 5 0 498403749 2776100864 615466 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9599/tid=9602] ppid=9597 vsize=2711036 CPUtime=1056.56 cores=0,2,4,6
/proc/9599/task/9602/stat : 9602 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 76820 0 0 0 105497 159 0 0 25 0 5 0 498403749 2776100864 615466 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9599/tid=9603] ppid=9597 vsize=2711036 CPUtime=1058.44 cores=0,2,4,6
/proc/9599/task/9603/stat : 9603 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 294122 0 0 0 105766 78 0 0 25 0 5 0 498403749 2776100864 615466 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9599/tid=9604] ppid=9597 vsize=2711036 CPUtime=1055.3 cores=0,2,4,6
/proc/9599/task/9604/stat : 9604 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 62622 0 0 0 105357 173 0 0 25 0 5 0 498403749 2776100864 615466 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4230.58
Current children cumulated vsize (KiB) 2711036

[startup+1122.3 s]
/proc/loadavg: 8.08 8.08 8.02 9/186 9674
/proc/meminfo: memFree=15774168/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=2758340 CPUtime=4469.99 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) S 9597 9599 9362 0 -1 4202496 628017 0 0 0 446517 482 0 0 25 0 5 0 498403420 2824540160 627951 33554432000 4194304 4338100 140736625587312 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9599/statm: 689585 627951 198 36 0 687490 0
[pid=9599/tid=9601] ppid=9597 vsize=2758340 CPUtime=1116.58 cores=0,2,4,6
/proc/9599/task/9601/stat : 9601 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 180934 0 0 0 111600 58 0 0 25 0 5 0 498403749 2824540160 627951 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9599/tid=9602] ppid=9597 vsize=2758340 CPUtime=1116.45 cores=0,2,4,6
/proc/9599/task/9602/stat : 9602 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 76903 0 0 0 111481 164 0 0 25 0 5 0 498403749 2824540160 627951 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9599/tid=9603] ppid=9597 vsize=2758340 CPUtime=1118.42 cores=0,2,4,6
/proc/9599/task/9603/stat : 9603 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 306402 0 0 0 111760 82 0 0 25 0 5 0 498403749 2824540160 627951 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9599/tid=9604] ppid=9597 vsize=2758340 CPUtime=1115.2 cores=0,2,4,6
/proc/9599/task/9604/stat : 9604 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 62710 0 0 0 111345 175 0 0 19 0 5 0 498403749 2824540160 627951 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4469.99
Current children cumulated vsize (KiB) 2758340

[startup+1182.3 s]
/proc/loadavg: 8.08 8.08 8.02 9/186 9676
/proc/meminfo: memFree=15696888/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=2786492 CPUtime=4709.58 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) S 9597 9599 9362 0 -1 4202496 634929 0 0 0 470469 489 0 0 25 0 5 0 498403420 2853367808 634863 33554432000 4194304 4338100 140736625587312 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9599/statm: 696623 634863 198 36 0 694528 0
[pid=9599/tid=9601] ppid=9597 vsize=2786492 CPUtime=1176.28 cores=0,2,4,6
/proc/9599/task/9601/stat : 9601 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 183233 0 0 0 117568 60 0 0 25 0 5 0 498403749 2853367808 634863 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9599/tid=9602] ppid=9597 vsize=2786492 CPUtime=1176.44 cores=0,2,4,6
/proc/9599/task/9602/stat : 9602 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 76925 0 0 0 117475 169 0 0 25 0 5 0 498403749 2853367808 634863 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9599/tid=9603] ppid=9597 vsize=2786492 CPUtime=1178.38 cores=0,2,4,6
/proc/9599/task/9603/stat : 9603 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 310968 0 0 0 117755 83 0 0 25 0 5 0 498403749 2853367808 634863 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9599/tid=9604] ppid=9597 vsize=2786492 CPUtime=1175.16 cores=0,2,4,6
/proc/9599/task/9604/stat : 9604 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 62735 0 0 0 117340 176 0 0 25 0 5 0 498403749 2853367808 634863 33554432000 4194304 4338100 140736625587312 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4709.58
Current children cumulated vsize (KiB) 2786492



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1205 s]
/proc/loadavg: 8.06 8.07 8.02 9/186 9676
/proc/meminfo: memFree=15681992/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=2796004 CPUtime=4800.27 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) S 9597 9599 9362 0 -1 4202496 637475 0 0 0 479534 493 0 0 25 0 5 0 498403420 2863108096 637409 33554432000 4194304 4338100 140736625587312 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9599/statm: 699001 637409 198 36 0 696906 0
[pid=9599/tid=9601] ppid=9597 vsize=2796004 CPUtime=1198.86 cores=0,2,4,6
/proc/9599/task/9601/stat : 9601 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 183242 0 0 0 119826 60 0 0 25 0 5 0 498403749 2863108096 637409 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9599/tid=9602] ppid=9597 vsize=2796004 CPUtime=1199.14 cores=0,2,4,6
/proc/9599/task/9602/stat : 9602 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 76938 0 0 0 119744 170 0 0 25 0 5 0 498403749 2863108096 637409 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9599/tid=9603] ppid=9597 vsize=2796004 CPUtime=1201.08 cores=0,2,4,6
/proc/9599/task/9603/stat : 9603 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 313477 0 0 0 120024 84 0 0 25 0 5 0 498403749 2863108096 637409 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9599/tid=9604] ppid=9597 vsize=2796004 CPUtime=1197.86 cores=0,2,4,6
/proc/9599/task/9604/stat : 9604 (ttsth-5-4) R 9597 9599 9362 0 -1 4202560 62750 0 0 0 119610 176 0 0 25 0 5 0 498403749 2863108096 637409 33554432000 4194304 4338100 140736625587312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.27
Current children cumulated vsize (KiB) 2796004

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1205.11 s]
/proc/loadavg: 8.06 8.07 8.02 9/186 9676
/proc/meminfo: memFree=15681992/32951124 swapFree=67111524/67111528
[pid=9599] ppid=9597 vsize=0 CPUtime=4800.4 cores=0,2,4,6
/proc/9599/stat : 9599 (ttsth-5-4) Z 9597 9599 9362 0 -1 4203548 637478 0 0 0 479537 503 0 0 18 0 2 0 498403420 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9599/statm: 0 0 0 0 0 0 0
[pid=9599/tid=9601] ppid=9597 vsize=0 CPUtime=1198.97 cores=0,2,4,6
/proc/9599/task/9601/stat : 9601 (ttsth-5-4) R 9597 9599 9362 0 -1 4203588 183242 0 0 0 119827 70 0 0 25 0 2 0 498403749 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.4
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1205.13
CPU time (s): 4800.41
CPU user time (s): 4795.37
CPU system time (s): 5.04323
CPU usage (%): 398.333
Max. virtual memory (cumulated for all children) (KiB): 2796004

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4795.37
system time used= 5.04323
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 637478
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 3620
involuntary context switches= 21994

runsolver used 1.86772 second user time and 4.28835 second system time

The end

Launcher Data 

Begin job on node144 at 2011-04-29 08:51:13
IDJOB=3274109
IDBENCH=83153
IDSOLVER=1654
FILE ID=node144/3274109-1304059873
RUNJOBID= node144-1304057438-9380
PBS_JOBID= 13170439
Free space on /tmp= 72412 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/spence/sgen/sgen3-n260-s62321009-sat.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3274109-1304059873/watcher-3274109-1304059873 -o /tmp/evaluation-result-3274109-1304059873/solver-3274109-1304059873 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3274109-1304059873.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= f2b430bdd9bc1c472e9e437567706bb7
RANDOM SEED=358079075

node144.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.16
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      19320360 kB
Buffers:       2207004 kB
Cached:       10554876 kB
SwapCached:          4 kB
Active:        4921972 kB
Inactive:      8145680 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      19320360 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:             224 kB
Writeback:           0 kB
AnonPages:      304996 kB
Mapped:          15228 kB
Slab:           499360 kB
PageTables:       5356 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   496312 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72408 MiB
End job on node144 at 2011-04-29 09:11:20