Trace number 3273545

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.29 1203.99

General information on the benchmark

Namecrafted/spence/sgen/
sgen3-n180-s34713902-sat.cnf
MD5SUM19292459de25c4f5e9e6f187c00d549d
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.431934
Satisfiable
(Un)Satisfiability was proved
Number of variables180
Number of clauses432
Sum of the clauses size1080
Maximum clause length5
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2360
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 572
Number of clauses of size over 50

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3273545-1304057438.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 180
0.00/0.00	c Clauses = 432
0.00/0.00	c Literals = 1080
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 180
0.00/0.00	c Clauses = 432
0.00/0.00	c Literals = 1080
0.00/0.00	c a2
0.00/0.00	c a1
2.10/2.11	c Independent sub-problems = 1
2.10/2.11	c Sub-problem with 180 variables and 432 clauses
2.10/2.11	c using permutation method 3
2.10/2.11	c Ordering finished 2.110 seconds
2.10/2.11	c mapped 2.110 seconds
2.10/2.11	c permuted 2.110 seconds
2.10/2.11	c about to sort
2.10/2.11	c sorted
2.10/2.12	c clauses now 432
2.10/2.12	c init finished 2.110 seconds
2.10/2.12	c tree built 2.120 seconds
2.10/2.12	c 
2.10/2.12	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
2.10/2.12	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
2.10/2.12	c ................................ | ..........                    |                                  2.1         0.0             
2.10/2.12	c ................................ | ..........                  1 |                                  2.1         0.5             
2.10/2.12	c ................................ | ..........                  2 |                                  2.1         0.9             
2.10/2.12	c ................................ | ..........                  3 |                                  2.1         1.4             
2.10/2.12	c ................................ | ..........                  4 |                                  2.1         1.9             
2.10/2.12	c ................................ | ..........                  5 |                                  2.1         2.4             
2.10/2.12	c ................................ | ..........               1000 | 83           639                 2.1       471.7        11366
96.30/25.71	c ................................ | ..........             410000 | 89594        128491             96.3      4255.3      7655959
191.74/49.69	c ................................ | ..........             629000 | 138060       175525            192.1      3273.8     11453939
287.98/73.77	c ................................ | ..........             781000 | 175082       222849            288.3      2709.4     15033736
383.79/97.77	c ................................ | ..........             975000 | 224121       263532            384.1      2538.5     18566073
479.98/121.86	c ................................ | *.........            1125000 | 263577       289414            480.2      2342.6     21154471
577.46/146.26	c ................................ | *.........            1205000 | 284807       307875            577.7      2085.8     23157586
672.06/169.93	c ................................ | *.........            1287000 | 306742       321039            672.2      1914.6     24398695
767.90/193.97	c ................................ | *.........            1396000 | 334588       334668            768.2      1817.3     24433693
864.93/218.27	c ................................ | *.........            1491000 | 358675       355637            865.2      1723.3     25833245
959.85/242.04	c ................................ | **........            1723000 | 413574       416822            960.0      1794.8     27099946
1056.05/266.19	c ................................ | **........            1897000 | 452383       455713           1056.4      1795.7     29424427
1151.86/290.15	c ................................ | **........            2022000 | 482328       483258           1152.1      1755.1     31061821
1248.00/314.26	c ................................ | **........            2186000 | 524036       510612           1248.2      1751.3     31801296
1344.24/338.37	c ................................ | **........            2333000 | 562615       532678           1344.5      1735.2     33535345
1441.30/362.60	c ................................ | ***.......            2412000 | 583324       561342           1441.3      1673.5     35624951
1536.34/386.45	c ................................ | ***.......            2513000 | 610770       579411           1536.6      1635.5     37698665
1632.19/410.41	c ................................ | ***.......            2606000 | 633851       598062           1632.2      1596.6     39190461
1728.75/434.65	c ................................ | ***.......            2720000 | 662284       614285           1729.0      1573.2     39432187
1824.56/458.68	c ................................ | ***.......            2809000 | 684751       628057           1824.9      1539.3     39802850
1920.38/482.66	c ................................ | ****......            2899000 | 707644       644228           1920.7      1509.4     40139674
2016.20/506.66	c ................................ | ****......            2999000 | 734308       667741           2016.5      1487.3     40704636
2112.42/530.79	c ................................ | ****......            3100000 | 759475       684647           2112.8      1467.3     41115820
2208.23/554.78	c ................................ | ****......            3189000 | 782238       701870           2208.6      1443.9     42004915
2304.89/578.99	c ................................ | ****......            3274000 | 805252       711705           2305.3      1420.2     43221248
2400.34/602.83	c ................................ | *****.....            3327000 | 819463       722304           2400.5      1386.0     43673606
2497.77/627.25	c ................................ | *****.....            3375000 | 831618       732107           2498.0      1351.1     43994972
2592.04/650.85	c ................................ | *****.....            3427000 | 845743       741362           2592.2      1322.0     45052373
2688.71/675.00	c ................................ | *****.....            3493000 | 863037       748912           2688.7      1299.1     45937305
2784.17/698.99	c ................................ | *****.....            3548000 | 876911       757270           2784.5      1274.2     47400234
2880.04/722.91	c ................................ | ******....            3604000 | 892411       766791           2880.1      1251.4     48326625
2975.87/746.99	c ................................ | ******....            3695000 | 917545       778471           2976.2      1241.5     49303157
3071.64/770.99	c ................................ | ******....            3798000 | 944334       797180           3072.0      1236.3     49341503
3167.91/795.05	c ................................ | ******....            3892000 | 971107       813849           3168.1      1228.5     50443294
3264.13/819.17	c ................................ | ******....            3998000 | 998511       827246           3264.4      1224.7     51718312
3360.34/843.24	c ................................ | *******...            4090000 | 1021185      840630           3360.5      1217.1     52522829
3455.71/867.19	c ................................ | *******...            4164000 | 1039372      852979           3456.1      1204.8     53060196
3552.37/891.33	c ................................ | *******...            4248000 | 1062112      861828           3552.5      1195.8     53720066
3648.18/915.36	c ................................ | *******...            4347000 | 1089615      871304           3648.4      1191.5     54613975
3744.35/939.49	c ................................ | *******...            4445000 | 1116366      883212           3744.7      1187.0     54934400
3839.68/963.39	c ................................ | ********..            4539000 | 1143480      890827           3840.1      1182.0     55347100
3935.83/987.48	c ................................ | ********..            4616000 | 1165527      896521           3936.2      1172.7     55727649
4032.09/1011.54	c ................................ | ********..            4666000 | 1179325      904392           4032.3      1157.2     55727649
4129.54/1035.93	c ................................ | ********..            4727000 | 1197243      909942           4129.7      1144.6     55727649
4223.61/1059.60	c ................................ | ********..            4803000 | 1218894      916386           4224.0      1137.1     55727649
4320.64/1083.88	c ................................ | *********.            4878000 | 1238854      931576           4321.0      1128.9     55727649
4416.77/1107.90	c ................................ | *********.            4950000 | 1259094      953348           4416.8      1120.7     55727649
4512.62/1131.90	c ................................ | *********.            5023000 | 1276570      977440           4512.6      1113.1     55874528
4608.07/1155.89	c ................................ | *********.            5092000 | 1293311      998268           4608.4      1104.9     56751198
4705.52/1180.28	c ................................ | *********.            5153000 | 1308550      1008369          4705.9      1095.0     57292588

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3273545-1304057438/watcher-3273545-1304057438 -o /tmp/evaluation-result-3273545-1304057438/solver-3273545-1304057438 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3273545-1304057438.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.87 7.85 7.96 3/176 9399
/proc/meminfo: memFree=19559984/32951124 swapFree=67111524/67111528
[pid=9397] ppid=9394 vsize=8840 CPUtime=0 cores=1,3,5,7
/proc/9397/stat : 9397 (ttsth-5-4) R 9394 9397 9362 0 -1 4202496 313 0 0 0 0 0 0 0 18 0 1 0 498159939 9052160 255 33554432000 4194304 4338100 140737232303104 18446744073709551615 4290788 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/9397/statm: 2210 255 163 36 0 115 0

[startup+0.0561719 s]
/proc/loadavg: 6.87 7.85 7.96 3/176 9399
/proc/meminfo: memFree=19559984/32951124 swapFree=67111524/67111528
[pid=9397] ppid=9394 vsize=8840 CPUtime=0.05 cores=1,3,5,7
/proc/9397/stat : 9397 (ttsth-5-4) R 9394 9397 9362 0 -1 4202496 313 0 0 0 5 0 0 0 18 0 1 0 498159939 9052160 255 33554432000 4194304 4338100 140737232303104 18446744073709551615 4245995 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/9397/statm: 2210 255 163 36 0 115 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 8840

[startup+0.100175 s]
/proc/loadavg: 6.87 7.85 7.96 3/176 9399
/proc/meminfo: memFree=19559984/32951124 swapFree=67111524/67111528
[pid=9397] ppid=9394 vsize=8840 CPUtime=0.09 cores=1,3,5,7
/proc/9397/stat : 9397 (ttsth-5-4) R 9394 9397 9362 0 -1 4202496 313 0 0 0 9 0 0 0 18 0 1 0 498159939 9052160 255 33554432000 4194304 4338100 140737232303104 18446744073709551615 4245895 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/9397/statm: 2210 255 163 36 0 115 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 8840

[startup+0.30013 s]
/proc/loadavg: 6.87 7.85 7.96 3/176 9399
/proc/meminfo: memFree=19559984/32951124 swapFree=67111524/67111528
[pid=9397] ppid=9394 vsize=8840 CPUtime=0.29 cores=1,3,5,7
/proc/9397/stat : 9397 (ttsth-5-4) R 9394 9397 9362 0 -1 4202496 324 0 0 0 29 0 0 0 18 0 1 0 498159939 9052160 266 33554432000 4194304 4338100 140737232303104 18446744073709551615 255110422866 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/9397/statm: 2210 266 173 36 0 115 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 8840

[startup+0.701059 s]
/proc/loadavg: 6.87 7.85 7.96 3/176 9399
/proc/meminfo: memFree=19559984/32951124 swapFree=67111524/67111528
[pid=9397] ppid=9394 vsize=8840 CPUtime=0.7 cores=1,3,5,7
/proc/9397/stat : 9397 (ttsth-5-4) R 9394 9397 9362 0 -1 4202496 324 0 0 0 70 0 0 0 18 0 1 0 498159939 9052160 266 33554432000 4194304 4338100 140737232303104 18446744073709551615 4301300 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/9397/statm: 2210 266 173 36 0 115 0
Current children cumulated CPU time (s) 0.7
Current children cumulated vsize (KiB) 8840

[startup+1.50093 s]
/proc/loadavg: 6.87 7.85 7.96 3/178 9401
/proc/meminfo: memFree=19559364/32951124 swapFree=67111524/67111528
[pid=9397] ppid=9394 vsize=8840 CPUtime=1.5 cores=1,3,5,7
/proc/9397/stat : 9397 (ttsth-5-4) R 9394 9397 9362 0 -1 4202496 326 0 0 0 150 0 0 0 20 0 1 0 498159939 9052160 268 33554432000 4194304 4338100 140737232303104 18446744073709551615 4245900 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/9397/statm: 2210 268 175 36 0 115 0
Current children cumulated CPU time (s) 1.5
Current children cumulated vsize (KiB) 8840

[startup+3.1006 s]
/proc/loadavg: 6.48 7.75 7.92 7/182 9405
/proc/meminfo: memFree=19551764/32951124 swapFree=67111524/67111528
[pid=9397] ppid=9394 vsize=227312 CPUtime=6.02 cores=1,3,5,7
/proc/9397/stat : 9397 (ttsth-5-4) S 9394 9397 9362 0 -1 4202496 10581 0 0 0 600 2 0 0 21 0 5 0 498159939 232767488 10523 33554432000 4194304 4338100 140737232303104 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9397/statm: 56828 10524 196 36 0 54733 0
Current children cumulated CPU time (s) 6.02
Current children cumulated vsize (KiB) 227312

[startup+6.30104 s]
/proc/loadavg: 6.48 7.75 7.92 10/186 9410
/proc/meminfo: memFree=19428728/32951124 swapFree=67111524/67111528
[pid=9397] ppid=9394 vsize=277620 CPUtime=18.8 cores=1,3,5,7
/proc/9397/stat : 9397 (ttsth-5-4) S 9394 9397 9362 0 -1 4202496 23379 0 0 0 1872 8 0 0 21 0 5 0 498159939 284282880 23321 33554432000 4194304 4338100 140737232303104 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9397/statm: 69405 23321 196 36 0 67310 0
[pid=9397/tid=9406] ppid=9394 vsize=277620 CPUtime=4.17 cores=1,3,5,7
/proc/9397/task/9406/stat : 9406 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 3888 0 0 0 414 3 0 0 25 0 5 0 498160151 284282880 23321 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9397/tid=9407] ppid=9394 vsize=277620 CPUtime=4.16 cores=1,3,5,7
/proc/9397/task/9407/stat : 9407 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 7025 0 0 0 415 1 0 0 25 0 5 0 498160151 284282880 23321 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=9397/tid=9408] ppid=9394 vsize=277620 CPUtime=4.14 cores=1,3,5,7
/proc/9397/task/9408/stat : 9408 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 3494 0 0 0 413 1 0 0 25 0 5 0 498160151 284282880 23321 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9397/tid=9409] ppid=9394 vsize=277620 CPUtime=4.17 cores=1,3,5,7
/proc/9397/task/9409/stat : 9409 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 8146 0 0 0 416 1 0 0 25 0 5 0 498160151 284282880 23321 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 18.8
Current children cumulated vsize (KiB) 277620

[startup+12.7009 s]
/proc/loadavg: 6.71 7.76 7.92 10/186 9410
/proc/meminfo: memFree=19318652/32951124 swapFree=67111524/67111528
[pid=9397] ppid=9394 vsize=338452 CPUtime=44.37 cores=1,3,5,7
/proc/9397/stat : 9397 (ttsth-5-4) S 9394 9397 9362 0 -1 4202496 37754 0 0 0 4422 15 0 0 21 0 5 0 498159939 346574848 37680 33554432000 4194304 4338100 140737232303104 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9397/statm: 84613 37680 198 36 0 82518 0
[pid=9397/tid=9406] ppid=9394 vsize=338452 CPUtime=10.57 cores=1,3,5,7
/proc/9397/task/9406/stat : 9406 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 5735 0 0 0 1052 5 0 0 25 0 5 0 498160151 346574848 37680 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9397/tid=9407] ppid=9394 vsize=338452 CPUtime=10.57 cores=1,3,5,7
/proc/9397/task/9407/stat : 9407 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 11229 0 0 0 1054 3 0 0 25 0 5 0 498160151 346574848 37680 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=9397/tid=9408] ppid=9394 vsize=338452 CPUtime=10.51 cores=1,3,5,7
/proc/9397/task/9408/stat : 9408 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 5422 0 0 0 1048 3 0 0 25 0 5 0 498160151 346574848 37680 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9397/tid=9409] ppid=9394 vsize=338452 CPUtime=10.57 cores=1,3,5,7
/proc/9397/task/9409/stat : 9409 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 14542 0 0 0 1054 3 0 0 25 0 5 0 498160151 346574848 37680 33554432000 4194304 4338100 140737232303104 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 44.37
Current children cumulated vsize (KiB) 338452

[startup+25.5007 s]
/proc/loadavg: 6.91 7.77 7.92 10/186 9410
/proc/meminfo: memFree=19161824/32951124 swapFree=67111524/67111528
[pid=9397] ppid=9394 vsize=429420 CPUtime=95.5 cores=1,3,5,7
/proc/9397/stat : 9397 (ttsth-5-4) S 9394 9397 9362 0 -1 4202496 61432 0 0 0 9522 28 0 0 21 0 5 0 498159939 439726080 61358 33554432000 4194304 4338100 140737232303104 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9397/statm: 107355 61358 198 36 0 105260 0
[pid=9397/tid=9406] ppid=9394 vsize=429420 CPUtime=23.37 cores=1,3,5,7
/proc/9397/task/9406/stat : 9406 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 9826 0 0 0 2328 9 0 0 25 0 5 0 498160151 439726080 61358 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9397/tid=9407] ppid=9394 vsize=429420 CPUtime=23.36 cores=1,3,5,7
/proc/9397/task/9407/stat : 9407 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 17216 0 0 0 2332 4 0 0 25 0 5 0 498160151 439726080 61358 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=9397/tid=9408] ppid=9394 vsize=429420 CPUtime=23.24 cores=1,3,5,7
/proc/9397/task/9408/stat : 9408 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 9262 0 0 0 2316 8 0 0 25 0 5 0 498160151 439726080 61358 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9397/tid=9409] ppid=9394 vsize=429420 CPUtime=23.37 cores=1,3,5,7
/proc/9397/task/9409/stat : 9409 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 24302 0 0 0 2332 5 0 0 25 0 5 0 498160151 439726080 61358 33554432000 4194304 4338100 140737232303104 18446744073709551615 4256798 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 95.5
Current children cumulated vsize (KiB) 429420

[startup+51.1012 s]
/proc/loadavg: 7.28 7.78 7.92 10/186 9411
/proc/meminfo: memFree=18922120/32951124 swapFree=67111524/67111528
[pid=9397] ppid=9394 vsize=559228 CPUtime=197.74 cores=1,3,5,7
/proc/9397/stat : 9397 (ttsth-5-4) S 9394 9397 9362 0 -1 4202496 93323 0 0 0 19727 47 0 0 21 0 5 0 498159939 572649472 93241 33554432000 4194304 4338100 140737232303104 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9397/statm: 139807 93241 198 36 0 137712 0
[pid=9397/tid=9406] ppid=9394 vsize=559228 CPUtime=48.96 cores=1,3,5,7
/proc/9397/task/9406/stat : 9406 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 15283 0 0 0 4881 15 0 0 25 0 5 0 498160151 572649472 93241 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9397/tid=9407] ppid=9394 vsize=559228 CPUtime=48.96 cores=1,3,5,7
/proc/9397/task/9407/stat : 9407 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 27012 0 0 0 4889 7 0 0 25 0 5 0 498160151 572649472 93241 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=9397/tid=9408] ppid=9394 vsize=559228 CPUtime=48.69 cores=1,3,5,7
/proc/9397/task/9408/stat : 9408 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 14289 0 0 0 4856 13 0 0 25 0 5 0 498160151 572649472 93241 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9397/tid=9409] ppid=9394 vsize=559228 CPUtime=48.96 cores=1,3,5,7
/proc/9397/task/9409/stat : 9409 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 35913 0 0 0 4886 10 0 0 25 0 5 0 498160151 572649472 93241 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 197.74
Current children cumulated vsize (KiB) 559228

[startup+102.307 s]
/proc/loadavg: 7.76 7.83 7.93 10/186 9412
/proc/meminfo: memFree=18526304/32951124 swapFree=67111524/67111528
[pid=9397] ppid=9394 vsize=798120 CPUtime=402.19 cores=1,3,5,7
/proc/9397/stat : 9397 (ttsth-5-4) S 9394 9397 9362 0 -1 4202496 152922 0 0 0 40132 87 0 0 21 0 5 0 498159939 817274880 152840 33554432000 4194304 4338100 140737232303104 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9397/statm: 199530 152840 198 36 0 197435 0
[pid=9397/tid=9406] ppid=9394 vsize=798120 CPUtime=100.13 cores=1,3,5,7
/proc/9397/task/9406/stat : 9406 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 24130 0 0 0 9984 29 0 0 25 0 5 0 498160151 817274880 152840 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9397/tid=9407] ppid=9394 vsize=798120 CPUtime=100.14 cores=1,3,5,7
/proc/9397/task/9407/stat : 9407 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 42136 0 0 0 10002 12 0 0 25 0 5 0 498160151 817274880 152840 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=9397/tid=9408] ppid=9394 vsize=798120 CPUtime=99.6 cores=1,3,5,7
/proc/9397/task/9408/stat : 9408 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 21837 0 0 0 9936 24 0 0 25 0 5 0 498160151 817274880 152840 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9397/tid=9409] ppid=9394 vsize=798120 CPUtime=100.15 cores=1,3,5,7
/proc/9397/task/9409/stat : 9409 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 63993 0 0 0 9996 19 0 0 25 0 5 0 498160151 817274880 152840 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 402.19
Current children cumulated vsize (KiB) 798120

[startup+162.301 s]
/proc/loadavg: 8.08 7.91 7.95 10/186 9414
/proc/meminfo: memFree=18243952/32951124 swapFree=67111524/67111528
[pid=9397] ppid=9394 vsize=961104 CPUtime=641.74 cores=1,3,5,7
/proc/9397/stat : 9397 (ttsth-5-4) S 9394 9397 9362 0 -1 4202496 193601 0 0 0 64050 124 0 0 21 0 5 0 498159939 984170496 193519 33554432000 4194304 4338100 140737232303104 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9397/statm: 240276 193519 198 36 0 238181 0
[pid=9397/tid=9406] ppid=9394 vsize=961104 CPUtime=160.1 cores=1,3,5,7
/proc/9397/task/9406/stat : 9406 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 29774 0 0 0 15968 42 0 0 25 0 5 0 498160151 984170496 193519 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9397/tid=9407] ppid=9394 vsize=961104 CPUtime=160.11 cores=1,3,5,7
/proc/9397/task/9407/stat : 9407 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 52381 0 0 0 15994 17 0 0 25 0 5 0 498160151 984170496 193519 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=9397/tid=9408] ppid=9394 vsize=961104 CPUtime=159.26 cores=1,3,5,7
/proc/9397/task/9408/stat : 9408 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 26037 0 0 0 15892 34 0 0 25 0 5 0 498160151 984170496 193519 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9397/tid=9409] ppid=9394 vsize=961104 CPUtime=160.11 cores=1,3,5,7
/proc/9397/task/9409/stat : 9409 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 84583 0 0 0 15982 29 0 0 25 0 5 0 498160151 984170496 193519 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 641.74
Current children cumulated vsize (KiB) 961104

[startup+222.301 s]
/proc/loadavg: 8.11 7.95 7.96 7/186 9415
/proc/meminfo: memFree=18050996/32951124 swapFree=67111524/67111528
[pid=9397] ppid=9394 vsize=1042244 CPUtime=881.23 cores=1,3,5,7
/proc/9397/stat : 9397 (ttsth-5-4) S 9394 9397 9362 0 -1 4202496 213582 0 0 0 87973 150 0 0 21 0 5 0 498159939 1067257856 213500 33554432000 4194304 4338100 140737232303104 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9397/statm: 260561 213500 198 36 0 258466 0
[pid=9397/tid=9406] ppid=9394 vsize=1042244 CPUtime=220.03 cores=1,3,5,7
/proc/9397/task/9406/stat : 9406 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 31853 0 0 0 21950 53 0 0 25 0 5 0 498160151 1067257856 213500 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9397/tid=9407] ppid=9394 vsize=1042244 CPUtime=220.09 cores=1,3,5,7
/proc/9397/task/9407/stat : 9407 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 58021 0 0 0 21990 19 0 0 25 0 5 0 498160151 1067257856 213500 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=9397/tid=9408] ppid=9394 vsize=1042244 CPUtime=218.86 cores=1,3,5,7
/proc/9397/task/9408/stat : 9408 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 29840 0 0 0 21842 44 0 0 25 0 5 0 498160151 1067257856 213500 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9397/tid=9409] ppid=9394 vsize=1042244 CPUtime=220.09 cores=1,3,5,7
/proc/9397/task/9409/stat : 9409 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 93042 0 0 0 21977 32 0 0 25 0 5 0 498160151 1067257856 213500 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 881.23
Current children cumulated vsize (KiB) 1042244

################
# More data... #
################

[pid=9397/tid=9407] ppid=9394 vsize=1795448 CPUtime=759.24 cores=1,3,5,7
/proc/9397/task/9407/stat : 9407 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 106694 0 0 0 75886 38 0 0 25 0 5 0 498160151 1838538752 401832 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=9397/tid=9408] ppid=9394 vsize=1795448 CPUtime=757.6 cores=1,3,5,7
/proc/9397/task/9408/stat : 9408 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 45027 0 0 0 75642 118 0 0 25 0 5 0 498160151 1838538752 401832 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9397/tid=9409] ppid=9394 vsize=1795448 CPUtime=759.32 cores=1,3,5,7
/proc/9397/task/9409/stat : 9409 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 197749 0 0 0 75849 83 0 0 25 0 5 0 498160151 1838538752 401832 33554432000 4194304 4338100 140737232303104 18446744073709551615 4283626 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3037.31
Current children cumulated vsize (KiB) 1795448

[startup+822.3 s]
/proc/loadavg: 8.16 8.09 8.01 10/186 9431
/proc/meminfo: memFree=16531880/32951124 swapFree=67111524/67111528
[pid=9397] ppid=9394 vsize=1870776 CPUtime=3276.91 cores=1,3,5,7
/proc/9397/stat : 9397 (ttsth-5-4) S 9394 9397 9362 0 -1 4202496 420884 0 0 0 327290 401 0 0 21 0 5 0 498159939 1915674624 420802 33554432000 4194304 4338100 140737232303104 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9397/statm: 467694 420802 198 36 0 465599 0
[pid=9397/tid=9406] ppid=9394 vsize=1870776 CPUtime=818.99 cores=1,3,5,7
/proc/9397/task/9406/stat : 9406 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 53582 0 0 0 81754 145 0 0 25 0 5 0 498160151 1915674624 420802 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9397/tid=9407] ppid=9394 vsize=1870776 CPUtime=819.22 cores=1,3,5,7
/proc/9397/task/9407/stat : 9407 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 113008 0 0 0 81881 41 0 0 25 0 5 0 498160151 1915674624 420802 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=9397/tid=9408] ppid=9394 vsize=1870776 CPUtime=817.26 cores=1,3,5,7
/proc/9397/task/9408/stat : 9408 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 45076 0 0 0 81601 125 0 0 25 0 5 0 498160151 1915674624 420802 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9397/tid=9409] ppid=9394 vsize=1870776 CPUtime=819.27 cores=1,3,5,7
/proc/9397/task/9409/stat : 9409 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 208392 0 0 0 81840 87 0 0 25 0 5 0 498160151 1915674624 420802 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3276.91
Current children cumulated vsize (KiB) 1870776

[startup+882.301 s]
/proc/loadavg: 8.34 8.14 8.03 10/186 9436
/proc/meminfo: memFree=16411848/32951124 swapFree=67111524/67111528
[pid=9397] ppid=9394 vsize=1921080 CPUtime=3516.42 cores=1,3,5,7
/proc/9397/stat : 9397 (ttsth-5-4) S 9394 9397 9362 0 -1 4202496 434225 0 0 0 351221 421 0 0 21 0 5 0 498159939 1967185920 434143 33554432000 4194304 4338100 140737232303104 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9397/statm: 480270 434143 198 36 0 478175 0
[pid=9397/tid=9406] ppid=9394 vsize=1921080 CPUtime=878.92 cores=1,3,5,7
/proc/9397/task/9406/stat : 9406 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 54144 0 0 0 87739 153 0 0 25 0 5 0 498160151 1967185920 434143 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9397/tid=9407] ppid=9394 vsize=1921080 CPUtime=879.22 cores=1,3,5,7
/proc/9397/task/9407/stat : 9407 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 117379 0 0 0 87878 44 0 0 25 0 5 0 498160151 1967185920 434143 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=9397/tid=9408] ppid=9394 vsize=1921080 CPUtime=876.88 cores=1,3,5,7
/proc/9397/task/9408/stat : 9408 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 45112 0 0 0 87556 132 0 0 25 0 5 0 498160151 1967185920 434143 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9397/tid=9409] ppid=9394 vsize=1921080 CPUtime=879.26 cores=1,3,5,7
/proc/9397/task/9409/stat : 9409 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 216764 0 0 0 87834 92 0 0 25 0 5 0 498160151 1967185920 434143 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3516.42
Current children cumulated vsize (KiB) 1921080

[startup+942.3 s]
/proc/loadavg: 8.12 8.12 8.03 10/186 9437
/proc/meminfo: memFree=16305516/32951124 swapFree=67111524/67111528
[pid=9397] ppid=9394 vsize=1971256 CPUtime=3755.91 cores=1,3,5,7
/proc/9397/stat : 9397 (ttsth-5-4) S 9394 9397 9362 0 -1 4202496 446453 0 0 0 375149 442 0 0 21 0 5 0 498159939 2018566144 446371 33554432000 4194304 4338100 140737232303104 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9397/statm: 492814 446371 198 36 0 490719 0
[pid=9397/tid=9406] ppid=9394 vsize=1971256 CPUtime=938.86 cores=1,3,5,7
/proc/9397/task/9406/stat : 9406 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 55693 0 0 0 93725 161 0 0 25 0 5 0 498160151 2018566144 446371 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9397/tid=9407] ppid=9394 vsize=1971256 CPUtime=939.19 cores=1,3,5,7
/proc/9397/task/9407/stat : 9407 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 121341 0 0 0 93874 45 0 0 25 0 5 0 498160151 2018566144 446371 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=9397/tid=9408] ppid=9394 vsize=1971256 CPUtime=936.47 cores=1,3,5,7
/proc/9397/task/9408/stat : 9408 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 45147 0 0 0 93509 138 0 0 25 0 5 0 498160151 2018566144 446371 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9397/tid=9409] ppid=9394 vsize=1971256 CPUtime=939.23 cores=1,3,5,7
/proc/9397/task/9409/stat : 9409 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 223446 0 0 0 93828 95 0 0 25 0 5 0 498160151 2018566144 446371 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3755.91
Current children cumulated vsize (KiB) 1971256

[startup+1002.3 s]
/proc/loadavg: 8.04 8.09 8.02 10/186 9439
/proc/meminfo: memFree=16209164/32951124 swapFree=67111524/67111528
[pid=9397] ppid=9394 vsize=2021000 CPUtime=3995.33 cores=1,3,5,7
/proc/9397/stat : 9397 (ttsth-5-4) S 9394 9397 9362 0 -1 4202496 455806 0 0 0 399075 458 0 0 21 0 5 0 498159939 2069504000 455724 33554432000 4194304 4338100 140737232303104 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9397/statm: 505250 455724 198 36 0 503155 0
[pid=9397/tid=9406] ppid=9394 vsize=2021000 CPUtime=998.48 cores=1,3,5,7
/proc/9397/task/9406/stat : 9406 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 57856 0 0 0 99679 169 0 0 25 0 5 0 498160151 2069504000 455724 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9397/tid=9407] ppid=9394 vsize=2021000 CPUtime=999.16 cores=1,3,5,7
/proc/9397/task/9407/stat : 9407 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 121964 0 0 0 99870 46 0 0 25 0 5 0 498160151 2069504000 455724 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=9397/tid=9408] ppid=9394 vsize=2021000 CPUtime=996.32 cores=1,3,5,7
/proc/9397/task/9408/stat : 9408 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 46462 0 0 0 99488 144 0 0 25 0 5 0 498160151 2069504000 455724 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9397/tid=9409] ppid=9394 vsize=2021000 CPUtime=999.22 cores=1,3,5,7
/proc/9397/task/9409/stat : 9409 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 228698 0 0 0 99825 97 0 0 25 0 5 0 498160151 2069504000 455724 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3995.33
Current children cumulated vsize (KiB) 2021000

[startup+1062.3 s]
/proc/loadavg: 8.01 8.07 8.02 10/186 9441
/proc/meminfo: memFree=16149748/32951124 swapFree=67111524/67111528
[pid=9397] ppid=9394 vsize=2021128 CPUtime=4234.76 cores=1,3,5,7
/proc/9397/stat : 9397 (ttsth-5-4) S 9394 9397 9362 0 -1 4202496 457909 0 0 0 423005 471 0 0 21 0 5 0 498159939 2069635072 457827 33554432000 4194304 4338100 140737232303104 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9397/statm: 505282 457827 198 36 0 503187 0
[pid=9397/tid=9406] ppid=9394 vsize=2021128 CPUtime=1058.11 cores=1,3,5,7
/proc/9397/task/9406/stat : 9406 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 59396 0 0 0 105634 177 0 0 25 0 5 0 498160151 2069635072 457827 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9397/tid=9407] ppid=9394 vsize=2021128 CPUtime=1059.09 cores=1,3,5,7
/proc/9397/task/9407/stat : 9407 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 122255 0 0 0 105863 46 0 0 25 0 5 0 498160151 2069635072 457827 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=9397/tid=9408] ppid=9394 vsize=2021128 CPUtime=1056.22 cores=1,3,5,7
/proc/9397/task/9408/stat : 9408 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 46491 0 0 0 105473 149 0 0 25 0 5 0 498160151 2069635072 457827 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9397/tid=9409] ppid=9394 vsize=2021128 CPUtime=1059.18 cores=1,3,5,7
/proc/9397/task/9409/stat : 9409 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 228941 0 0 0 105821 97 0 0 25 0 5 0 498160151 2069635072 457827 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4234.76
Current children cumulated vsize (KiB) 2021128

[startup+1122.3 s]
/proc/loadavg: 8.09 8.09 8.02 10/186 9442
/proc/meminfo: memFree=16109072/32951124 swapFree=67111524/67111528
[pid=9397] ppid=9394 vsize=2021512 CPUtime=4474.28 cores=1,3,5,7
/proc/9397/stat : 9397 (ttsth-5-4) S 9394 9397 9362 0 -1 4202496 458188 0 0 0 446943 485 0 0 21 0 5 0 498159939 2070028288 458106 33554432000 4194304 4338100 140737232303104 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9397/statm: 505378 458106 198 36 0 503283 0
[pid=9397/tid=9406] ppid=9394 vsize=2021512 CPUtime=1117.7 cores=1,3,5,7
/proc/9397/task/9406/stat : 9406 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 59462 0 0 0 111586 184 0 0 25 0 5 0 498160151 2070028288 458106 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9397/tid=9407] ppid=9394 vsize=2021512 CPUtime=1119.09 cores=1,3,5,7
/proc/9397/task/9407/stat : 9407 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 122321 0 0 0 111863 46 0 0 25 0 5 0 498160151 2070028288 458106 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=9397/tid=9408] ppid=9394 vsize=2021512 CPUtime=1116.15 cores=1,3,5,7
/proc/9397/task/9408/stat : 9408 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 46551 0 0 0 111459 156 0 0 25 0 5 0 498160151 2070028288 458106 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9397/tid=9409] ppid=9394 vsize=2021512 CPUtime=1119.18 cores=1,3,5,7
/proc/9397/task/9409/stat : 9409 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 229028 0 0 0 111821 97 0 0 25 0 5 0 498160151 2070028288 458106 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4474.28
Current children cumulated vsize (KiB) 2021512

[startup+1182.3 s]
/proc/loadavg: 7.95 8.04 8.00 10/186 9444
/proc/meminfo: memFree=15984552/32951124 swapFree=67111524/67111528
[pid=9397] ppid=9394 vsize=2075072 CPUtime=4713.9 cores=1,3,5,7
/proc/9397/stat : 9397 (ttsth-5-4) S 9394 9397 9362 0 -1 4202496 471319 0 0 0 470887 503 0 0 21 0 5 0 498159939 2124873728 471237 33554432000 4194304 4338100 140737232303104 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9397/statm: 518768 471237 198 36 0 516673 0
[pid=9397/tid=9406] ppid=9394 vsize=2075072 CPUtime=1177.35 cores=1,3,5,7
/proc/9397/task/9406/stat : 9406 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 59518 0 0 0 117546 189 0 0 25 0 5 0 498160151 2124873728 471237 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9397/tid=9407] ppid=9394 vsize=2075072 CPUtime=1179.08 cores=1,3,5,7
/proc/9397/task/9407/stat : 9407 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 123536 0 0 0 117861 47 0 0 25 0 5 0 498160151 2124873728 471237 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=9397/tid=9408] ppid=9394 vsize=2075072 CPUtime=1176.14 cores=1,3,5,7
/proc/9397/task/9408/stat : 9408 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 46618 0 0 0 117453 161 0 0 25 0 5 0 498160151 2124873728 471237 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9397/tid=9409] ppid=9394 vsize=2075072 CPUtime=1179.17 cores=1,3,5,7
/proc/9397/task/9409/stat : 9409 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 240821 0 0 0 117813 104 0 0 25 0 5 0 498160151 2124873728 471237 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4713.9
Current children cumulated vsize (KiB) 2075072



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.9 s]
/proc/loadavg: 8.10 8.07 8.01 9/186 9444
/proc/meminfo: memFree=15945120/32951124 swapFree=67111524/67111528
[pid=9397] ppid=9394 vsize=2090712 CPUtime=4800.19 cores=1,3,5,7
/proc/9397/stat : 9397 (ttsth-5-4) S 9394 9397 9362 0 -1 4202496 475152 0 0 0 479511 508 0 0 21 0 5 0 498159939 2140889088 475070 33554432000 4194304 4338100 140737232303104 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9397/statm: 522678 475070 198 36 0 520583 0
[pid=9397/tid=9406] ppid=9394 vsize=2090712 CPUtime=1198.83 cores=1,3,5,7
/proc/9397/task/9406/stat : 9406 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 59525 0 0 0 119692 191 0 0 25 0 5 0 498160151 2140889088 475070 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9397/tid=9407] ppid=9394 vsize=2090712 CPUtime=1200.69 cores=1,3,5,7
/proc/9397/task/9407/stat : 9407 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 124214 0 0 0 120022 47 0 0 25 0 5 0 498160151 2140889088 475070 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=9397/tid=9408] ppid=9394 vsize=2090712 CPUtime=1197.74 cores=1,3,5,7
/proc/9397/task/9408/stat : 9408 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 46624 0 0 0 119612 162 0 0 25 0 5 0 498160151 2140889088 475070 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9397/tid=9409] ppid=9394 vsize=2090712 CPUtime=1200.76 cores=1,3,5,7
/proc/9397/task/9409/stat : 9409 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 243963 0 0 0 119971 105 0 0 25 0 5 0 498160151 2140889088 475070 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.19
Current children cumulated vsize (KiB) 2090712

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.9 s]
/proc/loadavg: 8.10 8.07 8.01 9/186 9444
/proc/meminfo: memFree=15945120/32951124 swapFree=67111524/67111528
[pid=9397] ppid=9394 vsize=2090712 CPUtime=4800.19 cores=1,3,5,7
/proc/9397/stat : 9397 (ttsth-5-4) S 9394 9397 9362 0 -1 4202496 475152 0 0 0 479511 508 0 0 21 0 5 0 498159939 2140889088 475070 33554432000 4194304 4338100 140737232303104 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9397/statm: 522678 475070 198 36 0 520583 0
[pid=9397/tid=9406] ppid=9394 vsize=2090712 CPUtime=1198.83 cores=1,3,5,7
/proc/9397/task/9406/stat : 9406 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 59525 0 0 0 119692 191 0 0 25 0 5 0 498160151 2140889088 475070 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9397/tid=9407] ppid=9394 vsize=2090712 CPUtime=1200.69 cores=1,3,5,7
/proc/9397/task/9407/stat : 9407 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 124214 0 0 0 120022 47 0 0 25 0 5 0 498160151 2140889088 475070 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=9397/tid=9408] ppid=9394 vsize=2090712 CPUtime=1197.74 cores=1,3,5,7
/proc/9397/task/9408/stat : 9408 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 46624 0 0 0 119612 162 0 0 25 0 5 0 498160151 2140889088 475070 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9397/tid=9409] ppid=9394 vsize=2090712 CPUtime=1200.76 cores=1,3,5,7
/proc/9397/task/9409/stat : 9409 (ttsth-5-4) R 9394 9397 9362 0 -1 4202560 243963 0 0 0 119971 105 0 0 25 0 5 0 498160151 2140889088 475070 33554432000 4194304 4338100 140737232303104 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.19
Current children cumulated vsize (KiB) 2090712

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1203.99
CPU time (s): 4800.29
CPU user time (s): 4795.14
CPU system time (s): 5.15821
CPU usage (%): 398.699
Max. virtual memory (cumulated for all children) (KiB): 2090712

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4795.14
system time used= 5.15822
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 475152
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2341
involuntary context switches= 29771

runsolver used 1.86972 second user time and 4.70328 second system time

The end

Launcher Data 

Begin job on node144 at 2011-04-29 08:10:38
IDJOB=3273545
IDBENCH=83141
IDSOLVER=1654
FILE ID=node144/3273545-1304057438
RUNJOBID= node144-1304057438-9381
PBS_JOBID= 13170439
Free space on /tmp= 72412 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/spence/sgen/sgen3-n180-s34713902-sat.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3273545-1304057438/watcher-3273545-1304057438 -o /tmp/evaluation-result-3273545-1304057438/solver-3273545-1304057438 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3273545-1304057438.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 19292459de25c4f5e9e6f187c00d549d
RANDOM SEED=1010862323

node144.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.16
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      19560512 kB
Buffers:       2206876 kB
Cached:       10554308 kB
SwapCached:          4 kB
Active:        4678656 kB
Inactive:      8146248 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      19560512 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:             148 kB
Writeback:           0 kB
AnonPages:       62500 kB
Mapped:          14320 kB
Slab:           499660 kB
PageTables:       5028 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   182712 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72408 MiB
End job on node144 at 2011-04-29 08:30:44