Trace number 3259890

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.17 1206.58

General information on the benchmark

Namecrafted/mosoi/
289/289-sat-6x19.cnf
MD5SUM582c5df8c18789e4111dbdb4b380081b
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.009997
Satisfiable
(Un)Satisfiability was proved
Number of variables456
Number of clauses11058
Sum of the clauses size42864
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2684
Number of clauses of size 30
Number of clauses of size 410374
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.01	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259890-1304060359.cnf 
0.00/0.01	c Using SATTIMEOUT of 4800
0.00/0.01	c Using SATRAM of 15500
0.00/0.01	c Size of subset node = 32
0.00/0.01	c Variables = 456
0.00/0.01	c Clauses = 11058
0.00/0.01	c Literals = 42864
0.00/0.01	c Reduced size:
0.00/0.01	c Variables = 456
0.00/0.01	c Clauses = 11058
0.00/0.01	c Literals = 42864
0.00/0.01	c a2
0.00/0.03	c a1
0.49/0.53	c Independent sub-problems = 1
0.49/0.53	c Sub-problem with 456 variables and 11058 clauses
0.49/0.53	c using permutation method 4
0.49/0.53	c count = 2
0.49/0.53	c Ordering finished 0.520 seconds
0.49/0.53	c mapped 0.520 seconds
0.49/0.53	c permuted 0.520 seconds
0.49/0.53	c about to sort
0.49/0.53	c sorted
0.49/0.55	c clauses now 11058
0.49/0.55	c init finished 0.530 seconds
0.49/0.55	c tree built 0.540 seconds
0.49/0.55	c 
0.49/0.55	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.49/0.55	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.49/0.55	c ................................ | ..........                    |                                  0.5         0.0             
0.49/0.55	c ................................ | ..........                  1 |                                  0.5         1.9             
0.49/0.55	c ................................ | ..........                  2 |                                  0.5         3.7             
0.49/0.55	c ................................ | ..........                  3 |                                  0.5         5.6             
0.49/0.55	c ................................ | ..........                  4 |                                  0.5         7.4             
0.49/0.55	c ................................ | ..........                  5 |                                  0.5         9.3             
0.49/0.55	c ................................ | ..........               1000 | 9            1691                0.6      1785.7         9277
96.02/24.51	c ................................ | ..........             783000 | 185535       1416445            96.1      8149.5     10391329
191.91/48.68	c ................................ | ..........            1319000 | 321727       2285914           192.2      6861.2     13848083
287.84/72.74	c ................................ | ..........            1773000 | 448366       2918264           288.0      6155.8     16853505
384.21/96.90	c ................................ | ..........            2249000 | 574853       3609937           384.2      5853.6     19351641
480.14/121.02	c ................................ | *.........            2703000 | 701870       4200913           480.2      5628.7     22320393
575.72/145.08	c ................................ | *.........            3102000 | 817783       4711229           576.0      5384.9     25094054
672.02/169.24	c ................................ | *.........            3511000 | 939304       5209352           672.2      5223.3     27286792
767.89/193.39	c ................................ | *.........            3952000 | 1071471      5689230           768.3      5144.0     30327190
863.98/217.49	c ................................ | *.........            4256000 | 1156190      6071028           864.3      4924.0     31729498
959.95/241.51	c ................................ | **........            4590000 | 1249689      6484820           960.0      4781.2     32888614
1055.84/265.65	c ................................ | **........            4887000 | 1333387      6844586          1056.1      4627.5     34744268
1151.71/289.79	c ................................ | **........            5135000 | 1400432      7169859          1152.1      4457.2     35852801
1247.71/313.89	c ................................ | **........            5463000 | 1490566      7608530          1248.1      4377.1     36586508
1344.05/338.02	c ................................ | **........            5747000 | 1571751      7934319          1344.2      4275.6     38490489
1440.01/362.16	c ................................ | ***.......            5999000 | 1643400      8222258          1440.3      4165.2     39070653
1535.92/386.25	c ................................ | ***.......            6329000 | 1735489      8636159          1536.1      4120.1     39416570
1632.07/410.40	c ................................ | ***.......            6611000 | 1815648      8953519          1632.1      4050.6     39749918
1728.05/434.55	c ................................ | ***.......            6898000 | 1898298      9270955          1728.3      3991.3     40331808
1823.92/458.64	c ................................ | ***.......            7239000 | 1996495      9684893          1824.1      3968.6     41240099
1919.97/482.84	c ................................ | ****......            7499000 | 2076871      9937694          1920.1      3905.5     41682463
2016.19/507.02	c ................................ | ****......            7811000 | 2172052      10255771         2016.3      3874.0     42327194
2111.86/531.14	c ................................ | ****......            8128000 | 2269493      10577860         2112.0      3848.4     42457529
2208.01/555.32	c ................................ | ****......            8517000 | 2390863      10910134         2208.1      3857.1     43377149
2304.08/579.47	c ................................ | ****......            8770000 | 2468465      11160892         2304.4      3805.8     44352041
2400.07/603.51	c ................................ | *****.....            9006000 | 2536491      11410596         2400.1      3752.3     45011109
2495.75/627.58	c ................................ | *****.....            9252000 | 2607743      11668757         2496.1      3706.6     46416283
2591.86/651.68	c ................................ | *****.....            9505000 | 2679370      11924919         2592.2      3666.8     47483458
2688.20/675.83	c ................................ | *****.....            9721000 | 2739260      12150908         2688.3      3616.0     48767801
2784.25/699.92	c ................................ | *****.....            9999000 | 2812645      12488611         2784.3      3591.2     49577152
2880.08/724.03	c ................................ | ******....           10231000 | 2878130      12765385         2880.2      3552.2     50080503
2976.40/748.21	c ................................ | ******....           10477000 | 2950239      12995392         2976.4      3520.0     50333696
3072.42/772.30	c ................................ | ******....           10704000 | 3012888      13234559         3072.4      3483.9     51381985
3167.88/796.34	c ................................ | ******....           10962000 | 3083331      13550736         3168.0      3460.2     51620496
3264.22/820.52	c ................................ | ******....           11216000 | 3153926      13807486         3264.3      3436.0     52300625
3360.07/844.67	c ................................ | *******...           11471000 | 3226410      14078683         3360.4      3413.6     53688050
3455.68/868.69	c ................................ | *******...           11725000 | 3296992      14347541         3456.1      3392.6     54292028
3552.20/892.92	c ................................ | *******...           12002000 | 3373866      14691955         3552.3      3378.7     54432637
3647.71/916.98	c ................................ | *******...           12256000 | 3452003      14918958         3648.0      3359.6     54432637
3743.72/941.09	c ................................ | *******...           12534000 | 3533652      15198479         3744.1      3347.7     54432637
3840.06/965.24	c ................................ | ********..           12819000 | 3617260      15474668         3840.2      3338.1     54432637
3936.00/989.31	c ................................ | ********..           13095000 | 3701063      15728694         3936.1      3326.9     54499807
4032.05/1013.53	c ................................ | ********..           13437000 | 3803979      16024153         4032.2      3332.4     55025027
4128.06/1037.63	c ................................ | ********..           13679000 | 3876777      16248923         4128.2      3313.6     55832864
4224.14/1061.82	c ................................ | ********..           13886000 | 3936277      16458185         4224.2      3287.2     56699331
4320.01/1085.96	c ................................ | *********.           14129000 | 4000755      16740111         4320.3      3270.4     56924412
4416.06/1110.06	c ................................ | *********.           14396000 | 4076370      17035035         4416.3      3259.7     57490316
4512.06/1134.15	c ................................ | *********.           14631000 | 4141552      17268328         4512.2      3242.5     57894053
4608.12/1158.24	c ................................ | *********.           14865000 | 4207217      17515876         4608.3      3225.7     58839176
4703.87/1182.36	c ................................ | *********.           15078000 | 4264457      17765888         4704.1      3205.3     60249898
4799.73/1206.48	c CPU limit (4800) reached
4799.73/1206.48	c Time = 4800.07 seconds
4799.73/1206.48	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3259890-1304060359/watcher-3259890-1304060359 -o /tmp/evaluation-result-3259890-1304060359/solver-3259890-1304060359 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259890-1304060359.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.43 7.86 7.89 5/181 28467
/proc/meminfo: memFree=13617392/32950928 swapFree=67111528/67111528
[pid=28467] ppid=28465 vsize=9408 CPUtime=0 cores=1,3,5,7
/proc/28467/stat : 28467 (ttsth-5-4) R 28465 28467 28330 0 -1 4202496 406 0 0 0 0 0 0 0 25 0 1 0 596266562 9633792 349 33554432000 4194304 4338100 140734864416480 18446744073709551615 218880225989 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/28467/statm: 2352 349 147 36 0 257 0

[startup+0.0472599 s]
/proc/loadavg: 7.43 7.86 7.89 5/181 28467
/proc/meminfo: memFree=13617392/32950928 swapFree=67111528/67111528
[pid=28467] ppid=28465 vsize=12084 CPUtime=0.04 cores=1,3,5,7
/proc/28467/stat : 28467 (ttsth-5-4) R 28465 28467 28330 0 -1 4202496 1106 0 0 0 4 0 0 0 25 0 1 0 596266562 12374016 1048 33554432000 4194304 4338100 140734864416480 18446744073709551615 4245229 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/28467/statm: 3021 1048 154 36 0 926 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 12084

[startup+0.100249 s]
/proc/loadavg: 7.43 7.86 7.89 5/181 28467
/proc/meminfo: memFree=13617392/32950928 swapFree=67111528/67111528
[pid=28467] ppid=28465 vsize=12084 CPUtime=0.09 cores=1,3,5,7
/proc/28467/stat : 28467 (ttsth-5-4) R 28465 28467 28330 0 -1 4202496 1107 0 0 0 9 0 0 0 25 0 1 0 596266562 12374016 1049 33554432000 4194304 4338100 140734864416480 18446744073709551615 4295924 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/28467/statm: 3021 1049 155 36 0 926 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 12084

[startup+0.300213 s]
/proc/loadavg: 7.43 7.86 7.89 5/181 28467
/proc/meminfo: memFree=13617392/32950928 swapFree=67111528/67111528
[pid=28467] ppid=28465 vsize=12084 CPUtime=0.29 cores=1,3,5,7
/proc/28467/stat : 28467 (ttsth-5-4) R 28465 28467 28330 0 -1 4202496 1107 0 0 0 29 0 0 0 25 0 1 0 596266562 12374016 1049 33554432000 4194304 4338100 140734864416480 18446744073709551615 4295924 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/28467/statm: 3021 1049 155 36 0 926 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 12084

[startup+0.701071 s]
/proc/loadavg: 7.43 7.86 7.89 5/181 28467
/proc/meminfo: memFree=13617392/32950928 swapFree=67111528/67111528
[pid=28467] ppid=28465 vsize=201308 CPUtime=1.14 cores=1,3,5,7
/proc/28467/stat : 28467 (ttsth-5-4) S 28465 28467 28330 0 -1 4202496 5007 0 0 0 113 1 0 0 25 0 5 0 596266562 206139392 4805 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28467/statm: 50327 4805 175 36 0 48232 0
Current children cumulated CPU time (s) 1.14
Current children cumulated vsize (KiB) 201308

[startup+1.50093 s]
/proc/loadavg: 7.43 7.86 7.89 9/186 28472
/proc/meminfo: memFree=13555484/32950928 swapFree=67111528/67111528
[pid=28467] ppid=28465 vsize=239504 CPUtime=4.33 cores=1,3,5,7
/proc/28467/stat : 28467 (ttsth-5-4) S 28465 28467 28330 0 -1 4202496 12984 0 0 0 430 3 0 0 25 0 5 0 596266562 245252096 12782 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28467/statm: 59876 12782 175 36 0 57781 0
[pid=28467/tid=28469] ppid=28465 vsize=239504 CPUtime=0.94 cores=1,3,5,7
/proc/28467/task/28469/stat : 28469 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 2658 0 0 0 94 0 0 0 19 0 5 0 596266617 245252096 12782 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28467/tid=28470] ppid=28465 vsize=239504 CPUtime=0.94 cores=1,3,5,7
/proc/28467/task/28470/stat : 28470 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 2524 0 0 0 94 0 0 0 19 0 5 0 596266617 245252096 12782 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28467/tid=28471] ppid=28465 vsize=239504 CPUtime=0.93 cores=1,3,5,7
/proc/28467/task/28471/stat : 28471 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 2612 0 0 0 93 0 0 0 25 0 5 0 596266617 245252096 12782 33554432000 4194304 4338100 140734864416480 18446744073709551615 218880261685 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28467/tid=28472] ppid=28465 vsize=239504 CPUtime=0.93 cores=1,3,5,7
/proc/28467/task/28472/stat : 28472 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 2865 0 0 0 93 0 0 0 25 0 5 0 596266617 245252096 12782 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4.33
Current children cumulated vsize (KiB) 239504

[startup+3.10066 s]
/proc/loadavg: 7.43 7.86 7.89 9/186 28472
/proc/meminfo: memFree=13502908/32950928 swapFree=67111528/67111528
[pid=28467] ppid=28465 vsize=268580 CPUtime=10.71 cores=1,3,5,7
/proc/28467/stat : 28467 (ttsth-5-4) S 28465 28467 28330 0 -1 4202496 22085 0 0 0 1066 5 0 0 25 0 5 0 596266562 275025920 21883 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28467/statm: 67145 21883 175 36 0 65050 0
[pid=28467/tid=28469] ppid=28465 vsize=268580 CPUtime=2.54 cores=1,3,5,7
/proc/28467/task/28469/stat : 28469 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 4733 0 0 0 253 1 0 0 18 0 5 0 596266617 275025920 21883 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28467/tid=28470] ppid=28465 vsize=268580 CPUtime=2.54 cores=1,3,5,7
/proc/28467/task/28470/stat : 28470 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 4676 0 0 0 253 1 0 0 22 0 5 0 596266617 275025920 21883 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28467/tid=28471] ppid=28465 vsize=268580 CPUtime=2.52 cores=1,3,5,7
/proc/28467/task/28471/stat : 28471 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 4911 0 0 0 252 0 0 0 25 0 5 0 596266617 275025920 21884 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28467/tid=28472] ppid=28465 vsize=268580 CPUtime=2.53 cores=1,3,5,7
/proc/28467/task/28472/stat : 28472 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 5441 0 0 0 252 1 0 0 25 0 5 0 596266617 275025920 21884 33554432000 4194304 4338100 140734864416480 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 10.71
Current children cumulated vsize (KiB) 268580

[startup+6.3011 s]
/proc/loadavg: 7.47 7.86 7.89 9/186 28472
/proc/meminfo: memFree=13366740/32950928 swapFree=67111528/67111528
[pid=28467] ppid=28465 vsize=382428 CPUtime=23.47 cores=1,3,5,7
/proc/28467/stat : 28467 (ttsth-5-4) S 28465 28467 28330 0 -1 4202496 34213 0 0 0 2328 19 0 0 25 0 5 0 596266562 391606272 34011 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28467/statm: 95607 34011 175 36 0 93512 0
[pid=28467/tid=28469] ppid=28465 vsize=382428 CPUtime=5.72 cores=1,3,5,7
/proc/28467/task/28469/stat : 28469 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 7046 0 0 0 567 5 0 0 25 0 5 0 596266617 391606272 34011 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28467/tid=28470] ppid=28465 vsize=382428 CPUtime=5.73 cores=1,3,5,7
/proc/28467/task/28470/stat : 28470 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 7436 0 0 0 568 5 0 0 18 0 5 0 596266617 391606272 34011 33554432000 4194304 4338100 140734864416480 18446744073709551615 4253974 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28467/tid=28471] ppid=28465 vsize=382428 CPUtime=5.73 cores=1,3,5,7
/proc/28467/task/28471/stat : 28471 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 8470 0 0 0 570 3 0 0 25 0 5 0 596266617 391606272 34011 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28467/tid=28472] ppid=28465 vsize=382428 CPUtime=5.71 cores=1,3,5,7
/proc/28467/task/28472/stat : 28472 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 8936 0 0 0 567 4 0 0 25 0 5 0 596266617 391606272 34011 33554432000 4194304 4338100 140734864416480 18446744073709551615 218879960129 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 23.47
Current children cumulated vsize (KiB) 382428

[startup+12.701 s]
/proc/loadavg: 7.51 7.86 7.89 9/186 28472
/proc/meminfo: memFree=13082128/32950928 swapFree=67111528/67111528
[pid=28467] ppid=28465 vsize=480056 CPUtime=48.98 cores=1,3,5,7
/proc/28467/stat : 28467 (ttsth-5-4) S 28465 28467 28330 0 -1 4202496 58626 0 0 0 4873 25 0 0 25 0 5 0 596266562 491577344 58424 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28467/statm: 120014 58424 176 36 0 117919 0
[pid=28467/tid=28469] ppid=28465 vsize=480056 CPUtime=12.11 cores=1,3,5,7
/proc/28467/task/28469/stat : 28469 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 12748 0 0 0 1204 7 0 0 25 0 5 0 596266617 491577344 58424 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28467/tid=28470] ppid=28465 vsize=480056 CPUtime=12.12 cores=1,3,5,7
/proc/28467/task/28470/stat : 28470 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 13087 0 0 0 1205 7 0 0 25 0 5 0 596266617 491577344 58424 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28467/tid=28471] ppid=28465 vsize=480056 CPUtime=12.1 cores=1,3,5,7
/proc/28467/task/28471/stat : 28471 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 14326 0 0 0 1207 3 0 0 25 0 5 0 596266617 491577344 58424 33554432000 4194304 4338100 140734864416480 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28467/tid=28472] ppid=28465 vsize=480056 CPUtime=12.06 cores=1,3,5,7
/proc/28467/task/28472/stat : 28472 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 16140 0 0 0 1200 6 0 0 25 0 5 0 596266617 491577344 58424 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 48.98
Current children cumulated vsize (KiB) 480056

[startup+25.5008 s]
/proc/loadavg: 7.77 7.90 7.90 9/186 28473
/proc/meminfo: memFree=12577328/32950928 swapFree=67111528/67111528
[pid=28467] ppid=28465 vsize=604088 CPUtime=100.01 cores=1,3,5,7
/proc/28467/stat : 28467 (ttsth-5-4) S 28465 28467 28330 0 -1 4202496 91871 0 0 0 9940 61 0 0 25 0 5 0 596266562 618586112 91669 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28467/statm: 151022 91669 176 36 0 148927 0
[pid=28467/tid=28469] ppid=28465 vsize=604088 CPUtime=24.88 cores=1,3,5,7
/proc/28467/task/28469/stat : 28469 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 20286 0 0 0 2470 18 0 0 25 0 5 0 596266617 618586112 91669 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28467/tid=28470] ppid=28465 vsize=604088 CPUtime=24.9 cores=1,3,5,7
/proc/28467/task/28470/stat : 28470 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 20566 0 0 0 2472 18 0 0 25 0 5 0 596266617 618586112 91669 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28467/tid=28471] ppid=28465 vsize=604088 CPUtime=24.87 cores=1,3,5,7
/proc/28467/task/28471/stat : 28471 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 23685 0 0 0 2477 10 0 0 25 0 5 0 596266617 618586112 91669 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28467/tid=28472] ppid=28465 vsize=604088 CPUtime=24.78 cores=1,3,5,7
/proc/28467/task/28472/stat : 28472 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 25009 0 0 0 2465 13 0 0 25 0 5 0 596266617 618586112 91669 33554432000 4194304 4338100 140734864416480 18446744073709551615 4240524 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 100.01
Current children cumulated vsize (KiB) 604088

[startup+51.1003 s]
/proc/loadavg: 7.85 7.91 7.90 9/186 28473
/proc/meminfo: memFree=12180024/32950928 swapFree=67111528/67111528
[pid=28467] ppid=28465 vsize=723212 CPUtime=201.85 cores=1,3,5,7
/proc/28467/stat : 28467 (ttsth-5-4) S 28465 28467 28330 0 -1 4202496 123678 0 0 0 20077 108 0 0 25 0 5 0 596266562 740569088 123476 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28467/statm: 180803 123476 176 36 0 178708 0
[pid=28467/tid=28469] ppid=28465 vsize=723212 CPUtime=50.33 cores=1,3,5,7
/proc/28467/task/28469/stat : 28469 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 30365 0 0 0 5001 32 0 0 25 0 5 0 596266617 740569088 123476 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28467/tid=28470] ppid=28465 vsize=723212 CPUtime=50.38 cores=1,3,5,7
/proc/28467/task/28470/stat : 28470 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 26021 0 0 0 5010 28 0 0 25 0 5 0 596266617 740569088 123476 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28467/tid=28471] ppid=28465 vsize=723212 CPUtime=50.4 cores=1,3,5,7
/proc/28467/task/28471/stat : 28471 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 31202 0 0 0 5018 22 0 0 25 0 5 0 596266617 740569088 123476 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28467/tid=28472] ppid=28465 vsize=723212 CPUtime=50.15 cores=1,3,5,7
/proc/28467/task/28472/stat : 28472 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 33765 0 0 0 4991 24 0 0 25 0 5 0 596266617 740569088 123476 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 201.85
Current children cumulated vsize (KiB) 723212

[startup+102.307 s]
/proc/loadavg: 7.99 7.94 7.91 9/187 28476
/proc/meminfo: memFree=11513624/32950928 swapFree=67111528/67111528
[pid=28467] ppid=28465 vsize=934292 CPUtime=405.7 cores=1,3,5,7
/proc/28467/stat : 28467 (ttsth-5-4) S 28465 28467 28330 0 -1 4202496 179573 0 0 0 40367 203 0 0 25 0 5 0 596266562 956715008 179371 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28467/statm: 233573 179371 176 36 0 231478 0
[pid=28467/tid=28469] ppid=28465 vsize=934292 CPUtime=101.13 cores=1,3,5,7
/proc/28467/task/28469/stat : 28469 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 43307 0 0 0 10058 55 0 0 25 0 5 0 596266617 956715008 179371 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28467/tid=28470] ppid=28465 vsize=934292 CPUtime=101.34 cores=1,3,5,7
/proc/28467/task/28470/stat : 28470 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 34574 0 0 0 10081 53 0 0 25 0 5 0 596266617 956715008 179371 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28467/tid=28471] ppid=28465 vsize=934292 CPUtime=101.43 cores=1,3,5,7
/proc/28467/task/28471/stat : 28471 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 48501 0 0 0 10099 44 0 0 25 0 5 0 596266617 956715008 179371 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28467/tid=28472] ppid=28465 vsize=934292 CPUtime=101.2 cores=1,3,5,7
/proc/28467/task/28472/stat : 28472 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 50866 0 0 0 10072 48 0 0 18 0 5 0 596266617 956715008 179371 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 405.7
Current children cumulated vsize (KiB) 934292

[startup+162.301 s]
/proc/loadavg: 7.99 7.95 7.91 9/186 28485
/proc/meminfo: memFree=10508136/32950928 swapFree=67111528/67111528
[pid=28467] ppid=28465 vsize=1147840 CPUtime=644.57 cores=1,3,5,7
/proc/28467/stat : 28467 (ttsth-5-4) S 28465 28467 28330 0 -1 4202496 235944 0 0 0 64125 332 0 0 25 0 5 0 596266562 1175388160 235742 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28467/statm: 286960 235742 176 36 0 284865 0
[pid=28467/tid=28469] ppid=28465 vsize=1147840 CPUtime=160.63 cores=1,3,5,7
/proc/28467/task/28469/stat : 28469 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 58981 0 0 0 15971 92 0 0 25 0 5 0 596266617 1175388160 235742 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28467/tid=28470] ppid=28465 vsize=1147840 CPUtime=161.11 cores=1,3,5,7
/proc/28467/task/28470/stat : 28470 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 47073 0 0 0 16017 94 0 0 25 0 5 0 596266617 1175388160 235742 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28467/tid=28471] ppid=28465 vsize=1147840 CPUtime=161.22 cores=1,3,5,7
/proc/28467/task/28471/stat : 28471 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 65722 0 0 0 16051 71 0 0 25 0 5 0 596266617 1175388160 235742 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28467/tid=28472] ppid=28465 vsize=1147840 CPUtime=161.04 cores=1,3,5,7
/proc/28467/task/28472/stat : 28472 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 61843 0 0 0 16031 73 0 0 25 0 5 0 596266617 1175388160 235742 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 644.57
Current children cumulated vsize (KiB) 1147840


################
# More data... #
################

/proc/28467/task/28472/stat : 28472 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 111149 0 0 0 69619 283 0 0 25 0 5 0 596266617 1944113152 442594 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 2793.83
Current children cumulated vsize (KiB) 1898548

[startup+762.307 s]
/proc/loadavg: 8.09 8.06 7.97 9/186 28501
/proc/meminfo: memFree=6783684/32950928 swapFree=67111528/67111528
[pid=28467] ppid=28465 vsize=1935068 CPUtime=3032.6 cores=1,3,5,7
/proc/28467/stat : 28467 (ttsth-5-4) S 28465 28467 28330 0 -1 4202496 454391 0 0 0 301848 1412 0 0 25 0 5 0 596266562 1981509632 454189 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28467/statm: 483767 454189 176 36 0 481672 0
[pid=28467/tid=28469] ppid=28465 vsize=1935068 CPUtime=757.01 cores=1,3,5,7
/proc/28467/task/28469/stat : 28469 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 134688 0 0 0 75287 414 0 0 25 0 5 0 596266617 1981509632 454189 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28467/tid=28470] ppid=28465 vsize=1935068 CPUtime=758.02 cores=1,3,5,7
/proc/28467/task/28470/stat : 28470 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 85357 0 0 0 75390 412 0 0 25 0 5 0 596266617 1981509632 454189 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28467/tid=28471] ppid=28465 vsize=1935068 CPUtime=758.42 cores=1,3,5,7
/proc/28467/task/28471/stat : 28471 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 119269 0 0 0 75552 290 0 0 25 0 5 0 596266617 1981509632 454189 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28467/tid=28472] ppid=28465 vsize=1935068 CPUtime=758.57 cores=1,3,5,7
/proc/28467/task/28472/stat : 28472 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 112752 0 0 0 75563 294 0 0 25 0 5 0 596266617 1981509632 454189 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3032.6
Current children cumulated vsize (KiB) 1935068

[startup+822.301 s]
/proc/loadavg: 8.03 8.05 7.97 9/186 28503
/proc/meminfo: memFree=6625472/32950928 swapFree=67111528/67111528
[pid=28467] ppid=28465 vsize=1990620 CPUtime=3271.39 cores=1,3,5,7
/proc/28467/stat : 28467 (ttsth-5-4) S 28465 28467 28330 0 -1 4202496 469655 0 0 0 325598 1541 0 0 25 0 5 0 596266562 2038394880 469453 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28467/statm: 497655 469453 176 36 0 495560 0
[pid=28467/tid=28469] ppid=28465 vsize=1990620 CPUtime=816.38 cores=1,3,5,7
/proc/28467/task/28469/stat : 28469 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 138974 0 0 0 81187 451 0 0 25 0 5 0 596266617 2038394880 469453 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28467/tid=28470] ppid=28465 vsize=1990620 CPUtime=817.77 cores=1,3,5,7
/proc/28467/task/28470/stat : 28470 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 87344 0 0 0 81326 451 0 0 25 0 5 0 596266617 2038394880 469453 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28467/tid=28471] ppid=28465 vsize=1990620 CPUtime=818.24 cores=1,3,5,7
/proc/28467/task/28471/stat : 28471 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 123359 0 0 0 81508 316 0 0 25 0 5 0 596266617 2038394880 469453 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28467/tid=28472] ppid=28465 vsize=1990620 CPUtime=818.43 cores=1,3,5,7
/proc/28467/task/28472/stat : 28472 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 117653 0 0 0 81521 322 0 0 25 0 5 0 596266617 2038394880 469453 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3271.39
Current children cumulated vsize (KiB) 1990620

[startup+882.301 s]
/proc/loadavg: 8.12 8.07 7.98 9/186 28505
/proc/meminfo: memFree=6442788/32950928 swapFree=67111528/67111528
[pid=28467] ppid=28465 vsize=2070652 CPUtime=3509.9 cores=1,3,5,7
/proc/28467/stat : 28467 (ttsth-5-4) S 28465 28467 28330 0 -1 4202496 491154 0 0 0 349328 1662 0 0 25 0 5 0 596266562 2120347648 490952 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28467/statm: 517663 490952 176 36 0 515568 0
[pid=28467/tid=28469] ppid=28465 vsize=2070652 CPUtime=875.89 cores=1,3,5,7
/proc/28467/task/28469/stat : 28469 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 146455 0 0 0 87122 467 0 0 25 0 5 0 596266617 2120347648 490952 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28467/tid=28470] ppid=28465 vsize=2070652 CPUtime=877.4 cores=1,3,5,7
/proc/28467/task/28470/stat : 28470 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 89730 0 0 0 87249 491 0 0 25 0 5 0 596266617 2120347648 490952 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28467/tid=28471] ppid=28465 vsize=2070652 CPUtime=877.8 cores=1,3,5,7
/proc/28467/task/28471/stat : 28471 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 131247 0 0 0 87430 350 0 0 25 0 5 0 596266617 2120347648 490952 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28467/tid=28472] ppid=28465 vsize=2070652 CPUtime=878.23 cores=1,3,5,7
/proc/28467/task/28472/stat : 28472 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 121397 0 0 0 87471 352 0 0 25 0 5 0 596266617 2120347648 490952 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3509.9
Current children cumulated vsize (KiB) 2070652

[startup+942.3 s]
/proc/loadavg: 8.15 8.08 7.99 9/186 28506
/proc/meminfo: memFree=6415324/32950928 swapFree=67111528/67111528
[pid=28467] ppid=28465 vsize=2073028 CPUtime=3748.9 cores=1,3,5,7
/proc/28467/stat : 28467 (ttsth-5-4) S 28465 28467 28330 0 -1 4202496 494326 0 0 0 373090 1800 0 0 25 0 5 0 596266562 2122780672 494124 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28467/statm: 518257 494124 176 36 0 516162 0
[pid=28467/tid=28469] ppid=28465 vsize=2073028 CPUtime=935.59 cores=1,3,5,7
/proc/28467/task/28469/stat : 28469 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 147507 0 0 0 93050 509 0 0 25 0 5 0 596266617 2122780672 494124 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28467/tid=28470] ppid=28465 vsize=2073028 CPUtime=937.07 cores=1,3,5,7
/proc/28467/task/28470/stat : 28470 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 90381 0 0 0 93172 535 0 0 25 0 5 0 596266617 2122780672 494124 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28467/tid=28471] ppid=28465 vsize=2073028 CPUtime=937.56 cores=1,3,5,7
/proc/28467/task/28471/stat : 28471 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 131860 0 0 0 93380 376 0 0 25 0 5 0 596266617 2122780672 494124 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28467/tid=28472] ppid=28465 vsize=2073028 CPUtime=938.11 cores=1,3,5,7
/proc/28467/task/28472/stat : 28472 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 122253 0 0 0 93433 378 0 0 25 0 5 0 596266617 2122780672 494124 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3748.9
Current children cumulated vsize (KiB) 2073028

[startup+1002.3 s]
/proc/loadavg: 8.11 8.08 8.00 9/186 28508
/proc/meminfo: memFree=6391484/32950928 swapFree=67111528/67111528
[pid=28467] ppid=28465 vsize=2081396 CPUtime=3987.6 cores=1,3,5,7
/proc/28467/stat : 28467 (ttsth-5-4) S 28465 28467 28330 0 -1 4202496 498988 0 0 0 396831 1929 0 0 25 0 5 0 596266562 2131349504 498786 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28467/statm: 520349 498786 176 36 0 518254 0
[pid=28467/tid=28469] ppid=28465 vsize=2081396 CPUtime=995.05 cores=1,3,5,7
/proc/28467/task/28469/stat : 28469 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 150213 0 0 0 98959 546 0 0 25 0 5 0 596266617 2131349504 498786 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28467/tid=28470] ppid=28465 vsize=2081396 CPUtime=996.68 cores=1,3,5,7
/proc/28467/task/28470/stat : 28470 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 90993 0 0 0 99096 572 0 0 25 0 5 0 596266617 2131349504 498786 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28467/tid=28471] ppid=28465 vsize=2081396 CPUtime=997.41 cores=1,3,5,7
/proc/28467/task/28471/stat : 28471 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 132444 0 0 0 99338 403 0 0 25 0 5 0 596266617 2131349504 498786 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28467/tid=28472] ppid=28465 vsize=2081396 CPUtime=997.88 cores=1,3,5,7
/proc/28467/task/28472/stat : 28472 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 123013 0 0 0 99382 406 0 0 25 0 5 0 596266617 2131349504 498786 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3987.6
Current children cumulated vsize (KiB) 2081396

[startup+1062.3 s]
/proc/loadavg: 8.12 8.09 8.00 9/186 28509
/proc/meminfo: memFree=6306664/32950928 swapFree=67111528/67111528
[pid=28467] ppid=28465 vsize=2145672 CPUtime=4226.14 cores=1,3,5,7
/proc/28467/stat : 28467 (ttsth-5-4) S 28465 28467 28330 0 -1 4202496 516692 0 0 0 420579 2035 0 0 25 0 5 0 596266562 2197168128 516490 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28467/statm: 536418 516490 176 36 0 534323 0
[pid=28467/tid=28469] ppid=28465 vsize=2145672 CPUtime=1054.52 cores=1,3,5,7
/proc/28467/task/28469/stat : 28469 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 159106 0 0 0 104879 573 0 0 25 0 5 0 596266617 2197168128 516490 33554432000 4194304 4338100 140734864416480 18446744073709551615 4235770 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28467/tid=28470] ppid=28465 vsize=2145672 CPUtime=1056.1 cores=1,3,5,7
/proc/28467/task/28470/stat : 28470 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 93954 0 0 0 105005 605 0 0 25 0 5 0 596266617 2197168128 516490 33554432000 4194304 4338100 140734864416480 18446744073709551615 4235770 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28467/tid=28471] ppid=28465 vsize=2145672 CPUtime=1057.23 cores=1,3,5,7
/proc/28467/task/28471/stat : 28471 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 136387 0 0 0 105297 426 0 0 25 0 5 0 596266617 2197168128 516490 33554432000 4194304 4338100 140734864416480 18446744073709551615 4235847 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28467/tid=28472] ppid=28465 vsize=2145672 CPUtime=1057.7 cores=1,3,5,7
/proc/28467/task/28472/stat : 28472 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 124920 0 0 0 105341 429 0 0 25 0 5 0 596266617 2197168128 516490 33554432000 4194304 4338100 140734864416480 18446744073709551615 4235746 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4226.14
Current children cumulated vsize (KiB) 2145672

[startup+1122.3 s]
/proc/loadavg: 8.09 8.09 8.00 9/186 28511
/proc/meminfo: memFree=6256136/32950928 swapFree=67111528/67111528
[pid=28467] ppid=28465 vsize=2188712 CPUtime=4464.96 cores=1,3,5,7
/proc/28467/stat : 28467 (ttsth-5-4) S 28465 28467 28330 0 -1 4202496 528719 0 0 0 444318 2178 0 0 25 0 5 0 596266562 2241241088 528517 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28467/statm: 547178 528517 176 36 0 545083 0
[pid=28467/tid=28469] ppid=28465 vsize=2188712 CPUtime=1114.21 cores=1,3,5,7
/proc/28467/task/28469/stat : 28469 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 161440 0 0 0 110807 614 0 0 25 0 5 0 596266617 2241241088 528517 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28467/tid=28470] ppid=28465 vsize=2188712 CPUtime=1115.58 cores=1,3,5,7
/proc/28467/task/28470/stat : 28470 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 94601 0 0 0 110915 643 0 0 25 0 5 0 596266617 2241241088 528517 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28467/tid=28471] ppid=28465 vsize=2188712 CPUtime=1117.09 cores=1,3,5,7
/proc/28467/task/28471/stat : 28471 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 139237 0 0 0 111253 456 0 0 25 0 5 0 596266617 2241241088 528517 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28467/tid=28472] ppid=28465 vsize=2188712 CPUtime=1117.5 cores=1,3,5,7
/proc/28467/task/28472/stat : 28472 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 131116 0 0 0 111286 464 0 0 25 0 5 0 596266617 2241241088 528517 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4464.96
Current children cumulated vsize (KiB) 2188712

[startup+1182.3 s]
/proc/loadavg: 8.11 8.10 8.01 9/186 28545
/proc/meminfo: memFree=12183360/32950928 swapFree=67111528/67111528
[pid=28467] ppid=28465 vsize=2462240 CPUtime=4703.87 cores=1,3,5,7
/proc/28467/stat : 28467 (ttsth-5-4) S 28465 28467 28330 0 -1 4202496 549429 0 0 0 468148 2239 0 0 25 0 5 0 596266562 2521333760 549227 33554432000 4194304 4338100 140734864416480 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28467/statm: 615560 549227 176 36 0 613465 0
[pid=28467/tid=28469] ppid=28465 vsize=2462240 CPUtime=1173.79 cores=1,3,5,7
/proc/28467/task/28469/stat : 28469 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 168463 0 0 0 116748 631 0 0 25 0 5 0 596266617 2521333760 549227 33554432000 4194304 4338100 140734864416480 18446744073709551615 218879960506 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28467/tid=28470] ppid=28465 vsize=2462240 CPUtime=1175.23 cores=1,3,5,7
/proc/28467/task/28470/stat : 28470 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 97199 0 0 0 116865 658 0 0 25 0 5 0 596266617 2521333760 549227 33554432000 4194304 4338100 140734864416480 18446744073709551615 4286296 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28467/tid=28471] ppid=28465 vsize=2462240 CPUtime=1176.86 cores=1,3,5,7
/proc/28467/task/28471/stat : 28471 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 144211 0 0 0 117216 470 0 0 25 0 5 0 596266617 2521333760 549227 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28467/tid=28472] ppid=28465 vsize=2462240 CPUtime=1177.41 cores=1,3,5,7
/proc/28467/task/28472/stat : 28472 (ttsth-5-4) R 28465 28467 28330 0 -1 4202560 137231 0 0 0 117263 478 0 0 25 0 5 0 596266617 2521333760 549227 33554432000 4194304 4338100 140734864416480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4703.87
Current children cumulated vsize (KiB) 2462240



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1206.5 s]
/proc/loadavg: 8.07 8.09 8.01 7/183 28545
/proc/meminfo: memFree=11835512/32950928 swapFree=67111528/67111528
[pid=28467] ppid=28465 vsize=0 CPUtime=4800.09 cores=1,3,5,7
/proc/28467/stat : 28467 (ttsth-5-4) Z 28465 28467 28330 0 -1 4203548 556598 0 0 0 477700 2309 0 0 18 0 2 0 596266562 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28467/statm: 0 0 0 0 0 0 0
[pid=28467/tid=28472] ppid=28465 vsize=0 CPUtime=1201.55 cores=1,3,5,7
/proc/28467/task/28472/stat : 28472 (ttsth-5-4) R 28465 28467 28330 0 -1 4203588 139611 0 0 0 119661 494 0 0 25 0 2 0 596266617 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.09
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1206.5 s]
/proc/loadavg: 8.07 8.09 8.01 7/183 28545
/proc/meminfo: memFree=11835512/32950928 swapFree=67111528/67111528
[pid=28467] ppid=28465 vsize=0 CPUtime=4800.09 cores=1,3,5,7
/proc/28467/stat : 28467 (ttsth-5-4) Z 28465 28467 28330 0 -1 4203548 556598 0 0 0 477700 2309 0 0 18 0 2 0 596266562 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28467/statm: 0 0 0 0 0 0 0
[pid=28467/tid=28472] ppid=28465 vsize=0 CPUtime=1201.55 cores=1,3,5,7
/proc/28467/task/28472/stat : 28472 (ttsth-5-4) R 28465 28467 28330 0 -1 4203588 139611 0 0 0 119661 494 0 0 25 0 2 0 596266617 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.09
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1206.58
CPU time (s): 4800.17
CPU user time (s): 4777
CPU system time (s): 23.1715
CPU usage (%): 397.832
Max. virtual memory (cumulated for all children) (KiB): 2482328

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4777
system time used= 23.1715
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 556598
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 73867
involuntary context switches= 31479

runsolver used 2.08168 second user time and 4.51331 second system time

The end

Launcher Data 

Begin job on node112 at 2011-04-29 08:59:19
IDJOB=3259890
IDBENCH=83158
IDSOLVER=1654
FILE ID=node112/3259890-1304060359
RUNJOBID= node112-1304059146-28349
PBS_JOBID= 13170443
Free space on /tmp= 72564 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/mosoi/289/289-sat-6x19.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3259890-1304060359/watcher-3259890-1304060359 -o /tmp/evaluation-result-3259890-1304060359/solver-3259890-1304060359 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259890-1304060359.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 582c5df8c18789e4111dbdb4b380081b
RANDOM SEED=137941966

node112.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5261.27
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.93
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      13617796 kB
Buffers:       1995240 kB
Cached:       15797676 kB
SwapCached:          0 kB
Active:        8017452 kB
Inactive:     10759108 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      13617796 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:            1092 kB
Writeback:           0 kB
AnonPages:      982524 kB
Mapped:          15216 kB
Slab:           492548 kB
PageTables:       6448 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:  1173908 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72564 MiB
End job on node112 at 2011-04-29 09:19:28