Trace number 3262706

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.51 1203.23

General information on the benchmark

NameSAT03/handmade/gomes/bqwh/
bqwh.50.825.shuffled-as.sat03-1648.cnf
MD5SUM05a191325d89de504821b55dea51ca7f
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark19.758
Satisfiable
(Un)Satisfiability was proved
Number of variables4979
Number of clauses43754
Sum of the clauses size97507
Maximum clause length12
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 241316
Number of clauses of size 3117
Number of clauses of size 4287
Number of clauses of size 5525
Number of clauses of size over 51509

Solver Data 

0.02/0.03	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262706-1304094101.cnf 
0.02/0.03	c Using SATTIMEOUT of 4800
0.02/0.03	c Using SATRAM of 15500
0.02/0.03	c Size of subset node = 32
0.02/0.03	c Variables = 4979
0.02/0.03	c Clauses = 43754
0.02/0.03	c Literals = 97507
0.02/0.03	c Reduced size:
0.02/0.03	c Variables = 4979
0.02/0.03	c Clauses = 43754
0.02/0.03	c Literals = 97507
0.02/0.03	c a2
0.02/0.07	c a1
1.00/1.09	c Independent sub-problems = 1
1.00/1.09	c Sub-problem with 4979 variables and 43754 clauses
1.00/1.09	c using permutation method 4
1.00/1.09	c count = 2
1.00/1.09	c Ordering finished 1.080 seconds
1.00/1.09	c mapped 1.080 seconds
1.00/1.09	c permuted 1.080 seconds
1.00/1.09	c about to sort
1.00/1.09	c sorted
1.19/1.21	c clauses now 43754
1.19/1.21	c init finished 1.130 seconds
1.19/1.21	c tree built 1.200 seconds
1.19/1.21	c 
1.19/1.21	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
1.19/1.21	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
1.19/1.21	c ................................ | ..........                    |                                  1.2         0.0             
1.19/1.21	c ................................ | ..........                  1 |                                  1.2         0.8             
1.19/1.21	c ................................ | ..........                  2 |                                  1.2         1.7             
1.19/1.21	c ................................ | ..........                  3 |                                  1.2         2.5             
1.19/1.21	c ................................ | ..........                  4 |                                  1.2         3.3             
1.19/1.21	c ................................ | ..........                  5 |                                  1.2         4.1             
1.19/1.22	c ................................ | ..........               1000 |              91                  1.2       819.7          148
95.71/24.99	c ................................ | ..........            1615000 | 62068        1028253            96.1     16808.9      3720293
191.87/49.06	c ................................ | ..........            2501000 | 97495        1562391           192.1     13017.2      6355038
287.69/73.08	c ................................ | ..........            3221000 | 130866       1985206           288.0     11183.3      8635175
383.89/97.19	c ................................ | ..........            3993000 | 169995       2415625           384.2     10391.7     10361050
480.08/121.20	c ................................ | *.........            4481000 | 198770       2727030           480.1      9333.3     13236801
575.91/145.27	c ................................ | *.........            4989000 | 224695       2993951           576.2      8658.8     15536678
672.12/169.31	c ................................ | *.........            5536000 | 247025       3291307           672.1      8236.3     17105512
767.91/193.37	c ................................ | *.........            6041000 | 274109       3527445           768.2      7863.9     19235844
864.15/217.41	c ................................ | *.........            6428000 | 296068       3716859           864.2      7438.0     20791581
959.98/241.45	c ................................ | **........            6858000 | 316906       3915906           960.2      7142.3     22597113
1055.78/265.46	c ................................ | **........            7426000 | 344919       4160945          1056.0      7031.9     24541958
1151.99/289.51	c ................................ | **........            7924000 | 371567       4377172          1152.0      6878.2     26330440
1247.83/313.59	c ................................ | **........            8319000 | 389472       4544197          1248.2      6665.0     27643548
1344.03/337.63	c ................................ | **........            8878000 | 415516       4759076          1344.2      6604.9     29089667
1440.21/361.70	c ................................ | ***.......            9416000 | 437888       4972121          1440.2      6537.9     30617526
1536.03/385.71	c ................................ | ***.......            9801000 | 456030       5149914          1536.1      6380.5     32200980
1632.66/409.90	c ................................ | ***.......           10177000 | 475717       5319161          1632.7      6233.3     33352219
1728.09/433.80	c ................................ | ***.......           10556000 | 495678       5491364          1728.1      6108.4     34341531
1823.91/457.83	c ................................ | ***.......           10908000 | 515150       5636270          1824.0      5980.1     35591791
1919.77/481.88	c ................................ | ****......           11250000 | 534953       5780348          1920.1      5859.2     36765268
2015.98/505.92	c ................................ | ****......           11650000 | 556105       5949877          2016.0      5778.7     38439945
2111.80/529.96	c ................................ | ****......           11977000 | 570443       6100287          2112.1      5670.8     39723894
2208.04/554.02	c ................................ | ****......           12231000 | 583542       6229187          2208.1      5539.1     41137516
2303.88/578.09	c ................................ | ****......           12459000 | 593287       6356290          2304.3      5406.9     42245238
2400.12/602.13	c ................................ | *****.....           12710000 | 607129       6495254          2400.3      5295.2     43081143
2495.97/626.19	c ................................ | *****.....           12914000 | 618412       6620178          2496.3      5173.2     43945685
2592.24/650.27	c ................................ | *****.....           13095000 | 628637       6739472          2592.5      5051.1     44971449
2688.10/674.23	c ................................ | *****.....           13325000 | 641758       6885323          2688.2      4956.8     46076064
2783.93/698.22	c ................................ | *****.....           13635000 | 655655       7070605          2784.0      4897.6     47147516
2880.17/722.38	c ................................ | ******....           13939000 | 669573       7236065          2880.5      4839.1     47884747
2976.42/746.46	c ................................ | ******....           14143000 | 680177       7371809          2976.7      4751.3     49072116
3072.26/770.42	c ................................ | ******....           14305000 | 688470       7493259          3072.4      4656.0     49871428
3168.11/794.41	c ................................ | ******....           14515000 | 699924       7635598          3168.2      4581.5     50799055
3263.99/818.41	c ................................ | ******....           14670000 | 707273       7740045          3264.0      4494.4     51618192
3360.25/842.56	c ................................ | *******...           14780000 | 714996       7840766          3360.5      4398.2     52389655
3456.06/866.50	c ................................ | *******...           15073000 | 730832       7973324          3456.1      4361.3     53734292
3551.91/890.53	c ................................ | *******...           15306000 | 742171       8096158          3552.0      4309.1     54695030
3648.14/914.68	c ................................ | *******...           15573000 | 754679       8234008          3648.4      4268.4     55695327
3743.99/938.62	c ................................ | *******...           15798000 | 764870       8356290          3744.1      4219.5     56476834
3839.82/962.64	c ................................ | ********..           15972000 | 773018       8467657          3840.0      4159.4     57178556
3936.10/986.75	c ................................ | ********..           16130000 | 780794       8565811          3936.3      4097.8     57735980
4032.35/1010.83	c ................................ | ********..           16322000 | 790814       8678938          4032.4      4047.7     58297464
4128.18/1034.87	c ................................ | ********..           16451000 | 798597       8770089          4128.5      3984.8     59649428
4224.03/1058.85	c ................................ | ********..           16685000 | 811075       8889989          4224.2      3949.8     60596736
4320.27/1082.94	c ................................ | *********.           16945000 | 823319       9013693          4320.4      3922.0     60886308
4415.72/1106.88	c ................................ | *********.           17156000 | 836346       9125012          4416.1      3884.9     61577815
4512.77/1131.13	c ................................ | *********.           17283000 | 842675       9211852          4512.9      3829.7     62188003
4607.85/1154.99	c ................................ | *********.           17416000 | 849266       9303231          4608.2      3779.3     62733037
4704.09/1179.09	c ................................ | *********.           17525000 | 855217       9391268          4704.5      3725.2     63324838

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3262706-1304094101/watcher-3262706-1304094101 -o /tmp/evaluation-result-3262706-1304094101/solver-3262706-1304094101 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262706-1304094101.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.27 7.66 7.88 2/174 16350
/proc/meminfo: memFree=30926272/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=9544 CPUtime=0 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) R 16348 16350 16318 0 -1 4202496 461 0 0 0 0 0 0 0 25 0 1 0 521008818 9773056 403 33554432000 4194304 4338100 140736952084528 18446744073709551615 251813900997 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/16350/statm: 2386 404 147 36 0 291 0

[startup+0.0263809 s]
/proc/loadavg: 6.27 7.66 7.88 2/174 16350
/proc/meminfo: memFree=30926272/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=13120 CPUtime=0.02 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) R 16348 16350 16318 0 -1 4202496 1354 0 0 0 2 0 0 0 25 0 1 0 521008818 13434880 1296 33554432000 4194304 4338100 140736952084528 18446744073709551615 251813644191 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/16350/statm: 3280 1298 148 36 0 1185 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 13120

[startup+0.100483 s]
/proc/loadavg: 6.27 7.66 7.88 2/174 16350
/proc/meminfo: memFree=30926272/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=19656 CPUtime=0.09 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) R 16348 16350 16318 0 -1 4202496 3142 0 0 0 9 0 0 0 25 0 1 0 521008818 20127744 2912 33554432000 4194304 4338100 140736952084528 18446744073709551615 251813900997 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/16350/statm: 4914 2914 153 36 0 2819 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 19656

[startup+0.30046 s]
/proc/loadavg: 6.27 7.66 7.88 2/174 16350
/proc/meminfo: memFree=30926272/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=19920 CPUtime=0.29 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) R 16348 16350 16318 0 -1 4202496 3224 0 0 0 29 0 0 0 25 0 1 0 521008818 20398080 2994 33554432000 4194304 4338100 140736952084528 18446744073709551615 4245192 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/16350/statm: 4980 2994 154 36 0 2885 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 19920

[startup+0.700267 s]
/proc/loadavg: 6.27 7.66 7.88 2/174 16350
/proc/meminfo: memFree=30926272/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=19920 CPUtime=0.69 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) R 16348 16350 16318 0 -1 4202496 3238 0 0 0 69 0 0 0 25 0 1 0 521008818 20398080 3008 33554432000 4194304 4338100 140736952084528 18446744073709551615 4245192 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/16350/statm: 4980 3008 154 36 0 2885 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 19920

[startup+1.50014 s]
/proc/loadavg: 6.27 7.66 7.88 6/182 16361
/proc/meminfo: memFree=30785352/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=277496 CPUtime=2.34 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) S 16348 16350 16318 0 -1 4202496 22598 0 0 0 231 3 0 0 25 0 5 0 521008818 284155904 22192 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16350/statm: 69374 22192 176 36 0 67279 0
Current children cumulated CPU time (s) 2.34
Current children cumulated vsize (KiB) 277496

[startup+3.10087 s]
/proc/loadavg: 6.27 7.66 7.88 9/186 16365
/proc/meminfo: memFree=30698516/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=352456 CPUtime=8.73 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) S 16348 16350 16318 0 -1 4202496 26771 0 0 0 866 7 0 0 25 0 5 0 521008818 360914944 26365 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16350/statm: 88114 26365 176 36 0 86019 0
[pid=16350/tid=16362] ppid=16348 vsize=352456 CPUtime=1.86 cores=0,2,4,6
/proc/16350/task/16362/stat : 16362 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 1448 0 0 0 186 0 0 0 18 0 5 0 521008940 360914944 26365 33554432000 4194304 4338100 140736952084528 18446744073709551615 251813635822 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16350/tid=16363] ppid=16348 vsize=352456 CPUtime=1.88 cores=0,2,4,6
/proc/16350/task/16363/stat : 16363 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 1014 0 0 0 186 2 0 0 25 0 5 0 521008940 360914944 26365 33554432000 4194304 4338100 140736952084528 18446744073709551615 251813635518 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16350/tid=16364] ppid=16348 vsize=352456 CPUtime=1.87 cores=0,2,4,6
/proc/16350/task/16364/stat : 16364 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 2145 0 0 0 187 0 0 0 25 0 5 0 521008940 360914944 26365 33554432000 4194304 4338100 140736952084528 18446744073709551615 251813635321 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16350/tid=16365] ppid=16348 vsize=352456 CPUtime=1.87 cores=0,2,4,6
/proc/16350/task/16365/stat : 16365 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 1648 0 0 0 186 1 0 0 25 0 5 0 521008940 360914944 26365 33554432000 4194304 4338100 140736952084528 18446744073709551615 251813635302 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 8.73
Current children cumulated vsize (KiB) 352456

[startup+6.30033 s]
/proc/loadavg: 6.41 7.67 7.88 9/186 16365
/proc/meminfo: memFree=30654636/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=371528 CPUtime=21.49 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) S 16348 16350 16318 0 -1 4202496 32877 0 0 0 2129 20 0 0 25 0 5 0 521008818 380444672 32471 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16350/statm: 92882 32471 176 36 0 90787 0
[pid=16350/tid=16362] ppid=16348 vsize=371528 CPUtime=5.02 cores=0,2,4,6
/proc/16350/task/16362/stat : 16362 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 2730 0 0 0 501 1 0 0 18 0 5 0 521008940 380444672 32471 33554432000 4194304 4338100 140736952084528 18446744073709551615 4241503 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16350/tid=16363] ppid=16348 vsize=371528 CPUtime=5.08 cores=0,2,4,6
/proc/16350/task/16363/stat : 16363 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 2232 0 0 0 500 8 0 0 25 0 5 0 521008940 380444672 32471 33554432000 4194304 4338100 140736952084528 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16350/tid=16364] ppid=16348 vsize=371528 CPUtime=5.07 cores=0,2,4,6
/proc/16350/task/16364/stat : 16364 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 3958 0 0 0 505 2 0 0 25 0 5 0 521008940 380444672 32471 33554432000 4194304 4338100 140736952084528 18446744073709551615 4253972 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16350/tid=16365] ppid=16348 vsize=371528 CPUtime=5.07 cores=0,2,4,6
/proc/16350/task/16365/stat : 16365 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 3441 0 0 0 502 5 0 0 25 0 5 0 521008940 380444672 32471 33554432000 4194304 4338100 140736952084528 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 21.49
Current children cumulated vsize (KiB) 371528

[startup+12.7003 s]
/proc/loadavg: 6.53 7.67 7.88 9/186 16365
/proc/meminfo: memFree=30583460/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=413024 CPUtime=47.01 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) S 16348 16350 16318 0 -1 4202496 43778 0 0 0 4662 39 0 0 25 0 5 0 521008818 422936576 43372 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16350/statm: 103256 43372 176 36 0 101161 0
[pid=16350/tid=16362] ppid=16348 vsize=413024 CPUtime=11.35 cores=0,2,4,6
/proc/16350/task/16362/stat : 16362 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 5329 0 0 0 1131 4 0 0 18 0 5 0 521008940 422936576 43372 33554432000 4194304 4338100 140736952084528 18446744073709551615 251813635468 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16350/tid=16363] ppid=16348 vsize=413024 CPUtime=11.47 cores=0,2,4,6
/proc/16350/task/16363/stat : 16363 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 4146 0 0 0 1133 14 0 0 25 0 5 0 521008940 422936576 43372 33554432000 4194304 4338100 140736952084528 18446744073709551615 4286315 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16350/tid=16364] ppid=16348 vsize=413024 CPUtime=11.47 cores=0,2,4,6
/proc/16350/task/16364/stat : 16364 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 7575 0 0 0 1143 4 0 0 25 0 5 0 521008940 422936576 43372 33554432000 4194304 4338100 140736952084528 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16350/tid=16365] ppid=16348 vsize=413024 CPUtime=11.47 cores=0,2,4,6
/proc/16350/task/16365/stat : 16365 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 6212 0 0 0 1135 12 0 0 25 0 5 0 521008940 422936576 43372 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 47.01
Current children cumulated vsize (KiB) 413024

[startup+25.5011 s]
/proc/loadavg: 6.86 7.69 7.88 9/186 16365
/proc/meminfo: memFree=30470468/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=470160 CPUtime=98.11 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) S 16348 16350 16318 0 -1 4202496 60541 0 0 0 9725 86 0 0 25 0 5 0 521008818 481443840 60135 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16350/statm: 117540 60135 176 36 0 115445 0
[pid=16350/tid=16362] ppid=16348 vsize=470160 CPUtime=24.04 cores=0,2,4,6
/proc/16350/task/16362/stat : 16362 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 9853 0 0 0 2397 7 0 0 18 0 5 0 521008940 481443840 60135 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16350/tid=16363] ppid=16348 vsize=470160 CPUtime=24.27 cores=0,2,4,6
/proc/16350/task/16363/stat : 16363 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 6600 0 0 0 2394 33 0 0 25 0 5 0 521008940 481443840 60135 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16350/tid=16364] ppid=16348 vsize=470160 CPUtime=24.27 cores=0,2,4,6
/proc/16350/task/16364/stat : 16364 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 13693 0 0 0 2418 9 0 0 25 0 5 0 521008940 481443840 60135 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16350/tid=16365] ppid=16348 vsize=470160 CPUtime=24.27 cores=0,2,4,6
/proc/16350/task/16365/stat : 16365 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 9879 0 0 0 2396 31 0 0 25 0 5 0 521008940 481443840 60135 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 98.11
Current children cumulated vsize (KiB) 470160

[startup+51.1008 s]
/proc/loadavg: 7.25 7.71 7.88 9/186 16366
/proc/meminfo: memFree=30296712/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=559236 CPUtime=200.26 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) S 16348 16350 16318 0 -1 4202496 85720 0 0 0 19868 158 0 0 25 0 5 0 521008818 572657664 85314 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16350/statm: 139809 85314 176 36 0 137714 0
[pid=16350/tid=16362] ppid=16348 vsize=559236 CPUtime=49.4 cores=0,2,4,6
/proc/16350/task/16362/stat : 16362 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 16042 0 0 0 4928 12 0 0 18 0 5 0 521008940 572657664 85314 33554432000 4194304 4338100 140736952084528 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16350/tid=16363] ppid=16348 vsize=559236 CPUtime=49.88 cores=0,2,4,6
/proc/16350/task/16363/stat : 16363 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 11301 0 0 0 4927 61 0 0 25 0 5 0 521008940 572657664 85314 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16350/tid=16364] ppid=16348 vsize=559236 CPUtime=49.86 cores=0,2,4,6
/proc/16350/task/16364/stat : 16364 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 21791 0 0 0 4965 21 0 0 25 0 5 0 521008940 572657664 85314 33554432000 4194304 4338100 140736952084528 18446744073709551615 251813900997 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16350/tid=16365] ppid=16348 vsize=559236 CPUtime=49.87 cores=0,2,4,6
/proc/16350/task/16365/stat : 16365 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 16070 0 0 0 4928 59 0 0 25 0 5 0 521008940 572657664 85314 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 200.26
Current children cumulated vsize (KiB) 559236

[startup+102.306 s]
/proc/loadavg: 7.81 7.79 7.90 9/186 16368
/proc/meminfo: memFree=29979500/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=714344 CPUtime=404.67 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) S 16348 16350 16318 0 -1 4202496 128419 0 0 0 40173 294 0 0 25 0 5 0 521008818 731488256 128013 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16350/statm: 178586 128013 176 36 0 176491 0
[pid=16350/tid=16362] ppid=16348 vsize=714344 CPUtime=100.19 cores=0,2,4,6
/proc/16350/task/16362/stat : 16362 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 27262 0 0 0 10000 19 0 0 20 0 5 0 521008940 731488256 128013 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16350/tid=16363] ppid=16348 vsize=714344 CPUtime=101.08 cores=0,2,4,6
/proc/16350/task/16363/stat : 16363 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 19121 0 0 0 9992 116 0 0 25 0 5 0 521008940 731488256 128013 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16350/tid=16364] ppid=16348 vsize=714344 CPUtime=101.08 cores=0,2,4,6
/proc/16350/task/16364/stat : 16364 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 35990 0 0 0 10066 42 0 0 25 0 5 0 521008940 731488256 128013 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16350/tid=16365] ppid=16348 vsize=714344 CPUtime=101.08 cores=0,2,4,6
/proc/16350/task/16365/stat : 16365 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 25530 0 0 0 9995 113 0 0 25 0 5 0 521008940 731488256 128013 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 404.67
Current children cumulated vsize (KiB) 714344

[startup+162.301 s]
/proc/loadavg: 8.05 7.85 7.91 9/186 16369
/proc/meminfo: memFree=29644796/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=893248 CPUtime=644.17 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) S 16348 16350 16318 0 -1 4202496 177884 0 0 0 64000 417 0 0 25 0 5 0 521008818 914685952 177478 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16350/statm: 223312 177478 176 36 0 221217 0
[pid=16350/tid=16362] ppid=16348 vsize=893248 CPUtime=159.7 cores=0,2,4,6
/proc/16350/task/16362/stat : 16362 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 37847 0 0 0 15944 26 0 0 19 0 5 0 521008940 914685952 177478 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16350/tid=16363] ppid=16348 vsize=893248 CPUtime=161.07 cores=0,2,4,6
/proc/16350/task/16363/stat : 16363 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 30079 0 0 0 15944 163 0 0 25 0 5 0 521008940 914685952 177478 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16350/tid=16364] ppid=16348 vsize=893248 CPUtime=161.07 cores=0,2,4,6
/proc/16350/task/16364/stat : 16364 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 49080 0 0 0 16045 62 0 0 25 0 5 0 521008940 914685952 177478 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16350/tid=16365] ppid=16348 vsize=893248 CPUtime=161.07 cores=0,2,4,6
/proc/16350/task/16365/stat : 16365 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 40362 0 0 0 15946 161 0 0 25 0 5 0 521008940 914685952 177478 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 644.17
Current children cumulated vsize (KiB) 893248

[startup+222.301 s]
/proc/loadavg: 8.12 7.91 7.92 9/186 16371
/proc/meminfo: memFree=29367748/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=1035256 CPUtime=883.72 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) S 16348 16350 16318 0 -1 4202496 216316 0 0 0 87854 518 0 0 25 0 5 0 521008818 1060102144 215910 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16350/statm: 258814 215910 176 36 0 256719 0
[pid=16350/tid=16362] ppid=16348 vsize=1035256 CPUtime=219.26 cores=0,2,4,6
/proc/16350/task/16362/stat : 16362 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 46569 0 0 0 21893 33 0 0 25 0 5 0 521008940 1060102144 215910 33554432000 4194304 4338100 140736952084528 18446744073709551615 251813936683 0 0 4096 0 0 0 0 -1 6 0 0 0

################
# More data... #
################

/proc/meminfo: memFree=27482392/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=1961212 CPUtime=3039.93 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) S 16348 16350 16318 0 -1 4202496 467410 0 0 0 302768 1225 0 0 25 0 5 0 521008818 2008281088 467004 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16350/statm: 490303 467004 176 36 0 488208 0
[pid=16350/tid=16362] ppid=16348 vsize=1961212 CPUtime=756.59 cores=0,2,4,6
/proc/16350/task/16362/stat : 16362 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 114860 0 0 0 75565 94 0 0 25 0 5 0 521008940 2008281088 467004 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16350/tid=16363] ppid=16348 vsize=1961212 CPUtime=761.07 cores=0,2,4,6
/proc/16350/task/16363/stat : 16363 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 83089 0 0 0 75634 473 0 0 25 0 5 0 521008940 2008281088 467004 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16350/tid=16364] ppid=16348 vsize=1961212 CPUtime=759.95 cores=0,2,4,6
/proc/16350/task/16364/stat : 16364 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 142157 0 0 0 75809 186 0 0 25 0 5 0 521008940 2008281088 467004 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16350/tid=16365] ppid=16348 vsize=1961212 CPUtime=761.08 cores=0,2,4,6
/proc/16350/task/16365/stat : 16365 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 106788 0 0 0 75640 468 0 0 25 0 5 0 521008940 2008281088 467004 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3039.93
Current children cumulated vsize (KiB) 1961212

[startup+822.3 s]
/proc/loadavg: 8.09 8.17 8.06 9/186 16387
/proc/meminfo: memFree=27348080/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=2029540 CPUtime=3279.54 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) S 16348 16350 16318 0 -1 4202496 486039 0 0 0 326652 1302 0 0 25 0 5 0 521008818 2078248960 485633 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16350/statm: 507385 485633 176 36 0 505290 0
[pid=16350/tid=16362] ppid=16348 vsize=2029540 CPUtime=816.22 cores=0,2,4,6
/proc/16350/task/16362/stat : 16362 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 117466 0 0 0 81525 97 0 0 19 0 5 0 521008940 2078248960 485633 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16350/tid=16363] ppid=16348 vsize=2029540 CPUtime=821.06 cores=0,2,4,6
/proc/16350/task/16363/stat : 16363 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 87840 0 0 0 81598 508 0 0 25 0 5 0 521008940 2078248960 485633 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16350/tid=16364] ppid=16348 vsize=2029540 CPUtime=819.95 cores=0,2,4,6
/proc/16350/task/16364/stat : 16364 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 147704 0 0 0 81803 192 0 0 25 0 5 0 521008940 2078248960 485633 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16350/tid=16365] ppid=16348 vsize=2029540 CPUtime=821.07 cores=0,2,4,6
/proc/16350/task/16365/stat : 16365 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 112513 0 0 0 81606 501 0 0 25 0 5 0 521008940 2078248960 485633 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3279.54
Current children cumulated vsize (KiB) 2029540

[startup+882.3 s]
/proc/loadavg: 8.09 8.15 8.06 9/186 16389
/proc/meminfo: memFree=27184252/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=2116372 CPUtime=3519.15 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) S 16348 16350 16318 0 -1 4202496 509432 0 0 0 350538 1377 0 0 25 0 5 0 521008818 2167164928 509026 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16350/statm: 529093 509026 176 36 0 526998 0
[pid=16350/tid=16362] ppid=16348 vsize=2116372 CPUtime=875.82 cores=0,2,4,6
/proc/16350/task/16362/stat : 16362 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 123241 0 0 0 87483 99 0 0 22 0 5 0 521008940 2167164928 509026 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16350/tid=16363] ppid=16348 vsize=2116372 CPUtime=881.06 cores=0,2,4,6
/proc/16350/task/16363/stat : 16363 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 92194 0 0 0 87568 538 0 0 25 0 5 0 521008940 2167164928 509026 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16350/tid=16364] ppid=16348 vsize=2116372 CPUtime=879.94 cores=0,2,4,6
/proc/16350/task/16364/stat : 16364 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 155683 0 0 0 87791 203 0 0 25 0 5 0 521008940 2167164928 509026 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16350/tid=16365] ppid=16348 vsize=2116372 CPUtime=881.07 cores=0,2,4,6
/proc/16350/task/16365/stat : 16365 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 117798 0 0 0 87575 532 0 0 25 0 5 0 521008940 2167164928 509026 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3519.15
Current children cumulated vsize (KiB) 2116372

[startup+942.3 s]
/proc/loadavg: 8.29 8.19 8.08 10/186 16390
/proc/meminfo: memFree=27043480/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=2187432 CPUtime=3758.76 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) S 16348 16350 16318 0 -1 4202496 528529 0 0 0 374428 1448 0 0 25 0 5 0 521008818 2239930368 528123 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16350/statm: 546858 528123 176 36 0 544763 0
[pid=16350/tid=16362] ppid=16348 vsize=2187432 CPUtime=935.42 cores=0,2,4,6
/proc/16350/task/16362/stat : 16362 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 128590 0 0 0 93439 103 0 0 18 0 5 0 521008940 2239930368 528123 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16350/tid=16363] ppid=16348 vsize=2187432 CPUtime=941.07 cores=0,2,4,6
/proc/16350/task/16363/stat : 16363 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 94317 0 0 0 93541 566 0 0 25 0 5 0 521008940 2239930368 528123 33554432000 4194304 4338100 140736952084528 18446744073709551615 4206915 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16350/tid=16364] ppid=16348 vsize=2187432 CPUtime=939.95 cores=0,2,4,6
/proc/16350/task/16364/stat : 16364 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 164699 0 0 0 93781 214 0 0 25 0 5 0 521008940 2239930368 528123 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16350/tid=16365] ppid=16348 vsize=2187432 CPUtime=941.08 cores=0,2,4,6
/proc/16350/task/16365/stat : 16365 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 120407 0 0 0 93546 562 0 0 25 0 5 0 521008940 2239930368 528123 33554432000 4194304 4338100 140736952084528 18446744073709551615 251813635518 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3758.76
Current children cumulated vsize (KiB) 2187432

[startup+1002.3 s]
/proc/loadavg: 8.30 8.22 8.10 9/186 16392
/proc/meminfo: memFree=26911636/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=2298608 CPUtime=3998.39 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) S 16348 16350 16318 0 -1 4202496 541946 0 0 0 398334 1505 0 0 25 0 5 0 521008818 2353774592 541540 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16350/statm: 574652 541540 176 36 0 572557 0
[pid=16350/tid=16362] ppid=16348 vsize=2298608 CPUtime=995.1 cores=0,2,4,6
/proc/16350/task/16362/stat : 16362 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 132053 0 0 0 99403 107 0 0 25 0 5 0 521008940 2353774592 541540 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16350/tid=16363] ppid=16348 vsize=2298608 CPUtime=1001.06 cores=0,2,4,6
/proc/16350/task/16363/stat : 16363 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 95813 0 0 0 99517 589 0 0 25 0 5 0 521008940 2353774592 541540 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16350/tid=16364] ppid=16348 vsize=2298608 CPUtime=999.9 cores=0,2,4,6
/proc/16350/task/16364/stat : 16364 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 171157 0 0 0 99770 220 0 0 18 0 5 0 521008940 2353774592 541540 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16350/tid=16365] ppid=16348 vsize=2298608 CPUtime=1001.07 cores=0,2,4,6
/proc/16350/task/16365/stat : 16365 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 122407 0 0 0 99522 585 0 0 25 0 5 0 521008940 2353774592 541540 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3998.39
Current children cumulated vsize (KiB) 2298608

[startup+1062.3 s]
/proc/loadavg: 8.21 8.20 8.10 9/186 16393
/proc/meminfo: memFree=26759212/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=2384912 CPUtime=4238.02 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) S 16348 16350 16318 0 -1 4202496 564465 0 0 0 422230 1572 0 0 25 0 5 0 521008818 2442149888 564059 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16350/statm: 596228 564059 176 36 0 594133 0
[pid=16350/tid=16362] ppid=16348 vsize=2384912 CPUtime=1055.1 cores=0,2,4,6
/proc/16350/task/16362/stat : 16362 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 138430 0 0 0 105395 115 0 0 25 0 5 0 521008940 2442149888 564059 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16350/tid=16363] ppid=16348 vsize=2384912 CPUtime=1061.06 cores=0,2,4,6
/proc/16350/task/16363/stat : 16363 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 101153 0 0 0 105487 619 0 0 25 0 5 0 521008940 2442149888 564059 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16350/tid=16364] ppid=16348 vsize=2384912 CPUtime=1059.53 cores=0,2,4,6
/proc/16350/task/16364/stat : 16364 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 175369 0 0 0 105730 223 0 0 19 0 5 0 521008940 2442149888 564059 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16350/tid=16365] ppid=16348 vsize=2384912 CPUtime=1061.07 cores=0,2,4,6
/proc/16350/task/16365/stat : 16365 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 128997 0 0 0 105496 611 0 0 25 0 5 0 521008940 2442149888 564059 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4238.02
Current children cumulated vsize (KiB) 2384912

[startup+1122.3 s]
/proc/loadavg: 8.21 8.21 8.10 9/186 16395
/proc/meminfo: memFree=26616332/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=2428836 CPUtime=4477.62 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) S 16348 16350 16318 0 -1 4202496 577331 0 0 0 446126 1636 0 0 25 0 5 0 521008818 2487128064 576925 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16350/statm: 607209 576925 176 36 0 605114 0
[pid=16350/tid=16362] ppid=16348 vsize=2428836 CPUtime=1115.1 cores=0,2,4,6
/proc/16350/task/16362/stat : 16362 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 142650 0 0 0 111387 123 0 0 25 0 5 0 521008940 2487128064 576925 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16350/tid=16363] ppid=16348 vsize=2428836 CPUtime=1121.05 cores=0,2,4,6
/proc/16350/task/16363/stat : 16363 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 103964 0 0 0 111458 647 0 0 25 0 5 0 521008940 2487128064 576925 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16350/tid=16364] ppid=16348 vsize=2428836 CPUtime=1119.14 cores=0,2,4,6
/proc/16350/task/16364/stat : 16364 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 178832 0 0 0 111689 225 0 0 18 0 5 0 521008940 2487128064 576925 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254337 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16350/tid=16365] ppid=16348 vsize=2428836 CPUtime=1121.08 cores=0,2,4,6
/proc/16350/task/16365/stat : 16365 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 131369 0 0 0 111470 638 0 0 25 0 5 0 521008940 2487128064 576925 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4477.62
Current children cumulated vsize (KiB) 2428836

[startup+1182.3 s]
/proc/loadavg: 8.08 8.17 8.09 9/186 16397
/proc/meminfo: memFree=26509784/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=2474344 CPUtime=4717.27 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) S 16348 16350 16318 0 -1 4202496 589841 0 0 0 470025 1702 0 0 25 0 5 0 521008818 2533728256 589435 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16350/statm: 618586 589435 176 36 0 616491 0
[pid=16350/tid=16362] ppid=16348 vsize=2474344 CPUtime=1175.1 cores=0,2,4,6
/proc/16350/task/16362/stat : 16362 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 145277 0 0 0 117383 127 0 0 25 0 5 0 521008940 2533728256 589435 33554432000 4194304 4338100 140736952084528 18446744073709551615 251813635518 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16350/tid=16363] ppid=16348 vsize=2474344 CPUtime=1181.05 cores=0,2,4,6
/proc/16350/task/16363/stat : 16363 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 107643 0 0 0 117429 676 0 0 25 0 5 0 521008940 2533728256 589435 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16350/tid=16364] ppid=16348 vsize=2474344 CPUtime=1178.79 cores=0,2,4,6
/proc/16350/task/16364/stat : 16364 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 180906 0 0 0 117654 225 0 0 21 0 5 0 521008940 2533728256 589435 33554432000 4194304 4338100 140736952084528 18446744073709551615 251813635119 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16350/tid=16365] ppid=16348 vsize=2474344 CPUtime=1181.07 cores=0,2,4,6
/proc/16350/task/16365/stat : 16365 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 135499 0 0 0 117439 668 0 0 25 0 5 0 521008940 2533728256 589435 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4717.27
Current children cumulated vsize (KiB) 2474344



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.1 s]
/proc/loadavg: 8.05 8.16 8.09 6/182 16425
/proc/meminfo: memFree=28537176/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=2562304 CPUtime=4800.36 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) S 16348 16350 16318 0 -1 4202496 595770 0 0 0 478313 1723 0 0 25 0 5 0 521008818 2623799296 595364 33554432000 4194304 4338100 140736952084528 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16350/statm: 640576 595364 176 36 0 638481 0
[pid=16350/tid=16362] ppid=16348 vsize=2562304 CPUtime=1195.91 cores=0,2,4,6
/proc/16350/task/16362/stat : 16362 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 147067 0 0 0 119462 129 0 0 25 0 5 0 521008940 2623799296 595364 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16350/tid=16363] ppid=16348 vsize=2562304 CPUtime=1201.86 cores=0,2,4,6
/proc/16350/task/16363/stat : 16363 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 109105 0 0 0 119500 686 0 0 25 0 5 0 521008940 2623799296 595364 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16350/tid=16364] ppid=16348 vsize=2562304 CPUtime=1199.47 cores=0,2,4,6
/proc/16350/task/16364/stat : 16364 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 181795 0 0 0 119721 226 0 0 21 0 5 0 521008940 2623799296 595364 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16350/tid=16365] ppid=16348 vsize=2562304 CPUtime=1201.89 cores=0,2,4,6
/proc/16350/task/16365/stat : 16365 (ttsth-5-4) R 16348 16350 16318 0 -1 4202560 137287 0 0 0 119511 678 0 0 25 0 5 0 521008940 2623799296 595364 33554432000 4194304 4338100 140736952084528 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.36
Current children cumulated vsize (KiB) 2562304

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.21 s]
/proc/loadavg: 8.05 8.16 8.09 6/182 16425
/proc/meminfo: memFree=28537176/32950928 swapFree=67111448/67111528
[pid=16350] ppid=16348 vsize=0 CPUtime=4800.48 cores=0,2,4,6
/proc/16350/stat : 16350 (ttsth-5-4) Z 16348 16350 16318 0 -1 4203548 595770 0 0 0 478315 1733 0 0 18 0 2 0 521008818 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16350/statm: 0 0 0 0 0 0 0
[pid=16350/tid=16364] ppid=16348 vsize=0 CPUtime=1199.57 cores=0,2,4,6
/proc/16350/task/16364/stat : 16364 (ttsth-5-4) R 16348 16350 16318 0 -1 4203588 181795 0 0 0 119721 236 0 0 21 0 2 0 521008940 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.48
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1203.23
CPU time (s): 4800.51
CPU user time (s): 4783.16
CPU system time (s): 17.3494
CPU usage (%): 398.969
Max. virtual memory (cumulated for all children) (KiB): 2562304

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4783.16
system time used= 17.3494
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 595770
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 9940
involuntary context switches= 31858

runsolver used 1.93871 second user time and 4.6243 second system time

The end

Launcher Data 

Begin job on node126 at 2011-04-29 18:21:41
IDJOB=3262706
IDBENCH=83796
IDSOLVER=1654
FILE ID=node126/3262706-1304094101
RUNJOBID= node126-1304094101-16336
PBS_JOBID= 13170590
Free space on /tmp= 71600 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT03/handmade/gomes/bqwh/bqwh.50.825.shuffled-as.sat03-1648.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3262706-1304094101/watcher-3262706-1304094101 -o /tmp/evaluation-result-3262706-1304094101/solver-3262706-1304094101 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262706-1304094101.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 05a191325d89de504821b55dea51ca7f
RANDOM SEED=527536052

node126.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.56
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      30926676 kB
Buffers:        245324 kB
Cached:        1397520 kB
SwapCached:         76 kB
Active:         117460 kB
Inactive:      1593224 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      30926676 kB
SwapTotal:    67111528 kB
SwapFree:     67111448 kB
Dirty:            1288 kB
Writeback:           0 kB
AnonPages:       67704 kB
Mapped:          14332 kB
Slab:           246796 kB
PageTables:       4992 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   191240 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71596 MiB
End job on node126 at 2011-04-29 18:41:46