Trace number 3261870

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.55 1201.98

General information on the benchmark

Namecrafted/kullmann/GreenTao/
GreenTaok1k2/GreenTao_2-3-5_528.cnf
MD5SUM7c7070bd15b57ddd9a06dd3709baef54
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark157.607
Satisfiable
(Un)Satisfiability was proved
Number of variables527
Number of clauses12692
Sum of the clauses size38748
Maximum clause length5
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 312356
Number of clauses of size 40
Number of clauses of size 5336
Number of clauses of size over 50

Solver Data 

0.00/0.01	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3261870-1304090404.cnf 
0.00/0.01	c Using SATTIMEOUT of 4800
0.00/0.01	c Using SATRAM of 15500
0.00/0.01	c Size of subset node = 32
0.00/0.01	c Variables = 527
0.00/0.01	c Clauses = 12692
0.00/0.01	c Literals = 38748
0.00/0.01	c Reduced size:
0.00/0.01	c Variables = 527
0.00/0.01	c Clauses = 10739
0.00/0.01	c Literals = 32889
0.00/0.01	c a2
0.00/0.02	c a1
0.09/0.09	c Independent sub-problems = 2
0.09/0.09	c Sub-problem with 242 variables and 5130 clauses
0.09/0.09	c using permutation method 4
0.09/0.09	c count = 2
0.09/0.09	c Ordering finished 0.090 seconds
0.09/0.09	c mapped 0.090 seconds
0.09/0.09	c permuted 0.090 seconds
0.09/0.09	c about to sort
0.09/0.10	c sorted
0.09/0.11	c clauses now 5130
0.09/0.11	c init finished 0.100 seconds
0.09/0.11	c tree built 0.100 seconds
0.09/0.11	c 
0.09/0.11	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.09/0.11	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.09/0.11	c ................................ | ..........                    |                                  0.1         0.0             
0.09/0.11	c ................................ | ..........                  1 |                                  0.1        10.0             
0.09/0.11	c ................................ | ..........                  2 |                                  0.1        20.0             
0.09/0.11	c ................................ | ..........                  3 |                                  0.1        30.0             
0.09/0.11	c ................................ | ..........                  4 |                                  0.1        40.0             
0.09/0.11	c ................................ | ..........                  5 |                                  0.1        50.0             
0.09/0.11	c ................................ | ..........               1000 | 11           747                 0.1      8333.3         6387
95.94/24.14	c ................................ | ..........            1254000 | 78244        1272671            96.1     13053.0     18890690
191.79/48.17	c ................................ | ..........            1941000 | 129369       1950659           192.1     10104.1     28065802
288.05/72.21	c ................................ | ..........            2488000 | 170299       2456477           288.1      8635.9     32492229
383.92/96.22	c ................................ | ..........            2941000 | 205672       2899348           384.0      7658.5     36686085
479.80/120.30	c ................................ | *.........            3339000 | 239400       3281638           480.2      6953.5     39425003
576.06/144.36	c ................................ | *.........            3658000 | 260662       3643321           576.3      6347.2     44559234
671.94/168.34	c ................................ | *.........            3952000 | 282174       3956176           672.1      5880.1     48829350
767.80/192.36	c ................................ | *.........            4211000 | 301850       4208712           768.1      5482.6     52359490
864.06/216.43	c ................................ | *.........            4434000 | 319509       4432714           864.2      5130.8     56274555
959.92/240.49	c ................................ | **........            4672000 | 342173       4675297           960.3      4865.1     59989555
1055.80/264.48	c ................................ | **........            4918000 | 365913       4903775          1056.1      4656.6     64009321
1152.07/288.52	c ................................ | **........            5128000 | 385151       5140774          1152.2      4450.8     67789736
1247.94/312.56	c ................................ | **........            5310000 | 404627       5349500          1248.2      4254.2     71045774
1343.83/336.58	c ................................ | **........            5505000 | 419285       5534097          1344.2      4095.5     71682531
1440.09/360.65	c ................................ | ***.......            5698000 | 438533       5717334          1440.3      3956.1     72329966
1535.95/384.68	c ................................ | ***.......            5908000 | 459501       5916065          1536.3      3845.6     72637840
1632.22/408.73	c ................................ | ***.......            6121000 | 477282       6109671          1632.4      3749.8     72637840
1728.08/432.72	c ................................ | ***.......            6304000 | 493578       6286726          1728.2      3647.7     72637840
1823.96/456.71	c ................................ | ***.......            6482000 | 511134       6460232          1824.0      3553.7     72637840
1920.23/480.87	c ................................ | ****......            6657000 | 525281       6634237          1920.5      3466.3     73077861
2016.53/504.91	c ................................ | ****......            6824000 | 539057       6806144          2016.5      3384.0     73959902
2111.98/528.83	c ................................ | ****......            6987000 | 550552       6982308          2112.1      3308.1     75905560
2207.85/552.89	c ................................ | ****......            7130000 | 561622       7159366          2208.2      3228.8     78348487
2304.12/576.92	c ................................ | ****......            7262000 | 575481       7308688          2304.2      3151.6     81124325
2399.99/600.91	c ................................ | *****.....            7419000 | 589659       7451473          2400.0      3091.2     81635495
2496.27/625.06	c ................................ | *****.....            7587000 | 602890       7609048          2496.5      3039.0     82701280
2592.54/649.10	c ................................ | *****.....            7732000 | 617860       7751280          2592.6      2982.4     83323489
2687.62/672.99	c ................................ | *****.....            7872000 | 630214       7893237          2688.0      2928.6     84099374
2783.88/697.08	c ................................ | *****.....            8036000 | 644123       8032678          2784.2      2886.3     84771898
2880.56/721.24	c ................................ | ******....            8177000 | 655909       8183199          2880.7      2838.5     85917865
2976.02/745.12	c ................................ | ******....            8299000 | 666288       8320327          2976.1      2788.5     87197192
3072.69/769.33	c ................................ | ******....            8419000 | 676416       8451717          3072.8      2739.8     89287141
3167.79/793.18	c ................................ | ******....            8549000 | 685885       8590231          3168.1      2698.5     90836968
3264.47/817.38	c ................................ | ******....            8661000 | 694082       8746473          3264.8      2652.9     93227827
3359.91/841.29	c ................................ | *******...            8783000 | 706747       8884518          3360.3      2613.8     95198808
3455.78/865.29	c ................................ | *******...            8917000 | 720355       9021160          3456.2      2580.0     96736000
3552.41/889.41	c ................................ | *******...            9058000 | 733657       9150006          3552.4      2549.8     96989423
3647.88/913.39	c ................................ | *******...            9188000 | 745045       9291063          3648.3      2518.5     98121816
3744.56/937.54	c ................................ | *******...            9307000 | 755178       9420800          3744.7      2485.4     99784643
3840.42/961.50	c ................................ | ********..            9452000 | 767908       9559753          3840.4      2461.2    101750583
3936.32/985.53	c ................................ | ********..            9602000 | 782071       9688888          3936.4      2439.3    103391537
4032.58/1009.60	c ................................ | ********..            9747000 | 796348       9812764          4032.6      2417.1    103927970
4128.05/1033.59	c ................................ | ********..            9871000 | 808508       9938055          4128.4      2391.0    104283007
4224.32/1057.64	c ................................ | ********..            9999000 | 820153       10063697         4224.5      2366.9    104881378
4319.79/1081.56	c ................................ | *********.           10104000 | 830602       10184526         4320.0      2338.9    105493152
4416.46/1105.76	c ................................ | *********.           10222000 | 842199       10308789         4416.7      2314.4    106674481
4511.96/1129.61	c ................................ | *********.           10323000 | 851529       10416443         4512.0      2287.9    106742457
4608.21/1153.80	c ................................ | *********.           10449000 | 863307       10534128         4608.6      2267.3    107732055
4704.47/1177.81	c ................................ | *********.           10562000 | 874278       10651719         4704.5      2245.1    108310936

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3261870-1304090404/watcher-3261870-1304090404 -o /tmp/evaluation-result-3261870-1304090404/solver-3261870-1304090404 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3261870-1304090404.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.71 7.81 7.91 3/174 17351
/proc/meminfo: memFree=19982320/32951124 swapFree=67111524/67111528
[pid=17351] ppid=17348 vsize=9540 CPUtime=0 cores=0,2,4,6
/proc/17351/stat : 17351 (ttsth-5-4) R 17348 17351 16471 0 -1 4202496 460 0 0 0 0 0 0 0 20 0 1 0 501456067 9768960 402 33554432000 4194304 4338100 140733541479872 18446744073709551615 264726065861 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/17351/statm: 2385 403 146 36 0 290 0

[startup+0.095219 s]
/proc/loadavg: 6.71 7.81 7.91 3/174 17351
/proc/meminfo: memFree=19982320/32951124 swapFree=67111524/67111528
[pid=17351] ppid=17348 vsize=12028 CPUtime=0.09 cores=0,2,4,6
/proc/17351/stat : 17351 (ttsth-5-4) R 17348 17351 16471 0 -1 4202496 1099 0 0 0 9 0 0 0 20 0 1 0 501456067 12316672 1015 33554432000 4194304 4338100 140733541479872 18446744073709551615 4295949 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/17351/statm: 3007 1015 155 36 0 912 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 12028

[startup+0.100207 s]
/proc/loadavg: 6.71 7.81 7.91 3/174 17351
/proc/meminfo: memFree=19982320/32951124 swapFree=67111524/67111528
[pid=17351] ppid=17348 vsize=12028 CPUtime=0.09 cores=0,2,4,6
/proc/17351/stat : 17351 (ttsth-5-4) R 17348 17351 16471 0 -1 4202496 1104 0 0 0 9 0 0 0 20 0 1 0 501456067 12316672 1020 33554432000 4194304 4338100 140733541479872 18446744073709551615 264725800118 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/17351/statm: 3007 1020 160 36 0 912 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 12028

[startup+0.300136 s]
/proc/loadavg: 6.71 7.81 7.91 3/174 17351
/proc/meminfo: memFree=19982320/32951124 swapFree=67111524/67111528
[pid=17351] ppid=17348 vsize=201096 CPUtime=0.85 cores=0,2,4,6
/proc/17351/stat : 17351 (ttsth-5-4) S 17348 17351 16471 0 -1 4202496 5002 0 0 0 85 0 0 0 21 0 5 0 501456067 205922304 4918 33554432000 4194304 4338100 140733541479872 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/17351/statm: 50274 4919 175 36 0 48179 0
Current children cumulated CPU time (s) 0.85
Current children cumulated vsize (KiB) 201096

[startup+0.701068 s]
/proc/loadavg: 6.71 7.81 7.91 3/174 17351
/proc/meminfo: memFree=19982320/32951124 swapFree=67111524/67111528
[pid=17351] ppid=17348 vsize=227044 CPUtime=2.45 cores=0,2,4,6
/proc/17351/stat : 17351 (ttsth-5-4) S 17348 17351 16471 0 -1 4202496 10551 0 0 0 243 2 0 0 21 0 5 0 501456067 232493056 10467 33554432000 4194304 4338100 140733541479872 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/17351/statm: 56761 10468 175 36 0 54666 0
Current children cumulated CPU time (s) 2.45
Current children cumulated vsize (KiB) 227044

[startup+1.50088 s]
/proc/loadavg: 6.58 7.76 7.89 10/186 17363
/proc/meminfo: memFree=19829224/32951124 swapFree=67111524/67111528
[pid=17351] ppid=17348 vsize=265768 CPUtime=5.64 cores=0,2,4,6
/proc/17351/stat : 17351 (ttsth-5-4) S 17348 17351 16471 0 -1 4202496 20338 0 0 0 561 3 0 0 21 0 5 0 501456067 272146432 20254 33554432000 4194304 4338100 140733541479872 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/17351/statm: 66442 20254 175 36 0 64347 0
[pid=17351/tid=17356] ppid=17348 vsize=265768 CPUtime=1.38 cores=0,2,4,6
/proc/17351/task/17356/stat : 17356 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 4465 0 0 0 138 0 0 0 20 0 5 0 501456079 272146432 20254 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=17351/tid=17357] ppid=17348 vsize=265768 CPUtime=1.37 cores=0,2,4,6
/proc/17351/task/17357/stat : 17357 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 4967 0 0 0 137 0 0 0 19 0 5 0 501456079 272146432 20254 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17351/tid=17358] ppid=17348 vsize=265768 CPUtime=1.38 cores=0,2,4,6
/proc/17351/task/17358/stat : 17358 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 4423 0 0 0 138 0 0 0 25 0 5 0 501456079 272146432 20254 33554432000 4194304 4338100 140733541479872 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17351/tid=17359] ppid=17348 vsize=265768 CPUtime=1.37 cores=0,2,4,6
/proc/17351/task/17359/stat : 17359 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 4501 0 0 0 136 1 0 0 19 0 5 0 501456079 272146432 20254 33554432000 4194304 4338100 140733541479872 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 5.64
Current children cumulated vsize (KiB) 265768

[startup+3.10058 s]
/proc/loadavg: 6.58 7.76 7.89 10/186 17364
/proc/meminfo: memFree=19753196/32951124 swapFree=67111524/67111528
[pid=17351] ppid=17348 vsize=326916 CPUtime=12.03 cores=0,2,4,6
/proc/17351/stat : 17351 (ttsth-5-4) S 17348 17351 16471 0 -1 4202496 36612 0 0 0 1197 6 0 0 21 0 5 0 501456067 334761984 36528 33554432000 4194304 4338100 140733541479872 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/17351/statm: 81729 36529 175 36 0 79634 0
[pid=17351/tid=17356] ppid=17348 vsize=326916 CPUtime=2.98 cores=0,2,4,6
/proc/17351/task/17356/stat : 17356 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 8394 0 0 0 297 1 0 0 25 0 5 0 501456079 334761984 36529 33554432000 4194304 4338100 140733541479872 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=17351/tid=17357] ppid=17348 vsize=326916 CPUtime=2.98 cores=0,2,4,6
/proc/17351/task/17357/stat : 17357 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 9371 0 0 0 297 1 0 0 22 0 5 0 501456079 334761984 36529 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17351/tid=17358] ppid=17348 vsize=326916 CPUtime=2.98 cores=0,2,4,6
/proc/17351/task/17358/stat : 17358 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 8346 0 0 0 297 1 0 0 25 0 5 0 501456079 334761984 36529 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17351/tid=17359] ppid=17348 vsize=326916 CPUtime=2.96 cores=0,2,4,6
/proc/17351/task/17359/stat : 17359 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 8520 0 0 0 294 2 0 0 22 0 5 0 501456079 334761984 36529 33554432000 4194304 4338100 140733541479872 18446744073709551615 4265472 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 12.03
Current children cumulated vsize (KiB) 326916

[startup+6.30097 s]
/proc/loadavg: 6.58 7.76 7.89 10/186 17364
/proc/meminfo: memFree=19546688/32951124 swapFree=67111524/67111528
[pid=17351] ppid=17348 vsize=433656 CPUtime=24.82 cores=0,2,4,6
/proc/17351/stat : 17351 (ttsth-5-4) S 17348 17351 16471 0 -1 4202496 63579 0 0 0 2469 13 0 0 21 0 5 0 501456067 444063744 63441 33554432000 4194304 4338100 140733541479872 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/17351/statm: 108414 63441 176 36 0 106319 0
[pid=17351/tid=17356] ppid=17348 vsize=433656 CPUtime=6.18 cores=0,2,4,6
/proc/17351/task/17356/stat : 17356 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 15480 0 0 0 615 3 0 0 25 0 5 0 501456079 444063744 63441 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=17351/tid=17357] ppid=17348 vsize=433656 CPUtime=6.18 cores=0,2,4,6
/proc/17351/task/17357/stat : 17357 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 16019 0 0 0 615 3 0 0 25 0 5 0 501456079 444063744 63441 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17351/tid=17358] ppid=17348 vsize=433656 CPUtime=6.17 cores=0,2,4,6
/proc/17351/task/17358/stat : 17358 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 15167 0 0 0 615 2 0 0 25 0 5 0 501456079 444063744 63441 33554432000 4194304 4338100 140733541479872 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17351/tid=17359] ppid=17348 vsize=433656 CPUtime=6.14 cores=0,2,4,6
/proc/17351/task/17359/stat : 17359 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 14931 0 0 0 611 3 0 0 25 0 5 0 501456079 444063744 63441 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 24.82
Current children cumulated vsize (KiB) 433656

[startup+12.7008 s]
/proc/loadavg: 6.79 7.77 7.89 10/186 17364
/proc/meminfo: memFree=19202720/32951124 swapFree=67111524/67111528
[pid=17351] ppid=17348 vsize=581048 CPUtime=50.38 cores=0,2,4,6
/proc/17351/stat : 17351 (ttsth-5-4) S 17348 17351 16471 0 -1 4202496 102382 0 0 0 5018 20 0 0 21 0 5 0 501456067 594993152 102244 33554432000 4194304 4338100 140733541479872 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/17351/statm: 145262 102244 176 36 0 143167 0
[pid=17351/tid=17356] ppid=17348 vsize=581048 CPUtime=12.58 cores=0,2,4,6
/proc/17351/task/17356/stat : 17356 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 23787 0 0 0 1253 5 0 0 25 0 5 0 501456079 594993152 102244 33554432000 4194304 4338100 140733541479872 18446744073709551615 4289225 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=17351/tid=17357] ppid=17348 vsize=581048 CPUtime=12.58 cores=0,2,4,6
/proc/17351/task/17357/stat : 17357 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 25851 0 0 0 1254 4 0 0 25 0 5 0 501456079 594993152 102244 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17351/tid=17358] ppid=17348 vsize=581048 CPUtime=12.58 cores=0,2,4,6
/proc/17351/task/17358/stat : 17358 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 25771 0 0 0 1253 5 0 0 25 0 5 0 501456079 594993152 102244 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17351/tid=17359] ppid=17348 vsize=581048 CPUtime=12.51 cores=0,2,4,6
/proc/17351/task/17359/stat : 17359 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 24991 0 0 0 1246 5 0 0 25 0 5 0 501456079 594993152 102244 33554432000 4194304 4338100 140733541479872 18446744073709551615 264725800674 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 50.38
Current children cumulated vsize (KiB) 581048

[startup+25.5003 s]
/proc/loadavg: 6.98 7.77 7.89 9/186 17364
/proc/meminfo: memFree=18605932/32951124 swapFree=67111524/67111528
[pid=17351] ppid=17348 vsize=826524 CPUtime=101.51 cores=0,2,4,6
/proc/17351/stat : 17351 (ttsth-5-4) S 17348 17351 16471 0 -1 4202496 165329 0 0 0 10116 35 0 0 21 0 5 0 501456067 846360576 165191 33554432000 4194304 4338100 140733541479872 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/17351/statm: 206631 165192 176 36 0 204536 0
[pid=17351/tid=17356] ppid=17348 vsize=826524 CPUtime=25.38 cores=0,2,4,6
/proc/17351/task/17356/stat : 17356 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 38287 0 0 0 2530 8 0 0 25 0 5 0 501456079 846360576 165192 33554432000 4194304 4338100 140733541479872 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=17351/tid=17357] ppid=17348 vsize=826524 CPUtime=25.38 cores=0,2,4,6
/proc/17351/task/17357/stat : 17357 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 40060 0 0 0 2531 7 0 0 25 0 5 0 501456079 846360576 165192 33554432000 4194304 4338100 140733541479872 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17351/tid=17358] ppid=17348 vsize=826524 CPUtime=25.37 cores=0,2,4,6
/proc/17351/task/17358/stat : 17358 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 41978 0 0 0 2529 8 0 0 25 0 5 0 501456079 846360576 165192 33554432000 4194304 4338100 140733541479872 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17351/tid=17359] ppid=17348 vsize=826524 CPUtime=25.23 cores=0,2,4,6
/proc/17351/task/17359/stat : 17359 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 43023 0 0 0 2513 10 0 0 25 0 5 0 501456079 846360576 165192 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 101.51
Current children cumulated vsize (KiB) 826524

[startup+51.1005 s]
/proc/loadavg: 7.39 7.81 7.90 10/186 17365
/proc/meminfo: memFree=17621228/32951124 swapFree=67111524/67111528
[pid=17351] ppid=17348 vsize=1127924 CPUtime=203.77 cores=0,2,4,6
/proc/17351/stat : 17351 (ttsth-5-4) S 17348 17351 16471 0 -1 4202496 242122 0 0 0 20322 55 0 0 21 0 5 0 501456067 1154994176 241984 33554432000 4194304 4338100 140733541479872 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/17351/statm: 281981 241984 176 36 0 279886 0
[pid=17351/tid=17356] ppid=17348 vsize=1127924 CPUtime=50.98 cores=0,2,4,6
/proc/17351/task/17356/stat : 17356 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 60642 0 0 0 5083 15 0 0 25 0 5 0 501456079 1154994176 241984 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=17351/tid=17357] ppid=17348 vsize=1127924 CPUtime=50.98 cores=0,2,4,6
/proc/17351/task/17357/stat : 17357 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 55531 0 0 0 5088 10 0 0 25 0 5 0 501456079 1154994176 241984 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17351/tid=17358] ppid=17348 vsize=1127924 CPUtime=50.98 cores=0,2,4,6
/proc/17351/task/17358/stat : 17358 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 61173 0 0 0 5085 13 0 0 25 0 5 0 501456079 1154994176 241984 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17351/tid=17359] ppid=17348 vsize=1127924 CPUtime=50.7 cores=0,2,4,6
/proc/17351/task/17359/stat : 17359 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 62794 0 0 0 5054 16 0 0 25 0 5 0 501456079 1154994176 241984 33554432000 4194304 4338100 140733541479872 18446744073709551615 4256139 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 203.77
Current children cumulated vsize (KiB) 1127924

[startup+102.307 s]
/proc/loadavg: 7.75 7.84 7.91 10/186 17366
/proc/meminfo: memFree=16127408/32951124 swapFree=67111524/67111528
[pid=17351] ppid=17348 vsize=1417376 CPUtime=408.31 cores=0,2,4,6
/proc/17351/stat : 17351 (ttsth-5-4) S 17348 17351 16471 0 -1 4202496 318985 0 0 0 40750 81 0 0 21 0 5 0 501456067 1451393024 318847 33554432000 4194304 4338100 140733541479872 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/17351/statm: 354344 318847 176 36 0 352249 0
[pid=17351/tid=17356] ppid=17348 vsize=1417376 CPUtime=102.18 cores=0,2,4,6
/proc/17351/task/17356/stat : 17356 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 79989 0 0 0 10195 23 0 0 25 0 5 0 501456079 1451393024 318847 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=17351/tid=17357] ppid=17348 vsize=1417376 CPUtime=102.18 cores=0,2,4,6
/proc/17351/task/17357/stat : 17357 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 72846 0 0 0 10203 15 0 0 25 0 5 0 501456079 1451393024 318847 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17351/tid=17358] ppid=17348 vsize=1417376 CPUtime=102.11 cores=0,2,4,6
/proc/17351/task/17358/stat : 17358 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 79495 0 0 0 10193 18 0 0 25 0 5 0 501456079 1451393024 318847 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17351/tid=17359] ppid=17348 vsize=1417376 CPUtime=101.69 cores=0,2,4,6
/proc/17351/task/17359/stat : 17359 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 84673 0 0 0 10146 23 0 0 25 0 5 0 501456079 1451393024 318847 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 408.31
Current children cumulated vsize (KiB) 1417376

[startup+162.301 s]
/proc/loadavg: 7.91 7.86 7.91 10/186 17368
/proc/meminfo: memFree=14612620/32951124 swapFree=67111524/67111528
[pid=17351] ppid=17348 vsize=1753260 CPUtime=647.95 cores=0,2,4,6
/proc/17351/stat : 17351 (ttsth-5-4) S 17348 17351 16471 0 -1 4202496 404840 0 0 0 64685 110 0 0 21 0 5 0 501456067 1795338240 404702 33554432000 4194304 4338100 140733541479872 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/17351/statm: 438315 404702 176 36 0 436220 0
[pid=17351/tid=17356] ppid=17348 vsize=1753260 CPUtime=162.18 cores=0,2,4,6
/proc/17351/task/17356/stat : 17356 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 104680 0 0 0 16187 31 0 0 25 0 5 0 501456079 1795338240 404702 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=17351/tid=17357] ppid=17348 vsize=1753260 CPUtime=162.17 cores=0,2,4,6
/proc/17351/task/17357/stat : 17357 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 95370 0 0 0 16194 23 0 0 25 0 5 0 501456079 1795338240 404702 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17351/tid=17358] ppid=17348 vsize=1753260 CPUtime=162.08 cores=0,2,4,6
/proc/17351/task/17358/stat : 17358 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 92559 0 0 0 16184 24 0 0 25 0 5 0 501456079 1795338240 404702 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17351/tid=17359] ppid=17348 vsize=1753260 CPUtime=161.39 cores=0,2,4,6
/proc/17351/task/17359/stat : 17359 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 110249 0 0 0 16108 31 0 0 25 0 5 0 501456079 1795338240 404702 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 647.95
Current children cumulated vsize (KiB) 1753260


################
# More data... #
################

/proc/meminfo: memFree=6350640/32951124 swapFree=67111524/67111528
[pid=17351] ppid=17348 vsize=3229212 CPUtime=3044.75 cores=0,2,4,6
/proc/17351/stat : 17351 (ttsth-5-4) S 17348 17351 16471 0 -1 4202496 771910 0 0 0 304226 249 0 0 21 0 5 0 501456067 3306713088 771772 33554432000 4194304 4338100 140733541479872 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/17351/statm: 807303 771772 176 36 0 805208 0
[pid=17351/tid=17356] ppid=17348 vsize=3229212 CPUtime=761.37 cores=0,2,4,6
/proc/17351/task/17356/stat : 17356 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 205177 0 0 0 76064 73 0 0 25 0 5 0 501456079 3306713088 771772 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17351/tid=17357] ppid=17348 vsize=3229212 CPUtime=762.18 cores=0,2,4,6
/proc/17351/task/17357/stat : 17357 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 172384 0 0 0 76169 49 0 0 25 0 5 0 501456079 3306713088 771772 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17351/tid=17358] ppid=17348 vsize=3229212 CPUtime=760.2 cores=0,2,4,6
/proc/17351/task/17358/stat : 17358 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 174340 0 0 0 75964 56 0 0 25 0 5 0 501456079 3306713088 771772 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17351/tid=17359] ppid=17348 vsize=3229212 CPUtime=760.85 cores=0,2,4,6
/proc/17351/task/17359/stat : 17359 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 218027 0 0 0 76016 69 0 0 25 0 5 0 501456079 3306713088 771772 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3044.75
Current children cumulated vsize (KiB) 3229212

[startup+822.301 s]
/proc/loadavg: 8.11 8.05 7.99 10/186 17388
/proc/meminfo: memFree=5779504/32951124 swapFree=67111524/67111528
[pid=17351] ppid=17348 vsize=3386312 CPUtime=3284.42 cores=0,2,4,6
/proc/17351/stat : 17351 (ttsth-5-4) S 17348 17351 16471 0 -1 4202496 813331 0 0 0 328174 268 0 0 21 0 5 0 501456067 3467583488 813193 33554432000 4194304 4338100 140733541479872 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/17351/statm: 846578 813193 176 36 0 844483 0
[pid=17351/tid=17356] ppid=17348 vsize=3386312 CPUtime=821.37 cores=0,2,4,6
/proc/17351/task/17356/stat : 17356 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 213690 0 0 0 82059 78 0 0 25 0 5 0 501456079 3467583488 813193 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17351/tid=17357] ppid=17348 vsize=3386312 CPUtime=822.19 cores=0,2,4,6
/proc/17351/task/17357/stat : 17357 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 178177 0 0 0 82167 52 0 0 25 0 5 0 501456079 3467583488 813193 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17351/tid=17358] ppid=17348 vsize=3386312 CPUtime=819.88 cores=0,2,4,6
/proc/17351/task/17358/stat : 17358 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 185777 0 0 0 81927 61 0 0 25 0 5 0 501456079 3467583488 813193 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17351/tid=17359] ppid=17348 vsize=3386312 CPUtime=820.84 cores=0,2,4,6
/proc/17351/task/17359/stat : 17359 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 233705 0 0 0 82009 75 0 0 25 0 5 0 501456079 3467583488 813193 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3284.42
Current children cumulated vsize (KiB) 3386312

[startup+882.301 s]
/proc/loadavg: 8.11 8.06 8.00 10/186 17390
/proc/meminfo: memFree=5246668/32951124 swapFree=67111524/67111528
[pid=17351] ppid=17348 vsize=3558372 CPUtime=3524.09 cores=0,2,4,6
/proc/17351/stat : 17351 (ttsth-5-4) S 17348 17351 16471 0 -1 4202496 840733 0 0 0 352131 278 0 0 21 0 5 0 501456067 3643772928 840595 33554432000 4194304 4338100 140733541479872 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/17351/statm: 889593 840595 176 36 0 887498 0
[pid=17351/tid=17356] ppid=17348 vsize=3558372 CPUtime=881.37 cores=0,2,4,6
/proc/17351/task/17356/stat : 17356 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 222623 0 0 0 88055 82 0 0 25 0 5 0 501456079 3643772928 840595 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17351/tid=17357] ppid=17348 vsize=3558372 CPUtime=882.18 cores=0,2,4,6
/proc/17351/task/17357/stat : 17357 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 180334 0 0 0 88166 52 0 0 25 0 5 0 501456079 3643772928 840595 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17351/tid=17358] ppid=17348 vsize=3558372 CPUtime=879.55 cores=0,2,4,6
/proc/17351/task/17358/stat : 17358 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 192338 0 0 0 87891 64 0 0 25 0 5 0 501456079 3643772928 840595 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17351/tid=17359] ppid=17348 vsize=3558372 CPUtime=880.84 cores=0,2,4,6
/proc/17351/task/17359/stat : 17359 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 243456 0 0 0 88006 78 0 0 25 0 5 0 501456079 3643772928 840595 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3524.09
Current children cumulated vsize (KiB) 3558372

[startup+942.3 s]
/proc/loadavg: 8.11 8.06 8.00 9/186 17392
/proc/meminfo: memFree=4678008/32951124 swapFree=67111524/67111528
[pid=17351] ppid=17348 vsize=3661636 CPUtime=3763.73 cores=0,2,4,6
/proc/17351/stat : 17351 (ttsth-5-4) S 17348 17351 16471 0 -1 4202496 868754 0 0 0 376084 289 0 0 21 0 5 0 501456067 3749515264 868616 33554432000 4194304 4338100 140733541479872 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/17351/statm: 915409 868616 176 36 0 913314 0
[pid=17351/tid=17356] ppid=17348 vsize=3661636 CPUtime=941.36 cores=0,2,4,6
/proc/17351/task/17356/stat : 17356 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 229653 0 0 0 94050 86 0 0 25 0 5 0 501456079 3749515264 868616 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17351/tid=17357] ppid=17348 vsize=3661636 CPUtime=942.17 cores=0,2,4,6
/proc/17351/task/17357/stat : 17357 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 180788 0 0 0 94164 53 0 0 25 0 5 0 501456079 3749515264 868616 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17351/tid=17358] ppid=17348 vsize=3661636 CPUtime=939.23 cores=0,2,4,6
/proc/17351/task/17358/stat : 17358 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 202669 0 0 0 93856 67 0 0 25 0 5 0 501456079 3749515264 868616 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17351/tid=17359] ppid=17348 vsize=3661636 CPUtime=940.84 cores=0,2,4,6
/proc/17351/task/17359/stat : 17359 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 253662 0 0 0 94002 82 0 0 25 0 5 0 501456079 3749515264 868616 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3763.73
Current children cumulated vsize (KiB) 3661636

[startup+1002.3 s]
/proc/loadavg: 8.16 8.08 8.01 10/186 17393
/proc/meminfo: memFree=4103648/32951124 swapFree=67111524/67111528
[pid=17351] ppid=17348 vsize=3780672 CPUtime=4003.41 cores=0,2,4,6
/proc/17351/stat : 17351 (ttsth-5-4) S 17348 17351 16471 0 -1 4202496 899904 0 0 0 400039 302 0 0 21 0 5 0 501456067 3871408128 899766 33554432000 4194304 4338100 140733541479872 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/17351/statm: 945168 899766 176 36 0 943073 0
[pid=17351/tid=17356] ppid=17348 vsize=3780672 CPUtime=1001.35 cores=0,2,4,6
/proc/17351/task/17356/stat : 17356 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 241055 0 0 0 100045 90 0 0 25 0 5 0 501456079 3871408128 899766 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17351/tid=17357] ppid=17348 vsize=3780672 CPUtime=1002.17 cores=0,2,4,6
/proc/17351/task/17357/stat : 17357 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 181360 0 0 0 100164 53 0 0 25 0 5 0 501456079 3871408128 899766 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17351/tid=17358] ppid=17348 vsize=3780672 CPUtime=998.9 cores=0,2,4,6
/proc/17351/task/17358/stat : 17358 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 206811 0 0 0 99821 69 0 0 25 0 5 0 501456079 3871408128 899766 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17351/tid=17359] ppid=17348 vsize=3780672 CPUtime=1000.84 cores=0,2,4,6
/proc/17351/task/17359/stat : 17359 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 268696 0 0 0 99996 88 0 0 25 0 5 0 501456079 3871408128 899766 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4003.41
Current children cumulated vsize (KiB) 3780672

[startup+1062.3 s]
/proc/loadavg: 8.06 8.06 8.00 10/186 17395
/proc/meminfo: memFree=3659348/32951124 swapFree=67111524/67111528
[pid=17351] ppid=17348 vsize=3814988 CPUtime=4243.1 cores=0,2,4,6
/proc/17351/stat : 17351 (ttsth-5-4) S 17348 17351 16471 0 -1 4202496 909841 0 0 0 424002 308 0 0 21 0 5 0 501456067 3906547712 909703 33554432000 4194304 4338100 140733541479872 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/17351/statm: 953747 909704 176 36 0 951652 0
[pid=17351/tid=17356] ppid=17348 vsize=3814988 CPUtime=1061.36 cores=0,2,4,6
/proc/17351/task/17356/stat : 17356 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 246206 0 0 0 106043 93 0 0 25 0 5 0 501456079 3906547712 909704 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17351/tid=17357] ppid=17348 vsize=3814988 CPUtime=1062.18 cores=0,2,4,6
/proc/17351/task/17357/stat : 17357 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 185034 0 0 0 106163 55 0 0 25 0 5 0 501456079 3906547712 909704 33554432000 4194304 4338100 140733541479872 18446744073709551615 264726070808 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17351/tid=17358] ppid=17348 vsize=3814988 CPUtime=1058.58 cores=0,2,4,6
/proc/17351/task/17358/stat : 17358 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 207370 0 0 0 105788 70 0 0 25 0 5 0 501456079 3906547712 909704 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17351/tid=17359] ppid=17348 vsize=3814988 CPUtime=1060.84 cores=0,2,4,6
/proc/17351/task/17359/stat : 17359 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 269250 0 0 0 105995 89 0 0 25 0 5 0 501456079 3906547712 909704 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4243.1
Current children cumulated vsize (KiB) 3814988

[startup+1122.3 s]
/proc/loadavg: 8.02 8.05 8.00 9/186 17396
/proc/meminfo: memFree=3026284/32951124 swapFree=67111524/67111528
[pid=17351] ppid=17348 vsize=3874460 CPUtime=4482.77 cores=0,2,4,6
/proc/17351/stat : 17351 (ttsth-5-4) S 17348 17351 16471 0 -1 4202496 925836 0 0 0 447963 314 0 0 21 0 5 0 501456067 3967447040 925698 33554432000 4194304 4338100 140733541479872 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/17351/statm: 968615 925698 176 36 0 966520 0
[pid=17351/tid=17356] ppid=17348 vsize=3874460 CPUtime=1121.36 cores=0,2,4,6
/proc/17351/task/17356/stat : 17356 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 250754 0 0 0 112041 95 0 0 25 0 5 0 501456079 3967447040 925698 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17351/tid=17357] ppid=17348 vsize=3874460 CPUtime=1122.17 cores=0,2,4,6
/proc/17351/task/17357/stat : 17357 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 192675 0 0 0 112160 57 0 0 25 0 5 0 501456079 3967447040 925698 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17351/tid=17358] ppid=17348 vsize=3874460 CPUtime=1118.25 cores=0,2,4,6
/proc/17351/task/17358/stat : 17358 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 207947 0 0 0 111755 70 0 0 25 0 5 0 501456079 3967447040 925698 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17351/tid=17359] ppid=17348 vsize=3874460 CPUtime=1120.84 cores=0,2,4,6
/proc/17351/task/17359/stat : 17359 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 272478 0 0 0 111994 90 0 0 25 0 5 0 501456079 3967447040 925698 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4482.77
Current children cumulated vsize (KiB) 3874460

[startup+1182.3 s]
/proc/loadavg: 8.00 8.04 8.00 10/186 17398
/proc/meminfo: memFree=2438908/32951124 swapFree=67111524/67111528
[pid=17351] ppid=17348 vsize=3933668 CPUtime=4722.45 cores=0,2,4,6
/proc/17351/stat : 17351 (ttsth-5-4) S 17348 17351 16471 0 -1 4202496 941290 0 0 0 471923 322 0 0 21 0 5 0 501456067 4028076032 941152 33554432000 4194304 4338100 140733541479872 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/17351/statm: 983417 941152 176 36 0 981322 0
[pid=17351/tid=17356] ppid=17348 vsize=3933668 CPUtime=1181.37 cores=0,2,4,6
/proc/17351/task/17356/stat : 17356 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 261913 0 0 0 118037 100 0 0 25 0 5 0 501456079 4028076032 941152 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254213 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17351/tid=17357] ppid=17348 vsize=3933668 CPUtime=1182.18 cores=0,2,4,6
/proc/17351/task/17357/stat : 17357 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 193630 0 0 0 118160 58 0 0 25 0 5 0 501456079 4028076032 941152 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17351/tid=17358] ppid=17348 vsize=3933668 CPUtime=1177.93 cores=0,2,4,6
/proc/17351/task/17358/stat : 17358 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 208525 0 0 0 117722 71 0 0 25 0 5 0 501456079 4028076032 941152 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17351/tid=17359] ppid=17348 vsize=3933668 CPUtime=1180.84 cores=0,2,4,6
/proc/17351/task/17359/stat : 17359 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 275240 0 0 0 117992 92 0 0 25 0 5 0 501456079 4028076032 941152 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4722.45
Current children cumulated vsize (KiB) 3933668



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1201.8 s]
/proc/loadavg: 8.00 8.03 8.00 9/186 17398
/proc/meminfo: memFree=2242252/32951124 swapFree=67111524/67111528
[pid=17351] ppid=17348 vsize=3953228 CPUtime=4800.35 cores=0,2,4,6
/proc/17351/stat : 17351 (ttsth-5-4) S 17348 17351 16471 0 -1 4202496 947547 0 0 0 479711 324 0 0 21 0 5 0 501456067 4048105472 947409 33554432000 4194304 4338100 140733541479872 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/17351/statm: 988307 947409 176 36 0 986212 0
[pid=17351/tid=17356] ppid=17348 vsize=3953228 CPUtime=1200.86 cores=0,2,4,6
/proc/17351/task/17356/stat : 17356 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 263828 0 0 0 119986 100 0 0 25 0 5 0 501456079 4048105472 947409 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17351/tid=17357] ppid=17348 vsize=3953228 CPUtime=1201.68 cores=0,2,4,6
/proc/17351/task/17357/stat : 17357 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 195611 0 0 0 120110 58 0 0 25 0 5 0 501456079 4048105472 947409 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17351/tid=17358] ppid=17348 vsize=3953228 CPUtime=1197.32 cores=0,2,4,6
/proc/17351/task/17358/stat : 17358 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 208696 0 0 0 119661 71 0 0 25 0 5 0 501456079 4048105472 947409 33554432000 4194304 4338100 140733541479872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17351/tid=17359] ppid=17348 vsize=3953228 CPUtime=1200.34 cores=0,2,4,6
/proc/17351/task/17359/stat : 17359 (ttsth-5-4) R 17348 17351 16471 0 -1 4202560 277430 0 0 0 119942 92 0 0 25 0 5 0 501456079 4048105472 947409 33554432000 4194304 4338100 140733541479872 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.35
Current children cumulated vsize (KiB) 3953228

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1201.91 s]
/proc/loadavg: 8.00 8.03 8.00 9/186 17398
/proc/meminfo: memFree=2242252/32951124 swapFree=67111524/67111528
[pid=17351] ppid=17348 vsize=0 CPUtime=4800.47 cores=0,2,4,6
/proc/17351/stat : 17351 (ttsth-5-4) Z 17348 17351 16471 0 -1 4203548 947547 0 0 0 479713 334 0 0 18 0 2 0 501456067 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/17351/statm: 0 0 0 0 0 0 0
[pid=17351/tid=17358] ppid=17348 vsize=0 CPUtime=1197.43 cores=0,2,4,6
/proc/17351/task/17358/stat : 17358 (ttsth-5-4) R 17348 17351 16471 0 -1 4203588 208696 0 0 0 119662 81 0 0 25 0 2 0 501456079 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.47
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1201.98
CPU time (s): 4800.55
CPU user time (s): 4797.14
CPU system time (s): 3.41248
CPU usage (%): 399.388
Max. virtual memory (cumulated for all children) (KiB): 3953228

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4797.14
system time used= 3.41248
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 947547
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1394
involuntary context switches= 26841

runsolver used 1.90171 second user time and 4.65829 second system time

The end

Launcher Data 

Begin job on node141 at 2011-04-29 17:20:04
IDJOB=3261870
IDBENCH=83256
IDSOLVER=1654
FILE ID=node141/3261870-1304090404
RUNJOBID= node141-1304090404-17334
PBS_JOBID= 13170540
Free space on /tmp= 71424 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/kullmann/GreenTao/GreenTaok1k2/GreenTao_2-3-5_528.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3261870-1304090404/watcher-3261870-1304090404 -o /tmp/evaluation-result-3261870-1304090404/solver-3261870-1304090404 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3261870-1304090404.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 7c7070bd15b57ddd9a06dd3709baef54
RANDOM SEED=674306348

node141.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5362.96
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      19982616 kB
Buffers:       1543500 kB
Cached:       10781264 kB
SwapCached:          4 kB
Active:        3730576 kB
Inactive:      8657576 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      19982616 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:            2440 kB
Writeback:           8 kB
AnonPages:       63176 kB
Mapped:          14392 kB
Slab:           516348 kB
PageTables:       4600 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   183552 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71424 MiB
End job on node141 at 2011-04-29 17:40:08