Trace number 3250604

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.61 1202.75

General information on the benchmark

Namecrafted/kullmann/VanDerWaerden/
VanderWaerden3k/VanDerWaerden_2-3-20_389.cnf
MD5SUMf105570f47bdca662c2d3d185f8dc1f8
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables389
Number of clauses41426
Sum of the clauses size188708
Maximum clause length20
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 337636
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 53790

Solver Data 

0.00/0.05	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250604-1304092958.cnf 
0.00/0.05	c Using SATTIMEOUT of 4800
0.00/0.05	c Using SATRAM of 15500
0.00/0.05	c Size of subset node = 32
0.00/0.05	c Variables = 389
0.00/0.05	c Clauses = 41426
0.00/0.05	c Literals = 188708
0.00/0.05	c Reduced size:
0.00/0.05	c Variables = 389
0.00/0.05	c Clauses = 41426
0.00/0.05	c Literals = 188708
0.00/0.05	c a2
0.19/0.26	c a1
0.69/0.70	c Independent sub-problems = 1
0.69/0.70	c Sub-problem with 389 variables and 41426 clauses
0.69/0.70	c using permutation method 4
0.69/0.70	c count = 1
0.69/0.70	c Ordering finished 0.680 seconds
0.69/0.70	c mapped 0.690 seconds
0.69/0.70	c permuted 0.690 seconds
0.69/0.70	c about to sort
0.69/0.70	c sorted
0.79/0.88	c clauses now 41426
0.79/0.88	c init finished 0.700 seconds
0.79/0.88	c tree built 0.870 seconds
0.79/0.88	c 
0.79/0.88	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.79/0.88	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.79/0.88	c ................................ | ..........                    |                                  0.9         0.0             
0.79/0.88	c ................................ | ..........                  1 |                                  0.9         1.1             
0.79/0.88	c ................................ | ..........                  2 |                                  0.9         2.3             
0.79/0.88	c ................................ | ..........                  3 |                                  0.9         3.4             
0.79/0.88	c ................................ | ..........                  4 |                                  0.9         4.6             
0.79/0.88	c ................................ | ..........                  5 |                                  0.9         5.7             
0.93/0.92	c ................................ | ..........               1000 | 33           217                 1.0       990.1        36106
96.00/24.77	c ................................ | ..........             313000 | 38986        70349              96.3      3250.6     24428264
191.86/48.78	c ................................ | ..........             527000 | 67551        113597            192.2      2742.1     42153608
288.14/72.81	c ................................ | ..........             711000 | 92438        148748            288.2      2467.0     56299053
384.44/96.94	c ................................ | ..........             863000 | 112664       177345            384.6      2243.9     69707444
479.89/120.83	c ................................ | *.........            1001000 | 131713       204846            480.0      2085.2     81525885
576.15/144.91	c ................................ | *.........            1124000 | 149116       229461            576.2      1950.7     92676579
672.42/169.07	c ................................ | *.........            1263000 | 169391       252622            672.7      1877.5    103276758
768.29/193.08	c ................................ | *.........            1376000 | 185112       274560            768.6      1790.2    113187141
864.57/217.14	c ................................ | *.........            1490000 | 201762       293434            864.8      1723.0    121926646
960.43/241.19	c ................................ | **........            1613000 | 219096       319805            960.8      1678.8    130598569
1055.91/265.07	c ................................ | **........            1725000 | 233952       340948           1056.2      1633.2    140334593
1152.58/289.23	c ................................ | **........            1833000 | 249627       360629           1152.7      1590.2    149426246
1248.04/313.14	c ................................ | **........            1944000 | 265819       380444           1248.2      1557.4    158283149
1343.92/337.17	c ................................ | **........            2038000 | 278224       399448           1344.2      1516.1    166618522
1440.59/361.39	c ................................ | ***.......            2127000 | 289912       417159           1441.0      1476.1    175161482
1536.06/385.27	c ................................ | ***.......            2208000 | 301507       437268           1536.4      1437.2    182902303
1631.95/409.24	c ................................ | ***.......            2295000 | 315174       454230           1632.1      1406.1    190603189
1729.38/433.62	c ................................ | ***.......            2371000 | 326779       471088           1729.5      1370.9    198180839
1825.27/457.65	c ................................ | ***.......            2444000 | 337454       488693           1825.5      1338.8    203221506
1920.31/481.41	c ................................ | ****......            2508000 | 345925       507229           1920.4      1306.0    205933571
2015.79/505.35	c ................................ | ****......            2594000 | 357524       525181           2016.0      1286.7    207253642
2112.45/529.57	c ................................ | ****......            2685000 | 369928       544577           2112.7      1270.9    210837197
2208.33/553.50	c ................................ | ****......            2771000 | 381738       562403           2208.3      1254.8    216072458
2304.62/577.68	c ................................ | ****......            2856000 | 392885       577809           2304.9      1239.1    224067092
2401.28/601.81	c ................................ | *****.....            2947000 | 405553       595596           2401.3      1227.2    231070646
2496.35/625.60	c ................................ | *****.....            3029000 | 417430       612146           2496.4      1213.4    232922564
2593.41/649.94	c ................................ | *****.....            3100000 | 427899       627303           2593.6      1195.3    239089128
2688.89/673.86	c ................................ | *****.....            3171000 | 436937       644282           2689.1      1179.2    245286034
2784.35/697.78	c ................................ | *****.....            3242000 | 446544       660279           2784.7      1164.2    250783324
2880.63/721.83	c ................................ | ******....            3316000 | 456116       676834           2880.8      1151.1    256077671
2976.92/745.98	c ................................ | ******....            3382000 | 465199       690606           2977.2      1135.9    262021357
3072.77/769.95	c ................................ | ******....            3454000 | 475135       705947           3073.0      1124.0    268098721
3169.04/794.00	c ................................ | ******....            3529000 | 486255       722230           3169.1      1113.6    274314327
3264.12/817.87	c ................................ | ******....            3601000 | 496702       737858           3264.4      1103.1    280011918
3360.37/841.91	c ................................ | *******...            3668000 | 507807       753026           3360.4      1091.5    286241711
3456.66/866.07	c ................................ | *******...            3759000 | 522975       765941           3456.9      1087.4    290674821
3552.54/890.07	c ................................ | *******...            3846000 | 536629       779643           3552.8      1082.5    294408322
3648.00/913.96	c ................................ | *******...            3936000 | 552272       792971           3648.2      1078.9    300145874
3744.67/938.15	c ................................ | *******...            4030000 | 569574       806897           3744.9      1076.1    305171042
3840.93/962.21	c ................................ | ********..            4104000 | 580784       820393           3841.0      1068.5    310856240
3936.39/986.18	c ................................ | ********..            4192000 | 595361       834300           3936.7      1064.8    316788032
4032.26/1010.14	c ................................ | ********..            4269000 | 608089       847878           4032.4      1058.7    322298772
4128.94/1034.32	c ................................ | ********..            4341000 | 619843       860621           4129.0      1051.3    327478086
4224.42/1058.21	c ................................ | ********..            4414000 | 631750       872671           4224.5      1044.9    332707345
4321.49/1082.53	c ................................ | *********.            4479000 | 641715       885280           4321.6      1036.4    338233723
4416.52/1106.33	c ................................ | *********.            4543000 | 652011       897172           4416.6      1028.6    342794742
4512.39/1130.31	c ................................ | *********.            4615000 | 664257       908420           4512.4      1022.7    347219174
4608.22/1154.37	c ................................ | *********.            4688000 | 677187       920419           4608.5      1017.3    352115032
4705.29/1178.66	c ................................ | *********.            4767000 | 691209       931790           4705.6      1013.1    357136503
4799.97/1202.36	c CPU limit (4800) reached
4799.97/1202.36	c Time = 4800.21 seconds
4799.97/1202.36	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3250604-1304092958/watcher-3250604-1304092958 -o /tmp/evaluation-result-3250604-1304092958/solver-3250604-1304092958 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250604-1304092958.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.90 7.84 7.95 2/177 4802
/proc/meminfo: memFree=23623420/32950928 swapFree=67071984/67111528
[pid=4802] ppid=4800 vsize=9408 CPUtime=0 cores=1,3,5,7
/proc/4802/stat : 4802 (ttsth-5-4) R 4800 4802 4175 0 -1 4202496 435 0 0 0 0 0 0 0 22 0 1 0 599526475 9633792 377 33554432000 4194304 4338100 140737294761312 18446744073709551615 260905054917 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/4802/statm: 2352 378 146 36 0 257 0

[startup+0.105693 s]
/proc/loadavg: 6.90 7.84 7.95 2/177 4802
/proc/meminfo: memFree=23623420/32950928 swapFree=67071984/67111528
[pid=4802] ppid=4800 vsize=36980 CPUtime=0.1 cores=1,3,5,7
/proc/4802/stat : 4802 (ttsth-5-4) R 4800 4802 4175 0 -1 4202496 5921 0 0 0 10 0 0 0 23 0 1 0 599526475 37867520 5725 33554432000 4194304 4338100 140737294761312 18446744073709551615 260904789208 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/4802/statm: 9245 5725 448 36 0 7150 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 36980

[startup+0.200675 s]
/proc/loadavg: 6.90 7.84 7.95 2/177 4802
/proc/meminfo: memFree=23623420/32950928 swapFree=67071984/67111528
[pid=4802] ppid=4800 vsize=36980 CPUtime=0.19 cores=1,3,5,7
/proc/4802/stat : 4802 (ttsth-5-4) R 4800 4802 4175 0 -1 4202496 5928 0 0 0 19 0 0 0 23 0 1 0 599526475 37867520 5725 33554432000 4194304 4338100 140737294761312 18446744073709551615 260904789186 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/4802/statm: 9245 5725 441 36 0 7150 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 36980

[startup+0.300685 s]
/proc/loadavg: 6.90 7.84 7.95 2/177 4802
/proc/meminfo: memFree=23623420/32950928 swapFree=67071984/67111528
[pid=4802] ppid=4800 vsize=16416 CPUtime=0.29 cores=1,3,5,7
/proc/4802/stat : 4802 (ttsth-5-4) R 4800 4802 4175 0 -1 4202496 6397 0 0 0 29 0 0 0 23 0 1 0 599526475 16809984 2117 33554432000 4194304 4338100 140737294761312 18446744073709551615 260905054917 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/4802/statm: 4104 2117 155 36 0 2009 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 16416

[startup+0.700583 s]
/proc/loadavg: 6.90 7.84 7.95 2/177 4802
/proc/meminfo: memFree=23623420/32950928 swapFree=67071984/67111528
[pid=4802] ppid=4800 vsize=19980 CPUtime=0.69 cores=1,3,5,7
/proc/4802/stat : 4802 (ttsth-5-4) R 4800 4802 4175 0 -1 4202496 7292 0 0 0 69 0 0 0 25 0 1 0 599526475 20459520 3012 33554432000 4194304 4338100 140737294761312 18446744073709551615 260904789186 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/4802/statm: 4995 3012 161 36 0 2900 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 19980

[startup+1.50038 s]
/proc/loadavg: 6.90 7.84 7.95 11/186 4811
/proc/meminfo: memFree=23476788/32950928 swapFree=67071984/67111528
[pid=4802] ppid=4800 vsize=277156 CPUtime=3.32 cores=1,3,5,7
/proc/4802/stat : 4802 (ttsth-5-4) S 4800 4802 4175 0 -1 4202496 27290 0 0 0 329 3 0 0 25 0 5 0 599526475 283807744 22718 33554432000 4194304 4338100 140737294761312 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/4802/statm: 69289 22719 176 36 0 67194 0
[pid=4802/tid=4808] ppid=4800 vsize=277156 CPUtime=0.61 cores=1,3,5,7
/proc/4802/task/4808/stat : 4808 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 2487 0 0 0 61 0 0 0 18 0 5 0 599526563 283807744 22719 33554432000 4194304 4338100 140737294761312 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=4802/tid=4809] ppid=4800 vsize=277156 CPUtime=0.61 cores=1,3,5,7
/proc/4802/task/4809/stat : 4809 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 1696 0 0 0 61 0 0 0 18 0 5 0 599526563 283807744 22719 33554432000 4194304 4338100 140737294761312 18446744073709551615 260917206734 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=4802/tid=4810] ppid=4800 vsize=277156 CPUtime=0.59 cores=1,3,5,7
/proc/4802/task/4810/stat : 4810 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 1965 0 0 0 59 0 0 0 25 0 5 0 599526563 283807744 22719 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=4802/tid=4811] ppid=4800 vsize=277156 CPUtime=0.6 cores=1,3,5,7
/proc/4802/task/4811/stat : 4811 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 2135 0 0 0 60 0 0 0 18 0 5 0 599526563 283807744 22719 33554432000 4194304 4338100 140737294761312 18446744073709551615 4256403 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3.32
Current children cumulated vsize (KiB) 277156

[startup+3.10109 s]
/proc/loadavg: 6.90 7.84 7.95 9/186 4811
/proc/meminfo: memFree=23364952/32950928 swapFree=67071984/67111528
[pid=4802] ppid=4800 vsize=350160 CPUtime=9.72 cores=1,3,5,7
/proc/4802/stat : 4802 (ttsth-5-4) S 4800 4802 4175 0 -1 4202496 45369 0 0 0 965 7 0 0 25 0 5 0 599526475 358563840 40797 33554432000 4194304 4338100 140737294761312 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/4802/statm: 87540 40797 176 36 0 85445 0
[pid=4802/tid=4808] ppid=4800 vsize=350160 CPUtime=2.21 cores=1,3,5,7
/proc/4802/task/4808/stat : 4808 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 7602 0 0 0 220 1 0 0 21 0 5 0 599526563 358563840 40797 33554432000 4194304 4338100 140737294761312 18446744073709551615 4256115 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=4802/tid=4809] ppid=4800 vsize=350160 CPUtime=2.21 cores=1,3,5,7
/proc/4802/task/4809/stat : 4809 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 5265 0 0 0 220 1 0 0 21 0 5 0 599526563 358563840 40797 33554432000 4194304 4338100 140737294761312 18446744073709551615 4284177 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=4802/tid=4810] ppid=4800 vsize=350160 CPUtime=2.19 cores=1,3,5,7
/proc/4802/task/4810/stat : 4810 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 6540 0 0 0 218 1 0 0 25 0 5 0 599526563 358563840 40797 33554432000 4194304 4338100 140737294761312 18446744073709551615 4256400 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=4802/tid=4811] ppid=4800 vsize=350160 CPUtime=2.2 cores=1,3,5,7
/proc/4802/task/4811/stat : 4811 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 6954 0 0 0 220 0 0 0 21 0 5 0 599526563 358563840 40797 33554432000 4194304 4338100 140737294761312 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 9.72
Current children cumulated vsize (KiB) 350160

[startup+6.3005 s]
/proc/loadavg: 6.99 7.84 7.95 9/186 4811
/proc/meminfo: memFree=23094676/32950928 swapFree=67071984/67111528
[pid=4802] ppid=4800 vsize=472948 CPUtime=22.5 cores=1,3,5,7
/proc/4802/stat : 4802 (ttsth-5-4) S 4800 4802 4175 0 -1 4202496 76056 0 0 0 2237 13 0 0 25 0 5 0 599526475 484298752 71485 33554432000 4194304 4338100 140737294761312 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/4802/statm: 118237 71485 176 36 0 116142 0
[pid=4802/tid=4808] ppid=4800 vsize=472948 CPUtime=5.41 cores=1,3,5,7
/proc/4802/task/4808/stat : 4808 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 16488 0 0 0 538 3 0 0 25 0 5 0 599526563 484298752 71485 33554432000 4194304 4338100 140737294761312 18446744073709551615 4287731 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=4802/tid=4809] ppid=4800 vsize=472948 CPUtime=5.4 cores=1,3,5,7
/proc/4802/task/4809/stat : 4809 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 11837 0 0 0 538 2 0 0 25 0 5 0 599526563 484298752 71485 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=4802/tid=4810] ppid=4800 vsize=472948 CPUtime=5.37 cores=1,3,5,7
/proc/4802/task/4810/stat : 4810 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 14060 0 0 0 534 3 0 0 25 0 5 0 599526563 484298752 71485 33554432000 4194304 4338100 140737294761312 18446744073709551615 4284177 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=4802/tid=4811] ppid=4800 vsize=472948 CPUtime=5.41 cores=1,3,5,7
/proc/4802/task/4811/stat : 4811 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 14664 0 0 0 539 2 0 0 25 0 5 0 599526563 484298752 71485 33554432000 4194304 4338100 140737294761312 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 22.5
Current children cumulated vsize (KiB) 472948

[startup+12.7003 s]
/proc/loadavg: 7.07 7.84 7.95 9/186 4811
/proc/meminfo: memFree=22659816/32950928 swapFree=67071984/67111528
[pid=4802] ppid=4800 vsize=683588 CPUtime=48.07 cores=1,3,5,7
/proc/4802/stat : 4802 (ttsth-5-4) S 4800 4802 4175 0 -1 4202496 129621 0 0 0 4782 25 0 0 25 0 5 0 599526475 699994112 125049 33554432000 4194304 4338100 140737294761312 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/4802/statm: 170897 125049 176 36 0 168802 0
[pid=4802/tid=4808] ppid=4800 vsize=683588 CPUtime=11.81 cores=1,3,5,7
/proc/4802/task/4808/stat : 4808 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 31175 0 0 0 1175 6 0 0 25 0 5 0 599526563 699994112 125049 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=4802/tid=4809] ppid=4800 vsize=683588 CPUtime=11.8 cores=1,3,5,7
/proc/4802/task/4809/stat : 4809 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 23903 0 0 0 1176 4 0 0 25 0 5 0 599526563 699994112 125049 33554432000 4194304 4338100 140737294761312 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=4802/tid=4810] ppid=4800 vsize=683588 CPUtime=11.74 cores=1,3,5,7
/proc/4802/task/4810/stat : 4810 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 27752 0 0 0 1168 6 0 0 25 0 5 0 599526563 699994112 125049 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=4802/tid=4811] ppid=4800 vsize=683588 CPUtime=11.81 cores=1,3,5,7
/proc/4802/task/4811/stat : 4811 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 27783 0 0 0 1176 5 0 0 25 0 5 0 599526563 699994112 125049 33554432000 4194304 4338100 140737294761312 18446744073709551615 4284214 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 48.07
Current children cumulated vsize (KiB) 683588

[startup+25.501 s]
/proc/loadavg: 7.28 7.85 7.95 9/186 4812
/proc/meminfo: memFree=21936900/32950928 swapFree=67071984/67111528
[pid=4802] ppid=4800 vsize=1041368 CPUtime=99.2 cores=1,3,5,7
/proc/4802/stat : 4802 (ttsth-5-4) S 4800 4802 4175 0 -1 4202496 218583 0 0 0 9875 45 0 0 25 0 5 0 599526475 1066360832 214011 33554432000 4194304 4338100 140737294761312 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/4802/statm: 260342 214012 176 36 0 258247 0
[pid=4802/tid=4808] ppid=4800 vsize=1041368 CPUtime=24.61 cores=1,3,5,7
/proc/4802/task/4808/stat : 4808 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 55393 0 0 0 2450 11 0 0 25 0 5 0 599526563 1066360832 214012 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=4802/tid=4809] ppid=4800 vsize=1041368 CPUtime=24.6 cores=1,3,5,7
/proc/4802/task/4809/stat : 4809 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 44038 0 0 0 2452 8 0 0 25 0 5 0 599526563 1066360832 214012 33554432000 4194304 4338100 140737294761312 18446744073709551615 4285837 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=4802/tid=4810] ppid=4800 vsize=1041368 CPUtime=24.47 cores=1,3,5,7
/proc/4802/task/4810/stat : 4810 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 50334 0 0 0 2435 12 0 0 25 0 5 0 599526563 1066360832 214013 33554432000 4194304 4338100 140737294761312 18446744073709551615 4256032 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=4802/tid=4811] ppid=4800 vsize=1041368 CPUtime=24.61 cores=1,3,5,7
/proc/4802/task/4811/stat : 4811 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 49812 0 0 0 2450 11 0 0 25 0 5 0 599526563 1066360832 214013 33554432000 4194304 4338100 140737294761312 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 99.2
Current children cumulated vsize (KiB) 1041368

[startup+51.1003 s]
/proc/loadavg: 7.60 7.88 7.96 9/186 4812
/proc/meminfo: memFree=20808080/32950928 swapFree=67071984/67111528
[pid=4802] ppid=4800 vsize=1621632 CPUtime=201.46 cores=1,3,5,7
/proc/4802/stat : 4802 (ttsth-5-4) S 4800 4802 4175 0 -1 4202496 364612 0 0 0 20073 73 0 0 25 0 5 0 599526475 1660551168 360040 33554432000 4194304 4338100 140737294761312 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/4802/statm: 405408 360040 176 36 0 403313 0
[pid=4802/tid=4808] ppid=4800 vsize=1621632 CPUtime=50.21 cores=1,3,5,7
/proc/4802/task/4808/stat : 4808 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 94947 0 0 0 5002 19 0 0 25 0 5 0 599526563 1660551168 360040 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=4802/tid=4809] ppid=4800 vsize=1621632 CPUtime=50.2 cores=1,3,5,7
/proc/4802/task/4809/stat : 4809 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 76441 0 0 0 5006 14 0 0 25 0 5 0 599526563 1660551168 360040 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=4802/tid=4810] ppid=4800 vsize=1621632 CPUtime=49.93 cores=1,3,5,7
/proc/4802/task/4810/stat : 4810 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 89802 0 0 0 4975 18 0 0 25 0 5 0 599526563 1660551168 360040 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=4802/tid=4811] ppid=4800 vsize=1621632 CPUtime=50.21 cores=1,3,5,7
/proc/4802/task/4811/stat : 4811 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 84414 0 0 0 5003 18 0 0 25 0 5 0 599526563 1660551168 360040 33554432000 4194304 4338100 140737294761312 18446744073709551615 260917206734 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 201.46
Current children cumulated vsize (KiB) 1621632

[startup+102.301 s]
/proc/loadavg: 7.86 7.91 7.96 9/186 4814
/proc/meminfo: memFree=19033360/32950928 swapFree=67071984/67111528
[pid=4802] ppid=4800 vsize=2530448 CPUtime=406 cores=1,3,5,7
/proc/4802/stat : 4802 (ttsth-5-4) S 4800 4802 4175 0 -1 4202496 590940 0 0 0 40485 115 0 0 25 0 5 0 599526475 2591178752 586368 33554432000 4194304 4338100 140737294761312 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/4802/statm: 632612 586368 176 36 0 630517 0
[pid=4802/tid=4808] ppid=4800 vsize=2530448 CPUtime=101.42 cores=1,3,5,7
/proc/4802/task/4808/stat : 4808 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 153477 0 0 0 10110 32 0 0 25 0 5 0 599526563 2591178752 586368 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=4802/tid=4809] ppid=4800 vsize=2530448 CPUtime=101.41 cores=1,3,5,7
/proc/4802/task/4809/stat : 4809 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 123817 0 0 0 10118 23 0 0 25 0 5 0 599526563 2591178752 586368 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=4802/tid=4810] ppid=4800 vsize=2530448 CPUtime=100.86 cores=1,3,5,7
/proc/4802/task/4810/stat : 4810 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 151403 0 0 0 10055 31 0 0 25 0 5 0 599526563 2591178752 586368 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=4802/tid=4811] ppid=4800 vsize=2530448 CPUtime=101.42 cores=1,3,5,7
/proc/4802/task/4811/stat : 4811 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 143235 0 0 0 10115 27 0 0 25 0 5 0 599526563 2591178752 586368 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 406
Current children cumulated vsize (KiB) 2530448

[startup+162.301 s]
/proc/loadavg: 8.10 7.96 7.98 9/186 4815
/proc/meminfo: memFree=17338388/32950928 swapFree=67071984/67111528
[pid=4802] ppid=4800 vsize=3404352 CPUtime=645.65 cores=1,3,5,7
/proc/4802/stat : 4802 (ttsth-5-4) S 4800 4802 4175 0 -1 4202496 810955 0 0 0 64406 159 0 0 25 0 5 0 599526475 3486056448 806383 33554432000 4194304 4338100 140737294761312 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/4802/statm: 851088 806383 176 36 0 848993 0
[pid=4802/tid=4808] ppid=4800 vsize=3404352 CPUtime=161.29 cores=1,3,5,7
/proc/4802/task/4808/stat : 4808 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 211085 0 0 0 16087 42 0 0 25 0 5 0 599526563 3486056448 806383 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=4802/tid=4809] ppid=4800 vsize=3404352 CPUtime=161.4 cores=1,3,5,7
/proc/4802/task/4809/stat : 4809 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 174491 0 0 0 16109 31 0 0 25 0 5 0 599526563 3486056448 806383 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=4802/tid=4810] ppid=4800 vsize=3404352 CPUtime=160.64 cores=1,3,5,7
/proc/4802/task/4810/stat : 4810 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 212938 0 0 0 16020 44 0 0 25 0 5 0 599526563 3486056448 806383 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=4802/tid=4811] ppid=4800 vsize=3404352 CPUtime=161.41 cores=1,3,5,7
/proc/4802/task/4811/stat : 4811 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 193433 0 0 0 16103 38 0 0 25 0 5 0 599526563 3486056448 806383 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 645.65
Current children cumulated vsize (KiB) 3404352


################
# More data... #
################

/proc/4802/task/4811/stat : 4811 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 473300 0 0 0 70038 103 0 0 25 0 5 0 599526563 8504918016 2023726 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254400 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2802.73
Current children cumulated vsize (KiB) 8305584

[startup+762.301 s]
/proc/loadavg: 8.13 8.10 8.02 9/186 4831
/proc/meminfo: memFree=8240352/32950928 swapFree=67071984/67111528
[pid=4802] ppid=4800 vsize=8753416 CPUtime=3042.43 cores=1,3,5,7
/proc/4802/stat : 4802 (ttsth-5-4) S 4800 4802 4175 0 -1 4202496 2140058 0 0 0 303802 441 0 0 25 0 5 0 599526475 8963497984 2135486 33554432000 4194304 4338100 140737294761312 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/4802/statm: 2188354 2135486 176 36 0 2186259 0
[pid=4802/tid=4808] ppid=4800 vsize=8753416 CPUtime=758.03 cores=1,3,5,7
/proc/4802/task/4808/stat : 4808 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 580973 0 0 0 75685 118 0 0 25 0 5 0 599526563 8963497984 2135486 33554432000 4194304 4338100 140737294761312 18446744073709551615 4241570 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=4802/tid=4809] ppid=4800 vsize=8753416 CPUtime=761.43 cores=1,3,5,7
/proc/4802/task/4809/stat : 4809 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 482598 0 0 0 76044 99 0 0 25 0 5 0 599526563 8963497984 2135486 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=4802/tid=4810] ppid=4800 vsize=8753416 CPUtime=760.65 cores=1,3,5,7
/proc/4802/task/4810/stat : 4810 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 564645 0 0 0 75952 113 0 0 25 0 5 0 599526563 8963497984 2135486 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=4802/tid=4811] ppid=4800 vsize=8753416 CPUtime=761.41 cores=1,3,5,7
/proc/4802/task/4811/stat : 4811 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 492834 0 0 0 76034 107 0 0 25 0 5 0 599526563 8963497984 2135486 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3042.43
Current children cumulated vsize (KiB) 8753416

[startup+822.301 s]
/proc/loadavg: 8.13 8.11 8.03 9/186 4833
/proc/meminfo: memFree=7442132/32950928 swapFree=67071984/67111528
[pid=4802] ppid=4800 vsize=9223144 CPUtime=3282.09 cores=1,3,5,7
/proc/4802/stat : 4802 (ttsth-5-4) S 4800 4802 4175 0 -1 4202496 2258702 0 0 0 327744 465 0 0 25 0 5 0 599526475 9444499456 2254130 33554432000 4194304 4338100 140737294761312 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/4802/statm: 2305786 2254130 176 36 0 2303691 0
[pid=4802/tid=4808] ppid=4800 vsize=9223144 CPUtime=817.7 cores=1,3,5,7
/proc/4802/task/4808/stat : 4808 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 608260 0 0 0 81645 125 0 0 25 0 5 0 599526563 9444499456 2254130 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=4802/tid=4809] ppid=4800 vsize=9223144 CPUtime=821.43 cores=1,3,5,7
/proc/4802/task/4809/stat : 4809 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 518133 0 0 0 82037 106 0 0 25 0 5 0 599526563 9444499456 2254130 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=4802/tid=4810] ppid=4800 vsize=9223144 CPUtime=820.64 cores=1,3,5,7
/proc/4802/task/4810/stat : 4810 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 595516 0 0 0 81946 118 0 0 25 0 5 0 599526563 9444499456 2254130 33554432000 4194304 4338100 140737294761312 18446744073709551615 4284185 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=4802/tid=4811] ppid=4800 vsize=9223144 CPUtime=821.41 cores=1,3,5,7
/proc/4802/task/4811/stat : 4811 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 517785 0 0 0 82028 113 0 0 25 0 5 0 599526563 9444499456 2254130 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3282.09
Current children cumulated vsize (KiB) 9223144

[startup+882.307 s]
/proc/loadavg: 8.04 8.09 8.02 9/186 4834
/proc/meminfo: memFree=6752400/32950928 swapFree=67071984/67111528
[pid=4802] ppid=4800 vsize=9567752 CPUtime=3521.79 cores=1,3,5,7
/proc/4802/stat : 4802 (ttsth-5-4) S 4800 4802 4175 0 -1 4202496 2344140 0 0 0 351696 483 0 0 25 0 5 0 599526475 9797378048 2339568 33554432000 4194304 4338100 140737294761312 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/4802/statm: 2391938 2339568 176 36 0 2389843 0
[pid=4802/tid=4808] ppid=4800 vsize=9567752 CPUtime=877.38 cores=1,3,5,7
/proc/4802/task/4808/stat : 4808 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 631473 0 0 0 87609 129 0 0 25 0 5 0 599526563 9797378048 2339568 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=4802/tid=4809] ppid=4800 vsize=9567752 CPUtime=881.43 cores=1,3,5,7
/proc/4802/task/4809/stat : 4809 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 537133 0 0 0 88034 109 0 0 25 0 5 0 599526563 9797378048 2339568 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=4802/tid=4810] ppid=4800 vsize=9567752 CPUtime=880.65 cores=1,3,5,7
/proc/4802/task/4810/stat : 4810 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 617044 0 0 0 87942 123 0 0 25 0 5 0 599526563 9797378048 2339568 33554432000 4194304 4338100 140737294761312 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=4802/tid=4811] ppid=4800 vsize=9567752 CPUtime=881.41 cores=1,3,5,7
/proc/4802/task/4811/stat : 4811 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 539482 0 0 0 88023 118 0 0 25 0 5 0 599526563 9797378048 2339568 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3521.79
Current children cumulated vsize (KiB) 9567752

[startup+942.301 s]
/proc/loadavg: 8.01 8.07 8.02 9/186 4836
/proc/meminfo: memFree=5937064/32950928 swapFree=67071984/67111528
[pid=4802] ppid=4800 vsize=9996816 CPUtime=3761.45 cores=1,3,5,7
/proc/4802/stat : 4802 (ttsth-5-4) S 4800 4802 4175 0 -1 4202496 2452382 0 0 0 375636 509 0 0 25 0 5 0 599526475 10236739584 2447810 33554432000 4194304 4338100 140737294761312 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/4802/statm: 2499204 2447810 176 36 0 2497109 0
[pid=4802/tid=4808] ppid=4800 vsize=9996816 CPUtime=937.05 cores=1,3,5,7
/proc/4802/task/4808/stat : 4808 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 661731 0 0 0 93569 136 0 0 25 0 5 0 599526563 10236739584 2447810 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=4802/tid=4809] ppid=4800 vsize=9996816 CPUtime=941.42 cores=1,3,5,7
/proc/4802/task/4809/stat : 4809 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 563578 0 0 0 94026 116 0 0 25 0 5 0 599526563 10236739584 2447810 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=4802/tid=4810] ppid=4800 vsize=9996816 CPUtime=940.65 cores=1,3,5,7
/proc/4802/task/4810/stat : 4810 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 644979 0 0 0 93936 129 0 0 25 0 5 0 599526563 10236739584 2447810 33554432000 4194304 4338100 140737294761312 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=4802/tid=4811] ppid=4800 vsize=9996816 CPUtime=941.41 cores=1,3,5,7
/proc/4802/task/4811/stat : 4811 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 563086 0 0 0 94017 124 0 0 25 0 5 0 599526563 10236739584 2447810 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3761.45
Current children cumulated vsize (KiB) 9996816

[startup+1002.3 s]
/proc/loadavg: 8.00 8.05 8.01 9/186 4838
/proc/meminfo: memFree=5305732/32950928 swapFree=67071984/67111528
[pid=4802] ppid=4800 vsize=10453900 CPUtime=4001.1 cores=1,3,5,7
/proc/4802/stat : 4802 (ttsth-5-4) S 4800 4802 4175 0 -1 4202496 2566763 0 0 0 399575 535 0 0 25 0 5 0 599526475 10704793600 2562191 33554432000 4194304 4338100 140737294761312 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/4802/statm: 2613475 2562191 176 36 0 2611380 0
[pid=4802/tid=4808] ppid=4800 vsize=10453900 CPUtime=996.73 cores=1,3,5,7
/proc/4802/task/4808/stat : 4808 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 693936 0 0 0 99528 145 0 0 25 0 5 0 599526563 10704793600 2562191 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=4802/tid=4809] ppid=4800 vsize=10453900 CPUtime=1001.42 cores=1,3,5,7
/proc/4802/task/4809/stat : 4809 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 588186 0 0 0 100021 121 0 0 25 0 5 0 599526563 10704793600 2562191 33554432000 4194304 4338100 140737294761312 18446744073709551615 4256139 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=4802/tid=4810] ppid=4800 vsize=10453900 CPUtime=1000.63 cores=1,3,5,7
/proc/4802/task/4810/stat : 4810 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 678203 0 0 0 99928 135 0 0 25 0 5 0 599526563 10704793600 2562192 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=4802/tid=4811] ppid=4800 vsize=10453900 CPUtime=1001.41 cores=1,3,5,7
/proc/4802/task/4811/stat : 4811 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 587430 0 0 0 100011 130 0 0 25 0 5 0 599526563 10704793600 2562193 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4001.1
Current children cumulated vsize (KiB) 10453900

[startup+1062.3 s]
/proc/loadavg: 8.00 8.04 8.00 9/186 4839
/proc/meminfo: memFree=4560692/32950928 swapFree=67071984/67111528
[pid=4802] ppid=4800 vsize=10860936 CPUtime=4240.78 cores=1,3,5,7
/proc/4802/stat : 4802 (ttsth-5-4) S 4800 4802 4175 0 -1 4202496 2668332 0 0 0 423520 558 0 0 25 0 5 0 599526475 11121598464 2663760 33554432000 4194304 4338100 140737294761312 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/4802/statm: 2715234 2663760 176 36 0 2713139 0
[pid=4802/tid=4808] ppid=4800 vsize=10860936 CPUtime=1056.4 cores=1,3,5,7
/proc/4802/task/4808/stat : 4808 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 724447 0 0 0 105489 151 0 0 25 0 5 0 599526563 11121598464 2663760 33554432000 4194304 4338100 140737294761312 18446744073709551615 4256025 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=4802/tid=4809] ppid=4800 vsize=10860936 CPUtime=1061.43 cores=1,3,5,7
/proc/4802/task/4809/stat : 4809 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 612269 0 0 0 106016 127 0 0 25 0 5 0 599526563 11121598464 2663760 33554432000 4194304 4338100 140737294761312 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=4802/tid=4810] ppid=4800 vsize=10860936 CPUtime=1060.63 cores=1,3,5,7
/proc/4802/task/4810/stat : 4810 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 706535 0 0 0 105921 142 0 0 25 0 5 0 599526563 11121598464 2663760 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=4802/tid=4811] ppid=4800 vsize=10860936 CPUtime=1061.41 cores=1,3,5,7
/proc/4802/task/4811/stat : 4811 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 606073 0 0 0 106006 135 0 0 25 0 5 0 599526563 11121598464 2663760 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4240.78
Current children cumulated vsize (KiB) 10860936

[startup+1122.3 s]
/proc/loadavg: 8.00 8.03 8.00 9/186 4841
/proc/meminfo: memFree=3940756/32950928 swapFree=67071984/67111528
[pid=4802] ppid=4800 vsize=11224180 CPUtime=4480.44 cores=1,3,5,7
/proc/4802/stat : 4802 (ttsth-5-4) S 4800 4802 4175 0 -1 4202496 2759461 0 0 0 447464 580 0 0 25 0 5 0 599526475 11493560320 2754889 33554432000 4194304 4338100 140737294761312 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/4802/statm: 2806045 2754889 176 36 0 2803950 0
[pid=4802/tid=4808] ppid=4800 vsize=11224180 CPUtime=1116.07 cores=1,3,5,7
/proc/4802/task/4808/stat : 4808 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 750329 0 0 0 111450 157 0 0 25 0 5 0 599526563 11493560320 2754889 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254257 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=4802/tid=4809] ppid=4800 vsize=11224180 CPUtime=1121.43 cores=1,3,5,7
/proc/4802/task/4809/stat : 4809 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 626951 0 0 0 112013 130 0 0 25 0 5 0 599526563 11493560320 2754889 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=4802/tid=4810] ppid=4800 vsize=11224180 CPUtime=1120.61 cores=1,3,5,7
/proc/4802/task/4810/stat : 4810 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 736356 0 0 0 111911 150 0 0 25 0 5 0 599526563 11493560320 2754889 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=4802/tid=4811] ppid=4800 vsize=11224180 CPUtime=1121.42 cores=1,3,5,7
/proc/4802/task/4811/stat : 4811 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 626817 0 0 0 112002 140 0 0 25 0 5 0 599526563 11493560320 2754889 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4480.44
Current children cumulated vsize (KiB) 11224180

[startup+1182.3 s]
/proc/loadavg: 8.00 8.02 8.00 9/186 4842
/proc/meminfo: memFree=3221396/32950928 swapFree=67071984/67111528
[pid=4802] ppid=4800 vsize=11621700 CPUtime=4720.07 cores=1,3,5,7
/proc/4802/stat : 4802 (ttsth-5-4) S 4800 4802 4175 0 -1 4202496 2858196 0 0 0 471404 603 0 0 25 0 5 0 599526475 11900620800 2853624 33554432000 4194304 4338100 140737294761312 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/4802/statm: 2905425 2853624 176 36 0 2903330 0
[pid=4802/tid=4808] ppid=4800 vsize=11621700 CPUtime=1175.73 cores=1,3,5,7
/proc/4802/task/4808/stat : 4808 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 784259 0 0 0 117409 164 0 0 25 0 5 0 599526563 11900620800 2853624 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=4802/tid=4809] ppid=4800 vsize=11621700 CPUtime=1181.41 cores=1,3,5,7
/proc/4802/task/4809/stat : 4809 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 647859 0 0 0 118007 134 0 0 25 0 5 0 599526563 11900620800 2853624 33554432000 4194304 4338100 140737294761312 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=4802/tid=4810] ppid=4800 vsize=11621700 CPUtime=1180.6 cores=1,3,5,7
/proc/4802/task/4810/stat : 4810 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 763944 0 0 0 117903 157 0 0 25 0 5 0 599526563 11900620800 2853624 33554432000 4194304 4338100 140737294761312 18446744073709551615 4287678 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=4802/tid=4811] ppid=4800 vsize=11621700 CPUtime=1181.42 cores=1,3,5,7
/proc/4802/task/4811/stat : 4811 (ttsth-5-4) R 4800 4802 4175 0 -1 4202560 643126 0 0 0 117998 144 0 0 25 0 5 0 599526563 11900620800 2853624 33554432000 4194304 4338100 140737294761312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4720.07
Current children cumulated vsize (KiB) 11621700



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.41 s]
/proc/loadavg: 8.00 8.02 8.00 2/177 4844
/proc/meminfo: memFree=13072324/32950928 swapFree=67071984/67111528
[pid=4802] ppid=4800 vsize=0 CPUtime=4800.26 cores=1,3,5,7
/proc/4802/stat : 4802 (ttsth-5-4) Z 4800 4802 4175 0 -1 4203548 2894732 0 0 0 479410 616 0 0 18 0 2 0 599526475 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/4802/statm: 0 0 0 0 0 0 0
[pid=4802/tid=4809] ppid=4800 vsize=0 CPUtime=1201.52 cores=1,3,5,7
/proc/4802/task/4809/stat : 4809 (ttsth-5-4) R 4800 4802 4175 0 -1 4203588 658073 0 0 0 120011 141 0 0 25 0 2 0 599526563 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.26
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.7 s]
/proc/loadavg: 8.00 8.02 8.00 2/177 4844
/proc/meminfo: memFree=13072324/32950928 swapFree=67071984/67111528
[pid=4802] ppid=4800 vsize=0 CPUtime=4800.55 cores=1,3,5,7
/proc/4802/stat : 4802 (ttsth-5-4) Z 4800 4802 4175 0 -1 4203548 2894732 0 0 0 479410 645 0 0 18 0 2 0 599526475 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/4802/statm: 0 0 0 0 0 0 0
[pid=4802/tid=4809] ppid=4800 vsize=0 CPUtime=1201.81 cores=1,3,5,7
/proc/4802/task/4809/stat : 4809 (ttsth-5-4) R 4800 4802 4175 0 -1 4203588 658073 0 0 0 120011 170 0 0 25 0 2 0 599526563 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.55
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1202.75
CPU time (s): 4800.61
CPU user time (s): 4794.11
CPU system time (s): 6.49901
CPU usage (%): 399.137
Max. virtual memory (cumulated for all children) (KiB): 11768980

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4794.11
system time used= 6.49901
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2894732
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 851
involuntary context switches= 30975

runsolver used 1.90271 second user time and 4.66929 second system time

The end

Launcher Data 

Begin job on node116 at 2011-04-29 18:02:38
IDJOB=3250604
IDBENCH=83321
IDSOLVER=1654
FILE ID=node116/3250604-1304092958
RUNJOBID= node116-1304092958-4783
PBS_JOBID= 13170553
Free space on /tmp= 72512 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/kullmann/VanDerWaerden/VanderWaerden3k/VanDerWaerden_2-3-20_389.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3250604-1304092958/watcher-3250604-1304092958 -o /tmp/evaluation-result-3250604-1304092958/solver-3250604-1304092958 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250604-1304092958.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= f105570f47bdca662c2d3d185f8dc1f8
RANDOM SEED=251954979

node116.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      23623948 kB
Buffers:       1904212 kB
Cached:        6953084 kB
SwapCached:       9732 kB
Active:        4400900 kB
Inactive:      4498364 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      23623948 kB
SwapTotal:    67111528 kB
SwapFree:     67071984 kB
Dirty:            1324 kB
Writeback:           0 kB
AnonPages:       35848 kB
Mapped:          13880 kB
Slab:           365704 kB
PageTables:       4880 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   210952 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72516 MiB
End job on node116 at 2011-04-29 18:22:42