Trace number 3394082

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 40001.1 5008.65

General information on the benchmark

NameCRAFTED/rbsat/random/
forced/rbsat-v2640c305320gyes5.cnf
MD5SUMf892a2bda6fe6aec6f2fcb4e1c541df8
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2640
Number of clauses305320
Sum of the clauses size613120
Maximum clause length33
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2305240
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 580

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3394082-1306086005/watcher-3394082-1306086005 -o /tmp/evaluation-result-3394082-1306086005/solver-3394082-1306086005 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3394082-1306086005.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.06 8.05 8.00 2/379 29481
/proc/meminfo: memFree=10632104/32951124 swapFree=52307820/67111528
[pid=29481] ppid=29479 vsize=10440 CPUtime=0 cores=0-7
/proc/29481/stat : 29481 (plingeling) R 29479 29481 29455 0 -1 4202496 699 1252 0 0 0 0 0 0 25 0 1 0 720203282 10690560 628 33554432000 4194304 4417452 140734956598160 18446744073709551615 218370575997 0 0 4096 17442 0 0 0 17 0 0 0 0
/proc/29481/statm: 2610 628 147 55 0 496 0

[startup+0.0295621 s]
/proc/loadavg: 8.06 8.05 8.00 2/379 29481
/proc/meminfo: memFree=10632104/32951124 swapFree=52307820/67111528
[pid=29481] ppid=29479 vsize=12128 CPUtime=0.02 cores=0-7
/proc/29481/stat : 29481 (plingeling) R 29479 29481 29455 0 -1 4202496 1118 1252 0 0 2 0 0 0 25 0 1 0 720203282 12419072 1047 33554432000 4194304 4417452 140734956598160 18446744073709551615 218370957056 0 0 4096 17442 0 0 0 17 0 0 0 0
/proc/29481/statm: 3032 1047 148 55 0 918 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 12128

[startup+0.100537 s]
/proc/loadavg: 8.06 8.05 8.00 2/379 29481
/proc/meminfo: memFree=10632104/32951124 swapFree=52307820/67111528
[pid=29481] ppid=29479 vsize=27804 CPUtime=0.09 cores=0-7
/proc/29481/stat : 29481 (plingeling) R 29479 29481 29455 0 -1 4202496 5045 1252 0 0 8 1 0 0 25 0 1 0 720203282 28471296 4974 33554432000 4194304 4417452 140734956598160 18446744073709551615 4322355 0 0 4096 17442 0 0 0 17 0 0 0 0
/proc/29481/statm: 6951 4974 148 55 0 4837 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 27804

[startup+0.3005 s]
/proc/loadavg: 8.06 8.05 8.00 2/379 29481
/proc/meminfo: memFree=10632104/32951124 swapFree=52307820/67111528
[pid=29481] ppid=29479 vsize=76956 CPUtime=0.29 cores=0-7
/proc/29481/stat : 29481 (plingeling) R 29479 29481 29455 0 -1 4202496 17333 1252 0 0 27 2 0 0 25 0 1 0 720203282 78802944 17262 33554432000 4194304 4417452 140734956598160 18446744073709551615 4231265 0 0 4096 17442 0 0 0 17 0 0 0 0
/proc/29481/statm: 19239 17262 148 55 0 17125 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 76956

[startup+0.720999 s]
/proc/loadavg: 8.06 8.05 8.00 2/379 29481
/proc/meminfo: memFree=10632104/32951124 swapFree=52307820/67111528
[pid=29481] ppid=29479 vsize=249800 CPUtime=2.73 cores=0-7
/proc/29481/stat : 29481 (plingeling) S 29479 29481 29455 0 -1 4202496 27855 1252 0 0 269 4 0 0 18 0 9 0 720203282 255795200 22967 33554432000 4194304 4417452 140734956598160 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/29481/statm: 62450 22455 177 55 0 60336 0
Current children cumulated CPU time (s) 2.73
Current children cumulated vsize (KiB) 249800

[startup+1.50125 s]
/proc/loadavg: 8.05 8.05 8.00 9/388 29495
/proc/meminfo: memFree=10523904/32951124 swapFree=52307820/67111528
[pid=29481] ppid=29479 vsize=267440 CPUtime=8.96 cores=0-7
/proc/29481/stat : 29481 (plingeling) S 29479 29481 29455 0 -1 4202496 41962 1252 0 0 891 5 0 0 18 0 9 0 720203282 273858560 28094 33554432000 4194304 4417452 140734956598160 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/29481/statm: 66860 28094 180 55 0 64746 0
[pid=29481/tid=29488] ppid=29479 vsize=267440 CPUtime=1.06 cores=0-7
/proc/29481/task/29488/stat : 29488 (plingeling) R 29479 29481 29455 0 -1 4202560 2895 1252 0 0 106 0 0 0 19 0 9 0 720203325 273858560 28094 33554432000 4194304 4417452 140734956598160 18446744073709551615 4258462 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29481/tid=29489] ppid=29479 vsize=267440 CPUtime=1.06 cores=0-7
/proc/29481/task/29489/stat : 29489 (plingeling) R 29479 29481 29455 0 -1 4202560 1173 1252 0 0 106 0 0 0 25 0 9 0 720203325 273858560 28094 33554432000 4194304 4417452 140734956598160 18446744073709551615 4249601 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29481/tid=29490] ppid=29479 vsize=267440 CPUtime=1.07 cores=0-7
/proc/29481/task/29490/stat : 29490 (plingeling) R 29479 29481 29455 0 -1 4202560 3265 1252 0 0 107 0 0 0 25 0 9 0 720203325 273858560 28094 33554432000 4194304 4417452 140734956598160 18446744073709551615 4258434 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29481/tid=29491] ppid=29479 vsize=267440 CPUtime=1.07 cores=0-7
/proc/29481/task/29491/stat : 29491 (plingeling) R 29479 29481 29455 0 -1 4202560 4814 1252 0 0 107 0 0 0 19 0 9 0 720203325 273858560 28094 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=29481/tid=29492] ppid=29479 vsize=267440 CPUtime=1.05 cores=0-7
/proc/29481/task/29492/stat : 29492 (plingeling) R 29479 29481 29455 0 -1 4202560 3694 1252 0 0 105 0 0 0 19 0 9 0 720203325 273858560 28094 33554432000 4194304 4417452 140734956598160 18446744073709551615 4239393 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29481/tid=29493] ppid=29479 vsize=267440 CPUtime=1.06 cores=0-7
/proc/29481/task/29493/stat : 29493 (plingeling) R 29479 29481 29455 0 -1 4202560 2812 1252 0 0 106 0 0 0 21 0 9 0 720203325 273858560 28094 33554432000 4194304 4417452 140734956598160 18446744073709551615 4249584 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=29481/tid=29494] ppid=29479 vsize=267440 CPUtime=1.05 cores=0-7
/proc/29481/task/29494/stat : 29494 (plingeling) R 29479 29481 29455 0 -1 4202560 3156 1252 0 0 105 0 0 0 22 0 9 0 720203325 273858560 28094 33554432000 4194304 4417452 140734956598160 18446744073709551615 4249143 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=29481/tid=29495] ppid=29479 vsize=267440 CPUtime=1.06 cores=0-7
/proc/29481/task/29495/stat : 29495 (plingeling) R 29479 29481 29455 0 -1 4202560 2803 1252 0 0 106 0 0 0 24 0 9 0 720203325 273858560 28094 33554432000 4194304 4417452 140734956598160 18446744073709551615 4249110 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 8.96
Current children cumulated vsize (KiB) 267440

[startup+3.10097 s]
/proc/loadavg: 8.05 8.05 8.00 9/388 29495
/proc/meminfo: memFree=10514376/32951124 swapFree=52307820/67111528
[pid=29481] ppid=29479 vsize=292196 CPUtime=21.74 cores=0-7
/proc/29481/stat : 29481 (plingeling) S 29479 29481 29455 0 -1 4202496 53380 1252 0 0 2168 6 0 0 18 0 9 0 720203282 299208704 34283 33554432000 4194304 4417452 140734956598160 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/29481/statm: 73049 34283 180 55 0 70935 0
[pid=29481/tid=29488] ppid=29479 vsize=292196 CPUtime=2.66 cores=0-7
/proc/29481/task/29488/stat : 29488 (plingeling) R 29479 29481 29455 0 -1 4202560 4256 1252 0 0 266 0 0 0 23 0 9 0 720203325 299208704 34283 33554432000 4194304 4417452 140734956598160 18446744073709551615 4249675 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29481/tid=29489] ppid=29479 vsize=292196 CPUtime=2.66 cores=0-7
/proc/29481/task/29489/stat : 29489 (plingeling) R 29479 29481 29455 0 -1 4202560 2534 1252 0 0 266 0 0 0 25 0 9 0 720203325 299208704 34283 33554432000 4194304 4417452 140734956598160 18446744073709551615 4250381 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29481/tid=29490] ppid=29479 vsize=292196 CPUtime=2.67 cores=0-7
/proc/29481/task/29490/stat : 29490 (plingeling) R 29479 29481 29455 0 -1 4202560 3265 1252 0 0 267 0 0 0 25 0 9 0 720203325 299208704 34283 33554432000 4194304 4417452 140734956598160 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29481/tid=29491] ppid=29479 vsize=292196 CPUtime=2.67 cores=0-7
/proc/29481/task/29491/stat : 29491 (plingeling) R 29479 29481 29455 0 -1 4202560 10788 1252 0 0 266 1 0 0 22 0 9 0 720203325 299208704 34283 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=29481/tid=29492] ppid=29479 vsize=292196 CPUtime=2.64 cores=0-7
/proc/29481/task/29492/stat : 29492 (plingeling) R 29479 29481 29455 0 -1 4202560 5055 1252 0 0 264 0 0 0 22 0 9 0 720203325 299208704 34283 33554432000 4194304 4417452 140734956598160 18446744073709551615 4258565 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29481/tid=29493] ppid=29479 vsize=292196 CPUtime=2.66 cores=0-7
/proc/29481/task/29493/stat : 29493 (plingeling) R 29479 29481 29455 0 -1 4202560 4173 1252 0 0 266 0 0 0 25 0 9 0 720203325 299208704 34283 33554432000 4194304 4417452 140734956598160 18446744073709551615 4249110 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=29481/tid=29494] ppid=29479 vsize=292196 CPUtime=2.65 cores=0-7
/proc/29481/task/29494/stat : 29494 (plingeling) R 29479 29481 29455 0 -1 4202560 3156 1252 0 0 265 0 0 0 25 0 9 0 720203325 299208704 34283 33554432000 4194304 4417452 140734956598160 18446744073709551615 4250003 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=29481/tid=29495] ppid=29479 vsize=292196 CPUtime=2.66 cores=0-7
/proc/29481/task/29495/stat : 29495 (plingeling) R 29479 29481 29455 0 -1 4202560 2803 1252 0 0 266 0 0 0 25 0 9 0 720203325 299208704 34283 33554432000 4194304 4417452 140734956598160 18446744073709551615 4236249 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 21.74
Current children cumulated vsize (KiB) 292196

[startup+6.30039 s]
/proc/loadavg: 8.05 8.05 8.00 9/388 29495
/proc/meminfo: memFree=10464296/32951124 swapFree=52307820/67111528
[pid=29481] ppid=29479 vsize=298632 CPUtime=47.31 cores=0-7
/proc/29481/stat : 29481 (plingeling) S 29479 29481 29455 0 -1 4202496 69402 1252 0 0 4721 10 0 0 18 0 9 0 720203282 305799168 35892 33554432000 4194304 4417452 140734956598160 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/29481/statm: 74658 35892 180 55 0 72544 0
[pid=29481/tid=29488] ppid=29479 vsize=298632 CPUtime=5.86 cores=0-7
/proc/29481/task/29488/stat : 29488 (plingeling) R 29479 29481 29455 0 -1 4202560 4256 1252 0 0 586 0 0 0 25 0 9 0 720203325 305799168 35892 33554432000 4194304 4417452 140734956598160 18446744073709551615 4249143 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29481/tid=29489] ppid=29479 vsize=298632 CPUtime=5.86 cores=0-7
/proc/29481/task/29489/stat : 29489 (plingeling) R 29479 29481 29455 0 -1 4202560 2534 1252 0 0 586 0 0 0 25 0 9 0 720203325 305799168 35892 33554432000 4194304 4417452 140734956598160 18446744073709551615 4335984 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29481/tid=29490] ppid=29479 vsize=298632 CPUtime=5.86 cores=0-7
/proc/29481/task/29490/stat : 29490 (plingeling) R 29479 29481 29455 0 -1 4202560 3265 1252 0 0 586 0 0 0 25 0 9 0 720203325 305799168 35892 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29481/tid=29491] ppid=29479 vsize=298632 CPUtime=5.87 cores=0-7
/proc/29481/task/29491/stat : 29491 (plingeling) R 29479 29481 29455 0 -1 4202560 26810 1252 0 0 583 4 0 0 25 0 9 0 720203325 305799168 35892 33554432000 4194304 4417452 140734956598160 18446744073709551615 4265326 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=29481/tid=29492] ppid=29479 vsize=298632 CPUtime=5.8 cores=0-7
/proc/29481/task/29492/stat : 29492 (plingeling) R 29479 29481 29455 0 -1 4202560 5055 1252 0 0 580 0 0 0 25 0 9 0 720203325 305799168 35892 33554432000 4194304 4417452 140734956598160 18446744073709551615 4249143 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29481/tid=29493] ppid=29479 vsize=298632 CPUtime=5.86 cores=0-7
/proc/29481/task/29493/stat : 29493 (plingeling) R 29479 29481 29455 0 -1 4202560 4173 1252 0 0 586 0 0 0 25 0 9 0 720203325 305799168 35892 33554432000 4194304 4417452 140734956598160 18446744073709551615 4239395 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=29481/tid=29494] ppid=29479 vsize=298632 CPUtime=5.85 cores=0-7
/proc/29481/task/29494/stat : 29494 (plingeling) R 29479 29481 29455 0 -1 4202560 3156 1252 0 0 585 0 0 0 25 0 9 0 720203325 305799168 35892 33554432000 4194304 4417452 140734956598160 18446744073709551615 4277897 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=29481/tid=29495] ppid=29479 vsize=298632 CPUtime=5.86 cores=0-7
/proc/29481/task/29495/stat : 29495 (plingeling) R 29479 29481 29455 0 -1 4202560 2803 1252 0 0 586 0 0 0 25 0 9 0 720203325 305799168 35892 33554432000 4194304 4417452 140734956598160 18446744073709551615 4249601 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 47.31
Current children cumulated vsize (KiB) 298632

[startup+12.7013 s]
/proc/loadavg: 8.12 8.07 8.01 9/388 29495
/proc/meminfo: memFree=10419404/32951124 swapFree=52307820/67111528
[pid=29481] ppid=29479 vsize=373252 CPUtime=98.43 cores=0-7
/proc/29481/stat : 29481 (plingeling) S 29479 29481 29455 0 -1 4202496 100483 1252 0 0 9829 14 0 0 18 0 9 0 720203282 382210048 54547 33554432000 4194304 4417452 140734956598160 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/29481/statm: 93313 54547 180 55 0 91199 0
[pid=29481/tid=29488] ppid=29479 vsize=373252 CPUtime=12.26 cores=0-7
/proc/29481/task/29488/stat : 29488 (plingeling) R 29479 29481 29455 0 -1 4202560 5616 1252 0 0 1226 0 0 0 25 0 9 0 720203325 382210048 54547 33554432000 4194304 4417452 140734956598160 18446744073709551615 4249601 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29481/tid=29489] ppid=29479 vsize=373252 CPUtime=12.26 cores=0-7
/proc/29481/task/29489/stat : 29489 (plingeling) R 29479 29481 29455 0 -1 4202560 3894 1252 0 0 1226 0 0 0 25 0 9 0 720203325 382210048 54547 33554432000 4194304 4417452 140734956598160 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29481/tid=29490] ppid=29479 vsize=373252 CPUtime=12.27 cores=0-7
/proc/29481/task/29490/stat : 29490 (plingeling) R 29479 29481 29455 0 -1 4202560 3265 1252 0 0 1227 0 0 0 25 0 9 0 720203325 382210048 54547 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364835 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29481/tid=29491] ppid=29479 vsize=373252 CPUtime=12.27 cores=0-7
/proc/29481/task/29491/stat : 29491 (plingeling) R 29479 29481 29455 0 -1 4202560 52451 1252 0 0 1219 8 0 0 25 0 9 0 720203325 382210048 54547 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=29481/tid=29492] ppid=29479 vsize=373252 CPUtime=12.12 cores=0-7
/proc/29481/task/29492/stat : 29492 (plingeling) R 29479 29481 29455 0 -1 4202560 6415 1252 0 0 1212 0 0 0 25 0 9 0 720203325 382210048 54547 33554432000 4194304 4417452 140734956598160 18446744073709551615 4249143 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29481/tid=29493] ppid=29479 vsize=373252 CPUtime=12.26 cores=0-7
/proc/29481/task/29493/stat : 29493 (plingeling) R 29479 29481 29455 0 -1 4202560 5533 1252 0 0 1225 1 0 0 25 0 9 0 720203325 382210048 54547 33554432000 4194304 4417452 140734956598160 18446744073709551615 4248888 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=29481/tid=29494] ppid=29479 vsize=373252 CPUtime=12.25 cores=0-7
/proc/29481/task/29494/stat : 29494 (plingeling) R 29479 29481 29455 0 -1 4202560 3156 1252 0 0 1225 0 0 0 25 0 9 0 720203325 382210048 54547 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364616 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=29481/tid=29495] ppid=29479 vsize=373252 CPUtime=12.26 cores=0-7
/proc/29481/task/29495/stat : 29495 (plingeling) R 29479 29481 29455 0 -1 4202560 2803 1252 0 0 1226 0 0 0 25 0 9 0 720203325 382210048 54547 33554432000 4194304 4417452 140734956598160 18446744073709551615 4249193 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 98.43
Current children cumulated vsize (KiB) 373252

[startup+25.501 s]
/proc/loadavg: 8.10 8.06 8.00 9/388 29495
/proc/meminfo: memFree=10283788/32951124 swapFree=52307820/67111528
[pid=29481] ppid=29479 vsize=503944 CPUtime=200.76 cores=0-7
/proc/29481/stat : 29481 (plingeling) S 29479 29481 29455 0 -1 4202496 168212 1252 0 0 20053 23 0 0 18 0 9 0 720203282 516038656 87804 33554432000 4194304 4417452 140734956598160 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/29481/statm: 125986 87804 180 55 0 123872 0
[pid=29481/tid=29488] ppid=29479 vsize=503944 CPUtime=25.07 cores=0-7
/proc/29481/task/29488/stat : 29488 (plingeling) R 29479 29481 29455 0 -1 4202560 5616 1252 0 0 2507 0 0 0 25 0 9 0 720203325 516038656 87804 33554432000 4194304 4417452 140734956598160 18446744073709551615 4265429 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29481/tid=29489] ppid=29479 vsize=503944 CPUtime=25.07 cores=0-7
/proc/29481/task/29489/stat : 29489 (plingeling) R 29479 29481 29455 0 -1 4202560 6007 1252 0 0 2507 0 0 0 25 0 9 0 720203325 516038656 87804 33554432000 4194304 4417452 140734956598160 18446744073709551615 4357787 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29481/tid=29490] ppid=29479 vsize=503944 CPUtime=25.07 cores=0-7
/proc/29481/task/29490/stat : 29490 (plingeling) R 29479 29481 29455 0 -1 4202560 7358 1252 0 0 2507 0 0 0 25 0 9 0 720203325 516038656 87804 33554432000 4194304 4417452 140734956598160 18446744073709551615 4235421 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29481/tid=29491] ppid=29479 vsize=503944 CPUtime=25.08 cores=0-7
/proc/29481/task/29491/stat : 29491 (plingeling) R 29479 29481 29455 0 -1 4202560 104341 1252 0 0 2493 15 0 0 25 0 9 0 720203325 516038656 87804 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364572 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=29481/tid=29492] ppid=29479 vsize=503944 CPUtime=24.78 cores=0-7
/proc/29481/task/29492/stat : 29492 (plingeling) R 29479 29481 29455 0 -1 4202560 10853 1252 0 0 2477 1 0 0 25 0 9 0 720203325 516038656 87804 33554432000 4194304 4417452 140734956598160 18446744073709551615 4344131 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29481/tid=29493] ppid=29479 vsize=503944 CPUtime=25.08 cores=0-7
/proc/29481/task/29493/stat : 29493 (plingeling) R 29479 29481 29455 0 -1 4202560 6526 1252 0 0 2507 1 0 0 25 0 9 0 720203325 516038656 87804 33554432000 4194304 4417452 140734956598160 18446744073709551615 4277864 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=29481/tid=29494] ppid=29479 vsize=503944 CPUtime=25.06 cores=0-7
/proc/29481/task/29494/stat : 29494 (plingeling) R 29479 29481 29455 0 -1 4202560 6333 1252 0 0 2506 0 0 0 25 0 9 0 720203325 516038656 87804 33554432000 4194304 4417452 140734956598160 18446744073709551615 4248926 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=29481/tid=29495] ppid=29479 vsize=503944 CPUtime=25.06 cores=0-7
/proc/29481/task/29495/stat : 29495 (plingeling) R 29479 29481 29455 0 -1 4202560 3828 1252 0 0 2506 0 0 0 25 0 9 0 720203325 516038656 87804 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364604 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 200.76
Current children cumulated vsize (KiB) 503944

[startup+51.1004 s]
/proc/loadavg: 8.06 8.06 8.00 9/388 29496
/proc/meminfo: memFree=10322136/32951124 swapFree=52307820/67111528
[pid=29481] ppid=29479 vsize=459980 CPUtime=405.16 cores=0-7
/proc/29481/stat : 29481 (plingeling) S 29479 29481 29455 0 -1 4202496 250197 1252 0 0 40479 37 0 0 18 0 9 0 720203282 471019520 79384 33554432000 4194304 4417452 140734956598160 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/29481/statm: 114995 79384 180 55 0 112881 0
[pid=29481/tid=29488] ppid=29479 vsize=459980 CPUtime=50.66 cores=0-7
/proc/29481/task/29488/stat : 29488 (plingeling) R 29479 29481 29455 0 -1 4202560 7718 1252 0 0 5066 0 0 0 25 0 9 0 720203325 471019520 79384 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364629 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29481/tid=29489] ppid=29479 vsize=459980 CPUtime=50.65 cores=0-7
/proc/29481/task/29489/stat : 29489 (plingeling) R 29479 29481 29455 0 -1 4202560 8199 1252 0 0 5065 0 0 0 25 0 9 0 720203325 471019520 79384 33554432000 4194304 4417452 140734956598160 18446744073709551615 4356864 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29481/tid=29490] ppid=29479 vsize=459980 CPUtime=50.66 cores=0-7

################
# More data... #
################

[pid=29481/tid=29490] ppid=29479 vsize=5190812 CPUtime=4721.93 cores=0-7
/proc/29481/task/29490/stat : 29490 (plingeling) R 29479 29481 29455 0 -1 4202560 471915 1252 0 0 472131 62 0 0 25 0 9 0 720203325 5315391488 1274246 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364817 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29481/tid=29491] ppid=29479 vsize=5190812 CPUtime=4721.98 cores=0-7
/proc/29481/task/29491/stat : 29491 (plingeling) R 29479 29481 29455 0 -1 4202560 2415287 1252 0 0 471750 448 0 0 25 0 9 0 720203325 5315391488 1274246 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=29481/tid=29492] ppid=29479 vsize=5190812 CPUtime=4698.09 cores=0-7
/proc/29481/task/29492/stat : 29492 (plingeling) R 29479 29481 29455 0 -1 4202560 937264 1252 0 0 469681 128 0 0 25 0 9 0 720203325 5315391488 1274246 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29481/tid=29493] ppid=29479 vsize=5190812 CPUtime=4721.83 cores=0-7
/proc/29481/task/29493/stat : 29493 (plingeling) R 29479 29481 29455 0 -1 4202560 683056 1252 0 0 472078 105 0 0 25 0 9 0 720203325 5315391488 1274246 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364737 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=29481/tid=29494] ppid=29479 vsize=5190812 CPUtime=4721.73 cores=0-7
/proc/29481/task/29494/stat : 29494 (plingeling) R 29479 29481 29455 0 -1 4202560 627160 1252 0 0 472077 96 0 0 25 0 9 0 720203325 5315391488 1274246 33554432000 4194304 4417452 140734956598160 18446744073709551615 4366099 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=29481/tid=29495] ppid=29479 vsize=5190812 CPUtime=4720.95 cores=0-7
/proc/29481/task/29495/stat : 29495 (plingeling) R 29479 29481 29455 0 -1 4202560 693098 1252 0 0 471977 118 0 0 25 0 9 0 720203325 5315391488 1274246 33554432000 4194304 4417452 140734956598160 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 37715.6
Current children cumulated vsize (KiB) 5190812

[startup+4782.3 s]
/proc/loadavg: 8.09 8.04 8.00 9/388 29635
/proc/meminfo: memFree=5791200/32951124 swapFree=52307820/67111528
[pid=29481] ppid=29479 vsize=4929192 CPUtime=38194.9 cores=0-7
/proc/29481/stat : 29481 (plingeling) S 29479 29481 29455 0 -1 4202496 6920730 1252 0 0 3818372 1117 0 0 18 0 9 0 720203282 5047492608 1208841 33554432000 4194304 4417452 140734956598160 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/29481/statm: 1232298 1208841 180 55 0 1230184 0
[pid=29481/tid=29488] ppid=29479 vsize=4929192 CPUtime=4781.93 cores=0-7
/proc/29481/task/29488/stat : 29488 (plingeling) R 29479 29481 29455 0 -1 4202560 189267 1252 0 0 478167 26 0 0 25 0 9 0 720203325 5047492608 1208841 33554432000 4194304 4417452 140734956598160 18446744073709551615 4259903 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29481/tid=29489] ppid=29479 vsize=4929192 CPUtime=4745.94 cores=0-7
/proc/29481/task/29489/stat : 29489 (plingeling) R 29479 29481 29455 0 -1 4202560 827187 1252 0 0 474477 117 0 0 25 0 9 0 720203325 5047492608 1208841 33554432000 4194304 4417452 140734956598160 18446744073709551615 4237157 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29481/tid=29490] ppid=29479 vsize=4929192 CPUtime=4781.93 cores=0-7
/proc/29481/task/29490/stat : 29490 (plingeling) R 29479 29481 29455 0 -1 4202560 481212 1252 0 0 478130 63 0 0 25 0 9 0 720203325 5047492608 1208841 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29481/tid=29491] ppid=29479 vsize=4929192 CPUtime=4781.99 cores=0-7
/proc/29481/task/29491/stat : 29491 (plingeling) R 29479 29481 29455 0 -1 4202560 2415287 1252 0 0 477751 448 0 0 25 0 9 0 720203325 5047492608 1208841 33554432000 4194304 4417452 140734956598160 18446744073709551615 4354497 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=29481/tid=29492] ppid=29479 vsize=4929192 CPUtime=4758.09 cores=0-7
/proc/29481/task/29492/stat : 29492 (plingeling) R 29479 29481 29455 0 -1 4202560 957841 1252 0 0 475677 132 0 0 25 0 9 0 720203325 5047492608 1208841 33554432000 4194304 4417452 140734956598160 18446744073709551615 4357207 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29481/tid=29493] ppid=29479 vsize=4929192 CPUtime=4781.83 cores=0-7
/proc/29481/task/29493/stat : 29493 (plingeling) R 29479 29481 29455 0 -1 4202560 683056 1252 0 0 478078 105 0 0 25 0 9 0 720203325 5047492608 1208841 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=29481/tid=29494] ppid=29479 vsize=4929192 CPUtime=4781.73 cores=0-7
/proc/29481/task/29494/stat : 29494 (plingeling) R 29479 29481 29455 0 -1 4202560 643545 1252 0 0 478075 98 0 0 25 0 9 0 720203325 5047492608 1208841 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364629 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=29481/tid=29495] ppid=29479 vsize=4929192 CPUtime=4780.94 cores=0-7
/proc/29481/task/29495/stat : 29495 (plingeling) R 29479 29481 29455 0 -1 4202560 705985 1252 0 0 477974 120 0 0 25 0 9 0 720203325 5047492608 1208841 33554432000 4194304 4417452 140734956598160 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38194.9
Current children cumulated vsize (KiB) 4929192

[startup+4842.3 s]
/proc/loadavg: 8.08 8.05 8.00 9/388 29637
/proc/meminfo: memFree=5203276/32951124 swapFree=52307820/67111528
[pid=29481] ppid=29479 vsize=5515556 CPUtime=38674 cores=0-7
/proc/29481/stat : 29481 (plingeling) S 29479 29481 29455 0 -1 4202496 7099777 1252 0 0 3866241 1160 0 0 18 0 9 0 720203282 5647929344 1355432 33554432000 4194304 4417452 140734956598160 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/29481/statm: 1378889 1355432 180 55 0 1376775 0
[pid=29481/tid=29488] ppid=29479 vsize=5515556 CPUtime=4841.93 cores=0-7
/proc/29481/task/29488/stat : 29488 (plingeling) R 29479 29481 29455 0 -1 4202560 199658 1252 0 0 484165 28 0 0 25 0 9 0 720203325 5647929344 1355432 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364881 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29481/tid=29489] ppid=29479 vsize=5515556 CPUtime=4805.21 cores=0-7
/proc/29481/task/29489/stat : 29489 (plingeling) R 29479 29481 29455 0 -1 4202560 841327 1252 0 0 480402 119 0 0 25 0 9 0 720203325 5647929344 1355432 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364809 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29481/tid=29490] ppid=29479 vsize=5515556 CPUtime=4841.94 cores=0-7
/proc/29481/task/29490/stat : 29490 (plingeling) R 29479 29481 29455 0 -1 4202560 486651 1252 0 0 484130 64 0 0 25 0 9 0 720203325 5647929344 1355432 33554432000 4194304 4417452 140734956598160 18446744073709551615 4265534 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29481/tid=29491] ppid=29479 vsize=5515556 CPUtime=4841.99 cores=0-7
/proc/29481/task/29491/stat : 29491 (plingeling) R 29479 29481 29455 0 -1 4202560 2538518 1252 0 0 483716 483 0 0 25 0 9 0 720203325 5647929344 1355432 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364716 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=29481/tid=29492] ppid=29479 vsize=5515556 CPUtime=4818.09 cores=0-7
/proc/29481/task/29492/stat : 29492 (plingeling) R 29479 29481 29455 0 -1 4202560 966034 1252 0 0 481677 132 0 0 25 0 9 0 720203325 5647929344 1355432 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364756 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29481/tid=29493] ppid=29479 vsize=5515556 CPUtime=4841.83 cores=0-7
/proc/29481/task/29493/stat : 29493 (plingeling) R 29479 29481 29455 0 -1 4202560 683056 1252 0 0 484078 105 0 0 25 0 9 0 720203325 5647929344 1355432 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=29481/tid=29494] ppid=29479 vsize=5515556 CPUtime=4841.6 cores=0-7
/proc/29481/task/29494/stat : 29494 (plingeling) R 29479 29481 29455 0 -1 4202560 654209 1252 0 0 484060 100 0 0 25 0 9 0 720203325 5647929344 1355432 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364958 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=29481/tid=29495] ppid=29479 vsize=5515556 CPUtime=4840.93 cores=0-7
/proc/29481/task/29495/stat : 29495 (plingeling) R 29479 29481 29455 0 -1 4202560 712974 1252 0 0 483970 123 0 0 25 0 9 0 720203325 5647929344 1355432 33554432000 4194304 4417452 140734956598160 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38674
Current children cumulated vsize (KiB) 5515556

[startup+4902.3 s]
/proc/loadavg: 8.03 8.04 8.00 9/388 29647
/proc/meminfo: memFree=4885028/32951124 swapFree=52307820/67111528
[pid=29481] ppid=29479 vsize=5832948 CPUtime=39153.2 cores=0-7
/proc/29481/stat : 29481 (plingeling) S 29479 29481 29455 0 -1 4202496 7201918 1252 0 0 3914141 1182 0 0 18 0 9 0 720203282 5972938752 1434780 33554432000 4194304 4417452 140734956598160 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/29481/statm: 1458237 1434780 180 55 0 1456123 0
[pid=29481/tid=29488] ppid=29479 vsize=5832948 CPUtime=4901.93 cores=0-7
/proc/29481/task/29488/stat : 29488 (plingeling) R 29479 29481 29455 0 -1 4202560 209807 1252 0 0 490164 29 0 0 25 0 9 0 720203325 5972938752 1434780 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364958 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29481/tid=29489] ppid=29479 vsize=5832948 CPUtime=4864.46 cores=0-7
/proc/29481/task/29489/stat : 29489 (plingeling) R 29479 29481 29455 0 -1 4202560 850851 1252 0 0 486325 121 0 0 25 0 9 0 720203325 5972938752 1434780 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364724 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29481/tid=29490] ppid=29479 vsize=5832948 CPUtime=4901.94 cores=0-7
/proc/29481/task/29490/stat : 29490 (plingeling) R 29479 29481 29455 0 -1 4202560 491579 1252 0 0 490129 65 0 0 25 0 9 0 720203325 5972938752 1434780 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29481/tid=29491] ppid=29479 vsize=5832948 CPUtime=4901.99 cores=0-7
/proc/29481/task/29491/stat : 29491 (plingeling) R 29479 29481 29455 0 -1 4202560 2538518 1252 0 0 489716 483 0 0 25 0 9 0 720203325 5972938752 1434780 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=29481/tid=29492] ppid=29479 vsize=5832948 CPUtime=4878.1 cores=0-7
/proc/29481/task/29492/stat : 29492 (plingeling) R 29479 29481 29455 0 -1 4202560 977106 1252 0 0 487674 136 0 0 25 0 9 0 720203325 5972938752 1434780 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29481/tid=29493] ppid=29479 vsize=5832948 CPUtime=4901.82 cores=0-7
/proc/29481/task/29493/stat : 29493 (plingeling) R 29479 29481 29455 0 -1 4202560 737864 1252 0 0 490064 118 0 0 25 0 9 0 720203325 5972938752 1434780 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=29481/tid=29494] ppid=29479 vsize=5832948 CPUtime=4901.58 cores=0-7
/proc/29481/task/29494/stat : 29494 (plingeling) R 29479 29481 29455 0 -1 4202560 654209 1252 0 0 490058 100 0 0 25 0 9 0 720203325 5972938752 1434780 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364616 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=29481/tid=29495] ppid=29479 vsize=5832948 CPUtime=4900.92 cores=0-7
/proc/29481/task/29495/stat : 29495 (plingeling) R 29479 29481 29455 0 -1 4202560 724634 1252 0 0 489968 124 0 0 25 0 9 0 720203325 5972938752 1434780 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364838 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39153.2
Current children cumulated vsize (KiB) 5832948

[startup+4962.3 s]
/proc/loadavg: 8.01 8.03 8.00 9/388 29649
/proc/meminfo: memFree=4257060/32951124 swapFree=52307820/67111528
[pid=29481] ppid=29479 vsize=6459956 CPUtime=39632.6 cores=0-7
/proc/29481/stat : 29481 (plingeling) S 29479 29481 29455 0 -1 4202496 7369654 1252 0 0 3962035 1223 0 0 18 0 9 0 720203282 6614994944 1591532 33554432000 4194304 4417452 140734956598160 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/29481/statm: 1614989 1591532 180 55 0 1612875 0
[pid=29481/tid=29488] ppid=29479 vsize=6459956 CPUtime=4961.95 cores=0-7
/proc/29481/task/29488/stat : 29488 (plingeling) R 29479 29481 29455 0 -1 4202560 215311 1252 0 0 496165 30 0 0 25 0 9 0 720203325 6614994944 1591532 33554432000 4194304 4417452 140734956598160 18446744073709551615 4357714 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29481/tid=29489] ppid=29479 vsize=6459956 CPUtime=4923.72 cores=0-7
/proc/29481/task/29489/stat : 29489 (plingeling) R 29479 29481 29455 0 -1 4202560 874743 1252 0 0 492247 125 0 0 25 0 9 0 720203325 6614994944 1591532 33554432000 4194304 4417452 140734956598160 18446744073709551615 4251071 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29481/tid=29490] ppid=29479 vsize=6459956 CPUtime=4961.95 cores=0-7
/proc/29481/task/29490/stat : 29490 (plingeling) R 29479 29481 29455 0 -1 4202560 491579 1252 0 0 496130 65 0 0 25 0 9 0 720203325 6614994944 1591532 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29481/tid=29491] ppid=29479 vsize=6459956 CPUtime=4962 cores=0-7
/proc/29481/task/29491/stat : 29491 (plingeling) R 29479 29481 29455 0 -1 4202560 2650489 1252 0 0 495684 516 0 0 25 0 9 0 720203325 6614994944 1591532 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=29481/tid=29492] ppid=29479 vsize=6459956 CPUtime=4938.11 cores=0-7
/proc/29481/task/29492/stat : 29492 (plingeling) R 29479 29481 29455 0 -1 4202560 997139 1252 0 0 493672 139 0 0 25 0 9 0 720203325 6614994944 1591532 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364604 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29481/tid=29493] ppid=29479 vsize=6459956 CPUtime=4961.83 cores=0-7
/proc/29481/task/29493/stat : 29493 (plingeling) R 29479 29481 29455 0 -1 4202560 737864 1252 0 0 496065 118 0 0 25 0 9 0 720203325 6614994944 1591532 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=29481/tid=29494] ppid=29479 vsize=6459956 CPUtime=4961.6 cores=0-7
/proc/29481/task/29494/stat : 29494 (plingeling) R 29479 29481 29455 0 -1 4202560 660545 1252 0 0 496059 101 0 0 25 0 9 0 720203325 6614994944 1591532 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=29481/tid=29495] ppid=29479 vsize=6459956 CPUtime=4960.93 cores=0-7
/proc/29481/task/29495/stat : 29495 (plingeling) R 29479 29481 29455 0 -1 4202560 724634 1252 0 0 495969 124 0 0 25 0 9 0 720203325 6614994944 1591532 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364807 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39632.6
Current children cumulated vsize (KiB) 6459956



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5008.4 s]
/proc/loadavg: 8.06 8.04 8.00 9/388 29652
/proc/meminfo: memFree=4273036/32951124 swapFree=52307820/67111528
[pid=29481] ppid=29479 vsize=6443108 CPUtime=40000.8 cores=0-7
/proc/29481/stat : 29481 (plingeling) S 29479 29481 29455 0 -1 4202496 7467267 1252 0 0 3998831 1250 0 0 18 0 9 0 720203282 6597742592 1587320 33554432000 4194304 4417452 140734956598160 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/29481/statm: 1610777 1587320 180 55 0 1608663 0
[pid=29481/tid=29488] ppid=29479 vsize=6443108 CPUtime=5008.05 cores=0-7
/proc/29481/task/29488/stat : 29488 (plingeling) R 29479 29481 29455 0 -1 4202560 224735 1252 0 0 500773 32 0 0 25 0 9 0 720203325 6597742592 1587320 33554432000 4194304 4417452 140734956598160 18446744073709551615 4311399 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29481/tid=29489] ppid=29479 vsize=6443108 CPUtime=4969.26 cores=0-7
/proc/29481/task/29489/stat : 29489 (plingeling) R 29479 29481 29455 0 -1 4202560 887903 1252 0 0 496797 129 0 0 25 0 9 0 720203325 6597742592 1587320 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29481/tid=29490] ppid=29479 vsize=6443108 CPUtime=5008.05 cores=0-7
/proc/29481/task/29490/stat : 29490 (plingeling) R 29479 29481 29455 0 -1 4202560 491579 1252 0 0 500740 65 0 0 25 0 9 0 720203325 6597742592 1587320 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364537 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29481/tid=29491] ppid=29479 vsize=6443108 CPUtime=5008.11 cores=0-7
/proc/29481/task/29491/stat : 29491 (plingeling) R 29479 29481 29455 0 -1 4202560 2712667 1252 0 0 500277 534 0 0 25 0 9 0 720203325 6597742592 1587320 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=29481/tid=29492] ppid=29479 vsize=6443108 CPUtime=4984.21 cores=0-7
/proc/29481/task/29492/stat : 29492 (plingeling) R 29479 29481 29455 0 -1 4202560 999763 1252 0 0 498282 139 0 0 25 0 9 0 720203325 6597742592 1587320 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29481/tid=29493] ppid=29479 vsize=6443108 CPUtime=5007.93 cores=0-7
/proc/29481/task/29493/stat : 29493 (plingeling) R 29479 29481 29455 0 -1 4202560 739379 1252 0 0 500674 119 0 0 25 0 9 0 720203325 6597742592 1587320 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=29481/tid=29494] ppid=29479 vsize=6443108 CPUtime=5007.7 cores=0-7
/proc/29481/task/29494/stat : 29494 (plingeling) R 29479 29481 29455 0 -1 4202560 669257 1252 0 0 500668 102 0 0 25 0 9 0 720203325 6597742592 1587320 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=29481/tid=29495] ppid=29479 vsize=6443108 CPUtime=5007.02 cores=0-7
/proc/29481/task/29495/stat : 29495 (plingeling) R 29479 29481 29455 0 -1 4202560 724634 1252 0 0 500578 124 0 0 25 0 9 0 720203325 6597742592 1587320 33554432000 4194304 4417452 140734956598160 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 40000.8
Current children cumulated vsize (KiB) 6443108

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5008.52 s]
/proc/loadavg: 8.06 8.04 8.00 2/382 29653
/proc/meminfo: memFree=7050036/32951124 swapFree=52307820/67111528
[pid=29481] ppid=29479 vsize=0 CPUtime=40001 cores=0-7
/proc/29481/stat : 29481 (plingeling) Z 29479 29481 29455 0 -1 4203548 7467267 1252 0 0 3998836 1261 0 0 18 0 2 0 720203282 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29481/statm: 0 0 0 0 0 0 0
[pid=29481/tid=29489] ppid=29479 vsize=0 CPUtime=4969.37 cores=0-7
/proc/29481/task/29489/stat : 29489 (plingeling) R 29479 29481 29455 0 -1 4203588 887903 1252 0 0 496797 140 0 0 25 0 2 0 720203325 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 40001
Current children cumulated vsize (KiB) 0

[startup+5008.61 s]
/proc/loadavg: 8.06 8.04 8.00 2/382 29653
/proc/meminfo: memFree=7050036/32951124 swapFree=52307820/67111528
[pid=29481] ppid=29479 vsize=0 CPUtime=40001.1 cores=0-7
/proc/29481/stat : 29481 (plingeling) Z 29479 29481 29455 0 -1 4203548 7467267 1252 0 0 3998836 1270 0 0 18 0 2 0 720203282 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29481/statm: 0 0 0 0 0 0 0
[pid=29481/tid=29489] ppid=29479 vsize=0 CPUtime=4969.45 cores=0-7
/proc/29481/task/29489/stat : 29489 (plingeling) R 29479 29481 29455 0 -1 4203588 887903 1252 0 0 496797 148 0 0 25 0 2 0 720203325 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 40001.1
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5008.65
CPU time (s): 40001.1
CPU user time (s): 39988.4
CPU system time (s): 12.7371
CPU usage (%): 798.64
Max. virtual memory (cumulated for all children) (KiB): 6688048

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39988.4
system time used= 12.7371
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 7468519
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 662
involuntary context switches= 193830

runsolver used 17.7923 second user time and 43.9573 second system time

The end

Launcher Data 

Begin job on node101 at 2011-05-22 19:40:05
IDJOB=3394082
IDBENCH=71408
IDSOLVER=1851
FILE ID=node101/3394082-1306086005
RUNJOBID= node101-1306086005-29469
PBS_JOBID= 13387921
Free space on /tmp= 73576 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT09/CRAFTED/rbsat/random/forced/rbsat-v2640c305320gyes5.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3394082-1306086005/watcher-3394082-1306086005 -o /tmp/evaluation-result-3394082-1306086005/solver-3394082-1306086005 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3394082-1306086005.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= f892a2bda6fe6aec6f2fcb4e1c541df8
RANDOM SEED=481322797

node101.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      10632632 kB
Buffers:        369880 kB
Cached:       10445056 kB
SwapCached:      14796 kB
Active:       14808448 kB
Inactive:      7193572 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      10632632 kB
SwapTotal:    67111528 kB
SwapFree:     52307820 kB
Dirty:            4272 kB
Writeback:           0 kB
AnonPages:    11184176 kB
Mapped:          20160 kB
Slab:           172744 kB
PageTables:      74576 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 78578100 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73576 MiB
End job on node101 at 2011-05-22 21:03:36