Trace number 3372842

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 40000.5 5007.74

General information on the benchmark

NameSAT03/industrial/kukula/addm_bench/
am_7_7.shuffled-as.sat03-363.cnf
MD5SUMc7f51e1ee7556440ab29e1c4bfe29538
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables4264
Number of clauses14751
Sum of the clauses size40035
Maximum clause length3
Minimum clause length1
Number of clauses of size 12
Number of clauses of size 24214
Number of clauses of size 310535
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3372842-1305442901/watcher-3372842-1305442901 -o /tmp/evaluation-result-3372842-1305442901/solver-3372842-1305442901 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3372842-1305442901.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.00 1.47 4.62 2/347 22630
/proc/meminfo: memFree=21259136/32951124 swapFree=65581052/67111528
[pid=22630] ppid=22628 vsize=11296 CPUtime=0 cores=0-7
/proc/22630/stat : 22630 (plingeling) R 22628 22630 22604 0 -1 4202496 880 1253 0 0 0 0 0 0 25 0 1 0 655887783 11567104 809 33554432000 4194304 4417452 140737063085008 18446744073709551615 218905428176 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/22630/statm: 2824 823 148 55 0 710 0

[startup+0.0386599 s]
/proc/loadavg: 0.00 1.47 4.62 2/347 22630
/proc/meminfo: memFree=21259136/32951124 swapFree=65581052/67111528
[pid=22630] ppid=22628 vsize=17964 CPUtime=0.03 cores=0-7
/proc/22630/stat : 22630 (plingeling) R 22628 22630 22604 0 -1 4202496 2617 1253 0 0 3 0 0 0 25 0 1 0 655887783 18395136 2514 33554432000 4194304 4417452 140737063085008 18446744073709551615 4231321 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/22630/statm: 4491 2514 149 55 0 2377 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 17964

[startup+0.172265 s]
/proc/loadavg: 0.00 1.47 4.62 2/347 22630
/proc/meminfo: memFree=21259136/32951124 swapFree=65581052/67111528
[pid=22630] ppid=22628 vsize=168344 CPUtime=0.95 cores=0-7
/proc/22630/stat : 22630 (plingeling) S 22628 22630 22604 0 -1 4202496 4300 1253 0 0 95 0 0 0 25 0 9 0 655887783 172384256 3316 33554432000 4194304 4417452 140737063085008 18446744073709551615 218917533669 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/22630/statm: 42086 3316 179 55 0 39972 0
Current children cumulated CPU time (s) 0.95
Current children cumulated vsize (KiB) 168344

[startup+0.300544 s]
/proc/loadavg: 0.00 1.47 4.62 2/347 22630
/proc/meminfo: memFree=21259136/32951124 swapFree=65581052/67111528
[pid=22630] ppid=22628 vsize=169088 CPUtime=1.97 cores=0-7
/proc/22630/stat : 22630 (plingeling) S 22628 22630 22604 0 -1 4202496 4668 1253 0 0 197 0 0 0 25 0 9 0 655887783 173146112 3559 33554432000 4194304 4417452 140737063085008 18446744073709551615 218917533669 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/22630/statm: 42272 3559 179 55 0 40158 0
Current children cumulated CPU time (s) 1.97
Current children cumulated vsize (KiB) 169088

[startup+0.700472 s]
/proc/loadavg: 0.00 1.47 4.62 2/347 22630
/proc/meminfo: memFree=21259136/32951124 swapFree=65581052/67111528
[pid=22630] ppid=22628 vsize=170976 CPUtime=5.18 cores=0-7
/proc/22630/stat : 22630 (plingeling) S 22628 22630 22604 0 -1 4202496 8826 1253 0 0 516 2 0 0 25 0 9 0 655887783 175079424 3999 33554432000 4194304 4417452 140737063085008 18446744073709551615 218917533669 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/22630/statm: 42744 3999 179 55 0 40630 0
Current children cumulated CPU time (s) 5.18
Current children cumulated vsize (KiB) 170976

[startup+1.50033 s]
/proc/loadavg: 0.00 1.47 4.62 9/356 22644
/proc/meminfo: memFree=21235760/32951124 swapFree=65581052/67111528
[pid=22630] ppid=22628 vsize=178152 CPUtime=11.57 cores=0-7
/proc/22630/stat : 22630 (plingeling) S 22628 22630 22604 0 -1 4202496 17035 1253 0 0 1153 4 0 0 25 0 9 0 655887783 182427648 5794 33554432000 4194304 4417452 140737063085008 18446744073709551615 218917533669 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/22630/statm: 44538 5794 180 55 0 42424 0
[pid=22630/tid=22637] ppid=22628 vsize=178152 CPUtime=1.44 cores=0-7
/proc/22630/task/22637/stat : 22637 (plingeling) R 22628 22630 22604 0 -1 4202560 1786 1253 0 0 144 0 0 0 20 0 9 0 655887788 182427648 5794 33554432000 4194304 4417452 140737063085008 18446744073709551615 4355841 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22630/tid=22638] ppid=22628 vsize=178152 CPUtime=1.44 cores=0-7
/proc/22630/task/22638/stat : 22638 (plingeling) R 22628 22630 22604 0 -1 4202560 1655 1253 0 0 144 0 0 0 25 0 9 0 655887788 182427648 5794 33554432000 4194304 4417452 140737063085008 18446744073709551615 4257101 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22630/tid=22639] ppid=22628 vsize=178152 CPUtime=1.41 cores=0-7
/proc/22630/task/22639/stat : 22639 (plingeling) R 22628 22630 22604 0 -1 4202560 1727 1253 0 0 141 0 0 0 25 0 9 0 655887788 182427648 5794 33554432000 4194304 4417452 140737063085008 18446744073709551615 4309986 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22630/tid=22640] ppid=22628 vsize=178152 CPUtime=1.44 cores=0-7
/proc/22630/task/22640/stat : 22640 (plingeling) R 22628 22630 22604 0 -1 4202560 1801 1253 0 0 144 0 0 0 25 0 9 0 655887788 182427648 5794 33554432000 4194304 4417452 140737063085008 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22630/tid=22641] ppid=22628 vsize=178152 CPUtime=1.42 cores=0-7
/proc/22630/task/22641/stat : 22641 (plingeling) R 22628 22630 22604 0 -1 4202560 1418 1253 0 0 142 0 0 0 25 0 9 0 655887788 182427648 5794 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364703 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22630/tid=22642] ppid=22628 vsize=178152 CPUtime=1.44 cores=0-7
/proc/22630/task/22642/stat : 22642 (plingeling) R 22628 22630 22604 0 -1 4202560 1483 1253 0 0 144 0 0 0 25 0 9 0 655887788 182427648 5794 33554432000 4194304 4417452 140737063085008 18446744073709551615 4319276 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=22630/tid=22643] ppid=22628 vsize=178152 CPUtime=1.43 cores=0-7
/proc/22630/task/22643/stat : 22643 (plingeling) R 22628 22630 22604 0 -1 4202560 1955 1253 0 0 143 0 0 0 25 0 9 0 655887788 182427648 5794 33554432000 4194304 4417452 140737063085008 18446744073709551615 4236649 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22630/tid=22644] ppid=22628 vsize=178152 CPUtime=1.44 cores=0-7
/proc/22630/task/22644/stat : 22644 (plingeling) R 22628 22630 22604 0 -1 4202560 1552 1253 0 0 144 0 0 0 19 0 9 0 655887789 182427648 5794 33554432000 4194304 4417452 140737063085008 18446744073709551615 4309365 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 11.57
Current children cumulated vsize (KiB) 178152

[startup+3.10104 s]
/proc/loadavg: 0.00 1.47 4.62 9/356 22644
/proc/meminfo: memFree=21226460/32951124 swapFree=65581052/67111528
[pid=22630] ppid=22628 vsize=194168 CPUtime=24.36 cores=0-7
/proc/22630/stat : 22630 (plingeling) S 22628 22630 22604 0 -1 4202496 33857 1253 0 0 2428 8 0 0 25 0 9 0 655887783 198828032 9798 33554432000 4194304 4417452 140737063085008 18446744073709551615 218917533669 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/22630/statm: 48542 9798 180 55 0 46428 0
[pid=22630/tid=22637] ppid=22628 vsize=194168 CPUtime=3.04 cores=0-7
/proc/22630/task/22637/stat : 22637 (plingeling) R 22628 22630 22604 0 -1 4202560 3720 1253 0 0 304 0 0 0 25 0 9 0 655887788 198828032 9798 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364657 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22630/tid=22638] ppid=22628 vsize=194168 CPUtime=3.03 cores=0-7
/proc/22630/task/22638/stat : 22638 (plingeling) R 22628 22630 22604 0 -1 4202560 3721 1253 0 0 303 0 0 0 25 0 9 0 655887788 198828032 9798 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22630/tid=22639] ppid=22628 vsize=194168 CPUtime=3 cores=0-7
/proc/22630/task/22639/stat : 22639 (plingeling) R 22628 22630 22604 0 -1 4202560 3640 1253 0 0 299 1 0 0 25 0 9 0 655887788 198828032 9798 33554432000 4194304 4417452 140737063085008 18446744073709551615 4277545 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22630/tid=22640] ppid=22628 vsize=194168 CPUtime=3.04 cores=0-7
/proc/22630/task/22640/stat : 22640 (plingeling) R 22628 22630 22604 0 -1 4202560 3603 1253 0 0 304 0 0 0 25 0 9 0 655887788 198828032 9798 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364812 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22630/tid=22641] ppid=22628 vsize=194168 CPUtime=3.03 cores=0-7
/proc/22630/task/22641/stat : 22641 (plingeling) R 22628 22630 22604 0 -1 4202560 4201 1253 0 0 302 1 0 0 25 0 9 0 655887788 198828032 9798 33554432000 4194304 4417452 140737063085008 18446744073709551615 4342425 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22630/tid=22642] ppid=22628 vsize=194168 CPUtime=3.03 cores=0-7
/proc/22630/task/22642/stat : 22642 (plingeling) R 22628 22630 22604 0 -1 4202560 3368 1253 0 0 303 0 0 0 25 0 9 0 655887788 198828032 9798 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364812 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=22630/tid=22643] ppid=22628 vsize=194168 CPUtime=3.03 cores=0-7
/proc/22630/task/22643/stat : 22643 (plingeling) R 22628 22630 22604 0 -1 4202560 3995 1253 0 0 302 1 0 0 25 0 9 0 655887788 198828032 9798 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22630/tid=22644] ppid=22628 vsize=194168 CPUtime=3.03 cores=0-7
/proc/22630/task/22644/stat : 22644 (plingeling) R 22628 22630 22604 0 -1 4202560 3951 1253 0 0 303 0 0 0 25 0 9 0 655887789 198828032 9798 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364650 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 24.36
Current children cumulated vsize (KiB) 194168

[startup+6.30046 s]
/proc/loadavg: 0.64 1.58 4.64 9/356 22644
/proc/meminfo: memFree=21201924/32951124 swapFree=65581052/67111528
[pid=22630] ppid=22628 vsize=208868 CPUtime=49.92 cores=0-7
/proc/22630/stat : 22630 (plingeling) S 22628 22630 22604 0 -1 4202496 68905 1253 0 0 4976 16 0 0 25 0 9 0 655887783 213880832 13473 33554432000 4194304 4417452 140737063085008 18446744073709551615 218917533669 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/22630/statm: 52217 13473 180 55 0 50103 0
[pid=22630/tid=22637] ppid=22628 vsize=208868 CPUtime=6.24 cores=0-7
/proc/22630/task/22637/stat : 22637 (plingeling) R 22628 22630 22604 0 -1 4202560 7636 1253 0 0 623 1 0 0 25 0 9 0 655887788 213880832 13473 33554432000 4194304 4417452 140737063085008 18446744073709551615 4281450 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22630/tid=22638] ppid=22628 vsize=208868 CPUtime=6.23 cores=0-7
/proc/22630/task/22638/stat : 22638 (plingeling) R 22628 22630 22604 0 -1 4202560 8046 1253 0 0 622 1 0 0 25 0 9 0 655887788 213880832 13473 33554432000 4194304 4417452 140737063085008 18446744073709551615 4250936 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22630/tid=22639] ppid=22628 vsize=208868 CPUtime=6.16 cores=0-7
/proc/22630/task/22639/stat : 22639 (plingeling) R 22628 22630 22604 0 -1 4202560 8067 1253 0 0 614 2 0 0 25 0 9 0 655887788 213880832 13473 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364819 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22630/tid=22640] ppid=22628 vsize=208868 CPUtime=6.25 cores=0-7
/proc/22630/task/22640/stat : 22640 (plingeling) R 22628 22630 22604 0 -1 4202560 8383 1253 0 0 623 2 0 0 25 0 9 0 655887788 213880832 13473 33554432000 4194304 4417452 140737063085008 18446744073709551615 4250938 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22630/tid=22641] ppid=22628 vsize=208868 CPUtime=6.23 cores=0-7
/proc/22630/task/22641/stat : 22641 (plingeling) R 22628 22630 22604 0 -1 4202560 8396 1253 0 0 621 2 0 0 25 0 9 0 655887788 213880832 13473 33554432000 4194304 4417452 140737063085008 18446744073709551615 4321595 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22630/tid=22642] ppid=22628 vsize=208868 CPUtime=6.23 cores=0-7
/proc/22630/task/22642/stat : 22642 (plingeling) R 22628 22630 22604 0 -1 4202560 7912 1253 0 0 622 1 0 0 25 0 9 0 655887788 213880832 13473 33554432000 4194304 4417452 140737063085008 18446744073709551615 4357554 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=22630/tid=22643] ppid=22628 vsize=208868 CPUtime=6.23 cores=0-7
/proc/22630/task/22643/stat : 22643 (plingeling) R 22628 22630 22604 0 -1 4202560 8712 1253 0 0 621 2 0 0 25 0 9 0 655887788 213880832 13473 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364650 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22630/tid=22644] ppid=22628 vsize=208868 CPUtime=6.24 cores=0-7
/proc/22630/task/22644/stat : 22644 (plingeling) R 22628 22630 22604 0 -1 4202560 8095 1253 0 0 623 1 0 0 25 0 9 0 655887789 213880832 13473 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364564 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 49.92
Current children cumulated vsize (KiB) 208868

[startup+12.7013 s]
/proc/loadavg: 1.23 1.68 4.66 9/356 22644
/proc/meminfo: memFree=21188948/32951124 swapFree=65581052/67111528
[pid=22630] ppid=22628 vsize=226296 CPUtime=101.05 cores=0-7
/proc/22630/stat : 22630 (plingeling) S 22628 22630 22604 0 -1 4202496 124638 1253 0 0 10079 26 0 0 25 0 9 0 655887783 231727104 17836 33554432000 4194304 4417452 140737063085008 18446744073709551615 218917533669 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/22630/statm: 56574 17836 180 55 0 54460 0
[pid=22630/tid=22637] ppid=22628 vsize=226296 CPUtime=12.64 cores=0-7
/proc/22630/task/22637/stat : 22637 (plingeling) R 22628 22630 22604 0 -1 4202560 15638 1253 0 0 1261 3 0 0 25 0 9 0 655887788 231727104 17836 33554432000 4194304 4417452 140737063085008 18446744073709551615 4232787 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22630/tid=22638] ppid=22628 vsize=226296 CPUtime=12.64 cores=0-7
/proc/22630/task/22638/stat : 22638 (plingeling) R 22628 22630 22604 0 -1 4202560 14085 1253 0 0 1261 3 0 0 25 0 9 0 655887788 231727104 17836 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22630/tid=22639] ppid=22628 vsize=226296 CPUtime=12.49 cores=0-7
/proc/22630/task/22639/stat : 22639 (plingeling) R 22628 22630 22604 0 -1 4202560 14280 1253 0 0 1246 3 0 0 25 0 9 0 655887788 231727104 17836 33554432000 4194304 4417452 140737063085008 18446744073709551615 18446744073699065939 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22630/tid=22640] ppid=22628 vsize=226296 CPUtime=12.64 cores=0-7
/proc/22630/task/22640/stat : 22640 (plingeling) R 22628 22630 22604 0 -1 4202560 16160 1253 0 0 1261 3 0 0 25 0 9 0 655887788 231727104 17836 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364675 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22630/tid=22641] ppid=22628 vsize=226296 CPUtime=12.63 cores=0-7
/proc/22630/task/22641/stat : 22641 (plingeling) R 22628 22630 22604 0 -1 4202560 14233 1253 0 0 1260 3 0 0 25 0 9 0 655887788 231727104 17836 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22630/tid=22642] ppid=22628 vsize=226296 CPUtime=12.64 cores=0-7
/proc/22630/task/22642/stat : 22642 (plingeling) R 22628 22630 22604 0 -1 4202560 13856 1253 0 0 1262 2 0 0 25 0 9 0 655887788 231727104 17836 33554432000 4194304 4417452 140737063085008 18446744073709551615 4250721 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=22630/tid=22643] ppid=22628 vsize=226296 CPUtime=12.63 cores=0-7
/proc/22630/task/22643/stat : 22643 (plingeling) R 22628 22630 22604 0 -1 4202560 16633 1253 0 0 1260 3 0 0 25 0 9 0 655887788 231727104 17836 33554432000 4194304 4417452 140737063085008 18446744073709551615 4358022 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22630/tid=22644] ppid=22628 vsize=226296 CPUtime=12.63 cores=0-7
/proc/22630/task/22644/stat : 22644 (plingeling) R 22628 22630 22604 0 -1 4202560 16095 1253 0 0 1261 2 0 0 25 0 9 0 655887789 231727104 17836 33554432000 4194304 4417452 140737063085008 18446744073709551615 18446744073699065894 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 101.05
Current children cumulated vsize (KiB) 226296

[startup+25.501 s]
/proc/loadavg: 2.73 1.99 4.71 9/356 22644
/proc/meminfo: memFree=21161304/32951124 swapFree=65581052/67111528
[pid=22630] ppid=22628 vsize=254636 CPUtime=203.29 cores=0-7
/proc/22630/stat : 22630 (plingeling) S 22628 22630 22604 0 -1 4202496 223383 1253 0 0 20283 46 0 0 25 0 9 0 655887783 260747264 25183 33554432000 4194304 4417452 140737063085008 18446744073709551615 218917533669 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/22630/statm: 63659 25183 180 55 0 61545 0
[pid=22630/tid=22637] ppid=22628 vsize=254636 CPUtime=25.44 cores=0-7
/proc/22630/task/22637/stat : 22637 (plingeling) R 22628 22630 22604 0 -1 4202560 28668 1253 0 0 2538 6 0 0 25 0 9 0 655887788 260747264 25183 33554432000 4194304 4417452 140737063085008 18446744073709551615 4353761 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22630/tid=22638] ppid=22628 vsize=254636 CPUtime=25.44 cores=0-7
/proc/22630/task/22638/stat : 22638 (plingeling) R 22628 22630 22604 0 -1 4202560 25892 1253 0 0 2539 5 0 0 25 0 9 0 655887788 260747264 25183 33554432000 4194304 4417452 140737063085008 18446744073709551615 4365023 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22630/tid=22639] ppid=22628 vsize=254636 CPUtime=25.14 cores=0-7
/proc/22630/task/22639/stat : 22639 (plingeling) R 22628 22630 22604 0 -1 4202560 24438 1253 0 0 2509 5 0 0 25 0 9 0 655887788 260747264 25183 33554432000 4194304 4417452 140737063085008 18446744073709551615 4366385 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22630/tid=22640] ppid=22628 vsize=254636 CPUtime=25.45 cores=0-7
/proc/22630/task/22640/stat : 22640 (plingeling) R 22628 22630 22604 0 -1 4202560 26623 1253 0 0 2540 5 0 0 25 0 9 0 655887788 260747264 25183 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22630/tid=22641] ppid=22628 vsize=254636 CPUtime=25.42 cores=0-7
/proc/22630/task/22641/stat : 22641 (plingeling) R 22628 22630 22604 0 -1 4202560 29173 1253 0 0 2537 5 0 0 25 0 9 0 655887788 260747264 25183 33554432000 4194304 4417452 140737063085008 18446744073709551615 4353727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22630/tid=22642] ppid=22628 vsize=254636 CPUtime=25.43 cores=0-7
/proc/22630/task/22642/stat : 22642 (plingeling) R 22628 22630 22604 0 -1 4202560 27794 1253 0 0 2538 5 0 0 25 0 9 0 655887788 260747264 25183 33554432000 4194304 4417452 140737063085008 18446744073709551615 4354145 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=22630/tid=22643] ppid=22628 vsize=254636 CPUtime=25.43 cores=0-7
/proc/22630/task/22643/stat : 22643 (plingeling) R 22628 22630 22604 0 -1 4202560 28770 1253 0 0 2537 6 0 0 25 0 9 0 655887788 260747264 25183 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364684 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22630/tid=22644] ppid=22628 vsize=254636 CPUtime=25.44 cores=0-7
/proc/22630/task/22644/stat : 22644 (plingeling) R 22628 22630 22604 0 -1 4202560 28367 1253 0 0 2539 5 0 0 25 0 9 0 655887789 260747264 25183 33554432000 4194304 4417452 140737063085008 18446744073709551615 4250697 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 203.29
Current children cumulated vsize (KiB) 254636

[startup+51.1003 s]
/proc/loadavg: 4.58 2.49 4.80 9/356 22645
/proc/meminfo: memFree=21114404/32951124 swapFree=65581052/67111528
[pid=22630] ppid=22628 vsize=288544 CPUtime=407.82 cores=0-7
/proc/22630/stat : 22630 (plingeling) S 22628 22630 22604 0 -1 4202496 384896 1253 0 0 40703 79 0 0 25 0 9 0 655887783 295469056 33441 33554432000 4194304 4417452 140737063085008 18446744073709551615 218917533669 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/22630/statm: 72136 33441 180 55 0 70022 0
[pid=22630/tid=22637] ppid=22628 vsize=288544 CPUtime=51.04 cores=0-7
/proc/22630/task/22637/stat : 22637 (plingeling) R 22628 22630 22604 0 -1 4202560 50293 1253 0 0 5094 10 0 0 25 0 9 0 655887788 295469056 33441 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22630/tid=22638] ppid=22628 vsize=288544 CPUtime=51.04 cores=0-7
/proc/22630/task/22638/stat : 22638 (plingeling) R 22628 22630 22604 0 -1 4202560 48415 1253 0 0 5094 10 0 0 25 0 9 0 655887788 295469056 33441 33554432000 4194304 4417452 140737063085008 18446744073709551615 4237310 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22630/tid=22639] ppid=22628 vsize=288544 CPUtime=50.47 cores=0-7

################
# More data... #
################


[startup+4722.31 s]
/proc/loadavg: 8.01 8.03 8.00 9/356 22785
/proc/meminfo: memFree=18243224/32951124 swapFree=65581052/67111528
[pid=22630] ppid=22628 vsize=3164932 CPUtime=37721.2 cores=0-7
/proc/22630/stat : 22630 (plingeling) S 22628 22630 22604 0 -1 4202496 6536175 1253 0 0 3770884 1240 0 0 25 0 9 0 655887783 3240890368 752532 33554432000 4194304 4417452 140737063085008 18446744073709551615 218917533669 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/22630/statm: 791233 752532 180 55 0 789119 0
[pid=22630/tid=22637] ppid=22628 vsize=3164932 CPUtime=4722.38 cores=0-7
/proc/22630/task/22637/stat : 22637 (plingeling) R 22628 22630 22604 0 -1 4202560 785636 1253 0 0 472082 156 0 0 25 0 9 0 655887788 3240890368 752532 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22630/tid=22638] ppid=22628 vsize=3164932 CPUtime=4722.09 cores=0-7
/proc/22630/task/22638/stat : 22638 (plingeling) R 22628 22630 22604 0 -1 4202560 1067115 1253 0 0 472032 177 0 0 25 0 9 0 655887788 3240890368 752532 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22630/tid=22639] ppid=22628 vsize=3164932 CPUtime=4667.1 cores=0-7
/proc/22630/task/22639/stat : 22639 (plingeling) R 22628 22630 22604 0 -1 4202560 1069031 1253 0 0 466515 195 0 0 25 0 9 0 655887788 3240890368 752532 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22630/tid=22640] ppid=22628 vsize=3164932 CPUtime=4722.26 cores=0-7
/proc/22630/task/22640/stat : 22640 (plingeling) R 22628 22630 22604 0 -1 4202560 440273 1253 0 0 472136 90 0 0 25 0 9 0 655887788 3240890368 752532 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22630/tid=22641] ppid=22628 vsize=3164932 CPUtime=4722.38 cores=0-7
/proc/22630/task/22641/stat : 22641 (plingeling) R 22628 22630 22604 0 -1 4202560 730367 1253 0 0 472077 161 0 0 25 0 9 0 655887788 3240890368 752532 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364616 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22630/tid=22642] ppid=22628 vsize=3164932 CPUtime=4720.58 cores=0-7
/proc/22630/task/22642/stat : 22642 (plingeling) R 22628 22630 22604 0 -1 4202560 796130 1253 0 0 471900 158 0 0 25 0 9 0 655887788 3240890368 752532 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=22630/tid=22643] ppid=22628 vsize=3164932 CPUtime=4721.98 cores=0-7
/proc/22630/task/22643/stat : 22643 (plingeling) R 22628 22630 22604 0 -1 4202560 1075748 1253 0 0 472013 185 0 0 25 0 9 0 655887788 3240890368 752532 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364629 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22630/tid=22644] ppid=22628 vsize=3164932 CPUtime=4722.38 cores=0-7
/proc/22630/task/22644/stat : 22644 (plingeling) R 22628 22630 22604 0 -1 4202560 568217 1253 0 0 472123 115 0 0 25 0 9 0 655887789 3240890368 752532 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 37721.2
Current children cumulated vsize (KiB) 3164932

[startup+4782.3 s]
/proc/loadavg: 8.04 8.03 8.00 9/356 22786
/proc/meminfo: memFree=18526500/32951124 swapFree=65581052/67111528
[pid=22630] ppid=22628 vsize=2882608 CPUtime=38200.4 cores=0-7
/proc/22630/stat : 22630 (plingeling) S 22628 22630 22604 0 -1 4202496 6594115 1253 0 0 3818795 1249 0 0 25 0 9 0 655887783 2951790592 681983 33554432000 4194304 4417452 140737063085008 18446744073709551615 218917533669 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/22630/statm: 720652 681983 180 55 0 718538 0
[pid=22630/tid=22637] ppid=22628 vsize=2882608 CPUtime=4782.36 cores=0-7
/proc/22630/task/22637/stat : 22637 (plingeling) R 22628 22630 22604 0 -1 4202560 791888 1253 0 0 478080 156 0 0 25 0 9 0 655887788 2951790592 681983 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22630/tid=22638] ppid=22628 vsize=2882608 CPUtime=4782.07 cores=0-7
/proc/22630/task/22638/stat : 22638 (plingeling) R 22628 22630 22604 0 -1 4202560 1067115 1253 0 0 478030 177 0 0 25 0 9 0 655887788 2951790592 681983 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22630/tid=22639] ppid=22628 vsize=2882608 CPUtime=4726.39 cores=0-7
/proc/22630/task/22639/stat : 22639 (plingeling) R 22628 22630 22604 0 -1 4202560 1081428 1253 0 0 472442 197 0 0 25 0 9 0 655887788 2951790592 681983 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22630/tid=22640] ppid=22628 vsize=2882608 CPUtime=4782.25 cores=0-7
/proc/22630/task/22640/stat : 22640 (plingeling) R 22628 22630 22604 0 -1 4202560 440273 1253 0 0 478135 90 0 0 25 0 9 0 655887788 2951790592 681983 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364537 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22630/tid=22641] ppid=22628 vsize=2882608 CPUtime=4782.36 cores=0-7
/proc/22630/task/22641/stat : 22641 (plingeling) R 22628 22630 22604 0 -1 4202560 730367 1253 0 0 478075 161 0 0 25 0 9 0 655887788 2951790592 681983 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22630/tid=22642] ppid=22628 vsize=2882608 CPUtime=4780.56 cores=0-7
/proc/22630/task/22642/stat : 22642 (plingeling) R 22628 22630 22604 0 -1 4202560 835421 1253 0 0 477892 164 0 0 25 0 9 0 655887788 2951790592 681983 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=22630/tid=22643] ppid=22628 vsize=2882608 CPUtime=4781.97 cores=0-7
/proc/22630/task/22643/stat : 22643 (plingeling) R 22628 22630 22604 0 -1 4202560 1075748 1253 0 0 478012 185 0 0 25 0 9 0 655887788 2951790592 681983 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364621 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22630/tid=22644] ppid=22628 vsize=2882608 CPUtime=4782.37 cores=0-7
/proc/22630/task/22644/stat : 22644 (plingeling) R 22628 22630 22604 0 -1 4202560 568217 1253 0 0 478122 115 0 0 25 0 9 0 655887789 2951790592 681983 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 38200.4
Current children cumulated vsize (KiB) 2882608

[startup+4842.3 s]
/proc/loadavg: 8.12 8.05 8.01 9/356 22788
/proc/meminfo: memFree=18443768/32951124 swapFree=65581052/67111528
[pid=22630] ppid=22628 vsize=2965032 CPUtime=38679.7 cores=0-7
/proc/22630/stat : 22630 (plingeling) S 22628 22630 22604 0 -1 4202496 6614721 1253 0 0 3866720 1252 0 0 25 0 9 0 655887783 3036192768 702589 33554432000 4194304 4417452 140737063085008 18446744073709551615 218917533669 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/22630/statm: 741258 702589 180 55 0 739144 0
[pid=22630/tid=22637] ppid=22628 vsize=2965032 CPUtime=4842.36 cores=0-7
/proc/22630/task/22637/stat : 22637 (plingeling) R 22628 22630 22604 0 -1 4202560 791888 1253 0 0 484080 156 0 0 25 0 9 0 655887788 3036192768 702589 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22630/tid=22638] ppid=22628 vsize=2965032 CPUtime=4842.08 cores=0-7
/proc/22630/task/22638/stat : 22638 (plingeling) R 22628 22630 22604 0 -1 4202560 1067115 1253 0 0 484031 177 0 0 25 0 9 0 655887788 3036192768 702589 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22630/tid=22639] ppid=22628 vsize=2965032 CPUtime=4785.68 cores=0-7
/proc/22630/task/22639/stat : 22639 (plingeling) R 22628 22630 22604 0 -1 4202560 1093774 1253 0 0 478369 199 0 0 25 0 9 0 655887788 3036192768 702589 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22630/tid=22640] ppid=22628 vsize=2965032 CPUtime=4842.24 cores=0-7
/proc/22630/task/22640/stat : 22640 (plingeling) R 22628 22630 22604 0 -1 4202560 440273 1253 0 0 484134 90 0 0 25 0 9 0 655887788 3036192768 702589 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22630/tid=22641] ppid=22628 vsize=2965032 CPUtime=4842.37 cores=0-7
/proc/22630/task/22641/stat : 22641 (plingeling) R 22628 22630 22604 0 -1 4202560 730367 1253 0 0 484076 161 0 0 25 0 9 0 655887788 3036192768 702589 33554432000 4194304 4417452 140737063085008 18446744073709551615 4307968 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22630/tid=22642] ppid=22628 vsize=2965032 CPUtime=4840.55 cores=0-7
/proc/22630/task/22642/stat : 22642 (plingeling) R 22628 22630 22604 0 -1 4202560 843681 1253 0 0 483890 165 0 0 25 0 9 0 655887788 3036192768 702589 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364756 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=22630/tid=22643] ppid=22628 vsize=2965032 CPUtime=4841.97 cores=0-7
/proc/22630/task/22643/stat : 22643 (plingeling) R 22628 22630 22604 0 -1 4202560 1075748 1253 0 0 484012 185 0 0 25 0 9 0 655887788 3036192768 702589 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22630/tid=22644] ppid=22628 vsize=2965032 CPUtime=4842.37 cores=0-7
/proc/22630/task/22644/stat : 22644 (plingeling) R 22628 22630 22604 0 -1 4202560 568217 1253 0 0 484122 115 0 0 25 0 9 0 655887789 3036192768 702589 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 38679.7
Current children cumulated vsize (KiB) 2965032

[startup+4902.3 s]
/proc/loadavg: 8.04 8.04 8.00 9/356 22789
/proc/meminfo: memFree=18411756/32951124 swapFree=65581052/67111528
[pid=22630] ppid=22628 vsize=2998520 CPUtime=39159 cores=0-7
/proc/22630/stat : 22630 (plingeling) S 22628 22630 22604 0 -1 4202496 6754618 1253 0 0 3914617 1286 0 0 25 0 9 0 655887783 3070484480 710962 33554432000 4194304 4417452 140737063085008 18446744073709551615 218917533669 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/22630/statm: 749630 710962 180 55 0 747516 0
[pid=22630/tid=22637] ppid=22628 vsize=2998520 CPUtime=4902.37 cores=0-7
/proc/22630/task/22637/stat : 22637 (plingeling) R 22628 22630 22604 0 -1 4202560 796934 1253 0 0 490080 157 0 0 25 0 9 0 655887788 3070484480 710962 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22630/tid=22638] ppid=22628 vsize=2998520 CPUtime=4902.08 cores=0-7
/proc/22630/task/22638/stat : 22638 (plingeling) R 22628 22630 22604 0 -1 4202560 1157551 1253 0 0 490008 200 0 0 25 0 9 0 655887788 3070484480 710962 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22630/tid=22639] ppid=22628 vsize=2998520 CPUtime=4844.98 cores=0-7
/proc/22630/task/22639/stat : 22639 (plingeling) R 22628 22630 22604 0 -1 4202560 1094372 1253 0 0 484299 199 0 0 25 0 9 0 655887788 3070484480 710962 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364756 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22630/tid=22640] ppid=22628 vsize=2998520 CPUtime=4902.24 cores=0-7
/proc/22630/task/22640/stat : 22640 (plingeling) R 22628 22630 22604 0 -1 4202560 454949 1253 0 0 490129 95 0 0 25 0 9 0 655887788 3070484480 710962 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22630/tid=22641] ppid=22628 vsize=2998520 CPUtime=4902.36 cores=0-7
/proc/22630/task/22641/stat : 22641 (plingeling) R 22628 22630 22604 0 -1 4202560 758505 1253 0 0 490069 167 0 0 25 0 9 0 655887788 3070484480 710962 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22630/tid=22642] ppid=22628 vsize=2998520 CPUtime=4900.54 cores=0-7
/proc/22630/task/22642/stat : 22642 (plingeling) R 22628 22630 22604 0 -1 4202560 844684 1253 0 0 489889 165 0 0 25 0 9 0 655887788 3070484480 710962 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=22630/tid=22643] ppid=22628 vsize=2998520 CPUtime=4901.97 cores=0-7
/proc/22630/task/22643/stat : 22643 (plingeling) R 22628 22630 22604 0 -1 4202560 1075748 1253 0 0 490012 185 0 0 25 0 9 0 655887788 3070484480 710962 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22630/tid=22644] ppid=22628 vsize=2998520 CPUtime=4902.37 cores=0-7
/proc/22630/task/22644/stat : 22644 (plingeling) R 22628 22630 22604 0 -1 4202560 568217 1253 0 0 490122 115 0 0 25 0 9 0 655887789 3070484480 710962 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364629 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39159
Current children cumulated vsize (KiB) 2998520

[startup+4962.3 s]
/proc/loadavg: 8.05 8.05 8.00 9/356 22791
/proc/meminfo: memFree=18379940/32951124 swapFree=65581052/67111528
[pid=22630] ppid=22628 vsize=3028728 CPUtime=39638.3 cores=0-7
/proc/22630/stat : 22630 (plingeling) S 22628 22630 22604 0 -1 4202496 6787445 1253 0 0 3962542 1290 0 0 25 0 9 0 655887783 3101417472 718482 33554432000 4194304 4417452 140737063085008 18446744073709551615 218917533669 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/22630/statm: 757182 718482 180 55 0 755068 0
[pid=22630/tid=22637] ppid=22628 vsize=3028728 CPUtime=4962.37 cores=0-7
/proc/22630/task/22637/stat : 22637 (plingeling) R 22628 22630 22604 0 -1 4202560 809437 1253 0 0 496079 158 0 0 25 0 9 0 655887788 3101417472 718482 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22630/tid=22638] ppid=22628 vsize=3028728 CPUtime=4962.08 cores=0-7
/proc/22630/task/22638/stat : 22638 (plingeling) R 22628 22630 22604 0 -1 4202560 1157551 1253 0 0 496008 200 0 0 25 0 9 0 655887788 3101417472 718482 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364564 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22630/tid=22639] ppid=22628 vsize=3028728 CPUtime=4904.26 cores=0-7
/proc/22630/task/22639/stat : 22639 (plingeling) R 22628 22630 22604 0 -1 4202560 1112690 1253 0 0 490225 201 0 0 25 0 9 0 655887788 3101417472 718482 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364756 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22630/tid=22640] ppid=22628 vsize=3028728 CPUtime=4962.24 cores=0-7
/proc/22630/task/22640/stat : 22640 (plingeling) R 22628 22630 22604 0 -1 4202560 454949 1253 0 0 496129 95 0 0 25 0 9 0 655887788 3101417472 718482 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364735 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22630/tid=22641] ppid=22628 vsize=3028728 CPUtime=4962.37 cores=0-7
/proc/22630/task/22641/stat : 22641 (plingeling) R 22628 22630 22604 0 -1 4202560 758505 1253 0 0 496070 167 0 0 25 0 9 0 655887788 3101417472 718482 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364842 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22630/tid=22642] ppid=22628 vsize=3028728 CPUtime=4960.53 cores=0-7
/proc/22630/task/22642/stat : 22642 (plingeling) R 22628 22630 22604 0 -1 4202560 846690 1253 0 0 495888 165 0 0 25 0 9 0 655887788 3101417472 718482 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=22630/tid=22643] ppid=22628 vsize=3028728 CPUtime=4961.97 cores=0-7
/proc/22630/task/22643/stat : 22643 (plingeling) R 22628 22630 22604 0 -1 4202560 1075748 1253 0 0 496012 185 0 0 25 0 9 0 655887788 3101417472 718482 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22630/tid=22644] ppid=22628 vsize=3028728 CPUtime=4962.37 cores=0-7
/proc/22630/task/22644/stat : 22644 (plingeling) R 22628 22630 22604 0 -1 4202560 568217 1253 0 0 496122 115 0 0 25 0 9 0 655887789 3101417472 718482 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39638.3
Current children cumulated vsize (KiB) 3028728



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5007.6 s]
/proc/loadavg: 8.02 8.04 8.00 9/356 22792
/proc/meminfo: memFree=18342720/32951124 swapFree=65581052/67111528
[pid=22630] ppid=22628 vsize=3065880 CPUtime=40000.2 cores=0-7
/proc/22630/stat : 22630 (plingeling) S 22628 22630 22604 0 -1 4202496 6796733 1253 0 0 3998732 1291 0 0 25 0 9 0 655887783 3139461120 727770 33554432000 4194304 4417452 140737063085008 18446744073709551615 218917533669 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/22630/statm: 766470 727770 180 55 0 764356 0
[pid=22630/tid=22637] ppid=22628 vsize=3065880 CPUtime=5007.68 cores=0-7
/proc/22630/task/22637/stat : 22637 (plingeling) R 22628 22630 22604 0 -1 4202560 818725 1253 0 0 500610 158 0 0 25 0 9 0 655887788 3139461120 727770 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22630/tid=22638] ppid=22628 vsize=3065880 CPUtime=5007.4 cores=0-7
/proc/22630/task/22638/stat : 22638 (plingeling) R 22628 22630 22604 0 -1 4202560 1157551 1253 0 0 500540 200 0 0 25 0 9 0 655887788 3139461120 727770 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22630/tid=22639] ppid=22628 vsize=3065880 CPUtime=4949.03 cores=0-7
/proc/22630/task/22639/stat : 22639 (plingeling) R 22628 22630 22604 0 -1 4202560 1112690 1253 0 0 494702 201 0 0 25 0 9 0 655887788 3139461120 727770 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22630/tid=22640] ppid=22628 vsize=3065880 CPUtime=5007.56 cores=0-7
/proc/22630/task/22640/stat : 22640 (plingeling) R 22628 22630 22604 0 -1 4202560 454949 1253 0 0 500661 95 0 0 25 0 9 0 655887788 3139461120 727770 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22630/tid=22641] ppid=22628 vsize=3065880 CPUtime=5007.68 cores=0-7
/proc/22630/task/22641/stat : 22641 (plingeling) R 22628 22630 22604 0 -1 4202560 758505 1253 0 0 500601 167 0 0 25 0 9 0 655887788 3139461120 727770 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364881 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22630/tid=22642] ppid=22628 vsize=3065880 CPUtime=5005.83 cores=0-7
/proc/22630/task/22642/stat : 22642 (plingeling) R 22628 22630 22604 0 -1 4202560 846690 1253 0 0 500418 165 0 0 25 0 9 0 655887788 3139461120 727770 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364616 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=22630/tid=22643] ppid=22628 vsize=3065880 CPUtime=5007.29 cores=0-7
/proc/22630/task/22643/stat : 22643 (plingeling) R 22628 22630 22604 0 -1 4202560 1075748 1253 0 0 500544 185 0 0 25 0 9 0 655887788 3139461120 727770 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22630/tid=22644] ppid=22628 vsize=3065880 CPUtime=5007.69 cores=0-7
/proc/22630/task/22644/stat : 22644 (plingeling) R 22628 22630 22604 0 -1 4202560 568217 1253 0 0 500654 115 0 0 25 0 9 0 655887789 3139461120 727770 33554432000 4194304 4417452 140737063085008 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40000.2
Current children cumulated vsize (KiB) 3065880

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5007.72 s]
/proc/loadavg: 8.02 8.04 8.00 9/356 22792
/proc/meminfo: memFree=18342720/32951124 swapFree=65581052/67111528
[pid=22630] ppid=22628 vsize=0 CPUtime=40000.4 cores=0-7
/proc/22630/stat : 22630 (plingeling) Z 22628 22630 22604 0 -1 4203548 6796733 1253 0 0 3998741 1301 0 0 18 0 2 0 655887783 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/22630/statm: 0 0 0 0 0 0 0
[pid=22630/tid=22639] ppid=22628 vsize=0 CPUtime=4949.13 cores=0-7
/proc/22630/task/22639/stat : 22639 (plingeling) R 22628 22630 22604 0 -1 4203588 1112690 1253 0 0 494702 211 0 0 25 0 2 0 655887788 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 40000.4
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5007.74
CPU time (s): 40000.5
CPU user time (s): 39987.4
CPU system time (s): 13.038
CPU usage (%): 798.772
Max. virtual memory (cumulated for all children) (KiB): 3295452

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39987.4
system time used= 13.038
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 6797986
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 282
involuntary context switches= 193689

runsolver used 16.1535 second user time and 42.4725 second system time

The end

Launcher Data 

Begin job on node137 at 2011-05-15 09:01:41
IDJOB=3372842
IDBENCH=84237
IDSOLVER=1851
FILE ID=node137/3372842-1305442901
RUNJOBID= node137-1305442901-22618
PBS_JOBID= 13324829
Free space on /tmp= 73800 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT03/industrial/kukula/addm_bench/am_7_7.shuffled-as.sat03-363.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3372842-1305442901/watcher-3372842-1305442901 -o /tmp/evaluation-result-3372842-1305442901/solver-3372842-1305442901 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3372842-1305442901.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= c7f51e1ee7556440ab29e1c4bfe29538
RANDOM SEED=1712215791

node137.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      21259176 kB
Buffers:        293880 kB
Cached:        3733324 kB
SwapCached:      36328 kB
Active:        1670892 kB
Inactive:      9795384 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      21259176 kB
SwapTotal:    67111528 kB
SwapFree:     65581052 kB
Dirty:             392 kB
Writeback:           0 kB
AnonPages:     7412124 kB
Mapped:          18392 kB
Slab:           121052 kB
PageTables:      39072 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 75889660 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73796 MiB
End job on node137 at 2011-05-15 10:25:10