Trace number 3370896

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 4800.28 1203.99

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m37-q505.75-n35-p15-s48276711.cnf
MD5SUM1bac7b96ea6eb843b1de7405a721cd87
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.861868
Satisfiable
(Un)Satisfiability was proved
Number of variables1244
Number of clauses150604
Sum of the clauses size302382
Maximum clause length37
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2150569
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 535

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3370896-1305432991/watcher-3370896-1305432991 -o /tmp/evaluation-result-3370896-1305432991/solver-3370896-1305432991 -C 4800 -W 1300 -M 15500 plingeling -t 4 HOME/instance-3370896-1305432991.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.71 5.27 6.61 2/380 21354
/proc/meminfo: memFree=14808356/32951124 swapFree=65586204/67111528
[pid=21354] ppid=21352 vsize=10528 CPUtime=0.01 cores=1,3,5,7
/proc/21354/stat : 21354 (plingeling) R 21352 21354 14218 0 -1 4202496 774 1252 0 0 1 0 0 0 23 0 1 0 733529785 10780672 655 33554432000 4194304 4417452 140735749956096 18446744073709551615 216735178496 0 0 4096 17442 0 0 0 17 1 0 0 0
/proc/21354/statm: 2632 655 148 55 0 518 0

[startup+0.0182011 s]
/proc/loadavg: 3.71 5.27 6.61 2/380 21354
/proc/meminfo: memFree=14808356/32951124 swapFree=65586204/67111528
[pid=21354] ppid=21352 vsize=11552 CPUtime=0.01 cores=1,3,5,7
/proc/21354/stat : 21354 (plingeling) R 21352 21354 14218 0 -1 4202496 1030 1252 0 0 1 0 0 0 23 0 1 0 733529785 11829248 911 33554432000 4194304 4417452 140735749956096 18446744073709551615 4262065 0 0 4096 17442 0 0 0 17 1 0 0 0
/proc/21354/statm: 2888 911 148 55 0 774 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 11552

[startup+0.100189 s]
/proc/loadavg: 3.71 5.27 6.61 2/380 21354
/proc/meminfo: memFree=14808356/32951124 swapFree=65586204/67111528
[pid=21354] ppid=21352 vsize=66860 CPUtime=0.09 cores=1,3,5,7
/proc/21354/stat : 21354 (plingeling) S 21352 21354 14218 0 -1 4202496 4639 1252 0 0 9 0 0 0 24 0 5 0 733529785 68464640 4516 33554432000 4194304 4417452 140735749956096 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/21354/statm: 16715 4516 164 55 0 14601 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 66860

[startup+0.300174 s]
/proc/loadavg: 3.71 5.27 6.61 2/380 21354
/proc/meminfo: memFree=14808356/32951124 swapFree=65586204/67111528
[pid=21354] ppid=21352 vsize=133836 CPUtime=0.89 cores=1,3,5,7
/proc/21354/stat : 21354 (plingeling) S 21352 21354 14218 0 -1 4202496 9220 1252 0 0 88 1 0 0 24 0 5 0 733529785 137048064 4896 33554432000 4194304 4417452 140735749956096 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/21354/statm: 33459 4896 176 55 0 31345 0
Current children cumulated CPU time (s) 0.89
Current children cumulated vsize (KiB) 133836

[startup+0.701085 s]
/proc/loadavg: 3.71 5.27 6.61 2/380 21354
/proc/meminfo: memFree=14808356/32951124 swapFree=65586204/67111528
[pid=21354] ppid=21352 vsize=144400 CPUtime=2.49 cores=1,3,5,7
/proc/21354/stat : 21354 (plingeling) S 21352 21354 14218 0 -1 4202496 13145 1252 0 0 247 2 0 0 24 0 5 0 733529785 147865600 7541 33554432000 4194304 4417452 140735749956096 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/21354/statm: 36100 7541 180 55 0 33986 0
Current children cumulated CPU time (s) 2.49
Current children cumulated vsize (KiB) 144400

[startup+1.50094 s]
/proc/loadavg: 4.05 5.31 6.61 9/392 21379
/proc/meminfo: memFree=14724316/32951124 swapFree=65586204/67111528
[pid=21354] ppid=21352 vsize=152024 CPUtime=5.67 cores=1,3,5,7
/proc/21354/stat : 21354 (plingeling) S 21352 21354 14218 0 -1 4202496 17000 1252 0 0 565 2 0 0 24 0 5 0 733529785 155672576 9217 33554432000 4194304 4417452 140735749956096 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/21354/statm: 38006 9217 180 55 0 35892 0
[pid=21354/tid=21372] ppid=21352 vsize=152024 CPUtime=1.39 cores=1,3,5,7
/proc/21354/task/21372/stat : 21372 (plingeling) R 21352 21354 14218 0 -1 4202560 1944 1252 0 0 139 0 0 0 25 0 5 0 733529795 155672576 9217 33554432000 4194304 4417452 140735749956096 18446744073709551615 4258244 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=21354/tid=21373] ppid=21352 vsize=152024 CPUtime=1.39 cores=1,3,5,7
/proc/21354/task/21373/stat : 21373 (plingeling) R 21352 21354 14218 0 -1 4202560 1623 1252 0 0 139 0 0 0 25 0 5 0 733529795 155672576 9217 33554432000 4194304 4417452 140735749956096 18446744073709551615 4249130 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=21354/tid=21374] ppid=21352 vsize=152024 CPUtime=1.39 cores=1,3,5,7
/proc/21354/task/21374/stat : 21374 (plingeling) R 21352 21354 14218 0 -1 4202560 1772 1252 0 0 139 0 0 0 25 0 5 0 733529795 155672576 9217 33554432000 4194304 4417452 140735749956096 18446744073709551615 4211488 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21354/tid=21375] ppid=21352 vsize=152024 CPUtime=1.38 cores=1,3,5,7
/proc/21354/task/21375/stat : 21375 (plingeling) R 21352 21354 14218 0 -1 4202560 7035 1252 0 0 137 1 0 0 25 0 5 0 733529795 155672576 9217 33554432000 4194304 4417452 140735749956096 18446744073709551615 4357256 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 5.67
Current children cumulated vsize (KiB) 152024

[startup+3.10067 s]
/proc/loadavg: 4.05 5.31 6.61 9/392 21379
/proc/meminfo: memFree=14702244/32951124 swapFree=65586204/67111528
[pid=21354] ppid=21352 vsize=159608 CPUtime=12.06 cores=1,3,5,7
/proc/21354/stat : 21354 (plingeling) S 21352 21354 14218 0 -1 4202496 24373 1252 0 0 1203 3 0 0 24 0 5 0 733529785 163438592 11343 33554432000 4194304 4417452 140735749956096 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/21354/statm: 39902 11343 180 55 0 37788 0
[pid=21354/tid=21372] ppid=21352 vsize=159608 CPUtime=2.99 cores=1,3,5,7
/proc/21354/task/21372/stat : 21372 (plingeling) R 21352 21354 14218 0 -1 4202560 1944 1252 0 0 299 0 0 0 25 0 5 0 733529795 163438592 11343 33554432000 4194304 4417452 140735749956096 18446744073709551615 4250047 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=21354/tid=21373] ppid=21352 vsize=159608 CPUtime=2.99 cores=1,3,5,7
/proc/21354/task/21373/stat : 21373 (plingeling) R 21352 21354 14218 0 -1 4202560 1623 1252 0 0 299 0 0 0 25 0 5 0 733529795 163438592 11343 33554432000 4194304 4417452 140735749956096 18446744073709551615 4235985 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=21354/tid=21374] ppid=21352 vsize=159608 CPUtime=2.99 cores=1,3,5,7
/proc/21354/task/21374/stat : 21374 (plingeling) R 21352 21354 14218 0 -1 4202560 2095 1252 0 0 299 0 0 0 25 0 5 0 733529795 163438592 11343 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364548 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21354/tid=21375] ppid=21352 vsize=159608 CPUtime=2.97 cores=1,3,5,7
/proc/21354/task/21375/stat : 21375 (plingeling) R 21352 21354 14218 0 -1 4202560 14085 1252 0 0 295 2 0 0 25 0 5 0 733529795 163438592 11343 33554432000 4194304 4417452 140735749956096 18446744073709551615 4237002 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 12.06
Current children cumulated vsize (KiB) 159608

[startup+6.30111 s]
/proc/loadavg: 4.37 5.36 6.62 9/392 21379
/proc/meminfo: memFree=14636900/32951124 swapFree=65586204/67111528
[pid=21354] ppid=21352 vsize=180624 CPUtime=24.82 cores=1,3,5,7
/proc/21354/stat : 21354 (plingeling) S 21352 21354 14218 0 -1 4202496 37904 1252 0 0 2477 5 0 0 24 0 5 0 733529785 184958976 16597 33554432000 4194304 4417452 140735749956096 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/21354/statm: 45156 16597 180 55 0 43042 0
[pid=21354/tid=21372] ppid=21352 vsize=180624 CPUtime=6.2 cores=1,3,5,7
/proc/21354/task/21372/stat : 21372 (plingeling) R 21352 21354 14218 0 -1 4202560 2222 1252 0 0 620 0 0 0 25 0 5 0 733529795 184958976 16597 33554432000 4194304 4417452 140735749956096 18446744073709551615 4249143 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=21354/tid=21373] ppid=21352 vsize=180624 CPUtime=6.19 cores=1,3,5,7
/proc/21354/task/21373/stat : 21373 (plingeling) R 21352 21354 14218 0 -1 4202560 2263 1252 0 0 619 0 0 0 25 0 5 0 733529795 184958976 16597 33554432000 4194304 4417452 140735749956096 18446744073709551615 4249991 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=21354/tid=21374] ppid=21352 vsize=180624 CPUtime=6.19 cores=1,3,5,7
/proc/21354/task/21374/stat : 21374 (plingeling) R 21352 21354 14218 0 -1 4202560 2615 1252 0 0 619 0 0 0 25 0 5 0 733529795 184958976 16597 33554432000 4194304 4417452 140735749956096 18446744073709551615 4250623 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21354/tid=21375] ppid=21352 vsize=180624 CPUtime=6.13 cores=1,3,5,7
/proc/21354/task/21375/stat : 21375 (plingeling) R 21352 21354 14218 0 -1 4202560 26178 1252 0 0 609 4 0 0 25 0 5 0 733529795 184958976 16597 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 24.82
Current children cumulated vsize (KiB) 180624

[startup+12.701 s]
/proc/loadavg: 4.66 5.40 6.63 9/392 21379
/proc/meminfo: memFree=14553044/32951124 swapFree=65586204/67111528
[pid=21354] ppid=21352 vsize=211480 CPUtime=50.35 cores=1,3,5,7
/proc/21354/stat : 21354 (plingeling) S 21352 21354 14218 0 -1 4202496 58382 1252 0 0 5026 9 0 0 24 0 5 0 733529785 216555520 24308 33554432000 4194304 4417452 140735749956096 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/21354/statm: 52870 24308 180 55 0 50756 0
[pid=21354/tid=21372] ppid=21352 vsize=211480 CPUtime=12.59 cores=1,3,5,7
/proc/21354/task/21372/stat : 21372 (plingeling) R 21352 21354 14218 0 -1 4202560 3391 1252 0 0 1259 0 0 0 25 0 5 0 733529795 216555520 24308 33554432000 4194304 4417452 140735749956096 18446744073709551615 4311324 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=21354/tid=21373] ppid=21352 vsize=211480 CPUtime=12.59 cores=1,3,5,7
/proc/21354/task/21373/stat : 21373 (plingeling) R 21352 21354 14218 0 -1 4202560 2647 1252 0 0 1259 0 0 0 25 0 5 0 733529795 216555520 24308 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=21354/tid=21374] ppid=21352 vsize=211480 CPUtime=12.59 cores=1,3,5,7
/proc/21354/task/21374/stat : 21374 (plingeling) R 21352 21354 14218 0 -1 4202560 5781 1252 0 0 1259 0 0 0 25 0 5 0 733529795 216555520 24308 33554432000 4194304 4417452 140735749956096 18446744073709551615 4249228 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21354/tid=21375] ppid=21352 vsize=211480 CPUtime=12.45 cores=1,3,5,7
/proc/21354/task/21375/stat : 21375 (plingeling) R 21352 21354 14218 0 -1 4202560 41937 1252 0 0 1238 7 0 0 25 0 5 0 733529795 216555520 24308 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 50.35
Current children cumulated vsize (KiB) 211480

[startup+25.5007 s]
/proc/loadavg: 5.24 5.50 6.65 9/392 21380
/proc/meminfo: memFree=14425072/32951124 swapFree=65586204/67111528
[pid=21354] ppid=21352 vsize=322140 CPUtime=101.38 cores=1,3,5,7
/proc/21354/stat : 21354 (plingeling) S 21352 21354 14218 0 -1 4202496 107892 1252 0 0 10122 16 0 0 24 0 5 0 733529785 329871360 51973 33554432000 4194304 4417452 140735749956096 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/21354/statm: 80535 51973 180 55 0 78421 0
[pid=21354/tid=21372] ppid=21352 vsize=322140 CPUtime=25.4 cores=1,3,5,7
/proc/21354/task/21372/stat : 21372 (plingeling) R 21352 21354 14218 0 -1 4202560 9131 1252 0 0 2539 1 0 0 25 0 5 0 733529795 329871360 51973 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=21354/tid=21373] ppid=21352 vsize=322140 CPUtime=25.39 cores=1,3,5,7
/proc/21354/task/21373/stat : 21373 (plingeling) R 21352 21354 14218 0 -1 4202560 13177 1252 0 0 2538 1 0 0 25 0 5 0 733529795 329871360 51973 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364542 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=21354/tid=21374] ppid=21352 vsize=322140 CPUtime=25.39 cores=1,3,5,7
/proc/21354/task/21374/stat : 21374 (plingeling) R 21352 21354 14218 0 -1 4202560 14234 1252 0 0 2538 1 0 0 25 0 5 0 733529795 329871360 51973 33554432000 4194304 4417452 140735749956096 18446744073709551615 4233920 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21354/tid=21375] ppid=21352 vsize=322140 CPUtime=25.09 cores=1,3,5,7
/proc/21354/task/21375/stat : 21375 (plingeling) R 21352 21354 14218 0 -1 4202560 66724 1252 0 0 2498 11 0 0 25 0 5 0 733529795 329871360 51973 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364756 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 101.38
Current children cumulated vsize (KiB) 322140

[startup+51.1003 s]
/proc/loadavg: 6.33 5.74 6.69 9/392 21381
/proc/meminfo: memFree=14177252/32951124 swapFree=65586204/67111528
[pid=21354] ppid=21352 vsize=414916 CPUtime=203.48 cores=1,3,5,7
/proc/21354/stat : 21354 (plingeling) S 21352 21354 14218 0 -1 4202496 174814 1252 0 0 20320 28 0 0 24 0 5 0 733529785 424873984 75167 33554432000 4194304 4417452 140735749956096 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/21354/statm: 103729 75167 180 55 0 101615 0
[pid=21354/tid=21372] ppid=21352 vsize=414916 CPUtime=50.99 cores=1,3,5,7
/proc/21354/task/21372/stat : 21372 (plingeling) R 21352 21354 14218 0 -1 4202560 14823 1252 0 0 5098 1 0 0 25 0 5 0 733529795 424873984 75167 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364812 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=21354/tid=21373] ppid=21352 vsize=414916 CPUtime=50.99 cores=1,3,5,7
/proc/21354/task/21373/stat : 21373 (plingeling) R 21352 21354 14218 0 -1 4202560 25257 1252 0 0 5095 4 0 0 25 0 5 0 733529795 424873984 75167 33554432000 4194304 4417452 140735749956096 18446744073709551615 4251081 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=21354/tid=21374] ppid=21352 vsize=414916 CPUtime=50.99 cores=1,3,5,7
/proc/21354/task/21374/stat : 21374 (plingeling) R 21352 21354 14218 0 -1 4202560 27299 1252 0 0 5096 3 0 0 25 0 5 0 733529795 424873984 75167 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21354/tid=21375] ppid=21352 vsize=414916 CPUtime=50.38 cores=1,3,5,7
/proc/21354/task/21375/stat : 21375 (plingeling) R 21352 21354 14218 0 -1 4202560 102809 1252 0 0 5020 18 0 0 25 0 5 0 733529795 424873984 75167 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 203.48
Current children cumulated vsize (KiB) 414916

[startup+102.314 s]
/proc/loadavg: 7.32 6.10 6.76 9/392 21382
/proc/meminfo: memFree=13861288/32951124 swapFree=65586204/67111528
[pid=21354] ppid=21352 vsize=562328 CPUtime=407.69 cores=1,3,5,7
/proc/21354/stat : 21354 (plingeling) S 21352 21354 14218 0 -1 4202496 333869 1252 0 0 40717 52 0 0 24 0 5 0 733529785 575823872 112020 33554432000 4194304 4417452 140735749956096 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/21354/statm: 140582 112020 180 55 0 138468 0
[pid=21354/tid=21372] ppid=21352 vsize=562328 CPUtime=102.21 cores=1,3,5,7
/proc/21354/task/21372/stat : 21372 (plingeling) R 21352 21354 14218 0 -1 4202560 46385 1252 0 0 10215 6 0 0 25 0 5 0 733529795 575823872 112020 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364838 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=21354/tid=21373] ppid=21352 vsize=562328 CPUtime=102.21 cores=1,3,5,7
/proc/21354/task/21373/stat : 21373 (plingeling) R 21352 21354 14218 0 -1 4202560 44326 1252 0 0 10214 7 0 0 25 0 5 0 733529795 575823872 112020 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364819 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=21354/tid=21374] ppid=21352 vsize=562328 CPUtime=102.2 cores=1,3,5,7
/proc/21354/task/21374/stat : 21374 (plingeling) R 21352 21354 14218 0 -1 4202560 92648 1252 0 0 10209 11 0 0 25 0 5 0 733529795 575823872 112020 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21354/tid=21375] ppid=21352 vsize=562328 CPUtime=100.94 cores=1,3,5,7
/proc/21354/task/21375/stat : 21375 (plingeling) R 21352 21354 14218 0 -1 4202560 145884 1252 0 0 10068 26 0 0 25 0 5 0 733529795 575823872 112020 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364812 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 407.69
Current children cumulated vsize (KiB) 562328

[startup+162.301 s]
/proc/loadavg: 7.79 6.46 6.84 9/392 21383
/proc/meminfo: memFree=13547296/32951124 swapFree=65586204/67111528
[pid=21354] ppid=21352 vsize=690772 CPUtime=646.89 cores=1,3,5,7
/proc/21354/stat : 21354 (plingeling) S 21352 21354 14218 0 -1 4202496 514012 1252 0 0 64606 83 0 0 24 0 5 0 733529785 707350528 144131 33554432000 4194304 4417452 140735749956096 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/21354/statm: 172693 144131 180 55 0 170579 0
[pid=21354/tid=21372] ppid=21352 vsize=690772 CPUtime=162.2 cores=1,3,5,7
/proc/21354/task/21372/stat : 21372 (plingeling) R 21352 21354 14218 0 -1 4202560 66713 1252 0 0 16210 10 0 0 25 0 5 0 733529795 707350528 144131 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=21354/tid=21373] ppid=21352 vsize=690772 CPUtime=162.19 cores=1,3,5,7
/proc/21354/task/21373/stat : 21373 (plingeling) R 21352 21354 14218 0 -1 4202560 91131 1252 0 0 16206 13 0 0 25 0 5 0 733529795 707350528 144131 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=21354/tid=21374] ppid=21352 vsize=690772 CPUtime=162.19 cores=1,3,5,7
/proc/21354/task/21374/stat : 21374 (plingeling) R 21352 21354 14218 0 -1 4202560 150122 1252 0 0 16201 18 0 0 25 0 5 0 733529795 707350528 144131 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21354/tid=21375] ppid=21352 vsize=690772 CPUtime=160.18 cores=1,3,5,7
/proc/21354/task/21375/stat : 21375 (plingeling) R 21352 21354 14218 0 -1 4202560 201420 1252 0 0 15979 39 0 0 25 0 5 0 733529795 707350528 144131 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 646.89
Current children cumulated vsize (KiB) 690772


################
# More data... #
################

[pid=21354/tid=21373] ppid=21352 vsize=1595140 CPUtime=762.22 cores=1,3,5,7
/proc/21354/task/21373/stat : 21373 (plingeling) R 21352 21354 14218 0 -1 4202560 351292 1252 0 0 76166 56 0 0 25 0 5 0 733529795 1633423360 370223 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364829 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=21354/tid=21374] ppid=21352 vsize=1595140 CPUtime=762.22 cores=1,3,5,7
/proc/21354/task/21374/stat : 21374 (plingeling) R 21352 21354 14218 0 -1 4202560 665609 1252 0 0 76131 91 0 0 25 0 5 0 733529795 1633423360 370223 33554432000 4194304 4417452 140735749956096 18446744073709551615 4366548 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21354/tid=21375] ppid=21352 vsize=1595140 CPUtime=752.58 cores=1,3,5,7
/proc/21354/task/21375/stat : 21375 (plingeling) R 21352 21354 14218 0 -1 4202560 424831 1252 0 0 75175 83 0 0 25 0 5 0 733529795 1633423360 370223 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364735 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3039.36
Current children cumulated vsize (KiB) 1595140

[startup+822.3 s]
/proc/loadavg: 8.03 7.89 7.45 9/392 21401
/proc/meminfo: memFree=11200860/32951124 swapFree=65586204/67111528
[pid=21354] ppid=21352 vsize=1509108 CPUtime=3278.56 cores=1,3,5,7
/proc/21354/stat : 21354 (plingeling) S 21352 21354 14218 0 -1 4202496 2012325 1252 0 0 327543 313 0 0 24 0 5 0 733529785 1545326592 348715 33554432000 4194304 4417452 140735749956096 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/21354/statm: 377277 348715 180 55 0 375163 0
[pid=21354/tid=21372] ppid=21352 vsize=1509108 CPUtime=822.21 cores=1,3,5,7
/proc/21354/task/21372/stat : 21372 (plingeling) R 21352 21354 14218 0 -1 4202560 484441 1252 0 0 82151 70 0 0 25 0 5 0 733529795 1545326592 348715 33554432000 4194304 4417452 140735749956096 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=21354/tid=21373] ppid=21352 vsize=1509108 CPUtime=822.2 cores=1,3,5,7
/proc/21354/task/21373/stat : 21373 (plingeling) R 21352 21354 14218 0 -1 4202560 383050 1252 0 0 82159 61 0 0 25 0 5 0 733529795 1545326592 348715 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=21354/tid=21374] ppid=21352 vsize=1509108 CPUtime=822.21 cores=1,3,5,7
/proc/21354/task/21374/stat : 21374 (plingeling) R 21352 21354 14218 0 -1 4202560 715377 1252 0 0 82124 97 0 0 25 0 5 0 733529795 1545326592 348715 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364611 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21354/tid=21375] ppid=21352 vsize=1509108 CPUtime=811.82 cores=1,3,5,7
/proc/21354/task/21375/stat : 21375 (plingeling) R 21352 21354 14218 0 -1 4202560 424831 1252 0 0 81099 83 0 0 25 0 5 0 733529795 1545326592 348715 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364632 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3278.56
Current children cumulated vsize (KiB) 1509108

[startup+882.301 s]
/proc/loadavg: 8.06 7.92 7.49 9/392 21403
/proc/meminfo: memFree=10811484/32951124 swapFree=65586204/67111528
[pid=21354] ppid=21352 vsize=1700524 CPUtime=3517.8 cores=1,3,5,7
/proc/21354/stat : 21354 (plingeling) S 21352 21354 14218 0 -1 4202496 2095493 1252 0 0 351454 326 0 0 24 0 5 0 733529785 1741336576 396569 33554432000 4194304 4417452 140735749956096 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/21354/statm: 425131 396569 180 55 0 423017 0
[pid=21354/tid=21372] ppid=21352 vsize=1700524 CPUtime=882.22 cores=1,3,5,7
/proc/21354/task/21372/stat : 21372 (plingeling) R 21352 21354 14218 0 -1 4202560 506213 1252 0 0 88147 75 0 0 25 0 5 0 733529795 1741336576 396569 33554432000 4194304 4417452 140735749956096 18446744073709551615 4310236 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=21354/tid=21373] ppid=21352 vsize=1700524 CPUtime=882.2 cores=1,3,5,7
/proc/21354/task/21373/stat : 21373 (plingeling) R 21352 21354 14218 0 -1 4202560 423372 1252 0 0 88154 66 0 0 25 0 5 0 733529795 1741336576 396569 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=21354/tid=21374] ppid=21352 vsize=1700524 CPUtime=882.21 cores=1,3,5,7
/proc/21354/task/21374/stat : 21374 (plingeling) R 21352 21354 14218 0 -1 4202560 728472 1252 0 0 88122 99 0 0 25 0 5 0 733529795 1741336576 396569 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21354/tid=21375] ppid=21352 vsize=1700524 CPUtime=871.04 cores=1,3,5,7
/proc/21354/task/21375/stat : 21375 (plingeling) R 21352 21354 14218 0 -1 4202560 432810 1252 0 0 87020 84 0 0 25 0 5 0 733529795 1741336576 396569 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3517.8
Current children cumulated vsize (KiB) 1700524

[startup+942.3 s]
/proc/loadavg: 8.06 7.95 7.52 9/392 21404
/proc/meminfo: memFree=10699740/32951124 swapFree=65586204/67111528
[pid=21354] ppid=21352 vsize=1744372 CPUtime=3757.05 cores=1,3,5,7
/proc/21354/stat : 21354 (plingeling) S 21352 21354 14218 0 -1 4202496 2208174 1252 0 0 375365 340 0 0 24 0 5 0 733529785 1786236928 407531 33554432000 4194304 4417452 140735749956096 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/21354/statm: 436093 407531 180 55 0 433979 0
[pid=21354/tid=21372] ppid=21352 vsize=1744372 CPUtime=942.21 cores=1,3,5,7
/proc/21354/task/21372/stat : 21372 (plingeling) R 21352 21354 14218 0 -1 4202560 565951 1252 0 0 94139 82 0 0 25 0 5 0 733529795 1786236928 407531 33554432000 4194304 4417452 140735749956096 18446744073709551615 4366385 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=21354/tid=21373] ppid=21352 vsize=1744372 CPUtime=942.21 cores=1,3,5,7
/proc/21354/task/21373/stat : 21373 (plingeling) R 21352 21354 14218 0 -1 4202560 447948 1252 0 0 94152 69 0 0 25 0 5 0 733529795 1786236928 407531 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=21354/tid=21374] ppid=21352 vsize=1744372 CPUtime=942.2 cores=1,3,5,7
/proc/21354/task/21374/stat : 21374 (plingeling) R 21352 21354 14218 0 -1 4202560 756839 1252 0 0 94118 102 0 0 25 0 5 0 733529795 1786236928 407531 33554432000 4194304 4417452 140735749956096 18446744073709551615 4309925 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21354/tid=21375] ppid=21352 vsize=1744372 CPUtime=930.29 cores=1,3,5,7
/proc/21354/task/21375/stat : 21375 (plingeling) R 21352 21354 14218 0 -1 4202560 432810 1252 0 0 92945 84 0 0 25 0 5 0 733529795 1786236928 407531 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364752 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3757.05
Current children cumulated vsize (KiB) 1744372

[startup+1002.3 s]
/proc/loadavg: 8.06 7.97 7.56 9/392 21406
/proc/meminfo: memFree=10462512/32951124 swapFree=65586204/67111528
[pid=21354] ppid=21352 vsize=1861796 CPUtime=3996.28 cores=1,3,5,7
/proc/21354/stat : 21354 (plingeling) S 21352 21354 14218 0 -1 4202496 2370614 1252 0 0 399267 361 0 0 24 0 5 0 733529785 1906479104 436887 33554432000 4194304 4417452 140735749956096 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/21354/statm: 465449 436887 180 55 0 463335 0
[pid=21354/tid=21372] ppid=21352 vsize=1861796 CPUtime=1002.21 cores=1,3,5,7
/proc/21354/task/21372/stat : 21372 (plingeling) R 21352 21354 14218 0 -1 4202560 635373 1252 0 0 100131 90 0 0 25 0 5 0 733529795 1906479104 436887 33554432000 4194304 4417452 140735749956096 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=21354/tid=21373] ppid=21352 vsize=1861796 CPUtime=1002.21 cores=1,3,5,7
/proc/21354/task/21373/stat : 21373 (plingeling) R 21352 21354 14218 0 -1 4202560 472524 1252 0 0 100148 73 0 0 25 0 5 0 733529795 1906479104 436887 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=21354/tid=21374] ppid=21352 vsize=1861796 CPUtime=1002.21 cores=1,3,5,7
/proc/21354/task/21374/stat : 21374 (plingeling) R 21352 21354 14218 0 -1 4202560 825281 1252 0 0 100109 112 0 0 25 0 5 0 733529795 1906479104 436887 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364735 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21354/tid=21375] ppid=21352 vsize=1861796 CPUtime=989.52 cores=1,3,5,7
/proc/21354/task/21375/stat : 21375 (plingeling) R 21352 21354 14218 0 -1 4202560 432810 1252 0 0 98868 84 0 0 25 0 5 0 733529795 1906479104 436887 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364809 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3996.28
Current children cumulated vsize (KiB) 1861796

[startup+1062.3 s]
/proc/loadavg: 8.06 7.99 7.58 9/392 21407
/proc/meminfo: memFree=11061164/32951124 swapFree=65586204/67111528
[pid=21354] ppid=21352 vsize=1724944 CPUtime=4235.53 cores=1,3,5,7
/proc/21354/stat : 21354 (plingeling) S 21352 21354 14218 0 -1 4202496 2583492 1252 0 0 423161 392 0 0 24 0 5 0 733529785 1766342656 402706 33554432000 4194304 4417452 140735749956096 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/21354/statm: 431236 402706 180 55 0 429122 0
[pid=21354/tid=21372] ppid=21352 vsize=1724944 CPUtime=1062.22 cores=1,3,5,7
/proc/21354/task/21372/stat : 21372 (plingeling) R 21352 21354 14218 0 -1 4202560 705011 1252 0 0 106124 98 0 0 25 0 5 0 733529795 1766342656 402706 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364756 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=21354/tid=21373] ppid=21352 vsize=1724944 CPUtime=1062.21 cores=1,3,5,7
/proc/21354/task/21373/stat : 21373 (plingeling) R 21352 21354 14218 0 -1 4202560 488908 1252 0 0 106146 75 0 0 25 0 5 0 733529795 1766342656 402706 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=21354/tid=21374] ppid=21352 vsize=1724944 CPUtime=1062.21 cores=1,3,5,7
/proc/21354/task/21374/stat : 21374 (plingeling) R 21352 21354 14218 0 -1 4202560 866241 1252 0 0 106104 117 0 0 25 0 5 0 733529795 1766342656 402706 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21354/tid=21375] ppid=21352 vsize=1724944 CPUtime=1048.77 cores=1,3,5,7
/proc/21354/task/21375/stat : 21375 (plingeling) R 21352 21354 14218 0 -1 4202560 518706 1252 0 0 104777 100 0 0 25 0 5 0 733529795 1766342656 402706 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4235.53
Current children cumulated vsize (KiB) 1724944

[startup+1122.3 s]
/proc/loadavg: 8.05 8.00 7.61 9/392 21409
/proc/meminfo: memFree=11004352/32951124 swapFree=65586204/67111528
[pid=21354] ppid=21352 vsize=1673020 CPUtime=4474.77 cores=1,3,5,7
/proc/21354/stat : 21354 (plingeling) S 21352 21354 14218 0 -1 4202496 2660591 1252 0 0 447076 401 0 0 24 0 5 0 733529785 1713172480 389693 33554432000 4194304 4417452 140735749956096 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/21354/statm: 418255 389693 180 55 0 416141 0
[pid=21354/tid=21372] ppid=21352 vsize=1673020 CPUtime=1122.22 cores=1,3,5,7
/proc/21354/task/21372/stat : 21372 (plingeling) R 21352 21354 14218 0 -1 4202560 734580 1252 0 0 112121 101 0 0 25 0 5 0 733529795 1713172480 389693 33554432000 4194304 4417452 140735749956096 18446744073709551615 4263050 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=21354/tid=21373] ppid=21352 vsize=1673020 CPUtime=1122.22 cores=1,3,5,7
/proc/21354/task/21373/stat : 21373 (plingeling) R 21352 21354 14218 0 -1 4202560 511862 1252 0 0 112144 78 0 0 25 0 5 0 733529795 1713172480 389693 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=21354/tid=21374] ppid=21352 vsize=1673020 CPUtime=1122.21 cores=1,3,5,7
/proc/21354/task/21374/stat : 21374 (plingeling) R 21352 21354 14218 0 -1 4202560 890817 1252 0 0 112101 120 0 0 25 0 5 0 733529795 1713172480 389693 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364821 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21354/tid=21375] ppid=21352 vsize=1673020 CPUtime=1108 cores=1,3,5,7
/proc/21354/task/21375/stat : 21375 (plingeling) R 21352 21354 14218 0 -1 4202560 518706 1252 0 0 110700 100 0 0 25 0 5 0 733529795 1713172480 389693 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4474.77
Current children cumulated vsize (KiB) 1673020

[startup+1182.3 s]
/proc/loadavg: 8.06 8.01 7.64 9/392 21411
/proc/meminfo: memFree=10726068/32951124 swapFree=65586204/67111528
[pid=21354] ppid=21352 vsize=1790508 CPUtime=4714.01 cores=1,3,5,7
/proc/21354/stat : 21354 (plingeling) S 21352 21354 14218 0 -1 4202496 2747307 1252 0 0 470989 412 0 0 24 0 5 0 733529785 1833480192 419065 33554432000 4194304 4417452 140735749956096 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/21354/statm: 447627 419065 180 55 0 445513 0
[pid=21354/tid=21372] ppid=21352 vsize=1790508 CPUtime=1182.22 cores=1,3,5,7
/proc/21354/task/21372/stat : 21372 (plingeling) R 21352 21354 14218 0 -1 4202560 775410 1252 0 0 118116 106 0 0 25 0 5 0 733529795 1833480192 419065 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=21354/tid=21373] ppid=21352 vsize=1790508 CPUtime=1182.22 cores=1,3,5,7
/proc/21354/task/21373/stat : 21373 (plingeling) R 21352 21354 14218 0 -1 4202560 511862 1252 0 0 118144 78 0 0 25 0 5 0 733529795 1833480192 419065 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=21354/tid=21374] ppid=21352 vsize=1790508 CPUtime=1182.21 cores=1,3,5,7
/proc/21354/task/21374/stat : 21374 (plingeling) R 21352 21354 14218 0 -1 4202560 936703 1252 0 0 118095 126 0 0 25 0 5 0 733529795 1833480192 419065 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364881 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21354/tid=21375] ppid=21352 vsize=1790508 CPUtime=1167.24 cores=1,3,5,7
/proc/21354/task/21375/stat : 21375 (plingeling) R 21352 21354 14218 0 -1 4202560 518706 1252 0 0 116624 100 0 0 25 0 5 0 733529795 1833480192 419065 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4714.01
Current children cumulated vsize (KiB) 1790508



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.9 s]
/proc/loadavg: 8.11 8.03 7.65 9/392 21411
/proc/meminfo: memFree=10623264/32951124 swapFree=65586204/67111528
[pid=21354] ppid=21352 vsize=1872804 CPUtime=4800.18 cores=1,3,5,7
/proc/21354/stat : 21354 (plingeling) S 21352 21354 14218 0 -1 4202496 2808388 1252 0 0 479597 421 0 0 24 0 5 0 733529785 1917751296 436683 33554432000 4194304 4417452 140735749956096 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/21354/statm: 468201 436683 180 55 0 466087 0
[pid=21354/tid=21372] ppid=21352 vsize=1872804 CPUtime=1203.83 cores=1,3,5,7
/proc/21354/task/21372/stat : 21372 (plingeling) R 21352 21354 14218 0 -1 4202560 828299 1252 0 0 120269 114 0 0 25 0 5 0 733529795 1917751296 436683 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=21354/tid=21373] ppid=21352 vsize=1872804 CPUtime=1203.83 cores=1,3,5,7
/proc/21354/task/21373/stat : 21373 (plingeling) R 21352 21354 14218 0 -1 4202560 511862 1252 0 0 120305 78 0 0 25 0 5 0 733529795 1917751296 436683 33554432000 4194304 4417452 140735749956096 18446744073709551615 4251133 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=21354/tid=21374] ppid=21352 vsize=1872804 CPUtime=1203.83 cores=1,3,5,7
/proc/21354/task/21374/stat : 21374 (plingeling) R 21352 21354 14218 0 -1 4202560 944895 1252 0 0 120256 127 0 0 25 0 5 0 733529795 1917751296 436683 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21354/tid=21375] ppid=21352 vsize=1872804 CPUtime=1188.56 cores=1,3,5,7
/proc/21354/task/21375/stat : 21375 (plingeling) R 21352 21354 14218 0 -1 4202560 518706 1252 0 0 118756 100 0 0 25 0 5 0 733529795 1917751296 436683 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4800.18
Current children cumulated vsize (KiB) 1872804

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.9 s]
/proc/loadavg: 8.11 8.03 7.65 9/392 21411
/proc/meminfo: memFree=10623264/32951124 swapFree=65586204/67111528
[pid=21354] ppid=21352 vsize=1872804 CPUtime=4800.18 cores=1,3,5,7
/proc/21354/stat : 21354 (plingeling) S 21352 21354 14218 0 -1 4202496 2808388 1252 0 0 479597 421 0 0 24 0 5 0 733529785 1917751296 436683 33554432000 4194304 4417452 140735749956096 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/21354/statm: 468201 436683 180 55 0 466087 0
[pid=21354/tid=21372] ppid=21352 vsize=1872804 CPUtime=1203.83 cores=1,3,5,7
/proc/21354/task/21372/stat : 21372 (plingeling) R 21352 21354 14218 0 -1 4202560 828299 1252 0 0 120269 114 0 0 25 0 5 0 733529795 1917751296 436683 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=21354/tid=21373] ppid=21352 vsize=1872804 CPUtime=1203.83 cores=1,3,5,7
/proc/21354/task/21373/stat : 21373 (plingeling) R 21352 21354 14218 0 -1 4202560 511862 1252 0 0 120305 78 0 0 25 0 5 0 733529795 1917751296 436683 33554432000 4194304 4417452 140735749956096 18446744073709551615 4251133 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=21354/tid=21374] ppid=21352 vsize=1872804 CPUtime=1203.83 cores=1,3,5,7
/proc/21354/task/21374/stat : 21374 (plingeling) R 21352 21354 14218 0 -1 4202560 944895 1252 0 0 120256 127 0 0 25 0 5 0 733529795 1917751296 436683 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21354/tid=21375] ppid=21352 vsize=1872804 CPUtime=1188.56 cores=1,3,5,7
/proc/21354/task/21375/stat : 21375 (plingeling) R 21352 21354 14218 0 -1 4202560 518706 1252 0 0 118756 100 0 0 25 0 5 0 733529795 1917751296 436683 33554432000 4194304 4417452 140735749956096 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4800.18
Current children cumulated vsize (KiB) 1872804

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1203.99
CPU time (s): 4800.28
CPU user time (s): 4796
CPU system time (s): 4.28935
CPU usage (%): 398.697
Max. virtual memory (cumulated for all children) (KiB): 2023204

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4796
system time used= 4.28935
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2809640
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 136
involuntary context switches= 32606

runsolver used 4.41533 second user time and 10.8663 second system time

The end

Launcher Data 

Begin job on node119 at 2011-05-15 06:16:31
IDJOB=3370896
IDBENCH=88212
IDSOLVER=1848
FILE ID=node119/3370896-1305432991
RUNJOBID= node119-1305432991-21341
PBS_JOBID= 13324599
Free space on /tmp= 73328 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m37-q505.75-n35-p15-s48276711.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3370896-1305432991/watcher-3370896-1305432991 -o /tmp/evaluation-result-3370896-1305432991/solver-3370896-1305432991 -C 4800 -W 1300 -M 15500  plingeling -t 4 HOME/instance-3370896-1305432991.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 1bac7b96ea6eb843b1de7405a721cd87
RANDOM SEED=553855767

node119.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.63
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      14808884 kB
Buffers:        290584 kB
Cached:        4638828 kB
SwapCached:       4756 kB
Active:       14695580 kB
Inactive:      3117564 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      14808884 kB
SwapTotal:    67111528 kB
SwapFree:     65586204 kB
Dirty:            4752 kB
Writeback:          16 kB
AnonPages:    12882408 kB
Mapped:          29528 kB
Slab:           208328 kB
PageTables:      56416 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 102465000 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73328 MiB
End job on node119 at 2011-05-15 06:36:37