Trace number 3250856

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587? (TO) 4800.37 1202.2

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m62-q1327.5-n60-p30-s52708253.cnf
MD5SUM472b10ce0a2683d9355981037eea091e
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark7.17391
Satisfiable
(Un)Satisfiability was proved
Number of variables3672
Number of clauses1163952
Sum of the clauses size2331456
Maximum clause length62
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 21163892
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 560

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3250856-1304111824/watcher-3250856-1304111824 -o /tmp/evaluation-result-3250856-1304111824/solver-3250856-1304111824 -C 4800 -W 1300 -M 15500 plingeling -t 4 HOME/instance-3250856-1304111824.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.18 8.19 7.59 6/179 10582
/proc/meminfo: memFree=29744576/32951132 swapFree=67080208/67111528
[pid=10582] ppid=10580 vsize=9144 CPUtime=0 cores=1,3,5,7
/proc/10582/stat : 10582 (plingeling) R 10580 10582 9622 0 -1 4202496 347 1250 0 0 0 0 0 0 25 0 1 0 522782062 9363456 276 33554432000 4194304 4417332 140733432864656 18446744073709551615 210646840286 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/10582/statm: 2286 276 146 55 0 172 0

[startup+0.0327941 s]
/proc/loadavg: 8.18 8.19 7.59 6/179 10582
/proc/meminfo: memFree=29744576/32951132 swapFree=67080208/67111528
[pid=10582] ppid=10580 vsize=12284 CPUtime=0.02 cores=1,3,5,7
/proc/10582/stat : 10582 (plingeling) R 10580 10582 9622 0 -1 4202496 1197 1250 0 0 2 0 0 0 25 0 1 0 522782062 12578816 1094 33554432000 4194304 4417332 140733432864656 18446744073709551615 210647146240 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/10582/statm: 3071 1094 147 55 0 957 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 12284

[startup+0.100764 s]
/proc/loadavg: 8.18 8.19 7.59 6/179 10582
/proc/meminfo: memFree=29744576/32951132 swapFree=67080208/67111528
[pid=10582] ppid=10580 vsize=18428 CPUtime=0.09 cores=1,3,5,7
/proc/10582/stat : 10582 (plingeling) R 10580 10582 9622 0 -1 4202496 2733 1250 0 0 9 0 0 0 25 0 1 0 522782062 18870272 2630 33554432000 4194304 4417332 140733432864656 18446744073709551615 210647146240 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/10582/statm: 4607 2630 147 55 0 2493 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 18428

[startup+0.300731 s]
/proc/loadavg: 8.18 8.19 7.59 6/179 10582
/proc/meminfo: memFree=29744576/32951132 swapFree=67080208/67111528
[pid=10582] ppid=10580 vsize=43004 CPUtime=0.29 cores=1,3,5,7
/proc/10582/stat : 10582 (plingeling) R 10580 10582 9622 0 -1 4202496 8877 1250 0 0 28 1 0 0 25 0 1 0 522782062 44036096 8774 33554432000 4194304 4417332 140733432864656 18446744073709551615 210647146240 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/10582/statm: 10751 8774 147 55 0 8637 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 43004

[startup+0.700675 s]
/proc/loadavg: 8.18 8.19 7.59 6/179 10582
/proc/meminfo: memFree=29744576/32951132 swapFree=67080208/67111528
[pid=10582] ppid=10580 vsize=141308 CPUtime=0.68 cores=1,3,5,7
/proc/10582/stat : 10582 (plingeling) R 10580 10582 9622 0 -1 4202496 33453 1250 0 0 64 4 0 0 25 0 1 0 522782062 144699392 33350 33554432000 4194304 4417332 140733432864656 18446744073709551615 4322344 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/10582/statm: 35327 33350 147 55 0 33213 0
Current children cumulated CPU time (s) 0.68
Current children cumulated vsize (KiB) 141308

[startup+1.50049 s]
/proc/loadavg: 8.18 8.19 7.59 9/184 10592
/proc/meminfo: memFree=29600448/32951132 swapFree=67080208/67111528
[pid=10582] ppid=10580 vsize=194504 CPUtime=3.69 cores=1,3,5,7
/proc/10582/stat : 10582 (plingeling) S 10580 10582 9622 0 -1 4202496 38995 1250 0 0 364 5 0 0 25 0 5 0 522782062 199172096 36409 33554432000 4194304 4417332 140733432864656 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/10582/statm: 48626 36409 175 55 0 46512 0
[pid=10582/tid=10589] ppid=10580 vsize=194504 CPUtime=0.73 cores=1,3,5,7
/proc/10582/task/10589/stat : 10589 (plingeling) R 10580 10582 9622 0 -1 4202560 35 1250 0 0 73 0 0 0 25 0 5 0 522782138 199172096 36409 33554432000 4194304 4417332 140733432864656 18446744073709551615 4264256 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=10582/tid=10590] ppid=10580 vsize=194504 CPUtime=0.73 cores=1,3,5,7
/proc/10582/task/10590/stat : 10590 (plingeling) R 10580 10582 9622 0 -1 4202560 159 1250 0 0 73 0 0 0 25 0 5 0 522782138 199172096 36409 33554432000 4194304 4417332 140733432864656 18446744073709551615 4236722 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=10582/tid=10591] ppid=10580 vsize=194504 CPUtime=0.73 cores=1,3,5,7
/proc/10582/task/10591/stat : 10591 (plingeling) R 10580 10582 9622 0 -1 4202560 204 1250 0 0 73 0 0 0 25 0 5 0 522782138 199172096 36409 33554432000 4194304 4417332 140733432864656 18446744073709551615 4326516 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=10582/tid=10592] ppid=10580 vsize=194504 CPUtime=0.72 cores=1,3,5,7
/proc/10582/task/10592/stat : 10592 (plingeling) R 10580 10582 9622 0 -1 4202560 5132 1250 0 0 72 0 0 0 25 0 5 0 522782138 199172096 36409 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364531 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3.69
Current children cumulated vsize (KiB) 194504

[startup+3.10122 s]
/proc/loadavg: 8.17 8.19 7.60 9/184 10592
/proc/meminfo: memFree=29581540/32951132 swapFree=67080208/67111528
[pid=10582] ppid=10580 vsize=226324 CPUtime=10.08 cores=1,3,5,7
/proc/10582/stat : 10582 (plingeling) S 10580 10582 9622 0 -1 4202496 79041 1250 0 0 998 10 0 0 25 0 5 0 522782062 231755776 44369 33554432000 4194304 4417332 140733432864656 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/10582/statm: 56581 44369 180 55 0 54467 0
[pid=10582/tid=10589] ppid=10580 vsize=226324 CPUtime=2.33 cores=1,3,5,7
/proc/10582/task/10589/stat : 10589 (plingeling) R 10580 10582 9622 0 -1 4202560 9254 1250 0 0 232 1 0 0 25 0 5 0 522782138 231755776 44369 33554432000 4194304 4417332 140733432864656 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=10582/tid=10590] ppid=10580 vsize=226324 CPUtime=2.33 cores=1,3,5,7
/proc/10582/task/10590/stat : 10590 (plingeling) R 10580 10582 9622 0 -1 4202560 9038 1250 0 0 232 1 0 0 25 0 5 0 522782138 231755776 44369 33554432000 4194304 4417332 140733432864656 18446744073709551615 4249991 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=10582/tid=10591] ppid=10580 vsize=226324 CPUtime=2.33 cores=1,3,5,7
/proc/10582/task/10591/stat : 10591 (plingeling) R 10580 10582 9622 0 -1 4202560 9546 1250 0 0 232 1 0 0 25 0 5 0 522782138 231755776 44369 33554432000 4194304 4417332 140733432864656 18446744073709551615 4249186 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=10582/tid=10592] ppid=10580 vsize=226324 CPUtime=2.31 cores=1,3,5,7
/proc/10582/task/10592/stat : 10592 (plingeling) R 10580 10582 9622 0 -1 4202560 17738 1250 0 0 229 2 0 0 25 0 5 0 522782138 231755776 44369 33554432000 4194304 4417332 140733432864656 18446744073709551615 4365977 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 10.08
Current children cumulated vsize (KiB) 226324

[startup+6.30069 s]
/proc/loadavg: 8.17 8.19 7.60 9/184 10592
/proc/meminfo: memFree=29494624/32951132 swapFree=67080208/67111528
[pid=10582] ppid=10580 vsize=299132 CPUtime=22.86 cores=1,3,5,7
/proc/10582/stat : 10582 (plingeling) S 10580 10582 9622 0 -1 4202496 107093 1250 0 0 2270 16 0 0 25 0 5 0 522782062 306311168 62571 33554432000 4194304 4417332 140733432864656 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/10582/statm: 74783 62571 180 55 0 72669 0
[pid=10582/tid=10589] ppid=10580 vsize=299132 CPUtime=5.53 cores=1,3,5,7
/proc/10582/task/10589/stat : 10589 (plingeling) R 10580 10582 9622 0 -1 4202560 10169 1250 0 0 552 1 0 0 25 0 5 0 522782138 306311168 62571 33554432000 4194304 4417332 140733432864656 18446744073709551615 4258401 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=10582/tid=10590] ppid=10580 vsize=299132 CPUtime=5.52 cores=1,3,5,7
/proc/10582/task/10590/stat : 10590 (plingeling) R 10580 10582 9622 0 -1 4202560 9971 1250 0 0 551 1 0 0 25 0 5 0 522782138 306311168 62571 33554432000 4194304 4417332 140733432864656 18446744073709551615 4258389 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=10582/tid=10591] ppid=10580 vsize=299132 CPUtime=5.52 cores=1,3,5,7
/proc/10582/task/10591/stat : 10591 (plingeling) R 10580 10582 9622 0 -1 4202560 11412 1250 0 0 551 1 0 0 25 0 5 0 522782138 306311168 62571 33554432000 4194304 4417332 140733432864656 18446744073709551615 4258561 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=10582/tid=10592] ppid=10580 vsize=299132 CPUtime=5.5 cores=1,3,5,7
/proc/10582/task/10592/stat : 10592 (plingeling) R 10580 10582 9622 0 -1 4202560 42076 1250 0 0 543 7 0 0 25 0 5 0 522782138 306311168 62571 33554432000 4194304 4417332 140733432864656 18446744073709551615 4356907 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 22.86
Current children cumulated vsize (KiB) 299132

[startup+12.7006 s]
/proc/loadavg: 8.39 8.23 7.61 9/184 10592
/proc/meminfo: memFree=29509164/32951132 swapFree=67080208/67111528
[pid=10582] ppid=10580 vsize=306196 CPUtime=48.43 cores=1,3,5,7
/proc/10582/stat : 10582 (plingeling) S 10580 10582 9622 0 -1 4202496 144529 1250 0 0 4822 21 0 0 25 0 5 0 522782062 313544704 64337 33554432000 4194304 4417332 140733432864656 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/10582/statm: 76549 64337 180 55 0 74435 0
[pid=10582/tid=10589] ppid=10580 vsize=306196 CPUtime=11.93 cores=1,3,5,7
/proc/10582/task/10589/stat : 10589 (plingeling) R 10580 10582 9622 0 -1 4202560 12035 1250 0 0 1192 1 0 0 25 0 5 0 522782138 313544704 64337 33554432000 4194304 4417332 140733432864656 18446744073709551615 4249110 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=10582/tid=10590] ppid=10580 vsize=306196 CPUtime=11.92 cores=1,3,5,7
/proc/10582/task/10590/stat : 10590 (plingeling) R 10580 10582 9622 0 -1 4202560 11837 1250 0 0 1191 1 0 0 25 0 5 0 522782138 313544704 64337 33554432000 4194304 4417332 140733432864656 18446744073709551615 4258632 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=10582/tid=10591] ppid=10580 vsize=306196 CPUtime=11.92 cores=1,3,5,7
/proc/10582/task/10591/stat : 10591 (plingeling) R 10580 10582 9622 0 -1 4202560 11412 1250 0 0 1191 1 0 0 25 0 5 0 522782138 313544704 64337 33554432000 4194304 4417332 140733432864656 18446744073709551615 4250003 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=10582/tid=10592] ppid=10580 vsize=306196 CPUtime=11.87 cores=1,3,5,7
/proc/10582/task/10592/stat : 10592 (plingeling) R 10580 10582 9622 0 -1 4202560 75780 1250 0 0 1176 11 0 0 25 0 5 0 522782138 313544704 64337 33554432000 4194304 4417332 140733432864656 18446744073709551615 4357984 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 48.43
Current children cumulated vsize (KiB) 306196

[startup+25.5005 s]
/proc/loadavg: 8.30 8.22 7.62 9/184 10592
/proc/meminfo: memFree=29314652/32951132 swapFree=67080208/67111528
[pid=10582] ppid=10580 vsize=506072 CPUtime=99.54 cores=1,3,5,7
/proc/10582/stat : 10582 (plingeling) S 10580 10582 9622 0 -1 4202496 194455 1250 0 0 9925 29 0 0 25 0 5 0 522782062 518217728 97955 33554432000 4194304 4417332 140733432864656 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/10582/statm: 126518 97955 180 55 0 124404 0
[pid=10582/tid=10589] ppid=10580 vsize=506072 CPUtime=24.72 cores=1,3,5,7
/proc/10582/task/10589/stat : 10589 (plingeling) R 10580 10582 9622 0 -1 4202560 13910 1250 0 0 2471 1 0 0 25 0 5 0 522782138 518217728 97955 33554432000 4194304 4417332 140733432864656 18446744073709551615 4259932 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=10582/tid=10590] ppid=10580 vsize=506072 CPUtime=24.71 cores=1,3,5,7
/proc/10582/task/10590/stat : 10590 (plingeling) R 10580 10582 9622 0 -1 4202560 15881 1250 0 0 2466 5 0 0 25 0 5 0 522782138 518217728 97955 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=10582/tid=10591] ppid=10580 vsize=506072 CPUtime=24.73 cores=1,3,5,7
/proc/10582/task/10591/stat : 10591 (plingeling) R 10580 10582 9622 0 -1 4202560 15508 1250 0 0 2471 2 0 0 25 0 5 0 522782138 518217728 97955 33554432000 4194304 4417332 140733432864656 18446744073709551615 4248816 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=10582/tid=10592] ppid=10580 vsize=506072 CPUtime=24.6 cores=1,3,5,7
/proc/10582/task/10592/stat : 10592 (plingeling) R 10580 10582 9622 0 -1 4202560 115691 1250 0 0 2445 15 0 0 25 0 5 0 522782138 518217728 97955 33554432000 4194304 4417332 140733432864656 18446744073709551615 4321565 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 99.54
Current children cumulated vsize (KiB) 506072

[startup+51.1012 s]
/proc/loadavg: 8.20 8.20 7.63 9/184 10593
/proc/meminfo: memFree=29343888/32951132 swapFree=67080208/67111528
[pid=10582] ppid=10580 vsize=485852 CPUtime=201.76 cores=1,3,5,7
/proc/10582/stat : 10582 (plingeling) S 10580 10582 9622 0 -1 4202496 287694 1250 0 0 20131 45 0 0 25 0 5 0 522782062 497512448 92932 33554432000 4194304 4417332 140733432864656 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/10582/statm: 121463 92932 180 55 0 119349 0
[pid=10582/tid=10589] ppid=10580 vsize=485852 CPUtime=50.32 cores=1,3,5,7
/proc/10582/task/10589/stat : 10589 (plingeling) R 10580 10582 9622 0 -1 4202560 17974 1250 0 0 5030 2 0 0 25 0 5 0 522782138 497512448 92932 33554432000 4194304 4417332 140733432864656 18446744073709551615 4310188 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=10582/tid=10590] ppid=10580 vsize=485852 CPUtime=50.28 cores=1,3,5,7
/proc/10582/task/10590/stat : 10590 (plingeling) R 10580 10582 9622 0 -1 4202560 24073 1250 0 0 5020 8 0 0 25 0 5 0 522782138 497512448 92932 33554432000 4194304 4417332 140733432864656 18446744073709551615 4239395 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=10582/tid=10591] ppid=10580 vsize=485852 CPUtime=50.32 cores=1,3,5,7
/proc/10582/task/10591/stat : 10591 (plingeling) R 10580 10582 9622 0 -1 4202560 17706 1250 0 0 5030 2 0 0 25 0 5 0 522782138 497512448 92932 33554432000 4194304 4417332 140733432864656 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=10582/tid=10592] ppid=10580 vsize=485852 CPUtime=50.05 cores=1,3,5,7
/proc/10582/task/10592/stat : 10592 (plingeling) R 10580 10582 9622 0 -1 4202560 194476 1250 0 0 4979 26 0 0 25 0 5 0 522782138 497512448 92932 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364713 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 201.76
Current children cumulated vsize (KiB) 485852

[startup+102.307 s]
/proc/loadavg: 8.08 8.17 7.65 9/184 10594
/proc/meminfo: memFree=28982244/32951132 swapFree=67080208/67111528
[pid=10582] ppid=10580 vsize=662368 CPUtime=406.29 cores=1,3,5,7
/proc/10582/stat : 10582 (plingeling) S 10580 10582 9622 0 -1 4202496 415981 1250 0 0 40566 63 0 0 25 0 5 0 522782062 678264832 136591 33554432000 4194304 4417332 140733432864656 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/10582/statm: 165592 136591 180 55 0 163478 0
[pid=10582/tid=10589] ppid=10580 vsize=662368 CPUtime=101.52 cores=1,3,5,7
/proc/10582/task/10589/stat : 10589 (plingeling) R 10580 10582 9622 0 -1 4202560 17974 1250 0 0 10150 2 0 0 25 0 5 0 522782138 678264832 136591 33554432000 4194304 4417332 140733432864656 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=10582/tid=10590] ppid=10580 vsize=662368 CPUtime=101.47 cores=1,3,5,7
/proc/10582/task/10590/stat : 10590 (plingeling) R 10580 10582 9622 0 -1 4202560 28170 1250 0 0 10138 9 0 0 25 0 5 0 522782138 678264832 136591 33554432000 4194304 4417332 140733432864656 18446744073709551615 4250080 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=10582/tid=10591] ppid=10580 vsize=662368 CPUtime=101.53 cores=1,3,5,7
/proc/10582/task/10591/stat : 10591 (plingeling) R 10580 10582 9622 0 -1 4202560 18746 1250 0 0 10151 2 0 0 25 0 5 0 522782138 678264832 136591 33554432000 4194304 4417332 140733432864656 18446744073709551615 4249209 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=10582/tid=10592] ppid=10580 vsize=662368 CPUtime=100.99 cores=1,3,5,7
/proc/10582/task/10592/stat : 10592 (plingeling) R 10580 10582 9622 0 -1 4202560 317626 1250 0 0 10055 44 0 0 25 0 5 0 522782138 678264832 136591 33554432000 4194304 4417332 140733432864656 18446744073709551615 4357865 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 406.29
Current children cumulated vsize (KiB) 662368

[startup+162.3 s]
/proc/loadavg: 8.12 8.16 7.67 9/184 10596
/proc/meminfo: memFree=28888424/32951132 swapFree=67080208/67111528
[pid=10582] ppid=10580 vsize=624836 CPUtime=645.93 cores=1,3,5,7
/proc/10582/stat : 10582 (plingeling) S 10580 10582 9622 0 -1 4202496 516313 1250 0 0 64516 77 0 0 25 0 5 0 522782062 639832064 127646 33554432000 4194304 4417332 140733432864656 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/10582/statm: 156209 127646 180 55 0 154095 0
[pid=10582/tid=10589] ppid=10580 vsize=624836 CPUtime=161.51 cores=1,3,5,7
/proc/10582/task/10589/stat : 10589 (plingeling) R 10580 10582 9622 0 -1 4202560 17974 1250 0 0 16149 2 0 0 25 0 5 0 522782138 639832064 127646 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364544 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=10582/tid=10590] ppid=10580 vsize=624836 CPUtime=161.45 cores=1,3,5,7
/proc/10582/task/10590/stat : 10590 (plingeling) R 10580 10582 9622 0 -1 4202560 30819 1250 0 0 16136 9 0 0 25 0 5 0 522782138 639832064 127646 33554432000 4194304 4417332 140733432864656 18446744073709551615 4235991 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=10582/tid=10591] ppid=10580 vsize=624836 CPUtime=161.34 cores=1,3,5,7
/proc/10582/task/10591/stat : 10591 (plingeling) R 10580 10582 9622 0 -1 4202560 31581 1250 0 0 16131 3 0 0 25 0 5 0 522782138 639832064 127646 33554432000 4194304 4417332 140733432864656 18446744073709551615 4249130 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=10582/tid=10592] ppid=10580 vsize=624836 CPUtime=160.85 cores=1,3,5,7
/proc/10582/task/10592/stat : 10592 (plingeling) R 10580 10582 9622 0 -1 4202560 402474 1250 0 0 16028 57 0 0 25 0 5 0 522782138 639832064 127646 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364556 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 645.93
Current children cumulated vsize (KiB) 624836


################
# More data... #
################

[pid=10582/tid=10590] ppid=10580 vsize=1293904 CPUtime=761.43 cores=1,3,5,7
/proc/10582/task/10590/stat : 10590 (plingeling) R 10580 10582 9622 0 -1 4202560 34991 1250 0 0 76134 9 0 0 25 0 5 0 522782138 1324957696 294913 33554432000 4194304 4417332 140733432864656 18446744073709551615 4366582 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=10582/tid=10591] ppid=10580 vsize=1293904 CPUtime=758.49 cores=1,3,5,7
/proc/10582/task/10591/stat : 10591 (plingeling) R 10580 10582 9622 0 -1 4202560 81391 1250 0 0 75839 10 0 0 25 0 5 0 522782138 1324957696 294913 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364765 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=10582/tid=10592] ppid=10580 vsize=1293904 CPUtime=760.86 cores=1,3,5,7
/proc/10582/task/10592/stat : 10592 (plingeling) R 10580 10582 9622 0 -1 4202560 902184 1250 0 0 75947 139 0 0 25 0 5 0 522782138 1324957696 294913 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3043.06
Current children cumulated vsize (KiB) 1293904

[startup+822.301 s]
/proc/loadavg: 4.03 4.74 6.11 5/174 10631
/proc/meminfo: memFree=29237332/32951132 swapFree=67080208/67111528
[pid=10582] ppid=10580 vsize=1574292 CPUtime=3282.76 cores=1,3,5,7
/proc/10582/stat : 10582 (plingeling) S 10580 10582 9622 0 -1 4202496 1162030 1250 0 0 328096 180 0 0 25 0 5 0 522782062 1612075008 365010 33554432000 4194304 4417332 140733432864656 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/10582/statm: 393573 365010 180 55 0 391459 0
[pid=10582/tid=10589] ppid=10580 vsize=1574292 CPUtime=821.5 cores=1,3,5,7
/proc/10582/task/10589/stat : 10589 (plingeling) R 10580 10582 9622 0 -1 4202560 25386 1250 0 0 82147 3 0 0 25 0 5 0 522782138 1612075008 365010 33554432000 4194304 4417332 140733432864656 18446744073709551615 4365108 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=10582/tid=10590] ppid=10580 vsize=1574292 CPUtime=821.42 cores=1,3,5,7
/proc/10582/task/10590/stat : 10590 (plingeling) R 10580 10582 9622 0 -1 4202560 40562 1250 0 0 82133 9 0 0 25 0 5 0 522782138 1612075008 365010 33554432000 4194304 4417332 140733432864656 18446744073709551615 4321685 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=10582/tid=10591] ppid=10580 vsize=1574292 CPUtime=818.21 cores=1,3,5,7
/proc/10582/task/10591/stat : 10591 (plingeling) R 10580 10582 9622 0 -1 4202560 96698 1250 0 0 81809 12 0 0 25 0 5 0 522782138 1612075008 365010 33554432000 4194304 4417332 140733432864656 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=10582/tid=10592] ppid=10580 vsize=1574292 CPUtime=820.85 cores=1,3,5,7
/proc/10582/task/10592/stat : 10592 (plingeling) R 10580 10582 9622 0 -1 4202560 965919 1250 0 0 81935 150 0 0 25 0 5 0 522782138 1612075008 365010 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364465 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3282.76
Current children cumulated vsize (KiB) 1574292

[startup+882.3 s]
/proc/loadavg: 4.01 4.59 5.97 5/174 10633
/proc/meminfo: memFree=29119140/32951132 swapFree=67080208/67111528
[pid=10582] ppid=10580 vsize=1691700 CPUtime=3522.49 cores=1,3,5,7
/proc/10582/stat : 10582 (plingeling) S 10580 10582 9622 0 -1 4202496 1206546 1250 0 0 352062 187 0 0 25 0 5 0 522782062 1732300800 394362 33554432000 4194304 4417332 140733432864656 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/10582/statm: 422925 394362 180 55 0 420811 0
[pid=10582/tid=10589] ppid=10580 vsize=1691700 CPUtime=881.5 cores=1,3,5,7
/proc/10582/task/10589/stat : 10589 (plingeling) R 10580 10582 9622 0 -1 4202560 25386 1250 0 0 88147 3 0 0 25 0 5 0 522782138 1732300800 394362 33554432000 4194304 4417332 140733432864656 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=10582/tid=10590] ppid=10580 vsize=1691700 CPUtime=881.43 cores=1,3,5,7
/proc/10582/task/10590/stat : 10590 (plingeling) R 10580 10582 9622 0 -1 4202560 61152 1250 0 0 88131 12 0 0 25 0 5 0 522782138 1732300800 394362 33554432000 4194304 4417332 140733432864656 18446744073709551615 4250977 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=10582/tid=10591] ppid=10580 vsize=1691700 CPUtime=877.93 cores=1,3,5,7
/proc/10582/task/10591/stat : 10591 (plingeling) R 10580 10582 9622 0 -1 4202560 96698 1250 0 0 87781 12 0 0 25 0 5 0 522782138 1732300800 394362 33554432000 4194304 4417332 140733432864656 18446744073709551615 4250017 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=10582/tid=10592] ppid=10580 vsize=1691700 CPUtime=880.85 cores=1,3,5,7
/proc/10582/task/10592/stat : 10592 (plingeling) R 10580 10582 9622 0 -1 4202560 989845 1250 0 0 87931 154 0 0 25 0 5 0 522782138 1732300800 394362 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3522.49
Current children cumulated vsize (KiB) 1691700

[startup+942.3 s]
/proc/loadavg: 4.00 4.49 5.85 5/174 10634
/proc/meminfo: memFree=28875212/32951132 swapFree=67080208/67111528
[pid=10582] ppid=10580 vsize=1948432 CPUtime=3762.21 cores=1,3,5,7
/proc/10582/stat : 10582 (plingeling) S 10580 10582 9622 0 -1 4202496 1275118 1250 0 0 376028 193 0 0 25 0 5 0 522782062 1995194368 458545 33554432000 4194304 4417332 140733432864656 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/10582/statm: 487108 458545 180 55 0 484994 0
[pid=10582/tid=10589] ppid=10580 vsize=1948432 CPUtime=941.49 cores=1,3,5,7
/proc/10582/task/10589/stat : 10589 (plingeling) R 10580 10582 9622 0 -1 4202560 43680 1250 0 0 94145 4 0 0 25 0 5 0 522782138 1995194368 458545 33554432000 4194304 4417332 140733432864656 18446744073709551615 4365975 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=10582/tid=10590] ppid=10580 vsize=1948432 CPUtime=941.42 cores=1,3,5,7
/proc/10582/task/10590/stat : 10590 (plingeling) R 10580 10582 9622 0 -1 4202560 68483 1250 0 0 94130 12 0 0 25 0 5 0 522782138 1995194368 458545 33554432000 4194304 4417332 140733432864656 18446744073709551615 4366384 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=10582/tid=10591] ppid=10580 vsize=1948432 CPUtime=937.66 cores=1,3,5,7
/proc/10582/task/10591/stat : 10591 (plingeling) R 10580 10582 9622 0 -1 4202560 106703 1250 0 0 93752 14 0 0 25 0 5 0 522782138 1995194368 458545 33554432000 4194304 4417332 140733432864656 18446744073709551615 4366384 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=10582/tid=10592] ppid=10580 vsize=1948432 CPUtime=940.85 cores=1,3,5,7
/proc/10582/task/10592/stat : 10592 (plingeling) R 10580 10582 9622 0 -1 4202560 1022787 1250 0 0 93928 157 0 0 25 0 5 0 522782138 1995194368 458545 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3762.21
Current children cumulated vsize (KiB) 1948432

[startup+1002.3 s]
/proc/loadavg: 4.00 4.39 5.72 5/174 10635
/proc/meminfo: memFree=28715588/32951132 swapFree=67080208/67111528
[pid=10582] ppid=10580 vsize=2095240 CPUtime=4001.94 cores=1,3,5,7
/proc/10582/stat : 10582 (plingeling) S 10580 10582 9622 0 -1 4202496 1320115 1250 0 0 399995 199 0 0 25 0 5 0 522782062 2145525760 495247 33554432000 4194304 4417332 140733432864656 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/10582/statm: 523810 495247 180 55 0 521696 0
[pid=10582/tid=10589] ppid=10580 vsize=2095240 CPUtime=1001.49 cores=1,3,5,7
/proc/10582/task/10589/stat : 10589 (plingeling) R 10580 10582 9622 0 -1 4202560 63647 1250 0 0 100143 6 0 0 25 0 5 0 522782138 2145525760 495247 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364549 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=10582/tid=10590] ppid=10580 vsize=2095240 CPUtime=1001.43 cores=1,3,5,7
/proc/10582/task/10590/stat : 10590 (plingeling) R 10580 10582 9622 0 -1 4202560 82026 1250 0 0 100129 14 0 0 25 0 5 0 522782138 2145525760 495247 33554432000 4194304 4417332 140733432864656 18446744073709551615 4356771 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=10582/tid=10591] ppid=10580 vsize=2095240 CPUtime=997.38 cores=1,3,5,7
/proc/10582/task/10591/stat : 10591 (plingeling) R 10580 10582 9622 0 -1 4202560 115012 1250 0 0 99723 15 0 0 25 0 5 0 522782138 2145525760 495247 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364556 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=10582/tid=10592] ppid=10580 vsize=2095240 CPUtime=1000.86 cores=1,3,5,7
/proc/10582/task/10592/stat : 10592 (plingeling) R 10580 10582 9622 0 -1 4202560 1025965 1250 0 0 99928 158 0 0 25 0 5 0 522782138 2145525760 495247 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4001.94
Current children cumulated vsize (KiB) 2095240

[startup+1062.3 s]
/proc/loadavg: 4.00 4.32 5.62 5/174 10637
/proc/meminfo: memFree=28610656/32951132 swapFree=67080208/67111528
[pid=10582] ppid=10580 vsize=2253044 CPUtime=4241.66 cores=1,3,5,7
/proc/10582/stat : 10582 (plingeling) S 10580 10582 9622 0 -1 4202496 1382259 1250 0 0 423956 210 0 0 25 0 5 0 522782062 2307117056 531039 33554432000 4194304 4417332 140733432864656 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/10582/statm: 563261 531062 180 55 0 561147 0
[pid=10582/tid=10589] ppid=10580 vsize=2253044 CPUtime=1061.49 cores=1,3,5,7
/proc/10582/task/10589/stat : 10589 (plingeling) R 10580 10582 9622 0 -1 4202560 70902 1250 0 0 106141 8 0 0 25 0 5 0 522782138 2307117056 531074 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364631 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=10582/tid=10590] ppid=10580 vsize=2253044 CPUtime=1061.43 cores=1,3,5,7
/proc/10582/task/10590/stat : 10590 (plingeling) R 10580 10582 9622 0 -1 4202560 91835 1250 0 0 106128 15 0 0 25 0 5 0 522782138 2307117056 531086 33554432000 4194304 4417332 140733432864656 18446744073709551615 210646843779 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=10582/tid=10591] ppid=10580 vsize=2253044 CPUtime=1057.1 cores=1,3,5,7
/proc/10582/task/10591/stat : 10591 (plingeling) R 10580 10582 9622 0 -1 4202560 140047 1250 0 0 105691 19 0 0 25 0 5 0 522782138 2307117056 531099 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364724 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=10582/tid=10592] ppid=10580 vsize=2253044 CPUtime=1060.86 cores=1,3,5,7
/proc/10582/task/10592/stat : 10592 (plingeling) R 10580 10582 9622 0 -1 4202560 1046058 1250 0 0 105924 162 0 0 25 0 5 0 522782138 2307117056 531111 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4241.66
Current children cumulated vsize (KiB) 2253044

[startup+1122.3 s]
/proc/loadavg: 4.04 4.27 5.51 5/174 10639
/proc/meminfo: memFree=28277316/32951132 swapFree=67080208/67111528
[pid=10582] ppid=10580 vsize=2532760 CPUtime=4481.39 cores=1,3,5,7
/proc/10582/stat : 10582 (plingeling) S 10580 10582 9622 0 -1 4202496 1478679 1250 0 0 447912 227 0 0 25 0 5 0 522782062 2593546240 604627 33554432000 4194304 4417332 140733432864656 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/10582/statm: 633190 604627 180 55 0 631076 0
[pid=10582/tid=10589] ppid=10580 vsize=2532760 CPUtime=1121.49 cores=1,3,5,7
/proc/10582/task/10589/stat : 10589 (plingeling) R 10580 10582 9622 0 -1 4202560 80995 1250 0 0 112139 10 0 0 25 0 5 0 522782138 2593546240 604627 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=10582/tid=10590] ppid=10580 vsize=2532760 CPUtime=1121.42 cores=1,3,5,7
/proc/10582/task/10590/stat : 10590 (plingeling) R 10580 10582 9622 0 -1 4202560 102266 1250 0 0 112126 16 0 0 25 0 5 0 522782138 2593546240 604627 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364887 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=10582/tid=10591] ppid=10580 vsize=2532760 CPUtime=1116.83 cores=1,3,5,7
/proc/10582/task/10591/stat : 10591 (plingeling) R 10580 10582 9622 0 -1 4202560 169559 1250 0 0 111660 23 0 0 25 0 5 0 522782138 2593546240 604627 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=10582/tid=10592] ppid=10580 vsize=2532760 CPUtime=1120.85 cores=1,3,5,7
/proc/10582/task/10592/stat : 10592 (plingeling) R 10580 10582 9622 0 -1 4202560 1092394 1250 0 0 111914 171 0 0 25 0 5 0 522782138 2593546240 604627 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364494 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4481.39
Current children cumulated vsize (KiB) 2532760

[startup+1182.3 s]
/proc/loadavg: 4.01 4.22 5.42 5/174 10641
/proc/meminfo: memFree=28178720/32951132 swapFree=67080208/67111528
[pid=10582] ppid=10580 vsize=2630468 CPUtime=4721.12 cores=1,3,5,7
/proc/10582/stat : 10582 (plingeling) S 10580 10582 9622 0 -1 4202496 1521416 1250 0 0 471879 233 0 0 25 0 5 0 522782062 2693599232 629054 33554432000 4194304 4417332 140733432864656 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/10582/statm: 657617 629054 180 55 0 655503 0
[pid=10582/tid=10589] ppid=10580 vsize=2630468 CPUtime=1181.49 cores=1,3,5,7
/proc/10582/task/10589/stat : 10589 (plingeling) R 10580 10582 9622 0 -1 4202560 85593 1250 0 0 118139 10 0 0 25 0 5 0 522782138 2693599232 629054 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=10582/tid=10590] ppid=10580 vsize=2630468 CPUtime=1181.43 cores=1,3,5,7
/proc/10582/task/10590/stat : 10590 (plingeling) R 10580 10582 9622 0 -1 4202560 140405 1250 0 0 118121 22 0 0 25 0 5 0 522782138 2693599232 629054 33554432000 4194304 4417332 140733432864656 18446744073709551615 4343668 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=10582/tid=10591] ppid=10580 vsize=2630468 CPUtime=1176.55 cores=1,3,5,7
/proc/10582/task/10591/stat : 10591 (plingeling) R 10580 10582 9622 0 -1 4202560 169559 1250 0 0 117632 23 0 0 25 0 5 0 522782138 2693599232 629054 33554432000 4194304 4417332 140733432864656 18446744073709551615 4310184 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=10582/tid=10592] ppid=10580 vsize=2630468 CPUtime=1180.86 cores=1,3,5,7
/proc/10582/task/10592/stat : 10592 (plingeling) R 10580 10582 9622 0 -1 4202560 1092394 1250 0 0 117915 171 0 0 25 0 5 0 522782138 2693599232 629054 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4721.12
Current children cumulated vsize (KiB) 2630468



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.1 s]
/proc/loadavg: 4.07 4.22 5.39 5/174 10641
/proc/meminfo: memFree=28131708/32951132 swapFree=67080208/67111528
[pid=10582] ppid=10580 vsize=2677984 CPUtime=4800.24 cores=1,3,5,7
/proc/10582/stat : 10582 (plingeling) S 10580 10582 9622 0 -1 4202496 1551986 1250 0 0 479787 237 0 0 25 0 5 0 522782062 2742255616 640933 33554432000 4194304 4417332 140733432864656 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/10582/statm: 669496 640933 180 55 0 667382 0
[pid=10582/tid=10589] ppid=10580 vsize=2677984 CPUtime=1201.3 cores=1,3,5,7
/proc/10582/task/10589/stat : 10589 (plingeling) R 10580 10582 9622 0 -1 4202560 107971 1250 0 0 120116 14 0 0 25 0 5 0 522782138 2742255616 640933 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364526 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=10582/tid=10590] ppid=10580 vsize=2677984 CPUtime=1201.24 cores=1,3,5,7
/proc/10582/task/10590/stat : 10590 (plingeling) R 10580 10582 9622 0 -1 4202560 140405 1250 0 0 120102 22 0 0 25 0 5 0 522782138 2742255616 640933 33554432000 4194304 4417332 140733432864656 18446744073709551615 4365960 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=10582/tid=10591] ppid=10580 vsize=2677984 CPUtime=1196.27 cores=1,3,5,7
/proc/10582/task/10591/stat : 10591 (plingeling) R 10580 10582 9622 0 -1 4202560 177751 1250 0 0 119602 25 0 0 25 0 5 0 522782138 2742255616 640933 33554432000 4194304 4417332 140733432864656 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=10582/tid=10592] ppid=10580 vsize=2677984 CPUtime=1200.66 cores=1,3,5,7
/proc/10582/task/10592/stat : 10592 (plingeling) R 10580 10582 9622 0 -1 4202560 1092394 1250 0 0 119895 171 0 0 25 0 5 0 522782138 2742255616 640933 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364750 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.24
Current children cumulated vsize (KiB) 2677984

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.1 s]
/proc/loadavg: 4.07 4.22 5.39 5/174 10641
/proc/meminfo: memFree=28131708/32951132 swapFree=67080208/67111528
[pid=10582] ppid=10580 vsize=2677984 CPUtime=4800.24 cores=1,3,5,7
/proc/10582/stat : 10582 (plingeling) S 10580 10582 9622 0 -1 4202496 1551986 1250 0 0 479787 237 0 0 25 0 5 0 522782062 2742255616 640933 33554432000 4194304 4417332 140733432864656 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/10582/statm: 669496 640933 180 55 0 667382 0
[pid=10582/tid=10589] ppid=10580 vsize=2677984 CPUtime=1201.3 cores=1,3,5,7
/proc/10582/task/10589/stat : 10589 (plingeling) R 10580 10582 9622 0 -1 4202560 107971 1250 0 0 120116 14 0 0 25 0 5 0 522782138 2742255616 640933 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364526 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=10582/tid=10590] ppid=10580 vsize=2677984 CPUtime=1201.24 cores=1,3,5,7
/proc/10582/task/10590/stat : 10590 (plingeling) R 10580 10582 9622 0 -1 4202560 140405 1250 0 0 120102 22 0 0 25 0 5 0 522782138 2742255616 640933 33554432000 4194304 4417332 140733432864656 18446744073709551615 4365960 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=10582/tid=10591] ppid=10580 vsize=2677984 CPUtime=1196.27 cores=1,3,5,7
/proc/10582/task/10591/stat : 10591 (plingeling) R 10580 10582 9622 0 -1 4202560 177751 1250 0 0 119602 25 0 0 25 0 5 0 522782138 2742255616 640933 33554432000 4194304 4417332 140733432864656 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=10582/tid=10592] ppid=10580 vsize=2677984 CPUtime=1200.66 cores=1,3,5,7
/proc/10582/task/10592/stat : 10592 (plingeling) R 10580 10582 9622 0 -1 4202560 1092394 1250 0 0 119895 171 0 0 25 0 5 0 522782138 2742255616 640933 33554432000 4194304 4417332 140733432864656 18446744073709551615 4364750 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.24
Current children cumulated vsize (KiB) 2677984

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.2
CPU time (s): 4800.37
CPU user time (s): 4797.9
CPU system time (s): 2.47162
CPU usage (%): 399.298
Max. virtual memory (cumulated for all children) (KiB): 2697980

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4797.9
system time used= 2.47162
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1553236
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 168
involuntary context switches= 33114

runsolver used 1.59276 second user time and 4.13137 second system time

The end

Launcher Data 

Begin job on node129 at 2011-04-29 23:17:04
IDJOB=3250856
IDBENCH=88203
IDSOLVER=1605
FILE ID=node129/3250856-1304111824
RUNJOBID= node129-1304111664-10410
PBS_JOBID= 13170649
Free space on /tmp= 66256 MiB

SOLVER NAME= Plingeling 587
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m62-q1327.5-n60-p30-s52708253.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3250856-1304111824/watcher-3250856-1304111824 -o /tmp/evaluation-result-3250856-1304111824/solver-3250856-1304111824 -C 4800 -W 1300 -M 15500  plingeling -t 4 HOME/instance-3250856-1304111824.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 472b10ce0a2683d9355981037eea091e
RANDOM SEED=1754982774

node129.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      29744864 kB
Buffers:        365408 kB
Cached:        1562972 kB
SwapCached:       5992 kB
Active:        1078932 kB
Inactive:      1737464 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      29744864 kB
SwapTotal:    67111528 kB
SwapFree:     67080208 kB
Dirty:           97804 kB
Writeback:          12 kB
AnonPages:      884292 kB
Mapped:          13008 kB
Slab:           324768 kB
PageTables:       6332 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS:  1099828 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 66356 MiB
End job on node129 at 2011-04-29 23:37:09