Trace number 3372761

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 40000.2 5007.54

General information on the benchmark

Nameapplication/kullmann/AES/Benchmarks/
32/aes_32_5_keyfind_1.cnf
MD5SUM3f4ea4cf17e734bbe947db7c1dc9fde7
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark7602.69
Satisfiable
(Un)Satisfiability was proved
Number of variables1116
Number of clauses4312
Sum of the clauses size13688
Maximum clause length5
Minimum clause length1
Number of clauses of size 184
Number of clauses of size 2200
Number of clauses of size 33008
Number of clauses of size 4920
Number of clauses of size 5100
Number of clauses of size over 50

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3372761-1305442297/watcher-3372761-1305442297 -o /tmp/evaluation-result-3372761-1305442297/solver-3372761-1305442297 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3372761-1305442297.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.97 7.82 7.93 2/351 24641
/proc/meminfo: memFree=25180832/32950928 swapFree=38710892/67111528
[pid=24641] ppid=24639 vsize=9620 CPUtime=0 cores=0-7
/proc/24641/stat : 24641 (plingeling) R 24639 24641 24299 0 -1 4202496 478 1250 0 0 0 0 0 0 25 0 1 0 734460334 9850880 408 33554432000 4194304 4417452 140736324574384 18446744073709551615 251367243531 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/24641/statm: 2405 408 148 55 0 291 0

[startup+0.111636 s]
/proc/loadavg: 6.97 7.82 7.93 2/351 24641
/proc/meminfo: memFree=25180832/32950928 swapFree=38710892/67111528
[pid=24641] ppid=24639 vsize=161184 CPUtime=0.67 cores=0-7
/proc/24641/stat : 24641 (plingeling) S 24639 24641 24299 0 -1 4202496 1722 1250 0 0 67 0 0 0 25 0 9 0 734460334 165052416 1546 33554432000 4194304 4417452 140736324574384 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/24641/statm: 40296 1546 180 55 0 38182 0
Current children cumulated CPU time (s) 0.67
Current children cumulated vsize (KiB) 161184

[startup+0.200561 s]
/proc/loadavg: 6.97 7.82 7.93 2/351 24641
/proc/meminfo: memFree=25180832/32950928 swapFree=38710892/67111528
[pid=24641] ppid=24639 vsize=161980 CPUtime=1.37 cores=0-7
/proc/24641/stat : 24641 (plingeling) S 24639 24641 24299 0 -1 4202496 1908 1250 0 0 137 0 0 0 25 0 9 0 734460334 165867520 1732 33554432000 4194304 4417452 140736324574384 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/24641/statm: 40495 1732 180 55 0 38381 0
Current children cumulated CPU time (s) 1.37
Current children cumulated vsize (KiB) 161980

[startup+0.300544 s]
/proc/loadavg: 6.97 7.82 7.93 2/351 24641
/proc/meminfo: memFree=25180832/32950928 swapFree=38710892/67111528
[pid=24641] ppid=24639 vsize=164408 CPUtime=2.16 cores=0-7
/proc/24641/stat : 24641 (plingeling) S 24639 24641 24299 0 -1 4202496 2818 1250 0 0 216 0 0 0 25 0 9 0 734460334 168353792 2473 33554432000 4194304 4417452 140736324574384 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/24641/statm: 41102 2473 180 55 0 38988 0
Current children cumulated CPU time (s) 2.16
Current children cumulated vsize (KiB) 164408

[startup+0.70047 s]
/proc/loadavg: 6.97 7.82 7.93 2/351 24641
/proc/meminfo: memFree=25180832/32950928 swapFree=38710892/67111528
[pid=24641] ppid=24639 vsize=167220 CPUtime=5.37 cores=0-7
/proc/24641/stat : 24641 (plingeling) S 24639 24641 24299 0 -1 4202496 5348 1250 0 0 536 1 0 0 25 0 9 0 734460334 171233280 3164 33554432000 4194304 4417452 140736324574384 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/24641/statm: 41805 3164 180 55 0 39691 0
Current children cumulated CPU time (s) 5.37
Current children cumulated vsize (KiB) 167220

[startup+1.50032 s]
/proc/loadavg: 6.97 7.82 7.93 9/360 24655
/proc/meminfo: memFree=25166372/32950928 swapFree=38710892/67111528
[pid=24641] ppid=24639 vsize=170884 CPUtime=11.75 cores=0-7
/proc/24641/stat : 24641 (plingeling) S 24639 24641 24299 0 -1 4202496 9806 1250 0 0 1173 2 0 0 25 0 9 0 734460334 174985216 4054 33554432000 4194304 4417452 140736324574384 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/24641/statm: 42721 4054 180 55 0 40607 0
[pid=24641/tid=24648] ppid=24639 vsize=170884 CPUtime=1.48 cores=0-7
/proc/24641/task/24648/stat : 24648 (plingeling) R 24639 24641 24299 0 -1 4202560 1391 1250 0 0 148 0 0 0 25 0 9 0 734460336 174985216 4054 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364653 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=24641/tid=24649] ppid=24639 vsize=170884 CPUtime=1.38 cores=0-7
/proc/24641/task/24649/stat : 24649 (plingeling) R 24639 24641 24299 0 -1 4202560 820 1250 0 0 138 0 0 0 25 0 9 0 734460336 174985216 4054 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24641/tid=24650] ppid=24639 vsize=170884 CPUtime=1.48 cores=0-7
/proc/24641/task/24650/stat : 24650 (plingeling) R 24639 24641 24299 0 -1 4202560 1008 1250 0 0 148 0 0 0 25 0 9 0 734460336 174985216 4054 33554432000 4194304 4417452 140736324574384 18446744073709551615 4353714 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=24641/tid=24651] ppid=24639 vsize=170884 CPUtime=1.45 cores=0-7
/proc/24641/task/24651/stat : 24651 (plingeling) R 24639 24641 24299 0 -1 4202560 1216 1250 0 0 145 0 0 0 25 0 9 0 734460336 174985216 4054 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=24641/tid=24652] ppid=24639 vsize=170884 CPUtime=1.48 cores=0-7
/proc/24641/task/24652/stat : 24652 (plingeling) R 24639 24641 24299 0 -1 4202560 1253 1250 0 0 148 0 0 0 25 0 9 0 734460336 174985216 4054 33554432000 4194304 4417452 140736324574384 18446744073709551615 4365973 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=24641/tid=24653] ppid=24639 vsize=170884 CPUtime=1.48 cores=0-7
/proc/24641/task/24653/stat : 24653 (plingeling) R 24639 24641 24299 0 -1 4202560 1110 1250 0 0 148 0 0 0 20 0 9 0 734460336 174985216 4054 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=24641/tid=24654] ppid=24639 vsize=170884 CPUtime=1.48 cores=0-7
/proc/24641/task/24654/stat : 24654 (plingeling) R 24639 24641 24299 0 -1 4202560 1140 1250 0 0 148 0 0 0 19 0 9 0 734460336 174985216 4054 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364756 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24641/tid=24655] ppid=24639 vsize=170884 CPUtime=1.48 cores=0-7
/proc/24641/task/24655/stat : 24655 (plingeling) R 24639 24641 24299 0 -1 4202560 941 1250 0 0 148 0 0 0 20 0 9 0 734460336 174985216 4054 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364650 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 11.75
Current children cumulated vsize (KiB) 170884

[startup+3.10103 s]
/proc/loadavg: 7.05 7.83 7.93 9/360 24655
/proc/meminfo: memFree=25160048/32950928 swapFree=38710892/67111528
[pid=24641] ppid=24639 vsize=176032 CPUtime=24.55 cores=0-7
/proc/24641/stat : 24641 (plingeling) S 24639 24641 24299 0 -1 4202496 19758 1250 0 0 2451 4 0 0 25 0 9 0 734460334 180256768 5384 33554432000 4194304 4417452 140736324574384 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/24641/statm: 44008 5384 180 55 0 41894 0
[pid=24641/tid=24648] ppid=24639 vsize=176032 CPUtime=3.08 cores=0-7
/proc/24641/task/24648/stat : 24648 (plingeling) R 24639 24641 24299 0 -1 4202560 2699 1250 0 0 308 0 0 0 25 0 9 0 734460336 180256768 5384 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364765 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=24641/tid=24649] ppid=24639 vsize=176032 CPUtime=2.96 cores=0-7
/proc/24641/task/24649/stat : 24649 (plingeling) R 24639 24641 24299 0 -1 4202560 2412 1250 0 0 296 0 0 0 25 0 9 0 734460336 180256768 5384 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24641/tid=24650] ppid=24639 vsize=176032 CPUtime=3.07 cores=0-7
/proc/24641/task/24650/stat : 24650 (plingeling) R 24639 24641 24299 0 -1 4202560 2135 1250 0 0 307 0 0 0 25 0 9 0 734460336 180256768 5384 33554432000 4194304 4417452 140736324574384 18446744073709551615 4237354 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=24641/tid=24651] ppid=24639 vsize=176032 CPUtime=3.05 cores=0-7
/proc/24641/task/24651/stat : 24651 (plingeling) R 24639 24641 24299 0 -1 4202560 2258 1250 0 0 305 0 0 0 25 0 9 0 734460336 180256768 5384 33554432000 4194304 4417452 140736324574384 18446744073709551615 4236956 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=24641/tid=24652] ppid=24639 vsize=176032 CPUtime=3.08 cores=0-7
/proc/24641/task/24652/stat : 24652 (plingeling) R 24639 24641 24299 0 -1 4202560 2253 1250 0 0 308 0 0 0 25 0 9 0 734460336 180256768 5384 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364877 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=24641/tid=24653] ppid=24639 vsize=176032 CPUtime=3.07 cores=0-7
/proc/24641/task/24653/stat : 24653 (plingeling) R 24639 24641 24299 0 -1 4202560 2382 1250 0 0 307 0 0 0 25 0 9 0 734460336 180256768 5384 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364542 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=24641/tid=24654] ppid=24639 vsize=176032 CPUtime=3.08 cores=0-7
/proc/24641/task/24654/stat : 24654 (plingeling) R 24639 24641 24299 0 -1 4202560 2309 1250 0 0 308 0 0 0 25 0 9 0 734460336 180256768 5384 33554432000 4194304 4417452 140736324574384 18446744073709551615 4353830 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24641/tid=24655] ppid=24639 vsize=176032 CPUtime=3.08 cores=0-7
/proc/24641/task/24655/stat : 24655 (plingeling) R 24639 24641 24299 0 -1 4202560 2383 1250 0 0 308 0 0 0 25 0 9 0 734460336 180256768 5384 33554432000 4194304 4417452 140736324574384 18446744073709551615 4353528 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 24.55
Current children cumulated vsize (KiB) 176032

[startup+6.30044 s]
/proc/loadavg: 7.05 7.83 7.93 9/360 24655
/proc/meminfo: memFree=25150532/32950928 swapFree=38710892/67111528
[pid=24641] ppid=24639 vsize=185456 CPUtime=50.11 cores=0-7
/proc/24641/stat : 24641 (plingeling) S 24639 24641 24299 0 -1 4202496 34779 1250 0 0 5003 8 0 0 25 0 9 0 734460334 189906944 7769 33554432000 4194304 4417452 140736324574384 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/24641/statm: 46364 7769 180 55 0 44250 0
[pid=24641/tid=24648] ppid=24639 vsize=185456 CPUtime=6.28 cores=0-7
/proc/24641/task/24648/stat : 24648 (plingeling) R 24639 24641 24299 0 -1 4202560 5114 1250 0 0 627 1 0 0 25 0 9 0 734460336 189906944 7769 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=24641/tid=24649] ppid=24639 vsize=185456 CPUtime=6.12 cores=0-7
/proc/24641/task/24649/stat : 24649 (plingeling) R 24639 24641 24299 0 -1 4202560 4502 1250 0 0 612 0 0 0 25 0 9 0 734460336 189906944 7769 33554432000 4194304 4417452 140736324574384 18446744073709551615 4365103 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24641/tid=24650] ppid=24639 vsize=185456 CPUtime=6.27 cores=0-7
/proc/24641/task/24650/stat : 24650 (plingeling) R 24639 24641 24299 0 -1 4202560 3770 1250 0 0 627 0 0 0 25 0 9 0 734460336 189906944 7769 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364798 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=24641/tid=24651] ppid=24639 vsize=185456 CPUtime=6.25 cores=0-7
/proc/24641/task/24651/stat : 24651 (plingeling) R 24639 24641 24299 0 -1 4202560 3899 1250 0 0 625 0 0 0 25 0 9 0 734460336 189906944 7769 33554432000 4194304 4417452 140736324574384 18446744073709551615 4251248 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=24641/tid=24652] ppid=24639 vsize=185456 CPUtime=6.27 cores=0-7
/proc/24641/task/24652/stat : 24652 (plingeling) R 24639 24641 24299 0 -1 4202560 4008 1250 0 0 627 0 0 0 25 0 9 0 734460336 189906944 7769 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=24641/tid=24653] ppid=24639 vsize=185456 CPUtime=6.28 cores=0-7
/proc/24641/task/24653/stat : 24653 (plingeling) R 24639 24641 24299 0 -1 4202560 4248 1250 0 0 627 1 0 0 25 0 9 0 734460336 189906944 7769 33554432000 4194304 4417452 140736324574384 18446744073709551615 4365996 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=24641/tid=24654] ppid=24639 vsize=185456 CPUtime=6.27 cores=0-7
/proc/24641/task/24654/stat : 24654 (plingeling) R 24639 24641 24299 0 -1 4202560 3903 1250 0 0 627 0 0 0 25 0 9 0 734460336 189906944 7769 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364903 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24641/tid=24655] ppid=24639 vsize=185456 CPUtime=6.28 cores=0-7
/proc/24641/task/24655/stat : 24655 (plingeling) R 24639 24641 24299 0 -1 4202560 4408 1250 0 0 627 1 0 0 25 0 9 0 734460336 189906944 7769 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364780 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 50.11
Current children cumulated vsize (KiB) 185456

[startup+12.7013 s]
/proc/loadavg: 7.20 7.83 7.93 9/360 24656
/proc/meminfo: memFree=25139372/32950928 swapFree=38710892/67111528
[pid=24641] ppid=24639 vsize=199524 CPUtime=101.23 cores=0-7
/proc/24641/stat : 24641 (plingeling) S 24639 24641 24299 0 -1 4202496 61711 1250 0 0 10110 13 0 0 25 0 9 0 734460334 204312576 11277 33554432000 4194304 4417452 140736324574384 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/24641/statm: 49881 11277 180 55 0 47767 0
[pid=24641/tid=24648] ppid=24639 vsize=199524 CPUtime=12.68 cores=0-7
/proc/24641/task/24648/stat : 24648 (plingeling) R 24639 24641 24299 0 -1 4202560 8640 1250 0 0 1267 1 0 0 25 0 9 0 734460336 204312576 11277 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364589 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=24641/tid=24649] ppid=24639 vsize=199524 CPUtime=12.45 cores=0-7
/proc/24641/task/24649/stat : 24649 (plingeling) R 24639 24641 24299 0 -1 4202560 7287 1250 0 0 1244 1 0 0 25 0 9 0 734460336 204312576 11277 33554432000 4194304 4417452 140736324574384 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24641/tid=24650] ppid=24639 vsize=199524 CPUtime=12.67 cores=0-7
/proc/24641/task/24650/stat : 24650 (plingeling) R 24639 24641 24299 0 -1 4202560 6938 1250 0 0 1266 1 0 0 25 0 9 0 734460336 204312576 11277 33554432000 4194304 4417452 140736324574384 18446744073709551615 4355812 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=24641/tid=24651] ppid=24639 vsize=199524 CPUtime=12.66 cores=0-7
/proc/24641/task/24651/stat : 24651 (plingeling) R 24639 24641 24299 0 -1 4202560 7007 1250 0 0 1265 1 0 0 25 0 9 0 734460336 204312576 11277 33554432000 4194304 4417452 140736324574384 18446744073709551615 4250932 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=24641/tid=24652] ppid=24639 vsize=199524 CPUtime=12.68 cores=0-7
/proc/24641/task/24652/stat : 24652 (plingeling) R 24639 24641 24299 0 -1 4202560 7913 1250 0 0 1267 1 0 0 25 0 9 0 734460336 204312576 11277 33554432000 4194304 4417452 140736324574384 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=24641/tid=24653] ppid=24639 vsize=199524 CPUtime=12.68 cores=0-7
/proc/24641/task/24653/stat : 24653 (plingeling) R 24639 24641 24299 0 -1 4202560 8455 1250 0 0 1266 2 0 0 20 0 9 0 734460336 204312576 11277 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364885 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=24641/tid=24654] ppid=24639 vsize=199524 CPUtime=12.67 cores=0-7
/proc/24641/task/24654/stat : 24654 (plingeling) R 24639 24641 24299 0 -1 4202560 6829 1250 0 0 1266 1 0 0 25 0 9 0 734460336 204312576 11277 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24641/tid=24655] ppid=24639 vsize=199524 CPUtime=12.67 cores=0-7
/proc/24641/task/24655/stat : 24655 (plingeling) R 24639 24641 24299 0 -1 4202560 7715 1250 0 0 1266 1 0 0 25 0 9 0 734460336 204312576 11277 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364737 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 101.23
Current children cumulated vsize (KiB) 199524

[startup+25.5009 s]
/proc/loadavg: 7.39 7.85 7.93 9/360 24656
/proc/meminfo: memFree=25129184/32950928 swapFree=38710892/67111528
[pid=24641] ppid=24639 vsize=206312 CPUtime=203.5 cores=0-7
/proc/24641/stat : 24641 (plingeling) S 24639 24641 24299 0 -1 4202496 107126 1250 0 0 20326 24 0 0 25 0 9 0 734460334 211263488 12984 33554432000 4194304 4417452 140736324574384 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/24641/statm: 51578 12984 180 55 0 49464 0
[pid=24641/tid=24648] ppid=24639 vsize=206312 CPUtime=25.48 cores=0-7
/proc/24641/task/24648/stat : 24648 (plingeling) R 24639 24641 24299 0 -1 4202560 15535 1250 0 0 2545 3 0 0 25 0 9 0 734460336 211263488 12984 33554432000 4194304 4417452 140736324574384 18446744073709551615 4366197 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=24641/tid=24649] ppid=24639 vsize=206312 CPUtime=25.11 cores=0-7
/proc/24641/task/24649/stat : 24649 (plingeling) R 24639 24641 24299 0 -1 4202560 11594 1250 0 0 2509 2 0 0 25 0 9 0 734460336 211263488 12984 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364572 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24641/tid=24650] ppid=24639 vsize=206312 CPUtime=25.48 cores=0-7
/proc/24641/task/24650/stat : 24650 (plingeling) R 24639 24641 24299 0 -1 4202560 11798 1250 0 0 2545 3 0 0 25 0 9 0 734460336 211263488 12984 33554432000 4194304 4417452 140736324574384 18446744073709551615 4366079 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=24641/tid=24651] ppid=24639 vsize=206312 CPUtime=25.45 cores=0-7
/proc/24641/task/24651/stat : 24651 (plingeling) R 24639 24641 24299 0 -1 4202560 13804 1250 0 0 2543 2 0 0 25 0 9 0 734460336 211263488 12984 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364756 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=24641/tid=24652] ppid=24639 vsize=206312 CPUtime=25.48 cores=0-7
/proc/24641/task/24652/stat : 24652 (plingeling) R 24639 24641 24299 0 -1 4202560 15390 1250 0 0 2545 3 0 0 25 0 9 0 734460336 211263488 12984 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=24641/tid=24653] ppid=24639 vsize=206312 CPUtime=25.48 cores=0-7
/proc/24641/task/24653/stat : 24653 (plingeling) R 24639 24641 24299 0 -1 4202560 14178 1250 0 0 2545 3 0 0 25 0 9 0 734460336 211263488 12984 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364650 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=24641/tid=24654] ppid=24639 vsize=206312 CPUtime=25.48 cores=0-7
/proc/24641/task/24654/stat : 24654 (plingeling) R 24639 24641 24299 0 -1 4202560 10966 1250 0 0 2546 2 0 0 25 0 9 0 734460336 211263488 12984 33554432000 4194304 4417452 140736324574384 18446744073709551615 4365075 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24641/tid=24655] ppid=24639 vsize=206312 CPUtime=25.47 cores=0-7
/proc/24641/task/24655/stat : 24655 (plingeling) R 24639 24641 24299 0 -1 4202560 12934 1250 0 0 2545 2 0 0 25 0 9 0 734460336 211263488 12984 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364835 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 203.5
Current children cumulated vsize (KiB) 206312

[startup+51.1012 s]
/proc/loadavg: 7.60 7.86 7.93 9/360 24657
/proc/meminfo: memFree=25104144/32950928 swapFree=38710892/67111528
[pid=24641] ppid=24639 vsize=235004 CPUtime=408.01 cores=0-7
/proc/24641/stat : 24641 (plingeling) S 24639 24641 24299 0 -1 4202496 189986 1250 0 0 40761 40 0 0 25 0 9 0 734460334 240644096 20198 33554432000 4194304 4417452 140736324574384 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/24641/statm: 58751 20198 180 55 0 56637 0
[pid=24641/tid=24648] ppid=24639 vsize=235004 CPUtime=51.08 cores=0-7
/proc/24641/task/24648/stat : 24648 (plingeling) R 24639 24641 24299 0 -1 4202560 28406 1250 0 0 5103 5 0 0 25 0 9 0 734460336 240644096 20198 33554432000 4194304 4417452 140736324574384 18446744073709551615 4366126 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=24641/tid=24649] ppid=24639 vsize=235004 CPUtime=50.43 cores=0-7
/proc/24641/task/24649/stat : 24649 (plingeling) R 24639 24641 24299 0 -1 4202560 20836 1250 0 0 5039 4 0 0 25 0 9 0 734460336 240644096 20198 33554432000 4194304 4417452 140736324574384 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24641/tid=24650] ppid=24639 vsize=235004 CPUtime=51.08 cores=0-7

################
# More data... #
################

/proc/24641/task/24651/stat : 24651 (plingeling) R 24639 24641 24299 0 -1 4202560 539568 1250 0 0 466056 96 0 0 25 0 9 0 734460336 644730880 119364 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=24641/tid=24652] ppid=24639 vsize=629620 CPUtime=4662.28 cores=0-7
/proc/24641/task/24652/stat : 24652 (plingeling) R 24639 24641 24299 0 -1 4202560 499208 1250 0 0 466139 89 0 0 25 0 9 0 734460336 644730880 119364 33554432000 4194304 4417452 140736324574384 18446744073709551615 4251060 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=24641/tid=24653] ppid=24639 vsize=629620 CPUtime=4662.42 cores=0-7
/proc/24641/task/24653/stat : 24653 (plingeling) R 24639 24641 24299 0 -1 4202560 550727 1250 0 0 466149 93 0 0 25 0 9 0 734460336 644730880 119364 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=24641/tid=24654] ppid=24639 vsize=629620 CPUtime=4662.28 cores=0-7
/proc/24641/task/24654/stat : 24654 (plingeling) R 24639 24641 24299 0 -1 4202560 535193 1250 0 0 466133 95 0 0 25 0 9 0 734460336 644730880 119364 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24641/tid=24655] ppid=24639 vsize=629620 CPUtime=4662.39 cores=0-7
/proc/24641/task/24655/stat : 24655 (plingeling) R 24639 24641 24299 0 -1 4202560 534363 1250 0 0 466153 86 0 0 25 0 9 0 734460336 644730880 119364 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 37242.5
Current children cumulated vsize (KiB) 629620

[startup+4722.3 s]
/proc/loadavg: 8.02 8.05 8.00 9/360 24816
/proc/meminfo: memFree=24698196/32950928 swapFree=38710892/67111528
[pid=24641] ppid=24639 vsize=636840 CPUtime=37721.9 cores=0-7
/proc/24641/stat : 24641 (plingeling) S 24639 24641 24299 0 -1 4202496 4339051 1250 0 0 3771441 745 0 0 25 0 9 0 734460334 652124160 121139 33554432000 4194304 4417452 140736324574384 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/24641/statm: 159210 121139 180 55 0 157096 0
[pid=24641/tid=24648] ppid=24639 vsize=636840 CPUtime=4721.79 cores=0-7
/proc/24641/task/24648/stat : 24648 (plingeling) R 24639 24641 24299 0 -1 4202560 718698 1250 0 0 472060 119 0 0 25 0 9 0 734460336 652124160 121139 33554432000 4194304 4417452 140736324574384 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=24641/tid=24649] ppid=24639 vsize=636840 CPUtime=4666.67 cores=0-7
/proc/24641/task/24649/stat : 24649 (plingeling) R 24639 24641 24299 0 -1 4202560 442956 1250 0 0 466592 75 0 0 25 0 9 0 734460336 652124160 121139 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364621 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24641/tid=24650] ppid=24639 vsize=636840 CPUtime=4722.42 cores=0-7
/proc/24641/task/24650/stat : 24650 (plingeling) R 24639 24641 24299 0 -1 4202560 497802 1250 0 0 472158 84 0 0 25 0 9 0 734460336 652124160 121139 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=24641/tid=24651] ppid=24639 vsize=636840 CPUtime=4721.53 cores=0-7
/proc/24641/task/24651/stat : 24651 (plingeling) R 24639 24641 24299 0 -1 4202560 544559 1250 0 0 472056 97 0 0 25 0 9 0 734460336 652124160 121139 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=24641/tid=24652] ppid=24639 vsize=636840 CPUtime=4722.29 cores=0-7
/proc/24641/task/24652/stat : 24652 (plingeling) R 24639 24641 24299 0 -1 4202560 501593 1250 0 0 472139 90 0 0 25 0 9 0 734460336 652124160 121139 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=24641/tid=24653] ppid=24639 vsize=636840 CPUtime=4722.44 cores=0-7
/proc/24641/task/24653/stat : 24653 (plingeling) R 24639 24641 24299 0 -1 4202560 553093 1250 0 0 472150 94 0 0 25 0 9 0 734460336 652124160 121139 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364629 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=24641/tid=24654] ppid=24639 vsize=636840 CPUtime=4722.28 cores=0-7
/proc/24641/task/24654/stat : 24654 (plingeling) R 24639 24641 24299 0 -1 4202560 540850 1250 0 0 472132 96 0 0 25 0 9 0 734460336 652124160 121139 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24641/tid=24655] ppid=24639 vsize=636840 CPUtime=4722.42 cores=0-7
/proc/24641/task/24655/stat : 24655 (plingeling) R 24639 24641 24299 0 -1 4202560 538573 1250 0 0 472155 87 0 0 25 0 9 0 734460336 652124160 121139 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 37721.9
Current children cumulated vsize (KiB) 636840

[startup+4782.3 s]
/proc/loadavg: 8.04 8.05 8.00 9/360 24818
/proc/meminfo: memFree=24719776/32950928 swapFree=38710892/67111528
[pid=24641] ppid=24639 vsize=614812 CPUtime=38201.1 cores=0-7
/proc/24641/stat : 24641 (plingeling) S 24639 24641 24299 0 -1 4202496 4373966 1250 0 0 3819357 753 0 0 25 0 9 0 734460334 629567488 115664 33554432000 4194304 4417452 140736324574384 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/24641/statm: 153703 115664 180 55 0 151589 0
[pid=24641/tid=24648] ppid=24639 vsize=614812 CPUtime=4781.78 cores=0-7
/proc/24641/task/24648/stat : 24648 (plingeling) R 24639 24641 24299 0 -1 4202560 721655 1250 0 0 478058 120 0 0 25 0 9 0 734460336 629567488 115664 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=24641/tid=24649] ppid=24639 vsize=614812 CPUtime=4725.96 cores=0-7
/proc/24641/task/24649/stat : 24649 (plingeling) R 24639 24641 24299 0 -1 4202560 444804 1250 0 0 472520 76 0 0 25 0 9 0 734460336 629567488 115664 33554432000 4194304 4417452 140736324574384 18446744073709551615 18446744073699065939 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24641/tid=24650] ppid=24639 vsize=614812 CPUtime=4782.4 cores=0-7
/proc/24641/task/24650/stat : 24650 (plingeling) R 24639 24641 24299 0 -1 4202560 504853 1250 0 0 478155 85 0 0 25 0 9 0 734460336 629567488 115664 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=24641/tid=24651] ppid=24639 vsize=614812 CPUtime=4781.52 cores=0-7
/proc/24641/task/24651/stat : 24651 (plingeling) R 24639 24641 24299 0 -1 4202560 549727 1250 0 0 478054 98 0 0 25 0 9 0 734460336 629567488 115664 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=24641/tid=24652] ppid=24639 vsize=614812 CPUtime=4782.28 cores=0-7
/proc/24641/task/24652/stat : 24652 (plingeling) R 24639 24641 24299 0 -1 4202560 505044 1250 0 0 478137 91 0 0 25 0 9 0 734460336 629567488 115664 33554432000 4194304 4417452 140736324574384 18446744073709551615 4366385 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=24641/tid=24653] ppid=24639 vsize=614812 CPUtime=4782.43 cores=0-7
/proc/24641/task/24653/stat : 24653 (plingeling) R 24639 24641 24299 0 -1 4202560 557133 1250 0 0 478149 94 0 0 25 0 9 0 734460336 629567488 115664 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=24641/tid=24654] ppid=24639 vsize=614812 CPUtime=4782.25 cores=0-7
/proc/24641/task/24654/stat : 24654 (plingeling) R 24639 24641 24299 0 -1 4202560 546494 1250 0 0 478128 97 0 0 25 0 9 0 734460336 629567488 115664 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24641/tid=24655] ppid=24639 vsize=614812 CPUtime=4782.4 cores=0-7
/proc/24641/task/24655/stat : 24655 (plingeling) R 24639 24641 24299 0 -1 4202560 543329 1250 0 0 478152 88 0 0 25 0 9 0 734460336 629567488 115664 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38201.1
Current children cumulated vsize (KiB) 614812

[startup+4842.31 s]
/proc/loadavg: 8.09 8.05 8.00 9/360 24820
/proc/meminfo: memFree=24721380/32950928 swapFree=38710892/67111528
[pid=24641] ppid=24639 vsize=613464 CPUtime=38680.5 cores=0-7
/proc/24641/stat : 24641 (plingeling) S 24639 24641 24299 0 -1 4202496 4407148 1250 0 0 3867291 759 0 0 25 0 9 0 734460334 628187136 115299 33554432000 4194304 4417452 140736324574384 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/24641/statm: 153366 115299 180 55 0 151252 0
[pid=24641/tid=24648] ppid=24639 vsize=613464 CPUtime=4841.8 cores=0-7
/proc/24641/task/24648/stat : 24648 (plingeling) R 24639 24641 24299 0 -1 4202560 727683 1250 0 0 484058 122 0 0 25 0 9 0 734460336 628187136 115299 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364611 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=24641/tid=24649] ppid=24639 vsize=613464 CPUtime=4785.26 cores=0-7
/proc/24641/task/24649/stat : 24649 (plingeling) R 24639 24641 24299 0 -1 4202560 450509 1250 0 0 478449 77 0 0 25 0 9 0 734460336 628187136 115299 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24641/tid=24650] ppid=24639 vsize=613464 CPUtime=4842.42 cores=0-7
/proc/24641/task/24650/stat : 24650 (plingeling) R 24639 24641 24299 0 -1 4202560 507257 1250 0 0 484156 86 0 0 25 0 9 0 734460336 628187136 115299 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364632 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=24641/tid=24651] ppid=24639 vsize=613464 CPUtime=4841.54 cores=0-7
/proc/24641/task/24651/stat : 24651 (plingeling) R 24639 24641 24299 0 -1 4202560 554135 1250 0 0 484055 99 0 0 25 0 9 0 734460336 628187136 115299 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364798 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=24641/tid=24652] ppid=24639 vsize=613464 CPUtime=4842.29 cores=0-7
/proc/24641/task/24652/stat : 24652 (plingeling) R 24639 24641 24299 0 -1 4202560 508677 1250 0 0 484138 91 0 0 25 0 9 0 734460336 628187136 115299 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=24641/tid=24653] ppid=24639 vsize=613464 CPUtime=4842.45 cores=0-7
/proc/24641/task/24653/stat : 24653 (plingeling) R 24639 24641 24299 0 -1 4202560 561548 1250 0 0 484150 95 0 0 25 0 9 0 734460336 628187136 115299 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364752 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=24641/tid=24654] ppid=24639 vsize=613464 CPUtime=4842.26 cores=0-7
/proc/24641/task/24654/stat : 24654 (plingeling) R 24639 24641 24299 0 -1 4202560 548837 1250 0 0 484128 98 0 0 25 0 9 0 734460336 628187136 115299 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364564 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24641/tid=24655] ppid=24639 vsize=613464 CPUtime=4842.41 cores=0-7
/proc/24641/task/24655/stat : 24655 (plingeling) R 24639 24641 24299 0 -1 4202560 547575 1250 0 0 484153 88 0 0 25 0 9 0 734460336 628187136 115299 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364877 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38680.5
Current children cumulated vsize (KiB) 613464

[startup+4902.3 s]
/proc/loadavg: 8.10 8.06 8.00 9/360 24821
/proc/meminfo: memFree=24686288/32950928 swapFree=38710892/67111528
[pid=24641] ppid=24639 vsize=648428 CPUtime=39159.7 cores=0-7
/proc/24641/stat : 24641 (plingeling) S 24639 24641 24299 0 -1 4202496 4440383 1250 0 0 3915205 764 0 0 25 0 9 0 734460334 663990272 124072 33554432000 4194304 4417452 140736324574384 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/24641/statm: 162107 124072 180 55 0 159993 0
[pid=24641/tid=24648] ppid=24639 vsize=648428 CPUtime=4901.78 cores=0-7
/proc/24641/task/24648/stat : 24648 (plingeling) R 24639 24641 24299 0 -1 4202560 732117 1250 0 0 490056 122 0 0 25 0 9 0 734460336 663990272 124072 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=24641/tid=24649] ppid=24639 vsize=648428 CPUtime=4844.54 cores=0-7
/proc/24641/task/24649/stat : 24649 (plingeling) R 24639 24641 24299 0 -1 4202560 454249 1250 0 0 484377 77 0 0 25 0 9 0 734460336 663990272 124072 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24641/tid=24650] ppid=24639 vsize=648428 CPUtime=4902.4 cores=0-7
/proc/24641/task/24650/stat : 24650 (plingeling) R 24639 24641 24299 0 -1 4202560 510978 1250 0 0 490154 86 0 0 25 0 9 0 734460336 663990272 124072 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364835 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=24641/tid=24651] ppid=24639 vsize=648428 CPUtime=4901.52 cores=0-7
/proc/24641/task/24651/stat : 24651 (plingeling) R 24639 24641 24299 0 -1 4202560 558717 1250 0 0 490053 99 0 0 25 0 9 0 734460336 663990272 124072 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=24641/tid=24652] ppid=24639 vsize=648428 CPUtime=4902.28 cores=0-7
/proc/24641/task/24652/stat : 24652 (plingeling) R 24639 24641 24299 0 -1 4202560 511561 1250 0 0 490137 91 0 0 25 0 9 0 734460336 663990272 124072 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=24641/tid=24653] ppid=24639 vsize=648428 CPUtime=4902.44 cores=0-7
/proc/24641/task/24653/stat : 24653 (plingeling) R 24639 24641 24299 0 -1 4202560 565606 1250 0 0 490148 96 0 0 25 0 9 0 734460336 663990272 124072 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364632 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=24641/tid=24654] ppid=24639 vsize=648428 CPUtime=4902.23 cores=0-7
/proc/24641/task/24654/stat : 24654 (plingeling) R 24639 24641 24299 0 -1 4202560 551962 1250 0 0 490124 99 0 0 25 0 9 0 734460336 663990272 124072 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24641/tid=24655] ppid=24639 vsize=648428 CPUtime=4902.4 cores=0-7
/proc/24641/task/24655/stat : 24655 (plingeling) R 24639 24641 24299 0 -1 4202560 554266 1250 0 0 490151 89 0 0 25 0 9 0 734460336 663990272 124072 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364629 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 39159.7
Current children cumulated vsize (KiB) 648428

[startup+4962.3 s]
/proc/loadavg: 8.10 8.06 8.00 9/360 24823
/proc/meminfo: memFree=24733900/32950928 swapFree=38710892/67111528
[pid=24641] ppid=24639 vsize=601040 CPUtime=39639 cores=0-7
/proc/24641/stat : 24641 (plingeling) S 24639 24641 24299 0 -1 4202496 4478529 1250 0 0 3963129 770 0 0 25 0 9 0 734460334 615464960 112195 33554432000 4194304 4417452 140736324574384 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/24641/statm: 150260 112195 180 55 0 148146 0
[pid=24641/tid=24648] ppid=24639 vsize=601040 CPUtime=4961.79 cores=0-7
/proc/24641/task/24648/stat : 24648 (plingeling) R 24639 24641 24299 0 -1 4202560 741769 1250 0 0 496055 124 0 0 25 0 9 0 734460336 615464960 112195 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=24641/tid=24649] ppid=24639 vsize=601040 CPUtime=4903.83 cores=0-7
/proc/24641/task/24649/stat : 24649 (plingeling) R 24639 24641 24299 0 -1 4202560 457528 1250 0 0 490305 78 0 0 25 0 9 0 734460336 615464960 112195 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24641/tid=24650] ppid=24639 vsize=601040 CPUtime=4962.41 cores=0-7
/proc/24641/task/24650/stat : 24650 (plingeling) R 24639 24641 24299 0 -1 4202560 513401 1250 0 0 496154 87 0 0 25 0 9 0 734460336 615464960 112195 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=24641/tid=24651] ppid=24639 vsize=601040 CPUtime=4961.53 cores=0-7
/proc/24641/task/24651/stat : 24651 (plingeling) R 24639 24641 24299 0 -1 4202560 564258 1250 0 0 496052 101 0 0 25 0 9 0 734460336 615464960 112195 33554432000 4194304 4417452 140736324574384 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=24641/tid=24652] ppid=24639 vsize=601040 CPUtime=4962.28 cores=0-7
/proc/24641/task/24652/stat : 24652 (plingeling) R 24639 24641 24299 0 -1 4202560 515774 1250 0 0 496137 91 0 0 25 0 9 0 734460336 615464960 112195 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=24641/tid=24653] ppid=24639 vsize=601040 CPUtime=4962.44 cores=0-7
/proc/24641/task/24653/stat : 24653 (plingeling) R 24639 24641 24299 0 -1 4202560 570023 1250 0 0 496148 96 0 0 25 0 9 0 734460336 615464960 112195 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=24641/tid=24654] ppid=24639 vsize=601040 CPUtime=4962.22 cores=0-7
/proc/24641/task/24654/stat : 24654 (plingeling) R 24639 24641 24299 0 -1 4202560 556779 1250 0 0 496123 99 0 0 25 0 9 0 734460336 615464960 112195 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24641/tid=24655] ppid=24639 vsize=601040 CPUtime=4962.4 cores=0-7
/proc/24641/task/24655/stat : 24655 (plingeling) R 24639 24641 24299 0 -1 4202560 558070 1250 0 0 496150 90 0 0 25 0 9 0 734460336 615464960 112195 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364657 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 39639
Current children cumulated vsize (KiB) 601040



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5007.5 s]
/proc/loadavg: 8.12 8.07 8.00 9/360 24824
/proc/meminfo: memFree=24693716/32950928 swapFree=38710892/67111528
[pid=24641] ppid=24639 vsize=640796 CPUtime=40000.1 cores=0-7
/proc/24641/stat : 24641 (plingeling) S 24639 24641 24299 0 -1 4202496 4498075 1250 0 0 3999237 773 0 0 25 0 9 0 734460334 656175104 122134 33554432000 4194304 4417452 140736324574384 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/24641/statm: 160199 122134 180 55 0 158085 0
[pid=24641/tid=24648] ppid=24639 vsize=640796 CPUtime=5007 cores=0-7
/proc/24641/task/24648/stat : 24648 (plingeling) R 24639 24641 24299 0 -1 4202560 743990 1250 0 0 500576 124 0 0 25 0 9 0 734460336 656175104 122134 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=24641/tid=24649] ppid=24639 vsize=640796 CPUtime=4948.5 cores=0-7
/proc/24641/task/24649/stat : 24649 (plingeling) R 24639 24641 24299 0 -1 4202560 458790 1250 0 0 494772 78 0 0 25 0 9 0 734460336 656175104 122134 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24641/tid=24650] ppid=24639 vsize=640796 CPUtime=5007.62 cores=0-7
/proc/24641/task/24650/stat : 24650 (plingeling) R 24639 24641 24299 0 -1 4202560 517135 1250 0 0 500675 87 0 0 25 0 9 0 734460336 656175104 122134 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=24641/tid=24651] ppid=24639 vsize=640796 CPUtime=5006.74 cores=0-7
/proc/24641/task/24651/stat : 24651 (plingeling) R 24639 24641 24299 0 -1 4202560 568860 1250 0 0 500573 101 0 0 25 0 9 0 734460336 656175104 122134 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=24641/tid=24652] ppid=24639 vsize=640796 CPUtime=5007.49 cores=0-7
/proc/24641/task/24652/stat : 24652 (plingeling) R 24639 24641 24299 0 -1 4202560 515774 1250 0 0 500658 91 0 0 25 0 9 0 734460336 656175104 122134 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=24641/tid=24653] ppid=24639 vsize=640796 CPUtime=5007.65 cores=0-7
/proc/24641/task/24653/stat : 24653 (plingeling) R 24639 24641 24299 0 -1 4202560 571917 1250 0 0 500669 96 0 0 25 0 9 0 734460336 656175104 122134 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=24641/tid=24654] ppid=24639 vsize=640796 CPUtime=5007.43 cores=0-7
/proc/24641/task/24654/stat : 24654 (plingeling) R 24639 24641 24299 0 -1 4202560 556779 1250 0 0 500644 99 0 0 25 0 9 0 734460336 656175104 122134 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24641/tid=24655] ppid=24639 vsize=640796 CPUtime=5007.62 cores=0-7
/proc/24641/task/24655/stat : 24655 (plingeling) R 24639 24641 24299 0 -1 4202560 563903 1250 0 0 500671 91 0 0 25 0 9 0 734460336 656175104 122134 33554432000 4194304 4417452 140736324574384 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 40000.1
Current children cumulated vsize (KiB) 640796

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5007.54
CPU time (s): 40000.2
CPU user time (s): 39992.5
CPU system time (s): 7.75882
CPU usage (%): 798.8
Max. virtual memory (cumulated for all children) (KiB): 660524

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39992.5
system time used= 7.75882
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 4499325
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 204
involuntary context switches= 191003

runsolver used 17.3034 second user time and 41.7706 second system time

The end

Launcher Data 

Begin job on node113 at 2011-05-15 08:51:37
IDJOB=3372761
IDBENCH=83430
IDSOLVER=1851
FILE ID=node113/3372761-1305442297
RUNJOBID= node113-1305436997-24313
PBS_JOBID= 13324761
Free space on /tmp= 73484 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT11/application/kullmann/AES/Benchmarks/32/aes_32_5_keyfind_1.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3372761-1305442297/watcher-3372761-1305442297 -o /tmp/evaluation-result-3372761-1305442297/solver-3372761-1305442297 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3372761-1305442297.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 3f4ea4cf17e734bbe947db7c1dc9fde7
RANDOM SEED=1179783974

node113.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      25181236 kB
Buffers:        266660 kB
Cached:        2847460 kB
SwapCached:      93712 kB
Active:         787600 kB
Inactive:      6716804 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      25181236 kB
SwapTotal:    67111528 kB
SwapFree:     38710892 kB
Dirty:             404 kB
Writeback:           0 kB
AnonPages:     4308784 kB
Mapped:          13552 kB
Slab:           107184 kB
PageTables:      91484 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 87040308 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73484 MiB
End job on node113 at 2011-05-15 10:15:06