Trace number 3394494

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 40000.8 5012.25

General information on the benchmark

Namecrafted/skvortsov/automata-synchronization/
crn_40_1521_s.cnf
MD5SUM4e35cb9894b1ca841d462da2d6957cc5
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables63882
Number of clauses123981
Sum of the clauses size369562
Maximum clause length3
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 22381
Number of clauses of size 3121600
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3394494-1306098122/watcher-3394494-1306098122 -o /tmp/evaluation-result-3394494-1306098122/solver-3394494-1306098122 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3394494-1306098122.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.12 8.08 7.98 1/510 2389
/proc/meminfo: memFree=8872676/32950928 swapFree=45358860/67111528
[pid=2389] ppid=2387 vsize=16680 CPUtime=0 cores=0-7
/proc/2389/stat : 2389 (plingeling) R 2387 2389 1932 0 -1 4202496 2166 1252 0 0 0 0 0 0 25 0 1 0 721408850 17080320 2097 33554432000 4194304 4417452 140737130684976 18446744073709551615 254500385885 0 0 4096 17442 0 0 0 17 6 0 0 0
/proc/2389/statm: 4170 2110 141 55 0 2056 0

[startup+0.0902839 s]
/proc/loadavg: 8.12 8.08 7.98 1/510 2389
/proc/meminfo: memFree=8872676/32950928 swapFree=45358860/67111528
[pid=2389] ppid=2387 vsize=33532 CPUtime=0.07 cores=0-7
/proc/2389/stat : 2389 (plingeling) R 2387 2389 1932 0 -1 4202496 6463 1252 0 0 7 0 0 0 25 0 1 0 721408850 34336768 6392 33554432000 4194304 4417452 140737130684976 18446744073709551615 4274358 0 0 4096 17442 0 0 0 17 6 0 0 0
/proc/2389/statm: 8383 6392 149 55 0 6269 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 33532

[startup+0.100261 s]
/proc/loadavg: 8.12 8.08 7.98 1/510 2389
/proc/meminfo: memFree=8872676/32950928 swapFree=45358860/67111528
[pid=2389] ppid=2387 vsize=40860 CPUtime=0.08 cores=0-7
/proc/2389/stat : 2389 (plingeling) R 2387 2389 1932 0 -1 4202496 8311 1252 0 0 8 0 0 0 25 0 1 0 721408850 41840640 8240 33554432000 4194304 4417452 140737130684976 18446744073709551615 4259600 0 0 4096 17442 0 0 0 17 6 0 0 0
/proc/2389/statm: 10215 8240 149 55 0 8101 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 40860

[startup+0.300233 s]
/proc/loadavg: 8.12 8.08 7.98 1/510 2389
/proc/meminfo: memFree=8872676/32950928 swapFree=45358860/67111528
[pid=2389] ppid=2387 vsize=80084 CPUtime=0.28 cores=0-7
/proc/2389/stat : 2389 (plingeling) R 2387 2389 1932 0 -1 4202496 18141 1252 0 0 27 1 0 0 25 0 1 0 721408850 82006016 18046 33554432000 4194304 4417452 140737130684976 18446744073709551615 4259651 0 0 4096 17442 0 0 0 17 6 0 0 0
/proc/2389/statm: 20021 18046 149 55 0 17907 0
Current children cumulated CPU time (s) 0.28
Current children cumulated vsize (KiB) 80084

[startup+1.37794 s]
/proc/loadavg: 8.12 8.08 7.98 1/510 2389
/proc/meminfo: memFree=8872676/32950928 swapFree=45358860/67111528
[pid=2389] ppid=2387 vsize=292944 CPUtime=8.34 cores=0-7
/proc/2389/stat : 2389 (plingeling) S 2387 2389 1932 0 -1 4202496 67628 1252 0 0 825 9 0 0 18 0 9 0 721408850 299974656 25699 33554432000 4194304 4417452 140737130684976 18446744073709551615 254512494565 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/2389/statm: 73236 25699 174 55 0 71122 0
Current children cumulated CPU time (s) 8.34
Current children cumulated vsize (KiB) 292944

[startup+1.5779 s]
/proc/loadavg: 8.12 8.08 7.98 1/510 2389
/proc/meminfo: memFree=8872676/32950928 swapFree=45358860/67111528
[pid=2389] ppid=2387 vsize=292944 CPUtime=9.93 cores=0-7
/proc/2389/stat : 2389 (plingeling) S 2387 2389 1932 0 -1 4202496 70811 1252 0 0 984 9 0 0 18 0 9 0 721408850 299974656 25347 33554432000 4194304 4417452 140737130684976 18446744073709551615 254512494565 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/2389/statm: 73236 25347 174 55 0 71122 0
Current children cumulated CPU time (s) 9.93
Current children cumulated vsize (KiB) 292944

[startup+3.10059 s]
/proc/loadavg: 8.19 8.09 7.98 9/519 2403
/proc/meminfo: memFree=8754316/32950928 swapFree=45358860/67111528
[pid=2389] ppid=2387 vsize=343956 CPUtime=22.08 cores=0-7
/proc/2389/stat : 2389 (plingeling) S 2387 2389 1932 0 -1 4202496 98674 1252 0 0 2195 13 0 0 18 0 9 0 721408850 352210944 32503 33554432000 4194304 4417452 140737130684976 18446744073709551615 254512494565 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/2389/statm: 85989 32503 179 55 0 83875 0
[pid=2389/tid=2396] ppid=2387 vsize=343956 CPUtime=2.71 cores=0-7
/proc/2389/task/2396/stat : 2396 (plingeling) R 2387 2389 1932 0 -1 4202560 7360 1252 0 0 271 0 0 0 23 0 9 0 721408887 352210944 32503 33554432000 4194304 4417452 140737130684976 18446744073709551615 4252863 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2389/tid=2397] ppid=2387 vsize=343956 CPUtime=2.71 cores=0-7
/proc/2389/task/2397/stat : 2397 (plingeling) R 2387 2389 1932 0 -1 4202560 16049 1252 0 0 269 2 0 0 23 0 9 0 721408887 352210944 32503 33554432000 4194304 4417452 140737130684976 18446744073709551615 4257792 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2389/tid=2398] ppid=2387 vsize=343956 CPUtime=2.71 cores=0-7
/proc/2389/task/2398/stat : 2398 (plingeling) R 2387 2389 1932 0 -1 4202560 15452 1252 0 0 269 2 0 0 23 0 9 0 721408887 352210944 32503 33554432000 4194304 4417452 140737130684976 18446744073709551615 4249186 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2389/tid=2399] ppid=2387 vsize=343956 CPUtime=2.72 cores=0-7
/proc/2389/task/2399/stat : 2399 (plingeling) R 2387 2389 1932 0 -1 4202560 613 1252 0 0 272 0 0 0 23 0 9 0 721408887 352210944 32503 33554432000 4194304 4417452 140737130684976 18446744073709551615 4232623 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=2389/tid=2400] ppid=2387 vsize=343956 CPUtime=2.71 cores=0-7
/proc/2389/task/2400/stat : 2400 (plingeling) R 2387 2389 1932 0 -1 4202560 10390 1252 0 0 270 1 0 0 23 0 9 0 721408887 352210944 32503 33554432000 4194304 4417452 140737130684976 18446744073709551615 4253152 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=2389/tid=2401] ppid=2387 vsize=343956 CPUtime=2.68 cores=0-7
/proc/2389/task/2401/stat : 2401 (plingeling) R 2387 2389 1932 0 -1 4202560 13057 1252 0 0 266 2 0 0 22 0 9 0 721408887 352210944 32503 33554432000 4194304 4417452 140737130684976 18446744073709551615 4252507 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2389/tid=2402] ppid=2387 vsize=343956 CPUtime=2.71 cores=0-7
/proc/2389/task/2402/stat : 2402 (plingeling) R 2387 2389 1932 0 -1 4202560 8488 1252 0 0 270 1 0 0 23 0 9 0 721408887 352210944 32503 33554432000 4194304 4417452 140737130684976 18446744073709551615 4249296 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2389/tid=2403] ppid=2387 vsize=343956 CPUtime=2.7 cores=0-7
/proc/2389/task/2403/stat : 2403 (plingeling) R 2387 2389 1932 0 -1 4202560 9108 1252 0 0 270 0 0 0 25 0 9 0 721408887 352210944 32503 33554432000 4194304 4417452 140737130684976 18446744073709551615 4252385 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 22.08
Current children cumulated vsize (KiB) 343956

[startup+6.31895 s]
/proc/loadavg: 8.17 8.09 7.98 9/519 2403
/proc/meminfo: memFree=8694548/32950928 swapFree=45358860/67111528
[pid=2389] ppid=2387 vsize=388896 CPUtime=47.78 cores=0-7
/proc/2389/stat : 2389 (plingeling) S 2387 2389 1932 0 -1 4202496 110653 1252 0 0 4763 15 0 0 18 0 9 0 721408850 398229504 44365 33554432000 4194304 4417452 140737130684976 18446744073709551615 254512494565 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/2389/statm: 97224 44365 180 55 0 95110 0
[pid=2389/tid=2396] ppid=2387 vsize=388896 CPUtime=5.93 cores=0-7
/proc/2389/task/2396/stat : 2396 (plingeling) R 2387 2389 1932 0 -1 4202560 9410 1252 0 0 592 1 0 0 25 0 9 0 721408887 398229504 44365 33554432000 4194304 4417452 140737130684976 18446744073709551615 4257368 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2389/tid=2397] ppid=2387 vsize=388896 CPUtime=5.93 cores=0-7
/proc/2389/task/2397/stat : 2397 (plingeling) R 2387 2389 1932 0 -1 4202560 17461 1252 0 0 591 2 0 0 25 0 9 0 721408887 398229504 44365 33554432000 4194304 4417452 140737130684976 18446744073709551615 4252188 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2389/tid=2398] ppid=2387 vsize=388896 CPUtime=5.93 cores=0-7
/proc/2389/task/2398/stat : 2398 (plingeling) R 2387 2389 1932 0 -1 4202560 15484 1252 0 0 591 2 0 0 25 0 9 0 721408887 398229504 44365 33554432000 4194304 4417452 140737130684976 18446744073709551615 4249143 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2389/tid=2399] ppid=2387 vsize=388896 CPUtime=5.94 cores=0-7
/proc/2389/task/2399/stat : 2399 (plingeling) R 2387 2389 1932 0 -1 4202560 613 1252 0 0 594 0 0 0 25 0 9 0 721408887 398229504 44365 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=2389/tid=2400] ppid=2387 vsize=388896 CPUtime=5.93 cores=0-7
/proc/2389/task/2400/stat : 2400 (plingeling) R 2387 2389 1932 0 -1 4202560 12440 1252 0 0 592 1 0 0 25 0 9 0 721408887 398229504 44365 33554432000 4194304 4417452 140737130684976 18446744073709551615 4253152 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=2389/tid=2401] ppid=2387 vsize=388896 CPUtime=5.84 cores=0-7
/proc/2389/task/2401/stat : 2401 (plingeling) R 2387 2389 1932 0 -1 4202560 15107 1252 0 0 582 2 0 0 25 0 9 0 721408887 398229504 44365 33554432000 4194304 4417452 140737130684976 18446744073709551615 4271831 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2389/tid=2402] ppid=2387 vsize=388896 CPUtime=5.92 cores=0-7
/proc/2389/task/2402/stat : 2402 (plingeling) R 2387 2389 1932 0 -1 4202560 10568 1252 0 0 591 1 0 0 25 0 9 0 721408887 398229504 44365 33554432000 4194304 4417452 140737130684976 18446744073709551615 4265437 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2389/tid=2403] ppid=2387 vsize=388896 CPUtime=5.93 cores=0-7
/proc/2389/task/2403/stat : 2403 (plingeling) R 2387 2389 1932 0 -1 4202560 11413 1252 0 0 592 1 0 0 25 0 9 0 721408887 398229504 44365 33554432000 4194304 4417452 140737130684976 18446744073709551615 4272899 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 47.78
Current children cumulated vsize (KiB) 388896

[startup+12.7007 s]
/proc/loadavg: 8.16 8.09 7.98 9/519 2403
/proc/meminfo: memFree=8688124/32950928 swapFree=45358860/67111528
[pid=2389] ppid=2387 vsize=385276 CPUtime=98.73 cores=0-7
/proc/2389/stat : 2389 (plingeling) S 2387 2389 1932 0 -1 4202496 115913 1252 0 0 9858 15 0 0 18 0 9 0 721408850 394522624 43472 33554432000 4194304 4417452 140737130684976 18446744073709551615 254512494565 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/2389/statm: 96319 43472 180 55 0 94205 0
[pid=2389/tid=2396] ppid=2387 vsize=385276 CPUtime=12.31 cores=0-7
/proc/2389/task/2396/stat : 2396 (plingeling) R 2387 2389 1932 0 -1 4202560 9410 1252 0 0 1230 1 0 0 25 0 9 0 721408887 394522624 43472 33554432000 4194304 4417452 140737130684976 18446744073709551615 4199406 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2389/tid=2397] ppid=2387 vsize=385276 CPUtime=12.31 cores=0-7
/proc/2389/task/2397/stat : 2397 (plingeling) R 2387 2389 1932 0 -1 4202560 19510 1252 0 0 1229 2 0 0 25 0 9 0 721408887 394522624 43472 33554432000 4194304 4417452 140737130684976 18446744073709551615 4264796 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2389/tid=2398] ppid=2387 vsize=385276 CPUtime=12.31 cores=0-7
/proc/2389/task/2398/stat : 2398 (plingeling) R 2387 2389 1932 0 -1 4202560 16064 1252 0 0 1229 2 0 0 25 0 9 0 721408887 394522624 43472 33554432000 4194304 4417452 140737130684976 18446744073709551615 4253214 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2389/tid=2399] ppid=2387 vsize=385276 CPUtime=12.32 cores=0-7
/proc/2389/task/2399/stat : 2399 (plingeling) R 2387 2389 1932 0 -1 4202560 613 1252 0 0 1232 0 0 0 25 0 9 0 721408887 394522624 43472 33554432000 4194304 4417452 140737130684976 18446744073709551615 4232680 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=2389/tid=2400] ppid=2387 vsize=385276 CPUtime=12.31 cores=0-7
/proc/2389/task/2400/stat : 2400 (plingeling) R 2387 2389 1932 0 -1 4202560 14489 1252 0 0 1230 1 0 0 25 0 9 0 721408887 394522624 43472 33554432000 4194304 4417452 140737130684976 18446744073709551615 4365023 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=2389/tid=2401] ppid=2387 vsize=385276 CPUtime=12.13 cores=0-7
/proc/2389/task/2401/stat : 2401 (plingeling) R 2387 2389 1932 0 -1 4202560 15107 1252 0 0 1211 2 0 0 25 0 9 0 721408887 394522624 43472 33554432000 4194304 4417452 140737130684976 18446744073709551615 4234242 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2389/tid=2402] ppid=2387 vsize=385276 CPUtime=12.3 cores=0-7
/proc/2389/task/2402/stat : 2402 (plingeling) R 2387 2389 1932 0 -1 4202560 11150 1252 0 0 1229 1 0 0 25 0 9 0 721408887 394522624 43472 33554432000 4194304 4417452 140737130684976 18446744073709551615 4252394 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2389/tid=2403] ppid=2387 vsize=385276 CPUtime=12.31 cores=0-7
/proc/2389/task/2403/stat : 2403 (plingeling) R 2387 2389 1932 0 -1 4202560 11413 1252 0 0 1230 1 0 0 25 0 9 0 721408887 394522624 43472 33554432000 4194304 4417452 140737130684976 18446744073709551615 4232615 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 98.73
Current children cumulated vsize (KiB) 385276

[startup+25.5011 s]
/proc/loadavg: 8.13 8.09 7.98 9/519 2403
/proc/meminfo: memFree=8684332/32950928 swapFree=45358860/67111528
[pid=2389] ppid=2387 vsize=395648 CPUtime=200.92 cores=0-7
/proc/2389/stat : 2389 (plingeling) S 2387 2389 1932 0 -1 4202496 137177 1252 0 0 20072 20 0 0 18 0 9 0 721408850 405143552 47092 33554432000 4194304 4417452 140737130684976 18446744073709551615 254512494565 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/2389/statm: 98912 47092 180 55 0 96798 0
[pid=2389/tid=2396] ppid=2387 vsize=395648 CPUtime=25.11 cores=0-7
/proc/2389/task/2396/stat : 2396 (plingeling) R 2387 2389 1932 0 -1 4202560 11414 1252 0 0 2510 1 0 0 25 0 9 0 721408887 405143552 47092 33554432000 4194304 4417452 140737130684976 18446744073709551615 4253176 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2389/tid=2397] ppid=2387 vsize=395648 CPUtime=25.11 cores=0-7
/proc/2389/task/2397/stat : 2397 (plingeling) R 2387 2389 1932 0 -1 4202560 25249 1252 0 0 2508 3 0 0 25 0 9 0 721408887 405143552 47092 33554432000 4194304 4417452 140737130684976 18446744073709551615 4343760 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2389/tid=2398] ppid=2387 vsize=395648 CPUtime=25.12 cores=0-7
/proc/2389/task/2398/stat : 2398 (plingeling) R 2387 2389 1932 0 -1 4202560 18170 1252 0 0 2509 3 0 0 25 0 9 0 721408887 405143552 47092 33554432000 4194304 4417452 140737130684976 18446744073709551615 4236930 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2389/tid=2399] ppid=2387 vsize=395648 CPUtime=25.1 cores=0-7
/proc/2389/task/2399/stat : 2399 (plingeling) R 2387 2389 1932 0 -1 4202560 3685 1252 0 0 2510 0 0 0 25 0 9 0 721408887 405143552 47092 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=2389/tid=2400] ppid=2387 vsize=395648 CPUtime=25.11 cores=0-7
/proc/2389/task/2400/stat : 2400 (plingeling) R 2387 2389 1932 0 -1 4202560 16597 1252 0 0 2509 2 0 0 25 0 9 0 721408887 405143552 47092 33554432000 4194304 4417452 140737130684976 18446744073709551615 4257702 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=2389/tid=2401] ppid=2387 vsize=395648 CPUtime=24.71 cores=0-7
/proc/2389/task/2401/stat : 2401 (plingeling) R 2387 2389 1932 0 -1 4202560 15167 1252 0 0 2469 2 0 0 25 0 9 0 721408887 405143552 47092 33554432000 4194304 4417452 140737130684976 18446744073709551615 4252316 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2389/tid=2402] ppid=2387 vsize=395648 CPUtime=25.11 cores=0-7
/proc/2389/task/2402/stat : 2402 (plingeling) R 2387 2389 1932 0 -1 4202560 13229 1252 0 0 2509 2 0 0 25 0 9 0 721408887 405143552 47092 33554432000 4194304 4417452 140737130684976 18446744073709551615 4252176 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2389/tid=2403] ppid=2387 vsize=395648 CPUtime=25.11 cores=0-7
/proc/2389/task/2403/stat : 2403 (plingeling) R 2387 2389 1932 0 -1 4202560 15509 1252 0 0 2510 1 0 0 25 0 9 0 721408887 405143552 47092 33554432000 4194304 4417452 140737130684976 18446744073709551615 4252286 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 200.92
Current children cumulated vsize (KiB) 395648

[startup+51.1011 s]
/proc/loadavg: 8.09 8.08 7.98 9/519 2404
/proc/meminfo: memFree=8615072/32950928 swapFree=45358860/67111528
[pid=2389] ppid=2387 vsize=472520 CPUtime=405.29 cores=0-7
/proc/2389/stat : 2389 (plingeling) S 2387 2389 1932 0 -1 4202496 226140 1252 0 0 40498 31 0 0 18 0 9 0 721408850 483860480 67834 33554432000 4194304 4417452 140737130684976 18446744073709551615 254512494565 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/2389/statm: 118130 67834 180 55 0 116016 0
[pid=2389/tid=2396] ppid=2387 vsize=472520 CPUtime=50.71 cores=0-7
/proc/2389/task/2396/stat : 2396 (plingeling) R 2387 2389 1932 0 -1 4202560 27685 1252 0 0 5068 3 0 0 25 0 9 0 721408887 483860480 67834 33554432000 4194304 4417452 140737130684976 18446744073709551615 4306923 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2389/tid=2397] ppid=2387 vsize=472520 CPUtime=50.71 cores=0-7
/proc/2389/task/2397/stat : 2397 (plingeling) R 2387 2389 1932 0 -1 4202560 39723 1252 0 0 5066 5 0 0 25 0 9 0 721408887 483860480 67834 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364765 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2389/tid=2398] ppid=2387 vsize=472520 CPUtime=50.72 cores=0-7
/proc/2389/task/2398/stat : 2398 (plingeling) R 2387 2389 1932 0 -1 4202560 26447 1252 0 0 5068 4 0 0 25 0 9 0 721408887 483860480 67834 33554432000 4194304 4417452 140737130684976 18446744073709551615 4232623 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2389/tid=2399] ppid=2387 vsize=472520 CPUtime=50.7 cores=0-7
/proc/2389/task/2399/stat : 2399 (plingeling) R 2387 2389 1932 0 -1 4202560 16099 1252 0 0 5068 2 0 0 25 0 9 0 721408887 483860480 67834 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=2389/tid=2400] ppid=2387 vsize=472520 CPUtime=50.71 cores=0-7
/proc/2389/task/2400/stat : 2400 (plingeling) R 2387 2389 1932 0 -1 4202560 24896 1252 0 0 5068 3 0 0 25 0 9 0 721408887 483860480 67834 33554432000 4194304 4417452 140737130684976 18446744073709551615 4357204 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=2389/tid=2401] ppid=2387 vsize=472520 CPUtime=49.87 cores=0-7
/proc/2389/task/2401/stat : 2401 (plingeling) R 2387 2389 1932 0 -1 4202560 23457 1252 0 0 4984 3 0 0 25 0 9 0 721408887 483860480 67834 33554432000 4194304 4417452 140737130684976 18446744073709551615 4232664 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2389/tid=2402] ppid=2387 vsize=472520 CPUtime=50.7 cores=0-7
/proc/2389/task/2402/stat : 2402 (plingeling) R 2387 2389 1932 0 -1 4202560 25712 1252 0 0 5067 3 0 0 25 0 9 0 721408887 483860480 67834 33554432000 4194304 4417452 140737130684976 18446744073709551615 4253228 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2389/tid=2403] ppid=2387 vsize=472520 CPUtime=50.71 cores=0-7
/proc/2389/task/2403/stat : 2403 (plingeling) R 2387 2389 1932 0 -1 4202560 23964 1252 0 0 5069 2 0 0 25 0 9 0 721408887 483860480 67834 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364690 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 405.29
Current children cumulated vsize (KiB) 472520

[startup+102.301 s]
/proc/loadavg: 8.03 8.06 7.98 9/519 2405

################
# More data... #
################

[pid=2389/tid=2398] ppid=2387 vsize=8604024 CPUtime=4721.98 cores=0-7
/proc/2389/task/2398/stat : 2398 (plingeling) R 2387 2389 1932 0 -1 4202560 2416064 1252 0 0 471756 442 0 0 25 0 9 0 721408887 8810520576 2097968 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2389/tid=2399] ppid=2387 vsize=8604024 CPUtime=4720.54 cores=0-7
/proc/2389/task/2399/stat : 2399 (plingeling) R 2387 2389 1932 0 -1 4202560 538038 1252 0 0 471931 123 0 0 25 0 9 0 721408887 8810520576 2097968 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=2389/tid=2400] ppid=2387 vsize=8604024 CPUtime=4722.11 cores=0-7
/proc/2389/task/2400/stat : 2400 (plingeling) R 2387 2389 1932 0 -1 4202560 3028509 1252 0 0 471628 583 0 0 25 0 9 0 721408887 8810520576 2097968 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=2389/tid=2401] ppid=2387 vsize=8604024 CPUtime=4706.22 cores=0-7
/proc/2389/task/2401/stat : 2401 (plingeling) R 2387 2389 1932 0 -1 4202560 1862838 1252 0 0 470281 341 0 0 25 0 9 0 721408887 8810520576 2097968 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2389/tid=2402] ppid=2387 vsize=8604024 CPUtime=4722.12 cores=0-7
/proc/2389/task/2402/stat : 2402 (plingeling) R 2387 2389 1932 0 -1 4202560 745308 1252 0 0 472069 143 0 0 25 0 9 0 721408887 8810520576 2097968 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2389/tid=2403] ppid=2387 vsize=8604024 CPUtime=4721.7 cores=0-7
/proc/2389/task/2403/stat : 2403 (plingeling) R 2387 2389 1932 0 -1 4202560 2910773 1252 0 0 471727 443 0 0 25 0 9 0 721408887 8810520576 2097968 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 37692.7
Current children cumulated vsize (KiB) 8604024

[startup+4782.3 s]
/proc/loadavg: 8.07 8.06 8.01 9/519 2546
/proc/meminfo: memFree=327848/32950928 swapFree=45211384/67111528
[pid=2389] ppid=2387 vsize=8904428 CPUtime=38170.3 cores=0-7
/proc/2389/stat : 2389 (plingeling) S 2387 2389 1932 0 -1 4202496 17102580 1252 0 0 3813912 3120 0 0 18 0 9 0 721408850 9118134272 2173069 33554432000 4194304 4417452 140737130684976 18446744073709551615 254512494565 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/2389/statm: 2226107 2173069 180 55 0 2223993 0
[pid=2389/tid=2396] ppid=2387 vsize=8904428 CPUtime=4782.04 cores=0-7
/proc/2389/task/2396/stat : 2396 (plingeling) R 2387 2389 1932 0 -1 4202560 2687697 1252 0 0 477729 475 0 0 25 0 9 0 721408887 9118134272 2173069 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=2389/tid=2397] ppid=2387 vsize=8904428 CPUtime=4714.07 cores=0-7
/proc/2389/task/2397/stat : 2397 (plingeling) R 2387 2389 1932 0 -1 4202560 2745665 1252 0 0 470876 531 0 0 25 0 9 0 721408887 9118134272 2173069 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 7 0 0 33
[pid=2389/tid=2398] ppid=2387 vsize=8904428 CPUtime=4781.99 cores=0-7
/proc/2389/task/2398/stat : 2398 (plingeling) R 2387 2389 1932 0 -1 4202560 2416094 1252 0 0 477757 442 0 0 25 0 9 0 721408887 9118134272 2173069 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2389/tid=2399] ppid=2387 vsize=8904428 CPUtime=4780.39 cores=0-7
/proc/2389/task/2399/stat : 2399 (plingeling) R 2387 2389 1932 0 -1 4202560 538038 1252 0 0 477916 123 0 0 25 0 9 0 721408887 9118134272 2173069 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2389/tid=2400] ppid=2387 vsize=8904428 CPUtime=4781.94 cores=0-7
/proc/2389/task/2400/stat : 2400 (plingeling) R 2387 2389 1932 0 -1 4202560 3178010 1252 0 0 477582 612 0 0 25 0 9 0 721408887 9118134272 2173069 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=2389/tid=2401] ppid=2387 vsize=8904428 CPUtime=4766.22 cores=0-7
/proc/2389/task/2401/stat : 2401 (plingeling) R 2387 2389 1932 0 -1 4202560 1862838 1252 0 0 476281 341 0 0 25 0 9 0 721408887 9118134272 2173069 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2389/tid=2402] ppid=2387 vsize=8904428 CPUtime=4781.87 cores=0-7
/proc/2389/task/2402/stat : 2402 (plingeling) R 2387 2389 1932 0 -1 4202560 745308 1252 0 0 478044 143 0 0 25 0 9 0 721408887 9118134272 2173069 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2389/tid=2403] ppid=2387 vsize=8904428 CPUtime=4781.35 cores=0-7
/proc/2389/task/2403/stat : 2403 (plingeling) R 2387 2389 1932 0 -1 4202560 2910773 1252 0 0 477688 447 0 0 25 0 9 0 721408887 9118134272 2173069 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38170.3
Current children cumulated vsize (KiB) 8904428

[startup+4842.3 s]
/proc/loadavg: 8.07 8.06 8.01 9/519 2548
/proc/meminfo: memFree=1072120/32950928 swapFree=45211384/67111528
[pid=2389] ppid=2387 vsize=8161412 CPUtime=38649.3 cores=0-7
/proc/2389/stat : 2389 (plingeling) S 2387 2389 1932 0 -1 4202496 17293152 1252 0 0 3861770 3157 0 0 18 0 9 0 721408850 8357285888 1987315 33554432000 4194304 4417452 140737130684976 18446744073709551615 254512494565 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/2389/statm: 2040353 1987315 180 55 0 2038239 0
[pid=2389/tid=2396] ppid=2387 vsize=8161412 CPUtime=4842.05 cores=0-7
/proc/2389/task/2396/stat : 2396 (plingeling) R 2387 2389 1932 0 -1 4202560 2687697 1252 0 0 483729 476 0 0 25 0 9 0 721408887 8357285888 1987315 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=2389/tid=2397] ppid=2387 vsize=8161412 CPUtime=4773.03 cores=0-7
/proc/2389/task/2397/stat : 2397 (plingeling) R 2387 2389 1932 0 -1 4202560 2745665 1252 0 0 476769 534 0 0 25 0 9 0 721408887 8357285888 1987315 33554432000 4194304 4417452 140737130684976 18446744073709551615 4260714 0 0 4096 17442 0 0 0 -1 7 0 0 33
[pid=2389/tid=2398] ppid=2387 vsize=8161412 CPUtime=4841.99 cores=0-7
/proc/2389/task/2398/stat : 2398 (plingeling) R 2387 2389 1932 0 -1 4202560 2416094 1252 0 0 483757 442 0 0 25 0 9 0 721408887 8357285888 1987315 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2389/tid=2399] ppid=2387 vsize=8161412 CPUtime=4840.4 cores=0-7
/proc/2389/task/2399/stat : 2399 (plingeling) R 2387 2389 1932 0 -1 4202560 538038 1252 0 0 483917 123 0 0 25 0 9 0 721408887 8357285888 1987315 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2389/tid=2400] ppid=2387 vsize=8161412 CPUtime=4841.94 cores=0-7
/proc/2389/task/2400/stat : 2400 (plingeling) R 2387 2389 1932 0 -1 4202560 3243546 1252 0 0 483575 619 0 0 25 0 9 0 721408887 8357285888 1987315 33554432000 4194304 4417452 140737130684976 18446744073709551615 4237290 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=2389/tid=2401] ppid=2387 vsize=8161412 CPUtime=4826.22 cores=0-7
/proc/2389/task/2401/stat : 2401 (plingeling) R 2387 2389 1932 0 -1 4202560 1877996 1252 0 0 482277 345 0 0 25 0 9 0 721408887 8357285888 1987315 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2389/tid=2402] ppid=2387 vsize=8161412 CPUtime=4841.88 cores=0-7
/proc/2389/task/2402/stat : 2402 (plingeling) R 2387 2389 1932 0 -1 4202560 761220 1252 0 0 484041 147 0 0 25 0 9 0 721408887 8357285888 1987315 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364711 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2389/tid=2403] ppid=2387 vsize=8161412 CPUtime=4841.33 cores=0-7
/proc/2389/task/2403/stat : 2403 (plingeling) R 2387 2389 1932 0 -1 4202560 3004739 1252 0 0 483667 466 0 0 25 0 9 0 721408887 8357285888 1987315 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38649.3
Current children cumulated vsize (KiB) 8161412

[startup+4902.3 s]
/proc/loadavg: 8.11 8.08 8.02 9/519 2549
/proc/meminfo: memFree=619788/32950928 swapFree=45211384/67111528
[pid=2389] ppid=2387 vsize=8614748 CPUtime=39128.3 cores=0-7
/proc/2389/stat : 2389 (plingeling) S 2387 2389 1932 0 -1 4202496 17521406 1252 0 0 3909619 3210 0 0 18 0 9 0 721408850 8821501952 2100649 33554432000 4194304 4417452 140737130684976 18446744073709551615 254512494565 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/2389/statm: 2153687 2100649 180 55 0 2151573 0
[pid=2389/tid=2396] ppid=2387 vsize=8614748 CPUtime=4902.07 cores=0-7
/proc/2389/task/2396/stat : 2396 (plingeling) R 2387 2389 1932 0 -1 4202560 2753233 1252 0 0 489723 484 0 0 25 0 9 0 721408887 8821501952 2100649 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=2389/tid=2397] ppid=2387 vsize=8614748 CPUtime=4832 cores=0-7
/proc/2389/task/2397/stat : 2397 (plingeling) R 2387 2389 1932 0 -1 4202560 2824443 1252 0 0 482646 554 0 0 25 0 9 0 721408887 8821501952 2100649 33554432000 4194304 4417452 140737130684976 18446744073709551615 4366113 0 0 4096 17442 0 0 0 -1 7 0 0 33
[pid=2389/tid=2398] ppid=2387 vsize=8614748 CPUtime=4902.01 cores=0-7
/proc/2389/task/2398/stat : 2398 (plingeling) R 2387 2389 1932 0 -1 4202560 2416094 1252 0 0 489759 442 0 0 25 0 9 0 721408887 8821501952 2100649 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2389/tid=2399] ppid=2387 vsize=8614748 CPUtime=4900.42 cores=0-7
/proc/2389/task/2399/stat : 2399 (plingeling) R 2387 2389 1932 0 -1 4202560 538038 1252 0 0 489919 123 0 0 25 0 9 0 721408887 8821501952 2100649 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2389/tid=2400] ppid=2387 vsize=8614748 CPUtime=4901.96 cores=0-7
/proc/2389/task/2400/stat : 2400 (plingeling) R 2387 2389 1932 0 -1 4202560 3243546 1252 0 0 489577 619 0 0 25 0 9 0 721408887 8821501952 2100649 33554432000 4194304 4417452 140737130684976 18446744073709551615 4365058 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=2389/tid=2401] ppid=2387 vsize=8614748 CPUtime=4886.25 cores=0-7
/proc/2389/task/2401/stat : 2401 (plingeling) R 2387 2389 1932 0 -1 4202560 1961936 1252 0 0 488255 370 0 0 25 0 9 0 721408887 8821501952 2100649 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2389/tid=2402] ppid=2387 vsize=8614748 CPUtime=4901.91 cores=0-7
/proc/2389/task/2402/stat : 2402 (plingeling) R 2387 2389 1932 0 -1 4202560 761220 1252 0 0 490044 147 0 0 25 0 9 0 721408887 8821501952 2100649 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2389/tid=2403] ppid=2387 vsize=8614748 CPUtime=4901.34 cores=0-7
/proc/2389/task/2403/stat : 2403 (plingeling) R 2387 2389 1932 0 -1 4202560 3004739 1252 0 0 489668 466 0 0 25 0 9 0 721408887 8821501952 2100649 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39128.3
Current children cumulated vsize (KiB) 8614748

[startup+4962.3 s]
/proc/loadavg: 8.08 8.08 8.02 9/519 2551
/proc/meminfo: memFree=94136/32950928 swapFree=45211384/67111528
[pid=2389] ppid=2387 vsize=9139132 CPUtime=39607.2 cores=0-7
/proc/2389/stat : 2389 (plingeling) S 2387 2389 1932 0 -1 4202496 17652505 1252 0 0 3957496 3225 0 0 18 0 9 0 721408850 9358471168 2231745 33554432000 4194304 4417452 140737130684976 18446744073709551615 254512494565 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/2389/statm: 2284783 2231745 180 55 0 2282669 0
[pid=2389/tid=2396] ppid=2387 vsize=9139132 CPUtime=4962.05 cores=0-7
/proc/2389/task/2396/stat : 2396 (plingeling) R 2387 2389 1932 0 -1 4202560 2753233 1252 0 0 495721 484 0 0 25 0 9 0 721408887 9358471168 2231745 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=2389/tid=2397] ppid=2387 vsize=9139132 CPUtime=4890.96 cores=0-7
/proc/2389/task/2397/stat : 2397 (plingeling) R 2387 2389 1932 0 -1 4202560 2889979 1252 0 0 488533 563 0 0 25 0 9 0 721408887 9358471168 2231745 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 33
[pid=2389/tid=2398] ppid=2387 vsize=9139132 CPUtime=4961.99 cores=0-7
/proc/2389/task/2398/stat : 2398 (plingeling) R 2387 2389 1932 0 -1 4202560 2416094 1252 0 0 495757 442 0 0 25 0 9 0 721408887 9358471168 2231745 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2389/tid=2399] ppid=2387 vsize=9139132 CPUtime=4960.4 cores=0-7
/proc/2389/task/2399/stat : 2399 (plingeling) R 2387 2389 1932 0 -1 4202560 538038 1252 0 0 495917 123 0 0 25 0 9 0 721408887 9358471168 2231745 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2389/tid=2400] ppid=2387 vsize=9139132 CPUtime=4961.95 cores=0-7
/proc/2389/task/2400/stat : 2400 (plingeling) R 2387 2389 1932 0 -1 4202560 3243573 1252 0 0 495576 619 0 0 25 0 9 0 721408887 9358471168 2231745 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=2389/tid=2401] ppid=2387 vsize=9139132 CPUtime=4946.23 cores=0-7
/proc/2389/task/2401/stat : 2401 (plingeling) R 2387 2389 1932 0 -1 4202560 2027472 1252 0 0 494247 376 0 0 25 0 9 0 721408887 9358471168 2231745 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2389/tid=2402] ppid=2387 vsize=9139132 CPUtime=4961.89 cores=0-7
/proc/2389/task/2402/stat : 2402 (plingeling) R 2387 2389 1932 0 -1 4202560 761220 1252 0 0 496042 147 0 0 25 0 9 0 721408887 9358471168 2231745 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2389/tid=2403] ppid=2387 vsize=9139132 CPUtime=4961.31 cores=0-7
/proc/2389/task/2403/stat : 2403 (plingeling) R 2387 2389 1932 0 -1 4202560 3004739 1252 0 0 495665 466 0 0 25 0 9 0 721408887 9358471168 2231745 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364838 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39607.2
Current children cumulated vsize (KiB) 9139132



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5011.9 s]
/proc/loadavg: 8.16 8.11 8.03 9/519 2552
/proc/meminfo: memFree=102088/32950928 swapFree=44933860/67111528
[pid=2389] ppid=2387 vsize=9401276 CPUtime=40000.3 cores=0-7
/proc/2389/stat : 2389 (plingeling) S 2387 2389 1932 0 -1 4202496 17718041 1252 0 0 3996733 3299 0 0 18 0 9 0 721408850 9626906624 2297281 33554432000 4194304 4417452 140737130684976 18446744073709551615 254512494565 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/2389/statm: 2350319 2297281 180 55 0 2348205 0
[pid=2389/tid=2396] ppid=2387 vsize=9401276 CPUtime=5010.89 cores=0-7
/proc/2389/task/2396/stat : 2396 (plingeling) R 2387 2389 1932 0 -1 4202560 2753233 1252 0 0 500604 485 0 0 25 0 9 0 721408887 9626906624 2297281 33554432000 4194304 4417452 140737130684976 18446744073709551615 4357902 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=2389/tid=2397] ppid=2387 vsize=9401276 CPUtime=4940.23 cores=0-7
/proc/2389/task/2397/stat : 2397 (plingeling) R 2387 2389 1932 0 -1 4202560 2889979 1252 0 0 493460 563 0 0 25 0 9 0 721408887 9626906624 2297281 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 33
[pid=2389/tid=2398] ppid=2387 vsize=9401276 CPUtime=5011.58 cores=0-7
/proc/2389/task/2398/stat : 2398 (plingeling) R 2387 2389 1932 0 -1 4202560 2416094 1252 0 0 500715 443 0 0 25 0 9 0 721408887 9626906624 2297281 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=2389/tid=2399] ppid=2387 vsize=9401276 CPUtime=5009.87 cores=0-7
/proc/2389/task/2399/stat : 2399 (plingeling) R 2387 2389 1932 0 -1 4202560 538038 1252 0 0 500864 123 0 0 25 0 9 0 721408887 9626906624 2297281 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2389/tid=2400] ppid=2387 vsize=9401276 CPUtime=5011.39 cores=0-7
/proc/2389/task/2400/stat : 2400 (plingeling) R 2387 2389 1932 0 -1 4202560 3243573 1252 0 0 500520 619 0 0 25 0 9 0 721408887 9626906624 2297281 33554432000 4194304 4417452 140737130684976 18446744073709551615 4365023 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=2389/tid=2401] ppid=2387 vsize=9401276 CPUtime=4995.79 cores=0-7
/proc/2389/task/2401/stat : 2401 (plingeling) R 2387 2389 1932 0 -1 4202560 2027472 1252 0 0 499203 376 0 0 25 0 9 0 721408887 9626906624 2297281 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2389/tid=2402] ppid=2387 vsize=9401276 CPUtime=5010.56 cores=0-7
/proc/2389/task/2402/stat : 2402 (plingeling) R 2387 2389 1932 0 -1 4202560 761220 1252 0 0 500909 147 0 0 25 0 9 0 721408887 9626906624 2297281 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2389/tid=2403] ppid=2387 vsize=9401276 CPUtime=5009.65 cores=0-7
/proc/2389/task/2403/stat : 2403 (plingeling) R 2387 2389 1932 0 -1 4202560 3070275 1252 0 0 500428 537 0 0 25 0 9 0 721408887 9626906624 2297281 33554432000 4194304 4417452 140737130684976 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 4 0 0 102
Current children cumulated CPU time (s) 40000.3
Current children cumulated vsize (KiB) 9401276

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5012.1 s]
/proc/loadavg: 8.16 8.11 8.03 9/519 2552
/proc/meminfo: memFree=102088/32950928 swapFree=44933860/67111528
[pid=2389] ppid=2387 vsize=0 CPUtime=40000.6 cores=0-7
/proc/2389/stat : 2389 (plingeling) Z 2387 2389 1932 0 -1 4203548 17718041 1252 0 0 3996747 3317 0 0 18 0 2 0 721408850 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 6 0 0 0
/proc/2389/statm: 0 0 0 0 0 0 0
[pid=2389/tid=2402] ppid=2387 vsize=0 CPUtime=5010.74 cores=0-7
/proc/2389/task/2402/stat : 2402 (plingeling) R 2387 2389 1932 0 -1 4203588 761220 1252 0 0 500910 164 0 0 25 0 2 0 721408887 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 40000.6
Current children cumulated vsize (KiB) 0

[startup+5012.2 s]
/proc/loadavg: 8.16 8.11 8.03 9/519 2552
/proc/meminfo: memFree=102088/32950928 swapFree=44933860/67111528
[pid=2389] ppid=2387 vsize=0 CPUtime=40000.7 cores=0-7
/proc/2389/stat : 2389 (plingeling) Z 2387 2389 1932 0 -1 4203548 17718041 1252 0 0 3996747 3327 0 0 18 0 2 0 721408850 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 6 0 0 0
/proc/2389/statm: 0 0 0 0 0 0 0
[pid=2389/tid=2402] ppid=2387 vsize=0 CPUtime=5010.84 cores=0-7
/proc/2389/task/2402/stat : 2402 (plingeling) R 2387 2389 1932 0 -1 4203588 761220 1252 0 0 500910 174 0 0 25 0 2 0 721408887 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 40000.7
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5012.25
CPU time (s): 40000.8
CPU user time (s): 39967.5
CPU system time (s): 33.3279
CPU usage (%): 798.06
Max. virtual memory (cumulated for all children) (KiB): 9401276

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39967.5
system time used= 33.3279
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 17719293
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 435
involuntary context switches= 193483

runsolver used 23.9174 second user time and 62.6295 second system time

The end

Launcher Data 

Begin job on node149 at 2011-05-22 23:02:02
IDJOB=3394494
IDBENCH=83191
IDSOLVER=1851
FILE ID=node149/3394494-1306098122
RUNJOBID= node149-1306093720-1946
PBS_JOBID= 13387951
Free space on /tmp= 73412 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT11/crafted/skvortsov/automata-synchronization/crn_40_1521_s.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3394494-1306098122/watcher-3394494-1306098122 -o /tmp/evaluation-result-3394494-1306098122/solver-3394494-1306098122 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3394494-1306098122.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 4e35cb9894b1ca841d462da2d6957cc5
RANDOM SEED=713863592

node149.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.73
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:       8872832 kB
Buffers:        222484 kB
Cached:        5042208 kB
SwapCached:      36428 kB
Active:       11503536 kB
Inactive:     12251684 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:       8872832 kB
SwapTotal:    67111528 kB
SwapFree:     45358860 kB
Dirty:            2728 kB
Writeback:           4 kB
AnonPages:    18467204 kB
Mapped:          31736 kB
Slab:           132388 kB
PageTables:     117264 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 173084880 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73412 MiB
End job on node149 at 2011-05-23 00:25:36