Trace number 3309957

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587? (TO) 40001.2 5004.52

General information on the benchmark

NameSAT03/industrial/li/li-exam/
li-exam-61.shuffled-as.sat03-366.cnf
MD5SUM3be662940f394fa876137a905247d755
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables28147
Number of clauses108436
Sum of the clauses size373110
Maximum clause length166
Minimum clause length1
Number of clauses of size 114
Number of clauses of size 218786
Number of clauses of size 351588
Number of clauses of size 437440
Number of clauses of size 53
Number of clauses of size over 5605

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3309957-1304016603/watcher-3309957-1304016603 -o /tmp/evaluation-result-3309957-1304016603/solver-3309957-1304016603 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3309957-1304016603.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.04 7.01 7.00 2/170 16529
/proc/meminfo: memFree=12818508/32950928 swapFree=67111524/67111528
[pid=16529] ppid=16527 vsize=13836 CPUtime=0 cores=0-7
/proc/16529/stat : 16529 (plingeling) R 16527 16529 16061 0 -1 4202496 1525 1261 0 0 0 0 0 0 23 0 1 0 591890968 14168064 1454 33554432000 4194304 4417332 140736023020368 18446744073709551615 256415088395 0 0 4096 17442 0 0 0 17 6 0 0 0
/proc/16529/statm: 3459 1454 147 55 0 1345 0

[startup+0.0218779 s]
/proc/loadavg: 7.04 7.01 7.00 2/170 16529
/proc/meminfo: memFree=12818508/32950928 swapFree=67111524/67111528
[pid=16529] ppid=16527 vsize=17788 CPUtime=0.01 cores=0-7
/proc/16529/stat : 16529 (plingeling) R 16527 16529 16061 0 -1 4202496 2524 1261 0 0 1 0 0 0 23 0 1 0 591890968 18214912 2453 33554432000 4194304 4417332 140736023020368 18446744073709551615 4264920 0 0 4096 17442 0 0 0 17 6 0 0 0
/proc/16529/statm: 4447 2453 148 55 0 2333 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 17788

[startup+0.100836 s]
/proc/loadavg: 7.04 7.01 7.00 2/170 16529
/proc/meminfo: memFree=12818508/32950928 swapFree=67111524/67111528
[pid=16529] ppid=16527 vsize=33496 CPUtime=0.08 cores=0-7
/proc/16529/stat : 16529 (plingeling) R 16527 16529 16061 0 -1 4202496 6467 1261 0 0 8 0 0 0 23 0 1 0 591890968 34299904 6396 33554432000 4194304 4417332 140736023020368 18446744073709551615 256415391488 0 0 4096 17442 0 0 0 17 6 0 0 0
/proc/16529/statm: 8374 6396 148 55 0 6260 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 33496

[startup+0.300806 s]
/proc/loadavg: 7.04 7.01 7.00 2/170 16529
/proc/meminfo: memFree=12818508/32950928 swapFree=67111524/67111528
[pid=16529] ppid=16527 vsize=49120 CPUtime=0.29 cores=0-7
/proc/16529/stat : 16529 (plingeling) R 16527 16529 16061 0 -1 4202496 10373 1261 0 0 28 1 0 0 24 0 1 0 591890968 50298880 10302 33554432000 4194304 4417332 140736023020368 18446744073709551615 4274337 0 0 4096 17442 0 0 0 17 6 0 0 0
/proc/16529/statm: 12280 10302 148 55 0 10166 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 49120

[startup+0.700653 s]
/proc/loadavg: 7.04 7.01 7.00 2/170 16529
/proc/meminfo: memFree=12818508/32950928 swapFree=67111524/67111528
[pid=16529] ppid=16527 vsize=224132 CPUtime=1.21 cores=0-7
/proc/16529/stat : 16529 (plingeling) S 16527 16529 16061 0 -1 4202496 21066 1261 0 0 118 3 0 0 18 0 9 0 591890968 229511168 17409 33554432000 4194304 4417332 140736023020368 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/16529/statm: 56033 17409 169 55 0 53919 0
Current children cumulated CPU time (s) 1.21
Current children cumulated vsize (KiB) 224132

[startup+1.5005 s]
/proc/loadavg: 7.04 7.01 7.00 9/179 16543
/proc/meminfo: memFree=12749872/32950928 swapFree=67111524/67111528
[pid=16529] ppid=16527 vsize=225896 CPUtime=7.59 cores=0-7
/proc/16529/stat : 16529 (plingeling) S 16527 16529 16061 0 -1 4202496 25220 1261 0 0 756 3 0 0 18 0 9 0 591890968 231317504 17667 33554432000 4194304 4417332 140736023020368 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/16529/statm: 56474 17667 174 55 0 54360 0
[pid=16529/tid=16536] ppid=16527 vsize=225896 CPUtime=0.87 cores=0-7
/proc/16529/task/16536/stat : 16536 (plingeling) R 16527 16529 16061 0 -1 4202560 1540 1261 0 0 87 0 0 0 25 0 9 0 591891031 231317504 17667 33554432000 4194304 4417332 140736023020368 18446744073709551615 4301955 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=16529/tid=16537] ppid=16527 vsize=225896 CPUtime=0.87 cores=0-7
/proc/16529/task/16537/stat : 16537 (plingeling) R 16527 16529 16061 0 -1 4202560 1078 1261 0 0 87 0 0 0 25 0 9 0 591891031 231317504 17667 33554432000 4194304 4417332 140736023020368 18446744073709551615 4234289 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16529/tid=16538] ppid=16527 vsize=225896 CPUtime=0.87 cores=0-7
/proc/16529/task/16538/stat : 16538 (plingeling) R 16527 16529 16061 0 -1 4202560 917 1261 0 0 87 0 0 0 25 0 9 0 591891031 231317504 17667 33554432000 4194304 4417332 140736023020368 18446744073709551615 4302921 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16529/tid=16539] ppid=16527 vsize=225896 CPUtime=0.87 cores=0-7
/proc/16529/task/16539/stat : 16539 (plingeling) R 16527 16529 16061 0 -1 4202560 473 1261 0 0 87 0 0 0 21 0 9 0 591891031 231317504 17667 33554432000 4194304 4417332 140736023020368 18446744073709551615 4232787 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=16529/tid=16540] ppid=16527 vsize=225896 CPUtime=0.87 cores=0-7
/proc/16529/task/16540/stat : 16540 (plingeling) R 16527 16529 16061 0 -1 4202560 1699 1261 0 0 87 0 0 0 19 0 9 0 591891031 231317504 17667 33554432000 4194304 4417332 140736023020368 18446744073709551615 4310712 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=16529/tid=16541] ppid=16527 vsize=225896 CPUtime=0.86 cores=0-7
/proc/16529/task/16541/stat : 16541 (plingeling) R 16527 16529 16061 0 -1 4202560 1671 1261 0 0 86 0 0 0 21 0 9 0 591891031 231317504 17667 33554432000 4194304 4417332 140736023020368 18446744073709551615 4303592 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=16529/tid=16542] ppid=16527 vsize=225896 CPUtime=0.86 cores=0-7
/proc/16529/task/16542/stat : 16542 (plingeling) R 16527 16529 16061 0 -1 4202560 1016 1261 0 0 86 0 0 0 21 0 9 0 591891031 231317504 17667 33554432000 4194304 4417332 140736023020368 18446744073709551615 4284514 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=16529/tid=16543] ppid=16527 vsize=225896 CPUtime=0.86 cores=0-7
/proc/16529/task/16543/stat : 16543 (plingeling) R 16527 16529 16061 0 -1 4202560 1317 1261 0 0 86 0 0 0 21 0 9 0 591891031 231317504 17667 33554432000 4194304 4417332 140736023020368 18446744073709551615 4234242 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 7.59
Current children cumulated vsize (KiB) 225896

[startup+3.1012 s]
/proc/loadavg: 7.12 7.03 7.01 9/179 16543
/proc/meminfo: memFree=12749004/32950928 swapFree=67111524/67111528
[pid=16529] ppid=16527 vsize=253100 CPUtime=20.39 cores=0-7
/proc/16529/stat : 16529 (plingeling) S 16527 16529 16061 0 -1 4202496 35186 1261 0 0 2034 5 0 0 18 0 9 0 591890968 259174400 24472 33554432000 4194304 4417332 140736023020368 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/16529/statm: 63275 24472 178 55 0 61161 0
[pid=16529/tid=16536] ppid=16527 vsize=253100 CPUtime=2.47 cores=0-7
/proc/16529/task/16536/stat : 16536 (plingeling) R 16527 16529 16061 0 -1 4202560 2564 1261 0 0 247 0 0 0 25 0 9 0 591891031 259174400 24472 33554432000 4194304 4417332 140736023020368 18446744073709551615 4277545 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=16529/tid=16537] ppid=16527 vsize=253100 CPUtime=2.46 cores=0-7
/proc/16529/task/16537/stat : 16537 (plingeling) R 16527 16529 16061 0 -1 4202560 2681 1261 0 0 246 0 0 0 25 0 9 0 591891031 259174400 24472 33554432000 4194304 4417332 140736023020368 18446744073709551615 4271012 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16529/tid=16538] ppid=16527 vsize=253100 CPUtime=2.47 cores=0-7
/proc/16529/task/16538/stat : 16538 (plingeling) R 16527 16529 16061 0 -1 4202560 1944 1261 0 0 247 0 0 0 25 0 9 0 591891031 259174400 24472 33554432000 4194304 4417332 140736023020368 18446744073709551615 4233242 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16529/tid=16539] ppid=16527 vsize=253100 CPUtime=2.47 cores=0-7
/proc/16529/task/16539/stat : 16539 (plingeling) R 16527 16529 16061 0 -1 4202560 473 1261 0 0 247 0 0 0 25 0 9 0 591891031 259174400 24472 33554432000 4194304 4417332 140736023020368 18446744073709551615 4264940 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=16529/tid=16540] ppid=16527 vsize=253100 CPUtime=2.46 cores=0-7
/proc/16529/task/16540/stat : 16540 (plingeling) R 16527 16529 16061 0 -1 4202560 3435 1261 0 0 246 0 0 0 22 0 9 0 591891031 259174400 24472 33554432000 4194304 4417332 140736023020368 18446744073709551615 4278627 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=16529/tid=16541] ppid=16527 vsize=253100 CPUtime=2.45 cores=0-7
/proc/16529/task/16541/stat : 16541 (plingeling) R 16527 16529 16061 0 -1 4202560 3274 1261 0 0 245 0 0 0 25 0 9 0 591891031 259174400 24472 33554432000 4194304 4417332 140736023020368 18446744073709551615 4278271 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=16529/tid=16542] ppid=16527 vsize=253100 CPUtime=2.46 cores=0-7
/proc/16529/task/16542/stat : 16542 (plingeling) R 16527 16529 16061 0 -1 4202560 2040 1261 0 0 246 0 0 0 25 0 9 0 591891031 259174400 24472 33554432000 4194304 4417332 140736023020368 18446744073709551615 4260714 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=16529/tid=16543] ppid=16527 vsize=253100 CPUtime=2.46 cores=0-7
/proc/16529/task/16543/stat : 16543 (plingeling) R 16527 16529 16061 0 -1 4202560 3266 1261 0 0 246 0 0 0 25 0 9 0 591891031 259174400 24472 33554432000 4194304 4417332 140736023020368 18446744073709551615 4346564 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 20.39
Current children cumulated vsize (KiB) 253100

[startup+6.30059 s]
/proc/loadavg: 7.12 7.03 7.01 9/179 16543
/proc/meminfo: memFree=12715652/32950928 swapFree=67111524/67111528
[pid=16529] ppid=16527 vsize=258872 CPUtime=45.97 cores=0-7
/proc/16529/stat : 16529 (plingeling) S 16527 16529 16061 0 -1 4202496 36666 1261 0 0 4591 6 0 0 18 0 9 0 591890968 265084928 25948 33554432000 4194304 4417332 140736023020368 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/16529/statm: 64718 25948 181 55 0 62604 0
[pid=16529/tid=16536] ppid=16527 vsize=258872 CPUtime=5.67 cores=0-7
/proc/16529/task/16536/stat : 16536 (plingeling) R 16527 16529 16061 0 -1 4202560 2775 1261 0 0 567 0 0 0 25 0 9 0 591891031 265084928 25948 33554432000 4194304 4417332 140736023020368 18446744073709551615 4248500 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=16529/tid=16537] ppid=16527 vsize=258872 CPUtime=5.66 cores=0-7
/proc/16529/task/16537/stat : 16537 (plingeling) R 16527 16529 16061 0 -1 4202560 2891 1261 0 0 566 0 0 0 25 0 9 0 591891031 265084928 25948 33554432000 4194304 4417332 140736023020368 18446744073709551615 4235999 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16529/tid=16538] ppid=16527 vsize=258872 CPUtime=5.67 cores=0-7
/proc/16529/task/16538/stat : 16538 (plingeling) R 16527 16529 16061 0 -1 4202560 2159 1261 0 0 567 0 0 0 25 0 9 0 591891031 265084928 25948 33554432000 4194304 4417332 140736023020368 18446744073709551615 4376510 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16529/tid=16539] ppid=16527 vsize=258872 CPUtime=5.67 cores=0-7
/proc/16529/task/16539/stat : 16539 (plingeling) R 16527 16529 16061 0 -1 4202560 474 1261 0 0 567 0 0 0 25 0 9 0 591891031 265084928 25948 33554432000 4194304 4417332 140736023020368 18446744073709551615 4232584 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=16529/tid=16540] ppid=16527 vsize=258872 CPUtime=5.66 cores=0-7
/proc/16529/task/16540/stat : 16540 (plingeling) R 16527 16529 16061 0 -1 4202560 3645 1261 0 0 566 0 0 0 25 0 9 0 591891031 265084928 25948 33554432000 4194304 4417332 140736023020368 18446744073709551615 4252475 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=16529/tid=16541] ppid=16527 vsize=258872 CPUtime=5.63 cores=0-7
/proc/16529/task/16541/stat : 16541 (plingeling) R 16527 16529 16061 0 -1 4202560 3484 1261 0 0 563 0 0 0 25 0 9 0 591891031 265084928 25948 33554432000 4194304 4417332 140736023020368 18446744073709551615 4253415 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=16529/tid=16542] ppid=16527 vsize=258872 CPUtime=5.66 cores=0-7
/proc/16529/task/16542/stat : 16542 (plingeling) R 16527 16529 16061 0 -1 4202560 2253 1261 0 0 566 0 0 0 25 0 9 0 591891031 265084928 25948 33554432000 4194304 4417332 140736023020368 18446744073709551615 4258499 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=16529/tid=16543] ppid=16527 vsize=258872 CPUtime=5.66 cores=0-7
/proc/16529/task/16543/stat : 16543 (plingeling) R 16527 16529 16061 0 -1 4202560 3476 1261 0 0 566 0 0 0 25 0 9 0 591891031 265084928 25948 33554432000 4194304 4417332 140736023020368 18446744073709551615 4248926 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 45.97
Current children cumulated vsize (KiB) 258872

[startup+12.7004 s]
/proc/loadavg: 7.25 7.06 7.02 9/179 16543
/proc/meminfo: memFree=12702648/32950928 swapFree=67111524/67111528
[pid=16529] ppid=16527 vsize=272900 CPUtime=97.14 cores=0-7
/proc/16529/stat : 16529 (plingeling) S 16527 16529 16061 0 -1 4202496 40179 1261 0 0 9707 7 0 0 18 0 9 0 591890968 279449600 29455 33554432000 4194304 4417332 140736023020368 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/16529/statm: 68225 29455 181 55 0 66111 0
[pid=16529/tid=16536] ppid=16527 vsize=272900 CPUtime=12.07 cores=0-7
/proc/16529/task/16536/stat : 16536 (plingeling) R 16527 16529 16061 0 -1 4202560 3185 1261 0 0 1207 0 0 0 25 0 9 0 591891031 279449600 29455 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364498 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=16529/tid=16537] ppid=16527 vsize=272900 CPUtime=12.06 cores=0-7
/proc/16529/task/16537/stat : 16537 (plingeling) R 16527 16529 16061 0 -1 4202560 3299 1261 0 0 1206 0 0 0 25 0 9 0 591891031 279449600 29455 33554432000 4194304 4417332 140736023020368 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16529/tid=16538] ppid=16527 vsize=272900 CPUtime=12.07 cores=0-7
/proc/16529/task/16538/stat : 16538 (plingeling) R 16527 16529 16061 0 -1 4202560 2586 1261 0 0 1207 0 0 0 25 0 9 0 591891031 279449600 29455 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364512 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16529/tid=16539] ppid=16527 vsize=272900 CPUtime=12.07 cores=0-7
/proc/16529/task/16539/stat : 16539 (plingeling) R 16527 16529 16061 0 -1 4202560 1498 1261 0 0 1207 0 0 0 25 0 9 0 591891031 279449600 29455 33554432000 4194304 4417332 140736023020368 18446744073709551615 4264888 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=16529/tid=16540] ppid=16527 vsize=272900 CPUtime=12.06 cores=0-7
/proc/16529/task/16540/stat : 16540 (plingeling) R 16527 16529 16061 0 -1 4202560 3645 1261 0 0 1206 0 0 0 25 0 9 0 591891031 279449600 29455 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364502 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=16529/tid=16541] ppid=16527 vsize=272900 CPUtime=11.99 cores=0-7
/proc/16529/task/16541/stat : 16541 (plingeling) R 16527 16529 16061 0 -1 4202560 3893 1261 0 0 1199 0 0 0 25 0 9 0 591891031 279449600 29455 33554432000 4194304 4417332 140736023020368 18446744073709551615 4342442 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=16529/tid=16542] ppid=16527 vsize=272900 CPUtime=12.06 cores=0-7
/proc/16529/task/16542/stat : 16542 (plingeling) R 16527 16529 16061 0 -1 4202560 2680 1261 0 0 1206 0 0 0 25 0 9 0 591891031 279449600 29455 33554432000 4194304 4417332 140736023020368 18446744073709551615 4265687 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=16529/tid=16543] ppid=16527 vsize=272900 CPUtime=12.06 cores=0-7
/proc/16529/task/16543/stat : 16543 (plingeling) R 16527 16529 16061 0 -1 4202560 3884 1261 0 0 1206 0 0 0 25 0 9 0 591891031 279449600 29455 33554432000 4194304 4417332 140736023020368 18446744073709551615 4264783 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 97.14
Current children cumulated vsize (KiB) 272900

[startup+25.501 s]
/proc/loadavg: 7.45 7.11 7.03 9/179 16543
/proc/meminfo: memFree=12699236/32950928 swapFree=67111524/67111528
[pid=16529] ppid=16527 vsize=280488 CPUtime=199.47 cores=0-7
/proc/16529/stat : 16529 (plingeling) S 16527 16529 16061 0 -1 4202496 42112 1261 0 0 19940 7 0 0 18 0 9 0 591890968 287219712 31352 33554432000 4194304 4417332 140736023020368 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/16529/statm: 70122 31352 181 55 0 68008 0
[pid=16529/tid=16536] ppid=16527 vsize=280488 CPUtime=24.87 cores=0-7
/proc/16529/task/16536/stat : 16536 (plingeling) R 16527 16529 16061 0 -1 4202560 3192 1261 0 0 2487 0 0 0 25 0 9 0 591891031 287219712 31352 33554432000 4194304 4417332 140736023020368 18446744073709551615 4365906 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=16529/tid=16537] ppid=16527 vsize=280488 CPUtime=24.86 cores=0-7
/proc/16529/task/16537/stat : 16537 (plingeling) R 16527 16529 16061 0 -1 4202560 3317 1261 0 0 2486 0 0 0 25 0 9 0 591891031 287219712 31352 33554432000 4194304 4417332 140736023020368 18446744073709551615 4236759 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16529/tid=16538] ppid=16527 vsize=280488 CPUtime=24.87 cores=0-7
/proc/16529/task/16538/stat : 16538 (plingeling) R 16527 16529 16061 0 -1 4202560 2586 1261 0 0 2487 0 0 0 25 0 9 0 591891031 287219712 31352 33554432000 4194304 4417332 140736023020368 18446744073709551615 4342381 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16529/tid=16539] ppid=16527 vsize=280488 CPUtime=24.87 cores=0-7
/proc/16529/task/16539/stat : 16539 (plingeling) R 16527 16529 16061 0 -1 4202560 2365 1261 0 0 2487 0 0 0 25 0 9 0 591891031 287219712 31352 33554432000 4194304 4417332 140736023020368 18446744073709551615 4357874 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=16529/tid=16540] ppid=16527 vsize=280488 CPUtime=24.86 cores=0-7
/proc/16529/task/16540/stat : 16540 (plingeling) R 16527 16529 16061 0 -1 4202560 4215 1261 0 0 2486 0 0 0 25 0 9 0 591891031 287219712 31352 33554432000 4194304 4417332 140736023020368 18446744073709551615 4265465 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=16529/tid=16541] ppid=16527 vsize=280488 CPUtime=24.72 cores=0-7
/proc/16529/task/16541/stat : 16541 (plingeling) R 16527 16529 16061 0 -1 4202560 3893 1261 0 0 2472 0 0 0 25 0 9 0 591891031 287219712 31352 33554432000 4194304 4417332 140736023020368 18446744073709551615 4265408 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=16529/tid=16542] ppid=16527 vsize=280488 CPUtime=24.86 cores=0-7
/proc/16529/task/16542/stat : 16542 (plingeling) R 16527 16529 16061 0 -1 4202560 2680 1261 0 0 2486 0 0 0 25 0 9 0 591891031 287219712 31352 33554432000 4194304 4417332 140736023020368 18446744073709551615 4232473 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=16529/tid=16543] ppid=16527 vsize=280488 CPUtime=24.86 cores=0-7
/proc/16529/task/16543/stat : 16543 (plingeling) R 16527 16529 16061 0 -1 4202560 4355 1261 0 0 2486 0 0 0 25 0 9 0 591891031 287219712 31352 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364565 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 199.47
Current children cumulated vsize (KiB) 280488

[startup+51.1012 s]
/proc/loadavg: 7.64 7.18 7.06 9/179 16544
/proc/meminfo: memFree=12663836/32950928 swapFree=67111524/67111528
[pid=16529] ppid=16527 vsize=311868 CPUtime=404.14 cores=0-7
/proc/16529/stat : 16529 (plingeling) S 16527 16529 16061 0 -1 4202496 62921 1261 0 0 40404 10 0 0 18 0 9 0 591890968 319352832 39197 33554432000 4194304 4417332 140736023020368 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/16529/statm: 77967 39197 181 55 0 75853 0
[pid=16529/tid=16536] ppid=16527 vsize=311868 CPUtime=50.47 cores=0-7
/proc/16529/task/16536/stat : 16536 (plingeling) R 16527 16529 16061 0 -1 4202560 3845 1261 0 0 5047 0 0 0 25 0 9 0 591891031 319352832 39197 33554432000 4194304 4417332 140736023020368 18446744073709551615 4236693 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=16529/tid=16537] ppid=16527 vsize=311868 CPUtime=50.47 cores=0-7
/proc/16529/task/16537/stat : 16537 (plingeling) R 16527 16529 16061 0 -1 4202560 4007 1261 0 0 5046 1 0 0 25 0 9 0 591891031 319352832 39197 33554432000 4194304 4417332 140736023020368 18446744073709551615 4365035 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16529/tid=16538] ppid=16527 vsize=311868 CPUtime=50.47 cores=0-7

################
# More data... #
################

/proc/16529/task/16543/stat : 16543 (plingeling) R 16527 16529 16061 0 -1 4202560 1334116 1261 0 0 471915 263 0 0 25 0 9 0 591891031 9200889856 2207455 33554432000 4194304 4417332 140736023020368 18446744073709551615 4237169 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 37747.9
Current children cumulated vsize (KiB) 8985244

[startup+4782.3 s]
/proc/loadavg: 8.05 8.05 8.00 9/179 16691
/proc/meminfo: memFree=3790916/32950928 swapFree=67111524/67111528
[pid=16529] ppid=16527 vsize=9168364 CPUtime=38227.5 cores=0-7
/proc/16529/stat : 16529 (plingeling) S 16527 16529 16061 0 -1 4202496 11687810 1261 0 0 3820208 2541 0 0 18 0 9 0 591890968 9388404736 2253235 33554432000 4194304 4417332 140736023020368 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/16529/statm: 2292091 2253235 181 55 0 2289977 0
[pid=16529/tid=16536] ppid=16527 vsize=9168364 CPUtime=4781.78 cores=0-7
/proc/16529/task/16536/stat : 16536 (plingeling) R 16527 16529 16061 0 -1 4202560 1383357 1261 0 0 477918 260 0 0 25 0 9 0 591891031 9388404736 2253235 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=16529/tid=16537] ppid=16527 vsize=9168364 CPUtime=4781.13 cores=0-7
/proc/16529/task/16537/stat : 16537 (plingeling) R 16527 16529 16061 0 -1 4202560 1085699 1261 0 0 477898 215 0 0 25 0 9 0 591891031 9388404736 2253235 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16529/tid=16538] ppid=16527 vsize=9168364 CPUtime=4781.06 cores=0-7
/proc/16529/task/16538/stat : 16538 (plingeling) R 16527 16529 16061 0 -1 4202560 1387731 1261 0 0 477782 324 0 0 25 0 9 0 591891031 9388404736 2253235 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16529/tid=16539] ppid=16527 vsize=9168364 CPUtime=4781.83 cores=0-7
/proc/16529/task/16539/stat : 16539 (plingeling) R 16527 16529 16061 0 -1 4202560 2354673 1261 0 0 477529 654 0 0 25 0 9 0 591891031 9388404736 2253235 33554432000 4194304 4417332 140736023020368 18446744073709551615 4357448 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=16529/tid=16540] ppid=16527 vsize=9168364 CPUtime=4781.81 cores=0-7
/proc/16529/task/16540/stat : 16540 (plingeling) R 16527 16529 16061 0 -1 4202560 1292357 1261 0 0 477931 250 0 0 25 0 9 0 591891031 9388404736 2253235 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=16529/tid=16541] ppid=16527 vsize=9168364 CPUtime=4755.75 cores=0-7
/proc/16529/task/16541/stat : 16541 (plingeling) R 16527 16529 16061 0 -1 4202560 1443175 1261 0 0 475276 299 0 0 25 0 9 0 591891031 9388404736 2253235 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364742 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=16529/tid=16542] ppid=16527 vsize=9168364 CPUtime=4781.68 cores=0-7
/proc/16529/task/16542/stat : 16542 (plingeling) R 16527 16529 16061 0 -1 4202560 1391193 1261 0 0 477897 271 0 0 25 0 9 0 591891031 9388404736 2253235 33554432000 4194304 4417332 140736023020368 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=16529/tid=16543] ppid=16527 vsize=9168364 CPUtime=4781.77 cores=0-7
/proc/16529/task/16543/stat : 16543 (plingeling) R 16527 16529 16061 0 -1 4202560 1334116 1261 0 0 477914 263 0 0 25 0 9 0 591891031 9388404736 2253235 33554432000 4194304 4417332 140736023020368 18446744073709551615 4237290 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38227.5
Current children cumulated vsize (KiB) 9168364

[startup+4842.3 s]
/proc/loadavg: 8.13 8.07 8.01 9/179 16693
/proc/meminfo: memFree=3367120/32950928 swapFree=67111524/67111528
[pid=16529] ppid=16527 vsize=9591516 CPUtime=38707.2 cores=0-7
/proc/16529/stat : 16529 (plingeling) S 16527 16529 16061 0 -1 4202496 11809984 1261 0 0 3868149 2568 0 0 18 0 9 0 591890968 9821712384 2359023 33554432000 4194304 4417332 140736023020368 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/16529/statm: 2397879 2359023 181 55 0 2395765 0
[pid=16529/tid=16536] ppid=16527 vsize=9591516 CPUtime=4841.78 cores=0-7
/proc/16529/task/16536/stat : 16536 (plingeling) R 16527 16529 16061 0 -1 4202560 1383357 1261 0 0 483918 260 0 0 25 0 9 0 591891031 9821712384 2359023 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=16529/tid=16537] ppid=16527 vsize=9591516 CPUtime=4841.13 cores=0-7
/proc/16529/task/16537/stat : 16537 (plingeling) R 16527 16529 16061 0 -1 4202560 1085699 1261 0 0 483898 215 0 0 25 0 9 0 591891031 9821712384 2359023 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364750 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16529/tid=16538] ppid=16527 vsize=9591516 CPUtime=4841.07 cores=0-7
/proc/16529/task/16538/stat : 16538 (plingeling) R 16527 16529 16061 0 -1 4202560 1387747 1261 0 0 483783 324 0 0 25 0 9 0 591891031 9821712384 2359023 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16529/tid=16539] ppid=16527 vsize=9591516 CPUtime=4841.82 cores=0-7
/proc/16529/task/16539/stat : 16539 (plingeling) R 16527 16529 16061 0 -1 4202560 2419754 1261 0 0 483513 669 0 0 25 0 9 0 591891031 9821712384 2359023 33554432000 4194304 4417332 140736023020368 18446744073709551615 4357022 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=16529/tid=16540] ppid=16527 vsize=9591516 CPUtime=4841.81 cores=0-7
/proc/16529/task/16540/stat : 16540 (plingeling) R 16527 16529 16061 0 -1 4202560 1292357 1261 0 0 483931 250 0 0 25 0 9 0 591891031 9821712384 2359023 33554432000 4194304 4417332 140736023020368 18446744073709551615 4357783 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=16529/tid=16541] ppid=16527 vsize=9591516 CPUtime=4815.41 cores=0-7
/proc/16529/task/16541/stat : 16541 (plingeling) R 16527 16529 16061 0 -1 4202560 1456869 1261 0 0 481241 300 0 0 25 0 9 0 591891031 9821712384 2359023 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=16529/tid=16542] ppid=16527 vsize=9591516 CPUtime=4841.68 cores=0-7
/proc/16529/task/16542/stat : 16542 (plingeling) R 16527 16529 16061 0 -1 4202560 1407577 1261 0 0 483896 272 0 0 25 0 9 0 591891031 9821712384 2359023 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=16529/tid=16543] ppid=16527 vsize=9591516 CPUtime=4841.77 cores=0-7
/proc/16529/task/16543/stat : 16543 (plingeling) R 16527 16529 16061 0 -1 4202560 1361115 1261 0 0 483905 272 0 0 25 0 9 0 591891031 9821712384 2359023 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364470 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38707.2
Current children cumulated vsize (KiB) 9591516

[startup+4902.3 s]
/proc/loadavg: 8.05 8.05 8.00 9/179 16694
/proc/meminfo: memFree=2478196/32950928 swapFree=67111524/67111528
[pid=16529] ppid=16527 vsize=10521012 CPUtime=39186.9 cores=0-7
/proc/16529/stat : 16529 (plingeling) S 16527 16529 16061 0 -1 4202496 12166860 1261 0 0 3916031 2654 0 0 18 0 9 0 591890968 10773516288 2591397 33554432000 4194304 4417332 140736023020368 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/16529/statm: 2630253 2591397 181 55 0 2628139 0
[pid=16529/tid=16536] ppid=16527 vsize=10521012 CPUtime=4901.78 cores=0-7
/proc/16529/task/16536/stat : 16536 (plingeling) R 16527 16529 16061 0 -1 4202560 1383357 1261 0 0 489918 260 0 0 25 0 9 0 591891031 10773516288 2591397 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=16529/tid=16537] ppid=16527 vsize=10521012 CPUtime=4901.13 cores=0-7
/proc/16529/task/16537/stat : 16537 (plingeling) R 16527 16529 16061 0 -1 4202560 1085699 1261 0 0 489898 215 0 0 25 0 9 0 591891031 10773516288 2591397 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16529/tid=16538] ppid=16527 vsize=10521012 CPUtime=4901.07 cores=0-7
/proc/16529/task/16538/stat : 16538 (plingeling) R 16527 16529 16061 0 -1 4202560 1387747 1261 0 0 489783 324 0 0 25 0 9 0 591891031 10773516288 2591397 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16529/tid=16539] ppid=16527 vsize=10521012 CPUtime=4901.83 cores=0-7
/proc/16529/task/16539/stat : 16539 (plingeling) R 16527 16529 16061 0 -1 4202560 2468906 1261 0 0 489497 686 0 0 25 0 9 0 591891031 10773516288 2591397 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364494 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=16529/tid=16540] ppid=16527 vsize=10521012 CPUtime=4901.81 cores=0-7
/proc/16529/task/16540/stat : 16540 (plingeling) R 16527 16529 16061 0 -1 4202560 1440223 1261 0 0 489896 285 0 0 25 0 9 0 591891031 10773516288 2591397 33554432000 4194304 4417332 140736023020368 18446744073709551615 4259603 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=16529/tid=16541] ppid=16527 vsize=10521012 CPUtime=4875.09 cores=0-7
/proc/16529/task/16541/stat : 16541 (plingeling) R 16527 16529 16061 0 -1 4202560 1456869 1261 0 0 487209 300 0 0 25 0 9 0 591891031 10773516288 2591397 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=16529/tid=16542] ppid=16527 vsize=10521012 CPUtime=4901.68 cores=0-7
/proc/16529/task/16542/stat : 16542 (plingeling) R 16527 16529 16061 0 -1 4202560 1407577 1261 0 0 489896 272 0 0 25 0 9 0 591891031 10773516288 2591397 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364796 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=16529/tid=16543] ppid=16527 vsize=10521012 CPUtime=4901.77 cores=0-7
/proc/16529/task/16543/stat : 16543 (plingeling) R 16527 16529 16061 0 -1 4202560 1520973 1261 0 0 489871 306 0 0 25 0 9 0 591891031 10773516288 2591397 33554432000 4194304 4417332 140736023020368 18446744073709551615 4199435 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39186.8
Current children cumulated vsize (KiB) 10521012

[startup+4962.3 s]
/proc/loadavg: 8.02 8.04 8.00 9/179 16696
/proc/meminfo: memFree=2366100/32950928 swapFree=67111524/67111528
[pid=16529] ppid=16527 vsize=10590124 CPUtime=39666.5 cores=0-7
/proc/16529/stat : 16529 (plingeling) S 16527 16529 16061 0 -1 4202496 12216906 1261 0 0 3963991 2662 0 0 18 0 9 0 591890968 10844286976 2608675 33554432000 4194304 4417332 140736023020368 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/16529/statm: 2647531 2608675 181 55 0 2645417 0
[pid=16529/tid=16536] ppid=16527 vsize=10590124 CPUtime=4961.78 cores=0-7
/proc/16529/task/16536/stat : 16536 (plingeling) R 16527 16529 16061 0 -1 4202560 1383357 1261 0 0 495918 260 0 0 25 0 9 0 591891031 10844286976 2608675 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=16529/tid=16537] ppid=16527 vsize=10590124 CPUtime=4961.12 cores=0-7
/proc/16529/task/16537/stat : 16537 (plingeling) R 16527 16529 16061 0 -1 4202560 1085699 1261 0 0 495897 215 0 0 25 0 9 0 591891031 10844286976 2608675 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16529/tid=16538] ppid=16527 vsize=10590124 CPUtime=4961.07 cores=0-7
/proc/16529/task/16538/stat : 16538 (plingeling) R 16527 16529 16061 0 -1 4202560 1387747 1261 0 0 495783 324 0 0 25 0 9 0 591891031 10844286976 2608675 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364796 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16529/tid=16539] ppid=16527 vsize=10590124 CPUtime=4961.83 cores=0-7
/proc/16529/task/16539/stat : 16539 (plingeling) R 16527 16529 16061 0 -1 4202560 2496943 1261 0 0 495493 690 0 0 25 0 9 0 591891031 10844286976 2608675 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=16529/tid=16540] ppid=16527 vsize=10590124 CPUtime=4961.81 cores=0-7
/proc/16529/task/16540/stat : 16540 (plingeling) R 16527 16529 16061 0 -1 4202560 1444133 1261 0 0 495895 286 0 0 25 0 9 0 591891031 10844286976 2608675 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=16529/tid=16541] ppid=16527 vsize=10590124 CPUtime=4934.76 cores=0-7
/proc/16529/task/16541/stat : 16541 (plingeling) R 16527 16529 16061 0 -1 4202560 1474968 1261 0 0 493174 302 0 0 25 0 9 0 591891031 10844286976 2608675 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364680 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=16529/tid=16542] ppid=16527 vsize=10590124 CPUtime=4961.68 cores=0-7
/proc/16529/task/16542/stat : 16542 (plingeling) R 16527 16529 16061 0 -1 4202560 1407577 1261 0 0 495896 272 0 0 25 0 9 0 591891031 10844286976 2608675 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364491 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=16529/tid=16543] ppid=16527 vsize=10590124 CPUtime=4961.78 cores=0-7
/proc/16529/task/16543/stat : 16543 (plingeling) R 16527 16529 16061 0 -1 4202560 1520973 1261 0 0 495872 306 0 0 25 0 9 0 591891031 10844286976 2608675 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364683 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39666.5
Current children cumulated vsize (KiB) 10590124



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5004.1 s]
/proc/loadavg: 8.01 8.04 8.00 9/179 16697
/proc/meminfo: memFree=2044036/32950928 swapFree=67111524/67111528
[pid=16529] ppid=16527 vsize=10912328 CPUtime=40000.8 cores=0-7
/proc/16529/stat : 16529 (plingeling) S 16527 16529 16061 0 -1 4202496 12297457 1261 0 0 3997398 2677 0 0 18 0 9 0 591890968 11174223872 2689226 33554432000 4194304 4417332 140736023020368 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/16529/statm: 2728082 2689226 181 55 0 2725968 0
[pid=16529/tid=16536] ppid=16527 vsize=10912328 CPUtime=5003.59 cores=0-7
/proc/16529/task/16536/stat : 16536 (plingeling) R 16527 16529 16061 0 -1 4202560 1383357 1261 0 0 500099 260 0 0 25 0 9 0 591891031 11174223872 2689226 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=16529/tid=16537] ppid=16527 vsize=10912328 CPUtime=5002.93 cores=0-7
/proc/16529/task/16537/stat : 16537 (plingeling) R 16527 16529 16061 0 -1 4202560 1102083 1261 0 0 500075 218 0 0 25 0 9 0 591891031 11174223872 2689226 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16529/tid=16538] ppid=16527 vsize=10912328 CPUtime=5002.88 cores=0-7
/proc/16529/task/16538/stat : 16538 (plingeling) R 16527 16529 16061 0 -1 4202560 1387747 1261 0 0 499964 324 0 0 25 0 9 0 591891031 11174223872 2689226 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16529/tid=16539] ppid=16527 vsize=10912328 CPUtime=5003.64 cores=0-7
/proc/16529/task/16539/stat : 16539 (plingeling) R 16527 16529 16061 0 -1 4202560 2519311 1261 0 0 499670 694 0 0 25 0 9 0 591891031 11174223872 2689226 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=16529/tid=16540] ppid=16527 vsize=10912328 CPUtime=5003.62 cores=0-7
/proc/16529/task/16540/stat : 16540 (plingeling) R 16527 16529 16061 0 -1 4202560 1444133 1261 0 0 500076 286 0 0 25 0 9 0 591891031 11174223872 2689226 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=16529/tid=16541] ppid=16527 vsize=10912328 CPUtime=4976.34 cores=0-7
/proc/16529/task/16541/stat : 16541 (plingeling) R 16527 16529 16061 0 -1 4202560 1487356 1261 0 0 497328 306 0 0 25 0 9 0 591891031 11174223872 2689226 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=16529/tid=16542] ppid=16527 vsize=10912328 CPUtime=5003.48 cores=0-7
/proc/16529/task/16542/stat : 16542 (plingeling) R 16527 16529 16061 0 -1 4202560 1436988 1261 0 0 500071 277 0 0 25 0 9 0 591891031 11174223872 2689226 33554432000 4194304 4417332 140736023020368 18446744073709551615 4236713 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=16529/tid=16543] ppid=16527 vsize=10912328 CPUtime=5003.58 cores=0-7
/proc/16529/task/16543/stat : 16543 (plingeling) R 16527 16529 16061 0 -1 4202560 1520973 1261 0 0 500052 306 0 0 25 0 9 0 591891031 11174223872 2689226 33554432000 4194304 4417332 140736023020368 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 40000.8
Current children cumulated vsize (KiB) 10912328

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5004.21 s]
/proc/loadavg: 8.01 8.04 8.00 9/179 16697
/proc/meminfo: memFree=2044036/32950928 swapFree=67111524/67111528
[pid=16529] ppid=16527 vsize=0 CPUtime=40000.9 cores=0-7
/proc/16529/stat : 16529 (plingeling) Z 16527 16529 16061 0 -1 4203548 12297457 1261 0 0 3997402 2687 0 0 18 0 2 0 591890968 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 6 0 0 0
/proc/16529/statm: 0 0 0 0 0 0 0
[pid=16529/tid=16541] ppid=16527 vsize=0 CPUtime=4976.44 cores=0-7
/proc/16529/task/16541/stat : 16541 (plingeling) R 16527 16529 16061 0 -1 4203588 1487356 1261 0 0 497328 316 0 0 25 0 2 0 591891031 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40000.9
Current children cumulated vsize (KiB) 0

[startup+5004.4 s]
/proc/loadavg: 8.01 8.04 8.00 9/179 16697
/proc/meminfo: memFree=2044036/32950928 swapFree=67111524/67111528
[pid=16529] ppid=16527 vsize=0 CPUtime=40001.1 cores=0-7
/proc/16529/stat : 16529 (plingeling) Z 16527 16529 16061 0 -1 4203548 12297457 1261 0 0 3997402 2706 0 0 18 0 2 0 591890968 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 6 0 0 0
/proc/16529/statm: 0 0 0 0 0 0 0
[pid=16529/tid=16541] ppid=16527 vsize=0 CPUtime=4976.63 cores=0-7
/proc/16529/task/16541/stat : 16541 (plingeling) R 16527 16529 16061 0 -1 4203588 1487356 1261 0 0 497328 335 0 0 25 0 2 0 591891031 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40001.1
Current children cumulated vsize (KiB) 0

[startup+5004.5 s]
/proc/loadavg: 8.01 8.04 8.00 9/179 16697
/proc/meminfo: memFree=2044036/32950928 swapFree=67111524/67111528
[pid=16529] ppid=16527 vsize=0 CPUtime=40001.2 cores=0-7
/proc/16529/stat : 16529 (plingeling) Z 16527 16529 16061 0 -1 4203548 12297457 1261 0 0 3997402 2716 0 0 18 0 2 0 591890968 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 6 0 0 0
/proc/16529/statm: 0 0 0 0 0 0 0
[pid=16529/tid=16541] ppid=16527 vsize=0 CPUtime=4976.73 cores=0-7
/proc/16529/task/16541/stat : 16541 (plingeling) R 16527 16529 16061 0 -1 4203588 1487356 1261 0 0 497328 345 0 0 25 0 2 0 591891031 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40001.2
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5004.52
CPU time (s): 40001.2
CPU user time (s): 39974
CPU system time (s): 27.1839
CPU usage (%): 799.302
Max. virtual memory (cumulated for all children) (KiB): 11730216

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39974
system time used= 27.1839
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 12298718
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 361
involuntary context switches= 192691

runsolver used 7.82781 second user time and 19.474 second system time

The end

Launcher Data 

Begin job on node115 at 2011-04-28 20:50:03
IDJOB=3309957
IDBENCH=84223
IDSOLVER=1695
FILE ID=node115/3309957-1304016603
RUNJOBID= node115-1304006040-16075
PBS_JOBID= 13168822
Free space on /tmp= 72732 MiB

SOLVER NAME= Plingeling 587
BENCH NAME= SAT03/industrial/li/li-exam/li-exam-61.shuffled-as.sat03-366.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3309957-1304016603/watcher-3309957-1304016603 -o /tmp/evaluation-result-3309957-1304016603/solver-3309957-1304016603 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3309957-1304016603.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 3be662940f394fa876137a905247d755
RANDOM SEED=804311847

node115.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.62
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      12818788 kB
Buffers:       2313740 kB
Cached:       17193012 kB
SwapCached:          4 kB
Active:        7519572 kB
Inactive:     12054236 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      12818788 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:            2744 kB
Writeback:           0 kB
AnonPages:       66480 kB
Mapped:          14716 kB
Slab:           491836 kB
PageTables:       4284 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   187524 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72728 MiB
End job on node115 at 2011-04-28 22:13:29