Trace number 3438446

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
SArTagnan 2011-05-15 (fixed)? (TO) 40000.6 5012.87

General information on the benchmark

NameCRAFTED/rbsat/random/
forced/rbsat-v2400c266431gyes7.cnf
MD5SUM92e61742efd2e322c59b4442206bebf1
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2400
Number of clauses266431
Sum of the clauses size535112
Maximum clause length32
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2266356
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 575

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 977)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3438446-1306917136/watcher-3438446-1306917136 -o /tmp/evaluation-result-3438446-1306917136/solver-3438446-1306917136 -C 40000 -W 5100 -M 31000 SArTagnan HOME/instance-3438446-1306917136.cnf -threads=8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.97 1.55 1.82 1/551 19569
/proc/meminfo: memFree=28994724/32951124 swapFree=34426104/67111528
[pid=19569] ppid=19567 vsize=25884 CPUtime=0.01 cores=0-7
/proc/19569/stat : 19569 (SArTagnan) R 19567 19569 19543 0 -1 4202496 965 0 0 0 1 0 0 0 22 0 1 0 881944276 26505216 583 33554432000 4194304 4625164 140734789047920 18446744073709551615 4339447 0 0 4096 2 0 0 0 17 4 0 0 0
/proc/19569/statm: 6471 583 296 106 0 1974 0

[startup+0.06801 s]
/proc/loadavg: 0.97 1.55 1.82 1/551 19569
/proc/meminfo: memFree=28994724/32951124 swapFree=34426104/67111528
[pid=19569] ppid=19567 vsize=28820 CPUtime=0.06 cores=0-7
/proc/19569/stat : 19569 (SArTagnan) R 19567 19569 19543 0 -1 4202496 3046 0 0 0 6 0 0 0 22 0 1 0 881944276 29511680 1195 33554432000 4194304 4625164 140734789047920 18446744073709551615 224284925696 0 0 4096 2 0 0 0 17 4 0 0 0
/proc/19569/statm: 7205 1195 296 106 0 2708 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 28820

[startup+0.100994 s]
/proc/loadavg: 0.97 1.55 1.82 1/551 19569
/proc/meminfo: memFree=28994724/32951124 swapFree=34426104/67111528
[pid=19569] ppid=19567 vsize=127128 CPUtime=0.16 cores=0-7
/proc/19569/stat : 19569 (SArTagnan) R 19567 19569 19543 0 -1 4202496 9287 0 0 0 16 0 0 0 18 0 8 0 881944276 130179072 7435 33554432000 4194304 4625164 140734789047920 18446744073709551615 47357196359606 0 0 4096 2 18446744073709551615 0 0 17 4 0 0 0
/proc/19569/statm: 31782 7435 301 106 0 27285 0
Current children cumulated CPU time (s) 0.16
Current children cumulated vsize (KiB) 127128

[startup+0.300933 s]
/proc/loadavg: 0.97 1.55 1.82 1/551 19569
/proc/meminfo: memFree=28994724/32951124 swapFree=34426104/67111528
[pid=19569] ppid=19567 vsize=467908 CPUtime=1.48 cores=0-7
/proc/19569/stat : 19569 (SArTagnan) R 19567 19569 19543 0 -1 4202496 25257 0 0 0 144 4 0 0 18 0 8 0 881944276 479137792 17231 33554432000 4194304 4625164 140734789047920 18446744073709551615 4338597 0 0 4096 2 18446744073709551615 0 0 17 4 0 0 0
/proc/19569/statm: 116977 17231 345 106 0 112480 0
Current children cumulated CPU time (s) 1.48
Current children cumulated vsize (KiB) 467908

[startup+0.700867 s]
/proc/loadavg: 0.97 1.55 1.82 1/551 19569
/proc/meminfo: memFree=28994724/32951124 swapFree=34426104/67111528
[pid=19569] ppid=19567 vsize=474524 CPUtime=4.67 cores=0-7
/proc/19569/stat : 19569 (SArTagnan) R 19567 19569 19543 0 -1 4202496 31650 0 0 0 462 5 0 0 18 0 8 0 881944276 485912576 22090 33554432000 4194304 4625164 140734789047920 18446744073709551615 4512312 0 0 4096 2 18446744073709551615 0 0 17 4 0 0 0
/proc/19569/statm: 118631 22090 350 106 0 114134 0
Current children cumulated CPU time (s) 4.67
Current children cumulated vsize (KiB) 474524

[startup+1.50073 s]
/proc/loadavg: 0.97 1.55 1.82 9/559 19577
/proc/meminfo: memFree=28883316/32951124 swapFree=34426104/67111528
[pid=19569] ppid=19567 vsize=481604 CPUtime=11.05 cores=0-7
/proc/19569/stat : 19569 (SArTagnan) R 19567 19569 19543 0 -1 4202496 40846 0 0 0 1099 6 0 0 19 0 8 0 881944276 493162496 30669 33554432000 4194304 4625164 140734789047920 18446744073709551615 4511904 0 0 4096 2 18446744073709551615 0 0 17 4 0 0 0
/proc/19569/statm: 120401 30670 352 106 0 115904 0
[pid=19569/tid=19571] ppid=19567 vsize=481604 CPUtime=1.41 cores=0-7
/proc/19569/task/19571/stat : 19571 (SArTagnan) R 19567 19569 19543 0 -1 4202560 2983 0 0 0 141 0 0 0 20 0 8 0 881944285 493162496 30670 33554432000 4194304 4625164 140734789047920 18446744073709551615 4464320 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=19569/tid=19572] ppid=19567 vsize=481604 CPUtime=1.31 cores=0-7
/proc/19569/task/19572/stat : 19572 (SArTagnan) R 19567 19569 19543 0 -1 4202560 2953 0 0 0 131 0 0 0 25 0 8 0 881944285 493162496 30670 33554432000 4194304 4625164 140734789047920 18446744073709551615 4469147 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=19569/tid=19573] ppid=19567 vsize=481604 CPUtime=1.18 cores=0-7
/proc/19569/task/19573/stat : 19573 (SArTagnan) R 19567 19569 19543 0 -1 4202560 3090 0 0 0 118 0 0 0 25 0 8 0 881944285 493162496 30670 33554432000 4194304 4625164 140734789047920 18446744073709551615 4546849 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=19569/tid=19574] ppid=19567 vsize=481604 CPUtime=1.4 cores=0-7
/proc/19569/task/19574/stat : 19574 (SArTagnan) R 19567 19569 19543 0 -1 4202560 2697 0 0 0 139 1 0 0 19 0 8 0 881944285 493162496 30670 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351781 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=19569/tid=19575] ppid=19567 vsize=481604 CPUtime=1.4 cores=0-7
/proc/19569/task/19575/stat : 19575 (SArTagnan) R 19567 19569 19543 0 -1 4202560 2269 0 0 0 140 0 0 0 19 0 8 0 881944285 493162496 30670 33554432000 4194304 4625164 140734789047920 18446744073709551615 4340391 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=19569/tid=19576] ppid=19567 vsize=481604 CPUtime=1.4 cores=0-7
/proc/19569/task/19576/stat : 19576 (SArTagnan) R 19567 19569 19543 0 -1 4202560 2900 0 0 0 140 0 0 0 19 0 8 0 881944285 493162496 30670 33554432000 4194304 4625164 140734789047920 18446744073709551615 4352021 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=19569/tid=19577] ppid=19567 vsize=481604 CPUtime=1.4 cores=0-7
/proc/19569/task/19577/stat : 19577 (SArTagnan) R 19567 19569 19543 0 -1 4202560 2888 0 0 0 140 0 0 0 19 0 8 0 881944285 493162496 30670 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 11.05
Current children cumulated vsize (KiB) 481604

[startup+3.10047 s]
/proc/loadavg: 0.97 1.55 1.82 9/559 19577
/proc/meminfo: memFree=28852068/32951124 swapFree=34426104/67111528
[pid=19569] ppid=19567 vsize=508824 CPUtime=23.82 cores=0-7
/proc/19569/stat : 19569 (SArTagnan) R 19567 19569 19543 0 -1 4202496 60287 0 0 0 2374 8 0 0 24 0 8 0 881944276 521035776 43475 33554432000 4194304 4625164 140734789047920 18446744073709551615 4512057 0 0 4096 2 18446744073709551615 0 0 17 4 0 0 0
/proc/19569/statm: 127206 43475 359 106 0 122709 0
[pid=19569/tid=19571] ppid=19567 vsize=508824 CPUtime=3 cores=0-7
/proc/19569/task/19571/stat : 19571 (SArTagnan) R 19567 19569 19543 0 -1 4202560 6098 0 0 0 300 0 0 0 25 0 8 0 881944285 521035776 43475 33554432000 4194304 4625164 140734789047920 18446744073709551615 4352280 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=19569/tid=19572] ppid=19567 vsize=508824 CPUtime=2.89 cores=0-7
/proc/19569/task/19572/stat : 19572 (SArTagnan) R 19567 19569 19543 0 -1 4202560 3546 0 0 0 289 0 0 0 25 0 8 0 881944285 521035776 43475 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351781 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=19569/tid=19573] ppid=19567 vsize=508824 CPUtime=2.78 cores=0-7
/proc/19569/task/19573/stat : 19573 (SArTagnan) R 19567 19569 19543 0 -1 4202560 4620 0 0 0 278 0 0 0 25 0 8 0 881944285 521035776 43475 33554432000 4194304 4625164 140734789047920 18446744073709551615 4547280 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=19569/tid=19574] ppid=19567 vsize=508824 CPUtime=3 cores=0-7
/proc/19569/task/19574/stat : 19574 (SArTagnan) R 19567 19569 19543 0 -1 4202560 3958 0 0 0 299 1 0 0 25 0 8 0 881944285 521035776 43475 33554432000 4194304 4625164 140734789047920 18446744073709551615 4462955 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=19569/tid=19575] ppid=19567 vsize=508824 CPUtime=3 cores=0-7
/proc/19569/task/19575/stat : 19575 (SArTagnan) R 19567 19569 19543 0 -1 4202560 5400 0 0 0 299 1 0 0 18 0 8 0 881944285 521035776 43475 33554432000 4194304 4625164 140734789047920 18446744073709551615 4318609 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=19569/tid=19576] ppid=19567 vsize=508824 CPUtime=3 cores=0-7
/proc/19569/task/19576/stat : 19576 (SArTagnan) R 19567 19569 19543 0 -1 4202560 5790 0 0 0 300 0 0 0 24 0 8 0 881944285 521035776 43475 33554432000 4194304 4625164 140734789047920 18446744073709551615 4317294 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=19569/tid=19577] ppid=19567 vsize=508824 CPUtime=2.99 cores=0-7
/proc/19569/task/19577/stat : 19577 (SArTagnan) R 19567 19569 19543 0 -1 4202560 4166 0 0 0 299 0 0 0 24 0 8 0 881944285 521035776 43475 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 23.82
Current children cumulated vsize (KiB) 508824

[startup+6.30093 s]
/proc/loadavg: 1.53 1.66 1.85 9/559 19577
/proc/meminfo: memFree=28763304/32951124 swapFree=34426104/67111528
[pid=19569] ppid=19567 vsize=524536 CPUtime=49.36 cores=0-7
/proc/19569/stat : 19569 (SArTagnan) R 19567 19569 19543 0 -1 4202496 94683 0 0 0 4921 15 0 0 25 0 8 0 881944276 537124864 59585 33554432000 4194304 4625164 140734789047920 18446744073709551615 4511930 0 0 4096 2 18446744073709551615 0 0 17 4 0 0 0
/proc/19569/statm: 131134 59585 382 106 0 126637 0
[pid=19569/tid=19571] ppid=19567 vsize=524536 CPUtime=6.21 cores=0-7
/proc/19569/task/19571/stat : 19571 (SArTagnan) R 19567 19569 19543 0 -1 4202560 9071 0 0 0 620 1 0 0 25 0 8 0 881944285 537124864 59585 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=19569/tid=19572] ppid=19567 vsize=524536 CPUtime=6.02 cores=0-7
/proc/19569/task/19572/stat : 19572 (SArTagnan) R 19567 19569 19543 0 -1 4202560 5672 0 0 0 602 0 0 0 25 0 8 0 881944285 537124864 59585 33554432000 4194304 4625164 140734789047920 18446744073709551615 4352296 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=19569/tid=19573] ppid=19567 vsize=524536 CPUtime=5.98 cores=0-7
/proc/19569/task/19573/stat : 19573 (SArTagnan) R 19567 19569 19543 0 -1 4202560 16320 0 0 0 596 2 0 0 25 0 8 0 881944285 537124864 59585 33554432000 4194304 4625164 140734789047920 18446744073709551615 4334728 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=19569/tid=19574] ppid=19567 vsize=524536 CPUtime=6.2 cores=0-7
/proc/19569/task/19574/stat : 19574 (SArTagnan) R 19567 19569 19543 0 -1 4202560 6586 0 0 0 619 1 0 0 25 0 8 0 881944285 537124864 59585 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351975 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=19569/tid=19575] ppid=19567 vsize=524536 CPUtime=6.2 cores=0-7
/proc/19569/task/19575/stat : 19575 (SArTagnan) R 19567 19569 19543 0 -1 4202560 8494 0 0 0 618 2 0 0 25 0 8 0 881944285 537124864 59585 33554432000 4194304 4625164 140734789047920 18446744073709551615 224284593154 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=19569/tid=19576] ppid=19567 vsize=524536 CPUtime=6.2 cores=0-7
/proc/19569/task/19576/stat : 19576 (SArTagnan) R 19567 19569 19543 0 -1 4202560 6375 0 0 0 620 0 0 0 25 0 8 0 881944285 537124864 59585 33554432000 4194304 4625164 140734789047920 18446744073709551615 47357193847360 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=19569/tid=19577] ppid=19567 vsize=524536 CPUtime=6.2 cores=0-7
/proc/19569/task/19577/stat : 19577 (SArTagnan) R 19567 19569 19543 0 -1 4202560 6614 0 0 0 619 1 0 0 25 0 8 0 881944285 537124864 59585 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351901 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 49.36
Current children cumulated vsize (KiB) 524536

[startup+12.7009 s]
/proc/loadavg: 2.05 1.76 1.88 9/559 19577
/proc/meminfo: memFree=28676668/32951124 swapFree=34426104/67111528
[pid=19569] ppid=19567 vsize=668156 CPUtime=100.42 cores=0-7
/proc/19569/stat : 19569 (SArTagnan) R 19567 19569 19543 0 -1 4202496 147608 0 0 0 10018 24 0 0 25 0 8 0 881944276 684191744 80737 33554432000 4194304 4625164 140734789047920 18446744073709551615 4511904 0 0 4096 2 18446744073709551615 0 0 17 4 0 0 0
/proc/19569/statm: 167039 80738 433 106 0 162542 0
[pid=19569/tid=19571] ppid=19567 vsize=668156 CPUtime=12.6 cores=0-7
/proc/19569/task/19571/stat : 19571 (SArTagnan) R 19567 19569 19543 0 -1 4202560 10762 0 0 0 1259 1 0 0 25 0 8 0 881944285 684191744 80739 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=19569/tid=19572] ppid=19567 vsize=668156 CPUtime=12.3 cores=0-7
/proc/19569/task/19572/stat : 19572 (SArTagnan) R 19567 19569 19543 0 -1 4202560 8501 0 0 0 1230 0 0 0 25 0 8 0 881944285 684191744 80739 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=19569/tid=19573] ppid=19567 vsize=668156 CPUtime=12.38 cores=0-7
/proc/19569/task/19573/stat : 19573 (SArTagnan) R 19567 19569 19543 0 -1 4202560 40540 0 0 0 1231 7 0 0 25 0 8 0 881944285 684191744 80739 33554432000 4194304 4625164 140734789047920 18446744073709551615 4339447 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=19569/tid=19574] ppid=19567 vsize=668156 CPUtime=12.6 cores=0-7
/proc/19569/task/19574/stat : 19574 (SArTagnan) R 19567 19569 19543 0 -1 4202560 9213 0 0 0 1258 2 0 0 25 0 8 0 881944285 684191744 80740 33554432000 4194304 4625164 140734789047920 18446744073709551615 4352021 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=19569/tid=19575] ppid=19567 vsize=668156 CPUtime=12.6 cores=0-7
/proc/19569/task/19575/stat : 19575 (SArTagnan) R 19567 19569 19543 0 -1 4202560 12685 0 0 0 1258 2 0 0 25 0 8 0 881944285 684191744 80740 33554432000 4194304 4625164 140734789047920 18446744073709551615 4348903 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=19569/tid=19576] ppid=19567 vsize=668156 CPUtime=12.6 cores=0-7
/proc/19569/task/19576/stat : 19576 (SArTagnan) R 19567 19569 19543 0 -1 4202560 10371 0 0 0 1259 1 0 0 25 0 8 0 881944285 684191744 80741 33554432000 4194304 4625164 140734789047920 18446744073709551615 4339639 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=19569/tid=19577] ppid=19567 vsize=668156 CPUtime=12.6 cores=0-7
/proc/19569/task/19577/stat : 19577 (SArTagnan) R 19567 19569 19543 0 -1 4202560 7265 0 0 0 1259 1 0 0 25 0 8 0 881944285 684191744 80741 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 100.42
Current children cumulated vsize (KiB) 668156

[startup+25.5007 s]
/proc/loadavg: 3.37 2.07 1.98 9/559 19577
/proc/meminfo: memFree=28570068/32951124 swapFree=34426104/67111528
[pid=19569] ppid=19567 vsize=623216 CPUtime=202.57 cores=0-7
/proc/19569/stat : 19569 (SArTagnan) R 19567 19569 19543 0 -1 4202496 236192 0 0 0 20217 40 0 0 25 0 8 0 881944276 638173184 109713 33554432000 4194304 4625164 140734789047920 18446744073709551615 4519334 0 0 4096 2 18446744073709551615 0 0 17 4 0 0 0
/proc/19569/statm: 155804 109713 560 106 0 151307 0
[pid=19569/tid=19571] ppid=19567 vsize=623216 CPUtime=25.41 cores=0-7
/proc/19569/task/19571/stat : 19571 (SArTagnan) R 19567 19569 19543 0 -1 4202560 15366 0 0 0 2539 2 0 0 25 0 8 0 881944285 638173184 109713 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351764 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=19569/tid=19572] ppid=19567 vsize=623216 CPUtime=24.85 cores=0-7
/proc/19569/task/19572/stat : 19572 (SArTagnan) R 19567 19569 19543 0 -1 4202560 12510 0 0 0 2483 2 0 0 25 0 8 0 881944285 638173184 109713 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=19569/tid=19573] ppid=19567 vsize=623216 CPUtime=25.18 cores=0-7
/proc/19569/task/19573/stat : 19573 (SArTagnan) R 19567 19569 19543 0 -1 4202560 85542 0 0 0 2504 14 0 0 25 0 8 0 881944285 638173184 109714 33554432000 4194304 4625164 140734789047920 18446744073709551615 4334728 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=19569/tid=19574] ppid=19567 vsize=623216 CPUtime=25.4 cores=0-7
/proc/19569/task/19574/stat : 19574 (SArTagnan) R 19567 19569 19543 0 -1 4202560 13710 0 0 0 2537 3 0 0 25 0 8 0 881944285 638173184 109714 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=19569/tid=19575] ppid=19567 vsize=623216 CPUtime=25.39 cores=0-7
/proc/19569/task/19575/stat : 19575 (SArTagnan) R 19567 19569 19543 0 -1 4202560 18574 0 0 0 2536 3 0 0 25 0 8 0 881944285 638173184 109714 33554432000 4194304 4625164 140734789047920 18446744073709551615 4349061 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=19569/tid=19576] ppid=19567 vsize=623216 CPUtime=25.4 cores=0-7
/proc/19569/task/19576/stat : 19576 (SArTagnan) R 19567 19569 19543 0 -1 4202560 16566 0 0 0 2537 3 0 0 25 0 8 0 881944285 638173184 109714 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=19569/tid=19577] ppid=19567 vsize=623216 CPUtime=25.39 cores=0-7
/proc/19569/task/19577/stat : 19577 (SArTagnan) R 19567 19569 19543 0 -1 4202560 10190 0 0 0 2538 1 0 0 25 0 8 0 881944285 638173184 109714 33554432000 4194304 4625164 140734789047920 18446744073709551615 4352035 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 202.57
Current children cumulated vsize (KiB) 623216

[startup+51.1004 s]
/proc/loadavg: 4.95 2.54 2.14 9/559 19577
/proc/meminfo: memFree=28421488/32951124 swapFree=34426104/67111528
[pid=19569] ppid=19567 vsize=928980 CPUtime=406.86 cores=0-7
/proc/19569/stat : 19569 (SArTagnan) R 19567 19569 19543 0 -1 4202496 362364 0 0 0 40624 62 0 0 25 0 8 0 881944276 951275520 143994 33554432000 4194304 4625164 140734789047920 18446744073709551615 4511904 0 0 4096 2 18446744073709551615 0 0 17 4 0 0 0
/proc/19569/statm: 232245 143994 672 106 0 227748 0
[pid=19569/tid=19571] ppid=19567 vsize=928980 CPUtime=51 cores=0-7
/proc/19569/task/19571/stat : 19571 (SArTagnan) R 19567 19569 19543 0 -1 4202560 23451 0 0 0 5097 3 0 0 25 0 8 0 881944285 951275520 143994 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351777 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=19569/tid=19572] ppid=19567 vsize=928980 CPUtime=49.96 cores=0-7
/proc/19569/task/19572/stat : 19572 (SArTagnan) R 19567 19569 19543 0 -1 4202560 14754 0 0 0 4994 2 0 0 25 0 8 0 881944285 951275520 143994 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=19569/tid=19573] ppid=19567 vsize=928980 CPUtime=50.78 cores=0-7
/proc/19569/task/19573/stat : 19573 (SArTagnan) R 19567 19569 19543 0 -1 4202560 158763 0 0 0 5053 25 0 0 25 0 8 0 881944285 951275520 143994 33554432000 4194304 4625164 140734789047920 18446744073709551615 4375482 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=19569/tid=19574] ppid=19567 vsize=928980 CPUtime=50.99 cores=0-7
/proc/19569/task/19574/stat : 19574 (SArTagnan) R 19567 19569 19543 0 -1 4202560 14765 0 0 0 5096 3 0 0 25 0 8 0 881944285 951275520 143994 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=19569/tid=19575] ppid=19567 vsize=928980 CPUtime=50.99 cores=0-7
/proc/19569/task/19575/stat : 19575 (SArTagnan) R 19567 19569 19543 0 -1 4202560 21760 0 0 0 5095 4 0 0 25 0 8 0 881944285 951275520 143994 33554432000 4194304 4625164 140734789047920 18446744073709551615 4348941 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=19569/tid=19576] ppid=19567 vsize=928980 CPUtime=51 cores=0-7
/proc/19569/task/19576/stat : 19576 (SArTagnan) R 19567 19569 19543 0 -1 4202560 27812 0 0 0 5095 5 0 0 25 0 8 0 881944285 951275520 143994 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=19569/tid=19577] ppid=19567 vsize=928980 CPUtime=50.99 cores=0-7
/proc/19569/task/19577/stat : 19577 (SArTagnan) R 19567 19569 19543 0 -1 4202560 11832 0 0 0 5097 2 0 0 25 0 8 0 881944285 951275520 143994 33554432000 4194304 4625164 140734789047920 18446744073709551615 4352739 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 406.86

################
# More data... #
################

[pid=19569/tid=19573] ppid=19567 vsize=5237780 CPUtime=4641.39 cores=0-7
/proc/19569/task/19573/stat : 19573 (SArTagnan) R 19567 19569 19543 0 -1 4202560 25196240 0 0 0 460058 4081 0 0 25 0 8 0 881944285 5363486720 1066571 33554432000 4194304 4625164 140734789047920 18446744073709551615 4353075 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=19569/tid=19574] ppid=19567 vsize=5237780 CPUtime=4662.23 cores=0-7
/proc/19569/task/19574/stat : 19574 (SArTagnan) R 19567 19569 19543 0 -1 4202560 78789 0 0 0 466200 23 0 0 25 0 8 0 881944285 5363486720 1066571 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=19569/tid=19575] ppid=19567 vsize=5237780 CPUtime=4662.26 cores=0-7
/proc/19569/task/19575/stat : 19575 (SArTagnan) R 19567 19569 19543 0 -1 4202560 134056 0 0 0 466195 31 0 0 25 0 8 0 881944285 5363486720 1066571 33554432000 4194304 4625164 140734789047920 18446744073709551615 4329981 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=19569/tid=19576] ppid=19567 vsize=5237780 CPUtime=4662.26 cores=0-7
/proc/19569/task/19576/stat : 19576 (SArTagnan) R 19567 19569 19543 0 -1 4202560 168646 0 0 0 466186 40 0 0 25 0 8 0 881944285 5363486720 1066571 33554432000 4194304 4625164 140734789047920 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=19569/tid=19577] ppid=19567 vsize=5237780 CPUtime=4596.27 cores=0-7
/proc/19569/task/19577/stat : 19577 (SArTagnan) R 19567 19569 19543 0 -1 4202560 89170 0 0 0 459579 48 0 0 25 0 8 0 881944285 5363486720 1066571 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 37204.4
Current children cumulated vsize (KiB) 5237780

[startup+4722.32 s]
/proc/loadavg: 8.10 8.08 8.01 9/559 19725
/proc/meminfo: memFree=24696180/32951124 swapFree=34426104/67111528
[pid=19569] ppid=19567 vsize=5285780 CPUtime=37683.4 cores=0-7
/proc/19569/stat : 19569 (SArTagnan) R 19567 19569 19543 0 -1 4202496 28091461 0 0 0 3763616 4727 0 0 25 0 8 0 881944276 5412638720 1072633 33554432000 4194304 4625164 140734789047920 18446744073709551615 4511904 0 0 4096 2 18446744073709551615 0 0 17 4 0 0 0
/proc/19569/statm: 1321445 1072633 349 106 0 1316948 0
[pid=19569/tid=19571] ppid=19567 vsize=5285780 CPUtime=4722.28 cores=0-7
/proc/19569/task/19571/stat : 19571 (SArTagnan) R 19567 19569 19543 0 -1 4202560 138095 0 0 0 472197 31 0 0 25 0 8 0 881944285 5412638720 1072633 33554432000 4194304 4625164 140734789047920 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=19569/tid=19572] ppid=19567 vsize=5285780 CPUtime=4716.77 cores=0-7
/proc/19569/task/19572/stat : 19572 (SArTagnan) R 19567 19569 19543 0 -1 4202560 162508 0 0 0 471571 106 0 0 25 0 8 0 881944285 5412638720 1072633 33554432000 4194304 4625164 140734789047920 18446744073709551615 4339646 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=19569/tid=19573] ppid=19567 vsize=5285780 CPUtime=4700.24 cores=0-7
/proc/19569/task/19573/stat : 19573 (SArTagnan) R 19567 19569 19543 0 -1 4202560 25526189 0 0 0 465882 4142 0 0 25 0 8 0 881944285 5412638720 1072633 33554432000 4194304 4625164 140734789047920 18446744073709551615 4334728 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=19569/tid=19574] ppid=19567 vsize=5285780 CPUtime=4722.25 cores=0-7
/proc/19569/task/19574/stat : 19574 (SArTagnan) R 19567 19569 19543 0 -1 4202560 78816 0 0 0 472202 23 0 0 25 0 8 0 881944285 5412638720 1072633 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=19569/tid=19575] ppid=19567 vsize=5285780 CPUtime=4722.28 cores=0-7
/proc/19569/task/19575/stat : 19575 (SArTagnan) R 19567 19569 19543 0 -1 4202560 134449 0 0 0 472197 31 0 0 25 0 8 0 881944285 5412638720 1072633 33554432000 4194304 4625164 140734789047920 18446744073709551615 4329721 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=19569/tid=19576] ppid=19567 vsize=5285780 CPUtime=4722.29 cores=0-7
/proc/19569/task/19576/stat : 19576 (SArTagnan) R 19567 19569 19543 0 -1 4202560 181048 0 0 0 472187 42 0 0 25 0 8 0 881944285 5412638720 1072633 33554432000 4194304 4625164 140734789047920 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=19569/tid=19577] ppid=19567 vsize=5285780 CPUtime=4656.29 cores=0-7
/proc/19569/task/19577/stat : 19577 (SArTagnan) R 19567 19569 19543 0 -1 4202560 89316 0 0 0 465581 48 0 0 25 0 8 0 881944285 5412638720 1072633 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 37683.4
Current children cumulated vsize (KiB) 5285780

[startup+4782.3 s]
/proc/loadavg: 8.11 8.09 8.01 9/559 19727
/proc/meminfo: memFree=24669608/32951124 swapFree=34426104/67111528
[pid=19569] ppid=19567 vsize=5285780 CPUtime=38162.1 cores=0-7
/proc/19569/stat : 19569 (SArTagnan) R 19567 19569 19543 0 -1 4202496 28445006 0 0 0 3811413 4795 0 0 25 0 8 0 881944276 5412638720 1079321 33554432000 4194304 4625164 140734789047920 18446744073709551615 4392981 0 0 4096 2 18446744073709551615 0 0 17 4 0 0 0
/proc/19569/statm: 1321445 1079321 349 106 0 1316948 0
[pid=19569/tid=19571] ppid=19567 vsize=5285780 CPUtime=4782.26 cores=0-7
/proc/19569/task/19571/stat : 19571 (SArTagnan) R 19567 19569 19543 0 -1 4202560 138249 0 0 0 478195 31 0 0 25 0 8 0 881944285 5412638720 1079321 33554432000 4194304 4625164 140734789047920 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=19569/tid=19572] ppid=19567 vsize=5285780 CPUtime=4776.75 cores=0-7
/proc/19569/task/19572/stat : 19572 (SArTagnan) R 19567 19569 19543 0 -1 4202560 163431 0 0 0 477569 106 0 0 25 0 8 0 881944285 5412638720 1079321 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=19569/tid=19573] ppid=19567 vsize=5285780 CPUtime=4759.07 cores=0-7
/proc/19569/task/19573/stat : 19573 (SArTagnan) R 19567 19569 19543 0 -1 4202560 25842236 0 0 0 471704 4203 0 0 25 0 8 0 881944285 5412638720 1079321 33554432000 4194304 4625164 140734789047920 18446744073709551615 4334728 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=19569/tid=19574] ppid=19567 vsize=5285780 CPUtime=4782.23 cores=0-7
/proc/19569/task/19574/stat : 19574 (SArTagnan) R 19567 19569 19543 0 -1 4202560 79353 0 0 0 478200 23 0 0 25 0 8 0 881944285 5412638720 1079321 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=19569/tid=19575] ppid=19567 vsize=5285780 CPUtime=4782.26 cores=0-7
/proc/19569/task/19575/stat : 19575 (SArTagnan) R 19567 19569 19543 0 -1 4202560 134688 0 0 0 478195 31 0 0 25 0 8 0 881944285 5412638720 1079321 33554432000 4194304 4625164 140734789047920 18446744073709551615 4329981 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=19569/tid=19576] ppid=19567 vsize=5285780 CPUtime=4782.27 cores=0-7
/proc/19569/task/19576/stat : 19576 (SArTagnan) R 19567 19569 19543 0 -1 4202560 181267 0 0 0 478185 42 0 0 25 0 8 0 881944285 5412638720 1079321 33554432000 4194304 4625164 140734789047920 18446744073709551615 4330613 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=19569/tid=19577] ppid=19567 vsize=5285780 CPUtime=4716.26 cores=0-7
/proc/19569/task/19577/stat : 19577 (SArTagnan) R 19567 19569 19543 0 -1 4202560 89339 0 0 0 471578 48 0 0 25 0 8 0 881944285 5412638720 1079321 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 38162.1
Current children cumulated vsize (KiB) 5285780

[startup+4842.3 s]
/proc/loadavg: 8.07 8.08 8.01 9/559 19727
/proc/meminfo: memFree=24636308/32951124 swapFree=34426104/67111528
[pid=19569] ppid=19567 vsize=5294908 CPUtime=38640.9 cores=0-7
/proc/19569/stat : 19569 (SArTagnan) R 19567 19569 19543 0 -1 4202496 28700201 0 0 0 3859247 4843 0 0 25 0 8 0 881944276 5421985792 1087777 33554432000 4194304 4625164 140734789047920 18446744073709551615 4511904 0 0 4096 2 18446744073709551615 0 0 17 4 0 0 0
/proc/19569/statm: 1323727 1087777 349 106 0 1319230 0
[pid=19569/tid=19571] ppid=19567 vsize=5294908 CPUtime=4842.26 cores=0-7
/proc/19569/task/19571/stat : 19571 (SArTagnan) R 19567 19569 19543 0 -1 4202560 138482 0 0 0 484195 31 0 0 25 0 8 0 881944285 5421985792 1087777 33554432000 4194304 4625164 140734789047920 18446744073709551615 4488046 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=19569/tid=19572] ppid=19567 vsize=5294908 CPUtime=4836.75 cores=0-7
/proc/19569/task/19572/stat : 19572 (SArTagnan) R 19567 19569 19543 0 -1 4202560 166119 0 0 0 483567 108 0 0 25 0 8 0 881944285 5421985792 1087777 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=19569/tid=19573] ppid=19567 vsize=5294908 CPUtime=4817.9 cores=0-7
/proc/19569/task/19573/stat : 19573 (SArTagnan) R 19567 19569 19543 0 -1 4202560 26072451 0 0 0 477545 4245 0 0 25 0 8 0 881944285 5421985792 1087777 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=19569/tid=19574] ppid=19567 vsize=5294908 CPUtime=4842.23 cores=0-7
/proc/19569/task/19574/stat : 19574 (SArTagnan) R 19567 19569 19543 0 -1 4202560 80195 0 0 0 484200 23 0 0 25 0 8 0 881944285 5421985792 1087777 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351804 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=19569/tid=19575] ppid=19567 vsize=5294908 CPUtime=4842.26 cores=0-7
/proc/19569/task/19575/stat : 19575 (SArTagnan) R 19567 19569 19543 0 -1 4202560 134850 0 0 0 484195 31 0 0 25 0 8 0 881944285 5421985792 1087777 33554432000 4194304 4625164 140734789047920 18446744073709551615 4329673 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=19569/tid=19576] ppid=19567 vsize=5294908 CPUtime=4842.27 cores=0-7
/proc/19569/task/19576/stat : 19576 (SArTagnan) R 19567 19569 19543 0 -1 4202560 181498 0 0 0 484185 42 0 0 25 0 8 0 881944285 5421985792 1087777 33554432000 4194304 4625164 140734789047920 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=19569/tid=19577] ppid=19567 vsize=5294908 CPUtime=4776.26 cores=0-7
/proc/19569/task/19577/stat : 19577 (SArTagnan) R 19567 19569 19543 0 -1 4202560 89405 0 0 0 477578 48 0 0 25 0 8 0 881944285 5421985792 1087777 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 38640.9
Current children cumulated vsize (KiB) 5294908

[startup+4902.3 s]
/proc/loadavg: 8.02 8.07 8.01 9/559 19729
/proc/meminfo: memFree=24604164/32951124 swapFree=34426104/67111528
[pid=19569] ppid=19567 vsize=5294908 CPUtime=39119.8 cores=0-7
/proc/19569/stat : 19569 (SArTagnan) R 19567 19569 19543 0 -1 4202496 29079887 0 0 0 3907063 4913 0 0 25 0 8 0 881944276 5421985792 1096787 33554432000 4194304 4625164 140734789047920 18446744073709551615 4512009 0 0 4096 2 18446744073709551615 0 0 17 4 0 0 0
/proc/19569/statm: 1323727 1096787 349 106 0 1319230 0
[pid=19569/tid=19571] ppid=19567 vsize=5294908 CPUtime=4902.26 cores=0-7
/proc/19569/task/19571/stat : 19571 (SArTagnan) R 19567 19569 19543 0 -1 4202560 138653 0 0 0 490195 31 0 0 25 0 8 0 881944285 5421985792 1096787 33554432000 4194304 4625164 140734789047920 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=19569/tid=19572] ppid=19567 vsize=5294908 CPUtime=4896.74 cores=0-7
/proc/19569/task/19572/stat : 19572 (SArTagnan) R 19567 19569 19543 0 -1 4202560 168157 0 0 0 489566 108 0 0 25 0 8 0 881944285 5421985792 1096787 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=19569/tid=19573] ppid=19567 vsize=5294908 CPUtime=4876.76 cores=0-7
/proc/19569/task/19573/stat : 19573 (SArTagnan) R 19567 19569 19543 0 -1 4202560 26438854 0 0 0 483365 4311 0 0 25 0 8 0 881944285 5421985792 1096787 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351731 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=19569/tid=19574] ppid=19567 vsize=5294908 CPUtime=4902.23 cores=0-7
/proc/19569/task/19574/stat : 19574 (SArTagnan) R 19567 19569 19543 0 -1 4202560 80383 0 0 0 490200 23 0 0 25 0 8 0 881944285 5421985792 1096787 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=19569/tid=19575] ppid=19567 vsize=5294908 CPUtime=4902.26 cores=0-7
/proc/19569/task/19575/stat : 19575 (SArTagnan) R 19567 19569 19543 0 -1 4202560 135155 0 0 0 490195 31 0 0 25 0 8 0 881944285 5421985792 1096787 33554432000 4194304 4625164 140734789047920 18446744073709551615 4330074 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=19569/tid=19576] ppid=19567 vsize=5294908 CPUtime=4902.27 cores=0-7
/proc/19569/task/19576/stat : 19576 (SArTagnan) R 19567 19569 19543 0 -1 4202560 181950 0 0 0 490184 43 0 0 25 0 8 0 881944285 5421985792 1096787 33554432000 4194304 4625164 140734789047920 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=19569/tid=19577] ppid=19567 vsize=5294908 CPUtime=4836.26 cores=0-7
/proc/19569/task/19577/stat : 19577 (SArTagnan) R 19567 19569 19543 0 -1 4202560 89426 0 0 0 483578 48 0 0 25 0 8 0 881944285 5421985792 1096787 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 39119.8
Current children cumulated vsize (KiB) 5294908

[startup+4962.3 s]
/proc/loadavg: 8.01 8.05 8.00 9/559 19731
/proc/meminfo: memFree=24557280/32951124 swapFree=34426104/67111528
[pid=19569] ppid=19567 vsize=5360444 CPUtime=39598.6 cores=0-7
/proc/19569/stat : 19569 (SArTagnan) R 19567 19569 19543 0 -1 4202496 29395686 0 0 0 3954890 4968 0 0 25 0 8 0 881944276 5489094656 1107470 33554432000 4194304 4625164 140734789047920 18446744073709551615 4393089 0 0 4096 2 18446744073709551615 0 0 17 4 0 0 0
/proc/19569/statm: 1340111 1107470 349 106 0 1335614 0
[pid=19569/tid=19571] ppid=19567 vsize=5360444 CPUtime=4962.26 cores=0-7
/proc/19569/task/19571/stat : 19571 (SArTagnan) R 19567 19569 19543 0 -1 4202560 138818 0 0 0 496195 31 0 0 25 0 8 0 881944285 5489094656 1107470 33554432000 4194304 4625164 140734789047920 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=19569/tid=19572] ppid=19567 vsize=5360444 CPUtime=4956.75 cores=0-7
/proc/19569/task/19572/stat : 19572 (SArTagnan) R 19567 19569 19543 0 -1 4202560 171752 0 0 0 495564 111 0 0 25 0 8 0 881944285 5489094656 1107470 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=19569/tid=19573] ppid=19567 vsize=5360444 CPUtime=4935.61 cores=0-7
/proc/19569/task/19573/stat : 19573 (SArTagnan) R 19567 19569 19543 0 -1 4202560 26727128 0 0 0 489204 4357 0 0 25 0 8 0 881944285 5489094656 1107470 33554432000 4194304 4625164 140734789047920 18446744073709551615 4547280 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=19569/tid=19574] ppid=19567 vsize=5360444 CPUtime=4962.23 cores=0-7
/proc/19569/task/19574/stat : 19574 (SArTagnan) R 19567 19569 19543 0 -1 4202560 80507 0 0 0 496200 23 0 0 25 0 8 0 881944285 5489094656 1107470 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=19569/tid=19575] ppid=19567 vsize=5360444 CPUtime=4962.26 cores=0-7
/proc/19569/task/19575/stat : 19575 (SArTagnan) R 19567 19569 19543 0 -1 4202560 135279 0 0 0 496195 31 0 0 25 0 8 0 881944285 5489094656 1107470 33554432000 4194304 4625164 140734789047920 18446744073709551615 4329981 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=19569/tid=19576] ppid=19567 vsize=5360444 CPUtime=4962.27 cores=0-7
/proc/19569/task/19576/stat : 19576 (SArTagnan) R 19567 19569 19543 0 -1 4202560 182176 0 0 0 496184 43 0 0 25 0 8 0 881944285 5489094656 1107470 33554432000 4194304 4625164 140734789047920 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=19569/tid=19577] ppid=19567 vsize=5360444 CPUtime=4896.26 cores=0-7
/proc/19569/task/19577/stat : 19577 (SArTagnan) R 19567 19569 19543 0 -1 4202560 90902 0 0 0 489577 49 0 0 25 0 8 0 881944285 5489094656 1107470 33554432000 4194304 4625164 140734789047920 18446744073709551615 4352021 0 0 4096 2 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 39598.6
Current children cumulated vsize (KiB) 5360444



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5012.6 s]
/proc/loadavg: 8.04 8.06 8.00 10/559 19733
/proc/meminfo: memFree=24535120/32951124 swapFree=34426104/67111528
[pid=19569] ppid=19567 vsize=5361576 CPUtime=40000.3 cores=0-7
/proc/19569/stat : 19569 (SArTagnan) R 19567 19569 19543 0 -1 4202496 29747588 0 0 0 3995001 5027 0 0 25 0 8 0 881944276 5490253824 1112827 33554432000 4194304 4625164 140734789047920 18446744073709551615 4511904 0 0 4096 2 18446744073709551615 0 0 17 4 0 0 0
/proc/19569/statm: 1340394 1112827 349 106 0 1335897 0
[pid=19569/tid=19571] ppid=19567 vsize=5361576 CPUtime=5012.61 cores=0-7
/proc/19569/task/19571/stat : 19571 (SArTagnan) R 19567 19569 19543 0 -1 4202560 139033 0 0 0 501229 32 0 0 25 0 8 0 881944285 5490253824 1112827 33554432000 4194304 4625164 140734789047920 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=19569/tid=19572] ppid=19567 vsize=5361576 CPUtime=5007.09 cores=0-7
/proc/19569/task/19572/stat : 19572 (SArTagnan) R 19567 19569 19543 0 -1 4202560 172240 0 0 0 500597 112 0 0 25 0 8 0 881944285 5490253824 1112827 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351996 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=19569/tid=19573] ppid=19567 vsize=5361576 CPUtime=4984.97 cores=0-7
/proc/19569/task/19573/stat : 19573 (SArTagnan) R 19567 19569 19543 0 -1 4202560 27064486 0 0 0 494085 4412 0 0 25 0 8 0 881944285 5490253824 1112827 33554432000 4194304 4625164 140734789047920 18446744073709551615 4547280 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=19569/tid=19574] ppid=19567 vsize=5361576 CPUtime=5012.57 cores=0-7
/proc/19569/task/19574/stat : 19574 (SArTagnan) R 19567 19569 19543 0 -1 4202560 80644 0 0 0 501234 23 0 0 25 0 8 0 881944285 5490253824 1112827 33554432000 4194304 4625164 140734789047920 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=19569/tid=19575] ppid=19567 vsize=5361576 CPUtime=5012.6 cores=0-7
/proc/19569/task/19575/stat : 19575 (SArTagnan) R 19567 19569 19543 0 -1 4202560 135553 0 0 0 501229 31 0 0 25 0 8 0 881944285 5490253824 1112827 33554432000 4194304 4625164 140734789047920 18446744073709551615 4329981 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=19569/tid=19576] ppid=19567 vsize=5361576 CPUtime=5012.62 cores=0-7
/proc/19569/task/19576/stat : 19576 (SArTagnan) R 19567 19569 19543 0 -1 4202560 182576 0 0 0 501219 43 0 0 25 0 8 0 881944285 5490253824 1112827 33554432000 4194304 4625164 140734789047920 18446744073709551615 4330841 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=19569/tid=19577] ppid=19567 vsize=5361576 CPUtime=4946.6 cores=0-7
/proc/19569/task/19577/stat : 19577 (SArTagnan) R 19567 19569 19543 0 -1 4202560 91548 0 0 0 494610 50 0 0 25 0 8 0 881944285 5490253824 1112827 33554432000 4194304 4625164 140734789047920 18446744073709551615 4352280 0 0 4096 2 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 40000.3
Current children cumulated vsize (KiB) 5361576

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5012.8 s]
/proc/loadavg: 8.04 8.06 8.00 10/559 19733
/proc/meminfo: memFree=24535120/32951124 swapFree=34426104/67111528
[pid=19569] ppid=19567 vsize=0 CPUtime=40000.5 cores=0-7
/proc/19569/stat : 19569 (SArTagnan) Z 19567 19569 19543 0 -1 4203548 29747634 0 0 0 3995010 5042 0 0 25 0 2 0 881944276 0 0 33554432000 0 0 0 0 0 0 0 4096 2 18446744073709551615 0 0 17 4 0 0 0
/proc/19569/statm: 0 0 0 0 0 0 0
[pid=19569/tid=19573] ppid=19567 vsize=0 CPUtime=4985.12 cores=0-7
/proc/19569/task/19573/stat : 19573 (SArTagnan) R 19567 19569 19543 0 -1 4203588 27064532 0 0 0 494085 4427 0 0 25 0 2 0 881944285 0 0 33554432000 0 0 0 0 0 0 0 4096 2 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40000.5
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5012.87
CPU time (s): 40000.6
CPU user time (s): 39950.1
CPU system time (s): 50.4973
CPU usage (%): 797.957
Max. virtual memory (cumulated for all children) (KiB): 5592312

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39950.1
system time used= 50.4973
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 29747634
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 9659
involuntary context switches= 170368

runsolver used 26.326 second user time and 70.3583 second system time

The end

Launcher Data 

Begin job on node118 at 2011-06-01 10:32:16
IDJOB=3438446
IDBENCH=71353
IDSOLVER=1936
FILE ID=node118/3438446-1306917136
RUNJOBID= node118-1306917136-19557
PBS_JOBID= 13488520
Free space on /tmp= 73656 MiB

SOLVER NAME= SArTagnan 2011-05-15 (fixed)
BENCH NAME= SAT09/CRAFTED/rbsat/random/forced/rbsat-v2400c266431gyes7.cnf
COMMAND LINE= SArTagnan BENCHNAME -threads=NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3438446-1306917136/watcher-3438446-1306917136 -o /tmp/evaluation-result-3438446-1306917136/solver-3438446-1306917136 -C 40000 -W 5100 -M 31000  SArTagnan HOME/instance-3438446-1306917136.cnf -threads=8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 92e61742efd2e322c59b4442206bebf1
RANDOM SEED=463255371

node118.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5374.59
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      28994880 kB
Buffers:        281640 kB
Cached:        1775684 kB
SwapCached:      56136 kB
Active:         625228 kB
Inactive:      2991760 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      28994880 kB
SwapTotal:    67111528 kB
SwapFree:     34426104 kB
Dirty:            3960 kB
Writeback:          24 kB
AnonPages:     1517992 kB
Mapped:          13492 kB
Slab:           150744 kB
PageTables:     115384 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 183774188 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73652 MiB
End job on node118 at 2011-06-01 11:55:51