Trace number 3386004

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
SArTagnan 2011-05-15 (fixed)? (TO) 3887.32 1300.06

General information on the benchmark

NameSAT02/handmade/simon/satex-challenges/
par32-3.shuffled.cnf
MD5SUM0115a0e26236f3f01aa829c2004452c1
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.6469
Satisfiable
(Un)Satisfiability was proved
Number of variables3176
Number of clauses10297
Sum of the clauses size27581
Maximum clause length3
Minimum clause length1
Number of clauses of size 1141
Number of clauses of size 23028
Number of clauses of size 37128
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3386004-1305583434/watcher-3386004-1305583434 -o /tmp/evaluation-result-3386004-1305583434/solver-3386004-1305583434 -C 4800 -W 1300 -M 15500 SArTagnan HOME/instance-3386004-1305583434.cnf -threads=4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 10.29 10.19 8.22 8/399 6872
/proc/meminfo: memFree=11914084/32951124 swapFree=63308196/67111528
[pid=6872] ppid=6870 vsize=52440 CPUtime=0.01 cores=0,2,4,6
/proc/6872/stat : 6872 (SArTagnan) R 6870 6872 6726 0 -1 4202496 1114 0 0 0 1 0 0 0 25 0 4 0 669941266 53698560 1052 33554432000 4194304 4625164 140735519991376 18446744073709551615 47744152979378 0 0 4096 2 18446744073709551615 0 0 17 4 0 0 0
/proc/6872/statm: 13110 1052 304 106 0 8613 0
[pid=6872/tid=6873] ppid=6870 vsize=52440 CPUtime=0 cores=0,2,4,6
/proc/6872/task/6873/stat : 6873 (SArTagnan) R 6870 6872 6726 0 -1 4202560 0 0 0 0 0 0 0 0 25 0 4 0 669941267 53698560 1052 33554432000 4194304 4625164 140735519991376 18446744073709551615 255377357809 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=6872/tid=6874] ppid=6870 vsize=52440 CPUtime=0 cores=0,2,4,6
/proc/6872/task/6874/stat : 6874 (SArTagnan) R 6870 6872 6726 0 -1 4202560 0 0 0 0 0 0 0 0 25 0 4 0 669941267 53698560 1052 33554432000 4194304 4625164 140735519991376 18446744073709551615 255377357809 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=6872/tid=6875] ppid=6870 vsize=52440 CPUtime=0 cores=0,2,4,6
/proc/6872/task/6875/stat : 6875 (SArTagnan) R 6870 6872 6726 0 -1 4202560 0 0 0 0 0 0 0 0 25 0 4 0 669941267 53698560 1052 33554432000 4194304 4625164 140735519991376 18446744073709551615 255377357809 0 0 4096 2 0 0 0 -1 4 0 0 0

[startup+0.111787 s]
/proc/loadavg: 10.29 10.19 8.22 8/399 6872
/proc/meminfo: memFree=11914084/32951124 swapFree=63308196/67111528
[pid=6872] ppid=6870 vsize=119008 CPUtime=0.19 cores=0,2,4,6
/proc/6872/stat : 6872 (SArTagnan) R 6870 6872 6726 0 -1 4202496 1554 0 0 0 19 0 0 0 25 0 4 0 669941266 121864192 1493 33554432000 4194304 4625164 140735519991376 18446744073709551615 255376998608 0 0 4096 2 18446744073709551615 0 0 17 4 0 0 0
/proc/6872/statm: 29752 1510 341 106 0 25255 0
[pid=6872/tid=6873] ppid=6870 vsize=119008 CPUtime=0.04 cores=0,2,4,6
/proc/6872/task/6873/stat : 6873 (SArTagnan) D 6870 6872 6726 0 -1 4202560 0 0 0 0 4 0 0 0 25 0 4 0 669941267 121864192 1510 33554432000 4194304 4625164 140735519991376 18446744073709551615 4273465 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=6872/tid=6874] ppid=6870 vsize=119008 CPUtime=0.02 cores=0,2,4,6
/proc/6872/task/6874/stat : 6874 (SArTagnan) R 6870 6872 6726 0 -1 4202560 26 0 0 0 2 0 0 0 25 0 4 0 669941267 121864192 1510 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351827 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=6872/tid=6875] ppid=6870 vsize=119008 CPUtime=0.09 cores=0,2,4,6
/proc/6872/task/6875/stat : 6875 (SArTagnan) R 6870 6872 6726 0 -1 4202560 215 0 0 0 9 0 0 0 25 0 4 0 669941267 121864192 1510 33554432000 4194304 4625164 140735519991376 18446744073709551615 4279928 0 0 4096 2 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 119008

[startup+0.200997 s]
/proc/loadavg: 10.29 10.19 8.22 8/399 6872
/proc/meminfo: memFree=11914084/32951124 swapFree=63308196/67111528
[pid=6872] ppid=6870 vsize=186012 CPUtime=0.36 cores=0,2,4,6
/proc/6872/stat : 6872 (SArTagnan) R 6870 6872 6726 0 -1 4202496 2610 0 0 0 36 0 0 0 25 0 4 0 669941266 190476288 1966 33554432000 4194304 4625164 140735519991376 18446744073709551615 4492284 0 0 4096 2 18446744073709551615 0 0 17 2 0 0 0
/proc/6872/statm: 46503 1966 343 106 0 42006 0
[pid=6872/tid=6873] ppid=6870 vsize=186012 CPUtime=0.12 cores=0,2,4,6
/proc/6872/task/6873/stat : 6873 (SArTagnan) R 6870 6872 6726 0 -1 4202560 371 0 0 0 12 0 0 0 25 0 4 0 669941267 190476288 1966 33554432000 4194304 4625164 140735519991376 18446744073709551615 4352208 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=6872/tid=6874] ppid=6870 vsize=186012 CPUtime=0.09 cores=0,2,4,6
/proc/6872/task/6874/stat : 6874 (SArTagnan) R 6870 6872 6726 0 -1 4202560 334 0 0 0 9 0 0 0 25 0 4 0 669941267 190476288 1966 33554432000 4194304 4625164 140735519991376 18446744073709551615 4352592 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=6872/tid=6875] ppid=6870 vsize=186012 CPUtime=0.1 cores=0,2,4,6
/proc/6872/task/6875/stat : 6875 (SArTagnan) R 6870 6872 6726 0 -1 4202560 431 0 0 0 10 0 0 0 25 0 4 0 669941267 190476288 1966 33554432000 4194304 4625164 140735519991376 18446744073709551615 255376998511 0 0 4096 2 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 0.36
Current children cumulated vsize (KiB) 186012

[startup+0.300978 s]
/proc/loadavg: 10.29 10.19 8.22 8/399 6872
/proc/meminfo: memFree=11914084/32951124 swapFree=63308196/67111528
[pid=6872] ppid=6870 vsize=188116 CPUtime=0.57 cores=0,2,4,6
/proc/6872/stat : 6872 (SArTagnan) R 6870 6872 6726 0 -1 4202496 3344 0 0 0 57 0 0 0 25 0 4 0 669941266 192630784 2479 33554432000 4194304 4625164 140735519991376 18446744073709551615 255376957447 0 0 4096 2 18446744073709551615 0 0 17 2 0 0 0
/proc/6872/statm: 47029 2479 343 106 0 42532 0
[pid=6872/tid=6873] ppid=6870 vsize=188116 CPUtime=0.22 cores=0,2,4,6
/proc/6872/task/6873/stat : 6873 (SArTagnan) R 6870 6872 6726 0 -1 4202560 393 0 0 0 22 0 0 0 25 0 4 0 669941267 192630784 2479 33554432000 4194304 4625164 140735519991376 18446744073709551615 4288724 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=6872/tid=6874] ppid=6870 vsize=188116 CPUtime=0.19 cores=0,2,4,6
/proc/6872/task/6874/stat : 6874 (SArTagnan) R 6870 6872 6726 0 -1 4202560 342 0 0 0 19 0 0 0 25 0 4 0 669941267 192630784 2479 33554432000 4194304 4625164 140735519991376 18446744073709551615 4462504 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=6872/tid=6875] ppid=6870 vsize=188116 CPUtime=0.1 cores=0,2,4,6
/proc/6872/task/6875/stat : 6875 (SArTagnan) R 6870 6872 6726 0 -1 4202560 441 0 0 0 10 0 0 0 25 0 4 0 669941267 192630784 2479 33554432000 4194304 4625164 140735519991376 18446744073709551615 4352636 0 0 4096 2 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 0.57
Current children cumulated vsize (KiB) 188116

[startup+0.700904 s]
/proc/loadavg: 10.29 10.19 8.22 8/399 6872
/proc/meminfo: memFree=11914084/32951124 swapFree=63308196/67111528
[pid=6872] ppid=6870 vsize=255516 CPUtime=1.57 cores=0,2,4,6
/proc/6872/stat : 6872 (SArTagnan) R 6870 6872 6726 0 -1 4202496 5263 0 0 0 157 0 0 0 25 0 4 0 669941266 261648384 3245 33554432000 4194304 4625164 140735519991376 18446744073709551615 4513492 0 0 4096 2 18446744073709551615 0 0 17 2 0 0 0
/proc/6872/statm: 63879 3245 346 106 0 59382 0
[pid=6872/tid=6873] ppid=6870 vsize=255516 CPUtime=0.62 cores=0,2,4,6
/proc/6872/task/6873/stat : 6873 (SArTagnan) R 6870 6872 6726 0 -1 4202560 680 0 0 0 62 0 0 0 25 0 4 0 669941267 261648384 3245 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351781 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=6872/tid=6874] ppid=6870 vsize=255516 CPUtime=0.39 cores=0,2,4,6
/proc/6872/task/6874/stat : 6874 (SArTagnan) R 6870 6872 6726 0 -1 4202560 646 0 0 0 39 0 0 0 25 0 4 0 669941267 261648384 3245 33554432000 4194304 4625164 140735519991376 18446744073709551615 4353016 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=6872/tid=6875] ppid=6870 vsize=255516 CPUtime=0.3 cores=0,2,4,6
/proc/6872/task/6875/stat : 6875 (SArTagnan) R 6870 6872 6726 0 -1 4202560 1540 0 0 0 30 0 0 0 25 0 4 0 669941267 261648384 3245 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 1.57
Current children cumulated vsize (KiB) 255516

[startup+1.50075 s]
/proc/loadavg: 10.29 10.19 8.22 11/403 6876
/proc/meminfo: memFree=11896692/32951124 swapFree=63308196/67111528
[pid=6872] ppid=6870 vsize=257552 CPUtime=3.86 cores=0,2,4,6
/proc/6872/stat : 6872 (SArTagnan) R 6870 6872 6726 0 -1 4202496 8858 0 0 0 385 1 0 0 25 0 4 0 669941266 263733248 4115 33554432000 4194304 4625164 140735519991376 18446744073709551615 4396480 0 0 4096 2 18446744073709551615 0 0 17 2 0 0 0
/proc/6872/statm: 64388 4115 346 106 0 59891 0
[pid=6872/tid=6873] ppid=6870 vsize=257552 CPUtime=1.42 cores=0,2,4,6
/proc/6872/task/6873/stat : 6873 (SArTagnan) R 6870 6872 6726 0 -1 4202560 1150 0 0 0 142 0 0 0 25 0 4 0 669941267 263733248 4115 33554432000 4194304 4625164 140735519991376 18446744073709551615 4462375 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=6872/tid=6874] ppid=6870 vsize=257552 CPUtime=0.77 cores=0,2,4,6
/proc/6872/task/6874/stat : 6874 (SArTagnan) R 6870 6872 6726 0 -1 4202560 962 0 0 0 77 0 0 0 25 0 4 0 669941267 263733248 4115 33554432000 4194304 4625164 140735519991376 18446744073709551615 4352035 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=6872/tid=6875] ppid=6870 vsize=257552 CPUtime=0.7 cores=0,2,4,6
/proc/6872/task/6875/stat : 6875 (SArTagnan) R 6870 6872 6726 0 -1 4202560 3765 0 0 0 70 0 0 0 25 0 4 0 669941267 263733248 4115 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3.86
Current children cumulated vsize (KiB) 257552

[startup+3.10046 s]
/proc/loadavg: 10.27 10.19 8.23 11/403 6876
/proc/meminfo: memFree=11891360/32951124 swapFree=63308196/67111528
[pid=6872] ppid=6870 vsize=261824 CPUtime=8.64 cores=0,2,4,6
/proc/6872/stat : 6872 (SArTagnan) R 6870 6872 6726 0 -1 4202496 12323 0 0 0 863 1 0 0 25 0 4 0 669941266 268107776 6163 33554432000 4194304 4625164 140735519991376 18446744073709551615 4338085 0 0 4096 2 18446744073709551615 0 0 17 2 0 0 0
/proc/6872/statm: 65456 6163 347 106 0 60959 0
[pid=6872/tid=6873] ppid=6870 vsize=261824 CPUtime=3.02 cores=0,2,4,6
/proc/6872/task/6873/stat : 6873 (SArTagnan) R 6870 6872 6726 0 -1 4202560 2382 0 0 0 302 0 0 0 25 0 4 0 669941267 268107776 6163 33554432000 4194304 4625164 140735519991376 18446744073709551615 4291987 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=6872/tid=6874] ppid=6870 vsize=261824 CPUtime=1.56 cores=0,2,4,6
/proc/6872/task/6874/stat : 6874 (SArTagnan) R 6870 6872 6726 0 -1 4202560 1810 0 0 0 156 0 0 0 25 0 4 0 669941267 268107776 6163 33554432000 4194304 4625164 140735519991376 18446744073709551615 4352010 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=6872/tid=6875] ppid=6870 vsize=261824 CPUtime=1.51 cores=0,2,4,6
/proc/6872/task/6875/stat : 6875 (SArTagnan) R 6870 6872 6726 0 -1 4202560 4736 0 0 0 150 1 0 0 25 0 4 0 669941267 268107776 6163 33554432000 4194304 4625164 140735519991376 18446744073709551615 4552402 0 0 4096 2 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 8.64
Current children cumulated vsize (KiB) 261824

[startup+6.30087 s]
/proc/loadavg: 10.27 10.19 8.23 11/403 6876
/proc/meminfo: memFree=11868180/32951124 swapFree=63308196/67111528
[pid=6872] ppid=6870 vsize=271528 CPUtime=18.2 cores=0,2,4,6
/proc/6872/stat : 6872 (SArTagnan) R 6870 6872 6726 0 -1 4202496 29125 0 0 0 1816 4 0 0 25 0 4 0 669941266 278044672 9762 33554432000 4194304 4625164 140735519991376 18446744073709551615 4519344 0 0 4096 2 18446744073709551615 0 0 17 2 0 0 0
/proc/6872/statm: 67882 9762 360 106 0 63385 0
[pid=6872/tid=6873] ppid=6870 vsize=271528 CPUtime=5.03 cores=0,2,4,6
/proc/6872/task/6873/stat : 6873 (SArTagnan) R 6870 6872 6726 0 -1 4202560 3943 0 0 0 503 0 0 0 25 0 4 0 669941267 278044672 9762 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351901 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=6872/tid=6874] ppid=6870 vsize=271528 CPUtime=3.53 cores=0,2,4,6
/proc/6872/task/6874/stat : 6874 (SArTagnan) R 6870 6872 6726 0 -1 4202560 4083 0 0 0 353 0 0 0 25 0 4 0 669941267 278044672 9762 33554432000 4194304 4625164 140735519991376 18446744073709551615 4352170 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=6872/tid=6875] ppid=6870 vsize=271528 CPUtime=3.87 cores=0,2,4,6
/proc/6872/task/6875/stat : 6875 (SArTagnan) R 6870 6872 6726 0 -1 4202560 16851 0 0 0 384 3 0 0 25 0 4 0 669941267 278044672 9762 33554432000 4194304 4625164 140735519991376 18446744073709551615 4552388 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 18.2
Current children cumulated vsize (KiB) 271528

[startup+12.7007 s]
/proc/loadavg: 10.25 10.18 8.24 11/403 6876
/proc/meminfo: memFree=11843728/32951124 swapFree=63308196/67111528
[pid=6872] ppid=6870 vsize=278708 CPUtime=37.31 cores=0,2,4,6
/proc/6872/stat : 6872 (SArTagnan) R 6870 6872 6726 0 -1 4202496 47620 0 0 0 3722 9 0 0 25 0 4 0 669941266 285396992 14715 33554432000 4194304 4625164 140735519991376 18446744073709551615 4511956 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/6872/statm: 69677 14715 358 106 0 65180 0
[pid=6872/tid=6873] ppid=6870 vsize=278708 CPUtime=10.46 cores=0,2,4,6
/proc/6872/task/6873/stat : 6873 (SArTagnan) R 6870 6872 6726 0 -1 4202560 6579 0 0 0 1045 1 0 0 25 0 4 0 669941267 285396992 14715 33554432000 4194304 4625164 140735519991376 18446744073709551615 4462617 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=6872/tid=6874] ppid=6870 vsize=278708 CPUtime=7.71 cores=0,2,4,6
/proc/6872/task/6874/stat : 6874 (SArTagnan) R 6870 6872 6726 0 -1 4202560 6985 0 0 0 770 1 0 0 25 0 4 0 669941267 285396992 14715 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351777 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=6872/tid=6875] ppid=6870 vsize=278708 CPUtime=7.01 cores=0,2,4,6
/proc/6872/task/6875/stat : 6875 (SArTagnan) R 6870 6872 6726 0 -1 4202560 28930 0 0 0 696 5 0 0 25 0 4 0 669941267 285396992 14715 33554432000 4194304 4625164 140735519991376 18446744073709551615 4335407 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 37.31
Current children cumulated vsize (KiB) 278708

[startup+25.5003 s]
/proc/loadavg: 10.19 10.17 8.26 12/403 6876
/proc/meminfo: memFree=11777272/32951124 swapFree=63308196/67111528
[pid=6872] ppid=6870 vsize=295852 CPUtime=75.53 cores=0,2,4,6
/proc/6872/stat : 6872 (SArTagnan) R 6870 6872 6726 0 -1 4202496 58551 0 0 0 7541 12 0 0 25 0 4 0 669941266 302952448 21681 33554432000 4194304 4625164 140735519991376 18446744073709551615 4508758 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/6872/statm: 73963 21681 391 106 0 69466 0
[pid=6872/tid=6873] ppid=6870 vsize=295852 CPUtime=18.53 cores=0,2,4,6
/proc/6872/task/6873/stat : 6873 (SArTagnan) R 6870 6872 6726 0 -1 4202560 9590 0 0 0 1851 2 0 0 25 0 4 0 669941267 302952448 21681 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351804 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=6872/tid=6874] ppid=6870 vsize=295852 CPUtime=15.11 cores=0,2,4,6
/proc/6872/task/6874/stat : 6874 (SArTagnan) R 6870 6872 6726 0 -1 4202560 10660 0 0 0 1509 2 0 0 25 0 4 0 669941267 302952448 21681 33554432000 4194304 4625164 140735519991376 18446744073709551615 4352285 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=6872/tid=6875] ppid=6870 vsize=295852 CPUtime=17.46 cores=0,2,4,6
/proc/6872/task/6875/stat : 6875 (SArTagnan) R 6870 6872 6726 0 -1 4202560 31954 0 0 0 1740 6 0 0 25 0 4 0 669941267 302952448 21681 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351901 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 75.53
Current children cumulated vsize (KiB) 295852

[startup+51.1006 s]
/proc/loadavg: 10.35 10.21 8.32 12/404 6880
/proc/meminfo: memFree=11651572/32951124 swapFree=63308196/67111528
[pid=6872] ppid=6870 vsize=319780 CPUtime=151.5 cores=0,2,4,6
/proc/6872/stat : 6872 (SArTagnan) R 6870 6872 6726 0 -1 4202496 158449 0 0 0 15121 29 0 0 25 0 4 0 669941266 327454720 39001 33554432000 4194304 4625164 140735519991376 18446744073709551615 4512009 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/6872/statm: 79945 39001 620 106 0 75448 0
[pid=6872/tid=6873] ppid=6870 vsize=319780 CPUtime=31.26 cores=0,2,4,6
/proc/6872/task/6873/stat : 6873 (SArTagnan) R 6870 6872 6726 0 -1 4202560 14131 0 0 0 3122 4 0 0 25 0 4 0 669941267 327454720 39001 33554432000 4194304 4625164 140735519991376 18446744073709551615 4352728 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=6872/tid=6874] ppid=6870 vsize=319780 CPUtime=28.6 cores=0,2,4,6
/proc/6872/task/6874/stat : 6874 (SArTagnan) R 6870 6872 6726 0 -1 4202560 16495 0 0 0 2857 3 0 0 25 0 4 0 669941267 327454720 39001 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351792 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=6872/tid=6875] ppid=6870 vsize=319780 CPUtime=42.56 cores=0,2,4,6
/proc/6872/task/6875/stat : 6875 (SArTagnan) R 6870 6872 6726 0 -1 4202560 112846 0 0 0 4238 18 0 0 25 0 4 0 669941267 327454720 39001 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 151.5
Current children cumulated vsize (KiB) 319780

[startup+102.301 s]
/proc/loadavg: 10.41 10.26 8.45 12/403 6881
/proc/meminfo: memFree=11544628/32951124 swapFree=63308196/67111528
[pid=6872] ppid=6870 vsize=386272 CPUtime=304.16 cores=0,2,4,6
/proc/6872/stat : 6872 (SArTagnan) R 6870 6872 6726 0 -1 4202496 262171 0 0 0 30370 46 0 0 25 0 4 0 669941266 395542528 51683 33554432000 4194304 4625164 140735519991376 18446744073709551615 4519248 0 0 4096 2 18446744073709551615 0 0 17 2 0 0 0
/proc/6872/statm: 96568 51683 395 106 0 92071 0
[pid=6872/tid=6873] ppid=6870 vsize=386272 CPUtime=56.93 cores=0,2,4,6
/proc/6872/task/6873/stat : 6873 (SArTagnan) R 6870 6872 6726 0 -1 4202560 19445 0 0 0 5687 6 0 0 25 0 4 0 669941267 395542528 51683 33554432000 4194304 4625164 140735519991376 18446744073709551615 4352145 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=6872/tid=6874] ppid=6870 vsize=386272 CPUtime=72.24 cores=0,2,4,6
/proc/6872/task/6874/stat : 6874 (SArTagnan) R 6870 6872 6726 0 -1 4202560 36677 0 0 0 7217 7 0 0 25 0 4 0 669941267 395542528 51683 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351814 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=6872/tid=6875] ppid=6870 vsize=386272 CPUtime=75.29 cores=0,2,4,6
/proc/6872/task/6875/stat : 6875 (SArTagnan) R 6870 6872 6726 0 -1 4202560 182130 0 0 0 7500 29 0 0 25 0 4 0 669941267 395542528 51683 33554432000 4194304 4625164 140735519991376 18446744073709551615 4352061 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 304.16
Current children cumulated vsize (KiB) 386272

[startup+162.301 s]

################
# More data... #
################

[pid=6872] ppid=6870 vsize=988756 CPUtime=2281.07 cores=0,2,4,6
/proc/6872/stat : 6872 (SArTagnan) R 6870 6872 6726 0 -1 4202496 2074675 0 0 0 227766 341 0 0 25 0 4 0 669941266 1012486144 174677 33554432000 4194304 4625164 140735519991376 18446744073709551615 4511904 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/6872/statm: 247189 174677 390 106 0 242692 0
[pid=6872/tid=6873] ppid=6870 vsize=988756 CPUtime=644.04 cores=0,2,4,6
/proc/6872/task/6873/stat : 6873 (SArTagnan) R 6870 6872 6726 0 -1 4202560 70002 0 0 0 64381 23 0 0 25 0 4 0 669941267 1012486144 174677 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351818 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=6872/tid=6874] ppid=6870 vsize=988756 CPUtime=475.55 cores=0,2,4,6
/proc/6872/task/6874/stat : 6874 (SArTagnan) R 6870 6872 6726 0 -1 4202560 126311 0 0 0 47526 29 0 0 25 0 4 0 669941267 1012486144 174677 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351804 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=6872/tid=6875] ppid=6870 vsize=988756 CPUtime=536.59 cores=0,2,4,6
/proc/6872/task/6875/stat : 6875 (SArTagnan) R 6870 6872 6726 0 -1 4202560 1800574 0 0 0 53383 276 0 0 25 0 4 0 669941267 1012486144 174677 33554432000 4194304 4625164 140735519991376 18446744073709551615 4552315 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 2281.07
Current children cumulated vsize (KiB) 988756

[startup+822.301 s]
/proc/loadavg: 10.31 10.30 9.41 11/403 6932
/proc/meminfo: memFree=11298232/32951124 swapFree=63308196/67111528
[pid=6872] ppid=6870 vsize=995224 CPUtime=2460.27 cores=0,2,4,6
/proc/6872/stat : 6872 (SArTagnan) R 6870 6872 6726 0 -1 4202496 2223185 0 0 0 245661 366 0 0 25 0 4 0 669941266 1019109376 180335 33554432000 4194304 4625164 140735519991376 18446744073709551615 4512818 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/6872/statm: 248806 180335 386 106 0 244309 0
[pid=6872/tid=6873] ppid=6870 vsize=995224 CPUtime=674.04 cores=0,2,4,6
/proc/6872/task/6873/stat : 6873 (SArTagnan) R 6870 6872 6726 0 -1 4202560 71072 0 0 0 67379 25 0 0 25 0 4 0 669941267 1019109376 180335 33554432000 4194304 4625164 140735519991376 18446744073709551615 4352739 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=6872/tid=6874] ppid=6870 vsize=995224 CPUtime=535.53 cores=0,2,4,6
/proc/6872/task/6874/stat : 6874 (SArTagnan) R 6870 6872 6726 0 -1 4202560 129076 0 0 0 53522 31 0 0 25 0 4 0 669941267 1019109376 180335 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=6872/tid=6875] ppid=6870 vsize=995224 CPUtime=566.58 cores=0,2,4,6
/proc/6872/task/6875/stat : 6875 (SArTagnan) R 6870 6872 6726 0 -1 4202560 1944608 0 0 0 56360 298 0 0 25 0 4 0 669941267 1019109376 180335 33554432000 4194304 4625164 140735519991376 18446744073709551615 4552315 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 2460.27
Current children cumulated vsize (KiB) 995224

[startup+882.315 s]
/proc/loadavg: 10.25 10.28 9.47 11/403 6933
/proc/meminfo: memFree=11220224/32951124 swapFree=63308196/67111528
[pid=6872] ppid=6870 vsize=1031140 CPUtime=2639.54 cores=0,2,4,6
/proc/6872/stat : 6872 (SArTagnan) R 6870 6872 6726 0 -1 4202496 2391822 0 0 0 263558 396 0 0 25 0 4 0 669941266 1055887360 187383 33554432000 4194304 4625164 140735519991376 18446744073709551615 4512801 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/6872/statm: 257785 187383 423 106 0 253288 0
[pid=6872/tid=6873] ppid=6870 vsize=1031140 CPUtime=704.18 cores=0,2,4,6
/proc/6872/task/6873/stat : 6873 (SArTagnan) R 6870 6872 6726 0 -1 4202560 72028 0 0 0 70392 26 0 0 25 0 4 0 669941267 1055887360 187383 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351777 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=6872/tid=6874] ppid=6870 vsize=1031140 CPUtime=595.53 cores=0,2,4,6
/proc/6872/task/6874/stat : 6874 (SArTagnan) R 6870 6872 6726 0 -1 4202560 131428 0 0 0 59521 32 0 0 25 0 4 0 669941267 1055887360 187383 33554432000 4194304 4625164 140735519991376 18446744073709551615 4352728 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=6872/tid=6875] ppid=6870 vsize=1031140 CPUtime=596.51 cores=0,2,4,6
/proc/6872/task/6875/stat : 6875 (SArTagnan) R 6870 6872 6726 0 -1 4202560 2093093 0 0 0 59329 322 0 0 25 0 4 0 669941267 1055887360 187383 33554432000 4194304 4625164 140735519991376 18446744073709551615 4552315 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 2639.54
Current children cumulated vsize (KiB) 1031140

[startup+942.301 s]
/proc/loadavg: 10.24 10.27 9.51 11/403 6935
/proc/meminfo: memFree=11083184/32951124 swapFree=63308196/67111528
[pid=6872] ppid=6870 vsize=1037080 CPUtime=2818.72 cores=0,2,4,6
/proc/6872/stat : 6872 (SArTagnan) R 6870 6872 6726 0 -1 4202496 2549026 0 0 0 281447 425 0 0 25 0 4 0 669941266 1061969920 193021 33554432000 4194304 4625164 140735519991376 18446744073709551615 4511936 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/6872/statm: 259270 193021 426 106 0 254773 0
[pid=6872/tid=6873] ppid=6870 vsize=1037080 CPUtime=734.18 cores=0,2,4,6
/proc/6872/task/6873/stat : 6873 (SArTagnan) R 6870 6872 6726 0 -1 4202560 72873 0 0 0 73390 28 0 0 25 0 4 0 669941267 1061969920 193021 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351764 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=6872/tid=6874] ppid=6870 vsize=1037080 CPUtime=655.5 cores=0,2,4,6
/proc/6872/task/6874/stat : 6874 (SArTagnan) R 6870 6872 6726 0 -1 4202560 133772 0 0 0 65516 34 0 0 25 0 4 0 669941267 1061969920 193021 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351814 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=6872/tid=6875] ppid=6870 vsize=1037080 CPUtime=626.5 cores=0,2,4,6
/proc/6872/task/6875/stat : 6875 (SArTagnan) R 6870 6872 6726 0 -1 4202560 2245709 0 0 0 62302 348 0 0 25 0 4 0 669941267 1061969920 193021 33554432000 4194304 4625164 140735519991376 18446744073709551615 4339612 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 2818.72
Current children cumulated vsize (KiB) 1037080

[startup+1002.3 s]
/proc/loadavg: 10.24 10.28 9.56 11/403 6936
/proc/meminfo: memFree=10998356/32951124 swapFree=63308196/67111528
[pid=6872] ppid=6870 vsize=1042624 CPUtime=2997.95 cores=0,2,4,6
/proc/6872/stat : 6872 (SArTagnan) R 6870 6872 6726 0 -1 4202496 2707525 0 0 0 299345 450 0 0 25 0 4 0 669941266 1067646976 197767 33554432000 4194304 4625164 140735519991376 18446744073709551615 4511930 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/6872/statm: 260656 197767 426 106 0 256159 0
[pid=6872/tid=6873] ppid=6870 vsize=1042624 CPUtime=764.18 cores=0,2,4,6
/proc/6872/task/6873/stat : 6873 (SArTagnan) R 6870 6872 6726 0 -1 4202560 74215 0 0 0 76389 29 0 0 25 0 4 0 669941267 1067646976 197767 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351827 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=6872/tid=6874] ppid=6870 vsize=1042624 CPUtime=715.5 cores=0,2,4,6
/proc/6872/task/6874/stat : 6874 (SArTagnan) R 6870 6872 6726 0 -1 4202560 136112 0 0 0 71515 35 0 0 25 0 4 0 669941267 1067646976 197767 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=6872/tid=6875] ppid=6870 vsize=1042624 CPUtime=656.5 cores=0,2,4,6
/proc/6872/task/6875/stat : 6875 (SArTagnan) R 6870 6872 6726 0 -1 4202560 2400155 0 0 0 65280 370 0 0 25 0 4 0 669941267 1067646976 197767 33554432000 4194304 4625164 140735519991376 18446744073709551615 4552369 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 2997.95
Current children cumulated vsize (KiB) 1042624

[startup+1062.3 s]
/proc/loadavg: 10.18 10.26 9.59 11/403 6938
/proc/meminfo: memFree=10919476/32951124 swapFree=63308196/67111528
[pid=6872] ppid=6870 vsize=1048036 CPUtime=3177.19 cores=0,2,4,6
/proc/6872/stat : 6872 (SArTagnan) R 6870 6872 6726 0 -1 4202496 2853582 0 0 0 317245 474 0 0 25 0 4 0 669941266 1073188864 201341 33554432000 4194304 4625164 140735519991376 18446744073709551615 4511904 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/6872/statm: 262009 201341 401 106 0 257512 0
[pid=6872/tid=6873] ppid=6870 vsize=1048036 CPUtime=794.19 cores=0,2,4,6
/proc/6872/task/6873/stat : 6873 (SArTagnan) R 6870 6872 6726 0 -1 4202560 74986 0 0 0 79389 30 0 0 25 0 4 0 669941267 1073188864 201341 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=6872/tid=6874] ppid=6870 vsize=1048036 CPUtime=775.49 cores=0,2,4,6
/proc/6872/task/6874/stat : 6874 (SArTagnan) R 6870 6872 6726 0 -1 4202560 137482 0 0 0 77513 36 0 0 25 0 4 0 669941267 1073188864 201341 33554432000 4194304 4625164 140735519991376 18446744073709551615 4352739 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=6872/tid=6875] ppid=6870 vsize=1048036 CPUtime=686.5 cores=0,2,4,6
/proc/6872/task/6875/stat : 6875 (SArTagnan) R 6870 6872 6726 0 -1 4202560 2542983 0 0 0 68258 392 0 0 25 0 4 0 669941267 1073188864 201341 33554432000 4194304 4625164 140735519991376 18446744073709551615 4552432 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3177.19
Current children cumulated vsize (KiB) 1048036

[startup+1122.3 s]
/proc/loadavg: 10.15 10.23 9.63 11/403 6940
/proc/meminfo: memFree=10849896/32951124 swapFree=63308196/67111528
[pid=6872] ppid=6870 vsize=1089108 CPUtime=3356.4 cores=0,2,4,6
/proc/6872/stat : 6872 (SArTagnan) R 6870 6872 6726 0 -1 4202496 3024665 0 0 0 335137 503 0 0 25 0 4 0 669941266 1115246592 208501 33554432000 4194304 4625164 140735519991376 18446744073709551615 4519334 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/6872/statm: 272277 208501 445 106 0 267780 0
[pid=6872/tid=6873] ppid=6870 vsize=1089108 CPUtime=824.2 cores=0,2,4,6
/proc/6872/task/6873/stat : 6873 (SArTagnan) R 6870 6872 6726 0 -1 4202560 96039 0 0 0 82386 34 0 0 25 0 4 0 669941267 1115246592 208501 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351769 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=6872/tid=6874] ppid=6870 vsize=1089108 CPUtime=835.47 cores=0,2,4,6
/proc/6872/task/6874/stat : 6874 (SArTagnan) R 6870 6872 6726 0 -1 4202560 138742 0 0 0 83510 37 0 0 25 0 4 0 669941267 1115246592 208501 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351792 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=6872/tid=6875] ppid=6870 vsize=1089108 CPUtime=716.5 cores=0,2,4,6
/proc/6872/task/6875/stat : 6875 (SArTagnan) R 6870 6872 6726 0 -1 4202560 2691181 0 0 0 71234 416 0 0 25 0 4 0 669941267 1115246592 208501 33554432000 4194304 4625164 140735519991376 18446744073709551615 4339612 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3356.4
Current children cumulated vsize (KiB) 1089108

[startup+1182.3 s]
/proc/loadavg: 10.18 10.23 9.66 11/403 6941
/proc/meminfo: memFree=10782192/32951124 swapFree=63308196/67111528
[pid=6872] ppid=6870 vsize=1093860 CPUtime=3535.65 cores=0,2,4,6
/proc/6872/stat : 6872 (SArTagnan) R 6870 6872 6726 0 -1 4202496 3180675 0 0 0 353038 527 0 0 25 0 4 0 669941266 1120112640 213574 33554432000 4194304 4625164 140735519991376 18446744073709551615 4512009 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/6872/statm: 273465 213574 436 106 0 268968 0
[pid=6872/tid=6873] ppid=6870 vsize=1093860 CPUtime=854.21 cores=0,2,4,6
/proc/6872/task/6873/stat : 6873 (SArTagnan) R 6870 6872 6726 0 -1 4202560 96853 0 0 0 85386 35 0 0 25 0 4 0 669941267 1120112640 213574 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351781 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=6872/tid=6874] ppid=6870 vsize=1093860 CPUtime=895.46 cores=0,2,4,6
/proc/6872/task/6874/stat : 6874 (SArTagnan) R 6870 6872 6726 0 -1 4202560 140948 0 0 0 89508 38 0 0 25 0 4 0 669941267 1120112640 213574 33554432000 4194304 4625164 140735519991376 18446744073709551615 4352021 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=6872/tid=6875] ppid=6870 vsize=1093860 CPUtime=746.48 cores=0,2,4,6
/proc/6872/task/6875/stat : 6875 (SArTagnan) R 6870 6872 6726 0 -1 4202560 2843335 0 0 0 74211 437 0 0 25 0 4 0 669941267 1120112640 213574 33554432000 4194304 4625164 140735519991376 18446744073709551615 4552369 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3535.65
Current children cumulated vsize (KiB) 1093860

[startup+1242.3 s]
/proc/loadavg: 10.20 10.23 9.69 11/403 6943
/proc/meminfo: memFree=10735796/32951124 swapFree=63308196/67111528
[pid=6872] ppid=6870 vsize=1098480 CPUtime=3714.87 cores=0,2,4,6
/proc/6872/stat : 6872 (SArTagnan) R 6870 6872 6726 0 -1 4202496 3335266 0 0 0 370934 553 0 0 25 0 4 0 669941266 1124843520 219885 33554432000 4194304 4625164 140735519991376 18446744073709551615 4512009 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/6872/statm: 274620 219885 436 106 0 270123 0
[pid=6872/tid=6873] ppid=6870 vsize=1098480 CPUtime=884.2 cores=0,2,4,6
/proc/6872/task/6873/stat : 6873 (SArTagnan) R 6870 6872 6726 0 -1 4202560 97627 0 0 0 88384 36 0 0 25 0 4 0 669941267 1124843520 219885 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351827 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=6872/tid=6874] ppid=6870 vsize=1098480 CPUtime=955.4 cores=0,2,4,6
/proc/6872/task/6874/stat : 6874 (SArTagnan) R 6870 6872 6726 0 -1 4202560 143603 0 0 0 95501 39 0 0 25 0 4 0 669941267 1124843520 219885 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351777 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=6872/tid=6875] ppid=6870 vsize=1098480 CPUtime=776.54 cores=0,2,4,6
/proc/6872/task/6875/stat : 6875 (SArTagnan) R 6870 6872 6726 0 -1 4202560 2992460 0 0 0 77193 461 0 0 25 0 4 0 669941267 1124843520 219885 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3714.87
Current children cumulated vsize (KiB) 1098480



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+1300 s]
/proc/loadavg: 10.26 10.25 9.73 11/403 6944
/proc/meminfo: memFree=10685188/32951124 swapFree=63308196/67111528
[pid=6872] ppid=6870 vsize=1102704 CPUtime=3887.24 cores=0,2,4,6
/proc/6872/stat : 6872 (SArTagnan) R 6870 6872 6726 0 -1 4202496 3486889 0 0 0 388149 575 0 0 25 0 4 0 669941266 1129168896 223589 33554432000 4194304 4625164 140735519991376 18446744073709551615 4519334 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/6872/statm: 275676 223589 462 106 0 271179 0
[pid=6872/tid=6873] ppid=6870 vsize=1102704 CPUtime=913.09 cores=0,2,4,6
/proc/6872/task/6873/stat : 6873 (SArTagnan) R 6870 6872 6726 0 -1 4202560 98303 0 0 0 91272 37 0 0 25 0 4 0 669941267 1129168896 223589 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351781 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=6872/tid=6874] ppid=6870 vsize=1102704 CPUtime=1013.09 cores=0,2,4,6
/proc/6872/task/6874/stat : 6874 (SArTagnan) R 6870 6872 6726 0 -1 4202560 144772 0 0 0 101270 39 0 0 25 0 4 0 669941267 1129168896 223589 33554432000 4194304 4625164 140735519991376 18446744073709551615 4352280 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=6872/tid=6875] ppid=6870 vsize=1102704 CPUtime=805.38 cores=0,2,4,6
/proc/6872/task/6875/stat : 6875 (SArTagnan) R 6870 6872 6726 0 -1 4202560 3141604 0 0 0 80056 482 0 0 25 0 4 0 669941267 1129168896 223589 33554432000 4194304 4625164 140735519991376 18446744073709551615 4352021 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3887.24
Current children cumulated vsize (KiB) 1102704

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1300 s]
/proc/loadavg: 10.26 10.25 9.73 11/403 6944
/proc/meminfo: memFree=10685188/32951124 swapFree=63308196/67111528
[pid=6872] ppid=6870 vsize=1102704 CPUtime=3887.24 cores=0,2,4,6
/proc/6872/stat : 6872 (SArTagnan) R 6870 6872 6726 0 -1 4202496 3486889 0 0 0 388149 575 0 0 25 0 4 0 669941266 1129168896 223589 33554432000 4194304 4625164 140735519991376 18446744073709551615 4519334 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/6872/statm: 275676 223589 462 106 0 271179 0
[pid=6872/tid=6873] ppid=6870 vsize=1102704 CPUtime=913.09 cores=0,2,4,6
/proc/6872/task/6873/stat : 6873 (SArTagnan) R 6870 6872 6726 0 -1 4202560 98303 0 0 0 91272 37 0 0 25 0 4 0 669941267 1129168896 223589 33554432000 4194304 4625164 140735519991376 18446744073709551615 4351781 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=6872/tid=6874] ppid=6870 vsize=1102704 CPUtime=1013.09 cores=0,2,4,6
/proc/6872/task/6874/stat : 6874 (SArTagnan) R 6870 6872 6726 0 -1 4202560 144772 0 0 0 101270 39 0 0 25 0 4 0 669941267 1129168896 223589 33554432000 4194304 4625164 140735519991376 18446744073709551615 4352280 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=6872/tid=6875] ppid=6870 vsize=1102704 CPUtime=805.38 cores=0,2,4,6
/proc/6872/task/6875/stat : 6875 (SArTagnan) R 6870 6872 6726 0 -1 4202560 3141604 0 0 0 80056 482 0 0 25 0 4 0 669941267 1129168896 223589 33554432000 4194304 4625164 140735519991376 18446744073709551615 4352021 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3887.24
Current children cumulated vsize (KiB) 1102704

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1300.06
CPU time (s): 3887.32
CPU user time (s): 3881.52
CPU system time (s): 5.79712
CPU usage (%): 299.01
Max. virtual memory (cumulated for all children) (KiB): 1154048

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 3881.52
system time used= 5.79712
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 3486889
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 605
involuntary context switches= 88508

runsolver used 4.6233 second user time and 11.9382 second system time

The end

Launcher Data 

Begin job on node135 at 2011-05-17 00:03:54
IDJOB=3386004
IDBENCH=87739
IDSOLVER=1871
FILE ID=node135/3386004-1305583434
RUNJOBID= node135-1305582131-6744
PBS_JOBID= 13350294
Free space on /tmp= 73504 MiB

SOLVER NAME= SArTagnan 2011-05-15 (fixed)
BENCH NAME= SAT02/handmade/simon/satex-challenges/par32-3.shuffled.cnf
COMMAND LINE= SArTagnan BENCHNAME -threads=NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3386004-1305583434/watcher-3386004-1305583434 -o /tmp/evaluation-result-3386004-1305583434/solver-3386004-1305583434 -C 4800 -W 1300 -M 15500  SArTagnan HOME/instance-3386004-1305583434.cnf -threads=4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 0115a0e26236f3f01aa829c2004452c1
RANDOM SEED=412212672

node135.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 6640.14
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.78
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      11914480 kB
Buffers:        323596 kB
Cached:        6471844 kB
SwapCached:      18100 kB
Active:       12191404 kB
Inactive:      8572028 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      11914480 kB
SwapTotal:    67111528 kB
SwapFree:     63308196 kB
Dirty:             396 kB
Writeback:           0 kB
AnonPages:    13954984 kB
Mapped:          22700 kB
Slab:           145792 kB
PageTables:      62440 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 106351696 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73500 MiB
End job on node135 at 2011-05-17 00:25:36