Trace number 3273547

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
SArTagnan 2011-03-24? (TO) 4800.36 1201.94

General information on the benchmark

Namecrafted/spence/sgen/
sgen3-n180-s34713902-sat.cnf
MD5SUM19292459de25c4f5e9e6f187c00d549d
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.431934
Satisfiable
(Un)Satisfiability was proved
Number of variables180
Number of clauses432
Sum of the clauses size1080
Maximum clause length5
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2360
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 572
Number of clauses of size over 50

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3273547-1304084904/watcher-3273547-1304084904 -o /tmp/evaluation-result-3273547-1304084904/solver-3273547-1304084904 -C 4800 -W 1300 -M 15500 SArTagnan HOME/instance-3273547-1304084904.cnf -threads=4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.48 7.95 7.97 1/172 32519
/proc/meminfo: memFree=24412108/32951124 swapFree=67111524/67111528
[pid=32519] ppid=32517 vsize=49208 CPUtime=0 cores=1,3,5,7
/proc/32519/stat : 32519 (SArTagnan) R 32517 32519 31855 0 -1 4202496 447 0 0 0 0 0 0 0 23 0 4 0 520088118 50388992 383 33554432000 4194304 4623224 140734238586032 18446744073709551615 47056091097010 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/32519/statm: 12302 383 301 105 0 7806 0
[pid=32519/tid=32520] ppid=32517 vsize=49208 CPUtime=0 cores=1,3,5,7
/proc/32519/task/32520/stat : 32520 (SArTagnan) R 32517 32519 31855 0 -1 4202560 1 0 0 0 0 0 0 0 24 0 4 0 520088118 50388992 383 33554432000 4194304 4623224 140734238586032 18446744073709551615 47056091097010 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=32519/tid=32521] ppid=32517 vsize=49208 CPUtime=0 cores=1,3,5,7
/proc/32519/task/32521/stat : 32521 (SArTagnan) R 32517 32519 31855 0 -1 4202560 0 0 0 0 0 0 0 0 25 0 4 0 520088118 50388992 383 33554432000 4194304 4623224 140734238586032 18446744073709551615 212687731697 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=32519/tid=32522] ppid=32517 vsize=49208 CPUtime=0 cores=1,3,5,7
/proc/32519/task/32522/stat : 32522 (SArTagnan) R 32517 32519 31855 0 -1 4202560 0 0 0 0 0 0 0 0 25 0 4 0 520088118 50388992 383 33554432000 4194304 4623224 140734238586032 18446744073709551615 212699472950 0 0 4096 2 0 0 0 -1 5 0 0 0

[startup+0.074214 s]
/proc/loadavg: 7.48 7.95 7.97 1/172 32519
/proc/meminfo: memFree=24412108/32951124 swapFree=67111524/67111528
[pid=32519] ppid=32517 vsize=247236 CPUtime=0.28 cores=1,3,5,7
/proc/32519/stat : 32519 (SArTagnan) R 32517 32519 31855 0 -1 4202496 1367 0 0 0 28 0 0 0 25 0 4 0 520088118 253169664 1047 33554432000 4194304 4623224 140734238586032 18446744073709551615 4517479 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/32519/statm: 61809 1047 343 105 0 57313 0
[pid=32519/tid=32520] ppid=32517 vsize=247236 CPUtime=0.07 cores=1,3,5,7
/proc/32519/task/32520/stat : 32520 (SArTagnan) R 32517 32519 31855 0 -1 4202560 224 0 0 0 7 0 0 0 25 0 4 0 520088118 253169664 1047 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376773 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=32519/tid=32521] ppid=32517 vsize=247236 CPUtime=0.06 cores=1,3,5,7
/proc/32519/task/32521/stat : 32521 (SArTagnan) R 32517 32519 31855 0 -1 4202560 224 0 0 0 6 0 0 0 25 0 4 0 520088118 253169664 1047 33554432000 4194304 4623224 140734238586032 18446744073709551615 4377579 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=32519/tid=32522] ppid=32517 vsize=247236 CPUtime=0.06 cores=1,3,5,7
/proc/32519/task/32522/stat : 32522 (SArTagnan) R 32517 32519 31855 0 -1 4202560 245 0 0 0 6 0 0 0 25 0 4 0 520088118 253169664 1047 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376883 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 0.28
Current children cumulated vsize (KiB) 247236

[startup+0.100209 s]
/proc/loadavg: 7.48 7.95 7.97 1/172 32519
/proc/meminfo: memFree=24412108/32951124 swapFree=67111524/67111528
[pid=32519] ppid=32517 vsize=247476 CPUtime=0.38 cores=1,3,5,7
/proc/32519/stat : 32519 (SArTagnan) R 32517 32519 31855 0 -1 4202496 1590 0 0 0 38 0 0 0 25 0 4 0 520088118 253415424 1216 33554432000 4194304 4623224 140734238586032 18446744073709551615 4510235 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/32519/statm: 61869 1217 344 105 0 57373 0
[pid=32519/tid=32520] ppid=32517 vsize=247476 CPUtime=0.09 cores=1,3,5,7
/proc/32519/task/32520/stat : 32520 (SArTagnan) R 32517 32519 31855 0 -1 4202560 279 0 0 0 9 0 0 0 25 0 4 0 520088118 253415424 1217 33554432000 4194304 4623224 140734238586032 18446744073709551615 212687336481 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=32519/tid=32521] ppid=32517 vsize=247476 CPUtime=0.09 cores=1,3,5,7
/proc/32519/task/32521/stat : 32521 (SArTagnan) R 32517 32519 31855 0 -1 4202560 267 0 0 0 9 0 0 0 25 0 4 0 520088118 253415424 1217 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376672 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=32519/tid=32522] ppid=32517 vsize=247476 CPUtime=0.09 cores=1,3,5,7
/proc/32519/task/32522/stat : 32522 (SArTagnan) R 32517 32519 31855 0 -1 4202560 338 0 0 0 9 0 0 0 25 0 4 0 520088118 253415424 1217 33554432000 4194304 4623224 140734238586032 18446744073709551615 4364669 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 0.38
Current children cumulated vsize (KiB) 247476

[startup+0.300174 s]
/proc/loadavg: 7.48 7.95 7.97 1/172 32519
/proc/meminfo: memFree=24412108/32951124 swapFree=67111524/67111528
[pid=32519] ppid=32517 vsize=249708 CPUtime=1.18 cores=1,3,5,7
/proc/32519/stat : 32519 (SArTagnan) R 32517 32519 31855 0 -1 4202496 3511 0 0 0 118 0 0 0 25 0 4 0 520088118 255700992 2270 33554432000 4194304 4623224 140734238586032 18446744073709551615 4415933 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/32519/statm: 62427 2270 344 105 0 57931 0
[pid=32519/tid=32520] ppid=32517 vsize=249708 CPUtime=0.29 cores=1,3,5,7
/proc/32519/task/32520/stat : 32520 (SArTagnan) R 32517 32519 31855 0 -1 4202560 765 0 0 0 29 0 0 0 25 0 4 0 520088118 255700992 2270 33554432000 4194304 4623224 140734238586032 18446744073709551615 4377018 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=32519/tid=32521] ppid=32517 vsize=249708 CPUtime=0.29 cores=1,3,5,7
/proc/32519/task/32521/stat : 32521 (SArTagnan) R 32517 32519 31855 0 -1 4202560 809 0 0 0 29 0 0 0 25 0 4 0 520088118 255700992 2270 33554432000 4194304 4623224 140734238586032 18446744073709551615 4366341 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=32519/tid=32522] ppid=32517 vsize=249708 CPUtime=0.29 cores=1,3,5,7
/proc/32519/task/32522/stat : 32522 (SArTagnan) R 32517 32519 31855 0 -1 4202560 983 0 0 0 29 0 0 0 25 0 4 0 520088118 255700992 2270 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376666 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 1.18
Current children cumulated vsize (KiB) 249708

[startup+0.701102 s]
/proc/loadavg: 7.48 7.95 7.97 1/172 32519
/proc/meminfo: memFree=24412108/32951124 swapFree=67111524/67111528
[pid=32519] ppid=32517 vsize=253768 CPUtime=2.78 cores=1,3,5,7
/proc/32519/stat : 32519 (SArTagnan) R 32517 32519 31855 0 -1 4202496 7463 0 0 0 278 0 0 0 25 0 4 0 520088118 259858432 3828 33554432000 4194304 4623224 140734238586032 18446744073709551615 4517479 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/32519/statm: 63442 3828 344 105 0 58946 0
[pid=32519/tid=32520] ppid=32517 vsize=253768 CPUtime=0.69 cores=1,3,5,7
/proc/32519/task/32520/stat : 32520 (SArTagnan) R 32517 32519 31855 0 -1 4202560 1685 0 0 0 69 0 0 0 25 0 4 0 520088118 259858432 3828 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376853 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=32519/tid=32521] ppid=32517 vsize=253768 CPUtime=0.69 cores=1,3,5,7
/proc/32519/task/32521/stat : 32521 (SArTagnan) R 32517 32519 31855 0 -1 4202560 1808 0 0 0 69 0 0 0 25 0 4 0 520088118 259858432 3828 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376853 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=32519/tid=32522] ppid=32517 vsize=253768 CPUtime=0.69 cores=1,3,5,7
/proc/32519/task/32522/stat : 32522 (SArTagnan) R 32517 32519 31855 0 -1 4202560 2768 0 0 0 69 0 0 0 25 0 4 0 520088118 259858432 3828 33554432000 4194304 4623224 140734238586032 18446744073709551615 4557786 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 2.78
Current children cumulated vsize (KiB) 253768

[startup+1.50096 s]
/proc/loadavg: 7.48 7.95 7.97 10/184 32536
/proc/meminfo: memFree=24375452/32951124 swapFree=67111524/67111528
[pid=32519] ppid=32517 vsize=257200 CPUtime=5.97 cores=1,3,5,7
/proc/32519/stat : 32519 (SArTagnan) R 32517 32519 31855 0 -1 4202496 16355 0 0 0 596 1 0 0 25 0 4 0 520088118 263372800 5993 33554432000 4194304 4623224 140734238586032 18446744073709551615 4510064 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/32519/statm: 64300 5993 345 105 0 59804 0
[pid=32519/tid=32520] ppid=32517 vsize=257200 CPUtime=1.49 cores=1,3,5,7
/proc/32519/task/32520/stat : 32520 (SArTagnan) R 32517 32519 31855 0 -1 4202560 3526 0 0 0 149 0 0 0 25 0 4 0 520088118 263372800 5993 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376901 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=32519/tid=32521] ppid=32517 vsize=257200 CPUtime=1.48 cores=1,3,5,7
/proc/32519/task/32521/stat : 32521 (SArTagnan) R 32517 32519 31855 0 -1 4202560 2829 0 0 0 148 0 0 0 25 0 4 0 520088118 263372800 5993 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376901 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=32519/tid=32522] ppid=32517 vsize=257200 CPUtime=1.48 cores=1,3,5,7
/proc/32519/task/32522/stat : 32522 (SArTagnan) R 32517 32519 31855 0 -1 4202560 8415 0 0 0 148 0 0 0 25 0 4 0 520088118 263372800 5993 33554432000 4194304 4623224 140734238586032 18446744073709551615 212687333914 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 5.97
Current children cumulated vsize (KiB) 257200

[startup+3.10067 s]
/proc/loadavg: 7.48 7.95 7.97 9/184 32536
/proc/meminfo: memFree=24353372/32951124 swapFree=67111524/67111528
[pid=32519] ppid=32517 vsize=265516 CPUtime=12.36 cores=1,3,5,7
/proc/32519/stat : 32519 (SArTagnan) R 32517 32519 31855 0 -1 4202496 31612 0 0 0 1233 3 0 0 25 0 4 0 520088118 271888384 9314 33554432000 4194304 4623224 140734238586032 18446744073709551615 4510042 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/32519/statm: 66379 9314 351 105 0 61883 0
[pid=32519/tid=32520] ppid=32517 vsize=265516 CPUtime=3.08 cores=1,3,5,7
/proc/32519/task/32520/stat : 32520 (SArTagnan) R 32517 32519 31855 0 -1 4202560 5490 0 0 0 308 0 0 0 25 0 4 0 520088118 271888384 9314 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376629 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=32519/tid=32521] ppid=32517 vsize=265516 CPUtime=3.08 cores=1,3,5,7
/proc/32519/task/32521/stat : 32521 (SArTagnan) R 32517 32519 31855 0 -1 4202560 5531 0 0 0 307 1 0 0 25 0 4 0 520088118 271888384 9314 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376711 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=32519/tid=32522] ppid=32517 vsize=265516 CPUtime=3.08 cores=1,3,5,7
/proc/32519/task/32522/stat : 32522 (SArTagnan) R 32517 32519 31855 0 -1 4202560 18491 0 0 0 307 1 0 0 25 0 4 0 520088118 271888384 9314 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376827 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 12.36
Current children cumulated vsize (KiB) 265516

[startup+6.30108 s]
/proc/loadavg: 7.60 7.97 7.97 9/184 32536
/proc/meminfo: memFree=24311328/32951124 swapFree=67111524/67111528
[pid=32519] ppid=32517 vsize=275264 CPUtime=25.15 cores=1,3,5,7
/proc/32519/stat : 32519 (SArTagnan) R 32517 32519 31855 0 -1 4202496 56975 0 0 0 2508 7 0 0 25 0 4 0 520088118 281870336 12649 33554432000 4194304 4623224 140734238586032 18446744073709551615 4360016 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/32519/statm: 68816 12651 355 105 0 64320 0
[pid=32519/tid=32520] ppid=32517 vsize=275264 CPUtime=6.29 cores=1,3,5,7
/proc/32519/task/32520/stat : 32520 (SArTagnan) R 32517 32519 31855 0 -1 4202560 7955 0 0 0 628 1 0 0 25 0 4 0 520088118 281870336 12652 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376629 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=32519/tid=32521] ppid=32517 vsize=275264 CPUtime=6.25 cores=1,3,5,7
/proc/32519/task/32521/stat : 32521 (SArTagnan) R 32517 32519 31855 0 -1 4202560 8090 0 0 0 624 1 0 0 25 0 4 0 520088118 281870336 12652 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376869 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=32519/tid=32522] ppid=32517 vsize=275264 CPUtime=6.28 cores=1,3,5,7
/proc/32519/task/32522/stat : 32522 (SArTagnan) R 32517 32519 31855 0 -1 4202560 38147 0 0 0 624 4 0 0 25 0 4 0 520088118 281870336 12653 33554432000 4194304 4623224 140734238586032 18446744073709551615 4366247 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 25.15
Current children cumulated vsize (KiB) 275264

[startup+12.7009 s]
/proc/loadavg: 7.63 7.97 7.97 9/184 32537
/proc/meminfo: memFree=24266900/32951124 swapFree=67111524/67111528
[pid=32519] ppid=32517 vsize=281124 CPUtime=50.71 cores=1,3,5,7
/proc/32519/stat : 32519 (SArTagnan) R 32517 32519 31855 0 -1 4202496 65553 0 0 0 5062 9 0 0 25 0 4 0 520088118 287870976 18138 33554432000 4194304 4623224 140734238586032 18446744073709551615 4517494 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/32519/statm: 70281 18138 400 105 0 65785 0
[pid=32519/tid=32520] ppid=32517 vsize=281124 CPUtime=12.68 cores=1,3,5,7
/proc/32519/task/32520/stat : 32520 (SArTagnan) R 32517 32519 31855 0 -1 4202560 11844 0 0 0 1267 1 0 0 25 0 4 0 520088118 287870976 18138 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376827 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=32519/tid=32521] ppid=32517 vsize=281124 CPUtime=12.62 cores=1,3,5,7
/proc/32519/task/32521/stat : 32521 (SArTagnan) R 32517 32519 31855 0 -1 4202560 9375 0 0 0 1260 2 0 0 25 0 4 0 520088118 287870976 18138 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376652 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=32519/tid=32522] ppid=32517 vsize=281124 CPUtime=12.69 cores=1,3,5,7
/proc/32519/task/32522/stat : 32522 (SArTagnan) R 32517 32519 31855 0 -1 4202560 40264 0 0 0 1264 5 0 0 25 0 4 0 520088118 287870976 18138 33554432000 4194304 4623224 140734238586032 18446744073709551615 4366391 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 50.71
Current children cumulated vsize (KiB) 281124

[startup+25.5006 s]
/proc/loadavg: 7.71 7.97 7.97 9/184 32537
/proc/meminfo: memFree=24199196/32951124 swapFree=67111524/67111528
[pid=32519] ppid=32517 vsize=304300 CPUtime=101.84 cores=1,3,5,7
/proc/32519/stat : 32519 (SArTagnan) R 32517 32519 31855 0 -1 4202496 78443 0 0 0 10172 12 0 0 25 0 4 0 520088118 311603200 24083 33554432000 4194304 4623224 140734238586032 18446744073709551615 4517501 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/32519/statm: 76075 24083 364 105 0 71579 0
[pid=32519/tid=32520] ppid=32517 vsize=304300 CPUtime=25.49 cores=1,3,5,7
/proc/32519/task/32520/stat : 32520 (SArTagnan) R 32517 32519 31855 0 -1 4202560 17917 0 0 0 2546 3 0 0 25 0 4 0 520088118 311603200 24083 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376711 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=32519/tid=32521] ppid=32517 vsize=304300 CPUtime=25.35 cores=1,3,5,7
/proc/32519/task/32521/stat : 32521 (SArTagnan) R 32517 32519 31855 0 -1 4202560 13089 0 0 0 2532 3 0 0 25 0 4 0 520088118 311603200 24083 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376827 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=32519/tid=32522] ppid=32517 vsize=304300 CPUtime=25.48 cores=1,3,5,7
/proc/32519/task/32522/stat : 32522 (SArTagnan) R 32517 32519 31855 0 -1 4202560 42939 0 0 0 2543 5 0 0 25 0 4 0 520088118 311603200 24083 33554432000 4194304 4623224 140734238586032 18446744073709551615 47056091097638 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 101.84
Current children cumulated vsize (KiB) 304300

[startup+51.1009 s]
/proc/loadavg: 7.81 7.97 7.97 9/184 32538
/proc/meminfo: memFree=24063884/32951124 swapFree=67111524/67111528
[pid=32519] ppid=32517 vsize=346220 CPUtime=204.1 cores=1,3,5,7
/proc/32519/stat : 32519 (SArTagnan) R 32517 32519 31855 0 -1 4202496 153198 0 0 0 20384 26 0 0 25 0 4 0 520088118 354529280 41828 33554432000 4194304 4623224 140734238586032 18446744073709551615 4510090 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/32519/statm: 86555 41828 628 105 0 82059 0
[pid=32519/tid=32520] ppid=32517 vsize=346220 CPUtime=51.09 cores=1,3,5,7
/proc/32519/task/32520/stat : 32520 (SArTagnan) R 32517 32519 31855 0 -1 4202560 26641 0 0 0 5104 5 0 0 25 0 4 0 520088118 354529280 41828 33554432000 4194304 4623224 140734238586032 18446744073709551615 4366412 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=32519/tid=32521] ppid=32517 vsize=346220 CPUtime=50.8 cores=1,3,5,7
/proc/32519/task/32521/stat : 32521 (SArTagnan) R 32517 32519 31855 0 -1 4202560 19311 0 0 0 5076 4 0 0 25 0 4 0 520088118 354529280 41828 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376629 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=32519/tid=32522] ppid=32517 vsize=346220 CPUtime=51.09 cores=1,3,5,7
/proc/32519/task/32522/stat : 32522 (SArTagnan) R 32517 32519 31855 0 -1 4202560 94615 0 0 0 5094 15 0 0 25 0 4 0 520088118 354529280 41828 33554432000 4194304 4623224 140734238586032 18446744073709551615 4545373 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 204.1
Current children cumulated vsize (KiB) 346220

[startup+102.307 s]
/proc/loadavg: 7.96 7.99 7.98 9/184 32539
/proc/meminfo: memFree=23895936/32951124 swapFree=67111524/67111528
[pid=32519] ppid=32517 vsize=375648 CPUtime=408.65 cores=1,3,5,7
/proc/32519/stat : 32519 (SArTagnan) R 32517 32519 31855 0 -1 4202496 179756 0 0 0 40831 34 0 0 25 0 4 0 520088118 384663552 61678 33554432000 4194304 4623224 140734238586032 18446744073709551615 4510090 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/32519/statm: 93912 61678 1072 105 0 89416 0
[pid=32519/tid=32520] ppid=32517 vsize=375648 CPUtime=102.3 cores=1,3,5,7
/proc/32519/task/32520/stat : 32520 (SArTagnan) R 32517 32519 31855 0 -1 4202560 31027 0 0 0 10223 7 0 0 25 0 4 0 520088118 384663552 61678 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376829 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=32519/tid=32521] ppid=32517 vsize=375648 CPUtime=101.73 cores=1,3,5,7
/proc/32519/task/32521/stat : 32521 (SArTagnan) R 32517 32519 31855 0 -1 4202560 27995 0 0 0 10166 7 0 0 25 0 4 0 520088118 384663552 61678 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376827 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=32519/tid=32522] ppid=32517 vsize=375648 CPUtime=102.29 cores=1,3,5,7
/proc/32519/task/32522/stat : 32522 (SArTagnan) R 32517 32519 31855 0 -1 4202560 100154 0 0 0 10213 16 0 0 25 0 4 0 520088118 384663552 61678 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376827 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 408.65
Current children cumulated vsize (KiB) 375648

[startup+162.301 s]

################
# More data... #
################

[pid=32519] ppid=32517 vsize=767912 CPUtime=2805.41 cores=1,3,5,7
/proc/32519/stat : 32519 (SArTagnan) R 32517 32519 31855 0 -1 4202496 2246108 0 0 0 280140 401 0 0 25 0 4 0 520088118 786341888 134588 33554432000 4194304 4623224 140734238586032 18446744073709551615 4510086 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/32519/statm: 191978 134588 344 105 0 187482 0
[pid=32519/tid=32520] ppid=32517 vsize=767912 CPUtime=702.3 cores=1,3,5,7
/proc/32519/task/32520/stat : 32520 (SArTagnan) R 32517 32519 31855 0 -1 4202560 93729 0 0 0 70209 21 0 0 25 0 4 0 520088118 786341888 134588 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376711 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=32519/tid=32521] ppid=32517 vsize=767912 CPUtime=698.47 cores=1,3,5,7
/proc/32519/task/32521/stat : 32521 (SArTagnan) R 32517 32519 31855 0 -1 4202560 43911 0 0 0 69833 14 0 0 25 0 4 0 520088118 786341888 134588 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376827 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=32519/tid=32522] ppid=32517 vsize=767912 CPUtime=702.3 cores=1,3,5,7
/proc/32519/task/32522/stat : 32522 (SArTagnan) R 32517 32519 31855 0 -1 4202560 1810102 0 0 0 69924 306 0 0 25 0 4 0 520088118 786341888 134588 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376823 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 2805.41
Current children cumulated vsize (KiB) 767912

[startup+762.306 s]
/proc/loadavg: 8.00 8.02 8.00 9/185 32558
/proc/meminfo: memFree=23007804/32951124 swapFree=67111524/67111528
[pid=32519] ppid=32517 vsize=812668 CPUtime=3045.12 cores=1,3,5,7
/proc/32519/stat : 32519 (SArTagnan) R 32517 32519 31855 0 -1 4202496 2541231 0 0 0 304062 450 0 0 25 0 4 0 520088118 832172032 147152 33554432000 4194304 4623224 140734238586032 18446744073709551615 4510169 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/32519/statm: 203167 147152 477 105 0 198671 0
[pid=32519/tid=32520] ppid=32517 vsize=812668 CPUtime=762.32 cores=1,3,5,7
/proc/32519/task/32520/stat : 32520 (SArTagnan) R 32517 32519 31855 0 -1 4202560 95563 0 0 0 76210 22 0 0 25 0 4 0 520088118 832172032 147152 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376711 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=32519/tid=32521] ppid=32517 vsize=812668 CPUtime=758.14 cores=1,3,5,7
/proc/32519/task/32521/stat : 32521 (SArTagnan) R 32517 32519 31855 0 -1 4202560 44016 0 0 0 75800 14 0 0 25 0 4 0 520088118 832172032 147152 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376827 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=32519/tid=32522] ppid=32517 vsize=812668 CPUtime=762.31 cores=1,3,5,7
/proc/32519/task/32522/stat : 32522 (SArTagnan) R 32517 32519 31855 0 -1 4202560 1964893 0 0 0 75903 328 0 0 25 0 4 0 520088118 832172032 147152 33554432000 4194304 4623224 140734238586032 18446744073709551615 4377128 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3045.12
Current children cumulated vsize (KiB) 812668

[startup+822.3 s]
/proc/loadavg: 8.00 8.02 8.00 9/184 32567
/proc/meminfo: memFree=22960380/32951124 swapFree=67111524/67111528
[pid=32519] ppid=32517 vsize=812668 CPUtime=3284.76 cores=1,3,5,7
/proc/32519/stat : 32519 (SArTagnan) R 32517 32519 31855 0 -1 4202496 2694077 0 0 0 327997 479 0 0 25 0 4 0 520088118 832172032 148965 33554432000 4194304 4623224 140734238586032 18446744073709551615 4511014 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/32519/statm: 203167 148965 466 105 0 198671 0
[pid=32519/tid=32520] ppid=32517 vsize=812668 CPUtime=822.3 cores=1,3,5,7
/proc/32519/task/32520/stat : 32520 (SArTagnan) R 32517 32519 31855 0 -1 4202560 96795 0 0 0 82208 22 0 0 25 0 4 0 520088118 832172032 148965 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376629 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=32519/tid=32521] ppid=32517 vsize=812668 CPUtime=817.81 cores=1,3,5,7
/proc/32519/task/32521/stat : 32521 (SArTagnan) R 32517 32519 31855 0 -1 4202560 44101 0 0 0 81767 14 0 0 25 0 4 0 520088118 832172032 148965 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376827 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=32519/tid=32522] ppid=32517 vsize=812668 CPUtime=822.3 cores=1,3,5,7
/proc/32519/task/32522/stat : 32522 (SArTagnan) R 32517 32519 31855 0 -1 4202560 1973654 0 0 0 81901 329 0 0 25 0 4 0 520088118 832172032 148965 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376711 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3284.76
Current children cumulated vsize (KiB) 812668

[startup+882.301 s]
/proc/loadavg: 8.04 8.03 8.00 9/184 32569
/proc/meminfo: memFree=22923012/32951124 swapFree=67111524/67111528
[pid=32519] ppid=32517 vsize=836668 CPUtime=3524.44 cores=1,3,5,7
/proc/32519/stat : 32519 (SArTagnan) R 32517 32519 31855 0 -1 4202496 2845936 0 0 0 351937 507 0 0 25 0 4 0 520088118 856748032 154717 33554432000 4194304 4623224 140734238586032 18446744073709551615 4510064 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/32519/statm: 209167 154717 466 105 0 204671 0
[pid=32519/tid=32520] ppid=32517 vsize=836668 CPUtime=882.3 cores=1,3,5,7
/proc/32519/task/32520/stat : 32520 (SArTagnan) R 32517 32519 31855 0 -1 4202560 106309 0 0 0 88206 24 0 0 25 0 4 0 520088118 856748032 154717 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376827 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=32519/tid=32521] ppid=32517 vsize=836668 CPUtime=877.48 cores=1,3,5,7
/proc/32519/task/32521/stat : 32521 (SArTagnan) R 32517 32519 31855 0 -1 4202560 44331 0 0 0 87734 14 0 0 25 0 4 0 520088118 856748032 154717 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376711 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=32519/tid=32522] ppid=32517 vsize=836668 CPUtime=882.31 cores=1,3,5,7
/proc/32519/task/32522/stat : 32522 (SArTagnan) R 32517 32519 31855 0 -1 4202560 1973957 0 0 0 87901 330 0 0 25 0 4 0 520088118 856748032 154717 33554432000 4194304 4623224 140734238586032 18446744073709551615 4550521 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3524.44
Current children cumulated vsize (KiB) 836668

[startup+942.301 s]
/proc/loadavg: 8.01 8.02 8.00 9/184 32570
/proc/meminfo: memFree=22879568/32951124 swapFree=67111524/67111528
[pid=32519] ppid=32517 vsize=871128 CPUtime=3764.12 cores=1,3,5,7
/proc/32519/stat : 32519 (SArTagnan) R 32517 32519 31855 0 -1 4202496 3001588 0 0 0 375873 539 0 0 25 0 4 0 520088118 892035072 161177 33554432000 4194304 4623224 140734238586032 18446744073709551615 4510316 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/32519/statm: 217782 161177 466 105 0 213286 0
[pid=32519/tid=32520] ppid=32517 vsize=871128 CPUtime=942.31 cores=1,3,5,7
/proc/32519/task/32520/stat : 32520 (SArTagnan) R 32517 32519 31855 0 -1 4202560 107350 0 0 0 94206 25 0 0 25 0 4 0 520088118 892035072 161177 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376742 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=32519/tid=32521] ppid=32517 vsize=871128 CPUtime=937.16 cores=1,3,5,7
/proc/32519/task/32521/stat : 32521 (SArTagnan) R 32517 32519 31855 0 -1 4202560 45614 0 0 0 93700 16 0 0 25 0 4 0 520088118 892035072 161177 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376629 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=32519/tid=32522] ppid=32517 vsize=871128 CPUtime=942.31 cores=1,3,5,7
/proc/32519/task/32522/stat : 32522 (SArTagnan) R 32517 32519 31855 0 -1 4202560 1974709 0 0 0 93901 330 0 0 25 0 4 0 520088118 892035072 161177 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376711 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3764.12
Current children cumulated vsize (KiB) 871128

[startup+1002.3 s]
/proc/loadavg: 8.21 8.08 8.02 9/184 32575
/proc/meminfo: memFree=22844580/32951124 swapFree=67111524/67111528
[pid=32519] ppid=32517 vsize=898616 CPUtime=4002.96 cores=1,3,5,7
/proc/32519/stat : 32519 (SArTagnan) R 32517 32519 31855 0 -1 4202496 3153190 0 0 0 399730 566 0 0 25 0 4 0 520088118 920182784 165024 33554432000 4194304 4623224 140734238586032 18446744073709551615 4510316 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/32519/statm: 224654 165024 466 105 0 220158 0
[pid=32519/tid=32520] ppid=32517 vsize=898616 CPUtime=1002.3 cores=1,3,5,7
/proc/32519/task/32520/stat : 32520 (SArTagnan) R 32517 32519 31855 0 -1 4202560 107686 0 0 0 100205 25 0 0 25 0 4 0 520088118 920182784 165024 33554432000 4194304 4623224 140734238586032 18446744073709551615 4366412 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=32519/tid=32521] ppid=32517 vsize=898616 CPUtime=995.99 cores=1,3,5,7
/proc/32519/task/32521/stat : 32521 (SArTagnan) R 32517 32519 31855 0 -1 4202560 58691 0 0 0 99581 18 0 0 25 0 4 0 520088118 920182784 165024 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376827 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=32519/tid=32522] ppid=32517 vsize=898616 CPUtime=1002.31 cores=1,3,5,7
/proc/32519/task/32522/stat : 32522 (SArTagnan) R 32517 32519 31855 0 -1 4202560 1975693 0 0 0 99901 330 0 0 25 0 4 0 520088118 920182784 165024 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376827 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4002.96
Current children cumulated vsize (KiB) 898616

[startup+1062.3 s]
/proc/loadavg: 8.11 8.07 8.02 9/184 32577
/proc/meminfo: memFree=22801892/32951124 swapFree=67111524/67111528
[pid=32519] ppid=32517 vsize=968840 CPUtime=4242.64 cores=1,3,5,7
/proc/32519/stat : 32519 (SArTagnan) R 32517 32519 31855 0 -1 4202496 3302450 0 0 0 423672 592 0 0 25 0 4 0 520088118 992092160 168171 33554432000 4194304 4623224 140734238586032 18446744073709551615 4510949 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/32519/statm: 242210 168171 466 105 0 237714 0
[pid=32519/tid=32520] ppid=32517 vsize=968840 CPUtime=1062.3 cores=1,3,5,7
/proc/32519/task/32520/stat : 32520 (SArTagnan) R 32517 32519 31855 0 -1 4202560 108101 0 0 0 106205 25 0 0 25 0 4 0 520088118 992092160 168171 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376827 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=32519/tid=32521] ppid=32517 vsize=968840 CPUtime=1055.67 cores=1,3,5,7
/proc/32519/task/32521/stat : 32521 (SArTagnan) R 32517 32519 31855 0 -1 4202560 59194 0 0 0 105548 19 0 0 25 0 4 0 520088118 992092160 168171 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376820 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=32519/tid=32522] ppid=32517 vsize=968840 CPUtime=1062.31 cores=1,3,5,7
/proc/32519/task/32522/stat : 32522 (SArTagnan) R 32517 32519 31855 0 -1 4202560 1976884 0 0 0 105900 331 0 0 25 0 4 0 520088118 992092160 168171 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376827 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4242.64
Current children cumulated vsize (KiB) 968840

[startup+1122.3 s]
/proc/loadavg: 8.11 8.09 8.02 9/184 32578
/proc/meminfo: memFree=22755084/32951124 swapFree=67111524/67111528
[pid=32519] ppid=32517 vsize=1032036 CPUtime=4482.32 cores=1,3,5,7
/proc/32519/stat : 32519 (SArTagnan) R 32517 32519 31855 0 -1 4202496 3475798 0 0 0 447611 621 0 0 25 0 4 0 520088118 1056804864 174166 33554432000 4194304 4623224 140734238586032 18446744073709551615 4355951 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/32519/statm: 258009 174166 462 105 0 253513 0
[pid=32519/tid=32520] ppid=32517 vsize=1032036 CPUtime=1122.31 cores=1,3,5,7
/proc/32519/task/32520/stat : 32520 (SArTagnan) R 32517 32519 31855 0 -1 4202560 140500 0 0 0 112202 29 0 0 25 0 4 0 520088118 1056804864 174166 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376827 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=32519/tid=32521] ppid=32517 vsize=1032036 CPUtime=1115.34 cores=1,3,5,7
/proc/32519/task/32521/stat : 32521 (SArTagnan) R 32517 32519 31855 0 -1 4202560 59360 0 0 0 111515 19 0 0 25 0 4 0 520088118 1056804864 174166 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376827 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=32519/tid=32522] ppid=32517 vsize=1032036 CPUtime=1122.32 cores=1,3,5,7
/proc/32519/task/32522/stat : 32522 (SArTagnan) R 32517 32519 31855 0 -1 4202560 1978083 0 0 0 111900 332 0 0 25 0 4 0 520088118 1056804864 174166 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376711 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4482.32
Current children cumulated vsize (KiB) 1032036

[startup+1182.3 s]
/proc/loadavg: 8.07 8.08 8.02 9/184 32580
/proc/meminfo: memFree=22697240/32951124 swapFree=67111524/67111528
[pid=32519] ppid=32517 vsize=1067400 CPUtime=4721.99 cores=1,3,5,7
/proc/32519/stat : 32519 (SArTagnan) R 32517 32519 31855 0 -1 4202496 3877458 0 0 0 471508 691 0 0 25 0 4 0 520088118 1093017600 179413 33554432000 4194304 4623224 140734238586032 18446744073709551615 4510169 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/32519/statm: 266850 179413 344 105 0 262354 0
[pid=32519/tid=32520] ppid=32517 vsize=1067400 CPUtime=1182.31 cores=1,3,5,7
/proc/32519/task/32520/stat : 32520 (SArTagnan) R 32517 32519 31855 0 -1 4202560 141609 0 0 0 118201 30 0 0 25 0 4 0 520088118 1093017600 179413 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376625 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=32519/tid=32521] ppid=32517 vsize=1067400 CPUtime=1175.01 cores=1,3,5,7
/proc/32519/task/32521/stat : 32521 (SArTagnan) R 32517 32519 31855 0 -1 4202560 60653 0 0 0 117480 21 0 0 25 0 4 0 520088118 1093017600 179413 33554432000 4194304 4623224 140734238586032 18446744073709551615 4289875 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=32519/tid=32522] ppid=32517 vsize=1067400 CPUtime=1182.32 cores=1,3,5,7
/proc/32519/task/32522/stat : 32522 (SArTagnan) R 32517 32519 31855 0 -1 4202560 2244337 0 0 0 117857 375 0 0 25 0 4 0 520088118 1093017600 179416 33554432000 4194304 4623224 140734238586032 18446744073709551615 4577678 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4721.99
Current children cumulated vsize (KiB) 1067400



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1201.9 s]
/proc/loadavg: 8.05 8.08 8.02 5/179 32581
/proc/meminfo: memFree=23686528/32951124 swapFree=67111524/67111528
[pid=32519] ppid=32517 vsize=1067400 CPUtime=4800.3 cores=1,3,5,7
/proc/32519/stat : 32519 (SArTagnan) R 32517 32519 31855 0 -1 4202496 4023253 0 0 0 479318 712 0 0 25 0 4 0 520088118 1093017600 180418 33554432000 4194304 4623224 140734238586032 18446744073709551615 212687718759 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/32519/statm: 266850 180418 353 105 0 262354 0
[pid=32519/tid=32520] ppid=32517 vsize=1067400 CPUtime=1201.92 cores=1,3,5,7
/proc/32519/task/32520/stat : 32520 (SArTagnan) R 32517 32519 31855 0 -1 4202560 141937 0 0 0 120162 30 0 0 25 0 4 0 520088118 1093017600 180418 33554432000 4194304 4623224 140734238586032 18446744073709551615 4278653 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=32519/tid=32521] ppid=32517 vsize=1067400 CPUtime=1194.51 cores=1,3,5,7
/proc/32519/task/32521/stat : 32521 (SArTagnan) R 32517 32519 31855 0 -1 4202560 60727 0 0 0 119430 21 0 0 25 0 4 0 520088118 1093017600 180418 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376625 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=32519/tid=32522] ppid=32517 vsize=1067400 CPUtime=1201.92 cores=1,3,5,7
/proc/32519/task/32522/stat : 32522 (SArTagnan) R 32517 32519 31855 0 -1 4202560 2351862 0 0 0 119802 390 0 0 25 0 4 0 520088118 1093017600 180418 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376827 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.3
Current children cumulated vsize (KiB) 1067400

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1201.9 s]
/proc/loadavg: 8.05 8.08 8.02 5/179 32581
/proc/meminfo: memFree=23686528/32951124 swapFree=67111524/67111528
[pid=32519] ppid=32517 vsize=1067400 CPUtime=4800.3 cores=1,3,5,7
/proc/32519/stat : 32519 (SArTagnan) R 32517 32519 31855 0 -1 4202496 4023253 0 0 0 479318 712 0 0 25 0 4 0 520088118 1093017600 180418 33554432000 4194304 4623224 140734238586032 18446744073709551615 212687718759 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/32519/statm: 266850 180418 353 105 0 262354 0
[pid=32519/tid=32520] ppid=32517 vsize=1067400 CPUtime=1201.92 cores=1,3,5,7
/proc/32519/task/32520/stat : 32520 (SArTagnan) R 32517 32519 31855 0 -1 4202560 141937 0 0 0 120162 30 0 0 25 0 4 0 520088118 1093017600 180418 33554432000 4194304 4623224 140734238586032 18446744073709551615 4278653 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=32519/tid=32521] ppid=32517 vsize=1067400 CPUtime=1194.51 cores=1,3,5,7
/proc/32519/task/32521/stat : 32521 (SArTagnan) R 32517 32519 31855 0 -1 4202560 60727 0 0 0 119430 21 0 0 25 0 4 0 520088118 1093017600 180418 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376625 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=32519/tid=32522] ppid=32517 vsize=1067400 CPUtime=1201.92 cores=1,3,5,7
/proc/32519/task/32522/stat : 32522 (SArTagnan) R 32517 32519 31855 0 -1 4202560 2351862 0 0 0 119802 390 0 0 25 0 4 0 520088118 1093017600 180418 33554432000 4194304 4623224 140734238586032 18446744073709551615 4376827 0 0 4096 2 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.3
Current children cumulated vsize (KiB) 1067400

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1201.94
CPU time (s): 4800.36
CPU user time (s): 4793.2
CPU system time (s): 7.16191
CPU usage (%): 399.383
Max. virtual memory (cumulated for all children) (KiB): 1148100

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4793.2
system time used= 7.16191
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 4023254
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 325
involuntary context switches= 32615

runsolver used 1.72374 second user time and 4.84026 second system time

The end

Launcher Data 

Begin job on node139 at 2011-04-29 15:48:24
IDJOB=3273547
IDBENCH=83141
IDSOLVER=1656
FILE ID=node139/3273547-1304084904
RUNJOBID= node139-1304084904-32510
PBS_JOBID= 13170525
Free space on /tmp= 72552 MiB

SOLVER NAME= SArTagnan 2011-03-24
BENCH NAME= SAT11/crafted/spence/sgen/sgen3-n180-s34713902-sat.cnf
COMMAND LINE= SArTagnan BENCHNAME -threads=NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3273547-1304084904/watcher-3273547-1304084904 -o /tmp/evaluation-result-3273547-1304084904/solver-3273547-1304084904 -C 4800 -W 1300 -M 15500  SArTagnan HOME/instance-3273547-1304084904.cnf -threads=4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 19292459de25c4f5e9e6f187c00d549d
RANDOM SEED=1566208462

node139.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5321.49
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      24412264 kB
Buffers:       2040444 kB
Cached:        5917152 kB
SwapCached:          4 kB
Active:         908244 kB
Inactive:      7112208 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      24412264 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:             440 kB
Writeback:          12 kB
AnonPages:       62744 kB
Mapped:          14388 kB
Slab:           453516 kB
PageTables:       4460 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   186608 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72552 MiB
End job on node139 at 2011-04-29 16:08:28