Trace number 3438636

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
SArTagnan 2011-05-15 (fixed)? (TO) 40000.6 5010.85

General information on the benchmark

Namecrafted/spence/sgen/
sgen3-n200-s39840374-sat.cnf
MD5SUM085a3847e14185843eddf941accab006
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark1.84272
Satisfiable
(Un)Satisfiability was proved
Number of variables200
Number of clauses480
Sum of the clauses size1200
Maximum clause length5
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2400
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 580
Number of clauses of size over 50

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 977)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3438636-1306920802/watcher-3438636-1306920802 -o /tmp/evaluation-result-3438636-1306920802/solver-3438636-1306920802 -C 40000 -W 5100 -M 31000 SArTagnan HOME/instance-3438636-1306920802.cnf -threads=8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.04 8.04 8.00 1/482 5105
/proc/meminfo: memFree=27647040/32951124 swapFree=63649284/67111528
[pid=5105] ppid=5103 vsize=90456 CPUtime=0.01 cores=0-7
/proc/5105/stat : 5105 (SArTagnan) R 5103 5105 4818 0 -1 4202496 484 0 0 0 1 0 0 0 25 0 8 0 803678710 92626944 420 33554432000 4194304 4625164 140736782465792 18446744073709551615 47846803576754 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/5105/statm: 22614 420 300 106 0 18117 0
[pid=5105/tid=5106] ppid=5103 vsize=287196 CPUtime=0.01 cores=0-7
/proc/5105/task/5106/stat : 5106 (SArTagnan) R 5103 5105 4818 0 -1 4202560 50 0 0 0 1 0 0 0 25 0 8 0 803678711 294088704 749 33554432000 4194304 4625164 140736782465792 18446744073709551615 210204752366 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=5105/tid=5107] ppid=5103 vsize=287196 CPUtime=0 cores=0-7
/proc/5105/task/5107/stat : 5107 (SArTagnan) R 5103 5105 4818 0 -1 4202560 2 0 0 0 0 0 0 0 25 0 8 0 803678711 294088704 750 33554432000 4194304 4625164 140736782465792 18446744073709551615 4340391 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=5105/tid=5108] ppid=5103 vsize=287196 CPUtime=0 cores=0-7
/proc/5105/task/5108/stat : 5108 (SArTagnan) R 5103 5105 4818 0 -1 4202560 29 0 0 0 0 0 0 0 25 0 8 0 803678711 294088704 750 33554432000 4194304 4625164 140736782465792 18446744073709551615 47846803577382 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=5105/tid=5109] ppid=5103 vsize=287196 CPUtime=0 cores=0-7
/proc/5105/task/5109/stat : 5109 (SArTagnan) R 5103 5105 4818 0 -1 4202560 54 0 0 0 0 0 0 0 25 0 8 0 803678711 294088704 750 33554432000 4194304 4625164 140736782465792 18446744073709551615 210204308165 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=5105/tid=5110] ppid=5103 vsize=287196 CPUtime=0 cores=0-7
/proc/5105/task/5110/stat : 5110 (SArTagnan) S 5103 5105 4818 0 -1 4202560 7 0 0 0 0 0 0 0 25 0 8 0 803678711 294088704 750 33554432000 4194304 4625164 140736782465792 18446744073709551615 210204752366 0 0 4096 2 18446604454944608320 0 0 -1 1 0 0 0
[pid=5105/tid=5111] ppid=5103 vsize=287196 CPUtime=0 cores=0-7
/proc/5105/task/5111/stat : 5111 (SArTagnan) R 5103 5105 4818 0 -1 4202560 52 0 0 0 0 0 0 0 25 0 8 0 803678711 294088704 751 33554432000 4194304 4625164 140736782465792 18446744073709551615 210220631067 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=5105/tid=5112] ppid=5103 vsize=287196 CPUtime=0 cores=0-7
/proc/5105/task/5112/stat : 5112 (SArTagnan) R 5103 5105 4818 0 -1 4202560 1 0 0 0 0 0 0 0 25 0 8 0 803678711 294088704 752 33554432000 4194304 4625164 140736782465792 18446744073709551615 210204752366 0 0 4096 2 0 0 0 -1 5 0 0 0

[startup+0.0442251 s]
/proc/loadavg: 8.04 8.04 8.00 1/482 5105
/proc/meminfo: memFree=27647040/32951124 swapFree=63649284/67111528
[pid=5105] ppid=5103 vsize=288320 CPUtime=0.22 cores=0-7
/proc/5105/stat : 5105 (SArTagnan) R 5103 5105 4818 0 -1 4202496 1314 0 0 0 22 0 0 0 18 0 8 0 803678710 295239680 1182 33554432000 4194304 4625164 140736782465792 18446744073709551615 4512009 0 0 4096 2 18446744073709551615 0 0 17 1 0 0 0
/proc/5105/statm: 72080 1182 352 106 0 67583 0
[pid=5105/tid=5106] ppid=5103 vsize=288320 CPUtime=0.02 cores=0-7
/proc/5105/task/5106/stat : 5106 (SArTagnan) R 5103 5105 4818 0 -1 4202560 70 0 0 0 2 0 0 0 25 0 8 0 803678711 295239680 1182 33554432000 4194304 4625164 140736782465792 18446744073709551615 4352843 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=5105/tid=5107] ppid=5103 vsize=288320 CPUtime=0.02 cores=0-7
/proc/5105/task/5107/stat : 5107 (SArTagnan) R 5103 5105 4818 0 -1 4202560 85 0 0 0 2 0 0 0 25 0 8 0 803678711 295239680 1182 33554432000 4194304 4625164 140736782465792 18446744073709551615 4465169 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=5105/tid=5108] ppid=5103 vsize=288320 CPUtime=0.03 cores=0-7
/proc/5105/task/5108/stat : 5108 (SArTagnan) R 5103 5105 4818 0 -1 4202560 107 0 0 0 3 0 0 0 25 0 8 0 803678711 295239680 1182 33554432000 4194304 4625164 140736782465792 18446744073709551615 4559888 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=5105/tid=5109] ppid=5103 vsize=288320 CPUtime=0.03 cores=0-7
/proc/5105/task/5109/stat : 5109 (SArTagnan) R 5103 5105 4818 0 -1 4202560 133 0 0 0 3 0 0 0 25 0 8 0 803678711 295239680 1182 33554432000 4194304 4625164 140736782465792 18446744073709551615 4352543 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=5105/tid=5110] ppid=5103 vsize=288320 CPUtime=0.02 cores=0-7
/proc/5105/task/5110/stat : 5110 (SArTagnan) R 5103 5105 4818 0 -1 4202560 91 0 0 0 2 0 0 0 25 0 8 0 803678711 295239680 1182 33554432000 4194304 4625164 140736782465792 18446744073709551615 4348864 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=5105/tid=5111] ppid=5103 vsize=288320 CPUtime=0.03 cores=0-7
/proc/5105/task/5111/stat : 5111 (SArTagnan) R 5103 5105 4818 0 -1 4202560 138 0 0 0 3 0 0 0 25 0 8 0 803678711 295239680 1182 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351839 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=5105/tid=5112] ppid=5103 vsize=288320 CPUtime=0 cores=0-7
/proc/5105/task/5112/stat : 5112 (SArTagnan) R 5103 5105 4818 0 -1 4202560 11 0 0 0 0 0 0 0 25 0 8 0 803678711 295239680 1182 33554432000 4194304 4625164 140736782465792 18446744073709551615 210204752366 0 0 4096 2 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 0.22
Current children cumulated vsize (KiB) 288320

[startup+0.14421 s]
/proc/loadavg: 8.04 8.04 8.00 1/482 5105
/proc/meminfo: memFree=27647040/32951124 swapFree=63649284/67111528
[pid=5105] ppid=5103 vsize=291132 CPUtime=0.93 cores=0-7
/proc/5105/stat : 5105 (SArTagnan) R 5103 5105 4818 0 -1 4202496 3450 0 0 0 93 0 0 0 18 0 8 0 803678710 298119168 2389 33554432000 4194304 4625164 140736782465792 18446744073709551615 4511904 0 0 4096 2 18446744073709551615 0 0 17 1 0 0 0
/proc/5105/statm: 72783 2390 352 106 0 68286 0
[pid=5105/tid=5106] ppid=5103 vsize=291132 CPUtime=0.12 cores=0-7
/proc/5105/task/5106/stat : 5106 (SArTagnan) R 5103 5105 4818 0 -1 4202560 392 0 0 0 12 0 0 0 25 0 8 0 803678711 298119168 2392 33554432000 4194304 4625164 140736782465792 18446744073709551615 4340391 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=5105/tid=5107] ppid=5103 vsize=291132 CPUtime=0.12 cores=0-7
/proc/5105/task/5107/stat : 5107 (SArTagnan) R 5103 5105 4818 0 -1 4202560 380 0 0 0 12 0 0 0 25 0 8 0 803678711 298119168 2392 33554432000 4194304 4625164 140736782465792 18446744073709551615 4465256 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=5105/tid=5108] ppid=5103 vsize=291132 CPUtime=0.12 cores=0-7
/proc/5105/task/5108/stat : 5108 (SArTagnan) R 5103 5105 4818 0 -1 4202560 721 0 0 0 12 0 0 0 18 0 8 0 803678711 298119168 2393 33554432000 4194304 4625164 140736782465792 18446744073709551615 210204308165 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=5105/tid=5109] ppid=5103 vsize=291132 CPUtime=0.13 cores=0-7
/proc/5105/task/5109/stat : 5109 (SArTagnan) R 5103 5105 4818 0 -1 4202560 349 0 0 0 13 0 0 0 25 0 8 0 803678711 298119168 2393 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=5105/tid=5110] ppid=5103 vsize=291132 CPUtime=0.12 cores=0-7
/proc/5105/task/5110/stat : 5110 (SArTagnan) R 5103 5105 4818 0 -1 4202560 383 0 0 0 12 0 0 0 25 0 8 0 803678711 298119168 2398 33554432000 4194304 4625164 140736782465792 18446744073709551615 4435080 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=5105/tid=5111] ppid=5103 vsize=291132 CPUtime=0.12 cores=0-7
/proc/5105/task/5111/stat : 5111 (SArTagnan) R 5103 5105 4818 0 -1 4202560 356 0 0 0 12 0 0 0 25 0 8 0 803678711 298119168 2399 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351777 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=5105/tid=5112] ppid=5103 vsize=291132 CPUtime=0.02 cores=0-7
/proc/5105/task/5112/stat : 5112 (SArTagnan) R 5103 5105 4818 0 -1 4202560 65 0 0 0 2 0 0 0 25 0 8 0 803678711 298119168 2400 33554432000 4194304 4625164 140736782465792 18446744073709551615 4352072 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 0.93
Current children cumulated vsize (KiB) 291132

[startup+0.301186 s]
/proc/loadavg: 8.04 8.04 8.00 1/482 5105
/proc/meminfo: memFree=27647040/32951124 swapFree=63649284/67111528
[pid=5105] ppid=5103 vsize=294636 CPUtime=2.19 cores=0-7
/proc/5105/stat : 5105 (SArTagnan) R 5103 5105 4818 0 -1 4202496 6893 0 0 0 218 1 0 0 18 0 8 0 803678710 301707264 4503 33554432000 4194304 4625164 140736782465792 18446744073709551615 4512475 0 0 4096 2 18446744073709551615 0 0 17 1 0 0 0
/proc/5105/statm: 73659 4503 355 106 0 69162 0
[pid=5105/tid=5106] ppid=5103 vsize=294636 CPUtime=0.28 cores=0-7
/proc/5105/task/5106/stat : 5106 (SArTagnan) R 5103 5105 4818 0 -1 4202560 618 0 0 0 28 0 0 0 25 0 8 0 803678711 301707264 4503 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351764 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=5105/tid=5107] ppid=5103 vsize=294636 CPUtime=0.28 cores=0-7
/proc/5105/task/5107/stat : 5107 (SArTagnan) R 5103 5105 4818 0 -1 4202560 846 0 0 0 28 0 0 0 25 0 8 0 803678711 301707264 4503 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=5105/tid=5108] ppid=5103 vsize=294636 CPUtime=0.28 cores=0-7
/proc/5105/task/5108/stat : 5108 (SArTagnan) R 5103 5105 4818 0 -1 4202560 1730 0 0 0 28 0 0 0 18 0 8 0 803678711 301707264 4503 33554432000 4194304 4625164 140736782465792 18446744073709551615 210220631067 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=5105/tid=5109] ppid=5103 vsize=294636 CPUtime=0.28 cores=0-7
/proc/5105/task/5109/stat : 5109 (SArTagnan) R 5103 5105 4818 0 -1 4202560 748 0 0 0 28 0 0 0 25 0 8 0 803678711 301707264 4503 33554432000 4194304 4625164 140736782465792 18446744073709551615 4352280 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=5105/tid=5110] ppid=5103 vsize=294636 CPUtime=0.28 cores=0-7
/proc/5105/task/5110/stat : 5110 (SArTagnan) R 5103 5105 4818 0 -1 4202560 654 0 0 0 28 0 0 0 25 0 8 0 803678711 301707264 4503 33554432000 4194304 4625164 140736782465792 18446744073709551615 4349027 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=5105/tid=5111] ppid=5103 vsize=294636 CPUtime=0.28 cores=0-7
/proc/5105/task/5111/stat : 5111 (SArTagnan) R 5103 5105 4818 0 -1 4202560 817 0 0 0 28 0 0 0 25 0 8 0 803678711 301707264 4503 33554432000 4194304 4625164 140736782465792 18446744073709551615 4465130 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=5105/tid=5112] ppid=5103 vsize=294636 CPUtime=0.18 cores=0-7
/proc/5105/task/5112/stat : 5112 (SArTagnan) R 5103 5105 4818 0 -1 4202560 497 0 0 0 18 0 0 0 25 0 8 0 803678711 301707264 4503 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351839 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 2.19
Current children cumulated vsize (KiB) 294636

[startup+0.70112 s]
/proc/loadavg: 8.04 8.04 8.00 1/482 5105
/proc/meminfo: memFree=27647040/32951124 swapFree=63649284/67111528
[pid=5105] ppid=5103 vsize=303960 CPUtime=5.38 cores=0-7
/proc/5105/stat : 5105 (SArTagnan) R 5103 5105 4818 0 -1 4202496 12993 0 0 0 537 1 0 0 18 0 8 0 803678710 311255040 7991 33554432000 4194304 4625164 140736782465792 18446744073709551615 210204308165 0 0 4096 2 18446744073709551615 0 0 17 1 0 0 0
/proc/5105/statm: 75990 7991 352 106 0 71493 0
[pid=5105/tid=5106] ppid=5103 vsize=303960 CPUtime=0.68 cores=0-7
/proc/5105/task/5106/stat : 5106 (SArTagnan) R 5103 5105 4818 0 -1 4202560 1325 0 0 0 68 0 0 0 25 0 8 0 803678711 311255040 7991 33554432000 4194304 4625164 140736782465792 18446744073709551615 4464310 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=5105/tid=5107] ppid=5103 vsize=303960 CPUtime=0.68 cores=0-7
/proc/5105/task/5107/stat : 5107 (SArTagnan) R 5103 5105 4818 0 -1 4202560 1388 0 0 0 68 0 0 0 25 0 8 0 803678711 311255040 7991 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351792 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=5105/tid=5108] ppid=5103 vsize=303960 CPUtime=0.68 cores=0-7
/proc/5105/task/5108/stat : 5108 (SArTagnan) R 5103 5105 4818 0 -1 4202560 3767 0 0 0 68 0 0 0 18 0 8 0 803678711 311255040 7991 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351918 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=5105/tid=5109] ppid=5103 vsize=303960 CPUtime=0.68 cores=0-7
/proc/5105/task/5109/stat : 5109 (SArTagnan) R 5103 5105 4818 0 -1 4202560 1266 0 0 0 68 0 0 0 25 0 8 0 803678711 311255040 7991 33554432000 4194304 4625164 140736782465792 18446744073709551615 4339639 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=5105/tid=5110] ppid=5103 vsize=303960 CPUtime=0.68 cores=0-7
/proc/5105/task/5110/stat : 5110 (SArTagnan) R 5103 5105 4818 0 -1 4202560 1368 0 0 0 68 0 0 0 25 0 8 0 803678711 311255040 7992 33554432000 4194304 4625164 140736782465792 18446744073709551615 4348867 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=5105/tid=5111] ppid=5103 vsize=303960 CPUtime=0.68 cores=0-7
/proc/5105/task/5111/stat : 5111 (SArTagnan) R 5103 5105 4818 0 -1 4202560 1713 0 0 0 68 0 0 0 25 0 8 0 803678711 311255040 7992 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=5105/tid=5112] ppid=5103 vsize=303960 CPUtime=0.58 cores=0-7
/proc/5105/task/5112/stat : 5112 (SArTagnan) R 5103 5105 4818 0 -1 4202560 894 0 0 0 58 0 0 0 25 0 8 0 803678711 311255040 7992 33554432000 4194304 4625164 140736782465792 18446744073709551615 4484694 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 5.38
Current children cumulated vsize (KiB) 303960

[startup+1.50099 s]
/proc/loadavg: 8.04 8.04 8.00 9/490 5113
/proc/meminfo: memFree=27606064/32951124 swapFree=63649284/67111528
[pid=5105] ppid=5103 vsize=316864 CPUtime=11.77 cores=0-7
/proc/5105/stat : 5105 (SArTagnan) R 5103 5105 4818 0 -1 4202496 23995 0 0 0 1174 3 0 0 20 0 8 0 803678710 324468736 13044 33554432000 4194304 4625164 140736782465792 18446744073709551615 4512046 0 0 4096 2 18446744073709551615 0 0 17 1 0 0 0
/proc/5105/statm: 79216 13044 352 106 0 74719 0
[pid=5105/tid=5106] ppid=5103 vsize=316864 CPUtime=1.47 cores=0-7
/proc/5105/task/5106/stat : 5106 (SArTagnan) R 5103 5105 4818 0 -1 4202560 2216 0 0 0 147 0 0 0 25 0 8 0 803678711 324468736 13044 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351830 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=5105/tid=5107] ppid=5103 vsize=316864 CPUtime=1.48 cores=0-7
/proc/5105/task/5107/stat : 5107 (SArTagnan) R 5103 5105 4818 0 -1 4202560 2099 0 0 0 148 0 0 0 25 0 8 0 803678711 324468736 13044 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351781 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=5105/tid=5108] ppid=5103 vsize=316864 CPUtime=1.48 cores=0-7
/proc/5105/task/5108/stat : 5108 (SArTagnan) R 5103 5105 4818 0 -1 4202560 8384 0 0 0 147 1 0 0 19 0 8 0 803678711 324468736 13044 33554432000 4194304 4625164 140736782465792 18446744073709551615 4339612 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=5105/tid=5109] ppid=5103 vsize=316864 CPUtime=1.48 cores=0-7
/proc/5105/task/5109/stat : 5109 (SArTagnan) R 5103 5105 4818 0 -1 4202560 2028 0 0 0 148 0 0 0 25 0 8 0 803678711 324468736 13044 33554432000 4194304 4625164 140736782465792 18446744073709551615 4340391 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=5105/tid=5110] ppid=5103 vsize=316864 CPUtime=1.46 cores=0-7
/proc/5105/task/5110/stat : 5110 (SArTagnan) R 5103 5105 4818 0 -1 4202560 2706 0 0 0 146 0 0 0 25 0 8 0 803678711 324468736 13044 33554432000 4194304 4625164 140736782465792 18446744073709551615 4349768 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=5105/tid=5111] ppid=5103 vsize=316864 CPUtime=1.48 cores=0-7
/proc/5105/task/5111/stat : 5111 (SArTagnan) R 5103 5105 4818 0 -1 4202560 2776 0 0 0 148 0 0 0 25 0 8 0 803678711 324468736 13044 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=5105/tid=5112] ppid=5103 vsize=316864 CPUtime=1.38 cores=0-7
/proc/5105/task/5112/stat : 5112 (SArTagnan) R 5103 5105 4818 0 -1 4202560 2051 0 0 0 138 0 0 0 25 0 8 0 803678711 324468736 13044 33554432000 4194304 4625164 140736782465792 18446744073709551615 4339612 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 11.77
Current children cumulated vsize (KiB) 316864

[startup+3.10072 s]
/proc/loadavg: 8.11 8.06 8.01 9/490 5113
/proc/meminfo: memFree=27580340/32951124 swapFree=63649284/67111528
[pid=5105] ppid=5103 vsize=331512 CPUtime=24.54 cores=0-7
/proc/5105/stat : 5105 (SArTagnan) R 5103 5105 4818 0 -1 4202496 43187 0 0 0 2449 5 0 0 25 0 8 0 803678710 339468288 20603 33554432000 4194304 4625164 140736782465792 18446744073709551615 4393268 0 0 4096 2 18446744073709551615 0 0 17 1 0 0 0
/proc/5105/statm: 82878 20603 352 106 0 78381 0
[pid=5105/tid=5106] ppid=5103 vsize=331512 CPUtime=3.07 cores=0-7
/proc/5105/task/5106/stat : 5106 (SArTagnan) R 5103 5105 4818 0 -1 4202560 3539 0 0 0 307 0 0 0 25 0 8 0 803678711 339468288 20603 33554432000 4194304 4625164 140736782465792 18446744073709551615 4352044 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=5105/tid=5107] ppid=5103 vsize=331512 CPUtime=3.07 cores=0-7
/proc/5105/task/5107/stat : 5107 (SArTagnan) R 5103 5105 4818 0 -1 4202560 4412 0 0 0 307 0 0 0 25 0 8 0 803678711 339468288 20603 33554432000 4194304 4625164 140736782465792 18446744073709551615 4352280 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=5105/tid=5108] ppid=5103 vsize=331512 CPUtime=3.08 cores=0-7
/proc/5105/task/5108/stat : 5108 (SArTagnan) R 5103 5105 4818 0 -1 4202560 16479 0 0 0 306 2 0 0 25 0 8 0 803678711 339468288 20603 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351777 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=5105/tid=5109] ppid=5103 vsize=331512 CPUtime=3.08 cores=0-7
/proc/5105/task/5109/stat : 5109 (SArTagnan) R 5103 5105 4818 0 -1 4202560 4420 0 0 0 308 0 0 0 25 0 8 0 803678711 339468288 20603 33554432000 4194304 4625164 140736782465792 18446744073709551615 4462583 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=5105/tid=5110] ppid=5103 vsize=331512 CPUtime=3.04 cores=0-7
/proc/5105/task/5110/stat : 5110 (SArTagnan) R 5103 5105 4818 0 -1 4202560 4339 0 0 0 304 0 0 0 25 0 8 0 803678711 339468288 20603 33554432000 4194304 4625164 140736782465792 18446744073709551615 4348817 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=5105/tid=5111] ppid=5103 vsize=331512 CPUtime=3.08 cores=0-7
/proc/5105/task/5111/stat : 5111 (SArTagnan) R 5103 5105 4818 0 -1 4202560 4462 0 0 0 308 0 0 0 25 0 8 0 803678711 339468288 20603 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351824 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=5105/tid=5112] ppid=5103 vsize=331512 CPUtime=2.97 cores=0-7
/proc/5105/task/5112/stat : 5112 (SArTagnan) R 5103 5105 4818 0 -1 4202560 3353 0 0 0 297 0 0 0 25 0 8 0 803678711 339468288 20603 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 24.54
Current children cumulated vsize (KiB) 331512

[startup+6.30119 s]
/proc/loadavg: 8.11 8.06 8.01 9/490 5113
/proc/meminfo: memFree=27534744/32951124 swapFree=63649284/67111528
[pid=5105] ppid=5103 vsize=361004 CPUtime=50.09 cores=0-7
/proc/5105/stat : 5105 (SArTagnan) R 5103 5105 4818 0 -1 4202496 77311 0 0 0 4999 10 0 0 25 0 8 0 803678710 369668096 31584 33554432000 4194304 4625164 140736782465792 18446744073709551615 4339559 0 0 4096 2 18446744073709551615 0 0 17 1 0 0 0
/proc/5105/statm: 90251 31584 375 106 0 85754 0
[pid=5105/tid=5106] ppid=5103 vsize=361004 CPUtime=6.27 cores=0-7
/proc/5105/task/5106/stat : 5106 (SArTagnan) R 5103 5105 4818 0 -1 4202560 5589 0 0 0 627 0 0 0 25 0 8 0 803678711 369668096 31584 33554432000 4194304 4625164 140736782465792 18446744073709551615 4353037 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=5105/tid=5107] ppid=5103 vsize=361004 CPUtime=6.27 cores=0-7
/proc/5105/task/5107/stat : 5107 (SArTagnan) R 5103 5105 4818 0 -1 4202560 6439 0 0 0 627 0 0 0 25 0 8 0 803678711 369668096 31584 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=5105/tid=5108] ppid=5103 vsize=361004 CPUtime=6.28 cores=0-7
/proc/5105/task/5108/stat : 5108 (SArTagnan) R 5103 5105 4818 0 -1 4202560 33166 0 0 0 624 4 0 0 25 0 8 0 803678711 369668096 31584 33554432000 4194304 4625164 140736782465792 18446744073709551615 210220631067 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=5105/tid=5109] ppid=5103 vsize=361004 CPUtime=6.28 cores=0-7
/proc/5105/task/5109/stat : 5109 (SArTagnan) R 5103 5105 4818 0 -1 4202560 6835 0 0 0 628 0 0 0 25 0 8 0 803678711 369668096 31584 33554432000 4194304 4625164 140736782465792 18446744073709551615 4339612 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=5105/tid=5110] ppid=5103 vsize=361004 CPUtime=6.18 cores=0-7
/proc/5105/task/5110/stat : 5110 (SArTagnan) R 5103 5105 4818 0 -1 4202560 6172 0 0 0 618 0 0 0 25 0 8 0 803678711 369668096 31584 33554432000 4194304 4625164 140736782465792 18446744073709551615 4348864 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=5105/tid=5111] ppid=5103 vsize=361004 CPUtime=6.28 cores=0-7
/proc/5105/task/5111/stat : 5111 (SArTagnan) R 5103 5105 4818 0 -1 4202560 8852 0 0 0 627 1 0 0 25 0 8 0 803678711 369668096 31584 33554432000 4194304 4625164 140736782465792 18446744073709551615 4352098 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=5105/tid=5112] ppid=5103 vsize=361004 CPUtime=6.18 cores=0-7
/proc/5105/task/5112/stat : 5112 (SArTagnan) R 5103 5105 4818 0 -1 4202560 7198 0 0 0 617 1 0 0 25 0 8 0 803678711 369668096 31584 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351801 0 0 4096 2 0 0 0 -1 2 0 0 0

################
# More data... #
################

[pid=5105/tid=5108] ppid=5103 vsize=5400876 CPUtime=4607.46 cores=0-7
/proc/5105/task/5108/stat : 5108 (SArTagnan) R 5103 5105 4818 0 -1 4202560 14604009 0 0 0 458266 2480 0 0 25 0 8 0 803678711 5530497024 1092765 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=5105/tid=5109] ppid=5103 vsize=5400876 CPUtime=4662.2 cores=0-7
/proc/5105/task/5109/stat : 5109 (SArTagnan) R 5103 5105 4818 0 -1 4202560 281822 0 0 0 466164 56 0 0 25 0 8 0 803678711 5530497024 1092765 33554432000 4194304 4625164 140736782465792 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=5105/tid=5110] ppid=5103 vsize=5400876 CPUtime=4639.63 cores=0-7
/proc/5105/task/5110/stat : 5110 (SArTagnan) R 5103 5105 4818 0 -1 4202560 174099 0 0 0 463924 39 0 0 25 0 8 0 803678711 5530497024 1092765 33554432000 4194304 4625164 140736782465792 18446744073709551615 4348821 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=5105/tid=5111] ppid=5103 vsize=5400876 CPUtime=4662.33 cores=0-7
/proc/5105/task/5111/stat : 5111 (SArTagnan) R 5103 5105 4818 0 -1 4202560 382027 0 0 0 466154 79 0 0 25 0 8 0 803678711 5530497024 1092765 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=5105/tid=5112] ppid=5103 vsize=5400876 CPUtime=4662.21 cores=0-7
/proc/5105/task/5112/stat : 5112 (SArTagnan) R 5103 5105 4818 0 -1 4202560 229386 0 0 0 466177 44 0 0 25 0 8 0 803678711 5530497024 1092765 33554432000 4194304 4625164 140736782465792 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 37219.7
Current children cumulated vsize (KiB) 5400876

[startup+4722.32 s]
/proc/loadavg: 8.08 8.04 8.00 9/490 5251
/proc/meminfo: memFree=23247852/32951124 swapFree=63649284/67111528
[pid=5105] ppid=5103 vsize=5402016 CPUtime=37698.8 cores=0-7
/proc/5105/stat : 5105 (SArTagnan) R 5103 5105 4818 0 -1 4202496 21454051 0 0 0 3766160 3719 0 0 25 0 8 0 803678710 5531664384 1097098 33554432000 4194304 4625164 140736782465792 18446744073709551615 4511904 0 0 4096 2 18446744073709551615 0 0 17 1 0 0 0
/proc/5105/statm: 1350504 1097098 352 106 0 1346007 0
[pid=5105/tid=5106] ppid=5103 vsize=5402016 CPUtime=4722.33 cores=0-7
/proc/5105/task/5106/stat : 5106 (SArTagnan) R 5103 5105 4818 0 -1 4202560 215241 0 0 0 472186 47 0 0 25 0 8 0 803678711 5531664384 1097098 33554432000 4194304 4625164 140736782465792 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=5105/tid=5107] ppid=5103 vsize=5402016 CPUtime=4721.12 cores=0-7
/proc/5105/task/5107/stat : 5107 (SArTagnan) R 5103 5105 4818 0 -1 4202560 273271 0 0 0 472061 51 0 0 25 0 8 0 803678711 5531664384 1097098 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351781 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=5105/tid=5108] ppid=5103 vsize=5402016 CPUtime=4666.48 cores=0-7
/proc/5105/task/5108/stat : 5108 (SArTagnan) R 5103 5105 4818 0 -1 4202560 14817565 0 0 0 464130 2518 0 0 25 0 8 0 803678711 5531664384 1097098 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=5105/tid=5109] ppid=5103 vsize=5402016 CPUtime=4722.22 cores=0-7
/proc/5105/task/5109/stat : 5109 (SArTagnan) R 5103 5105 4818 0 -1 4202560 281907 0 0 0 472166 56 0 0 25 0 8 0 803678711 5531664384 1097098 33554432000 4194304 4625164 140736782465792 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=5105/tid=5110] ppid=5103 vsize=5402016 CPUtime=4699.64 cores=0-7
/proc/5105/task/5110/stat : 5110 (SArTagnan) R 5103 5105 4818 0 -1 4202560 174808 0 0 0 469925 39 0 0 25 0 8 0 803678711 5531664384 1097098 33554432000 4194304 4625164 140736782465792 18446744073709551615 4348864 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=5105/tid=5111] ppid=5103 vsize=5402016 CPUtime=4722.35 cores=0-7
/proc/5105/task/5111/stat : 5111 (SArTagnan) R 5103 5105 4818 0 -1 4202560 382229 0 0 0 472156 79 0 0 25 0 8 0 803678711 5531664384 1097098 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=5105/tid=5112] ppid=5103 vsize=5402016 CPUtime=4722.24 cores=0-7
/proc/5105/task/5112/stat : 5112 (SArTagnan) R 5103 5105 4818 0 -1 4202560 229474 0 0 0 472179 45 0 0 25 0 8 0 803678711 5531664384 1097098 33554432000 4194304 4625164 140736782465792 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 37698.8
Current children cumulated vsize (KiB) 5402016

[startup+4782.3 s]
/proc/loadavg: 8.08 8.05 8.00 9/490 5252
/proc/meminfo: memFree=23227204/32951124 swapFree=63649284/67111528
[pid=5105] ppid=5103 vsize=5403312 CPUtime=38177.6 cores=0-7
/proc/5105/stat : 5105 (SArTagnan) R 5103 5105 4818 0 -1 4202496 21639510 0 0 0 3814012 3752 0 0 25 0 8 0 803678710 5532991488 1102450 33554432000 4194304 4625164 140736782465792 18446744073709551615 4511930 0 0 4096 2 18446744073709551615 0 0 17 1 0 0 0
/proc/5105/statm: 1350828 1102450 355 106 0 1346331 0
[pid=5105/tid=5106] ppid=5103 vsize=5403312 CPUtime=4782.31 cores=0-7
/proc/5105/task/5106/stat : 5106 (SArTagnan) R 5103 5105 4818 0 -1 4202560 215330 0 0 0 478184 47 0 0 25 0 8 0 803678711 5532991488 1102450 33554432000 4194304 4625164 140736782465792 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=5105/tid=5107] ppid=5103 vsize=5403312 CPUtime=4781.09 cores=0-7
/proc/5105/task/5107/stat : 5107 (SArTagnan) R 5103 5105 4818 0 -1 4202560 273428 0 0 0 478058 51 0 0 25 0 8 0 803678711 5532991488 1102450 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351795 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=5105/tid=5108] ppid=5103 vsize=5403312 CPUtime=4725.48 cores=0-7
/proc/5105/task/5108/stat : 5108 (SArTagnan) R 5103 5105 4818 0 -1 4202560 14936275 0 0 0 470008 2540 0 0 25 0 8 0 803678711 5532991488 1102450 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=5105/tid=5109] ppid=5103 vsize=5403312 CPUtime=4782.2 cores=0-7
/proc/5105/task/5109/stat : 5109 (SArTagnan) R 5103 5105 4818 0 -1 4202560 281994 0 0 0 478164 56 0 0 25 0 8 0 803678711 5532991488 1102450 33554432000 4194304 4625164 140736782465792 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=5105/tid=5110] ppid=5103 vsize=5403312 CPUtime=4759.63 cores=0-7
/proc/5105/task/5110/stat : 5110 (SArTagnan) R 5103 5105 4818 0 -1 4202560 175058 0 0 0 475923 40 0 0 25 0 8 0 803678711 5532991488 1102450 33554432000 4194304 4625164 140736782465792 18446744073709551615 4349029 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=5105/tid=5111] ppid=5103 vsize=5403312 CPUtime=4782.33 cores=0-7
/proc/5105/task/5111/stat : 5111 (SArTagnan) R 5103 5105 4818 0 -1 4202560 384449 0 0 0 478153 80 0 0 25 0 8 0 803678711 5532991488 1102450 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=5105/tid=5112] ppid=5103 vsize=5403312 CPUtime=4782.22 cores=0-7
/proc/5105/task/5112/stat : 5112 (SArTagnan) R 5103 5105 4818 0 -1 4202560 230010 0 0 0 478177 45 0 0 25 0 8 0 803678711 5532991488 1102450 33554432000 4194304 4625164 140736782465792 18446744073709551615 4488046 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 38177.6
Current children cumulated vsize (KiB) 5403312

[startup+4842.3 s]
/proc/loadavg: 8.08 8.05 8.00 9/490 5254
/proc/meminfo: memFree=23210156/32951124 swapFree=63649284/67111528
[pid=5105] ppid=5103 vsize=5405684 CPUtime=38656.6 cores=0-7
/proc/5105/stat : 5105 (SArTagnan) R 5103 5105 4818 0 -1 4202496 21842258 0 0 0 3861875 3788 0 0 25 0 8 0 803678710 5535420416 1106668 33554432000 4194304 4625164 140736782465792 18446744073709551615 4511933 0 0 4096 2 18446744073709551615 0 0 17 1 0 0 0
/proc/5105/statm: 1351421 1106668 352 106 0 1346924 0
[pid=5105/tid=5106] ppid=5103 vsize=5405684 CPUtime=4842.31 cores=0-7
/proc/5105/task/5106/stat : 5106 (SArTagnan) R 5103 5105 4818 0 -1 4202560 215416 0 0 0 484184 47 0 0 25 0 8 0 803678711 5535420416 1106668 33554432000 4194304 4625164 140736782465792 18446744073709551615 4330859 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=5105/tid=5107] ppid=5103 vsize=5405684 CPUtime=4841.07 cores=0-7
/proc/5105/task/5107/stat : 5107 (SArTagnan) R 5103 5105 4818 0 -1 4202560 273683 0 0 0 484056 51 0 0 25 0 8 0 803678711 5535420416 1106668 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=5105/tid=5108] ppid=5103 vsize=5405684 CPUtime=4784.49 cores=0-7
/proc/5105/task/5108/stat : 5108 (SArTagnan) R 5103 5105 4818 0 -1 4202560 15036524 0 0 0 475889 2560 0 0 25 0 8 0 803678711 5535420416 1106668 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=5105/tid=5109] ppid=5103 vsize=5405684 CPUtime=4842.19 cores=0-7
/proc/5105/task/5109/stat : 5109 (SArTagnan) R 5103 5105 4818 0 -1 4202560 282071 0 0 0 484163 56 0 0 25 0 8 0 803678711 5535420416 1106668 33554432000 4194304 4625164 140736782465792 18446744073709551615 4488020 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=5105/tid=5110] ppid=5103 vsize=5405684 CPUtime=4819.63 cores=0-7
/proc/5105/task/5110/stat : 5110 (SArTagnan) R 5103 5105 4818 0 -1 4202560 175239 0 0 0 481923 40 0 0 25 0 8 0 803678711 5535420416 1106668 33554432000 4194304 4625164 140736782465792 18446744073709551615 4348864 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=5105/tid=5111] ppid=5103 vsize=5405684 CPUtime=4842.33 cores=0-7
/proc/5105/task/5111/stat : 5111 (SArTagnan) R 5103 5105 4818 0 -1 4202560 385504 0 0 0 484153 80 0 0 25 0 8 0 803678711 5535420416 1106668 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=5105/tid=5112] ppid=5103 vsize=5405684 CPUtime=4842.22 cores=0-7
/proc/5105/task/5112/stat : 5112 (SArTagnan) R 5103 5105 4818 0 -1 4202560 230099 0 0 0 484177 45 0 0 25 0 8 0 803678711 5535420416 1106668 33554432000 4194304 4625164 140736782465792 18446744073709551615 4488046 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 38656.6
Current children cumulated vsize (KiB) 5405684

[startup+4902.3 s]
/proc/loadavg: 8.17 8.08 8.01 9/490 5256
/proc/meminfo: memFree=23195468/32951124 swapFree=63649284/67111528
[pid=5105] ppid=5103 vsize=5408220 CPUtime=39135.6 cores=0-7
/proc/5105/stat : 5105 (SArTagnan) R 5103 5105 4818 0 -1 4202496 22054293 0 0 0 3909738 3826 0 0 25 0 8 0 803678710 5538017280 1109926 33554432000 4194304 4625164 140736782465792 18446744073709551615 4511904 0 0 4096 2 18446744073709551615 0 0 17 1 0 0 0
/proc/5105/statm: 1352055 1109926 352 106 0 1347558 0
[pid=5105/tid=5106] ppid=5103 vsize=5408220 CPUtime=4902.31 cores=0-7
/proc/5105/task/5106/stat : 5106 (SArTagnan) R 5103 5105 4818 0 -1 4202560 215503 0 0 0 490184 47 0 0 25 0 8 0 803678711 5538017280 1109926 33554432000 4194304 4625164 140736782465792 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=5105/tid=5107] ppid=5103 vsize=5408220 CPUtime=4901.06 cores=0-7
/proc/5105/task/5107/stat : 5107 (SArTagnan) R 5103 5105 4818 0 -1 4202560 273802 0 0 0 490055 51 0 0 25 0 8 0 803678711 5538017280 1109926 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351792 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=5105/tid=5108] ppid=5103 vsize=5408220 CPUtime=4843.51 cores=0-7
/proc/5105/task/5108/stat : 5108 (SArTagnan) R 5103 5105 4818 0 -1 4202560 15199820 0 0 0 481762 2589 0 0 25 0 8 0 803678711 5538017280 1109926 33554432000 4194304 4625164 140736782465792 18446744073709551615 4335407 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=5105/tid=5109] ppid=5103 vsize=5408220 CPUtime=4902.19 cores=0-7
/proc/5105/task/5109/stat : 5109 (SArTagnan) R 5103 5105 4818 0 -1 4202560 282157 0 0 0 490163 56 0 0 25 0 8 0 803678711 5538017280 1109926 33554432000 4194304 4625164 140736782465792 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=5105/tid=5110] ppid=5103 vsize=5408220 CPUtime=4879.63 cores=0-7
/proc/5105/task/5110/stat : 5110 (SArTagnan) R 5103 5105 4818 0 -1 4202560 175500 0 0 0 487923 40 0 0 25 0 8 0 803678711 5538017280 1109926 33554432000 4194304 4625164 140736782465792 18446744073709551615 4339616 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=5105/tid=5111] ppid=5103 vsize=5408220 CPUtime=4902.33 cores=0-7
/proc/5105/task/5111/stat : 5111 (SArTagnan) R 5103 5105 4818 0 -1 4202560 385605 0 0 0 490153 80 0 0 25 0 8 0 803678711 5538017280 1109926 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=5105/tid=5112] ppid=5103 vsize=5408220 CPUtime=4902.22 cores=0-7
/proc/5105/task/5112/stat : 5112 (SArTagnan) R 5103 5105 4818 0 -1 4202560 230188 0 0 0 490177 45 0 0 25 0 8 0 803678711 5538017280 1109926 33554432000 4194304 4625164 140736782465792 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 39135.6
Current children cumulated vsize (KiB) 5408220

[startup+4962.3 s]
/proc/loadavg: 8.15 8.10 8.02 9/490 5257
/proc/meminfo: memFree=23181528/32951124 swapFree=63649284/67111528
[pid=5105] ppid=5103 vsize=5410624 CPUtime=39614.6 cores=0-7
/proc/5105/stat : 5105 (SArTagnan) R 5103 5105 4818 0 -1 4202496 22321197 0 0 0 3957588 3876 0 0 25 0 8 0 803678710 5540478976 1113826 33554432000 4194304 4625164 140736782465792 18446744073709551615 4512009 0 0 4096 2 18446744073709551615 0 0 17 1 0 0 0
/proc/5105/statm: 1352656 1113826 353 106 0 1348159 0
[pid=5105/tid=5106] ppid=5103 vsize=5410624 CPUtime=4962.31 cores=0-7
/proc/5105/task/5106/stat : 5106 (SArTagnan) R 5103 5105 4818 0 -1 4202560 215586 0 0 0 496184 47 0 0 25 0 8 0 803678711 5540478976 1113826 33554432000 4194304 4625164 140736782465792 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=5105/tid=5107] ppid=5103 vsize=5410624 CPUtime=4961.04 cores=0-7
/proc/5105/task/5107/stat : 5107 (SArTagnan) R 5103 5105 4818 0 -1 4202560 273957 0 0 0 496053 51 0 0 25 0 8 0 803678711 5540478976 1113826 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=5105/tid=5108] ppid=5103 vsize=5410624 CPUtime=4902.5 cores=0-7
/proc/5105/task/5108/stat : 5108 (SArTagnan) R 5103 5105 4818 0 -1 4202560 15367145 0 0 0 487629 2621 0 0 25 0 8 0 803678711 5540478976 1113826 33554432000 4194304 4625164 140736782465792 18446744073709551615 4352002 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=5105/tid=5109] ppid=5103 vsize=5410624 CPUtime=4962.2 cores=0-7
/proc/5105/task/5109/stat : 5109 (SArTagnan) R 5103 5105 4818 0 -1 4202560 282240 0 0 0 496163 57 0 0 25 0 8 0 803678711 5540478976 1113826 33554432000 4194304 4625164 140736782465792 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=5105/tid=5110] ppid=5103 vsize=5410624 CPUtime=4939.63 cores=0-7
/proc/5105/task/5110/stat : 5110 (SArTagnan) R 5103 5105 4818 0 -1 4202560 175590 0 0 0 493923 40 0 0 25 0 8 0 803678711 5540478976 1113826 33554432000 4194304 4625164 140736782465792 18446744073709551615 210204314501 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=5105/tid=5111] ppid=5103 vsize=5410624 CPUtime=4962.33 cores=0-7
/proc/5105/task/5111/stat : 5111 (SArTagnan) R 5103 5105 4818 0 -1 4202560 385748 0 0 0 496153 80 0 0 25 0 8 0 803678711 5540478976 1113826 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=5105/tid=5112] ppid=5103 vsize=5410624 CPUtime=4962.22 cores=0-7
/proc/5105/task/5112/stat : 5112 (SArTagnan) R 5103 5105 4818 0 -1 4202560 230282 0 0 0 496177 45 0 0 25 0 8 0 803678711 5540478976 1113826 33554432000 4194304 4625164 140736782465792 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 39614.6
Current children cumulated vsize (KiB) 5410624



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5010.6 s]
/proc/loadavg: 8.12 8.09 8.02 9/490 5258
/proc/meminfo: memFree=23169924/32951124 swapFree=63649284/67111528
[pid=5105] ppid=5103 vsize=5414036 CPUtime=40000.3 cores=0-7
/proc/5105/stat : 5105 (SArTagnan) R 5103 5105 4818 0 -1 4202496 22526495 0 0 0 3996117 3911 0 0 25 0 8 0 803678710 5543972864 1116675 33554432000 4194304 4625164 140736782465792 18446744073709551615 4519268 0 0 4096 2 18446744073709551615 0 0 17 1 0 0 0
/proc/5105/statm: 1353509 1116675 352 106 0 1349012 0
[pid=5105/tid=5106] ppid=5103 vsize=5414036 CPUtime=5010.63 cores=0-7
/proc/5105/task/5106/stat : 5106 (SArTagnan) R 5103 5105 4818 0 -1 4202560 215650 0 0 0 501016 47 0 0 25 0 8 0 803678711 5543972864 1116675 33554432000 4194304 4625164 140736782465792 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=5105/tid=5107] ppid=5103 vsize=5414036 CPUtime=5009.34 cores=0-7
/proc/5105/task/5107/stat : 5107 (SArTagnan) R 5103 5105 4818 0 -1 4202560 274092 0 0 0 500883 51 0 0 25 0 8 0 803678711 5543972864 1116675 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=5105/tid=5108] ppid=5103 vsize=5414036 CPUtime=4950.01 cores=0-7
/proc/5105/task/5108/stat : 5108 (SArTagnan) R 5103 5105 4818 0 -1 4202560 15517931 0 0 0 492353 2648 0 0 25 0 8 0 803678711 5543972864 1116675 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=5105/tid=5109] ppid=5103 vsize=5414036 CPUtime=5010.52 cores=0-7
/proc/5105/task/5109/stat : 5109 (SArTagnan) R 5103 5105 4818 0 -1 4202560 282301 0 0 0 500995 57 0 0 25 0 8 0 803678711 5543972864 1116675 33554432000 4194304 4625164 140736782465792 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=5105/tid=5110] ppid=5103 vsize=5414036 CPUtime=4987.95 cores=0-7
/proc/5105/task/5110/stat : 5110 (SArTagnan) R 5103 5105 4818 0 -1 4202560 175699 0 0 0 498755 40 0 0 25 0 8 0 803678711 5543972864 1116675 33554432000 4194304 4625164 140736782465792 18446744073709551615 4349019 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=5105/tid=5111] ppid=5103 vsize=5414036 CPUtime=5010.65 cores=0-7
/proc/5105/task/5111/stat : 5111 (SArTagnan) R 5103 5105 4818 0 -1 4202560 386135 0 0 0 500985 80 0 0 25 0 8 0 803678711 5543972864 1116675 33554432000 4194304 4625164 140736782465792 18446744073709551615 4351764 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=5105/tid=5112] ppid=5103 vsize=5414036 CPUtime=5010.54 cores=0-7
/proc/5105/task/5112/stat : 5112 (SArTagnan) R 5103 5105 4818 0 -1 4202560 230354 0 0 0 501009 45 0 0 25 0 8 0 803678711 5543972864 1116675 33554432000 4194304 4625164 140736782465792 18446744073709551615 4488020 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 40000.3
Current children cumulated vsize (KiB) 5414036

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5010.8 s]
/proc/loadavg: 8.12 8.09 8.02 9/490 5258
/proc/meminfo: memFree=23169924/32951124 swapFree=63649284/67111528
[pid=5105] ppid=5103 vsize=0 CPUtime=40000.6 cores=0-7
/proc/5105/stat : 5105 (SArTagnan) Z 5103 5105 4818 0 -1 4203548 22526495 0 0 0 3996128 3928 0 0 25 0 2 0 803678710 0 0 33554432000 0 0 0 0 0 0 0 4096 2 18446744073709551615 0 0 17 1 0 0 0
/proc/5105/statm: 0 0 0 0 0 0 0
[pid=5105/tid=5108] ppid=5103 vsize=0 CPUtime=4950.18 cores=0-7
/proc/5105/task/5108/stat : 5108 (SArTagnan) R 5103 5105 4818 0 -1 4203588 15517931 0 0 0 492353 2665 0 0 25 0 2 0 803678711 0 0 33554432000 0 0 0 0 0 0 0 4096 2 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 40000.6
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5010.85
CPU time (s): 40000.6
CPU user time (s): 39961.3
CPU system time (s): 39.33
CPU usage (%): 798.281
Max. virtual memory (cumulated for all children) (KiB): 5414036

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39961.3
system time used= 39.33
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 22526495
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2322
involuntary context switches= 167090

runsolver used 22.2506 second user time and 60.2148 second system time

The end

Launcher Data 

Begin job on node130 at 2011-06-01 11:33:22
IDJOB=3438636
IDBENCH=83142
IDSOLVER=1936
FILE ID=node130/3438636-1306920802
RUNJOBID= node130-1306915788-4832
PBS_JOBID= 13488513
Free space on /tmp= 73808 MiB

SOLVER NAME= SArTagnan 2011-05-15 (fixed)
BENCH NAME= SAT11/crafted/spence/sgen/sgen3-n200-s39840374-sat.cnf
COMMAND LINE= SArTagnan BENCHNAME -threads=NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3438636-1306920802/watcher-3438636-1306920802 -o /tmp/evaluation-result-3438636-1306920802/solver-3438636-1306920802 -C 40000 -W 5100 -M 31000  SArTagnan HOME/instance-3438636-1306920802.cnf -threads=8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 085a3847e14185843eddf941accab006
RANDOM SEED=1125135518

node130.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      27647428 kB
Buffers:        123752 kB
Cached:         123652 kB
SwapCached:       9700 kB
Active:        4899912 kB
Inactive:       127264 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      27647428 kB
SwapTotal:    67111528 kB
SwapFree:     63649284 kB
Dirty:             428 kB
Writeback:           0 kB
AnonPages:     4778540 kB
Mapped:          30332 kB
Slab:           151464 kB
PageTables:      53520 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 145071808 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73808 MiB
End job on node130 at 2011-06-01 12:56:55