Trace number 3438671

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
SArTagnan 2011-05-15 (fixed)? (TO) 40001 5010.65

General information on the benchmark

Namecrafted/spence/sgen/
sgen3-n280-s93756482-sat.cnf
MD5SUMbadb3b4ac12b29ada7dd8ed96ed75c80
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark68.6746
Satisfiable
(Un)Satisfiability was proved
Number of variables280
Number of clauses672
Sum of the clauses size1680
Maximum clause length5
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2560
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 5112
Number of clauses of size over 50

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 977)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3438671-1306921347/watcher-3438671-1306921347 -o /tmp/evaluation-result-3438671-1306921347/solver-3438671-1306921347 -C 40000 -W 5100 -M 31000 SArTagnan HOME/instance-3438671-1306921347.cnf -threads=8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.38 7.91 7.96 2/470 9968
/proc/meminfo: memFree=22314744/32951124 swapFree=67057928/67111528
[pid=9968] ppid=9966 vsize=90452 CPUtime=0.03 cores=0-7
/proc/9968/stat : 9968 (SArTagnan) R 9966 9968 9942 0 -1 4202496 508 0 0 0 3 0 0 0 25 0 8 0 784550021 92622848 444 33554432000 4194304 4625164 140733345230448 18446744073709551615 47574777296812 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/9968/statm: 22613 444 301 106 0 18116 0
[pid=9968/tid=9969] ppid=9966 vsize=90452 CPUtime=0.01 cores=0-7
/proc/9968/task/9969/stat : 9969 (SArTagnan) R 9966 9968 9942 0 -1 4202560 1 0 0 0 1 0 0 0 25 0 8 0 784550021 92622848 444 33554432000 4194304 4625164 140733345230448 18446744073709551615 47574777296808 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=9968/tid=9970] ppid=9966 vsize=90452 CPUtime=0.01 cores=0-7
/proc/9968/task/9970/stat : 9970 (SArTagnan) R 9966 9968 9942 0 -1 4202560 0 0 0 0 1 0 0 0 25 0 8 0 784550021 92622848 444 33554432000 4194304 4625164 140733345230448 18446744073709551615 47574777296818 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=9968/tid=9971] ppid=9966 vsize=90452 CPUtime=0 cores=0-7
/proc/9968/task/9971/stat : 9971 (SArTagnan) R 9966 9968 9942 0 -1 4202560 0 0 0 0 0 0 0 0 25 0 8 0 784550021 92622848 444 33554432000 4194304 4625164 140733345230448 18446744073709551615 254261672945 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=9968/tid=9972] ppid=9966 vsize=90452 CPUtime=0 cores=0-7
/proc/9968/task/9972/stat : 9972 (SArTagnan) R 9966 9968 9942 0 -1 4202560 0 0 0 0 0 0 0 0 25 0 8 0 784550021 92622848 444 33554432000 4194304 4625164 140733345230448 18446744073709551615 47574777296818 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=9968/tid=9973] ppid=9966 vsize=90452 CPUtime=0 cores=0-7
/proc/9968/task/9973/stat : 9973 (SArTagnan) R 9966 9968 9942 0 -1 4202560 0 0 0 0 0 0 0 0 25 0 8 0 784550021 92622848 444 33554432000 4194304 4625164 140733345230448 18446744073709551615 47574777296818 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=9968/tid=9974] ppid=9966 vsize=90452 CPUtime=0.01 cores=0-7
/proc/9968/task/9974/stat : 9974 (SArTagnan) R 9966 9968 9942 0 -1 4202560 0 0 0 0 1 0 0 0 25 0 8 0 784550021 92622848 444 33554432000 4194304 4625164 140733345230448 18446744073709551615 47574777296818 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=9968/tid=9975] ppid=9966 vsize=90452 CPUtime=0.01 cores=0-7
/proc/9968/task/9975/stat : 9975 (SArTagnan) R 9966 9968 9942 0 -1 4202560 0 0 0 0 1 0 0 0 25 0 8 0 784550021 92622848 444 33554432000 4194304 4625164 140733345230448 18446744073709551615 47574777296812 0 0 4096 2 0 0 0 -1 4 0 0 0

[startup+0.0301869 s]
/proc/loadavg: 7.38 7.91 7.96 2/470 9968
/proc/meminfo: memFree=22314744/32951124 swapFree=67057928/67111528
[pid=9968] ppid=9966 vsize=90452 CPUtime=0.16 cores=0-7
/proc/9968/stat : 9968 (SArTagnan) R 9966 9968 9942 0 -1 4202496 508 0 0 0 16 0 0 0 25 0 8 0 784550021 92622848 444 33554432000 4194304 4625164 140733345230448 18446744073709551615 47574777296808 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/9968/statm: 22613 444 301 106 0 18116 0
[pid=9968/tid=9969] ppid=9966 vsize=90452 CPUtime=0.02 cores=0-7
/proc/9968/task/9969/stat : 9969 (SArTagnan) R 9966 9968 9942 0 -1 4202560 1 0 0 0 2 0 0 0 25 0 8 0 784550021 92622848 444 33554432000 4194304 4625164 140733345230448 18446744073709551615 47574777296808 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=9968/tid=9970] ppid=9966 vsize=90452 CPUtime=0.02 cores=0-7
/proc/9968/task/9970/stat : 9970 (SArTagnan) R 9966 9968 9942 0 -1 4202560 0 0 0 0 2 0 0 0 25 0 8 0 784550021 92622848 444 33554432000 4194304 4625164 140733345230448 18446744073709551615 47574777296818 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=9968/tid=9971] ppid=9966 vsize=90452 CPUtime=0 cores=0-7
/proc/9968/task/9971/stat : 9971 (SArTagnan) R 9966 9968 9942 0 -1 4202560 0 0 0 0 0 0 0 0 25 0 8 0 784550021 92622848 444 33554432000 4194304 4625164 140733345230448 18446744073709551615 254261672945 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=9968/tid=9972] ppid=9966 vsize=90452 CPUtime=0.01 cores=0-7
/proc/9968/task/9972/stat : 9972 (SArTagnan) R 9966 9968 9942 0 -1 4202560 0 0 0 0 1 0 0 0 25 0 8 0 784550021 92622848 444 33554432000 4194304 4625164 140733345230448 18446744073709551615 47574777296822 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=9968/tid=9973] ppid=9966 vsize=90452 CPUtime=0.01 cores=0-7
/proc/9968/task/9973/stat : 9973 (SArTagnan) R 9966 9968 9942 0 -1 4202560 0 0 0 0 1 0 0 0 25 0 8 0 784550021 92622848 444 33554432000 4194304 4625164 140733345230448 18446744073709551615 47574777296818 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=9968/tid=9974] ppid=9966 vsize=90452 CPUtime=0.02 cores=0-7
/proc/9968/task/9974/stat : 9974 (SArTagnan) R 9966 9968 9942 0 -1 4202560 0 0 0 0 2 0 0 0 25 0 8 0 784550021 92622848 444 33554432000 4194304 4625164 140733345230448 18446744073709551615 47574777296818 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=9968/tid=9975] ppid=9966 vsize=90452 CPUtime=0.02 cores=0-7
/proc/9968/task/9975/stat : 9975 (SArTagnan) R 9966 9968 9942 0 -1 4202560 0 0 0 0 2 0 0 0 25 0 8 0 784550021 92622848 444 33554432000 4194304 4625164 140733345230448 18446744073709551615 47574777296808 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 0.16
Current children cumulated vsize (KiB) 90452

[startup+0.10966 s]
/proc/loadavg: 7.38 7.91 7.96 2/470 9968
/proc/meminfo: memFree=22314744/32951124 swapFree=67057928/67111528
[pid=9968] ppid=9966 vsize=420560 CPUtime=0.77 cores=0-7
/proc/9968/stat : 9968 (SArTagnan) R 9966 9968 9942 0 -1 4202496 2237 0 0 0 77 0 0 0 25 0 8 0 784550021 430653440 1742 33554432000 4194304 4625164 140733345230448 18446744073709551615 4512312 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/9968/statm: 105140 1742 352 106 0 100643 0
[pid=9968/tid=9969] ppid=9966 vsize=420560 CPUtime=0.1 cores=0-7
/proc/9968/task/9969/stat : 9969 (SArTagnan) R 9966 9968 9942 0 -1 4202560 177 0 0 0 10 0 0 0 18 0 8 0 784550021 430653440 1742 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351984 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=9968/tid=9970] ppid=9966 vsize=420560 CPUtime=0.1 cores=0-7
/proc/9968/task/9970/stat : 9970 (SArTagnan) R 9966 9968 9942 0 -1 4202560 202 0 0 0 10 0 0 0 18 0 8 0 784550021 430653440 1742 33554432000 4194304 4625164 140733345230448 18446744073709551615 254261277729 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=9968/tid=9971] ppid=9966 vsize=420560 CPUtime=0.05 cores=0-7
/proc/9968/task/9971/stat : 9971 (SArTagnan) R 9966 9968 9942 0 -1 4202560 250 0 0 0 5 0 0 0 25 0 8 0 784550021 430653440 1742 33554432000 4194304 4625164 140733345230448 18446744073709551615 4339447 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=9968/tid=9972] ppid=9966 vsize=420560 CPUtime=0.09 cores=0-7
/proc/9968/task/9972/stat : 9972 (SArTagnan) D 9966 9968 9942 0 -1 4202560 214 0 0 0 9 0 0 0 18 0 8 0 784550021 430653440 1742 33554432000 4194304 4625164 140733345230448 18446744073709551615 254261659831 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=9968/tid=9973] ppid=9966 vsize=420560 CPUtime=0.09 cores=0-7
/proc/9968/task/9973/stat : 9973 (SArTagnan) R 9966 9968 9942 0 -1 4202560 232 0 0 0 9 0 0 0 18 0 8 0 784550021 430653440 1742 33554432000 4194304 4625164 140733345230448 18446744073709551615 4348821 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=9968/tid=9974] ppid=9966 vsize=420560 CPUtime=0.1 cores=0-7
/proc/9968/task/9974/stat : 9974 (SArTagnan) R 9966 9968 9942 0 -1 4202560 232 0 0 0 10 0 0 0 25 0 8 0 784550021 430653440 1742 33554432000 4194304 4625164 140733345230448 18446744073709551615 4464251 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=9968/tid=9975] ppid=9966 vsize=420560 CPUtime=0.1 cores=0-7
/proc/9968/task/9975/stat : 9975 (SArTagnan) R 9966 9968 9942 0 -1 4202560 139 0 0 0 10 0 0 0 25 0 8 0 784550021 430653440 1742 33554432000 4194304 4625164 140733345230448 18446744073709551615 4465408 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 0.77
Current children cumulated vsize (KiB) 420560

[startup+0.30111 s]
/proc/loadavg: 7.38 7.91 7.96 2/470 9968
/proc/meminfo: memFree=22314744/32951124 swapFree=67057928/67111528
[pid=9968] ppid=9966 vsize=426012 CPUtime=2.3 cores=0-7
/proc/9968/stat : 9968 (SArTagnan) R 9966 9968 9942 0 -1 4202496 6708 0 0 0 230 0 0 0 25 0 8 0 784550021 436236288 4331 33554432000 4194304 4625164 140733345230448 18446744073709551615 4519334 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/9968/statm: 106503 4332 352 106 0 102006 0
[pid=9968/tid=9969] ppid=9966 vsize=426012 CPUtime=0.29 cores=0-7
/proc/9968/task/9969/stat : 9969 (SArTagnan) R 9966 9968 9942 0 -1 4202560 680 0 0 0 29 0 0 0 18 0 8 0 784550021 436236288 4332 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351839 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=9968/tid=9970] ppid=9966 vsize=426012 CPUtime=0.29 cores=0-7
/proc/9968/task/9970/stat : 9970 (SArTagnan) R 9966 9968 9942 0 -1 4202560 614 0 0 0 29 0 0 0 18 0 8 0 784550021 436236288 4333 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351824 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=9968/tid=9971] ppid=9966 vsize=426012 CPUtime=0.24 cores=0-7
/proc/9968/task/9971/stat : 9971 (SArTagnan) R 9966 9968 9942 0 -1 4202560 1561 0 0 0 24 0 0 0 25 0 8 0 784550021 436236288 4333 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=9968/tid=9972] ppid=9966 vsize=426012 CPUtime=0.28 cores=0-7
/proc/9968/task/9972/stat : 9972 (SArTagnan) R 9966 9968 9942 0 -1 4202560 623 0 0 0 28 0 0 0 18 0 8 0 784550021 436236288 4334 33554432000 4194304 4625164 140733345230448 18446744073709551615 4340391 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=9968/tid=9973] ppid=9966 vsize=426012 CPUtime=0.28 cores=0-7
/proc/9968/task/9973/stat : 9973 (SArTagnan) R 9966 9968 9942 0 -1 4202560 687 0 0 0 28 0 0 0 18 0 8 0 784550021 436236288 4334 33554432000 4194304 4625164 140733345230448 18446744073709551615 4349019 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=9968/tid=9974] ppid=9966 vsize=426012 CPUtime=0.29 cores=0-7
/proc/9968/task/9974/stat : 9974 (SArTagnan) R 9966 9968 9942 0 -1 4202560 852 0 0 0 29 0 0 0 25 0 8 0 784550021 436236288 4335 33554432000 4194304 4625164 140733345230448 18446744073709551615 4352021 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=9968/tid=9975] ppid=9966 vsize=426012 CPUtime=0.29 cores=0-7
/proc/9968/task/9975/stat : 9975 (SArTagnan) R 9966 9968 9942 0 -1 4202560 614 0 0 0 29 0 0 0 25 0 8 0 784550021 436236288 4335 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351764 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 2.3
Current children cumulated vsize (KiB) 426012

[startup+0.70104 s]
/proc/loadavg: 7.38 7.91 7.96 2/470 9968
/proc/meminfo: memFree=22314744/32951124 swapFree=67057928/67111528
[pid=9968] ppid=9966 vsize=432788 CPUtime=5.49 cores=0-7
/proc/9968/stat : 9968 (SArTagnan) R 9966 9968 9942 0 -1 4202496 14700 0 0 0 548 1 0 0 25 0 8 0 784550021 443174912 8678 33554432000 4194304 4625164 140733345230448 18446744073709551615 4396709 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/9968/statm: 108197 8678 352 106 0 103700 0
[pid=9968/tid=9969] ppid=9966 vsize=432788 CPUtime=0.69 cores=0-7
/proc/9968/task/9969/stat : 9969 (SArTagnan) R 9966 9968 9942 0 -1 4202560 1581 0 0 0 69 0 0 0 18 0 8 0 784550021 443174912 8678 33554432000 4194304 4625164 140733345230448 18446744073709551615 4339639 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=9968/tid=9970] ppid=9966 vsize=432788 CPUtime=0.69 cores=0-7
/proc/9968/task/9970/stat : 9970 (SArTagnan) R 9966 9968 9942 0 -1 4202560 1421 0 0 0 69 0 0 0 18 0 8 0 784550021 443174912 8678 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=9968/tid=9971] ppid=9966 vsize=432788 CPUtime=0.64 cores=0-7
/proc/9968/task/9971/stat : 9971 (SArTagnan) R 9966 9968 9942 0 -1 4202560 4341 0 0 0 64 0 0 0 25 0 8 0 784550021 443174912 8678 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=9968/tid=9972] ppid=9966 vsize=432788 CPUtime=0.68 cores=0-7
/proc/9968/task/9972/stat : 9972 (SArTagnan) R 9966 9968 9942 0 -1 4202560 1452 0 0 0 68 0 0 0 18 0 8 0 784550021 443174912 8678 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351804 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=9968/tid=9973] ppid=9966 vsize=432788 CPUtime=0.68 cores=0-7
/proc/9968/task/9973/stat : 9973 (SArTagnan) R 9966 9968 9942 0 -1 4202560 1477 0 0 0 68 0 0 0 18 0 8 0 784550021 443174912 8678 33554432000 4194304 4625164 140733345230448 18446744073709551615 4349050 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=9968/tid=9974] ppid=9966 vsize=432788 CPUtime=0.69 cores=0-7
/proc/9968/task/9974/stat : 9974 (SArTagnan) R 9966 9968 9942 0 -1 4202560 1475 0 0 0 69 0 0 0 25 0 8 0 784550021 443174912 8678 33554432000 4194304 4625164 140733345230448 18446744073709551615 4352285 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=9968/tid=9975] ppid=9966 vsize=432788 CPUtime=0.69 cores=0-7
/proc/9968/task/9975/stat : 9975 (SArTagnan) R 9966 9968 9942 0 -1 4202560 1455 0 0 0 69 0 0 0 25 0 8 0 784550021 443174912 8678 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351830 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 5.49
Current children cumulated vsize (KiB) 432788

[startup+1.5009 s]
/proc/loadavg: 7.38 7.91 7.96 9/478 9976
/proc/meminfo: memFree=22271536/32951124 swapFree=67057928/67111528
[pid=9968] ppid=9966 vsize=442016 CPUtime=11.88 cores=0-7
/proc/9968/stat : 9968 (SArTagnan) R 9966 9968 9942 0 -1 4202496 27250 0 0 0 1185 3 0 0 25 0 8 0 784550021 452624384 14787 33554432000 4194304 4625164 140733345230448 18446744073709551615 254261721631 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/9968/statm: 110504 14787 354 106 0 106007 0
[pid=9968/tid=9969] ppid=9966 vsize=442016 CPUtime=1.49 cores=0-7
/proc/9968/task/9969/stat : 9969 (SArTagnan) R 9966 9968 9942 0 -1 4202560 2525 0 0 0 149 0 0 0 20 0 8 0 784550021 452624384 14787 33554432000 4194304 4625164 140733345230448 18446744073709551615 4352072 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=9968/tid=9970] ppid=9966 vsize=442016 CPUtime=1.49 cores=0-7
/proc/9968/task/9970/stat : 9970 (SArTagnan) R 9966 9968 9942 0 -1 4202560 2360 0 0 0 149 0 0 0 20 0 8 0 784550021 452624384 14787 33554432000 4194304 4625164 140733345230448 18446744073709551615 4352032 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=9968/tid=9971] ppid=9966 vsize=442016 CPUtime=1.43 cores=0-7
/proc/9968/task/9971/stat : 9971 (SArTagnan) R 9966 9968 9942 0 -1 4202560 10339 0 0 0 142 1 0 0 25 0 8 0 784550021 452624384 14787 33554432000 4194304 4625164 140733345230448 18446744073709551615 4335177 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=9968/tid=9972] ppid=9966 vsize=442016 CPUtime=1.47 cores=0-7
/proc/9968/task/9972/stat : 9972 (SArTagnan) R 9966 9968 9942 0 -1 4202560 2304 0 0 0 147 0 0 0 19 0 8 0 784550021 452624384 14787 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351824 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=9968/tid=9973] ppid=9966 vsize=442016 CPUtime=1.48 cores=0-7
/proc/9968/task/9973/stat : 9973 (SArTagnan) R 9966 9968 9942 0 -1 4202560 2300 0 0 0 148 0 0 0 20 0 8 0 784550021 452624384 14787 33554432000 4194304 4625164 140733345230448 18446744073709551615 254261721631 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=9968/tid=9974] ppid=9966 vsize=442016 CPUtime=1.49 cores=0-7
/proc/9968/task/9974/stat : 9974 (SArTagnan) R 9966 9968 9942 0 -1 4202560 2926 0 0 0 149 0 0 0 25 0 8 0 784550021 452624384 14787 33554432000 4194304 4625164 140733345230448 18446744073709551615 4352005 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=9968/tid=9975] ppid=9966 vsize=442016 CPUtime=1.49 cores=0-7
/proc/9968/task/9975/stat : 9975 (SArTagnan) R 9966 9968 9942 0 -1 4202560 2395 0 0 0 149 0 0 0 25 0 8 0 784550021 452624384 14787 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 11.88
Current children cumulated vsize (KiB) 442016

[startup+3.10062 s]
/proc/loadavg: 7.38 7.91 7.96 9/478 9976
/proc/meminfo: memFree=22238552/32951124 swapFree=67057928/67111528
[pid=9968] ppid=9966 vsize=457100 CPUtime=24.66 cores=0-7
/proc/9968/stat : 9968 (SArTagnan) R 9966 9968 9942 0 -1 4202496 48587 0 0 0 2459 7 0 0 25 0 8 0 784550021 468070400 24204 33554432000 4194304 4625164 140733345230448 18446744073709551615 4511904 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/9968/statm: 114275 24204 356 106 0 109778 0
[pid=9968/tid=9969] ppid=9966 vsize=457100 CPUtime=3.09 cores=0-7
/proc/9968/task/9969/stat : 9969 (SArTagnan) R 9966 9968 9942 0 -1 4202560 5285 0 0 0 309 0 0 0 25 0 8 0 784550021 468070400 24204 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=9968/tid=9970] ppid=9966 vsize=457100 CPUtime=3.08 cores=0-7
/proc/9968/task/9970/stat : 9970 (SArTagnan) R 9966 9968 9942 0 -1 4202560 3655 0 0 0 308 0 0 0 25 0 8 0 784550021 468070400 24204 33554432000 4194304 4625164 140733345230448 18446744073709551615 4339612 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=9968/tid=9971] ppid=9966 vsize=457100 CPUtime=3 cores=0-7
/proc/9968/task/9971/stat : 9971 (SArTagnan) R 9966 9968 9942 0 -1 4202560 19789 0 0 0 298 2 0 0 25 0 8 0 784550021 468070400 24204 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=9968/tid=9972] ppid=9966 vsize=457100 CPUtime=3.07 cores=0-7
/proc/9968/task/9972/stat : 9972 (SArTagnan) R 9966 9968 9942 0 -1 4202560 3675 0 0 0 307 0 0 0 20 0 8 0 784550021 468070400 24204 33554432000 4194304 4625164 140733345230448 18446744073709551615 4352739 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=9968/tid=9973] ppid=9966 vsize=457100 CPUtime=3.08 cores=0-7
/proc/9968/task/9973/stat : 9973 (SArTagnan) R 9966 9968 9942 0 -1 4202560 4710 0 0 0 308 0 0 0 25 0 8 0 784550021 468070400 24204 33554432000 4194304 4625164 140733345230448 18446744073709551615 4349197 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=9968/tid=9974] ppid=9966 vsize=457100 CPUtime=3.08 cores=0-7
/proc/9968/task/9974/stat : 9974 (SArTagnan) R 9966 9968 9942 0 -1 4202560 4765 0 0 0 308 0 0 0 25 0 8 0 784550021 468070400 24204 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351781 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=9968/tid=9975] ppid=9966 vsize=457100 CPUtime=3.08 cores=0-7
/proc/9968/task/9975/stat : 9975 (SArTagnan) R 9966 9968 9942 0 -1 4202560 3630 0 0 0 308 0 0 0 25 0 8 0 784550021 468070400 24204 33554432000 4194304 4625164 140733345230448 18446744073709551615 4288653 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 24.66
Current children cumulated vsize (KiB) 457100

[startup+6.30105 s]
/proc/loadavg: 7.43 7.91 7.96 9/478 9976
/proc/meminfo: memFree=22172844/32951124 swapFree=67057928/67111528
[pid=9968] ppid=9966 vsize=491484 CPUtime=50.21 cores=0-7
/proc/9968/stat : 9968 (SArTagnan) R 9966 9968 9942 0 -1 4202496 89868 0 0 0 5008 13 0 0 25 0 8 0 784550021 503279616 38024 33554432000 4194304 4625164 140733345230448 18446744073709551615 4511904 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/9968/statm: 122871 38024 380 106 0 118374 0
[pid=9968/tid=9969] ppid=9966 vsize=491484 CPUtime=6.29 cores=0-7
/proc/9968/task/9969/stat : 9969 (SArTagnan) R 9966 9968 9942 0 -1 4202560 8118 0 0 0 628 1 0 0 25 0 8 0 784550021 503279616 38024 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351824 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=9968/tid=9970] ppid=9966 vsize=491484 CPUtime=6.29 cores=0-7
/proc/9968/task/9970/stat : 9970 (SArTagnan) R 9966 9968 9942 0 -1 4202560 5762 0 0 0 628 1 0 0 25 0 8 0 784550021 503279616 38024 33554432000 4194304 4625164 140733345230448 18446744073709551615 4352021 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=9968/tid=9971] ppid=9966 vsize=491484 CPUtime=6.15 cores=0-7
/proc/9968/task/9971/stat : 9971 (SArTagnan) R 9966 9968 9942 0 -1 4202560 41010 0 0 0 610 5 0 0 25 0 8 0 784550021 503279616 38024 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=9968/tid=9972] ppid=9966 vsize=491484 CPUtime=6.28 cores=0-7
/proc/9968/task/9972/stat : 9972 (SArTagnan) R 9966 9968 9942 0 -1 4202560 7230 0 0 0 627 1 0 0 25 0 8 0 784550021 503279616 38024 33554432000 4194304 4625164 140733345230448 18446744073709551615 254261274995 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=9968/tid=9973] ppid=9966 vsize=491484 CPUtime=6.28 cores=0-7
/proc/9968/task/9973/stat : 9973 (SArTagnan) R 9966 9968 9942 0 -1 4202560 7241 0 0 0 627 1 0 0 25 0 8 0 784550021 503279616 38024 33554432000 4194304 4625164 140733345230448 18446744073709551615 4436904 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=9968/tid=9974] ppid=9966 vsize=491484 CPUtime=6.29 cores=0-7
/proc/9968/task/9974/stat : 9974 (SArTagnan) R 9966 9968 9942 0 -1 4202560 9239 0 0 0 628 1 0 0 25 0 8 0 784550021 503279616 38024 33554432000 4194304 4625164 140733345230448 18446744073709551615 4352739 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=9968/tid=9975] ppid=9966 vsize=491484 CPUtime=6.28 cores=0-7
/proc/9968/task/9975/stat : 9975 (SArTagnan) R 9966 9968 9942 0 -1 4202560 7099 0 0 0 627 1 0 0 25 0 8 0 784550021 503279616 38024 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351827 0 0 4096 2 0 0 0 -1 4 0 0 0

################
# More data... #
################

[pid=9968/tid=9971] ppid=9966 vsize=5319200 CPUtime=4592.86 cores=0-7
/proc/9968/task/9971/stat : 9971 (SArTagnan) R 9966 9968 9942 0 -1 4202560 16912653 0 0 0 456420 2866 0 0 25 0 8 0 784550021 5446860800 1064085 33554432000 4194304 4625164 140733345230448 18446744073709551615 4335119 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=9968/tid=9972] ppid=9966 vsize=5319200 CPUtime=4656.32 cores=0-7
/proc/9968/task/9972/stat : 9972 (SArTagnan) R 9966 9968 9942 0 -1 4202560 204536 0 0 0 465582 50 0 0 25 0 8 0 784550021 5446860800 1064085 33554432000 4194304 4625164 140733345230448 18446744073709551615 4330841 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=9968/tid=9973] ppid=9966 vsize=5319200 CPUtime=4662.39 cores=0-7
/proc/9968/task/9973/stat : 9973 (SArTagnan) R 9966 9968 9942 0 -1 4202560 268587 0 0 0 466174 65 0 0 25 0 8 0 784550021 5446860800 1064085 33554432000 4194304 4625164 140733345230448 18446744073709551615 4329721 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=9968/tid=9974] ppid=9966 vsize=5319200 CPUtime=4662.25 cores=0-7
/proc/9968/task/9974/stat : 9974 (SArTagnan) R 9966 9968 9942 0 -1 4202560 353239 0 0 0 466144 81 0 0 25 0 8 0 784550021 5446860800 1064085 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=9968/tid=9975] ppid=9966 vsize=5319200 CPUtime=4661.74 cores=0-7
/proc/9968/task/9975/stat : 9975 (SArTagnan) R 9966 9968 9942 0 -1 4202560 210335 0 0 0 466125 49 0 0 25 0 8 0 784550021 5446860800 1064085 33554432000 4194304 4625164 140733345230448 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 37221.4
Current children cumulated vsize (KiB) 5319200

[startup+4722.32 s]
/proc/loadavg: 8.00 8.02 8.00 9/478 10123
/proc/meminfo: memFree=18046276/32951124 swapFree=67057928/67111528
[pid=9968] ppid=9966 vsize=5319200 CPUtime=37700.6 cores=0-7
/proc/9968/stat : 9968 (SArTagnan) R 9966 9968 9942 0 -1 4202496 23748937 0 0 0 3765798 4258 0 0 25 0 8 0 784550021 5446860800 1064902 33554432000 4194304 4625164 140733345230448 18446744073709551615 4519319 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/9968/statm: 1329800 1064902 506 106 0 1325303 0
[pid=9968/tid=9969] ppid=9966 vsize=5319200 CPUtime=4721.06 cores=0-7
/proc/9968/task/9969/stat : 9969 (SArTagnan) R 9966 9968 9942 0 -1 4202560 211252 0 0 0 472061 45 0 0 25 0 8 0 784550021 5446860800 1064902 33554432000 4194304 4625164 140733345230448 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=9968/tid=9970] ppid=9966 vsize=5319200 CPUtime=4722.41 cores=0-7
/proc/9968/task/9970/stat : 9970 (SArTagnan) R 9966 9968 9942 0 -1 4202560 289940 0 0 0 472162 79 0 0 25 0 8 0 784550021 5446860800 1064902 33554432000 4194304 4625164 140733345230448 18446744073709551615 4330841 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=9968/tid=9971] ppid=9966 vsize=5319200 CPUtime=4651.88 cores=0-7
/proc/9968/task/9971/stat : 9971 (SArTagnan) R 9966 9968 9942 0 -1 4202560 17226892 0 0 0 462267 2921 0 0 25 0 8 0 784550021 5446860800 1064902 33554432000 4194304 4625164 140733345230448 18446744073709551615 4334485 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=9968/tid=9972] ppid=9966 vsize=5319200 CPUtime=4716.34 cores=0-7
/proc/9968/task/9972/stat : 9972 (SArTagnan) R 9966 9968 9942 0 -1 4202560 204552 0 0 0 471584 50 0 0 25 0 8 0 784550021 5446860800 1064902 33554432000 4194304 4625164 140733345230448 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=9968/tid=9973] ppid=9966 vsize=5319200 CPUtime=4722.39 cores=0-7
/proc/9968/task/9973/stat : 9973 (SArTagnan) R 9966 9968 9942 0 -1 4202560 268604 0 0 0 472174 65 0 0 25 0 8 0 784550021 5446860800 1064902 33554432000 4194304 4625164 140733345230448 18446744073709551615 4460412 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=9968/tid=9974] ppid=9966 vsize=5319200 CPUtime=4722.26 cores=0-7
/proc/9968/task/9974/stat : 9974 (SArTagnan) R 9966 9968 9942 0 -1 4202560 354032 0 0 0 472144 82 0 0 25 0 8 0 784550021 5446860800 1064902 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=9968/tid=9975] ppid=9966 vsize=5319200 CPUtime=4721.76 cores=0-7
/proc/9968/task/9975/stat : 9975 (SArTagnan) R 9966 9968 9942 0 -1 4202560 210351 0 0 0 472127 49 0 0 25 0 8 0 784550021 5446860800 1064902 33554432000 4194304 4625164 140733345230448 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 37700.6
Current children cumulated vsize (KiB) 5319200

[startup+4782.3 s]
/proc/loadavg: 8.11 8.04 8.01 9/478 10124
/proc/meminfo: memFree=18048924/32951124 swapFree=67057928/67111528
[pid=9968] ppid=9966 vsize=5319200 CPUtime=38179.4 cores=0-7
/proc/9968/stat : 9968 (SArTagnan) R 9966 9968 9942 0 -1 4202496 24127199 0 0 0 3813617 4326 0 0 25 0 8 0 784550021 5446860800 1064063 33554432000 4194304 4625164 140733345230448 18446744073709551615 4519334 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/9968/statm: 1329800 1064063 358 106 0 1325303 0
[pid=9968/tid=9969] ppid=9966 vsize=5319200 CPUtime=4781.02 cores=0-7
/proc/9968/task/9969/stat : 9969 (SArTagnan) R 9966 9968 9942 0 -1 4202560 211268 0 0 0 478057 45 0 0 25 0 8 0 784550021 5446860800 1064063 33554432000 4194304 4625164 140733345230448 18446744073709551615 4330841 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=9968/tid=9970] ppid=9966 vsize=5319200 CPUtime=4782.4 cores=0-7
/proc/9968/task/9970/stat : 9970 (SArTagnan) R 9966 9968 9942 0 -1 4202560 289952 0 0 0 478161 79 0 0 25 0 8 0 784550021 5446860800 1064063 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=9968/tid=9971] ppid=9966 vsize=5319200 CPUtime=4710.89 cores=0-7
/proc/9968/task/9971/stat : 9971 (SArTagnan) R 9966 9968 9942 0 -1 4202560 17509415 0 0 0 468117 2972 0 0 25 0 8 0 784550021 5446860800 1064063 33554432000 4194304 4625164 140733345230448 18446744073709551615 4339612 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=9968/tid=9972] ppid=9966 vsize=5319200 CPUtime=4776.32 cores=0-7
/proc/9968/task/9972/stat : 9972 (SArTagnan) R 9966 9968 9942 0 -1 4202560 204567 0 0 0 477582 50 0 0 25 0 8 0 784550021 5446860800 1064063 33554432000 4194304 4625164 140733345230448 18446744073709551615 4488068 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=9968/tid=9973] ppid=9966 vsize=5319200 CPUtime=4782.37 cores=0-7
/proc/9968/task/9973/stat : 9973 (SArTagnan) R 9966 9968 9942 0 -1 4202560 268621 0 0 0 478172 65 0 0 25 0 8 0 784550021 5446860800 1064063 33554432000 4194304 4625164 140733345230448 18446744073709551615 4329721 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=9968/tid=9974] ppid=9966 vsize=5319200 CPUtime=4782.24 cores=0-7
/proc/9968/task/9974/stat : 9974 (SArTagnan) R 9966 9968 9942 0 -1 4202560 354284 0 0 0 478142 82 0 0 25 0 8 0 784550021 5446860800 1064063 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=9968/tid=9975] ppid=9966 vsize=5319200 CPUtime=4781.75 cores=0-7
/proc/9968/task/9975/stat : 9975 (SArTagnan) R 9966 9968 9942 0 -1 4202560 210366 0 0 0 478126 49 0 0 25 0 8 0 784550021 5446860800 1064063 33554432000 4194304 4625164 140733345230448 18446744073709551615 4330841 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 38179.4
Current children cumulated vsize (KiB) 5319200

[startup+4842.3 s]
/proc/loadavg: 8.11 8.05 8.00 9/478 10126
/proc/meminfo: memFree=18038120/32951124 swapFree=67057928/67111528
[pid=9968] ppid=9966 vsize=5319200 CPUtime=38658.4 cores=0-7
/proc/9968/stat : 9968 (SArTagnan) R 9966 9968 9942 0 -1 4202496 24461482 0 0 0 3861461 4382 0 0 25 0 8 0 784550021 5446860800 1066498 33554432000 4194304 4625164 140733345230448 18446744073709551615 4512009 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/9968/statm: 1329800 1066498 385 106 0 1325303 0
[pid=9968/tid=9969] ppid=9966 vsize=5319200 CPUtime=4841.01 cores=0-7
/proc/9968/task/9969/stat : 9969 (SArTagnan) R 9966 9968 9942 0 -1 4202560 211292 0 0 0 484056 45 0 0 25 0 8 0 784550021 5446860800 1066498 33554432000 4194304 4625164 140733345230448 18446744073709551615 4330841 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=9968/tid=9970] ppid=9966 vsize=5319200 CPUtime=4842.4 cores=0-7
/proc/9968/task/9970/stat : 9970 (SArTagnan) R 9966 9968 9942 0 -1 4202560 289965 0 0 0 484161 79 0 0 25 0 8 0 784550021 5446860800 1066498 33554432000 4194304 4625164 140733345230448 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=9968/tid=9971] ppid=9966 vsize=5319200 CPUtime=4769.9 cores=0-7
/proc/9968/task/9971/stat : 9971 (SArTagnan) R 9966 9968 9942 0 -1 4202560 17742149 0 0 0 473980 3010 0 0 25 0 8 0 784550021 5446860800 1066498 33554432000 4194304 4625164 140733345230448 18446744073709551615 4547280 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=9968/tid=9972] ppid=9966 vsize=5319200 CPUtime=4836.32 cores=0-7
/proc/9968/task/9972/stat : 9972 (SArTagnan) R 9966 9968 9942 0 -1 4202560 204582 0 0 0 483582 50 0 0 25 0 8 0 784550021 5446860800 1066498 33554432000 4194304 4625164 140733345230448 18446744073709551615 4488006 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=9968/tid=9973] ppid=9966 vsize=5319200 CPUtime=4842.37 cores=0-7
/proc/9968/task/9973/stat : 9973 (SArTagnan) R 9966 9968 9942 0 -1 4202560 268637 0 0 0 484172 65 0 0 25 0 8 0 784550021 5446860800 1066498 33554432000 4194304 4625164 140733345230448 18446744073709551615 4329721 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=9968/tid=9974] ppid=9966 vsize=5319200 CPUtime=4842.24 cores=0-7
/proc/9968/task/9974/stat : 9974 (SArTagnan) R 9966 9968 9942 0 -1 4202560 355304 0 0 0 484141 83 0 0 25 0 8 0 784550021 5446860800 1066498 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=9968/tid=9975] ppid=9966 vsize=5319200 CPUtime=4841.75 cores=0-7
/proc/9968/task/9975/stat : 9975 (SArTagnan) R 9966 9968 9942 0 -1 4202560 210380 0 0 0 484126 49 0 0 25 0 8 0 784550021 5446860800 1066498 33554432000 4194304 4625164 140733345230448 18446744073709551615 4488046 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 38658.4
Current children cumulated vsize (KiB) 5319200

[startup+4902.3 s]
/proc/loadavg: 8.11 8.05 8.00 9/478 10127
/proc/meminfo: memFree=18013788/32951124 swapFree=67057928/67111528
[pid=9968] ppid=9966 vsize=5454860 CPUtime=39137.4 cores=0-7
/proc/9968/stat : 9968 (SArTagnan) R 9966 9968 9942 0 -1 4202496 24856813 0 0 0 3909290 4454 0 0 25 0 8 0 784550021 5585776640 1072937 33554432000 4194304 4625164 140733345230448 18446744073709551615 4511904 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/9968/statm: 1363715 1072937 447 106 0 1359218 0
[pid=9968/tid=9969] ppid=9966 vsize=5454860 CPUtime=4901 cores=0-7
/proc/9968/task/9969/stat : 9969 (SArTagnan) R 9966 9968 9942 0 -1 4202560 211312 0 0 0 490055 45 0 0 25 0 8 0 784550021 5585776640 1072937 33554432000 4194304 4625164 140733345230448 18446744073709551615 4330841 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=9968/tid=9970] ppid=9966 vsize=5454860 CPUtime=4902.4 cores=0-7
/proc/9968/task/9970/stat : 9970 (SArTagnan) R 9966 9968 9942 0 -1 4202560 289977 0 0 0 490161 79 0 0 25 0 8 0 784550021 5585776640 1072937 33554432000 4194304 4625164 140733345230448 18446744073709551615 4331096 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=9968/tid=9971] ppid=9966 vsize=5454860 CPUtime=4828.92 cores=0-7
/proc/9968/task/9971/stat : 9971 (SArTagnan) R 9966 9968 9942 0 -1 4202560 18002444 0 0 0 479834 3058 0 0 25 0 8 0 784550021 5585776640 1072937 33554432000 4194304 4625164 140733345230448 18446744073709551615 4552369 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=9968/tid=9972] ppid=9966 vsize=5454860 CPUtime=4896.32 cores=0-7
/proc/9968/task/9972/stat : 9972 (SArTagnan) R 9966 9968 9942 0 -1 4202560 204596 0 0 0 489582 50 0 0 25 0 8 0 784550021 5585776640 1072937 33554432000 4194304 4625164 140733345230448 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=9968/tid=9973] ppid=9966 vsize=5454860 CPUtime=4902.37 cores=0-7
/proc/9968/task/9973/stat : 9973 (SArTagnan) R 9966 9968 9942 0 -1 4202560 268652 0 0 0 490172 65 0 0 25 0 8 0 784550021 5585776640 1072937 33554432000 4194304 4625164 140733345230448 18446744073709551615 4329721 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=9968/tid=9974] ppid=9966 vsize=5454860 CPUtime=4902.24 cores=0-7
/proc/9968/task/9974/stat : 9974 (SArTagnan) R 9966 9968 9942 0 -1 4202560 355364 0 0 0 490141 83 0 0 25 0 8 0 784550021 5585776640 1072937 33554432000 4194304 4625164 140733345230448 18446744073709551615 4352728 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=9968/tid=9975] ppid=9966 vsize=5454860 CPUtime=4901.75 cores=0-7
/proc/9968/task/9975/stat : 9975 (SArTagnan) R 9966 9968 9942 0 -1 4202560 210395 0 0 0 490126 49 0 0 25 0 8 0 784550021 5585776640 1072937 33554432000 4194304 4625164 140733345230448 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 39137.4
Current children cumulated vsize (KiB) 5454860

[startup+4962.3 s]
/proc/loadavg: 8.16 8.07 8.01 9/478 10129
/proc/meminfo: memFree=18003728/32951124 swapFree=67057928/67111528
[pid=9968] ppid=9966 vsize=5454860 CPUtime=39616.4 cores=0-7
/proc/9968/stat : 9968 (SArTagnan) R 9966 9968 9942 0 -1 4202496 25197545 0 0 0 3957127 4518 0 0 25 0 8 0 784550021 5585776640 1075668 33554432000 4194304 4625164 140733345230448 18446744073709551615 4512180 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/9968/statm: 1363715 1075668 495 106 0 1359218 0
[pid=9968/tid=9969] ppid=9966 vsize=5454860 CPUtime=4960.98 cores=0-7
/proc/9968/task/9969/stat : 9969 (SArTagnan) R 9966 9968 9942 0 -1 4202560 211332 0 0 0 496053 45 0 0 25 0 8 0 784550021 5585776640 1075668 33554432000 4194304 4625164 140733345230448 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=9968/tid=9970] ppid=9966 vsize=5454860 CPUtime=4962.4 cores=0-7
/proc/9968/task/9970/stat : 9970 (SArTagnan) R 9966 9968 9942 0 -1 4202560 289991 0 0 0 496161 79 0 0 25 0 8 0 784550021 5585776640 1075668 33554432000 4194304 4625164 140733345230448 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=9968/tid=9971] ppid=9966 vsize=5454860 CPUtime=4887.93 cores=0-7
/proc/9968/task/9971/stat : 9971 (SArTagnan) R 9966 9968 9942 0 -1 4202560 18252600 0 0 0 485689 3104 0 0 25 0 8 0 784550021 5585776640 1075668 33554432000 4194304 4625164 140733345230448 18446744073709551615 4334728 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=9968/tid=9972] ppid=9966 vsize=5454860 CPUtime=4956.32 cores=0-7
/proc/9968/task/9972/stat : 9972 (SArTagnan) R 9966 9968 9942 0 -1 4202560 204611 0 0 0 495582 50 0 0 25 0 8 0 784550021 5585776640 1075668 33554432000 4194304 4625164 140733345230448 18446744073709551615 4488046 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=9968/tid=9973] ppid=9966 vsize=5454860 CPUtime=4962.37 cores=0-7
/proc/9968/task/9973/stat : 9973 (SArTagnan) R 9966 9968 9942 0 -1 4202560 268678 0 0 0 496172 65 0 0 25 0 8 0 784550021 5585776640 1075668 33554432000 4194304 4625164 140733345230448 18446744073709551615 4329721 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=9968/tid=9974] ppid=9966 vsize=5454860 CPUtime=4962.25 cores=0-7
/proc/9968/task/9974/stat : 9974 (SArTagnan) R 9966 9968 9942 0 -1 4202560 355484 0 0 0 496142 83 0 0 25 0 8 0 784550021 5585776640 1075668 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=9968/tid=9975] ppid=9966 vsize=5454860 CPUtime=4961.75 cores=0-7
/proc/9968/task/9975/stat : 9975 (SArTagnan) R 9966 9968 9942 0 -1 4202560 210412 0 0 0 496126 49 0 0 25 0 8 0 784550021 5585776640 1075668 33554432000 4194304 4625164 140733345230448 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 39616.4
Current children cumulated vsize (KiB) 5454860



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5010.4 s]
/proc/loadavg: 8.14 8.08 8.01 10/478 10130
/proc/meminfo: memFree=18002224/32951124 swapFree=67057928/67111528
[pid=9968] ppid=9966 vsize=5454860 CPUtime=40000.7 cores=0-7
/proc/9968/stat : 9968 (SArTagnan) R 9966 9968 9942 0 -1 4202496 25400852 0 0 0 3995513 4555 0 0 25 0 8 0 784550021 5585776640 1076067 33554432000 4194304 4625164 140733345230448 18446744073709551615 4511904 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/9968/statm: 1363715 1076067 408 106 0 1359218 0
[pid=9968/tid=9969] ppid=9966 vsize=5454860 CPUtime=5009.11 cores=0-7
/proc/9968/task/9969/stat : 9969 (SArTagnan) R 9966 9968 9942 0 -1 4202560 211345 0 0 0 500866 45 0 0 25 0 8 0 784550021 5585776640 1076067 33554432000 4194304 4625164 140733345230448 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=9968/tid=9970] ppid=9966 vsize=5454860 CPUtime=5010.54 cores=0-7
/proc/9968/task/9970/stat : 9970 (SArTagnan) R 9966 9968 9942 0 -1 4202560 290004 0 0 0 500975 79 0 0 25 0 8 0 784550021 5585776640 1076067 33554432000 4194304 4625164 140733345230448 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=9968/tid=9971] ppid=9966 vsize=5454860 CPUtime=4935.25 cores=0-7
/proc/9968/task/9971/stat : 9971 (SArTagnan) R 9966 9968 9942 0 -1 4202560 18454965 0 0 0 490384 3141 0 0 25 0 8 0 784550021 5585776640 1076067 33554432000 4194304 4625164 140733345230448 18446744073709551615 4547216 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=9968/tid=9972] ppid=9966 vsize=5454860 CPUtime=5004.46 cores=0-7
/proc/9968/task/9972/stat : 9972 (SArTagnan) R 9966 9968 9942 0 -1 4202560 204624 0 0 0 500396 50 0 0 25 0 8 0 784550021 5585776640 1076067 33554432000 4194304 4625164 140733345230448 18446744073709551615 4331101 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=9968/tid=9973] ppid=9966 vsize=5454860 CPUtime=5010.51 cores=0-7
/proc/9968/task/9973/stat : 9973 (SArTagnan) R 9966 9968 9942 0 -1 4202560 268694 0 0 0 500986 65 0 0 25 0 8 0 784550021 5585776640 1076067 33554432000 4194304 4625164 140733345230448 18446744073709551615 4329981 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=9968/tid=9974] ppid=9966 vsize=5454860 CPUtime=5010.38 cores=0-7
/proc/9968/task/9974/stat : 9974 (SArTagnan) R 9966 9968 9942 0 -1 4202560 355974 0 0 0 500955 83 0 0 25 0 8 0 784550021 5585776640 1076067 33554432000 4194304 4625164 140733345230448 18446744073709551615 4351844 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=9968/tid=9975] ppid=9966 vsize=5454860 CPUtime=5009.89 cores=0-7
/proc/9968/task/9975/stat : 9975 (SArTagnan) R 9966 9968 9942 0 -1 4202560 210425 0 0 0 500940 49 0 0 25 0 8 0 784550021 5585776640 1076067 33554432000 4194304 4625164 140733345230448 18446744073709551615 4330958 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 40000.7
Current children cumulated vsize (KiB) 5454860

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5010.6 s]
/proc/loadavg: 8.14 8.08 8.01 10/478 10130
/proc/meminfo: memFree=18002224/32951124 swapFree=67057928/67111528
[pid=9968] ppid=9966 vsize=0 CPUtime=40000.9 cores=0-7
/proc/9968/stat : 9968 (SArTagnan) Z 9966 9968 9942 0 -1 4203548 25400874 0 0 0 3995521 4571 0 0 25 0 2 0 784550021 0 0 33554432000 0 0 0 0 0 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/9968/statm: 0 0 0 0 0 0 0
[pid=9968/tid=9971] ppid=9966 vsize=0 CPUtime=4935.42 cores=0-7
/proc/9968/task/9971/stat : 9971 (SArTagnan) R 9966 9968 9942 0 -1 4203588 18454987 0 0 0 490385 3157 0 0 25 0 2 0 784550021 0 0 33554432000 0 0 0 0 0 0 0 4096 2 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 40000.9
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5010.65
CPU time (s): 40001
CPU user time (s): 39955.2
CPU system time (s): 45.766
CPU usage (%): 798.319
Max. virtual memory (cumulated for all children) (KiB): 5634260

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39955.2
system time used= 45.766
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 25400874
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 902
involuntary context switches= 173311

runsolver used 22.1776 second user time and 59.072 second system time

The end

Launcher Data 

Begin job on node140 at 2011-06-01 11:42:27
IDJOB=3438671
IDBENCH=83149
IDSOLVER=1936
FILE ID=node140/3438671-1306921347
RUNJOBID= node140-1306921347-9956
PBS_JOBID= 13488551
Free space on /tmp= 73780 MiB

SOLVER NAME= SArTagnan 2011-05-15 (fixed)
BENCH NAME= SAT11/crafted/spence/sgen/sgen3-n280-s93756482-sat.cnf
COMMAND LINE= SArTagnan BENCHNAME -threads=NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3438671-1306921347/watcher-3438671-1306921347 -o /tmp/evaluation-result-3438671-1306921347/solver-3438671-1306921347 -C 40000 -W 5100 -M 31000  SArTagnan HOME/instance-3438671-1306921347.cnf -threads=8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= badb3b4ac12b29ada7dd8ed96ed75c80
RANDOM SEED=760119780

node140.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.14
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      22315156 kB
Buffers:        238832 kB
Cached:        3146952 kB
SwapCached:       6932 kB
Active:        7099908 kB
Inactive:      3199952 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      22315156 kB
SwapTotal:    67111528 kB
SwapFree:     67057928 kB
Dirty:             440 kB
Writeback:           0 kB
AnonPages:     6913004 kB
Mapped:          37252 kB
Slab:           213428 kB
PageTables:      52436 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 141375712 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73780 MiB
End job on node140 at 2011-06-01 13:05:59