Trace number 3321503

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095? (TO) 40001.1 5008.5

General information on the benchmark

NameCRAFTED/rbsat/crafted/unforced/
rbsat-v2400c266431g6.cnf
MD5SUMd0bb6612a0c88017c970eb8fd1a4be9e
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2400
Number of clauses266431
Sum of the clauses size535112
Maximum clause length32
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2266356
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 575

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...356581/instance-3321503-1304356581.cnf
1.39/1.48	c Solving...
40000.08/5007.51	
40000.08/5007.51	*** INTERRUPTED! ***
40000.08/5007.51	*** Shutting down threads...
40001.01/5008.49	s UNKNOWN
40001.01/5008.49	
40001.01/5008.49	c INTERRUPTED   : 1
40001.01/5008.49	c Models        : 0+    
40001.01/5008.49	c Time          : 5007.516s (Solving: 5006.03s 1st Model: 0.00s Unsat: 0.00s)
40001.01/5008.49	c CPU Time      : 40000.130s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3321503-1304356581/watcher-3321503-1304356581 -o /tmp/evaluation-result-3321503-1304356581/solver-3321503-1304356581 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3321503-1304356581.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.16 2.76 4.39 2/278 30417
/proc/meminfo: memFree=8314220/32951124 swapFree=67111524/67111528
[pid=30417] ppid=30415 vsize=22544 CPUtime=0 cores=0-7
/proc/30417/stat : 30417 (claspmt) R 30415 30417 29577 0 -1 4202496 1027 0 0 0 0 0 0 0 22 0 1 0 279849420 23085056 849 33554432000 4194304 4964704 140734711428160 18446744073709551615 268660323013 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/30417/statm: 5636 850 386 189 0 510 0

[startup+0.090131 s]
/proc/loadavg: 2.16 2.76 4.39 2/278 30417
/proc/meminfo: memFree=8314220/32951124 swapFree=67111524/67111528
[pid=30417] ppid=30415 vsize=37700 CPUtime=0.08 cores=0-7
/proc/30417/stat : 30417 (claspmt) R 30415 30417 29577 0 -1 4202496 5793 0 0 0 8 0 0 0 22 0 1 0 279849420 38604800 4532 33554432000 4194304 4964704 140734711428160 18446744073709551615 268660323013 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/30417/statm: 9425 4534 397 189 0 4299 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 37700

[startup+0.101111 s]
/proc/loadavg: 2.16 2.76 4.39 2/278 30417
/proc/meminfo: memFree=8314220/32951124 swapFree=67111524/67111528
[pid=30417] ppid=30415 vsize=37832 CPUtime=0.09 cores=0-7
/proc/30417/stat : 30417 (claspmt) R 30415 30417 29577 0 -1 4202496 5837 0 0 0 9 0 0 0 22 0 1 0 279849420 38739968 4576 33554432000 4194304 4964704 140734711428160 18446744073709551615 4607644 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/30417/statm: 9458 4576 399 189 0 4332 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 37832

[startup+0.301077 s]
/proc/loadavg: 2.16 2.76 4.39 2/278 30417
/proc/meminfo: memFree=8314220/32951124 swapFree=67111524/67111528
[pid=30417] ppid=30415 vsize=37832 CPUtime=0.29 cores=0-7
/proc/30417/stat : 30417 (claspmt) R 30415 30417 29577 0 -1 4202496 5837 0 0 0 29 0 0 0 23 0 1 0 279849420 38739968 4576 33554432000 4194304 4964704 140734711428160 18446744073709551615 4607547 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/30417/statm: 9458 4576 399 189 0 4332 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 37832

[startup+0.701024 s]
/proc/loadavg: 2.16 2.76 4.39 2/278 30417
/proc/meminfo: memFree=8314220/32951124 swapFree=67111524/67111528
[pid=30417] ppid=30415 vsize=37964 CPUtime=0.69 cores=0-7
/proc/30417/stat : 30417 (claspmt) R 30415 30417 29577 0 -1 4202496 5849 0 0 0 69 0 0 0 25 0 1 0 279849420 38875136 4588 33554432000 4194304 4964704 140734711428160 18446744073709551615 4600360 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/30417/statm: 9491 4588 400 189 0 4365 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 37964

[startup+1.50083 s]
/proc/loadavg: 2.16 2.76 4.39 2/279 30418
/proc/meminfo: memFree=8295496/32951124 swapFree=67111524/67111528
[pid=30417] ppid=30415 vsize=464980 CPUtime=1.5 cores=0-7
/proc/30417/stat : 30417 (claspmt) S 30415 30417 29577 0 -1 4202496 7601 0 0 0 149 1 0 0 18 0 8 0 279849420 476139520 5799 33554432000 4194304 4964704 140734711428160 18446744073709551615 268672478105 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/30417/statm: 116245 5801 432 189 0 110594 0
Current children cumulated CPU time (s) 1.5
Current children cumulated vsize (KiB) 464980

[startup+3.10055 s]
/proc/loadavg: 2.16 2.76 4.39 9/286 30425
/proc/meminfo: memFree=8268804/32951124 swapFree=67111524/67111528
[pid=30417] ppid=30415 vsize=464980 CPUtime=12.72 cores=0-7
/proc/30417/stat : 30417 (claspmt) R 30415 30417 29577 0 -1 4202496 20462 0 0 0 1263 9 0 0 17 0 8 0 279849420 476139520 18660 33554432000 4194304 4964704 140734711428160 18446744073709551615 4320944 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/30417/statm: 116245 18660 464 189 0 110594 0
[pid=30417/tid=30419] ppid=30415 vsize=464980 CPUtime=1.38 cores=0-7
/proc/30417/task/30419/stat : 30419 (claspmt) R 30415 30417 29577 0 -1 4202560 1513 0 0 0 137 1 0 0 17 0 8 0 279849569 476139520 18660 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30417/tid=30420] ppid=30415 vsize=464980 CPUtime=1.36 cores=0-7
/proc/30417/task/30420/stat : 30420 (claspmt) R 30415 30417 29577 0 -1 4202560 1565 0 0 0 135 1 0 0 17 0 8 0 279849569 476139520 18660 33554432000 4194304 4964704 140734711428160 18446744073709551615 4658341 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30417/tid=30421] ppid=30415 vsize=464980 CPUtime=1.39 cores=0-7
/proc/30417/task/30421/stat : 30421 (claspmt) R 30415 30417 29577 0 -1 4202560 1571 0 0 0 138 1 0 0 17 0 8 0 279849569 476139520 18660 33554432000 4194304 4964704 140734711428160 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30417/tid=30422] ppid=30415 vsize=464980 CPUtime=1.38 cores=0-7
/proc/30417/task/30422/stat : 30422 (claspmt) R 30415 30417 29577 0 -1 4202560 1571 0 0 0 138 0 0 0 17 0 8 0 279849569 476139520 18660 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670184 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30417/tid=30423] ppid=30415 vsize=464980 CPUtime=1.59 cores=0-7
/proc/30417/task/30423/stat : 30423 (claspmt) R 30415 30417 29577 0 -1 4202560 1129 0 0 0 159 0 0 0 20 0 8 0 279849569 476139520 18660 33554432000 4194304 4964704 140734711428160 18446744073709551615 4627272 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30417/tid=30424] ppid=30415 vsize=464980 CPUtime=1.33 cores=0-7
/proc/30417/task/30424/stat : 30424 (claspmt) R 30415 30417 29577 0 -1 4202560 2007 0 0 0 132 1 0 0 17 0 8 0 279849569 476139520 18660 33554432000 4194304 4964704 140734711428160 18446744073709551615 268660767263 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30417/tid=30425] ppid=30415 vsize=464980 CPUtime=1.39 cores=0-7
/proc/30417/task/30425/stat : 30425 (claspmt) R 30415 30417 29577 0 -1 4202560 2882 0 0 0 138 1 0 0 17 0 8 0 279849569 476139520 18660 33554432000 4194304 4964704 140734711428160 18446744073709551615 268660767263 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 12.72
Current children cumulated vsize (KiB) 464980

[startup+6.30099 s]
/proc/loadavg: 2.63 2.85 4.41 9/286 30425
/proc/meminfo: memFree=8198012/32951124 swapFree=67111524/67111528
[pid=30417] ppid=30415 vsize=472640 CPUtime=38.29 cores=0-7
/proc/30417/stat : 30417 (claspmt) R 30415 30417 29577 0 -1 4202496 36731 0 0 0 3811 18 0 0 23 0 8 0 279849420 483983360 34929 33554432000 4194304 4964704 140734711428160 18446744073709551615 4627170 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/30417/statm: 118160 34929 466 189 0 112509 0
[pid=30417/tid=30419] ppid=30415 vsize=472640 CPUtime=4.54 cores=0-7
/proc/30417/task/30419/stat : 30419 (claspmt) R 30415 30417 29577 0 -1 4202560 3672 0 0 0 452 2 0 0 23 0 8 0 279849569 483983360 34929 33554432000 4194304 4964704 140734711428160 18446744073709551615 4342208 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30417/tid=30420] ppid=30415 vsize=472640 CPUtime=4.55 cores=0-7
/proc/30417/task/30420/stat : 30420 (claspmt) R 30415 30417 29577 0 -1 4202560 3207 0 0 0 453 2 0 0 23 0 8 0 279849569 483983360 34929 33554432000 4194304 4964704 140734711428160 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30417/tid=30421] ppid=30415 vsize=472640 CPUtime=4.59 cores=0-7
/proc/30417/task/30421/stat : 30421 (claspmt) R 30415 30417 29577 0 -1 4202560 3341 0 0 0 457 2 0 0 23 0 8 0 279849569 483983360 34929 33554432000 4194304 4964704 140734711428160 18446744073709551615 4668942 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30417/tid=30422] ppid=30415 vsize=472640 CPUtime=4.59 cores=0-7
/proc/30417/task/30422/stat : 30422 (claspmt) R 30415 30417 29577 0 -1 4202560 3860 0 0 0 457 2 0 0 23 0 8 0 279849569 483983360 34929 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670281 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30417/tid=30423] ppid=30415 vsize=472640 CPUtime=4.78 cores=0-7
/proc/30417/task/30423/stat : 30423 (claspmt) R 30415 30417 29577 0 -1 4202560 1952 0 0 0 478 0 0 0 25 0 8 0 279849569 483983360 34929 33554432000 4194304 4964704 140734711428160 18446744073709551615 4320924 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30417/tid=30424] ppid=30415 vsize=472640 CPUtime=4.53 cores=0-7
/proc/30417/task/30424/stat : 30424 (claspmt) R 30415 30417 29577 0 -1 4202560 3470 0 0 0 451 2 0 0 23 0 8 0 279849569 483983360 34929 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30417/tid=30425] ppid=30415 vsize=472640 CPUtime=4.59 cores=0-7
/proc/30417/task/30425/stat : 30425 (claspmt) R 30415 30417 29577 0 -1 4202560 7435 0 0 0 456 3 0 0 24 0 8 0 279849569 483983360 34929 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38.29
Current children cumulated vsize (KiB) 472640

[startup+12.7009 s]
/proc/loadavg: 3.06 2.93 4.43 9/286 30425
/proc/meminfo: memFree=8088936/32951124 swapFree=67111524/67111528
[pid=30417] ppid=30415 vsize=487332 CPUtime=89.42 cores=0-7
/proc/30417/stat : 30417 (claspmt) R 30415 30417 29577 0 -1 4202496 61799 0 0 0 8910 32 0 0 25 0 8 0 279849420 499027968 59997 33554432000 4194304 4964704 140734711428160 18446744073709551615 4627204 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/30417/statm: 121833 59997 466 189 0 116182 0
[pid=30417/tid=30419] ppid=30415 vsize=487332 CPUtime=10.88 cores=0-7
/proc/30417/task/30419/stat : 30419 (claspmt) R 30415 30417 29577 0 -1 4202560 6623 0 0 0 1085 3 0 0 25 0 8 0 279849569 499027968 59997 33554432000 4194304 4964704 140734711428160 18446744073709551615 4627248 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30417/tid=30420] ppid=30415 vsize=487332 CPUtime=10.95 cores=0-7
/proc/30417/task/30420/stat : 30420 (claspmt) R 30415 30417 29577 0 -1 4202560 6164 0 0 0 1091 4 0 0 25 0 8 0 279849569 499027968 59997 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30417/tid=30421] ppid=30415 vsize=487332 CPUtime=10.99 cores=0-7
/proc/30417/task/30421/stat : 30421 (claspmt) R 30415 30417 29577 0 -1 4202560 4910 0 0 0 1096 3 0 0 25 0 8 0 279849569 499027968 59997 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670339 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30417/tid=30422] ppid=30415 vsize=487332 CPUtime=10.98 cores=0-7
/proc/30417/task/30422/stat : 30422 (claspmt) R 30415 30417 29577 0 -1 4202560 6469 0 0 0 1095 3 0 0 25 0 8 0 279849569 499027968 59997 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30417/tid=30423] ppid=30415 vsize=487332 CPUtime=11.19 cores=0-7
/proc/30417/task/30423/stat : 30423 (claspmt) R 30415 30417 29577 0 -1 4202560 4829 0 0 0 1116 3 0 0 25 0 8 0 279849569 499027968 59997 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30417/tid=30424] ppid=30415 vsize=487332 CPUtime=10.93 cores=0-7
/proc/30417/task/30424/stat : 30424 (claspmt) R 30415 30417 29577 0 -1 4202560 5161 0 0 0 1091 2 0 0 25 0 8 0 279849569 499027968 59997 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30417/tid=30425] ppid=30415 vsize=487332 CPUtime=10.99 cores=0-7
/proc/30417/task/30425/stat : 30425 (claspmt) R 30415 30417 29577 0 -1 4202560 15243 0 0 0 1092 7 0 0 25 0 8 0 279849569 499027968 59998 33554432000 4194304 4964704 140734711428160 18446744073709551615 4320966 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 89.42
Current children cumulated vsize (KiB) 487332

[startup+25.5006 s]
/proc/loadavg: 4.15 3.18 4.48 9/286 30425
/proc/meminfo: memFree=8005304/32951124 swapFree=67111524/67111528
[pid=30417] ppid=30415 vsize=505280 CPUtime=191.68 cores=0-7
/proc/30417/stat : 30417 (claspmt) R 30415 30417 29577 0 -1 4202496 80770 0 0 0 19125 43 0 0 25 0 8 0 279849420 517406720 78968 33554432000 4194304 4964704 140734711428160 18446744073709551615 4668937 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/30417/statm: 126320 78968 466 189 0 120669 0
[pid=30417/tid=30419] ppid=30415 vsize=505280 CPUtime=23.56 cores=0-7
/proc/30417/task/30419/stat : 30419 (claspmt) R 30415 30417 29577 0 -1 4202560 9064 0 0 0 2351 5 0 0 25 0 8 0 279849569 517406720 78968 33554432000 4194304 4964704 140734711428160 18446744073709551615 4668931 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30417/tid=30420] ppid=30415 vsize=505280 CPUtime=23.74 cores=0-7
/proc/30417/task/30420/stat : 30420 (claspmt) R 30415 30417 29577 0 -1 4202560 7918 0 0 0 2369 5 0 0 25 0 8 0 279849569 517406720 78968 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30417/tid=30421] ppid=30415 vsize=505280 CPUtime=23.78 cores=0-7
/proc/30417/task/30421/stat : 30421 (claspmt) R 30415 30417 29577 0 -1 4202560 5991 0 0 0 2375 3 0 0 25 0 8 0 279849569 517406720 78968 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670454 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30417/tid=30422] ppid=30415 vsize=505280 CPUtime=23.78 cores=0-7
/proc/30417/task/30422/stat : 30422 (claspmt) R 30415 30417 29577 0 -1 4202560 7690 0 0 0 2375 3 0 0 25 0 8 0 279849569 517406720 78968 33554432000 4194304 4964704 140734711428160 18446744073709551615 4684027 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30417/tid=30423] ppid=30415 vsize=505280 CPUtime=23.99 cores=0-7
/proc/30417/task/30423/stat : 30423 (claspmt) R 30415 30417 29577 0 -1 4202560 6095 0 0 0 2395 4 0 0 19 0 8 0 279849569 517406720 78968 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30417/tid=30424] ppid=30415 vsize=505280 CPUtime=23.73 cores=0-7
/proc/30417/task/30424/stat : 30424 (claspmt) R 30415 30417 29577 0 -1 4202560 6426 0 0 0 2370 3 0 0 25 0 8 0 279849569 517406720 78968 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30417/tid=30425] ppid=30415 vsize=505280 CPUtime=23.79 cores=0-7
/proc/30417/task/30425/stat : 30425 (claspmt) R 30415 30417 29577 0 -1 4202560 22901 0 0 0 2367 12 0 0 25 0 8 0 279849569 517406720 78968 33554432000 4194304 4964704 140734711428160 18446744073709551615 4653154 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 191.68
Current children cumulated vsize (KiB) 505280

[startup+51.1011 s]
/proc/loadavg: 5.46 3.57 4.57 9/286 30426
/proc/meminfo: memFree=7942976/32951124 swapFree=67111524/67111528
[pid=30417] ppid=30415 vsize=505280 CPUtime=396.24 cores=0-7
/proc/30417/stat : 30417 (claspmt) R 30415 30417 29577 0 -1 4202496 96309 0 0 0 39568 56 0 0 25 0 8 0 279849420 517406720 94507 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670193 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/30417/statm: 126320 94507 466 189 0 120669 0
[pid=30417/tid=30419] ppid=30415 vsize=505280 CPUtime=48.92 cores=0-7
/proc/30417/task/30419/stat : 30419 (claspmt) R 30415 30417 29577 0 -1 4202560 11348 0 0 0 4885 7 0 0 25 0 8 0 279849569 517406720 94507 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30417/tid=30420] ppid=30415 vsize=505280 CPUtime=49.35 cores=0-7
/proc/30417/task/30420/stat : 30420 (claspmt) R 30415 30417 29577 0 -1 4202560 9570 0 0 0 4928 7 0 0 25 0 8 0 279849569 517406720 94507 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30417/tid=30421] ppid=30415 vsize=505280 CPUtime=49.38 cores=0-7
/proc/30417/task/30421/stat : 30421 (claspmt) R 30415 30417 29577 0 -1 4202560 7032 0 0 0 4934 4 0 0 25 0 8 0 279849569 517406720 94507 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670283 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30417/tid=30422] ppid=30415 vsize=505280 CPUtime=49.38 cores=0-7
/proc/30417/task/30422/stat : 30422 (claspmt) R 30415 30417 29577 0 -1 4202560 10175 0 0 0 4934 4 0 0 25 0 8 0 279849569 517406720 94507 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30417/tid=30423] ppid=30415 vsize=505280 CPUtime=49.59 cores=0-7
/proc/30417/task/30423/stat : 30423 (claspmt) R 30415 30417 29577 0 -1 4202560 6909 0 0 0 4954 5 0 0 25 0 8 0 279849569 517406720 94507 33554432000 4194304 4964704 140734711428160 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30417/tid=30424] ppid=30415 vsize=505280 CPUtime=49.33 cores=0-7
/proc/30417/task/30424/stat : 30424 (claspmt) R 30415 30417 29577 0 -1 4202560 7442 0 0 0 4929 4 0 0 25 0 8 0 279849569 517406720 94507 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30417/tid=30425] ppid=30415 vsize=505280 CPUtime=49.39 cores=0-7
/proc/30417/task/30425/stat : 30425 (claspmt) R 30415 30417 29577 0 -1 4202560 27535 0 0 0 4924 15 0 0 25 0 8 0 279849569 517406720 94507 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670263 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 396.24
Current children cumulated vsize (KiB) 505280

[startup+102.311 s]
/proc/loadavg: 6.90 4.25 4.75 9/286 30427
/proc/meminfo: memFree=7845628/32951124 swapFree=67111524/67111528
[pid=30417] ppid=30415 vsize=780544 CPUtime=805.41 cores=0-7
/proc/30417/stat : 30417 (claspmt) R 30415 30417 29577 0 -1 4202496 120807 0 0 0 80462 79 0 0 25 0 8 0 279849420 799277056 118929 33554432000 4194304 4964704 140734711428160 18446744073709551615 4342193 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/30417/statm: 195136 118929 466 189 0 189485 0
[pid=30417/tid=30419] ppid=30415 vsize=780544 CPUtime=99.66 cores=0-7
/proc/30417/task/30419/stat : 30419 (claspmt) R 30415 30417 29577 0 -1 4202560 14176 0 0 0 9956 10 0 0 25 0 8 0 279849569 799277056 118929 33554432000 4194304 4964704 140734711428160 18446744073709551615 4653275 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30417/tid=30420] ppid=30415 vsize=780544 CPUtime=100.54 cores=0-7
/proc/30417/task/30420/stat : 30420 (claspmt) R 30415 30417 29577 0 -1 4202560 11321 0 0 0 10046 8 0 0 25 0 8 0 279849569 799277056 118929 33554432000 4194304 4964704 140734711428160 18446744073709551615 4627227 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30417/tid=30421] ppid=30415 vsize=780544 CPUtime=100.58 cores=0-7
/proc/30417/task/30421/stat : 30421 (claspmt) R 30415 30417 29577 0 -1 4202560 7670 0 0 0 10053 5 0 0 25 0 8 0 279849569 799277056 118929 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0

################
# More data... #
################

[pid=30417] ppid=30415 vsize=2669256 CPUtime=37721.1 cores=0-7
/proc/30417/stat : 30417 (claspmt) R 30415 30417 29577 0 -1 4202496 621249 0 0 0 3771365 750 0 0 25 0 8 0 279849420 2733318144 611749 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/30417/statm: 667314 611749 466 189 0 661663 0
[pid=30417/tid=30419] ppid=30415 vsize=2669256 CPUtime=4716.65 cores=0-7
/proc/30417/task/30419/stat : 30419 (claspmt) R 30415 30417 29577 0 -1 4202560 49713 0 0 0 471604 61 0 0 25 0 8 0 279849569 2733318144 611749 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30417/tid=30420] ppid=30415 vsize=2669256 CPUtime=4719.86 cores=0-7
/proc/30417/task/30420/stat : 30420 (claspmt) R 30415 30417 29577 0 -1 4202560 31684 0 0 0 471956 30 0 0 25 0 8 0 279849569 2733318144 611749 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30417/tid=30421] ppid=30415 vsize=2669256 CPUtime=4720.26 cores=0-7
/proc/30417/task/30421/stat : 30421 (claspmt) R 30415 30417 29577 0 -1 4202560 19829 0 0 0 471997 29 0 0 25 0 8 0 279849569 2733318144 611749 33554432000 4194304 4964704 140734711428160 18446744073709551615 4320966 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30417/tid=30422] ppid=30415 vsize=2669256 CPUtime=4681.18 cores=0-7
/proc/30417/task/30422/stat : 30422 (claspmt) R 30415 30417 29577 0 -1 4202560 33765 0 0 0 468081 37 0 0 25 0 8 0 279849569 2733318144 611749 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30417/tid=30423] ppid=30415 vsize=2669256 CPUtime=4720.13 cores=0-7
/proc/30417/task/30423/stat : 30423 (claspmt) R 30415 30417 29577 0 -1 4202560 24849 0 0 0 471990 23 0 0 25 0 8 0 279849569 2733318144 611749 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30417/tid=30424] ppid=30415 vsize=2669256 CPUtime=4720.47 cores=0-7
/proc/30417/task/30424/stat : 30424 (claspmt) R 30415 30417 29577 0 -1 4202560 29270 0 0 0 472005 42 0 0 25 0 8 0 279849569 2733318144 611749 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30417/tid=30425] ppid=30415 vsize=2669256 CPUtime=4720.52 cores=0-7
/proc/30417/task/30425/stat : 30425 (claspmt) R 30415 30417 29577 0 -1 4202560 358805 0 0 0 471615 437 0 0 25 0 8 0 279849569 2733318144 611749 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670263 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 37721.1
Current children cumulated vsize (KiB) 2669256

[startup+4782.3 s]
/proc/loadavg: 8.00 8.02 8.00 9/286 30564
/proc/meminfo: memFree=5867000/32951124 swapFree=67111524/67111528
[pid=30417] ppid=30415 vsize=2669256 CPUtime=38200.5 cores=0-7
/proc/30417/stat : 30417 (claspmt) R 30415 30417 29577 0 -1 4202496 621257 0 0 0 3819303 750 0 0 25 0 8 0 279849420 2733318144 611757 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670454 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/30417/statm: 667314 611757 466 189 0 661663 0
[pid=30417/tid=30419] ppid=30415 vsize=2669256 CPUtime=4776.64 cores=0-7
/proc/30417/task/30419/stat : 30419 (claspmt) R 30415 30417 29577 0 -1 4202560 49715 0 0 0 477603 61 0 0 25 0 8 0 279849569 2733318144 611757 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30417/tid=30420] ppid=30415 vsize=2669256 CPUtime=4779.85 cores=0-7
/proc/30417/task/30420/stat : 30420 (claspmt) R 30415 30417 29577 0 -1 4202560 31684 0 0 0 477955 30 0 0 25 0 8 0 279849569 2733318144 611757 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30417/tid=30421] ppid=30415 vsize=2669256 CPUtime=4780.25 cores=0-7
/proc/30417/task/30421/stat : 30421 (claspmt) R 30415 30417 29577 0 -1 4202560 19829 0 0 0 477996 29 0 0 25 0 8 0 279849569 2733318144 611757 33554432000 4194304 4964704 140734711428160 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30417/tid=30422] ppid=30415 vsize=2669256 CPUtime=4740.63 cores=0-7
/proc/30417/task/30422/stat : 30422 (claspmt) R 30415 30417 29577 0 -1 4202560 33765 0 0 0 474026 37 0 0 25 0 8 0 279849569 2733318144 611757 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30417/tid=30423] ppid=30415 vsize=2669256 CPUtime=4780.12 cores=0-7
/proc/30417/task/30423/stat : 30423 (claspmt) R 30415 30417 29577 0 -1 4202560 24849 0 0 0 477989 23 0 0 25 0 8 0 279849569 2733318144 611757 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30417/tid=30424] ppid=30415 vsize=2669256 CPUtime=4780.46 cores=0-7
/proc/30417/task/30424/stat : 30424 (claspmt) R 30415 30417 29577 0 -1 4202560 29270 0 0 0 478004 42 0 0 25 0 8 0 279849569 2733318144 611757 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30417/tid=30425] ppid=30415 vsize=2669256 CPUtime=4780.51 cores=0-7
/proc/30417/task/30425/stat : 30425 (claspmt) R 30415 30417 29577 0 -1 4202560 358811 0 0 0 477614 437 0 0 25 0 8 0 279849569 2733318144 611757 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670233 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38200.5
Current children cumulated vsize (KiB) 2669256

[startup+4842.3 s]
/proc/loadavg: 8.00 8.02 8.00 9/286 30566
/proc/meminfo: memFree=5813632/32951124 swapFree=67111524/67111528
[pid=30417] ppid=30415 vsize=2734792 CPUtime=38680 cores=0-7
/proc/30417/stat : 30417 (claspmt) R 30415 30417 29577 0 -1 4202496 634625 0 0 0 3867224 773 0 0 25 0 8 0 279849420 2800427008 625125 33554432000 4194304 4964704 140734711428160 18446744073709551615 4321029 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/30417/statm: 683698 625125 466 189 0 678047 0
[pid=30417/tid=30419] ppid=30415 vsize=2734792 CPUtime=4836.65 cores=0-7
/proc/30417/task/30419/stat : 30419 (claspmt) R 30415 30417 29577 0 -1 4202560 49715 0 0 0 483604 61 0 0 25 0 8 0 279849569 2800427008 625125 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30417/tid=30420] ppid=30415 vsize=2734792 CPUtime=4839.83 cores=0-7
/proc/30417/task/30420/stat : 30420 (claspmt) R 30415 30417 29577 0 -1 4202560 31715 0 0 0 483953 30 0 0 25 0 8 0 279849569 2800427008 625125 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30417/tid=30421] ppid=30415 vsize=2734792 CPUtime=4840.25 cores=0-7
/proc/30417/task/30421/stat : 30421 (claspmt) R 30415 30417 29577 0 -1 4202560 19862 0 0 0 483996 29 0 0 25 0 8 0 279849569 2800427008 625125 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30417/tid=30422] ppid=30415 vsize=2734792 CPUtime=4800.09 cores=0-7
/proc/30417/task/30422/stat : 30422 (claspmt) R 30415 30417 29577 0 -1 4202560 33798 0 0 0 479971 38 0 0 25 0 8 0 279849569 2800427008 625125 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30417/tid=30423] ppid=30415 vsize=2734792 CPUtime=4840.13 cores=0-7
/proc/30417/task/30423/stat : 30423 (claspmt) R 30415 30417 29577 0 -1 4202560 24849 0 0 0 483990 23 0 0 25 0 8 0 279849569 2800427008 625125 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30417/tid=30424] ppid=30415 vsize=2734792 CPUtime=4840.46 cores=0-7
/proc/30417/task/30424/stat : 30424 (claspmt) R 30415 30417 29577 0 -1 4202560 29270 0 0 0 484004 42 0 0 25 0 8 0 279849569 2800427008 625125 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30417/tid=30425] ppid=30415 vsize=2734792 CPUtime=4840.51 cores=0-7
/proc/30417/task/30425/stat : 30425 (claspmt) R 30415 30417 29577 0 -1 4202560 372050 0 0 0 483591 460 0 0 25 0 8 0 279849569 2800427008 625125 33554432000 4194304 4964704 140734711428160 18446744073709551615 4321069 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38680
Current children cumulated vsize (KiB) 2734792

[startup+4902.3 s]
/proc/loadavg: 8.00 8.01 8.00 9/286 30567
/proc/meminfo: memFree=5802688/32951124 swapFree=67111524/67111528
[pid=30417] ppid=30415 vsize=2734792 CPUtime=39159.4 cores=0-7
/proc/30417/stat : 30417 (claspmt) R 30415 30417 29577 0 -1 4202496 637457 0 0 0 3915166 777 0 0 25 0 8 0 279849420 2800427008 627957 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/30417/statm: 683698 627957 466 189 0 678047 0
[pid=30417/tid=30419] ppid=30415 vsize=2734792 CPUtime=4896.65 cores=0-7
/proc/30417/task/30419/stat : 30419 (claspmt) R 30415 30417 29577 0 -1 4202560 49757 0 0 0 489604 61 0 0 25 0 8 0 279849569 2800427008 627957 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30417/tid=30420] ppid=30415 vsize=2734792 CPUtime=4899.82 cores=0-7
/proc/30417/task/30420/stat : 30420 (claspmt) R 30415 30417 29577 0 -1 4202560 33104 0 0 0 489951 31 0 0 25 0 8 0 279849569 2800427008 627957 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30417/tid=30421] ppid=30415 vsize=2734792 CPUtime=4900.23 cores=0-7
/proc/30417/task/30421/stat : 30421 (claspmt) R 30415 30417 29577 0 -1 4202560 19890 0 0 0 489994 29 0 0 25 0 8 0 279849569 2800427008 627957 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30417/tid=30422] ppid=30415 vsize=2734792 CPUtime=4859.55 cores=0-7
/proc/30417/task/30422/stat : 30422 (claspmt) R 30415 30417 29577 0 -1 4202560 33828 0 0 0 485917 38 0 0 25 0 8 0 279849569 2800427008 627957 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30417/tid=30423] ppid=30415 vsize=2734792 CPUtime=4900.13 cores=0-7
/proc/30417/task/30423/stat : 30423 (claspmt) R 30415 30417 29577 0 -1 4202560 25130 0 0 0 489990 23 0 0 25 0 8 0 279849569 2800427008 627957 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30417/tid=30424] ppid=30415 vsize=2734792 CPUtime=4900.47 cores=0-7
/proc/30417/task/30424/stat : 30424 (claspmt) R 30415 30417 29577 0 -1 4202560 30275 0 0 0 490003 44 0 0 25 0 8 0 279849569 2800427008 627957 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30417/tid=30425] ppid=30415 vsize=2734792 CPUtime=4900.52 cores=0-7
/proc/30417/task/30425/stat : 30425 (claspmt) R 30415 30417 29577 0 -1 4202560 372056 0 0 0 489592 460 0 0 25 0 8 0 279849569 2800427008 627957 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 39159.4
Current children cumulated vsize (KiB) 2734792

[startup+4962.3 s]
/proc/loadavg: 8.00 8.01 8.00 9/286 30569
/proc/meminfo: memFree=5800656/32951124 swapFree=67111524/67111528
[pid=30417] ppid=30415 vsize=2734792 CPUtime=39638.8 cores=0-7
/proc/30417/stat : 30417 (claspmt) R 30415 30417 29577 0 -1 4202496 637945 0 0 0 3963103 779 0 0 25 0 8 0 279849420 2800427008 628442 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670219 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/30417/statm: 683698 628442 466 189 0 678047 0
[pid=30417/tid=30419] ppid=30415 vsize=2734792 CPUtime=4956.61 cores=0-7
/proc/30417/task/30419/stat : 30419 (claspmt) R 30415 30417 29577 0 -1 4202560 49866 0 0 0 495599 62 0 0 25 0 8 0 279849569 2800427008 628442 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30417/tid=30420] ppid=30415 vsize=2734792 CPUtime=4959.82 cores=0-7
/proc/30417/task/30420/stat : 30420 (claspmt) R 30415 30417 29577 0 -1 4202560 33104 0 0 0 495951 31 0 0 25 0 8 0 279849569 2800427008 628442 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30417/tid=30421] ppid=30415 vsize=2734792 CPUtime=4960.23 cores=0-7
/proc/30417/task/30421/stat : 30421 (claspmt) R 30415 30417 29577 0 -1 4202560 19896 0 0 0 495994 29 0 0 25 0 8 0 279849569 2800427008 628442 33554432000 4194304 4964704 140734711428160 18446744073709551615 4659143 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30417/tid=30422] ppid=30415 vsize=2734792 CPUtime=4919 cores=0-7
/proc/30417/task/30422/stat : 30422 (claspmt) R 30415 30417 29577 0 -1 4202560 33828 0 0 0 491862 38 0 0 25 0 8 0 279849569 2800427008 628442 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30417/tid=30423] ppid=30415 vsize=2734792 CPUtime=4960.12 cores=0-7
/proc/30417/task/30423/stat : 30423 (claspmt) R 30415 30417 29577 0 -1 4202560 25132 0 0 0 495989 23 0 0 25 0 8 0 279849569 2800427008 628442 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30417/tid=30424] ppid=30415 vsize=2734792 CPUtime=4960.47 cores=0-7
/proc/30417/task/30424/stat : 30424 (claspmt) R 30415 30417 29577 0 -1 4202560 30275 0 0 0 496003 44 0 0 25 0 8 0 279849569 2800427008 628442 33554432000 4194304 4964704 140734711428160 18446744073709551615 4627263 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30417/tid=30425] ppid=30415 vsize=2734792 CPUtime=4960.51 cores=0-7
/proc/30417/task/30425/stat : 30425 (claspmt) R 30415 30417 29577 0 -1 4202560 372427 0 0 0 495590 461 0 0 25 0 8 0 279849569 2800427008 628442 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 39638.8
Current children cumulated vsize (KiB) 2734792



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5007.5 s]
/proc/loadavg: 8.00 8.00 8.00 9/286 30570
/proc/meminfo: memFree=5800620/32951124 swapFree=67111524/67111528
[pid=30417] ppid=30415 vsize=2734792 CPUtime=40000.1 cores=0-7
/proc/30417/stat : 30417 (claspmt) R 30415 30417 29577 0 -1 4202496 637963 0 0 0 3999228 780 0 0 25 0 8 0 279849420 2800427008 628460 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/30417/statm: 683698 628460 466 189 0 678047 0
[pid=30417/tid=30419] ppid=30415 vsize=2734792 CPUtime=5001.82 cores=0-7
/proc/30417/task/30419/stat : 30419 (claspmt) R 30415 30417 29577 0 -1 4202560 49879 0 0 0 500120 62 0 0 25 0 8 0 279849569 2800427008 628460 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30417/tid=30420] ppid=30415 vsize=2734792 CPUtime=5005.02 cores=0-7
/proc/30417/task/30420/stat : 30420 (claspmt) R 30415 30417 29577 0 -1 4202560 33104 0 0 0 500471 31 0 0 25 0 8 0 279849569 2800427008 628460 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30417/tid=30421] ppid=30415 vsize=2734792 CPUtime=5005.44 cores=0-7
/proc/30417/task/30421/stat : 30421 (claspmt) R 30415 30417 29577 0 -1 4202560 19896 0 0 0 500515 29 0 0 25 0 8 0 279849569 2800427008 628460 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30417/tid=30422] ppid=30415 vsize=2734792 CPUtime=4963.79 cores=0-7
/proc/30417/task/30422/stat : 30422 (claspmt) R 30415 30417 29577 0 -1 4202560 33828 0 0 0 496341 38 0 0 25 0 8 0 279849569 2800427008 628460 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30417/tid=30423] ppid=30415 vsize=2734792 CPUtime=5005.34 cores=0-7
/proc/30417/task/30423/stat : 30423 (claspmt) R 30415 30417 29577 0 -1 4202560 25132 0 0 0 500511 23 0 0 25 0 8 0 279849569 2800427008 628460 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30417/tid=30424] ppid=30415 vsize=2734792 CPUtime=5005.67 cores=0-7
/proc/30417/task/30424/stat : 30424 (claspmt) R 30415 30417 29577 0 -1 4202560 30277 0 0 0 500523 44 0 0 25 0 8 0 279849569 2800427008 628460 33554432000 4194304 4964704 140734711428160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30417/tid=30425] ppid=30415 vsize=2734792 CPUtime=5005.72 cores=0-7
/proc/30417/task/30425/stat : 30425 (claspmt) R 30415 30417 29577 0 -1 4202560 372430 0 0 0 500111 461 0 0 25 0 8 0 279849569 2800427008 628460 33554432000 4194304 4964704 140734711428160 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 40000.1
Current children cumulated vsize (KiB) 2734792

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5007.8 s]
/proc/loadavg: 8.00 8.00 8.00 9/286 30570
/proc/meminfo: memFree=5800620/32951124 swapFree=67111524/67111528
[pid=30417] ppid=30415 vsize=2726284 CPUtime=40000.4 cores=0-7
/proc/30417/stat : 30417 (claspmt) R 30415 30417 29577 0 -1 4202496 637974 0 0 0 3999261 780 0 0 25 0 1 0 279849420 2791714816 626546 33554432000 4194304 4964704 140734711428160 18446744073709551615 4646560 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/30417/statm: 681571 626546 476 189 0 675920 0
Current children cumulated CPU time (s) 40000.4
Current children cumulated vsize (KiB) 2726284

[startup+5008.2 s]
/proc/loadavg: 8.00 8.00 8.00 2/280 30571
/proc/meminfo: memFree=5814308/32951124 swapFree=67111524/67111528
[pid=30417] ppid=30415 vsize=2718080 CPUtime=40000.8 cores=0-7
/proc/30417/stat : 30417 (claspmt) R 30415 30417 29577 0 -1 4202496 637974 0 0 0 3999302 780 0 0 25 0 1 0 279849420 2783313920 624851 33554432000 4194304 4964704 140734711428160 18446744073709551615 268660317537 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/30417/statm: 679520 624851 476 189 0 673869 0
Current children cumulated CPU time (s) 40000.8
Current children cumulated vsize (KiB) 2718080

[startup+5008.4 s]
/proc/loadavg: 8.00 8.00 8.00 2/280 30571
/proc/meminfo: memFree=5814308/32951124 swapFree=67111524/67111528
[pid=30417] ppid=30415 vsize=2505064 CPUtime=40001 cores=0-7
/proc/30417/stat : 30417 (claspmt) R 30415 30417 29577 0 -1 4202496 637974 0 0 0 3999320 781 0 0 25 0 1 0 279849420 2565185536 560962 33554432000 4194304 4964704 140734711428160 18446744073709551615 268660705639 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/30417/statm: 626266 560687 476 189 0 620615 0
Current children cumulated CPU time (s) 40001
Current children cumulated vsize (KiB) 2505064

Child status: 0
Real time (s): 5008.5
CPU time (s): 40001.1
CPU user time (s): 39993.2
CPU system time (s): 7.9088
CPU usage (%): 798.665
Max. virtual memory (cumulated for all children) (KiB): 2734792

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39993.2
system time used= 7.9088
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 637975
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 63986
involuntary context switches= 190065

runsolver used 12.927 second user time and 32.621 second system time

The end

Launcher Data 

Begin job on node107 at 2011-05-02 19:16:22
IDJOB=3321503
IDBENCH=71471
IDSOLVER=1711
FILE ID=node107/3321503-1304356581
RUNJOBID= node107-1304348805-29591
PBS_JOBID= 13178668
Free space on /tmp= 72924 MiB

SOLVER NAME= claspmt 2.0-R4095
BENCH NAME= SAT09/CRAFTED/rbsat/crafted/unforced/rbsat-v2400c266431g6.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3321503-1304356581/watcher-3321503-1304356581 -o /tmp/evaluation-result-3321503-1304356581/solver-3321503-1304356581 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3321503-1304356581.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= d0bb6612a0c88017c970eb8fd1a4be9e
RANDOM SEED=1471365132

node107.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5359.47
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:       8314632 kB
Buffers:        305120 kB
Cached:        3145340 kB
SwapCached:          4 kB
Active:       20927348 kB
Inactive:      3315804 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:       8314632 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:            4292 kB
Writeback:           0 kB
AnonPages:    20793744 kB
Mapped:          25548 kB
Slab:           266896 kB
PageTables:      58152 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 54926988 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72920 MiB
End job on node107 at 2011-05-02 20:39:51