Trace number 3380763

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 39713.6 5102.44

General information on the benchmark

Nameapplication/rintanen/SATPlanning/
sokoban-sequential-p145-microban-sequential.060-NOTKNOWN.cnf
MD5SUMe44b288003a3eaf34673a2c0fb8c6f3e
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark1095.17
Satisfiable
(Un)Satisfiability was proved
Number of variables131484
Number of clauses2120376
Sum of the clauses size4841016
Maximum clause length78
Minimum clause length1
Number of clauses of size 1696
Number of clauses of size 22046240
Number of clauses of size 35760
Number of clauses of size 426640
Number of clauses of size 511520
Number of clauses of size over 529520

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...491401/instance-3380763-1305491401.cnf
1.99/2.04	c Solving...
39711.10/5100.01	
39711.10/5100.01	*** INTERRUPTED! ***
39711.10/5100.01	*** Shutting down threads...

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3380763-1305491401/watcher-3380763-1305491401 -o /tmp/evaluation-result-3380763-1305491401/solver-3380763-1305491401 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380763-1305491401.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.96 8.00 8.17 3/300 21040
/proc/meminfo: memFree=15824588/32951124 swapFree=52307768/67111528
[pid=21040] ppid=21038 vsize=37576 CPUtime=0 cores=0-7
/proc/21040/stat : 21040 (claspmt) R 21038 21040 21014 0 -1 4202496 3038 0 0 0 0 0 0 0 23 0 1 0 660742981 38477824 2968 33554432000 4194304 4964800 140735592082128 18446744073709551615 4700412 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/21040/statm: 9394 2976 376 189 0 4268 0

[startup+0.108845 s]
/proc/loadavg: 6.96 8.00 8.17 3/300 21040
/proc/meminfo: memFree=15824588/32951124 swapFree=52307768/67111528
[pid=21040] ppid=21038 vsize=50412 CPUtime=0.1 cores=0-7
/proc/21040/stat : 21040 (claspmt) R 21038 21040 21014 0 -1 4202496 9086 0 0 0 9 1 0 0 24 0 1 0 660742981 51621888 7697 33554432000 4194304 4964800 140735592082128 18446744073709551615 218370618053 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/21040/statm: 12603 7698 389 189 0 7477 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 50412

[startup+0.200827 s]
/proc/loadavg: 6.96 8.00 8.17 3/300 21040
/proc/meminfo: memFree=15824588/32951124 swapFree=52307768/67111528
[pid=21040] ppid=21038 vsize=65084 CPUtime=0.19 cores=0-7
/proc/21040/stat : 21040 (claspmt) R 21038 21040 21014 0 -1 4202496 14223 0 0 0 17 2 0 0 24 0 1 0 660742981 66646016 11368 33554432000 4194304 4964800 140735592082128 18446744073709551615 218370618053 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/21040/statm: 16271 11369 389 189 0 11145 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 65084

[startup+0.300821 s]
/proc/loadavg: 6.96 8.00 8.17 3/300 21040
/proc/meminfo: memFree=15824588/32951124 swapFree=52307768/67111528
[pid=21040] ppid=21038 vsize=82840 CPUtime=0.29 cores=0-7
/proc/21040/stat : 21040 (claspmt) R 21038 21040 21014 0 -1 4202496 20587 0 0 0 27 2 0 0 25 0 1 0 660742981 84828160 15453 33554432000 4194304 4964800 140735592082128 18446744073709551615 218370618053 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/21040/statm: 20710 15454 389 189 0 15584 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 82840

[startup+0.700738 s]
/proc/loadavg: 6.96 8.00 8.17 3/300 21040
/proc/meminfo: memFree=15824588/32951124 swapFree=52307768/67111528
[pid=21040] ppid=21038 vsize=147768 CPUtime=0.69 cores=0-7
/proc/21040/stat : 21040 (claspmt) R 21038 21040 21014 0 -1 4202496 42291 0 0 0 64 5 0 0 25 0 1 0 660742981 151314432 30713 33554432000 4194304 4964800 140735592082128 18446744073709551615 4615747 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/21040/statm: 36942 30713 399 189 0 31816 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 147768

[startup+1.5006 s]
/proc/loadavg: 6.96 8.00 8.17 2/301 21041
/proc/meminfo: memFree=15681120/32951124 swapFree=52307768/67111528
[pid=21040] ppid=21038 vsize=171920 CPUtime=1.49 cores=0-7
/proc/21040/stat : 21040 (claspmt) R 21038 21040 21014 0 -1 4202496 47811 0 0 0 144 5 0 0 25 0 1 0 660742981 176046080 36233 33554432000 4194304 4964800 140735592082128 18446744073709551615 4607536 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/21040/statm: 42980 36233 403 189 0 37854 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 171920

[startup+3.10031 s]
/proc/loadavg: 6.96 8.00 8.17 2/308 21049
/proc/meminfo: memFree=15590420/32951124 swapFree=52307768/67111528
[pid=21040] ppid=21038 vsize=519320 CPUtime=3.43 cores=0-7
/proc/21040/stat : 21040 (claspmt) S 21038 21040 21014 0 -1 4202496 80530 0 0 0 327 16 0 0 18 0 8 0 660742981 531783680 59471 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/21040/statm: 129830 59471 443 189 0 124179 0
[pid=21040/tid=21043] ppid=21038 vsize=519320 CPUtime=0.03 cores=0-7
/proc/21040/task/21043/stat : 21043 (claspmt) S 21038 21040 21014 0 -1 4202560 982 0 0 0 3 0 0 0 18 0 8 0 660743185 531783680 59471 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 1 0 0 0
[pid=21040/tid=21044] ppid=21038 vsize=519320 CPUtime=0.07 cores=0-7
/proc/21040/task/21044/stat : 21044 (claspmt) S 21038 21040 21014 0 -1 4202560 5204 0 0 0 6 1 0 0 18 0 8 0 660743185 531783680 59471 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 7 0 0 0
[pid=21040/tid=21045] ppid=21038 vsize=519320 CPUtime=0.07 cores=0-7
/proc/21040/task/21045/stat : 21045 (claspmt) S 21038 21040 21014 0 -1 4202560 5956 0 0 0 6 1 0 0 18 0 8 0 660743185 531783680 59471 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 5 0 0 0
[pid=21040/tid=21046] ppid=21038 vsize=519320 CPUtime=0.03 cores=0-7
/proc/21040/task/21046/stat : 21046 (claspmt) S 21038 21040 21014 0 -1 4202560 4480 0 0 0 2 1 0 0 18 0 8 0 660743185 531783680 59471 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 0 0 0 0
[pid=21040/tid=21047] ppid=21038 vsize=519320 CPUtime=0.99 cores=0-7
/proc/21040/task/21047/stat : 21047 (claspmt) R 21038 21040 21014 0 -1 4202560 6445 0 0 0 97 2 0 0 19 0 8 0 660743185 531783680 59471 33554432000 4194304 4964800 140735592082128 18446744073709551615 4399292 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=21040/tid=21048] ppid=21038 vsize=519320 CPUtime=0.06 cores=0-7
/proc/21040/task/21048/stat : 21048 (claspmt) S 21038 21040 21014 0 -1 4202560 3895 0 0 0 6 0 0 0 18 0 8 0 660743185 531783680 59471 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 6 0 0 0
[pid=21040/tid=21049] ppid=21038 vsize=519320 CPUtime=0.07 cores=0-7
/proc/21040/task/21049/stat : 21049 (claspmt) S 21038 21040 21014 0 -1 4202560 5670 0 0 0 6 1 0 0 18 0 8 0 660743185 531783680 59471 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3.43
Current children cumulated vsize (KiB) 519320

[startup+6.30075 s]
/proc/loadavg: 6.48 7.89 8.13 2/308 21049
/proc/meminfo: memFree=15588980/32951124 swapFree=52307768/67111528
[pid=21040] ppid=21038 vsize=519320 CPUtime=6.62 cores=0-7
/proc/21040/stat : 21040 (claspmt) S 21038 21040 21014 0 -1 4202496 80685 0 0 0 646 16 0 0 18 0 8 0 660742981 531783680 59626 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/21040/statm: 129830 59626 443 189 0 124179 0
[pid=21040/tid=21043] ppid=21038 vsize=519320 CPUtime=0.03 cores=0-7
/proc/21040/task/21043/stat : 21043 (claspmt) S 21038 21040 21014 0 -1 4202560 982 0 0 0 3 0 0 0 18 0 8 0 660743185 531783680 59626 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 1 0 0 0
[pid=21040/tid=21044] ppid=21038 vsize=519320 CPUtime=0.07 cores=0-7
/proc/21040/task/21044/stat : 21044 (claspmt) S 21038 21040 21014 0 -1 4202560 5204 0 0 0 6 1 0 0 18 0 8 0 660743185 531783680 59626 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 7 0 0 0
[pid=21040/tid=21045] ppid=21038 vsize=519320 CPUtime=0.07 cores=0-7
/proc/21040/task/21045/stat : 21045 (claspmt) S 21038 21040 21014 0 -1 4202560 5956 0 0 0 6 1 0 0 18 0 8 0 660743185 531783680 59626 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 5 0 0 0
[pid=21040/tid=21046] ppid=21038 vsize=519320 CPUtime=0.03 cores=0-7
/proc/21040/task/21046/stat : 21046 (claspmt) S 21038 21040 21014 0 -1 4202560 4480 0 0 0 2 1 0 0 18 0 8 0 660743185 531783680 59626 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 0 0 0 0
[pid=21040/tid=21047] ppid=21038 vsize=519320 CPUtime=4.2 cores=0-7
/proc/21040/task/21047/stat : 21047 (claspmt) R 21038 21040 21014 0 -1 4202560 6600 0 0 0 417 3 0 0 25 0 8 0 660743185 531783680 59626 33554432000 4194304 4964800 140735592082128 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=21040/tid=21048] ppid=21038 vsize=519320 CPUtime=0.06 cores=0-7
/proc/21040/task/21048/stat : 21048 (claspmt) S 21038 21040 21014 0 -1 4202560 3895 0 0 0 6 0 0 0 18 0 8 0 660743185 531783680 59626 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 6 0 0 0
[pid=21040/tid=21049] ppid=21038 vsize=519320 CPUtime=0.07 cores=0-7
/proc/21040/task/21049/stat : 21049 (claspmt) S 21038 21040 21014 0 -1 4202560 5670 0 0 0 6 1 0 0 18 0 8 0 660743185 531783680 59626 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 6.62
Current children cumulated vsize (KiB) 519320

[startup+12.7006 s]
/proc/loadavg: 6.04 7.77 8.09 2/308 21049
/proc/meminfo: memFree=15590040/32951124 swapFree=52307768/67111528
[pid=21040] ppid=21038 vsize=519320 CPUtime=13.02 cores=0-7
/proc/21040/stat : 21040 (claspmt) S 21038 21040 21014 0 -1 4202496 80809 0 0 0 1286 16 0 0 18 0 8 0 660742981 531783680 59750 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/21040/statm: 129830 59750 443 189 0 124179 0
[pid=21040/tid=21043] ppid=21038 vsize=519320 CPUtime=0.03 cores=0-7
/proc/21040/task/21043/stat : 21043 (claspmt) S 21038 21040 21014 0 -1 4202560 982 0 0 0 3 0 0 0 18 0 8 0 660743185 531783680 59750 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 1 0 0 0
[pid=21040/tid=21044] ppid=21038 vsize=519320 CPUtime=0.07 cores=0-7
/proc/21040/task/21044/stat : 21044 (claspmt) S 21038 21040 21014 0 -1 4202560 5204 0 0 0 6 1 0 0 18 0 8 0 660743185 531783680 59750 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 7 0 0 0
[pid=21040/tid=21045] ppid=21038 vsize=519320 CPUtime=0.07 cores=0-7
/proc/21040/task/21045/stat : 21045 (claspmt) S 21038 21040 21014 0 -1 4202560 5956 0 0 0 6 1 0 0 18 0 8 0 660743185 531783680 59750 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 5 0 0 0
[pid=21040/tid=21046] ppid=21038 vsize=519320 CPUtime=0.03 cores=0-7
/proc/21040/task/21046/stat : 21046 (claspmt) S 21038 21040 21014 0 -1 4202560 4480 0 0 0 2 1 0 0 18 0 8 0 660743185 531783680 59750 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 0 0 0 0
[pid=21040/tid=21047] ppid=21038 vsize=519320 CPUtime=10.59 cores=0-7
/proc/21040/task/21047/stat : 21047 (claspmt) R 21038 21040 21014 0 -1 4202560 6724 0 0 0 1056 3 0 0 25 0 8 0 660743185 531783680 59750 33554432000 4194304 4964800 140735592082128 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=21040/tid=21048] ppid=21038 vsize=519320 CPUtime=0.06 cores=0-7
/proc/21040/task/21048/stat : 21048 (claspmt) S 21038 21040 21014 0 -1 4202560 3895 0 0 0 6 0 0 0 18 0 8 0 660743185 531783680 59750 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 6 0 0 0
[pid=21040/tid=21049] ppid=21038 vsize=519320 CPUtime=0.07 cores=0-7
/proc/21040/task/21049/stat : 21049 (claspmt) S 21038 21040 21014 0 -1 4202560 5670 0 0 0 6 1 0 0 18 0 8 0 660743185 531783680 59750 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 13.02
Current children cumulated vsize (KiB) 519320

[startup+25.5013 s]
/proc/loadavg: 4.92 7.44 7.98 2/308 21049
/proc/meminfo: memFree=15590856/32951124 swapFree=52307768/67111528
[pid=21040] ppid=21038 vsize=519320 CPUtime=25.82 cores=0-7
/proc/21040/stat : 21040 (claspmt) S 21038 21040 21014 0 -1 4202496 80941 0 0 0 2566 16 0 0 18 0 8 0 660742981 531783680 59882 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/21040/statm: 129830 59882 443 189 0 124179 0
[pid=21040/tid=21043] ppid=21038 vsize=519320 CPUtime=0.03 cores=0-7
/proc/21040/task/21043/stat : 21043 (claspmt) S 21038 21040 21014 0 -1 4202560 982 0 0 0 3 0 0 0 18 0 8 0 660743185 531783680 59882 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 1 0 0 0
[pid=21040/tid=21044] ppid=21038 vsize=519320 CPUtime=0.07 cores=0-7
/proc/21040/task/21044/stat : 21044 (claspmt) S 21038 21040 21014 0 -1 4202560 5204 0 0 0 6 1 0 0 18 0 8 0 660743185 531783680 59882 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 7 0 0 0
[pid=21040/tid=21045] ppid=21038 vsize=519320 CPUtime=0.07 cores=0-7
/proc/21040/task/21045/stat : 21045 (claspmt) S 21038 21040 21014 0 -1 4202560 5956 0 0 0 6 1 0 0 18 0 8 0 660743185 531783680 59882 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 5 0 0 0
[pid=21040/tid=21046] ppid=21038 vsize=519320 CPUtime=0.03 cores=0-7
/proc/21040/task/21046/stat : 21046 (claspmt) S 21038 21040 21014 0 -1 4202560 4480 0 0 0 2 1 0 0 18 0 8 0 660743185 531783680 59882 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 0 0 0 0
[pid=21040/tid=21047] ppid=21038 vsize=519320 CPUtime=23.39 cores=0-7
/proc/21040/task/21047/stat : 21047 (claspmt) R 21038 21040 21014 0 -1 4202560 6856 0 0 0 2336 3 0 0 25 0 8 0 660743185 531783680 59882 33554432000 4194304 4964800 140735592082128 18446744073709551615 4399324 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=21040/tid=21048] ppid=21038 vsize=519320 CPUtime=0.06 cores=0-7
/proc/21040/task/21048/stat : 21048 (claspmt) S 21038 21040 21014 0 -1 4202560 3895 0 0 0 6 0 0 0 18 0 8 0 660743185 531783680 59882 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 6 0 0 0
[pid=21040/tid=21049] ppid=21038 vsize=519320 CPUtime=0.07 cores=0-7
/proc/21040/task/21049/stat : 21049 (claspmt) S 21038 21040 21014 0 -1 4202560 5670 0 0 0 6 1 0 0 18 0 8 0 660743185 531783680 59882 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 25.82
Current children cumulated vsize (KiB) 519320

[startup+51.1008 s]
/proc/loadavg: 3.58 6.92 7.79 2/308 21050
/proc/meminfo: memFree=15589456/32951124 swapFree=52307768/67111528
[pid=21040] ppid=21038 vsize=519320 CPUtime=51.43 cores=0-7
/proc/21040/stat : 21040 (claspmt) S 21038 21040 21014 0 -1 4202496 81148 0 0 0 5126 17 0 0 18 0 8 0 660742981 531783680 60089 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/21040/statm: 129830 60089 443 189 0 124179 0
[pid=21040/tid=21043] ppid=21038 vsize=519320 CPUtime=0.03 cores=0-7
/proc/21040/task/21043/stat : 21043 (claspmt) S 21038 21040 21014 0 -1 4202560 982 0 0 0 3 0 0 0 18 0 8 0 660743185 531783680 60089 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 1 0 0 0
[pid=21040/tid=21044] ppid=21038 vsize=519320 CPUtime=0.07 cores=0-7
/proc/21040/task/21044/stat : 21044 (claspmt) S 21038 21040 21014 0 -1 4202560 5204 0 0 0 6 1 0 0 18 0 8 0 660743185 531783680 60089 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 7 0 0 0
[pid=21040/tid=21045] ppid=21038 vsize=519320 CPUtime=0.07 cores=0-7
/proc/21040/task/21045/stat : 21045 (claspmt) S 21038 21040 21014 0 -1 4202560 5956 0 0 0 6 1 0 0 18 0 8 0 660743185 531783680 60089 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 5 0 0 0
[pid=21040/tid=21046] ppid=21038 vsize=519320 CPUtime=0.03 cores=0-7
/proc/21040/task/21046/stat : 21046 (claspmt) S 21038 21040 21014 0 -1 4202560 4480 0 0 0 2 1 0 0 18 0 8 0 660743185 531783680 60089 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 0 0 0 0
[pid=21040/tid=21047] ppid=21038 vsize=519320 CPUtime=48.99 cores=0-7
/proc/21040/task/21047/stat : 21047 (claspmt) R 21038 21040 21014 0 -1 4202560 7063 0 0 0 4896 3 0 0 25 0 8 0 660743185 531783680 60089 33554432000 4194304 4964800 140735592082128 18446744073709551615 4668588 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=21040/tid=21048] ppid=21038 vsize=519320 CPUtime=0.06 cores=0-7
/proc/21040/task/21048/stat : 21048 (claspmt) S 21038 21040 21014 0 -1 4202560 3895 0 0 0 6 0 0 0 18 0 8 0 660743185 531783680 60089 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 6 0 0 0
[pid=21040/tid=21049] ppid=21038 vsize=519320 CPUtime=0.07 cores=0-7
/proc/21040/task/21049/stat : 21049 (claspmt) S 21038 21040 21014 0 -1 4202560 5670 0 0 0 6 1 0 0 18 0 8 0 660743185 531783680 60089 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 51.43
Current children cumulated vsize (KiB) 519320

[startup+102.301 s]
/proc/loadavg: 2.12 6.01 7.43 2/308 21051
/proc/meminfo: memFree=15588808/32951124 swapFree=52307768/67111528
[pid=21040] ppid=21038 vsize=519320 CPUtime=102.64 cores=0-7
/proc/21040/stat : 21040 (claspmt) S 21038 21040 21014 0 -1 4202496 81197 0 0 0 10247 17 0 0 18 0 8 0 660742981 531783680 60138 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/21040/statm: 129830 60138 443 189 0 124179 0
[pid=21040/tid=21043] ppid=21038 vsize=519320 CPUtime=0.03 cores=0-7
/proc/21040/task/21043/stat : 21043 (claspmt) S 21038 21040 21014 0 -1 4202560 982 0 0 0 3 0 0 0 18 0 8 0 660743185 531783680 60138 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 1 0 0 0
[pid=21040/tid=21044] ppid=21038 vsize=519320 CPUtime=0.07 cores=0-7
/proc/21040/task/21044/stat : 21044 (claspmt) S 21038 21040 21014 0 -1 4202560 5204 0 0 0 6 1 0 0 18 0 8 0 660743185 531783680 60138 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 7 0 0 0
[pid=21040/tid=21045] ppid=21038 vsize=519320 CPUtime=0.07 cores=0-7
/proc/21040/task/21045/stat : 21045 (claspmt) S 21038 21040 21014 0 -1 4202560 5956 0 0 0 6 1 0 0 18 0 8 0 660743185 531783680 60138 33554432000 4194304 4964800 140735592082128 18446744073709551615 218382773145 0 0 4096 18946 18446604469113483712 0 0 -1 5 0 0 0

################
# More data... #
################

/proc/meminfo: memFree=3450364/32951124 swapFree=52307768/67111528
[pid=21040] ppid=21038 vsize=12624948 CPUtime=38131.4 cores=0-7
/proc/21040/stat : 21040 (claspmt) R 21038 21040 21014 0 -1 4202496 3111600 0 0 0 3811107 2034 0 0 25 0 8 0 660742981 12927946752 3089650 33554432000 4194304 4964800 140735592082128 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/21040/statm: 3156237 3089650 466 189 0 3150586 0
[pid=21040/tid=21043] ppid=21038 vsize=12624948 CPUtime=4732.36 cores=0-7
/proc/21040/task/21043/stat : 21043 (claspmt) R 21038 21040 21014 0 -1 4202560 180780 0 0 0 473044 192 0 0 25 0 8 0 660743185 12927946752 3089650 33554432000 4194304 4964800 140735592082128 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=21040/tid=21044] ppid=21038 vsize=12624948 CPUtime=4728.59 cores=0-7
/proc/21040/task/21044/stat : 21044 (claspmt) R 21038 21040 21014 0 -1 4202560 965921 0 0 0 472370 489 0 0 25 0 8 0 660743185 12927946752 3089650 33554432000 4194304 4964800 140735592082128 18446744073709551615 4627252 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=21040/tid=21045] ppid=21038 vsize=12624948 CPUtime=4754.01 cores=0-7
/proc/21040/task/21045/stat : 21045 (claspmt) R 21038 21040 21014 0 -1 4202560 334738 0 0 0 475180 221 0 0 25 0 8 0 660743185 12927946752 3089650 33554432000 4194304 4964800 140735592082128 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=21040/tid=21046] ppid=21038 vsize=12624948 CPUtime=4754.04 cores=0-7
/proc/21040/task/21046/stat : 21046 (claspmt) R 21038 21040 21014 0 -1 4202560 229333 0 0 0 475249 155 0 0 25 0 8 0 660743185 12927946752 3089650 33554432000 4194304 4964800 140735592082128 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=21040/tid=21047] ppid=21038 vsize=12624948 CPUtime=4900.18 cores=0-7
/proc/21040/task/21047/stat : 21047 (claspmt) R 21038 21040 21014 0 -1 4202560 7144 0 0 0 490014 4 0 0 25 0 8 0 660743185 12927946752 3089650 33554432000 4194304 4964800 140735592082128 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=21040/tid=21048] ppid=21038 vsize=12624948 CPUtime=4752.15 cores=0-7
/proc/21040/task/21048/stat : 21048 (claspmt) R 21038 21040 21014 0 -1 4202560 314884 0 0 0 475005 210 0 0 25 0 8 0 660743185 12927946752 3089650 33554432000 4194304 4964800 140735592082128 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=21040/tid=21049] ppid=21038 vsize=12624948 CPUtime=4754.04 cores=0-7
/proc/21040/task/21049/stat : 21049 (claspmt) R 21038 21040 21014 0 -1 4202560 874300 0 0 0 474798 606 0 0 25 0 8 0 660743185 12927946752 3089650 33554432000 4194304 4964800 140735592082128 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 38131.4
Current children cumulated vsize (KiB) 12624948

[startup+4962.3 s]
/proc/loadavg: 8.10 8.04 8.00 9/308 21193
/proc/meminfo: memFree=3316148/32951124 swapFree=52307768/67111528
[pid=21040] ppid=21038 vsize=12756020 CPUtime=38610.8 cores=0-7
/proc/21040/stat : 21040 (claspmt) R 21038 21040 21014 0 -1 4202496 3145305 0 0 0 3859021 2057 0 0 25 0 8 0 660742981 13062164480 3123355 33554432000 4194304 4964800 140735592082128 18446744073709551615 4653507 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/21040/statm: 3189005 3123355 466 189 0 3183354 0
[pid=21040/tid=21043] ppid=21038 vsize=12756020 CPUtime=4791.73 cores=0-7
/proc/21040/task/21043/stat : 21043 (claspmt) R 21038 21040 21014 0 -1 4202560 181047 0 0 0 478980 193 0 0 25 0 8 0 660743185 13062164480 3123355 33554432000 4194304 4964800 140735592082128 18446744073709551615 4669509 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=21040/tid=21044] ppid=21038 vsize=12756020 CPUtime=4788.59 cores=0-7
/proc/21040/task/21044/stat : 21044 (claspmt) R 21038 21040 21014 0 -1 4202560 982660 0 0 0 478362 497 0 0 25 0 8 0 660743185 13062164480 3123355 33554432000 4194304 4964800 140735592082128 18446744073709551615 4670357 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=21040/tid=21045] ppid=21038 vsize=12756020 CPUtime=4814.02 cores=0-7
/proc/21040/task/21045/stat : 21045 (claspmt) R 21038 21040 21014 0 -1 4202560 334738 0 0 0 481181 221 0 0 25 0 8 0 660743185 13062164480 3123355 33554432000 4194304 4964800 140735592082128 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=21040/tid=21046] ppid=21038 vsize=12756020 CPUtime=4814.04 cores=0-7
/proc/21040/task/21046/stat : 21046 (claspmt) R 21038 21040 21014 0 -1 4202560 229647 0 0 0 481249 155 0 0 25 0 8 0 660743185 13062164480 3123355 33554432000 4194304 4964800 140735592082128 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=21040/tid=21047] ppid=21038 vsize=12756020 CPUtime=4960.18 cores=0-7
/proc/21040/task/21047/stat : 21047 (claspmt) R 21038 21040 21014 0 -1 4202560 7144 0 0 0 496014 4 0 0 25 0 8 0 660743185 13062164480 3123355 33554432000 4194304 4964800 140735592082128 18446744073709551615 4669131 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=21040/tid=21048] ppid=21038 vsize=12756020 CPUtime=4812.15 cores=0-7
/proc/21040/task/21048/stat : 21048 (claspmt) R 21038 21040 21014 0 -1 4202560 323987 0 0 0 480997 218 0 0 25 0 8 0 660743185 13062164480 3123355 33554432000 4194304 4964800 140735592082128 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=21040/tid=21049] ppid=21038 vsize=12756020 CPUtime=4814.05 cores=0-7
/proc/21040/task/21049/stat : 21049 (claspmt) R 21038 21040 21014 0 -1 4202560 881582 0 0 0 480792 613 0 0 25 0 8 0 660743185 13062164480 3123355 33554432000 4194304 4964800 140735592082128 18446744073709551615 4670499 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 38610.8
Current children cumulated vsize (KiB) 12756020

[startup+5022.3 s]
/proc/loadavg: 8.10 8.05 8.00 9/308 21194
/proc/meminfo: memFree=3153924/32951124 swapFree=52307768/67111528
[pid=21040] ppid=21038 vsize=12887092 CPUtime=39090.2 cores=0-7
/proc/21040/stat : 21040 (claspmt) R 21038 21040 21014 0 -1 4202496 3186503 0 0 0 3906934 2087 0 0 25 0 8 0 660742981 13196382208 3164553 33554432000 4194304 4964800 140735592082128 18446744073709551615 4670346 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/21040/statm: 3221773 3164553 466 189 0 3216122 0
[pid=21040/tid=21043] ppid=21038 vsize=12887092 CPUtime=4851.17 cores=0-7
/proc/21040/task/21043/stat : 21043 (claspmt) R 21038 21040 21014 0 -1 4202560 181047 0 0 0 484924 193 0 0 25 0 8 0 660743185 13196382208 3164553 33554432000 4194304 4964800 140735592082128 18446744073709551615 4668900 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=21040/tid=21044] ppid=21038 vsize=12887092 CPUtime=4848.59 cores=0-7
/proc/21040/task/21044/stat : 21044 (claspmt) R 21038 21040 21014 0 -1 4202560 982660 0 0 0 484362 497 0 0 25 0 8 0 660743185 13196382208 3164553 33554432000 4194304 4964800 140735592082128 18446744073709551615 4684107 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=21040/tid=21045] ppid=21038 vsize=12887092 CPUtime=4874.02 cores=0-7
/proc/21040/task/21045/stat : 21045 (claspmt) R 21038 21040 21014 0 -1 4202560 334738 0 0 0 487181 221 0 0 25 0 8 0 660743185 13196382208 3164553 33554432000 4194304 4964800 140735592082128 18446744073709551615 4668987 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=21040/tid=21046] ppid=21038 vsize=12887092 CPUtime=4874.04 cores=0-7
/proc/21040/task/21046/stat : 21046 (claspmt) R 21038 21040 21014 0 -1 4202560 229647 0 0 0 487249 155 0 0 25 0 8 0 660743185 13196382208 3164553 33554432000 4194304 4964800 140735592082128 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=21040/tid=21047] ppid=21038 vsize=12887092 CPUtime=5020.18 cores=0-7
/proc/21040/task/21047/stat : 21047 (claspmt) R 21038 21040 21014 0 -1 4202560 7144 0 0 0 502014 4 0 0 25 0 8 0 660743185 13196382208 3164553 33554432000 4194304 4964800 140735592082128 18446744073709551615 4669476 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=21040/tid=21048] ppid=21038 vsize=12887092 CPUtime=4872.14 cores=0-7
/proc/21040/task/21048/stat : 21048 (claspmt) R 21038 21040 21014 0 -1 4202560 333876 0 0 0 486988 226 0 0 25 0 8 0 660743185 13196382208 3164553 33554432000 4194304 4964800 140735592082128 18446744073709551615 4668900 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=21040/tid=21049] ppid=21038 vsize=12887092 CPUtime=4874.05 cores=0-7
/proc/21040/task/21049/stat : 21049 (claspmt) R 21038 21040 21014 0 -1 4202560 912891 0 0 0 486770 635 0 0 25 0 8 0 660743185 13196382208 3164553 33554432000 4194304 4964800 140735592082128 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39090.2
Current children cumulated vsize (KiB) 12887092

[startup+5082.3 s]
/proc/loadavg: 8.04 8.04 8.00 9/308 21196
/proc/meminfo: memFree=3073272/32951124 swapFree=52307768/67111528
[pid=21040] ppid=21038 vsize=12952628 CPUtime=39569.6 cores=0-7
/proc/21040/stat : 21040 (claspmt) R 21038 21040 21014 0 -1 4202496 3206039 0 0 0 3954860 2100 0 0 25 0 8 0 660742981 13263491072 3184089 33554432000 4194304 4964800 140735592082128 18446744073709551615 4669479 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/21040/statm: 3238157 3184089 466 189 0 3232506 0
[pid=21040/tid=21043] ppid=21038 vsize=12952628 CPUtime=4910.57 cores=0-7
/proc/21040/task/21043/stat : 21043 (claspmt) R 21038 21040 21014 0 -1 4202560 181047 0 0 0 490864 193 0 0 25 0 8 0 660743185 13263491072 3184089 33554432000 4194304 4964800 140735592082128 18446744073709551615 4669089 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=21040/tid=21044] ppid=21038 vsize=12952628 CPUtime=4908.59 cores=0-7
/proc/21040/task/21044/stat : 21044 (claspmt) R 21038 21040 21014 0 -1 4202560 982660 0 0 0 490362 497 0 0 25 0 8 0 660743185 13263491072 3184089 33554432000 4194304 4964800 140735592082128 18446744073709551615 4313751 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=21040/tid=21045] ppid=21038 vsize=12952628 CPUtime=4934.02 cores=0-7
/proc/21040/task/21045/stat : 21045 (claspmt) R 21038 21040 21014 0 -1 4202560 334847 0 0 0 493181 221 0 0 25 0 8 0 660743185 13263491072 3184089 33554432000 4194304 4964800 140735592082128 18446744073709551615 4669376 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=21040/tid=21046] ppid=21038 vsize=12952628 CPUtime=4934.04 cores=0-7
/proc/21040/task/21046/stat : 21046 (claspmt) R 21038 21040 21014 0 -1 4202560 229647 0 0 0 493249 155 0 0 25 0 8 0 660743185 13263491072 3184089 33554432000 4194304 4964800 140735592082128 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=21040/tid=21047] ppid=21038 vsize=12952628 CPUtime=5080.18 cores=0-7
/proc/21040/task/21047/stat : 21047 (claspmt) R 21038 21040 21014 0 -1 4202560 7144 0 0 0 508014 4 0 0 25 0 8 0 660743185 13263491072 3184089 33554432000 4194304 4964800 140735592082128 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=21040/tid=21048] ppid=21038 vsize=12952628 CPUtime=4932.13 cores=0-7
/proc/21040/task/21048/stat : 21048 (claspmt) R 21038 21040 21014 0 -1 4202560 349896 0 0 0 492978 235 0 0 25 0 8 0 660743185 13263491072 3184089 33554432000 4194304 4964800 140735592082128 18446744073709551615 4653507 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=21040/tid=21049] ppid=21038 vsize=12952628 CPUtime=4934.05 cores=0-7
/proc/21040/task/21049/stat : 21049 (claspmt) R 21038 21040 21014 0 -1 4202560 916298 0 0 0 492767 638 0 0 25 0 8 0 660743185 13263491072 3184089 33554432000 4194304 4964800 140735592082128 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39569.6
Current children cumulated vsize (KiB) 12952628



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 8.02 8.03 8.00 9/308 21196
/proc/meminfo: memFree=3031720/32951124 swapFree=52307768/67111528
[pid=21040] ppid=21038 vsize=13083700 CPUtime=39711.1 cores=0-7
/proc/21040/stat : 21040 (claspmt) R 21038 21040 21014 0 -1 4202496 3216256 0 0 0 3969001 2109 0 0 25 0 8 0 660742981 13397708800 3194306 33554432000 4194304 4964800 140735592082128 18446744073709551615 4669350 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/21040/statm: 3270925 3194306 466 189 0 3265274 0
[pid=21040/tid=21043] ppid=21038 vsize=13083700 CPUtime=4928.09 cores=0-7
/proc/21040/task/21043/stat : 21043 (claspmt) R 21038 21040 21014 0 -1 4202560 181047 0 0 0 492616 193 0 0 25 0 8 0 660743185 13397708800 3194306 33554432000 4194304 4964800 140735592082128 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=21040/tid=21044] ppid=21038 vsize=13083700 CPUtime=4926.3 cores=0-7
/proc/21040/task/21044/stat : 21044 (claspmt) R 21038 21040 21014 0 -1 4202560 982660 0 0 0 492133 497 0 0 25 0 8 0 660743185 13397708800 3194306 33554432000 4194304 4964800 140735592082128 18446744073709551615 4313804 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=21040/tid=21045] ppid=21038 vsize=13083700 CPUtime=4951.73 cores=0-7
/proc/21040/task/21045/stat : 21045 (claspmt) R 21038 21040 21014 0 -1 4202560 334847 0 0 0 494952 221 0 0 25 0 8 0 660743185 13397708800 3194306 33554432000 4194304 4964800 140735592082128 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=21040/tid=21046] ppid=21038 vsize=13083700 CPUtime=4951.75 cores=0-7
/proc/21040/task/21046/stat : 21046 (claspmt) R 21038 21040 21014 0 -1 4202560 229647 0 0 0 495020 155 0 0 25 0 8 0 660743185 13397708800 3194306 33554432000 4194304 4964800 140735592082128 18446744073709551615 4627263 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=21040/tid=21047] ppid=21038 vsize=13083700 CPUtime=5097.89 cores=0-7
/proc/21040/task/21047/stat : 21047 (claspmt) R 21038 21040 21014 0 -1 4202560 7144 0 0 0 509785 4 0 0 25 0 8 0 660743185 13397708800 3194306 33554432000 4194304 4964800 140735592082128 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=21040/tid=21048] ppid=21038 vsize=13083700 CPUtime=4949.84 cores=0-7
/proc/21040/task/21048/stat : 21048 (claspmt) R 21038 21040 21014 0 -1 4202560 353436 0 0 0 494747 237 0 0 25 0 8 0 660743185 13397708800 3194306 33554432000 4194304 4964800 140735592082128 18446744073709551615 4342180 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=21040/tid=21049] ppid=21038 vsize=13083700 CPUtime=4951.76 cores=0-7
/proc/21040/task/21049/stat : 21049 (claspmt) R 21038 21040 21014 0 -1 4202560 922975 0 0 0 494531 645 0 0 25 0 8 0 660743185 13397708800 3194306 33554432000 4194304 4964800 140735592082128 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39711.1
Current children cumulated vsize (KiB) 13083700

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)

Sending SIGKILL to -21040

Solver just ended. Dumping a history of the last processes samples

[startup+5100.7 s]
/proc/loadavg: 8.02 8.03 8.00 2/302 21197
/proc/meminfo: memFree=3031768/32951124 swapFree=52307768/67111528
[pid=21040] ppid=21038 vsize=13083700 CPUtime=39711.9 cores=0-7
/proc/21040/stat : 21040 (claspmt) R 21038 21040 21014 0 -1 4202496 3216270 0 0 0 3969076 2109 0 0 22 0 1 0 660742981 13397708800 3194320 33554432000 4194304 4964800 140735592082128 18446744073709551615 4454134 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/21040/statm: 3270925 3194320 476 189 0 3265274 0
Current children cumulated CPU time (s) 39711.9
Current children cumulated vsize (KiB) 13083700

[startup+5101.5 s]
/proc/loadavg: 8.02 8.03 8.00 2/302 21197
/proc/meminfo: memFree=3031768/32951124 swapFree=52307768/67111528
[pid=21040] ppid=21038 vsize=13001596 CPUtime=39712.6 cores=0-7
/proc/21040/stat : 21040 (claspmt) R 21038 21040 21014 0 -1 4202496 3216270 0 0 0 3969154 2109 0 0 25 0 1 0 660742981 13313634304 3187169 33554432000 4194304 4964800 140735592082128 18446744073709551615 218370613819 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/21040/statm: 3250399 3187169 476 189 0 3244748 0
Current children cumulated CPU time (s) 39712.6
Current children cumulated vsize (KiB) 13001596

[startup+5101.91 s]
/proc/loadavg: 8.02 8.03 8.00 2/302 21197
/proc/meminfo: memFree=3066848/32951124 swapFree=52307768/67111528
[pid=21040] ppid=21038 vsize=12084092 CPUtime=39713.1 cores=0-7
/proc/21040/stat : 21040 (claspmt) R 21038 21040 21014 0 -1 4202496 3216270 0 0 0 3969192 2113 0 0 25 0 1 0 660742981 12374110208 2953484 33554432000 4194304 4964800 140735592082128 18446744073709551615 218370614260 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/21040/statm: 3021023 2953484 476 189 0 3015372 0
Current children cumulated CPU time (s) 39713.1
Current children cumulated vsize (KiB) 12084092

[startup+5102.3 s]
/proc/loadavg: 8.02 8.03 8.00 2/302 21197
/proc/meminfo: memFree=3066848/32951124 swapFree=52307768/67111528
[pid=21040] ppid=21038 vsize=0 CPUtime=39713.4 cores=0-7
/proc/21040/stat : 21040 (claspmt) R 21038 21040 21014 0 -1 4203524 3216270 0 0 0 3969192 2151 0 0 25 0 1 0 660742981 0 0 33554432000 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/21040/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 39713.4
Current children cumulated vsize (KiB) 0

[startup+5102.4 s]
/proc/loadavg: 8.02 8.03 8.00 2/302 21197
/proc/meminfo: memFree=3066848/32951124 swapFree=52307768/67111528
[pid=21040] ppid=21038 vsize=0 CPUtime=39713.5 cores=0-7
/proc/21040/stat : 21040 (claspmt) R 21038 21040 21014 0 -1 4203524 3216270 0 0 0 3969192 2161 0 0 25 0 1 0 660742981 0 0 33554432000 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/21040/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 39713.5
Current children cumulated vsize (KiB) 0

Child ended because it received signal 9 (SIGKILL)
Real time (s): 5102.44
CPU time (s): 39713.6
CPU user time (s): 39691.9
CPU system time (s): 21.6617
CPU usage (%): 778.325
Max. virtual memory (cumulated for all children) (KiB): 13083700

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39691.9
system time used= 21.6617
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 3216270
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 30915
involuntary context switches= 185107

runsolver used 13.8849 second user time and 36.3335 second system time

The end

Launcher Data 

Begin job on node101 at 2011-05-15 22:30:02
IDJOB=3380763
IDBENCH=82366
IDSOLVER=1865
FILE ID=node101/3380763-1305491401
RUNJOBID= node101-1305491401-21028
PBS_JOBID= 13325415
Free space on /tmp= 73720 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT11/application/rintanen/SATPlanning/sokoban-sequential-p145-microban-sequential.060-NOTKNOWN.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3380763-1305491401/watcher-3380763-1305491401 -o /tmp/evaluation-result-3380763-1305491401/solver-3380763-1305491401 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380763-1305491401.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= e44b288003a3eaf34673a2c0fb8c6f3e
RANDOM SEED=1541416347

node101.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      15824868 kB
Buffers:        348900 kB
Cached:        6247084 kB
SwapCached:      14848 kB
Active:       11648932 kB
Inactive:      5183368 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      15824868 kB
SwapTotal:    67111528 kB
SwapFree:     52307768 kB
Dirty:           38044 kB
Writeback:           4 kB
AnonPages:    10233248 kB
Mapped:          15468 kB
Slab:           161216 kB
PageTables:      64136 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 42582572 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73720 MiB
End job on node101 at 2011-05-15 23:55:05