Trace number 3380877

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40001.6 5011.49

General information on the benchmark

Nameapplication/jarvisalo/HWMCC10-combination-timeframe-expansion/
hwmcc10-timeframe-expansion-k45-pdtvisns3p02-tseitin.cnf
MD5SUMa0ec1baa890da62d0c6df7f1e62c6af3
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark2133.18
Satisfiable
(Un)Satisfiability was proved
Number of variables163622
Number of clauses488120
Sum of the clauses size1138944
Maximum clause length3
Minimum clause length1
Number of clauses of size 12
Number of clauses of size 2325412
Number of clauses of size 3162706
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...494111/instance-3380877-1305494111.cnf
1.29/1.37	c Solving...
40000.75/5010.72	
40000.75/5010.72	*** INTERRUPTED! ***
40000.75/5010.72	*** Shutting down threads...
40001.50/5011.48	s UNKNOWN
40001.50/5011.48	
40001.50/5011.48	c INTERRUPTED   : 1
40001.50/5011.48	c Models        : 0+    
40001.50/5011.48	c Time          : 5010.722s (Solving: 5009.35s 1st Model: 0.00s Unsat: 0.00s)
40001.50/5011.48	c CPU Time      : 40000.820s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3380877-1305494111/watcher-3380877-1305494111 -o /tmp/evaluation-result-3380877-1305494111/solver-3380877-1305494111 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380877-1305494111.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.53 4.32 5.88 3/374 30971
/proc/meminfo: memFree=20600280/32951132 swapFree=64495988/67111528
[pid=30971] ppid=30969 vsize=41652 CPUtime=0 cores=0-7
/proc/30971/stat : 30971 (claspmt) R 30969 30971 30539 0 -1 4202496 3823 0 0 0 0 0 0 0 18 0 1 0 661010955 42651648 3754 33554432000 4194304 4964800 140736539827376 18446744073709551615 4700412 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/30971/statm: 10413 3760 376 189 0 5287 0

[startup+0.067264 s]
/proc/loadavg: 3.53 4.32 5.88 3/374 30971
/proc/meminfo: memFree=20600280/32951132 swapFree=64495988/67111528
[pid=30971] ppid=30969 vsize=60904 CPUtime=0.06 cores=0-7
/proc/30971/stat : 30971 (claspmt) R 30969 30971 30539 0 -1 4202496 11916 0 0 0 5 1 0 0 18 0 1 0 661010955 62365696 10095 33554432000 4194304 4964800 140736539827376 18446744073709551615 210646843861 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/30971/statm: 15226 10110 386 189 0 10100 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 60904

[startup+0.100258 s]
/proc/loadavg: 3.53 4.32 5.88 3/374 30971
/proc/meminfo: memFree=20600280/32951132 swapFree=64495988/67111528
[pid=30971] ppid=30969 vsize=65116 CPUtime=0.08 cores=0-7
/proc/30971/stat : 30971 (claspmt) R 30969 30971 30539 0 -1 4202496 13737 0 0 0 7 1 0 0 18 0 1 0 661010955 66678784 11239 33554432000 4194304 4964800 140736539827376 18446744073709551615 4599599 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/30971/statm: 16279 11240 386 189 0 11153 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 65116

[startup+0.300222 s]
/proc/loadavg: 3.53 4.32 5.88 3/374 30971
/proc/meminfo: memFree=20600280/32951132 swapFree=64495988/67111528
[pid=30971] ppid=30969 vsize=101952 CPUtime=0.29 cores=0-7
/proc/30971/stat : 30971 (claspmt) R 30969 30971 30539 0 -1 4202496 25477 0 0 0 26 3 0 0 18 0 1 0 661010955 104398848 19822 33554432000 4194304 4964800 140736539827376 18446744073709551615 4607590 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/30971/statm: 25488 19822 403 189 0 20362 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 101952

[startup+0.700145 s]
/proc/loadavg: 3.53 4.32 5.88 3/374 30971
/proc/meminfo: memFree=20600280/32951132 swapFree=64495988/67111528
[pid=30971] ppid=30969 vsize=111984 CPUtime=0.69 cores=0-7
/proc/30971/stat : 30971 (claspmt) R 30969 30971 30539 0 -1 4202496 28018 0 0 0 66 3 0 0 18 0 1 0 661010955 114671616 22363 33554432000 4194304 4964800 140736539827376 18446744073709551615 210646813568 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/30971/statm: 27996 22363 404 189 0 22870 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 111984

[startup+1.50102 s]
/proc/loadavg: 3.49 4.30 5.86 2/375 30972
/proc/meminfo: memFree=20506652/32951132 swapFree=64495988/67111528
[pid=30971] ppid=30969 vsize=535600 CPUtime=1.65 cores=0-7
/proc/30971/stat : 30971 (claspmt) S 30969 30971 30539 0 -1 4202496 72425 0 0 0 146 19 0 0 18 0 8 0 661010955 548454400 55141 33554432000 4194304 4964800 140736539827376 18446744073709551615 210658962329 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/30971/statm: 133900 55141 442 189 0 128249 0
Current children cumulated CPU time (s) 1.65
Current children cumulated vsize (KiB) 535600

[startup+3.10071 s]
/proc/loadavg: 3.49 4.30 5.86 10/382 30979
/proc/meminfo: memFree=20375900/32951132 swapFree=64495988/67111528
[pid=30971] ppid=30969 vsize=535600 CPUtime=11.89 cores=0-7
/proc/30971/stat : 30971 (claspmt) R 30969 30971 30539 0 -1 4202496 74199 0 0 0 1168 21 0 0 17 0 8 0 661010955 548454400 56915 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/30971/statm: 133900 56915 464 189 0 128249 0
[pid=30971/tid=30973] ppid=30969 vsize=535600 CPUtime=1.33 cores=0-7
/proc/30971/task/30973/stat : 30973 (claspmt) R 30969 30971 30539 0 -1 4202560 4447 0 0 0 132 1 0 0 17 0 8 0 661011093 548454400 56915 33554432000 4194304 4964800 140736539827376 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30971/tid=30974] ppid=30969 vsize=535600 CPUtime=1.01 cores=0-7
/proc/30971/task/30974/stat : 30974 (claspmt) R 30969 30971 30539 0 -1 4202560 7208 0 0 0 99 2 0 0 16 0 8 0 661011093 548454400 56915 33554432000 4194304 4964800 140736539827376 18446744073709551615 4629451 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30971/tid=30975] ppid=30969 vsize=535600 CPUtime=1.32 cores=0-7
/proc/30971/task/30975/stat : 30975 (claspmt) R 30969 30971 30539 0 -1 4202560 7262 0 0 0 129 3 0 0 17 0 8 0 661011093 548454400 56915 33554432000 4194304 4964800 140736539827376 18446744073709551615 4668715 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30971/tid=30976] ppid=30969 vsize=535600 CPUtime=1.32 cores=0-7
/proc/30971/task/30976/stat : 30976 (claspmt) R 30969 30971 30539 0 -1 4202560 5513 0 0 0 131 1 0 0 17 0 8 0 661011093 548454400 56915 33554432000 4194304 4964800 140736539827376 18446744073709551615 4668715 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=30971/tid=30977] ppid=30969 vsize=535600 CPUtime=1.64 cores=0-7
/proc/30971/task/30977/stat : 30977 (claspmt) R 30969 30971 30539 0 -1 4202560 6116 0 0 0 161 3 0 0 20 0 8 0 661011093 548454400 56915 33554432000 4194304 4964800 140736539827376 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30971/tid=30978] ppid=30969 vsize=535600 CPUtime=1.2 cores=0-7
/proc/30971/task/30978/stat : 30978 (claspmt) R 30969 30971 30539 0 -1 4202560 6882 0 0 0 119 1 0 0 17 0 8 0 661011093 548454400 56915 33554432000 4194304 4964800 140736539827376 18446744073709551615 4669119 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30971/tid=30979] ppid=30969 vsize=535600 CPUtime=1.32 cores=0-7
/proc/30971/task/30979/stat : 30979 (claspmt) R 30969 30971 30539 0 -1 4202560 6114 0 0 0 130 2 0 0 17 0 8 0 661011093 548454400 56915 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 11.89
Current children cumulated vsize (KiB) 535600

[startup+6.30117 s]
/proc/loadavg: 3.85 4.36 5.87 9/382 30979
/proc/meminfo: memFree=20370568/32951132 swapFree=64495988/67111528
[pid=30971] ppid=30969 vsize=535600 CPUtime=36.54 cores=0-7
/proc/30971/stat : 30971 (claspmt) R 30969 30971 30539 0 -1 4202496 75713 0 0 0 3630 24 0 0 22 0 8 0 661010955 548454400 58429 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/30971/statm: 133900 58429 464 189 0 128249 0
[pid=30971/tid=30973] ppid=30969 vsize=535600 CPUtime=4.53 cores=0-7
/proc/30971/task/30973/stat : 30973 (claspmt) R 30969 30971 30539 0 -1 4202560 4568 0 0 0 451 2 0 0 23 0 8 0 661011093 548454400 58429 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670499 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30971/tid=30974] ppid=30969 vsize=535600 CPUtime=4.12 cores=0-7
/proc/30971/task/30974/stat : 30974 (claspmt) R 30969 30971 30539 0 -1 4202560 7473 0 0 0 410 2 0 0 21 0 8 0 661011093 548454400 58429 33554432000 4194304 4964800 140736539827376 18446744073709551615 4669406 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30971/tid=30975] ppid=30969 vsize=535600 CPUtime=4.51 cores=0-7
/proc/30971/task/30975/stat : 30975 (claspmt) R 30969 30971 30539 0 -1 4202560 7504 0 0 0 448 3 0 0 23 0 8 0 661011093 548454400 58429 33554432000 4194304 4964800 140736539827376 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30971/tid=30976] ppid=30969 vsize=535600 CPUtime=4.26 cores=0-7
/proc/30971/task/30976/stat : 30976 (claspmt) R 30969 30971 30539 0 -1 4202560 5688 0 0 0 424 2 0 0 22 0 8 0 661011093 548454400 58429 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30971/tid=30977] ppid=30969 vsize=535600 CPUtime=4.49 cores=0-7
/proc/30971/task/30977/stat : 30977 (claspmt) R 30969 30971 30539 0 -1 4202560 6119 0 0 0 446 3 0 0 25 0 8 0 661011093 548454400 58429 33554432000 4194304 4964800 140736539827376 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30971/tid=30978] ppid=30969 vsize=535600 CPUtime=4.39 cores=0-7
/proc/30971/task/30978/stat : 30978 (claspmt) R 30969 30971 30539 0 -1 4202560 7211 0 0 0 437 2 0 0 22 0 8 0 661011093 548454400 58429 33554432000 4194304 4964800 140736539827376 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=30971/tid=30979] ppid=30969 vsize=535600 CPUtime=4.52 cores=0-7
/proc/30971/task/30979/stat : 30979 (claspmt) R 30969 30971 30539 0 -1 4202560 6334 0 0 0 449 3 0 0 23 0 8 0 661011093 548454400 58429 33554432000 4194304 4964800 140736539827376 18446744073709551615 4668748 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 36.54
Current children cumulated vsize (KiB) 535600

[startup+12.7011 s]
/proc/loadavg: 4.18 4.42 5.88 9/382 30980
/proc/meminfo: memFree=20360552/32951132 swapFree=64495988/67111528
[pid=30971] ppid=30969 vsize=538248 CPUtime=87.64 cores=0-7
/proc/30971/stat : 30971 (claspmt) R 30969 30971 30539 0 -1 4202496 78528 0 0 0 8737 27 0 0 25 0 8 0 661010955 551165952 61244 33554432000 4194304 4964800 140736539827376 18446744073709551615 4321079 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/30971/statm: 134562 61244 464 189 0 128911 0
[pid=30971/tid=30973] ppid=30969 vsize=538248 CPUtime=10.92 cores=0-7
/proc/30971/task/30973/stat : 30973 (claspmt) R 30969 30971 30539 0 -1 4202560 4729 0 0 0 1090 2 0 0 25 0 8 0 661011093 551165952 61244 33554432000 4194304 4964800 140736539827376 18446744073709551615 4668987 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30971/tid=30974] ppid=30969 vsize=538248 CPUtime=10.52 cores=0-7
/proc/30971/task/30974/stat : 30974 (claspmt) R 30969 30971 30539 0 -1 4202560 8254 0 0 0 1049 3 0 0 25 0 8 0 661011093 551165952 61244 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30971/tid=30975] ppid=30969 vsize=538248 CPUtime=10.92 cores=0-7
/proc/30971/task/30975/stat : 30975 (claspmt) R 30969 30971 30539 0 -1 4202560 7929 0 0 0 1088 4 0 0 25 0 8 0 661011093 551165952 61244 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670203 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30971/tid=30976] ppid=30969 vsize=538248 CPUtime=10.57 cores=0-7
/proc/30971/task/30976/stat : 30976 (claspmt) R 30969 30971 30539 0 -1 4202560 6042 0 0 0 1055 2 0 0 25 0 8 0 661011093 551165952 61244 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30971/tid=30977] ppid=30969 vsize=538248 CPUtime=10.89 cores=0-7
/proc/30971/task/30977/stat : 30977 (claspmt) R 30969 30971 30539 0 -1 4202560 6184 0 0 0 1086 3 0 0 25 0 8 0 661011093 551165952 61244 33554432000 4194304 4964800 140736539827376 18446744073709551615 4399339 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30971/tid=30978] ppid=30969 vsize=538248 CPUtime=10.79 cores=0-7
/proc/30971/task/30978/stat : 30978 (claspmt) R 30969 30971 30539 0 -1 4202560 7859 0 0 0 1076 3 0 0 25 0 8 0 661011093 551165952 61244 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=30971/tid=30979] ppid=30969 vsize=538248 CPUtime=10.91 cores=0-7
/proc/30971/task/30979/stat : 30979 (claspmt) R 30969 30971 30539 0 -1 4202560 6541 0 0 0 1088 3 0 0 25 0 8 0 661011093 551165952 61244 33554432000 4194304 4964800 140736539827376 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 87.64
Current children cumulated vsize (KiB) 538248

[startup+25.5009 s]
/proc/loadavg: 5.03 4.59 5.92 9/382 30980
/proc/meminfo: memFree=20336708/32951132 swapFree=64495988/67111528
[pid=30971] ppid=30969 vsize=542576 CPUtime=189.85 cores=0-7
/proc/30971/stat : 30971 (claspmt) R 30969 30971 30539 0 -1 4202496 84701 0 0 0 18948 37 0 0 25 0 8 0 661010955 555597824 67417 33554432000 4194304 4964800 140736539827376 18446744073709551615 4678036 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/30971/statm: 135644 67417 464 189 0 129993 0
[pid=30971/tid=30973] ppid=30969 vsize=542576 CPUtime=23.72 cores=0-7
/proc/30971/task/30973/stat : 30973 (claspmt) R 30969 30971 30539 0 -1 4202560 5004 0 0 0 2370 2 0 0 25 0 8 0 661011093 555597824 67417 33554432000 4194304 4964800 140736539827376 18446744073709551615 4370352 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30971/tid=30974] ppid=30969 vsize=542576 CPUtime=23.32 cores=0-7
/proc/30971/task/30974/stat : 30974 (claspmt) R 30969 30971 30539 0 -1 4202560 10049 0 0 0 2327 5 0 0 25 0 8 0 661011093 555597824 67417 33554432000 4194304 4964800 140736539827376 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30971/tid=30975] ppid=30969 vsize=542576 CPUtime=23.71 cores=0-7
/proc/30971/task/30975/stat : 30975 (claspmt) R 30969 30971 30539 0 -1 4202560 8730 0 0 0 2367 4 0 0 25 0 8 0 661011093 555597824 67417 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670678 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30971/tid=30976] ppid=30969 vsize=542576 CPUtime=23.21 cores=0-7
/proc/30971/task/30976/stat : 30976 (claspmt) R 30969 30971 30539 0 -1 4202560 6973 0 0 0 2317 4 0 0 25 0 8 0 661011093 555597824 67417 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30971/tid=30977] ppid=30969 vsize=542576 CPUtime=23.69 cores=0-7
/proc/30971/task/30977/stat : 30977 (claspmt) R 30969 30971 30539 0 -1 4202560 6247 0 0 0 2365 4 0 0 25 0 8 0 661011093 555597824 67417 33554432000 4194304 4964800 140736539827376 18446744073709551615 4668998 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30971/tid=30978] ppid=30969 vsize=542576 CPUtime=23.59 cores=0-7
/proc/30971/task/30978/stat : 30978 (claspmt) R 30969 30971 30539 0 -1 4202560 9123 0 0 0 2354 5 0 0 25 0 8 0 661011093 555597824 67417 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=30971/tid=30979] ppid=30969 vsize=542576 CPUtime=23.71 cores=0-7
/proc/30971/task/30979/stat : 30979 (claspmt) R 30969 30971 30539 0 -1 4202560 7402 0 0 0 2366 5 0 0 25 0 8 0 661011093 555597824 67417 33554432000 4194304 4964800 140736539827376 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 189.85
Current children cumulated vsize (KiB) 542576

[startup+51.1006 s]
/proc/loadavg: 6.11 4.88 5.98 9/382 30980
/proc/meminfo: memFree=20298856/32951132 swapFree=64495988/67111528
[pid=30971] ppid=30969 vsize=543640 CPUtime=394.27 cores=0-7
/proc/30971/stat : 30971 (claspmt) R 30969 30971 30539 0 -1 4202496 94183 0 0 0 39374 53 0 0 25 0 8 0 661010955 556687360 76899 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670299 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/30971/statm: 135910 76899 466 189 0 130259 0
[pid=30971/tid=30973] ppid=30969 vsize=543640 CPUtime=49.32 cores=0-7
/proc/30971/task/30973/stat : 30973 (claspmt) R 30969 30971 30539 0 -1 4202560 5526 0 0 0 4929 3 0 0 25 0 8 0 661011093 556687360 76899 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30971/tid=30974] ppid=30969 vsize=543640 CPUtime=48.91 cores=0-7
/proc/30971/task/30974/stat : 30974 (claspmt) R 30969 30971 30539 0 -1 4202560 13047 0 0 0 4880 11 0 0 25 0 8 0 661011093 556687360 76899 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30971/tid=30975] ppid=30969 vsize=543640 CPUtime=49.31 cores=0-7
/proc/30971/task/30975/stat : 30975 (claspmt) R 30969 30971 30539 0 -1 4202560 9846 0 0 0 4925 6 0 0 25 0 8 0 661011093 556687360 76899 33554432000 4194304 4964800 140736539827376 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30971/tid=30976] ppid=30969 vsize=543640 CPUtime=48.48 cores=0-7
/proc/30971/task/30976/stat : 30976 (claspmt) R 30969 30971 30539 0 -1 4202560 8182 0 0 0 4843 5 0 0 25 0 8 0 661011093 556687360 76899 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30971/tid=30977] ppid=30969 vsize=543640 CPUtime=49.29 cores=0-7
/proc/30971/task/30977/stat : 30977 (claspmt) R 30969 30971 30539 0 -1 4202560 6335 0 0 0 4925 4 0 0 25 0 8 0 661011093 556687360 76899 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30971/tid=30978] ppid=30969 vsize=543640 CPUtime=49.16 cores=0-7
/proc/30971/task/30978/stat : 30978 (claspmt) R 30969 30971 30539 0 -1 4202560 10322 0 0 0 4906 10 0 0 25 0 8 0 661011093 556687360 76899 33554432000 4194304 4964800 140736539827376 18446744073709551615 4378499 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=30971/tid=30979] ppid=30969 vsize=543640 CPUtime=49.29 cores=0-7
/proc/30971/task/30979/stat : 30979 (claspmt) R 30969 30971 30539 0 -1 4202560 9392 0 0 0 4923 6 0 0 25 0 8 0 661011093 556687360 76899 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670437 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 394.27
Current children cumulated vsize (KiB) 543640

[startup+102.315 s]
/proc/loadavg: 7.26 5.38 6.09 9/382 30982
/proc/meminfo: memFree=20206344/32951132 swapFree=64495988/67111528
[pid=30971] ppid=30969 vsize=559408 CPUtime=803.27 cores=0-7
/proc/30971/stat : 30971 (claspmt) R 30969 30971 30539 0 -1 4202496 117114 0 0 0 80245 82 0 0 25 0 8 0 661010955 572833792 99830 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670180 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/30971/statm: 139852 99830 466 189 0 134201 0
[pid=30971/tid=30973] ppid=30969 vsize=559408 CPUtime=100.53 cores=0-7
/proc/30971/task/30973/stat : 30973 (claspmt) R 30969 30971 30539 0 -1 4202560 6531 0 0 0 10048 5 0 0 25 0 8 0 661011093 572833792 99830 33554432000 4194304 4964800 140736539827376 18446744073709551615 4370259 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30971/tid=30974] ppid=30969 vsize=559408 CPUtime=100.11 cores=0-7
/proc/30971/task/30974/stat : 30974 (claspmt) R 30969 30971 30539 0 -1 4202560 17829 0 0 0 9993 18 0 0 25 0 8 0 661011093 572833792 99830 33554432000 4194304 4964800 140736539827376 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30971/tid=30975] ppid=30969 vsize=559408 CPUtime=100.52 cores=0-7
/proc/30971/task/30975/stat : 30975 (claspmt) R 30969 30971 30539 0 -1 4202560 11843 0 0 0 10043 9 0 0 25 0 8 0 661011093 572833792 99830 33554432000 4194304 4964800 140736539827376 18446744073709551615 4669280 0 0 4096 18946 0 0 0 -1 4 0 0 0

################
# More data... #
################

/proc/30971/task/30978/stat : 30978 (claspmt) R 30969 30971 30539 0 -1 4202560 36484 0 0 0 462778 59 0 0 25 0 8 0 661011093 2546167808 556271 33554432000 4194304 4964800 140736539827376 18446744073709551615 4361114 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30971/tid=30979] ppid=30969 vsize=2486492 CPUtime=4659.37 cores=0-7
/proc/30971/task/30979/stat : 30979 (claspmt) R 30969 30971 30539 0 -1 4202560 195726 0 0 0 465674 263 0 0 25 0 8 0 661011093 2546167808 556271 33554432000 4194304 4964800 140736539827376 18446744073709551615 210646807585 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 37218.4
Current children cumulated vsize (KiB) 2486492

[startup+4722.31 s]
/proc/loadavg: 8.05 8.03 8.00 9/382 31117
/proc/meminfo: memFree=18333488/32951132 swapFree=64495988/67111528
[pid=30971] ppid=30969 vsize=2555156 CPUtime=37697.6 cores=0-7
/proc/30971/stat : 30971 (claspmt) R 30969 30971 30539 0 -1 4202496 584729 0 0 0 3768959 806 0 0 25 0 8 0 661010955 2616479744 566932 33554432000 4194304 4964800 140736539827376 18446744073709551615 4669476 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/30971/statm: 638789 566932 466 189 0 633138 0
[pid=30971/tid=30973] ppid=30969 vsize=2555156 CPUtime=4711.57 cores=0-7
/proc/30971/task/30973/stat : 30973 (claspmt) R 30969 30971 30539 0 -1 4202560 22201 0 0 0 471126 31 0 0 25 0 8 0 661011093 2616479744 566932 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30971/tid=30974] ppid=30969 vsize=2555156 CPUtime=4716.54 cores=0-7
/proc/30971/task/30974/stat : 30974 (claspmt) R 30969 30971 30539 0 -1 4202560 95328 0 0 0 471519 135 0 0 25 0 8 0 661011093 2616479744 566932 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30971/tid=30975] ppid=30969 vsize=2555156 CPUtime=4719.81 cores=0-7
/proc/30971/task/30975/stat : 30975 (claspmt) R 30969 30971 30539 0 -1 4202560 33104 0 0 0 471938 43 0 0 25 0 8 0 661011093 2616479744 566932 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30971/tid=30976] ppid=30969 vsize=2555156 CPUtime=4712.96 cores=0-7
/proc/30971/task/30976/stat : 30976 (claspmt) R 30969 30971 30539 0 -1 4202560 77987 0 0 0 471176 120 0 0 25 0 8 0 661011093 2616479744 566932 33554432000 4194304 4964800 140736539827376 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30971/tid=30977] ppid=30969 vsize=2555156 CPUtime=4707.56 cores=0-7
/proc/30971/task/30977/stat : 30977 (claspmt) R 30969 30971 30539 0 -1 4202560 62282 0 0 0 470670 86 0 0 25 0 8 0 661011093 2616479744 566932 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=30971/tid=30978] ppid=30969 vsize=2555156 CPUtime=4688.38 cores=0-7
/proc/30971/task/30978/stat : 30978 (claspmt) R 30969 30971 30539 0 -1 4202560 36486 0 0 0 468779 59 0 0 25 0 8 0 661011093 2616479744 566932 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30971/tid=30979] ppid=30969 vsize=2555156 CPUtime=4719.38 cores=0-7
/proc/30971/task/30979/stat : 30979 (claspmt) R 30969 30971 30539 0 -1 4202560 204749 0 0 0 471657 281 0 0 25 0 8 0 661011093 2616479744 566932 33554432000 4194304 4964800 140736539827376 18446744073709551615 4669011 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 37697.6
Current children cumulated vsize (KiB) 2555156

[startup+4782.3 s]
/proc/loadavg: 8.06 8.03 8.00 9/382 31119
/proc/meminfo: memFree=18281288/32951132 swapFree=64495988/67111528
[pid=30971] ppid=30969 vsize=2566700 CPUtime=38176.7 cores=0-7
/proc/30971/stat : 30971 (claspmt) R 30969 30971 30539 0 -1 4202496 598022 0 0 0 3816852 822 0 0 25 0 8 0 661010955 2628300800 580225 33554432000 4194304 4964800 140736539827376 18446744073709551615 4669350 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/30971/statm: 641675 580225 466 189 0 636024 0
[pid=30971/tid=30973] ppid=30969 vsize=2566700 CPUtime=4770.8 cores=0-7
/proc/30971/task/30973/stat : 30973 (claspmt) R 30969 30971 30539 0 -1 4202560 22283 0 0 0 477049 31 0 0 25 0 8 0 661011093 2628300800 580225 33554432000 4194304 4964800 140736539827376 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30971/tid=30974] ppid=30969 vsize=2566700 CPUtime=4776.52 cores=0-7
/proc/30971/task/30974/stat : 30974 (claspmt) R 30969 30971 30539 0 -1 4202560 98709 0 0 0 477514 138 0 0 25 0 8 0 661011093 2628300800 580226 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670357 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30971/tid=30975] ppid=30969 vsize=2566700 CPUtime=4779.8 cores=0-7
/proc/30971/task/30975/stat : 30975 (claspmt) R 30969 30971 30539 0 -1 4202560 33898 0 0 0 477936 44 0 0 25 0 8 0 661011093 2628300800 580226 33554432000 4194304 4964800 140736539827376 18446744073709551615 4668773 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30971/tid=30976] ppid=30969 vsize=2566700 CPUtime=4772.92 cores=0-7
/proc/30971/task/30976/stat : 30976 (claspmt) R 30969 30971 30539 0 -1 4202560 79276 0 0 0 477171 121 0 0 25 0 8 0 661011093 2628300800 580226 33554432000 4194304 4964800 140736539827376 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30971/tid=30977] ppid=30969 vsize=2566700 CPUtime=4767.55 cores=0-7
/proc/30971/task/30977/stat : 30977 (claspmt) R 30969 30971 30539 0 -1 4202560 64888 0 0 0 476665 90 0 0 25 0 8 0 661011093 2628300800 580226 33554432000 4194304 4964800 140736539827376 18446744073709551615 210646807585 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=30971/tid=30978] ppid=30969 vsize=2566700 CPUtime=4748.37 cores=0-7
/proc/30971/task/30978/stat : 30978 (claspmt) R 30969 30971 30539 0 -1 4202560 37015 0 0 0 474778 59 0 0 25 0 8 0 661011093 2628300800 580226 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670264 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30971/tid=30979] ppid=30969 vsize=2566700 CPUtime=4779.36 cores=0-7
/proc/30971/task/30979/stat : 30979 (claspmt) R 30969 30971 30539 0 -1 4202560 208441 0 0 0 477650 286 0 0 25 0 8 0 661011093 2628300800 580226 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 38176.7
Current children cumulated vsize (KiB) 2566700

[startup+4842.3 s]
/proc/loadavg: 8.06 8.04 8.00 9/382 31120
/proc/meminfo: memFree=18221516/32951132 swapFree=64495988/67111528
[pid=30971] ppid=30969 vsize=2583340 CPUtime=38655.9 cores=0-7
/proc/30971/stat : 30971 (claspmt) R 30969 30971 30539 0 -1 4202496 612931 0 0 0 3864742 844 0 0 25 0 8 0 661010955 2645340160 594837 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/30971/statm: 645835 594837 466 189 0 640184 0
[pid=30971/tid=30973] ppid=30969 vsize=2583340 CPUtime=4830 cores=0-7
/proc/30971/task/30973/stat : 30973 (claspmt) R 30969 30971 30539 0 -1 4202560 22908 0 0 0 482968 32 0 0 25 0 8 0 661011093 2645340160 594837 33554432000 4194304 4964800 140736539827376 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30971/tid=30974] ppid=30969 vsize=2583340 CPUtime=4836.5 cores=0-7
/proc/30971/task/30974/stat : 30974 (claspmt) R 30969 30971 30539 0 -1 4202560 100203 0 0 0 483510 140 0 0 25 0 8 0 661011093 2645340160 594837 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30971/tid=30975] ppid=30969 vsize=2583340 CPUtime=4839.77 cores=0-7
/proc/30971/task/30975/stat : 30975 (claspmt) R 30969 30971 30539 0 -1 4202560 34799 0 0 0 483932 45 0 0 25 0 8 0 661011093 2645340160 594837 33554432000 4194304 4964800 140736539827376 18446744073709551615 4363007 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30971/tid=30976] ppid=30969 vsize=2583340 CPUtime=4832.92 cores=0-7
/proc/30971/task/30976/stat : 30976 (claspmt) R 30969 30971 30539 0 -1 4202560 79676 0 0 0 483170 122 0 0 25 0 8 0 661011093 2645340160 594837 33554432000 4194304 4964800 140736539827376 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30971/tid=30977] ppid=30969 vsize=2583340 CPUtime=4827.55 cores=0-7
/proc/30971/task/30977/stat : 30977 (claspmt) R 30969 30971 30539 0 -1 4202560 68415 0 0 0 482657 98 0 0 25 0 8 0 661011093 2645340160 594837 33554432000 4194304 4964800 140736539827376 18446744073709551615 4669538 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=30971/tid=30978] ppid=30969 vsize=2583340 CPUtime=4808.37 cores=0-7
/proc/30971/task/30978/stat : 30978 (claspmt) R 30969 30971 30539 0 -1 4202560 37501 0 0 0 480777 60 0 0 25 0 8 0 661011093 2645340160 594837 33554432000 4194304 4964800 140736539827376 18446744073709551615 4669131 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30971/tid=30979] ppid=30969 vsize=2583340 CPUtime=4839.35 cores=0-7
/proc/30971/task/30979/stat : 30979 (claspmt) R 30969 30971 30539 0 -1 4202560 215819 0 0 0 483642 293 0 0 25 0 8 0 661011093 2645340160 594837 33554432000 4194304 4964800 140736539827376 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 38655.9
Current children cumulated vsize (KiB) 2583340

[startup+4902.3 s]
/proc/loadavg: 8.10 8.06 8.01 9/382 31122
/proc/meminfo: memFree=18206640/32951132 swapFree=64495988/67111528
[pid=30971] ppid=30969 vsize=2583916 CPUtime=39135 cores=0-7
/proc/30971/stat : 30971 (claspmt) R 30969 30971 30539 0 -1 4202496 616694 0 0 0 3912652 851 0 0 25 0 8 0 661010955 2645929984 598561 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670499 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/30971/statm: 645979 598561 466 189 0 640328 0
[pid=30971/tid=30973] ppid=30969 vsize=2583916 CPUtime=4889.24 cores=0-7
/proc/30971/task/30973/stat : 30973 (claspmt) R 30969 30971 30539 0 -1 4202560 22908 0 0 0 488892 32 0 0 25 0 8 0 661011093 2645929984 598561 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30971/tid=30974] ppid=30969 vsize=2583916 CPUtime=4896.49 cores=0-7
/proc/30971/task/30974/stat : 30974 (claspmt) R 30969 30971 30539 0 -1 4202560 100456 0 0 0 489509 140 0 0 25 0 8 0 661011093 2645929984 598561 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30971/tid=30975] ppid=30969 vsize=2583916 CPUtime=4899.75 cores=0-7
/proc/30971/task/30975/stat : 30975 (claspmt) R 30969 30971 30539 0 -1 4202560 34844 0 0 0 489930 45 0 0 25 0 8 0 661011093 2645929984 598561 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30971/tid=30976] ppid=30969 vsize=2583916 CPUtime=4892.92 cores=0-7
/proc/30971/task/30976/stat : 30976 (claspmt) R 30969 30971 30539 0 -1 4202560 79873 0 0 0 489170 122 0 0 25 0 8 0 661011093 2645929984 598561 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670437 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30971/tid=30977] ppid=30969 vsize=2583916 CPUtime=4887.51 cores=0-7
/proc/30971/task/30977/stat : 30977 (claspmt) R 30969 30971 30539 0 -1 4202560 68454 0 0 0 488652 99 0 0 25 0 8 0 661011093 2645929984 598561 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=30971/tid=30978] ppid=30969 vsize=2583916 CPUtime=4868.37 cores=0-7
/proc/30971/task/30978/stat : 30978 (claspmt) R 30969 30971 30539 0 -1 4202560 38186 0 0 0 486776 61 0 0 25 0 8 0 661011093 2645929984 598561 33554432000 4194304 4964800 140736539827376 18446744073709551615 4669523 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30971/tid=30979] ppid=30969 vsize=2583916 CPUtime=4899.36 cores=0-7
/proc/30971/task/30979/stat : 30979 (claspmt) R 30969 30971 30539 0 -1 4202560 218274 0 0 0 489638 298 0 0 25 0 8 0 661011093 2645929984 598561 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670371 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 39135
Current children cumulated vsize (KiB) 2583916

[startup+4962.3 s]
/proc/loadavg: 8.03 8.04 8.00 9/382 31124
/proc/meminfo: memFree=18206512/32951132 swapFree=64495988/67111528
[pid=30971] ppid=30969 vsize=2583916 CPUtime=39614.2 cores=0-7
/proc/30971/stat : 30971 (claspmt) R 30969 30971 30539 0 -1 4202496 616724 0 0 0 3960567 853 0 0 25 0 8 0 661010955 2645929984 598591 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670299 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/30971/statm: 645979 598591 466 189 0 640328 0
[pid=30971/tid=30973] ppid=30969 vsize=2583916 CPUtime=4948.47 cores=0-7
/proc/30971/task/30973/stat : 30973 (claspmt) R 30969 30971 30539 0 -1 4202560 22908 0 0 0 494815 32 0 0 25 0 8 0 661011093 2645929984 598591 33554432000 4194304 4964800 140736539827376 18446744073709551615 4342144 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30971/tid=30974] ppid=30969 vsize=2583916 CPUtime=4956.48 cores=0-7
/proc/30971/task/30974/stat : 30974 (claspmt) R 30969 30971 30539 0 -1 4202560 100460 0 0 0 495508 140 0 0 25 0 8 0 661011093 2645929984 598591 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30971/tid=30975] ppid=30969 vsize=2583916 CPUtime=4959.73 cores=0-7
/proc/30971/task/30975/stat : 30975 (claspmt) R 30969 30971 30539 0 -1 4202560 34844 0 0 0 495928 45 0 0 25 0 8 0 661011093 2645929984 598591 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30971/tid=30976] ppid=30969 vsize=2583916 CPUtime=4952.9 cores=0-7
/proc/30971/task/30976/stat : 30976 (claspmt) R 30969 30971 30539 0 -1 4202560 79898 0 0 0 495168 122 0 0 25 0 8 0 661011093 2645929984 598591 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30971/tid=30977] ppid=30969 vsize=2583916 CPUtime=4947.51 cores=0-7
/proc/30971/task/30977/stat : 30977 (claspmt) R 30969 30971 30539 0 -1 4202560 68454 0 0 0 494652 99 0 0 25 0 8 0 661011093 2645929984 598591 33554432000 4194304 4964800 140736539827376 18446744073709551615 4321020 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=30971/tid=30978] ppid=30969 vsize=2583916 CPUtime=4928.37 cores=0-7
/proc/30971/task/30978/stat : 30978 (claspmt) R 30969 30971 30539 0 -1 4202560 38186 0 0 0 492776 61 0 0 25 0 8 0 661011093 2645929984 598591 33554432000 4194304 4964800 140736539827376 18446744073709551615 4669560 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30971/tid=30979] ppid=30969 vsize=2583916 CPUtime=4959.35 cores=0-7
/proc/30971/task/30979/stat : 30979 (claspmt) R 30969 30971 30539 0 -1 4202560 218274 0 0 0 495637 298 0 0 25 0 8 0 661011093 2645929984 598591 33554432000 4194304 4964800 140736539827376 18446744073709551615 4669376 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 39614.2
Current children cumulated vsize (KiB) 2583916



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5010.7 s]
/proc/loadavg: 8.01 8.04 8.00 9/382 31125
/proc/meminfo: memFree=18208500/32951132 swapFree=64495988/67111528
[pid=30971] ppid=30969 vsize=2583916 CPUtime=40000.8 cores=0-7
/proc/30971/stat : 30971 (claspmt) R 30969 30971 30539 0 -1 4202496 616740 0 0 0 3999221 854 0 0 25 0 8 0 661010955 2645929984 597927 33554432000 4194304 4964800 140736539827376 18446744073709551615 4669350 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/30971/statm: 645979 597927 466 189 0 640328 0
[pid=30971/tid=30973] ppid=30969 vsize=2583916 CPUtime=4996.25 cores=0-7
/proc/30971/task/30973/stat : 30973 (claspmt) R 30969 30971 30539 0 -1 4202560 22908 0 0 0 499593 32 0 0 25 0 8 0 661011093 2645929984 597927 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30971/tid=30974] ppid=30969 vsize=2583916 CPUtime=5004.82 cores=0-7
/proc/30971/task/30974/stat : 30974 (claspmt) R 30969 30971 30539 0 -1 4202560 100461 0 0 0 500342 140 0 0 25 0 8 0 661011093 2645929984 597927 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30971/tid=30975] ppid=30969 vsize=2583916 CPUtime=5008.11 cores=0-7
/proc/30971/task/30975/stat : 30975 (claspmt) R 30969 30971 30539 0 -1 4202560 34848 0 0 0 500766 45 0 0 25 0 8 0 661011093 2645929984 597927 33554432000 4194304 4964800 140736539827376 18446744073709551615 4668987 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30971/tid=30976] ppid=30969 vsize=2583916 CPUtime=5001.32 cores=0-7
/proc/30971/task/30976/stat : 30976 (claspmt) R 30969 30971 30539 0 -1 4202560 79898 0 0 0 500010 122 0 0 25 0 8 0 661011093 2645929984 597927 33554432000 4194304 4964800 140736539827376 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30971/tid=30977] ppid=30969 vsize=2583916 CPUtime=4995.92 cores=0-7
/proc/30971/task/30977/stat : 30977 (claspmt) R 30969 30971 30539 0 -1 4202560 68464 0 0 0 499493 99 0 0 25 0 8 0 661011093 2645929984 597927 33554432000 4194304 4964800 140736539827376 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=30971/tid=30978] ppid=30969 vsize=2583916 CPUtime=4976.78 cores=0-7
/proc/30971/task/30978/stat : 30978 (claspmt) R 30969 30971 30539 0 -1 4202560 38186 0 0 0 497617 61 0 0 25 0 8 0 661011093 2645929984 597927 33554432000 4194304 4964800 140736539827376 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30971/tid=30979] ppid=30969 vsize=2583916 CPUtime=5007.77 cores=0-7
/proc/30971/task/30979/stat : 30979 (claspmt) R 30969 30971 30539 0 -1 4202560 218274 0 0 0 500479 298 0 0 25 0 8 0 661011093 2645929984 597927 33554432000 4194304 4964800 140736539827376 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 40000.8
Current children cumulated vsize (KiB) 2583916

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5011 s]
/proc/loadavg: 8.01 8.04 8.00 9/382 31125
/proc/meminfo: memFree=18208500/32951132 swapFree=64495988/67111528
[pid=30971] ppid=30969 vsize=2547956 CPUtime=40001.1 cores=0-7
/proc/30971/stat : 30971 (claspmt) R 30969 30971 30539 0 -1 4202496 616750 0 0 0 3999255 855 0 0 25 0 1 0 661010955 2609106944 588768 33554432000 4194304 4964800 140736539827376 18446744073709551615 210646803444 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/30971/statm: 636989 588768 476 189 0 631338 0
Current children cumulated CPU time (s) 40001.1
Current children cumulated vsize (KiB) 2547956

[startup+5011.4 s]
/proc/loadavg: 8.01 8.04 8.00 9/382 31125
/proc/meminfo: memFree=18208500/32951132 swapFree=64495988/67111528
[pid=30971] ppid=30969 vsize=2186972 CPUtime=40001.5 cores=0-7
/proc/30971/stat : 30971 (claspmt) R 30969 30971 30539 0 -1 4202496 616750 0 0 0 3999294 856 0 0 25 0 1 0 661010955 2239459328 501091 33554432000 4194304 4964800 140736539827376 18446744073709551615 210647189863 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/30971/statm: 546743 501091 476 189 0 541092 0
Current children cumulated CPU time (s) 40001.5
Current children cumulated vsize (KiB) 2186972

Child status: 0
Real time (s): 5011.49
CPU time (s): 40001.6
CPU user time (s): 39992.9
CPU system time (s): 8.65068
CPU usage (%): 798.198
Max. virtual memory (cumulated for all children) (KiB): 2583916

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39992.9
system time used= 8.65068
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 616751
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 38857
involuntary context switches= 138421

runsolver used 17.3774 second user time and 46.4849 second system time

The end

Launcher Data 

Begin job on node129 at 2011-05-15 23:15:12
IDJOB=3380877
IDBENCH=82468
IDSOLVER=1865
FILE ID=node129/3380877-1305494111
RUNJOBID= node129-1305493483-30553
PBS_JOBID= 13325426
Free space on /tmp= 73384 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT11/application/jarvisalo/HWMCC10-combination-timeframe-expansion/hwmcc10-timeframe-expansion-k45-pdtvisns3p02-tseitin.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3380877-1305494111/watcher-3380877-1305494111 -o /tmp/evaluation-result-3380877-1305494111/solver-3380877-1305494111 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380877-1305494111.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= a0ec1baa890da62d0c6df7f1e62c6af3
RANDOM SEED=256847199

node129.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      20600428 kB
Buffers:        278940 kB
Cached:        6425328 kB
SwapCached:      52664 kB
Active:        3567316 kB
Inactive:      8538416 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      20600428 kB
SwapTotal:    67111528 kB
SwapFree:     64495988 kB
Dirty:           10120 kB
Writeback:           0 kB
AnonPages:     5358016 kB
Mapped:          20300 kB
Slab:           135140 kB
PageTables:      40336 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS: 93361856 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73384 MiB
End job on node129 at 2011-05-16 00:38:46