Trace number 3321694

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095? (TO) 40001.1 5007.44

General information on the benchmark

Nameapplication/jarvisalo/HWMCC10-combination-timeframe-expansion/
hwmcc10-timeframe-expansion-k50-pdtvisns3p00-tseitin.cnf
MD5SUMa8ad0173824ed9404a8c153da07c5f9c
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark891.683
Satisfiable
(Un)Satisfiability was proved
Number of variables183325
Number of clauses546914
Sum of the clauses size1276130
Maximum clause length3
Minimum clause length1
Number of clauses of size 12
Number of clauses of size 2364608
Number of clauses of size 3182304
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...366474/instance-3321694-1304366474.cnf
1.49/1.56	c Solving...
40000.19/5006.61	
40000.19/5006.61	*** INTERRUPTED! ***
40000.19/5006.61	*** Shutting down threads...
40001.04/5007.43	s UNKNOWN
40001.04/5007.43	
40001.04/5007.43	c INTERRUPTED   : 1
40001.04/5007.43	c Models        : 0+    
40001.04/5007.43	c Time          : 5006.618s (Solving: 5005.06s 1st Model: 0.00s Unsat: 0.00s)
40001.04/5007.43	c CPU Time      : 40000.260s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3321694-1304366474/watcher-3321694-1304366474 -o /tmp/evaluation-result-3321694-1304366474/solver-3321694-1304366474 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3321694-1304366474.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.15 0.03 0.06 2/221 11279
/proc/meminfo: memFree=20782808/32950928 swapFree=66698992/67111528
[pid=11279] ppid=11277 vsize=44152 CPUtime=0 cores=0-7
/proc/11279/stat : 11279 (claspmt) R 11277 11279 11185 0 -1 4202496 1729 0 0 0 0 0 0 0 22 0 1 0 548247222 45211648 1658 33554432000 4194304 4964704 140733769205248 18446744073709551615 4700332 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/11279/statm: 11038 1666 376 189 0 5912 0

[startup+0.0898049 s]
/proc/loadavg: 0.15 0.03 0.06 2/221 11279
/proc/meminfo: memFree=20782808/32950928 swapFree=66698992/67111528
[pid=11279] ppid=11277 vsize=67164 CPUtime=0.08 cores=0-7
/proc/11279/stat : 11279 (claspmt) R 11277 11279 11185 0 -1 4202496 14291 0 0 0 7 1 0 0 22 0 1 0 548247222 68775936 11638 33554432000 4194304 4964704 140733769205248 18446744073709551615 4599599 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/11279/statm: 16791 11639 386 189 0 11665 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 67164

[startup+0.100804 s]
/proc/loadavg: 0.15 0.03 0.06 2/221 11279
/proc/meminfo: memFree=20782808/32950928 swapFree=66698992/67111528
[pid=11279] ppid=11277 vsize=68748 CPUtime=0.09 cores=0-7
/proc/11279/stat : 11279 (claspmt) R 11277 11279 11185 0 -1 4202496 14753 0 0 0 8 1 0 0 22 0 1 0 548247222 70397952 12100 33554432000 4194304 4964704 140733769205248 18446744073709551615 248089359045 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/11279/statm: 17187 12102 386 189 0 12061 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 68748

[startup+0.300769 s]
/proc/loadavg: 0.15 0.03 0.06 2/221 11279
/proc/meminfo: memFree=20782808/32950928 swapFree=66698992/67111528
[pid=11279] ppid=11277 vsize=109304 CPUtime=0.29 cores=0-7
/proc/11279/stat : 11279 (claspmt) R 11277 11279 11185 0 -1 4202496 27675 0 0 0 26 3 0 0 23 0 1 0 548247222 111927296 21667 33554432000 4194304 4964704 140733769205248 18446744073709551615 4615965 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/11279/statm: 27326 21668 399 189 0 22200 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 109304

[startup+0.700699 s]
/proc/loadavg: 0.15 0.03 0.06 2/221 11279
/proc/meminfo: memFree=20782808/32950928 swapFree=66698992/67111528
[pid=11279] ppid=11277 vsize=120000 CPUtime=0.69 cores=0-7
/proc/11279/stat : 11279 (claspmt) R 11277 11279 11185 0 -1 4202496 30544 0 0 0 66 3 0 0 25 0 1 0 548247222 122880000 24536 33554432000 4194304 4964704 140733769205248 18446744073709551615 4612561 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/11279/statm: 30000 24537 404 189 0 24874 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 120000

[startup+1.50057 s]
/proc/loadavg: 0.15 0.03 0.06 2/222 11280
/proc/meminfo: memFree=20680508/32950928 swapFree=66698992/67111528
[pid=11279] ppid=11277 vsize=129432 CPUtime=1.49 cores=0-7
/proc/11279/stat : 11279 (claspmt) R 11277 11279 11185 0 -1 4202496 32913 0 0 0 145 4 0 0 25 0 1 0 548247222 132538368 26905 33554432000 4194304 4964704 140733769205248 18446744073709551615 248089359045 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/11279/statm: 32358 26905 406 189 0 27232 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 129432

[startup+3.10128 s]
/proc/loadavg: 0.15 0.03 0.06 9/229 11287
/proc/meminfo: memFree=20536612/32950928 swapFree=66698992/67111528
[pid=11279] ppid=11277 vsize=675792 CPUtime=10.63 cores=0-7
/proc/11279/stat : 11279 (claspmt) R 11277 11279 11185 0 -1 4202496 82324 0 0 0 1042 21 0 0 25 0 8 0 548247222 692011008 62675 33554432000 4194304 4964704 140733769205248 18446744073709551615 4668635 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11279/statm: 168948 62675 464 189 0 163297 0
[pid=11279/tid=11281] ppid=11277 vsize=675792 CPUtime=1.09 cores=0-7
/proc/11279/task/11281/stat : 11281 (claspmt) R 11277 11279 11185 0 -1 4202560 5667 0 0 0 108 1 0 0 16 0 8 0 548247379 692011008 62675 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11279/tid=11282] ppid=11277 vsize=675792 CPUtime=1.08 cores=0-7
/proc/11279/task/11282/stat : 11282 (claspmt) R 11277 11279 11185 0 -1 4202560 7608 0 0 0 106 2 0 0 16 0 8 0 548247379 692011008 62675 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669365 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11279/tid=11283] ppid=11277 vsize=675792 CPUtime=1.08 cores=0-7
/proc/11279/task/11283/stat : 11283 (claspmt) R 11277 11279 11185 0 -1 4202560 7864 0 0 0 106 2 0 0 16 0 8 0 548247379 692011008 62675 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670100 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11279/tid=11284] ppid=11277 vsize=675792 CPUtime=1.08 cores=0-7
/proc/11279/task/11284/stat : 11284 (claspmt) R 11277 11279 11185 0 -1 4202560 6125 0 0 0 107 1 0 0 16 0 8 0 548247379 692011008 62675 33554432000 4194304 4964704 140733769205248 18446744073709551615 4370264 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=11279/tid=11285] ppid=11277 vsize=675792 CPUtime=1.46 cores=0-7
/proc/11279/task/11285/stat : 11285 (claspmt) R 11277 11279 11185 0 -1 4202560 6456 0 0 0 144 2 0 0 19 0 8 0 548247379 692011008 62675 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669856 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=11279/tid=11286] ppid=11277 vsize=675792 CPUtime=1.09 cores=0-7
/proc/11279/task/11286/stat : 11286 (claspmt) R 11277 11279 11185 0 -1 4202560 7638 0 0 0 107 2 0 0 17 0 8 0 548247379 692011008 62675 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670100 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11279/tid=11287] ppid=11277 vsize=675792 CPUtime=1.1 cores=0-7
/proc/11279/task/11287/stat : 11287 (claspmt) R 11277 11279 11185 0 -1 4202560 7014 0 0 0 108 2 0 0 16 0 8 0 548247379 692011008 62675 33554432000 4194304 4964704 140733769205248 18446744073709551615 4321082 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 10.63
Current children cumulated vsize (KiB) 675792

[startup+6.30071 s]
/proc/loadavg: 0.94 0.20 0.11 9/229 11287
/proc/meminfo: memFree=20528564/32950928 swapFree=66698992/67111528
[pid=11279] ppid=11277 vsize=675952 CPUtime=36.2 cores=0-7
/proc/11279/stat : 11279 (claspmt) R 11277 11279 11185 0 -1 4202496 84215 0 0 0 3597 23 0 0 25 0 8 0 548247222 692174848 64566 33554432000 4194304 4964704 140733769205248 18446744073709551615 4370264 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11279/statm: 168988 64566 464 189 0 163337 0
[pid=11279/tid=11281] ppid=11277 vsize=675952 CPUtime=4.29 cores=0-7
/proc/11279/task/11281/stat : 11281 (claspmt) R 11277 11279 11185 0 -1 4202560 5829 0 0 0 428 1 0 0 22 0 8 0 548247379 692174848 64566 33554432000 4194304 4964704 140733769205248 18446744073709551615 4370273 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11279/tid=11282] ppid=11277 vsize=675952 CPUtime=4.29 cores=0-7
/proc/11279/task/11282/stat : 11282 (claspmt) R 11277 11279 11185 0 -1 4202560 7977 0 0 0 426 3 0 0 22 0 8 0 548247379 692174848 64566 33554432000 4194304 4964704 140733769205248 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11279/tid=11283] ppid=11277 vsize=675952 CPUtime=4.28 cores=0-7
/proc/11279/task/11283/stat : 11283 (claspmt) R 11277 11279 11185 0 -1 4202560 8096 0 0 0 426 2 0 0 22 0 8 0 548247379 692174848 64566 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670100 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11279/tid=11284] ppid=11277 vsize=675952 CPUtime=4.29 cores=0-7
/proc/11279/task/11284/stat : 11284 (claspmt) R 11277 11279 11185 0 -1 4202560 6309 0 0 0 427 2 0 0 22 0 8 0 548247379 692174848 64566 33554432000 4194304 4964704 140733769205248 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=11279/tid=11285] ppid=11277 vsize=675952 CPUtime=4.66 cores=0-7
/proc/11279/task/11285/stat : 11285 (claspmt) R 11277 11279 11185 0 -1 4202560 6505 0 0 0 464 2 0 0 25 0 8 0 548247379 692174848 64566 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670100 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=11279/tid=11286] ppid=11277 vsize=675952 CPUtime=4.29 cores=0-7
/proc/11279/task/11286/stat : 11286 (claspmt) R 11277 11279 11185 0 -1 4202560 8097 0 0 0 427 2 0 0 22 0 8 0 548247379 692174848 64566 33554432000 4194304 4964704 140733769205248 18446744073709551615 4668683 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11279/tid=11287] ppid=11277 vsize=675952 CPUtime=4.31 cores=0-7
/proc/11279/task/11287/stat : 11287 (claspmt) R 11277 11279 11185 0 -1 4202560 7314 0 0 0 428 3 0 0 22 0 8 0 548247379 692174848 64566 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 36.2
Current children cumulated vsize (KiB) 675952

[startup+12.7006 s]
/proc/loadavg: 1.51 0.33 0.16 9/229 11287
/proc/meminfo: memFree=20515828/32950928 swapFree=66698992/67111528
[pid=11279] ppid=11277 vsize=676672 CPUtime=87.35 cores=0-7
/proc/11279/stat : 11279 (claspmt) R 11277 11279 11185 0 -1 4202496 87441 0 0 0 8708 27 0 0 25 0 8 0 548247222 692912128 67792 33554432000 4194304 4964704 140733769205248 18446744073709551615 4342180 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11279/statm: 169168 67792 464 189 0 163517 0
[pid=11279/tid=11281] ppid=11277 vsize=676672 CPUtime=10.69 cores=0-7
/proc/11279/task/11281/stat : 11281 (claspmt) R 11277 11279 11185 0 -1 4202560 6054 0 0 0 1067 2 0 0 25 0 8 0 548247379 692912128 67792 33554432000 4194304 4964704 140733769205248 18446744073709551615 4668907 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11279/tid=11282] ppid=11277 vsize=676672 CPUtime=10.68 cores=0-7
/proc/11279/task/11282/stat : 11282 (claspmt) R 11277 11279 11185 0 -1 4202560 8603 0 0 0 1065 3 0 0 25 0 8 0 548247379 692912128 67792 33554432000 4194304 4964704 140733769205248 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11279/tid=11283] ppid=11277 vsize=676672 CPUtime=10.68 cores=0-7
/proc/11279/task/11283/stat : 11283 (claspmt) R 11277 11279 11185 0 -1 4202560 8707 0 0 0 1065 3 0 0 25 0 8 0 548247379 692912128 67792 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670233 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11279/tid=11284] ppid=11277 vsize=676672 CPUtime=10.68 cores=0-7
/proc/11279/task/11284/stat : 11284 (claspmt) R 11277 11279 11185 0 -1 4202560 6786 0 0 0 1066 2 0 0 25 0 8 0 548247379 692912128 67792 33554432000 4194304 4964704 140733769205248 18446744073709551615 4668654 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=11279/tid=11285] ppid=11277 vsize=676672 CPUtime=11.06 cores=0-7
/proc/11279/task/11285/stat : 11285 (claspmt) R 11277 11279 11185 0 -1 4202560 6551 0 0 0 1104 2 0 0 25 0 8 0 548247379 692912128 67792 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=11279/tid=11286] ppid=11277 vsize=676672 CPUtime=10.69 cores=0-7
/proc/11279/task/11286/stat : 11286 (claspmt) R 11277 11279 11185 0 -1 4202560 8708 0 0 0 1066 3 0 0 25 0 8 0 548247379 692912128 67792 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670611 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11279/tid=11287] ppid=11277 vsize=676672 CPUtime=10.71 cores=0-7
/proc/11279/task/11287/stat : 11287 (claspmt) R 11277 11279 11185 0 -1 4202560 7754 0 0 0 1067 4 0 0 25 0 8 0 548247379 692912128 67792 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 87.35
Current children cumulated vsize (KiB) 676672

[startup+25.5013 s]
/proc/loadavg: 2.94 0.70 0.28 9/229 11288
/proc/meminfo: memFree=20489524/32950928 swapFree=66698992/67111528
[pid=11279] ppid=11277 vsize=678388 CPUtime=189.66 cores=0-7
/proc/11279/stat : 11279 (claspmt) R 11277 11279 11185 0 -1 4202496 94179 0 0 0 18931 35 0 0 25 0 8 0 548247222 694669312 74530 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670219 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11279/statm: 169597 74530 464 189 0 163946 0
[pid=11279/tid=11281] ppid=11277 vsize=678388 CPUtime=23.49 cores=0-7
/proc/11279/task/11281/stat : 11281 (claspmt) R 11277 11279 11185 0 -1 4202560 6331 0 0 0 2347 2 0 0 25 0 8 0 548247379 694669312 74530 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11279/tid=11282] ppid=11277 vsize=678388 CPUtime=23.48 cores=0-7
/proc/11279/task/11282/stat : 11282 (claspmt) R 11277 11279 11185 0 -1 4202560 10218 0 0 0 2343 5 0 0 25 0 8 0 548247379 694669312 74530 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669051 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11279/tid=11283] ppid=11277 vsize=678388 CPUtime=23.48 cores=0-7
/proc/11279/task/11283/stat : 11283 (claspmt) R 11277 11279 11185 0 -1 4202560 10149 0 0 0 2344 4 0 0 25 0 8 0 548247379 694669312 74530 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11279/tid=11284] ppid=11277 vsize=678388 CPUtime=23.49 cores=0-7
/proc/11279/task/11284/stat : 11284 (claspmt) R 11277 11279 11185 0 -1 4202560 7857 0 0 0 2345 4 0 0 25 0 8 0 548247379 694669312 74530 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=11279/tid=11285] ppid=11277 vsize=678388 CPUtime=23.86 cores=0-7
/proc/11279/task/11285/stat : 11285 (claspmt) R 11277 11279 11185 0 -1 4202560 6625 0 0 0 2384 2 0 0 25 0 8 0 548247379 694669312 74530 33554432000 4194304 4964704 140733769205248 18446744073709551615 4668668 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=11279/tid=11286] ppid=11277 vsize=678388 CPUtime=23.49 cores=0-7
/proc/11279/task/11286/stat : 11286 (claspmt) R 11277 11279 11185 0 -1 4202560 9667 0 0 0 2345 4 0 0 25 0 8 0 548247379 694669312 74530 33554432000 4194304 4964704 140733769205248 18446744073709551615 4668654 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11279/tid=11287] ppid=11277 vsize=678388 CPUtime=23.51 cores=0-7
/proc/11279/task/11287/stat : 11287 (claspmt) R 11277 11279 11185 0 -1 4202560 8813 0 0 0 2346 5 0 0 25 0 8 0 548247379 694669312 74530 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669504 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 189.66
Current children cumulated vsize (KiB) 678388

[startup+51.1008 s]
/proc/loadavg: 4.67 1.29 0.48 9/229 11288
/proc/meminfo: memFree=20453152/32950928 swapFree=66698992/67111528
[pid=11279] ppid=11277 vsize=684760 CPUtime=394.27 cores=0-7
/proc/11279/stat : 11279 (claspmt) R 11277 11279 11185 0 -1 4202496 103287 0 0 0 39380 47 0 0 25 0 8 0 548247222 701194240 83638 33554432000 4194304 4964704 140733769205248 18446744073709551615 4668488 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11279/statm: 171190 83638 466 189 0 165539 0
[pid=11279/tid=11281] ppid=11277 vsize=684760 CPUtime=49.09 cores=0-7
/proc/11279/task/11281/stat : 11281 (claspmt) R 11277 11279 11185 0 -1 4202560 6586 0 0 0 4907 2 0 0 25 0 8 0 548247379 701194240 83638 33554432000 4194304 4964704 140733769205248 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11279/tid=11282] ppid=11277 vsize=684760 CPUtime=49.08 cores=0-7
/proc/11279/task/11282/stat : 11282 (claspmt) R 11277 11279 11185 0 -1 4202560 12577 0 0 0 4900 8 0 0 25 0 8 0 548247379 701194240 83638 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670598 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11279/tid=11283] ppid=11277 vsize=684760 CPUtime=49.07 cores=0-7
/proc/11279/task/11283/stat : 11283 (claspmt) R 11277 11279 11185 0 -1 4202560 11749 0 0 0 4901 6 0 0 25 0 8 0 548247379 701194240 83638 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669477 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11279/tid=11284] ppid=11277 vsize=684760 CPUtime=49.09 cores=0-7
/proc/11279/task/11284/stat : 11284 (claspmt) R 11277 11279 11185 0 -1 4202560 9391 0 0 0 4903 6 0 0 25 0 8 0 548247379 701194240 83638 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=11279/tid=11285] ppid=11277 vsize=684760 CPUtime=49.47 cores=0-7
/proc/11279/task/11285/stat : 11285 (claspmt) R 11277 11279 11185 0 -1 4202560 6720 0 0 0 4944 3 0 0 25 0 8 0 548247379 701194240 83638 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=11279/tid=11286] ppid=11277 vsize=684760 CPUtime=49.09 cores=0-7
/proc/11279/task/11286/stat : 11286 (claspmt) R 11277 11279 11185 0 -1 4202560 11400 0 0 0 4902 7 0 0 25 0 8 0 548247379 701194240 83638 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11279/tid=11287] ppid=11277 vsize=684760 CPUtime=49.1 cores=0-7
/proc/11279/task/11287/stat : 11287 (claspmt) R 11277 11279 11185 0 -1 4202560 10232 0 0 0 4904 6 0 0 25 0 8 0 548247379 701194240 83638 33554432000 4194304 4964704 140733769205248 18446744073709551615 4668609 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 394.27
Current children cumulated vsize (KiB) 684760

[startup+102.309 s]
/proc/loadavg: 6.55 2.32 0.88 9/229 11290
/proc/meminfo: memFree=20389132/32950928 swapFree=66698992/67111528
[pid=11279] ppid=11277 vsize=688852 CPUtime=803.55 cores=0-7
/proc/11279/stat : 11279 (claspmt) R 11277 11279 11185 0 -1 4202496 119014 0 0 0 80287 68 0 0 25 0 8 0 548247222 705384448 99365 33554432000 4194304 4964704 140733769205248 18446744073709551615 4668632 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11279/statm: 172213 99365 466 189 0 166562 0
[pid=11279/tid=11281] ppid=11277 vsize=688852 CPUtime=100.3 cores=0-7
/proc/11279/task/11281/stat : 11281 (claspmt) R 11277 11279 11185 0 -1 4202560 6967 0 0 0 10027 3 0 0 25 0 8 0 548247379 705384448 99365 33554432000 4194304 4964704 140733769205248 18446744073709551615 4668632 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11279/tid=11282] ppid=11277 vsize=688852 CPUtime=100.3 cores=0-7
/proc/11279/task/11282/stat : 11282 (claspmt) R 11277 11279 11185 0 -1 4202560 16436 0 0 0 10016 14 0 0 25 0 8 0 548247379 705384448 99365 33554432000 4194304 4964704 140733769205248 18446744073709551615 4668632 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11279/tid=11283] ppid=11277 vsize=688852 CPUtime=100.28 cores=0-7
/proc/11279/task/11283/stat : 11283 (claspmt) R 11277 11279 11185 0 -1 4202560 13110 0 0 0 10020 8 0 0 25 0 8 0 548247379 705384448 99365 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 6 0 0 0

################
# More data... #
################

/proc/11279/task/11286/stat : 11286 (claspmt) R 11277 11279 11185 0 -1 4202560 40513 0 0 0 465889 46 0 0 25 0 8 0 548247379 2842247168 629083 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670252 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11279/tid=11287] ppid=11277 vsize=2775632 CPUtime=4659.21 cores=0-7
/proc/11279/task/11287/stat : 11287 (claspmt) R 11277 11279 11185 0 -1 4202560 232308 0 0 0 465618 303 0 0 25 0 8 0 548247379 2842247168 629083 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670357 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 37248.3
Current children cumulated vsize (KiB) 2775632

[startup+4722.31 s]
/proc/loadavg: 8.06 8.06 7.97 9/229 11428
/proc/meminfo: memFree=18262048/32950928 swapFree=66698992/67111528
[pid=11279] ppid=11277 vsize=2775632 CPUtime=37727.9 cores=0-7
/proc/11279/stat : 11279 (claspmt) R 11277 11279 11185 0 -1 4202496 650664 0 0 0 3771976 817 0 0 25 0 8 0 548247222 2842247168 629199 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/11279/statm: 693908 629199 466 189 0 688257 0
[pid=11279/tid=11281] ppid=11277 vsize=2775632 CPUtime=4709.16 cores=0-7
/proc/11279/task/11281/stat : 11281 (claspmt) R 11277 11279 11185 0 -1 4202560 23065 0 0 0 470889 27 0 0 25 0 8 0 548247379 2842247168 629199 33554432000 4194304 4964704 140733769205248 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=11279/tid=11282] ppid=11277 vsize=2775632 CPUtime=4719.79 cores=0-7
/proc/11279/task/11282/stat : 11282 (claspmt) R 11277 11279 11185 0 -1 4202560 71075 0 0 0 471894 85 0 0 25 0 8 0 548247379 2842247168 629199 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11279/tid=11283] ppid=11277 vsize=2775632 CPUtime=4719.63 cores=0-7
/proc/11279/task/11283/stat : 11283 (claspmt) R 11277 11279 11185 0 -1 4202560 55230 0 0 0 471891 72 0 0 25 0 8 0 548247379 2842247168 629199 33554432000 4194304 4964704 140733769205248 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11279/tid=11284] ppid=11277 vsize=2775632 CPUtime=4720.36 cores=0-7
/proc/11279/task/11284/stat : 11284 (claspmt) R 11277 11279 11185 0 -1 4202560 75683 0 0 0 471923 113 0 0 25 0 8 0 548247379 2842247168 629199 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=11279/tid=11285] ppid=11277 vsize=2775632 CPUtime=4720.7 cores=0-7
/proc/11279/task/11285/stat : 11285 (claspmt) R 11277 11279 11185 0 -1 4202560 92749 0 0 0 471948 122 0 0 25 0 8 0 548247379 2842247168 629199 33554432000 4194304 4964704 140733769205248 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=11279/tid=11286] ppid=11277 vsize=2775632 CPUtime=4719.36 cores=0-7
/proc/11279/task/11286/stat : 11286 (claspmt) R 11277 11279 11185 0 -1 4202560 40513 0 0 0 471890 46 0 0 25 0 8 0 548247379 2842247168 629199 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11279/tid=11287] ppid=11277 vsize=2775632 CPUtime=4719.22 cores=0-7
/proc/11279/task/11287/stat : 11287 (claspmt) R 11277 11279 11185 0 -1 4202560 232308 0 0 0 471619 303 0 0 25 0 8 0 548247379 2842247168 629199 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669429 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 37727.9
Current children cumulated vsize (KiB) 2775632

[startup+4782.3 s]
/proc/loadavg: 8.06 8.06 7.98 9/229 11429
/proc/meminfo: memFree=18187232/32950928 swapFree=66698992/67111528
[pid=11279] ppid=11277 vsize=2841168 CPUtime=38207.4 cores=0-7
/proc/11279/stat : 11279 (claspmt) R 11277 11279 11185 0 -1 4202496 669322 0 0 0 3819894 849 0 0 25 0 8 0 548247222 2909356032 647857 33554432000 4194304 4964704 140733769205248 18446744073709551615 4668488 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/11279/statm: 710292 647857 466 189 0 704641 0
[pid=11279/tid=11281] ppid=11277 vsize=2841168 CPUtime=4768.72 cores=0-7
/proc/11279/task/11281/stat : 11281 (claspmt) R 11277 11279 11185 0 -1 4202560 23109 0 0 0 476845 27 0 0 25 0 8 0 548247379 2909356032 647857 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=11279/tid=11282] ppid=11277 vsize=2841168 CPUtime=4779.78 cores=0-7
/proc/11279/task/11282/stat : 11282 (claspmt) R 11277 11279 11185 0 -1 4202560 71075 0 0 0 477893 85 0 0 25 0 8 0 548247379 2909356032 647857 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11279/tid=11283] ppid=11277 vsize=2841168 CPUtime=4779.62 cores=0-7
/proc/11279/task/11283/stat : 11283 (claspmt) R 11277 11279 11185 0 -1 4202560 55230 0 0 0 477890 72 0 0 25 0 8 0 548247379 2909356032 647857 33554432000 4194304 4964704 140733769205248 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11279/tid=11284] ppid=11277 vsize=2841168 CPUtime=4780.35 cores=0-7
/proc/11279/task/11284/stat : 11284 (claspmt) R 11277 11279 11185 0 -1 4202560 75683 0 0 0 477922 113 0 0 25 0 8 0 548247379 2909356032 647857 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=11279/tid=11285] ppid=11277 vsize=2841168 CPUtime=4780.69 cores=0-7
/proc/11279/task/11285/stat : 11285 (claspmt) R 11277 11279 11185 0 -1 4202560 92816 0 0 0 477947 122 0 0 25 0 8 0 548247379 2909356032 647857 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669272 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=11279/tid=11286] ppid=11277 vsize=2841168 CPUtime=4779.35 cores=0-7
/proc/11279/task/11286/stat : 11286 (claspmt) R 11277 11279 11185 0 -1 4202560 40513 0 0 0 477889 46 0 0 25 0 8 0 548247379 2909356032 647857 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669333 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11279/tid=11287] ppid=11277 vsize=2841168 CPUtime=4779.21 cores=0-7
/proc/11279/task/11287/stat : 11287 (claspmt) R 11277 11279 11185 0 -1 4202560 250855 0 0 0 477586 335 0 0 25 0 8 0 548247379 2909356032 647857 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669399 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38207.4
Current children cumulated vsize (KiB) 2841168

[startup+4842.3 s]
/proc/loadavg: 8.13 8.08 7.99 9/229 11431
/proc/meminfo: memFree=18186056/32950928 swapFree=66698992/67111528
[pid=11279] ppid=11277 vsize=2841168 CPUtime=38687 cores=0-7
/proc/11279/stat : 11279 (claspmt) R 11277 11279 11185 0 -1 4202496 669632 0 0 0 3867846 851 0 0 25 0 8 0 548247222 2909356032 648167 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/11279/statm: 710292 648167 466 189 0 704641 0
[pid=11279/tid=11281] ppid=11277 vsize=2841168 CPUtime=4828.26 cores=0-7
/proc/11279/task/11281/stat : 11281 (claspmt) R 11277 11279 11185 0 -1 4202560 23148 0 0 0 482798 28 0 0 25 0 8 0 548247379 2909356032 648167 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=11279/tid=11282] ppid=11277 vsize=2841168 CPUtime=4839.78 cores=0-7
/proc/11279/task/11282/stat : 11282 (claspmt) R 11277 11279 11185 0 -1 4202560 71300 0 0 0 483893 85 0 0 25 0 8 0 548247379 2909356032 648167 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11279/tid=11283] ppid=11277 vsize=2841168 CPUtime=4839.61 cores=0-7
/proc/11279/task/11283/stat : 11283 (claspmt) R 11277 11279 11185 0 -1 4202560 55231 0 0 0 483889 72 0 0 25 0 8 0 548247379 2909356032 648167 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670269 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11279/tid=11284] ppid=11277 vsize=2841168 CPUtime=4840.36 cores=0-7
/proc/11279/task/11284/stat : 11284 (claspmt) R 11277 11279 11185 0 -1 4202560 75683 0 0 0 483923 113 0 0 25 0 8 0 548247379 2909356032 648167 33554432000 4194304 4964704 140733769205248 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=11279/tid=11285] ppid=11277 vsize=2841168 CPUtime=4840.69 cores=0-7
/proc/11279/task/11285/stat : 11285 (claspmt) R 11277 11279 11185 0 -1 4202560 92858 0 0 0 483947 122 0 0 25 0 8 0 548247379 2909356032 648167 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=11279/tid=11286] ppid=11277 vsize=2841168 CPUtime=4839.35 cores=0-7
/proc/11279/task/11286/stat : 11286 (claspmt) R 11277 11279 11185 0 -1 4202560 40514 0 0 0 483889 46 0 0 25 0 8 0 548247379 2909356032 648167 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11279/tid=11287] ppid=11277 vsize=2841168 CPUtime=4839.21 cores=0-7
/proc/11279/task/11287/stat : 11287 (claspmt) R 11277 11279 11185 0 -1 4202560 250857 0 0 0 483586 335 0 0 25 0 8 0 548247379 2909356032 648167 33554432000 4194304 4964704 140733769205248 18446744073709551615 4342180 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38687
Current children cumulated vsize (KiB) 2841168

[startup+4902.3 s]
/proc/loadavg: 8.12 8.08 8.00 9/229 11433
/proc/meminfo: memFree=18183776/32950928 swapFree=66698992/67111528
[pid=11279] ppid=11277 vsize=2841168 CPUtime=39166.5 cores=0-7
/proc/11279/stat : 11279 (claspmt) R 11277 11279 11185 0 -1 4202496 670151 0 0 0 3915803 851 0 0 25 0 8 0 548247222 2909356032 648686 33554432000 4194304 4964704 140733769205248 18446744073709551615 4668635 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/11279/statm: 710292 648686 466 189 0 704641 0
[pid=11279/tid=11281] ppid=11277 vsize=2841168 CPUtime=4887.83 cores=0-7
/proc/11279/task/11281/stat : 11281 (claspmt) R 11277 11279 11185 0 -1 4202560 23148 0 0 0 488755 28 0 0 25 0 8 0 548247379 2909356032 648686 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669296 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=11279/tid=11282] ppid=11277 vsize=2841168 CPUtime=4899.77 cores=0-7
/proc/11279/task/11282/stat : 11282 (claspmt) R 11277 11279 11185 0 -1 4202560 71588 0 0 0 489892 85 0 0 25 0 8 0 548247379 2909356032 648686 33554432000 4194304 4964704 140733769205248 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11279/tid=11283] ppid=11277 vsize=2841168 CPUtime=4899.61 cores=0-7
/proc/11279/task/11283/stat : 11283 (claspmt) R 11277 11279 11185 0 -1 4202560 55231 0 0 0 489889 72 0 0 25 0 8 0 548247379 2909356032 648686 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11279/tid=11284] ppid=11277 vsize=2841168 CPUtime=4900.36 cores=0-7
/proc/11279/task/11284/stat : 11284 (claspmt) R 11277 11279 11185 0 -1 4202560 75683 0 0 0 489923 113 0 0 25 0 8 0 548247379 2909356032 648686 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=11279/tid=11285] ppid=11277 vsize=2841168 CPUtime=4900.69 cores=0-7
/proc/11279/task/11285/stat : 11285 (claspmt) R 11277 11279 11185 0 -1 4202560 93089 0 0 0 489947 122 0 0 25 0 8 0 548247379 2909356032 648686 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669477 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=11279/tid=11286] ppid=11277 vsize=2841168 CPUtime=4899.35 cores=0-7
/proc/11279/task/11286/stat : 11286 (claspmt) R 11277 11279 11185 0 -1 4202560 40514 0 0 0 489889 46 0 0 25 0 8 0 548247379 2909356032 648686 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669396 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11279/tid=11287] ppid=11277 vsize=2841168 CPUtime=4899.21 cores=0-7
/proc/11279/task/11287/stat : 11287 (claspmt) R 11277 11279 11185 0 -1 4202560 250857 0 0 0 489586 335 0 0 25 0 8 0 548247379 2909356032 648686 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669309 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 39166.5
Current children cumulated vsize (KiB) 2841168

[startup+4962.3 s]
/proc/loadavg: 8.11 8.07 8.00 9/229 11434
/proc/meminfo: memFree=18183224/32950928 swapFree=66698992/67111528
[pid=11279] ppid=11277 vsize=2841168 CPUtime=39646.1 cores=0-7
/proc/11279/stat : 11279 (claspmt) R 11277 11279 11185 0 -1 4202496 670352 0 0 0 3963755 852 0 0 25 0 8 0 548247222 2909356032 648887 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669270 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/11279/statm: 710292 648887 466 189 0 704641 0
[pid=11279/tid=11281] ppid=11277 vsize=2841168 CPUtime=4947.39 cores=0-7
/proc/11279/task/11281/stat : 11281 (claspmt) R 11277 11279 11185 0 -1 4202560 23148 0 0 0 494711 28 0 0 25 0 8 0 548247379 2909356032 648887 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669429 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=11279/tid=11282] ppid=11277 vsize=2841168 CPUtime=4959.78 cores=0-7
/proc/11279/task/11282/stat : 11282 (claspmt) R 11277 11279 11185 0 -1 4202560 71592 0 0 0 495893 85 0 0 25 0 8 0 548247379 2909356032 648887 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11279/tid=11283] ppid=11277 vsize=2841168 CPUtime=4959.6 cores=0-7
/proc/11279/task/11283/stat : 11283 (claspmt) R 11277 11279 11185 0 -1 4202560 55231 0 0 0 495888 72 0 0 25 0 8 0 548247379 2909356032 648887 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11279/tid=11284] ppid=11277 vsize=2841168 CPUtime=4960.35 cores=0-7
/proc/11279/task/11284/stat : 11284 (claspmt) R 11277 11279 11185 0 -1 4202560 75771 0 0 0 495922 113 0 0 25 0 8 0 548247379 2909356032 648887 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=11279/tid=11285] ppid=11277 vsize=2841168 CPUtime=4960.67 cores=0-7
/proc/11279/task/11285/stat : 11285 (claspmt) R 11277 11279 11185 0 -1 4202560 93173 0 0 0 495944 123 0 0 25 0 8 0 548247379 2909356032 648887 33554432000 4194304 4964704 140733769205248 18446744073709551615 4668654 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=11279/tid=11286] ppid=11277 vsize=2841168 CPUtime=4959.35 cores=0-7
/proc/11279/task/11286/stat : 11286 (claspmt) R 11277 11279 11185 0 -1 4202560 40514 0 0 0 495889 46 0 0 25 0 8 0 548247379 2909356032 648887 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670419 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11279/tid=11287] ppid=11277 vsize=2841168 CPUtime=4959.21 cores=0-7
/proc/11279/task/11287/stat : 11287 (claspmt) R 11277 11279 11185 0 -1 4202560 250857 0 0 0 495586 335 0 0 25 0 8 0 548247379 2909356032 648887 33554432000 4194304 4964704 140733769205248 18446744073709551615 4660415 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 39646.1
Current children cumulated vsize (KiB) 2841168



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5006.6 s]
/proc/loadavg: 8.13 8.08 8.01 9/229 11435
/proc/meminfo: memFree=18183168/32950928 swapFree=66698992/67111528
[pid=11279] ppid=11277 vsize=2841168 CPUtime=40000.2 cores=0-7
/proc/11279/stat : 11279 (claspmt) R 11277 11279 11185 0 -1 4202496 670356 0 0 0 3999167 852 0 0 25 0 8 0 548247222 2909356032 648891 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669309 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/11279/statm: 710292 648891 466 189 0 704641 0
[pid=11279/tid=11281] ppid=11277 vsize=2841168 CPUtime=4991.37 cores=0-7
/proc/11279/task/11281/stat : 11281 (claspmt) R 11277 11279 11185 0 -1 4202560 23148 0 0 0 499109 28 0 0 25 0 8 0 548247379 2909356032 648891 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=11279/tid=11282] ppid=11277 vsize=2841168 CPUtime=5004.08 cores=0-7
/proc/11279/task/11282/stat : 11282 (claspmt) R 11277 11279 11185 0 -1 4202560 71595 0 0 0 500323 85 0 0 25 0 8 0 548247379 2909356032 648891 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11279/tid=11283] ppid=11277 vsize=2841168 CPUtime=5003.9 cores=0-7
/proc/11279/task/11283/stat : 11283 (claspmt) R 11277 11279 11185 0 -1 4202560 55231 0 0 0 500318 72 0 0 25 0 8 0 548247379 2909356032 648891 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670291 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11279/tid=11284] ppid=11277 vsize=2841168 CPUtime=5004.66 cores=0-7
/proc/11279/task/11284/stat : 11284 (claspmt) R 11277 11279 11185 0 -1 4202560 75771 0 0 0 500353 113 0 0 25 0 8 0 548247379 2909356032 648891 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670250 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=11279/tid=11285] ppid=11277 vsize=2841168 CPUtime=5004.98 cores=0-7
/proc/11279/task/11285/stat : 11285 (claspmt) R 11277 11279 11185 0 -1 4202560 93173 0 0 0 500375 123 0 0 25 0 8 0 548247379 2909356032 648891 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669051 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=11279/tid=11286] ppid=11277 vsize=2841168 CPUtime=5003.66 cores=0-7
/proc/11279/task/11286/stat : 11286 (claspmt) R 11277 11279 11185 0 -1 4202560 40514 0 0 0 500320 46 0 0 25 0 8 0 548247379 2909356032 648891 33554432000 4194304 4964704 140733769205248 18446744073709551615 4669333 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11279/tid=11287] ppid=11277 vsize=2841168 CPUtime=5003.51 cores=0-7
/proc/11279/task/11287/stat : 11287 (claspmt) R 11277 11279 11185 0 -1 4202560 250857 0 0 0 500016 335 0 0 25 0 8 0 548247379 2909356032 648891 33554432000 4194304 4964704 140733769205248 18446744073709551615 4670100 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 40000.2
Current children cumulated vsize (KiB) 2841168

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5007 s]
/proc/loadavg: 8.13 8.08 8.01 9/229 11435
/proc/meminfo: memFree=18183168/32950928 swapFree=66698992/67111528
[pid=11279] ppid=11277 vsize=2792284 CPUtime=40000.6 cores=0-7
/proc/11279/stat : 11279 (claspmt) R 11277 11279 11185 0 -1 4202496 670366 0 0 0 3999211 852 0 0 25 0 1 0 548247222 2859298816 636172 33554432000 4194304 4964704 140733769205248 18446744073709551615 248089354139 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/11279/statm: 698071 636172 476 189 0 692420 0
Current children cumulated CPU time (s) 40000.6
Current children cumulated vsize (KiB) 2792284

[startup+5007.4 s]
/proc/loadavg: 8.13 8.08 8.01 2/223 11437
/proc/meminfo: memFree=18250672/32950928 swapFree=66698992/67111528
[pid=11279] ppid=11277 vsize=973556 CPUtime=40001 cores=0-7
/proc/11279/stat : 11279 (claspmt) R 11277 11279 11185 0 -1 4202496 670366 0 0 0 3999244 860 0 0 25 0 1 0 548247222 996921344 176412 33554432000 4194304 4964704 140733769205248 18446744073709551615 248089741671 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/11279/statm: 243389 176006 476 189 0 237738 0
Current children cumulated CPU time (s) 40001
Current children cumulated vsize (KiB) 973556

Child status: 0
Real time (s): 5007.44
CPU time (s): 40001.1
CPU user time (s): 39992.4
CPU system time (s): 8.64368
CPU usage (%): 798.833
Max. virtual memory (cumulated for all children) (KiB): 2841168

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39992.4
system time used= 8.64368
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 670367
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 28150
involuntary context switches= 188698

runsolver used 10.2484 second user time and 26.111 second system time

The end

Launcher Data 

Begin job on node123 at 2011-05-02 22:01:14
IDJOB=3321694
IDBENCH=82481
IDSOLVER=1711
FILE ID=node123/3321694-1304366474
RUNJOBID= node123-1304366467-11199
PBS_JOBID= 13181748
Free space on /tmp= 73672 MiB

SOLVER NAME= claspmt 2.0-R4095
BENCH NAME= SAT11/application/jarvisalo/HWMCC10-combination-timeframe-expansion/hwmcc10-timeframe-expansion-k50-pdtvisns3p00-tseitin.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3321694-1304366474/watcher-3321694-1304366474 -o /tmp/evaluation-result-3321694-1304366474/solver-3321694-1304366474 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3321694-1304366474.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= a8ad0173824ed9404a8c153da07c5f9c
RANDOM SEED=1397071266

node123.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      20783088 kB
Buffers:         83468 kB
Cached:         798852 kB
SwapCached:      16260 kB
Active:        9272968 kB
Inactive:      2655460 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      20783088 kB
SwapTotal:    67111528 kB
SwapFree:     66698992 kB
Dirty:           10824 kB
Writeback:           0 kB
AnonPages:    11037112 kB
Mapped:          19072 kB
Slab:           144620 kB
PageTables:      30264 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 12835104 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73672 MiB
End job on node123 at 2011-05-02 23:24:43