Trace number 3317087

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (MO) 2006.69 518.118

General information on the benchmark

Namerandom/large/
unif-k7-r85-v400-c34000-S1897807474-073-UNKNOWN.cnf
MD5SUM6ed0ea2d6f55cf35f843b9274cf5be29
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables400
Number of clauses34000
Sum of the clauses size238000
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 534000

Solver Data 

0.61/12.92	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.61/12.92	c INFO: solving HOME/instance-3317087-1304267293.cnf
0.61/12.92	c INFO: computing task features
0.80/13.15	c INFO: computed variable-clause graph statistics
0.80/13.16	c INFO: computed clause balance statistics
2.00/14.32	c INFO: computed variable balance statistics
2.49/14.80	c INFO: computed small-clause counts
2.49/14.80	c INFO: computed Horn-clause counts
2.49/14.84	c INFO: computed variable graph statistics
2.49/14.84	c INFO: collected features for HOME/instance-3317087-1304267293.cnf in 1.91 s
2.49/14.87	c INFO: running gnovelty+2@0 for 97 with 4798 remaining (b = 0.38)
2.59/14.90	c INFO: running gnovelty+2@0 for 97 with 4798 remaining (b = 0.13)
2.59/14.93	c INFO: running adaptg2wsat2009++@0 for 97 with 4798 remaining (b = 0.08)
2.59/14.96	c INFO: running adaptg2wsat2009++@0 for 97 with 4797 remaining (b = 0.07)
398.36/114.09	c INFO: running adaptg2wsat2009++@0 for 97 with 4698 remaining (b = 0.05)
398.76/114.13	c INFO: running gnovelty+2@0 for 97 with 4698 remaining (b = 0.04)
398.76/114.17	c INFO: running FH@0 for 97 with 4698 remaining (b = 0.03)
402.29/115.09	c INFO: running adaptg2wsat2009++@0 for 97 with 4697 remaining (b = 0.03)
797.58/214.25	c INFO: running adaptg2wsat2009++@0 for 97 with 4598 remaining (b = 0.02)
797.93/214.35	c INFO: running FH@0 for 97 with 4598 remaining (b = 0.02)
797.93/214.39	c INFO: running hybridGM3@0 for 194 with 4598 remaining (b = 0.04)
800.71/215.25	c INFO: running gnovelty+2@0 for 97 with 4597 remaining (b = 0.02)
1196.70/314.42	c INFO: running hybridGM3@0 for 194 with 4498 remaining (b = 0.03)
1197.11/314.52	c INFO: running adaptg2wsat2009++@0 for 97 with 4498 remaining (b = 0.01)
1199.81/315.48	c INFO: running hybridGM3@0 for 194 with 4497 remaining (b = 0.03)
1580.64/410.87	c INFO: running hybridGM3@99 for 97 with 4402 remaining (b = 0.01)
1596.19/414.72	c INFO: running hybridGM3@0 for 194 with 4398 remaining (b = 0.02)
1978.09/510.86	c INFO: running hybridGM3@99 for 97 with 4302 remaining (b = 0.01)
1982.84/512.07	c INFO: running TNM@0 for 1552 with 4300 remaining (b = 0.18)
2002.31/516.93	c INFO: running FH@0 for 97 with 4296 remaining (b = 0.01)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317087-1304267293/watcher-3317087-1304267293 -o /tmp/evaluation-result-3317087-1304267293/solver-3317087-1304267293 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317087-1304267293.cnf -seed 832098950 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.00 0.00 0.00 1/174 21454
/proc/meminfo: memFree=28954160/32951124 swapFree=67111348/67111528
[pid=21454] ppid=21451 vsize=11800 CPUtime=0 cores=1,3,5,7
/proc/21454/stat : 21454 (runsolver) S 21451 21454 21419 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 1 0 616959987 12083200 54 33554432000 4194304 5434984 140733677013728 18446744073709551615 5055495 0 2147483391 4096 24578 18446744071699199166 0 0 17 5 0 0 0
/proc/21454/statm: 2950 54 34 303 0 2644 0

[startup+0.0618819 s]
/proc/loadavg: 0.00 0.00 0.00 1/174 21454
/proc/meminfo: memFree=28954160/32951124 swapFree=67111348/67111528
[pid=21454] ppid=21451 vsize=8300 CPUtime=0 cores=1,3,5,7
/proc/21454/stat : 21454 (python) S 21451 21454 21419 0 -1 4194304 101 0 2 0 0 0 0 0 17 0 1 0 616959987 8499200 55 33554432000 4194304 5683644 140733669425232 18446744073709551615 252453142151 0 2147483391 4096 0 18446744071699199166 0 0 17 7 0 0 1
/proc/21454/statm: 2075 55 43 364 0 42 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8300

[startup+0.100826 s]
/proc/loadavg: 0.00 0.00 0.00 1/174 21454
/proc/meminfo: memFree=28954160/32951124 swapFree=67111348/67111528
[pid=21454] ppid=21451 vsize=14820 CPUtime=0 cores=1,3,5,7
/proc/21454/stat : 21454 (python) S 21451 21454 21419 0 -1 4202496 507 0 6 0 0 0 0 0 17 0 1 0 616959987 15175680 420 33554432000 4194304 5683644 140733669425232 18446744073709551615 252458061669 0 2147483391 4096 0 18446744071699199166 0 0 17 7 0 0 3
/proc/21454/statm: 3705 420 259 364 0 192 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 14820

[startup+0.300829 s]
/proc/loadavg: 0.00 0.00 0.00 1/174 21454
/proc/meminfo: memFree=28954160/32951124 swapFree=67111348/67111528
[pid=21454] ppid=21451 vsize=15184 CPUtime=0 cores=1,3,5,7
/proc/21454/stat : 21454 (python) S 21451 21454 21419 0 -1 4202496 719 0 6 0 0 0 0 0 16 0 1 0 616959987 15548416 615 33554432000 4194304 5683644 140733669425232 18446744073709551615 252458063872 0 2147483391 16781312 2 18446744071699199166 0 0 17 7 0 0 5
/proc/21454/statm: 3796 615 321 364 0 283 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15184

[startup+0.700721 s]
/proc/loadavg: 0.00 0.00 0.00 1/174 21454
/proc/meminfo: memFree=28954160/32951124 swapFree=67111348/67111528
[pid=21454] ppid=21451 vsize=15776 CPUtime=0 cores=1,3,5,7
/proc/21454/stat : 21454 (python) S 21451 21454 21419 0 -1 4202496 930 0 6 0 0 0 0 0 16 0 1 0 616959987 16154624 814 33554432000 4194304 5683644 140733669425232 18446744073709551615 252458063872 0 2147483391 16781312 2 18446744071699199166 0 0 17 7 0 0 8
/proc/21454/statm: 3944 814 356 364 0 431 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15776

[startup+1.50059 s]
/proc/loadavg: 0.08 0.02 0.01 1/178 21458
/proc/meminfo: memFree=28946368/32951124 swapFree=67111348/67111528
[pid=21454] ppid=21451 vsize=16400 CPUtime=0.02 cores=1,3,5,7
/proc/21454/stat : 21454 (python) D 21451 21454 21419 0 -1 4202496 1124 0 6 0 1 1 0 0 15 0 1 0 616959987 16793600 996 33554432000 4194304 5683644 140733669425232 18446744073709551615 252458065664 0 0 16781312 2 18446744071562234166 0 0 17 3 0 0 10
/proc/21454/statm: 4100 996 413 364 0 587 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 16400

[startup+3.10029 s]
/proc/loadavg: 0.08 0.02 0.01 3/178 21458
/proc/meminfo: memFree=28944944/32951124 swapFree=67111348/67111528
[pid=21454] ppid=21451 vsize=18884 CPUtime=0.04 cores=1,3,5,7
/proc/21454/stat : 21454 (python) D 21451 21454 21419 0 -1 4202496 1282 0 7 0 2 2 0 0 16 0 1 0 616959987 19337216 1141 33554432000 4194304 5683644 140733669425232 18446744073709551615 252458063872 0 0 16781312 2 0 0 0 17 3 0 0 42
/proc/21454/statm: 4721 1141 421 364 0 690 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 18884
heavy processes:

[startup+6.3007 s]
/proc/loadavg: 0.08 0.02 0.01 1/178 21458
/proc/meminfo: memFree=28940656/32951124 swapFree=67111348/67111528
[pid=21454] ppid=21451 vsize=54536 CPUtime=0.09 cores=1,3,5,7
/proc/21454/stat : 21454 (python) S 21451 21454 21419 0 -1 4202496 2053 0 19 0 4 5 0 0 15 0 1 0 616959987 55844864 1828 33554432000 4194304 5683644 140733669425232 18446744073709551615 252458063872 0 2147483391 16781312 2 18446744071699199166 0 0 17 3 0 0 61
/proc/21454/statm: 13634 1828 492 364 0 1284 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 54536
heavy processes:

[startup+12.7005 s]
/proc/loadavg: 0.07 0.02 0.00 1/178 21466
/proc/meminfo: memFree=28885512/32951124 swapFree=67111348/67111528
[pid=21454] ppid=21451 vsize=257000 CPUtime=0.58 cores=1,3,5,7
/proc/21454/stat : 21454 (python) S 21451 21454 21419 0 -1 4202496 8445 926 107 9 30 28 0 0 16 0 1 0 616959987 263168000 6411 33554432000 4194304 5683644 140733669425232 18446744073709551615 252458063872 0 2147483391 16781312 2 18446744071699199166 0 0 17 7 0 0 159
/proc/21454/statm: 64250 6411 1648 364 0 5129 0
Current children cumulated CPU time (s) 0.58
Current children cumulated vsize (KiB) 257000
heavy processes:

[startup+25.5011 s]
/proc/loadavg: 1.29 0.28 0.09 9/202 21491
/proc/meminfo: memFree=28723376/32951124 swapFree=67111348/67111528
[pid=21454] ppid=21451 vsize=323400 CPUtime=2.65 cores=1,3,5,7
/proc/21454/stat : 21454 (python) S 21451 21454 21419 0 -1 4202496 16660 926 115 9 234 31 0 0 21 0 5 0 616959987 331161600 8049 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469892555 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 162
/proc/21454/statm: 80850 8049 1883 364 0 16759 0
[pid=21480] ppid=21454 vsize=282244 CPUtime=0 cores=1,3,5,7
/proc/21480/stat : 21480 (python) S 21454 21454 21419 0 -1 4202560 908 0 0 0 0 0 0 0 18 0 1 0 616961475 289017856 6355 33554432000 4194304 5683644 140733669425232 18446744073709551615 252458085903 0 0 16781312 514 16720774700891865352 0 0 17 7 0 0 0
/proc/21480/statm: 70561 6355 378 364 0 6479 0
[pid=21482] ppid=21480 vsize=1020188 CPUtime=10.61 cores=1,3,5,7
/proc/21482/stat : 21482 (gnovelty+2) R 21480 21482 21482 0 -1 4202496 1136 0 0 0 1061 0 0 0 25 0 1 0 616961475 1044672512 877 33554432000 134512640 135663704 4289689360 18446744073709551615 134524548 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/21482/statm: 255047 877 113 282 0 254763 0
[pid=21483] ppid=21454 vsize=293128 CPUtime=0.01 cores=1,3,5,7
/proc/21483/stat : 21483 (python) S 21454 21454 21419 0 -1 4202560 925 0 0 0 0 1 0 0 18 0 1 0 616961478 300163072 6392 33554432000 4194304 5683644 140733669425232 18446744073709551615 252458085958 0 0 16781312 514 16720774700891865352 0 0 17 1 0 0 0
/proc/21483/statm: 73282 6392 372 364 0 9197 0
[pid=21485] ppid=21483 vsize=1020188 CPUtime=10.57 cores=1,3,5,7
/proc/21485/stat : 21485 (gnovelty+2) R 21483 21485 21485 0 -1 4202496 1140 0 0 0 1057 0 0 0 25 0 1 0 616961478 1044672512 877 33554432000 134512640 135663704 4289295056 18446744073709551615 134525678 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/21485/statm: 255047 877 113 282 0 254763 0
[pid=21486] ppid=21454 vsize=302900 CPUtime=0 cores=1,3,5,7
/proc/21486/stat : 21486 (python) S 21454 21454 21419 0 -1 4202560 904 0 0 0 0 0 0 0 18 0 1 0 616961480 310169600 6529 33554432000 4194304 5683644 140733669425232 18446744073709551615 252458085958 0 0 16781312 514 16720774700891865352 0 0 17 5 0 0 0
/proc/21486/statm: 75725 6529 370 364 0 11637 0
[pid=21488] ppid=21486 vsize=421804 CPUtime=10.48 cores=1,3,5,7
/proc/21488/stat : 21488 (adaptg2wsat2009) R 21486 21488 21488 0 -1 4202496 2136 0 0 0 1048 0 0 0 25 0 1 0 616961481 431927296 1872 33554432000 134512640 135107411 4289232736 18446744073709551615 134531781 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/21488/statm: 105451 1872 59 146 0 104327 0
[pid=21489] ppid=21454 vsize=313156 CPUtime=0 cores=1,3,5,7
/proc/21489/stat : 21489 (python) S 21454 21454 21419 0 -1 4202560 922 0 0 0 0 0 0 0 18 0 1 0 616961483 320671744 6533 33554432000 4194304 5683644 140733669425232 18446744073709551615 252458085958 0 0 16781312 514 16720774700891865352 0 0 17 1 0 0 0
/proc/21489/statm: 78289 6533 370 364 0 14198 0
[pid=21491] ppid=21489 vsize=421804 CPUtime=10.49 cores=1,3,5,7
/proc/21491/stat : 21491 (adaptg2wsat2009) R 21489 21491 21491 0 -1 4202496 2136 0 0 0 1049 0 0 0 25 0 1 0 616961484 431927296 1871 33554432000 134512640 135107411 4294859472 18446744073709551615 134531613 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/21491/statm: 105451 1871 59 146 0 104327 0
[pid=21454/tid=21481] ppid=21451 vsize=323400 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21481/stat : 21481 (python) S 21451 21454 21419 0 -1 4202560 26 926 0 9 0 0 0 0 20 0 5 0 616961475 331161600 8049 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 7 0 0 0
[pid=21454/tid=21484] ppid=21451 vsize=323400 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21484/stat : 21484 (python) S 21451 21454 21419 0 -1 4202560 25 926 0 9 0 0 0 0 22 0 5 0 616961478 331161600 8049 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 7 0 0 0
[pid=21454/tid=21487] ppid=21451 vsize=323400 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21487/stat : 21487 (python) S 21451 21454 21419 0 -1 4202560 25 926 0 9 0 0 0 0 20 0 5 0 616961481 331161600 8049 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 1 0 0 0
[pid=21454/tid=21490] ppid=21451 vsize=323400 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21490/stat : 21490 (python) S 21451 21454 21419 0 -1 4202560 21 926 0 9 0 0 0 0 22 0 5 0 616961484 331161600 8049 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 44.81
Current children cumulated vsize (KiB) 4398812

[startup+51.1013 s]
/proc/loadavg: 3.58 0.90 0.30 9/202 21491
/proc/meminfo: memFree=28717436/32951124 swapFree=67111348/67111528
[pid=21454] ppid=21451 vsize=323400 CPUtime=2.65 cores=1,3,5,7
/proc/21454/stat : 21454 (python) S 21451 21454 21419 0 -1 4202496 16660 926 115 9 234 31 0 0 21 0 5 0 616959987 331161600 8049 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469892555 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 162
/proc/21454/statm: 80850 8049 1883 364 0 16759 0
[pid=21480] ppid=21454 vsize=282244 CPUtime=0.04 cores=1,3,5,7
/proc/21480/stat : 21480 (python) S 21454 21454 21419 0 -1 4202560 915 0 0 0 2 2 0 0 18 0 1 0 616961475 289017856 6355 33554432000 4194304 5683644 140733669425232 18446744073709551615 252458085903 0 0 16781312 514 16720774700891865352 0 0 17 7 0 0 0
/proc/21480/statm: 70561 6355 378 364 0 6479 0
[pid=21482] ppid=21480 vsize=1020188 CPUtime=36.22 cores=1,3,5,7
/proc/21482/stat : 21482 (gnovelty+2) R 21480 21482 21482 0 -1 4202496 1136 0 0 0 3622 0 0 0 25 0 1 0 616961475 1044672512 877 33554432000 134512640 135663704 4289689360 18446744073709551615 134545635 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/21482/statm: 255047 877 113 282 0 254763 0
[pid=21483] ppid=21454 vsize=293128 CPUtime=0.04 cores=1,3,5,7
/proc/21483/stat : 21483 (python) S 21454 21454 21419 0 -1 4202560 931 0 0 0 2 2 0 0 18 0 1 0 616961478 300163072 6392 33554432000 4194304 5683644 140733669425232 18446744073709551615 252458085958 0 0 16781312 514 16720774700891865352 0 0 17 1 0 0 0
/proc/21483/statm: 73282 6392 372 364 0 9197 0
[pid=21485] ppid=21483 vsize=1020188 CPUtime=36.14 cores=1,3,5,7
/proc/21485/stat : 21485 (gnovelty+2) R 21483 21485 21485 0 -1 4202496 1140 0 0 0 3614 0 0 0 25 0 1 0 616961478 1044672512 877 33554432000 134512640 135663704 4289295056 18446744073709551615 134524534 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/21485/statm: 255047 877 113 282 0 254763 0
[pid=21486] ppid=21454 vsize=302900 CPUtime=0.04 cores=1,3,5,7
/proc/21486/stat : 21486 (python) S 21454 21454 21419 0 -1 4202560 913 0 0 0 2 2 0 0 18 0 1 0 616961480 310169600 6529 33554432000 4194304 5683644 140733669425232 18446744073709551615 252458085958 0 0 16781312 514 16720774700891865352 0 0 17 5 0 0 0
/proc/21486/statm: 75725 6529 370 364 0 11637 0
[pid=21488] ppid=21486 vsize=421804 CPUtime=35.91 cores=1,3,5,7
/proc/21488/stat : 21488 (adaptg2wsat2009) R 21486 21488 21488 0 -1 4202496 2136 0 0 0 3591 0 0 0 25 0 1 0 616961481 431927296 1872 33554432000 134512640 135107411 4289232736 18446744073709551615 134531600 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/21488/statm: 105451 1872 59 146 0 104327 0
[pid=21489] ppid=21454 vsize=313156 CPUtime=0.04 cores=1,3,5,7
/proc/21489/stat : 21489 (python) S 21454 21454 21419 0 -1 4202560 930 0 0 0 2 2 0 0 18 0 1 0 616961483 320671744 6533 33554432000 4194304 5683644 140733669425232 18446744073709551615 252458085958 0 0 16781312 514 16720774700891865352 0 0 17 1 0 0 0
/proc/21489/statm: 78289 6533 370 364 0 14198 0
[pid=21491] ppid=21489 vsize=421804 CPUtime=36.03 cores=1,3,5,7
/proc/21491/stat : 21491 (adaptg2wsat2009) R 21489 21491 21491 0 -1 4202496 2136 0 0 0 3603 0 0 0 25 0 1 0 616961484 431927296 1871 33554432000 134512640 135107411 4294859472 18446744073709551615 134531246 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/21491/statm: 105451 1871 59 146 0 104327 0
[pid=21454/tid=21481] ppid=21451 vsize=323400 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21481/stat : 21481 (python) S 21451 21454 21419 0 -1 4202560 26 926 0 9 0 0 0 0 20 0 5 0 616961475 331161600 8049 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 7 0 0 0
[pid=21454/tid=21484] ppid=21451 vsize=323400 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21484/stat : 21484 (python) S 21451 21454 21419 0 -1 4202560 25 926 0 9 0 0 0 0 22 0 5 0 616961478 331161600 8049 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 7 0 0 0
[pid=21454/tid=21487] ppid=21451 vsize=323400 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21487/stat : 21487 (python) S 21451 21454 21419 0 -1 4202560 25 926 0 9 0 0 0 0 20 0 5 0 616961481 331161600 8049 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 1 0 0 0
[pid=21454/tid=21490] ppid=21451 vsize=323400 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21490/stat : 21490 (python) S 21451 21454 21419 0 -1 4202560 21 926 0 9 0 0 0 0 22 0 5 0 616961484 331161600 8049 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 147.11
Current children cumulated vsize (KiB) 4398812

[startup+102.307 s]
/proc/loadavg: 6.32 2.12 0.75 9/202 21493
/proc/meminfo: memFree=28717780/32951124 swapFree=67111348/67111528
[pid=21454] ppid=21451 vsize=323400 CPUtime=2.65 cores=1,3,5,7
/proc/21454/stat : 21454 (python) S 21451 21454 21419 0 -1 4202496 16660 926 115 9 234 31 0 0 21 0 5 0 616959987 331161600 8049 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469892555 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 162
/proc/21454/statm: 80850 8049 1883 364 0 16759 0
[pid=21480] ppid=21454 vsize=282244 CPUtime=0.11 cores=1,3,5,7
/proc/21480/stat : 21480 (python) S 21454 21454 21419 0 -1 4202560 933 0 0 0 6 5 0 0 18 0 1 0 616961475 289017856 6355 33554432000 4194304 5683644 140733669425232 18446744073709551615 252458085903 0 0 16781312 514 16720774700891865352 0 0 17 7 0 0 0
/proc/21480/statm: 70561 6355 378 364 0 6479 0
[pid=21482] ppid=21480 vsize=1020188 CPUtime=87.42 cores=1,3,5,7
/proc/21482/stat : 21482 (gnovelty+2) R 21480 21482 21482 0 -1 4202496 1136 0 0 0 8742 0 0 0 25 0 1 0 616961475 1044672512 877 33554432000 134512640 135663704 4289689360 18446744073709551615 134525276 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/21482/statm: 255047 877 113 282 0 254763 0
[pid=21483] ppid=21454 vsize=293128 CPUtime=0.1 cores=1,3,5,7
/proc/21483/stat : 21483 (python) S 21454 21454 21419 0 -1 4202560 948 0 0 0 5 5 0 0 18 0 1 0 616961478 300163072 6392 33554432000 4194304 5683644 140733669425232 18446744073709551615 252458085958 0 0 16781312 514 16720774700891865352 0 0 17 1 0 0 0
/proc/21483/statm: 73282 6392 372 364 0 9197 0

################
# More data... #
################

/proc/21489/stat : 21489 (python) S 21454 21454 21419 0 -1 4202560 1114 0 0 0 7 5 0 0 19 0 2 0 616961483 320671744 6563 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469892555 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/21489/statm: 78289 6563 399 364 0 14198 0
[pid=21491] ppid=21489 vsize=421804 CPUtime=98.89 cores=1,3,5,7
/proc/21491/stat : 21491 (adaptg2wsat2009) T 21489 21491 21491 0 -1 4202496 2136 0 0 0 9889 0 0 0 25 0 1 0 616961484 431927296 1871 33554432000 134512640 135107411 4294859472 18446744073709551615 134531561 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/21491/statm: 105451 1871 59 146 0 104327 0
[pid=21489/tid=21499] ppid=21454 vsize=313156 CPUtime=0 cores=1,3,5,7
/proc/21489/task/21499/stat : 21499 (python) S 21454 21454 21419 0 -1 4202560 51 0 0 0 0 0 0 0 19 0 2 0 616971400 320671744 6563 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 514 18446604465560722112 0 0 -1 3 0 0 0
[pid=21496] ppid=21454 vsize=323544 CPUtime=0.13 cores=1,3,5,7
/proc/21496/stat : 21496 (python) S 21454 21454 21419 0 -1 4202560 1141 0 0 0 5 8 0 0 19 0 2 0 616971397 331309056 6568 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469892555 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/21496/statm: 80886 6568 400 364 0 16792 0
[pid=21498] ppid=21496 vsize=421804 CPUtime=99.97 cores=1,3,5,7
/proc/21498/stat : 21498 (adaptg2wsat2009) T 21496 21498 21498 0 -1 4202496 2138 0 0 0 9997 0 0 0 25 0 1 0 616971397 431927296 1871 33554432000 134512640 135107411 4289599616 18446744073709551615 134531582 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/21498/statm: 105451 1871 59 146 0 104327 0
[pid=21496/tid=21607] ppid=21454 vsize=323544 CPUtime=0 cores=1,3,5,7
/proc/21496/task/21607/stat : 21607 (python) S 21454 21454 21419 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 616981409 331309056 6568 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 514 18446604471128009664 0 0 -1 5 0 0 0
[pid=21500] ppid=21454 vsize=333668 CPUtime=0.12 cores=1,3,5,7
/proc/21500/stat : 21500 (python) S 21454 21454 21419 0 -1 4202560 1140 0 0 0 6 6 0 0 19 0 2 0 616971401 341676032 6571 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469892555 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/21500/statm: 83417 6571 399 364 0 19320 0
[pid=21502] ppid=21500 vsize=1020188 CPUtime=99.91 cores=1,3,5,7
/proc/21502/stat : 21502 (gnovelty+2) T 21500 21502 21502 0 -1 4202496 1139 0 0 0 9991 0 0 0 25 0 1 0 616971401 1044672512 877 33554432000 134512640 135663704 4288534496 18446744073709551615 134524548 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/21502/statm: 255047 877 113 282 0 254763 0
[pid=21500/tid=21611] ppid=21454 vsize=333668 CPUtime=0 cores=1,3,5,7
/proc/21500/task/21611/stat : 21611 (python) S 21454 21454 21419 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 616981419 341676032 6571 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 514 18446604471128007872 0 0 -1 5 0 0 0
[pid=21503] ppid=21454 vsize=344628 CPUtime=0.13 cores=1,3,5,7
/proc/21503/stat : 21503 (python) S 21454 21454 21419 0 -1 4202560 1199 0 0 0 7 6 0 0 19 0 2 0 616971404 352899072 6582 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469892555 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/21503/statm: 86157 6582 406 364 0 22057 0
[pid=21505] ppid=21503 vsize=251292 CPUtime=98.65 cores=1,3,5,7
/proc/21505/stat : 21505 (FH) T 21503 21505 21505 0 -1 4202496 1960 0 0 0 9865 0 0 0 25 0 1 0 616971405 257323008 1693 33554432000 134512640 134936988 4290403248 18446744073709551615 134553498 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/21505/statm: 62823 1693 41 104 0 62327 0
[pid=21503/tid=21612] ppid=21454 vsize=344628 CPUtime=0 cores=1,3,5,7
/proc/21503/task/21612/stat : 21612 (python) S 21454 21454 21419 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 616981419 352899072 6582 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 514 18446604471125314432 0 0 -1 1 0 0 0
[pid=21513] ppid=21454 vsize=354400 CPUtime=0.12 cores=1,3,5,7
/proc/21513/stat : 21513 (python) S 21454 21454 21419 0 -1 4202560 1186 0 0 0 6 6 0 0 19 0 2 0 616971496 362905600 6640 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469892555 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/21513/statm: 88600 6640 405 364 0 24497 0
[pid=21515] ppid=21513 vsize=421804 CPUtime=99.98 cores=1,3,5,7
/proc/21515/stat : 21515 (adaptg2wsat2009) T 21513 21515 21515 0 -1 4202496 2140 0 0 0 9998 0 0 0 25 0 1 0 616971496 431927296 1871 33554432000 134512640 135107411 4289399024 18446744073709551615 134531392 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/21515/statm: 105451 1871 59 146 0 104327 0
[pid=21513/tid=21619] ppid=21454 vsize=354400 CPUtime=0 cores=1,3,5,7
/proc/21513/task/21619/stat : 21619 (python) S 21454 21454 21419 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 616981508 362905600 6640 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 514 18446604471125311744 0 0 -1 1 0 0 0
[pid=21608] ppid=21454 vsize=364824 CPUtime=0.13 cores=1,3,5,7
/proc/21608/stat : 21608 (python) S 21454 21454 21419 0 -1 4202560 1172 0 0 0 7 6 0 0 19 0 2 0 616981413 373579776 6639 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469892555 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/21608/statm: 91206 6639 400 364 0 27100 0
[pid=21610] ppid=21608 vsize=421804 CPUtime=100.1 cores=1,3,5,7
/proc/21610/stat : 21610 (adaptg2wsat2009) T 21608 21610 21610 0 -1 4202496 2143 0 0 0 10010 0 0 0 25 0 1 0 616981413 431927296 1872 33554432000 134512640 135107411 4286621552 18446744073709551615 134531731 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/21610/statm: 105451 1872 59 146 0 104327 0
[pid=21608/tid=21669] ppid=21454 vsize=364824 CPUtime=0 cores=1,3,5,7
/proc/21608/task/21669/stat : 21669 (python) S 21454 21454 21419 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 616991425 373579776 6639 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 514 18446604471146458048 0 0 -1 1 0 0 0
[pid=21613] ppid=21454 vsize=375304 CPUtime=0.13 cores=1,3,5,7
/proc/21613/stat : 21613 (python) S 21454 21454 21419 0 -1 4202560 1178 0 0 0 7 6 0 0 19 0 2 0 616981423 384311296 6643 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469892555 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/21613/statm: 93826 6643 400 364 0 29717 0
[pid=21615] ppid=21613 vsize=251292 CPUtime=99.97 cores=1,3,5,7
/proc/21615/stat : 21615 (FH) T 21613 21615 21615 0 -1 4202496 1963 0 0 0 9997 0 0 0 25 0 1 0 616981423 257323008 1694 33554432000 134512640 134936988 4290123536 18446744073709551615 134552073 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/21615/statm: 62823 1694 41 104 0 62327 0
[pid=21613/tid=21673] ppid=21454 vsize=375304 CPUtime=0 cores=1,3,5,7
/proc/21613/task/21673/stat : 21673 (python) S 21454 21454 21419 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 616991435 384311296 6643 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 514 18446604471144165824 0 0 -1 5 0 0 0
[pid=21616] ppid=21454 vsize=385680 CPUtime=0.45 cores=1,3,5,7
/proc/21616/stat : 21616 (python) S 21454 21454 21419 0 -1 4202560 1199 0 0 0 23 22 0 0 18 0 2 0 616981427 394936320 6647 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469892555 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/21616/statm: 96420 6647 400 364 0 32308 0
[pid=21618] ppid=21616 vsize=268420 CPUtime=296.56 cores=1,3,5,7
/proc/21618/stat : 21618 (hybridGM3) T 21616 21618 21618 0 -1 4202496 5646 0 0 0 29656 0 0 0 25 0 1 0 616981427 274862080 5377 33554432000 134512640 135204385 4289843392 18446744073709551615 134545302 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/21618/statm: 67105 5377 63 169 0 66935 0
[pid=21616/tid=21700] ppid=21454 vsize=385680 CPUtime=0 cores=1,3,5,7
/proc/21616/task/21700/stat : 21700 (python) S 21454 21454 21419 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 617001070 394936320 6647 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 514 18446604459674908672 0 0 -1 7 0 0 0
[pid=21620] ppid=21454 vsize=395856 CPUtime=0.13 cores=1,3,5,7
/proc/21620/stat : 21620 (python) S 21454 21454 21419 0 -1 4202560 1184 0 0 0 6 7 0 0 19 0 2 0 616981513 405356544 6658 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469892555 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/21620/statm: 98964 6658 400 364 0 34849 0
[pid=21622] ppid=21620 vsize=1020188 CPUtime=98.62 cores=1,3,5,7
/proc/21622/stat : 21622 (gnovelty+2) T 21620 21622 21622 0 -1 4202496 1148 0 0 0 9862 0 0 0 25 0 1 0 616981513 1044672512 877 33554432000 134512640 135663704 4294189056 18446744073709551615 134524894 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/21622/statm: 255047 877 113 282 0 254763 0
[pid=21620/tid=21678] ppid=21454 vsize=395856 CPUtime=0 cores=1,3,5,7
/proc/21620/task/21678/stat : 21678 (python) S 21454 21454 21419 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 616991531 405356544 6658 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 514 18446604459674906880 0 0 -1 7 0 0 0
[pid=21670] ppid=21454 vsize=406248 CPUtime=0.31 cores=1,3,5,7
/proc/21670/stat : 21670 (python) S 21454 21454 21419 0 -1 4202560 1192 0 0 0 16 15 0 0 18 0 2 0 616991430 415997952 6662 33554432000 4194304 5683644 140733669425232 18446744073709551615 252458085958 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/21670/statm: 101562 6662 400 364 0 37444 0
[pid=21672] ppid=21670 vsize=268428 CPUtime=203.5 cores=1,3,5,7
/proc/21672/stat : 21672 (hybridGM3) R 21670 21672 21672 0 -1 4202496 5655 0 0 0 20350 0 0 0 25 0 1 0 616991430 274870272 5378 33554432000 134512640 135204385 4293592912 18446744073709551615 134537935 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/21672/statm: 67107 5378 63 169 0 66937 0
[pid=21670/tid=21724] ppid=21454 vsize=406248 CPUtime=0 cores=1,3,5,7
/proc/21670/task/21724/stat : 21724 (python) S 21454 21454 21419 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 617011069 415997952 6662 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 514 18446604465560721216 0 0 -1 3 0 0 0
[pid=21674] ppid=21454 vsize=416412 CPUtime=0.14 cores=1,3,5,7
/proc/21674/stat : 21674 (python) S 21454 21454 21419 0 -1 4202560 1189 0 0 0 7 7 0 0 19 0 2 0 616991440 426405888 6679 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469892555 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/21674/statm: 104103 6679 400 364 0 39982 0
[pid=21676] ppid=21674 vsize=421804 CPUtime=100.13 cores=1,3,5,7
/proc/21676/stat : 21676 (adaptg2wsat2009) T 21674 21676 21676 0 -1 4202496 2144 0 0 0 10013 0 0 0 25 0 1 0 616991440 431927296 1871 33554432000 134512640 135107411 4290251776 18446744073709551615 134532128 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/21676/statm: 105451 1871 59 146 0 104327 0
[pid=21674/tid=21701] ppid=21454 vsize=416412 CPUtime=0 cores=1,3,5,7
/proc/21674/task/21701/stat : 21701 (python) S 21454 21454 21419 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 617001454 426405888 6679 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 514 18446604471129392320 0 0 -1 5 0 0 0
[pid=21679] ppid=21454 vsize=426820 CPUtime=0.32 cores=1,3,5,7
/proc/21679/stat : 21679 (python) S 21454 21454 21419 0 -1 4202560 1239 0 0 0 17 15 0 0 19 0 2 0 616991536 437063680 6691 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469892555 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/21679/statm: 106705 6691 408 364 0 42581 0
[pid=21681] ppid=21679 vsize=268428 CPUtime=197.07 cores=1,3,5,7
/proc/21681/stat : 21681 (hybridGM3) T 21679 21681 21681 0 -1 4202496 5653 0 0 0 19706 1 0 0 25 0 1 0 616991536 274870272 5378 33554432000 134512640 135204385 4287346464 18446744073709551615 134538284 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/21681/statm: 67107 5378 63 169 0 66937 0
[pid=21679/tid=21728] ppid=21454 vsize=426820 CPUtime=0 cores=1,3,5,7
/proc/21679/task/21728/stat : 21728 (python) S 21454 21454 21419 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 617011675 437063680 6691 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 514 18446604467824619840 0 0 -1 7 0 0 0
[pid=21702] ppid=21454 vsize=437128 CPUtime=0.16 cores=1,3,5,7
/proc/21702/stat : 21702 (python) S 21454 21454 21419 0 -1 4202560 1098 0 0 0 8 8 0 0 18 0 1 0 617001459 447619072 6681 33554432000 4194304 5683644 140733669425232 18446744073709551615 252458085958 0 0 16781312 514 16720774700891865350 0 0 17 5 0 0 0
/proc/21702/statm: 109282 6681 380 364 0 45155 0
[pid=21704] ppid=21702 vsize=268420 CPUtime=102.65 cores=1,3,5,7
/proc/21704/stat : 21704 (hybridGM3) R 21702 21704 21704 0 -1 4202496 5653 0 0 0 10265 0 0 0 25 0 1 0 617001461 274862080 5377 33554432000 134512640 135204385 4287966480 18446744073709551615 134538041 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/21704/statm: 67105 5377 63 169 0 66935 0
[pid=21725] ppid=21454 vsize=447412 CPUtime=0 cores=1,3,5,7
/proc/21725/stat : 21725 (python) S 21454 21454 21419 0 -1 4202560 1058 0 0 0 0 0 0 0 18 0 1 0 617011194 458149888 6693 33554432000 4194304 5683644 140733669425232 18446744073709551615 252458085958 0 0 16781312 514 8706012512740409607 0 0 17 5 0 0 0
/proc/21725/statm: 111853 6693 374 364 0 47723 0
[pid=21727] ppid=21725 vsize=455276 CPUtime=5.88 cores=1,3,5,7
/proc/21727/stat : 21727 (TNM) R 21725 21727 21727 0 -1 4202496 1575 0 0 0 588 0 0 0 25 0 1 0 617011195 466202624 1302 33554432000 134512640 135008748 4288985968 18446744073709551615 134523389 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/21727/statm: 113819 1302 45 122 0 112719 0
[pid=21729] ppid=21454 vsize=457784 CPUtime=0 cores=1,3,5,7
/proc/21729/stat : 21729 (python) S 21454 21454 21419 0 -1 4202560 844 0 0 0 0 0 0 0 18 0 1 0 617011681 468770816 6681 33554432000 4194304 5683644 140733669425232 18446744073709551615 252458085958 0 0 16781312 514 16313735225706283007 0 0 17 5 0 0 0
/proc/21729/statm: 114446 6681 351 364 0 50313 0
[pid=21731] ppid=21729 vsize=249144 CPUtime=1.04 cores=1,3,5,7
/proc/21731/stat : 21731 (FH) R 21729 21731 21731 0 -1 4202496 1567 0 0 0 104 0 0 0 24 0 1 0 617011681 255123456 1296 33554432000 134512640 134936988 4291532864 18446744073709551615 134523665 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/21731/statm: 62286 1296 37 104 0 61790 0
[pid=21454/tid=21481] ppid=21451 vsize=467344 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21481/stat : 21481 (python) S 21451 21454 21419 0 -1 4202560 26 926 0 9 0 0 0 0 20 0 19 0 616961475 478560256 8245 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 7 0 0 0
[pid=21454/tid=21484] ppid=21451 vsize=467344 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21484/stat : 21484 (python) S 21451 21454 21419 0 -1 4202560 25 926 0 9 0 0 0 0 22 0 19 0 616961478 478560256 8245 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 7 0 0 0
[pid=21454/tid=21487] ppid=21451 vsize=467344 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21487/stat : 21487 (python) S 21451 21454 21419 0 -1 4202560 25 926 0 9 0 0 0 0 20 0 19 0 616961481 478560256 8245 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 1 0 0 0
[pid=21454/tid=21490] ppid=21451 vsize=467344 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21490/stat : 21490 (python) S 21451 21454 21419 0 -1 4202560 21 926 0 9 0 0 0 0 22 0 19 0 616961484 478560256 8245 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 1 0 0 0
[pid=21454/tid=21497] ppid=21451 vsize=467344 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21497/stat : 21497 (python) S 21451 21454 21419 0 -1 4202560 30 926 0 9 0 0 0 0 20 0 19 0 616971397 478560256 8245 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 5 0 0 0
[pid=21454/tid=21501] ppid=21451 vsize=467344 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21501/stat : 21501 (python) S 21451 21454 21419 0 -1 4202560 25 926 0 9 0 0 0 0 20 0 19 0 616971401 478560256 8245 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 1 0 0 0
[pid=21454/tid=21504] ppid=21451 vsize=467344 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21504/stat : 21504 (python) S 21451 21454 21419 0 -1 4202560 31 926 0 9 0 0 0 0 22 0 19 0 616971405 478560256 8245 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 1 0 0 0
[pid=21454/tid=21514] ppid=21451 vsize=467344 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21514/stat : 21514 (python) S 21451 21454 21419 0 -1 4202560 28 926 0 9 0 0 0 0 20 0 19 0 616971496 478560256 8245 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 1 0 0 0
[pid=21454/tid=21609] ppid=21451 vsize=467344 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21609/stat : 21609 (python) S 21451 21454 21419 0 -1 4202560 30 926 0 9 0 0 0 0 20 0 19 0 616981413 478560256 8245 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 1 0 0 0
[pid=21454/tid=21614] ppid=21451 vsize=467344 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21614/stat : 21614 (python) S 21451 21454 21419 0 -1 4202560 24 926 0 9 0 0 0 0 21 0 19 0 616981423 478560256 8245 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 5 0 0 0
[pid=21454/tid=21617] ppid=21451 vsize=467344 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21617/stat : 21617 (python) S 21451 21454 21419 0 -1 4202560 29 926 0 9 0 0 0 0 18 0 19 0 616981427 478560256 8245 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 5 0 0 0
[pid=21454/tid=21621] ppid=21451 vsize=467344 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21621/stat : 21621 (python) S 21451 21454 21419 0 -1 4202560 29 926 0 9 0 0 0 0 20 0 19 0 616981513 478560256 8245 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 5 0 0 0
[pid=21454/tid=21671] ppid=21451 vsize=467344 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21671/stat : 21671 (python) S 21451 21454 21419 0 -1 4202560 33 926 0 9 0 0 0 0 18 0 19 0 616991430 478560256 8245 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 5 0 0 0
[pid=21454/tid=21675] ppid=21451 vsize=467344 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21675/stat : 21675 (python) S 21451 21454 21419 0 -1 4202560 28 926 0 9 0 0 0 0 21 0 19 0 616991440 478560256 8245 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 5 0 0 0
[pid=21454/tid=21680] ppid=21451 vsize=467344 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21680/stat : 21680 (python) S 21451 21454 21419 0 -1 4202560 27 926 0 9 0 0 0 0 20 0 19 0 616991536 478560256 8245 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 5 0 0 0
[pid=21454/tid=21703] ppid=21451 vsize=467344 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21703/stat : 21703 (python) S 21451 21454 21419 0 -1 4202560 29 926 0 9 0 0 0 0 20 0 19 0 617001460 478560256 8245 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 5 0 0 0
[pid=21454/tid=21726] ppid=21451 vsize=467344 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21726/stat : 21726 (python) S 21451 21454 21419 0 -1 4202560 26 926 0 9 0 0 0 0 20 0 19 0 617011194 478560256 8245 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 5 0 0 0
[pid=21454/tid=21730] ppid=21451 vsize=467344 CPUtime=0 cores=1,3,5,7
/proc/21454/task/21730/stat : 21730 (python) S 21451 21454 21419 0 -1 4202560 27 926 0 9 0 0 0 0 20 0 19 0 617011681 478560256 8245 33554432000 4194304 5683644 140733669425232 18446744073709551615 252469889457 0 0 16781312 2 18446604464696454208 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 2006.69
Current children cumulated vsize (KiB) 16031000

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 21454 and gives
#  childrusage.ru_utime.tv_sec=3
#  childrusage.ru_utime.tv_usec=29539
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=361944
# CPU time returned by wait4() is 3.39148
# while last known CPU time is 2006.69
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 518.118
CPU time (s): 2006.69
CPU user time (s): 2004.98
CPU system time (s): 1.71
CPU usage (%): 387.303
Max. virtual memory (cumulated for all children) (KiB): 16031000

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 3.02954
system time used= 0.361944
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 31784
page faults= 124
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 19590
involuntary context switches= 149

runsolver used 2.5926 second user time and 5.46117 second system time

The end

Launcher Data 

Begin job on node111 at 2011-05-01 18:28:13
IDJOB=3317087
IDBENCH=82552
IDSOLVER=1705
FILE ID=node111/3317087-1304267293
RUNJOBID= node111-1304267292-21438
PBS_JOBID= 13173967
Free space on /tmp= 72552 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k7-r85-v400-c34000-S1897807474-073-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317087-1304267293/watcher-3317087-1304267293 -o /tmp/evaluation-result-3317087-1304267293/solver-3317087-1304267293 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317087-1304267293.cnf -seed 832098950 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 6ed0ea2d6f55cf35f843b9274cf5be29
RANDOM SEED=832098950

node111.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5428.12
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.86
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      28954704 kB
Buffers:       1046840 kB
Cached:        2431652 kB
SwapCached:          0 kB
Active:         379256 kB
Inactive:      3163200 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      28954704 kB
SwapTotal:    67111528 kB
SwapFree:     67111348 kB
Dirty:           65168 kB
Writeback:           0 kB
AnonPages:       63852 kB
Mapped:          14540 kB
Slab:           389176 kB
PageTables:       4652 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   187664 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72492 MiB
End job on node111 at 2011-05-01 18:36:55