Trace number 3317165

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (MO) 1974.93 498.344

General information on the benchmark

Namerandom/large/
unif-k5-r20-v2000-c40000-S1264065752-051-UNKNOWN.cnf
MD5SUMfafa522767a4e86ea9e03f34c2fb7311
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark279.102
Satisfiable
(Un)Satisfiability was proved
Number of variables2000
Number of clauses40000
Sum of the clauses size200000
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 540000
Number of clauses of size over 50

Solver Data 

0.36/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.36/0.67	c INFO: solving HOME/instance-3317165-1304267903.cnf
0.36/0.67	c INFO: computing task features
0.54/0.88	c INFO: computed variable-clause graph statistics
0.54/0.88	c INFO: computed clause balance statistics
1.64/1.92	c INFO: computed variable balance statistics
2.14/2.49	c INFO: computed small-clause counts
2.14/2.49	c INFO: computed Horn-clause counts
2.24/2.54	c INFO: computed variable graph statistics
2.24/2.54	c INFO: collected features for HOME/instance-3317165-1304267903.cnf in 1.86 s
2.24/2.57	c INFO: running gnovelty+2@0 for 97 with 4798 remaining (b = 0.38)
2.24/2.59	c INFO: running gnovelty+2@0 for 97 with 4798 remaining (b = 0.13)
2.33/2.63	c INFO: running adaptg2wsat2009++@0 for 97 with 4798 remaining (b = 0.08)
2.33/2.65	c INFO: running adaptg2wsat2009++@0 for 97 with 4798 remaining (b = 0.07)
401.10/102.79	c INFO: running adaptg2wsat2009++@0 for 97 with 4697 remaining (b = 0.05)
401.50/102.84	c INFO: running gnovelty+2@0 for 97 with 4697 remaining (b = 0.05)
401.50/102.87	c INFO: running FH@0 for 97 with 4697 remaining (b = 0.03)
401.65/102.91	c INFO: running adaptg2wsat2009++@0 for 97 with 4697 remaining (b = 0.03)
800.11/202.98	c INFO: running adaptg2wsat2009++@0 for 97 with 4597 remaining (b = 0.02)
800.49/203.09	c INFO: running FH@0 for 97 with 4597 remaining (b = 0.02)
800.74/203.13	c INFO: running hybridGM3@0 for 194 with 4597 remaining (b = 0.04)
800.74/203.17	c INFO: running gnovelty+2@0 for 97 with 4597 remaining (b = 0.02)
1195.46/302.30	c INFO: running hybridGM3@0 for 194 with 4498 remaining (b = 0.03)
1199.02/303.19	c INFO: running adaptg2wsat2009++@0 for 97 with 4497 remaining (b = 0.01)
1199.99/303.44	c INFO: running hybridGM3@0 for 194 with 4497 remaining (b = 0.03)
1582.79/399.58	c INFO: running hybridGM3@100 for 97 with 4401 remaining (b = 0.01)
1598.31/403.43	c INFO: running hybridGM3@0 for 194 with 4397 remaining (b = 0.02)
1961.00/494.79	c INFO: running TNM@0 for 1552 with 4305 remaining (b = 0.19)
1969.32/496.86	c INFO: running hybridGM3@99 for 97 with 4303 remaining (b = 0.01)
1973.69/497.93	c INFO: running FH@0 for 97 with 4302 remaining (b = 0.01)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317165-1304267903/watcher-3317165-1304267903 -o /tmp/evaluation-result-3317165-1304267903/solver-3317165-1304267903 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317165-1304267903.cnf -seed 1755707011 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.84 6.95 3.82 6/251 11267
/proc/meminfo: memFree=22962484/32951124 swapFree=67102140/67111528
[pid=11267] ppid=11265 vsize=15384 CPUtime=0 cores=1,3,5,7
/proc/11267/stat : 11267 (python) R 11265 11267 10986 0 -1 4202496 796 0 0 0 0 0 0 0 23 0 1 0 617020959 15753216 682 33554432000 4194304 5683644 140736178709536 18446744073709551615 216735178496 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/11267/statm: 3846 684 322 364 0 333 0

[startup+0.051018 s]
/proc/loadavg: 7.84 6.95 3.82 6/251 11267
/proc/meminfo: memFree=22962484/32951124 swapFree=67102140/67111528
[pid=11267] ppid=11265 vsize=23184 CPUtime=0.02 cores=1,3,5,7
/proc/11267/stat : 11267 (python) R 11265 11267 10986 0 -1 4202496 1319 0 0 0 2 0 0 0 23 0 1 0 617020959 23740416 1169 33554432000 4194304 5683644 140736178709536 18446744073709551615 216730255031 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/11267/statm: 5796 1172 432 364 0 723 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 23184

[startup+0.101009 s]
/proc/loadavg: 7.84 6.95 3.82 6/251 11267
/proc/meminfo: memFree=22962484/32951124 swapFree=67102140/67111528
[pid=11267] ppid=11265 vsize=89768 CPUtime=0.06 cores=1,3,5,7
/proc/11267/stat : 11267 (python) S 11265 11267 10986 0 -1 4202496 2673 0 0 0 5 1 0 0 17 0 1 0 617020959 91922432 2388 33554432000 4194304 5683644 140736178709536 18446744073709551615 216735176704 0 2147483391 16781312 2 18446744071699199166 0 0 17 3 0 0 0
/proc/11267/statm: 22442 2388 750 364 0 1622 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 89768

[startup+0.30101 s]
/proc/loadavg: 7.84 6.95 3.82 6/251 11267
/proc/meminfo: memFree=22962484/32951124 swapFree=67102140/67111528
[pid=11267] ppid=11265 vsize=135748 CPUtime=0.18 cores=1,3,5,7
/proc/11267/stat : 11267 (python) S 11265 11267 10986 0 -1 4202496 5962 932 1 0 14 4 0 0 17 0 1 0 617020959 139005952 4151 33554432000 4194304 5683644 140736178709536 18446744073709551615 216735174661 0 2147483391 16781312 2 18446744071699199166 0 0 17 1 0 0 0
/proc/11267/statm: 33937 4151 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.18
Current children cumulated vsize (KiB) 135748

[startup+0.700901 s]
/proc/loadavg: 7.84 6.95 3.82 6/251 11267
/proc/meminfo: memFree=22962484/32951124 swapFree=67102140/67111528
[pid=11267] ppid=11265 vsize=280824 CPUtime=0.44 cores=1,3,5,7
/proc/11267/stat : 11267 (python) R 11265 11267 10986 0 -1 4202496 9857 932 1 0 33 11 0 0 15 0 1 0 617020959 287563776 7540 33554432000 4194304 5683644 140736178709536 18446744073709551615 4571108 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/11267/statm: 70206 7540 1761 364 0 6130 0
Current children cumulated CPU time (s) 0.44
Current children cumulated vsize (KiB) 280824

[startup+1.50076 s]
/proc/loadavg: 7.84 6.95 3.82 6/252 11272
/proc/meminfo: memFree=22927136/32951124 swapFree=67102140/67111528
[pid=11267] ppid=11265 vsize=290808 CPUtime=1.24 cores=1,3,5,7
/proc/11267/stat : 11267 (python) R 11265 11267 10986 0 -1 4202496 14010 932 1 0 113 11 0 0 16 0 1 0 617020959 297787392 10087 33554432000 4194304 5683644 140736178709536 18446744073709551615 216738587333 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/11267/statm: 72702 10087 1826 364 0 8626 0
Current children cumulated CPU time (s) 1.24
Current children cumulated vsize (KiB) 290808

[startup+3.10045 s]
/proc/loadavg: 7.84 6.95 3.82 6/252 11272
/proc/meminfo: memFree=22927136/32951124 swapFree=67102140/67111528
[pid=11267] ppid=11265 vsize=325516 CPUtime=2.39 cores=1,3,5,7
/proc/11267/stat : 11267 (python) S 11265 11267 10986 0 -1 4202496 16958 932 1 0 227 12 0 0 24 0 5 0 617020959 333328384 8578 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747005387 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/11267/statm: 81379 8578 1883 364 0 17288 0
Current children cumulated CPU time (s) 2.39
Current children cumulated vsize (KiB) 325516
heavy processes:

[startup+6.30089 s]
/proc/loadavg: 7.94 6.99 3.84 9/264 11285
/proc/meminfo: memFree=22893736/32951124 swapFree=67102140/67111528
[pid=11267] ppid=11265 vsize=325516 CPUtime=2.39 cores=1,3,5,7
/proc/11267/stat : 11267 (python) S 11265 11267 10986 0 -1 4202496 16958 932 1 0 227 12 0 0 24 0 5 0 617020959 333328384 8578 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747005387 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/11267/statm: 81379 8578 1883 364 0 17288 0
[pid=11273] ppid=11267 vsize=284988 CPUtime=0 cores=1,3,5,7
/proc/11273/stat : 11273 (python) S 11267 11267 10986 0 -1 4202560 733 0 0 0 0 0 0 0 18 0 1 0 617021216 291827712 6900 33554432000 4194304 5683644 140736178709536 18446744073709551615 216735198735 0 0 16781312 514 10652168434869043461 0 0 17 5 0 0 0
/proc/11273/statm: 71247 6900 355 364 0 7165 0
[pid=11275] ppid=11273 vsize=1020308 CPUtime=3.72 cores=1,3,5,7
/proc/11275/stat : 11275 (gnovelty+2) R 11273 11275 11275 0 -1 4202496 1154 0 0 0 372 0 0 0 25 0 1 0 617021217 1044795392 892 33554432000 134512640 135663704 4286752544 18446744073709551615 134524688 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/11275/statm: 255077 892 113 282 0 254793 0
[pid=11276] ppid=11267 vsize=295008 CPUtime=0 cores=1,3,5,7
/proc/11276/stat : 11276 (python) S 11267 11267 10986 0 -1 4202560 741 0 0 0 0 0 0 0 18 0 1 0 617021219 302088192 7033 33554432000 4194304 5683644 140736178709536 18446744073709551615 216735198790 0 0 16781312 514 14882127930510508293 0 0 17 1 0 0 0
/proc/11276/statm: 73752 7033 349 364 0 9667 0
[pid=11278] ppid=11276 vsize=1020308 CPUtime=3.66 cores=1,3,5,7
/proc/11278/stat : 11278 (gnovelty+2) R 11276 11278 11278 0 -1 4202496 1155 0 0 0 366 0 0 0 25 0 1 0 617021220 1044795392 892 33554432000 134512640 135663704 4286897936 18446744073709551615 134524918 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/11278/statm: 255077 892 113 282 0 254793 0
[pid=11279] ppid=11267 vsize=305196 CPUtime=0 cores=1,3,5,7
/proc/11279/stat : 11279 (python) S 11267 11267 10986 0 -1 4202560 736 0 0 0 0 0 0 0 18 0 1 0 617021222 312520704 7037 33554432000 4194304 5683644 140736178709536 18446744073709551615 216735198790 0 0 16781312 514 13471190629283234053 0 0 17 5 0 0 0
/proc/11279/statm: 76299 7037 349 364 0 12211 0
[pid=11281] ppid=11279 vsize=423400 CPUtime=3.65 cores=1,3,5,7
/proc/11281/stat : 11281 (adaptg2wsat2009) R 11279 11281 11281 0 -1 4202496 2626 0 0 0 365 0 0 0 25 0 1 0 617021223 433561600 2359 33554432000 134512640 135107411 4287144208 18446744073709551615 134531582 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/11281/statm: 105850 2359 59 146 0 104726 0
[pid=11282] ppid=11267 vsize=316060 CPUtime=0 cores=1,3,5,7
/proc/11282/stat : 11282 (python) S 11267 11267 10986 0 -1 4202560 744 0 0 0 0 0 0 0 18 0 1 0 617021225 323645440 7041 33554432000 4194304 5683644 140736178709536 18446744073709551615 216735198790 0 0 16781312 514 4513037289722446087 0 0 17 1 0 0 0
/proc/11282/statm: 79015 7041 349 364 0 14924 0
[pid=11284] ppid=11282 vsize=423400 CPUtime=3.63 cores=1,3,5,7
/proc/11284/stat : 11284 (adaptg2wsat2009) R 11282 11284 11284 0 -1 4202496 2620 0 0 0 363 0 0 0 25 0 1 0 617021226 433561600 2359 33554432000 134512640 135107411 4292305360 18446744073709551615 134531561 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/11284/statm: 105850 2359 59 146 0 104726 0
[pid=11267/tid=11274] ppid=11265 vsize=325516 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11274/stat : 11274 (python) S 11265 11267 10986 0 -1 4202560 24 932 0 0 0 0 0 0 20 0 5 0 617021217 333328384 8578 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 3 0 0 0
[pid=11267/tid=11277] ppid=11265 vsize=325516 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11277/stat : 11277 (python) S 11265 11267 10986 0 -1 4202560 22 932 0 0 0 0 0 0 20 0 5 0 617021220 333328384 8578 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 5 0 0 0
[pid=11267/tid=11280] ppid=11265 vsize=325516 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11280/stat : 11280 (python) S 11265 11267 10986 0 -1 4202560 19 932 0 0 0 0 0 0 22 0 5 0 617021222 333328384 8578 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 5 0 0 0
[pid=11267/tid=11283] ppid=11265 vsize=325516 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11283/stat : 11283 (python) S 11265 11267 10986 0 -1 4202560 26 932 0 0 0 0 0 0 24 0 5 0 617021226 333328384 8578 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 17.05
Current children cumulated vsize (KiB) 4414184

[startup+12.7008 s]
/proc/loadavg: 7.94 7.00 3.87 9/265 11286
/proc/meminfo: memFree=22890236/32951124 swapFree=67102140/67111528
[pid=11267] ppid=11265 vsize=325516 CPUtime=2.39 cores=1,3,5,7
/proc/11267/stat : 11267 (python) S 11265 11267 10986 0 -1 4202496 16958 932 1 0 227 12 0 0 24 0 5 0 617020959 333328384 8578 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747005387 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/11267/statm: 81379 8578 1883 364 0 17288 0
[pid=11273] ppid=11267 vsize=284988 CPUtime=0.01 cores=1,3,5,7
/proc/11273/stat : 11273 (python) S 11267 11267 10986 0 -1 4202560 892 0 0 0 1 0 0 0 18 0 1 0 617021216 291827712 6923 33554432000 4194304 5683644 140736178709536 18446744073709551615 216735198735 0 0 16781312 514 4026605094962168069 0 0 17 5 0 0 0
/proc/11273/statm: 71247 6923 378 364 0 7165 0
[pid=11275] ppid=11273 vsize=1020308 CPUtime=10.12 cores=1,3,5,7
/proc/11275/stat : 11275 (gnovelty+2) R 11273 11275 11275 0 -1 4202496 1154 0 0 0 1012 0 0 0 25 0 1 0 617021217 1044795392 892 33554432000 134512640 135663704 4286752544 18446744073709551615 134525479 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/11275/statm: 255077 892 113 282 0 254793 0
[pid=11276] ppid=11267 vsize=295008 CPUtime=0.01 cores=1,3,5,7
/proc/11276/stat : 11276 (python) S 11267 11267 10986 0 -1 4202560 901 0 0 0 1 0 0 0 18 0 1 0 617021219 302088192 7056 33554432000 4194304 5683644 140736178709536 18446744073709551615 216735198790 0 0 16781312 514 4026605094962168069 0 0 17 1 0 0 0
/proc/11276/statm: 73752 7056 372 364 0 9667 0
[pid=11278] ppid=11276 vsize=1020308 CPUtime=10.01 cores=1,3,5,7
/proc/11278/stat : 11278 (gnovelty+2) R 11276 11278 11278 0 -1 4202496 1155 0 0 0 1001 0 0 0 25 0 1 0 617021220 1044795392 892 33554432000 134512640 135663704 4286897936 18446744073709551615 134524498 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/11278/statm: 255077 892 113 282 0 254793 0
[pid=11279] ppid=11267 vsize=305196 CPUtime=0.01 cores=1,3,5,7
/proc/11279/stat : 11279 (python) S 11267 11267 10986 0 -1 4202560 900 0 0 0 1 0 0 0 18 0 1 0 617021222 312520704 7060 33554432000 4194304 5683644 140736178709536 18446744073709551615 216735198790 0 0 16781312 514 4026605094962168069 0 0 17 5 0 0 0
/proc/11279/statm: 76299 7060 372 364 0 12211 0
[pid=11281] ppid=11279 vsize=423400 CPUtime=10.02 cores=1,3,5,7
/proc/11281/stat : 11281 (adaptg2wsat2009) R 11279 11281 11281 0 -1 4202496 2626 0 0 0 1002 0 0 0 25 0 1 0 617021223 433561600 2359 33554432000 134512640 135107411 4287144208 18446744073709551615 134531456 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/11281/statm: 105850 2359 59 146 0 104726 0
[pid=11282] ppid=11267 vsize=316060 CPUtime=0 cores=1,3,5,7
/proc/11282/stat : 11282 (python) R 11267 11267 10986 0 -1 4202560 901 0 0 0 0 0 0 0 18 0 1 0 617021225 323645440 7063 33554432000 4194304 5683644 140736178709536 18446744073709551615 216735221895 0 0 16781312 514 0 0 0 17 1 0 0 0
/proc/11282/statm: 79015 7063 371 364 0 14924 0
[pid=11284] ppid=11282 vsize=423400 CPUtime=9.99 cores=1,3,5,7
/proc/11284/stat : 11284 (adaptg2wsat2009) R 11282 11284 11284 0 -1 4202496 2620 0 0 0 999 0 0 0 25 0 1 0 617021226 433561600 2359 33554432000 134512640 135107411 4292305360 18446744073709551615 134531600 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/11284/statm: 105850 2359 59 146 0 104726 0
[pid=11267/tid=11274] ppid=11265 vsize=325516 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11274/stat : 11274 (python) S 11265 11267 10986 0 -1 4202560 24 932 0 0 0 0 0 0 20 0 5 0 617021217 333328384 8578 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 3 0 0 0
[pid=11267/tid=11277] ppid=11265 vsize=325516 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11277/stat : 11277 (python) S 11265 11267 10986 0 -1 4202560 22 932 0 0 0 0 0 0 20 0 5 0 617021220 333328384 8578 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 5 0 0 0
[pid=11267/tid=11280] ppid=11265 vsize=325516 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11280/stat : 11280 (python) S 11265 11267 10986 0 -1 4202560 19 932 0 0 0 0 0 0 22 0 5 0 617021222 333328384 8578 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 5 0 0 0
[pid=11267/tid=11283] ppid=11265 vsize=325516 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11283/stat : 11283 (python) S 11265 11267 10986 0 -1 4202560 26 932 0 0 0 0 0 0 24 0 5 0 617021226 333328384 8578 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 42.56
Current children cumulated vsize (KiB) 4414184

[startup+25.5015 s]
/proc/loadavg: 8.03 7.07 3.94 9/267 11290
/proc/meminfo: memFree=22889356/32951124 swapFree=67102140/67111528
[pid=11267] ppid=11265 vsize=325516 CPUtime=2.39 cores=1,3,5,7
/proc/11267/stat : 11267 (python) S 11265 11267 10986 0 -1 4202496 16958 932 1 0 227 12 0 0 24 0 5 0 617020959 333328384 8578 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747005387 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/11267/statm: 81379 8578 1883 364 0 17288 0
[pid=11273] ppid=11267 vsize=284988 CPUtime=0.03 cores=1,3,5,7
/proc/11273/stat : 11273 (python) S 11267 11267 10986 0 -1 4202560 898 0 0 0 2 1 0 0 18 0 1 0 617021216 291827712 6923 33554432000 4194304 5683644 140736178709536 18446744073709551615 216735198735 0 0 16781312 514 4026605094962168069 0 0 17 5 0 0 0
/proc/11273/statm: 71247 6923 378 364 0 7165 0
[pid=11275] ppid=11273 vsize=1020308 CPUtime=22.92 cores=1,3,5,7
/proc/11275/stat : 11275 (gnovelty+2) R 11273 11275 11275 0 -1 4202496 1154 0 0 0 2292 0 0 0 25 0 1 0 617021217 1044795392 892 33554432000 134512640 135663704 4286752544 18446744073709551615 134524918 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/11275/statm: 255077 892 113 282 0 254793 0
[pid=11276] ppid=11267 vsize=295008 CPUtime=0.03 cores=1,3,5,7
/proc/11276/stat : 11276 (python) S 11267 11267 10986 0 -1 4202560 906 0 0 0 2 1 0 0 18 0 1 0 617021219 302088192 7056 33554432000 4194304 5683644 140736178709536 18446744073709551615 216735198790 0 0 16781312 514 4026605094962168069 0 0 17 1 0 0 0
/proc/11276/statm: 73752 7056 372 364 0 9667 0
[pid=11278] ppid=11276 vsize=1020308 CPUtime=22.72 cores=1,3,5,7
/proc/11278/stat : 11278 (gnovelty+2) R 11276 11278 11278 0 -1 4202496 1155 0 0 0 2272 0 0 0 25 0 1 0 617021220 1044795392 892 33554432000 134512640 135663704 4286897936 18446744073709551615 134525678 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/11278/statm: 255077 892 113 282 0 254793 0
[pid=11279] ppid=11267 vsize=305196 CPUtime=0.03 cores=1,3,5,7
/proc/11279/stat : 11279 (python) S 11267 11267 10986 0 -1 4202560 905 0 0 0 2 1 0 0 18 0 1 0 617021222 312520704 7060 33554432000 4194304 5683644 140736178709536 18446744073709551615 216735198790 0 0 16781312 514 4026605094962168069 0 0 17 5 0 0 0
/proc/11279/statm: 76299 7060 372 364 0 12211 0
[pid=11281] ppid=11279 vsize=423400 CPUtime=22.78 cores=1,3,5,7
/proc/11281/stat : 11281 (adaptg2wsat2009) R 11279 11281 11281 0 -1 4202496 2626 0 0 0 2278 0 0 0 25 0 1 0 617021223 433561600 2359 33554432000 134512640 135107411 4287144208 18446744073709551615 134531709 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/11281/statm: 105850 2359 59 146 0 104726 0
[pid=11282] ppid=11267 vsize=316060 CPUtime=0.02 cores=1,3,5,7
/proc/11282/stat : 11282 (python) S 11267 11267 10986 0 -1 4202560 906 0 0 0 1 1 0 0 18 0 1 0 617021225 323645440 7064 33554432000 4194304 5683644 140736178709536 18446744073709551615 216735198790 0 0 16781312 514 4026605094962168071 0 0 17 1 0 0 0
/proc/11282/statm: 79015 7064 372 364 0 14924 0
[pid=11284] ppid=11282 vsize=423400 CPUtime=22.75 cores=1,3,5,7
/proc/11284/stat : 11284 (adaptg2wsat2009) R 11282 11284 11284 0 -1 4202496 2620 0 0 0 2275 0 0 0 25 0 1 0 617021226 433561600 2359 33554432000 134512640 135107411 4292305360 18446744073709551615 134531613 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/11284/statm: 105850 2359 59 146 0 104726 0
[pid=11267/tid=11274] ppid=11265 vsize=325516 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11274/stat : 11274 (python) S 11265 11267 10986 0 -1 4202560 24 932 0 0 0 0 0 0 20 0 5 0 617021217 333328384 8578 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 3 0 0 0
[pid=11267/tid=11277] ppid=11265 vsize=325516 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11277/stat : 11277 (python) S 11265 11267 10986 0 -1 4202560 22 932 0 0 0 0 0 0 20 0 5 0 617021220 333328384 8578 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 5 0 0 0
[pid=11267/tid=11280] ppid=11265 vsize=325516 CPUtime=0 cores=1,3,5,7

################
# More data... #
################

/proc/11282/stat : 11282 (python) S 11267 11267 10986 0 -1 4202560 1093 0 0 0 7 8 0 0 19 0 2 0 617021225 323645440 7093 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/11282/statm: 79015 7093 400 364 0 14924 0
[pid=11284] ppid=11282 vsize=423400 CPUtime=99.83 cores=1,3,5,7
/proc/11284/stat : 11284 (adaptg2wsat2009) T 11282 11284 11284 0 -1 4202496 2620 0 0 0 9983 0 0 0 25 0 1 0 617021226 433561600 2359 33554432000 134512640 135107411 4292305360 18446744073709551615 134532318 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/11284/statm: 105850 2359 59 146 0 104726 0
[pid=11282/tid=11300] ppid=11267 vsize=316060 CPUtime=0 cores=1,3,5,7
/proc/11282/task/11300/stat : 11300 (python) S 11267 11267 10986 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 617031242 323645440 7093 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 514 18446604471143335808 0 0 -1 1 0 0 0
[pid=11295] ppid=11267 vsize=326216 CPUtime=0.15 cores=1,3,5,7
/proc/11295/stat : 11295 (python) S 11267 11267 10986 0 -1 4202560 1110 0 0 0 7 8 0 0 19 0 2 0 617031239 334045184 7097 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/11295/statm: 81554 7097 400 364 0 17460 0
[pid=11297] ppid=11295 vsize=423400 CPUtime=98.51 cores=1,3,5,7
/proc/11297/stat : 11297 (adaptg2wsat2009) T 11295 11297 11297 0 -1 4202496 2623 0 0 0 9851 0 0 0 25 0 1 0 617031239 433561600 2358 33554432000 134512640 135107411 4294766672 18446744073709551615 134531239 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/11297/statm: 105850 2358 59 146 0 104726 0
[pid=11295/tid=11366] ppid=11267 vsize=326216 CPUtime=0 cores=1,3,5,7
/proc/11295/task/11366/stat : 11366 (python) S 11267 11267 10986 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 617041254 334045184 7097 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 514 18446604471121204160 0 0 -1 1 0 0 0
[pid=11301] ppid=11267 vsize=336132 CPUtime=0.15 cores=1,3,5,7
/proc/11301/stat : 11301 (python) S 11267 11267 10986 0 -1 4202560 1122 0 0 0 7 8 0 0 18 0 2 0 617031243 344199168 7152 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/11301/statm: 84033 7152 400 364 0 19936 0
[pid=11303] ppid=11301 vsize=1020308 CPUtime=99.9 cores=1,3,5,7
/proc/11303/stat : 11303 (gnovelty+2) T 11301 11303 11303 0 -1 4202496 1156 0 0 0 9990 0 0 0 25 0 1 0 617031244 1044795392 892 33554432000 134512640 135663704 4293358480 18446744073709551615 134524894 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/11303/statm: 255077 892 113 282 0 254793 0
[pid=11301/tid=11371] ppid=11267 vsize=336132 CPUtime=0 cores=1,3,5,7
/proc/11301/task/11371/stat : 11371 (python) S 11267 11267 10986 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 617041265 344199168 7152 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 514 18446604458000829504 0 0 -1 3 0 0 0
[pid=11304] ppid=11267 vsize=346244 CPUtime=0.15 cores=1,3,5,7
/proc/11304/stat : 11304 (python) S 11267 11267 10986 0 -1 4202560 1169 0 0 0 8 7 0 0 19 0 2 0 617031247 354553856 7161 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/11304/statm: 86561 7161 405 364 0 22461 0
[pid=11306] ppid=11304 vsize=253036 CPUtime=99.85 cores=1,3,5,7
/proc/11306/stat : 11306 (FH) T 11304 11306 11306 0 -1 4202496 2495 0 0 0 9985 0 0 0 25 0 1 0 617031247 259108864 2234 33554432000 134512640 134936988 4286977984 18446744073709551615 134552507 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/11306/statm: 63259 2234 41 104 0 62763 0
[pid=11304/tid=11370] ppid=11267 vsize=346244 CPUtime=0 cores=1,3,5,7
/proc/11304/task/11370/stat : 11370 (python) S 11267 11267 10986 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 617041264 354553856 7161 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 514 18446604456345953408 0 0 -1 1 0 0 0
[pid=11307] ppid=11267 vsize=357188 CPUtime=0.15 cores=1,3,5,7
/proc/11307/stat : 11307 (python) S 11267 11267 10986 0 -1 4202560 1162 0 0 0 8 7 0 0 19 0 2 0 617031251 365760512 7166 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/11307/statm: 89297 7166 406 364 0 25194 0
[pid=11309] ppid=11307 vsize=423400 CPUtime=100.13 cores=1,3,5,7
/proc/11309/stat : 11309 (adaptg2wsat2009) T 11307 11309 11309 0 -1 4202496 2629 0 0 0 10013 0 0 0 25 0 1 0 617031251 433561600 2359 33554432000 134512640 135107411 4287841232 18446744073709551615 134531613 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/11309/statm: 105850 2359 59 146 0 104726 0
[pid=11307/tid=11372] ppid=11267 vsize=357188 CPUtime=0 cores=1,3,5,7
/proc/11307/task/11372/stat : 11372 (python) S 11267 11267 10986 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 617041266 365760512 7166 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 514 18446604458000826816 0 0 -1 3 0 0 0
[pid=11367] ppid=11267 vsize=366960 CPUtime=0.16 cores=1,3,5,7
/proc/11367/stat : 11367 (python) S 11267 11267 10986 0 -1 4202560 1136 0 0 0 7 9 0 0 19 0 2 0 617041258 375767040 7214 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/11367/statm: 91740 7214 399 364 0 27634 0
[pid=11369] ppid=11367 vsize=423400 CPUtime=99.81 cores=1,3,5,7
/proc/11369/stat : 11369 (adaptg2wsat2009) T 11367 11369 11369 0 -1 4202496 2629 0 0 0 9981 0 0 0 25 0 1 0 617041258 433561600 2358 33554432000 134512640 135107411 4293501936 18446744073709551615 134532364 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/11369/statm: 105850 2358 59 146 0 104726 0
[pid=11367/tid=11404] ppid=11267 vsize=366960 CPUtime=0 cores=1,3,5,7
/proc/11367/task/11404/stat : 11404 (python) S 11267 11267 10986 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 617051274 375767040 7214 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 514 18446604459687809088 0 0 -1 3 0 0 0
[pid=11373] ppid=11267 vsize=377408 CPUtime=0.16 cores=1,3,5,7
/proc/11373/stat : 11373 (python) S 11267 11267 10986 0 -1 4202560 1143 0 0 0 7 9 0 0 19 0 2 0 617041268 386465792 7219 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/11373/statm: 94352 7219 400 364 0 30243 0
[pid=11375] ppid=11373 vsize=253036 CPUtime=97.12 cores=1,3,5,7
/proc/11375/stat : 11375 (FH) T 11373 11375 11375 0 -1 4202496 2502 0 0 0 9712 0 0 0 25 0 1 0 617041269 259108864 2233 33554432000 134512640 134936988 4290534176 18446744073709551615 134552455 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/11375/statm: 63259 2233 41 104 0 62763 0
[pid=11373/tid=11400] ppid=11267 vsize=377408 CPUtime=0 cores=1,3,5,7
/proc/11373/task/11400/stat : 11400 (python) S 11267 11267 10986 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 617051185 386465792 7219 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 514 18446604456345952512 0 0 -1 7 0 0 0
[pid=11376] ppid=11267 vsize=387604 CPUtime=0.51 cores=1,3,5,7
/proc/11376/stat : 11376 (python) S 11267 11267 10986 0 -1 4202560 1181 0 0 0 25 26 0 0 18 0 2 0 617041273 396906496 7223 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/11376/statm: 96901 7223 400 364 0 32789 0
[pid=11378] ppid=11376 vsize=268428 CPUtime=291.45 cores=1,3,5,7
/proc/11378/stat : 11378 (hybridGM3) T 11376 11378 11378 0 -1 4202496 6185 0 0 0 29145 0 0 0 25 0 1 0 617041273 274870272 5919 33554432000 134512640 135204385 4291447440 18446744073709551615 134538427 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/11378/statm: 67107 5919 63 169 0 66937 0
[pid=11376/tid=11431] ppid=11267 vsize=387604 CPUtime=0 cores=1,3,5,7
/proc/11376/task/11431/stat : 11431 (python) S 11267 11267 10986 0 -1 4202560 46 0 0 0 0 0 0 0 18 0 2 0 617060913 396906496 7223 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 514 18446604471128781056 0 0 -1 3 0 0 0
[pid=11379] ppid=11267 vsize=397988 CPUtime=0.16 cores=1,3,5,7
/proc/11379/stat : 11379 (python) S 11267 11267 10986 0 -1 4202560 1153 0 0 0 9 7 0 0 19 0 2 0 617041277 407539712 7227 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/11379/statm: 99497 7227 400 364 0 35382 0
[pid=11381] ppid=11379 vsize=1020308 CPUtime=100.18 cores=1,3,5,7
/proc/11381/stat : 11381 (gnovelty+2) T 11379 11381 11381 0 -1 4202496 1164 0 0 0 10018 0 0 0 25 0 1 0 617041277 1044795392 892 33554432000 134512640 135663704 4288466736 18446744073709551615 134524897 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/11381/statm: 255077 892 113 282 0 254793 0
[pid=11379/tid=11408] ppid=11267 vsize=397988 CPUtime=0 cores=1,3,5,7
/proc/11379/task/11408/stat : 11408 (python) S 11267 11267 10986 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 617051298 407539712 7227 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 514 18446604466106737984 0 0 -1 7 0 0 0
[pid=11401] ppid=11267 vsize=408220 CPUtime=0.36 cores=1,3,5,7
/proc/11401/stat : 11401 (python) S 11267 11267 10986 0 -1 4202560 1174 0 0 0 20 16 0 0 18 0 2 0 617051189 418017280 7231 33554432000 4194304 5683644 140736178709536 18446744073709551615 216735198790 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/11401/statm: 102055 7231 400 364 0 37937 0
[pid=11403] ppid=11401 vsize=268420 CPUtime=195.41 cores=1,3,5,7
/proc/11403/stat : 11403 (hybridGM3) R 11401 11403 11403 0 -1 4202496 6189 0 0 0 19541 0 0 0 25 0 1 0 617051190 274862080 5919 33554432000 134512640 135204385 4289073536 18446744073709551615 134537711 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/11403/statm: 67105 5919 63 169 0 66935 0
[pid=11401/tid=11453] ppid=11267 vsize=408220 CPUtime=0 cores=1,3,5,7
/proc/11401/task/11453/stat : 11453 (python) S 11267 11267 10986 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 617070641 418017280 7231 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 514 18446604463215220736 0 0 -1 7 0 0 0
[pid=11405] ppid=11267 vsize=418620 CPUtime=0.18 cores=1,3,5,7
/proc/11405/stat : 11405 (python) S 11267 11267 10986 0 -1 4202560 1164 0 0 0 10 8 0 0 19 0 2 0 617051279 428666880 7235 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/11405/statm: 104655 7235 400 364 0 40534 0
[pid=11407] ppid=11405 vsize=423400 CPUtime=97.86 cores=1,3,5,7
/proc/11407/stat : 11407 (adaptg2wsat2009) T 11405 11407 11407 0 -1 4202496 2630 0 0 0 9786 0 0 0 25 0 1 0 617051279 433561600 2359 33554432000 134512640 135107411 4292157872 18446744073709551615 134532277 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/11407/statm: 105850 2359 59 146 0 104726 0
[pid=11405/tid=11432] ppid=11267 vsize=418620 CPUtime=0 cores=1,3,5,7
/proc/11405/task/11432/stat : 11432 (python) S 11267 11267 10986 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 617061297 428666880 7235 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 514 18446604454223470656 0 0 -1 5 0 0 0
[pid=11409] ppid=11267 vsize=428844 CPUtime=0.36 cores=1,3,5,7
/proc/11409/stat : 11409 (python) S 11267 11267 10986 0 -1 4202560 1213 0 0 0 19 17 0 0 19 0 2 0 617051303 439136256 7247 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/11409/statm: 107211 7247 408 364 0 43087 0
[pid=11411] ppid=11409 vsize=268424 CPUtime=194.43 cores=1,3,5,7
/proc/11411/stat : 11411 (hybridGM3) T 11409 11411 11411 0 -1 4202496 6194 0 0 0 19442 1 0 0 25 0 1 0 617051304 274866176 5918 33554432000 134512640 135204385 4294532608 18446744073709551615 134537792 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/11411/statm: 67106 5918 63 169 0 66936 0
[pid=11409/tid=11454] ppid=11267 vsize=428844 CPUtime=0 cores=1,3,5,7
/proc/11409/task/11454/stat : 11454 (python) S 11267 11267 10986 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 617070748 439136256 7247 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 514 18446604459687810880 0 0 -1 7 0 0 0
[pid=11433] ppid=11267 vsize=439144 CPUtime=0.17 cores=1,3,5,7
/proc/11433/stat : 11433 (python) S 11267 11267 10986 0 -1 4202560 1082 0 0 0 10 7 0 0 18 0 1 0 617061302 449683456 7227 33554432000 4194304 5683644 140736178709536 18446744073709551615 216735198790 0 0 16781312 514 4026605094962168071 0 0 17 7 0 0 0
/proc/11433/statm: 109786 7227 380 364 0 45659 0
[pid=11435] ppid=11433 vsize=268424 CPUtime=91.79 cores=1,3,5,7
/proc/11435/stat : 11435 (hybridGM3) R 11433 11435 11435 0 -1 4202496 6194 0 0 0 9179 0 0 0 25 0 1 0 617061303 274866176 5918 33554432000 134512640 135204385 4287425472 18446744073709551615 134538041 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/11435/statm: 67106 5918 63 169 0 66936 0
[pid=11450] ppid=11267 vsize=449820 CPUtime=0 cores=1,3,5,7
/proc/11450/stat : 11450 (python) S 11267 11267 10986 0 -1 4202560 830 0 0 0 0 0 0 0 18 0 1 0 617070439 460615680 7202 33554432000 4194304 5683644 140736178709536 18446744073709551615 216735198790 0 0 16781312 514 16234970610738888968 0 0 17 3 0 0 0
/proc/11450/statm: 112455 7202 351 364 0 48325 0
[pid=11452] ppid=11450 vsize=461508 CPUtime=3.33 cores=1,3,5,7
/proc/11452/stat : 11452 (TNM) R 11450 11452 11452 0 -1 4202496 2468 0 0 0 333 0 0 0 25 0 1 0 617070439 472584192 2195 33554432000 134512640 135008748 4287068608 18446744073709551615 134527773 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/11452/statm: 115377 2195 46 122 0 114277 0
[pid=11455] ppid=11267 vsize=459896 CPUtime=0 cores=1,3,5,7
/proc/11455/stat : 11455 (python) S 11267 11267 10986 0 -1 4202560 818 0 0 0 0 0 0 0 19 0 1 0 617070753 470933504 7302 33554432000 4194304 5683644 140736178709536 18446744073709551615 216735198790 0 0 16781312 514 16234966762448224255 0 0 17 7 0 0 0
/proc/11455/statm: 114974 7302 347 364 0 50841 0
[pid=11457] ppid=11455 vsize=248616 CPUtime=0.27 cores=1,3,5,7
/proc/11457/stat : 11457 (FH) R 11455 11457 11457 0 -1 4202496 1485 0 0 0 27 0 0 0 21 0 1 0 617070753 254582784 1213 33554432000 134512640 134936988 4293065328 18446744073709551615 134523622 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/11457/statm: 62154 1213 37 104 0 61658 0
[pid=11267/tid=11274] ppid=11265 vsize=469928 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11274/stat : 11274 (python) S 11265 11267 10986 0 -1 4202560 24 932 0 0 0 0 0 0 20 0 19 0 617021217 481206272 8870 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 3 0 0 0
[pid=11267/tid=11277] ppid=11265 vsize=469928 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11277/stat : 11277 (python) S 11265 11267 10986 0 -1 4202560 22 932 0 0 0 0 0 0 20 0 19 0 617021220 481206272 8870 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 5 0 0 0
[pid=11267/tid=11280] ppid=11265 vsize=469928 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11280/stat : 11280 (python) S 11265 11267 10986 0 -1 4202560 19 932 0 0 0 0 0 0 22 0 19 0 617021222 481206272 8870 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 5 0 0 0
[pid=11267/tid=11283] ppid=11265 vsize=469928 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11283/stat : 11283 (python) S 11265 11267 10986 0 -1 4202560 26 932 0 0 0 0 0 0 24 0 19 0 617021226 481206272 8870 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 1 0 0 0
[pid=11267/tid=11296] ppid=11265 vsize=469928 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11296/stat : 11296 (python) S 11265 11267 10986 0 -1 4202560 31 932 0 0 0 0 0 0 20 0 19 0 617031239 481206272 8870 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 1 0 0 0
[pid=11267/tid=11302] ppid=11265 vsize=469928 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11302/stat : 11302 (python) S 11265 11267 10986 0 -1 4202560 26 932 0 0 0 0 0 0 21 0 19 0 617031244 481206272 8870 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 3 0 0 0
[pid=11267/tid=11305] ppid=11265 vsize=469928 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11305/stat : 11305 (python) S 11265 11267 10986 0 -1 4202560 24 932 0 0 0 0 0 0 23 0 19 0 617031247 481206272 8870 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 1 0 0 0
[pid=11267/tid=11308] ppid=11265 vsize=469928 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11308/stat : 11308 (python) S 11265 11267 10986 0 -1 4202560 30 932 0 0 0 0 0 0 20 0 19 0 617031251 481206272 8870 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 3 0 0 0
[pid=11267/tid=11368] ppid=11265 vsize=469928 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11368/stat : 11368 (python) S 11265 11267 10986 0 -1 4202560 29 932 0 0 0 0 0 0 20 0 19 0 617041258 481206272 8870 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 3 0 0 0
[pid=11267/tid=11374] ppid=11265 vsize=469928 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11374/stat : 11374 (python) S 11265 11267 10986 0 -1 4202560 24 932 0 0 0 0 0 0 21 0 19 0 617041269 481206272 8870 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 1 0 0 0
[pid=11267/tid=11377] ppid=11265 vsize=469928 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11377/stat : 11377 (python) S 11265 11267 10986 0 -1 4202560 30 932 0 0 0 0 0 0 18 0 19 0 617041273 481206272 8870 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 3 0 0 0
[pid=11267/tid=11380] ppid=11265 vsize=469928 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11380/stat : 11380 (python) S 11265 11267 10986 0 -1 4202560 26 932 0 0 0 0 0 0 22 0 19 0 617041277 481206272 8870 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 7 0 0 0
[pid=11267/tid=11402] ppid=11265 vsize=469928 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11402/stat : 11402 (python) S 11265 11267 10986 0 -1 4202560 31 932 0 0 0 0 0 0 18 0 19 0 617051190 481206272 8870 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 7 0 0 0
[pid=11267/tid=11406] ppid=11265 vsize=469928 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11406/stat : 11406 (python) S 11265 11267 10986 0 -1 4202560 28 932 0 0 0 0 0 0 20 0 19 0 617051279 481206272 8870 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 7 0 0 0
[pid=11267/tid=11410] ppid=11265 vsize=469928 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11410/stat : 11410 (python) S 11265 11267 10986 0 -1 4202560 29 932 0 0 0 0 0 0 20 0 19 0 617051304 481206272 8870 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 7 0 0 0
[pid=11267/tid=11434] ppid=11265 vsize=469928 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11434/stat : 11434 (python) S 11265 11267 10986 0 -1 4202560 32 932 0 0 0 0 0 0 20 0 19 0 617061303 481206272 8870 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 7 0 0 0
[pid=11267/tid=11451] ppid=11265 vsize=469928 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11451/stat : 11451 (python) S 11265 11267 10986 0 -1 4202560 27 932 0 0 0 0 0 0 20 0 19 0 617070439 481206272 8870 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 7 0 0 0
[pid=11267/tid=11456] ppid=11265 vsize=469928 CPUtime=0 cores=1,3,5,7
/proc/11267/task/11456/stat : 11456 (python) S 11265 11267 10986 0 -1 4202560 24 932 0 0 0 0 0 0 20 0 19 0 617070753 481206272 8870 33554432000 4194304 5683644 140736178709536 18446744073709551615 216747002289 0 0 16781312 2 18446604468433845184 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 1974.93
Current children cumulated vsize (KiB) 16093232

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 11267 and gives
#  childrusage.ru_utime.tv_sec=2
#  childrusage.ru_utime.tv_usec=946552
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=175973
# CPU time returned by wait4() is 3.12252
# while last known CPU time is 1974.93
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 498.344
CPU time (s): 1974.93
CPU user time (s): 1973.17
CPU system time (s): 1.76
CPU usage (%): 396.299
Max. virtual memory (cumulated for all children) (KiB): 16093232

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2.94655
system time used= 0.175973
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 31917
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 3073
involuntary context switches= 74

runsolver used 2.78558 second user time and 5.75612 second system time

The end

Launcher Data 

Begin job on node119 at 2011-05-01 18:38:23
IDJOB=3317165
IDBENCH=82630
IDSOLVER=1705
FILE ID=node119/3317165-1304267903
RUNJOBID= node119-1304267283-11005
PBS_JOBID= 13173960
Free space on /tmp= 73012 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k5-r20-v2000-c40000-S1264065752-051-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317165-1304267903/watcher-3317165-1304267903 -o /tmp/evaluation-result-3317165-1304267903/solver-3317165-1304267903 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317165-1304267903.cnf -seed 1755707011 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= fafa522767a4e86ea9e03f34c2fb7311
RANDOM SEED=1755707011

node119.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.63
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      22962640 kB
Buffers:       1964296 kB
Cached:        7088072 kB
SwapCached:       3688 kB
Active:         928104 kB
Inactive:      8463520 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      22962640 kB
SwapTotal:    67111528 kB
SwapFree:     67102140 kB
Dirty:           33104 kB
Writeback:           0 kB
AnonPages:      339052 kB
Mapped:          22860 kB
Slab:           517676 kB
PageTables:      18256 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 18004480 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72972 MiB
End job on node119 at 2011-05-01 18:46:43