Trace number 3378375

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (MO) 2766.25 696.335

General information on the benchmark

Namerandom/medium/
unif-k7-r89-v90-c8010-S764895749-094.UNKNOWN.cnf
MD5SUM86d70d98b4a1fa931b287202356d4839
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables90
Number of clauses8010
Sum of the clauses size56070
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 58010

Solver Data 

0.29/0.46	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.29/0.46	c INFO: solving HOME/instance-3378375-1305465121.cnf
0.29/0.46	c INFO: computing task features
0.38/0.52	c INFO: computed variable-clause graph statistics
0.38/0.52	c INFO: computed clause balance statistics
0.58/0.80	c INFO: computed variable balance statistics
0.78/0.91	c INFO: computed small-clause counts
0.78/0.91	c INFO: computed Horn-clause counts
0.78/0.92	c INFO: computed variable graph statistics
0.78/0.92	c INFO: collected features for HOME/instance-3378375-1305465121.cnf in 0.45 s
0.78/0.93	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.46)
0.78/0.94	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.11)
0.78/0.95	c INFO: running adaptg2wsat2009++@0 for 97 with 1199 remaining (b = 0.06)
0.78/0.96	c INFO: running adaptg2wsat2009++@0 for 97 with 1199 remaining (b = 0.05)
400.42/101.27	c INFO: running adaptg2wsat2009++@0 for 97 with 1099 remaining (b = 0.04)
400.42/101.29	c INFO: running gnovelty+2@0 for 97 with 1099 remaining (b = 0.03)
400.73/101.30	c INFO: running FH@0 for 97 with 1099 remaining (b = 0.02)
400.73/101.31	c INFO: running adaptg2wsat2009++@0 for 97 with 1099 remaining (b = 0.02)
796.40/200.70	c INFO: running adaptg2wsat2009++@0 for 97 with 999 remaining (b = 0.02)
800.37/201.61	c INFO: running hybridGM3@0 for 194 with 998 remaining (b = 0.03)
800.37/201.67	c INFO: running FH@0 for 97 with 998 remaining (b = 0.01)
800.37/201.70	c INFO: running hybridGM3@0 for 194 with 998 remaining (b = 0.02)
1195.45/301.04	c INFO: running gnovelty+2@0 for 97 with 899 remaining (b = 0.01)
1199.40/302.02	c INFO: running adaptg2wsat2009++@0 for 97 with 898 remaining (b = 0.01)
1578.24/397.38	c INFO: running hybridGM3@97 for 97 with 803 remaining (b = 0.01)
1581.89/398.28	c INFO: running hybridGM3@100 for 97 with 802 remaining (b = 0.01)
1586.24/399.37	c INFO: running hybridGM3@0 for 194 with 801 remaining (b = 0.02)
1594.55/401.42	c INFO: running adaptg2wsat2009++@98 for 97 with 799 remaining (b = 0.01)
1961.05/493.61	c INFO: running FH@0 for 97 with 706 remaining (b = 0.01)
1981.29/498.74	c INFO: running hybridGM3@294 for 97 with 701 remaining (b = 0.01)
1981.29/498.77	c INFO: running march_hi@0 for 388 with 701 remaining (b = 0.04)
2359.69/593.97	c INFO: running adaptg2wsat2009++@99 for 97 with 606 remaining (b = 0.01)
2360.10/594.08	c INFO: running hybridGM3@389 for 97 with 606 remaining (b = 0.01)
2368.01/596.07	c INFO: running hybridGM3@99 for 97 with 604 remaining (b = 0.01)
2746.73/691.32	c INFO: running adaptg2wsat2009++@97 for 97 with 509 remaining (b = 0.01)
2759.04/694.44	c INFO: running gnovelty+2@0 for 97 with 506 remaining (b = 0.01)
2763.00/695.47	c INFO: running hybridGM3@0 for 194 with 504 remaining (b = 0.02)
2766.25/696.33	c INFO: running march_hi@0 for 388 with 504 remaining (b = 0.03)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378375-1305465121/watcher-3378375-1305465121 -o /tmp/evaluation-result-3378375-1305465121/solver-3378375-1305465121 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378375-1305465121.cnf -seed 969329219 -budget 4800 -cores 4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.00 8.13 7.62 5/440 21940
/proc/meminfo: memFree=21986316/32951124 swapFree=51873460/67111528
[pid=21940] ppid=21938 vsize=16236 CPUtime=0 cores=0,2,4,6
/proc/21940/stat : 21940 (python) R 21938 21940 20819 0 -1 4202496 1049 0 0 0 0 0 0 0 25 0 1 0 638928425 16625664 921 33554432000 4194304 5683644 140735096548976 18446744073709551615 212039349199 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/21940/statm: 4059 921 381 364 0 546 0

[startup+0.0667801 s]
/proc/loadavg: 8.00 8.13 7.62 5/440 21940
/proc/meminfo: memFree=21986316/32951124 swapFree=51873460/67111528
[pid=21940] ppid=21938 vsize=50332 CPUtime=0.03 cores=0,2,4,6
/proc/21940/stat : 21940 (python) R 21938 21940 20819 0 -1 4202496 2002 0 0 0 3 0 0 0 25 0 1 0 638928425 51539968 1762 33554432000 4194304 5683644 140735096548976 18446744073709551615 4540219 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/21940/statm: 12583 1762 480 364 0 1283 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 50332

[startup+0.100748 s]
/proc/loadavg: 8.00 8.13 7.62 5/440 21940
/proc/meminfo: memFree=21986316/32951124 swapFree=51873460/67111528
[pid=21940] ppid=21938 vsize=104228 CPUtime=0.06 cores=0,2,4,6
/proc/21940/stat : 21940 (python) R 21938 21940 20819 0 -1 4202496 3083 0 0 0 5 1 0 0 17 0 1 0 638928425 106729472 2764 33554432000 4194304 5683644 140735096548976 18446744073709551615 4540219 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/21940/statm: 26057 2765 847 364 0 1805 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 104228

[startup+0.300714 s]
/proc/loadavg: 8.00 8.13 7.62 5/440 21940
/proc/meminfo: memFree=21986316/32951124 swapFree=51873460/67111528
[pid=21940] ppid=21938 vsize=187688 CPUtime=0.22 cores=0,2,4,6
/proc/21940/stat : 21940 (python) S 21938 21940 20819 0 -1 4202496 6823 934 1 0 16 6 0 0 18 0 1 0 638928425 192192512 4882 33554432000 4194304 5683644 140735096548976 18446744073709551615 212039653376 0 2147483391 16781312 2 18446744071699199166 0 0 17 4 0 0 0
/proc/21940/statm: 46922 4882 1399 364 0 3723 0
Current children cumulated CPU time (s) 0.22
Current children cumulated vsize (KiB) 187688

[startup+0.700641 s]
/proc/loadavg: 8.00 8.13 7.62 5/440 21940
/proc/meminfo: memFree=21986316/32951124 swapFree=51873460/67111528
[pid=21940] ppid=21938 vsize=280232 CPUtime=0.58 cores=0,2,4,6
/proc/21940/stat : 21940 (python) R 21938 21940 20819 0 -1 4202496 9992 934 1 0 50 8 0 0 18 0 1 0 638928425 286957568 7443 33554432000 4194304 5683644 140735096548976 18446744073709551615 212039311921 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/21940/statm: 70058 7443 1826 364 0 5982 0
Current children cumulated CPU time (s) 0.58
Current children cumulated vsize (KiB) 280232

[startup+1.50043 s]
/proc/loadavg: 8.00 8.13 7.62 9/453 21957
/proc/meminfo: memFree=21943556/32951124 swapFree=51873460/67111528
[pid=21940] ppid=21938 vsize=320744 CPUtime=0.84 cores=0,2,4,6
/proc/21940/stat : 21940 (python) S 21938 21940 20819 0 -1 4202496 12930 934 1 0 75 9 0 0 24 0 5 0 638928425 328441856 7385 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051482059 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/21940/statm: 80186 7385 1883 364 0 16095 0
[pid=21946] ppid=21940 vsize=279412 CPUtime=0 cores=0,2,4,6
/proc/21946/stat : 21946 (python) S 21940 21940 20819 0 -1 4202560 719 0 0 0 0 0 0 0 20 0 1 0 638928518 286117888 5792 33554432000 4194304 5683644 140735096548976 18446744073709551615 212039675407 0 0 16781312 514 6032167634634539007 0 0 17 4 0 0 0
/proc/21946/statm: 69853 5792 354 364 0 5771 0
[pid=21948] ppid=21946 vsize=1018088 CPUtime=0.54 cores=0,2,4,6
/proc/21948/stat : 21948 (gnovelty+2) R 21946 21948 21948 0 -1 4202496 569 0 0 0 54 0 0 0 23 0 1 0 638928519 1042522112 307 33554432000 134512640 135663704 4292466848 18446744073709551615 134525243 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/21948/statm: 254522 307 113 282 0 254238 0
[pid=21949] ppid=21940 vsize=289988 CPUtime=0 cores=0,2,4,6
/proc/21949/stat : 21949 (python) S 21940 21940 20819 0 -1 4202560 731 0 0 0 0 0 0 0 19 0 1 0 638928519 296947712 5838 33554432000 4194304 5683644 140735096548976 18446744073709551615 212039675462 0 0 16781312 514 1696696529290526719 0 0 17 4 0 0 0
/proc/21949/statm: 72497 5838 348 364 0 8412 0
[pid=21951] ppid=21949 vsize=1018088 CPUtime=0.55 cores=0,2,4,6
/proc/21951/stat : 21951 (gnovelty+2) R 21949 21951 21951 0 -1 4202496 569 0 0 0 55 0 0 0 22 0 1 0 638928520 1042522112 307 33554432000 134512640 135663704 4290279632 18446744073709551615 134524897 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/21951/statm: 254522 307 113 282 0 254238 0
[pid=21952] ppid=21940 vsize=300244 CPUtime=0 cores=0,2,4,6
/proc/21952/stat : 21952 (python) S 21940 21940 20819 0 -1 4202560 717 0 0 0 0 0 0 0 21 0 1 0 638928520 307449856 5838 33554432000 4194304 5683644 140735096548976 18446744073709551615 212039675462 0 0 16781312 514 2914182181314101247 0 0 17 0 0 0 0
/proc/21952/statm: 75061 5838 344 364 0 10973 0
[pid=21954] ppid=21952 vsize=417048 CPUtime=0.54 cores=0,2,4,6
/proc/21954/stat : 21954 (adaptg2wsat2009) R 21952 21954 21954 0 -1 4202496 683 0 0 0 54 0 0 0 25 0 1 0 638928520 427057152 420 33554432000 134512640 135107411 4293591776 18446744073709551615 134523452 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/21954/statm: 104262 420 49 146 0 103138 0
[pid=21955] ppid=21940 vsize=310500 CPUtime=0 cores=0,2,4,6
/proc/21955/stat : 21955 (python) S 21940 21940 20819 0 -1 4202560 729 0 0 0 0 0 0 0 24 0 1 0 638928521 317952000 5842 33554432000 4194304 5683644 140735096548976 18446744073709551615 212039675462 0 0 16781312 514 13583115415432724479 0 0 17 4 0 0 0
/proc/21955/statm: 77625 5842 344 364 0 13534 0
[pid=21957] ppid=21955 vsize=417048 CPUtime=0.53 cores=0,2,4,6
/proc/21957/stat : 21957 (adaptg2wsat2009) R 21955 21957 21957 0 -1 4202496 682 0 0 0 53 0 0 0 25 0 1 0 638928521 427057152 418 33554432000 134512640 135107411 4289443536 18446744073709551615 134523424 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/21957/statm: 104262 418 49 146 0 103138 0
[pid=21940/tid=21947] ppid=21938 vsize=320744 CPUtime=0 cores=0,2,4,6
/proc/21940/task/21947/stat : 21947 (python) S 21938 21940 20819 0 -1 4202560 24 934 0 0 0 0 0 0 21 0 5 0 638928518 328441856 7385 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 2 0 0 0
[pid=21940/tid=21950] ppid=21938 vsize=320744 CPUtime=0 cores=0,2,4,6
/proc/21940/task/21950/stat : 21950 (python) S 21938 21940 20819 0 -1 4202560 24 934 0 0 0 0 0 0 20 0 5 0 638928519 328441856 7385 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 2 0 0 0
[pid=21940/tid=21953] ppid=21938 vsize=320744 CPUtime=0 cores=0,2,4,6
/proc/21940/task/21953/stat : 21953 (python) S 21938 21940 20819 0 -1 4202560 24 934 0 0 0 0 0 0 22 0 5 0 638928520 328441856 7385 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 0 0 0 0
[pid=21940/tid=21956] ppid=21938 vsize=320744 CPUtime=0 cores=0,2,4,6
/proc/21940/task/21956/stat : 21956 (python) S 21938 21940 20819 0 -1 4202560 28 934 0 0 0 0 0 0 24 0 5 0 638928521 328441856 7385 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3
Current children cumulated vsize (KiB) 4371160

[startup+3.10115 s]
/proc/loadavg: 8.00 8.13 7.62 9/453 21957
/proc/meminfo: memFree=21942812/32951124 swapFree=51873460/67111528
[pid=21940] ppid=21938 vsize=320744 CPUtime=0.84 cores=0,2,4,6
/proc/21940/stat : 21940 (python) S 21938 21940 20819 0 -1 4202496 12930 934 1 0 75 9 0 0 24 0 5 0 638928425 328441856 7385 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051482059 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/21940/statm: 80186 7385 1883 364 0 16095 0
[pid=21946] ppid=21940 vsize=279412 CPUtime=0 cores=0,2,4,6
/proc/21946/stat : 21946 (python) S 21940 21940 20819 0 -1 4202560 719 0 0 0 0 0 0 0 18 0 1 0 638928518 286117888 5792 33554432000 4194304 5683644 140735096548976 18446744073709551615 212039675407 0 0 16781312 514 6032167634634506499 0 0 17 4 0 0 0
/proc/21946/statm: 69853 5792 354 364 0 5771 0
[pid=21948] ppid=21946 vsize=1018088 CPUtime=2.13 cores=0,2,4,6
/proc/21948/stat : 21948 (gnovelty+2) R 21946 21948 21948 0 -1 4202496 569 0 0 0 213 0 0 0 25 0 1 0 638928519 1042522112 307 33554432000 134512640 135663704 4292466848 18446744073709551615 134546906 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/21948/statm: 254522 307 113 282 0 254238 0
[pid=21949] ppid=21940 vsize=289988 CPUtime=0 cores=0,2,4,6
/proc/21949/stat : 21949 (python) S 21940 21940 20819 0 -1 4202560 731 0 0 0 0 0 0 0 18 0 1 0 638928519 296947712 5838 33554432000 4194304 5683644 140735096548976 18446744073709551615 212039675462 0 0 16781312 514 1696696529290494209 0 0 17 4 0 0 0
/proc/21949/statm: 72497 5838 348 364 0 8412 0
[pid=21951] ppid=21949 vsize=1018088 CPUtime=2.15 cores=0,2,4,6
/proc/21951/stat : 21951 (gnovelty+2) R 21949 21951 21951 0 -1 4202496 569 0 0 0 215 0 0 0 25 0 1 0 638928520 1042522112 307 33554432000 134512640 135663704 4290279632 18446744073709551615 134545583 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/21951/statm: 254522 307 113 282 0 254238 0
[pid=21952] ppid=21940 vsize=300244 CPUtime=0 cores=0,2,4,6
/proc/21952/stat : 21952 (python) S 21940 21940 20819 0 -1 4202560 735 0 0 0 0 0 0 0 18 0 1 0 638928520 307449856 5842 33554432000 4194304 5683644 140735096548976 18446744073709551615 212039675462 0 0 16781312 514 2914182181314068737 0 0 17 0 0 0 0
/proc/21952/statm: 75061 5842 348 364 0 10973 0
[pid=21954] ppid=21952 vsize=417192 CPUtime=2.14 cores=0,2,4,6
/proc/21954/stat : 21954 (adaptg2wsat2009) R 21952 21954 21954 0 -1 4202496 759 0 0 0 214 0 0 0 25 0 1 0 638928520 427204608 496 33554432000 134512640 135107411 4293591776 18446744073709551615 134531239 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/21954/statm: 104298 496 59 146 0 103174 0
[pid=21955] ppid=21940 vsize=310500 CPUtime=0 cores=0,2,4,6
/proc/21955/stat : 21955 (python) S 21940 21940 20819 0 -1 4202560 748 0 0 0 0 0 0 0 18 0 1 0 638928521 317952000 5846 33554432000 4194304 5683644 140735096548976 18446744073709551615 212039675462 0 0 16781312 514 13583115415432691971 0 0 17 4 0 0 0
/proc/21955/statm: 77625 5846 348 364 0 13534 0
[pid=21957] ppid=21955 vsize=417192 CPUtime=2.13 cores=0,2,4,6
/proc/21957/stat : 21957 (adaptg2wsat2009) R 21955 21957 21957 0 -1 4202496 759 0 0 0 213 0 0 0 25 0 1 0 638928521 427204608 495 33554432000 134512640 135107411 4289443536 18446744073709551615 134531935 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/21957/statm: 104298 495 59 146 0 103174 0
[pid=21940/tid=21947] ppid=21938 vsize=320744 CPUtime=0 cores=0,2,4,6
/proc/21940/task/21947/stat : 21947 (python) S 21938 21940 20819 0 -1 4202560 24 934 0 0 0 0 0 0 21 0 5 0 638928518 328441856 7385 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 2 0 0 0
[pid=21940/tid=21950] ppid=21938 vsize=320744 CPUtime=0 cores=0,2,4,6
/proc/21940/task/21950/stat : 21950 (python) S 21938 21940 20819 0 -1 4202560 24 934 0 0 0 0 0 0 20 0 5 0 638928519 328441856 7385 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 2 0 0 0
[pid=21940/tid=21953] ppid=21938 vsize=320744 CPUtime=0 cores=0,2,4,6
/proc/21940/task/21953/stat : 21953 (python) S 21938 21940 20819 0 -1 4202560 24 934 0 0 0 0 0 0 22 0 5 0 638928520 328441856 7385 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 0 0 0 0
[pid=21940/tid=21956] ppid=21938 vsize=320744 CPUtime=0 cores=0,2,4,6
/proc/21940/task/21956/stat : 21956 (python) S 21938 21940 20819 0 -1 4202560 28 934 0 0 0 0 0 0 24 0 5 0 638928521 328441856 7385 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 9.39
Current children cumulated vsize (KiB) 4371448

[startup+6.30057 s]
/proc/loadavg: 8.16 8.16 7.64 6/436 21990
/proc/meminfo: memFree=21985036/32951124 swapFree=51873460/67111528
[pid=21940] ppid=21938 vsize=320744 CPUtime=0.84 cores=0,2,4,6
/proc/21940/stat : 21940 (python) S 21938 21940 20819 0 -1 4202496 12930 934 1 0 75 9 0 0 24 0 5 0 638928425 328441856 7385 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051482059 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/21940/statm: 80186 7385 1883 364 0 16095 0
[pid=21946] ppid=21940 vsize=279412 CPUtime=0.01 cores=0,2,4,6
/proc/21946/stat : 21946 (python) S 21940 21940 20819 0 -1 4202560 881 0 0 0 0 1 0 0 18 0 1 0 638928518 286117888 5815 33554432000 4194304 5683644 140735096548976 18446744073709551615 212039675407 0 0 16781312 514 6032167634634506499 0 0 17 2 0 0 0
/proc/21946/statm: 69853 5815 377 364 0 5771 0
[pid=21948] ppid=21946 vsize=1018088 CPUtime=5.28 cores=0,2,4,6
/proc/21948/stat : 21948 (gnovelty+2) R 21946 21948 21948 0 -1 4202496 569 0 0 0 528 0 0 0 25 0 1 0 638928519 1042522112 307 33554432000 134512640 135663704 4292466848 18446744073709551615 134524897 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/21948/statm: 254522 307 113 282 0 254238 0
[pid=21949] ppid=21940 vsize=289988 CPUtime=0.01 cores=0,2,4,6
/proc/21949/stat : 21949 (python) S 21940 21940 20819 0 -1 4202560 894 0 0 0 0 1 0 0 18 0 1 0 638928519 296947712 5861 33554432000 4194304 5683644 140735096548976 18446744073709551615 212039675462 0 0 16781312 514 1696696529290494209 0 0 17 4 0 0 0
/proc/21949/statm: 72497 5861 371 364 0 8412 0
[pid=21951] ppid=21949 vsize=1018088 CPUtime=5.35 cores=0,2,4,6
/proc/21951/stat : 21951 (gnovelty+2) R 21949 21951 21951 0 -1 4202496 569 0 0 0 535 0 0 0 25 0 1 0 638928520 1042522112 307 33554432000 134512640 135663704 4290279632 18446744073709551615 134525276 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/21951/statm: 254522 307 113 282 0 254238 0
[pid=21952] ppid=21940 vsize=300244 CPUtime=0.01 cores=0,2,4,6
/proc/21952/stat : 21952 (python) S 21940 21940 20819 0 -1 4202560 898 0 0 0 0 1 0 0 18 0 1 0 638928520 307449856 5864 33554432000 4194304 5683644 140735096548976 18446744073709551615 212039675462 0 0 16781312 514 2914182181314068737 0 0 17 0 0 0 0
/proc/21952/statm: 75061 5864 370 364 0 10973 0
[pid=21954] ppid=21952 vsize=417192 CPUtime=5.32 cores=0,2,4,6
/proc/21954/stat : 21954 (adaptg2wsat2009) R 21952 21954 21954 0 -1 4202496 759 0 0 0 532 0 0 0 25 0 1 0 638928520 427204608 496 33554432000 134512640 135107411 4293591776 18446744073709551615 134539280 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/21954/statm: 104298 496 59 146 0 103174 0
[pid=21955] ppid=21940 vsize=310500 CPUtime=0.01 cores=0,2,4,6
/proc/21955/stat : 21955 (python) S 21940 21940 20819 0 -1 4202560 912 0 0 0 0 1 0 0 18 0 1 0 638928521 317952000 5869 33554432000 4194304 5683644 140735096548976 18446744073709551615 212039675462 0 0 16781312 514 13583115415432691971 0 0 17 4 0 0 0
/proc/21955/statm: 77625 5869 371 364 0 13534 0
[pid=21957] ppid=21955 vsize=417192 CPUtime=5.27 cores=0,2,4,6
/proc/21957/stat : 21957 (adaptg2wsat2009) R 21955 21957 21957 0 -1 4202496 759 0 0 0 527 0 0 0 25 0 1 0 638928521 427204608 495 33554432000 134512640 135107411 4289443536 18446744073709551615 134531709 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/21957/statm: 104298 495 59 146 0 103174 0
[pid=21940/tid=21947] ppid=21938 vsize=320744 CPUtime=0 cores=0,2,4,6
/proc/21940/task/21947/stat : 21947 (python) S 21938 21940 20819 0 -1 4202560 24 934 0 0 0 0 0 0 21 0 5 0 638928518 328441856 7385 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 2 0 0 0
[pid=21940/tid=21950] ppid=21938 vsize=320744 CPUtime=0 cores=0,2,4,6
/proc/21940/task/21950/stat : 21950 (python) S 21938 21940 20819 0 -1 4202560 24 934 0 0 0 0 0 0 20 0 5 0 638928519 328441856 7385 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 2 0 0 0
[pid=21940/tid=21953] ppid=21938 vsize=320744 CPUtime=0 cores=0,2,4,6
/proc/21940/task/21953/stat : 21953 (python) S 21938 21940 20819 0 -1 4202560 24 934 0 0 0 0 0 0 22 0 5 0 638928520 328441856 7385 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 0 0 0 0
[pid=21940/tid=21956] ppid=21938 vsize=320744 CPUtime=0 cores=0,2,4,6
/proc/21940/task/21956/stat : 21956 (python) S 21938 21940 20819 0 -1 4202560 28 934 0 0 0 0 0 0 24 0 5 0 638928521 328441856 7385 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 22.1
Current children cumulated vsize (KiB) 4371448

[startup+12.7004 s]
/proc/loadavg: 8.14 8.16 7.64 9/448 22002
/proc/meminfo: memFree=21963800/32951124 swapFree=51873460/67111528
[pid=21940] ppid=21938 vsize=320744 CPUtime=0.84 cores=0,2,4,6
/proc/21940/stat : 21940 (python) S 21938 21940 20819 0 -1 4202496 12930 934 1 0 75 9 0 0 24 0 5 0 638928425 328441856 7385 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051482059 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/21940/statm: 80186 7385 1883 364 0 16095 0
[pid=21946] ppid=21940 vsize=279412 CPUtime=0.03 cores=0,2,4,6
/proc/21946/stat : 21946 (python) S 21940 21940 20819 0 -1 4202560 884 0 0 0 2 1 0 0 18 0 1 0 638928518 286117888 5815 33554432000 4194304 5683644 140735096548976 18446744073709551615 212039675407 0 0 16781312 514 6545964466196939011 0 0 17 2 0 0 0
/proc/21946/statm: 69853 5815 377 364 0 5771 0
[pid=21948] ppid=21946 vsize=1018088 CPUtime=11.58 cores=0,2,4,6
/proc/21948/stat : 21948 (gnovelty+2) R 21946 21948 21948 0 -1 4202496 569 0 0 0 1158 0 0 0 25 0 1 0 638928519 1042522112 307 33554432000 134512640 135663704 4292466848 18446744073709551615 134525243 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/21948/statm: 254522 307 113 282 0 254238 0
[pid=21949] ppid=21940 vsize=289988 CPUtime=0.03 cores=0,2,4,6

################
# More data... #
################

/proc/21955/stat : 21955 (python) S 21940 21940 20819 0 -1 4202560 1135 0 0 0 34 33 0 0 18 0 2 0 638928521 317952000 5899 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/21955/statm: 77625 5899 400 364 0 13534 0
[pid=21957] ppid=21955 vsize=417192 CPUtime=194.62 cores=0,2,4,6
/proc/21957/stat : 21957 (adaptg2wsat2009) T 21955 21957 21957 0 -1 4202496 759 0 0 0 19462 0 0 0 25 0 1 0 638928521 427204608 495 33554432000 134512640 135107411 4289443536 18446744073709551615 134531741 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/21957/statm: 104298 495 59 146 0 103174 0
[pid=21955/tid=22014] ppid=21940 vsize=310500 CPUtime=0 cores=0,2,4,6
/proc/21955/task/22014/stat : 22014 (python) S 21940 21940 20819 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 638938555 317952000 5899 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 514 18446604447485324160 0 0 -1 2 0 0 0
[pid=22008] ppid=21940 vsize=320756 CPUtime=0.69 cores=0,2,4,6
/proc/22008/stat : 22008 (python) S 21940 21940 20819 0 -1 4202560 1147 0 0 0 36 33 0 0 18 0 2 0 638938553 328454144 5903 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/22008/statm: 80189 5903 400 364 0 16095 0
[pid=22011] ppid=22008 vsize=417192 CPUtime=194.49 cores=0,2,4,6
/proc/22011/stat : 22011 (adaptg2wsat2009) T 22008 22011 22011 0 -1 4202496 756 0 0 0 19449 0 0 0 25 0 1 0 638938553 427204608 495 33554432000 134512640 135107411 4289788464 18446744073709551615 134532264 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/22011/statm: 104298 495 59 146 0 103174 0
[pid=22008/tid=22044] ppid=21940 vsize=320756 CPUtime=0 cores=0,2,4,6
/proc/22008/task/22044/stat : 22044 (python) S 21940 21940 20819 0 -1 4202560 49 0 0 0 0 0 0 0 18 0 2 0 638948586 328454144 5903 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 514 18446604444491711744 0 0 -1 4 0 0 0
[pid=22012] ppid=21940 vsize=331012 CPUtime=0.33 cores=0,2,4,6
/proc/22012/stat : 22012 (python) S 21940 21940 20819 0 -1 4202560 1097 0 0 0 17 16 0 0 19 0 2 0 638938554 338956288 5907 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/22012/statm: 82753 5907 400 364 0 18656 0
[pid=22015] ppid=22012 vsize=1018088 CPUtime=100.36 cores=0,2,4,6
/proc/22015/stat : 22015 (gnovelty+2) T 22012 22015 22015 0 -1 4202496 567 0 0 0 10036 0 0 0 25 0 1 0 638938555 1042522112 306 33554432000 134512640 135663704 4288639728 18446744073709551615 134524553 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/22015/statm: 254522 306 113 282 0 254238 0
[pid=22012/tid=22048] ppid=21940 vsize=331012 CPUtime=0 cores=0,2,4,6
/proc/22012/task/22048/stat : 22048 (python) S 21940 21940 20819 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 638948592 338956288 5907 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 514 18446604453945230272 0 0 -1 2 0 0 0
[pid=22016] ppid=21940 vsize=341268 CPUtime=0.34 cores=0,2,4,6
/proc/22016/stat : 22016 (python) S 21940 21940 20819 0 -1 4202560 1097 0 0 0 18 16 0 0 19 0 2 0 638938556 349458432 5911 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/22016/statm: 85317 5911 400 364 0 21217 0
[pid=22018] ppid=22016 vsize=247348 CPUtime=97.55 cores=0,2,4,6
/proc/22018/stat : 22018 (FH) T 22016 22018 22018 0 -1 4202496 706 0 0 0 9755 0 0 0 25 0 1 0 638938556 253284352 444 33554432000 134512640 134936988 4290904464 18446744073709551615 134553435 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/22018/statm: 61837 444 41 104 0 61341 0
[pid=22016/tid=22040] ppid=21940 vsize=341268 CPUtime=0 cores=0,2,4,6
/proc/22016/task/22040/stat : 22040 (python) S 21940 21940 20819 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 638948494 349458432 5911 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 514 18446604444800842688 0 0 -1 6 0 0 0
[pid=22019] ppid=21940 vsize=351524 CPUtime=0.35 cores=0,2,4,6
/proc/22019/stat : 22019 (python) S 21940 21940 20819 0 -1 4202560 1155 0 0 0 18 17 0 0 18 0 2 0 638938557 359960576 5921 33554432000 4194304 5683644 140735096548976 18446744073709551615 212039675462 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/22019/statm: 87881 5921 406 364 0 23778 0
[pid=22021] ppid=22019 vsize=417192 CPUtime=104.87 cores=0,2,4,6
/proc/22021/stat : 22021 (adaptg2wsat2009) R 22019 22021 22021 0 -1 4202496 761 0 0 0 10487 0 0 0 25 0 1 0 638938557 427204608 496 33554432000 134512640 135107411 4294677040 18446744073709551615 134532107 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/22021/statm: 104298 496 59 146 0 103174 0
[pid=22019/tid=22052] ppid=21940 vsize=351524 CPUtime=0 cores=0,2,4,6
/proc/22019/task/22052/stat : 22052 (python) S 21940 21940 20819 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 638948594 359960576 5921 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 514 18446604453944322176 0 0 -1 0 0 0 0
[pid=22041] ppid=21940 vsize=361780 CPUtime=0.33 cores=0,2,4,6
/proc/22041/stat : 22041 (python) S 21940 21940 20819 0 -1 4202560 1141 0 0 0 17 16 0 0 19 0 2 0 638948495 370462720 5925 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/22041/statm: 90445 5925 406 364 0 26339 0
[pid=22043] ppid=22041 vsize=417192 CPUtime=97.91 cores=0,2,4,6
/proc/22043/stat : 22043 (adaptg2wsat2009) T 22041 22043 22043 0 -1 4202496 764 0 0 0 9791 0 0 0 25 0 1 0 638948496 427204608 495 33554432000 134512640 135107411 4294890848 18446744073709551615 134532336 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/22043/statm: 104298 495 59 146 0 103174 0
[pid=22041/tid=22068] ppid=21940 vsize=361780 CPUtime=0 cores=0,2,4,6
/proc/22041/task/22068/stat : 22068 (python) S 21940 21940 20819 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 638958528 370462720 5925 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 514 18446604440361206720 0 0 -1 2 0 0 0
[pid=22045] ppid=21940 vsize=372036 CPUtime=0.99 cores=0,2,4,6
/proc/22045/stat : 22045 (python) S 21940 21940 20819 0 -1 4202560 1168 0 0 0 53 46 0 0 18 0 2 0 638948587 380964864 5923 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/22045/statm: 93009 5923 400 364 0 28900 0
[pid=22047] ppid=22045 vsize=268424 CPUtime=291.22 cores=0,2,4,6
/proc/22047/stat : 22047 (hybridGM3) T 22045 22047 22047 0 -1 4202496 4936 0 0 0 29122 0 0 0 25 0 1 0 638948587 274866176 4672 33554432000 134512640 135204385 4290488912 18446744073709551615 134536993 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/22047/statm: 67106 4672 64 169 0 66936 0
[pid=22045/tid=22084] ppid=21940 vsize=372036 CPUtime=0 cores=0,2,4,6
/proc/22045/task/22084/stat : 22084 (python) S 21940 21940 20819 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 638968253 380964864 5923 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 514 18446604442071686016 0 0 -1 4 0 0 0
[pid=22049] ppid=21940 vsize=382292 CPUtime=0.33 cores=0,2,4,6
/proc/22049/stat : 22049 (python) S 21940 21940 20819 0 -1 4202560 1134 0 0 0 16 17 0 0 19 0 2 0 638948593 391467008 5927 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/22049/statm: 95573 5927 400 364 0 31461 0
[pid=22051] ppid=22049 vsize=247348 CPUtime=99.97 cores=0,2,4,6
/proc/22051/stat : 22051 (FH) T 22049 22051 22051 0 -1 4202496 710 0 0 0 9996 1 0 0 25 0 1 0 638948593 253284352 444 33554432000 134512640 134936988 4286631808 18446744073709551615 134552382 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/22051/statm: 61837 444 41 104 0 61341 0
[pid=22049/tid=22072] ppid=21940 vsize=382292 CPUtime=0 cores=0,2,4,6
/proc/22049/task/22072/stat : 22072 (python) S 21940 21940 20819 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 638958626 391467008 5927 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 514 18446604450022202112 0 0 -1 2 0 0 0
[pid=22053] ppid=21940 vsize=392548 CPUtime=1.71 cores=0,2,4,6
/proc/22053/stat : 22053 (python) S 21940 21940 20819 0 -1 4202560 1179 0 0 0 89 82 0 0 18 0 2 0 638948595 401969152 5961 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/22053/statm: 98137 5961 400 364 0 34022 0
[pid=22055] ppid=22053 vsize=268424 CPUtime=488.14 cores=0,2,4,6
/proc/22055/stat : 22055 (hybridGM3) T 22053 22055 22055 0 -1 4202496 4941 0 0 0 48813 1 0 0 25 0 1 0 638948595 274866176 4672 33554432000 134512640 135204385 4287558320 18446744073709551615 134537272 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/22055/statm: 67106 4672 64 169 0 66936 0
[pid=22053/tid=22083] ppid=21940 vsize=392548 CPUtime=0 cores=0,2,4,6
/proc/22053/task/22083/stat : 22083 (python) S 21940 21940 20819 0 -1 4202560 50 0 0 0 0 0 0 0 18 0 2 0 638968163 401969152 5961 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 514 18446604453973748672 0 0 -1 0 0 0 0
[pid=22069] ppid=21940 vsize=402804 CPUtime=0.34 cores=0,2,4,6
/proc/22069/stat : 22069 (python) S 21940 21940 20819 0 -1 4202560 1148 0 0 0 19 15 0 0 19 0 2 0 638958530 412471296 5967 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/22069/statm: 100701 5967 400 364 0 36583 0
[pid=22071] ppid=22069 vsize=1018088 CPUtime=99.64 cores=0,2,4,6
/proc/22071/stat : 22071 (gnovelty+2) T 22069 22071 22071 0 -1 4202496 575 0 0 0 9964 0 0 0 25 0 1 0 638958532 1042522112 306 33554432000 134512640 135663704 4291643776 18446744073709551615 134524551 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/22071/statm: 254522 306 113 282 0 254238 0
[pid=22069/tid=22088] ppid=21940 vsize=402804 CPUtime=0 cores=0,2,4,6
/proc/22069/task/22088/stat : 22088 (python) S 21940 21940 20819 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 638968566 412471296 5967 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 514 18446604448509380416 0 0 -1 4 0 0 0
[pid=22085] ppid=21940 vsize=413324 CPUtime=1.04 cores=0,2,4,6
/proc/22085/stat : 22085 (python) S 21940 21940 20819 0 -1 4202560 1193 0 0 0 54 50 0 0 18 0 2 0 638968362 423243776 6010 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/22085/statm: 103331 6010 400 364 0 39210 0
[pid=22087] ppid=22085 vsize=268428 CPUtime=292.87 cores=0,2,4,6
/proc/22087/stat : 22087 (hybridGM3) T 22085 22087 22087 0 -1 4202496 4943 0 0 0 29282 5 0 0 25 0 1 0 638968363 274870272 4673 33554432000 134512640 135204385 4292142816 18446744073709551615 134536891 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/22087/statm: 67107 4673 64 169 0 66937 0
[pid=22085/tid=22106] ppid=21940 vsize=413324 CPUtime=0 cores=0,2,4,6
/proc/22085/task/22106/stat : 22106 (python) S 21940 21940 20819 0 -1 4202560 48 0 0 0 0 0 0 0 18 0 2 0 638988032 423243776 6010 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 514 18446604440683049152 0 0 -1 0 0 0 0
[pid=22095] ppid=21940 vsize=423580 CPUtime=0.35 cores=0,2,4,6
/proc/22095/stat : 22095 (python) S 21940 21940 20819 0 -1 4202560 1222 0 0 0 20 15 0 0 19 0 2 0 638977787 433745920 6023 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/22095/statm: 105895 6023 407 364 0 41771 0
[pid=22097] ppid=22095 vsize=247348 CPUtime=100.25 cores=0,2,4,6
/proc/22097/stat : 22097 (FH) T 22095 22097 22097 0 -1 4202496 723 0 0 0 10025 0 0 0 25 0 1 0 638977788 253284352 444 33554432000 134512640 134936988 4292714448 18446744073709551615 134554603 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/22097/statm: 61837 444 41 104 0 61341 0
[pid=22095/tid=22104] ppid=21940 vsize=423580 CPUtime=0 cores=0,2,4,6
/proc/22095/task/22104/stat : 22104 (python) S 21940 21940 20819 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 638987822 433745920 6023 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 514 18446604444491713536 0 0 -1 4 0 0 0
[pid=22098] ppid=21940 vsize=434016 CPUtime=0.73 cores=0,2,4,6
/proc/22098/stat : 22098 (python) S 21940 21940 20819 0 -1 4202560 1073 0 0 0 36 37 0 0 15 0 1 0 638978302 444432384 6036 33554432000 4194304 5683644 140735096548976 18446744073709551615 212039675462 0 0 16781312 514 6545964466196939010 0 0 17 0 0 0 0
/proc/22098/statm: 108504 6036 379 364 0 44377 0
[pid=22100] ppid=22098 vsize=30084 CPUtime=197.41 cores=0,2,4,6
/proc/22100/stat : 22100 (march_hi) R 22098 22100 22100 0 -1 4202496 7791 0 0 0 19740 1 0 0 25 0 1 0 638978303 30806016 6403 33554432000 4194304 4955518 140735864885296 18446744073709551615 4208624 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/22100/statm: 7521 6403 106 186 0 7334 0
[pid=22111] ppid=21940 vsize=444272 CPUtime=0 cores=0,2,4,6
/proc/22111/stat : 22111 (python) S 21940 21940 20819 0 -1 4202560 863 0 0 0 0 0 0 0 18 0 1 0 638997869 454934528 6028 33554432000 4194304 5683644 140735096548976 18446744073709551615 212039675462 0 0 16781312 514 7003703805518774529 0 0 17 6 0 0 0
/proc/22111/statm: 111068 6028 356 364 0 46938 0
[pid=22113] ppid=22111 vsize=1018088 CPUtime=1.7 cores=0,2,4,6
/proc/22113/stat : 22113 (gnovelty+2) R 22111 22113 22113 0 -1 4202496 585 0 0 0 170 0 0 0 25 0 1 0 638997870 1042522112 306 33554432000 134512640 135663704 4291882496 18446744073709551615 134524739 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/22113/statm: 254522 306 113 282 0 254238 0
[pid=22114] ppid=21940 vsize=454528 CPUtime=0 cores=0,2,4,6
/proc/22114/stat : 22114 (python) S 21940 21940 20819 0 -1 4202560 850 0 0 0 0 0 0 0 19 0 1 0 638997973 465436672 6025 33554432000 4194304 5683644 140735096548976 18446744073709551615 212039675462 0 0 16781312 514 889725088369213439 0 0 17 0 0 0 0
/proc/22114/statm: 113632 6025 349 364 0 49499 0
[pid=22116] ppid=22114 vsize=268424 CPUtime=0.74 cores=0,2,4,6
/proc/22116/stat : 22116 (hybridGM3) R 22114 22116 22116 0 -1 4202496 4945 0 0 0 74 0 0 0 22 0 1 0 638997973 274866176 4672 33554432000 134512640 135204385 4292890352 18446744073709551615 134537383 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/22116/statm: 67106 4672 64 169 0 66936 0
[pid=21940/tid=21947] ppid=21938 vsize=464772 CPUtime=0 cores=0,2,4,6
/proc/21940/task/21947/stat : 21947 (python) S 21938 21940 20819 0 -1 4202560 24 934 0 0 0 0 0 0 21 0 19 0 638928518 475926528 7592 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 2 0 0 0
[pid=21940/tid=21950] ppid=21938 vsize=464772 CPUtime=0 cores=0,2,4,6
/proc/21940/task/21950/stat : 21950 (python) S 21938 21940 20819 0 -1 4202560 24 934 0 0 0 0 0 0 20 0 19 0 638928519 475926528 7592 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 2 0 0 0
[pid=21940/tid=21953] ppid=21938 vsize=464772 CPUtime=0 cores=0,2,4,6
/proc/21940/task/21953/stat : 21953 (python) S 21938 21940 20819 0 -1 4202560 28 934 0 0 0 0 0 0 18 0 19 0 638928520 475926528 7592 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 0 0 0 0
[pid=21940/tid=21956] ppid=21938 vsize=464772 CPUtime=0 cores=0,2,4,6
/proc/21940/task/21956/stat : 21956 (python) S 21938 21940 20819 0 -1 4202560 32 934 0 0 0 0 0 0 18 0 19 0 638928521 475926528 7592 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 4 0 0 0
[pid=21940/tid=22010] ppid=21938 vsize=464772 CPUtime=0 cores=0,2,4,6
/proc/21940/task/22010/stat : 22010 (python) S 21938 21940 20819 0 -1 4202560 29 934 0 0 0 0 0 0 18 0 19 0 638938553 475926528 7592 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 0 0 0 0
[pid=21940/tid=22013] ppid=21938 vsize=464772 CPUtime=0 cores=0,2,4,6
/proc/21940/task/22013/stat : 22013 (python) S 21938 21940 20819 0 -1 4202560 24 934 0 0 0 0 0 0 18 0 19 0 638938554 475926528 7592 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 2 0 0 0
[pid=21940/tid=22017] ppid=21938 vsize=464772 CPUtime=0 cores=0,2,4,6
/proc/21940/task/22017/stat : 22017 (python) S 21938 21940 20819 0 -1 4202560 25 934 0 0 0 0 0 0 24 0 19 0 638938556 475926528 7592 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 6 0 0 0
[pid=21940/tid=22020] ppid=21938 vsize=464772 CPUtime=0 cores=0,2,4,6
/proc/21940/task/22020/stat : 22020 (python) S 21938 21940 20819 0 -1 4202560 31 934 0 0 0 0 0 0 18 0 19 0 638938557 475926528 7592 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 2 0 0 0
[pid=21940/tid=22042] ppid=21938 vsize=464772 CPUtime=0 cores=0,2,4,6
/proc/21940/task/22042/stat : 22042 (python) S 21938 21940 20819 0 -1 4202560 21 934 0 0 0 0 0 0 20 0 19 0 638948495 475926528 7592 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 2 0 0 0
[pid=21940/tid=22046] ppid=21938 vsize=464772 CPUtime=0 cores=0,2,4,6
/proc/21940/task/22046/stat : 22046 (python) S 21938 21940 20819 0 -1 4202560 28 934 0 0 0 0 0 0 18 0 19 0 638948587 475926528 7592 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 2 0 0 0
[pid=21940/tid=22050] ppid=21938 vsize=464772 CPUtime=0 cores=0,2,4,6
/proc/21940/task/22050/stat : 22050 (python) S 21938 21940 20819 0 -1 4202560 26 934 0 0 0 0 0 0 21 0 19 0 638948593 475926528 7592 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 2 0 0 0
[pid=21940/tid=22054] ppid=21938 vsize=464772 CPUtime=0 cores=0,2,4,6
/proc/21940/task/22054/stat : 22054 (python) S 21938 21940 20819 0 -1 4202560 39 934 0 0 0 0 0 0 18 0 19 0 638948595 475926528 7592 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 0 0 0 0
[pid=21940/tid=22070] ppid=21938 vsize=464772 CPUtime=0 cores=0,2,4,6
/proc/21940/task/22070/stat : 22070 (python) S 21938 21940 20819 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 19 0 638958530 475926528 7592 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 0 0 0 0
[pid=21940/tid=22086] ppid=21938 vsize=464772 CPUtime=0 cores=0,2,4,6
/proc/21940/task/22086/stat : 22086 (python) S 21938 21940 20819 0 -1 4202560 36 934 0 0 0 0 0 0 18 0 19 0 638968363 475926528 7592 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 0 0 0 0
[pid=21940/tid=22096] ppid=21938 vsize=464772 CPUtime=0 cores=0,2,4,6
/proc/21940/task/22096/stat : 22096 (python) S 21938 21940 20819 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 19 0 638977787 475926528 7592 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 0 0 0 0
[pid=21940/tid=22099] ppid=21938 vsize=464772 CPUtime=0 cores=0,2,4,6
/proc/21940/task/22099/stat : 22099 (python) S 21938 21940 20819 0 -1 4202560 30 934 0 0 0 0 0 0 20 0 19 0 638978303 475926528 7592 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 0 0 0 0
[pid=21940/tid=22112] ppid=21938 vsize=464772 CPUtime=0 cores=0,2,4,6
/proc/21940/task/22112/stat : 22112 (python) S 21938 21940 20819 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 19 0 638997870 475926528 7592 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 0 0 0 0
[pid=21940/tid=22115] ppid=21938 vsize=464772 CPUtime=0 cores=0,2,4,6
/proc/21940/task/22115/stat : 22115 (python) S 21938 21940 20819 0 -1 4202560 26 934 0 0 0 0 0 0 20 0 19 0 638997973 475926528 7592 33554432000 4194304 5683644 140735096548976 18446744073709551615 212051478961 0 0 16781312 2 18446604444491710848 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2766.25
Current children cumulated vsize (KiB) 16103128

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 21940 and gives
#  childrusage.ru_utime.tv_sec=206
#  childrusage.ru_utime.tv_usec=178656
#  childrusage.ru_stime.tv_sec=4
#  childrusage.ru_stime.tv_usec=809268
# CPU time returned by wait4() is 210.988
# while last known CPU time is 2766.25
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 696.335
CPU time (s): 2766.25
CPU user time (s): 2761.48
CPU system time (s): 4.77
CPU usage (%): 397.258
Max. virtual memory (cumulated for all children) (KiB): 16103128

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 206.179
system time used= 4.80927
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 62254
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 5870
involuntary context switches= 371

runsolver used 4.82027 second user time and 10.4164 second system time

The end

Launcher Data 

Begin job on node146 at 2011-05-15 15:12:01
IDJOB=3378375
IDBENCH=82784
IDSOLVER=1859
FILE ID=node146/3378375-1305465121
RUNJOBID= node146-1305463158-20837
PBS_JOBID= 13324984
Free space on /tmp= 73268 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/medium/unif-k7-r89-v90-c8010-S764895749-094.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378375-1305465121/watcher-3378375-1305465121 -o /tmp/evaluation-result-3378375-1305465121/solver-3378375-1305465121 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378375-1305465121.cnf -seed 969329219 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 86d70d98b4a1fa931b287202356d4839
RANDOM SEED=969329219

node146.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.86
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      21986472 kB
Buffers:        149548 kB
Cached:        1074664 kB
SwapCached:      56148 kB
Active:        8181224 kB
Inactive:      2517968 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      21986472 kB
SwapTotal:    67111528 kB
SwapFree:     51873460 kB
Dirty:           32536 kB
Writeback:           0 kB
AnonPages:     9426132 kB
Mapped:          28560 kB
Slab:           113876 kB
PageTables:      83164 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 114042292 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73236 MiB
End job on node146 at 2011-05-15 15:23:40