Trace number 3378104

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (MO) 2010.68 508.237

General information on the benchmark

Namerandom/large/
unif-k7-r85-v400-c34000-S832901479-042-UNKNOWN.cnf
MD5SUMa5a87dc48ba0573fb8f2b8721bafcab2
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables400
Number of clauses34000
Sum of the clauses size238000
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 534000

Solver Data 

0.38/0.79	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.38/0.79	c INFO: solving HOME/instance-3378104-1305462668.cnf
0.38/0.79	c INFO: computing task features
0.66/1.02	c INFO: computed variable-clause graph statistics
0.66/1.03	c INFO: computed clause balance statistics
1.86/2.24	c INFO: computed variable balance statistics
2.45/3.00	c INFO: computed small-clause counts
2.45/3.00	c INFO: computed Horn-clause counts
2.45/3.00	c INFO: computed variable graph statistics
2.45/3.00	c INFO: collected features for HOME/instance-3378104-1305462668.cnf in 1.97 s
2.45/3.00	c INFO: running gnovelty+2@0 for 97 with 1198 remaining (b = 0.38)
2.45/3.00	c INFO: running gnovelty+2@0 for 97 with 1198 remaining (b = 0.13)
2.45/3.00	c INFO: running adaptg2wsat2009++@0 for 97 with 1198 remaining (b = 0.08)
2.45/3.00	c INFO: running adaptg2wsat2009++@0 for 97 with 1197 remaining (b = 0.07)
394.23/101.20	c INFO: running adaptg2wsat2009++@0 for 97 with 1099 remaining (b = 0.05)
397.26/102.18	c INFO: running gnovelty+2@0 for 97 with 1098 remaining (b = 0.04)
402.11/103.11	c INFO: running FH@0 for 97 with 1097 remaining (b = 0.03)
402.11/103.15	c INFO: running adaptg2wsat2009++@0 for 97 with 1097 remaining (b = 0.03)
789.70/200.60	c INFO: running adaptg2wsat2009++@0 for 97 with 1000 remaining (b = 0.02)
793.26/201.50	c INFO: running FH@0 for 97 with 999 remaining (b = 0.02)
793.26/201.58	c INFO: running hybridGM3@0 for 194 with 999 remaining (b = 0.04)
816.66/207.48	c INFO: running gnovelty+2@0 for 97 with 993 remaining (b = 0.02)
1188.28/300.97	c INFO: running hybridGM3@0 for 194 with 899 remaining (b = 0.03)
1208.14/305.97	c INFO: running adaptg2wsat2009++@0 for 97 with 894 remaining (b = 0.01)
1212.16/306.93	c INFO: running hybridGM3@0 for 194 with 893 remaining (b = 0.03)
1567.28/396.39	c INFO: running hybridGM3@100 for 97 with 804 remaining (b = 0.01)
1607.01/406.36	c INFO: running hybridGM3@0 for 194 with 794 remaining (b = 0.02)
1965.42/496.77	c INFO: running TNM@0 for 704 with 704 remaining (b = 0.08)
1977.69/499.81	c INFO: running hybridGM3@100 for 97 with 700 remaining (b = 0.01)
2008.99/507.77	c INFO: running FH@0 for 97 with 693 remaining (b = 0.01)
2010.68/508.21	c INFO: running hybridGM3@0 for 194 with 692 remaining (b = 0.02)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378104-1305462668/watcher-3378104-1305462668 -o /tmp/evaluation-result-3378104-1305462668/solver-3378104-1305462668 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378104-1305462668.cnf -seed 1912783864 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.39 3.86 3.31 3/408 17015
/proc/meminfo: memFree=25237596/32951124 swapFree=53501244/67111528
[pid=17015] ppid=17013 vsize=15912 CPUtime=0 cores=1,3,5,7
/proc/17015/stat : 17015 (python) R 17013 17015 16982 0 -1 4202496 957 0 0 0 0 0 0 0 17 0 1 0 657869428 16293888 834 33554432000 4194304 5683644 140733913895760 18446744073709551615 216634513408 0 2147483391 16781312 2 0 0 0 17 1 0 0 0
/proc/17015/statm: 3978 834 357 364 0 465 0

[startup+0.0493309 s]
/proc/loadavg: 3.39 3.86 3.31 3/408 17015
/proc/meminfo: memFree=25237596/32951124 swapFree=53501244/67111528
[pid=17015] ppid=17013 vsize=18888 CPUtime=0.02 cores=1,3,5,7
/proc/17015/stat : 17015 (python) R 17013 17015 16982 0 -1 4202496 1283 0 0 0 2 0 0 0 17 0 1 0 657869428 19341312 1137 33554432000 4194304 5683644 140733913895760 18446744073709551615 4877038 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/17015/statm: 4722 1137 421 364 0 691 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 18888

[startup+0.100323 s]
/proc/loadavg: 3.39 3.86 3.31 3/408 17015
/proc/meminfo: memFree=25237596/32951124 swapFree=53501244/67111528
[pid=17015] ppid=17013 vsize=50340 CPUtime=0.05 cores=1,3,5,7
/proc/17015/stat : 17015 (python) R 17013 17015 16982 0 -1 4202496 2009 0 0 0 3 2 0 0 17 0 1 0 657869428 51548160 1770 33554432000 4194304 5683644 140733913895760 18446744073709551615 216634513408 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/17015/statm: 12585 1770 480 364 0 1285 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 50340

[startup+0.300286 s]
/proc/loadavg: 3.39 3.86 3.31 3/408 17015
/proc/meminfo: memFree=25237596/32951124 swapFree=53501244/67111528
[pid=17015] ppid=17013 vsize=131296 CPUtime=0.18 cores=1,3,5,7
/proc/17015/stat : 17015 (python) R 17013 17015 16982 0 -1 4202496 5714 936 1 0 12 6 0 0 17 0 1 0 657869428 134447104 4101 33554432000 4194304 5683644 140733913895760 18446744073709551615 216634513408 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/17015/statm: 32824 4101 1015 364 0 3085 0
Current children cumulated CPU time (s) 0.18
Current children cumulated vsize (KiB) 131296

[startup+0.70021 s]
/proc/loadavg: 3.39 3.86 3.31 3/408 17015
/proc/meminfo: memFree=25237596/32951124 swapFree=53501244/67111528
[pid=17015] ppid=17013 vsize=246316 CPUtime=0.38 cores=1,3,5,7
/proc/17015/stat : 17015 (python) R 17013 17015 16982 0 -1 4202496 8131 936 1 0 25 13 0 0 15 0 1 0 657869428 252227584 6030 33554432000 4194304 5683644 140733913895760 18446744073709551615 216634176197 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/17015/statm: 61579 6030 1621 364 0 4581 0
Current children cumulated CPU time (s) 0.38
Current children cumulated vsize (KiB) 246316

[startup+1.50107 s]
/proc/loadavg: 3.39 3.86 3.31 3/414 17029
/proc/meminfo: memFree=25177772/32951124 swapFree=53501244/67111528
[pid=17015] ppid=17013 vsize=289496 CPUtime=1.14 cores=1,3,5,7
/proc/17015/stat : 17015 (python) R 17013 17015 16982 0 -1 4202496 13879 936 1 0 99 15 0 0 16 0 1 0 657869428 296443904 9758 33554432000 4194304 5683644 140733913895760 18446744073709551615 47014798377557 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/17015/statm: 72374 9758 1826 364 0 8298 0
Current children cumulated CPU time (s) 1.14
Current children cumulated vsize (KiB) 289496

[startup+3.10076 s]
/proc/loadavg: 3.39 3.86 3.31 7/426 17041
/proc/meminfo: memFree=25161028/32951124 swapFree=53501244/67111528
[pid=17015] ppid=17013 vsize=322832 CPUtime=2.45 cores=1,3,5,7
/proc/17015/stat : 17015 (python) S 17013 17015 16982 0 -1 4202496 16644 936 1 0 229 16 0 0 22 0 5 0 657869428 330579968 7915 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646342091 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/17015/statm: 80708 7915 1875 364 0 16617 0
Current children cumulated CPU time (s) 2.45
Current children cumulated vsize (KiB) 322832
heavy processes:

[startup+6.30119 s]
/proc/loadavg: 3.76 3.92 3.33 9/438 17053
/proc/meminfo: memFree=25127248/32951124 swapFree=53501244/67111528
[pid=17015] ppid=17013 vsize=322832 CPUtime=2.45 cores=1,3,5,7
/proc/17015/stat : 17015 (python) S 17013 17015 16982 0 -1 4202496 16644 936 1 0 229 16 0 0 22 0 5 0 657869428 330579968 7915 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646342091 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/17015/statm: 80708 7915 1875 364 0 16617 0
[pid=17042] ppid=17015 vsize=282236 CPUtime=0 cores=1,3,5,7
/proc/17042/stat : 17042 (python) S 17015 17015 16982 0 -1 4202560 723 0 0 0 0 0 0 0 18 0 1 0 657869706 289009664 6329 33554432000 4194304 5683644 140733913895760 18446744073709551615 216634535439 0 0 16781312 514 17342495754136027400 0 0 17 7 0 0 0
/proc/17042/statm: 70559 6329 355 364 0 6477 0
[pid=17044] ppid=17042 vsize=1020188 CPUtime=3.51 cores=1,3,5,7
/proc/17044/stat : 17044 (gnovelty+2) R 17042 17044 17044 0 -1 4202496 1138 0 0 0 351 0 0 0 25 0 1 0 657869706 1044672512 876 33554432000 134512640 135663704 4292144960 18446744073709551615 134524918 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/17044/statm: 255047 876 113 282 0 254763 0
[pid=17045] ppid=17015 vsize=293168 CPUtime=0 cores=1,3,5,7
/proc/17045/stat : 17045 (python) S 17015 17015 16982 0 -1 4202560 737 0 0 0 0 0 0 0 18 0 1 0 657869707 300204032 6378 33554432000 4194304 5683644 140733913895760 18446744073709551615 216634535494 0 0 16781312 514 17342488057554632967 0 0 17 7 0 0 0
/proc/17045/statm: 73292 6378 349 364 0 9207 0
[pid=17047] ppid=17045 vsize=1020188 CPUtime=3.5 cores=1,3,5,7
/proc/17047/stat : 17047 (gnovelty+2) R 17045 17047 17047 0 -1 4202496 1140 0 0 0 350 0 0 0 25 0 1 0 657869707 1044672512 877 33554432000 134512640 135663704 4287948880 18446744073709551615 134545619 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/17047/statm: 255047 877 113 282 0 254763 0
[pid=17048] ppid=17015 vsize=302884 CPUtime=0 cores=1,3,5,7
/proc/17048/stat : 17048 (python) S 17015 17015 16982 0 -1 4202560 741 0 0 0 0 0 0 0 18 0 1 0 657869708 310153216 6382 33554432000 4194304 5683644 140733913895760 18446744073709551615 216634535494 0 0 16781312 514 7573840991572951301 0 0 17 1 0 0 0
/proc/17048/statm: 75721 6382 349 364 0 11633 0
[pid=17050] ppid=17048 vsize=421140 CPUtime=3.5 cores=1,3,5,7
/proc/17050/stat : 17050 (adaptg2wsat2009) R 17048 17050 17050 0 -1 4202496 1799 0 0 0 350 0 0 0 25 0 1 0 657869708 431247360 1538 33554432000 134512640 135107411 4287391648 18446744073709551615 134523245 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/17050/statm: 105285 1538 49 146 0 104161 0
[pid=17051] ppid=17015 vsize=313140 CPUtime=0 cores=1,3,5,7
/proc/17051/stat : 17051 (python) S 17015 17015 16982 0 -1 4202560 749 0 0 0 0 0 0 0 18 0 1 0 657869709 320655360 6386 33554432000 4194304 5683644 140733913895760 18446744073709551615 216634535494 0 0 16781312 514 6615022871687168263 0 0 17 7 0 0 0
/proc/17051/statm: 78285 6386 349 364 0 14194 0
[pid=17053] ppid=17051 vsize=421140 CPUtime=3.43 cores=1,3,5,7
/proc/17053/stat : 17053 (adaptg2wsat2009) R 17051 17053 17053 0 -1 4202496 1805 0 0 0 343 0 0 0 25 0 1 0 657869709 431247360 1540 33554432000 134512640 135107411 4287009040 18446744073709551615 134523457 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/17053/statm: 105285 1540 49 146 0 104161 0
[pid=17015/tid=17043] ppid=17013 vsize=322832 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17043/stat : 17043 (python) S 17013 17015 16982 0 -1 4202560 25 936 0 0 0 0 0 0 20 0 5 0 657869706 330579968 7915 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
[pid=17015/tid=17046] ppid=17013 vsize=322832 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17046/stat : 17046 (python) S 17013 17015 16982 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 5 0 657869707 330579968 7915 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
[pid=17015/tid=17049] ppid=17013 vsize=322832 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17049/stat : 17049 (python) S 17013 17015 16982 0 -1 4202560 24 936 0 0 0 0 0 0 20 0 5 0 657869708 330579968 7915 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 1 0 0 0
[pid=17015/tid=17052] ppid=17013 vsize=322832 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17052/stat : 17052 (python) S 17013 17015 16982 0 -1 4202560 30 936 0 0 0 0 0 0 22 0 5 0 657869709 330579968 7915 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 16.39
Current children cumulated vsize (KiB) 4396916

[startup+12.7011 s]
/proc/loadavg: 4.10 3.99 3.36 9/438 17053
/proc/meminfo: memFree=25121224/32951124 swapFree=53501244/67111528
[pid=17015] ppid=17013 vsize=322832 CPUtime=2.45 cores=1,3,5,7
/proc/17015/stat : 17015 (python) S 17013 17015 16982 0 -1 4202496 16644 936 1 0 229 16 0 0 22 0 5 0 657869428 330579968 7915 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646342091 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/17015/statm: 80708 7915 1875 364 0 16617 0
[pid=17042] ppid=17015 vsize=282236 CPUtime=0.01 cores=1,3,5,7
/proc/17042/stat : 17042 (python) S 17015 17015 16982 0 -1 4202560 883 0 0 0 0 1 0 0 18 0 1 0 657869706 289009664 6352 33554432000 4194304 5683644 140733913895760 18446744073709551615 216634535439 0 0 16781312 514 17342495754136027400 0 0 17 7 0 0 0
/proc/17042/statm: 70559 6352 378 364 0 6477 0
[pid=17044] ppid=17042 vsize=1020188 CPUtime=9.89 cores=1,3,5,7
/proc/17044/stat : 17044 (gnovelty+2) R 17042 17044 17044 0 -1 4202496 1138 0 0 0 989 0 0 0 25 0 1 0 657869706 1044672512 876 33554432000 134512640 135663704 4292144960 18446744073709551615 134525276 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/17044/statm: 255047 876 113 282 0 254763 0
[pid=17045] ppid=17015 vsize=293168 CPUtime=0.01 cores=1,3,5,7
/proc/17045/stat : 17045 (python) S 17015 17015 16982 0 -1 4202560 895 0 0 0 1 0 0 0 18 0 1 0 657869707 300204032 6401 33554432000 4194304 5683644 140733913895760 18446744073709551615 216634535494 0 0 16781312 514 17342488057554632967 0 0 17 7 0 0 0
/proc/17045/statm: 73292 6401 372 364 0 9207 0
[pid=17047] ppid=17045 vsize=1020188 CPUtime=9.9 cores=1,3,5,7
/proc/17047/stat : 17047 (gnovelty+2) R 17045 17047 17047 0 -1 4202496 1140 0 0 0 990 0 0 0 25 0 1 0 657869707 1044672512 877 33554432000 134512640 135663704 4287948880 18446744073709551615 134524739 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/17047/statm: 255047 877 113 282 0 254763 0
[pid=17048] ppid=17015 vsize=302884 CPUtime=0.01 cores=1,3,5,7
/proc/17048/stat : 17048 (python) S 17015 17015 16982 0 -1 4202560 910 0 0 0 0 1 0 0 18 0 1 0 657869708 310153216 6405 33554432000 4194304 5683644 140733913895760 18446744073709551615 216634535494 0 0 16781312 514 7573840991572951301 0 0 17 1 0 0 0
/proc/17048/statm: 75721 6405 372 364 0 11633 0
[pid=17050] ppid=17048 vsize=421804 CPUtime=9.9 cores=1,3,5,7
/proc/17050/stat : 17050 (adaptg2wsat2009) R 17048 17050 17050 0 -1 4202496 2133 0 0 0 990 0 0 0 25 0 1 0 657869708 431927296 1872 33554432000 134512640 135107411 4287391648 18446744073709551615 134531615 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/17050/statm: 105451 1872 59 146 0 104327 0
[pid=17051] ppid=17015 vsize=313140 CPUtime=0.01 cores=1,3,5,7
/proc/17051/stat : 17051 (python) S 17015 17015 16982 0 -1 4202560 918 0 0 0 1 0 0 0 18 0 1 0 657869709 320655360 6409 33554432000 4194304 5683644 140733913895760 18446744073709551615 216634535494 0 0 16781312 514 6615022871687168263 0 0 17 7 0 0 0
/proc/17051/statm: 78285 6409 372 364 0 14194 0
[pid=17053] ppid=17051 vsize=421804 CPUtime=9.67 cores=1,3,5,7
/proc/17053/stat : 17053 (adaptg2wsat2009) R 17051 17053 17053 0 -1 4202496 2138 0 0 0 967 0 0 0 25 0 1 0 657869709 431927296 1873 33554432000 134512640 135107411 4287009040 18446744073709551615 134531843 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/17053/statm: 105451 1873 59 146 0 104327 0
[pid=17015/tid=17043] ppid=17013 vsize=322832 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17043/stat : 17043 (python) S 17013 17015 16982 0 -1 4202560 25 936 0 0 0 0 0 0 20 0 5 0 657869706 330579968 7915 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
[pid=17015/tid=17046] ppid=17013 vsize=322832 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17046/stat : 17046 (python) S 17013 17015 16982 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 5 0 657869707 330579968 7915 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
[pid=17015/tid=17049] ppid=17013 vsize=322832 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17049/stat : 17049 (python) S 17013 17015 16982 0 -1 4202560 24 936 0 0 0 0 0 0 20 0 5 0 657869708 330579968 7915 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 1 0 0 0
[pid=17015/tid=17052] ppid=17013 vsize=322832 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17052/stat : 17052 (python) S 17013 17015 16982 0 -1 4202560 30 936 0 0 0 0 0 0 22 0 5 0 657869709 330579968 7915 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 41.85
Current children cumulated vsize (KiB) 4398244

[startup+25.5008 s]
/proc/loadavg: 5.04 4.20 3.43 9/438 17054
/proc/meminfo: memFree=25122136/32951124 swapFree=53501244/67111528
[pid=17015] ppid=17013 vsize=322832 CPUtime=2.45 cores=1,3,5,7
/proc/17015/stat : 17015 (python) S 17013 17015 16982 0 -1 4202496 16644 936 1 0 229 16 0 0 22 0 5 0 657869428 330579968 7915 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646342091 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/17015/statm: 80708 7915 1875 364 0 16617 0
[pid=17042] ppid=17015 vsize=282236 CPUtime=0.07 cores=1,3,5,7
/proc/17042/stat : 17042 (python) S 17015 17015 16982 0 -1 4202560 893 0 0 0 3 4 0 0 18 0 1 0 657869706 289009664 6352 33554432000 4194304 5683644 140733913895760 18446744073709551615 216634535439 0 0 16781312 514 8730104732355363080 0 0 17 7 0 0 0
/proc/17042/statm: 70559 6352 378 364 0 6477 0
[pid=17044] ppid=17042 vsize=1020188 CPUtime=22.64 cores=1,3,5,7
/proc/17044/stat : 17044 (gnovelty+2) R 17042 17044 17044 0 -1 4202496 1138 0 0 0 2264 0 0 0 25 0 1 0 657869706 1044672512 876 33554432000 134512640 135663704 4292144960 18446744073709551615 134545670 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/17044/statm: 255047 876 113 282 0 254763 0
[pid=17045] ppid=17015 vsize=293168 CPUtime=0.06 cores=1,3,5,7
/proc/17045/stat : 17045 (python) S 17015 17015 16982 0 -1 4202560 906 0 0 0 3 3 0 0 18 0 1 0 657869707 300204032 6401 33554432000 4194304 5683644 140733913895760 18446744073709551615 216634535494 0 0 16781312 514 8730104732355363079 0 0 17 7 0 0 0
/proc/17045/statm: 73292 6401 372 364 0 9207 0
[pid=17047] ppid=17045 vsize=1020188 CPUtime=22.7 cores=1,3,5,7
/proc/17047/stat : 17047 (gnovelty+2) R 17045 17047 17047 0 -1 4202496 1140 0 0 0 2270 0 0 0 25 0 1 0 657869707 1044672512 877 33554432000 134512640 135663704 4287948880 18446744073709551615 134525222 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/17047/statm: 255047 877 113 282 0 254763 0
[pid=17048] ppid=17015 vsize=302884 CPUtime=0.06 cores=1,3,5,7
/proc/17048/stat : 17048 (python) S 17015 17015 16982 0 -1 4202560 918 0 0 0 3 3 0 0 18 0 1 0 657869708 310153216 6405 33554432000 4194304 5683644 140733913895760 18446744073709551615 216634535494 0 0 16781312 514 8730104732355363077 0 0 17 1 0 0 0
/proc/17048/statm: 75721 6405 372 364 0 11633 0
[pid=17050] ppid=17048 vsize=421804 CPUtime=22.7 cores=1,3,5,7
/proc/17050/stat : 17050 (adaptg2wsat2009) R 17048 17050 17050 0 -1 4202496 2133 0 0 0 2270 0 0 0 25 0 1 0 657869708 431927296 1872 33554432000 134512640 135107411 4287391648 18446744073709551615 134532224 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/17050/statm: 105451 1872 59 146 0 104327 0
[pid=17051] ppid=17015 vsize=313140 CPUtime=0.07 cores=1,3,5,7
/proc/17051/stat : 17051 (python) S 17015 17015 16982 0 -1 4202560 924 0 0 0 4 3 0 0 18 0 1 0 657869709 320655360 6409 33554432000 4194304 5683644 140733913895760 18446744073709551615 216634535494 0 0 16781312 514 8730104732355363079 0 0 17 7 0 0 0
/proc/17051/statm: 78285 6409 372 364 0 14194 0
[pid=17053] ppid=17051 vsize=421804 CPUtime=22.11 cores=1,3,5,7
/proc/17053/stat : 17053 (adaptg2wsat2009) R 17051 17053 17053 0 -1 4202496 2138 0 0 0 2211 0 0 0 25 0 1 0 657869709 431927296 1873 33554432000 134512640 135107411 4287009040 18446744073709551615 134531582 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/17053/statm: 105451 1873 59 146 0 104327 0
[pid=17015/tid=17043] ppid=17013 vsize=322832 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17043/stat : 17043 (python) S 17013 17015 16982 0 -1 4202560 25 936 0 0 0 0 0 0 20 0 5 0 657869706 330579968 7915 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
[pid=17015/tid=17046] ppid=17013 vsize=322832 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17046/stat : 17046 (python) S 17013 17015 16982 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 5 0 657869707 330579968 7915 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
[pid=17015/tid=17049] ppid=17013 vsize=322832 CPUtime=0 cores=1,3,5,7

################
# More data... #
################

/proc/17050/stat : 17050 (adaptg2wsat2009) T 17048 17050 17050 0 -1 4202496 2133 0 0 0 10030 0 0 0 25 0 1 0 657869708 431927296 1872 33554432000 134512640 135107411 4287391648 18446744073709551615 134532224 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/17050/statm: 105451 1872 59 146 0 104327 0
[pid=17048/tid=17081] ppid=17015 vsize=302884 CPUtime=0 cores=1,3,5,7
/proc/17048/task/17081/stat : 17081 (python) S 17015 17015 16982 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 657879739 310153216 6434 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 514 18446604454676115520 0 0 -1 1 0 0 0
[pid=17051] ppid=17015 vsize=313140 CPUtime=0.31 cores=1,3,5,7
/proc/17051/stat : 17051 (python) S 17015 17015 16982 0 -1 4202560 1105 0 0 0 17 14 0 0 19 0 2 0 657869709 320655360 6438 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/17051/statm: 78285 6438 400 364 0 14194 0
[pid=17053] ppid=17051 vsize=421804 CPUtime=97.91 cores=1,3,5,7
/proc/17053/stat : 17053 (adaptg2wsat2009) T 17051 17053 17053 0 -1 4202496 2138 0 0 0 9791 0 0 0 25 0 1 0 657869709 431927296 1873 33554432000 134512640 135107411 4287009040 18446744073709551615 134531435 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/17053/statm: 105451 1873 59 146 0 104327 0
[pid=17051/tid=17085] ppid=17015 vsize=313140 CPUtime=0 cores=1,3,5,7
/proc/17051/task/17085/stat : 17085 (python) S 17015 17015 16982 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 657879743 320655360 6438 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 514 18446604467337775168 0 0 -1 7 0 0 0
[pid=17058] ppid=17015 vsize=324004 CPUtime=0.33 cores=1,3,5,7
/proc/17058/stat : 17058 (python) S 17015 17015 16982 0 -1 4202560 1096 0 0 0 17 16 0 0 19 0 2 0 657879549 331780096 6459 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/17058/statm: 81001 6459 400 364 0 16907 0
[pid=17060] ppid=17058 vsize=421804 CPUtime=99.36 cores=1,3,5,7
/proc/17060/stat : 17060 (adaptg2wsat2009) T 17058 17060 17060 0 -1 4202496 2138 0 0 0 9936 0 0 0 25 0 1 0 657879549 431927296 1873 33554432000 134512640 135107411 4293636272 18446744073709551615 134532288 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/17060/statm: 105451 1873 59 146 0 104327 0
[pid=17058/tid=17091] ppid=17015 vsize=324004 CPUtime=0 cores=1,3,5,7
/proc/17058/task/17091/stat : 17091 (python) S 17015 17015 16982 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 657889488 331780096 6459 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 514 18446604454747147136 0 0 -1 5 0 0 0
[pid=17062] ppid=17015 vsize=333776 CPUtime=0.33 cores=1,3,5,7
/proc/17062/stat : 17062 (python) S 17015 17015 16982 0 -1 4202560 1099 0 0 0 18 15 0 0 19 0 2 0 657879647 341786624 6597 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/17062/statm: 83444 6597 399 364 0 19347 0
[pid=17064] ppid=17062 vsize=1020188 CPUtime=99.36 cores=1,3,5,7
/proc/17064/stat : 17064 (gnovelty+2) T 17062 17064 17064 0 -1 4202496 1135 0 0 0 9936 0 0 0 25 0 1 0 657879647 1044672512 876 33554432000 134512640 135663704 4288879920 18446744073709551615 134525049 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/17064/statm: 255047 876 113 282 0 254763 0
[pid=17062/tid=17099] ppid=17015 vsize=333776 CPUtime=0 cores=1,3,5,7
/proc/17062/task/17099/stat : 17099 (python) S 17015 17015 16982 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 657889586 341786624 6597 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 514 18446604454747146240 0 0 -1 7 0 0 0
[pid=17082] ppid=17015 vsize=344032 CPUtime=0.36 cores=1,3,5,7
/proc/17082/stat : 17082 (python) S 17015 17015 16982 0 -1 4202560 1117 0 0 0 19 17 0 0 19 0 2 0 657879740 352288768 6601 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/17082/statm: 86008 6601 399 364 0 21908 0
[pid=17084] ppid=17082 vsize=251296 CPUtime=100.67 cores=1,3,5,7
/proc/17084/stat : 17084 (FH) T 17082 17084 17084 0 -1 4202496 1955 0 0 0 10067 0 0 0 25 0 1 0 657879740 257327104 1694 33554432000 134512640 134936988 4288746960 18446744073709551615 134552375 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/17084/statm: 62824 1694 41 104 0 62328 0
[pid=17082/tid=17120] ppid=17015 vsize=344032 CPUtime=0 cores=1,3,5,7
/proc/17082/task/17120/stat : 17120 (python) S 17015 17015 16982 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 657890176 352288768 6601 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 514 18446604464176073472 0 0 -1 7 0 0 0
[pid=17086] ppid=17015 vsize=354604 CPUtime=0.33 cores=1,3,5,7
/proc/17086/stat : 17086 (python) S 17015 17015 16982 0 -1 4202560 1142 0 0 0 18 15 0 0 19 0 2 0 657879744 363114496 6612 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/17086/statm: 88651 6612 406 364 0 24548 0
[pid=17088] ppid=17086 vsize=421804 CPUtime=98.33 cores=1,3,5,7
/proc/17088/stat : 17088 (adaptg2wsat2009) T 17086 17088 17088 0 -1 4202496 2142 0 0 0 9833 0 0 0 25 0 1 0 657879744 431927296 1871 33554432000 134512640 135107411 4294417488 18446744073709551615 134531613 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/17088/statm: 105451 1871 59 146 0 104327 0
[pid=17086/tid=17095] ppid=17015 vsize=354604 CPUtime=0 cores=1,3,5,7
/proc/17086/task/17095/stat : 17095 (python) S 17015 17015 16982 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 657889578 363114496 6612 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 514 18446604467019464896 0 0 -1 1 0 0 0
[pid=17092] ppid=17015 vsize=364748 CPUtime=0.34 cores=1,3,5,7
/proc/17092/stat : 17092 (python) S 17015 17015 16982 0 -1 4202560 1151 0 0 0 17 17 0 0 19 0 2 0 657889489 373501952 6616 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/17092/statm: 91187 6616 406 364 0 27081 0
[pid=17094] ppid=17092 vsize=421804 CPUtime=100.25 cores=1,3,5,7
/proc/17094/stat : 17094 (adaptg2wsat2009) T 17092 17094 17094 0 -1 4202496 2139 0 0 0 10025 0 0 0 25 0 1 0 657889490 431927296 1872 33554432000 134512640 135107411 4288655968 18446744073709551615 134532347 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/17094/statm: 105451 1872 59 146 0 104327 0
[pid=17092/tid=17126] ppid=17015 vsize=364748 CPUtime=0 cores=1,3,5,7
/proc/17092/task/17126/stat : 17126 (python) S 17015 17015 16982 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 657899524 373501952 6616 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 514 18446604460774859968 0 0 -1 7 0 0 0
[pid=17096] ppid=17015 vsize=375116 CPUtime=0.36 cores=1,3,5,7
/proc/17096/stat : 17096 (python) S 17015 17015 16982 0 -1 4202560 1139 0 0 0 18 18 0 0 19 0 2 0 657889579 384118784 6614 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/17096/statm: 93779 6614 400 364 0 29670 0
[pid=17098] ppid=17096 vsize=251292 CPUtime=100.76 cores=1,3,5,7
/proc/17098/stat : 17098 (FH) T 17096 17098 17098 0 -1 4202496 1960 0 0 0 10076 0 0 0 25 0 1 0 657889579 257323008 1694 33554432000 134512640 134936988 4294009136 18446744073709551615 134553482 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/17098/statm: 62823 1694 41 104 0 62327 0
[pid=17096/tid=17138] ppid=17015 vsize=375116 CPUtime=0 cores=1,3,5,7
/proc/17096/task/17138/stat : 17138 (python) S 17015 17015 16982 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 657900024 384118784 6614 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 514 18446604460774859072 0 0 -1 3 0 0 0
[pid=17100] ppid=17015 vsize=385272 CPUtime=1.03 cores=1,3,5,7
/proc/17100/stat : 17100 (python) S 17015 17015 16982 0 -1 4202560 1192 0 0 0 55 48 0 0 18 0 2 0 657889587 394518528 6618 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/17100/statm: 96318 6618 400 364 0 32206 0
[pid=17102] ppid=17100 vsize=268424 CPUtime=294.53 cores=1,3,5,7
/proc/17102/stat : 17102 (hybridGM3) T 17100 17102 17102 0 -1 4202496 5644 0 0 0 29453 0 0 0 25 0 1 0 657889587 274866176 5378 33554432000 134512640 135204385 4290236352 18446744073709551615 134538034 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/17102/statm: 67106 5378 63 169 0 66936 0
[pid=17100/tid=17153] ppid=17015 vsize=385272 CPUtime=0 cores=1,3,5,7
/proc/17100/task/17153/stat : 17153 (python) S 17015 17015 16982 0 -1 4202560 49 0 0 0 0 0 0 0 18 0 2 0 657909067 394518528 6618 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 514 18446604468178615296 0 0 -1 5 0 0 0
[pid=17121] ppid=17015 vsize=395656 CPUtime=0.34 cores=1,3,5,7
/proc/17121/stat : 17121 (python) S 17015 17015 16982 0 -1 4202560 1143 0 0 0 18 16 0 0 19 0 2 0 657890177 405151744 6622 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/17121/statm: 98914 6622 400 364 0 34799 0
[pid=17123] ppid=17121 vsize=1020188 CPUtime=99.35 cores=1,3,5,7
/proc/17123/stat : 17123 (gnovelty+2) T 17121 17123 17123 0 -1 4202496 1143 0 0 0 9935 0 0 0 25 0 1 0 657890178 1044672512 876 33554432000 134512640 135663704 4294335808 18446744073709551615 134525286 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/17123/statm: 255047 876 113 282 0 254763 0
[pid=17121/tid=17142] ppid=17015 vsize=395656 CPUtime=0 cores=1,3,5,7
/proc/17121/task/17142/stat : 17142 (python) S 17015 17015 16982 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 657900121 405151744 6622 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 514 18446604467935917056 0 0 -1 7 0 0 0
[pid=17127] ppid=17015 vsize=406048 CPUtime=0.74 cores=1,3,5,7
/proc/17127/stat : 17127 (python) S 17015 17015 16982 0 -1 4202560 1187 0 0 0 38 36 0 0 18 0 2 0 657899526 415793152 6626 33554432000 4194304 5683644 140733913895760 18446744073709551615 216634535494 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/17127/statm: 101512 6626 400 364 0 37394 0
[pid=17129] ppid=17127 vsize=268420 CPUtime=206.85 cores=1,3,5,7
/proc/17129/stat : 17129 (hybridGM3) R 17127 17129 17129 0 -1 4202496 5645 0 0 0 20685 0 0 0 25 0 1 0 657899526 274862080 5377 33554432000 134512640 135204385 4289793280 18446744073709551615 134538349 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/17129/statm: 67105 5377 63 169 0 66935 0
[pid=17127/tid=17169] ppid=17015 vsize=406048 CPUtime=0 cores=1,3,5,7
/proc/17127/task/17169/stat : 17169 (python) S 17015 17015 16982 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 657919409 415793152 6626 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 514 18446604468205424768 0 0 -1 3 0 0 0
[pid=17139] ppid=17015 vsize=416208 CPUtime=0.35 cores=1,3,5,7
/proc/17139/stat : 17139 (python) S 17015 17015 16982 0 -1 4202560 1152 0 0 0 19 16 0 0 19 0 2 0 657900026 426196992 6630 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/17139/statm: 104052 6630 400 364 0 39931 0
[pid=17141] ppid=17139 vsize=421804 CPUtime=100.37 cores=1,3,5,7
/proc/17141/stat : 17141 (adaptg2wsat2009) T 17139 17141 17141 0 -1 4202496 2142 0 0 0 10037 0 0 0 25 0 1 0 657900026 431927296 1872 33554432000 134512640 135107411 4289725616 18446744073709551615 134532376 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/17141/statm: 105451 1872 59 146 0 104327 0
[pid=17139/tid=17159] ppid=17015 vsize=416208 CPUtime=0 cores=1,3,5,7
/proc/17139/task/17159/stat : 17159 (python) S 17015 17015 16982 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 657910063 426196992 6630 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 514 18446604468205425664 0 0 -1 7 0 0 0
[pid=17143] ppid=17015 vsize=426612 CPUtime=0.71 cores=1,3,5,7
/proc/17143/stat : 17143 (python) S 17015 17015 16982 0 -1 4202560 1248 0 0 0 38 33 0 0 19 0 2 0 657900122 436850688 6641 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/17143/statm: 106653 6641 407 364 0 42529 0
[pid=17145] ppid=17143 vsize=268424 CPUtime=197.17 cores=1,3,5,7
/proc/17145/stat : 17145 (hybridGM3) T 17143 17145 17145 0 -1 4202496 5655 0 0 0 19716 1 0 0 25 0 1 0 657900122 274866176 5378 33554432000 134512640 135204385 4292400176 18446744073709551615 134538395 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/17145/statm: 67106 5378 63 169 0 66936 0
[pid=17143/tid=17175] ppid=17015 vsize=426612 CPUtime=0 cores=1,3,5,7
/proc/17143/task/17175/stat : 17175 (python) S 17015 17015 16982 0 -1 4202560 44 0 0 0 0 0 0 0 19 0 2 0 657920205 436850688 6641 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 514 18446604459366494656 0 0 -1 7 0 0 0
[pid=17160] ppid=17015 vsize=436892 CPUtime=0.35 cores=1,3,5,7
/proc/17160/stat : 17160 (python) S 17015 17015 16982 0 -1 4202560 1053 0 0 0 18 17 0 0 18 0 1 0 657910064 447377408 6629 33554432000 4194304 5683644 140733913895760 18446744073709551615 216634535494 0 0 16781312 514 8730104732355363079 0 0 17 7 0 0 0
/proc/17160/statm: 109223 6629 380 364 0 45096 0
[pid=17162] ppid=17160 vsize=268428 CPUtime=97.27 cores=1,3,5,7
/proc/17162/stat : 17162 (hybridGM3) R 17160 17162 17162 0 -1 4202496 5652 0 0 0 9726 1 0 0 25 0 1 0 657910065 274870272 5378 33554432000 134512640 135204385 4287817920 18446744073709551615 134538034 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/17162/statm: 67107 5378 63 169 0 66937 0
[pid=17165] ppid=17015 vsize=447396 CPUtime=0.03 cores=1,3,5,7
/proc/17165/stat : 17165 (python) S 17015 17015 16982 0 -1 4202560 1034 0 0 0 2 1 0 0 18 0 1 0 657919106 458133504 6639 33554432000 4194304 5683644 140733913895760 18446744073709551615 216634535494 0 0 16781312 514 8730104732355363077 0 0 17 3 0 0 0
/proc/17165/statm: 111849 6639 380 364 0 47719 0
[pid=17167] ppid=17165 vsize=459772 CPUtime=10.85 cores=1,3,5,7
/proc/17167/stat : 17167 (TNM) R 17165 17167 17167 0 -1 4202496 1932 0 0 0 1085 0 0 0 25 0 1 0 657919107 470806528 1657 33554432000 134512640 135008748 4287707488 18446744073709551615 134527722 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/17167/statm: 114943 1657 46 122 0 113843 0
[pid=17176] ppid=17015 vsize=457712 CPUtime=0 cores=1,3,5,7
/proc/17176/stat : 17176 (python) S 17015 17015 16982 0 -1 4202560 825 0 0 0 0 0 0 0 19 0 1 0 657920206 468697088 6650 33554432000 4194304 5683644 140733913895760 18446744073709551615 216634535494 0 0 16781312 514 15604897671465140223 0 0 17 3 0 0 0
/proc/17176/statm: 114428 6650 346 364 0 50295 0
[pid=17178] ppid=17176 vsize=249144 CPUtime=0.35 cores=1,3,5,7
/proc/17178/stat : 17178 (FH) R 17176 17178 17178 0 -1 4202496 1564 0 0 0 35 0 0 0 21 0 1 0 657920207 255123456 1295 33554432000 134512640 134936988 4292755920 18446744073709551615 134523650 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/17178/statm: 62286 1295 37 104 0 61790 0
[pid=17015/tid=17043] ppid=17013 vsize=467328 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17043/stat : 17043 (python) S 17013 17015 16982 0 -1 4202560 25 936 0 0 0 0 0 0 20 0 19 0 657869706 478543872 8219 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
[pid=17015/tid=17046] ppid=17013 vsize=467328 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17046/stat : 17046 (python) S 17013 17015 16982 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 19 0 657869707 478543872 8219 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
[pid=17015/tid=17049] ppid=17013 vsize=467328 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17049/stat : 17049 (python) S 17013 17015 16982 0 -1 4202560 24 936 0 0 0 0 0 0 20 0 19 0 657869708 478543872 8219 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 1 0 0 0
[pid=17015/tid=17052] ppid=17013 vsize=467328 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17052/stat : 17052 (python) S 17013 17015 16982 0 -1 4202560 30 936 0 0 0 0 0 0 22 0 19 0 657869709 478543872 8219 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
[pid=17015/tid=17059] ppid=17013 vsize=467328 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17059/stat : 17059 (python) S 17013 17015 16982 0 -1 4202560 28 936 0 0 0 0 0 0 20 0 19 0 657879549 478543872 8219 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
[pid=17015/tid=17063] ppid=17013 vsize=467328 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17063/stat : 17063 (python) S 17013 17015 16982 0 -1 4202560 25 936 0 0 0 0 0 0 20 0 19 0 657879647 478543872 8219 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
[pid=17015/tid=17083] ppid=17013 vsize=467328 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17083/stat : 17083 (python) S 17013 17015 16982 0 -1 4202560 26 936 0 0 0 0 0 0 20 0 19 0 657879740 478543872 8219 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
[pid=17015/tid=17087] ppid=17013 vsize=467328 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17087/stat : 17087 (python) S 17013 17015 16982 0 -1 4202560 26 936 0 0 0 0 0 0 21 0 19 0 657879744 478543872 8219 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
[pid=17015/tid=17093] ppid=17013 vsize=467328 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17093/stat : 17093 (python) S 17013 17015 16982 0 -1 4202560 20 936 0 0 0 0 0 0 20 0 19 0 657889489 478543872 8219 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
[pid=17015/tid=17097] ppid=17013 vsize=467328 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17097/stat : 17097 (python) S 17013 17015 16982 0 -1 4202560 24 936 0 0 0 0 0 0 20 0 19 0 657889579 478543872 8219 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
[pid=17015/tid=17101] ppid=17013 vsize=467328 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17101/stat : 17101 (python) S 17013 17015 16982 0 -1 4202560 28 936 0 0 0 0 0 0 18 0 19 0 657889587 478543872 8219 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
[pid=17015/tid=17122] ppid=17013 vsize=467328 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17122/stat : 17122 (python) S 17013 17015 16982 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 19 0 657890178 478543872 8219 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
[pid=17015/tid=17128] ppid=17013 vsize=467328 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17128/stat : 17128 (python) S 17013 17015 16982 0 -1 4202560 27 936 0 0 0 0 0 0 18 0 19 0 657899526 478543872 8219 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
[pid=17015/tid=17140] ppid=17013 vsize=467328 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17140/stat : 17140 (python) S 17013 17015 16982 0 -1 4202560 30 936 0 0 0 0 0 0 20 0 19 0 657900026 478543872 8219 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
[pid=17015/tid=17144] ppid=17013 vsize=467328 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17144/stat : 17144 (python) S 17013 17015 16982 0 -1 4202560 26 936 0 0 0 0 0 0 20 0 19 0 657900122 478543872 8219 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
[pid=17015/tid=17161] ppid=17013 vsize=467328 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17161/stat : 17161 (python) S 17013 17015 16982 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 19 0 657910065 478543872 8219 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
[pid=17015/tid=17166] ppid=17013 vsize=467328 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17166/stat : 17166 (python) S 17013 17015 16982 0 -1 4202560 25 936 0 0 0 0 0 0 20 0 19 0 657919106 478543872 8219 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 7 0 0 0
[pid=17015/tid=17177] ppid=17013 vsize=467328 CPUtime=0 cores=1,3,5,7
/proc/17015/task/17177/stat : 17177 (python) S 17013 17015 16982 0 -1 4202560 26 936 0 0 0 0 0 0 20 0 19 0 657920206 478543872 8219 33554432000 4194304 5683644 140733913895760 18446744073709551615 216646338993 0 0 16781312 2 18446604467339633920 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2010.68
Current children cumulated vsize (KiB) 16033852

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 17015 and gives
#  childrusage.ru_utime.tv_sec=212
#  childrusage.ru_utime.tv_usec=298725
#  childrusage.ru_stime.tv_sec=2
#  childrusage.ru_stime.tv_usec=883561
# CPU time returned by wait4() is 215.182
# while last known CPU time is 2010.68
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 508.237
CPU time (s): 2010.68
CPU user time (s): 2007.21
CPU system time (s): 3.47
CPU usage (%): 395.619
Max. virtual memory (cumulated for all children) (KiB): 16033852

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 212.299
system time used= 2.88356
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 52190
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 14736
involuntary context switches= 879

runsolver used 3.90441 second user time and 8.94164 second system time

The end

Launcher Data 

Begin job on node103 at 2011-05-15 14:31:08
IDJOB=3378104
IDBENCH=82513
IDSOLVER=1859
FILE ID=node103/3378104-1305462668
RUNJOBID= node103-1305462668-17001
PBS_JOBID= 13324971
Free space on /tmp= 72988 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/large/unif-k7-r85-v400-c34000-S832901479-042-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378104-1305462668/watcher-3378104-1305462668 -o /tmp/evaluation-result-3378104-1305462668/solver-3378104-1305462668 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378104-1305462668.cnf -seed 1912783864 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= a5a87dc48ba0573fb8f2b8721bafcab2
RANDOM SEED=1912783864

node103.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      25237876 kB
Buffers:        326052 kB
Cached:        4173976 kB
SwapCached:     340184 kB
Active:        2206384 kB
Inactive:      5212664 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      25237876 kB
SwapTotal:    67111528 kB
SwapFree:     53501244 kB
Dirty:           65428 kB
Writeback:           0 kB
AnonPages:     2590524 kB
Mapped:          16860 kB
Slab:           163396 kB
PageTables:      61396 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 115408512 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72924 MiB
End job on node103 at 2011-05-15 14:39:38