Trace number 3317164

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (MO) 2011.55 507.119

General information on the benchmark

Namerandom/large/
unif-k7-r85-v400-c34000-S1275797759-041-UNKNOWN.cnf
MD5SUM374e3ba6783d13d6d478972802d8dba0
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables400
Number of clauses34000
Sum of the clauses size238000
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 534000

Solver Data 

0.29/0.49	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.29/0.49	c INFO: solving HOME/instance-3317164-1304267884.cnf
0.29/0.49	c INFO: computing task features
0.57/0.72	c INFO: computed variable-clause graph statistics
0.57/0.73	c INFO: computed clause balance statistics
1.78/1.95	c INFO: computed variable balance statistics
2.28/2.44	c INFO: computed small-clause counts
2.28/2.44	c INFO: computed Horn-clause counts
2.28/2.48	c INFO: computed variable graph statistics
2.28/2.48	c INFO: collected features for HOME/instance-3317164-1304267884.cnf in 1.99 s
2.38/2.51	c INFO: running gnovelty+2@0 for 97 with 4797 remaining (b = 0.38)
2.38/2.54	c INFO: running gnovelty+2@0 for 97 with 4797 remaining (b = 0.13)
2.38/2.57	c INFO: running adaptg2wsat2009++@0 for 97 with 4797 remaining (b = 0.08)
2.47/2.60	c INFO: running adaptg2wsat2009++@0 for 97 with 4797 remaining (b = 0.07)
398.35/101.76	c INFO: running adaptg2wsat2009++@0 for 97 with 4698 remaining (b = 0.05)
398.69/101.80	c INFO: running gnovelty+2@0 for 97 with 4698 remaining (b = 0.04)
402.28/102.73	c INFO: running FH@0 for 97 with 4697 remaining (b = 0.03)
402.28/102.76	c INFO: running adaptg2wsat2009++@0 for 97 with 4697 remaining (b = 0.03)
793.91/201.04	c INFO: running adaptg2wsat2009++@0 for 97 with 4599 remaining (b = 0.02)
797.48/201.96	c INFO: running FH@0 for 97 with 4598 remaining (b = 0.02)
797.79/202.00	c INFO: running hybridGM3@0 for 194 with 4598 remaining (b = 0.04)
801.41/202.91	c INFO: running gnovelty+2@0 for 97 with 4597 remaining (b = 0.02)
1192.57/301.17	c INFO: running hybridGM3@0 for 194 with 4499 remaining (b = 0.03)
1192.94/301.23	c INFO: running adaptg2wsat2009++@0 for 97 with 4499 remaining (b = 0.01)
1216.42/307.17	c INFO: running hybridGM3@0 for 194 with 4493 remaining (b = 0.03)
1579.95/398.44	c INFO: running hybridGM3@99 for 97 with 4402 remaining (b = 0.01)
1607.85/405.46	c INFO: running hybridGM3@0 for 194 with 4395 remaining (b = 0.02)
1970.21/496.69	c INFO: running TNM@0 for 1552 with 4303 remaining (b = 0.18)
1986.09/500.69	c INFO: running hybridGM3@100 for 97 with 4299 remaining (b = 0.01)
2009.93/506.69	c INFO: running FH@0 for 97 with 4293 remaining (b = 0.01)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317164-1304267884/watcher-3317164-1304267884 -o /tmp/evaluation-result-3317164-1304267884/solver-3317164-1304267884 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317164-1304267884.cnf -seed 824782661 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.92 6.92 3.75 5/212 2028
/proc/meminfo: memFree=30272332/32950928 swapFree=67111448/67111528
[pid=2028] ppid=2026 vsize=15188 CPUtime=0 cores=1,3,5,7
/proc/2028/stat : 2028 (python) R 2026 2028 1708 0 -1 4202496 717 0 0 0 0 0 0 0 25 0 1 0 538387075 15552512 608 33554432000 4194304 5683644 140735922807280 18446744073709551615 251814240000 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/2028/statm: 3797 608 319 364 0 284 0

[startup+0.092556 s]
/proc/loadavg: 7.92 6.92 3.75 5/212 2028
/proc/meminfo: memFree=30272332/32950928 swapFree=67111448/67111528
[pid=2028] ppid=2026 vsize=99344 CPUtime=0.06 cores=1,3,5,7
/proc/2028/stat : 2028 (python) R 2026 2028 1708 0 -1 4202496 2850 0 0 0 5 1 0 0 25 0 1 0 538387075 101728256 2551 33554432000 4194304 5683644 140735922807280 18446744073709551615 47926984785917 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/2028/statm: 24836 2556 818 364 0 1687 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 99344

[startup+0.100553 s]
/proc/loadavg: 7.92 6.92 3.75 5/212 2028
/proc/meminfo: memFree=30272332/32950928 swapFree=67111448/67111528
[pid=2028] ppid=2026 vsize=102012 CPUtime=0.07 cores=1,3,5,7
/proc/2028/stat : 2028 (python) R 2026 2028 1708 0 -1 4202496 3028 0 0 0 6 1 0 0 25 0 1 0 538387075 104460288 2718 33554432000 4194304 5683644 140735922807280 18446744073709551615 4870783 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/2028/statm: 25503 2718 841 364 0 1803 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 102012

[startup+0.300586 s]
/proc/loadavg: 7.92 6.92 3.75 5/212 2028
/proc/meminfo: memFree=30272332/32950928 swapFree=67111448/67111528
[pid=2028] ppid=2026 vsize=151084 CPUtime=0.21 cores=1,3,5,7
/proc/2028/stat : 2028 (python) R 2026 2028 1708 0 -1 4202496 6304 934 1 0 15 6 0 0 17 0 1 0 538387075 154710016 4432 33554432000 4194304 5683644 140735922807280 18446744073709551615 251814240000 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/2028/statm: 37771 4438 1220 364 0 3182 0
Current children cumulated CPU time (s) 0.21
Current children cumulated vsize (KiB) 151084

[startup+0.700517 s]
/proc/loadavg: 7.92 6.92 3.75 5/212 2028
/proc/meminfo: memFree=30272332/32950928 swapFree=67111448/67111528
[pid=2028] ppid=2026 vsize=288808 CPUtime=0.57 cores=1,3,5,7
/proc/2028/stat : 2028 (python) R 2026 2028 1708 0 -1 4202496 11677 934 1 0 48 9 0 0 17 0 1 0 538387075 295739392 9359 33554432000 4194304 5683644 140735922807280 18446744073709551615 47926982023493 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/2028/statm: 72202 9359 1761 364 0 8126 0
Current children cumulated CPU time (s) 0.57
Current children cumulated vsize (KiB) 288808

[startup+1.50038 s]
/proc/loadavg: 7.92 6.92 3.75 7/213 2033
/proc/meminfo: memFree=30239464/32950928 swapFree=67111448/67111528
[pid=2028] ppid=2026 vsize=289500 CPUtime=1.38 cores=1,3,5,7
/proc/2028/stat : 2028 (python) R 2026 2028 1708 0 -1 4202496 13883 934 1 0 128 10 0 0 18 0 1 0 538387075 296448000 9760 33554432000 4194304 5683644 140735922807280 18446744073709551615 47926982246561 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/2028/statm: 72375 9760 1826 364 0 8299 0
Current children cumulated CPU time (s) 1.38
Current children cumulated vsize (KiB) 289500

[startup+3.10105 s]
/proc/loadavg: 7.92 6.92 3.75 7/213 2033
/proc/meminfo: memFree=30238892/32950928 swapFree=67111448/67111528
[pid=2028] ppid=2026 vsize=323400 CPUtime=2.48 cores=1,3,5,7
/proc/2028/stat : 2028 (python) S 2026 2028 1708 0 -1 4202496 16761 934 1 0 237 11 0 0 25 0 5 0 538387075 331161600 8049 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/2028/statm: 80850 8049 1883 364 0 16759 0
Current children cumulated CPU time (s) 2.48
Current children cumulated vsize (KiB) 323400
heavy processes:

[startup+6.30051 s]
/proc/loadavg: 7.92 6.93 3.77 10/225 2045
/proc/meminfo: memFree=30212256/32950928 swapFree=67111448/67111528
[pid=2028] ppid=2026 vsize=323400 CPUtime=2.48 cores=1,3,5,7
/proc/2028/stat : 2028 (python) S 2026 2028 1708 0 -1 4202496 16761 934 1 0 237 11 0 0 25 0 5 0 538387075 331161600 8049 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/2028/statm: 80850 8049 1883 364 0 16759 0
[pid=2034] ppid=2028 vsize=282244 CPUtime=0 cores=1,3,5,7
/proc/2034/stat : 2034 (python) S 2028 2028 1708 0 -1 4202560 737 0 0 0 0 0 0 0 18 0 1 0 538387326 289017856 6332 33554432000 4194304 5683644 140735922807280 18446744073709551615 251814260239 0 0 16781312 514 4579382645976170760 0 0 17 3 0 0 0
/proc/2034/statm: 70561 6332 355 364 0 6479 0
[pid=2036] ppid=2034 vsize=1020188 CPUtime=3.78 cores=1,3,5,7
/proc/2036/stat : 2036 (gnovelty+2) R 2034 2036 2036 0 -1 4202496 1137 0 0 0 378 0 0 0 25 0 1 0 538387326 1044672512 877 33554432000 134512640 135663704 4292698880 18446744073709551615 134524747 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/2036/statm: 255047 877 113 282 0 254763 0
[pid=2037] ppid=2028 vsize=293128 CPUtime=0 cores=1,3,5,7
/proc/2037/stat : 2037 (python) S 2028 2028 1708 0 -1 4202560 749 0 0 0 0 0 0 0 18 0 1 0 538387329 300163072 6369 33554432000 4194304 5683644 140735922807280 18446744073709551615 251814260294 0 0 16781312 514 7670394880044007688 0 0 17 1 0 0 0
/proc/2037/statm: 73282 6369 349 364 0 9197 0
[pid=2039] ppid=2037 vsize=1020188 CPUtime=3.75 cores=1,3,5,7
/proc/2039/stat : 2039 (gnovelty+2) R 2037 2039 2039 0 -1 4202496 1140 0 0 0 375 0 0 0 25 0 1 0 538387329 1044672512 877 33554432000 134512640 135663704 4293166272 18446744073709551615 134525213 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/2039/statm: 255047 877 113 282 0 254763 0
[pid=2040] ppid=2028 vsize=302900 CPUtime=0 cores=1,3,5,7
/proc/2040/stat : 2040 (python) S 2028 2028 1708 0 -1 4202560 741 0 0 0 0 0 0 0 18 0 1 0 538387332 310169600 6506 33554432000 4194304 5683644 140735922807280 18446744073709551615 251814260294 0 0 16781312 514 4312166685810327816 0 0 17 7 0 0 0
/proc/2040/statm: 75725 6506 347 364 0 11637 0
[pid=2042] ppid=2040 vsize=421140 CPUtime=3.69 cores=1,3,5,7
/proc/2042/stat : 2042 (adaptg2wsat2009) R 2040 2042 2042 0 -1 4202496 1803 0 0 0 369 0 0 0 25 0 1 0 538387332 431247360 1538 33554432000 134512640 135107411 4292311152 18446744073709551615 134523455 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/2042/statm: 105285 1538 49 146 0 104161 0
[pid=2043] ppid=2028 vsize=313156 CPUtime=0 cores=1,3,5,7
/proc/2043/stat : 2043 (python) S 2028 2028 1708 0 -1 4202560 750 0 0 0 0 0 0 0 18 0 1 0 538387335 320671744 6510 33554432000 4194304 5683644 140735922807280 18446744073709551615 251814260294 0 0 16781312 514 7670387183462613254 0 0 17 1 0 0 0
/proc/2043/statm: 78289 6510 347 364 0 14198 0
[pid=2045] ppid=2043 vsize=421140 CPUtime=3.67 cores=1,3,5,7
/proc/2045/stat : 2045 (adaptg2wsat2009) R 2043 2045 2045 0 -1 4202496 1807 0 0 0 367 0 0 0 25 0 1 0 538387335 431247360 1539 33554432000 134512640 135107411 4290547328 18446744073709551615 134523300 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/2045/statm: 105285 1539 49 146 0 104161 0
[pid=2028/tid=2035] ppid=2026 vsize=323400 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2035/stat : 2035 (python) S 2026 2028 1708 0 -1 4202560 26 934 0 0 0 0 0 0 22 0 5 0 538387326 331161600 8049 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 7 0 0 0
[pid=2028/tid=2038] ppid=2026 vsize=323400 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2038/stat : 2038 (python) S 2026 2028 1708 0 -1 4202560 24 934 0 0 0 0 0 0 24 0 5 0 538387329 331161600 8049 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 3 0 0 0
[pid=2028/tid=2041] ppid=2026 vsize=323400 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2041/stat : 2041 (python) S 2026 2028 1708 0 -1 4202560 26 934 0 0 0 0 0 0 25 0 5 0 538387332 331161600 8049 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 7 0 0 0
[pid=2028/tid=2044] ppid=2026 vsize=323400 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2044/stat : 2044 (python) S 2026 2028 1708 0 -1 4202560 26 934 0 0 0 0 0 0 25 0 5 0 538387335 331161600 8049 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 17.37
Current children cumulated vsize (KiB) 4397484

[startup+12.7014 s]
/proc/loadavg: 7.93 6.95 3.79 10/225 2045
/proc/meminfo: memFree=30208940/32950928 swapFree=67111448/67111528
[pid=2028] ppid=2026 vsize=323400 CPUtime=2.48 cores=1,3,5,7
/proc/2028/stat : 2028 (python) S 2026 2028 1708 0 -1 4202496 16761 934 1 0 237 11 0 0 25 0 5 0 538387075 331161600 8049 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/2028/statm: 80850 8049 1883 364 0 16759 0
[pid=2034] ppid=2028 vsize=282244 CPUtime=0.01 cores=1,3,5,7
/proc/2034/stat : 2034 (python) S 2028 2028 1708 0 -1 4202560 904 0 0 0 1 0 0 0 18 0 1 0 538387326 289017856 6355 33554432000 4194304 5683644 140735922807280 18446744073709551615 251814260239 0 0 16781312 514 15830838785462993160 0 0 17 3 0 0 0
/proc/2034/statm: 70561 6355 378 364 0 6479 0
[pid=2036] ppid=2034 vsize=1020188 CPUtime=10.18 cores=1,3,5,7
/proc/2036/stat : 2036 (gnovelty+2) R 2034 2036 2036 0 -1 4202496 1137 0 0 0 1018 0 0 0 25 0 1 0 538387326 1044672512 877 33554432000 134512640 135663704 4292698880 18446744073709551615 134524894 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/2036/statm: 255047 877 113 282 0 254763 0
[pid=2037] ppid=2028 vsize=293128 CPUtime=0.01 cores=1,3,5,7
/proc/2037/stat : 2037 (python) S 2028 2028 1708 0 -1 4202560 917 0 0 0 1 0 0 0 18 0 1 0 538387329 300163072 6392 33554432000 4194304 5683644 140735922807280 18446744073709551615 251814260294 0 0 16781312 514 15830838785462993160 0 0 17 1 0 0 0
/proc/2037/statm: 73282 6392 372 364 0 9197 0
[pid=2039] ppid=2037 vsize=1020188 CPUtime=10.13 cores=1,3,5,7
/proc/2039/stat : 2039 (gnovelty+2) R 2037 2039 2039 0 -1 4202496 1140 0 0 0 1013 0 0 0 25 0 1 0 538387329 1044672512 877 33554432000 134512640 135663704 4293166272 18446744073709551615 134545631 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/2039/statm: 255047 877 113 282 0 254763 0
[pid=2040] ppid=2028 vsize=302900 CPUtime=0.01 cores=1,3,5,7
/proc/2040/stat : 2040 (python) S 2028 2028 1708 0 -1 4202560 905 0 0 0 0 1 0 0 18 0 1 0 538387332 310169600 6529 33554432000 4194304 5683644 140735922807280 18446744073709551615 251814260294 0 0 16781312 514 15830838785462993160 0 0 17 7 0 0 0
/proc/2040/statm: 75725 6529 370 364 0 11637 0
[pid=2042] ppid=2040 vsize=421804 CPUtime=10.03 cores=1,3,5,7
/proc/2042/stat : 2042 (adaptg2wsat2009) R 2040 2042 2042 0 -1 4202496 2137 0 0 0 1003 0 0 0 25 0 1 0 538387332 431927296 1872 33554432000 134512640 135107411 4292311152 18446744073709551615 134531942 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/2042/statm: 105451 1872 59 146 0 104327 0
[pid=2043] ppid=2028 vsize=313156 CPUtime=0 cores=1,3,5,7
/proc/2043/stat : 2043 (python) S 2028 2028 1708 0 -1 4202560 910 0 0 0 0 0 0 0 18 0 1 0 538387335 320671744 6533 33554432000 4194304 5683644 140735922807280 18446744073709551615 251814260294 0 0 16781312 514 15830838785462993158 0 0 17 1 0 0 0
/proc/2043/statm: 78289 6533 370 364 0 14198 0
[pid=2045] ppid=2043 vsize=421804 CPUtime=10.04 cores=1,3,5,7
/proc/2045/stat : 2045 (adaptg2wsat2009) R 2043 2045 2045 0 -1 4202496 2140 0 0 0 1004 0 0 0 25 0 1 0 538387335 431927296 1872 33554432000 134512640 135107411 4290547328 18446744073709551615 134532416 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/2045/statm: 105451 1872 59 146 0 104327 0
[pid=2028/tid=2035] ppid=2026 vsize=323400 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2035/stat : 2035 (python) S 2026 2028 1708 0 -1 4202560 26 934 0 0 0 0 0 0 22 0 5 0 538387326 331161600 8049 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 7 0 0 0
[pid=2028/tid=2038] ppid=2026 vsize=323400 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2038/stat : 2038 (python) S 2026 2028 1708 0 -1 4202560 24 934 0 0 0 0 0 0 24 0 5 0 538387329 331161600 8049 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 3 0 0 0
[pid=2028/tid=2041] ppid=2026 vsize=323400 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2041/stat : 2041 (python) S 2026 2028 1708 0 -1 4202560 26 934 0 0 0 0 0 0 25 0 5 0 538387332 331161600 8049 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 7 0 0 0
[pid=2028/tid=2044] ppid=2026 vsize=323400 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2044/stat : 2044 (python) S 2026 2028 1708 0 -1 4202560 26 934 0 0 0 0 0 0 25 0 5 0 538387335 331161600 8049 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 42.89
Current children cumulated vsize (KiB) 4398812

[startup+25.5013 s]
/proc/loadavg: 7.94 7.00 3.86 10/225 2046
/proc/meminfo: memFree=30208708/32950928 swapFree=67111448/67111528
[pid=2028] ppid=2026 vsize=323400 CPUtime=2.48 cores=1,3,5,7
/proc/2028/stat : 2028 (python) S 2026 2028 1708 0 -1 4202496 16761 934 1 0 237 11 0 0 25 0 5 0 538387075 331161600 8049 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/2028/statm: 80850 8049 1883 364 0 16759 0
[pid=2034] ppid=2028 vsize=282244 CPUtime=0.02 cores=1,3,5,7
/proc/2034/stat : 2034 (python) S 2028 2028 1708 0 -1 4202560 910 0 0 0 1 1 0 0 18 0 1 0 538387326 289017856 6355 33554432000 4194304 5683644 140735922807280 18446744073709551615 251814260239 0 0 16781312 514 15830838785462993160 0 0 17 3 0 0 0
/proc/2034/statm: 70561 6355 378 364 0 6479 0
[pid=2036] ppid=2034 vsize=1020188 CPUtime=22.98 cores=1,3,5,7
/proc/2036/stat : 2036 (gnovelty+2) R 2034 2036 2036 0 -1 4202496 1137 0 0 0 2298 0 0 0 25 0 1 0 538387326 1044672512 877 33554432000 134512640 135663704 4292698880 18446744073709551615 134545583 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/2036/statm: 255047 877 113 282 0 254763 0
[pid=2037] ppid=2028 vsize=293128 CPUtime=0.02 cores=1,3,5,7
/proc/2037/stat : 2037 (python) S 2028 2028 1708 0 -1 4202560 921 0 0 0 1 1 0 0 18 0 1 0 538387329 300163072 6392 33554432000 4194304 5683644 140735922807280 18446744073709551615 251814260294 0 0 16781312 514 15830838785462993160 0 0 17 1 0 0 0
/proc/2037/statm: 73282 6392 372 364 0 9197 0
[pid=2039] ppid=2037 vsize=1020188 CPUtime=22.91 cores=1,3,5,7
/proc/2039/stat : 2039 (gnovelty+2) R 2037 2039 2039 0 -1 4202496 1140 0 0 0 2291 0 0 0 25 0 1 0 538387329 1044672512 877 33554432000 134512640 135663704 4293166272 18446744073709551615 134524509 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/2039/statm: 255047 877 113 282 0 254763 0
[pid=2040] ppid=2028 vsize=302900 CPUtime=0.02 cores=1,3,5,7
/proc/2040/stat : 2040 (python) S 2028 2028 1708 0 -1 4202560 909 0 0 0 1 1 0 0 18 0 1 0 538387332 310169600 6529 33554432000 4194304 5683644 140735922807280 18446744073709551615 251814260294 0 0 16781312 514 15830838785462993160 0 0 17 7 0 0 0
/proc/2040/statm: 75725 6529 370 364 0 11637 0
[pid=2042] ppid=2040 vsize=421804 CPUtime=22.73 cores=1,3,5,7
/proc/2042/stat : 2042 (adaptg2wsat2009) R 2040 2042 2042 0 -1 4202496 2137 0 0 0 2273 0 0 0 25 0 1 0 538387332 431927296 1872 33554432000 134512640 135107411 4292311152 18446744073709551615 134531613 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/2042/statm: 105451 1872 59 146 0 104327 0
[pid=2043] ppid=2028 vsize=313156 CPUtime=0.02 cores=1,3,5,7
/proc/2043/stat : 2043 (python) S 2028 2028 1708 0 -1 4202560 914 0 0 0 1 1 0 0 18 0 1 0 538387335 320671744 6533 33554432000 4194304 5683644 140735922807280 18446744073709551615 251814260294 0 0 16781312 514 15830838785462993158 0 0 17 1 0 0 0
/proc/2043/statm: 78289 6533 370 364 0 14198 0
[pid=2045] ppid=2043 vsize=421804 CPUtime=22.81 cores=1,3,5,7
/proc/2045/stat : 2045 (adaptg2wsat2009) R 2043 2045 2045 0 -1 4202496 2140 0 0 0 2281 0 0 0 25 0 1 0 538387335 431927296 1872 33554432000 134512640 135107411 4290547328 18446744073709551615 134531856 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/2045/statm: 105451 1872 59 146 0 104327 0
[pid=2028/tid=2035] ppid=2026 vsize=323400 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2035/stat : 2035 (python) S 2026 2028 1708 0 -1 4202560 26 934 0 0 0 0 0 0 22 0 5 0 538387326 331161600 8049 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 7 0 0 0
[pid=2028/tid=2038] ppid=2026 vsize=323400 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2038/stat : 2038 (python) S 2026 2028 1708 0 -1 4202560 24 934 0 0 0 0 0 0 24 0 5 0 538387329 331161600 8049 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 3 0 0 0
[pid=2028/tid=2041] ppid=2026 vsize=323400 CPUtime=0 cores=1,3,5,7

################
# More data... #
################

/proc/2043/stat : 2043 (python) S 2028 2028 1708 0 -1 4202560 1104 0 0 0 6 7 0 0 19 0 2 0 538387335 320671744 6563 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/2043/statm: 78289 6563 399 364 0 14198 0
[pid=2045] ppid=2043 vsize=421804 CPUtime=98.88 cores=1,3,5,7
/proc/2045/stat : 2045 (adaptg2wsat2009) T 2043 2045 2045 0 -1 4202496 2140 0 0 0 9888 0 0 0 25 0 1 0 538387335 431927296 1872 33554432000 134512640 135107411 4290547328 18446744073709551615 134531674 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/2045/statm: 105451 1872 59 146 0 104327 0
[pid=2043/tid=2054] ppid=2028 vsize=313156 CPUtime=0 cores=1,3,5,7
/proc/2043/task/2054/stat : 2054 (python) S 2028 2028 1708 0 -1 4202560 52 0 0 0 0 0 0 0 19 0 2 0 538397251 320671744 6563 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 514 18446604467583496320 0 0 -1 1 0 0 0
[pid=2055] ppid=2028 vsize=323548 CPUtime=0.14 cores=1,3,5,7
/proc/2055/stat : 2055 (python) S 2028 2028 1708 0 -1 4202560 1125 0 0 0 7 7 0 0 19 0 2 0 538397251 331313152 6568 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/2055/statm: 80887 6568 400 364 0 16793 0
[pid=2057] ppid=2055 vsize=421804 CPUtime=100.12 cores=1,3,5,7
/proc/2057/stat : 2057 (adaptg2wsat2009) T 2055 2057 2057 0 -1 4202496 2140 0 0 0 10012 0 0 0 25 0 1 0 538397252 431927296 1872 33554432000 134512640 135107411 4293596816 18446744073709551615 134532349 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/2057/statm: 105451 1872 59 146 0 104327 0
[pid=2055/tid=2079] ppid=2028 vsize=323548 CPUtime=0 cores=1,3,5,7
/proc/2055/task/2079/stat : 2079 (python) S 2028 2028 1708 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 538407266 331313152 6568 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 514 18446604460702572864 0 0 -1 1 0 0 0
[pid=2058] ppid=2028 vsize=333884 CPUtime=0.15 cores=1,3,5,7
/proc/2058/stat : 2058 (python) S 2028 2028 1708 0 -1 4202560 1124 0 0 0 8 7 0 0 19 0 2 0 538397255 341897216 6572 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2058/statm: 83471 6572 400 364 0 19374 0
[pid=2060] ppid=2058 vsize=1020188 CPUtime=99.18 cores=1,3,5,7
/proc/2060/stat : 2060 (gnovelty+2) T 2058 2060 2060 0 -1 4202496 1139 0 0 0 9918 0 0 0 25 0 1 0 538397255 1044672512 877 33554432000 134512640 135663704 4291272560 18446744073709551615 134524945 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/2060/statm: 255047 877 113 282 0 254763 0
[pid=2058/tid=2075] ppid=2028 vsize=333884 CPUtime=0 cores=1,3,5,7
/proc/2058/task/2075/stat : 2075 (python) S 2028 2028 1708 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 538407175 341897216 6572 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 514 18446604460702573760 0 0 -1 3 0 0 0
[pid=2063] ppid=2028 vsize=344232 CPUtime=0.15 cores=1,3,5,7
/proc/2063/stat : 2063 (python) S 2028 2028 1708 0 -1 4202560 1186 0 0 0 9 6 0 0 19 0 2 0 538397348 352493568 6582 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2063/statm: 86058 6582 406 364 0 21958 0
[pid=2065] ppid=2063 vsize=251296 CPUtime=99.98 cores=1,3,5,7
/proc/2065/stat : 2065 (FH) T 2063 2065 2065 0 -1 4202496 1960 0 0 0 9998 0 0 0 25 0 1 0 538397348 257327104 1694 33554432000 134512640 134936988 4287411808 18446744073709551615 134553505 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/2065/statm: 62824 1694 41 104 0 62328 0
[pid=2063/tid=2087] ppid=2028 vsize=344232 CPUtime=0 cores=1,3,5,7
/proc/2063/task/2087/stat : 2087 (python) S 2028 2028 1708 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538407362 352493568 6582 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 514 18446604460702575552 0 0 -1 7 0 0 0
[pid=2066] ppid=2028 vsize=354584 CPUtime=0.14 cores=1,3,5,7
/proc/2066/stat : 2066 (python) S 2028 2028 1708 0 -1 4202560 1179 0 0 0 8 6 0 0 19 0 2 0 538397351 363094016 6586 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/2066/statm: 88646 6586 406 364 0 24543 0
[pid=2068] ppid=2066 vsize=421804 CPUtime=97.2 cores=1,3,5,7
/proc/2068/stat : 2068 (adaptg2wsat2009) T 2066 2068 2068 0 -1 4202496 2141 0 0 0 9720 0 0 0 25 0 1 0 538397352 431927296 1871 33554432000 134512640 135107411 4293413776 18446744073709551615 134531600 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/2068/statm: 105451 1871 59 146 0 104327 0
[pid=2066/tid=2080] ppid=2028 vsize=354584 CPUtime=0 cores=1,3,5,7
/proc/2066/task/2080/stat : 2080 (python) S 2028 2028 1708 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 538407269 363094016 6586 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 514 18446604466917079936 0 0 -1 3 0 0 0
[pid=2076] ppid=2028 vsize=364840 CPUtime=0.15 cores=1,3,5,7
/proc/2076/stat : 2076 (python) S 2028 2028 1708 0 -1 4202560 1160 0 0 0 8 7 0 0 19 0 2 0 538407179 373596160 6584 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2076/statm: 91210 6584 400 364 0 27104 0
[pid=2078] ppid=2076 vsize=421804 CPUtime=100.12 cores=1,3,5,7
/proc/2078/stat : 2078 (adaptg2wsat2009) T 2076 2078 2078 0 -1 4202496 2142 0 0 0 10012 0 0 0 25 0 1 0 538407179 431927296 1873 33554432000 134512640 135107411 4290741216 18446744073709551615 134532224 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/2078/statm: 105451 1873 59 146 0 104327 0
[pid=2076/tid=2098] ppid=2028 vsize=364840 CPUtime=0 cores=1,3,5,7
/proc/2076/task/2098/stat : 2098 (python) S 2028 2028 1708 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 538417194 373596160 6584 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 514 18446604457842759616 0 0 -1 7 0 0 0
[pid=2081] ppid=2028 vsize=375208 CPUtime=0.16 cores=1,3,5,7
/proc/2081/stat : 2081 (python) S 2028 2028 1708 0 -1 4202560 1161 0 0 0 7 9 0 0 19 0 2 0 538407271 384212992 6588 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2081/statm: 93802 6588 400 364 0 29693 0
[pid=2083] ppid=2081 vsize=251292 CPUtime=102.8 cores=1,3,5,7
/proc/2083/stat : 2083 (FH) T 2081 2083 2083 0 -1 4202496 1963 0 0 0 10280 0 0 0 25 0 1 0 538407271 257323008 1694 33554432000 134512640 134936988 4287903632 18446744073709551615 134553554 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/2083/statm: 62823 1694 41 104 0 62327 0
[pid=2081/tid=2102] ppid=2028 vsize=375208 CPUtime=0 cores=1,3,5,7
/proc/2081/task/2102/stat : 2102 (python) S 2028 2028 1708 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538417787 384212992 6588 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 514 18446604457842757824 0 0 -1 1 0 0 0
[pid=2084] ppid=2028 vsize=385584 CPUtime=0.48 cores=1,3,5,7
/proc/2084/stat : 2084 (python) S 2028 2028 1708 0 -1 4202560 1199 0 0 0 26 22 0 0 18 0 2 0 538407275 394838016 6600 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2084/statm: 96396 6600 400 364 0 32284 0
[pid=2086] ppid=2084 vsize=268424 CPUtime=294.38 cores=1,3,5,7
/proc/2086/stat : 2086 (hybridGM3) T 2084 2086 2086 0 -1 4202496 5647 0 0 0 29438 0 0 0 25 0 1 0 538407276 274866176 5378 33554432000 134512640 135204385 4289177664 18446744073709551615 134538395 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/2086/statm: 67106 5378 63 169 0 66936 0
[pid=2084/tid=2116] ppid=2028 vsize=385584 CPUtime=0 cores=1,3,5,7
/proc/2084/task/2116/stat : 2116 (python) S 2028 2028 1708 0 -1 4202560 46 0 0 0 0 0 0 0 18 0 2 0 538426914 394838016 6600 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 514 18446604471107760576 0 0 -1 3 0 0 0
[pid=2088] ppid=2028 vsize=395848 CPUtime=0.15 cores=1,3,5,7
/proc/2088/stat : 2088 (python) S 2028 2028 1708 0 -1 4202560 1164 0 0 0 8 7 0 0 19 0 2 0 538407366 405348352 6634 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2088/statm: 98962 6634 400 364 0 34847 0
[pid=2090] ppid=2088 vsize=1020188 CPUtime=98.2 cores=1,3,5,7
/proc/2090/stat : 2090 (gnovelty+2) T 2088 2090 2090 0 -1 4202496 1149 0 0 0 9820 0 0 0 25 0 1 0 538407367 1044672512 876 33554432000 134512640 135663704 4291754928 18446744073709551615 134524897 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/2090/statm: 255047 876 113 282 0 254763 0
[pid=2088/tid=2094] ppid=2028 vsize=395848 CPUtime=0 cores=1,3,5,7
/proc/2088/task/2094/stat : 2094 (python) S 2028 2028 1708 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 538417187 405348352 6634 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 514 18446604471107761472 0 0 -1 1 0 0 0
[pid=2095] ppid=2028 vsize=405976 CPUtime=0.34 cores=1,3,5,7
/proc/2095/stat : 2095 (python) S 2028 2028 1708 0 -1 4202560 1194 0 0 0 17 17 0 0 18 0 2 0 538417192 415719424 6640 33554432000 4194304 5683644 140735922807280 18446744073709551615 251814260294 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2095/statm: 101494 6640 400 364 0 37376 0
[pid=2097] ppid=2095 vsize=268424 CPUtime=205.75 cores=1,3,5,7
/proc/2097/stat : 2097 (hybridGM3) R 2095 2097 2097 0 -1 4202496 5650 0 0 0 20575 0 0 0 25 0 1 0 538417192 274866176 5378 33554432000 134512640 135204385 4291388416 18446744073709551615 134538368 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/2097/statm: 67106 5378 63 169 0 66936 0
[pid=2095/tid=2134] ppid=2028 vsize=405976 CPUtime=0 cores=1,3,5,7
/proc/2095/task/2134/stat : 2134 (python) S 2028 2028 1708 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538437139 415719424 6640 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 514 18446604471158782144 0 0 -1 7 0 0 0
[pid=2099] ppid=2028 vsize=416360 CPUtime=0.17 cores=1,3,5,7
/proc/2099/stat : 2099 (python) S 2028 2028 1708 0 -1 4202560 1172 0 0 0 10 7 0 0 19 0 2 0 538417198 426352640 6644 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2099/statm: 104090 6644 400 364 0 39969 0
[pid=2101] ppid=2099 vsize=421804 CPUtime=101.93 cores=1,3,5,7
/proc/2101/stat : 2101 (adaptg2wsat2009) T 2099 2101 2101 0 -1 4202496 2148 0 0 0 10193 0 0 0 25 0 1 0 538417199 431927296 1871 33554432000 134512640 135107411 4294871296 18446744073709551615 134531239 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/2101/statm: 105451 1871 59 146 0 104327 0
[pid=2099/tid=2117] ppid=2028 vsize=416360 CPUtime=0 cores=1,3,5,7
/proc/2099/task/2117/stat : 2117 (python) S 2028 2028 1708 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538427617 426352640 6644 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 514 18446604471096568768 0 0 -1 3 0 0 0
[pid=2103] ppid=2028 vsize=426624 CPUtime=0.33 cores=1,3,5,7
/proc/2103/stat : 2103 (python) S 2028 2028 1708 0 -1 4202560 1236 0 0 0 17 16 0 0 19 0 2 0 538417792 436862976 6656 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2103/statm: 106656 6656 408 364 0 42532 0
[pid=2105] ppid=2103 vsize=268424 CPUtime=199.45 cores=1,3,5,7
/proc/2105/stat : 2105 (hybridGM3) T 2103 2105 2105 0 -1 4202496 5652 0 0 0 19945 0 0 0 25 0 1 0 538417793 274866176 5378 33554432000 134512640 135204385 4286895504 18446744073709551615 134538494 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/2105/statm: 67106 5378 63 169 0 66936 0
[pid=2103/tid=2135] ppid=2028 vsize=426624 CPUtime=0 cores=1,3,5,7
/proc/2103/task/2135/stat : 2135 (python) S 2028 2028 1708 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 538437739 436862976 6656 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 514 18446604454225329088 0 0 -1 1 0 0 0
[pid=2118] ppid=2028 vsize=436728 CPUtime=0.16 cores=1,3,5,7
/proc/2118/stat : 2118 (python) S 2028 2028 1708 0 -1 4202560 1086 0 0 0 9 7 0 0 18 0 1 0 538427622 447209472 6672 33554432000 4194304 5683644 140735922807280 18446744073709551615 251814260294 0 0 16781312 514 15830838785462993159 0 0 17 7 0 0 0
/proc/2118/statm: 109182 6672 380 364 0 45055 0
[pid=2120] ppid=2118 vsize=268420 CPUtime=98.35 cores=1,3,5,7
/proc/2120/stat : 2120 (hybridGM3) R 2118 2120 2120 0 -1 4202496 5654 0 0 0 9835 0 0 0 25 0 1 0 538427624 274862080 5377 33554432000 134512640 135204385 4293772224 18446744073709551615 134538044 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/2120/statm: 67105 5377 63 169 0 66935 0
[pid=2131] ppid=2028 vsize=447848 CPUtime=0.01 cores=1,3,5,7
/proc/2131/stat : 2131 (python) S 2028 2028 1708 0 -1 4202560 1024 0 0 0 0 1 0 0 18 0 1 0 538436744 458596352 6671 33554432000 4194304 5683644 140735922807280 18446744073709551615 251814260294 0 0 16781312 514 15830838785462993157 0 0 17 3 0 0 0
/proc/2131/statm: 111962 6671 374 364 0 47832 0
[pid=2133] ppid=2131 vsize=459768 CPUtime=10.1 cores=1,3,5,7
/proc/2133/stat : 2133 (TNM) R 2131 2133 2133 0 -1 4202496 1929 0 0 0 1010 0 0 0 25 0 1 0 538436745 470802432 1657 33554432000 134512640 135008748 4294122544 18446744073709551615 134529517 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/2133/statm: 114942 1657 46 122 0 113842 0
[pid=2136] ppid=2028 vsize=457848 CPUtime=0 cores=1,3,5,7
/proc/2136/stat : 2136 (python) S 2028 2028 1708 0 -1 4202560 822 0 0 0 0 0 0 0 19 0 1 0 538437744 468836352 6722 33554432000 4194304 5683644 140735922807280 18446744073709551615 251814260294 0 0 16781312 514 3624765210264338431 0 0 17 3 0 0 0
/proc/2136/statm: 114462 6722 347 364 0 50329 0
[pid=2138] ppid=2136 vsize=249144 CPUtime=0.31 cores=1,3,5,7
/proc/2138/stat : 2138 (FH) R 2136 2138 2138 0 -1 4202496 1568 0 0 0 31 0 0 0 21 0 1 0 538437744 255123456 1295 33554432000 134512640 134936988 4288457136 18446744073709551615 134523354 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/2138/statm: 62286 1295 37 104 0 61790 0
[pid=2028/tid=2035] ppid=2026 vsize=467608 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2035/stat : 2035 (python) S 2026 2028 1708 0 -1 4202560 26 934 0 0 0 0 0 0 22 0 19 0 538387326 478830592 8290 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 7 0 0 0
[pid=2028/tid=2038] ppid=2026 vsize=467608 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2038/stat : 2038 (python) S 2026 2028 1708 0 -1 4202560 24 934 0 0 0 0 0 0 24 0 19 0 538387329 478830592 8290 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 3 0 0 0
[pid=2028/tid=2041] ppid=2026 vsize=467608 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2041/stat : 2041 (python) S 2026 2028 1708 0 -1 4202560 26 934 0 0 0 0 0 0 25 0 19 0 538387332 478830592 8290 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 7 0 0 0
[pid=2028/tid=2044] ppid=2026 vsize=467608 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2044/stat : 2044 (python) S 2026 2028 1708 0 -1 4202560 26 934 0 0 0 0 0 0 25 0 19 0 538387335 478830592 8290 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 1 0 0 0
[pid=2028/tid=2056] ppid=2026 vsize=467608 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2056/stat : 2056 (python) S 2026 2028 1708 0 -1 4202560 26 934 0 0 0 0 0 0 20 0 19 0 538397252 478830592 8290 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 1 0 0 0
[pid=2028/tid=2059] ppid=2026 vsize=467608 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2059/stat : 2059 (python) S 2026 2028 1708 0 -1 4202560 29 934 0 0 0 0 0 0 22 0 19 0 538397255 478830592 8290 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 7 0 0 0
[pid=2028/tid=2064] ppid=2026 vsize=467608 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2064/stat : 2064 (python) S 2026 2028 1708 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 19 0 538397348 478830592 8290 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 7 0 0 0
[pid=2028/tid=2067] ppid=2026 vsize=467608 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2067/stat : 2067 (python) S 2026 2028 1708 0 -1 4202560 29 934 0 0 0 0 0 0 22 0 19 0 538397352 478830592 8290 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 7 0 0 0
[pid=2028/tid=2077] ppid=2026 vsize=467608 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2077/stat : 2077 (python) S 2026 2028 1708 0 -1 4202560 30 934 0 0 0 0 0 0 20 0 19 0 538407179 478830592 8290 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 7 0 0 0
[pid=2028/tid=2082] ppid=2026 vsize=467608 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2082/stat : 2082 (python) S 2026 2028 1708 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 19 0 538407271 478830592 8290 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 7 0 0 0
[pid=2028/tid=2085] ppid=2026 vsize=467608 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2085/stat : 2085 (python) S 2026 2028 1708 0 -1 4202560 33 934 0 0 0 0 0 0 18 0 19 0 538407276 478830592 8290 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 7 0 0 0
[pid=2028/tid=2089] ppid=2026 vsize=467608 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2089/stat : 2089 (python) S 2026 2028 1708 0 -1 4202560 30 934 0 0 0 0 0 0 20 0 19 0 538407367 478830592 8290 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 7 0 0 0
[pid=2028/tid=2096] ppid=2026 vsize=467608 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2096/stat : 2096 (python) S 2026 2028 1708 0 -1 4202560 31 934 0 0 0 0 0 0 18 0 19 0 538417192 478830592 8290 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 7 0 0 0
[pid=2028/tid=2100] ppid=2026 vsize=467608 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2100/stat : 2100 (python) S 2026 2028 1708 0 -1 4202560 31 934 0 0 0 0 0 0 21 0 19 0 538417199 478830592 8290 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 7 0 0 0
[pid=2028/tid=2104] ppid=2026 vsize=467608 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2104/stat : 2104 (python) S 2026 2028 1708 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 19 0 538417792 478830592 8290 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 7 0 0 0
[pid=2028/tid=2119] ppid=2026 vsize=467608 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2119/stat : 2119 (python) S 2026 2028 1708 0 -1 4202560 28 934 0 0 0 0 0 0 20 0 19 0 538427622 478830592 8290 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 7 0 0 0
[pid=2028/tid=2132] ppid=2026 vsize=467608 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2132/stat : 2132 (python) S 2026 2028 1708 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 19 0 538436745 478830592 8290 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 3 0 0 0
[pid=2028/tid=2137] ppid=2026 vsize=467608 CPUtime=0 cores=1,3,5,7
/proc/2028/task/2137/stat : 2137 (python) S 2026 2028 1708 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 19 0 538437744 478830592 8290 33554432000 4194304 5683644 140735922807280 18446744073709551615 251826063793 0 0 16781312 2 18446604471162978368 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2011.55
Current children cumulated vsize (KiB) 16035160

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 2028 and gives
#  childrusage.ru_utime.tv_sec=3
#  childrusage.ru_utime.tv_usec=48536
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=158975
# CPU time returned by wait4() is 3.20751
# while last known CPU time is 2011.55
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 507.119
CPU time (s): 2011.55
CPU user time (s): 2009.97
CPU system time (s): 1.58
CPU usage (%): 396.662
Max. virtual memory (cumulated for all children) (KiB): 16035160

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 3.04854
system time used= 0.158975
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 31901
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1427
involuntary context switches= 74

runsolver used 2.26865 second user time and 6.09907 second system time

The end

Launcher Data 

Begin job on node126 at 2011-05-01 18:38:04
IDJOB=3317164
IDBENCH=82629
IDSOLVER=1705
FILE ID=node126/3317164-1304267884
RUNJOBID= node126-1304267283-1727
PBS_JOBID= 13173953
Free space on /tmp= 73224 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k7-r85-v400-c34000-S1275797759-041-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317164-1304267884/watcher-3317164-1304267884 -o /tmp/evaluation-result-3317164-1304267884/solver-3317164-1304267884 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317164-1304267884.cnf -seed 824782661 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 374e3ba6783d13d6d478972802d8dba0
RANDOM SEED=824782661

node126.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.56
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      30272612 kB
Buffers:        314440 kB
Cached:        1663332 kB
SwapCached:         76 kB
Active:         672780 kB
Inactive:      1679984 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      30272612 kB
SwapTotal:    67111528 kB
SwapFree:     67111448 kB
Dirty:           33448 kB
Writeback:           0 kB
AnonPages:      374548 kB
Mapped:          21240 kB
Slab:           253992 kB
PageTables:      12424 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:  3938620 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73192 MiB
End job on node126 at 2011-05-01 18:46:33