Trace number 3378106

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (MO) 2742.62 691.939

General information on the benchmark

Namerandom/large/
unif-k7-r85-v250-c21250-S1682211323-009-UNKNOWN.cnf
MD5SUM917ee0efab4f0e89027865c1129d272b
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables250
Number of clauses21250
Sum of the clauses size148750
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 521250

Solver Data 

0.41/0.79	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.41/0.79	c INFO: solving HOME/instance-3378106-1305462668.cnf
0.41/0.79	c INFO: computing task features
0.57/0.93	c INFO: computed variable-clause graph statistics
0.57/0.93	c INFO: computed clause balance statistics
1.37/1.70	c INFO: computed variable balance statistics
1.67/2.02	c INFO: computed small-clause counts
4.99/2.91	c INFO: computed Horn-clause counts
4.99/2.91	c INFO: computed variable graph statistics
4.99/2.91	c INFO: collected features for HOME/instance-3378106-1305462668.cnf in 1.26 s
4.99/2.91	c INFO: running gnovelty+2@0 for 97 with 1198 remaining (b = 0.36)
4.99/2.91	c INFO: running gnovelty+2@0 for 97 with 1198 remaining (b = 0.13)
4.99/2.91	c INFO: running adaptg2wsat2009++@0 for 97 with 1198 remaining (b = 0.08)
4.99/2.91	c INFO: running adaptg2wsat2009++@0 for 97 with 1198 remaining (b = 0.07)
401.51/102.40	c INFO: running adaptg2wsat2009++@0 for 97 with 1098 remaining (b = 0.05)
401.51/102.43	c INFO: running gnovelty+2@0 for 97 with 1098 remaining (b = 0.04)
401.51/102.44	c INFO: running FH@0 for 97 with 1098 remaining (b = 0.03)
401.51/102.46	c INFO: running adaptg2wsat2009++@0 for 97 with 1098 remaining (b = 0.03)
796.72/201.76	c INFO: running adaptg2wsat2009++@0 for 97 with 999 remaining (b = 0.02)
801.08/202.80	c INFO: running FH@0 for 97 with 997 remaining (b = 0.02)
801.08/202.82	c INFO: running hybridGM3@0 for 194 with 997 remaining (b = 0.04)
816.60/206.81	c INFO: running gnovelty+2@0 for 97 with 993 remaining (b = 0.02)
1195.74/302.11	c INFO: running hybridGM3@0 for 194 with 898 remaining (b = 0.03)
1199.69/303.15	c INFO: running adaptg2wsat2009++@0 for 97 with 897 remaining (b = 0.02)
1235.86/312.24	c INFO: running hybridGM3@0 for 194 with 888 remaining (b = 0.03)
1598.67/403.51	c INFO: running FH@0 for 97 with 797 remaining (b = 0.01)
1598.67/403.56	c INFO: running hybridGM3@101 for 97 with 797 remaining (b = 0.01)
1973.24/497.84	c INFO: running hybridGM3@99 for 97 with 702 remaining (b = 0.01)
1977.19/498.90	c INFO: running hybridGM3@297 for 97 with 701 remaining (b = 0.01)
1997.10/503.88	c INFO: running hybridGM3@0 for 194 with 696 remaining (b = 0.02)
2041.41/515.02	c INFO: running hybridGM3@98 for 97 with 685 remaining (b = 0.01)
2355.51/594.22	c INFO: running adaptg2wsat2009++@0 for 97 with 606 remaining (b = 0.01)
2374.97/599.18	c INFO: running hybridGM3@193 for 97 with 601 remaining (b = 0.01)
2419.38/610.33	c INFO: running adaptg2wsat2009++@100 for 97 with 590 remaining (b = 0.01)
2741.34/691.57	c INFO: running gnovelty+2@0 for 97 with 509 remaining (b = 0.01)
2742.62/691.87	c INFO: running hybridGM3@397 for 97 with 508 remaining (b = 0.01)
2742.62/691.91	c INFO: running hybridGM3@0 for 194 with 508 remaining (b = 0.02)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378106-1305462668/watcher-3378106-1305462668 -o /tmp/evaluation-result-3378106-1305462668/solver-3378106-1305462668 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378106-1305462668.cnf -seed 1912783864 -budget 4800 -cores 4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.39 3.86 3.31 3/412 17019
/proc/meminfo: memFree=25236820/32951124 swapFree=53501244/67111528
[pid=17019] ppid=17017 vsize=16240 CPUtime=0 cores=0,2,4,6
/proc/17019/stat : 17019 (python) R 17017 17019 16982 0 -1 4202496 1033 0 0 0 0 0 0 0 17 0 1 0 657869429 16629760 906 33554432000 4194304 5683644 140736280637904 18446744073709551615 216633973072 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/17019/statm: 4060 906 380 364 0 547 0

[startup+0.051614 s]
/proc/loadavg: 3.39 3.86 3.31 3/412 17019
/proc/meminfo: memFree=25236820/32951124 swapFree=53501244/67111528
[pid=17019] ppid=17017 vsize=18884 CPUtime=0.02 cores=0,2,4,6
/proc/17019/stat : 17019 (python) D 17017 17019 16982 0 -1 4202496 1287 0 0 0 2 0 0 0 17 0 1 0 657869429 19337216 1141 33554432000 4194304 5683644 140736280637904 18446744073709551615 216634513408 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/17019/statm: 4721 1141 421 364 0 690 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 18884

[startup+0.100594 s]
/proc/loadavg: 3.39 3.86 3.31 3/412 17019
/proc/meminfo: memFree=25236820/32951124 swapFree=53501244/67111528
[pid=17019] ppid=17017 vsize=54536 CPUtime=0.06 cores=0,2,4,6
/proc/17019/stat : 17019 (python) S 17017 17019 16982 0 -1 4202496 2055 0 0 0 4 2 0 0 17 0 1 0 657869429 55844864 1814 33554432000 4194304 5683644 140736280637904 18446744073709551615 216634513408 0 2147483391 16781312 2 18446744071699945619 0 0 17 4 0 0 0
/proc/17019/statm: 13634 1814 492 364 0 1284 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 54536

[startup+0.300544 s]
/proc/loadavg: 3.39 3.86 3.31 3/412 17019
/proc/meminfo: memFree=25236820/32951124 swapFree=53501244/67111528
[pid=17019] ppid=17017 vsize=135584 CPUtime=0.19 cores=0,2,4,6
/proc/17019/stat : 17019 (python) S 17017 17019 16982 0 -1 4202496 5799 933 1 0 12 7 0 0 18 0 1 0 657869429 138838016 4121 33554432000 4194304 5683644 140736280637904 18446744073709551615 216634511205 0 2147483391 16781312 2 18446744071699199166 0 0 17 4 0 0 0
/proc/17019/statm: 33896 4121 1031 364 0 3082 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 135584

[startup+0.700513 s]
/proc/loadavg: 3.39 3.86 3.31 3/412 17019
/proc/meminfo: memFree=25236820/32951124 swapFree=53501244/67111528
[pid=17019] ppid=17017 vsize=246316 CPUtime=0.41 cores=0,2,4,6
/proc/17019/stat : 17019 (python) R 17017 17019 16982 0 -1 4202496 8149 933 1 0 28 13 0 0 16 0 1 0 657869429 252227584 6049 33554432000 4194304 5683644 140736280637904 18446744073709551615 216634176197 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/17019/statm: 61579 6049 1621 364 0 4581 0
Current children cumulated CPU time (s) 0.41
Current children cumulated vsize (KiB) 246316

[startup+1.5004 s]
/proc/loadavg: 3.39 3.86 3.31 4/414 17029
/proc/meminfo: memFree=25177648/32951124 swapFree=53501244/67111528
[pid=17019] ppid=17017 vsize=283916 CPUtime=1.17 cores=0,2,4,6
/proc/17019/stat : 17019 (python) R 17017 17019 16982 0 -1 4202496 11840 933 1 0 102 15 0 0 16 0 1 0 657869429 290729984 8364 33554432000 4194304 5683644 140736280637904 18446744073709551615 47599583671826 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/17019/statm: 70979 8364 1826 364 0 6903 0
Current children cumulated CPU time (s) 1.17
Current children cumulated vsize (KiB) 283916

[startup+3.10101 s]
/proc/loadavg: 3.39 3.86 3.31 9/426 17041
/proc/meminfo: memFree=25161524/32951124 swapFree=53501244/67111528
[pid=17019] ppid=17017 vsize=321108 CPUtime=1.76 cores=0,2,4,6
/proc/17019/stat : 17019 (python) S 17017 17019 16982 0 -1 4202496 14588 933 1 0 160 16 0 0 23 0 5 0 657869429 328814592 7468 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646342091 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/17019/statm: 80277 7468 1875 364 0 16186 0
[pid=17030] ppid=17019 vsize=279940 CPUtime=0 cores=0,2,4,6
/proc/17030/stat : 17030 (python) S 17019 17019 16982 0 -1 4202560 723 0 0 0 0 0 0 0 18 0 1 0 657869634 286658560 5773 33554432000 4194304 5683644 140736280637904 18446744073709551615 216634535439 0 0 16781312 514 13813231654254149631 0 0 17 6 0 0 0
/proc/17030/statm: 69985 5773 355 364 0 5903 0
[pid=17032] ppid=17030 vsize=1018820 CPUtime=1 cores=0,2,4,6
/proc/17032/stat : 17032 (gnovelty+2) R 17030 17032 17032 0 -1 4202496 857 0 0 0 100 0 0 0 22 0 1 0 657869635 1043271680 595 33554432000 134512640 135663704 4290886928 18446744073709551615 134525255 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/17032/statm: 254705 595 113 282 0 254421 0
[pid=17033] ppid=17019 vsize=290836 CPUtime=0 cores=0,2,4,6
/proc/17033/stat : 17033 (python) S 17019 17019 16982 0 -1 4202560 731 0 0 0 0 0 0 0 20 0 1 0 657869636 297816064 5796 33554432000 4194304 5683644 140736280637904 18446744073709551615 216634535494 0 0 16781312 514 10763622080286031871 0 0 17 0 0 0 0
/proc/17033/statm: 72709 5796 349 364 0 8624 0
[pid=17037] ppid=17033 vsize=1018820 CPUtime=1.01 cores=0,2,4,6
/proc/17037/stat : 17037 (gnovelty+2) R 17033 17037 17037 0 -1 4202496 857 0 0 0 101 0 0 0 25 0 1 0 657869637 1043271680 595 33554432000 134512640 135663704 4293265920 18446744073709551615 134524548 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/17037/statm: 254705 595 113 282 0 254421 0
[pid=17035] ppid=17019 vsize=300608 CPUtime=0 cores=0,2,4,6
/proc/17035/stat : 17035 (python) S 17019 17019 16982 0 -1 4202560 732 0 0 0 0 0 0 0 18 0 1 0 657869637 307822592 5932 33554432000 4194304 5683644 140736280637904 18446744073709551615 216634535494 0 0 16781312 514 6729586210865414143 0 0 17 4 0 0 0
/proc/17035/statm: 75152 5932 347 364 0 11064 0
[pid=17038] ppid=17035 vsize=419160 CPUtime=1.01 cores=0,2,4,6
/proc/17038/stat : 17038 (adaptg2wsat2009) R 17035 17038 17038 0 -1 4202496 1255 0 0 0 101 0 0 0 25 0 1 0 657869637 429219840 990 33554432000 134512640 135107411 4292458960 18446744073709551615 134523438 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/17038/statm: 104790 990 49 146 0 103666 0
[pid=17039] ppid=17019 vsize=310864 CPUtime=0 cores=0,2,4,6
/proc/17039/stat : 17039 (python) S 17019 17019 16982 0 -1 4202560 725 0 0 0 0 0 0 0 23 0 1 0 657869638 318324736 5933 33554432000 4194304 5683644 140736280637904 18446744073709551615 216634535494 0 0 16781312 514 7651730945040449535 0 0 17 4 0 0 0
/proc/17039/statm: 77716 5933 343 364 0 13625 0
[pid=17041] ppid=17039 vsize=419160 CPUtime=1 cores=0,2,4,6
/proc/17041/stat : 17041 (adaptg2wsat2009) R 17039 17041 17041 0 -1 4202496 1255 0 0 0 100 0 0 0 25 0 1 0 657869638 429219840 990 33554432000 134512640 135107411 4292094640 18446744073709551615 134523455 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/17041/statm: 104790 990 49 146 0 103666 0
[pid=17019/tid=17031] ppid=17017 vsize=321108 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17031/stat : 17031 (python) S 17017 17019 16982 0 -1 4202560 25 933 0 0 0 0 0 0 19 0 5 0 657869635 328814592 7468 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 4 0 0 0
[pid=17019/tid=17034] ppid=17017 vsize=321108 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17034/stat : 17034 (python) S 17017 17019 16982 0 -1 4202560 40 933 0 0 0 0 0 0 21 0 5 0 657869636 328814592 7468 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 4 0 0 0
[pid=17019/tid=17036] ppid=17017 vsize=321108 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17036/stat : 17036 (python) S 17017 17019 16982 0 -1 4202560 15 933 0 0 0 0 0 0 23 0 5 0 657869637 328814592 7468 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 4 0 0 0
[pid=17019/tid=17040] ppid=17017 vsize=321108 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17040/stat : 17040 (python) S 17017 17019 16982 0 -1 4202560 26 933 0 0 0 0 0 0 24 0 5 0 657869638 328814592 7468 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 5.78
Current children cumulated vsize (KiB) 4379316

[startup+6.30144 s]
/proc/loadavg: 3.76 3.92 3.33 10/438 17053
/proc/meminfo: memFree=25127248/32951124 swapFree=53501244/67111528
[pid=17019] ppid=17017 vsize=321108 CPUtime=1.76 cores=0,2,4,6
/proc/17019/stat : 17019 (python) S 17017 17019 16982 0 -1 4202496 14588 933 1 0 160 16 0 0 23 0 5 0 657869429 328814592 7468 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646342091 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/17019/statm: 80277 7468 1875 364 0 16186 0
[pid=17030] ppid=17019 vsize=279940 CPUtime=0.01 cores=0,2,4,6
/proc/17030/stat : 17030 (python) S 17019 17019 16982 0 -1 4202560 882 0 0 0 1 0 0 0 18 0 1 0 657869634 286658560 5796 33554432000 4194304 5683644 140736280637904 18446744073709551615 216634535439 0 0 16781312 514 13813231654254117122 0 0 17 6 0 0 0
/proc/17030/statm: 69985 5796 378 364 0 5903 0
[pid=17032] ppid=17030 vsize=1018820 CPUtime=4.14 cores=0,2,4,6
/proc/17032/stat : 17032 (gnovelty+2) R 17030 17032 17032 0 -1 4202496 857 0 0 0 414 0 0 0 25 0 1 0 657869635 1043271680 595 33554432000 134512640 135663704 4290886928 18446744073709551615 134525255 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/17032/statm: 254705 595 113 282 0 254421 0
[pid=17033] ppid=17019 vsize=290836 CPUtime=0.01 cores=0,2,4,6
/proc/17033/stat : 17033 (python) S 17019 17019 16982 0 -1 4202560 888 0 0 0 1 0 0 0 18 0 1 0 657869636 297816064 5819 33554432000 4194304 5683644 140736280637904 18446744073709551615 216634535494 0 0 16781312 514 10763622080285999362 0 0 17 0 0 0 0
/proc/17033/statm: 72709 5819 372 364 0 8624 0
[pid=17037] ppid=17033 vsize=1018820 CPUtime=4.19 cores=0,2,4,6
/proc/17037/stat : 17037 (gnovelty+2) R 17033 17037 17037 0 -1 4202496 857 0 0 0 419 0 0 0 25 0 1 0 657869637 1043271680 595 33554432000 134512640 135663704 4293265920 18446744073709551615 134525224 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/17037/statm: 254705 595 113 282 0 254421 0
[pid=17035] ppid=17019 vsize=300608 CPUtime=0.01 cores=0,2,4,6
/proc/17035/stat : 17035 (python) S 17019 17019 16982 0 -1 4202560 892 0 0 0 1 0 0 0 18 0 1 0 657869637 307822592 5955 33554432000 4194304 5683644 140736280637904 18446744073709551615 216634535494 0 0 16781312 514 6729586210865381636 0 0 17 4 0 0 0
/proc/17035/statm: 75152 5955 370 364 0 11064 0
[pid=17038] ppid=17035 vsize=419428 CPUtime=4.21 cores=0,2,4,6
/proc/17038/stat : 17038 (adaptg2wsat2009) R 17035 17038 17038 0 -1 4202496 1439 0 0 0 421 0 0 0 25 0 1 0 657869637 429494272 1174 33554432000 134512640 135107411 4292458960 18446744073709551615 134531817 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/17038/statm: 104857 1174 59 146 0 103733 0
[pid=17039] ppid=17019 vsize=310864 CPUtime=0.01 cores=0,2,4,6
/proc/17039/stat : 17039 (python) S 17019 17019 16982 0 -1 4202560 904 0 0 0 0 1 0 0 18 0 1 0 657869638 318324736 5959 33554432000 4194304 5683644 140736280637904 18446744073709551615 216634535494 0 0 16781312 514 7651730945040417025 0 0 17 4 0 0 0
/proc/17039/statm: 77716 5959 369 364 0 13625 0
[pid=17041] ppid=17039 vsize=419428 CPUtime=4.16 cores=0,2,4,6
/proc/17041/stat : 17041 (adaptg2wsat2009) R 17039 17041 17041 0 -1 4202496 1440 0 0 0 416 0 0 0 25 0 1 0 657869638 429494272 1175 33554432000 134512640 135107411 4292094640 18446744073709551615 134532277 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/17041/statm: 104857 1175 59 146 0 103733 0
[pid=17019/tid=17031] ppid=17017 vsize=321108 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17031/stat : 17031 (python) S 17017 17019 16982 0 -1 4202560 25 933 0 0 0 0 0 0 19 0 5 0 657869635 328814592 7468 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 4 0 0 0
[pid=17019/tid=17034] ppid=17017 vsize=321108 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17034/stat : 17034 (python) S 17017 17019 16982 0 -1 4202560 40 933 0 0 0 0 0 0 21 0 5 0 657869636 328814592 7468 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 4 0 0 0
[pid=17019/tid=17036] ppid=17017 vsize=321108 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17036/stat : 17036 (python) S 17017 17019 16982 0 -1 4202560 15 933 0 0 0 0 0 0 23 0 5 0 657869637 328814592 7468 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 4 0 0 0
[pid=17019/tid=17040] ppid=17017 vsize=321108 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17040/stat : 17040 (python) S 17017 17019 16982 0 -1 4202560 26 933 0 0 0 0 0 0 24 0 5 0 657869638 328814592 7468 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 18.5
Current children cumulated vsize (KiB) 4379852

[startup+12.7013 s]
/proc/loadavg: 4.10 3.99 3.36 10/438 17053
/proc/meminfo: memFree=25121224/32951124 swapFree=53501244/67111528
[pid=17019] ppid=17017 vsize=321108 CPUtime=1.76 cores=0,2,4,6
/proc/17019/stat : 17019 (python) S 17017 17019 16982 0 -1 4202496 14588 933 1 0 160 16 0 0 23 0 5 0 657869429 328814592 7468 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646342091 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/17019/statm: 80277 7468 1875 364 0 16186 0
[pid=17030] ppid=17019 vsize=279940 CPUtime=0.03 cores=0,2,4,6
/proc/17030/stat : 17030 (python) S 17019 17019 16982 0 -1 4202560 885 0 0 0 2 1 0 0 18 0 1 0 657869634 286658560 5796 33554432000 4194304 5683644 140736280637904 18446744073709551615 216634535439 0 0 16781312 514 11606743809966178562 0 0 17 6 0 0 0
/proc/17030/statm: 69985 5796 378 364 0 5903 0
[pid=17032] ppid=17030 vsize=1018820 CPUtime=10.43 cores=0,2,4,6
/proc/17032/stat : 17032 (gnovelty+2) R 17030 17032 17032 0 -1 4202496 857 0 0 0 1043 0 0 0 25 0 1 0 657869635 1043271680 595 33554432000 134512640 135663704 4290886928 18446744073709551615 134524882 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/17032/statm: 254705 595 113 282 0 254421 0
[pid=17033] ppid=17019 vsize=290836 CPUtime=0.02 cores=0,2,4,6
/proc/17033/stat : 17033 (python) S 17019 17019 16982 0 -1 4202560 893 0 0 0 1 1 0 0 18 0 1 0 657869636 297816064 5819 33554432000 4194304 5683644 140736280637904 18446744073709551615 216634535494 0 0 16781312 514 11606743809966178562 0 0 17 0 0 0 0
/proc/17033/statm: 72709 5819 372 364 0 8624 0
[pid=17037] ppid=17033 vsize=1018820 CPUtime=10.58 cores=0,2,4,6
/proc/17037/stat : 17037 (gnovelty+2) R 17033 17037 17037 0 -1 4202496 857 0 0 0 1058 0 0 0 25 0 1 0 657869637 1043271680 595 33554432000 134512640 135663704 4293265920 18446744073709551615 134524548 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/17037/statm: 254705 595 113 282 0 254421 0
[pid=17035] ppid=17019 vsize=300608 CPUtime=0.03 cores=0,2,4,6
/proc/17035/stat : 17035 (python) S 17019 17019 16982 0 -1 4202560 895 0 0 0 2 1 0 0 18 0 1 0 657869637 307822592 5955 33554432000 4194304 5683644 140736280637904 18446744073709551615 216634535494 0 0 16781312 514 11606743809966178564 0 0 17 4 0 0 0
/proc/17035/statm: 75152 5955 370 364 0 11064 0
[pid=17038] ppid=17035 vsize=419428 CPUtime=10.61 cores=0,2,4,6
/proc/17038/stat : 17038 (adaptg2wsat2009) R 17035 17038 17038 0 -1 4202496 1439 0 0 0 1061 0 0 0 25 0 1 0 657869637 429494272 1174 33554432000 134512640 135107411 4292458960 18446744073709551615 134531781 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/17038/statm: 104857 1174 59 146 0 103733 0
[pid=17039] ppid=17019 vsize=310864 CPUtime=0.03 cores=0,2,4,6
/proc/17039/stat : 17039 (python) S 17019 17019 16982 0 -1 4202560 907 0 0 0 2 1 0 0 18 0 1 0 657869638 318324736 5960 33554432000 4194304 5683644 140736280637904 18446744073709551615 216634535494 0 0 16781312 514 11606743809966178561 0 0 17 4 0 0 0
/proc/17039/statm: 77716 5960 370 364 0 13625 0
[pid=17041] ppid=17039 vsize=419428 CPUtime=10.52 cores=0,2,4,6
/proc/17041/stat : 17041 (adaptg2wsat2009) R 17039 17041 17041 0 -1 4202496 1440 0 0 0 1052 0 0 0 25 0 1 0 657869638 429494272 1175 33554432000 134512640 135107411 4292094640 18446744073709551615 134538035 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/17041/statm: 104857 1175 59 146 0 103733 0
[pid=17019/tid=17031] ppid=17017 vsize=321108 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17031/stat : 17031 (python) S 17017 17019 16982 0 -1 4202560 25 933 0 0 0 0 0 0 19 0 5 0 657869635 328814592 7468 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 4 0 0 0
[pid=17019/tid=17034] ppid=17017 vsize=321108 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17034/stat : 17034 (python) S 17017 17019 16982 0 -1 4202560 40 933 0 0 0 0 0 0 21 0 5 0 657869636 328814592 7468 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 4 0 0 0
[pid=17019/tid=17036] ppid=17017 vsize=321108 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17036/stat : 17036 (python) S 17017 17019 16982 0 -1 4202560 15 933 0 0 0 0 0 0 23 0 5 0 657869637 328814592 7468 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 4 0 0 0
[pid=17019/tid=17040] ppid=17017 vsize=321108 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17040/stat : 17040 (python) S 17017 17019 16982 0 -1 4202560 26 933 0 0 0 0 0 0 24 0 5 0 657869638 328814592 7468 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 44.01
Current children cumulated vsize (KiB) 4379852

[startup+25.5011 s]
/proc/loadavg: 5.04 4.20 3.43 10/438 17054
/proc/meminfo: memFree=25122136/32951124 swapFree=53501244/67111528
[pid=17019] ppid=17017 vsize=321108 CPUtime=1.76 cores=0,2,4,6

################
# More data... #
################

[pid=17041] ppid=17039 vsize=419428 CPUtime=181.18 cores=0,2,4,6
/proc/17041/stat : 17041 (adaptg2wsat2009) R 17039 17041 17041 0 -1 4202496 1440 0 0 0 18118 0 0 0 25 0 1 0 657869638 429494272 1175 33554432000 134512640 135107411 4292094640 18446744073709551615 134531674 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/17041/statm: 104857 1175 59 146 0 103733 0
[pid=17039/tid=17071] ppid=17019 vsize=376400 CPUtime=0 cores=0,2,4,6
/proc/17039/task/17071/stat : 17071 (python) S 17019 17019 16982 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 657879672 385433600 5998 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 514 18446604450399283392 0 0 -1 6 0 0 0
[pid=17066] ppid=17019 vsize=321120 CPUtime=0.32 cores=0,2,4,6
/proc/17066/stat : 17066 (python) S 17019 17019 16982 0 -1 4202560 1084 0 0 0 18 14 0 0 19 0 2 0 657879669 328826880 5994 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/17066/statm: 80280 5994 399 364 0 16186 0
[pid=17070] ppid=17066 vsize=419428 CPUtime=99.29 cores=0,2,4,6
/proc/17070/stat : 17070 (adaptg2wsat2009) T 17066 17070 17070 0 -1 4202496 1435 0 0 0 9929 0 0 0 25 0 1 0 657879671 429494272 1175 33554432000 134512640 135107411 4290337520 18446744073709551615 134532242 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/17070/statm: 104857 1175 59 146 0 103733 0
[pid=17066/tid=17103] ppid=17019 vsize=321120 CPUtime=0 cores=0,2,4,6
/proc/17066/task/17103/stat : 17103 (python) S 17019 17019 16982 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 657889604 328826880 5994 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 514 18446604450139292608 0 0 -1 6 0 0 0
[pid=17072] ppid=17019 vsize=331512 CPUtime=0.33 cores=0,2,4,6
/proc/17072/stat : 17072 (python) S 17019 17019 16982 0 -1 4202560 1095 0 0 0 19 14 0 0 18 0 2 0 657879672 339468288 5999 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/17072/statm: 82878 5999 400 364 0 18781 0
[pid=17074] ppid=17072 vsize=1018820 CPUtime=100.01 cores=0,2,4,6
/proc/17074/stat : 17074 (gnovelty+2) T 17072 17074 17074 0 -1 4202496 857 0 0 0 10001 0 0 0 25 0 1 0 657879672 1043271680 595 33554432000 134512640 135663704 4289785328 18446744073709551615 134525286 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/17074/statm: 254705 595 113 282 0 254421 0
[pid=17072/tid=17108] ppid=17019 vsize=331512 CPUtime=0 cores=0,2,4,6
/proc/17072/task/17108/stat : 17108 (python) S 17019 17019 16982 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 657889708 339468288 5999 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 514 18446604444824865600 0 0 -1 4 0 0 0
[pid=17075] ppid=17019 vsize=341768 CPUtime=0.34 cores=0,2,4,6
/proc/17075/stat : 17075 (python) S 17019 17019 16982 0 -1 4202560 1099 0 0 0 17 17 0 0 19 0 2 0 657879673 349970432 6003 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/17075/statm: 85442 6003 400 364 0 21342 0
[pid=17077] ppid=17075 vsize=249188 CPUtime=101.8 cores=0,2,4,6
/proc/17077/stat : 17077 (FH) T 17075 17077 17077 0 -1 4202496 1316 0 0 0 10178 2 0 0 25 0 1 0 657879674 255168512 1058 33554432000 134512640 134936988 4290592288 18446744073709551615 134552455 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/17077/statm: 62297 1058 41 104 0 61801 0
[pid=17075/tid=17116] ppid=17019 vsize=341768 CPUtime=0 cores=0,2,4,6
/proc/17075/task/17116/stat : 17116 (python) S 17019 17019 16982 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 657890109 349970432 6003 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 514 18446604443232208000 0 0 -1 6 0 0 0
[pid=17078] ppid=17019 vsize=352220 CPUtime=0.32 cores=0,2,4,6
/proc/17078/stat : 17078 (python) S 17019 17019 16982 0 -1 4202560 1119 0 0 0 17 15 0 0 19 0 2 0 657879675 360673280 6013 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/17078/statm: 88055 6013 406 364 0 23952 0
[pid=17080] ppid=17078 vsize=419428 CPUtime=100.02 cores=0,2,4,6
/proc/17080/stat : 17080 (adaptg2wsat2009) T 17078 17080 17080 0 -1 4202496 1444 0 0 0 10000 2 0 0 25 0 1 0 657879675 429494272 1175 33554432000 134512640 135107411 4287801872 18446744073709551615 134532128 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/17080/statm: 104857 1175 59 146 0 103733 0
[pid=17078/tid=17107] ppid=17019 vsize=352220 CPUtime=0 cores=0,2,4,6
/proc/17078/task/17107/stat : 17107 (python) S 17019 17019 16982 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 657889708 360673280 6013 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 514 18446604444824864704 0 0 -1 4 0 0 0
[pid=17104] ppid=17019 vsize=362144 CPUtime=0.33 cores=0,2,4,6
/proc/17104/stat : 17104 (python) S 17019 17019 16982 0 -1 4202560 1134 0 0 0 19 14 0 0 19 0 2 0 657889605 370835456 6016 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/17104/statm: 90536 6016 405 364 0 26430 0
[pid=17106] ppid=17104 vsize=419428 CPUtime=100.27 cores=0,2,4,6
/proc/17106/stat : 17106 (adaptg2wsat2009) T 17104 17106 17106 0 -1 4202496 1443 0 0 0 10027 0 0 0 25 0 1 0 657889606 429494272 1176 33554432000 134512640 135107411 4294375408 18446744073709551615 134531856 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/17106/statm: 104857 1176 59 146 0 103733 0
[pid=17104/tid=17130] ppid=17019 vsize=362144 CPUtime=0 cores=0,2,4,6
/proc/17104/task/17130/stat : 17130 (python) S 17019 17019 16982 0 -1 4202560 52 0 0 0 0 0 0 0 18 0 2 0 657899638 370835456 6016 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 514 18446604449178028928 0 0 -1 0 0 0 0
[pid=17109] ppid=17019 vsize=373124 CPUtime=0.33 cores=0,2,4,6
/proc/17109/stat : 17109 (python) S 17019 17019 16982 0 -1 4202560 1117 0 0 0 19 14 0 0 19 0 2 0 657889709 382078976 6015 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/17109/statm: 93281 6015 400 364 0 29172 0
[pid=17111] ppid=17109 vsize=249196 CPUtime=99.88 cores=0,2,4,6
/proc/17111/stat : 17111 (FH) T 17109 17111 17111 0 -1 4202496 1323 0 0 0 9988 0 0 0 25 0 1 0 657889710 255176704 1058 33554432000 134512640 134936988 4294596624 18446744073709551615 134552493 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/17111/statm: 62299 1058 41 104 0 61803 0
[pid=17109/tid=17134] ppid=17019 vsize=373124 CPUtime=0 cores=0,2,4,6
/proc/17109/task/17134/stat : 17134 (python) S 17019 17019 16982 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 657899743 382078976 6015 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 514 18446604441594642560 0 0 -1 0 0 0 0
[pid=17112] ppid=17019 vsize=383084 CPUtime=1.34 cores=0,2,4,6
/proc/17112/stat : 17112 (python) S 17019 17019 16982 0 -1 4202560 1183 0 0 0 66 68 0 0 18 0 2 0 657889711 392278016 6079 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/17112/statm: 95771 6079 400 364 0 31659 0
[pid=17114] ppid=17112 vsize=268420 CPUtime=389 cores=0,2,4,6
/proc/17114/stat : 17114 (hybridGM3) T 17112 17114 17114 0 -1 4202496 5274 0 0 0 38898 2 0 0 25 0 1 0 657889711 274862080 5008 33554432000 134512640 135204385 4290407344 18446744073709551615 134537949 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/17114/statm: 67105 5008 63 169 0 66935 0
[pid=17112/tid=17158] ppid=17019 vsize=383084 CPUtime=0 cores=0,2,4,6
/proc/17112/task/17158/stat : 17158 (python) S 17019 17019 16982 0 -1 4202560 50 0 0 0 0 0 0 0 18 0 2 0 657909784 392278016 6079 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 514 18446604450399281600 0 0 -1 2 0 0 0
[pid=17117] ppid=17019 vsize=393332 CPUtime=0.35 cores=0,2,4,6
/proc/17117/stat : 17117 (python) S 17019 17019 16982 0 -1 4202560 1131 0 0 0 19 16 0 0 19 0 2 0 657890110 402771968 6083 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/17117/statm: 98333 6083 400 364 0 34218 0
[pid=17119] ppid=17117 vsize=1018820 CPUtime=102.89 cores=0,2,4,6
/proc/17119/stat : 17119 (gnovelty+2) T 17117 17119 17119 0 -1 4202496 860 0 0 0 10289 0 0 0 25 0 1 0 657890113 1043271680 595 33554432000 134512640 135663704 4293833184 18446744073709551615 134525361 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/17119/statm: 254705 595 113 282 0 254421 0
[pid=17117/tid=17146] ppid=17019 vsize=393332 CPUtime=0 cores=0,2,4,6
/proc/17117/task/17146/stat : 17146 (python) S 17019 17019 16982 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 657900652 402771968 6083 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 514 18446604450399282496 0 0 -1 4 0 0 0
[pid=17131] ppid=17019 vsize=403408 CPUtime=1.34 cores=0,2,4,6
/proc/17131/stat : 17131 (python) S 17019 17019 16982 0 -1 4202560 1180 0 0 0 71 63 0 0 18 0 2 0 657899640 413089792 6086 33554432000 4194304 5683644 140736280637904 18446744073709551615 216634535494 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/17131/statm: 100852 6086 399 364 0 36734 0
[pid=17133] ppid=17131 vsize=268428 CPUtime=388.29 cores=0,2,4,6
/proc/17133/stat : 17133 (hybridGM3) R 17131 17133 17133 0 -1 4202496 5277 0 0 0 38826 3 0 0 25 0 1 0 657899640 274870272 5010 33554432000 134512640 135204385 4289968704 18446744073709551615 134538432 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/17133/statm: 67107 5010 63 169 0 66937 0
[pid=17131/tid=17168] ppid=17019 vsize=403408 CPUtime=0 cores=0,2,4,6
/proc/17131/task/17168/stat : 17168 (python) S 17019 17019 16982 0 -1 4202560 48 0 0 0 0 0 0 0 18 0 2 0 657919212 413089792 6086 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 514 18446604444655086336 0 0 -1 0 0 0 0
[pid=17135] ppid=17019 vsize=414296 CPUtime=0.35 cores=0,2,4,6
/proc/17135/stat : 17135 (python) S 17019 17019 16982 0 -1 4202560 1140 0 0 0 18 17 0 0 19 0 2 0 657899744 424239104 6091 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/17135/statm: 103574 6091 400 364 0 39453 0
[pid=17137] ppid=17135 vsize=419428 CPUtime=98.83 cores=0,2,4,6
/proc/17137/stat : 17137 (adaptg2wsat2009) T 17135 17137 17137 0 -1 4202496 1447 0 0 0 9883 0 0 0 25 0 1 0 657899744 429494272 1176 33554432000 134512640 135107411 4287902608 18446744073709551615 134532224 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/17137/statm: 104857 1176 59 146 0 103733 0
[pid=17135/tid=17154] ppid=17019 vsize=414296 CPUtime=0 cores=0,2,4,6
/proc/17135/task/17154/stat : 17154 (python) S 17019 17019 16982 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 657909779 424239104 6091 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 514 18446604444655083648 0 0 -1 4 0 0 0
[pid=17147] ppid=17019 vsize=424068 CPUtime=1.03 cores=0,2,4,6
/proc/17147/stat : 17147 (python) S 17019 17019 16982 0 -1 4202560 1237 0 0 0 52 51 0 0 18 0 2 0 657900654 434245632 6137 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/17147/statm: 106017 6137 406 364 0 41893 0
[pid=17149] ppid=17147 vsize=268424 CPUtime=291.86 cores=0,2,4,6
/proc/17149/stat : 17149 (hybridGM3) T 17147 17149 17149 0 -1 4202496 5285 0 0 0 29184 2 0 0 25 0 1 0 657900654 274866176 5010 33554432000 134512640 135204385 4291033120 18446744073709551615 134538432 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/17149/statm: 67106 5010 63 169 0 66936 0
[pid=17147/tid=17228] ppid=17019 vsize=424068 CPUtime=0 cores=0,2,4,6
/proc/17147/task/17228/stat : 17228 (python) S 17019 17019 16982 0 -1 4202560 45 0 0 0 0 0 0 0 18 0 2 0 657920930 434245632 6137 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 514 18446604449263579456 0 0 -1 4 0 0 0
[pid=17155] ppid=17019 vsize=434896 CPUtime=0.35 cores=0,2,4,6
/proc/17155/stat : 17155 (python) S 17019 17019 16982 0 -1 4202560 1188 0 0 0 19 16 0 0 19 0 2 0 657909780 445333504 6144 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/17155/statm: 108724 6144 408 364 0 44597 0
[pid=17157] ppid=17155 vsize=249196 CPUtime=100.3 cores=0,2,4,6
/proc/17157/stat : 17157 (FH) T 17155 17157 17157 0 -1 4202496 1335 0 0 0 10030 0 0 0 25 0 1 0 657909780 255176704 1057 33554432000 134512640 134936988 4288363696 18446744073709551615 134554229 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/17157/statm: 62299 1057 41 104 0 61803 0
[pid=17155/tid=17171] ppid=17019 vsize=434896 CPUtime=0 cores=0,2,4,6
/proc/17155/task/17171/stat : 17171 (python) S 17019 17019 16982 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 657919816 445333504 6144 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 514 18446604447872192768 0 0 -1 4 0 0 0
[pid=17172] ppid=17019 vsize=444808 CPUtime=0.64 cores=0,2,4,6
/proc/17172/stat : 17172 (python) S 17019 17019 16982 0 -1 4202560 1060 0 0 0 37 27 0 0 18 0 1 0 657919818 455483392 6140 33554432000 4194304 5683644 140736280637904 18446744073709551615 216634535494 0 0 16781312 514 11606743809966178562 0 0 17 0 0 0 0
/proc/17172/statm: 111202 6140 379 364 0 47072 0
[pid=17174] ppid=17172 vsize=268428 CPUtime=181.79 cores=0,2,4,6
/proc/17174/stat : 17174 (hybridGM3) R 17172 17174 17174 0 -1 4202496 5289 0 0 0 18178 1 0 0 25 0 1 0 657919818 274870272 5010 33554432000 134512640 135204385 4291282944 18446744073709551615 134537704 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/17174/statm: 67107 5010 63 169 0 66937 0
[pid=17250] ppid=17019 vsize=455500 CPUtime=0 cores=0,2,4,6
/proc/17250/stat : 17250 (python) S 17019 17019 16982 0 -1 4202560 853 0 0 0 0 0 0 0 19 0 1 0 657938586 466432000 6115 33554432000 4194304 5683644 140736280637904 18446744073709551615 216634535494 0 0 16781312 514 4602451774316478463 0 0 17 4 0 0 0
/proc/17250/statm: 113875 6115 350 364 0 49742 0
[pid=17252] ppid=17250 vsize=1018820 CPUtime=0.25 cores=0,2,4,6
/proc/17252/stat : 17252 (gnovelty+2) R 17250 17252 17252 0 -1 4202496 869 0 0 0 25 0 0 0 21 0 1 0 657938586 1043271680 595 33554432000 134512640 135663704 4292917888 18446744073709551615 134524498 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/17252/statm: 254705 595 113 282 0 254421 0
[pid=17019/tid=17031] ppid=17017 vsize=465080 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17031/stat : 17031 (python) S 17017 17019 16982 0 -1 4202560 25 933 0 0 0 0 0 0 19 0 19 0 657869635 476241920 7680 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 4 0 0 0
[pid=17019/tid=17034] ppid=17017 vsize=465080 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17034/stat : 17034 (python) S 17017 17019 16982 0 -1 4202560 40 933 0 0 0 0 0 0 21 0 19 0 657869636 476241920 7680 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 4 0 0 0
[pid=17019/tid=17036] ppid=17017 vsize=465080 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17036/stat : 17036 (python) S 17017 17019 16982 0 -1 4202560 19 933 0 0 0 0 0 0 18 0 19 0 657869637 476241920 7680 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 4 0 0 0
[pid=17019/tid=17040] ppid=17017 vsize=465080 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17040/stat : 17040 (python) S 17017 17019 16982 0 -1 4202560 30 933 0 0 0 0 0 0 18 0 19 0 657869638 476241920 7680 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 4 0 0 0
[pid=17019/tid=17067] ppid=17017 vsize=465080 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17067/stat : 17067 (python) S 17017 17019 16982 0 -1 4202560 21 933 0 0 0 0 0 0 20 0 19 0 657879670 476241920 7680 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 6 0 0 0
[pid=17019/tid=17073] ppid=17017 vsize=465080 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17073/stat : 17073 (python) S 17017 17019 16982 0 -1 4202560 23 933 0 0 0 0 0 0 21 0 19 0 657879672 476241920 7680 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 4 0 0 0
[pid=17019/tid=17076] ppid=17017 vsize=465080 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17076/stat : 17076 (python) S 17017 17019 16982 0 -1 4202560 25 933 0 0 0 0 0 0 23 0 19 0 657879673 476241920 7680 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 6 0 0 0
[pid=17019/tid=17079] ppid=17017 vsize=465080 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17079/stat : 17079 (python) S 17017 17019 16982 0 -1 4202560 27 933 0 0 0 0 0 0 25 0 19 0 657879675 476241920 7680 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 0 0 0 0
[pid=17019/tid=17105] ppid=17017 vsize=465080 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17105/stat : 17105 (python) S 17017 17019 16982 0 -1 4202560 23 933 0 0 0 0 0 0 20 0 19 0 657889606 476241920 7680 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 0 0 0 0
[pid=17019/tid=17110] ppid=17017 vsize=465080 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17110/stat : 17110 (python) S 17017 17019 16982 0 -1 4202560 31 933 0 0 0 0 0 0 20 0 19 0 657889709 476241920 7680 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 4 0 0 0
[pid=17019/tid=17113] ppid=17017 vsize=465080 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17113/stat : 17113 (python) S 17017 17019 16982 0 -1 4202560 30 933 0 0 0 0 0 0 15 0 19 0 657889711 476241920 7680 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 4 0 0 0
[pid=17019/tid=17118] ppid=17017 vsize=465080 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17118/stat : 17118 (python) S 17017 17019 16982 0 -1 4202560 26 933 0 0 0 0 0 0 20 0 19 0 657890110 476241920 7680 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 4 0 0 0
[pid=17019/tid=17132] ppid=17017 vsize=465080 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17132/stat : 17132 (python) S 17017 17019 16982 0 -1 4202560 37 933 0 0 0 0 0 0 18 0 19 0 657899640 476241920 7680 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 0 0 0 0
[pid=17019/tid=17136] ppid=17017 vsize=465080 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17136/stat : 17136 (python) S 17017 17019 16982 0 -1 4202560 29 933 0 0 0 0 0 0 20 0 19 0 657899744 476241920 7680 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 4 0 0 0
[pid=17019/tid=17148] ppid=17017 vsize=465080 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17148/stat : 17148 (python) S 17017 17019 16982 0 -1 4202560 29 933 0 0 0 0 0 0 18 0 19 0 657900654 476241920 7680 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 4 0 0 0
[pid=17019/tid=17156] ppid=17017 vsize=465080 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17156/stat : 17156 (python) S 17017 17019 16982 0 -1 4202560 24 933 0 0 0 0 0 0 20 0 19 0 657909780 476241920 7680 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 4 0 0 0
[pid=17019/tid=17173] ppid=17017 vsize=465080 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17173/stat : 17173 (python) S 17017 17019 16982 0 -1 4202560 27 933 0 0 0 0 0 0 20 0 19 0 657919818 476241920 7680 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 0 0 0 0
[pid=17019/tid=17251] ppid=17017 vsize=465080 CPUtime=0 cores=0,2,4,6
/proc/17019/task/17251/stat : 17251 (python) S 17017 17019 16982 0 -1 4202560 26 933 0 0 0 0 0 0 20 0 19 0 657938586 476241920 7680 33554432000 4194304 5683644 140736280637904 18446744073709551615 216646338993 0 0 16781312 2 18446604448253643456 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2742.62
Current children cumulated vsize (KiB) 16590336

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 17019 and gives
#  childrusage.ru_utime.tv_sec=575
#  childrusage.ru_utime.tv_usec=311539
#  childrusage.ru_stime.tv_sec=3
#  childrusage.ru_stime.tv_usec=755429
# CPU time returned by wait4() is 579.067
# while last known CPU time is 2742.62
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 691.939
CPU time (s): 2742.62
CPU user time (s): 2737.93
CPU system time (s): 4.69
CPU usage (%): 396.367
Max. virtual memory (cumulated for all children) (KiB): 16590336

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 575.312
system time used= 3.75543
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 52376
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 18525
involuntary context switches= 2926

runsolver used 5.51916 second user time and 12.0672 second system time

The end

Launcher Data 

Begin job on node103 at 2011-05-15 14:31:08
IDJOB=3378106
IDBENCH=82515
IDSOLVER=1859
FILE ID=node103/3378106-1305462668
RUNJOBID= node103-1305462668-17000
PBS_JOBID= 13324971
Free space on /tmp= 72988 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/large/unif-k7-r85-v250-c21250-S1682211323-009-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378106-1305462668/watcher-3378106-1305462668 -o /tmp/evaluation-result-3378106-1305462668/solver-3378106-1305462668 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378106-1305462668.cnf -seed 1912783864 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 917ee0efab4f0e89027865c1129d272b
RANDOM SEED=1912783864

node103.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      25237596 kB
Buffers:        326052 kB
Cached:        4173976 kB
SwapCached:     340184 kB
Active:        2207748 kB
Inactive:      5211628 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      25237596 kB
SwapTotal:    67111528 kB
SwapFree:     53501244 kB
Dirty:           65428 kB
Writeback:           0 kB
AnonPages:     2590912 kB
Mapped:          17248 kB
Slab:           163396 kB
PageTables:      61396 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 115419580 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72916 MiB
End job on node103 at 2011-05-15 14:42:42