Trace number 3378230

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (MO) 1999.95 504.937

General information on the benchmark

Namerandom/large/
unif-k5-r20-v1500-c30000-S377999798-061-UNKNOWN.cnf
MD5SUMa4ba28fb1550ba8a4eca95d91f4669c6
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark872.659
Satisfiable
(Un)Satisfiability was proved
Number of variables1500
Number of clauses30000
Sum of the clauses size150000
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 530000
Number of clauses of size over 50

Solver Data 

0.38/0.66	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.38/0.66	c INFO: solving HOME/instance-3378230-1305463938.cnf
0.38/0.66	c INFO: computing task features
0.55/0.82	c INFO: computed variable-clause graph statistics
0.55/0.82	c INFO: computed clause balance statistics
1.35/1.62	c INFO: computed variable balance statistics
1.75/2.07	c INFO: computed small-clause counts
1.75/2.07	c INFO: computed Horn-clause counts
1.75/2.10	c INFO: computed variable graph statistics
1.75/2.10	c INFO: collected features for HOME/instance-3378230-1305463938.cnf in 1.44 s
1.86/2.11	c INFO: running gnovelty+2@0 for 97 with 1198 remaining (b = 0.38)
1.86/2.12	c INFO: running gnovelty+2@0 for 97 with 1198 remaining (b = 0.14)
1.86/2.13	c INFO: running adaptg2wsat2009++@0 for 97 with 1198 remaining (b = 0.09)
1.86/2.14	c INFO: running adaptg2wsat2009++@0 for 97 with 1198 remaining (b = 0.07)
401.46/102.49	c INFO: running adaptg2wsat2009++@0 for 97 with 1098 remaining (b = 0.06)
401.85/102.51	c INFO: running gnovelty+2@0 for 97 with 1098 remaining (b = 0.05)
401.85/102.52	c INFO: running FH@0 for 97 with 1098 remaining (b = 0.04)
401.85/102.54	c INFO: running adaptg2wsat2009++@0 for 97 with 1098 remaining (b = 0.03)
800.58/202.81	c INFO: running adaptg2wsat2009++@0 for 97 with 997 remaining (b = 0.03)
800.88/202.93	c INFO: running FH@0 for 97 with 997 remaining (b = 0.02)
800.88/202.94	c INFO: running hybridGM3@0 for 194 with 997 remaining (b = 0.04)
820.84/207.92	c INFO: running gnovelty+2@0 for 97 with 992 remaining (b = 0.02)
1199.25/303.16	c INFO: running hybridGM3@0 for 194 with 897 remaining (b = 0.03)
1199.59/303.27	c INFO: running adaptg2wsat2009++@0 for 97 with 897 remaining (b = 0.02)
1219.84/308.38	c INFO: running hybridGM3@0 for 194 with 892 remaining (b = 0.03)
1598.63/403.63	c INFO: running FH@0 for 97 with 797 remaining (b = 0.01)
1601.61/404.70	c INFO: running hybridGM3@102 for 97 with 795 remaining (b = 0.01)
1980.85/500.03	c INFO: running hybridGM3@0 for 194 with 700 remaining (b = 0.02)
1996.29/503.99	c INFO: running TNM@0 for 696 with 696 remaining (b = 0.08)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378230-1305463938/watcher-3378230-1305463938 -o /tmp/evaluation-result-3378230-1305463938/solver-3378230-1305463938 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378230-1305463938.cnf -seed 2046337918 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.19 7.42 5.75 5/463 9733
/proc/meminfo: memFree=29957652/32950928 swapFree=41934208/67111528
[pid=9733] ppid=9731 vsize=15780 CPUtime=0 cores=1,3,5,7
/proc/9733/stat : 9733 (python) R 9731 9733 9506 0 -1 4202496 899 0 0 0 0 0 0 0 25 0 1 0 657992514 16158720 780 33554432000 4194304 5683644 140737119991792 18446744073709551615 4540219 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/9733/statm: 3945 781 343 364 0 432 0

[startup+0.092507 s]
/proc/loadavg: 8.19 7.42 5.75 5/463 9733
/proc/meminfo: memFree=29957652/32950928 swapFree=41934208/67111528
[pid=9733] ppid=9731 vsize=84816 CPUtime=0.06 cores=1,3,5,7
/proc/9733/stat : 9733 (python) R 9731 9733 9506 0 -1 4202496 2426 0 0 0 4 2 0 0 25 0 1 0 657992514 86851584 2153 33554432000 4194304 5683644 140737119991792 18446744073709551615 251814240000 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/9733/statm: 21204 2170 707 364 0 1421 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 84816

[startup+0.100504 s]
/proc/loadavg: 8.19 7.42 5.75 5/463 9733
/proc/meminfo: memFree=29957652/32950928 swapFree=41934208/67111528
[pid=9733] ppid=9731 vsize=91832 CPUtime=0.06 cores=1,3,5,7
/proc/9733/stat : 9733 (python) R 9731 9733 9506 0 -1 4202496 2678 0 0 0 4 2 0 0 25 0 1 0 657992514 94035968 2394 33554432000 4194304 5683644 140737119991792 18446744073709551615 251814238208 0 2147483391 16781312 2 0 0 0 17 5 0 0 0
/proc/9733/statm: 22958 2394 755 364 0 1623 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 91832

[startup+0.300474 s]
/proc/loadavg: 8.19 7.42 5.75 5/463 9733
/proc/meminfo: memFree=29957652/32950928 swapFree=41934208/67111528
[pid=9733] ppid=9731 vsize=135752 CPUtime=0.19 cores=1,3,5,7
/proc/9733/stat : 9733 (python) S 9731 9733 9506 0 -1 4202496 5962 934 1 0 14 5 0 0 25 0 1 0 657992514 139010048 4151 33554432000 4194304 5683644 140737119991792 18446744073709551615 251814236165 0 2147483391 16781312 2 18446744071699215550 0 0 17 5 0 0 0
/proc/9733/statm: 33938 4151 1032 364 0 3124 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 135752

[startup+0.700403 s]
/proc/loadavg: 8.19 7.42 5.75 5/463 9733
/proc/meminfo: memFree=29957652/32950928 swapFree=41934208/67111528
[pid=9733] ppid=9731 vsize=280960 CPUtime=0.45 cores=1,3,5,7
/proc/9733/stat : 9733 (python) R 9731 9733 9506 0 -1 4202496 9900 934 1 0 33 12 0 0 17 0 1 0 657992514 287703040 7584 33554432000 4194304 5683644 140737119991792 18446744073709551615 4273416 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/9733/statm: 70240 7584 1761 364 0 6164 0
Current children cumulated CPU time (s) 0.45
Current children cumulated vsize (KiB) 280960

[startup+1.50027 s]
/proc/loadavg: 8.19 7.42 5.75 6/464 9738
/proc/meminfo: memFree=29924652/32950928 swapFree=41934208/67111528
[pid=9733] ppid=9731 vsize=287588 CPUtime=1.25 cores=1,3,5,7
/proc/9733/stat : 9733 (python) R 9731 9733 9506 0 -1 4202496 12679 934 1 0 113 12 0 0 18 0 1 0 657992514 294490112 9281 33554432000 4194304 5683644 140737119991792 18446744073709551615 47574313239645 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/9733/statm: 71897 9281 1826 364 0 7821 0
Current children cumulated CPU time (s) 1.25
Current children cumulated vsize (KiB) 287588

[startup+3.101 s]
/proc/loadavg: 8.19 7.42 5.75 6/464 9738
/proc/meminfo: memFree=29922792/32950928 swapFree=41934208/67111528
[pid=9733] ppid=9731 vsize=324044 CPUtime=1.89 cores=1,3,5,7
/proc/9733/stat : 9733 (python) S 9731 9733 9506 0 -1 4202496 15469 934 1 0 176 13 0 0 25 0 5 0 657992514 331821056 8201 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/9733/statm: 81011 8201 1875 364 0 16920 0
Current children cumulated CPU time (s) 1.89
Current children cumulated vsize (KiB) 324044
heavy processes:

[startup+6.30947 s]
/proc/loadavg: 8.25 7.45 5.77 9/476 9750
/proc/meminfo: memFree=29894364/32950928 swapFree=41934208/67111528
[pid=9733] ppid=9731 vsize=324044 CPUtime=1.89 cores=1,3,5,7
/proc/9733/stat : 9733 (python) S 9731 9733 9506 0 -1 4202496 15469 934 1 0 176 13 0 0 25 0 5 0 657992514 331821056 8201 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/9733/statm: 81011 8201 1875 364 0 16920 0
[pid=9739] ppid=9733 vsize=283516 CPUtime=0.01 cores=1,3,5,7
/proc/9739/stat : 9739 (python) S 9733 9733 9506 0 -1 4202560 889 0 0 0 1 0 0 0 18 0 1 0 657992725 290320384 6606 33554432000 4194304 5683644 140737119991792 18446744073709551615 251814260239 0 0 16781312 514 11239727107787424008 0 0 17 1 0 0 0
/proc/9739/statm: 70879 6606 378 364 0 6797 0
[pid=9741] ppid=9739 vsize=1018996 CPUtime=4.17 cores=1,3,5,7
/proc/9741/stat : 9741 (gnovelty+2) R 9739 9741 9741 0 -1 4202496 962 0 0 0 417 0 0 0 25 0 1 0 657992726 1043451904 697 33554432000 134512640 135663704 4291866560 18446744073709551615 134524918 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/9741/statm: 254749 697 113 282 0 254465 0
[pid=9742] ppid=9733 vsize=293704 CPUtime=0.01 cores=1,3,5,7
/proc/9742/stat : 9742 (python) S 9733 9733 9506 0 -1 4202560 905 0 0 0 1 0 0 0 18 0 1 0 657992726 300752896 6687 33554432000 4194304 5683644 140737119991792 18446744073709551615 251814260294 0 0 16781312 514 13719786085154783495 0 0 17 5 0 0 0
/proc/9742/statm: 73426 6687 372 364 0 9341 0
[pid=9744] ppid=9742 vsize=1018996 CPUtime=4.17 cores=1,3,5,7
/proc/9744/stat : 9744 (gnovelty+2) R 9742 9744 9744 0 -1 4202496 959 0 0 0 417 0 0 0 25 0 1 0 657992727 1043451904 697 33554432000 134512640 135663704 4293629200 18446744073709551615 134525302 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/9744/statm: 254749 697 113 282 0 254465 0
[pid=9745] ppid=9733 vsize=303960 CPUtime=0.01 cores=1,3,5,7
/proc/9745/stat : 9745 (python) S 9733 9733 9506 0 -1 4202560 905 0 0 0 0 1 0 0 18 0 1 0 657992727 311255040 6691 33554432000 4194304 5683644 140737119991792 18446744073709551615 251814260294 0 0 16781312 514 2080179521901789446 0 0 17 5 0 0 0
/proc/9745/statm: 75990 6691 372 364 0 11902 0
[pid=9747] ppid=9745 vsize=421428 CPUtime=4.1 cores=1,3,5,7
/proc/9747/stat : 9747 (adaptg2wsat2009) R 9745 9747 9747 0 -1 4202496 2047 0 0 0 410 0 0 0 25 0 1 0 657992727 431542272 1783 33554432000 134512640 135107411 4291981296 18446744073709551615 134532408 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/9747/statm: 105357 1783 59 146 0 104233 0
[pid=9748] ppid=9733 vsize=314216 CPUtime=0.01 cores=1,3,5,7
/proc/9748/stat : 9748 (python) S 9733 9733 9506 0 -1 4202560 916 0 0 0 1 0 0 0 18 0 1 0 657992728 321757184 6695 33554432000 4194304 5683644 140737119991792 18446744073709551615 251814260294 0 0 16781312 514 9700608740994023685 0 0 17 5 0 0 0
/proc/9748/statm: 78554 6695 372 364 0 14463 0
[pid=9750] ppid=9748 vsize=421428 CPUtime=4.09 cores=1,3,5,7
/proc/9750/stat : 9750 (adaptg2wsat2009) R 9748 9750 9750 0 -1 4202496 2046 0 0 0 409 0 0 0 25 0 1 0 657992728 431542272 1781 33554432000 134512640 135107411 4292214096 18446744073709551615 134531613 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/9750/statm: 105357 1781 59 146 0 104233 0
[pid=9733/tid=9740] ppid=9731 vsize=324044 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9740/stat : 9740 (python) S 9731 9733 9506 0 -1 4202560 24 934 0 0 0 0 0 0 21 0 5 0 657992725 331821056 8201 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 5 0 0 0
[pid=9733/tid=9743] ppid=9731 vsize=324044 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9743/stat : 9743 (python) S 9731 9733 9506 0 -1 4202560 25 934 0 0 0 0 0 0 23 0 5 0 657992726 331821056 8201 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 5 0 0 0
[pid=9733/tid=9746] ppid=9731 vsize=324044 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9746/stat : 9746 (python) S 9731 9733 9506 0 -1 4202560 25 934 0 0 0 0 0 0 25 0 5 0 657992727 331821056 8201 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 7 0 0 0
[pid=9733/tid=9749] ppid=9731 vsize=324044 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9749/stat : 9749 (python) S 9731 9733 9506 0 -1 4202560 27 934 0 0 0 0 0 0 25 0 5 0 657992728 331821056 8201 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 18.46
Current children cumulated vsize (KiB) 4400288

[startup+12.7014 s]
/proc/loadavg: 8.39 7.49 5.79 9/476 9750
/proc/meminfo: memFree=29892572/32950928 swapFree=41934208/67111528
[pid=9733] ppid=9731 vsize=324044 CPUtime=1.89 cores=1,3,5,7
/proc/9733/stat : 9733 (python) S 9731 9733 9506 0 -1 4202496 15469 934 1 0 176 13 0 0 25 0 5 0 657992514 331821056 8201 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/9733/statm: 81011 8201 1875 364 0 16920 0
[pid=9739] ppid=9733 vsize=283516 CPUtime=0.03 cores=1,3,5,7
/proc/9739/stat : 9739 (python) S 9733 9733 9506 0 -1 4202560 892 0 0 0 2 1 0 0 18 0 1 0 657992725 290320384 6606 33554432000 4194304 5683644 140737119991792 18446744073709551615 251814260239 0 0 16781312 514 13762103259755938056 0 0 17 1 0 0 0
/proc/9739/statm: 70879 6606 378 364 0 6797 0
[pid=9741] ppid=9739 vsize=1018996 CPUtime=10.54 cores=1,3,5,7
/proc/9741/stat : 9741 (gnovelty+2) R 9739 9741 9741 0 -1 4202496 962 0 0 0 1054 0 0 0 25 0 1 0 657992726 1043451904 697 33554432000 134512640 135663704 4291866560 18446744073709551615 134544196 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/9741/statm: 254749 697 113 282 0 254465 0
[pid=9742] ppid=9733 vsize=293704 CPUtime=0.03 cores=1,3,5,7
/proc/9742/stat : 9742 (python) S 9733 9733 9506 0 -1 4202560 908 0 0 0 2 1 0 0 18 0 1 0 657992726 300752896 6687 33554432000 4194304 5683644 140737119991792 18446744073709551615 251814260294 0 0 16781312 514 13762103259755938055 0 0 17 5 0 0 0
/proc/9742/statm: 73426 6687 372 364 0 9341 0
[pid=9744] ppid=9742 vsize=1018996 CPUtime=10.57 cores=1,3,5,7
/proc/9744/stat : 9744 (gnovelty+2) R 9742 9744 9744 0 -1 4202496 959 0 0 0 1057 0 0 0 25 0 1 0 657992727 1043451904 697 33554432000 134512640 135663704 4293629200 18446744073709551615 134524918 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/9744/statm: 254749 697 113 282 0 254465 0
[pid=9745] ppid=9733 vsize=303960 CPUtime=0.02 cores=1,3,5,7
/proc/9745/stat : 9745 (python) S 9733 9733 9506 0 -1 4202560 907 0 0 0 1 1 0 0 18 0 1 0 657992727 311255040 6691 33554432000 4194304 5683644 140737119991792 18446744073709551615 251814260294 0 0 16781312 514 13762103259755938054 0 0 17 5 0 0 0
/proc/9745/statm: 75990 6691 372 364 0 11902 0
[pid=9747] ppid=9745 vsize=421428 CPUtime=10.4 cores=1,3,5,7
/proc/9747/stat : 9747 (adaptg2wsat2009) R 9745 9747 9747 0 -1 4202496 2047 0 0 0 1040 0 0 0 25 0 1 0 657992727 431542272 1783 33554432000 134512640 135107411 4291981296 18446744073709551615 134531600 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/9747/statm: 105357 1783 59 146 0 104233 0
[pid=9748] ppid=9733 vsize=314216 CPUtime=0.03 cores=1,3,5,7
/proc/9748/stat : 9748 (python) S 9733 9733 9506 0 -1 4202560 918 0 0 0 2 1 0 0 18 0 1 0 657992728 321757184 6695 33554432000 4194304 5683644 140737119991792 18446744073709551615 251814260294 0 0 16781312 514 13762103259755938053 0 0 17 5 0 0 0
/proc/9748/statm: 78554 6695 372 364 0 14463 0
[pid=9750] ppid=9748 vsize=421428 CPUtime=10.42 cores=1,3,5,7
/proc/9750/stat : 9750 (adaptg2wsat2009) R 9748 9750 9750 0 -1 4202496 2046 0 0 0 1042 0 0 0 25 0 1 0 657992728 431542272 1781 33554432000 134512640 135107411 4292214096 18446744073709551615 134531707 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/9750/statm: 105357 1781 59 146 0 104233 0
[pid=9733/tid=9740] ppid=9731 vsize=324044 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9740/stat : 9740 (python) S 9731 9733 9506 0 -1 4202560 24 934 0 0 0 0 0 0 21 0 5 0 657992725 331821056 8201 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 5 0 0 0
[pid=9733/tid=9743] ppid=9731 vsize=324044 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9743/stat : 9743 (python) S 9731 9733 9506 0 -1 4202560 25 934 0 0 0 0 0 0 23 0 5 0 657992726 331821056 8201 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 5 0 0 0
[pid=9733/tid=9746] ppid=9731 vsize=324044 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9746/stat : 9746 (python) S 9731 9733 9506 0 -1 4202560 25 934 0 0 0 0 0 0 25 0 5 0 657992727 331821056 8201 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 7 0 0 0
[pid=9733/tid=9749] ppid=9731 vsize=324044 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9749/stat : 9749 (python) S 9731 9733 9506 0 -1 4202560 27 934 0 0 0 0 0 0 25 0 5 0 657992728 331821056 8201 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 43.93
Current children cumulated vsize (KiB) 4400288

[startup+25.5012 s]
/proc/loadavg: 8.30 7.52 5.83 9/476 9750
/proc/meminfo: memFree=29892964/32950928 swapFree=41934208/67111528
[pid=9733] ppid=9731 vsize=324044 CPUtime=1.89 cores=1,3,5,7
/proc/9733/stat : 9733 (python) S 9731 9733 9506 0 -1 4202496 15469 934 1 0 176 13 0 0 25 0 5 0 657992514 331821056 8201 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/9733/statm: 81011 8201 1875 364 0 16920 0
[pid=9739] ppid=9733 vsize=283516 CPUtime=0.07 cores=1,3,5,7
/proc/9739/stat : 9739 (python) S 9733 9733 9506 0 -1 4202560 898 0 0 0 4 3 0 0 18 0 1 0 657992725 290320384 6606 33554432000 4194304 5683644 140737119991792 18446744073709551615 251814260239 0 0 16781312 514 13762103259755938056 0 0 17 1 0 0 0
/proc/9739/statm: 70879 6606 378 364 0 6797 0
[pid=9741] ppid=9739 vsize=1018996 CPUtime=23.3 cores=1,3,5,7
/proc/9741/stat : 9741 (gnovelty+2) R 9739 9741 9741 0 -1 4202496 962 0 0 0 2330 0 0 0 25 0 1 0 657992726 1043451904 697 33554432000 134512640 135663704 4291866560 18446744073709551615 134524957 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/9741/statm: 254749 697 113 282 0 254465 0
[pid=9742] ppid=9733 vsize=293704 CPUtime=0.07 cores=1,3,5,7
/proc/9742/stat : 9742 (python) S 9733 9733 9506 0 -1 4202560 914 0 0 0 4 3 0 0 18 0 1 0 657992726 300752896 6687 33554432000 4194304 5683644 140737119991792 18446744073709551615 251814260294 0 0 16781312 514 13762103259755938055 0 0 17 5 0 0 0
/proc/9742/statm: 73426 6687 372 364 0 9341 0
[pid=9744] ppid=9742 vsize=1018996 CPUtime=23.37 cores=1,3,5,7
/proc/9744/stat : 9744 (gnovelty+2) R 9742 9744 9744 0 -1 4202496 959 0 0 0 2337 0 0 0 25 0 1 0 657992727 1043451904 697 33554432000 134512640 135663704 4293629200 18446744073709551615 134544204 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/9744/statm: 254749 697 113 282 0 254465 0
[pid=9745] ppid=9733 vsize=303960 CPUtime=0.07 cores=1,3,5,7
/proc/9745/stat : 9745 (python) S 9733 9733 9506 0 -1 4202560 912 0 0 0 3 4 0 0 18 0 1 0 657992727 311255040 6691 33554432000 4194304 5683644 140737119991792 18446744073709551615 251814260294 0 0 16781312 514 13762103259755938054 0 0 17 5 0 0 0
/proc/9745/statm: 75990 6691 372 364 0 11902 0
[pid=9747] ppid=9745 vsize=421428 CPUtime=23 cores=1,3,5,7
/proc/9747/stat : 9747 (adaptg2wsat2009) R 9745 9747 9747 0 -1 4202496 2047 0 0 0 2300 0 0 0 25 0 1 0 657992727 431542272 1783 33554432000 134512640 135107411 4291981296 18446744073709551615 134531792 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/9747/statm: 105357 1783 59 146 0 104233 0
[pid=9748] ppid=9733 vsize=314216 CPUtime=0.07 cores=1,3,5,7
/proc/9748/stat : 9748 (python) S 9733 9733 9506 0 -1 4202560 922 0 0 0 4 3 0 0 18 0 1 0 657992728 321757184 6695 33554432000 4194304 5683644 140737119991792 18446744073709551615 251814260294 0 0 16781312 514 13762103259755938053 0 0 17 5 0 0 0
/proc/9748/statm: 78554 6695 372 364 0 14463 0
[pid=9750] ppid=9748 vsize=421428 CPUtime=23.11 cores=1,3,5,7
/proc/9750/stat : 9750 (adaptg2wsat2009) R 9748 9750 9750 0 -1 4202496 2046 0 0 0 2311 0 0 0 25 0 1 0 657992728 431542272 1781 33554432000 134512640 135107411 4292214096 18446744073709551615 134531795 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/9750/statm: 105357 1781 59 146 0 104233 0
[pid=9733/tid=9740] ppid=9731 vsize=324044 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9740/stat : 9740 (python) S 9731 9733 9506 0 -1 4202560 24 934 0 0 0 0 0 0 21 0 5 0 657992725 331821056 8201 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 5 0 0 0
[pid=9733/tid=9743] ppid=9731 vsize=324044 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9743/stat : 9743 (python) S 9731 9733 9506 0 -1 4202560 25 934 0 0 0 0 0 0 23 0 5 0 657992726 331821056 8201 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 5 0 0 0
[pid=9733/tid=9746] ppid=9731 vsize=324044 CPUtime=0 cores=1,3,5,7

################
# More data... #
################

/proc/9745/task/9853/stat : 9853 (python) S 9733 9733 9506 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 658002762 311255040 6720 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 514 18446604457691990080 0 0 -1 7 0 0 0
[pid=9748] ppid=9733 vsize=314216 CPUtime=0.32 cores=1,3,5,7
/proc/9748/stat : 9748 (python) S 9733 9733 9506 0 -1 4202560 1099 0 0 0 17 15 0 0 19 0 2 0 657992728 321757184 6724 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/9748/statm: 78554 6724 400 364 0 14463 0
[pid=9750] ppid=9748 vsize=421428 CPUtime=99.38 cores=1,3,5,7
/proc/9750/stat : 9750 (adaptg2wsat2009) T 9748 9750 9750 0 -1 4202496 2046 0 0 0 9938 0 0 0 25 0 1 0 657992728 431542272 1781 33554432000 134512640 135107411 4292214096 18446744073709551615 134532408 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/9750/statm: 105357 1781 59 146 0 104233 0
[pid=9748/tid=9852] ppid=9733 vsize=314216 CPUtime=0 cores=1,3,5,7
/proc/9748/task/9852/stat : 9852 (python) S 9733 9733 9506 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 658002760 321757184 6724 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 514 18446604468056194112 0 0 -1 5 0 0 0
[pid=9854] ppid=9733 vsize=324472 CPUtime=0.31 cores=1,3,5,7
/proc/9854/stat : 9854 (python) S 9733 9733 9506 0 -1 4202560 1097 0 0 0 16 15 0 0 19 0 2 0 658002763 332259328 6728 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/9854/statm: 81118 6728 400 364 0 17024 0
[pid=9856] ppid=9854 vsize=421428 CPUtime=100.06 cores=1,3,5,7
/proc/9856/stat : 9856 (adaptg2wsat2009) T 9854 9856 9856 0 -1 4202496 2047 0 0 0 10006 0 0 0 25 0 1 0 658002763 431542272 1781 33554432000 134512640 135107411 4289776048 18446744073709551615 134531411 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/9856/statm: 105357 1781 59 146 0 104233 0
[pid=9854/tid=9875] ppid=9733 vsize=324472 CPUtime=0 cores=1,3,5,7
/proc/9854/task/9875/stat : 9875 (python) S 9733 9733 9506 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 658012794 332259328 6728 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 514 18446604471120757888 0 0 -1 5 0 0 0
[pid=9859] ppid=9733 vsize=334728 CPUtime=0.32 cores=1,3,5,7
/proc/9859/stat : 9859 (python) S 9733 9733 9506 0 -1 4202560 1101 0 0 0 16 16 0 0 19 0 2 0 658002765 342761472 6732 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/9859/statm: 83682 6732 400 364 0 19585 0
[pid=9861] ppid=9859 vsize=1018996 CPUtime=100.39 cores=1,3,5,7
/proc/9861/stat : 9861 (gnovelty+2) T 9859 9861 9861 0 -1 4202496 960 0 0 0 10039 0 0 0 25 0 1 0 658002766 1043451904 698 33554432000 134512640 135663704 4287727072 18446744073709551615 134544196 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/9861/statm: 254749 698 113 282 0 254465 0
[pid=9859/tid=9879] ppid=9733 vsize=334728 CPUtime=0 cores=1,3,5,7
/proc/9859/task/9879/stat : 9879 (python) S 9733 9733 9506 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 658012806 342761472 6732 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 514 18446604466077573376 0 0 -1 1 0 0 0
[pid=9862] ppid=9733 vsize=345076 CPUtime=0.33 cores=1,3,5,7
/proc/9862/stat : 9862 (python) S 9733 9733 9506 0 -1 4202560 1102 0 0 0 16 17 0 0 19 0 2 0 658002767 353357824 6736 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/9862/statm: 86269 6736 400 364 0 22169 0
[pid=9864] ppid=9862 vsize=251292 CPUtime=102.67 cores=1,3,5,7
/proc/9864/stat : 9864 (FH) T 9862 9864 9864 0 -1 4202496 1945 0 0 0 10267 0 0 0 25 0 1 0 658002767 257323008 1682 33554432000 134512640 134936988 4288805520 18446744073709551615 134552472 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/9864/statm: 62823 1682 41 104 0 62327 0
[pid=9862/tid=9891] ppid=9733 vsize=345076 CPUtime=0 cores=1,3,5,7
/proc/9862/task/9891/stat : 9891 (python) S 9733 9733 9506 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 658013305 353357824 6736 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 514 18446604462478072512 0 0 -1 7 0 0 0
[pid=9865] ppid=9733 vsize=355336 CPUtime=0.32 cores=1,3,5,7
/proc/9865/stat : 9865 (python) S 9733 9733 9506 0 -1 4202560 1151 0 0 0 17 15 0 0 19 0 2 0 658002768 363864064 6752 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/9865/statm: 88834 6752 406 364 0 24731 0
[pid=9867] ppid=9865 vsize=421428 CPUtime=99.63 cores=1,3,5,7
/proc/9867/stat : 9867 (adaptg2wsat2009) T 9865 9867 9867 0 -1 4202496 2052 0 0 0 9963 0 0 0 25 0 1 0 658002768 431542272 1781 33554432000 134512640 135107411 4288646496 18446744073709551615 134531584 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/9867/statm: 105357 1781 59 146 0 104233 0
[pid=9865/tid=9880] ppid=9733 vsize=355336 CPUtime=0 cores=1,3,5,7
/proc/9865/task/9880/stat : 9880 (python) S 9733 9733 9506 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 658012806 363864064 6752 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 514 18446604455129835712 0 0 -1 3 0 0 0
[pid=9876] ppid=9733 vsize=365804 CPUtime=0.33 cores=1,3,5,7
/proc/9876/stat : 9876 (python) S 9733 9733 9506 0 -1 4202560 1147 0 0 0 17 16 0 0 19 0 2 0 658012796 374583296 6757 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/9876/statm: 91451 6757 406 364 0 27345 0
[pid=9878] ppid=9876 vsize=421428 CPUtime=100.29 cores=1,3,5,7
/proc/9878/stat : 9878 (adaptg2wsat2009) T 9876 9878 9878 0 -1 4202496 2050 0 0 0 10029 0 0 0 25 0 1 0 658012796 431542272 1781 33554432000 134512640 135107411 4290255664 18446744073709551615 134531215 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/9878/statm: 105357 1781 59 146 0 104233 0
[pid=9876/tid=9910] ppid=9733 vsize=365804 CPUtime=0 cores=1,3,5,7
/proc/9876/task/9910/stat : 9910 (python) S 9733 9733 9506 0 -1 4202560 51 0 0 0 0 0 0 0 19 0 2 0 658022828 374583296 6757 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 514 18446604463523517440 0 0 -1 5 0 0 0
[pid=9881] ppid=9733 vsize=375932 CPUtime=0.33 cores=1,3,5,7
/proc/9881/stat : 9881 (python) S 9733 9733 9506 0 -1 4202560 1118 0 0 0 17 16 0 0 19 0 2 0 658012807 384954368 6808 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/9881/statm: 93983 6808 400 364 0 29874 0
[pid=9883] ppid=9881 vsize=251288 CPUtime=97.85 cores=1,3,5,7
/proc/9883/stat : 9883 (FH) T 9881 9883 9883 0 -1 4202496 1949 0 0 0 9785 0 0 0 25 0 1 0 658012807 257318912 1682 33554432000 134512640 134936988 4293108480 18446744073709551615 134554998 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/9883/statm: 62822 1682 41 104 0 62326 0
[pid=9881/tid=9914] ppid=9733 vsize=375932 CPUtime=0 cores=1,3,5,7
/proc/9881/task/9914/stat : 9914 (python) S 9733 9733 9506 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 658022840 384954368 6808 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 514 18446604468056193216 0 0 -1 5 0 0 0
[pid=9884] ppid=9733 vsize=386428 CPUtime=1.02 cores=1,3,5,7
/proc/9884/stat : 9884 (python) S 9733 9733 9506 0 -1 4202560 1175 0 0 0 50 52 0 0 18 0 2 0 658012808 395702272 6812 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/9884/statm: 96607 6812 400 364 0 32495 0
[pid=9886] ppid=9884 vsize=268428 CPUtime=293.72 cores=1,3,5,7
/proc/9886/stat : 9886 (hybridGM3) T 9884 9886 9886 0 -1 4202496 5811 0 0 0 29372 0 0 0 25 0 1 0 658012809 274870272 5544 33554432000 134512640 135204385 4289210512 18446744073709551615 134537739 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/9886/statm: 67107 5544 63 169 0 66937 0
[pid=9884/tid=9940] ppid=9733 vsize=386428 CPUtime=0 cores=1,3,5,7
/proc/9884/task/9940/stat : 9940 (python) S 9733 9733 9506 0 -1 4202560 50 0 0 0 0 0 0 0 18 0 2 0 658032982 395702272 6812 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 514 18446604469813852928 0 0 -1 1 0 0 0
[pid=9892] ppid=9733 vsize=396520 CPUtime=0.33 cores=1,3,5,7
/proc/9892/stat : 9892 (python) S 9733 9733 9506 0 -1 4202560 1142 0 0 0 18 15 0 0 18 0 2 0 658013306 406036480 6844 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/9892/statm: 99130 6844 400 364 0 35015 0
[pid=9894] ppid=9892 vsize=1018996 CPUtime=99.78 cores=1,3,5,7
/proc/9894/stat : 9894 (gnovelty+2) T 9892 9894 9894 0 -1 4202496 966 0 0 0 9978 0 0 0 25 0 1 0 658013306 1043451904 698 33554432000 134512640 135663704 4294055728 18446744073709551615 134544186 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/9894/statm: 254749 698 113 282 0 254465 0
[pid=9892/tid=9918] ppid=9733 vsize=396520 CPUtime=0 cores=1,3,5,7
/proc/9892/task/9918/stat : 9918 (python) S 9733 9733 9506 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 658023349 406036480 6844 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 514 18446604469813851136 0 0 -1 3 0 0 0
[pid=9911] ppid=9733 vsize=407044 CPUtime=0.68 cores=1,3,5,7
/proc/9911/stat : 9911 (python) S 9733 9733 9506 0 -1 4202560 1011 0 0 0 36 32 0 0 18 0 1 0 658022830 416813056 6819 33554432000 4194304 5683644 140737119991792 18446744073709551615 251814260294 0 0 16781312 514 13762103259755938053 0 0 17 3 0 0 0
/proc/9911/statm: 101761 6819 372 364 0 37643 0
[pid=9913] ppid=9911 vsize=268424 CPUtime=195.58 cores=1,3,5,7
/proc/9913/stat : 9913 (hybridGM3) R 9911 9913 9913 0 -1 4202496 5813 0 0 0 19557 1 0 0 25 0 1 0 658022830 274866176 5544 33554432000 134512640 135204385 4287090512 18446744073709551615 134538541 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/9913/statm: 67106 5544 63 169 0 66936 0
[pid=9915] ppid=9733 vsize=417060 CPUtime=0.35 cores=1,3,5,7
/proc/9915/stat : 9915 (python) S 9733 9733 9506 0 -1 4202560 1150 0 0 0 17 18 0 0 19 0 2 0 658022841 427069440 6878 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/9915/statm: 104265 6878 400 364 0 40144 0
[pid=9917] ppid=9915 vsize=421428 CPUtime=100.27 cores=1,3,5,7
/proc/9917/stat : 9917 (adaptg2wsat2009) T 9915 9917 9917 0 -1 4202496 2052 0 0 0 10027 0 0 0 25 0 1 0 658022841 431542272 1781 33554432000 134512640 135107411 4287769904 18446744073709551615 134531905 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/9917/statm: 105357 1781 59 146 0 104233 0
[pid=9915/tid=9936] ppid=9733 vsize=417060 CPUtime=0 cores=1,3,5,7
/proc/9915/task/9936/stat : 9936 (python) S 9733 9733 9506 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 658032876 427069440 6878 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 514 18446604466094707136 0 0 -1 1 0 0 0
[pid=9919] ppid=9733 vsize=427616 CPUtime=0.66 cores=1,3,5,7
/proc/9919/stat : 9919 (python) S 9733 9733 9506 0 -1 4202560 1095 0 0 0 38 28 0 0 18 0 1 0 658023352 437878784 6861 33554432000 4194304 5683644 140737119991792 18446744073709551615 251814260294 0 0 16781312 514 13762103259755938055 0 0 17 3 0 0 0
/proc/9919/statm: 106904 6861 380 364 0 42780 0
[pid=9921] ppid=9919 vsize=268428 CPUtime=196.41 cores=1,3,5,7
/proc/9921/stat : 9921 (hybridGM3) R 9919 9921 9921 0 -1 4202496 5818 0 0 0 19641 0 0 0 25 0 1 0 658023353 274870272 5543 33554432000 134512640 135204385 4288993264 18446744073709551615 134538534 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/9921/statm: 67107 5543 63 169 0 66937 0
[pid=9937] ppid=9733 vsize=437604 CPUtime=0.35 cores=1,3,5,7
/proc/9937/stat : 9937 (python) S 9733 9733 9506 0 -1 4202560 1197 0 0 0 19 16 0 0 19 0 2 0 658032877 448106496 6909 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/9937/statm: 109401 6909 408 364 0 45274 0
[pid=9939] ppid=9937 vsize=251292 CPUtime=100.31 cores=1,3,5,7
/proc/9939/stat : 9939 (FH) T 9937 9939 9939 0 -1 4202496 1959 0 0 0 10031 0 0 0 25 0 1 0 658032877 257323008 1683 33554432000 134512640 134936988 4288866000 18446744073709551615 134552472 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/9939/statm: 62823 1683 41 104 0 62327 0
[pid=9937/tid=9955] ppid=9733 vsize=437604 CPUtime=0 cores=1,3,5,7
/proc/9937/task/9955/stat : 9955 (python) S 9733 9733 9506 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 658042912 448106496 6909 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 514 18446604463190169536 0 0 -1 1 0 0 0
[pid=9952] ppid=9733 vsize=447876 CPUtime=0.01 cores=1,3,5,7
/proc/9952/stat : 9952 (python) S 9733 9733 9506 0 -1 4202560 1037 0 0 0 0 1 0 0 18 0 1 0 658042517 458625024 6899 33554432000 4194304 5683644 140737119991792 18446744073709551615 251814260294 0 0 16781312 514 17976664099345367303 0 0 17 3 0 0 0
/proc/9952/statm: 111969 6899 380 364 0 47839 0
[pid=9954] ppid=9952 vsize=268420 CPUtime=4.76 cores=1,3,5,7
/proc/9954/stat : 9954 (hybridGM3) R 9952 9954 9954 0 -1 4202496 5821 0 0 0 476 0 0 0 25 0 1 0 658042517 274862080 5543 33554432000 134512640 135204385 4294019872 18446744073709551615 134538041 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/9954/statm: 67105 5543 63 169 0 66935 0
[pid=9956] ppid=9733 vsize=458476 CPUtime=0 cores=1,3,5,7
/proc/9956/stat : 9956 (python) S 9733 9733 9506 0 -1 4202560 821 0 0 0 0 0 0 0 20 0 1 0 658042913 469479424 6870 33554432000 4194304 5683644 140737119991792 18446744073709551615 251814260294 0 0 16781312 514 5108450049075970047 0 0 17 3 0 0 0
/proc/9956/statm: 114619 6870 347 364 0 50486 0
[pid=9958] ppid=9956 vsize=459768 CPUtime=0.83 cores=1,3,5,7
/proc/9958/stat : 9958 (TNM) R 9956 9958 9958 0 -1 4202496 1927 0 0 0 83 0 0 0 25 0 1 0 658042914 470802432 1654 33554432000 134512640 135008748 4292273616 18446744073709551615 134524772 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/9958/statm: 114942 1654 46 122 0 113842 0
[pid=9733/tid=9740] ppid=9731 vsize=468084 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9740/stat : 9740 (python) S 9731 9733 9506 0 -1 4202560 24 934 0 0 0 0 0 0 21 0 19 0 657992725 479318016 8430 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 5 0 0 0
[pid=9733/tid=9743] ppid=9731 vsize=468084 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9743/stat : 9743 (python) S 9731 9733 9506 0 -1 4202560 25 934 0 0 0 0 0 0 23 0 19 0 657992726 479318016 8430 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 5 0 0 0
[pid=9733/tid=9746] ppid=9731 vsize=468084 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9746/stat : 9746 (python) S 9731 9733 9506 0 -1 4202560 25 934 0 0 0 0 0 0 25 0 19 0 657992727 479318016 8430 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 7 0 0 0
[pid=9733/tid=9749] ppid=9731 vsize=468084 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9749/stat : 9749 (python) S 9731 9733 9506 0 -1 4202560 27 934 0 0 0 0 0 0 25 0 19 0 657992728 479318016 8430 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 5 0 0 0
[pid=9733/tid=9855] ppid=9731 vsize=468084 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9855/stat : 9855 (python) S 9731 9733 9506 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 19 0 658002763 479318016 8430 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 5 0 0 0
[pid=9733/tid=9860] ppid=9731 vsize=468084 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9860/stat : 9860 (python) S 9731 9733 9506 0 -1 4202560 24 934 0 0 0 0 0 0 21 0 19 0 658002766 479318016 8430 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 1 0 0 0
[pid=9733/tid=9863] ppid=9731 vsize=468084 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9863/stat : 9863 (python) S 9731 9733 9506 0 -1 4202560 27 934 0 0 0 0 0 0 23 0 19 0 658002767 479318016 8430 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 7 0 0 0
[pid=9733/tid=9866] ppid=9731 vsize=468084 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9866/stat : 9866 (python) S 9731 9733 9506 0 -1 4202560 28 934 0 0 0 0 0 0 25 0 19 0 658002768 479318016 8430 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 5 0 0 0
[pid=9733/tid=9877] ppid=9731 vsize=468084 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9877/stat : 9877 (python) S 9731 9733 9506 0 -1 4202560 24 934 0 0 0 0 0 0 20 0 19 0 658012796 479318016 8430 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 5 0 0 0
[pid=9733/tid=9882] ppid=9731 vsize=468084 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9882/stat : 9882 (python) S 9731 9733 9506 0 -1 4202560 22 934 0 0 0 0 0 0 21 0 19 0 658012807 479318016 8430 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 5 0 0 0
[pid=9733/tid=9885] ppid=9731 vsize=468084 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9885/stat : 9885 (python) S 9731 9733 9506 0 -1 4202560 26 934 0 0 0 0 0 0 18 0 19 0 658012808 479318016 8430 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 3 0 0 0
[pid=9733/tid=9893] ppid=9731 vsize=468084 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9893/stat : 9893 (python) S 9731 9733 9506 0 -1 4202560 26 934 0 0 0 0 0 0 20 0 19 0 658013306 479318016 8430 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 3 0 0 0
[pid=9733/tid=9912] ppid=9731 vsize=468084 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9912/stat : 9912 (python) S 9731 9733 9506 0 -1 4202560 28 934 0 0 0 0 0 0 20 0 19 0 658022830 479318016 8430 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 3 0 0 0
[pid=9733/tid=9916] ppid=9731 vsize=468084 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9916/stat : 9916 (python) S 9731 9733 9506 0 -1 4202560 26 934 0 0 0 0 0 0 21 0 19 0 658022841 479318016 8430 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 3 0 0 0
[pid=9733/tid=9920] ppid=9731 vsize=468084 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9920/stat : 9920 (python) S 9731 9733 9506 0 -1 4202560 24 934 0 0 0 0 0 0 20 0 19 0 658023353 479318016 8430 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 3 0 0 0
[pid=9733/tid=9938] ppid=9731 vsize=468084 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9938/stat : 9938 (python) S 9731 9733 9506 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 19 0 658032877 479318016 8430 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 3 0 0 0
[pid=9733/tid=9953] ppid=9731 vsize=468084 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9953/stat : 9953 (python) S 9731 9733 9506 0 -1 4202560 26 934 0 0 0 0 0 0 20 0 19 0 658042517 479318016 8430 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 3 0 0 0
[pid=9733/tid=9957] ppid=9731 vsize=468084 CPUtime=0 cores=1,3,5,7
/proc/9733/task/9957/stat : 9957 (python) S 9731 9733 9506 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 19 0 658042913 479318016 8430 33554432000 4194304 5683644 140737119991792 18446744073709551615 251826063793 0 0 16781312 2 18446604457236279360 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 1999.95
Current children cumulated vsize (KiB) 16045588

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 9733 and gives
#  childrusage.ru_utime.tv_sec=1
#  childrusage.ru_utime.tv_usec=953702
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=159975
# CPU time returned by wait4() is 2.11368
# while last known CPU time is 1999.95
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 504.937
CPU time (s): 1999.95
CPU user time (s): 1996.61
CPU system time (s): 3.34
CPU usage (%): 396.079
Max. virtual memory (cumulated for all children) (KiB): 16045588

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1.9537
system time used= 0.159975
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 29720
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 3000
involuntary context switches= 81

runsolver used 3.79442 second user time and 8.60369 second system time

The end

Launcher Data 

Begin job on node126 at 2011-05-15 14:52:18
IDJOB=3378230
IDBENCH=82639
IDSOLVER=1859
FILE ID=node126/3378230-1305463938
RUNJOBID= node126-1305463427-9526
PBS_JOBID= 13324997
Free space on /tmp= 73324 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/large/unif-k5-r20-v1500-c30000-S377999798-061-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378230-1305463938/watcher-3378230-1305463938 -o /tmp/evaluation-result-3378230-1305463938/solver-3378230-1305463938 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378230-1305463938.cnf -seed 2046337918 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= a4ba28fb1550ba8a4eca95d91f4669c6
RANDOM SEED=2046337918

node126.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.56
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      29957932 kB
Buffers:        176736 kB
Cached:        1839980 kB
SwapCached:      49560 kB
Active:         447420 kB
Inactive:      2276604 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      29957932 kB
SwapTotal:    67111528 kB
SwapFree:     41934208 kB
Dirty:           32552 kB
Writeback:          36 kB
AnonPages:      670684 kB
Mapped:          18332 kB
Slab:           111620 kB
PageTables:      89412 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 105663232 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73280 MiB
End job on node126 at 2011-05-15 15:00:45