Trace number 3378260

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (MO) 2008.2 507.13

General information on the benchmark

Namerandom/large/
unif-k7-r85-v400-c34000-S586055573-067-UNKNOWN.cnf
MD5SUMd673a60901b4a946b283fdbb02f99fdc
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables400
Number of clauses34000
Sum of the clauses size238000
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 534000

Solver Data 

0.31/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.31/0.67	c INFO: solving HOME/instance-3378260-1305464336.cnf
0.31/0.67	c INFO: computing task features
0.59/0.90	c INFO: computed variable-clause graph statistics
0.59/0.91	c INFO: computed clause balance statistics
1.79/2.16	c INFO: computed variable balance statistics
2.29/2.65	c INFO: computed small-clause counts
2.29/2.65	c INFO: computed Horn-clause counts
2.29/2.69	c INFO: computed variable graph statistics
2.29/2.69	c INFO: collected features for HOME/instance-3378260-1305464336.cnf in 2.02 s
2.38/2.70	c INFO: running gnovelty+2@0 for 97 with 1197 remaining (b = 0.38)
2.38/2.71	c INFO: running gnovelty+2@0 for 97 with 1197 remaining (b = 0.13)
2.38/2.72	c INFO: running adaptg2wsat2009++@0 for 97 with 1197 remaining (b = 0.08)
2.38/2.73	c INFO: running adaptg2wsat2009++@0 for 97 with 1197 remaining (b = 0.07)
397.92/101.99	c INFO: running adaptg2wsat2009++@0 for 97 with 1098 remaining (b = 0.05)
401.90/102.98	c INFO: running gnovelty+2@0 for 97 with 1097 remaining (b = 0.04)
402.27/103.07	c INFO: running FH@0 for 97 with 1097 remaining (b = 0.03)
402.53/103.12	c INFO: running adaptg2wsat2009++@0 for 97 with 1097 remaining (b = 0.03)
784.98/199.28	c INFO: running adaptg2wsat2009++@0 for 97 with 1001 remaining (b = 0.02)
797.35/202.33	c INFO: running FH@0 for 97 with 998 remaining (b = 0.02)
800.34/203.39	c INFO: running hybridGM3@0 for 194 with 997 remaining (b = 0.04)
813.23/206.38	c INFO: running gnovelty+2@0 for 97 with 994 remaining (b = 0.02)
1176.78/297.70	c INFO: running hybridGM3@0 for 194 with 902 remaining (b = 0.03)
1192.28/301.62	c INFO: running adaptg2wsat2009++@0 for 97 with 899 remaining (b = 0.01)
1232.42/311.73	c INFO: running hybridGM3@0 for 194 with 888 remaining (b = 0.03)
1579.58/399.05	c INFO: running hybridGM3@100 for 97 with 801 remaining (b = 0.01)
1591.10/401.91	c INFO: running hybridGM3@0 for 194 with 798 remaining (b = 0.02)
1953.79/493.33	c INFO: running hybridGM3@99 for 97 with 707 remaining (b = 0.01)
1965.70/496.36	c INFO: running TNM@0 for 704 with 704 remaining (b = 0.08)
2005.37/506.39	c INFO: running FH@0 for 97 with 694 remaining (b = 0.01)
2008.20/507.10	c INFO: running hybridGM3@0 for 194 with 693 remaining (b = 0.02)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378260-1305464336/watcher-3378260-1305464336 -o /tmp/evaluation-result-3378260-1305464336/solver-3378260-1305464336 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378260-1305464336.cnf -seed 868295949 -budget 4800 -cores 4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.20 8.04 6.76 6/359 1647
/proc/meminfo: memFree=23652044/32951124 swapFree=67057896/67111528
[pid=1647] ppid=1645 vsize=15184 CPUtime=0 cores=0,2,4,6
/proc/1647/stat : 1647 (python) S 1645 1647 1220 0 -1 4202496 724 0 0 0 0 0 0 0 25 0 1 0 638848980 15548416 615 33554432000 4194304 5683644 140734174288272 18446744073709551615 254261614592 0 2147483391 16781312 2 18446744071699207358 0 0 17 2 0 0 0
/proc/1647/statm: 3796 615 321 364 0 283 0

[startup+0.01316 s]
/proc/loadavg: 8.20 8.04 6.76 6/359 1647
/proc/meminfo: memFree=23652044/32951124 swapFree=67057896/67111528
[pid=1647] ppid=1645 vsize=15780 CPUtime=0 cores=0,2,4,6
/proc/1647/stat : 1647 (python) S 1645 1647 1220 0 -1 4202496 903 0 0 0 0 0 0 0 25 0 1 0 638848980 16158720 784 33554432000 4194304 5683644 140734174288272 18446744073709551615 254261614592 0 2147483391 16781312 2 18446744071699207358 0 0 17 2 0 0 0
/proc/1647/statm: 3945 784 346 364 0 432 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15780

[startup+0.100143 s]
/proc/loadavg: 8.20 8.04 6.76 6/359 1647
/proc/meminfo: memFree=23652044/32951124 swapFree=67057896/67111528
[pid=1647] ppid=1645 vsize=89772 CPUtime=0.05 cores=0,2,4,6
/proc/1647/stat : 1647 (python) R 1645 1647 1220 0 -1 4202496 2640 0 0 0 4 1 0 0 25 0 1 0 638848980 91926528 2356 33554432000 4194304 5683644 140734174288272 18446744073709551615 254256646944 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/1647/statm: 22443 2356 718 364 0 1623 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 89772

[startup+0.300138 s]
/proc/loadavg: 8.20 8.04 6.76 6/359 1647
/proc/meminfo: memFree=23652044/32951124 swapFree=67057896/67111528
[pid=1647] ppid=1645 vsize=135752 CPUtime=0.16 cores=0,2,4,6
/proc/1647/stat : 1647 (python) S 1645 1647 1220 0 -1 4202496 5962 932 1 0 13 3 0 0 25 0 1 0 638848980 139010048 4151 33554432000 4194304 5683644 140734174288272 18446744073709551615 254261612549 0 2147483391 16781312 2 18446744071699207358 0 0 17 2 0 0 0
/proc/1647/statm: 33938 4151 1032 364 0 3124 0
Current children cumulated CPU time (s) 0.16
Current children cumulated vsize (KiB) 135752

[startup+0.701079 s]
/proc/loadavg: 8.20 8.04 6.76 6/359 1647
/proc/meminfo: memFree=23652044/32951124 swapFree=67057896/67111528
[pid=1647] ppid=1645 vsize=280572 CPUtime=0.38 cores=0,2,4,6
/proc/1647/stat : 1647 (python) R 1645 1647 1220 0 -1 4202496 9794 932 1 0 30 8 0 0 25 0 1 0 638848980 287305728 7478 33554432000 4194304 5683644 140734174288272 18446744073709551615 47054986929463 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/1647/statm: 70143 7478 1761 364 0 6067 0
Current children cumulated CPU time (s) 0.38
Current children cumulated vsize (KiB) 280572

[startup+1.50094 s]
/proc/loadavg: 8.20 8.04 6.76 6/361 1653
/proc/meminfo: memFree=23616308/32951124 swapFree=67057896/67111528
[pid=1647] ppid=1645 vsize=289488 CPUtime=1.19 cores=0,2,4,6
/proc/1647/stat : 1647 (python) R 1645 1647 1220 0 -1 4202496 14017 932 1 0 110 9 0 0 25 0 1 0 638848980 296435712 9756 33554432000 4194304 5683644 140734174288272 18446744073709551615 47054816223282 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/1647/statm: 72372 9756 1826 364 0 8296 0
Current children cumulated CPU time (s) 1.19
Current children cumulated vsize (KiB) 289488

[startup+3.10061 s]
/proc/loadavg: 7.94 7.99 6.75 7/361 1653
/proc/meminfo: memFree=23615936/32951124 swapFree=67057896/67111528
[pid=1647] ppid=1645 vsize=322828 CPUtime=2.41 cores=0,2,4,6
/proc/1647/stat : 1647 (python) S 1645 1647 1220 0 -1 4202496 16793 932 1 0 232 9 0 0 22 0 5 0 638848980 330575872 7922 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273443275 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/1647/statm: 80707 7922 1883 364 0 16616 0
Current children cumulated CPU time (s) 2.41
Current children cumulated vsize (KiB) 322828
heavy processes:

[startup+6.30105 s]
/proc/loadavg: 7.94 7.99 6.75 9/373 1665
/proc/meminfo: memFree=23590488/32951124 swapFree=67057896/67111528
[pid=1647] ppid=1645 vsize=322828 CPUtime=2.41 cores=0,2,4,6
/proc/1647/stat : 1647 (python) S 1645 1647 1220 0 -1 4202496 16793 932 1 0 232 9 0 0 22 0 5 0 638848980 330575872 7922 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273443275 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/1647/statm: 80707 7922 1883 364 0 16616 0
[pid=1654] ppid=1647 vsize=282232 CPUtime=0 cores=0,2,4,6
/proc/1654/stat : 1654 (python) S 1647 1647 1220 0 -1 4202560 726 0 0 0 0 0 0 0 18 0 1 0 638849250 289005568 6328 33554432000 4194304 5683644 140734174288272 18446744073709551615 254261636623 0 0 16781312 514 10740206880660881666 0 0 17 6 0 0 0
/proc/1654/statm: 70558 6328 355 364 0 6476 0
[pid=1656] ppid=1654 vsize=1020188 CPUtime=3.53 cores=0,2,4,6
/proc/1656/stat : 1656 (gnovelty+2) R 1654 1656 1656 0 -1 4202496 1141 0 0 0 353 0 0 0 25 0 1 0 638849251 1044672512 877 33554432000 134512640 135663704 4289774992 18446744073709551615 134545682 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/1656/statm: 255047 877 113 282 0 254763 0
[pid=1657] ppid=1647 vsize=293164 CPUtime=0 cores=0,2,4,6
/proc/1657/stat : 1657 (python) S 1647 1647 1220 0 -1 4202560 744 0 0 0 0 0 0 0 18 0 1 0 638849251 300199936 6377 33554432000 4194304 5683644 140734174288272 18446744073709551615 254261636678 0 0 16781312 514 9910242177701609729 0 0 17 2 0 0 0
/proc/1657/statm: 73291 6377 349 364 0 9206 0
[pid=1659] ppid=1657 vsize=1020188 CPUtime=3.57 cores=0,2,4,6
/proc/1659/stat : 1659 (gnovelty+2) R 1657 1659 1659 0 -1 4202496 1139 0 0 0 357 0 0 0 25 0 1 0 638849252 1044672512 876 33554432000 134512640 135663704 4290592992 18446744073709551615 134525271 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/1659/statm: 255047 876 113 282 0 254763 0
[pid=1660] ppid=1647 vsize=302880 CPUtime=0 cores=0,2,4,6
/proc/1660/stat : 1660 (python) S 1647 1647 1220 0 -1 4202560 744 0 0 0 0 0 0 0 18 0 1 0 638849252 310149120 6381 33554432000 4194304 5683644 140734174288272 18446744073709551615 254261636678 0 0 16781312 514 4428969455296413956 0 0 17 2 0 0 0
/proc/1660/statm: 75720 6381 349 364 0 11632 0
[pid=1662] ppid=1660 vsize=421140 CPUtime=3.56 cores=0,2,4,6
/proc/1662/stat : 1662 (adaptg2wsat2009) R 1660 1662 1662 0 -1 4202496 1806 0 0 0 356 0 0 0 25 0 1 0 638849253 431247360 1538 33554432000 134512640 135107411 4288629968 18446744073709551615 134523455 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/1662/statm: 105285 1538 49 146 0 104161 0
[pid=1663] ppid=1647 vsize=313136 CPUtime=0 cores=0,2,4,6
/proc/1663/stat : 1663 (python) S 1647 1647 1220 0 -1 4202560 742 0 0 0 0 0 0 0 18 0 1 0 638849254 320651264 6385 33554432000 4194304 5683644 140734174288272 18446744073709551615 254261636678 0 0 16781312 514 15982237967489925380 0 0 17 6 0 0 0
/proc/1663/statm: 78284 6385 349 364 0 14193 0
[pid=1665] ppid=1663 vsize=421140 CPUtime=3.55 cores=0,2,4,6
/proc/1665/stat : 1665 (adaptg2wsat2009) R 1663 1665 1665 0 -1 4202496 1799 0 0 0 355 0 0 0 25 0 1 0 638849254 431247360 1538 33554432000 134512640 135107411 4289231600 18446744073709551615 134523245 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/1665/statm: 105285 1538 49 146 0 104161 0
[pid=1647/tid=1655] ppid=1645 vsize=322828 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1655/stat : 1655 (python) S 1645 1647 1220 0 -1 4202560 23 932 0 0 0 0 0 0 25 0 5 0 638849251 330575872 7922 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1658] ppid=1645 vsize=322828 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1658/stat : 1658 (python) S 1645 1647 1220 0 -1 4202560 24 932 0 0 0 0 0 0 25 0 5 0 638849252 330575872 7922 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1661] ppid=1645 vsize=322828 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1661/stat : 1661 (python) S 1645 1647 1220 0 -1 4202560 24 932 0 0 0 0 0 0 20 0 5 0 638849253 330575872 7922 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1664] ppid=1645 vsize=322828 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1664/stat : 1664 (python) S 1645 1647 1220 0 -1 4202560 32 932 0 0 0 0 0 0 22 0 5 0 638849254 330575872 7922 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 16.62
Current children cumulated vsize (KiB) 4396896

[startup+12.7009 s]
/proc/loadavg: 8.03 8.01 6.77 10/373 1666
/proc/meminfo: memFree=23585668/32951124 swapFree=67057896/67111528
[pid=1647] ppid=1645 vsize=322828 CPUtime=2.41 cores=0,2,4,6
/proc/1647/stat : 1647 (python) S 1645 1647 1220 0 -1 4202496 16793 932 1 0 232 9 0 0 22 0 5 0 638848980 330575872 7922 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273443275 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/1647/statm: 80707 7922 1883 364 0 16616 0
[pid=1654] ppid=1647 vsize=282232 CPUtime=0 cores=0,2,4,6
/proc/1654/stat : 1654 (python) S 1647 1647 1220 0 -1 4202560 885 0 0 0 0 0 0 0 18 0 1 0 638849250 289005568 6351 33554432000 4194304 5683644 140734174288272 18446744073709551615 254261636623 0 0 16781312 514 10740206880660881666 0 0 17 6 0 0 0
/proc/1654/statm: 70558 6351 378 364 0 6476 0
[pid=1656] ppid=1654 vsize=1020188 CPUtime=9.83 cores=0,2,4,6
/proc/1656/stat : 1656 (gnovelty+2) R 1654 1656 1656 0 -1 4202496 1141 0 0 0 983 0 0 0 25 0 1 0 638849251 1044672512 877 33554432000 134512640 135663704 4289774992 18446744073709551615 134525218 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/1656/statm: 255047 877 113 282 0 254763 0
[pid=1657] ppid=1647 vsize=293164 CPUtime=0.01 cores=0,2,4,6
/proc/1657/stat : 1657 (python) S 1647 1647 1220 0 -1 4202560 916 0 0 0 1 0 0 0 18 0 1 0 638849251 300199936 6400 33554432000 4194304 5683644 140734174288272 18446744073709551615 254261636678 0 0 16781312 514 9910242177701609729 0 0 17 2 0 0 0
/proc/1657/statm: 73291 6400 372 364 0 9206 0
[pid=1659] ppid=1657 vsize=1020188 CPUtime=9.93 cores=0,2,4,6
/proc/1659/stat : 1659 (gnovelty+2) R 1657 1659 1659 0 -1 4202496 1139 0 0 0 993 0 0 0 25 0 1 0 638849252 1044672512 876 33554432000 134512640 135663704 4290592992 18446744073709551615 134525302 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/1659/statm: 255047 876 113 282 0 254763 0
[pid=1660] ppid=1647 vsize=302880 CPUtime=0 cores=0,2,4,6
/proc/1660/stat : 1660 (python) S 1647 1647 1220 0 -1 4202560 902 0 0 0 0 0 0 0 18 0 1 0 638849252 310149120 6403 33554432000 4194304 5683644 140734174288272 18446744073709551615 254261636678 0 0 16781312 514 4428969455296413956 0 0 17 2 0 0 0
/proc/1660/statm: 75720 6403 371 364 0 11632 0
[pid=1662] ppid=1660 vsize=421804 CPUtime=9.96 cores=0,2,4,6
/proc/1662/stat : 1662 (adaptg2wsat2009) R 1660 1662 1662 0 -1 4202496 2139 0 0 0 996 0 0 0 25 0 1 0 638849253 431927296 1871 33554432000 134512640 135107411 4288629968 18446744073709551615 134531613 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/1662/statm: 105451 1871 59 146 0 104327 0
[pid=1663] ppid=1647 vsize=313136 CPUtime=0.01 cores=0,2,4,6
/proc/1663/stat : 1663 (python) S 1647 1647 1220 0 -1 4202560 909 0 0 0 1 0 0 0 18 0 1 0 638849254 320651264 6407 33554432000 4194304 5683644 140734174288272 18446744073709551615 254261636678 0 0 16781312 514 15982237967489925380 0 0 17 6 0 0 0
/proc/1663/statm: 78284 6407 371 364 0 14193 0
[pid=1665] ppid=1663 vsize=421804 CPUtime=9.95 cores=0,2,4,6
/proc/1665/stat : 1665 (adaptg2wsat2009) R 1663 1665 1665 0 -1 4202496 2132 0 0 0 995 0 0 0 25 0 1 0 638849254 431927296 1871 33554432000 134512640 135107411 4289231600 18446744073709551615 134532135 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/1665/statm: 105451 1871 59 146 0 104327 0
[pid=1647/tid=1655] ppid=1645 vsize=322828 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1655/stat : 1655 (python) S 1645 1647 1220 0 -1 4202560 23 932 0 0 0 0 0 0 25 0 5 0 638849251 330575872 7922 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1658] ppid=1645 vsize=322828 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1658/stat : 1658 (python) S 1645 1647 1220 0 -1 4202560 24 932 0 0 0 0 0 0 25 0 5 0 638849252 330575872 7922 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1661] ppid=1645 vsize=322828 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1661/stat : 1661 (python) S 1645 1647 1220 0 -1 4202560 24 932 0 0 0 0 0 0 20 0 5 0 638849253 330575872 7922 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1664] ppid=1645 vsize=322828 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1664/stat : 1664 (python) S 1645 1647 1220 0 -1 4202560 32 932 0 0 0 0 0 0 22 0 5 0 638849254 330575872 7922 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 42.1
Current children cumulated vsize (KiB) 4398224

[startup+25.5007 s]
/proc/loadavg: 8.03 8.01 6.78 9/373 1666
/proc/meminfo: memFree=23585348/32951124 swapFree=67057896/67111528
[pid=1647] ppid=1645 vsize=322828 CPUtime=2.41 cores=0,2,4,6
/proc/1647/stat : 1647 (python) S 1645 1647 1220 0 -1 4202496 16793 932 1 0 232 9 0 0 22 0 5 0 638848980 330575872 7922 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273443275 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/1647/statm: 80707 7922 1883 364 0 16616 0
[pid=1654] ppid=1647 vsize=282232 CPUtime=0.05 cores=0,2,4,6
/proc/1654/stat : 1654 (python) S 1647 1647 1220 0 -1 4202560 894 0 0 0 2 3 0 0 18 0 1 0 638849250 289005568 6351 33554432000 4194304 5683644 140734174288272 18446744073709551615 254261636623 0 0 16781312 514 14596008062633345282 0 0 17 6 0 0 0
/proc/1654/statm: 70558 6351 378 364 0 6476 0
[pid=1656] ppid=1654 vsize=1020188 CPUtime=22.4 cores=0,2,4,6
/proc/1656/stat : 1656 (gnovelty+2) R 1654 1656 1656 0 -1 4202496 1141 0 0 0 2240 0 0 0 25 0 1 0 638849251 1044672512 877 33554432000 134512640 135663704 4289774992 18446744073709551615 134524918 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/1656/statm: 255047 877 113 282 0 254763 0
[pid=1657] ppid=1647 vsize=293164 CPUtime=0.05 cores=0,2,4,6
/proc/1657/stat : 1657 (python) S 1647 1647 1220 0 -1 4202560 927 0 0 0 3 2 0 0 18 0 1 0 638849251 300199936 6400 33554432000 4194304 5683644 140734174288272 18446744073709551615 254261636678 0 0 16781312 514 14596008062633345281 0 0 17 2 0 0 0
/proc/1657/statm: 73291 6400 372 364 0 9206 0
[pid=1659] ppid=1657 vsize=1020188 CPUtime=22.64 cores=0,2,4,6
/proc/1659/stat : 1659 (gnovelty+2) R 1657 1659 1659 0 -1 4202496 1139 0 0 0 2264 0 0 0 25 0 1 0 638849252 1044672512 876 33554432000 134512640 135663704 4290592992 18446744073709551615 134524918 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/1659/statm: 255047 876 113 282 0 254763 0
[pid=1660] ppid=1647 vsize=302880 CPUtime=0.05 cores=0,2,4,6
/proc/1660/stat : 1660 (python) S 1647 1647 1220 0 -1 4202560 912 0 0 0 3 2 0 0 18 0 1 0 638849252 310149120 6404 33554432000 4194304 5683644 140734174288272 18446744073709551615 254261636678 0 0 16781312 514 14596008062633345284 0 0 17 2 0 0 0
/proc/1660/statm: 75720 6404 372 364 0 11632 0
[pid=1662] ppid=1660 vsize=421804 CPUtime=22.76 cores=0,2,4,6
/proc/1662/stat : 1662 (adaptg2wsat2009) R 1660 1662 1662 0 -1 4202496 2139 0 0 0 2276 0 0 0 25 0 1 0 638849253 431927296 1871 33554432000 134512640 135107411 4288629968 18446744073709551615 134532028 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/1662/statm: 105451 1871 59 146 0 104327 0
[pid=1663] ppid=1647 vsize=313136 CPUtime=0.05 cores=0,2,4,6
/proc/1663/stat : 1663 (python) S 1647 1647 1220 0 -1 4202560 916 0 0 0 3 2 0 0 18 0 1 0 638849254 320651264 6408 33554432000 4194304 5683644 140734174288272 18446744073709551615 254261636678 0 0 16781312 514 14596008062633345284 0 0 17 6 0 0 0
/proc/1663/statm: 78284 6408 372 364 0 14193 0
[pid=1665] ppid=1663 vsize=421804 CPUtime=22.75 cores=0,2,4,6
/proc/1665/stat : 1665 (adaptg2wsat2009) R 1663 1665 1665 0 -1 4202496 2132 0 0 0 2275 0 0 0 25 0 1 0 638849254 431927296 1871 33554432000 134512640 135107411 4289231600 18446744073709551615 134531830 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/1665/statm: 105451 1871 59 146 0 104327 0
[pid=1647/tid=1655] ppid=1645 vsize=322828 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1655/stat : 1655 (python) S 1645 1647 1220 0 -1 4202560 23 932 0 0 0 0 0 0 25 0 5 0 638849251 330575872 7922 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1658] ppid=1645 vsize=322828 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1658/stat : 1658 (python) S 1645 1647 1220 0 -1 4202560 24 932 0 0 0 0 0 0 25 0 5 0 638849252 330575872 7922 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1661] ppid=1645 vsize=322828 CPUtime=0 cores=0,2,4,6

################
# More data... #
################

/proc/1663/stat : 1663 (python) S 1647 1647 1220 0 -1 4202560 1097 0 0 0 13 11 0 0 19 0 2 0 638849254 320651264 6437 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/1663/statm: 78284 6437 400 364 0 14193 0
[pid=1665] ppid=1663 vsize=421804 CPUtime=99.24 cores=0,2,4,6
/proc/1665/stat : 1665 (adaptg2wsat2009) T 1663 1665 1665 0 -1 4202496 2132 0 0 0 9924 0 0 0 25 0 1 0 638849254 431927296 1871 33554432000 134512640 135107411 4289231600 18446744073709551615 134531696 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/1665/statm: 105451 1871 59 146 0 104327 0
[pid=1663/tid=1678] ppid=1647 vsize=313136 CPUtime=0 cores=0,2,4,6
/proc/1663/task/1678/stat : 1678 (python) S 1647 1647 1220 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 638859178 320651264 6437 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 514 18446604443748946112 0 0 -1 6 0 0 0
[pid=1679] ppid=1647 vsize=324000 CPUtime=0.27 cores=0,2,4,6
/proc/1679/stat : 1679 (python) S 1647 1647 1220 0 -1 4202560 1097 0 0 0 16 11 0 0 19 0 2 0 638859179 331776000 6458 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/1679/statm: 81000 6458 400 364 0 16906 0
[pid=1681] ppid=1679 vsize=421804 CPUtime=97.24 cores=0,2,4,6
/proc/1681/stat : 1681 (adaptg2wsat2009) T 1679 1681 1681 0 -1 4202496 2138 0 0 0 9723 1 0 0 25 0 1 0 638859179 431927296 1872 33554432000 134512640 135107411 4292966304 18446744073709551615 134532135 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/1681/statm: 105451 1872 59 146 0 104327 0
[pid=1679/tid=1709] ppid=1647 vsize=324000 CPUtime=0 cores=0,2,4,6
/proc/1679/task/1709/stat : 1709 (python) S 1647 1647 1220 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 638868907 331776000 6458 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 514 18446604453968145472 0 0 -1 4 0 0 0
[pid=1683] ppid=1647 vsize=333772 CPUtime=0.27 cores=0,2,4,6
/proc/1683/stat : 1683 (python) S 1647 1647 1220 0 -1 4202560 1099 0 0 0 16 11 0 0 19 0 2 0 638859278 341782528 6596 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/1683/statm: 83443 6596 399 364 0 19346 0
[pid=1685] ppid=1683 vsize=1020188 CPUtime=99.31 cores=0,2,4,6
/proc/1685/stat : 1685 (gnovelty+2) T 1683 1685 1685 0 -1 4202496 1135 0 0 0 9931 0 0 0 25 0 1 0 638859278 1044672512 876 33554432000 134512640 135663704 4289978896 18446744073709551615 134524509 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/1685/statm: 255047 876 113 282 0 254763 0
[pid=1683/tid=1713] ppid=1647 vsize=333772 CPUtime=0 cores=0,2,4,6
/proc/1683/task/1713/stat : 1713 (python) S 1647 1647 1220 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 638869212 341782528 6596 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 514 18446604450896360512 0 0 -1 6 0 0 0
[pid=1687] ppid=1647 vsize=344028 CPUtime=0.28 cores=0,2,4,6
/proc/1687/stat : 1687 (python) S 1647 1647 1220 0 -1 4202560 1109 0 0 0 17 11 0 0 19 0 2 0 638859287 352284672 6600 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/1687/statm: 86007 6600 399 364 0 21907 0
[pid=1689] ppid=1687 vsize=251292 CPUtime=100.25 cores=0,2,4,6
/proc/1689/stat : 1689 (FH) T 1687 1689 1689 0 -1 4202496 1956 0 0 0 10025 0 0 0 25 0 1 0 638859287 257323008 1693 33554432000 134512640 134936988 4290067552 18446744073709551615 134553297 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/1689/statm: 62823 1693 41 104 0 62327 0
[pid=1687/tid=1721] ppid=1647 vsize=344028 CPUtime=0 cores=0,2,4,6
/proc/1687/task/1721/stat : 1721 (python) S 1647 1647 1220 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 638869617 352284672 6600 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 514 18446604441448907776 0 0 -1 6 0 0 0
[pid=1691] ppid=1647 vsize=354604 CPUtime=0.27 cores=0,2,4,6
/proc/1691/stat : 1691 (python) S 1647 1647 1220 0 -1 4202560 1148 0 0 0 15 12 0 0 19 0 2 0 638859292 363114496 6611 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/1691/statm: 88651 6611 406 364 0 24548 0
[pid=1693] ppid=1691 vsize=421804 CPUtime=100.22 cores=0,2,4,6
/proc/1693/stat : 1693 (adaptg2wsat2009) T 1691 1693 1693 0 -1 4202496 2141 0 0 0 10022 0 0 0 25 0 1 0 638859292 431927296 1873 33554432000 134512640 135107411 4290581264 18446744073709551615 134531584 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/1693/statm: 105451 1873 59 146 0 104327 0
[pid=1691/tid=1717] ppid=1647 vsize=354604 CPUtime=0 cores=0,2,4,6
/proc/1691/task/1717/stat : 1717 (python) S 1647 1647 1220 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 638869318 363114496 6611 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 514 18446604443748947008 0 0 -1 6 0 0 0
[pid=1710] ppid=1647 vsize=364744 CPUtime=0.27 cores=0,2,4,6
/proc/1710/stat : 1710 (python) S 1647 1647 1220 0 -1 4202560 1137 0 0 0 15 12 0 0 19 0 2 0 638868908 373497856 6615 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/1710/statm: 91186 6615 406 364 0 27080 0
[pid=1712] ppid=1710 vsize=421804 CPUtime=98.12 cores=0,2,4,6
/proc/1712/stat : 1712 (adaptg2wsat2009) T 1710 1712 1712 0 -1 4202496 2143 0 0 0 9812 0 0 0 25 0 1 0 638868909 431927296 1872 33554432000 134512640 135107411 4289913520 18446744073709551615 134531554 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/1712/statm: 105451 1872 59 146 0 104327 0
[pid=1710/tid=1736] ppid=1647 vsize=364744 CPUtime=0 cores=0,2,4,6
/proc/1710/task/1736/stat : 1736 (python) S 1647 1647 1220 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 638878749 373497856 6615 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 514 18446604453961272064 0 0 -1 2 0 0 0
[pid=1714] ppid=1647 vsize=375112 CPUtime=0.26 cores=0,2,4,6
/proc/1714/stat : 1714 (python) S 1647 1647 1220 0 -1 4202560 1129 0 0 0 13 13 0 0 19 0 2 0 638869213 384114688 6613 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/1714/statm: 93778 6613 400 364 0 29669 0
[pid=1716] ppid=1714 vsize=251292 CPUtime=99.27 cores=0,2,4,6
/proc/1716/stat : 1716 (FH) T 1714 1716 1716 0 -1 4202496 1961 0 0 0 9927 0 0 0 25 0 1 0 638869213 257323008 1693 33554432000 134512640 134936988 4291288464 18446744073709551615 134552504 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/1716/statm: 62823 1693 41 104 0 62327 0
[pid=1714/tid=1740] ppid=1647 vsize=375112 CPUtime=0 cores=0,2,4,6
/proc/1714/task/1740/stat : 1740 (python) S 1647 1647 1220 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 638879141 384114688 6613 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 514 18446604450109097280 0 0 -1 0 0 0 0
[pid=1718] ppid=1647 vsize=385268 CPUtime=0.8 cores=0,2,4,6
/proc/1718/stat : 1718 (python) S 1647 1647 1220 0 -1 4202560 1173 0 0 0 40 40 0 0 18 0 2 0 638869319 394514432 6617 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/1718/statm: 96317 6617 400 364 0 32205 0
[pid=1720] ppid=1718 vsize=268424 CPUtime=292.7 cores=0,2,4,6
/proc/1720/stat : 1720 (hybridGM3) T 1718 1720 1720 0 -1 4202496 5644 0 0 0 29266 4 0 0 25 0 1 0 638869323 274866176 5378 33554432000 134512640 135204385 4287453328 18446744073709551615 134538494 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/1720/statm: 67106 5378 63 169 0 66936 0
[pid=1718/tid=1761] ppid=1647 vsize=385268 CPUtime=0 cores=0,2,4,6
/proc/1718/task/1761/stat : 1761 (python) S 1647 1647 1220 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 638888884 394514432 6617 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 514 18446604446630210432 0 0 -1 6 0 0 0
[pid=1722] ppid=1647 vsize=395652 CPUtime=0.28 cores=0,2,4,6
/proc/1722/stat : 1722 (python) S 1647 1647 1220 0 -1 4202560 1128 0 0 0 16 12 0 0 19 0 2 0 638869618 405147648 6621 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/1722/statm: 98913 6621 400 364 0 34798 0
[pid=1724] ppid=1722 vsize=1020188 CPUtime=102.45 cores=0,2,4,6
/proc/1724/stat : 1724 (gnovelty+2) T 1722 1724 1724 0 -1 4202496 1143 0 0 0 10245 0 0 0 25 0 1 0 638869618 1044672512 876 33554432000 134512640 135663704 4294093728 18446744073709551615 134525276 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/1724/statm: 255047 876 113 282 0 254763 0
[pid=1722/tid=1745] ppid=1647 vsize=395652 CPUtime=0 cores=0,2,4,6
/proc/1722/task/1745/stat : 1745 (python) S 1647 1647 1220 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 638880152 405147648 6621 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 514 18446604453943891264 0 0 -1 2 0 0 0
[pid=1737] ppid=1647 vsize=406044 CPUtime=0.56 cores=0,2,4,6
/proc/1737/stat : 1737 (python) S 1647 1647 1220 0 -1 4202560 1182 0 0 0 29 27 0 0 18 0 2 0 638878750 415789056 6625 33554432000 4194304 5683644 140734174288272 18446744073709551615 254261636678 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/1737/statm: 101511 6625 400 364 0 37393 0
[pid=1739] ppid=1737 vsize=268420 CPUtime=207.77 cores=0,2,4,6
/proc/1739/stat : 1739 (hybridGM3) R 1737 1739 1739 0 -1 4202496 5642 0 0 0 20776 1 0 0 25 0 1 0 638878751 274862080 5376 33554432000 134512640 135204385 4291931712 18446744073709551615 134538444 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/1739/statm: 67105 5376 63 169 0 66935 0
[pid=1737/tid=1920] ppid=1647 vsize=406044 CPUtime=0 cores=0,2,4,6
/proc/1737/task/1920/stat : 1920 (python) S 1647 1647 1220 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 638898312 415789056 6625 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 514 18446604440928577728 0 0 -1 2 0 0 0
[pid=1741] ppid=1647 vsize=416204 CPUtime=0.27 cores=0,2,4,6
/proc/1741/stat : 1741 (python) S 1647 1647 1220 0 -1 4202560 1149 0 0 0 13 14 0 0 19 0 2 0 638879142 426192896 6629 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/1741/statm: 104051 6629 400 364 0 39930 0
[pid=1743] ppid=1741 vsize=421804 CPUtime=97.72 cores=0,2,4,6
/proc/1743/stat : 1743 (adaptg2wsat2009) T 1741 1743 1743 0 -1 4202496 2141 0 0 0 9772 0 0 0 25 0 1 0 638879142 431927296 1871 33554432000 134512640 135107411 4290452912 18446744073709551615 134531600 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/1743/statm: 105451 1871 59 146 0 104327 0
[pid=1741/tid=1762] ppid=1647 vsize=416204 CPUtime=0 cores=0,2,4,6
/proc/1741/task/1762/stat : 1762 (python) S 1647 1647 1220 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 638889170 426192896 6629 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 514 18446604453952734528 0 0 -1 4 0 0 0
[pid=1746] ppid=1647 vsize=426608 CPUtime=0.54 cores=0,2,4,6
/proc/1746/stat : 1746 (python) S 1647 1647 1220 0 -1 4202560 1232 0 0 0 26 28 0 0 18 0 2 0 638880153 436846592 6640 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/1746/statm: 106652 6640 407 364 0 42528 0
[pid=1748] ppid=1746 vsize=268428 CPUtime=194.63 cores=0,2,4,6
/proc/1748/stat : 1748 (hybridGM3) T 1746 1748 1748 0 -1 4202496 5653 0 0 0 19463 0 0 0 25 0 1 0 638880154 274870272 5378 33554432000 134512640 135204385 4291389296 18446744073709551615 134538284 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/1748/statm: 67107 5378 63 169 0 66937 0
[pid=1746/tid=1925] ppid=1647 vsize=426608 CPUtime=0 cores=0,2,4,6
/proc/1746/task/1925/stat : 1925 (python) S 1647 1647 1220 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 638899618 436846592 6640 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 514 18446604453956682944 0 0 -1 2 0 0 0
[pid=1763] ppid=1647 vsize=436888 CPUtime=0.28 cores=0,2,4,6
/proc/1763/stat : 1763 (python) S 1647 1647 1220 0 -1 4202560 1038 0 0 0 14 14 0 0 18 0 1 0 638889171 447373312 6628 33554432000 4194304 5683644 140734174288272 18446744073709551615 254261636678 0 0 16781312 514 14596008062633345282 0 0 17 6 0 0 0
/proc/1763/statm: 109222 6628 380 364 0 45095 0
[pid=1765] ppid=1763 vsize=268428 CPUtime=101.59 cores=0,2,4,6
/proc/1765/stat : 1765 (hybridGM3) R 1763 1765 1765 0 -1 4202496 5653 0 0 0 10157 2 0 0 25 0 1 0 638889171 274870272 5378 33554432000 134512640 135204385 4290467520 18446744073709551615 134538041 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/1765/statm: 67107 5378 63 169 0 66937 0
[pid=1922] ppid=1647 vsize=447300 CPUtime=0.03 cores=0,2,4,6
/proc/1922/stat : 1922 (python) S 1647 1647 1220 0 -1 4202560 1031 0 0 0 2 1 0 0 18 0 1 0 638898616 458035200 6638 33554432000 4194304 5683644 140734174288272 18446744073709551615 254261636678 0 0 16781312 514 14596008062633345283 0 0 17 6 0 0 0
/proc/1922/statm: 111825 6638 380 364 0 47695 0
[pid=1924] ppid=1922 vsize=459764 CPUtime=10.3 cores=0,2,4,6
/proc/1924/stat : 1924 (TNM) R 1922 1924 1924 0 -1 4202496 1932 0 0 0 1030 0 0 0 25 0 1 0 638898616 470798336 1657 33554432000 134512640 135008748 4288761984 18446744073709551615 134527104 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/1924/statm: 114941 1657 46 122 0 113841 0
[pid=1926] ppid=1647 vsize=457708 CPUtime=0 cores=0,2,4,6
/proc/1926/stat : 1926 (python) S 1647 1647 1220 0 -1 4202560 822 0 0 0 0 0 0 0 19 0 1 0 638899619 468692992 6647 33554432000 4194304 5683644 140734174288272 18446744073709551615 254261636678 0 0 16781312 514 9509739294848188415 0 0 17 6 0 0 0
/proc/1926/statm: 114427 6647 346 364 0 50294 0
[pid=1928] ppid=1926 vsize=249144 CPUtime=0.6 cores=0,2,4,6
/proc/1928/stat : 1928 (FH) R 1926 1928 1928 0 -1 4202496 1568 0 0 0 60 0 0 0 22 0 1 0 638899621 255123456 1296 33554432000 134512640 134936988 4291929568 18446744073709551615 134523436 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/1928/statm: 62286 1296 37 104 0 61790 0
[pid=1647/tid=1655] ppid=1645 vsize=467316 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1655/stat : 1655 (python) S 1645 1647 1220 0 -1 4202560 23 932 0 0 0 0 0 0 25 0 19 0 638849251 478531584 8216 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1658] ppid=1645 vsize=467316 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1658/stat : 1658 (python) S 1645 1647 1220 0 -1 4202560 24 932 0 0 0 0 0 0 25 0 19 0 638849252 478531584 8216 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1661] ppid=1645 vsize=467316 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1661/stat : 1661 (python) S 1645 1647 1220 0 -1 4202560 24 932 0 0 0 0 0 0 20 0 19 0 638849253 478531584 8216 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1664] ppid=1645 vsize=467316 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1664/stat : 1664 (python) S 1645 1647 1220 0 -1 4202560 32 932 0 0 0 0 0 0 22 0 19 0 638849254 478531584 8216 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1680] ppid=1645 vsize=467316 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1680/stat : 1680 (python) S 1645 1647 1220 0 -1 4202560 24 932 0 0 0 0 0 0 20 0 19 0 638859179 478531584 8216 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1684] ppid=1645 vsize=467316 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1684/stat : 1684 (python) S 1645 1647 1220 0 -1 4202560 29 932 0 0 0 0 0 0 20 0 19 0 638859278 478531584 8216 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1688] ppid=1645 vsize=467316 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1688/stat : 1688 (python) S 1645 1647 1220 0 -1 4202560 27 932 0 0 0 0 0 0 21 0 19 0 638859287 478531584 8216 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1692] ppid=1645 vsize=467316 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1692/stat : 1692 (python) S 1645 1647 1220 0 -1 4202560 26 932 0 0 0 0 0 0 22 0 19 0 638859292 478531584 8216 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1711] ppid=1645 vsize=467316 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1711/stat : 1711 (python) S 1645 1647 1220 0 -1 4202560 20 932 0 0 0 0 0 0 20 0 19 0 638868909 478531584 8216 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1715] ppid=1645 vsize=467316 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1715/stat : 1715 (python) S 1645 1647 1220 0 -1 4202560 24 932 0 0 0 0 0 0 20 0 19 0 638869213 478531584 8216 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1719] ppid=1645 vsize=467316 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1719/stat : 1719 (python) S 1645 1647 1220 0 -1 4202560 28 932 0 0 0 0 0 0 18 0 19 0 638869320 478531584 8216 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1723] ppid=1645 vsize=467316 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1723/stat : 1723 (python) S 1645 1647 1220 0 -1 4202560 26 932 0 0 0 0 0 0 20 0 19 0 638869618 478531584 8216 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1738] ppid=1645 vsize=467316 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1738/stat : 1738 (python) S 1645 1647 1220 0 -1 4202560 32 932 0 0 0 0 0 0 18 0 19 0 638878751 478531584 8216 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1742] ppid=1645 vsize=467316 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1742/stat : 1742 (python) S 1645 1647 1220 0 -1 4202560 28 932 0 0 0 0 0 0 20 0 19 0 638879142 478531584 8216 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1747] ppid=1645 vsize=467316 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1747/stat : 1747 (python) S 1645 1647 1220 0 -1 4202560 27 932 0 0 0 0 0 0 20 0 19 0 638880154 478531584 8216 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1764] ppid=1645 vsize=467316 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1764/stat : 1764 (python) S 1645 1647 1220 0 -1 4202560 29 932 0 0 0 0 0 0 20 0 19 0 638889171 478531584 8216 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1923] ppid=1645 vsize=467316 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1923/stat : 1923 (python) S 1645 1647 1220 0 -1 4202560 30 932 0 0 0 0 0 0 20 0 19 0 638898616 478531584 8216 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
[pid=1647/tid=1927] ppid=1645 vsize=467316 CPUtime=0 cores=0,2,4,6
/proc/1647/task/1927/stat : 1927 (python) S 1645 1647 1220 0 -1 4202560 25 932 0 0 0 0 0 0 20 0 19 0 638899619 478531584 8216 33554432000 4194304 5683644 140734174288272 18446744073709551615 254273440177 0 0 16781312 2 18446604441448908672 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 2008.2
Current children cumulated vsize (KiB) 16033672

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 1647 and gives
#  childrusage.ru_utime.tv_sec=212
#  childrusage.ru_utime.tv_usec=975622
#  childrusage.ru_stime.tv_sec=2
#  childrusage.ru_stime.tv_usec=500619
# CPU time returned by wait4() is 215.476
# while last known CPU time is 2008.2
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 507.13
CPU time (s): 2008.2
CPU user time (s): 2005.45
CPU system time (s): 2.75
CPU usage (%): 395.993
Max. virtual memory (cumulated for all children) (KiB): 16033672

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 212.976
system time used= 2.50062
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 53197
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 5514
involuntary context switches= 1416

runsolver used 3.55746 second user time and 7.38688 second system time

The end

Launcher Data 

Begin job on node140 at 2011-05-15 14:58:56
IDJOB=3378260
IDBENCH=82669
IDSOLVER=1859
FILE ID=node140/3378260-1305464336
RUNJOBID= node140-1305463229-1238
PBS_JOBID= 13324987
Free space on /tmp= 73384 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/large/unif-k7-r85-v400-c34000-S586055573-067-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378260-1305464336/watcher-3378260-1305464336 -o /tmp/evaluation-result-3378260-1305464336/solver-3378260-1305464336 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378260-1305464336.cnf -seed 868295949 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= d673a60901b4a946b283fdbb02f99fdc
RANDOM SEED=868295949

node140.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.14
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      23652324 kB
Buffers:        317900 kB
Cached:        4479208 kB
SwapCached:       8088 kB
Active:        5727060 kB
Inactive:      3241972 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      23652324 kB
SwapTotal:    67111528 kB
SwapFree:     67057896 kB
Dirty:           53356 kB
Writeback:           0 kB
AnonPages:     4169264 kB
Mapped:          32388 kB
Slab:           228440 kB
PageTables:      36592 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 73128936 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73412 MiB
End job on node140 at 2011-05-15 15:07:26