Trace number 3317111

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (TO) 4800.05 1208.7

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v25000-c105000-S109959541-056-UNKNOWN.cnf
MD5SUM5f909806405bfbe25b7db5afcabe8fc9
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark87.7936
Satisfiable
(Un)Satisfiability was proved
Number of variables24999
Number of clauses105000
Sum of the clauses size315000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3105000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.39/0.65	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.39/0.65	c INFO: solving HOME/instance-3317111-1304267330.cnf
0.39/0.65	c INFO: computing task features
0.77/1.00	c INFO: computed variable-clause graph statistics
0.77/1.02	c INFO: computed clause balance statistics
2.57/2.88	c INFO: computed variable balance statistics
4.07/4.33	c INFO: computed small-clause counts
4.07/4.34	c INFO: computed Horn-clause counts
4.87/5.17	c INFO: computed variable graph statistics
4.87/5.17	c INFO: collected features for HOME/instance-3317111-1304267330.cnf in 4.51 s
4.97/5.20	c INFO: running hybridGM3@0 for 97 with 4795 remaining (b = 0.37)
4.97/5.23	c INFO: running hybridGM3@0 for 97 with 4795 remaining (b = 0.24)
4.97/5.26	c INFO: running hybridGM3@0 for 97 with 4795 remaining (b = 0.17)
4.97/5.29	c INFO: running hybridGM3@0 for 194 with 4795 remaining (b = 0.22)
406.97/105.95	c INFO: running hybridGM3@0 for 97 with 4694 remaining (b = 0.10)
406.97/105.99	c INFO: running hybridGM3@0 for 97 with 4694 remaining (b = 0.10)
407.35/106.02	c INFO: running hybridGM3@0 for 97 with 4694 remaining (b = 0.10)
786.76/201.09	c INFO: running hybridGM3@100 for 97 with 4599 remaining (b = 0.10)
787.16/201.13	c INFO: running hybridGM3@100 for 97 with 4599 remaining (b = 0.10)
787.16/201.16	c INFO: running TNM@0 for 97 with 4599 remaining (b = 0.12)
791.13/202.17	c INFO: running hybridGM3@94 for 97 with 4598 remaining (b = 0.09)
1187.27/301.34	c INFO: running hybridGM3@100 for 97 with 4499 remaining (b = 0.09)
1190.86/302.26	c INFO: running hybridGM3@0 for 194 with 4498 remaining (b = 0.16)
1191.24/302.30	c INFO: running hybridGM3@0 for 194 with 4498 remaining (b = 0.12)
1191.24/302.34	c INFO: running hybridGM3@0 for 194 with 4498 remaining (b = 0.09)
1574.41/398.51	c INFO: running TNM@196 for 97 with 4402 remaining (b = 0.04)
1961.23/495.68	c INFO: running TNM@0 for 97 with 4304 remaining (b = 0.05)
1975.17/499.19	c INFO: running hybridGM3@100 for 97 with 4301 remaining (b = 0.04)
1975.58/499.23	c INFO: running hybridGM3@99 for 97 with 4301 remaining (b = 0.04)
1975.58/499.27	c INFO: running hybridGM3@95 for 97 with 4301 remaining (b = 0.04)
2354.46/594.34	c INFO: running hybridGM3@0 for 194 with 4206 remaining (b = 0.07)
2354.46/594.38	c INFO: running hybridGM3@297 for 97 with 4206 remaining (b = 0.03)
2360.41/595.85	c INFO: running TNM@293 for 97 with 4204 remaining (b = 0.03)
2378.69/600.43	c INFO: running hybridGM3@190 for 97 with 4200 remaining (b = 0.03)
2747.66/693.01	c INFO: running hybridGM3@294 for 97 with 4107 remaining (b = 0.03)
2757.58/695.54	c INFO: running TNM@392 for 194 with 4105 remaining (b = 0.05)
2757.58/695.58	c INFO: running hybridGM3@0 for 194 with 4105 remaining (b = 0.05)
3138.80/791.28	c INFO: running hybridGM3@94 for 97 with 4009 remaining (b = 0.02)
3154.34/795.19	c INFO: running hybridGM3@397 for 97 with 4005 remaining (b = 0.02)
3535.56/890.85	c INFO: running TNM@486 for 97 with 3909 remaining (b = 0.03)
3537.95/891.46	c INFO: running hybridGM3@395 for 97 with 3909 remaining (b = 0.02)
3541.54/892.37	c INFO: running TNM@0 for 97 with 3908 remaining (b = 0.02)
3562.23/897.55	c INFO: running hybridGM3@0 for 194 with 3903 remaining (b = 0.04)
3914.64/986.03	c INFO: running TNM@586 for 873 with 3814 remaining (b = 0.15)
3921.03/987.65	c INFO: running hybridGM3@0 for 194 with 3813 remaining (b = 0.03)
3956.46/996.60	c INFO: running TNM@0 for 97 with 3804 remaining (b = 0.01)
4355.79/1096.81	c INFO: running TNM@196 for 1843 with 3703 remaining (b = 0.20)
4358.60/1097.55	c INFO: running gnovelty+2@0 for 97 with 3703 remaining (b = 0.02)
4728.49/1190.65	c INFO: running hybridGM3@95 for 97 with 3610 remaining (b = 0.02)
4753.16/1196.87	c INFO: running hybridGM3@101 for 97 with 3603 remaining (b = 0.02)
4800.05/1208.67	c INFO: running hybridGM3@0 for 194 with 3592 remaining (b = 0.03)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317111-1304267330/watcher-3317111-1304267330 -o /tmp/evaluation-result-3317111-1304267330/solver-3317111-1304267330 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317111-1304267330.cnf -seed 867666104 -budget 4800 -cores 4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.74 0.65 0.22 5/189 1808
/proc/meminfo: memFree=30529796/32950928 swapFree=67111448/67111528
[pid=1808] ppid=1806 vsize=15080 CPUtime=0 cores=0,2,4,6
/proc/1808/stat : 1808 (python) S 1806 1808 1708 0 -1 4202496 650 0 0 0 0 0 0 0 25 0 1 0 538331684 15441920 545 33554432000 4194304 5683644 140734496419376 18446744073709551615 251814238208 0 2147483391 16781312 2 18446744071699215550 0 0 17 2 0 0 0
/proc/1808/statm: 3770 545 312 364 0 257 0

[startup+0.0910901 s]
/proc/loadavg: 2.74 0.65 0.22 5/189 1808
/proc/meminfo: memFree=30529796/32950928 swapFree=67111448/67111528
[pid=1808] ppid=1806 vsize=84600 CPUtime=0.06 cores=0,2,4,6
/proc/1808/stat : 1808 (python) R 1806 1808 1708 0 -1 4202496 2408 0 0 0 4 2 0 0 25 0 1 0 538331684 86630400 2135 33554432000 4194304 5683644 140734496419376 18446744073709551615 251814236005 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/1808/statm: 21150 2135 707 364 0 1367 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 84600

[startup+0.101088 s]
/proc/loadavg: 2.74 0.65 0.22 5/189 1808
/proc/meminfo: memFree=30529796/32950928 swapFree=67111448/67111528
[pid=1808] ppid=1806 vsize=93956 CPUtime=0.07 cores=0,2,4,6
/proc/1808/stat : 1808 (python) S 1806 1808 1708 0 -1 4202496 2692 0 0 0 5 2 0 0 25 0 1 0 538331684 96210944 2406 33554432000 4194304 5683644 140734496419376 18446744073709551615 251814238208 0 2147483391 16781312 2 18446744071699215550 0 0 17 2 0 0 0
/proc/1808/statm: 23489 2406 764 364 0 1621 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 93956

[startup+0.301056 s]
/proc/loadavg: 2.74 0.65 0.22 5/189 1808
/proc/meminfo: memFree=30529796/32950928 swapFree=67111448/67111528
[pid=1808] ppid=1806 vsize=135748 CPUtime=0.19 cores=0,2,4,6
/proc/1808/stat : 1808 (python) R 1806 1808 1708 0 -1 4202496 5964 937 1 0 13 6 0 0 25 0 1 0 538331684 139005952 4151 33554432000 4194304 5683644 140734496419376 18446744073709551615 251814045467 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/1808/statm: 33937 4151 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 135748

[startup+0.700987 s]
/proc/loadavg: 2.74 0.65 0.22 5/189 1808
/proc/meminfo: memFree=30529796/32950928 swapFree=67111448/67111528
[pid=1808] ppid=1806 vsize=283324 CPUtime=0.47 cores=0,2,4,6
/proc/1808/stat : 1808 (python) R 1806 1808 1708 0 -1 4202496 10457 937 1 0 34 13 0 0 25 0 1 0 538331684 290123776 8138 33554432000 4194304 5683644 140734496419376 18446744073709551615 4952114 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/1808/statm: 70831 8138 1761 364 0 6755 0
Current children cumulated CPU time (s) 0.47
Current children cumulated vsize (KiB) 283324

[startup+1.50085 s]
/proc/loadavg: 2.74 0.65 0.22 6/190 1813
/proc/meminfo: memFree=30483040/32950928 swapFree=67111448/67111528
[pid=1808] ppid=1806 vsize=302956 CPUtime=1.27 cores=0,2,4,6
/proc/1808/stat : 1808 (python) R 1806 1808 1708 0 -1 4202496 18311 937 1 0 113 14 0 0 25 0 1 0 538331684 310226944 13124 33554432000 4194304 5683644 140734496419376 18446744073709551615 4509415 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/1808/statm: 75739 13124 1826 364 0 11663 0
Current children cumulated CPU time (s) 1.27
Current children cumulated vsize (KiB) 302956

[startup+3.10058 s]
/proc/loadavg: 3.00 0.74 0.25 6/190 1813
/proc/meminfo: memFree=30482344/32950928 swapFree=67111448/67111528
[pid=1808] ppid=1806 vsize=302956 CPUtime=2.87 cores=0,2,4,6
/proc/1808/stat : 1808 (python) R 1806 1808 1708 0 -1 4202496 18312 937 1 0 273 14 0 0 25 0 1 0 538331684 310226944 13125 33554432000 4194304 5683644 140734496419376 18446744073709551615 5005565 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/1808/statm: 75739 13125 1827 364 0 11663 0
Current children cumulated CPU time (s) 2.87
Current children cumulated vsize (KiB) 302956

[startup+6.30105 s]
/proc/loadavg: 3.00 0.74 0.25 9/202 1825
/proc/meminfo: memFree=30423596/32950928 swapFree=67111448/67111528
[pid=1808] ppid=1806 vsize=342040 CPUtime=5.06 cores=0,2,4,6
/proc/1808/stat : 1808 (python) S 1806 1808 1708 0 -1 4202496 21248 937 1 0 491 15 0 0 25 0 5 0 538331684 350248960 12707 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/1808/statm: 85510 12707 1881 364 0 21419 0
[pid=1814] ppid=1808 vsize=301028 CPUtime=0 cores=0,2,4,6
/proc/1814/stat : 1814 (python) S 1808 1808 1708 0 -1 4202560 742 0 0 0 0 0 0 0 20 0 1 0 538332205 308252672 11163 33554432000 4194304 5683644 140734496419376 18446744073709551615 251814260239 0 0 16781312 514 4581278753778302975 0 0 17 0 0 0 0
/proc/1814/statm: 75257 11163 353 364 0 11175 0
[pid=1816] ppid=1814 vsize=268420 CPUtime=1.08 cores=0,2,4,6
/proc/1816/stat : 1816 (hybridGM3) R 1814 1816 1816 0 -1 4202496 7128 0 0 0 107 1 0 0 25 0 1 0 538332205 274862080 6868 33554432000 134512640 135204385 4293386880 18446744073709551615 134545880 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/1816/statm: 67105 6868 64 169 0 66935 0
[pid=1817] ppid=1808 vsize=311284 CPUtime=0 cores=0,2,4,6
/proc/1817/stat : 1817 (python) S 1808 1808 1708 0 -1 4202560 745 0 0 0 0 0 0 0 20 0 1 0 538332208 318754816 11161 33554432000 4194304 5683644 140734496419376 18446744073709551615 251814260294 0 0 16781312 514 4140397131028496383 0 0 17 4 0 0 0
/proc/1817/statm: 77821 11161 347 364 0 13736 0
[pid=1819] ppid=1817 vsize=268420 CPUtime=1.05 cores=0,2,4,6
/proc/1819/stat : 1819 (hybridGM3) R 1817 1819 1819 0 -1 4202496 7129 0 0 0 104 1 0 0 25 0 1 0 538332208 274862080 6867 33554432000 134512640 135204385 4286739024 18446744073709551615 134545851 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/1819/statm: 67105 6867 64 169 0 66935 0
[pid=1820] ppid=1808 vsize=321540 CPUtime=0 cores=0,2,4,6
/proc/1820/stat : 1820 (python) S 1808 1808 1708 0 -1 4202560 743 0 0 0 0 0 0 0 20 0 1 0 538332211 329256960 11165 33554432000 4194304 5683644 140734496419376 18446744073709551615 251814260294 0 0 16781312 514 4172245859716562943 0 0 17 0 0 0 0
/proc/1820/statm: 80385 11165 347 364 0 16297 0
[pid=1822] ppid=1820 vsize=268424 CPUtime=1.02 cores=0,2,4,6
/proc/1822/stat : 1822 (hybridGM3) R 1820 1822 1822 0 -1 4202496 7130 0 0 0 102 0 0 0 25 0 1 0 538332211 274866176 6869 33554432000 134512640 135204385 4289747568 18446744073709551615 134537357 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/1822/statm: 67106 6869 64 169 0 66936 0
[pid=1823] ppid=1808 vsize=331796 CPUtime=0 cores=0,2,4,6
/proc/1823/stat : 1823 (python) S 1808 1808 1708 0 -1 4202560 761 0 0 0 0 0 0 0 20 0 1 0 538332214 339759104 11169 33554432000 4194304 5683644 140734496419376 18446744073709551615 251814260294 0 0 16781312 514 4140400979319193599 0 0 17 4 0 0 0
/proc/1823/statm: 82949 11169 347 364 0 18858 0
[pid=1825] ppid=1823 vsize=268424 CPUtime=0.99 cores=0,2,4,6
/proc/1825/stat : 1825 (hybridGM3) R 1823 1825 1825 0 -1 4202496 7131 0 0 0 99 0 0 0 25 0 1 0 538332214 274866176 6870 33554432000 134512640 135204385 4287534864 18446744073709551615 134536646 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/1825/statm: 67106 6870 64 169 0 66936 0
[pid=1808/tid=1815] ppid=1806 vsize=342040 CPUtime=0 cores=0,2,4,6
/proc/1808/task/1815/stat : 1815 (python) S 1806 1808 1708 0 -1 4202560 26 937 0 0 0 0 0 0 25 0 5 0 538332205 350248960 12707 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 0 0 0 0
[pid=1808/tid=1818] ppid=1806 vsize=342040 CPUtime=0 cores=0,2,4,6
/proc/1808/task/1818/stat : 1818 (python) S 1806 1808 1708 0 -1 4202560 23 937 0 0 0 0 0 0 25 0 5 0 538332208 350248960 12707 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 6 0 0 0
[pid=1808/tid=1821] ppid=1806 vsize=342040 CPUtime=0 cores=0,2,4,6
/proc/1808/task/1821/stat : 1821 (python) S 1806 1808 1708 0 -1 4202560 23 937 0 0 0 0 0 0 25 0 5 0 538332211 350248960 12707 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 0 0 0 0
[pid=1808/tid=1824] ppid=1806 vsize=342040 CPUtime=0 cores=0,2,4,6
/proc/1808/task/1824/stat : 1824 (python) S 1806 1808 1708 0 -1 4202560 27 937 0 0 0 0 0 0 25 0 5 0 538332214 350248960 12707 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 9.2
Current children cumulated vsize (KiB) 2681376

[startup+12.701 s]
/proc/loadavg: 3.85 1.00 0.33 9/202 1825
/proc/meminfo: memFree=30358696/32950928 swapFree=67111448/67111528
[pid=1808] ppid=1806 vsize=342040 CPUtime=5.06 cores=0,2,4,6
/proc/1808/stat : 1808 (python) S 1806 1808 1708 0 -1 4202496 21248 937 1 0 491 15 0 0 25 0 5 0 538331684 350248960 12707 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/1808/statm: 85510 12707 1881 364 0 21419 0
[pid=1814] ppid=1808 vsize=301028 CPUtime=0 cores=0,2,4,6
/proc/1814/stat : 1814 (python) S 1808 1808 1708 0 -1 4202560 909 0 0 0 0 0 0 0 18 0 1 0 538332205 308252672 11186 33554432000 4194304 5683644 140734496419376 18446744073709551615 251814260239 0 0 16781312 514 4581278753778270468 0 0 17 0 0 0 0
/proc/1814/statm: 75257 11186 376 364 0 11175 0
[pid=1816] ppid=1814 vsize=268420 CPUtime=7.48 cores=0,2,4,6
/proc/1816/stat : 1816 (hybridGM3) R 1814 1816 1816 0 -1 4202496 7130 0 0 0 747 1 0 0 25 0 1 0 538332205 274862080 6870 33554432000 134512640 135204385 4293386880 18446744073709551615 134537233 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/1816/statm: 67105 6870 64 169 0 66935 0
[pid=1817] ppid=1808 vsize=311284 CPUtime=0 cores=0,2,4,6
/proc/1817/stat : 1817 (python) S 1808 1808 1708 0 -1 4202560 916 0 0 0 0 0 0 0 18 0 1 0 538332208 318754816 11184 33554432000 4194304 5683644 140734496419376 18446744073709551615 251814260294 0 0 16781312 514 4140397131028463876 0 0 17 4 0 0 0
/proc/1817/statm: 77821 11184 370 364 0 13736 0
[pid=1819] ppid=1817 vsize=268420 CPUtime=7.41 cores=0,2,4,6
/proc/1819/stat : 1819 (hybridGM3) R 1817 1819 1819 0 -1 4202496 7130 0 0 0 740 1 0 0 25 0 1 0 538332208 274862080 6868 33554432000 134512640 135204385 4286739024 18446744073709551615 134536641 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/1819/statm: 67105 6868 64 169 0 66935 0
[pid=1820] ppid=1808 vsize=321540 CPUtime=0 cores=0,2,4,6
/proc/1820/stat : 1820 (python) S 1808 1808 1708 0 -1 4202560 903 0 0 0 0 0 0 0 18 0 1 0 538332211 329256960 11188 33554432000 4194304 5683644 140734496419376 18446744073709551615 251814260294 0 0 16781312 514 4172245859716530435 0 0 17 0 0 0 0
/proc/1820/statm: 80385 11188 370 364 0 16297 0
[pid=1822] ppid=1820 vsize=268424 CPUtime=7.4 cores=0,2,4,6
/proc/1822/stat : 1822 (hybridGM3) R 1820 1822 1822 0 -1 4202496 7130 0 0 0 740 0 0 0 25 0 1 0 538332211 274866176 6869 33554432000 134512640 135204385 4289747568 18446744073709551615 134536737 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/1822/statm: 67106 6869 64 169 0 66936 0
[pid=1823] ppid=1808 vsize=331796 CPUtime=0 cores=0,2,4,6
/proc/1823/stat : 1823 (python) S 1808 1808 1708 0 -1 4202560 941 0 0 0 0 0 0 0 18 0 1 0 538332214 339759104 11192 33554432000 4194304 5683644 140734496419376 18446744073709551615 251814260294 0 0 16781312 514 4140400979319161092 0 0 17 4 0 0 0
/proc/1823/statm: 82949 11192 370 364 0 18858 0
[pid=1825] ppid=1823 vsize=268424 CPUtime=7.37 cores=0,2,4,6
/proc/1825/stat : 1825 (hybridGM3) R 1823 1825 1825 0 -1 4202496 7131 0 0 0 737 0 0 0 25 0 1 0 538332214 274866176 6870 33554432000 134512640 135204385 4287534864 18446744073709551615 134545851 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/1825/statm: 67106 6870 64 169 0 66936 0
[pid=1808/tid=1815] ppid=1806 vsize=342040 CPUtime=0 cores=0,2,4,6
/proc/1808/task/1815/stat : 1815 (python) S 1806 1808 1708 0 -1 4202560 26 937 0 0 0 0 0 0 25 0 5 0 538332205 350248960 12707 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 0 0 0 0
[pid=1808/tid=1818] ppid=1806 vsize=342040 CPUtime=0 cores=0,2,4,6
/proc/1808/task/1818/stat : 1818 (python) S 1806 1808 1708 0 -1 4202560 23 937 0 0 0 0 0 0 25 0 5 0 538332208 350248960 12707 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 6 0 0 0
[pid=1808/tid=1821] ppid=1806 vsize=342040 CPUtime=0 cores=0,2,4,6
/proc/1808/task/1821/stat : 1821 (python) S 1806 1808 1708 0 -1 4202560 23 937 0 0 0 0 0 0 25 0 5 0 538332211 350248960 12707 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 0 0 0 0
[pid=1808/tid=1824] ppid=1806 vsize=342040 CPUtime=0 cores=0,2,4,6
/proc/1808/task/1824/stat : 1824 (python) S 1806 1808 1708 0 -1 4202560 27 937 0 0 0 0 0 0 25 0 5 0 538332214 350248960 12707 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 34.72
Current children cumulated vsize (KiB) 2681376

[startup+25.5008 s]
/proc/loadavg: 4.65 1.26 0.43 9/202 1825
/proc/meminfo: memFree=30358456/32950928 swapFree=67111448/67111528
[pid=1808] ppid=1806 vsize=342040 CPUtime=5.06 cores=0,2,4,6
/proc/1808/stat : 1808 (python) S 1806 1808 1708 0 -1 4202496 21248 937 1 0 491 15 0 0 25 0 5 0 538331684 350248960 12707 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/1808/statm: 85510 12707 1881 364 0 21419 0
[pid=1814] ppid=1808 vsize=301028 CPUtime=0.02 cores=0,2,4,6
/proc/1814/stat : 1814 (python) S 1808 1808 1708 0 -1 4202560 914 0 0 0 1 1 0 0 18 0 1 0 538332205 308252672 11186 33554432000 4194304 5683644 140734496419376 18446744073709551615 251814260239 0 0 16781312 514 15325819900685943044 0 0 17 0 0 0 0
/proc/1814/statm: 75257 11186 376 364 0 11175 0
[pid=1816] ppid=1814 vsize=268420 CPUtime=20.28 cores=0,2,4,6
/proc/1816/stat : 1816 (hybridGM3) R 1814 1816 1816 0 -1 4202496 7130 0 0 0 2027 1 0 0 25 0 1 0 538332205 274862080 6870 33554432000 134512640 135204385 4293386880 18446744073709551615 134547418 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/1816/statm: 67105 6870 64 169 0 66935 0
[pid=1817] ppid=1808 vsize=311284 CPUtime=0.02 cores=0,2,4,6
/proc/1817/stat : 1817 (python) S 1808 1808 1708 0 -1 4202560 922 0 0 0 1 1 0 0 18 0 1 0 538332208 318754816 11184 33554432000 4194304 5683644 140734496419376 18446744073709551615 251814260294 0 0 16781312 514 15325819900685943044 0 0 17 4 0 0 0
/proc/1817/statm: 77821 11184 370 364 0 13736 0
[pid=1819] ppid=1817 vsize=268420 CPUtime=20.14 cores=0,2,4,6
/proc/1819/stat : 1819 (hybridGM3) R 1817 1819 1819 0 -1 4202496 7130 0 0 0 2013 1 0 0 25 0 1 0 538332208 274862080 6868 33554432000 134512640 135204385 4286739024 18446744073709551615 134537292 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/1819/statm: 67105 6868 64 169 0 66935 0
[pid=1820] ppid=1808 vsize=321540 CPUtime=0.02 cores=0,2,4,6
/proc/1820/stat : 1820 (python) S 1808 1808 1708 0 -1 4202560 908 0 0 0 1 1 0 0 18 0 1 0 538332211 329256960 11188 33554432000 4194304 5683644 140734496419376 18446744073709551615 251814260294 0 0 16781312 514 15325819900685943043 0 0 17 0 0 0 0
/proc/1820/statm: 80385 11188 370 364 0 16297 0
[pid=1822] ppid=1820 vsize=268424 CPUtime=20.17 cores=0,2,4,6
/proc/1822/stat : 1822 (hybridGM3) R 1820 1822 1822 0 -1 4202496 7130 0 0 0 2017 0 0 0 25 0 1 0 538332211 274866176 6869 33554432000 134512640 135204385 4289747568 18446744073709551615 134549096 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/1822/statm: 67106 6869 64 169 0 66936 0
[pid=1823] ppid=1808 vsize=331796 CPUtime=0.02 cores=0,2,4,6
/proc/1823/stat : 1823 (python) S 1808 1808 1708 0 -1 4202560 946 0 0 0 1 1 0 0 18 0 1 0 538332214 339759104 11192 33554432000 4194304 5683644 140734496419376 18446744073709551615 251814260294 0 0 16781312 514 15325819900685943044 0 0 17 4 0 0 0
/proc/1823/statm: 82949 11192 370 364 0 18858 0
[pid=1825] ppid=1823 vsize=268424 CPUtime=20.14 cores=0,2,4,6
/proc/1825/stat : 1825 (hybridGM3) R 1823 1825 1825 0 -1 4202496 7131 0 0 0 2014 0 0 0 25 0 1 0 538332214 274866176 6870 33554432000 134512640 135204385 4287534864 18446744073709551615 134537573 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/1825/statm: 67106 6870 64 169 0 66936 0
[pid=1808/tid=1815] ppid=1806 vsize=342040 CPUtime=0 cores=0,2,4,6
/proc/1808/task/1815/stat : 1815 (python) S 1806 1808 1708 0 -1 4202560 26 937 0 0 0 0 0 0 25 0 5 0 538332205 350248960 12707 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 0 0 0 0
[pid=1808/tid=1818] ppid=1806 vsize=342040 CPUtime=0 cores=0,2,4,6
/proc/1808/task/1818/stat : 1818 (python) S 1806 1808 1708 0 -1 4202560 23 937 0 0 0 0 0 0 25 0 5 0 538332208 350248960 12707 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 6 0 0 0
[pid=1808/tid=1821] ppid=1806 vsize=342040 CPUtime=0 cores=0,2,4,6
/proc/1808/task/1821/stat : 1821 (python) S 1806 1808 1708 0 -1 4202560 23 937 0 0 0 0 0 0 25 0 5 0 538332211 350248960 12707 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 0 0 0 0

################
# More data... #
################

[pid=1819] ppid=1817 vsize=268420 CPUtime=296.07 cores=0,2,4,6
/proc/1819/stat : 1819 (hybridGM3) T 1817 1819 1819 0 -1 4202496 7132 0 0 0 29605 2 0 0 25 0 1 0 538332208 274862080 6870 33554432000 134512640 135204385 4286739024 18446744073709551615 134537292 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/1819/statm: 67105 6870 64 169 0 66935 0
[pid=1817/tid=1845] ppid=1808 vsize=311284 CPUtime=0 cores=0,2,4,6
/proc/1817/task/1845/stat : 1845 (python) S 1808 1808 1708 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 538342281 318754816 11214 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 514 18446604453876704576 0 0 -1 2 0 0 0
[pid=1820] ppid=1808 vsize=321540 CPUtime=0.41 cores=0,2,4,6
/proc/1820/stat : 1820 (python) S 1808 1808 1708 0 -1 4202560 1154 0 0 0 21 20 0 0 18 0 2 0 538332211 329256960 11218 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/1820/statm: 80385 11218 399 364 0 16297 0
[pid=1822] ppid=1820 vsize=268424 CPUtime=294.79 cores=0,2,4,6
/proc/1822/stat : 1822 (hybridGM3) T 1820 1822 1822 0 -1 4202496 7135 0 0 0 29479 0 0 0 25 0 1 0 538332211 274866176 6874 33554432000 134512640 135204385 4289747568 18446744073709551615 134536646 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/1822/statm: 67106 6874 64 169 0 66936 0
[pid=1820/tid=1846] ppid=1808 vsize=321540 CPUtime=0 cores=0,2,4,6
/proc/1820/task/1846/stat : 1846 (python) S 1808 1808 1708 0 -1 4202560 45 0 0 0 0 0 0 0 18 0 2 0 538342283 329256960 11218 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 514 18446604453884119936 0 0 -1 0 0 0 0
[pid=1823] ppid=1808 vsize=331796 CPUtime=0.42 cores=0,2,4,6
/proc/1823/stat : 1823 (python) S 1808 1808 1708 0 -1 4202560 1176 0 0 0 21 21 0 0 18 0 2 0 538332214 339759104 11222 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/1823/statm: 82949 11222 399 364 0 18858 0
[pid=1825] ppid=1823 vsize=268424 CPUtime=291.87 cores=0,2,4,6
/proc/1825/stat : 1825 (hybridGM3) T 1823 1825 1825 0 -1 4202496 7133 0 0 0 29185 2 0 0 25 0 1 0 538332214 274866176 6872 33554432000 134512640 135204385 4287534864 18446744073709551615 134536737 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/1825/statm: 67106 6872 64 169 0 66936 0
[pid=1823/tid=1869] ppid=1808 vsize=331796 CPUtime=0 cores=0,2,4,6
/proc/1823/task/1869/stat : 1869 (python) S 1808 1808 1708 0 -1 4202560 48 0 0 0 0 0 0 0 18 0 2 0 538351899 339759104 11222 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 514 18446604453876705472 0 0 -1 6 0 0 0
[pid=1866] ppid=1808 vsize=342052 CPUtime=1.26 cores=0,2,4,6
/proc/1866/stat : 1866 (python) S 1808 1808 1708 0 -1 4202560 1174 0 0 0 65 61 0 0 18 0 2 0 538351801 350261248 11226 33554432000 4194304 5683644 140734496419376 18446744073709551615 251814260294 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/1866/statm: 85513 11226 399 364 0 21419 0
[pid=1868] ppid=1866 vsize=460952 CPUtime=806.3 cores=0,2,4,6
/proc/1868/stat : 1868 (TNM) R 1866 1868 1868 0 -1 4202496 4400 0 0 0 80627 3 0 0 25 0 1 0 538351801 472014848 4134 33554432000 134512640 135008748 4291004336 18446744073709551615 134527693 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/1868/statm: 115238 4134 46 122 0 114138 0
[pid=1866/tid=1885] ppid=1808 vsize=342052 CPUtime=0 cores=0,2,4,6
/proc/1866/task/1885/stat : 1885 (python) S 1808 1808 1708 0 -1 4202560 48 0 0 0 0 0 0 0 18 0 2 0 538361816 350261248 11226 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 514 18446604452414602048 0 0 -1 2 0 0 0
[pid=1886] ppid=1808 vsize=352308 CPUtime=0.72 cores=0,2,4,6
/proc/1886/stat : 1886 (python) S 1808 1808 1708 0 -1 4202560 1188 0 0 0 40 32 0 0 15 0 2 0 538361911 360763392 11239 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/1886/statm: 88077 11239 408 364 0 23980 0
[pid=1888] ppid=1886 vsize=268428 CPUtime=487.73 cores=0,2,4,6
/proc/1888/stat : 1888 (hybridGM3) T 1886 1888 1888 0 -1 4202496 7141 0 0 0 48768 5 0 0 25 0 1 0 538361912 274870272 6875 33554432000 134512640 135204385 4292172320 18446744073709551615 134536698 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/1888/statm: 67107 6875 64 169 0 66937 0
[pid=1886/tid=1954] ppid=1808 vsize=352308 CPUtime=0 cores=0,2,4,6
/proc/1886/task/1954/stat : 1954 (python) S 1808 1808 1708 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 538381600 360763392 11239 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 514 18446604453882025152 0 0 -1 0 0 0 0
[pid=1889] ppid=1808 vsize=362564 CPUtime=0.75 cores=0,2,4,6
/proc/1889/stat : 1889 (python) S 1808 1808 1708 0 -1 4202560 1225 0 0 0 43 32 0 0 15 0 2 0 538361915 371265536 11240 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/1889/statm: 90641 11240 405 364 0 26541 0
[pid=1891] ppid=1889 vsize=268424 CPUtime=489.52 cores=0,2,4,6
/proc/1891/stat : 1891 (hybridGM3) T 1889 1891 1891 0 -1 4202496 7138 0 0 0 48950 2 0 0 25 0 1 0 538361916 274866176 6873 33554432000 134512640 135204385 4291015712 18446744073709551615 134536777 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/1891/statm: 67106 6873 64 169 0 66936 0
[pid=1889/tid=1955] ppid=1808 vsize=362564 CPUtime=0 cores=0,2,4,6
/proc/1889/task/1955/stat : 1955 (python) S 1808 1808 1708 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 538381603 371265536 11240 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 514 18446604453882026048 0 0 -1 4 0 0 0
[pid=1892] ppid=1808 vsize=372820 CPUtime=0.56 cores=0,2,4,6
/proc/1892/stat : 1892 (python) S 1808 1808 1708 0 -1 4202560 1216 0 0 0 31 25 0 0 15 0 2 0 538361919 381767680 11244 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/1892/statm: 93205 11244 405 364 0 29102 0
[pid=1894] ppid=1892 vsize=268420 CPUtime=388.13 cores=0,2,4,6
/proc/1894/stat : 1894 (hybridGM3) T 1892 1894 1894 0 -1 4202496 7142 0 0 0 38811 2 0 0 25 0 1 0 538361920 274862080 6872 33554432000 134512640 135204385 4288770752 18446744073709551615 134546270 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/1894/statm: 67105 6872 64 169 0 66935 0
[pid=1892/tid=1956] ppid=1808 vsize=372820 CPUtime=0 cores=0,2,4,6
/proc/1892/task/1956/stat : 1956 (python) S 1808 1808 1708 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 538381606 381767680 11244 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 514 18446604441936149504 0 0 -1 2 0 0 0
[pid=1950] ppid=1808 vsize=383076 CPUtime=0.32 cores=0,2,4,6
/proc/1950/stat : 1950 (python) S 1808 1808 1708 0 -1 4202560 1182 0 0 0 17 15 0 0 18 0 2 0 538381253 392269824 11251 33554432000 4194304 5683644 140734496419376 18446744073709551615 251814260294 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/1950/statm: 95769 11251 408 364 0 31663 0
[pid=1952] ppid=1950 vsize=460952 CPUtime=211.78 cores=0,2,4,6
/proc/1952/stat : 1952 (TNM) R 1950 1952 1952 0 -1 4202496 4404 0 0 0 21176 2 0 0 25 0 1 0 538381253 472014848 4135 33554432000 134512640 135008748 4290168224 18446744073709551615 134529475 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/1952/statm: 115238 4135 46 122 0 114138 0
[pid=1950/tid=2050] ppid=1808 vsize=383076 CPUtime=0 cores=0,2,4,6
/proc/1950/task/2050/stat : 2050 (python) S 1808 1808 1708 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 538391266 392269824 11251 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 514 18446604453876930688 0 0 -1 2 0 0 0
[pid=2047] ppid=1808 vsize=393332 CPUtime=0.44 cores=0,2,4,6
/proc/2047/stat : 2047 (python) S 1808 1808 1708 0 -1 4202560 1195 0 0 0 23 21 0 0 15 0 2 0 538391119 402771968 11246 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/2047/statm: 98333 11246 399 364 0 34224 0
[pid=2049] ppid=2047 vsize=268424 CPUtime=296.79 cores=0,2,4,6
/proc/2049/stat : 2049 (hybridGM3) T 2047 2049 2049 0 -1 4202496 7140 0 0 0 29678 1 0 0 25 0 1 0 538391119 274866176 6873 33554432000 134512640 135204385 4289513904 18446744073709551615 134536767 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/2049/statm: 67106 6873 64 169 0 66936 0
[pid=2047/tid=2091] ppid=1808 vsize=393332 CPUtime=0 cores=0,2,4,6
/proc/2047/task/2091/stat : 2091 (python) S 1808 1808 1708 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 538410809 402771968 11246 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 514 18446604453985333312 0 0 -1 6 0 0 0
[pid=2070] ppid=1808 vsize=403588 CPUtime=0.31 cores=0,2,4,6
/proc/2070/stat : 2070 (python) S 1808 1808 1708 0 -1 4202560 1201 0 0 0 16 15 0 0 16 0 2 0 538401243 413274112 11250 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/2070/statm: 100897 11250 399 364 0 36785 0
[pid=2072] ppid=2070 vsize=268420 CPUtime=199.76 cores=0,2,4,6
/proc/2072/stat : 2072 (hybridGM3) T 2070 2072 2072 0 -1 4202496 7143 0 0 0 19975 1 0 0 25 0 1 0 538401243 274862080 6873 33554432000 134512640 135204385 4290813280 18446744073709551615 134545864 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/2072/statm: 67105 6873 64 169 0 66935 0
[pid=2070/tid=2110] ppid=1808 vsize=403588 CPUtime=0 cores=0,2,4,6
/proc/2070/task/2110/stat : 2110 (python) S 1808 1808 1708 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 538421437 413274112 11250 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 514 18446604453744989440 0 0 -1 6 0 0 0
[pid=2107] ppid=1808 vsize=413844 CPUtime=0.18 cores=0,2,4,6
/proc/2107/stat : 2107 (python) S 1808 1808 1708 0 -1 4202560 1168 0 0 0 8 10 0 0 19 0 2 0 538420922 423776256 11254 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/2107/statm: 103461 11254 399 364 0 39346 0
[pid=2109] ppid=2107 vsize=460952 CPUtime=101.88 cores=0,2,4,6
/proc/2109/stat : 2109 (TNM) T 2107 2109 2109 0 -1 4202496 4408 0 0 0 10187 1 0 0 25 0 1 0 538420922 472014848 4134 33554432000 134512640 135008748 4291875120 18446744073709551615 134529176 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/2109/statm: 115238 4134 46 122 0 114138 0
[pid=2107/tid=2125] ppid=1808 vsize=413844 CPUtime=0 cores=0,2,4,6
/proc/2107/task/2125/stat : 2125 (python) S 1808 1808 1708 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538431341 423776256 11254 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 514 18446604453902491776 0 0 -1 2 0 0 0
[pid=2111] ppid=1808 vsize=424100 CPUtime=0.36 cores=0,2,4,6
/proc/2111/stat : 2111 (python) S 1808 1808 1708 0 -1 4202560 1198 0 0 0 19 17 0 0 15 0 2 0 538421440 434278400 11258 33554432000 4194304 5683644 140734496419376 18446744073709551615 251814260294 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/2111/statm: 106025 11258 399 364 0 41907 0
[pid=2113] ppid=2111 vsize=268424 CPUtime=215.33 cores=0,2,4,6
/proc/2113/stat : 2113 (hybridGM3) R 2111 2113 2113 0 -1 4202496 7146 0 0 0 21532 1 0 0 25 0 1 0 538421441 274866176 6874 33554432000 134512640 135204385 4289350880 18446744073709551615 134537455 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/2113/statm: 67106 6874 64 169 0 66936 0
[pid=2111/tid=2187] ppid=1808 vsize=424100 CPUtime=0 cores=0,2,4,6
/proc/2111/task/2187/stat : 2187 (python) S 1808 1808 1708 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 538441435 434278400 11258 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 514 18446604453889174528 0 0 -1 6 0 0 0
[pid=2122] ppid=1808 vsize=434356 CPUtime=0.35 cores=0,2,4,6
/proc/2122/stat : 2122 (python) S 1808 1808 1708 0 -1 4202560 1202 0 0 0 18 17 0 0 15 0 2 0 538430450 444780544 11262 33554432000 4194304 5683644 140734496419376 18446744073709551615 251814260294 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/2122/statm: 108589 11262 399 364 0 44468 0
[pid=2124] ppid=2122 vsize=268420 CPUtime=210.99 cores=0,2,4,6
/proc/2124/stat : 2124 (hybridGM3) R 2122 2124 2124 0 -1 4202496 7146 0 0 0 21094 5 0 0 25 0 1 0 538430451 274862080 6875 33554432000 134512640 135204385 4292305280 18446744073709551615 134537363 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/2124/statm: 67105 6875 64 169 0 66935 0
[pid=2122/tid=2238] ppid=1808 vsize=434356 CPUtime=0 cores=0,2,4,6
/proc/2122/task/2238/stat : 2238 (python) S 1808 1808 1708 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 538450746 444780544 11262 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 514 18446604453874711424 0 0 -1 6 0 0 0
[pid=2126] ppid=1808 vsize=444612 CPUtime=0.17 cores=0,2,4,6
/proc/2126/stat : 2126 (python) S 1808 1808 1708 0 -1 4202560 1224 0 0 0 9 8 0 0 19 0 2 0 538431345 455282688 11274 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/2126/statm: 111153 11274 407 364 0 47029 0
[pid=2128] ppid=2126 vsize=460952 CPUtime=100.15 cores=0,2,4,6
/proc/2128/stat : 2128 (TNM) T 2126 2128 2128 0 -1 4202496 4409 0 0 0 10014 1 0 0 25 0 1 0 538431346 472014848 4133 33554432000 134512640 135008748 4287037968 18446744073709551615 134527693 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/2128/statm: 115238 4133 46 122 0 114138 0
[pid=2126/tid=2186] ppid=1808 vsize=444612 CPUtime=0 cores=0,2,4,6
/proc/2126/task/2186/stat : 2186 (python) S 1808 1808 1708 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538441362 455282688 11274 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 514 18446604453869824768 0 0 -1 0 0 0 0
[pid=2188] ppid=1808 vsize=454868 CPUtime=0.16 cores=0,2,4,6
/proc/2188/stat : 2188 (python) S 1808 1808 1708 0 -1 4202560 1183 0 0 0 9 7 0 0 19 0 2 0 538441439 465784832 11278 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/2188/statm: 113717 11278 407 364 0 49590 0
[pid=2190] ppid=2188 vsize=1025020 CPUtime=99.06 cores=0,2,4,6
/proc/2190/stat : 2190 (gnovelty+2) T 2188 2190 2190 0 -1 4202496 2625 0 0 0 9903 3 0 0 25 0 1 0 538441440 1049620480 2319 33554432000 134512640 135663704 4290252192 18446744073709551615 134544196 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/2190/statm: 256255 2319 113 282 0 255971 0
[pid=2188/tid=2239] ppid=1808 vsize=454868 CPUtime=0 cores=0,2,4,6
/proc/2188/task/2239/stat : 2239 (python) S 1808 1808 1708 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538451368 465784832 11278 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 514 18446604453857019072 0 0 -1 0 0 0 0
[pid=1808/tid=1815] ppid=1806 vsize=465112 CPUtime=0 cores=0,2,4,6
/proc/1808/task/1815/stat : 1815 (python) S 1806 1808 1708 0 -1 4202560 30 937 0 0 0 0 0 0 18 0 17 0 538332205 476274688 12780 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 0 0 0 0
[pid=1808/tid=1818] ppid=1806 vsize=465112 CPUtime=0 cores=0,2,4,6
/proc/1808/task/1818/stat : 1818 (python) S 1806 1808 1708 0 -1 4202560 27 937 0 0 0 0 0 0 15 0 17 0 538332208 476274688 12780 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 6 0 0 0
[pid=1808/tid=1821] ppid=1806 vsize=465112 CPUtime=0 cores=0,2,4,6
/proc/1808/task/1821/stat : 1821 (python) S 1806 1808 1708 0 -1 4202560 30 937 0 0 0 0 0 0 18 0 17 0 538332211 476274688 12780 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 0 0 0 0
[pid=1808/tid=1824] ppid=1806 vsize=465112 CPUtime=0 cores=0,2,4,6
/proc/1808/task/1824/stat : 1824 (python) S 1806 1808 1708 0 -1 4202560 30 937 0 0 0 0 0 0 18 0 17 0 538332214 476274688 12780 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 4 0 0 0
[pid=1808/tid=1867] ppid=1806 vsize=465112 CPUtime=0 cores=0,2,4,6
/proc/1808/task/1867/stat : 1867 (python) S 1806 1808 1708 0 -1 4202560 43 937 0 0 0 0 0 0 18 0 17 0 538351801 476274688 12780 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 4 0 0 0
[pid=1808/tid=1887] ppid=1806 vsize=465112 CPUtime=0 cores=0,2,4,6
/proc/1808/task/1887/stat : 1887 (python) S 1806 1808 1708 0 -1 4202560 34 937 0 0 0 0 0 0 15 0 17 0 538361911 476274688 12780 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 0 0 0 0
[pid=1808/tid=1890] ppid=1806 vsize=465112 CPUtime=0 cores=0,2,4,6
/proc/1808/task/1890/stat : 1890 (python) S 1806 1808 1708 0 -1 4202560 38 937 0 0 0 0 0 0 15 0 17 0 538361916 476274688 12780 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 4 0 0 0
[pid=1808/tid=1893] ppid=1806 vsize=465112 CPUtime=0 cores=0,2,4,6
/proc/1808/task/1893/stat : 1893 (python) S 1806 1808 1708 0 -1 4202560 85 937 0 0 0 0 0 0 18 0 17 0 538361920 476274688 12780 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 2 0 0 0
[pid=1808/tid=1951] ppid=1806 vsize=465112 CPUtime=0 cores=0,2,4,6
/proc/1808/task/1951/stat : 1951 (python) S 1806 1808 1708 0 -1 4202560 32 937 0 0 0 0 0 0 18 0 17 0 538381253 476274688 12780 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 2 0 0 0
[pid=1808/tid=2048] ppid=1806 vsize=465112 CPUtime=0 cores=0,2,4,6
/proc/1808/task/2048/stat : 2048 (python) S 1806 1808 1708 0 -1 4202560 36 937 0 0 0 0 0 0 18 0 17 0 538391119 476274688 12780 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 6 0 0 0
[pid=1808/tid=2071] ppid=1806 vsize=465112 CPUtime=0 cores=0,2,4,6
/proc/1808/task/2071/stat : 2071 (python) S 1806 1808 1708 0 -1 4202560 29 937 0 0 0 0 0 0 20 0 17 0 538401243 476274688 12780 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 6 0 0 0
[pid=1808/tid=2108] ppid=1806 vsize=465112 CPUtime=0 cores=0,2,4,6
/proc/1808/task/2108/stat : 2108 (python) S 1806 1808 1708 0 -1 4202560 29 937 0 0 0 0 0 0 20 0 17 0 538420922 476274688 12780 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 6 0 0 0
[pid=1808/tid=2112] ppid=1806 vsize=465112 CPUtime=0 cores=0,2,4,6
/proc/1808/task/2112/stat : 2112 (python) S 1806 1808 1708 0 -1 4202560 32 937 0 0 0 0 0 0 18 0 17 0 538421441 476274688 12780 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 6 0 0 0
[pid=1808/tid=2123] ppid=1806 vsize=465112 CPUtime=0 cores=0,2,4,6
/proc/1808/task/2123/stat : 2123 (python) S 1806 1808 1708 0 -1 4202560 37 937 0 0 0 0 0 0 18 0 17 0 538430451 476274688 12780 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 6 0 0 0
[pid=1808/tid=2127] ppid=1806 vsize=465112 CPUtime=0 cores=0,2,4,6
/proc/1808/task/2127/stat : 2127 (python) S 1806 1808 1708 0 -1 4202560 27 937 0 0 0 0 0 0 20 0 17 0 538431346 476274688 12780 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 6 0 0 0
[pid=1808/tid=2189] ppid=1806 vsize=465112 CPUtime=0 cores=0,2,4,6
/proc/1808/task/2189/stat : 2189 (python) S 1806 1808 1708 0 -1 4202560 27 937 0 0 0 0 0 0 20 0 17 0 538441440 476274688 12780 33554432000 4194304 5683644 140734496419376 18446744073709551615 251826063793 0 0 16781312 2 18446604452414600256 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4800.05
Current children cumulated vsize (KiB) 12344000

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 1808 and gives
#  childrusage.ru_utime.tv_sec=1027
#  childrusage.ru_utime.tv_usec=564786
#  childrusage.ru_stime.tv_sec=3
#  childrusage.ru_stime.tv_usec=140522
# CPU time returned by wait4() is 1030.71
# while last known CPU time is 4800.05
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1208.7
CPU time (s): 4800.05
CPU user time (s): 4796.12
CPU system time (s): 3.93
CPU usage (%): 397.124
Max. virtual memory (cumulated for all children) (KiB): 12344004

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1027.56
system time used= 3.14052
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 57824
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7590
involuntary context switches= 2790

runsolver used 4.87826 second user time and 13.04 second system time

The end

Launcher Data 

Begin job on node126 at 2011-05-01 18:28:50
IDJOB=3317111
IDBENCH=82576
IDSOLVER=1705
FILE ID=node126/3317111-1304267330
RUNJOBID= node126-1304267283-1726
PBS_JOBID= 13173953
Free space on /tmp= 73224 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v25000-c105000-S109959541-056-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317111-1304267330/watcher-3317111-1304267330 -o /tmp/evaluation-result-3317111-1304267330/solver-3317111-1304267330 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317111-1304267330.cnf -seed 867666104 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 5f909806405bfbe25b7db5afcabe8fc9
RANDOM SEED=867666104

node126.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.56
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      30530076 kB
Buffers:        313332 kB
Cached:        1662468 kB
SwapCached:         76 kB
Active:         416912 kB
Inactive:      1683520 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      30530076 kB
SwapTotal:    67111528 kB
SwapFree:     67111448 kB
Dirty:           42844 kB
Writeback:          12 kB
AnonPages:      124328 kB
Mapped:          21160 kB
Slab:           254136 kB
PageTables:       7988 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:  3333180 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73180 MiB
End job on node126 at 2011-05-01 18:49:01