Trace number 3317104

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (TO) 4800.2 1209.01

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v25000-c105000-S5267991-057-UNKNOWN.cnf
MD5SUM1a8d3d5d5a6ad4a4b0bb6bd83ad87f53
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark312.282
Satisfiable
(Un)Satisfiability was proved
Number of variables25000
Number of clauses105000
Sum of the clauses size315000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3105000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.36/0.65	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.36/0.65	c INFO: solving HOME/instance-3317104-1304267317.cnf
0.36/0.65	c INFO: computing task features
0.73/1.01	c INFO: computed variable-clause graph statistics
0.73/1.03	c INFO: computed clause balance statistics
2.64/2.97	c INFO: computed variable balance statistics
4.24/4.56	c INFO: computed small-clause counts
4.24/4.57	c INFO: computed Horn-clause counts
5.14/5.40	c INFO: computed variable graph statistics
5.14/5.40	c INFO: collected features for HOME/instance-3317104-1304267317.cnf in 4.74 s
5.14/5.43	c INFO: running hybridGM3@0 for 97 with 4795 remaining (b = 0.40)
5.14/5.46	c INFO: running hybridGM3@0 for 97 with 4795 remaining (b = 0.24)
5.14/5.49	c INFO: running hybridGM3@0 for 97 with 4795 remaining (b = 0.17)
5.24/5.52	c INFO: running hybridGM3@0 for 194 with 4795 remaining (b = 0.23)
403.11/105.18	c INFO: running hybridGM3@0 for 97 with 4695 remaining (b = 0.10)
403.51/105.24	c INFO: running hybridGM3@0 for 97 with 4695 remaining (b = 0.10)
407.50/106.21	c INFO: running hybridGM3@0 for 97 with 4694 remaining (b = 0.10)
783.36/200.32	c INFO: running hybridGM3@99 for 97 with 4600 remaining (b = 0.10)
783.36/200.38	c INFO: running hybridGM3@100 for 97 with 4600 remaining (b = 0.10)
787.38/201.35	c INFO: running TNM@0 for 97 with 4599 remaining (b = 0.12)
791.38/202.38	c INFO: running hybridGM3@95 for 97 with 4598 remaining (b = 0.10)
1183.13/300.48	c INFO: running hybridGM3@99 for 97 with 4500 remaining (b = 0.10)
1183.52/300.51	c INFO: running hybridGM3@0 for 194 with 4500 remaining (b = 0.17)
1183.52/300.56	c INFO: running hybridGM3@0 for 194 with 4500 remaining (b = 0.13)
1187.48/301.53	c INFO: running hybridGM3@0 for 194 with 4499 remaining (b = 0.10)
1594.30/403.65	c INFO: running TNM@194 for 194 with 4397 remaining (b = 0.08)
1968.04/497.42	c INFO: running TNM@0 for 97 with 4303 remaining (b = 0.05)
1968.04/497.46	c INFO: running hybridGM3@98 for 97 with 4303 remaining (b = 0.04)
1971.99/498.44	c INFO: running hybridGM3@100 for 97 with 4302 remaining (b = 0.04)
2347.30/592.62	c INFO: running hybridGM3@101 for 97 with 4208 remaining (b = 0.04)
2367.24/597.60	c INFO: running hybridGM3@0 for 194 with 4203 remaining (b = 0.07)
2371.20/598.61	c INFO: running hybridGM3@294 for 97 with 4202 remaining (b = 0.03)
2384.36/601.97	c INFO: running TNM@391 for 194 with 4198 remaining (b = 0.05)
2734.14/689.79	c INFO: running hybridGM3@0 for 194 with 4110 remaining (b = 0.05)
2758.01/695.79	c INFO: running TNM@0 for 97 with 4104 remaining (b = 0.02)
3151.56/794.54	c INFO: running hybridGM3@99 for 97 with 4006 remaining (b = 0.02)
3157.15/795.97	c INFO: running hybridGM3@0 for 194 with 4004 remaining (b = 0.04)
3170.27/799.28	c INFO: running TNM@486 for 97 with 4001 remaining (b = 0.03)
3518.83/886.73	c INFO: running hybridGM3@97 for 97 with 3913 remaining (b = 0.02)
3530.79/889.70	c INFO: running TNM@0 for 97 with 3910 remaining (b = 0.02)
3549.49/894.45	c INFO: running TNM@683 for 776 with 3906 remaining (b = 0.14)
3898.05/981.92	c INFO: running hybridGM3@192 for 97 with 3818 remaining (b = 0.02)
3929.50/989.90	c INFO: running hybridGM3@0 for 194 with 3810 remaining (b = 0.03)
3965.69/998.93	c INFO: running TNM@0 for 97 with 3801 remaining (b = 0.01)
4300.29/1083.11	c INFO: running hybridGM3@292 for 97 with 3717 remaining (b = 0.02)
4363.86/1099.14	c INFO: running TNM@196 for 1843 with 3701 remaining (b = 0.20)
4678.18/1178.30	c INFO: running gnovelty+2@0 for 97 with 3622 remaining (b = 0.01)
4736.17/1192.88	c INFO: running hybridGM3@392 for 97 with 3607 remaining (b = 0.01)
4800.20/1208.98	c INFO: running hybridGM3@0 for 194 with 3591 remaining (b = 0.03)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317104-1304267317/watcher-3317104-1304267317 -o /tmp/evaluation-result-3317104-1304267317/solver-3317104-1304267317 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317104-1304267317.cnf -seed 2121985107 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.25 0.28 0.09 5/189 23585
/proc/meminfo: memFree=30696344/32951124 swapFree=67107188/67111528
[pid=23585] ppid=23583 vsize=15384 CPUtime=0 cores=1,3,5,7
/proc/23585/stat : 23585 (python) R 23583 23585 23486 0 -1 4202496 786 0 0 0 0 0 0 0 22 0 1 0 538330256 15753216 674 33554432000 4194304 5683644 140734581824880 18446744073709551615 210204341229 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/23585/statm: 3846 676 322 364 0 333 0

[startup+0.081288 s]
/proc/loadavg: 1.25 0.28 0.09 5/189 23585
/proc/meminfo: memFree=30696344/32951124 swapFree=67107188/67111528
[pid=23585] ppid=23583 vsize=50332 CPUtime=0.05 cores=1,3,5,7
/proc/23585/stat : 23585 (python) S 23583 23585 23486 0 -1 4202496 2010 0 0 0 3 2 0 0 17 0 1 0 538330256 51539968 1769 33554432000 4194304 5683644 140734581824880 18446744073709551615 210204645376 0 2147483391 16781312 2 18446744071699207358 0 0 17 7 0 0 0
/proc/23585/statm: 12583 1769 480 364 0 1283 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 50332

[startup+0.100379 s]
/proc/loadavg: 1.25 0.28 0.09 5/189 23585
/proc/meminfo: memFree=30696344/32951124 swapFree=67107188/67111528
[pid=23585] ppid=23583 vsize=93956 CPUtime=0.06 cores=1,3,5,7
/proc/23585/stat : 23585 (python) S 23583 23585 23486 0 -1 4202496 2692 0 0 0 4 2 0 0 17 0 1 0 538330256 96210944 2406 33554432000 4194304 5683644 140734581824880 18446744073709551615 210204645376 0 2147483391 16781312 2 18446744071699207358 0 0 17 7 0 0 0
/proc/23585/statm: 23489 2406 764 364 0 1621 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 93956

[startup+0.300368 s]
/proc/loadavg: 1.25 0.28 0.09 5/189 23585
/proc/meminfo: memFree=30696344/32951124 swapFree=67107188/67111528
[pid=23585] ppid=23583 vsize=135748 CPUtime=0.19 cores=1,3,5,7
/proc/23585/stat : 23585 (python) S 23583 23585 23486 0 -1 4202496 5965 937 1 0 13 6 0 0 17 0 1 0 538330256 139005952 4151 33554432000 4194304 5683644 140734581824880 18446744073709551615 210204643333 0 2147483391 16781312 2 18446744071699207358 0 0 17 7 0 0 0
/proc/23585/statm: 33937 4151 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 135748

[startup+0.705318 s]
/proc/loadavg: 1.25 0.28 0.09 5/189 23585
/proc/meminfo: memFree=30696344/32951124 swapFree=67107188/67111528
[pid=23585] ppid=23583 vsize=283088 CPUtime=0.44 cores=1,3,5,7
/proc/23585/stat : 23585 (python) R 23583 23585 23486 0 -1 4202496 10400 937 1 0 34 10 0 0 16 0 1 0 538330256 289882112 8082 33554432000 4194304 5683644 140734581824880 18446744073709551615 4603502 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/23585/statm: 70772 8083 1761 364 0 6696 0
Current children cumulated CPU time (s) 0.44
Current children cumulated vsize (KiB) 283088

[startup+1.50016 s]
/proc/loadavg: 1.25 0.28 0.09 6/190 23590
/proc/meminfo: memFree=30652308/32951124 swapFree=67107188/67111528
[pid=23585] ppid=23583 vsize=302956 CPUtime=1.24 cores=1,3,5,7
/proc/23585/stat : 23585 (python) R 23583 23585 23486 0 -1 4202496 18309 937 1 0 113 11 0 0 17 0 1 0 538330256 310226944 13124 33554432000 4194304 5683644 140734581824880 18446744073709551615 4602993 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/23585/statm: 75739 13124 1826 364 0 11663 0
Current children cumulated CPU time (s) 1.24
Current children cumulated vsize (KiB) 302956

[startup+3.10091 s]
/proc/loadavg: 1.55 0.35 0.12 6/190 23590
/proc/meminfo: memFree=30648712/32951124 swapFree=67107188/67111528
[pid=23585] ppid=23583 vsize=302956 CPUtime=2.84 cores=1,3,5,7
/proc/23585/stat : 23585 (python) R 23583 23585 23486 0 -1 4202496 18310 937 1 0 273 11 0 0 20 0 1 0 538330256 310226944 13125 33554432000 4194304 5683644 140734581824880 18446744073709551615 210204306863 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/23585/statm: 75739 13125 1827 364 0 11663 0
Current children cumulated CPU time (s) 2.84
Current children cumulated vsize (KiB) 302956

[startup+6.30033 s]
/proc/loadavg: 1.55 0.35 0.12 6/190 23590
/proc/meminfo: memFree=30648556/32951124 swapFree=67107188/67111528
[pid=23585] ppid=23583 vsize=342040 CPUtime=5.26 cores=1,3,5,7
/proc/23585/stat : 23585 (python) S 23583 23585 23486 0 -1 4202496 21258 937 1 0 514 12 0 0 20 0 5 0 538330256 350248960 12707 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216474059 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/23585/statm: 85510 12707 1881 364 0 21419 0
Current children cumulated CPU time (s) 5.26
Current children cumulated vsize (KiB) 342040

[startup+12.7013 s]
/proc/loadavg: 2.54 0.60 0.20 9/202 23602
/proc/meminfo: memFree=30524312/32951124 swapFree=67107188/67111528
[pid=23585] ppid=23583 vsize=342040 CPUtime=5.26 cores=1,3,5,7
/proc/23585/stat : 23585 (python) S 23583 23585 23486 0 -1 4202496 21258 937 1 0 514 12 0 0 20 0 5 0 538330256 350248960 12707 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216474059 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/23585/statm: 85510 12707 1881 364 0 21419 0
[pid=23591] ppid=23585 vsize=301028 CPUtime=0 cores=1,3,5,7
/proc/23591/stat : 23591 (python) S 23585 23585 23486 0 -1 4202560 910 0 0 0 0 0 0 0 18 0 1 0 538330799 308252672 11186 33554432000 4194304 5683644 140734581824880 18446744073709551615 210204667407 0 0 16781312 514 4576122593999814920 0 0 17 7 0 0 0
/proc/23591/statm: 75257 11186 376 364 0 11175 0
[pid=23593] ppid=23591 vsize=268420 CPUtime=7.24 cores=1,3,5,7
/proc/23593/stat : 23593 (hybridGM3) R 23591 23593 23593 0 -1 4202496 7132 0 0 0 724 0 0 0 25 0 1 0 538330800 274862080 6867 33554432000 134512640 135204385 4288409552 18446744073709551615 134546059 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/23593/statm: 67105 6867 64 169 0 66935 0
[pid=23594] ppid=23585 vsize=311284 CPUtime=0 cores=1,3,5,7
/proc/23594/stat : 23594 (python) S 23585 23585 23486 0 -1 4202560 900 0 0 0 0 0 0 0 18 0 1 0 538330802 318754816 11184 33554432000 4194304 5683644 140734581824880 18446744073709551615 210204667462 0 0 16781312 514 10936845464317690120 0 0 17 7 0 0 0
/proc/23594/statm: 77821 11184 370 364 0 13736 0
[pid=23596] ppid=23594 vsize=268424 CPUtime=7.22 cores=1,3,5,7
/proc/23596/stat : 23596 (hybridGM3) R 23594 23596 23596 0 -1 4202496 7136 0 0 0 722 0 0 0 25 0 1 0 538330803 274866176 6872 33554432000 134512640 135204385 4288922640 18446744073709551615 134537027 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/23596/statm: 67106 6872 64 169 0 66936 0
[pid=23597] ppid=23585 vsize=321540 CPUtime=0 cores=1,3,5,7
/proc/23597/stat : 23597 (python) S 23585 23585 23486 0 -1 4202560 903 0 0 0 0 0 0 0 15 0 1 0 538330805 329256960 11188 33554432000 4194304 5683644 140734581824880 18446744073709551615 210204667462 0 0 16781312 514 4438385947754594568 0 0 17 7 0 0 0
/proc/23597/statm: 80385 11188 370 364 0 16297 0
[pid=23599] ppid=23597 vsize=268428 CPUtime=7.19 cores=1,3,5,7
/proc/23599/stat : 23599 (hybridGM3) R 23597 23599 23599 0 -1 4202496 7132 0 0 0 719 0 0 0 25 0 1 0 538330806 274870272 6868 33554432000 134512640 135204385 4288800192 18446744073709551615 134536996 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/23599/statm: 67107 6868 64 169 0 66937 0
[pid=23600] ppid=23585 vsize=331796 CPUtime=0 cores=1,3,5,7
/proc/23600/stat : 23600 (python) S 23585 23585 23486 0 -1 4202560 937 0 0 0 0 0 0 0 15 0 1 0 538330808 339759104 11192 33554432000 4194304 5683644 140734581824880 18446744073709551615 210204667462 0 0 16781312 514 16156923427108651272 0 0 17 5 0 0 0
/proc/23600/statm: 82949 11192 370 364 0 18858 0
[pid=23602] ppid=23600 vsize=268424 CPUtime=7.1 cores=1,3,5,7
/proc/23602/stat : 23602 (hybridGM3) R 23600 23602 23602 0 -1 4202496 7133 0 0 0 709 1 0 0 25 0 1 0 538330809 274866176 6871 33554432000 134512640 135204385 4293183024 18446744073709551615 134546086 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/23602/statm: 67106 6871 64 169 0 66936 0
[pid=23585/tid=23592] ppid=23583 vsize=342040 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23592/stat : 23592 (python) S 23583 23585 23486 0 -1 4202560 26 937 0 0 0 0 0 0 25 0 5 0 538330800 350248960 12707 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 1 0 0 0
[pid=23585/tid=23595] ppid=23583 vsize=342040 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23595/stat : 23595 (python) S 23583 23585 23486 0 -1 4202560 22 937 0 0 0 0 0 0 25 0 5 0 538330803 350248960 12707 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 3 0 0 0
[pid=23585/tid=23598] ppid=23583 vsize=342040 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23598/stat : 23598 (python) S 23583 23585 23486 0 -1 4202560 23 937 0 0 0 0 0 0 20 0 5 0 538330806 350248960 12707 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 7 0 0 0
[pid=23585/tid=23601] ppid=23583 vsize=342040 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23601/stat : 23601 (python) S 23583 23585 23486 0 -1 4202560 28 937 0 0 0 0 0 0 20 0 5 0 538330809 350248960 12707 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 34.01
Current children cumulated vsize (KiB) 2681384

[startup+25.5011 s]
/proc/loadavg: 3.38 0.85 0.28 9/202 23603
/proc/meminfo: memFree=30524312/32951124 swapFree=67107188/67111528
[pid=23585] ppid=23583 vsize=342040 CPUtime=5.26 cores=1,3,5,7
/proc/23585/stat : 23585 (python) S 23583 23585 23486 0 -1 4202496 21258 937 1 0 514 12 0 0 20 0 5 0 538330256 350248960 12707 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216474059 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/23585/statm: 85510 12707 1881 364 0 21419 0
[pid=23591] ppid=23585 vsize=301028 CPUtime=0.02 cores=1,3,5,7
/proc/23591/stat : 23591 (python) S 23585 23585 23486 0 -1 4202560 920 0 0 0 1 1 0 0 18 0 1 0 538330799 308252672 11186 33554432000 4194304 5683644 140734581824880 18446744073709551615 210204667407 0 0 16781312 514 1956989959952892168 0 0 17 7 0 0 0
/proc/23591/statm: 75257 11186 376 364 0 11175 0
[pid=23593] ppid=23591 vsize=268420 CPUtime=20.02 cores=1,3,5,7
/proc/23593/stat : 23593 (hybridGM3) R 23591 23593 23593 0 -1 4202496 7132 0 0 0 2002 0 0 0 25 0 1 0 538330800 274862080 6867 33554432000 134512640 135204385 4288409552 18446744073709551615 134545866 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/23593/statm: 67105 6867 64 169 0 66935 0
[pid=23594] ppid=23585 vsize=311284 CPUtime=0.02 cores=1,3,5,7
/proc/23594/stat : 23594 (python) S 23585 23585 23486 0 -1 4202560 906 0 0 0 1 1 0 0 18 0 1 0 538330802 318754816 11184 33554432000 4194304 5683644 140734581824880 18446744073709551615 210204667462 0 0 16781312 514 1956989959952892168 0 0 17 7 0 0 0
/proc/23594/statm: 77821 11184 370 364 0 13736 0
[pid=23596] ppid=23594 vsize=268424 CPUtime=20.02 cores=1,3,5,7
/proc/23596/stat : 23596 (hybridGM3) R 23594 23596 23596 0 -1 4202496 7136 0 0 0 2002 0 0 0 25 0 1 0 538330803 274866176 6872 33554432000 134512640 135204385 4288922640 18446744073709551615 134537383 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/23596/statm: 67106 6872 64 169 0 66936 0
[pid=23597] ppid=23585 vsize=321540 CPUtime=0.02 cores=1,3,5,7
/proc/23597/stat : 23597 (python) S 23585 23585 23486 0 -1 4202560 906 0 0 0 1 1 0 0 15 0 1 0 538330805 329256960 11188 33554432000 4194304 5683644 140734581824880 18446744073709551615 210204667462 0 0 16781312 514 1956989959952892168 0 0 17 7 0 0 0
/proc/23597/statm: 80385 11188 370 364 0 16297 0
[pid=23599] ppid=23597 vsize=268428 CPUtime=19.99 cores=1,3,5,7
/proc/23599/stat : 23599 (hybridGM3) R 23597 23599 23599 0 -1 4202496 7133 0 0 0 1999 0 0 0 25 0 1 0 538330806 274870272 6869 33554432000 134512640 135204385 4288800192 18446744073709551615 134537403 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/23599/statm: 67107 6869 64 169 0 66937 0
[pid=23600] ppid=23585 vsize=331796 CPUtime=0.02 cores=1,3,5,7
/proc/23600/stat : 23600 (python) S 23585 23585 23486 0 -1 4202560 941 0 0 0 1 1 0 0 15 0 1 0 538330808 339759104 11192 33554432000 4194304 5683644 140734581824880 18446744073709551615 210204667462 0 0 16781312 514 1956989959952892168 0 0 17 5 0 0 0
/proc/23600/statm: 82949 11192 370 364 0 18858 0
[pid=23602] ppid=23600 vsize=268424 CPUtime=19.78 cores=1,3,5,7
/proc/23602/stat : 23602 (hybridGM3) R 23600 23602 23602 0 -1 4202496 7134 0 0 0 1977 1 0 0 25 0 1 0 538330809 274866176 6872 33554432000 134512640 135204385 4293183024 18446744073709551615 134537128 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/23602/statm: 67106 6872 64 169 0 66936 0
[pid=23585/tid=23592] ppid=23583 vsize=342040 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23592/stat : 23592 (python) S 23583 23585 23486 0 -1 4202560 26 937 0 0 0 0 0 0 25 0 5 0 538330800 350248960 12707 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 1 0 0 0
[pid=23585/tid=23595] ppid=23583 vsize=342040 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23595/stat : 23595 (python) S 23583 23585 23486 0 -1 4202560 22 937 0 0 0 0 0 0 25 0 5 0 538330803 350248960 12707 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 3 0 0 0
[pid=23585/tid=23598] ppid=23583 vsize=342040 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23598/stat : 23598 (python) S 23583 23585 23486 0 -1 4202560 23 937 0 0 0 0 0 0 20 0 5 0 538330806 350248960 12707 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 7 0 0 0
[pid=23585/tid=23601] ppid=23583 vsize=342040 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23601/stat : 23601 (python) S 23583 23585 23486 0 -1 4202560 28 937 0 0 0 0 0 0 20 0 5 0 538330809 350248960 12707 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 85.15
Current children cumulated vsize (KiB) 2681384

[startup+51.1008 s]
/proc/loadavg: 4.95 1.42 0.49 9/202 23603
/proc/meminfo: memFree=30524264/32951124 swapFree=67107188/67111528
[pid=23585] ppid=23583 vsize=342040 CPUtime=5.26 cores=1,3,5,7
/proc/23585/stat : 23585 (python) S 23583 23585 23486 0 -1 4202496 21258 937 1 0 514 12 0 0 20 0 5 0 538330256 350248960 12707 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216474059 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/23585/statm: 85510 12707 1881 364 0 21419 0
[pid=23591] ppid=23585 vsize=301028 CPUtime=0.05 cores=1,3,5,7
/proc/23591/stat : 23591 (python) S 23585 23585 23486 0 -1 4202560 927 0 0 0 3 2 0 0 18 0 1 0 538330799 308252672 11186 33554432000 4194304 5683644 140734581824880 18446744073709551615 210204667407 0 0 16781312 514 1956989959952892168 0 0 17 7 0 0 0
/proc/23591/statm: 75257 11186 376 364 0 11175 0
[pid=23593] ppid=23591 vsize=268420 CPUtime=45.59 cores=1,3,5,7
/proc/23593/stat : 23593 (hybridGM3) R 23591 23593 23593 0 -1 4202496 7132 0 0 0 4559 0 0 0 25 0 1 0 538330800 274862080 6867 33554432000 134512640 135204385 4288409552 18446744073709551615 134546082 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/23593/statm: 67105 6867 64 169 0 66935 0
[pid=23594] ppid=23585 vsize=311284 CPUtime=0.05 cores=1,3,5,7
/proc/23594/stat : 23594 (python) S 23585 23585 23486 0 -1 4202560 915 0 0 0 2 3 0 0 18 0 1 0 538330802 318754816 11184 33554432000 4194304 5683644 140734581824880 18446744073709551615 210204667462 0 0 16781312 514 1956989959952892168 0 0 17 7 0 0 0
/proc/23594/statm: 77821 11184 370 364 0 13736 0
[pid=23596] ppid=23594 vsize=268424 CPUtime=45.62 cores=1,3,5,7
/proc/23596/stat : 23596 (hybridGM3) R 23594 23596 23596 0 -1 4202496 7136 0 0 0 4562 0 0 0 25 0 1 0 538330803 274866176 6872 33554432000 134512640 135204385 4288922640 18446744073709551615 134536663 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/23596/statm: 67106 6872 64 169 0 66936 0
[pid=23597] ppid=23585 vsize=321540 CPUtime=0.05 cores=1,3,5,7
/proc/23597/stat : 23597 (python) S 23585 23585 23486 0 -1 4202560 916 0 0 0 3 2 0 0 15 0 1 0 538330805 329256960 11188 33554432000 4194304 5683644 140734581824880 18446744073709551615 210204667462 0 0 16781312 514 1956989959952892168 0 0 17 7 0 0 0
/proc/23597/statm: 80385 11188 370 364 0 16297 0
[pid=23599] ppid=23597 vsize=268428 CPUtime=45.59 cores=1,3,5,7
/proc/23599/stat : 23599 (hybridGM3) R 23597 23599 23599 0 -1 4202496 7133 0 0 0 4559 0 0 0 25 0 1 0 538330806 274870272 6869 33554432000 134512640 135204385 4288800192 18446744073709551615 134536737 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/23599/statm: 67107 6869 64 169 0 66937 0
[pid=23600] ppid=23585 vsize=331796 CPUtime=0.05 cores=1,3,5,7
/proc/23600/stat : 23600 (python) S 23585 23585 23486 0 -1 4202560 950 0 0 0 2 3 0 0 15 0 1 0 538330808 339759104 11192 33554432000 4194304 5683644 140734581824880 18446744073709551615 210204667462 0 0 16781312 514 1956989959952892168 0 0 17 5 0 0 0
/proc/23600/statm: 82949 11192 370 364 0 18858 0

################
# More data... #
################

[pid=23594/tid=23612] ppid=23585 vsize=311284 CPUtime=0 cores=1,3,5,7
/proc/23594/task/23612/stat : 23612 (python) S 23585 23585 23486 0 -1 4202560 50 0 0 0 0 0 0 0 18 0 2 0 538340874 318754816 11214 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 514 18446604464049060608 0 0 -1 1 0 0 0
[pid=23597] ppid=23585 vsize=321540 CPUtime=0.39 cores=1,3,5,7
/proc/23597/stat : 23597 (python) S 23585 23585 23486 0 -1 4202560 1155 0 0 0 19 20 0 0 16 0 2 0 538330805 329256960 11218 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/23597/statm: 80385 11218 399 364 0 16297 0
[pid=23599] ppid=23597 vsize=268428 CPUtime=295.8 cores=1,3,5,7
/proc/23599/stat : 23599 (hybridGM3) T 23597 23599 23599 0 -1 4202496 7141 0 0 0 29580 0 0 0 25 0 1 0 538330806 274870272 6877 33554432000 134512640 135204385 4288800192 18446744073709551615 134537383 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/23599/statm: 67107 6877 64 169 0 66937 0
[pid=23597/tid=23611] ppid=23585 vsize=321540 CPUtime=0 cores=1,3,5,7
/proc/23597/task/23611/stat : 23611 (python) S 23585 23585 23486 0 -1 4202560 51 0 0 0 0 0 0 0 15 0 2 0 538340777 329256960 11218 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 514 18446604471125954688 0 0 -1 7 0 0 0
[pid=23600] ppid=23585 vsize=331796 CPUtime=0.4 cores=1,3,5,7
/proc/23600/stat : 23600 (python) S 23585 23585 23486 0 -1 4202560 1177 0 0 0 19 21 0 0 15 0 2 0 538330808 339759104 11222 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/23600/statm: 82949 11222 399 364 0 18858 0
[pid=23602] ppid=23600 vsize=268424 CPUtime=296.49 cores=1,3,5,7
/proc/23602/stat : 23602 (hybridGM3) T 23600 23602 23602 0 -1 4202496 7134 0 0 0 29648 1 0 0 25 0 1 0 538330809 274866176 6872 33554432000 134512640 135204385 4293183024 18446744073709551615 134536737 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/23602/statm: 67106 6872 64 169 0 66936 0
[pid=23600/tid=23693] ppid=23585 vsize=331796 CPUtime=0 cores=1,3,5,7
/proc/23600/task/23693/stat : 23693 (python) S 23585 23585 23486 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 538350491 339759104 11222 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 514 18446604460969487424 0 0 -1 5 0 0 0
[pid=23690] ppid=23585 vsize=342052 CPUtime=1.37 cores=1,3,5,7
/proc/23690/stat : 23690 (python) S 23585 23585 23486 0 -1 4202560 1164 0 0 0 72 65 0 0 18 0 2 0 538350391 350261248 11226 33554432000 4194304 5683644 140734581824880 18446744073709551615 210204667462 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/23690/statm: 85513 11226 399 364 0 21419 0
[pid=23692] ppid=23690 vsize=460952 CPUtime=896.47 cores=1,3,5,7
/proc/23692/stat : 23692 (TNM) R 23690 23692 23692 0 -1 4202496 4400 0 0 0 89647 0 0 0 25 0 1 0 538350392 472014848 4134 33554432000 134512640 135008748 4293821344 18446744073709551615 134529448 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/23692/statm: 115238 4134 46 122 0 114138 0
[pid=23690/tid=23709] ppid=23585 vsize=342052 CPUtime=0 cores=1,3,5,7
/proc/23690/task/23709/stat : 23709 (python) S 23585 23585 23486 0 -1 4202560 48 0 0 0 0 0 0 0 18 0 2 0 538360305 350261248 11226 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 514 18446604468155771008 0 0 -1 5 0 0 0
[pid=23710] ppid=23585 vsize=352308 CPUtime=0.56 cores=1,3,5,7
/proc/23710/stat : 23710 (python) S 23585 23585 23486 0 -1 4202560 1186 0 0 0 32 24 0 0 15 0 2 0 538360308 360763392 11230 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/23710/statm: 88077 11230 399 364 0 23980 0
[pid=23712] ppid=23710 vsize=268424 CPUtime=388.56 cores=1,3,5,7
/proc/23712/stat : 23712 (hybridGM3) T 23710 23712 23712 0 -1 4202496 7137 0 0 0 38856 0 0 0 25 0 1 0 538360309 274866176 6871 33554432000 134512640 135204385 4291812896 18446744073709551615 134537510 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/23712/statm: 67106 6871 64 169 0 66936 0
[pid=23710/tid=23731] ppid=23585 vsize=352308 CPUtime=0 cores=1,3,5,7
/proc/23710/task/23731/stat : 23731 (python) S 23585 23585 23486 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 538379994 360763392 11230 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 514 18446604454224049216 0 0 -1 5 0 0 0
[pid=23713] ppid=23585 vsize=362564 CPUtime=0.42 cores=1,3,5,7
/proc/23713/stat : 23713 (python) S 23585 23585 23486 0 -1 4202560 1227 0 0 0 21 21 0 0 15 0 2 0 538360312 371265536 11240 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/23713/statm: 90641 11240 405 364 0 26541 0
[pid=23715] ppid=23713 vsize=268424 CPUtime=296 cores=1,3,5,7
/proc/23715/stat : 23715 (hybridGM3) T 23713 23715 23715 0 -1 4202496 7142 0 0 0 29600 0 0 0 25 0 1 0 538360313 274866176 6874 33554432000 134512640 135204385 4294592592 18446744073709551615 134536961 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/23715/statm: 67106 6874 64 169 0 66936 0
[pid=23713/tid=23732] ppid=23585 vsize=362564 CPUtime=0 cores=1,3,5,7
/proc/23713/task/23732/stat : 23732 (python) S 23585 23585 23486 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 538379997 371265536 11240 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 514 18446604471126690496 0 0 -1 1 0 0 0
[pid=23716] ppid=23585 vsize=372820 CPUtime=0.42 cores=1,3,5,7
/proc/23716/stat : 23716 (python) S 23585 23585 23486 0 -1 4202560 1225 0 0 0 23 19 0 0 15 0 2 0 538360410 381767680 11244 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/23716/statm: 93205 11244 405 364 0 29102 0
[pid=23718] ppid=23716 vsize=268424 CPUtime=292.54 cores=1,3,5,7
/proc/23718/stat : 23718 (hybridGM3) T 23716 23718 23718 0 -1 4202496 7142 0 0 0 29254 0 0 0 25 0 1 0 538360410 274866176 6873 33554432000 134512640 135204385 4291372960 18446744073709551615 134537608 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/23718/statm: 67106 6873 64 169 0 66936 0
[pid=23716/tid=23736] ppid=23585 vsize=372820 CPUtime=0 cores=1,3,5,7
/proc/23716/task/23736/stat : 23736 (python) S 23585 23585 23486 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 538380096 381767680 11244 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 514 18446604471127142784 0 0 -1 1 0 0 0
[pid=23733] ppid=23585 vsize=383076 CPUtime=0.34 cores=1,3,5,7
/proc/23733/stat : 23733 (python) S 23585 23585 23486 0 -1 4202560 1151 0 0 0 17 17 0 0 18 0 2 0 538379998 392269824 11242 33554432000 4194304 5683644 140734581824880 18446744073709551615 210204667462 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/23733/statm: 95769 11242 399 364 0 31663 0
[pid=23735] ppid=23733 vsize=460952 CPUtime=209.37 cores=1,3,5,7
/proc/23735/stat : 23735 (TNM) R 23733 23735 23735 0 -1 4202496 4398 0 0 0 20936 1 0 0 25 0 1 0 538379999 472014848 4133 33554432000 134512640 135008748 4291214432 18446744073709551615 134527542 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/23735/statm: 115238 4133 46 122 0 114138 0
[pid=23733/tid=23750] ppid=23585 vsize=383076 CPUtime=0 cores=1,3,5,7
/proc/23733/task/23750/stat : 23750 (python) S 23585 23585 23486 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 538390013 392269824 11242 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 514 18446604471127365632 0 0 -1 5 0 0 0
[pid=23751] ppid=23585 vsize=393332 CPUtime=0.43 cores=1,3,5,7
/proc/23751/stat : 23751 (python) S 23585 23585 23486 0 -1 4202560 1171 0 0 0 21 22 0 0 15 0 2 0 538390016 402771968 11246 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/23751/statm: 98333 11246 399 364 0 34224 0
[pid=23753] ppid=23751 vsize=268424 CPUtime=291.54 cores=1,3,5,7
/proc/23753/stat : 23753 (hybridGM3) T 23751 23753 23753 0 -1 4202496 7147 0 0 0 29154 0 0 0 25 0 1 0 538390017 274866176 6878 33554432000 134512640 135204385 4289534928 18446744073709551615 134537363 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/23753/statm: 67106 6878 64 169 0 66936 0
[pid=23751/tid=23775] ppid=23585 vsize=393332 CPUtime=0 cores=1,3,5,7
/proc/23751/task/23775/stat : 23775 (python) S 23585 23585 23486 0 -1 4202560 45 0 0 0 0 0 0 0 15 0 2 0 538409706 402771968 11246 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 514 18446604465913390912 0 0 -1 1 0 0 0
[pid=23756] ppid=23585 vsize=403588 CPUtime=0.81 cores=1,3,5,7
/proc/23756/stat : 23756 (python) S 23585 23585 23486 0 -1 4202560 1198 0 0 0 37 44 0 0 15 0 2 0 538399236 413274112 11250 33554432000 4194304 5683644 140734581824880 18446744073709551615 210204667462 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/23756/statm: 100897 11250 399 364 0 36785 0
[pid=23758] ppid=23756 vsize=281016 CPUtime=503.49 cores=1,3,5,7
/proc/23758/stat : 23758 (hybridGM3) R 23756 23758 23758 0 -1 4202496 23726 0 0 0 50347 2 0 0 25 0 1 0 538399236 287760384 10537 33554432000 134512640 135204385 4292263712 18446744073709551615 134537435 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/23758/statm: 70254 10537 85 169 0 70084 0
[pid=23756/tid=23782] ppid=23585 vsize=403588 CPUtime=0 cores=1,3,5,7
/proc/23756/task/23782/stat : 23782 (python) S 23585 23585 23486 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 538418926 413274112 11250 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 514 18446604467076019392 0 0 -1 1 0 0 0
[pid=23764] ppid=23585 vsize=413844 CPUtime=0.15 cores=1,3,5,7
/proc/23764/stat : 23764 (python) S 23585 23585 23486 0 -1 4202560 1170 0 0 0 8 7 0 0 19 0 2 0 538399835 423776256 11254 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/23764/statm: 103461 11254 399 364 0 39346 0
[pid=23766] ppid=23764 vsize=460952 CPUtime=100.13 cores=1,3,5,7
/proc/23766/stat : 23766 (TNM) T 23764 23766 23766 0 -1 4202496 4409 0 0 0 10013 0 0 0 25 0 1 0 538399835 472014848 4135 33554432000 134512640 135008748 4291839200 18446744073709551615 134529125 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/23766/statm: 115238 4135 46 122 0 114138 0
[pid=23764/tid=23776] ppid=23585 vsize=413844 CPUtime=0 cores=1,3,5,7
/proc/23764/task/23776/stat : 23776 (python) S 23585 23585 23486 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538409849 423776256 11254 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 514 18446604454223471552 0 0 -1 1 0 0 0
[pid=23777] ppid=23585 vsize=424100 CPUtime=0.31 cores=1,3,5,7
/proc/23777/stat : 23777 (python) S 23585 23585 23486 0 -1 4202560 1201 0 0 0 15 16 0 0 17 0 2 0 538409853 434278400 11258 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/23777/statm: 106025 11258 399 364 0 41907 0
[pid=23779] ppid=23777 vsize=268420 CPUtime=199.56 cores=1,3,5,7
/proc/23779/stat : 23779 (hybridGM3) T 23777 23779 23779 0 -1 4202496 7146 0 0 0 19954 2 0 0 25 0 1 0 538409854 274862080 6873 33554432000 134512640 135204385 4287008064 18446744073709551615 134536646 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/23779/statm: 67105 6873 64 169 0 66935 0
[pid=23777/tid=23805] ppid=23585 vsize=424100 CPUtime=0 cores=1,3,5,7
/proc/23777/task/23805/stat : 23805 (python) S 23585 23585 23486 0 -1 4202560 48 0 0 0 0 0 0 0 17 0 2 0 538430145 434278400 11258 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 514 18446604471127366528 0 0 -1 1 0 0 0
[pid=23783] ppid=23585 vsize=434356 CPUtime=0.16 cores=1,3,5,7
/proc/23783/stat : 23783 (python) S 23585 23585 23486 0 -1 4202560 1184 0 0 0 7 9 0 0 19 0 2 0 538419227 444780544 11262 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/23783/statm: 108589 11262 399 364 0 44468 0
[pid=23785] ppid=23783 vsize=460952 CPUtime=99.97 cores=1,3,5,7
/proc/23785/stat : 23785 (TNM) T 23783 23785 23785 0 -1 4202496 4408 0 0 0 9996 1 0 0 25 0 1 0 538419227 472014848 4134 33554432000 134512640 135008748 4288611968 18446744073709551615 134527750 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/23785/statm: 115238 4134 46 122 0 114138 0
[pid=23783/tid=23795] ppid=23585 vsize=434356 CPUtime=0 cores=1,3,5,7
/proc/23783/task/23795/stat : 23795 (python) S 23585 23585 23486 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538429243 444780544 11262 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 514 18446604471127144576 0 0 -1 5 0 0 0
[pid=23796] ppid=23585 vsize=444612 CPUtime=0.33 cores=1,3,5,7
/proc/23796/stat : 23796 (python) S 23585 23585 23486 0 -1 4202560 1263 0 0 0 15 18 0 0 16 0 2 0 538429246 455282688 11273 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/23796/statm: 111153 11273 406 364 0 47029 0
[pid=23798] ppid=23796 vsize=268420 CPUtime=197.78 cores=1,3,5,7
/proc/23798/stat : 23798 (hybridGM3) T 23796 23798 23798 0 -1 4202496 7154 0 0 0 19776 2 0 0 25 0 1 0 538429247 274862080 6875 33554432000 134512640 135204385 4292493184 18446744073709551615 134537608 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/23798/statm: 67105 6875 64 169 0 66935 0
[pid=23796/tid=23833] ppid=23585 vsize=444612 CPUtime=0 cores=1,3,5,7
/proc/23796/task/23833/stat : 23833 (python) S 23585 23585 23486 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 538449541 455282688 11273 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 514 18446604465913390016 0 0 -1 1 0 0 0
[pid=23806] ppid=23585 vsize=454868 CPUtime=0.17 cores=1,3,5,7
/proc/23806/stat : 23806 (python) S 23585 23585 23486 0 -1 4202560 1208 0 0 0 8 9 0 0 19 0 2 0 538430149 465784832 11278 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/23806/statm: 113717 11278 407 364 0 49590 0
[pid=23808] ppid=23806 vsize=460952 CPUtime=100.16 cores=1,3,5,7
/proc/23808/stat : 23808 (TNM) T 23806 23808 23808 0 -1 4202496 4408 0 0 0 10015 1 0 0 25 0 1 0 538430150 472014848 4133 33554432000 134512640 135008748 4294558272 18446744073709551615 134529552 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/23808/statm: 115238 4133 46 122 0 114138 0
[pid=23806/tid=23815] ppid=23585 vsize=454868 CPUtime=0 cores=1,3,5,7
/proc/23806/task/23815/stat : 23815 (python) S 23585 23585 23486 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 538440166 465784832 11278 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 514 18446604471125956480 0 0 -1 3 0 0 0
[pid=23822] ppid=23585 vsize=465124 CPUtime=0.05 cores=1,3,5,7
/proc/23822/stat : 23822 (python) S 23585 23585 23486 0 -1 4202560 983 0 0 0 3 2 0 0 18 0 1 0 538448086 476286976 11245 33554432000 4194304 5683644 140734581824880 18446744073709551615 210204667462 0 0 16781312 514 1956989959952892165 0 0 17 1 0 0 0
/proc/23822/statm: 116281 11245 371 364 0 52151 0
[pid=23824] ppid=23822 vsize=1025020 CPUtime=30.11 cores=1,3,5,7
/proc/23824/stat : 23824 (gnovelty+2) R 23822 23824 23824 0 -1 4202496 2629 0 0 0 3011 0 0 0 25 0 1 0 538448087 1049620480 2312 33554432000 134512640 135663704 4294489200 18446744073709551615 134524548 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/23824/statm: 256255 2312 113 282 0 255971 0
[pid=23585/tid=23592] ppid=23583 vsize=475368 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23592/stat : 23592 (python) S 23583 23585 23486 0 -1 4202560 29 937 0 0 0 0 0 0 18 0 18 0 538330800 486776832 12786 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 1 0 0 0
[pid=23585/tid=23595] ppid=23583 vsize=475368 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23595/stat : 23595 (python) S 23583 23585 23486 0 -1 4202560 30 937 0 0 0 0 0 0 18 0 18 0 538330803 486776832 12786 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 3 0 0 0
[pid=23585/tid=23598] ppid=23583 vsize=475368 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23598/stat : 23598 (python) S 23583 23585 23486 0 -1 4202560 25 937 0 0 0 0 0 0 18 0 18 0 538330806 486776832 12786 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 7 0 0 0
[pid=23585/tid=23601] ppid=23583 vsize=475368 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23601/stat : 23601 (python) S 23583 23585 23486 0 -1 4202560 31 937 0 0 0 0 0 0 18 0 18 0 538330809 486776832 12786 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 5 0 0 0
[pid=23585/tid=23691] ppid=23583 vsize=475368 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23691/stat : 23691 (python) S 23583 23585 23486 0 -1 4202560 47 937 0 0 0 0 0 0 18 0 18 0 538350392 486776832 12786 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 5 0 0 0
[pid=23585/tid=23711] ppid=23583 vsize=475368 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23711/stat : 23711 (python) S 23583 23585 23486 0 -1 4202560 35 937 0 0 0 0 0 0 18 0 18 0 538360308 486776832 12786 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 1 0 0 0
[pid=23585/tid=23714] ppid=23583 vsize=475368 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23714/stat : 23714 (python) S 23583 23585 23486 0 -1 4202560 32 937 0 0 0 0 0 0 18 0 18 0 538360313 486776832 12786 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 1 0 0 0
[pid=23585/tid=23717] ppid=23583 vsize=475368 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23717/stat : 23717 (python) S 23583 23585 23486 0 -1 4202560 85 937 0 0 0 0 0 0 18 0 18 0 538360410 486776832 12786 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 1 0 0 0
[pid=23585/tid=23734] ppid=23583 vsize=475368 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23734/stat : 23734 (python) S 23583 23585 23486 0 -1 4202560 27 937 0 0 0 0 0 0 18 0 18 0 538379999 486776832 12786 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 5 0 0 0
[pid=23585/tid=23752] ppid=23583 vsize=475368 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23752/stat : 23752 (python) S 23583 23585 23486 0 -1 4202560 36 937 0 0 0 0 0 0 18 0 18 0 538390017 486776832 12786 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 1 0 0 0
[pid=23585/tid=23757] ppid=23583 vsize=475368 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23757/stat : 23757 (python) S 23583 23585 23486 0 -1 4202560 47 937 0 0 0 0 0 0 18 0 18 0 538399236 486776832 12786 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 1 0 0 0
[pid=23585/tid=23765] ppid=23583 vsize=475368 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23765/stat : 23765 (python) S 23583 23585 23486 0 -1 4202560 29 937 0 0 0 0 0 0 20 0 18 0 538399835 486776832 12786 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 1 0 0 0
[pid=23585/tid=23778] ppid=23583 vsize=475368 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23778/stat : 23778 (python) S 23583 23585 23486 0 -1 4202560 29 937 0 0 0 0 0 0 20 0 18 0 538409853 486776832 12786 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 1 0 0 0
[pid=23585/tid=23784] ppid=23583 vsize=475368 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23784/stat : 23784 (python) S 23583 23585 23486 0 -1 4202560 32 937 0 0 0 0 0 0 20 0 18 0 538419227 486776832 12786 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 1 0 0 0
[pid=23585/tid=23797] ppid=23583 vsize=475368 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23797/stat : 23797 (python) S 23583 23585 23486 0 -1 4202560 31 937 0 0 0 0 0 0 20 0 18 0 538429247 486776832 12786 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 1 0 0 0
[pid=23585/tid=23807] ppid=23583 vsize=475368 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23807/stat : 23807 (python) S 23583 23585 23486 0 -1 4202560 25 937 0 0 0 0 0 0 20 0 18 0 538430149 486776832 12786 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 1 0 0 0
[pid=23585/tid=23823] ppid=23583 vsize=475368 CPUtime=0 cores=1,3,5,7
/proc/23585/task/23823/stat : 23823 (python) S 23583 23585 23486 0 -1 4202560 31 937 0 0 0 0 0 0 20 0 18 0 538448086 486776832 12786 33554432000 4194304 5683644 140734581824880 18446744073709551615 210216470961 0 0 16781312 2 18446604467076021184 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.2
Current children cumulated vsize (KiB) 13292932

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 23585 and gives
#  childrusage.ru_utime.tv_sec=1409
#  childrusage.ru_utime.tv_usec=511721
#  childrusage.ru_stime.tv_sec=3
#  childrusage.ru_stime.tv_usec=299498
# CPU time returned by wait4() is 1412.81
# while last known CPU time is 4800.2
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1209.01
CPU time (s): 4800.2
CPU user time (s): 4796.36
CPU system time (s): 3.84
CPU usage (%): 397.034
Max. virtual memory (cumulated for all children) (KiB): 13292936

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1409.51
system time used= 3.2995
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 79631
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7891
involuntary context switches= 12452

runsolver used 4.64229 second user time and 13.8709 second system time

The end

Launcher Data 

Begin job on node130 at 2011-05-01 18:28:37
IDJOB=3317104
IDBENCH=82569
IDSOLVER=1705
FILE ID=node130/3317104-1304267317
RUNJOBID= node130-1304267282-23505
PBS_JOBID= 13173949
Free space on /tmp= 72688 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v25000-c105000-S5267991-057-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317104-1304267317/watcher-3317104-1304267317 -o /tmp/evaluation-result-3317104-1304267317/solver-3317104-1304267317 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317104-1304267317.cnf -seed 2121985107 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 1a8d3d5d5a6ad4a4b0bb6bd83ad87f53
RANDOM SEED=2121985107

node130.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30696500 kB
Buffers:        228964 kB
Cached:        1472436 kB
SwapCached:       1916 kB
Active:         717304 kB
Inactive:      1249784 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30696500 kB
SwapTotal:    67111528 kB
SwapFree:     67107188 kB
Dirty:           42492 kB
Writeback:           0 kB
AnonPages:      264964 kB
Mapped:          21000 kB
Slab:           220572 kB
PageTables:       8336 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  1719292 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72656 MiB
End job on node130 at 2011-05-01 18:48:49