Trace number 3386546

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-seq-random (fixed)? (MO) 3458.06 3444.14

General information on the benchmark

Namerandom/large/
unif-k5-r20-v1500-c30000-S969405384-096-UNKNOWN.cnf
MD5SUM6e345981a96012731da9bd2ce5420289
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark1088.25
Satisfiable
(Un)Satisfiability was proved
Number of variables1500
Number of clauses30000
Sum of the clauses size150000
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 530000
Number of clauses of size over 50

Solver Data 

0.32/0.45	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.32/0.45	c INFO: solving HOME/instance-3386546-1305586406.cnf
0.32/0.45	c INFO: computing task features
0.50/0.61	c INFO: computed variable-clause graph statistics
0.50/0.62	c INFO: computed clause balance statistics
1.20/1.39	c INFO: computed variable balance statistics
1.70/1.80	c INFO: computed small-clause counts
1.70/1.80	c INFO: computed Horn-clause counts
1.70/1.83	c INFO: computed variable graph statistics
1.70/1.83	c INFO: collected features for HOME/instance-3386546-1305586406.cnf in 1.38 s
1.70/1.86	c INFO: running gnovelty+2@0 for 97 with 4998 remaining (b = 0.38)
100.56/100.33	c INFO: running gnovelty+2@0 for 97 with 4900 remaining (b = 0.14)
200.44/199.81	c INFO: running adaptg2wsat2009++@0 for 97 with 4800 remaining (b = 0.09)
301.15/300.23	c INFO: running adaptg2wsat2009++@0 for 97 with 4700 remaining (b = 0.07)
401.79/400.65	c INFO: running adaptg2wsat2009++@0 for 97 with 4600 remaining (b = 0.06)
502.57/501.08	c INFO: running gnovelty+2@0 for 97 with 4499 remaining (b = 0.05)
602.48/600.58	c INFO: running FH@0 for 97 with 4400 remaining (b = 0.04)
702.38/700.05	c INFO: running adaptg2wsat2009++@0 for 97 with 4300 remaining (b = 0.03)
803.15/800.49	c INFO: running adaptg2wsat2009++@0 for 97 with 4200 remaining (b = 0.03)
902.05/898.95	c INFO: running FH@0 for 97 with 4101 remaining (b = 0.02)
1004.98/1001.45	c INFO: running hybridGM3@0 for 194 with 3999 remaining (b = 0.04)
1201.79/1197.43	c INFO: running gnovelty+2@0 for 97 with 3803 remaining (b = 0.02)
1302.72/1297.93	c INFO: running hybridGM3@0 for 194 with 3702 remaining (b = 0.03)
1501.40/1495.88	c INFO: running hybridGM3@0 for 194 with 3504 remaining (b = 0.03)
1698.29/1691.85	c INFO: running TNM@0 for 1552 with 3309 remaining (b = 0.21)
3259.78/3246.63	c INFO: running adaptg2wsat2009++@0 for 97 with 1754 remaining (b = 0.02)
3358.61/3345.10	c INFO: running FH@0 for 97 with 1655 remaining (b = 0.01)
3457.52/3443.58	c INFO: running hybridGM3@0 for 194 with 1557 remaining (b = 0.02)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3386546-1305586406/watcher-3386546-1305586406 -o /tmp/evaluation-result-3386546-1305586406/solver-3386546-1305586406 -C 5000 -W 5100 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386546-1305586406.cnf -seed 1302764447 -budget 5000 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 5030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 4.04 3.25 3.85 4/514 31406
/proc/meminfo: memFree=15830964/32951124 swapFree=45193380/67111528
[pid=31406] ppid=31404 vsize=16400 CPUtime=0.01 cores=0,2,4,6
/proc/31406/stat : 31406 (python) R 31404 31406 31236 0 -1 4202496 1090 0 0 0 1 0 0 0 25 0 1 0 748871276 16793600 960 33554432000 4194304 5683644 140734812124048 18446744073709551615 224284586693 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/31406/statm: 4100 966 394 364 0 587 0

[startup+0.065811 s]
/proc/loadavg: 4.04 3.25 3.85 4/514 31406
/proc/meminfo: memFree=15830964/32951124 swapFree=45193380/67111528
[pid=31406] ppid=31404 vsize=50336 CPUtime=0.05 cores=0,2,4,6
/proc/31406/stat : 31406 (python) S 31404 31406 31236 0 -1 4202496 2009 0 0 0 4 1 0 0 25 0 1 0 748871276 51544064 1770 33554432000 4194304 5683644 140734812124048 18446744073709551615 224280002183 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/31406/statm: 12584 1770 480 364 0 1284 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 50336

[startup+0.100959 s]
/proc/loadavg: 4.04 3.25 3.85 4/514 31406
/proc/meminfo: memFree=15830964/32951124 swapFree=45193380/67111528
[pid=31406] ppid=31404 vsize=104616 CPUtime=0.06 cores=0,2,4,6
/proc/31406/stat : 31406 (python) S 31404 31406 31236 0 -1 4202496 3144 0 0 0 5 1 0 0 17 0 1 0 748871276 107126784 2822 33554432000 4194304 5683644 140734812124048 18446744073709551615 224284923904 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/31406/statm: 26154 2822 847 364 0 1902 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 104616

[startup+0.306789 s]
/proc/loadavg: 4.04 3.25 3.85 4/514 31406
/proc/meminfo: memFree=15830964/32951124 swapFree=45193380/67111528
[pid=31406] ppid=31404 vsize=198916 CPUtime=0.23 cores=0,2,4,6
/proc/31406/stat : 31406 (python) R 31404 31406 31236 0 -1 4202496 7026 936 1 0 18 5 0 0 17 0 1 0 748871276 203689984 5053 33554432000 4194304 5683644 140734812124048 18446744073709551615 4620076 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/31406/statm: 49729 5053 1466 364 0 3756 0
Current children cumulated CPU time (s) 0.23
Current children cumulated vsize (KiB) 198916

[startup+0.70073 s]
/proc/loadavg: 4.04 3.25 3.85 4/514 31406
/proc/meminfo: memFree=15830964/32951124 swapFree=45193380/67111528
[pid=31406] ppid=31404 vsize=287584 CPUtime=0.6 cores=0,2,4,6
/proc/31406/stat : 31406 (python) R 31404 31406 31236 0 -1 4202496 12690 936 1 0 52 8 0 0 17 0 1 0 748871276 294486016 9281 33554432000 4194304 5683644 140734812124048 18446744073709551615 224284580857 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/31406/statm: 71896 9281 1826 364 0 7820 0
Current children cumulated CPU time (s) 0.6
Current children cumulated vsize (KiB) 287584

[startup+1.50058 s]
/proc/loadavg: 4.04 3.25 3.85 5/515 31411
/proc/meminfo: memFree=15798344/32951124 swapFree=45193380/67111528
[pid=31406] ppid=31404 vsize=287584 CPUtime=1.4 cores=0,2,4,6
/proc/31406/stat : 31406 (python) R 31404 31406 31236 0 -1 4202496 12691 936 1 0 132 8 0 0 19 0 1 0 748871276 294486016 9282 33554432000 4194304 5683644 140734812124048 18446744073709551615 4457125 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/31406/statm: 71896 9282 1827 364 0 7820 0
Current children cumulated CPU time (s) 1.4
Current children cumulated vsize (KiB) 287584

[startup+3.10036 s]
/proc/loadavg: 4.04 3.25 3.85 5/518 31414
/proc/meminfo: memFree=15798592/32951124 swapFree=45193380/67111528
[pid=31406] ppid=31404 vsize=292812 CPUtime=1.76 cores=0,2,4,6
/proc/31406/stat : 31406 (python) S 31404 31406 31236 0 -1 4202496 12992 936 1 0 168 8 0 0 21 0 2 0 748871276 299839488 8108 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296752587 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/31406/statm: 73203 8108 1877 364 0 9121 0
[pid=31412] ppid=31406 vsize=283512 CPUtime=0 cores=0,2,4,6
/proc/31412/stat : 31412 (python) S 31406 31406 31236 0 -1 4202560 742 0 0 0 0 0 0 0 18 0 1 0 748871463 290316288 6583 33554432000 4194304 5683644 140734812124048 18446744073709551615 224284945935 0 0 16781312 514 17573807711852658947 0 0 17 4 0 0 0
/proc/31412/statm: 70878 6583 355 364 0 6796 0
[pid=31414] ppid=31412 vsize=1018996 CPUtime=1.22 cores=0,2,4,6
/proc/31414/stat : 31414 (gnovelty+2) R 31412 31414 31414 0 -1 4202496 960 0 0 0 122 0 0 0 25 0 1 0 748871463 1043451904 698 33554432000 134512640 135663704 4291381344 18446744073709551615 134547147 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/31414/statm: 254749 698 113 282 0 254465 0
[pid=31406/tid=31413] ppid=31404 vsize=292812 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31413/stat : 31413 (python) S 31404 31406 31236 0 -1 4202560 31 936 0 0 0 0 0 0 21 0 2 0 748871463 299839488 8108 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 2.98
Current children cumulated vsize (KiB) 1595320

[startup+6.3008 s]
/proc/loadavg: 4.04 3.26 3.85 5/518 31414
/proc/meminfo: memFree=15798592/32951124 swapFree=45193380/67111528
[pid=31406] ppid=31404 vsize=292812 CPUtime=1.76 cores=0,2,4,6
/proc/31406/stat : 31406 (python) S 31404 31406 31236 0 -1 4202496 12992 936 1 0 168 8 0 0 21 0 2 0 748871276 299839488 8108 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296752587 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/31406/statm: 73203 8108 1877 364 0 9121 0
[pid=31412] ppid=31406 vsize=283512 CPUtime=0.02 cores=0,2,4,6
/proc/31412/stat : 31412 (python) S 31406 31406 31236 0 -1 4202560 895 0 0 0 1 1 0 0 18 0 1 0 748871463 290316288 6606 33554432000 4194304 5683644 140734812124048 18446744073709551615 224284945935 0 0 16781312 514 17573807711852691455 0 0 17 4 0 0 0
/proc/31412/statm: 70878 6606 378 364 0 6796 0
[pid=31414] ppid=31412 vsize=1018996 CPUtime=4.43 cores=0,2,4,6
/proc/31414/stat : 31414 (gnovelty+2) R 31412 31414 31414 0 -1 4202496 960 0 0 0 443 0 0 0 25 0 1 0 748871463 1043451904 698 33554432000 134512640 135663704 4291381344 18446744073709551615 134524897 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/31414/statm: 254749 698 113 282 0 254465 0
[pid=31406/tid=31413] ppid=31404 vsize=292812 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31413/stat : 31413 (python) S 31404 31406 31236 0 -1 4202560 31 936 0 0 0 0 0 0 21 0 2 0 748871463 299839488 8108 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 6.21
Current children cumulated vsize (KiB) 1595320

[startup+12.7007 s]
/proc/loadavg: 4.04 3.27 3.85 5/518 31414
/proc/meminfo: memFree=15797124/32951124 swapFree=45193380/67111528
[pid=31406] ppid=31404 vsize=292812 CPUtime=1.76 cores=0,2,4,6
/proc/31406/stat : 31406 (python) S 31404 31406 31236 0 -1 4202496 12992 936 1 0 168 8 0 0 21 0 2 0 748871276 299839488 8108 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296752587 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/31406/statm: 73203 8108 1877 364 0 9121 0
[pid=31412] ppid=31406 vsize=283512 CPUtime=0.03 cores=0,2,4,6
/proc/31412/stat : 31412 (python) S 31406 31406 31236 0 -1 4202560 896 0 0 0 2 1 0 0 18 0 1 0 748871463 290316288 6606 33554432000 4194304 5683644 140734812124048 18446744073709551615 224284945935 0 0 16781312 514 8195671712512049411 0 0 17 4 0 0 0
/proc/31412/statm: 70878 6606 378 364 0 6796 0
[pid=31414] ppid=31412 vsize=1018996 CPUtime=10.83 cores=0,2,4,6
/proc/31414/stat : 31414 (gnovelty+2) R 31412 31414 31414 0 -1 4202496 960 0 0 0 1083 0 0 0 25 0 1 0 748871463 1043451904 698 33554432000 134512640 135663704 4291381344 18446744073709551615 134525243 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/31414/statm: 254749 698 113 282 0 254465 0
[pid=31406/tid=31413] ppid=31404 vsize=292812 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31413/stat : 31413 (python) S 31404 31406 31236 0 -1 4202560 31 936 0 0 0 0 0 0 21 0 2 0 748871463 299839488 8108 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 12.62
Current children cumulated vsize (KiB) 1595320

[startup+25.5006 s]
/proc/loadavg: 4.03 3.31 3.85 5/518 31414
/proc/meminfo: memFree=15798100/32951124 swapFree=45193380/67111528
[pid=31406] ppid=31404 vsize=292812 CPUtime=1.76 cores=0,2,4,6
/proc/31406/stat : 31406 (python) S 31404 31406 31236 0 -1 4202496 12992 936 1 0 168 8 0 0 21 0 2 0 748871276 299839488 8108 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296752587 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/31406/statm: 73203 8108 1877 364 0 9121 0
[pid=31412] ppid=31406 vsize=283512 CPUtime=0.09 cores=0,2,4,6
/proc/31412/stat : 31412 (python) S 31406 31406 31236 0 -1 4202560 900 0 0 0 5 4 0 0 18 0 1 0 748871463 290316288 6606 33554432000 4194304 5683644 140734812124048 18446744073709551615 224284945935 0 0 16781312 514 8195671712512049411 0 0 17 4 0 0 0
/proc/31412/statm: 70878 6606 378 364 0 6796 0
[pid=31414] ppid=31412 vsize=1018996 CPUtime=23.63 cores=0,2,4,6
/proc/31414/stat : 31414 (gnovelty+2) R 31412 31414 31414 0 -1 4202496 960 0 0 0 2363 0 0 0 25 0 1 0 748871463 1043451904 698 33554432000 134512640 135663704 4291381344 18446744073709551615 134525222 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/31414/statm: 254749 698 113 282 0 254465 0
[pid=31406/tid=31413] ppid=31404 vsize=292812 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31413/stat : 31413 (python) S 31404 31406 31236 0 -1 4202560 31 936 0 0 0 0 0 0 21 0 2 0 748871463 299839488 8108 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 25.48
Current children cumulated vsize (KiB) 1595320

[startup+51.1003 s]
/proc/loadavg: 4.13 3.39 3.87 5/518 31414
/proc/meminfo: memFree=15798276/32951124 swapFree=45193380/67111528
[pid=31406] ppid=31404 vsize=292812 CPUtime=1.76 cores=0,2,4,6
/proc/31406/stat : 31406 (python) S 31404 31406 31236 0 -1 4202496 12992 936 1 0 168 8 0 0 21 0 2 0 748871276 299839488 8108 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296752587 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/31406/statm: 73203 8108 1877 364 0 9121 0
[pid=31412] ppid=31406 vsize=283512 CPUtime=0.17 cores=0,2,4,6
/proc/31412/stat : 31412 (python) S 31406 31406 31236 0 -1 4202560 908 0 0 0 10 7 0 0 18 0 1 0 748871463 290316288 6606 33554432000 4194304 5683644 140734812124048 18446744073709551615 224284945935 0 0 16781312 514 8195671712512049411 0 0 17 6 0 0 0
/proc/31412/statm: 70878 6606 378 364 0 6796 0
[pid=31414] ppid=31412 vsize=1018996 CPUtime=49.22 cores=0,2,4,6
/proc/31414/stat : 31414 (gnovelty+2) R 31412 31414 31414 0 -1 4202496 960 0 0 0 4922 0 0 0 25 0 1 0 748871463 1043451904 698 33554432000 134512640 135663704 4291381344 18446744073709551615 134524897 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/31414/statm: 254749 698 113 282 0 254465 0
[pid=31406/tid=31413] ppid=31404 vsize=292812 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31413/stat : 31413 (python) S 31404 31406 31236 0 -1 4202560 31 936 0 0 0 0 0 0 21 0 2 0 748871463 299839488 8108 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 51.15
Current children cumulated vsize (KiB) 1595320

[startup+102.32 s]
/proc/loadavg: 4.06 3.48 3.87 5/526 31424
/proc/meminfo: memFree=15756532/32951124 swapFree=45193380/67111528
[pid=31406] ppid=31404 vsize=303528 CPUtime=1.79 cores=0,2,4,6
/proc/31406/stat : 31406 (python) S 31404 31406 31236 0 -1 4202496 13897 936 1 0 171 8 0 0 20 0 3 0 748871276 310812672 8198 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296752587 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/31406/statm: 75882 8198 1880 364 0 11797 0
[pid=31412] ppid=31406 vsize=293756 CPUtime=0.39 cores=0,2,4,6
/proc/31412/stat : 31412 (python) S 31406 31406 31236 0 -1 4202560 1081 0 0 0 22 17 0 0 19 0 2 0 748871463 300806144 6643 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/31412/statm: 73439 6643 410 364 0 9357 0
[pid=31414] ppid=31412 vsize=1018996 CPUtime=98.43 cores=0,2,4,6
/proc/31414/stat : 31414 (gnovelty+2) T 31412 31414 31414 0 -1 4202496 960 0 0 0 9843 0 0 0 25 0 1 0 748871463 1043451904 698 33554432000 134512640 135663704 4291381344 18446744073709551615 134524918 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/31414/statm: 254749 698 113 282 0 254465 0
[pid=31412/tid=31421] ppid=31406 vsize=293756 CPUtime=0 cores=0,2,4,6
/proc/31412/task/31421/stat : 31421 (python) S 31406 31406 31236 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 748881307 300806144 6643 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 514 18446604452709446912 0 0 -1 6 0 0 0
[pid=31422] ppid=31406 vsize=293700 CPUtime=0 cores=0,2,4,6
/proc/31422/stat : 31422 (python) S 31406 31406 31236 0 -1 4202560 748 0 0 0 0 0 0 0 18 0 1 0 748881310 300748800 6664 33554432000 4194304 5683644 140734812124048 18446744073709551615 224284945990 0 0 16781312 514 13856468849504780546 0 0 17 4 0 0 0
/proc/31422/statm: 73425 6664 349 364 0 9340 0
[pid=31424] ppid=31422 vsize=1018996 CPUtime=1.97 cores=0,2,4,6
/proc/31424/stat : 31424 (gnovelty+2) R 31422 31424 31424 0 -1 4202496 965 0 0 0 197 0 0 0 25 0 1 0 748881310 1043451904 698 33554432000 134512640 135663704 4290835888 18446744073709551615 134525218 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/31424/statm: 254749 698 113 282 0 254465 0
[pid=31406/tid=31413] ppid=31404 vsize=303528 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31413/stat : 31413 (python) S 31404 31406 31236 0 -1 4202560 31 936 0 0 0 0 0 0 21 0 3 0 748871463 310812672 8198 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 4 0 0 0
[pid=31406/tid=31423] ppid=31404 vsize=303528 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31423/stat : 31423 (python) S 31404 31406 31236 0 -1 4202560 30 936 0 0 0 0 0 0 20 0 3 0 748881310 310812672 8198 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 102.58
Current children cumulated vsize (KiB) 2928976

[startup+162.301 s]
/proc/loadavg: 4.02 3.57 3.88 5/526 31426
/proc/meminfo: memFree=15753284/32951124 swapFree=45193380/67111528
[pid=31406] ppid=31404 vsize=303528 CPUtime=1.79 cores=0,2,4,6
/proc/31406/stat : 31406 (python) S 31404 31406 31236 0 -1 4202496 13897 936 1 0 171 8 0 0 20 0 3 0 748871276 310812672 8198 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296752587 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/31406/statm: 75882 8198 1880 364 0 11797 0
[pid=31412] ppid=31406 vsize=293756 CPUtime=0.39 cores=0,2,4,6
/proc/31412/stat : 31412 (python) S 31406 31406 31236 0 -1 4202560 1081 0 0 0 22 17 0 0 19 0 2 0 748871463 300806144 6643 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/31412/statm: 73439 6643 410 364 0 9357 0
[pid=31414] ppid=31412 vsize=1018996 CPUtime=98.43 cores=0,2,4,6
/proc/31414/stat : 31414 (gnovelty+2) T 31412 31414 31414 0 -1 4202496 960 0 0 0 9843 0 0 0 25 0 1 0 748871463 1043451904 698 33554432000 134512640 135663704 4291381344 18446744073709551615 134524918 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/31414/statm: 254749 698 113 282 0 254465 0
[pid=31412/tid=31421] ppid=31406 vsize=293756 CPUtime=0 cores=0,2,4,6
/proc/31412/task/31421/stat : 31421 (python) S 31406 31406 31236 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 748881307 300806144 6643 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 514 18446604452709446912 0 0 -1 6 0 0 0
[pid=31422] ppid=31406 vsize=293700 CPUtime=0.23 cores=0,2,4,6

################
# More data... #
################

/proc/31443/statm: 105357 1781 59 146 0 104233 0
[pid=31441/tid=31446] ppid=31406 vsize=314388 CPUtime=0 cores=0,2,4,6
/proc/31441/task/31446/stat : 31446 (python) S 31406 31406 31236 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 748911338 321933312 6728 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 514 18446604452709447808 0 0 -1 2 0 0 0
[pid=31447] ppid=31406 vsize=324740 CPUtime=0.4 cores=0,2,4,6
/proc/31447/stat : 31447 (python) S 31406 31406 31236 0 -1 4202560 1114 0 0 0 21 19 0 0 16 0 2 0 748911341 332533760 6754 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/31447/statm: 81185 6754 400 364 0 17091 0
[pid=31449] ppid=31447 vsize=421428 CPUtime=100.38 cores=0,2,4,6
/proc/31449/stat : 31449 (adaptg2wsat2009) T 31447 31449 31449 0 -1 4202496 2053 0 0 0 10038 0 0 0 25 0 1 0 748911342 431542272 1782 33554432000 134512640 135107411 4288262304 18446744073709551615 134531613 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/31449/statm: 105357 1782 59 146 0 104233 0
[pid=31447/tid=31464] ppid=31406 vsize=324740 CPUtime=0 cores=0,2,4,6
/proc/31447/task/31464/stat : 31464 (python) S 31406 31406 31236 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 748921381 332533760 6754 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 514 18446604445261574336 0 0 -1 6 0 0 0
[pid=31465] ppid=31406 vsize=335100 CPUtime=0.41 cores=0,2,4,6
/proc/31465/stat : 31465 (python) S 31406 31406 31236 0 -1 4202560 1129 0 0 0 23 18 0 0 19 0 2 0 748921385 343142400 6783 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/31465/statm: 83775 6783 400 364 0 19678 0
[pid=31467] ppid=31465 vsize=1018996 CPUtime=99.45 cores=0,2,4,6
/proc/31467/stat : 31467 (gnovelty+2) T 31465 31467 31467 0 -1 4202496 963 0 0 0 9945 0 0 0 25 0 1 0 748921385 1043451904 697 33554432000 134512640 135663704 4289481680 18446744073709551615 134544252 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/31467/statm: 254749 697 113 282 0 254465 0
[pid=31465/tid=31473] ppid=31406 vsize=335100 CPUtime=0 cores=0,2,4,6
/proc/31465/task/31473/stat : 31473 (python) S 31406 31406 31236 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 748931331 343142400 6783 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 514 18446604446990942592 0 0 -1 6 0 0 0
[pid=31474] ppid=31406 vsize=345468 CPUtime=0.4 cores=0,2,4,6
/proc/31474/stat : 31474 (python) S 31406 31406 31236 0 -1 4202560 1178 0 0 0 21 19 0 0 19 0 2 0 748931334 353759232 6819 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/31474/statm: 86367 6819 406 364 0 22267 0
[pid=31476] ppid=31474 vsize=251288 CPUtime=99.42 cores=0,2,4,6
/proc/31476/stat : 31476 (FH) T 31474 31476 31476 0 -1 4202496 1950 0 0 0 9942 0 0 0 25 0 1 0 748931335 257318912 1682 33554432000 134512640 134936988 4287303168 18446744073709551615 134554443 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/31476/statm: 62822 1682 41 104 0 62326 0
[pid=31474/tid=31482] ppid=31406 vsize=345468 CPUtime=0 cores=0,2,4,6
/proc/31474/task/31482/stat : 31482 (python) S 31406 31406 31236 0 -1 4202560 102 0 0 0 0 0 0 0 19 0 2 0 748941278 353759232 6819 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 514 18446604445474173120 0 0 -1 2 0 0 0
[pid=31483] ppid=31406 vsize=355732 CPUtime=0.4 cores=0,2,4,6
/proc/31483/stat : 31483 (python) S 31406 31406 31236 0 -1 4202560 1160 0 0 0 21 19 0 0 19 0 2 0 748941281 364269568 6851 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/31483/statm: 88933 6851 406 364 0 24830 0
[pid=31485] ppid=31483 vsize=421428 CPUtime=100.4 cores=0,2,4,6
/proc/31485/stat : 31485 (adaptg2wsat2009) T 31483 31485 31485 0 -1 4202496 2054 0 0 0 10040 0 0 0 25 0 1 0 748941282 431542272 1782 33554432000 134512640 135107411 4291407536 18446744073709551615 134532277 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/31485/statm: 105357 1782 59 146 0 104233 0
[pid=31483/tid=31488] ppid=31406 vsize=355732 CPUtime=0 cores=0,2,4,6
/proc/31483/task/31488/stat : 31488 (python) S 31406 31406 31236 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 748951322 364269568 6851 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 514 18446604452453078016 0 0 -1 6 0 0 0
[pid=31489] ppid=31406 vsize=365832 CPUtime=0.4 cores=0,2,4,6
/proc/31489/stat : 31489 (python) S 31406 31406 31236 0 -1 4202560 1144 0 0 0 22 18 0 0 19 0 2 0 748951326 374611968 6851 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/31489/statm: 91458 6851 400 364 0 27352 0
[pid=31491] ppid=31489 vsize=421428 CPUtime=98.41 cores=0,2,4,6
/proc/31491/stat : 31491 (adaptg2wsat2009) T 31489 31491 31491 0 -1 4202496 2054 0 0 0 9841 0 0 0 25 0 1 0 748951326 431542272 1782 33554432000 134512640 135107411 4291752656 18446744073709551615 134532288 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/31491/statm: 105357 1782 59 146 0 104233 0
[pid=31489/tid=31500] ppid=31406 vsize=365832 CPUtime=0 cores=0,2,4,6
/proc/31489/task/31500/stat : 31500 (python) S 31406 31406 31236 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 748961168 374611968 6851 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 514 18446604451486126400 0 0 -1 2 0 0 0
[pid=31501] ppid=31406 vsize=376356 CPUtime=0.43 cores=0,2,4,6
/proc/31501/stat : 31501 (python) S 31406 31406 31236 0 -1 4202560 1139 0 0 0 22 21 0 0 19 0 2 0 748961172 385388544 6855 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/31501/statm: 94089 6855 400 364 0 29980 0
[pid=31503] ppid=31501 vsize=251288 CPUtime=102.45 cores=0,2,4,6
/proc/31503/stat : 31503 (FH) T 31501 31503 31503 0 -1 4202496 1949 0 0 0 10245 0 0 0 25 0 1 0 748961172 257318912 1682 33554432000 134512640 134936988 4293688720 18446744073709551615 134545312 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/31503/statm: 62822 1682 41 104 0 62326 0
[pid=31501/tid=31506] ppid=31406 vsize=376356 CPUtime=0 cores=0,2,4,6
/proc/31501/task/31506/stat : 31506 (python) S 31406 31406 31236 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 748971418 385388544 6855 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 514 18446604442396742976 0 0 -1 6 0 0 0
[pid=31507] ppid=31406 vsize=386372 CPUtime=0.83 cores=0,2,4,6
/proc/31507/stat : 31507 (python) S 31406 31406 31236 0 -1 4202560 1179 0 0 0 44 39 0 0 19 0 2 0 748971422 395644928 6887 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/31507/statm: 96593 6887 400 364 0 32481 0
[pid=31509] ppid=31507 vsize=268420 CPUtime=195.91 cores=0,2,4,6
/proc/31509/stat : 31509 (hybridGM3) T 31507 31509 31509 0 -1 4202496 5815 0 0 0 19591 0 0 0 25 0 1 0 748971422 274862080 5543 33554432000 134512640 135204385 4289413824 18446744073709551615 134538475 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/31509/statm: 67105 5543 63 169 0 66935 0
[pid=31507/tid=31520] ppid=31406 vsize=386372 CPUtime=0 cores=0,2,4,6
/proc/31507/task/31520/stat : 31520 (python) S 31406 31406 31236 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 748991016 395644928 6887 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 514 18446604446429848768 0 0 -1 6 0 0 0
[pid=31521] ppid=31406 vsize=396928 CPUtime=0.42 cores=0,2,4,6
/proc/31521/stat : 31521 (python) S 31406 31406 31236 0 -1 4202560 1162 0 0 0 21 21 0 0 19 0 2 0 748991019 406454272 6891 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/31521/statm: 99232 6891 400 364 0 35117 0
[pid=31523] ppid=31521 vsize=1018996 CPUtime=100.46 cores=0,2,4,6
/proc/31523/stat : 31523 (gnovelty+2) T 31521 31523 31523 0 -1 4202496 975 0 0 0 10046 0 0 0 25 0 1 0 748991020 1043451904 698 33554432000 134512640 135663704 4292188304 18446744073709551615 134524897 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/31523/statm: 254749 698 113 282 0 254465 0
[pid=31521/tid=31530] ppid=31406 vsize=396928 CPUtime=0 cores=0,2,4,6
/proc/31521/task/31530/stat : 31530 (python) S 31406 31406 31236 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 749001066 406454272 6891 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 514 18446604442396742080 0 0 -1 6 0 0 0
[pid=31531] ppid=31406 vsize=406916 CPUtime=0.85 cores=0,2,4,6
/proc/31531/stat : 31531 (python) S 31406 31406 31236 0 -1 4202560 1175 0 0 0 45 40 0 0 19 0 2 0 749001070 416681984 6910 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/31531/statm: 101729 6910 400 364 0 37611 0
[pid=31533] ppid=31531 vsize=268424 CPUtime=197.9 cores=0,2,4,6
/proc/31533/stat : 31533 (hybridGM3) T 31531 31533 31533 0 -1 4202496 5817 0 0 0 19790 0 0 0 25 0 1 0 749001070 274866176 5545 33554432000 134512640 135204385 4288536224 18446744073709551615 134537726 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/31533/statm: 67106 5545 63 169 0 66936 0
[pid=31531/tid=31544] ppid=31406 vsize=406916 CPUtime=0 cores=0,2,4,6
/proc/31531/task/31544/stat : 31544 (python) S 31406 31406 31236 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 749020861 416681984 6910 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 514 18446604448358614336 0 0 -1 6 0 0 0
[pid=31545] ppid=31406 vsize=417500 CPUtime=0.85 cores=0,2,4,6
/proc/31545/stat : 31545 (python) S 31406 31406 31236 0 -1 4202560 1181 0 0 0 48 37 0 0 19 0 2 0 749020865 427520000 6914 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/31545/statm: 104375 6914 400 364 0 40254 0
[pid=31547] ppid=31545 vsize=268424 CPUtime=195.92 cores=0,2,4,6
/proc/31547/stat : 31547 (hybridGM3) T 31545 31547 31547 0 -1 4202496 5814 0 0 0 19592 0 0 0 25 0 1 0 749020865 274866176 5544 33554432000 134512640 135204385 4294518976 18446744073709551615 134538284 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/31547/statm: 67106 5544 63 169 0 66936 0
[pid=31545/tid=31556] ppid=31406 vsize=417500 CPUtime=0 cores=0,2,4,6
/proc/31545/task/31556/stat : 31556 (python) S 31406 31406 31236 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 749040458 427520000 6914 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 514 18446604448358613440 0 0 -1 6 0 0 0
[pid=31557] ppid=31406 vsize=427456 CPUtime=6.7 cores=0,2,4,6
/proc/31557/stat : 31557 (python) S 31406 31406 31236 0 -1 4202560 1236 0 0 0 375 295 0 0 19 0 2 0 749040462 437714944 6941 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/31557/statm: 106864 6941 408 364 0 42740 0
[pid=31559] ppid=31557 vsize=459764 CPUtime=1554.76 cores=0,2,4,6
/proc/31559/stat : 31559 (TNM) T 31557 31559 31559 0 -1 4202496 1925 0 0 0 155476 0 0 0 25 0 1 0 749040462 470798336 1652 33554432000 134512640 135008748 4293989504 18446744073709551615 134527524 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/31559/statm: 114941 1652 46 122 0 113841 0
[pid=31557/tid=31614] ppid=31406 vsize=427456 CPUtime=0 cores=0,2,4,6
/proc/31557/task/31614/stat : 31614 (python) S 31406 31406 31236 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 749195937 437714944 6941 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 514 18446604451261091968 0 0 -1 6 0 0 0
[pid=31615] ppid=31406 vsize=438072 CPUtime=0.44 cores=0,2,4,6
/proc/31615/stat : 31615 (python) S 31406 31406 31236 0 -1 4202560 1192 0 0 0 23 21 0 0 19 0 2 0 749195940 448585728 6944 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/31615/statm: 109518 6944 407 364 0 45391 0
[pid=31617] ppid=31615 vsize=421428 CPUtime=98.44 cores=0,2,4,6
/proc/31617/stat : 31617 (adaptg2wsat2009) T 31615 31617 31617 0 -1 4202496 2058 0 0 0 9844 0 0 0 25 0 1 0 749195940 431542272 1781 33554432000 134512640 135107411 4292107776 18446744073709551615 134531724 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/31617/statm: 105357 1781 59 146 0 104233 0
[pid=31615/tid=31620] ppid=31406 vsize=438072 CPUtime=0 cores=0,2,4,6
/proc/31615/task/31620/stat : 31620 (python) S 31406 31406 31236 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 749205785 448585728 6944 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 514 18446604451261092864 0 0 -1 6 0 0 0
[pid=31621] ppid=31406 vsize=448256 CPUtime=0.43 cores=0,2,4,6
/proc/31621/stat : 31621 (python) S 31406 31406 31236 0 -1 4202560 1167 0 0 0 22 21 0 0 19 0 2 0 749205787 459014144 6957 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/31621/statm: 112064 6957 401 364 0 47934 0
[pid=31623] ppid=31621 vsize=251288 CPUtime=98.44 cores=0,2,4,6
/proc/31623/stat : 31623 (FH) T 31621 31623 31623 0 -1 4202496 1955 0 0 0 9844 0 0 0 25 0 1 0 749205787 257318912 1682 33554432000 134512640 134936988 4291080208 18446744073709551615 134544737 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/31623/statm: 62822 1682 41 104 0 62326 0
[pid=31621/tid=31628] ppid=31406 vsize=448256 CPUtime=0 cores=0,2,4,6
/proc/31621/task/31628/stat : 31628 (python) S 31406 31406 31236 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 749215632 459014144 6957 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 514 18446604440321914944 0 0 -1 6 0 0 0
[pid=31629] ppid=31406 vsize=458644 CPUtime=0 cores=0,2,4,6
/proc/31629/stat : 31629 (python) S 31406 31406 31236 0 -1 4202560 844 0 0 0 0 0 0 0 19 0 1 0 749215634 469651456 6909 33554432000 4194304 5683644 140734812124048 18446744073709551615 224284945990 0 0 16781312 514 15157503366466109439 0 0 17 2 0 0 0
/proc/31629/statm: 114661 6909 350 364 0 50528 0
[pid=31631] ppid=31629 vsize=268424 CPUtime=0.45 cores=0,2,4,6
/proc/31631/stat : 31631 (hybridGM3) R 31629 31631 31631 0 -1 4202496 5821 0 0 0 45 0 0 0 21 0 1 0 749215635 274866176 5544 33554432000 134512640 135204385 4290953232 18446744073709551615 134660144 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/31631/statm: 67106 5544 63 169 0 66936 0
[pid=31406/tid=31413] ppid=31404 vsize=468224 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31413/stat : 31413 (python) S 31404 31406 31236 0 -1 4202560 31 936 0 0 0 0 0 0 21 0 19 0 748871463 479461376 8474 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 4 0 0 0
[pid=31406/tid=31423] ppid=31404 vsize=468224 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31423/stat : 31423 (python) S 31404 31406 31236 0 -1 4202560 30 936 0 0 0 0 0 0 20 0 19 0 748881310 479461376 8474 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 6 0 0 0
[pid=31406/tid=31433] ppid=31404 vsize=468224 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31433/stat : 31433 (python) S 31404 31406 31236 0 -1 4202560 28 936 0 0 0 0 0 0 20 0 19 0 748891258 479461376 8474 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 2 0 0 0
[pid=31406/tid=31442] ppid=31404 vsize=468224 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31442/stat : 31442 (python) S 31404 31406 31236 0 -1 4202560 24 936 0 0 0 0 0 0 20 0 19 0 748901300 479461376 8474 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 6 0 0 0
[pid=31406/tid=31448] ppid=31404 vsize=468224 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31448/stat : 31448 (python) S 31404 31406 31236 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 19 0 748911342 479461376 8474 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 2 0 0 0
[pid=31406/tid=31466] ppid=31404 vsize=468224 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31466/stat : 31466 (python) S 31404 31406 31236 0 -1 4202560 30 936 0 0 0 0 0 0 20 0 19 0 748921385 479461376 8474 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 2 0 0 0
[pid=31406/tid=31475] ppid=31404 vsize=468224 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31475/stat : 31475 (python) S 31404 31406 31236 0 -1 4202560 30 936 0 0 0 0 0 0 20 0 19 0 748931335 479461376 8474 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 6 0 0 0
[pid=31406/tid=31484] ppid=31404 vsize=468224 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31484/stat : 31484 (python) S 31404 31406 31236 0 -1 4202560 30 936 0 0 0 0 0 0 20 0 19 0 748941282 479461376 8474 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 6 0 0 0
[pid=31406/tid=31490] ppid=31404 vsize=468224 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31490/stat : 31490 (python) S 31404 31406 31236 0 -1 4202560 26 936 0 0 0 0 0 0 20 0 19 0 748951326 479461376 8474 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 6 0 0 0
[pid=31406/tid=31502] ppid=31404 vsize=468224 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31502/stat : 31502 (python) S 31404 31406 31236 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 19 0 748961172 479461376 8474 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 2 0 0 0
[pid=31406/tid=31508] ppid=31404 vsize=468224 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31508/stat : 31508 (python) S 31404 31406 31236 0 -1 4202560 26 936 0 0 0 0 0 0 20 0 19 0 748971422 479461376 8474 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 2 0 0 0
[pid=31406/tid=31522] ppid=31404 vsize=468224 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31522/stat : 31522 (python) S 31404 31406 31236 0 -1 4202560 31 936 0 0 0 0 0 0 20 0 19 0 748991020 479461376 8474 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 2 0 0 0
[pid=31406/tid=31532] ppid=31404 vsize=468224 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31532/stat : 31532 (python) S 31404 31406 31236 0 -1 4202560 28 936 0 0 0 0 0 0 20 0 19 0 749001070 479461376 8474 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 2 0 0 0
[pid=31406/tid=31546] ppid=31404 vsize=468224 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31546/stat : 31546 (python) S 31404 31406 31236 0 -1 4202560 32 936 0 0 0 0 0 0 20 0 19 0 749020865 479461376 8474 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 6 0 0 0
[pid=31406/tid=31558] ppid=31404 vsize=468224 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31558/stat : 31558 (python) S 31404 31406 31236 0 -1 4202560 28 936 0 0 0 0 0 0 20 0 19 0 749040462 479461376 8474 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 2 0 0 0
[pid=31406/tid=31616] ppid=31404 vsize=468224 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31616/stat : 31616 (python) S 31404 31406 31236 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 19 0 749195940 479461376 8474 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 2 0 0 0
[pid=31406/tid=31622] ppid=31404 vsize=468224 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31622/stat : 31622 (python) S 31404 31406 31236 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 19 0 749205787 479461376 8474 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 2 0 0 0
[pid=31406/tid=31630] ppid=31404 vsize=468224 CPUtime=0 cores=0,2,4,6
/proc/31406/task/31630/stat : 31630 (python) S 31404 31406 31236 0 -1 4202560 28 936 0 0 0 0 0 0 20 0 19 0 749215635 479461376 8474 33554432000 4194304 5683644 140734812124048 18446744073709551615 224296749489 0 0 16781312 2 18446604447627130112 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3458.06
Current children cumulated vsize (KiB) 16049352

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 31406 and gives
#  childrusage.ru_utime.tv_sec=2
#  childrusage.ru_utime.tv_usec=230660
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=133979
# CPU time returned by wait4() is 2.36464
# while last known CPU time is 3458.06
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 3444.14
CPU time (s): 3458.06
CPU user time (s): 3451.37
CPU system time (s): 6.69
CPU usage (%): 100.404
Max. virtual memory (cumulated for all children) (KiB): 16049352

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2.23066
system time used= 0.133979
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 31044
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1062
involuntary context switches= 53

runsolver used 38.4652 second user time and 65.3741 second system time

The end

Launcher Data 

Begin job on node118 at 2011-05-17 00:53:26
IDJOB=3386546
IDBENCH=82611
IDSOLVER=1872
FILE ID=node118/3386546-1305586406
RUNJOBID= node118-1305586147-31254
PBS_JOBID= 13350360
Free space on /tmp= 73108 MiB

SOLVER NAME= borg-sat 11.03.02-seq-random (fixed)
BENCH NAME= SAT11/random/large/unif-k5-r20-v1500-c30000-S969405384-096-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3386546-1305586406/watcher-3386546-1305586406 -o /tmp/evaluation-result-3386546-1305586406/solver-3386546-1305586406 -C 5000 -W 5100 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386546-1305586406.cnf -seed 1302764447 -budget 5000

TIME LIMIT= 5000 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 6e345981a96012731da9bd2ce5420289
RANDOM SEED=1302764447

node118.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5374.59
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      15831120 kB
Buffers:        322644 kB
Cached:        5016256 kB
SwapCached:      22068 kB
Active:       14031620 kB
Inactive:      2746760 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      15831120 kB
SwapTotal:    67111528 kB
SwapFree:     45193380 kB
Dirty:           67948 kB
Writeback:           0 kB
AnonPages:    11430596 kB
Mapped:          33080 kB
Slab:           161804 kB
PageTables:     110640 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 163048536 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73076 MiB
End job on node118 at 2011-05-17 01:50:52