Trace number 3317190

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (MO) 4733.06 1193.54

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v30000-c126000-S643238994-057-UNKNOWN.cnf
MD5SUM2d92a52914298e3c91073398b38892d9
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark265.737
Satisfiable
(Un)Satisfiability was proved
Number of variables30000
Number of clauses126000
Sum of the clauses size378000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3126000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.30/0.46	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.30/0.46	c INFO: solving HOME/instance-3317190-1304268034.cnf
0.30/0.46	c INFO: computing task features
0.79/0.91	c INFO: computed variable-clause graph statistics
0.79/0.93	c INFO: computed clause balance statistics
2.99/3.14	c INFO: computed variable balance statistics
4.69/4.85	c INFO: computed small-clause counts
4.69/4.87	c INFO: computed Horn-clause counts
5.89/6.03	c INFO: computed variable graph statistics
5.89/6.03	c INFO: collected features for HOME/instance-3317190-1304268034.cnf in 5.56 s
5.89/6.06	c INFO: running hybridGM3@0 for 97 with 4794 remaining (b = 0.38)
5.89/6.09	c INFO: running hybridGM3@0 for 97 with 4794 remaining (b = 0.23)
5.99/6.12	c INFO: running hybridGM3@0 for 97 with 4794 remaining (b = 0.15)
5.99/6.15	c INFO: running hybridGM3@0 for 97 with 4794 remaining (b = 0.11)
404.84/106.07	c INFO: running hybridGM3@0 for 97 with 4694 remaining (b = 0.09)
405.20/106.10	c INFO: running hybridGM3@0 for 97 with 4694 remaining (b = 0.09)
409.19/107.12	c INFO: running hybridGM3@0 for 97 with 4693 remaining (b = 0.09)
409.19/107.16	c INFO: running hybridGM3@99 for 97 with 4693 remaining (b = 0.09)
789.00/202.27	c INFO: running hybridGM3@100 for 97 with 4598 remaining (b = 0.09)
789.00/202.30	c INFO: running hybridGM3@0 for 194 with 4598 remaining (b = 0.16)
804.93/206.23	c INFO: running TNM@0 for 97 with 4594 remaining (b = 0.07)
804.93/206.27	c INFO: running hybridGM3@99 for 97 with 4594 remaining (b = 0.06)
1181.01/300.44	c INFO: running hybridGM3@99 for 97 with 4500 remaining (b = 0.06)
1192.98/303.43	c INFO: running hybridGM3@0 for 194 with 4497 remaining (b = 0.11)
1204.94/306.41	c INFO: running hybridGM3@0 for 194 with 4494 remaining (b = 0.08)
1568.23/397.62	c INFO: running TNM@0 for 97 with 4402 remaining (b = 0.03)
1575.37/399.49	c INFO: running hybridGM3@100 for 97 with 4401 remaining (b = 0.03)
1967.34/497.82	c INFO: running hybridGM3@0 for 194 with 4302 remaining (b = 0.06)
1974.50/499.65	c INFO: running hybridGM3@99 for 97 with 4300 remaining (b = 0.03)
1978.46/500.68	c INFO: running hybridGM3@0 for 194 with 4299 remaining (b = 0.05)
1990.38/503.62	c INFO: running hybridGM3@96 for 97 with 4296 remaining (b = 0.02)
2369.50/598.80	c INFO: running hybridGM3@191 for 97 with 4201 remaining (b = 0.02)
2373.49/599.85	c INFO: running TNM@0 for 97 with 4200 remaining (b = 0.02)
2763.75/697.89	c INFO: running TNM@0 for 97 with 4102 remaining (b = 0.02)
2772.08/700.00	c INFO: running hybridGM3@291 for 97 with 4100 remaining (b = 0.02)
2772.50/700.05	c INFO: running TNM@0 for 97 with 4100 remaining (b = 0.01)
2772.50/700.09	c INFO: running TNM@0 for 97 with 4100 remaining (b = 0.01)
3150.85/795.20	c INFO: running hybridGM3@0 for 194 with 4005 remaining (b = 0.03)
3162.75/798.10	c INFO: running TNM@0 for 97 with 4002 remaining (b = 0.01)
3171.07/800.27	c INFO: running TNM@195 for 1843 with 4000 remaining (b = 0.20)
3171.45/800.33	c INFO: running hybridGM3@0 for 194 with 4000 remaining (b = 0.02)
3560.76/898.32	c INFO: running hybridGM3@0 for 194 with 3902 remaining (b = 0.02)
3934.54/992.44	c INFO: running hybridGM3@98 for 97 with 3808 remaining (b = 0.01)
3954.81/997.57	c INFO: running hybridGM3@100 for 97 with 3802 remaining (b = 0.01)
4312.72/1087.64	c INFO: running hybridGM3@295 for 97 with 3712 remaining (b = 0.01)
4332.98/1092.78	c INFO: running adaptg2wsat2009++@0 for 97 with 3707 remaining (b = 0.01)
4344.14/1095.57	c INFO: running TNM@196 for 1843 with 3704 remaining (b = 0.14)
4714.74/1188.86	c INFO: running hybridGM3@389 for 97 with 3611 remaining (b = 0.01)
4731.43/1193.01	c INFO: running hybridGM3@0 for 194 with 3607 remaining (b = 0.02)
4733.06/1193.53	c INFO: running adaptg2wsat2009++@0 for 194 with 3606 remaining (b = 0.02)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317190-1304268034/watcher-3317190-1304268034 -o /tmp/evaluation-result-3317190-1304268034/solver-3317190-1304268034 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317190-1304268034.cnf -seed 886541200 -budget 4800 -cores 4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.21 7.41 4.38 5/244 22548
/proc/meminfo: memFree=29966328/32951124 swapFree=67111524/67111528
[pid=22548] ppid=22546 vsize=15764 CPUtime=0 cores=0,2,4,6
/proc/22548/stat : 22548 (python) S 22546 22548 22195 0 -1 4202496 846 0 0 0 0 0 0 0 17 0 1 0 270994733 16142336 728 33554432000 4194304 5683644 140737016005648 18446744073709551615 268660660224 0 2147483391 16781312 2 18446744071699199166 0 0 17 4 0 0 0
/proc/22548/statm: 3941 728 334 364 0 428 0

[startup+0.02086 s]
/proc/loadavg: 8.21 7.41 4.38 5/244 22548
/proc/meminfo: memFree=29966328/32951124 swapFree=67111524/67111528
[pid=22548] ppid=22546 vsize=16396 CPUtime=0.01 cores=0,2,4,6
/proc/22548/stat : 22548 (python) R 22546 22548 22195 0 -1 4202496 1118 0 0 0 1 0 0 0 17 0 1 0 270994733 16789504 985 33554432000 4194304 5683644 140737016005648 18446744073709551615 268660660224 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/22548/statm: 4099 985 409 364 0 586 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16396

[startup+0.100855 s]
/proc/loadavg: 8.21 7.41 4.38 5/244 22548
/proc/meminfo: memFree=29966328/32951124 swapFree=67111524/67111528
[pid=22548] ppid=22546 vsize=104232 CPUtime=0.07 cores=0,2,4,6
/proc/22548/stat : 22548 (python) R 22546 22548 22195 0 -1 4202496 3088 0 0 0 5 2 0 0 17 0 1 0 270994733 106733568 2768 33554432000 4194304 5683644 140737016005648 18446744073709551615 5116756 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/22548/statm: 26058 2768 847 364 0 1806 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 104232

[startup+0.300809 s]
/proc/loadavg: 8.21 7.41 4.38 5/244 22548
/proc/meminfo: memFree=29966328/32951124 swapFree=67111524/67111528
[pid=22548] ppid=22546 vsize=187688 CPUtime=0.22 cores=0,2,4,6
/proc/22548/stat : 22548 (python) R 22546 22548 22195 0 -1 4202496 6829 933 1 0 17 5 0 0 17 0 1 0 270994733 192192512 4884 33554432000 4194304 5683644 140737016005648 18446744073709551615 268660660224 0 2147483391 16781312 2 0 0 0 17 4 0 0 0
/proc/22548/statm: 46922 4884 1399 364 0 3723 0
Current children cumulated CPU time (s) 0.22
Current children cumulated vsize (KiB) 187688

[startup+0.700742 s]
/proc/loadavg: 8.21 7.41 4.38 5/244 22548
/proc/meminfo: memFree=29966328/32951124 swapFree=67111524/67111528
[pid=22548] ppid=22546 vsize=294652 CPUtime=0.59 cores=0,2,4,6
/proc/22548/stat : 22548 (python) R 22546 22548 22195 0 -1 4202496 13259 933 1 0 52 7 0 0 17 0 1 0 270994733 301723648 10940 33554432000 4194304 5683644 140737016005648 18446744073709551615 268660323013 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/22548/statm: 73663 10940 1761 364 0 9587 0
Current children cumulated CPU time (s) 0.59
Current children cumulated vsize (KiB) 294652

[startup+1.5006 s]
/proc/loadavg: 7.95 7.37 4.38 6/245 22553
/proc/meminfo: memFree=29914736/32951124 swapFree=67111524/67111528
[pid=22548] ppid=22546 vsize=307972 CPUtime=1.39 cores=0,2,4,6
/proc/22548/stat : 22548 (python) R 22546 22548 22195 0 -1 4202496 20142 933 1 0 131 8 0 0 19 0 1 0 270994733 315363328 14378 33554432000 4194304 5683644 140737016005648 18446744073709551615 4539699 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/22548/statm: 76993 14378 1826 364 0 12917 0
Current children cumulated CPU time (s) 1.39
Current children cumulated vsize (KiB) 307972

[startup+3.10031 s]
/proc/loadavg: 7.95 7.37 4.38 6/245 22554
/proc/meminfo: memFree=29914728/32951124 swapFree=67111524/67111528
[pid=22548] ppid=22546 vsize=307972 CPUtime=2.99 cores=0,2,4,6
/proc/22548/stat : 22548 (python) R 22546 22548 22195 0 -1 4202496 20142 933 1 0 291 8 0 0 22 0 1 0 270994733 315363328 14378 33554432000 4194304 5683644 140737016005648 18446744073709551615 47625474270465 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/22548/statm: 76993 14378 1826 364 0 12917 0
Current children cumulated CPU time (s) 2.99
Current children cumulated vsize (KiB) 307972

[startup+6.30071 s]
/proc/loadavg: 7.80 7.35 4.39 6/245 22554
/proc/meminfo: memFree=29913860/32951124 swapFree=67111524/67111528
[pid=22548] ppid=22546 vsize=346660 CPUtime=6.05 cores=0,2,4,6
/proc/22548/stat : 22548 (python) S 22546 22548 22195 0 -1 4202496 23099 933 1 0 596 9 0 0 25 0 5 0 270994733 354979840 13862 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672488907 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/22548/statm: 86665 13862 1881 364 0 22574 0
Current children cumulated CPU time (s) 6.05
Current children cumulated vsize (KiB) 346660

[startup+12.7006 s]
/proc/loadavg: 7.81 7.36 4.41 9/257 22566
/proc/meminfo: memFree=29782572/32951124 swapFree=67111524/67111528
[pid=22548] ppid=22546 vsize=346660 CPUtime=6.05 cores=0,2,4,6
/proc/22548/stat : 22548 (python) S 22546 22548 22195 0 -1 4202496 23099 933 1 0 596 9 0 0 25 0 5 0 270994733 354979840 13862 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672488907 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/22548/statm: 86665 13862 1881 364 0 22574 0
[pid=22555] ppid=22548 vsize=305648 CPUtime=0 cores=0,2,4,6
/proc/22555/stat : 22555 (python) S 22548 22548 22195 0 -1 4202560 894 0 0 0 0 0 0 0 18 0 1 0 270995340 312983552 12341 33554432000 4194304 5683644 140737016005648 18446744073709551615 268660682255 0 0 16781312 514 4317606244710842628 0 0 17 6 0 0 0
/proc/22555/statm: 76412 12341 376 364 0 12330 0
[pid=22557] ppid=22555 vsize=268424 CPUtime=6.62 cores=0,2,4,6
/proc/22557/stat : 22557 (hybridGM3) R 22555 22557 22557 0 -1 4202496 7611 0 0 0 661 1 0 0 25 0 1 0 270995340 274866176 7346 33554432000 134512640 135204385 4287576800 18446744073709551615 134536744 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/22557/statm: 67106 7346 64 169 0 66936 0
[pid=22558] ppid=22548 vsize=315904 CPUtime=0 cores=0,2,4,6
/proc/22558/stat : 22558 (python) S 22548 22548 22195 0 -1 4202560 907 0 0 0 0 0 0 0 18 0 1 0 270995343 323485696 12339 33554432000 4194304 5683644 140737016005648 18446744073709551615 268660682310 0 0 16781312 514 4317602396420145412 0 0 17 6 0 0 0
/proc/22558/statm: 78976 12339 370 364 0 14891 0
[pid=22560] ppid=22558 vsize=268424 CPUtime=6.59 cores=0,2,4,6
/proc/22560/stat : 22560 (hybridGM3) R 22558 22560 22560 0 -1 4202496 7605 0 0 0 658 1 0 0 25 0 1 0 270995343 274866176 7344 33554432000 134512640 135204385 4294265344 18446744073709551615 134537403 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/22560/statm: 67106 7344 64 169 0 66936 0
[pid=22561] ppid=22548 vsize=326160 CPUtime=0 cores=0,2,4,6
/proc/22561/stat : 22561 (python) S 22548 22548 22195 0 -1 4202560 907 0 0 0 0 0 0 0 18 0 1 0 270995346 333987840 12343 33554432000 4194304 5683644 140737016005648 18446744073709551615 268660682310 0 0 16781312 514 4357703234752577796 0 0 17 6 0 0 0
/proc/22561/statm: 81540 12343 370 364 0 17452 0
[pid=22563] ppid=22561 vsize=268420 CPUtime=6.48 cores=0,2,4,6
/proc/22563/stat : 22563 (hybridGM3) R 22561 22563 22563 0 -1 4202496 7608 0 0 0 648 0 0 0 25 0 1 0 270995346 274862080 7345 33554432000 134512640 135204385 4292477040 18446744073709551615 134536737 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/22563/statm: 67105 7345 64 169 0 66935 0
[pid=22564] ppid=22548 vsize=336416 CPUtime=0 cores=0,2,4,6
/proc/22564/stat : 22564 (python) S 22548 22548 22195 0 -1 4202560 930 0 0 0 0 0 0 0 18 0 1 0 270995349 344489984 12347 33554432000 4194304 5683644 140737016005648 18446744073709551615 268660682310 0 0 16781312 514 4357695538171183364 0 0 17 4 0 0 0
/proc/22564/statm: 84104 12347 370 364 0 20013 0
[pid=22566] ppid=22564 vsize=268420 CPUtime=6.52 cores=0,2,4,6
/proc/22566/stat : 22566 (hybridGM3) R 22564 22566 22566 0 -1 4202496 7606 0 0 0 652 0 0 0 25 0 1 0 270995349 274862080 7343 33554432000 134512640 135204385 4292763680 18446744073709551615 134546153 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/22566/statm: 67105 7343 64 169 0 66935 0
[pid=22548/tid=22556] ppid=22546 vsize=346660 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22556/stat : 22556 (python) S 22546 22548 22195 0 -1 4202560 27 933 0 0 0 0 0 0 25 0 5 0 270995340 354979840 13862 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 6 0 0 0
[pid=22548/tid=22559] ppid=22546 vsize=346660 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22559/stat : 22559 (python) S 22546 22548 22195 0 -1 4202560 24 933 0 0 0 0 0 0 25 0 5 0 270995343 354979840 13862 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 6 0 0 0
[pid=22548/tid=22562] ppid=22546 vsize=346660 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22562/stat : 22562 (python) S 22546 22548 22195 0 -1 4202560 23 933 0 0 0 0 0 0 25 0 5 0 270995346 354979840 13862 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 6 0 0 0
[pid=22548/tid=22565] ppid=22546 vsize=346660 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22565/stat : 22565 (python) S 22546 22548 22195 0 -1 4202560 29 933 0 0 0 0 0 0 25 0 5 0 270995349 354979840 13862 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 32.26
Current children cumulated vsize (KiB) 2704476

[startup+25.5013 s]
/proc/loadavg: 7.85 7.39 4.47 9/257 22566
/proc/meminfo: memFree=29783192/32951124 swapFree=67111524/67111528
[pid=22548] ppid=22546 vsize=346660 CPUtime=6.05 cores=0,2,4,6
/proc/22548/stat : 22548 (python) S 22546 22548 22195 0 -1 4202496 23099 933 1 0 596 9 0 0 25 0 5 0 270994733 354979840 13862 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672488907 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/22548/statm: 86665 13862 1881 364 0 22574 0
[pid=22555] ppid=22548 vsize=305648 CPUtime=0.02 cores=0,2,4,6
/proc/22555/stat : 22555 (python) S 22548 22548 22195 0 -1 4202560 902 0 0 0 1 1 0 0 18 0 1 0 270995340 312983552 12341 33554432000 4194304 5683644 140737016005648 18446744073709551615 268660682255 0 0 16781312 514 12136321386461233412 0 0 17 6 0 0 0
/proc/22555/statm: 76412 12341 376 364 0 12330 0
[pid=22557] ppid=22555 vsize=268424 CPUtime=19.42 cores=0,2,4,6
/proc/22557/stat : 22557 (hybridGM3) R 22555 22557 22557 0 -1 4202496 7611 0 0 0 1941 1 0 0 25 0 1 0 270995340 274866176 7346 33554432000 134512640 135204385 4287576800 18446744073709551615 134537363 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/22557/statm: 67106 7346 64 169 0 66936 0
[pid=22558] ppid=22548 vsize=315904 CPUtime=0.02 cores=0,2,4,6
/proc/22558/stat : 22558 (python) S 22548 22548 22195 0 -1 4202560 913 0 0 0 1 1 0 0 18 0 1 0 270995343 323485696 12339 33554432000 4194304 5683644 140737016005648 18446744073709551615 268660682310 0 0 16781312 514 12136321386461233412 0 0 17 6 0 0 0
/proc/22558/statm: 78976 12339 370 364 0 14891 0
[pid=22560] ppid=22558 vsize=268424 CPUtime=19.39 cores=0,2,4,6
/proc/22560/stat : 22560 (hybridGM3) R 22558 22560 22560 0 -1 4202496 7605 0 0 0 1938 1 0 0 25 0 1 0 270995343 274866176 7344 33554432000 134512640 135204385 4294265344 18446744073709551615 134546086 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/22560/statm: 67106 7344 64 169 0 66936 0
[pid=22561] ppid=22548 vsize=326160 CPUtime=0.02 cores=0,2,4,6
/proc/22561/stat : 22561 (python) S 22548 22548 22195 0 -1 4202560 913 0 0 0 1 1 0 0 18 0 1 0 270995346 333987840 12343 33554432000 4194304 5683644 140737016005648 18446744073709551615 268660682310 0 0 16781312 514 12136321386461233412 0 0 17 6 0 0 0
/proc/22561/statm: 81540 12343 370 364 0 17452 0
[pid=22563] ppid=22561 vsize=268420 CPUtime=19.14 cores=0,2,4,6
/proc/22563/stat : 22563 (hybridGM3) R 22561 22563 22563 0 -1 4202496 7608 0 0 0 1914 0 0 0 25 0 1 0 270995346 274862080 7345 33554432000 134512640 135204385 4292477040 18446744073709551615 134537383 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/22563/statm: 67105 7345 64 169 0 66935 0
[pid=22564] ppid=22548 vsize=336416 CPUtime=0.01 cores=0,2,4,6
/proc/22564/stat : 22564 (python) S 22548 22548 22195 0 -1 4202560 934 0 0 0 1 0 0 0 18 0 1 0 270995349 344489984 12347 33554432000 4194304 5683644 140737016005648 18446744073709551615 268660682310 0 0 16781312 514 12136321386461233412 0 0 17 4 0 0 0
/proc/22564/statm: 84104 12347 370 364 0 20013 0
[pid=22566] ppid=22564 vsize=268420 CPUtime=19.3 cores=0,2,4,6
/proc/22566/stat : 22566 (hybridGM3) R 22564 22566 22566 0 -1 4202496 7606 0 0 0 1930 0 0 0 25 0 1 0 270995349 274862080 7343 33554432000 134512640 135204385 4292763680 18446744073709551615 134546082 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/22566/statm: 67105 7343 64 169 0 66935 0
[pid=22548/tid=22556] ppid=22546 vsize=346660 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22556/stat : 22556 (python) S 22546 22548 22195 0 -1 4202560 27 933 0 0 0 0 0 0 25 0 5 0 270995340 354979840 13862 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 6 0 0 0
[pid=22548/tid=22559] ppid=22546 vsize=346660 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22559/stat : 22559 (python) S 22546 22548 22195 0 -1 4202560 24 933 0 0 0 0 0 0 25 0 5 0 270995343 354979840 13862 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 6 0 0 0
[pid=22548/tid=22562] ppid=22546 vsize=346660 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22562/stat : 22562 (python) S 22546 22548 22195 0 -1 4202560 23 933 0 0 0 0 0 0 25 0 5 0 270995346 354979840 13862 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 6 0 0 0
[pid=22548/tid=22565] ppid=22546 vsize=346660 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22565/stat : 22565 (python) S 22546 22548 22195 0 -1 4202560 29 933 0 0 0 0 0 0 25 0 5 0 270995349 354979840 13862 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 83.37
Current children cumulated vsize (KiB) 2704476

[startup+51.1008 s]
/proc/loadavg: 7.97 7.45 4.56 9/257 22567
/proc/meminfo: memFree=29782836/32951124 swapFree=67111524/67111528
[pid=22548] ppid=22546 vsize=346660 CPUtime=6.05 cores=0,2,4,6
/proc/22548/stat : 22548 (python) S 22546 22548 22195 0 -1 4202496 23099 933 1 0 596 9 0 0 25 0 5 0 270994733 354979840 13862 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672488907 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/22548/statm: 86665 13862 1881 364 0 22574 0
[pid=22555] ppid=22548 vsize=305648 CPUtime=0.05 cores=0,2,4,6
/proc/22555/stat : 22555 (python) S 22548 22548 22195 0 -1 4202560 908 0 0 0 3 2 0 0 18 0 1 0 270995340 312983552 12341 33554432000 4194304 5683644 140737016005648 18446744073709551615 268660682255 0 0 16781312 514 12136321386461233412 0 0 17 6 0 0 0
/proc/22555/statm: 76412 12341 376 364 0 12330 0
[pid=22557] ppid=22555 vsize=268424 CPUtime=45.02 cores=0,2,4,6
/proc/22557/stat : 22557 (hybridGM3) R 22555 22557 22557 0 -1 4202496 7611 0 0 0 4501 1 0 0 25 0 1 0 270995340 274866176 7346 33554432000 134512640 135204385 4287576800 18446744073709551615 134546059 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/22557/statm: 67106 7346 64 169 0 66936 0
[pid=22558] ppid=22548 vsize=315904 CPUtime=0.05 cores=0,2,4,6
/proc/22558/stat : 22558 (python) S 22548 22548 22195 0 -1 4202560 923 0 0 0 2 3 0 0 18 0 1 0 270995343 323485696 12339 33554432000 4194304 5683644 140737016005648 18446744073709551615 268660682310 0 0 16781312 514 12136321386461233412 0 0 17 6 0 0 0
/proc/22558/statm: 78976 12339 370 364 0 14891 0
[pid=22560] ppid=22558 vsize=268424 CPUtime=44.98 cores=0,2,4,6
/proc/22560/stat : 22560 (hybridGM3) R 22558 22560 22560 0 -1 4202496 7609 0 0 0 4497 1 0 0 25 0 1 0 270995343 274866176 7348 33554432000 134512640 135204385 4294265344 18446744073709551615 134536646 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/22560/statm: 67106 7348 64 169 0 66936 0
[pid=22561] ppid=22548 vsize=326160 CPUtime=0.05 cores=0,2,4,6
/proc/22561/stat : 22561 (python) S 22548 22548 22195 0 -1 4202560 924 0 0 0 3 2 0 0 18 0 1 0 270995346 333987840 12343 33554432000 4194304 5683644 140737016005648 18446744073709551615 268660682310 0 0 16781312 514 12136321386461233412 0 0 17 6 0 0 0
/proc/22561/statm: 81540 12343 370 364 0 17452 0
[pid=22563] ppid=22561 vsize=268420 CPUtime=44.48 cores=0,2,4,6
/proc/22563/stat : 22563 (hybridGM3) R 22561 22563 22563 0 -1 4202496 7608 0 0 0 4447 1 0 0 25 0 1 0 270995346 274862080 7345 33554432000 134512640 135204385 4292477040 18446744073709551615 134537412 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/22563/statm: 67105 7345 64 169 0 66935 0
[pid=22564] ppid=22548 vsize=336416 CPUtime=0.06 cores=0,2,4,6
/proc/22564/stat : 22564 (python) S 22548 22548 22195 0 -1 4202560 941 0 0 0 4 2 0 0 18 0 1 0 270995349 344489984 12347 33554432000 4194304 5683644 140737016005648 18446744073709551615 268660682310 0 0 16781312 514 12136321386461233412 0 0 17 4 0 0 0
/proc/22564/statm: 84104 12347 370 364 0 20013 0

################
# More data... #
################

[pid=22672] ppid=22670 vsize=268420 CPUtime=488.08 cores=0,2,4,6
/proc/22672/stat : 22672 (hybridGM3) T 22670 22672 22672 0 -1 4202496 7614 0 0 0 48804 4 0 0 25 0 1 0 271025375 274862080 7345 33554432000 134512640 135204385 4294782352 18446744073709551615 134536986 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/22672/statm: 67105 7345 64 169 0 66935 0
[pid=22670/tid=22706] ppid=22548 vsize=377440 CPUtime=0 cores=0,2,4,6
/proc/22670/task/22706/stat : 22706 (python) S 22548 22548 22195 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 271045092 386498560 12399 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 514 18446604453927368704 0 0 -1 0 0 0 0
[pid=22687] ppid=22548 vsize=387696 CPUtime=0.88 cores=0,2,4,6
/proc/22687/stat : 22687 (python) S 22548 22548 22195 0 -1 4202560 1192 0 0 0 49 39 0 0 18 0 2 0 271034495 397000704 12397 33554432000 4194304 5683644 140737016005648 18446744073709551615 268660682310 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/22687/statm: 96924 12397 399 364 0 32818 0
[pid=22689] ppid=22687 vsize=463080 CPUtime=480.71 cores=0,2,4,6
/proc/22689/stat : 22689 (TNM) R 22687 22689 22689 0 -1 4202496 5207 0 0 0 48070 1 0 0 25 0 1 0 271034496 474193920 4941 33554432000 134512640 135008748 4290656032 18446744073709551615 134529076 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/22689/statm: 115770 4941 46 122 0 114670 0
[pid=22687/tid=22698] ppid=22548 vsize=387696 CPUtime=0 cores=0,2,4,6
/proc/22687/task/22698/stat : 22698 (python) S 22548 22548 22195 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 271044512 397000704 12397 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 514 18446604453963334720 0 0 -1 6 0 0 0
[pid=22699] ppid=22548 vsize=397952 CPUtime=0.33 cores=0,2,4,6
/proc/22699/stat : 22699 (python) S 22548 22548 22195 0 -1 4202560 1178 0 0 0 17 16 0 0 19 0 2 0 271044515 407502848 12401 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672488907 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/22699/statm: 99488 12401 399 364 0 35379 0
[pid=22701] ppid=22699 vsize=268420 CPUtime=198.54 cores=0,2,4,6
/proc/22701/stat : 22701 (hybridGM3) T 22699 22701 22701 0 -1 4202496 7616 0 0 0 19853 1 0 0 25 0 1 0 271044516 274862080 7348 33554432000 134512640 135204385 4291693728 18446744073709551615 134537231 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/22701/statm: 67105 7348 64 169 0 66935 0
[pid=22699/tid=22725] ppid=22548 vsize=397952 CPUtime=0 cores=0,2,4,6
/proc/22699/task/22725/stat : 22725 (python) S 22548 22548 22195 0 -1 4202560 52 0 0 0 0 0 0 0 19 0 2 0 271064734 407502848 12401 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 514 18446604453938038976 0 0 -1 6 0 0 0
[pid=22703] ppid=22548 vsize=408208 CPUtime=0.32 cores=0,2,4,6
/proc/22703/stat : 22703 (python) S 22548 22548 22195 0 -1 4202560 1201 0 0 0 17 15 0 0 19 0 2 0 271044801 418004992 12405 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672488907 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/22703/statm: 102052 12405 399 364 0 37940 0
[pid=22705] ppid=22703 vsize=268424 CPUtime=196.65 cores=0,2,4,6
/proc/22705/stat : 22705 (hybridGM3) T 22703 22705 22705 0 -1 4202496 7617 0 0 0 19663 2 0 0 25 0 1 0 271044802 274866176 7350 33554432000 134512640 135204385 4293215072 18446744073709551615 134536835 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/22705/statm: 67106 7350 64 169 0 66936 0
[pid=22703/tid=22721] ppid=22548 vsize=408208 CPUtime=0 cores=0,2,4,6
/proc/22703/task/22721/stat : 22721 (python) S 22548 22548 22195 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 271064518 418004992 12405 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 514 18446604453938038080 0 0 -1 4 0 0 0
[pid=22712] ppid=22548 vsize=418464 CPUtime=0.35 cores=0,2,4,6
/proc/22712/stat : 22712 (python) S 22548 22548 22195 0 -1 4202560 1198 0 0 0 18 17 0 0 18 0 2 0 271054718 428507136 12409 33554432000 4194304 5683644 140737016005648 18446744073709551615 268660682310 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/22712/statm: 104616 12409 399 364 0 40501 0
[pid=22714] ppid=22712 vsize=463080 CPUtime=197.63 cores=0,2,4,6
/proc/22714/stat : 22714 (TNM) R 22712 22714 22714 0 -1 4202496 5215 0 0 0 19763 0 0 0 25 0 1 0 271054719 474193920 4942 33554432000 134512640 135008748 4294196144 18446744073709551615 134527795 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/22714/statm: 115770 4942 46 122 0 114670 0
[pid=22712/tid=22726] ppid=22548 vsize=418464 CPUtime=0 cores=0,2,4,6
/proc/22712/task/22726/stat : 22726 (python) S 22548 22548 22195 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 271064734 428507136 12409 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 514 18446604453934339904 0 0 -1 2 0 0 0
[pid=22722] ppid=22548 vsize=428720 CPUtime=0.18 cores=0,2,4,6
/proc/22722/stat : 22722 (python) S 22548 22548 22195 0 -1 4202560 1175 0 0 0 9 9 0 0 19 0 2 0 271064522 439009280 12413 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672488907 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/22722/statm: 107180 12413 399 364 0 43062 0
[pid=22724] ppid=22722 vsize=463080 CPUtime=98.13 cores=0,2,4,6
/proc/22724/stat : 22724 (TNM) T 22722 22724 22724 0 -1 4202496 5212 0 0 0 9811 2 0 0 25 0 1 0 271064523 474193920 4941 33554432000 134512640 135008748 4292982896 18446744073709551615 134528940 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/22724/statm: 115770 4941 46 122 0 114670 0
[pid=22722/tid=22789] ppid=22548 vsize=428720 CPUtime=0 cores=0,2,4,6
/proc/22722/task/22789/stat : 22789 (python) S 22548 22548 22195 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 271074539 439009280 12413 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 514 18446604453985339200 0 0 -1 0 0 0 0
[pid=22727] ppid=22548 vsize=438976 CPUtime=0.18 cores=0,2,4,6
/proc/22727/stat : 22727 (python) S 22548 22548 22195 0 -1 4202560 1192 0 0 0 10 8 0 0 19 0 2 0 271064738 449511424 12417 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672488907 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/22727/statm: 109744 12417 399 364 0 45623 0
[pid=22729] ppid=22727 vsize=463080 CPUtime=99.63 cores=0,2,4,6
/proc/22729/stat : 22729 (TNM) T 22727 22729 22729 0 -1 4202496 5219 0 0 0 9961 2 0 0 25 0 1 0 271064739 474193920 4942 33554432000 134512640 135008748 4290445552 18446744073709551615 134527729 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/22729/statm: 115770 4942 46 122 0 114670 0
[pid=22727/tid=22793] ppid=22548 vsize=438976 CPUtime=0 cores=0,2,4,6
/proc/22727/task/22793/stat : 22793 (python) S 22548 22548 22195 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 271074756 449511424 12417 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 514 18446604453916342080 0 0 -1 2 0 0 0
[pid=22730] ppid=22548 vsize=449232 CPUtime=0.17 cores=0,2,4,6
/proc/22730/stat : 22730 (python) S 22548 22548 22195 0 -1 4202560 1227 0 0 0 9 8 0 0 19 0 2 0 271064743 460013568 12428 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672488907 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/22730/statm: 112308 12428 406 364 0 48184 0
[pid=22732] ppid=22730 vsize=463080 CPUtime=99.98 cores=0,2,4,6
/proc/22732/stat : 22732 (TNM) T 22730 22732 22732 0 -1 4202496 5216 0 0 0 9998 0 0 0 25 0 1 0 271064744 474193920 4941 33554432000 134512640 135008748 4290206016 18446744073709551615 134527742 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/22732/statm: 115770 4941 46 122 0 114670 0
[pid=22730/tid=22794] ppid=22548 vsize=449232 CPUtime=0 cores=0,2,4,6
/proc/22730/task/22794/stat : 22794 (python) S 22548 22548 22195 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 271074762 460013568 12428 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 514 18446604453916340288 0 0 -1 0 0 0 0
[pid=22786] ppid=22548 vsize=459488 CPUtime=0.7 cores=0,2,4,6
/proc/22786/stat : 22786 (python) S 22548 22548 22195 0 -1 4202560 1228 0 0 0 40 30 0 0 18 0 2 0 271074253 470515712 12433 33554432000 4194304 5683644 140737016005648 18446744073709551615 268660682310 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/22786/statm: 114872 12433 407 364 0 50745 0
[pid=22788] ppid=22786 vsize=268420 CPUtime=397.03 cores=0,2,4,6
/proc/22788/stat : 22788 (hybridGM3) R 22786 22788 22788 0 -1 4202496 7624 0 0 0 39700 3 0 0 25 0 1 0 271074254 274862080 7349 33554432000 134512640 135204385 4291425200 18446744073709551615 134536698 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/22788/statm: 67105 7349 64 169 0 66935 0
[pid=22786/tid=22818] ppid=22548 vsize=459488 CPUtime=0 cores=0,2,4,6
/proc/22786/task/22818/stat : 22818 (python) S 22548 22548 22195 0 -1 4202560 46 0 0 0 0 0 0 0 18 0 2 0 271093973 470515712 12433 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 514 18446604453943806912 0 0 -1 4 0 0 0
[pid=22790] ppid=22548 vsize=469744 CPUtime=0.17 cores=0,2,4,6
/proc/22790/stat : 22790 (python) S 22548 22548 22195 0 -1 4202560 1190 0 0 0 9 8 0 0 19 0 2 0 271074544 481017856 12430 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672488907 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/22790/statm: 117436 12430 400 364 0 53306 0
[pid=22792] ppid=22790 vsize=463080 CPUtime=100.14 cores=0,2,4,6
/proc/22792/stat : 22792 (TNM) T 22790 22792 22792 0 -1 4202496 5215 0 0 0 10014 0 0 0 25 0 1 0 271074545 474193920 4942 33554432000 134512640 135008748 4290182384 18446744073709551615 134528491 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/22792/statm: 115770 4942 46 122 0 114670 0
[pid=22790/tid=22808] ppid=22548 vsize=469744 CPUtime=0 cores=0,2,4,6
/proc/22790/task/22808/stat : 22808 (python) S 22548 22548 22195 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 271084561 481017856 12430 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 514 18446604453916678912 0 0 -1 4 0 0 0
[pid=22795] ppid=22548 vsize=480000 CPUtime=0.53 cores=0,2,4,6
/proc/22795/stat : 22795 (python) S 22548 22548 22195 0 -1 4202560 1205 0 0 0 28 25 0 0 18 0 2 0 271074767 491520000 12435 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672488907 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/22795/statm: 120000 12435 401 364 0 55867 0
[pid=22797] ppid=22795 vsize=268424 CPUtime=291.66 cores=0,2,4,6
/proc/22797/stat : 22797 (hybridGM3) T 22795 22797 22797 0 -1 4202496 7621 0 0 0 29163 3 0 0 25 0 1 0 271074767 274866176 7349 33554432000 134512640 135204385 4289915216 18446744073709551615 134537440 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/22797/statm: 67106 7349 64 169 0 66936 0
[pid=22795/tid=22819] ppid=22548 vsize=480000 CPUtime=0 cores=0,2,4,6
/proc/22795/task/22819/stat : 22819 (python) S 22548 22548 22195 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 271094486 491520000 12435 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 514 18446604453936402816 0 0 -1 2 0 0 0
[pid=22809] ppid=22548 vsize=490256 CPUtime=0.36 cores=0,2,4,6
/proc/22809/stat : 22809 (python) S 22548 22548 22195 0 -1 4202560 1212 0 0 0 20 16 0 0 19 0 2 0 271084566 502022144 12438 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672488907 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/22809/statm: 122564 12438 400 364 0 58428 0
[pid=22811] ppid=22809 vsize=268428 CPUtime=196.47 cores=0,2,4,6
/proc/22811/stat : 22811 (hybridGM3) T 22809 22811 22811 0 -1 4202496 7629 0 0 0 19645 2 0 0 25 0 1 0 271084566 274870272 7349 33554432000 134512640 135204385 4289902224 18446744073709551615 134536698 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/22811/statm: 67107 7349 64 169 0 66937 0
[pid=22809/tid=22826] ppid=22548 vsize=490256 CPUtime=0 cores=0,2,4,6
/proc/22809/task/22826/stat : 22826 (python) S 22548 22548 22195 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 271104286 502022144 12438 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 514 18446604453960803328 0 0 -1 6 0 0 0
[pid=22823] ppid=22548 vsize=500512 CPUtime=0.18 cores=0,2,4,6
/proc/22823/stat : 22823 (python) S 22548 22548 22195 0 -1 4202560 1198 0 0 0 9 9 0 0 19 0 2 0 271104011 512524288 12443 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672488907 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/22823/statm: 125128 12443 401 364 0 60989 0
[pid=22825] ppid=22823 vsize=429460 CPUtime=99.8 cores=0,2,4,6
/proc/22825/stat : 22825 (adaptg2wsat2009) T 22823 22825 22825 0 -1 4202496 5419 0 0 0 9979 1 0 0 25 0 1 0 271104012 439767040 5139 33554432000 134512640 135107411 4291299984 18446744073709551615 134533880 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/22825/statm: 107365 5139 59 146 0 106241 0
[pid=22823/tid=22831] ppid=22548 vsize=500512 CPUtime=0 cores=0,2,4,6
/proc/22823/task/22831/stat : 22831 (python) S 22548 22548 22195 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 271114030 512524288 12443 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 514 18446604453925615680 0 0 -1 4 0 0 0
[pid=22832] ppid=22548 vsize=510768 CPUtime=0 cores=0,2,4,6
/proc/22832/stat : 22832 (python) S 22548 22548 22195 0 -1 4202560 841 0 0 0 0 0 0 0 19 0 1 0 271114034 523026432 12389 33554432000 4194304 5683644 140737016005648 18446744073709551615 268660682310 0 0 16781312 514 4454652397653852159 0 0 17 4 0 0 0
/proc/22832/statm: 127692 12389 344 364 0 63550 0
[pid=22834] ppid=22832 vsize=268420 CPUtime=0.39 cores=0,2,4,6
/proc/22834/stat : 22834 (hybridGM3) R 22832 22834 22834 0 -1 4202496 3617 0 0 0 39 0 0 0 21 0 1 0 271114035 274862080 3338 33554432000 134512640 135204385 4288392464 18446744073709551615 134543993 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/22834/statm: 67105 3338 57 169 0 66935 0
[pid=22548/tid=22556] ppid=22546 vsize=521012 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22556/stat : 22556 (python) S 22546 22548 22195 0 -1 4202560 33 933 0 0 0 0 0 0 16 0 22 0 270995340 533516288 13965 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 6 0 0 0
[pid=22548/tid=22559] ppid=22546 vsize=521012 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22559/stat : 22559 (python) S 22546 22548 22195 0 -1 4202560 32 933 0 0 0 0 0 0 18 0 22 0 270995343 533516288 13965 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 6 0 0 0
[pid=22548/tid=22562] ppid=22546 vsize=521012 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22562/stat : 22562 (python) S 22546 22548 22195 0 -1 4202560 25 933 0 0 0 0 0 0 18 0 22 0 270995346 533516288 13965 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 6 0 0 0
[pid=22548/tid=22565] ppid=22546 vsize=521012 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22565/stat : 22565 (python) S 22546 22548 22195 0 -1 4202560 29 933 0 0 0 0 0 0 18 0 22 0 270995349 533516288 13965 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 4 0 0 0
[pid=22548/tid=22642] ppid=22546 vsize=521012 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22642/stat : 22642 (python) S 22546 22548 22195 0 -1 4202560 34 933 0 0 0 0 0 0 18 0 22 0 271014964 533516288 13965 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 4 0 0 0
[pid=22548/tid=22645] ppid=22546 vsize=521012 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22645/stat : 22645 (python) S 22546 22548 22195 0 -1 4202560 24 933 0 0 0 0 0 0 20 0 22 0 271015357 533516288 13965 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 4 0 0 0
[pid=22548/tid=22667] ppid=22546 vsize=521012 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22667/stat : 22667 (python) S 22546 22548 22195 0 -1 4202560 32 933 0 0 0 0 0 0 18 0 22 0 271025077 533516288 13965 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 0 0 0 0
[pid=22548/tid=22671] ppid=22546 vsize=521012 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22671/stat : 22671 (python) S 22546 22548 22195 0 -1 4202560 34 933 0 0 0 0 0 0 18 0 22 0 271025375 533516288 13965 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 0 0 0 0
[pid=22548/tid=22688] ppid=22546 vsize=521012 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22688/stat : 22688 (python) S 22546 22548 22195 0 -1 4202560 35 933 0 0 0 0 0 0 18 0 22 0 271034496 533516288 13965 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 0 0 0 0
[pid=22548/tid=22700] ppid=22546 vsize=521012 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22700/stat : 22700 (python) S 22546 22548 22195 0 -1 4202560 29 933 0 0 0 0 0 0 20 0 22 0 271044516 533516288 13965 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 0 0 0 0
[pid=22548/tid=22704] ppid=22546 vsize=521012 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22704/stat : 22704 (python) S 22546 22548 22195 0 -1 4202560 31 933 0 0 0 0 0 0 20 0 22 0 271044802 533516288 13965 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 0 0 0 0
[pid=22548/tid=22713] ppid=22546 vsize=521012 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22713/stat : 22713 (python) S 22546 22548 22195 0 -1 4202560 32 933 0 0 0 0 0 0 18 0 22 0 271054718 533516288 13965 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 0 0 0 0
[pid=22548/tid=22723] ppid=22546 vsize=521012 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22723/stat : 22723 (python) S 22546 22548 22195 0 -1 4202560 29 933 0 0 0 0 0 0 20 0 22 0 271064523 533516288 13965 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 0 0 0 0
[pid=22548/tid=22728] ppid=22546 vsize=521012 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22728/stat : 22728 (python) S 22546 22548 22195 0 -1 4202560 27 933 0 0 0 0 0 0 20 0 22 0 271064739 533516288 13965 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 2 0 0 0
[pid=22548/tid=22731] ppid=22546 vsize=521012 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22731/stat : 22731 (python) S 22546 22548 22195 0 -1 4202560 29 933 0 0 0 0 0 0 20 0 22 0 271064744 533516288 13965 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 0 0 0 0
[pid=22548/tid=22787] ppid=22546 vsize=521012 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22787/stat : 22787 (python) S 22546 22548 22195 0 -1 4202560 40 933 0 0 0 0 0 0 18 0 22 0 271074254 533516288 13965 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 0 0 0 0
[pid=22548/tid=22791] ppid=22546 vsize=521012 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22791/stat : 22791 (python) S 22546 22548 22195 0 -1 4202560 26 933 0 0 0 0 0 0 20 0 22 0 271074545 533516288 13965 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 0 0 0 0
[pid=22548/tid=22796] ppid=22546 vsize=521012 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22796/stat : 22796 (python) S 22546 22548 22195 0 -1 4202560 28 933 0 0 0 0 0 0 18 0 22 0 271074767 533516288 13965 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 0 0 0 0
[pid=22548/tid=22810] ppid=22546 vsize=521012 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22810/stat : 22810 (python) S 22546 22548 22195 0 -1 4202560 24 933 0 0 0 0 0 0 20 0 22 0 271084566 533516288 13965 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 0 0 0 0
[pid=22548/tid=22824] ppid=22546 vsize=521012 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22824/stat : 22824 (python) S 22546 22548 22195 0 -1 4202560 24 933 0 0 0 0 0 0 20 0 22 0 271104012 533516288 13965 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 0 0 0 0
[pid=22548/tid=22833] ppid=22546 vsize=521012 CPUtime=0 cores=0,2,4,6
/proc/22548/task/22833/stat : 22833 (python) S 22546 22548 22195 0 -1 4202560 27 933 0 0 0 0 0 0 20 0 22 0 271114035 533516288 13965 33554432000 4194304 5683644 140737016005648 18446744073709551615 268672485809 0 0 16781312 2 18446604453966736256 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4733.06
Current children cumulated vsize (KiB) 16264140

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 22548 and gives
#  childrusage.ru_utime.tv_sec=606
#  childrusage.ru_utime.tv_usec=928732
#  childrusage.ru_stime.tv_sec=4
#  childrusage.ru_stime.tv_usec=2391
# CPU time returned by wait4() is 610.931
# while last known CPU time is 4733.06
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1193.54
CPU time (s): 4733.06
CPU user time (s): 4728.93
CPU system time (s): 4.13
CPU usage (%): 396.555
Max. virtual memory (cumulated for all children) (KiB): 16264140

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 606.929
system time used= 4.00239
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 84840
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 6559
involuntary context switches= 2817

runsolver used 6.16706 second user time and 15.3657 second system time

The end

Launcher Data 

Begin job on node107 at 2011-05-01 18:40:35
IDJOB=3317190
IDBENCH=82655
IDSOLVER=1705
FILE ID=node107/3317190-1304268034
RUNJOBID= node107-1304267294-22213
PBS_JOBID= 13173971
Free space on /tmp= 72620 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v30000-c126000-S643238994-057-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317190-1304268034/watcher-3317190-1304268034 -o /tmp/evaluation-result-3317190-1304268034/solver-3317190-1304268034 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317190-1304268034.cnf -seed 886541200 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 2d92a52914298e3c91073398b38892d9
RANDOM SEED=886541200

node107.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5359.47
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      29966608 kB
Buffers:        418068 kB
Cached:        1929808 kB
SwapCached:          4 kB
Active:         499192 kB
Inactive:      2112200 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      29966608 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:           34300 kB
Writeback:           0 kB
AnonPages:      263800 kB
Mapped:          22008 kB
Slab:           295220 kB
PageTables:      17396 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 10821304 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72572 MiB
End job on node107 at 2011-05-01 19:00:30