Trace number 3378270

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (MO) 4226.09 1069.25

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v40000-c168000-S907804052-003-UNKNOWN.cnf
MD5SUMeb4eadfedd2c4dc9b0ed7231ad324f1f
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark273.432
Satisfiable
(Un)Satisfiability was proved
Number of variables40000
Number of clauses168000
Sum of the clauses size504000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3168000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.30/0.47	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.30/0.47	c INFO: solving HOME/instance-3378270-1305464377.cnf
0.30/0.47	c INFO: computing task features
0.90/1.05	c INFO: computed variable-clause graph statistics
0.90/1.08	c INFO: computed clause balance statistics
3.98/4.16	c INFO: computed variable balance statistics
6.50/6.68	c INFO: computed small-clause counts
6.58/6.70	c INFO: computed Horn-clause counts
8.59/8.71	c INFO: computed variable graph statistics
8.59/8.72	c INFO: collected features for HOME/instance-3378270-1305464377.cnf in 8.22 s
8.59/8.72	c INFO: running hybridGM3@0 for 97 with 1191 remaining (b = 0.35)
8.59/8.73	c INFO: running hybridGM3@0 for 97 with 1191 remaining (b = 0.21)
8.59/8.74	c INFO: running hybridGM3@0 for 97 with 1191 remaining (b = 0.14)
8.59/8.76	c INFO: running hybridGM3@0 for 194 with 1191 remaining (b = 0.18)
406.06/108.57	c INFO: running hybridGM3@0 for 97 with 1091 remaining (b = 0.08)
410.06/109.51	c INFO: running hybridGM3@0 for 97 with 1090 remaining (b = 0.08)
410.06/109.55	c INFO: running hybridGM3@0 for 97 with 1090 remaining (b = 0.08)
786.25/203.90	c INFO: running hybridGM3@98 for 97 with 996 remaining (b = 0.08)
789.82/204.82	c INFO: running hybridGM3@100 for 97 with 995 remaining (b = 0.08)
789.82/204.86	c INFO: running hybridGM3@0 for 194 with 995 remaining (b = 0.14)
810.17/209.91	c INFO: running TNM@0 for 97 with 990 remaining (b = 0.06)
1181.87/303.24	c INFO: running hybridGM3@100 for 97 with 897 remaining (b = 0.05)
1193.42/306.15	c INFO: running hybridGM3@193 for 97 with 894 remaining (b = 0.05)
1209.82/310.25	c INFO: running hybridGM3@0 for 194 with 890 remaining (b = 0.09)
1483.75/379.05	c INFO: running hybridGM3@0 for 194 with 821 remaining (b = 0.07)
1574.44/401.84	c INFO: running hybridGM3@0 for 194 with 798 remaining (b = 0.05)
1574.81/401.95	c INFO: running hybridGM3@99 for 97 with 798 remaining (b = 0.02)
1978.11/503.27	c INFO: running TNM@0 for 97 with 697 remaining (b = 0.02)
1990.41/506.37	c INFO: running hybridGM3@72 for 97 with 694 remaining (b = 0.02)
2267.96/576.19	c INFO: running hybridGM3@92 for 97 with 624 remaining (b = 0.02)
2376.91/603.59	c INFO: running hybridGM3@0 for 194 with 596 remaining (b = 0.04)
2382.50/604.96	c INFO: running hybridGM3@0 for 194 with 595 remaining (b = 0.03)
2393.19/607.68	c INFO: running TNM@0 for 97 with 592 remaining (b = 0.01)
2646.80/671.50	c INFO: running hybridGM3@192 for 97 with 528 remaining (b = 0.02)
2792.32/708.02	c INFO: running hybridGM3@0 for 194 with 492 remaining (b = 0.03)
3049.89/772.83	c INFO: running hybridGM3@291 for 97 with 427 remaining (b = 0.01)
3160.41/800.71	c INFO: running TNM@0 for 97 with 399 remaining (b = 0.01)
3189.79/808.09	c INFO: running hybridGM3@100 for 97 with 392 remaining (b = 0.01)
3428.70/868.14	c INFO: running TNM@0 for 97 with 332 remaining (b = 0.01)
3559.44/901.05	c INFO: running TNM@0 for 97 with 299 remaining (b = 0.01)
3568.96/903.41	c INFO: running hybridGM3@0 for 194 with 297 remaining (b = 0.02)
3575.70/905.12	c INFO: running hybridGM3@100 for 97 with 295 remaining (b = 0.01)
3826.72/968.49	c INFO: running TNM@0 for 97 with 231 remaining (b = 0.01)
3957.17/1001.40	c INFO: running TNM@0 for 97 with 199 remaining (b = 0.01)
3977.44/1006.46	c INFO: running TNM@0 for 194 with 194 remaining (b = 0.01)
4224.81/1068.84	c INFO: running TNM@0 for 131 with 131 remaining (b = 0.01)
4226.09/1069.24	c INFO: running TNM@0 for 131 with 131 remaining (b = 0.01)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378270-1305464377/watcher-3378270-1305464377 -o /tmp/evaluation-result-3378270-1305464377/solver-3378270-1305464377 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378270-1305464377.cnf -seed 398813431 -budget 4800 -cores 4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.33 8.12 7.01 5/445 29310
/proc/meminfo: memFree=21187032/32950928 swapFree=65730864/67111528
[pid=29310] ppid=29308 vsize=16244 CPUtime=0 cores=0,2,4,6
/proc/29310/stat : 29310 (python) R 29308 29310 28586 0 -1 4202496 996 0 0 0 0 0 0 0 25 0 1 0 736668408 16633856 869 33554432000 4194304 5683644 140734649595888 18446744073709551615 4540219 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/29310/statm: 4061 870 368 364 0 548 0

[startup+0.046752 s]
/proc/loadavg: 8.33 8.12 7.01 5/445 29310
/proc/meminfo: memFree=21187032/32950928 swapFree=65730864/67111528
[pid=29310] ppid=29308 vsize=29792 CPUtime=0.03 cores=0,2,4,6
/proc/29310/stat : 29310 (python) R 29308 29310 28586 0 -1 4202496 1408 0 0 0 2 1 0 0 25 0 1 0 736668408 30507008 1252 33554432000 4194304 5683644 140734649595888 18446744073709551615 5376320 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/29310/statm: 7448 1252 443 364 0 818 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 29792

[startup+0.100751 s]
/proc/loadavg: 8.33 8.12 7.01 5/445 29310
/proc/meminfo: memFree=21187032/32950928 swapFree=65730864/67111528
[pid=29310] ppid=29308 vsize=104228 CPUtime=0.06 cores=0,2,4,6
/proc/29310/stat : 29310 (python) S 29308 29310 28586 0 -1 4202496 3058 0 0 0 4 2 0 0 25 0 1 0 736668408 106729472 2741 33554432000 4194304 5683644 140734649595888 18446744073709551615 260905392128 0 2147483391 16781312 2 18446744071699207358 0 0 17 4 0 0 0
/proc/29310/statm: 26057 2741 847 364 0 1805 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 104228

[startup+0.300707 s]
/proc/loadavg: 8.33 8.12 7.01 5/445 29310
/proc/meminfo: memFree=21187032/32950928 swapFree=65730864/67111528
[pid=29310] ppid=29308 vsize=170756 CPUtime=0.22 cores=0,2,4,6
/proc/29310/stat : 29310 (python) S 29308 29310 28586 0 -1 4202496 6733 932 1 0 16 6 0 0 17 0 1 0 736668408 174854144 4806 33554432000 4194304 5683644 140734649595888 18446744073709551615 260905392128 0 2147483391 16781312 2 18446744071699207358 0 0 17 4 0 0 0
/proc/29310/statm: 42689 4806 1336 364 0 3454 0
Current children cumulated CPU time (s) 0.22
Current children cumulated vsize (KiB) 170756

[startup+0.700634 s]
/proc/loadavg: 8.33 8.12 7.01 5/445 29310
/proc/meminfo: memFree=21187032/32950928 swapFree=65730864/67111528
[pid=29310] ppid=29308 vsize=296028 CPUtime=0.59 cores=0,2,4,6
/proc/29310/stat : 29310 (python) R 29308 29310 28586 0 -1 4202496 13611 932 1 0 49 10 0 0 17 0 1 0 736668408 303132672 11297 33554432000 4194304 5683644 140734649595888 18446744073709551615 4445115 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/29310/statm: 74007 11297 1761 364 0 9931 0
Current children cumulated CPU time (s) 0.59
Current children cumulated vsize (KiB) 296028

[startup+1.50049 s]
/proc/loadavg: 8.33 8.12 7.01 6/446 29315
/proc/meminfo: memFree=21143376/32950928 swapFree=65730864/67111528
[pid=29310] ppid=29308 vsize=317992 CPUtime=1.38 cores=0,2,4,6
/proc/29310/stat : 29310 (python) R 29308 29310 28586 0 -1 4202496 23794 932 1 0 127 11 0 0 19 0 1 0 736668408 325623808 16882 33554432000 4194304 5683644 140734649595888 18446744073709551615 47005647679790 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/29310/statm: 79498 16882 1826 364 0 15422 0
Current children cumulated CPU time (s) 1.38
Current children cumulated vsize (KiB) 317992

[startup+3.10019 s]
/proc/loadavg: 8.33 8.12 7.01 6/446 29315
/proc/meminfo: memFree=21124404/32950928 swapFree=65730864/67111528
[pid=29310] ppid=29308 vsize=317992 CPUtime=2.98 cores=0,2,4,6
/proc/29310/stat : 29310 (python) R 29308 29310 28586 0 -1 4202496 23794 932 1 0 287 11 0 0 22 0 1 0 736668408 325623808 16882 33554432000 4194304 5683644 140734649595888 18446744073709551615 4603005 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/29310/statm: 79498 16882 1826 364 0 15422 0
Current children cumulated CPU time (s) 2.98
Current children cumulated vsize (KiB) 317992

[startup+6.30061 s]
/proc/loadavg: 8.15 8.08 7.00 6/446 29315
/proc/meminfo: memFree=21124064/32950928 swapFree=65730864/67111528
[pid=29310] ppid=29308 vsize=317992 CPUtime=6.18 cores=0,2,4,6
/proc/29310/stat : 29310 (python) R 29308 29310 28586 0 -1 4202496 23795 932 1 0 607 11 0 0 25 0 1 0 736668408 325623808 16883 33554432000 4194304 5683644 140734649595888 18446744073709551615 260908802737 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/29310/statm: 79498 16883 1827 364 0 15422 0
Current children cumulated CPU time (s) 6.18
Current children cumulated vsize (KiB) 317992

[startup+12.7014 s]
/proc/loadavg: 7.89 8.03 6.99 9/462 29331
/proc/meminfo: memFree=20951752/32950928 swapFree=65730864/67111528
[pid=29310] ppid=29308 vsize=355900 CPUtime=8.64 cores=0,2,4,6
/proc/29310/stat : 29310 (python) S 29308 29310 28586 0 -1 4202496 26566 932 1 0 852 12 0 0 25 0 5 0 736668408 364441600 16163 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/29310/statm: 88975 16163 1873 364 0 24884 0
[pid=29320] ppid=29310 vsize=314888 CPUtime=0 cores=0,2,4,6
/proc/29320/stat : 29320 (python) S 29310 29310 28586 0 -1 4202560 730 0 0 0 0 0 0 0 15 0 1 0 736669281 322445312 14627 33554432000 4194304 5683644 140734649595888 18446744073709551615 260905414159 0 0 16781312 514 17374399183527575812 0 0 17 4 0 0 0
/proc/29320/statm: 78722 14627 353 364 0 14640 0
[pid=29322] ppid=29320 vsize=276236 CPUtime=3.92 cores=0,2,4,6
/proc/29322/stat : 29322 (hybridGM3) R 29320 29322 29322 0 -1 4202496 8566 0 0 0 391 1 0 0 25 0 1 0 736669281 282865664 8302 33554432000 134512640 135204385 4287329040 18446744073709551615 134545837 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/29322/statm: 69059 8302 64 169 0 68889 0
[pid=29323] ppid=29310 vsize=325144 CPUtime=0 cores=0,2,4,6
/proc/29323/stat : 29323 (python) S 29310 29310 28586 0 -1 4202560 736 0 0 0 0 0 0 0 15 0 1 0 736669282 332947456 14625 33554432000 4194304 5683644 140734649595888 18446744073709551615 260905414214 0 0 16781312 514 17374403581574086915 0 0 17 2 0 0 0
/proc/29323/statm: 81286 14625 347 364 0 17201 0
[pid=29325] ppid=29323 vsize=276240 CPUtime=3.94 cores=0,2,4,6
/proc/29325/stat : 29325 (hybridGM3) R 29323 29325 29325 0 -1 4202496 8562 0 0 0 394 0 0 0 25 0 1 0 736669282 282869760 8302 33554432000 134512640 135204385 4291703744 18446744073709551615 134537123 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/29325/statm: 69060 8302 64 169 0 68890 0
[pid=29326] ppid=29310 vsize=335400 CPUtime=0 cores=0,2,4,6
/proc/29326/stat : 29326 (python) S 29310 29310 28586 0 -1 4202560 744 0 0 0 0 0 0 0 15 0 1 0 736669283 343449600 14629 33554432000 4194304 5683644 140734649595888 18446744073709551615 260905414214 0 0 16781312 514 6924363596214272257 0 0 17 2 0 0 0
/proc/29326/statm: 83850 14629 347 364 0 19762 0
[pid=29328] ppid=29326 vsize=276244 CPUtime=3.93 cores=0,2,4,6
/proc/29328/stat : 29328 (hybridGM3) R 29326 29328 29328 0 -1 4202496 8560 0 0 0 392 1 0 0 25 0 1 0 736669283 282873856 8301 33554432000 134512640 135204385 4288850432 18446744073709551615 134537292 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/29328/statm: 69061 8301 64 169 0 68891 0
[pid=29329] ppid=29310 vsize=345656 CPUtime=0 cores=0,2,4,6
/proc/29329/stat : 29329 (python) S 29310 29310 28586 0 -1 4202560 753 0 0 0 0 0 0 0 15 0 1 0 736669284 353951744 14633 33554432000 4194304 5683644 140734649595888 18446744073709551615 260905414214 0 0 16781312 514 4002736900841373954 0 0 17 4 0 0 0
/proc/29329/statm: 86414 14633 347 364 0 22323 0
[pid=29331] ppid=29329 vsize=276240 CPUtime=3.93 cores=0,2,4,6
/proc/29331/stat : 29331 (hybridGM3) R 29329 29331 29331 0 -1 4202496 8558 0 0 0 392 1 0 0 25 0 1 0 736669284 282869760 8298 33554432000 134512640 135204385 4294775024 18446744073709551615 134537167 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/29331/statm: 69060 8298 64 169 0 68890 0
[pid=29310/tid=29321] ppid=29308 vsize=355900 CPUtime=0 cores=0,2,4,6
/proc/29310/task/29321/stat : 29321 (python) S 29308 29310 28586 0 -1 4202560 24 932 0 0 0 0 0 0 25 0 5 0 736669281 364441600 16163 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 6 0 0 0
[pid=29310/tid=29324] ppid=29308 vsize=355900 CPUtime=0 cores=0,2,4,6
/proc/29310/task/29324/stat : 29324 (python) S 29308 29310 28586 0 -1 4202560 23 932 0 0 0 0 0 0 25 0 5 0 736669282 364441600 16163 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 2 0 0 0
[pid=29310/tid=29327] ppid=29308 vsize=355900 CPUtime=0 cores=0,2,4,6
/proc/29310/task/29327/stat : 29327 (python) S 29308 29310 28586 0 -1 4202560 22 932 0 0 0 0 0 0 25 0 5 0 736669283 364441600 16163 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 2 0 0 0
[pid=29310/tid=29330] ppid=29308 vsize=355900 CPUtime=0 cores=0,2,4,6
/proc/29310/task/29330/stat : 29330 (python) S 29308 29310 28586 0 -1 4202560 27 932 0 0 0 0 0 0 25 0 5 0 736669284 364441600 16163 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 24.36
Current children cumulated vsize (KiB) 2781948

[startup+25.5011 s]
/proc/loadavg: 7.92 8.03 7.01 10/466 29336
/proc/meminfo: memFree=20921960/32950928 swapFree=65730864/67111528
[pid=29310] ppid=29308 vsize=355900 CPUtime=8.64 cores=0,2,4,6
/proc/29310/stat : 29310 (python) S 29308 29310 28586 0 -1 4202496 26566 932 1 0 852 12 0 0 25 0 5 0 736668408 364441600 16163 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/29310/statm: 88975 16163 1873 364 0 24884 0
[pid=29320] ppid=29310 vsize=314888 CPUtime=0.05 cores=0,2,4,6
/proc/29320/stat : 29320 (python) S 29310 29310 28586 0 -1 4202560 905 0 0 0 3 2 0 0 16 0 1 0 736669281 322445312 14650 33554432000 4194304 5683644 140734649595888 18446744073709551615 260905414159 0 0 16781312 514 6332236997943329028 0 0 17 4 0 0 0
/proc/29320/statm: 78722 14650 376 364 0 14640 0
[pid=29322] ppid=29320 vsize=276236 CPUtime=16.52 cores=0,2,4,6
/proc/29322/stat : 29322 (hybridGM3) R 29320 29322 29322 0 -1 4202496 8567 0 0 0 1651 1 0 0 25 0 1 0 736669281 282865664 8303 33554432000 134512640 135204385 4287329040 18446744073709551615 134549101 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/29322/statm: 69059 8303 64 169 0 68889 0
[pid=29323] ppid=29310 vsize=325144 CPUtime=0.05 cores=0,2,4,6
/proc/29323/stat : 29323 (python) S 29310 29310 28586 0 -1 4202560 922 0 0 0 3 2 0 0 16 0 1 0 736669282 332947456 14648 33554432000 4194304 5683644 140734649595888 18446744073709551615 260905414214 0 0 16781312 514 6332236997943329027 0 0 17 2 0 0 0
/proc/29323/statm: 81286 14648 370 364 0 17201 0
[pid=29325] ppid=29323 vsize=276240 CPUtime=16.74 cores=0,2,4,6
/proc/29325/stat : 29325 (hybridGM3) R 29323 29325 29325 0 -1 4202496 8562 0 0 0 1674 0 0 0 25 0 1 0 736669282 282869760 8302 33554432000 134512640 135204385 4291703744 18446744073709551615 134536646 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/29325/statm: 69060 8302 64 169 0 68890 0
[pid=29326] ppid=29310 vsize=335400 CPUtime=0.05 cores=0,2,4,6
/proc/29326/stat : 29326 (python) S 29310 29310 28586 0 -1 4202560 922 0 0 0 3 2 0 0 16 0 1 0 736669283 343449600 14652 33554432000 4194304 5683644 140734649595888 18446744073709551615 260905414214 0 0 16781312 514 6332236997943329025 0 0 17 2 0 0 0
/proc/29326/statm: 83850 14652 370 364 0 19762 0
[pid=29328] ppid=29326 vsize=276244 CPUtime=16.61 cores=0,2,4,6
/proc/29328/stat : 29328 (hybridGM3) R 29326 29328 29328 0 -1 4202496 8560 0 0 0 1660 1 0 0 25 0 1 0 736669283 282873856 8301 33554432000 134512640 135204385 4288850432 18446744073709551615 134536986 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/29328/statm: 69061 8301 64 169 0 68891 0
[pid=29329] ppid=29310 vsize=345656 CPUtime=0.05 cores=0,2,4,6
/proc/29329/stat : 29329 (python) S 29310 29310 28586 0 -1 4202560 920 0 0 0 2 3 0 0 16 0 1 0 736669284 353951744 14656 33554432000 4194304 5683644 140734649595888 18446744073709551615 260905414214 0 0 16781312 514 6332236997943329026 0 0 17 4 0 0 0
/proc/29329/statm: 86414 14656 370 364 0 22323 0
[pid=29331] ppid=29329 vsize=276240 CPUtime=16.61 cores=0,2,4,6
/proc/29331/stat : 29331 (hybridGM3) R 29329 29331 29331 0 -1 4202496 8558 0 0 0 1660 1 0 0 25 0 1 0 736669284 282869760 8298 33554432000 134512640 135204385 4294775024 18446744073709551615 134536676 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/29331/statm: 69060 8298 64 169 0 68890 0
[pid=29310/tid=29321] ppid=29308 vsize=355900 CPUtime=0 cores=0,2,4,6
/proc/29310/task/29321/stat : 29321 (python) S 29308 29310 28586 0 -1 4202560 24 932 0 0 0 0 0 0 25 0 5 0 736669281 364441600 16163 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 6 0 0 0
[pid=29310/tid=29324] ppid=29308 vsize=355900 CPUtime=0 cores=0,2,4,6
/proc/29310/task/29324/stat : 29324 (python) S 29308 29310 28586 0 -1 4202560 23 932 0 0 0 0 0 0 25 0 5 0 736669282 364441600 16163 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 2 0 0 0
[pid=29310/tid=29327] ppid=29308 vsize=355900 CPUtime=0 cores=0,2,4,6
/proc/29310/task/29327/stat : 29327 (python) S 29308 29310 28586 0 -1 4202560 22 932 0 0 0 0 0 0 25 0 5 0 736669283 364441600 16163 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 2 0 0 0
[pid=29310/tid=29330] ppid=29308 vsize=355900 CPUtime=0 cores=0,2,4,6
/proc/29310/task/29330/stat : 29330 (python) S 29308 29310 28586 0 -1 4202560 27 932 0 0 0 0 0 0 25 0 5 0 736669284 364441600 16163 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 75.32
Current children cumulated vsize (KiB) 2781948

[startup+51.1014 s]
/proc/loadavg: 8.01 8.04 7.04 9/467 29340
/proc/meminfo: memFree=20888048/32950928 swapFree=65730864/67111528
[pid=29310] ppid=29308 vsize=355900 CPUtime=8.64 cores=0,2,4,6
/proc/29310/stat : 29310 (python) S 29308 29310 28586 0 -1 4202496 26566 932 1 0 852 12 0 0 25 0 5 0 736668408 364441600 16163 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/29310/statm: 88975 16163 1873 364 0 24884 0
[pid=29320] ppid=29310 vsize=314888 CPUtime=0.12 cores=0,2,4,6
/proc/29320/stat : 29320 (python) S 29310 29310 28586 0 -1 4202560 915 0 0 0 7 5 0 0 15 0 1 0 736669281 322445312 14650 33554432000 4194304 5683644 140734649595888 18446744073709551615 260905414159 0 0 16781312 514 6332236997943329028 0 0 17 4 0 0 0
/proc/29320/statm: 78722 14650 376 364 0 14640 0
[pid=29322] ppid=29320 vsize=276236 CPUtime=41.76 cores=0,2,4,6
/proc/29322/stat : 29322 (hybridGM3) R 29320 29322 29322 0 -1 4202496 8567 0 0 0 4175 1 0 0 25 0 1 0 736669281 282865664 8303 33554432000 134512640 135204385 4287329040 18446744073709551615 134537272 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/29322/statm: 69059 8303 64 169 0 68889 0
[pid=29323] ppid=29310 vsize=325144 CPUtime=0.13 cores=0,2,4,6
/proc/29323/stat : 29323 (python) S 29310 29310 28586 0 -1 4202560 932 0 0 0 7 6 0 0 15 0 1 0 736669282 332947456 14648 33554432000 4194304 5683644 140734649595888 18446744073709551615 260905414214 0 0 16781312 514 6332236997943329027 0 0 17 2 0 0 0
/proc/29323/statm: 81286 14648 370 364 0 17201 0
[pid=29325] ppid=29323 vsize=276240 CPUtime=42.35 cores=0,2,4,6
/proc/29325/stat : 29325 (hybridGM3) R 29323 29325 29325 0 -1 4202496 8562 0 0 0 4235 0 0 0 25 0 1 0 736669282 282869760 8302 33554432000 134512640 135204385 4291703744 18446744073709551615 134537510 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/29325/statm: 69060 8302 64 169 0 68890 0
[pid=29326] ppid=29310 vsize=335400 CPUtime=0.13 cores=0,2,4,6
/proc/29326/stat : 29326 (python) S 29310 29310 28586 0 -1 4202560 931 0 0 0 7 6 0 0 15 0 1 0 736669283 343449600 14652 33554432000 4194304 5683644 140734649595888 18446744073709551615 260905414214 0 0 16781312 514 6332236997943329025 0 0 17 2 0 0 0
/proc/29326/statm: 83850 14652 370 364 0 19762 0
[pid=29328] ppid=29326 vsize=276244 CPUtime=42.04 cores=0,2,4,6
/proc/29328/stat : 29328 (hybridGM3) R 29326 29328 29328 0 -1 4202496 8560 0 0 0 4203 1 0 0 25 0 1 0 736669283 282873856 8301 33554432000 134512640 135204385 4288850432 18446744073709551615 134537403 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/29328/statm: 69061 8301 64 169 0 68891 0
[pid=29329] ppid=29310 vsize=345656 CPUtime=0.13 cores=0,2,4,6
/proc/29329/stat : 29329 (python) S 29310 29310 28586 0 -1 4202560 929 0 0 0 6 7 0 0 15 0 1 0 736669284 353951744 14656 33554432000 4194304 5683644 140734649595888 18446744073709551615 260905414214 0 0 16781312 514 6332236997943329026 0 0 17 4 0 0 0
/proc/29329/statm: 86414 14656 370 364 0 22323 0

################
# More data... #
################

/proc/29399/statm: 96670 14708 405 364 0 32567 0
[pid=29401] ppid=29399 vsize=276236 CPUtime=487.27 cores=0,2,4,6
/proc/29401/stat : 29401 (hybridGM3) T 29399 29401 29401 0 -1 4202496 8566 0 0 0 48724 3 0 0 25 0 1 0 736706314 282865664 8300 33554432000 134512640 135204385 4292203264 18446744073709551615 134536663 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/29401/statm: 69059 8300 64 169 0 68889 0
[pid=29399/tid=29466] ppid=29310 vsize=386680 CPUtime=0 cores=0,2,4,6
/proc/29399/task/29466/stat : 29466 (python) S 29310 29310 28586 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 736726026 395960320 14708 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 514 18446604446866490304 0 0 -1 4 0 0 0
[pid=29446] ppid=29310 vsize=396936 CPUtime=0.64 cores=0,2,4,6
/proc/29446/stat : 29446 (python) S 29310 29310 28586 0 -1 4202560 1194 0 0 0 33 31 0 0 16 0 2 0 736708593 406462464 14712 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/29446/statm: 99234 14712 405 364 0 35128 0
[pid=29448] ppid=29446 vsize=276240 CPUtime=198.78 cores=0,2,4,6
/proc/29448/stat : 29448 (hybridGM3) T 29446 29448 29448 0 -1 4202496 8578 0 0 0 19874 4 0 0 25 0 1 0 736708593 282869760 8310 33554432000 134512640 135204385 4286856816 18446744073709551615 134537573 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/29448/statm: 69060 8310 64 169 0 68890 0
[pid=29446/tid=29472] ppid=29310 vsize=396936 CPUtime=0 cores=0,2,4,6
/proc/29446/task/29472/stat : 29472 (python) S 29310 29310 28586 0 -1 4202560 51 0 0 0 0 0 0 0 16 0 2 0 736728903 406462464 14712 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 514 18446604450937346048 0 0 -1 4 0 0 0
[pid=29460] ppid=29310 vsize=407192 CPUtime=0.32 cores=0,2,4,6
/proc/29460/stat : 29460 (python) S 29310 29310 28586 0 -1 4202560 1139 0 0 0 16 16 0 0 19 0 2 0 736718735 416964608 14710 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/29460/statm: 101798 14710 399 364 0 37689 0
[pid=29462] ppid=29460 vsize=467024 CPUtime=100.26 cores=0,2,4,6
/proc/29462/stat : 29462 (TNM) T 29460 29462 29462 0 -1 4202496 6824 0 0 0 10025 1 0 0 25 0 1 0 736718735 478232576 6558 33554432000 134512640 135008748 4287727776 18446744073709551615 134527729 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/29462/statm: 116756 6558 46 122 0 115656 0
[pid=29460/tid=29468] ppid=29310 vsize=407192 CPUtime=0 cores=0,2,4,6
/proc/29460/task/29468/stat : 29468 (python) S 29310 29310 28586 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 736728766 416964608 14710 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 514 18446604449707337856 0 0 -1 4 0 0 0
[pid=29469] ppid=29310 vsize=417448 CPUtime=0.66 cores=0,2,4,6
/proc/29469/stat : 29469 (python) S 29310 29310 28586 0 -1 4202560 1171 0 0 0 37 29 0 0 16 0 2 0 736728767 427466752 14714 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/29469/statm: 104362 14714 399 364 0 40250 0
[pid=29471] ppid=29469 vsize=276236 CPUtime=197.03 cores=0,2,4,6
/proc/29471/stat : 29471 (hybridGM3) T 29469 29471 29471 0 -1 4202496 8563 0 0 0 19702 1 0 0 25 0 1 0 736728768 282865664 8295 33554432000 134512640 135204385 4288265248 18446744073709551615 134536863 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/29471/statm: 69059 8295 64 169 0 68889 0
[pid=29469/tid=29489] ppid=29310 vsize=417448 CPUtime=0 cores=0,2,4,6
/proc/29469/task/29489/stat : 29489 (python) S 29310 29310 28586 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 736748478 427466752 14714 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 514 18446604446866492096 0 0 -1 4 0 0 0
[pid=29473] ppid=29310 vsize=427704 CPUtime=0.99 cores=0,2,4,6
/proc/29473/stat : 29473 (python) S 29310 29310 28586 0 -1 4202560 1203 0 0 0 54 45 0 0 15 0 2 0 736728904 437968896 14718 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/29473/statm: 106926 14718 399 364 0 42811 0
[pid=29475] ppid=29473 vsize=276236 CPUtime=292.66 cores=0,2,4,6
/proc/29475/stat : 29475 (hybridGM3) T 29473 29475 29475 0 -1 4202496 8566 0 0 0 29265 1 0 0 25 0 1 0 736728905 282865664 8299 33554432000 134512640 135204385 4292780240 18446744073709551615 134536663 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/29475/statm: 69059 8299 64 169 0 68889 0
[pid=29473/tid=29494] ppid=29310 vsize=427704 CPUtime=0 cores=0,2,4,6
/proc/29473/task/29494/stat : 29494 (python) S 29310 29310 28586 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 736749216 437968896 14718 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 514 18446604446866491200 0 0 -1 4 0 0 0
[pid=29476] ppid=29310 vsize=437960 CPUtime=0.33 cores=0,2,4,6
/proc/29476/stat : 29476 (python) S 29310 29310 28586 0 -1 4202560 1148 0 0 0 18 15 0 0 19 0 2 0 736729177 448471040 14722 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/29476/statm: 109490 14722 399 364 0 45372 0
[pid=29478] ppid=29476 vsize=467024 CPUtime=99.44 cores=0,2,4,6
/proc/29478/stat : 29478 (TNM) T 29476 29478 29478 0 -1 4202496 6827 0 0 0 9943 1 0 0 25 0 1 0 736729177 478232576 6559 33554432000 134512640 135008748 4289791104 18446744073709551615 134527327 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/29478/statm: 116756 6559 46 122 0 115656 0
[pid=29476/tid=29482] ppid=29310 vsize=437960 CPUtime=0 cores=0,2,4,6
/proc/29476/task/29482/stat : 29482 (python) S 29310 29310 28586 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 736739209 448471040 14722 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 514 18446604447093065792 0 0 -1 0 0 0 0
[pid=29483] ppid=29310 vsize=448216 CPUtime=0.99 cores=0,2,4,6
/proc/29483/stat : 29483 (python) S 29310 29310 28586 0 -1 4202560 1201 0 0 0 52 47 0 0 16 0 2 0 736739211 458973184 14726 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/29483/statm: 112054 14726 399 364 0 47933 0
[pid=29485] ppid=29483 vsize=276240 CPUtime=296.65 cores=0,2,4,6
/proc/29485/stat : 29485 (hybridGM3) T 29483 29485 29485 0 -1 4202496 8573 0 0 0 29664 1 0 0 25 0 1 0 736739211 282869760 8303 33554432000 134512640 135204385 4290189376 18446744073709551615 134536585 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/29485/statm: 69060 8303 64 169 0 68890 0
[pid=29483/tid=29508] ppid=29310 vsize=448216 CPUtime=0 cores=0,2,4,6
/proc/29483/task/29508/stat : 29508 (python) S 29310 29310 28586 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 736758919 458973184 14726 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 514 18446604450127337600 0 0 -1 4 0 0 0
[pid=29490] ppid=29310 vsize=458472 CPUtime=0.34 cores=0,2,4,6
/proc/29490/stat : 29490 (python) S 29310 29310 28586 0 -1 4202560 1217 0 0 0 19 15 0 0 19 0 2 0 736748479 469475328 14737 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/29490/statm: 114618 14737 406 364 0 50494 0
[pid=29492] ppid=29490 vsize=467024 CPUtime=97.87 cores=0,2,4,6
/proc/29492/stat : 29492 (TNM) T 29490 29492 29492 0 -1 4202496 6835 0 0 0 9786 1 0 0 25 0 1 0 736748480 478232576 6559 33554432000 134512640 135008748 4293420576 18446744073709551615 134528974 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/29492/statm: 116756 6559 46 122 0 115656 0
[pid=29490/tid=29501] ppid=29310 vsize=458472 CPUtime=0 cores=0,2,4,6
/proc/29490/task/29501/stat : 29501 (python) S 29310 29310 28586 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 736758512 469475328 14737 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 514 18446604449015812480 0 0 -1 4 0 0 0
[pid=29496] ppid=29310 vsize=468728 CPUtime=0.33 cores=0,2,4,6
/proc/29496/stat : 29496 (python) S 29310 29310 28586 0 -1 4202560 1193 0 0 0 18 15 0 0 19 0 2 0 736755223 479977472 14742 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/29496/statm: 117182 14742 407 364 0 53055 0
[pid=29498] ppid=29496 vsize=467024 CPUtime=99.77 cores=0,2,4,6
/proc/29498/stat : 29498 (TNM) T 29496 29498 29498 0 -1 4202496 6831 0 0 0 9976 1 0 0 25 0 1 0 736755223 478232576 6557 33554432000 134512640 135008748 4290566448 18446744073709551615 134527847 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/29498/statm: 116756 6557 46 122 0 115656 0
[pid=29496/tid=29511] ppid=29310 vsize=468728 CPUtime=0 cores=0,2,4,6
/proc/29496/task/29511/stat : 29511 (python) S 29310 29310 28586 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 736765256 479977472 14742 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 514 18446604445752718400 0 0 -1 0 0 0 0
[pid=29502] ppid=29310 vsize=478984 CPUtime=0.34 cores=0,2,4,6
/proc/29502/stat : 29502 (python) S 29310 29310 28586 0 -1 4202560 1207 0 0 0 19 15 0 0 19 0 2 0 736758514 490479616 14746 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/29502/statm: 119746 14746 407 364 0 55616 0
[pid=29504] ppid=29502 vsize=467024 CPUtime=100.25 cores=0,2,4,6
/proc/29504/stat : 29504 (TNM) T 29502 29504 29504 0 -1 4202496 6831 0 0 0 10024 1 0 0 25 0 1 0 736758514 478232576 6557 33554432000 134512640 135008748 4291755328 18446744073709551615 134529183 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/29504/statm: 116756 6557 46 122 0 115656 0
[pid=29502/tid=29516] ppid=29310 vsize=478984 CPUtime=0 cores=0,2,4,6
/proc/29502/task/29516/stat : 29516 (python) S 29310 29310 28586 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 736768547 490479616 14746 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 514 18446604444275062720 0 0 -1 4 0 0 0
[pid=29505] ppid=29310 vsize=489240 CPUtime=0.54 cores=0,2,4,6
/proc/29505/stat : 29505 (python) S 29310 29310 28586 0 -1 4202560 1053 0 0 0 31 23 0 0 15 0 1 0 736758749 500981760 14714 33554432000 4194304 5683644 140734649595888 18446744073709551615 260905414214 0 0 16781312 514 6332236997943329027 0 0 17 4 0 0 0
/proc/29505/statm: 122310 14714 372 364 0 58177 0
[pid=29507] ppid=29505 vsize=276244 CPUtime=162.32 cores=0,2,4,6
/proc/29507/stat : 29507 (hybridGM3) R 29505 29507 29507 0 -1 4202496 8582 0 0 0 16231 1 0 0 25 0 1 0 736758750 282873856 8306 33554432000 134512640 135204385 4291348096 18446744073709551615 134537292 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/29507/statm: 69061 8306 64 169 0 68891 0
[pid=29512] ppid=29310 vsize=499496 CPUtime=0.34 cores=0,2,4,6
/proc/29512/stat : 29512 (python) S 29310 29310 28586 0 -1 4202560 1196 0 0 0 19 15 0 0 19 0 2 0 736765257 511483904 14748 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/29512/statm: 124874 14748 401 364 0 60738 0
[pid=29514] ppid=29512 vsize=467024 CPUtime=99.87 cores=0,2,4,6
/proc/29514/stat : 29514 (TNM) T 29512 29514 29514 0 -1 4202496 6831 0 0 0 9986 1 0 0 25 0 1 0 736765257 478232576 6557 33554432000 134512640 135008748 4294167888 18446744073709551615 134524694 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/29514/statm: 116756 6557 46 122 0 115656 0
[pid=29512/tid=29524] ppid=29310 vsize=499496 CPUtime=0 cores=0,2,4,6
/proc/29512/task/29524/stat : 29524 (python) S 29310 29310 28586 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 736775291 511483904 14748 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 514 18446604451218808896 0 0 -1 2 0 0 0
[pid=29517] ppid=29310 vsize=509752 CPUtime=0.22 cores=0,2,4,6
/proc/29517/stat : 29517 (python) S 29310 29310 28586 0 -1 4202560 1025 0 0 0 10 12 0 0 18 0 1 0 736768548 521986048 14722 33554432000 4194304 5683644 140734649595888 18446744073709551615 260905414214 0 0 16781312 514 6332236997943329026 0 0 17 4 0 0 0
/proc/29517/statm: 127438 14722 372 364 0 63299 0
[pid=29519] ppid=29517 vsize=467024 CPUtime=66.7 cores=0,2,4,6
/proc/29519/stat : 29519 (TNM) R 29517 29519 29519 0 -1 4202496 6834 0 0 0 6669 1 0 0 25 0 1 0 736768549 478232576 6558 33554432000 134512640 135008748 4286829824 18446744073709551615 134527542 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/29519/statm: 116756 6558 46 122 0 115656 0
[pid=29520] ppid=29310 vsize=520008 CPUtime=0.2 cores=0,2,4,6
/proc/29520/stat : 29520 (python) S 29310 29310 28586 0 -1 4202560 1040 0 0 0 10 10 0 0 18 0 1 0 736769055 532488192 14725 33554432000 4194304 5683644 140734649595888 18446744073709551615 260905414214 0 0 16781312 514 6332236997943329027 0 0 17 4 0 0 0
/proc/29520/statm: 130002 14725 371 364 0 65860 0
[pid=29522] ppid=29520 vsize=467024 CPUtime=60.48 cores=0,2,4,6
/proc/29522/stat : 29522 (TNM) R 29520 29522 29522 0 -1 4202496 6837 0 0 0 6047 1 0 0 25 0 1 0 736769055 478232576 6558 33554432000 134512640 135008748 4291229952 18446744073709551615 134528940 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/29522/statm: 116756 6558 46 122 0 115656 0
[pid=29525] ppid=29310 vsize=530264 CPUtime=0 cores=0,2,4,6
/proc/29525/stat : 29525 (python) S 29310 29310 28586 0 -1 4202560 855 0 0 0 0 0 0 0 19 0 1 0 736775292 542990336 14703 33554432000 4194304 5683644 140734649595888 18446744073709551615 260905414214 0 0 16781312 514 1709776319614550015 0 0 17 4 0 0 0
/proc/29525/statm: 132566 14703 345 364 0 68421 0
[pid=29527] ppid=29525 vsize=462800 CPUtime=0.27 cores=0,2,4,6
/proc/29527/stat : 29527 (TNM) R 29525 29527 29527 0 -1 4202496 4186 0 0 0 27 0 0 0 21 0 1 0 736775293 473907200 3907 33554432000 134512640 135008748 4293301344 18446744073709551615 134523393 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/29527/statm: 115700 3907 45 122 0 114600 0
[pid=29310/tid=29321] ppid=29308 vsize=540508 CPUtime=664.2 cores=0,2,4,6
/proc/29310/task/29321/stat : 29321 (python) S 29308 29310 28586 0 -1 4202560 32 20669 0 0 0 0 66251 169 18 0 23 0 736669281 553480192 16272 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 6 0 0 0
[pid=29310/tid=29324] ppid=29308 vsize=540508 CPUtime=664.2 cores=0,2,4,6
/proc/29310/task/29324/stat : 29324 (python) S 29308 29310 28586 0 -1 4202560 28 20669 0 0 0 0 66251 169 18 0 23 0 736669282 553480192 16272 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 2 0 0 0
[pid=29310/tid=29327] ppid=29308 vsize=540508 CPUtime=664.2 cores=0,2,4,6
/proc/29310/task/29327/stat : 29327 (python) S 29308 29310 28586 0 -1 4202560 25 20669 0 0 0 0 66251 169 18 0 23 0 736669283 553480192 16272 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 2 0 0 0
[pid=29310/tid=29330] ppid=29308 vsize=540508 CPUtime=664.2 cores=0,2,4,6
/proc/29310/task/29330/stat : 29330 (python) S 29308 29310 28586 0 -1 4202560 32 20669 0 0 0 0 66251 169 18 0 23 0 736669284 553480192 16272 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 4 0 0 0
[pid=29310/tid=29372] ppid=29308 vsize=540508 CPUtime=664.2 cores=0,2,4,6
/proc/29310/task/29372/stat : 29372 (python) S 29308 29310 28586 0 -1 4202560 32 20669 0 0 0 0 66251 169 18 0 23 0 736688895 553480192 16272 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 4 0 0 0
[pid=29310/tid=29376] ppid=29308 vsize=540508 CPUtime=664.2 cores=0,2,4,6
/proc/29310/task/29376/stat : 29376 (python) S 29308 29310 28586 0 -1 4202560 27 20669 0 0 0 0 66251 169 20 0 23 0 736689399 553480192 16272 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 4 0 0 0
[pid=29310/tid=29390] ppid=29308 vsize=540508 CPUtime=664.2 cores=0,2,4,6
/proc/29310/task/29390/stat : 29390 (python) S 29308 29310 28586 0 -1 4202560 35 20669 0 0 0 0 66251 169 18 0 23 0 736699434 553480192 16272 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 4 0 0 0
[pid=29310/tid=29400] ppid=29308 vsize=540508 CPUtime=664.2 cores=0,2,4,6
/proc/29310/task/29400/stat : 29400 (python) S 29308 29310 28586 0 -1 4202560 42 20669 0 0 0 0 66251 169 18 0 23 0 736706313 553480192 16272 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 4 0 0 0
[pid=29310/tid=29447] ppid=29308 vsize=540508 CPUtime=664.2 cores=0,2,4,6
/proc/29310/task/29447/stat : 29447 (python) S 29308 29310 28586 0 -1 4202560 23 20669 0 0 0 0 66251 169 20 0 23 0 736708593 553480192 16272 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 4 0 0 0
[pid=29310/tid=29461] ppid=29308 vsize=540508 CPUtime=664.2 cores=0,2,4,6
/proc/29310/task/29461/stat : 29461 (python) S 29308 29310 28586 0 -1 4202560 27 20669 0 0 0 0 66251 169 20 0 23 0 736718735 553480192 16272 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 4 0 0 0
[pid=29310/tid=29470] ppid=29308 vsize=540508 CPUtime=664.2 cores=0,2,4,6
/proc/29310/task/29470/stat : 29470 (python) S 29308 29310 28586 0 -1 4202560 28 20669 0 0 0 0 66251 169 20 0 23 0 736728768 553480192 16272 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 4 0 0 0
[pid=29310/tid=29474] ppid=29308 vsize=540508 CPUtime=664.2 cores=0,2,4,6
/proc/29310/task/29474/stat : 29474 (python) S 29308 29310 28586 0 -1 4202560 29 20669 0 0 0 0 66251 169 18 0 23 0 736728905 553480192 16272 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 4 0 0 0
[pid=29310/tid=29477] ppid=29308 vsize=540508 CPUtime=664.2 cores=0,2,4,6
/proc/29310/task/29477/stat : 29477 (python) S 29308 29310 28586 0 -1 4202560 25 20669 0 0 0 0 66251 169 20 0 23 0 736729177 553480192 16272 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 4 0 0 0
[pid=29310/tid=29484] ppid=29308 vsize=540508 CPUtime=664.2 cores=0,2,4,6
/proc/29310/task/29484/stat : 29484 (python) S 29308 29310 28586 0 -1 4202560 34 20669 0 0 0 0 66251 169 18 0 23 0 736739211 553480192 16272 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 4 0 0 0
[pid=29310/tid=29491] ppid=29308 vsize=540508 CPUtime=664.2 cores=0,2,4,6
/proc/29310/task/29491/stat : 29491 (python) S 29308 29310 28586 0 -1 4202560 29 20669 0 0 0 0 66251 169 20 0 23 0 736748479 553480192 16272 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 4 0 0 0
[pid=29310/tid=29497] ppid=29308 vsize=540508 CPUtime=664.2 cores=0,2,4,6
/proc/29310/task/29497/stat : 29497 (python) S 29308 29310 28586 0 -1 4202560 26 20669 0 0 0 0 66251 169 20 0 23 0 736755223 553480192 16272 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 4 0 0 0
[pid=29310/tid=29503] ppid=29308 vsize=540508 CPUtime=664.2 cores=0,2,4,6
/proc/29310/task/29503/stat : 29503 (python) S 29308 29310 28586 0 -1 4202560 24 20669 0 0 0 0 66251 169 20 0 23 0 736758514 553480192 16272 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 4 0 0 0
[pid=29310/tid=29506] ppid=29308 vsize=540508 CPUtime=664.2 cores=0,2,4,6
/proc/29310/task/29506/stat : 29506 (python) S 29308 29310 28586 0 -1 4202560 27 20669 0 0 0 0 66251 169 20 0 23 0 736758750 553480192 16272 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 4 0 0 0
[pid=29310/tid=29513] ppid=29308 vsize=540508 CPUtime=664.2 cores=0,2,4,6
/proc/29310/task/29513/stat : 29513 (python) S 29308 29310 28586 0 -1 4202560 28 20669 0 0 0 0 66251 169 20 0 23 0 736765257 553480192 16272 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 4 0 0 0
[pid=29310/tid=29518] ppid=29308 vsize=540508 CPUtime=664.2 cores=0,2,4,6
/proc/29310/task/29518/stat : 29518 (python) S 29308 29310 28586 0 -1 4202560 26 20669 0 0 0 0 66251 169 20 0 23 0 736768549 553480192 16272 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 4 0 0 0
[pid=29310/tid=29521] ppid=29308 vsize=540508 CPUtime=664.2 cores=0,2,4,6
/proc/29310/task/29521/stat : 29521 (python) S 29308 29310 28586 0 -1 4202560 33 20669 0 0 0 0 66251 169 20 0 23 0 736769055 553480192 16272 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 4 0 0 0
[pid=29310/tid=29526] ppid=29308 vsize=540508 CPUtime=664.2 cores=0,2,4,6
/proc/29310/task/29526/stat : 29526 (python) S 29308 29310 28586 0 -1 4202560 27 20669 0 0 0 0 66251 169 20 0 23 0 736775293 553480192 16272 33554432000 4194304 5683644 140734649595888 18446744073709551615 260917217713 0 0 16781312 2 18446604444659178240 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4226.09
Current children cumulated vsize (KiB) 16605044

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 29310 and gives
#  childrusage.ru_utime.tv_sec=805
#  childrusage.ru_utime.tv_usec=425556
#  childrusage.ru_stime.tv_sec=7
#  childrusage.ru_stime.tv_usec=551851
# CPU time returned by wait4() is 812.977
# while last known CPU time is 4226.09
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1069.25
CPU time (s): 4226.09
CPU user time (s): 4218.44
CPU system time (s): 7.65
CPU usage (%): 395.24
Max. virtual memory (cumulated for all children) (KiB): 16605044

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 805.426
system time used= 7.55185
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 107575
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7489
involuntary context switches= 6158

runsolver used 7.00293 second user time and 16.5475 second system time

The end

Launcher Data 

Begin job on node116 at 2011-05-15 14:59:37
IDJOB=3378270
IDBENCH=82679
IDSOLVER=1859
FILE ID=node116/3378270-1305464377
RUNJOBID= node116-1305463137-28604
PBS_JOBID= 13324982
Free space on /tmp= 72952 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v40000-c168000-S907804052-003-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378270-1305464377/watcher-3378270-1305464377 -o /tmp/evaluation-result-3378270-1305464377/solver-3378270-1305464377 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378270-1305464377.cnf -seed 398813431 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= eb4eadfedd2c4dc9b0ed7231ad324f1f
RANDOM SEED=398813431

node116.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      21187312 kB
Buffers:        122296 kB
Cached:        1493560 kB
SwapCached:      12356 kB
Active:       10450276 kB
Inactive:      1074044 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      21187312 kB
SwapTotal:    67111528 kB
SwapFree:     65730864 kB
Dirty:           35692 kB
Writeback:           0 kB
AnonPages:     9906636 kB
Mapped:          25772 kB
Slab:           113864 kB
PageTables:      57736 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 112142972 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73012 MiB
End job on node116 at 2011-05-15 15:17:29