Trace number 3321768

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (MO) 8547.9 1074.8

General information on the benchmark

Namerandom/large/
unif-k7-r85-v300-c25500-S1286362348-008-UNKNOWN.cnf
MD5SUM63b66954174098a654eacd27832fcf38
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables300
Number of clauses25500
Sum of the clauses size178500
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 525500

Solver Data 

0.35/0.68	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.35/0.68	c INFO: solving HOME/instance-3321768-1304366829.cnf
0.35/0.68	c INFO: computing task features
0.52/0.85	c INFO: computed variable-clause graph statistics
0.52/0.85	c INFO: computed clause balance statistics
1.43/1.72	c INFO: computed variable balance statistics
1.73/2.07	c INFO: computed small-clause counts
1.73/2.07	c INFO: computed Horn-clause counts
1.73/2.10	c INFO: computed variable graph statistics
1.73/2.10	c INFO: collected features for HOME/instance-3321768-1304366829.cnf in 1.41 s
1.83/2.13	c INFO: running gnovelty+2@0 for 97 with 39998 remaining (b = 0.37)
1.83/2.17	c INFO: running gnovelty+2@0 for 97 with 39998 remaining (b = 0.13)
1.92/2.21	c INFO: running adaptg2wsat2009++@0 for 97 with 39998 remaining (b = 0.08)
1.92/2.24	c INFO: running adaptg2wsat2009++@0 for 97 with 39998 remaining (b = 0.07)
1.92/2.28	c INFO: running adaptg2wsat2009++@0 for 97 with 39998 remaining (b = 0.05)
2.02/2.33	c INFO: running gnovelty+2@0 for 97 with 39998 remaining (b = 0.04)
2.02/2.36	c INFO: running FH@0 for 97 with 39998 remaining (b = 0.03)
2.11/2.40	c INFO: running adaptg2wsat2009++@0 for 97 with 39998 remaining (b = 0.03)
801.50/102.36	c INFO: running adaptg2wsat2009++@0 for 97 with 39898 remaining (b = 0.02)
802.16/102.40	c INFO: running FH@0 for 97 with 39898 remaining (b = 0.02)
802.16/102.45	c INFO: running hybridGM3@0 for 194 with 39898 remaining (b = 0.04)
802.62/102.50	c INFO: running gnovelty+2@0 for 97 with 39898 remaining (b = 0.02)
802.62/102.56	c INFO: running hybridGM3@0 for 194 with 39898 remaining (b = 0.03)
802.82/102.61	c INFO: running adaptg2wsat2009++@0 for 97 with 39898 remaining (b = 0.01)
802.82/102.67	c INFO: running hybridGM3@0 for 194 with 39898 remaining (b = 0.03)
802.92/102.73	c INFO: running hybridGM3@0 for 194 with 39897 remaining (b = 0.02)
1600.53/202.58	c INFO: running TNM@0 for 1552 with 39798 remaining (b = 0.19)
1601.29/202.66	c INFO: running FH@0 for 97 with 39798 remaining (b = 0.01)
1601.96/202.78	c INFO: running adaptg2wsat2009++@0 for 97 with 39797 remaining (b = 0.01)
1602.54/202.85	c INFO: running adaptg2wsat2009++@0 for 97 with 39797 remaining (b = 0.01)
2368.34/298.88	c INFO: running hybridGM3@99 for 97 with 39701 remaining (b = 0.01)
2369.12/299.00	c INFO: running hybridGM3@99 for 97 with 39701 remaining (b = 0.01)
2371.56/299.26	c INFO: running hybridGM3@100 for 97 with 39701 remaining (b = 0.01)
2372.27/299.33	c INFO: running hybridGM3@99 for 97 with 39701 remaining (b = 0.01)
2377.86/300.07	c INFO: running adaptg2wsat2009++@0 for 97 with 39700 remaining (b = 0.01)
2400.15/302.88	c INFO: running adaptg2wsat2009++@100 for 97 with 39697 remaining (b = 0.01)
2425.60/306.02	c INFO: running hybridGM3@0 for 194 with 39694 remaining (b = 0.02)
3152.79/397.30	c INFO: running hybridGM3@0 for 194 with 39603 remaining (b = 0.02)
3169.58/399.33	c INFO: running hybridGM3@296 for 97 with 39601 remaining (b = 0.01)
3175.84/400.11	c INFO: running TNM@0 for 1552 with 39600 remaining (b = 0.15)
3175.84/400.19	c INFO: running hybridGM3@295 for 97 with 39600 remaining (b = 0.01)
3178.24/400.49	c INFO: running adaptg2wsat2009++@99 for 97 with 39600 remaining (b = 0.01)
3178.97/400.59	c INFO: running gnovelty+2@0 for 97 with 39600 remaining (b = 0.01)
3928.36/494.57	c INFO: running hybridGM3@296 for 97 with 39506 remaining (b = 0.01)
3935.56/495.41	c INFO: running hybridGM3@392 for 97 with 39505 remaining (b = 0.01)
3953.90/497.72	c INFO: running iPAWS@0 for 97 with 39503 remaining (b = 0.01)
3971.42/499.94	c INFO: running hybridGM3@396 for 97 with 39500 remaining (b = 0.01)
3984.20/501.52	c INFO: running hybridGM3@97 for 97 with 39499 remaining (b = 0.01)
4656.16/585.80	c INFO: running hybridGM3@390 for 97 with 39414 remaining (b = 0.01)
4719.13/593.75	c INFO: running FH@0 for 291 with 39406 remaining (b = 0.02)
4735.07/595.76	c INFO: running adaptg2wsat2009++@100 for 97 with 39404 remaining (b = 0.01)
4746.20/597.18	c INFO: running hybridGM3@97 for 97 with 39403 remaining (b = 0.01)
4753.36/598.02	c INFO: running TNM@0 for 1552 with 39402 remaining (b = 0.11)
4782.86/601.78	c INFO: running hybridGM3@188 for 97 with 39398 remaining (b = 0.01)
5454.96/686.12	c INFO: running FH@100 for 291 with 39314 remaining (b = 0.02)
5501.93/692.02	c INFO: running iPAWS@0 for 97 with 39308 remaining (b = 0.01)
5529.02/695.44	c INFO: running adaptg2wsat2009++@98 for 97 with 39305 remaining (b = 0.01)
5573.60/701.06	c INFO: running gnovelty+2@0 for 97 with 39299 remaining (b = 0.01)
6292.66/791.32	c INFO: running hybridGM3@0 for 194 with 39209 remaining (b = 0.01)
6303.81/792.71	c INFO: running adaptg2wsat2009++@99 for 97 with 39208 remaining (b = 0.01)
6372.29/801.39	c INFO: running FH@0 for 97 with 39199 remaining (b = 0.01)
7065.04/888.37	c INFO: running hybridGM3@0 for 194 with 39112 remaining (b = 0.01)
7077.81/889.98	c INFO: running NCVWr@0 for 485 with 39110 remaining (b = 0.03)
7170.88/901.67	c INFO: running FH@293 for 97 with 39099 remaining (b = 0.01)
7752.93/974.73	c INFO: running iPAWS@0 for 97 with 39026 remaining (b = 0.01)
7857.29/987.86	c INFO: running hybridGM3@96 for 97 with 39012 remaining (b = 0.01)
7921.71/995.99	c INFO: running hybridGM3@0 for 194 with 39004 remaining (b = 0.01)
8543.04/1074.04	c INFO: running hybridGM3@0 for 194 with 38926 remaining (b = 0.01)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3321768-1304366829/watcher-3321768-1304366829 -o /tmp/evaluation-result-3321768-1304366829/solver-3321768-1304366829 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3321768-1304366829.cnf -seed 178252680 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.10 5.72 2.91 2/201 415
/proc/meminfo: memFree=14510792/32951124 swapFree=67111352/67111528
[pid=415] ppid=413 vsize=14644 CPUtime=0 cores=0-7
/proc/415/stat : 415 (python) R 413 415 342 0 -1 4202496 459 0 0 0 0 0 0 0 18 0 1 0 548280834 14995456 368 33554432000 4194304 5683644 140736800973600 18446744073709551615 4521712 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/415/statm: 3661 374 236 364 0 148 0

[startup+0.0107931 s]
/proc/loadavg: 8.10 5.72 2.91 2/201 415
/proc/meminfo: memFree=14510792/32951124 swapFree=67111352/67111528
[pid=415] ppid=413 vsize=15760 CPUtime=0 cores=0-7
/proc/415/stat : 415 (python) S 413 415 342 0 -1 4202496 844 0 0 0 0 0 0 0 17 0 1 0 548280834 16138240 727 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377299456 0 2147483391 16781312 2 18446744071699453118 0 0 17 2 0 0 0
/proc/415/statm: 3940 727 334 364 0 427 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15760

[startup+0.100734 s]
/proc/loadavg: 8.10 5.72 2.91 2/201 415
/proc/meminfo: memFree=14510792/32951124 swapFree=67111352/67111528
[pid=415] ppid=413 vsize=85616 CPUtime=0.05 cores=0-7
/proc/415/stat : 415 (python) R 413 415 342 0 -1 4202496 2629 0 0 0 4 1 0 0 17 0 1 0 548280834 87670784 2347 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377299456 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/415/statm: 21404 2347 713 364 0 1621 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 85616

[startup+0.300722 s]
/proc/loadavg: 8.10 5.72 2.91 2/201 415
/proc/meminfo: memFree=14510792/32951124 swapFree=67111352/67111528
[pid=415] ppid=413 vsize=135744 CPUtime=0.18 cores=0-7
/proc/415/stat : 415 (python) R 413 415 342 0 -1 4202496 5964 933 1 0 13 5 0 0 18 0 1 0 548280834 139001856 4150 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377297413 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/415/statm: 33936 4150 1032 364 0 3122 0
Current children cumulated CPU time (s) 0.18
Current children cumulated vsize (KiB) 135744

[startup+0.700616 s]
/proc/loadavg: 8.10 5.72 2.91 2/201 415
/proc/meminfo: memFree=14510792/32951124 swapFree=67111352/67111528
[pid=415] ppid=413 vsize=279508 CPUtime=0.42 cores=0-7
/proc/415/stat : 415 (python) R 413 415 342 0 -1 4202496 9517 933 1 0 31 11 0 0 17 0 1 0 548280834 286216192 7197 33554432000 4194304 5683644 140736800973600 18446744073709551615 255376662278 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/415/statm: 69877 7197 1761 364 0 5801 0
Current children cumulated CPU time (s) 0.42
Current children cumulated vsize (KiB) 279508

[startup+1.50048 s]
/proc/loadavg: 8.10 5.72 2.91 2/202 420
/proc/meminfo: memFree=14481832/32951124 swapFree=67111352/67111528
[pid=415] ppid=413 vsize=285632 CPUtime=1.23 cores=0-7
/proc/415/stat : 415 (python) R 413 415 342 0 -1 4202496 12503 933 1 0 111 12 0 0 18 0 1 0 548280834 292487168 8793 33554432000 4194304 5683644 140736800973600 18446744073709551615 47182018268440 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/415/statm: 71408 8793 1826 364 0 7332 0
Current children cumulated CPU time (s) 1.23
Current children cumulated vsize (KiB) 285632

[startup+3.10013 s]
/proc/loadavg: 7.53 5.64 2.90 2/202 420
/proc/meminfo: memFree=14481832/32951124 swapFree=67111352/67111528
[pid=415] ppid=413 vsize=362444 CPUtime=2.13 cores=0-7
/proc/415/stat : 415 (python) S 413 415 342 0 -1 4202496 18944 933 1 0 199 14 0 0 21 0 9 0 548280834 371142656 7578 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389128139 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/415/statm: 90611 7578 1891 364 0 26508 0
Current children cumulated CPU time (s) 2.13
Current children cumulated vsize (KiB) 362444
heavy processes:

[startup+6.30154 s]
/proc/loadavg: 7.53 5.64 2.90 9/226 444
/proc/meminfo: memFree=14422460/32951124 swapFree=67111352/67111528
[pid=415] ppid=413 vsize=362444 CPUtime=2.13 cores=0-7
/proc/415/stat : 415 (python) S 413 415 342 0 -1 4202496 18944 933 1 0 199 14 0 0 21 0 9 0 548280834 371142656 7578 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389128139 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/415/statm: 90611 7578 1891 364 0 26508 0
[pid=421] ppid=415 vsize=280756 CPUtime=0 cores=0-7
/proc/421/stat : 421 (python) S 415 415 342 0 -1 4202560 889 0 0 0 0 0 0 0 18 0 1 0 548281048 287494144 6000 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377321487 0 0 16781312 514 4108942027503141124 0 0 17 1 0 0 0
/proc/421/statm: 70189 6000 378 364 0 6107 0
[pid=423] ppid=421 vsize=1019708 CPUtime=4.15 cores=0-7
/proc/423/stat : 423 (gnovelty+2) R 421 423 423 0 -1 4202496 950 0 0 0 415 0 0 0 25 0 1 0 548281048 1044180992 689 33554432000 134512640 135663704 4294167024 18446744073709551615 134545631 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/423/statm: 254927 689 113 282 0 254643 0
[pid=424] ppid=415 vsize=291080 CPUtime=0 cores=0-7
/proc/424/stat : 424 (python) S 415 415 342 0 -1 4202560 879 0 0 0 0 0 0 0 18 0 1 0 548281052 298065920 5998 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377321542 0 0 16781312 514 4108945875793838338 0 0 17 4 0 0 0
/proc/424/statm: 72770 5998 372 364 0 8685 0
[pid=426] ppid=424 vsize=1019708 CPUtime=4.12 cores=0-7
/proc/426/stat : 426 (gnovelty+2) R 424 426 426 0 -1 4202496 953 0 0 0 412 0 0 0 25 0 1 0 548281052 1044180992 689 33554432000 134512640 135663704 4288055728 18446744073709551615 134525276 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/426/statm: 254927 689 113 282 0 254643 0
[pid=427] ppid=415 vsize=301268 CPUtime=0 cores=0-7
/proc/427/stat : 427 (python) S 415 415 342 0 -1 4202560 918 0 0 0 0 0 0 0 18 0 1 0 548281055 308498432 6019 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377321542 0 0 16781312 514 3988205480290779395 0 0 17 4 0 0 0
/proc/427/statm: 75317 6019 372 364 0 11229 0
[pid=429] ppid=427 vsize=420220 CPUtime=4.05 cores=0-7
/proc/429/stat : 429 (adaptg2wsat2009) R 427 429 429 0 -1 4202496 1662 0 0 0 405 0 0 0 25 0 1 0 548281056 430305280 1399 33554432000 134512640 135107411 4287286048 18446744073709551615 134531600 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/429/statm: 105055 1399 59 146 0 103931 0
[pid=430] ppid=415 vsize=311592 CPUtime=0 cores=0-7
/proc/430/stat : 430 (python) S 415 415 342 0 -1 4202560 933 0 0 0 0 0 0 0 18 0 1 0 548281059 319070208 6023 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377321542 0 0 16781312 514 4580024211010978052 0 0 17 2 0 0 0
/proc/430/statm: 77898 6023 372 364 0 13807 0
[pid=432] ppid=430 vsize=420220 CPUtime=4.02 cores=0-7
/proc/432/stat : 432 (adaptg2wsat2009) R 430 432 432 0 -1 4202496 1663 0 0 0 402 0 0 0 25 0 1 0 548281059 430305280 1399 33554432000 134512640 135107411 4291839104 18446744073709551615 134532277 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/432/statm: 105055 1399 59 146 0 103931 0
[pid=433] ppid=415 vsize=321784 CPUtime=0 cores=0-7
/proc/433/stat : 433 (python) R 415 415 342 0 -1 4202560 931 0 0 0 0 0 0 0 18 0 1 0 548281063 329506816 6026 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377301136 0 0 16781312 514 0 0 0 17 0 0 0 0
/proc/433/statm: 80445 6026 371 364 0 16351 0
[pid=435] ppid=433 vsize=420220 CPUtime=3.99 cores=0-7
/proc/435/stat : 435 (adaptg2wsat2009) R 433 435 435 0 -1 4202496 1582 0 0 0 399 0 0 0 25 0 1 0 548281064 430305280 1316 33554432000 134512640 135107411 4293446528 18446744073709551615 134537133 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/435/statm: 105055 1316 58 146 0 103931 0
[pid=436] ppid=415 vsize=332644 CPUtime=0 cores=0-7
/proc/436/stat : 436 (python) S 415 415 342 0 -1 4202560 763 0 0 0 0 0 0 0 18 0 1 0 548281067 340627456 6008 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377321542 0 0 16781312 514 1788686344994652420 0 0 17 0 0 0 0
/proc/436/statm: 83161 6008 349 364 0 19064 0
[pid=438] ppid=436 vsize=1019708 CPUtime=3.96 cores=0-7
/proc/438/stat : 438 (gnovelty+2) R 436 438 438 0 -1 4202496 954 0 0 0 396 0 0 0 25 0 1 0 548281067 1044180992 690 33554432000 134512640 135663704 4291178544 18446744073709551615 134525316 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/438/statm: 254927 690 113 282 0 254643 0
[pid=439] ppid=415 vsize=342360 CPUtime=0 cores=0-7
/proc/439/stat : 439 (python) S 415 415 342 0 -1 4202560 767 0 0 0 0 0 0 0 18 0 1 0 548281071 350576640 6012 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377321542 0 0 16781312 514 1788690193285349635 0 0 17 6 0 0 0
/proc/439/statm: 85590 6012 349 364 0 21490 0
[pid=441] ppid=439 vsize=247956 CPUtime=3.9 cores=0-7
/proc/441/stat : 441 (FH) R 439 441 441 0 -1 4202496 1249 0 0 0 390 0 0 0 25 0 1 0 548281071 253906944 987 33554432000 134512640 134936988 4288284304 18446744073709551615 134523497 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/441/statm: 61989 987 37 104 0 61493 0
[pid=442] ppid=415 vsize=353224 CPUtime=0 cores=0-7
/proc/442/stat : 442 (python) S 415 415 342 0 -1 4202560 797 0 0 0 0 0 0 0 18 0 1 0 548281075 361701376 6039 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377321542 0 0 16781312 514 14846525469125214466 0 0 17 0 0 0 0
/proc/442/statm: 88306 6039 355 364 0 24203 0
[pid=444] ppid=442 vsize=419820 CPUtime=3.87 cores=0-7
/proc/444/stat : 444 (adaptg2wsat2009) R 442 444 444 0 -1 4202496 1443 0 0 0 387 0 0 0 25 0 1 0 548281075 429895680 1172 33554432000 134512640 135107411 4289628864 18446744073709551615 134523300 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/444/statm: 104955 1172 49 146 0 103831 0
[pid=415/tid=422] ppid=413 vsize=362444 CPUtime=0 cores=0-7
/proc/415/task/422/stat : 422 (python) S 413 415 342 0 -1 4202560 26 933 0 0 0 0 0 0 20 0 9 0 548281048 371142656 7578 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 0 0 0 0
[pid=415/tid=425] ppid=413 vsize=362444 CPUtime=0 cores=0-7
/proc/415/task/425/stat : 425 (python) S 413 415 342 0 -1 4202560 21 933 0 0 0 0 0 0 20 0 9 0 548281052 371142656 7578 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 2 0 0 0
[pid=415/tid=428] ppid=413 vsize=362444 CPUtime=0 cores=0-7
/proc/415/task/428/stat : 428 (python) S 413 415 342 0 -1 4202560 23 933 0 0 0 0 0 0 22 0 9 0 548281055 371142656 7578 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 0 0 0 0
[pid=415/tid=431] ppid=413 vsize=362444 CPUtime=0 cores=0-7
/proc/415/task/431/stat : 431 (python) S 413 415 342 0 -1 4202560 27 933 0 0 0 0 0 0 20 0 9 0 548281059 371142656 7578 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 4 0 0 0
[pid=415/tid=434] ppid=413 vsize=362444 CPUtime=0 cores=0-7
/proc/415/task/434/stat : 434 (python) S 413 415 342 0 -1 4202560 30 933 0 0 0 0 0 0 20 0 9 0 548281063 371142656 7578 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 6 0 0 0
[pid=415/tid=437] ppid=413 vsize=362444 CPUtime=0 cores=0-7
/proc/415/task/437/stat : 437 (python) S 413 415 342 0 -1 4202560 21 933 0 0 0 0 0 0 22 0 9 0 548281067 371142656 7578 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 0 0 0 0
[pid=415/tid=440] ppid=413 vsize=362444 CPUtime=0 cores=0-7
/proc/415/task/440/stat : 440 (python) S 413 415 342 0 -1 4202560 25 933 0 0 0 0 0 0 20 0 9 0 548281071 371142656 7578 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 6 0 0 0
[pid=415/tid=443] ppid=413 vsize=362444 CPUtime=0 cores=0-7
/proc/415/task/443/stat : 443 (python) S 413 415 342 0 -1 4202560 86 933 0 0 0 0 0 0 23 0 9 0 548281075 371142656 7578 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 34.19
Current children cumulated vsize (KiB) 7884712

[startup+12.7014 s]
/proc/loadavg: 7.61 5.72 2.95 9/226 444
/proc/meminfo: memFree=14416296/32951124 swapFree=67111352/67111528
[pid=415] ppid=413 vsize=362444 CPUtime=2.13 cores=0-7
/proc/415/stat : 415 (python) S 413 415 342 0 -1 4202496 18944 933 1 0 199 14 0 0 21 0 9 0 548280834 371142656 7578 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389128139 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/415/statm: 90611 7578 1891 364 0 26508 0
[pid=421] ppid=415 vsize=280756 CPUtime=0.01 cores=0-7
/proc/421/stat : 421 (python) S 415 415 342 0 -1 4202560 892 0 0 0 0 1 0 0 18 0 1 0 548281048 287494144 6000 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377321487 0 0 16781312 514 7523192808778203396 0 0 17 1 0 0 0
/proc/421/statm: 70189 6000 378 364 0 6107 0
[pid=423] ppid=421 vsize=1019708 CPUtime=10.55 cores=0-7
/proc/423/stat : 423 (gnovelty+2) R 421 423 423 0 -1 4202496 950 0 0 0 1055 0 0 0 25 0 1 0 548281048 1044180992 689 33554432000 134512640 135663704 4294167024 18446744073709551615 134525276 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/423/statm: 254927 689 113 282 0 254643 0
[pid=424] ppid=415 vsize=291080 CPUtime=0.01 cores=0-7
/proc/424/stat : 424 (python) S 415 415 342 0 -1 4202560 884 0 0 0 1 0 0 0 18 0 1 0 548281052 298065920 5998 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377321542 0 0 16781312 514 7523192808778203394 0 0 17 4 0 0 0
/proc/424/statm: 72770 5998 372 364 0 8685 0
[pid=426] ppid=424 vsize=1019708 CPUtime=10.52 cores=0-7
/proc/426/stat : 426 (gnovelty+2) R 424 426 426 0 -1 4202496 953 0 0 0 1052 0 0 0 25 0 1 0 548281052 1044180992 689 33554432000 134512640 135663704 4288055728 18446744073709551615 134524993 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/426/statm: 254927 689 113 282 0 254643 0
[pid=427] ppid=415 vsize=301268 CPUtime=0 cores=0-7
/proc/427/stat : 427 (python) S 415 415 342 0 -1 4202560 919 0 0 0 0 0 0 0 18 0 1 0 548281055 308498432 6019 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377321542 0 0 16781312 514 7523192808778203395 0 0 17 4 0 0 0
/proc/427/statm: 75317 6019 372 364 0 11229 0
[pid=429] ppid=427 vsize=420220 CPUtime=10.4 cores=0-7
/proc/429/stat : 429 (adaptg2wsat2009) R 427 429 429 0 -1 4202496 1662 0 0 0 1040 0 0 0 25 0 1 0 548281056 430305280 1399 33554432000 134512640 135107411 4287286048 18446744073709551615 134532277 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/429/statm: 105055 1399 59 146 0 103931 0
[pid=430] ppid=415 vsize=311592 CPUtime=0.02 cores=0-7
/proc/430/stat : 430 (python) S 415 415 342 0 -1 4202560 940 0 0 0 1 1 0 0 18 0 1 0 548281059 319070208 6023 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377321542 0 0 16781312 514 7523192808778203396 0 0 17 2 0 0 0
/proc/430/statm: 77898 6023 372 364 0 13807 0
[pid=432] ppid=430 vsize=420220 CPUtime=10.4 cores=0-7
/proc/432/stat : 432 (adaptg2wsat2009) R 430 432 432 0 -1 4202496 1663 0 0 0 1040 0 0 0 25 0 1 0 548281059 430305280 1399 33554432000 134512640 135107411 4291839104 18446744073709551615 134531428 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/432/statm: 105055 1399 59 146 0 103931 0
[pid=433] ppid=415 vsize=321780 CPUtime=0 cores=0-7
/proc/433/stat : 433 (python) S 415 415 342 0 -1 4202560 940 0 0 0 0 0 0 0 18 0 1 0 548281063 329502720 6027 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377321542 0 0 16781312 514 7523192808778203393 0 0 17 0 0 0 0
/proc/433/statm: 80445 6027 372 364 0 16351 0
[pid=435] ppid=433 vsize=420220 CPUtime=10.38 cores=0-7
/proc/435/stat : 435 (adaptg2wsat2009) R 433 435 435 0 -1 4202496 1664 0 0 0 1038 0 0 0 25 0 1 0 548281064 430305280 1398 33554432000 134512640 135107411 4293446528 18446744073709551615 134531435 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/435/statm: 105055 1398 59 146 0 103931 0
[pid=436] ppid=415 vsize=332644 CPUtime=0.01 cores=0-7
/proc/436/stat : 436 (python) S 415 415 342 0 -1 4202560 939 0 0 0 0 1 0 0 18 0 1 0 548281067 340627456 6031 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377321542 0 0 16781312 514 7523192808778203396 0 0 17 0 0 0 0
/proc/436/statm: 83161 6031 372 364 0 19064 0
[pid=438] ppid=436 vsize=1019708 CPUtime=10.34 cores=0-7
/proc/438/stat : 438 (gnovelty+2) R 436 438 438 0 -1 4202496 954 0 0 0 1034 0 0 0 25 0 1 0 548281067 1044180992 690 33554432000 134512640 135663704 4291178544 18446744073709551615 134525255 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/438/statm: 254927 690 113 282 0 254643 0
[pid=439] ppid=415 vsize=342360 CPUtime=0.01 cores=0-7
/proc/439/stat : 439 (python) S 415 415 342 0 -1 4202560 947 0 0 0 1 0 0 0 18 0 1 0 548281071 350576640 6035 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377321542 0 0 16781312 514 7523192808778203395 0 0 17 6 0 0 0

################
# More data... #
################

[pid=524] ppid=522 vsize=1019708 CPUtime=98.93 cores=0-7
/proc/524/stat : 524 (gnovelty+2) T 522 524 524 0 -1 4202496 976 0 0 0 9893 0 0 0 25 0 1 0 548320894 1044180992 690 33554432000 134512640 135663704 4293101632 18446744073709551615 134525260 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/524/statm: 254927 690 113 282 0 254643 0
[pid=522/tid=530] ppid=415 vsize=562840 CPUtime=0 cores=0-7
/proc/522/task/530/stat : 530 (python) S 415 415 342 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 548330820 576348160 6597 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 514 18446604471138828480 0 0 -1 2 0 0 0
[pid=527] ppid=415 vsize=573096 CPUtime=0.18 cores=0-7
/proc/527/stat : 527 (python) S 415 415 342 0 -1 4202560 1279 0 0 0 11 7 0 0 19 0 2 0 548330607 586850304 6664 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389128139 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/527/statm: 143274 6664 408 364 0 79126 0
[pid=529] ppid=527 vsize=335996 CPUtime=99.82 cores=0-7
/proc/529/stat : 529 (iPAWS) T 527 529 529 0 -1 4202496 1819 0 0 0 9982 0 0 0 25 0 1 0 548330607 344059904 1534 33554432000 134512640 135114423 4293291536 18446744073709551615 134515629 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/529/statm: 83999 1534 59 147 0 83851 0
[pid=527/tid=536] ppid=415 vsize=573096 CPUtime=0 cores=0-7
/proc/527/task/536/stat : 536 (python) S 415 415 342 0 -1 4202560 44 0 0 0 0 0 0 0 19 0 2 0 548340628 586850304 6664 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 514 18446604471145539648 0 0 -1 0 0 0 0
[pid=537] ppid=415 vsize=583216 CPUtime=0.87 cores=0-7
/proc/537/stat : 537 (python) S 415 415 342 0 -1 4202560 984 0 0 0 46 41 0 0 18 0 1 0 548340637 597213184 6718 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377321542 0 0 16781312 514 7523192808778203397 0 0 17 1 0 0 0
/proc/537/statm: 145804 6718 378 364 0 81653 0
[pid=539] ppid=537 vsize=458348 CPUtime=473.2 cores=0-7
/proc/539/stat : 539 (TNM) R 537 539 539 0 -1 4202496 1519 0 0 0 47320 0 0 0 25 0 1 0 548340637 469348352 1240 33554432000 134512640 135008748 4289906000 18446744073709551615 134527320 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/539/statm: 114587 1240 46 122 0 113487 0
[pid=542] ppid=415 vsize=594012 CPUtime=0.18 cores=0-7
/proc/542/stat : 542 (python) S 415 415 342 0 -1 4202560 1241 0 0 0 8 10 0 0 19 0 2 0 548350037 608268288 6767 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389128139 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/542/statm: 148503 6767 401 364 0 84349 0
[pid=544] ppid=542 vsize=335996 CPUtime=98.94 cores=0-7
/proc/544/stat : 544 (iPAWS) T 542 544 544 0 -1 4202496 1820 0 0 0 9894 0 0 0 25 0 1 0 548350037 344059904 1536 33554432000 134512640 135114423 4288823488 18446744073709551615 134516646 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/544/statm: 83999 1536 59 147 0 83851 0
[pid=542/tid=551] ppid=415 vsize=594012 CPUtime=0 cores=0-7
/proc/542/task/551/stat : 551 (python) S 415 415 342 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 548359957 608268288 6767 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 514 18446604461464398016 0 0 -1 0 0 0 0
[pid=546] ppid=415 vsize=603728 CPUtime=0.18 cores=0-7
/proc/546/stat : 546 (python) S 415 415 342 0 -1 4202560 1296 0 0 0 10 8 0 0 19 0 2 0 548350941 618217472 6921 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389128139 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/546/statm: 150932 6921 400 364 0 86775 0
[pid=548] ppid=546 vsize=1019708 CPUtime=99.67 cores=0-7
/proc/548/stat : 548 (gnovelty+2) T 546 548 548 0 -1 4202496 969 0 0 0 9967 0 0 0 25 0 1 0 548350942 1044180992 689 33554432000 134512640 135663704 4290314880 18446744073709551615 134524894 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/548/statm: 254927 689 113 282 0 254643 0
[pid=546/tid=555] ppid=415 vsize=603728 CPUtime=0 cores=0-7
/proc/546/task/555/stat : 555 (python) S 415 415 342 0 -1 4202560 221 0 0 0 0 0 0 0 19 0 2 0 548360963 618217472 6921 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 514 18446604454662591808 0 0 -1 0 0 0 0
[pid=552] ppid=415 vsize=613984 CPUtime=0.51 cores=0-7
/proc/552/stat : 552 (python) S 415 415 342 0 -1 4202560 1193 0 0 0 29 22 0 0 18 0 2 0 548359966 628719616 6876 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377321542 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/552/statm: 153496 6876 400 364 0 89336 0
[pid=554] ppid=552 vsize=268428 CPUtime=282.8 cores=0-7
/proc/554/stat : 554 (hybridGM3) R 552 554 554 0 -1 4202496 5407 0 0 0 28280 0 0 0 25 0 1 0 548359967 274870272 5127 33554432000 134512640 135204385 4292102048 18446744073709551615 134537792 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/554/statm: 67107 5127 63 169 0 66937 0
[pid=552/tid=574] ppid=415 vsize=613984 CPUtime=0 cores=0-7
/proc/552/task/574/stat : 574 (python) S 415 415 342 0 -1 4202560 99 0 0 0 0 0 0 0 19 0 2 0 548379610 628719616 6876 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 514 18446604471165531392 0 0 -1 4 0 0 0
[pid=556] ppid=415 vsize=624240 CPUtime=0.36 cores=0-7
/proc/556/stat : 556 (python) S 415 415 342 0 -1 4202560 1201 0 0 0 20 16 0 0 18 0 2 0 548360973 639221760 6990 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389128139 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/556/statm: 156060 6990 401 364 0 91897 0
[pid=558] ppid=556 vsize=249872 CPUtime=194.07 cores=0-7
/proc/558/stat : 558 (FH) T 556 558 558 0 -1 4202496 1546 0 0 0 19405 2 0 0 25 0 1 0 548360973 255868928 1264 33554432000 134512640 134936988 4288423584 18446744073709551615 134545104 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/558/statm: 62468 1264 41 104 0 61972 0
[pid=556/tid=568] ppid=415 vsize=624240 CPUtime=0 cores=0-7
/proc/556/task/568/stat : 568 (python) S 415 415 342 0 -1 4202560 103 0 0 0 0 0 0 0 18 0 2 0 548370991 639221760 6990 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 514 18446604471165530496 0 0 -1 0 0 0 0
[pid=562] ppid=415 vsize=634496 CPUtime=0.33 cores=0-7
/proc/562/stat : 562 (python) S 415 415 342 0 -1 4202560 1020 0 0 0 19 14 0 0 18 0 1 0 548369672 649723904 7026 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377321542 0 0 16781312 514 7523192808778203396 0 0 17 1 0 0 0
/proc/562/statm: 158624 7026 378 364 0 94458 0
[pid=564] ppid=562 vsize=268420 CPUtime=178.73 cores=0-7
/proc/564/stat : 564 (hybridGM3) R 562 564 564 0 -1 4202496 5415 0 0 0 17872 1 0 0 25 0 1 0 548369673 274862080 5126 33554432000 134512640 135204385 4293603888 18446744073709551615 134538264 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/564/statm: 67105 5126 63 169 0 66935 0
[pid=565] ppid=415 vsize=644752 CPUtime=0.34 cores=0-7
/proc/565/stat : 565 (python) S 415 415 342 0 -1 4202560 1019 0 0 0 20 14 0 0 18 0 1 0 548369832 660226048 7001 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377321542 0 0 16781312 514 7523192808778203399 0 0 17 1 0 0 0
/proc/565/statm: 161188 7001 379 364 0 97019 0
[pid=567] ppid=565 vsize=419772 CPUtime=182.91 cores=0-7
/proc/567/stat : 567 (NCVWr) R 565 567 567 0 -1 4202496 1564 0 0 0 18291 0 0 0 25 0 1 0 548369833 429846528 1281 33554432000 134512640 135113558 4287533488 18446744073709551615 134534202 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/567/statm: 104943 1281 58 147 0 103818 0
[pid=571] ppid=415 vsize=655008 CPUtime=0.18 cores=0-7
/proc/571/stat : 571 (python) S 415 415 342 0 -1 4202560 1244 0 0 0 9 9 0 0 19 0 2 0 548378308 670728192 7068 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389128139 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/571/statm: 163752 7068 407 364 0 99580 0
[pid=573] ppid=571 vsize=335996 CPUtime=99.09 cores=0-7
/proc/573/stat : 573 (iPAWS) T 571 573 573 0 -1 4202496 1821 0 0 0 9909 0 0 0 25 0 1 0 548378308 344059904 1534 33554432000 134512640 135114423 4287442848 18446744073709551615 134514777 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/573/statm: 83999 1534 59 147 0 83851 0
[pid=571/tid=581] ppid=415 vsize=655008 CPUtime=0 cores=0-7
/proc/571/task/581/stat : 581 (python) S 415 415 342 0 -1 4202560 104 0 0 0 0 0 0 0 19 0 2 0 548388227 670728192 7068 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 514 18446604443484519232 0 0 -1 0 0 0 0
[pid=575] ppid=415 vsize=665264 CPUtime=0.14 cores=0-7
/proc/575/stat : 575 (python) S 415 415 342 0 -1 4202560 1042 0 0 0 7 7 0 0 18 0 1 0 548380433 681230336 7044 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377321542 0 0 16781312 514 7523192808778203394 0 0 17 0 0 0 0
/proc/575/statm: 166316 7044 378 364 0 102141 0
[pid=577] ppid=575 vsize=268424 CPUtime=78.33 cores=0-7
/proc/577/stat : 577 (hybridGM3) R 575 577 577 0 -1 4202496 5413 0 0 0 7832 1 0 0 25 0 1 0 548380434 274866176 5127 33554432000 134512640 135204385 4289254976 18446744073709551615 134538061 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/577/statm: 67106 5127 63 169 0 66936 0
[pid=582] ppid=415 vsize=675520 CPUtime=0 cores=0-7
/proc/582/stat : 582 (python) S 415 415 342 0 -1 4202560 940 0 0 0 0 0 0 0 19 0 1 0 548388238 691732480 6957 33554432000 4194304 5683644 140736800973600 18446744073709551615 255377321542 0 0 16781312 514 16865637561169608703 0 0 17 1 0 0 0
/proc/582/statm: 168880 6957 349 364 0 104702 0
[pid=584] ppid=582 vsize=268424 CPUtime=0.56 cores=0-7
/proc/584/stat : 584 (hybridGM3) R 582 584 584 0 -1 4202496 5419 0 0 0 56 0 0 0 22 0 1 0 548388239 274866176 5127 33554432000 134512640 135204385 4290675744 18446744073709551615 134538215 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/584/statm: 67106 5127 63 169 0 66936 0
[pid=415/tid=422] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/422/stat : 422 (python) S 413 415 342 0 -1 4202560 26 933 0 0 0 0 0 0 20 0 34 0 548281048 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 0 0 0 0
[pid=415/tid=425] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/425/stat : 425 (python) S 413 415 342 0 -1 4202560 21 933 0 0 0 0 0 0 20 0 34 0 548281052 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 2 0 0 0
[pid=415/tid=428] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/428/stat : 428 (python) S 413 415 342 0 -1 4202560 31 933 0 0 0 0 0 0 18 0 34 0 548281055 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 0 0 0 0
[pid=415/tid=431] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/431/stat : 431 (python) S 413 415 342 0 -1 4202560 31 933 0 0 0 0 0 0 18 0 34 0 548281059 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 4 0 0 0
[pid=415/tid=434] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/434/stat : 434 (python) S 413 415 342 0 -1 4202560 38 933 0 0 0 0 0 0 18 0 34 0 548281063 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 6 0 0 0
[pid=415/tid=437] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/437/stat : 437 (python) S 413 415 342 0 -1 4202560 21 933 0 0 0 0 0 0 22 0 34 0 548281067 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 0 0 0 0
[pid=415/tid=440] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/440/stat : 440 (python) S 413 415 342 0 -1 4202560 29 933 0 0 0 0 0 0 18 0 34 0 548281071 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 6 0 0 0
[pid=415/tid=443] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/443/stat : 443 (python) S 413 415 342 0 -1 4202560 89 933 0 0 0 0 0 0 18 0 34 0 548281075 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 0 0 0 0
[pid=415/tid=452] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/452/stat : 452 (python) S 413 415 342 0 -1 4202560 30 933 0 0 0 0 0 0 18 0 34 0 548291070 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 0 0 0 0
[pid=415/tid=456] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/456/stat : 456 (python) S 413 415 342 0 -1 4202560 32 933 0 0 0 0 0 0 18 0 34 0 548291075 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 4 0 0 0
[pid=415/tid=460] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/460/stat : 460 (python) S 413 415 342 0 -1 4202560 42 933 0 0 0 0 0 0 18 0 34 0 548291080 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 2 0 0 0
[pid=415/tid=472] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/472/stat : 472 (python) S 413 415 342 0 -1 4202560 24 933 0 0 0 0 0 0 25 0 34 0 548291085 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 5 0 0 0
[pid=415/tid=479] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/479/stat : 479 (python) S 413 415 342 0 -1 4202560 42 933 0 0 0 0 0 0 18 0 34 0 548291090 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 0 0 0 0
[pid=415/tid=482] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/482/stat : 482 (python) S 413 415 342 0 -1 4202560 36 933 0 0 0 0 0 0 18 0 34 0 548291096 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 7 0 0 0
[pid=415/tid=485] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/485/stat : 485 (python) S 413 415 342 0 -1 4202560 114 933 0 0 0 0 0 0 18 0 34 0 548291102 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 1 0 0 0
[pid=415/tid=488] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/488/stat : 488 (python) S 413 415 342 0 -1 4202560 47 933 0 0 0 0 0 0 18 0 34 0 548291108 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 4 0 0 0
[pid=415/tid=494] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/494/stat : 494 (python) S 413 415 342 0 -1 4202560 26 933 0 0 0 0 0 0 19 0 34 0 548301092 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 4 0 0 0
[pid=415/tid=498] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/498/stat : 498 (python) S 413 415 342 0 -1 4202560 28 933 0 0 0 0 0 0 20 0 34 0 548301100 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 4 0 0 0
[pid=415/tid=511] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/511/stat : 511 (python) S 413 415 342 0 -1 4202560 37 933 0 0 0 0 0 0 18 0 34 0 548311436 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 1 0 0 0
[pid=415/tid=516] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/516/stat : 516 (python) S 413 415 342 0 -1 4202560 33 933 0 0 0 0 0 0 18 0 34 0 548320565 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 1 0 0 0
[pid=415/tid=520] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/520/stat : 520 (python) S 413 415 342 0 -1 4202560 27 933 0 0 0 0 0 0 19 0 34 0 548320846 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 1 0 0 0
[pid=415/tid=523] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/523/stat : 523 (python) S 413 415 342 0 -1 4202560 34 933 0 0 0 0 0 0 20 0 34 0 548320894 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 1 0 0 0
[pid=415/tid=528] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/528/stat : 528 (python) S 413 415 342 0 -1 4202560 31 933 0 0 0 0 0 0 20 0 34 0 548330607 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 0 0 0 0
[pid=415/tid=538] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/538/stat : 538 (python) S 413 415 342 0 -1 4202560 48 933 0 0 0 0 0 0 20 0 34 0 548340637 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 0 0 0 0
[pid=415/tid=543] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/543/stat : 543 (python) S 413 415 342 0 -1 4202560 32 933 0 0 0 0 0 0 18 0 34 0 548350037 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 0 0 0 0
[pid=415/tid=547] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/547/stat : 547 (python) S 413 415 342 0 -1 4202560 175 933 0 0 0 0 0 0 19 0 34 0 548350941 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 1 0 0 0
[pid=415/tid=553] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/553/stat : 553 (python) S 413 415 342 0 -1 4202560 40 933 0 0 0 0 0 0 18 0 34 0 548359966 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 0 0 0 0
[pid=415/tid=557] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/557/stat : 557 (python) S 413 415 342 0 -1 4202560 42 933 0 0 0 0 0 0 18 0 34 0 548360973 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 1 0 0 0
[pid=415/tid=563] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/563/stat : 563 (python) S 413 415 342 0 -1 4202560 38 933 0 0 0 0 0 0 20 0 34 0 548369672 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 0 0 0 0
[pid=415/tid=566] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/566/stat : 566 (python) S 413 415 342 0 -1 4202560 35 933 0 0 0 0 0 0 20 0 34 0 548369833 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 1 0 0 0
[pid=415/tid=572] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/572/stat : 572 (python) S 413 415 342 0 -1 4202560 37 933 0 0 0 0 0 0 20 0 34 0 548378308 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 1 0 0 0
[pid=415/tid=576] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/576/stat : 576 (python) S 413 415 342 0 -1 4202560 39 933 0 0 0 0 0 0 20 0 34 0 548380434 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 1 0 0 0
[pid=415/tid=583] ppid=413 vsize=685764 CPUtime=0 cores=0-7
/proc/415/task/583/stat : 583 (python) S 413 415 342 0 -1 4202560 37 933 0 0 0 0 0 0 19 0 34 0 548388238 702222336 8554 33554432000 4194304 5683644 140736800973600 18446744073709551615 255389125041 0 0 16781312 2 18446604453329156032 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 8547.9
Current children cumulated vsize (KiB) 31965464

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 415 and gives
#  childrusage.ru_utime.tv_sec=5
#  childrusage.ru_utime.tv_usec=919100
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=250961
# CPU time returned by wait4() is 6.17006
# while last known CPU time is 8547.9
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1074.8
CPU time (s): 8547.9
CPU user time (s): 8540.63
CPU system time (s): 7.27
CPU usage (%): 795.3
Max. virtual memory (cumulated for all children) (KiB): 31965464

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 5.9191
system time used= 0.250961
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 58605
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 3248
involuntary context switches= 131

runsolver used 9.56954 second user time and 21.9357 second system time

The end

Launcher Data 

Begin job on node135 at 2011-05-02 22:07:09
IDJOB=3321768
IDBENCH=82546
IDSOLVER=1712
FILE ID=node135/3321768-1304366829
RUNJOBID= node135-1304366465-356
PBS_JOBID= 13181737
Free space on /tmp= 72936 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k7-r85-v300-c25500-S1286362348-008-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3321768-1304366829/watcher-3321768-1304366829 -o /tmp/evaluation-result-3321768-1304366829/solver-3321768-1304366829 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3321768-1304366829.cnf -seed 178252680 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 63b66954174098a654eacd27832fcf38
RANDOM SEED=178252680

node135.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 6640.14
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.78
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      14510824 kB
Buffers:        997072 kB
Cached:        3870444 kB
SwapCached:        172 kB
Active:       13696544 kB
Inactive:      4181176 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      14510824 kB
SwapTotal:    67111528 kB
SwapFree:     67111352 kB
Dirty:           32648 kB
Writeback:           0 kB
AnonPages:    13010028 kB
Mapped:          17712 kB
Slab:           469828 kB
PageTables:      31956 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 14072804 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72904 MiB
End job on node135 at 2011-05-02 22:25:07