Trace number 3378196

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (MO) 3956.76 1006.53

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v50000-c210000-S758437099-099-UNKNOWN.cnf
MD5SUM94d10ca78c6cee5b0badfb96b777ac08
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables49999
Number of clauses210000
Sum of the clauses size630000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3210000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.43/5.56	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.43/5.56	c INFO: solving HOME/instance-3378196-1307741596.cnf
0.43/5.56	c INFO: computing task features
1.08/6.29	c INFO: computed variable-clause graph statistics
1.18/6.32	c INFO: computed clause balance statistics
4.98/10.17	c INFO: computed variable balance statistics
8.18/13.31	c INFO: computed small-clause counts
8.18/13.32	c INFO: computed Horn-clause counts
11.29/16.41	c INFO: computed variable graph statistics
11.29/16.41	c INFO: collected features for HOME/instance-3378196-1307741596.cnf in 10.82 s
11.29/16.42	c INFO: running hybridGM3@0 for 97 with 1189 remaining (b = 0.33)
11.29/16.43	c INFO: running hybridGM3@0 for 97 with 1189 remaining (b = 0.21)
11.29/16.44	c INFO: running hybridGM3@0 for 97 with 1189 remaining (b = 0.14)
11.29/16.45	c INFO: running hybridGM3@0 for 194 with 1189 remaining (b = 0.17)
411.35/116.66	c INFO: running hybridGM3@0 for 97 with 1088 remaining (b = 0.08)
411.35/116.69	c INFO: running hybridGM3@0 for 97 with 1088 remaining (b = 0.08)
411.73/116.70	c INFO: running hybridGM3@0 for 97 with 1088 remaining (b = 0.08)
791.21/211.78	c INFO: running hybridGM3@99 for 97 with 993 remaining (b = 0.08)
791.60/211.82	c INFO: running TNM@0 for 97 with 993 remaining (b = 0.08)
791.60/211.83	c INFO: running hybridGM3@99 for 97 with 993 remaining (b = 0.07)
795.57/212.81	c INFO: running hybridGM3@94 for 97 with 992 remaining (b = 0.07)
1191.35/311.96	c INFO: running hybridGM3@100 for 97 with 893 remaining (b = 0.07)
1195.34/312.90	c INFO: running hybridGM3@0 for 97 with 892 remaining (b = 0.06)
1195.34/312.96	c INFO: running hybridGM3@0 for 194 with 892 remaining (b = 0.09)
1199.30/313.93	c INFO: running hybridGM3@0 for 194 with 891 remaining (b = 0.07)
1578.22/409.09	c INFO: running hybridGM3@0 for 194 with 796 remaining (b = 0.05)
1590.96/412.22	c INFO: running hybridGM3@101 for 97 with 793 remaining (b = 0.02)
1973.86/508.34	c INFO: running hybridGM3@95 for 97 with 697 remaining (b = 0.02)
1973.86/508.40	c INFO: running hybridGM3@100 for 97 with 697 remaining (b = 0.02)
1993.80/513.35	c INFO: running hybridGM3@296 for 97 with 692 remaining (b = 0.02)
2357.16/604.59	c INFO: running hybridGM3@99 for 97 with 600 remaining (b = 0.02)
2373.10/608.53	c INFO: running hybridGM3@295 for 97 with 597 remaining (b = 0.02)
2376.70/609.47	c INFO: running TNM@0 for 97 with 596 remaining (b = 0.02)
2380.67/610.48	c INFO: running hybridGM3@0 for 194 with 595 remaining (b = 0.04)
2756.31/704.74	c INFO: running TNM@0 for 97 with 500 remaining (b = 0.02)
2759.89/705.67	c INFO: running hybridGM3@0 for 194 with 499 remaining (b = 0.03)
2771.81/708.64	c INFO: running TNM@0 for 97 with 496 remaining (b = 0.01)
3155.43/804.90	c INFO: running hybridGM3@396 for 97 with 400 remaining (b = 0.01)
3159.42/805.96	c INFO: running hybridGM3@97 for 97 with 399 remaining (b = 0.01)
3170.56/808.80	c INFO: running hybridGM3@0 for 194 with 396 remaining (b = 0.02)
3541.92/902.05	c INFO: running TNM@0 for 97 with 303 remaining (b = 0.01)
3558.16/906.12	c INFO: running TNM@0 for 97 with 299 remaining (b = 0.01)
3558.16/906.18	c INFO: running hybridGM3@99 for 97 with 299 remaining (b = 0.01)
3936.50/1001.32	c INFO: running TNM@0 for 97 with 204 remaining (b = 0.01)
3940.04/1002.22	c INFO: running hybridGM3@0 for 203 with 203 remaining (b = 0.02)
3955.94/1006.28	c INFO: running TNM@0 for 97 with 199 remaining (b = 0.01)
3956.76/1006.52	c INFO: running TNM@0 for 97 with 199 remaining (b = 0.01)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 988)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378196-1307741596/watcher-3378196-1307741596 -o /tmp/evaluation-result-3378196-1307741596/solver-3378196-1307741596 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378196-1307741596.cnf -seed 960148277 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.00 0.00 0.00 2/176 2572
/proc/meminfo: memFree=31778320/32950928 swapFree=67051812/67111528
[pid=2572] ppid=2568 vsize=14644 CPUtime=0 cores=1,3,5,7
/proc/2572/stat : 2572 (python) R 2568 2572 2526 0 -1 4202496 427 0 0 0 0 0 0 0 20 0 1 0 964390363 14995456 336 33554432000 4194304 5683644 140733662686848 18446744073709551615 260905054917 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/2572/statm: 3661 338 217 364 0 148 0

[startup+0.0928599 s]
/proc/loadavg: 0.00 0.00 0.00 2/176 2572
/proc/meminfo: memFree=31778320/32950928 swapFree=67051812/67111528
[pid=2572] ppid=2568 vsize=15756 CPUtime=0 cores=1,3,5,7
/proc/2572/stat : 2572 (python) S 2568 2572 2526 0 -1 4202496 842 0 0 0 0 0 0 0 17 0 1 0 964390363 16134144 724 33554432000 4194304 5683644 140733662686848 18446744073709551615 260905392128 0 2147483391 16781312 2 18446744071699207358 0 0 17 5 0 0 0
/proc/2572/statm: 3939 724 334 364 0 426 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15756

[startup+0.100871 s]
/proc/loadavg: 0.00 0.00 0.00 2/176 2572
/proc/meminfo: memFree=31778320/32950928 swapFree=67051812/67111528
[pid=2572] ppid=2568 vsize=15776 CPUtime=0 cores=1,3,5,7
/proc/2572/stat : 2572 (python) S 2568 2572 2526 0 -1 4202496 929 0 0 0 0 0 0 0 17 0 1 0 964390363 16154624 807 33554432000 4194304 5683644 140733662686848 18446744073709551615 260905392128 0 2147483391 16781312 2 18446744071699953811 0 0 17 5 0 0 0
/proc/2572/statm: 3944 807 356 364 0 431 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15776

[startup+0.300839 s]
/proc/loadavg: 0.00 0.00 0.00 2/176 2572
/proc/meminfo: memFree=31778320/32950928 swapFree=67051812/67111528
[pid=2572] ppid=2568 vsize=29788 CPUtime=0.04 cores=1,3,5,7
/proc/2572/stat : 2572 (python) S 2568 2572 2526 0 -1 4202496 1462 0 0 0 3 1 0 0 16 0 1 0 964390363 30502912 1300 33554432000 4194304 5683644 140733662686848 18446744073709551615 260905392128 0 2147483391 16781312 2 18446744071699207358 0 0 17 5 0 0 0
/proc/2572/statm: 7447 1300 443 364 0 817 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 29788

[startup+0.700745 s]
/proc/loadavg: 0.00 0.00 0.00 2/176 2572
/proc/meminfo: memFree=31778320/32950928 swapFree=67051812/67111528
[pid=2572] ppid=2568 vsize=50012 CPUtime=0.05 cores=1,3,5,7
/proc/2572/stat : 2572 (python) S 2568 2572 2526 0 -1 4202496 1950 0 0 0 3 2 0 0 15 0 1 0 964390363 51212288 1713 33554432000 4194304 5683644 140733662686848 18446744073709551615 260905392128 0 2147483391 16781312 2 18446744071699207358 0 0 17 7 0 0 0
/proc/2572/statm: 12503 1713 476 364 0 1203 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 50012

[startup+1.5006 s]
/proc/loadavg: 0.00 0.00 0.00 1/178 2574
/proc/meminfo: memFree=31764016/32950928 swapFree=67051812/67111528
[pid=2572] ppid=2568 vsize=104616 CPUtime=0.09 cores=1,3,5,7
/proc/2572/stat : 2572 (python) S 2568 2572 2526 0 -1 4202496 3101 0 0 0 6 3 0 0 15 0 1 0 964390363 107126784 2779 33554432000 4194304 5683644 140733662686848 18446744073709551615 260905392128 0 2147483391 16781312 2 18446744071699207358 0 0 17 7 0 0 0
/proc/2572/statm: 26154 2779 847 364 0 1902 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 104616

[startup+3.10031 s]
/proc/loadavg: 0.00 0.00 0.00 1/178 2574
/proc/meminfo: memFree=31755956/32950928 swapFree=67051812/67111528
[pid=2572] ppid=2568 vsize=135752 CPUtime=0.24 cores=1,3,5,7
/proc/2572/stat : 2572 (python) S 2568 2572 2526 0 -1 4202496 5969 935 1 0 17 7 0 0 15 0 1 0 964390363 139010048 4152 33554432000 4194304 5683644 140733662686848 18446744073709551615 260905390085 0 2147483391 16781312 2 18446744071699953811 0 0 17 7 0 0 0
/proc/2572/statm: 33938 4152 1032 364 0 3124 0
Current children cumulated CPU time (s) 0.24
Current children cumulated vsize (KiB) 135752
heavy processes:

[startup+6.30072 s]
/proc/loadavg: 0.00 0.00 0.00 2/178 2582
/proc/meminfo: memFree=31737180/32950928 swapFree=67051812/67111528
[pid=2572] ppid=2568 vsize=327996 CPUtime=1.18 cores=1,3,5,7
/proc/2572/stat : 2572 (python) R 2568 2572 2526 0 -1 4202496 27449 935 1 0 101 17 0 0 16 0 1 0 964390363 335867904 19376 33554432000 4194304 5683644 140733662686848 18446744073709551615 47425664597138 0 0 16781312 2 0 0 0 17 3 0 0 19
/proc/2572/statm: 81999 19376 1818 364 0 17923 0
Current children cumulated CPU time (s) 1.18
Current children cumulated vsize (KiB) 327996
heavy processes:

[startup+12.7005 s]
/proc/loadavg: 0.55 0.12 0.04 7/190 2594
/proc/meminfo: memFree=31648672/32950928 swapFree=67051812/67111528
[pid=2572] ppid=2568 vsize=327996 CPUtime=7.58 cores=1,3,5,7
/proc/2572/stat : 2572 (python) R 2568 2572 2526 0 -1 4202496 27458 935 1 0 741 17 0 0 25 0 1 0 964390363 335867904 19385 33554432000 4194304 5683644 140733662686848 18446744073709551615 260905061253 0 0 16781312 2 0 0 0 17 3 0 0 19
/proc/2572/statm: 81999 19385 1827 364 0 17923 0
Current children cumulated CPU time (s) 7.58
Current children cumulated vsize (KiB) 327996
heavy processes:

[startup+25.5012 s]
/proc/loadavg: 1.69 0.37 0.12 10/202 2606
/proc/meminfo: memFree=31489168/32950928 swapFree=67051812/67111528
[pid=2572] ppid=2568 vsize=365128 CPUtime=11.33 cores=1,3,5,7
/proc/2572/stat : 2572 (python) S 2568 2572 2526 0 -1 4202496 30198 935 1 0 1115 18 0 0 25 0 5 0 964390363 373891072 18479 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 19
/proc/2572/statm: 91282 18479 1881 364 0 27191 0
[pid=2595] ppid=2572 vsize=324116 CPUtime=0 cores=1,3,5,7
/proc/2595/stat : 2595 (python) S 2572 2572 2526 0 -1 4202560 884 0 0 0 0 0 0 0 15 0 1 0 964392005 331894784 16957 33554432000 4194304 5683644 140733662686848 18446744073709551615 260905414159 0 0 16781312 514 4466312993344291080 0 0 17 5 0 0 0
/proc/2595/statm: 81029 16957 375 364 0 16947 0
[pid=2597] ppid=2595 vsize=276240 CPUtime=9.07 cores=1,3,5,7
/proc/2597/stat : 2597 (hybridGM3) R 2595 2597 2597 0 -1 4202496 9518 0 0 0 906 1 0 0 25 0 1 0 964392006 282869760 9255 33554432000 134512640 135204385 4291625712 18446744073709551615 134546073 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/2597/statm: 69060 9255 64 169 0 68890 0
[pid=2598] ppid=2572 vsize=334372 CPUtime=0 cores=1,3,5,7
/proc/2598/stat : 2598 (python) S 2572 2572 2526 0 -1 4202560 904 0 0 0 0 0 0 0 15 0 1 0 964392006 342396928 16955 33554432000 4194304 5683644 140733662686848 18446744073709551615 260905414214 0 0 16781312 514 4579250979458744584 0 0 17 5 0 0 0
/proc/2598/statm: 83593 16955 369 364 0 19508 0
[pid=2600] ppid=2598 vsize=276240 CPUtime=9.01 cores=1,3,5,7
/proc/2600/stat : 2600 (hybridGM3) R 2598 2600 2600 0 -1 4202496 9507 0 0 0 900 1 0 0 25 0 1 0 964392007 282869760 9249 33554432000 134512640 135204385 4287606320 18446744073709551615 134547435 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/2600/statm: 69060 9249 64 169 0 68890 0
[pid=2601] ppid=2572 vsize=344628 CPUtime=0 cores=1,3,5,7
/proc/2601/stat : 2601 (python) S 2572 2572 2526 0 -1 4202560 909 0 0 0 0 0 0 0 15 0 1 0 964392008 352899072 16960 33554432000 4194304 5683644 140733662686848 18446744073709551615 260905414214 0 0 16781312 514 4552753848740970760 0 0 17 5 0 0 0
/proc/2601/statm: 86157 16960 370 364 0 22069 0
[pid=2603] ppid=2601 vsize=276240 CPUtime=9.02 cores=1,3,5,7
/proc/2603/stat : 2603 (hybridGM3) R 2601 2603 2603 0 -1 4202496 9511 0 0 0 901 1 0 0 25 0 1 0 964392008 282869760 9252 33554432000 134512640 135204385 4293483632 18446744073709551615 134536646 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/2603/statm: 69060 9252 64 169 0 68890 0
[pid=2604] ppid=2572 vsize=354884 CPUtime=0 cores=1,3,5,7
/proc/2604/stat : 2604 (python) S 2572 2572 2526 0 -1 4202560 915 0 0 0 0 0 0 0 15 0 1 0 964392009 363401216 16964 33554432000 4194304 5683644 140733662686848 18446744073709551615 260905414214 0 0 16781312 514 4552765393613062408 0 0 17 3 0 0 0
/proc/2604/statm: 88721 16964 370 364 0 24630 0
[pid=2606] ppid=2604 vsize=276236 CPUtime=9.02 cores=1,3,5,7
/proc/2606/stat : 2606 (hybridGM3) R 2604 2606 2606 0 -1 4202496 9509 0 0 0 901 1 0 0 25 0 1 0 964392009 282865664 9251 33554432000 134512640 135204385 4288462896 18446744073709551615 134537357 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/2606/statm: 69059 9251 64 169 0 68889 0
[pid=2572/tid=2596] ppid=2568 vsize=365128 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2596/stat : 2596 (python) S 2568 2572 2526 0 -1 4202560 25 935 0 0 0 0 0 0 25 0 5 0 964392006 373891072 18479 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 1 0 0 0
[pid=2572/tid=2599] ppid=2568 vsize=365128 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2599/stat : 2599 (python) S 2568 2572 2526 0 -1 4202560 25 935 0 0 0 0 0 0 25 0 5 0 964392007 373891072 18479 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 7 0 0 0
[pid=2572/tid=2602] ppid=2568 vsize=365128 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2602/stat : 2602 (python) S 2568 2572 2526 0 -1 4202560 25 935 0 0 0 0 0 0 25 0 5 0 964392008 373891072 18479 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 5 0 0 0
[pid=2572/tid=2605] ppid=2568 vsize=365128 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2605/stat : 2605 (python) S 2568 2572 2526 0 -1 4202560 26 935 0 0 0 0 0 0 25 0 5 0 964392009 373891072 18479 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 47.45
Current children cumulated vsize (KiB) 2828084

[startup+51.1005 s]
/proc/loadavg: 3.92 1.00 0.34 10/202 2607
/proc/meminfo: memFree=31489184/32950928 swapFree=67051812/67111528
[pid=2572] ppid=2568 vsize=365128 CPUtime=11.33 cores=1,3,5,7
/proc/2572/stat : 2572 (python) S 2568 2572 2526 0 -1 4202496 30198 935 1 0 1115 18 0 0 25 0 5 0 964390363 373891072 18479 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 19
/proc/2572/statm: 91282 18479 1881 364 0 27191 0
[pid=2595] ppid=2572 vsize=324116 CPUtime=0.04 cores=1,3,5,7
/proc/2595/stat : 2595 (python) S 2572 2572 2526 0 -1 4202560 905 0 0 0 2 2 0 0 15 0 1 0 964392005 331894784 16958 33554432000 4194304 5683644 140733662686848 18446744073709551615 260905414159 0 0 16781312 514 2371321125670977800 0 0 17 5 0 0 0
/proc/2595/statm: 81029 16958 376 364 0 16947 0
[pid=2597] ppid=2595 vsize=276240 CPUtime=34.66 cores=1,3,5,7
/proc/2597/stat : 2597 (hybridGM3) R 2595 2597 2597 0 -1 4202496 9519 0 0 0 3465 1 0 0 25 0 1 0 964392006 282869760 9256 33554432000 134512640 135204385 4291625712 18446744073709551615 134536843 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/2597/statm: 69060 9256 64 169 0 68890 0
[pid=2598] ppid=2572 vsize=334372 CPUtime=0.05 cores=1,3,5,7
/proc/2598/stat : 2598 (python) S 2572 2572 2526 0 -1 4202560 921 0 0 0 3 2 0 0 15 0 1 0 964392006 342396928 16956 33554432000 4194304 5683644 140733662686848 18446744073709551615 260905414214 0 0 16781312 514 2371321125670977800 0 0 17 5 0 0 0
/proc/2598/statm: 83593 16956 370 364 0 19508 0
[pid=2600] ppid=2598 vsize=276240 CPUtime=34.46 cores=1,3,5,7
/proc/2600/stat : 2600 (hybridGM3) R 2598 2600 2600 0 -1 4202496 9507 0 0 0 3445 1 0 0 25 0 1 0 964392007 282869760 9249 33554432000 134512640 135204385 4287606320 18446744073709551615 134536744 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/2600/statm: 69060 9249 64 169 0 68890 0
[pid=2601] ppid=2572 vsize=344628 CPUtime=0.04 cores=1,3,5,7
/proc/2601/stat : 2601 (python) S 2572 2572 2526 0 -1 4202560 921 0 0 0 2 2 0 0 16 0 1 0 964392008 352899072 16960 33554432000 4194304 5683644 140733662686848 18446744073709551615 260905414214 0 0 16781312 514 2371321125670977800 0 0 17 5 0 0 0
/proc/2601/statm: 86157 16960 370 364 0 22069 0
[pid=2603] ppid=2601 vsize=276240 CPUtime=34.5 cores=1,3,5,7
/proc/2603/stat : 2603 (hybridGM3) R 2601 2603 2603 0 -1 4202496 9511 0 0 0 3449 1 0 0 25 0 1 0 964392008 282869760 9252 33554432000 134512640 135204385 4293483632 18446744073709551615 134549153 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/2603/statm: 69060 9252 64 169 0 68890 0
[pid=2604] ppid=2572 vsize=354884 CPUtime=0.04 cores=1,3,5,7
/proc/2604/stat : 2604 (python) S 2572 2572 2526 0 -1 4202560 926 0 0 0 2 2 0 0 16 0 1 0 964392009 363401216 16964 33554432000 4194304 5683644 140733662686848 18446744073709551615 260905414214 0 0 16781312 514 2371321125670977800 0 0 17 3 0 0 0
/proc/2604/statm: 88721 16964 370 364 0 24630 0
[pid=2606] ppid=2604 vsize=276236 CPUtime=34.58 cores=1,3,5,7
/proc/2606/stat : 2606 (hybridGM3) R 2604 2606 2606 0 -1 4202496 9509 0 0 0 3457 1 0 0 25 0 1 0 964392009 282865664 9251 33554432000 134512640 135204385 4288462896 18446744073709551615 134537290 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/2606/statm: 69059 9251 64 169 0 68889 0
[pid=2572/tid=2596] ppid=2568 vsize=365128 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2596/stat : 2596 (python) S 2568 2572 2526 0 -1 4202560 25 935 0 0 0 0 0 0 25 0 5 0 964392006 373891072 18479 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 1 0 0 0
[pid=2572/tid=2599] ppid=2568 vsize=365128 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2599/stat : 2599 (python) S 2568 2572 2526 0 -1 4202560 25 935 0 0 0 0 0 0 25 0 5 0 964392007 373891072 18479 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 7 0 0 0
[pid=2572/tid=2602] ppid=2568 vsize=365128 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2602/stat : 2602 (python) S 2568 2572 2526 0 -1 4202560 25 935 0 0 0 0 0 0 25 0 5 0 964392008 373891072 18479 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 5 0 0 0
[pid=2572/tid=2605] ppid=2568 vsize=365128 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2605/stat : 2605 (python) S 2568 2572 2526 0 -1 4202560 26 935 0 0 0 0 0 0 25 0 5 0 964392009 373891072 18479 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 149.7
Current children cumulated vsize (KiB) 2828084

[startup+102.307 s]
/proc/loadavg: 6.43 2.19 0.78 10/202 2664
/proc/meminfo: memFree=31487488/32950928 swapFree=67051812/67111528
[pid=2572] ppid=2568 vsize=365128 CPUtime=11.33 cores=1,3,5,7
/proc/2572/stat : 2572 (python) S 2568 2572 2526 0 -1 4202496 30198 935 1 0 1115 18 0 0 25 0 5 0 964390363 373891072 18479 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 19
/proc/2572/statm: 91282 18479 1881 364 0 27191 0
[pid=2595] ppid=2572 vsize=324116 CPUtime=0.11 cores=1,3,5,7
/proc/2595/stat : 2595 (python) S 2572 2572 2526 0 -1 4202560 923 0 0 0 5 6 0 0 15 0 1 0 964392005 331894784 16958 33554432000 4194304 5683644 140733662686848 18446744073709551615 260905414159 0 0 16781312 514 2371321125670977800 0 0 17 5 0 0 0
/proc/2595/statm: 81029 16958 376 364 0 16947 0
[pid=2597] ppid=2595 vsize=276240 CPUtime=85.86 cores=1,3,5,7
/proc/2597/stat : 2597 (hybridGM3) R 2595 2597 2597 0 -1 4202496 9519 0 0 0 8585 1 0 0 25 0 1 0 964392006 282869760 9256 33554432000 134512640 135204385 4291625712 18446744073709551615 134536605 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/2597/statm: 69060 9256 64 169 0 68890 0
[pid=2598] ppid=2572 vsize=334372 CPUtime=0.11 cores=1,3,5,7
/proc/2598/stat : 2598 (python) S 2572 2572 2526 0 -1 4202560 938 0 0 0 6 5 0 0 15 0 1 0 964392006 342396928 16956 33554432000 4194304 5683644 140733662686848 18446744073709551615 260905414214 0 0 16781312 514 2371321125670977800 0 0 17 5 0 0 0
/proc/2598/statm: 83593 16956 370 364 0 19508 0

################
# More data... #
################

[pid=2688] ppid=2686 vsize=276240 CPUtime=392.78 cores=1,3,5,7
/proc/2688/stat : 2688 (hybridGM3) T 2686 2688 2688 0 -1 4202496 9529 0 0 0 39277 1 0 0 25 0 1 0 964421655 282869760 9266 33554432000 134512640 135204385 4294070912 18446744073709551615 134537292 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/2688/statm: 69060 9266 64 169 0 68890 0
[pid=2686/tid=2704] ppid=2572 vsize=375396 CPUtime=0 cores=1,3,5,7
/proc/2686/task/2704/stat : 2704 (python) S 2572 2572 2526 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 964431585 384405504 17002 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 514 18446604459210032448 0 0 -1 7 0 0 0
[pid=2689] ppid=2572 vsize=385652 CPUtime=0.4 cores=1,3,5,7
/proc/2689/stat : 2689 (python) S 2572 2572 2526 0 -1 4202560 1162 0 0 0 20 20 0 0 15 0 2 0 964421660 394907648 17006 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2689/statm: 96413 17006 399 364 0 32313 0
[pid=2691] ppid=2689 vsize=276236 CPUtime=293.63 cores=1,3,5,7
/proc/2691/stat : 2691 (hybridGM3) T 2689 2691 2691 0 -1 4202496 9514 0 0 0 29361 2 0 0 25 0 1 0 964421660 282865664 9253 33554432000 134512640 135204385 4289383984 18446744073709551615 134537357 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/2691/statm: 69059 9253 64 169 0 68889 0
[pid=2689/tid=2708] ppid=2572 vsize=385652 CPUtime=0 cores=1,3,5,7
/proc/2689/task/2708/stat : 2708 (python) S 2572 2572 2526 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 964441202 394907648 17006 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 514 18446604471108813824 0 0 -1 7 0 0 0
[pid=2692] ppid=2572 vsize=395908 CPUtime=0.4 cores=1,3,5,7
/proc/2692/stat : 2692 (python) S 2572 2572 2526 0 -1 4202560 1194 0 0 0 20 20 0 0 15 0 2 0 964421757 405409792 17016 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/2692/statm: 98977 17016 405 364 0 34874 0
[pid=2694] ppid=2692 vsize=276236 CPUtime=295.49 cores=1,3,5,7
/proc/2694/stat : 2694 (hybridGM3) T 2692 2694 2694 0 -1 4202496 9525 0 0 0 29548 1 0 0 25 0 1 0 964421758 282865664 9259 33554432000 134512640 135204385 4289420384 18446744073709551615 134536663 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/2694/statm: 69059 9259 64 169 0 68889 0
[pid=2692/tid=2707] ppid=2572 vsize=395908 CPUtime=0 cores=1,3,5,7
/proc/2692/task/2707/stat : 2707 (python) S 2572 2572 2526 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 964441197 405409792 17016 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 514 18446604471150912576 0 0 -1 5 0 0 0
[pid=2701] ppid=2572 vsize=406164 CPUtime=0.41 cores=1,3,5,7
/proc/2701/stat : 2701 (python) S 2572 2572 2526 0 -1 4202560 1189 0 0 0 20 21 0 0 15 0 2 0 964431272 415911936 17020 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2701/statm: 101541 17020 405 364 0 37435 0
[pid=2703] ppid=2701 vsize=276240 CPUtime=293.27 cores=1,3,5,7
/proc/2703/stat : 2703 (hybridGM3) T 2701 2703 2703 0 -1 4202496 9520 0 0 0 29325 2 0 0 25 0 1 0 964431273 282869760 9252 33554432000 134512640 135204385 4292393968 18446744073709551615 134537383 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/2703/statm: 69060 9252 64 169 0 68890 0
[pid=2701/tid=2714] ppid=2572 vsize=406164 CPUtime=0 cores=1,3,5,7
/proc/2701/task/2714/stat : 2714 (python) S 2572 2572 2526 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 964450822 415911936 17020 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 514 18446604471149801600 0 0 -1 7 0 0 0
[pid=2715] ppid=2572 vsize=416420 CPUtime=0.15 cores=1,3,5,7
/proc/2715/stat : 2715 (python) S 2572 2572 2526 0 -1 4202560 1120 0 0 0 8 7 0 0 19 0 2 0 964451311 426414080 17018 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2715/statm: 104105 17018 399 364 0 39996 0
[pid=2717] ppid=2715 vsize=471116 CPUtime=99.1 cores=1,3,5,7
/proc/2717/stat : 2717 (TNM) T 2715 2717 2717 0 -1 4202496 8436 0 0 0 9909 1 0 0 25 0 1 0 964451311 482422784 8172 33554432000 134512640 135008748 4292604592 18446744073709551615 134529123 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/2717/statm: 117779 8172 46 122 0 116679 0
[pid=2715/tid=2733] ppid=2572 vsize=416420 CPUtime=0 cores=1,3,5,7
/proc/2715/task/2733/stat : 2733 (python) S 2572 2572 2526 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 964461226 426414080 17018 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 514 18446604471149803392 0 0 -1 5 0 0 0
[pid=2718] ppid=2572 vsize=426676 CPUtime=0.44 cores=1,3,5,7
/proc/2718/stat : 2718 (python) S 2572 2572 2526 0 -1 4202560 1179 0 0 0 22 22 0 0 15 0 2 0 964451412 436916224 17022 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2718/statm: 106669 17022 399 364 0 42557 0
[pid=2720] ppid=2718 vsize=276244 CPUtime=295.47 cores=1,3,5,7
/proc/2720/stat : 2720 (hybridGM3) T 2718 2720 2720 0 -1 4202496 9521 0 0 0 29545 2 0 0 25 0 1 0 964451412 282873856 9255 33554432000 134512640 135204385 4288866544 18446744073709551615 134537403 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/2720/statm: 69061 9255 64 169 0 68891 0
[pid=2718/tid=2746] ppid=2572 vsize=426676 CPUtime=0 cores=1,3,5,7
/proc/2718/task/2746/stat : 2746 (python) S 2572 2572 2526 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 964470959 436916224 17022 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 514 18446604471150913472 0 0 -1 1 0 0 0
[pid=2727] ppid=2572 vsize=436932 CPUtime=0.15 cores=1,3,5,7
/proc/2727/stat : 2727 (python) S 2572 2572 2526 0 -1 4202560 1123 0 0 0 8 7 0 0 19 0 2 0 964460838 447418368 17026 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2727/statm: 109233 17026 399 364 0 45118 0
[pid=2729] ppid=2727 vsize=471116 CPUtime=100.11 cores=1,3,5,7
/proc/2729/stat : 2729 (TNM) T 2727 2729 2729 0 -1 4202496 8439 0 0 0 10009 2 0 0 25 0 1 0 964460838 482422784 8172 33554432000 134512640 135008748 4292625312 18446744073709551615 134527524 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/2729/statm: 117779 8172 46 122 0 116679 0
[pid=2727/tid=2745] ppid=2572 vsize=436932 CPUtime=0 cores=1,3,5,7
/proc/2727/task/2745/stat : 2745 (python) S 2572 2572 2526 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 964470853 447418368 17026 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 514 18446604471118331968 0 0 -1 3 0 0 0
[pid=2730] ppid=2572 vsize=447188 CPUtime=0.44 cores=1,3,5,7
/proc/2730/stat : 2730 (python) S 2572 2572 2526 0 -1 4202560 1179 0 0 0 24 20 0 0 16 0 2 0 964460931 457920512 17030 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/2730/statm: 111797 17030 399 364 0 47679 0
[pid=2732] ppid=2730 vsize=276236 CPUtime=293.25 cores=1,3,5,7
/proc/2732/stat : 2732 (hybridGM3) T 2730 2732 2732 0 -1 4202496 9525 0 0 0 29322 3 0 0 25 0 1 0 964460932 282865664 9257 33554432000 134512640 135204385 4292416544 18446744073709551615 134536676 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/2732/statm: 69059 9257 64 169 0 68889 0
[pid=2730/tid=2760] ppid=2572 vsize=447188 CPUtime=0 cores=1,3,5,7
/proc/2730/task/2760/stat : 2760 (python) S 2572 2572 2526 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 964480980 457920512 17030 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 514 18446604471118333760 0 0 -1 7 0 0 0
[pid=2734] ppid=2572 vsize=457444 CPUtime=0.14 cores=1,3,5,7
/proc/2734/stat : 2734 (python) S 2572 2572 2526 0 -1 4202560 1141 0 0 0 7 7 0 0 19 0 2 0 964461228 468422656 17034 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2734/statm: 114361 17034 399 364 0 50240 0
[pid=2736] ppid=2734 vsize=471116 CPUtime=100.14 cores=1,3,5,7
/proc/2736/stat : 2736 (TNM) T 2734 2736 2736 0 -1 4202496 8442 0 0 0 10012 2 0 0 25 0 1 0 964461228 482422784 8170 33554432000 134512640 135008748 4292627904 18446744073709551615 134529176 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/2736/statm: 117779 8170 46 122 0 116679 0
[pid=2734/tid=2747] ppid=2572 vsize=457444 CPUtime=0 cores=1,3,5,7
/proc/2734/task/2747/stat : 2747 (python) S 2572 2572 2526 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 964471242 468422656 17034 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 514 18446604470887831360 0 0 -1 1 0 0 0
[pid=2748] ppid=2572 vsize=467700 CPUtime=0.29 cores=1,3,5,7
/proc/2748/stat : 2748 (python) S 2572 2572 2526 0 -1 4202560 1079 0 0 0 17 12 0 0 15 0 1 0 964471244 478924800 17016 33554432000 4194304 5683644 140733662686848 18446744073709551615 260905414214 0 0 16781312 514 2371321125670977799 0 0 17 7 0 0 0
/proc/2748/statm: 116925 17016 378 364 0 52801 0
[pid=2750] ppid=2748 vsize=276244 CPUtime=195.03 cores=1,3,5,7
/proc/2750/stat : 2750 (hybridGM3) R 2748 2750 2750 0 -1 4202496 9533 0 0 0 19498 5 0 0 25 0 1 0 964471244 282873856 9254 33554432000 134512640 135204385 4287621440 18446744073709551615 134537272 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/2750/statm: 69061 9254 64 169 0 68891 0
[pid=2754] ppid=2572 vsize=477956 CPUtime=0.15 cores=1,3,5,7
/proc/2754/stat : 2754 (python) S 2572 2572 2526 0 -1 4202560 1192 0 0 0 9 6 0 0 19 0 2 0 964480569 489426944 17050 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2754/statm: 119489 17050 407 364 0 55362 0
[pid=2756] ppid=2754 vsize=471116 CPUtime=97.26 cores=1,3,5,7
/proc/2756/stat : 2756 (TNM) T 2754 2756 2756 0 -1 4202496 8447 0 0 0 9724 2 0 0 25 0 1 0 964480570 482422784 8171 33554432000 134512640 135008748 4292583952 18446744073709551615 134527327 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/2756/statm: 117779 8171 46 122 0 116679 0
[pid=2754/tid=2768] ppid=2572 vsize=477956 CPUtime=0 cores=1,3,5,7
/proc/2754/task/2768/stat : 2768 (python) S 2572 2572 2526 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 964490584 489426944 17050 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 514 18446604465943512064 0 0 -1 7 0 0 0
[pid=2757] ppid=2572 vsize=488212 CPUtime=0.15 cores=1,3,5,7
/proc/2757/stat : 2757 (python) S 2572 2572 2526 0 -1 4202560 1199 0 0 0 7 8 0 0 19 0 2 0 964480975 499929088 17054 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/2757/statm: 122053 17054 407 364 0 57923 0
[pid=2759] ppid=2757 vsize=471116 CPUtime=100.11 cores=1,3,5,7
/proc/2759/stat : 2759 (TNM) T 2757 2759 2759 0 -1 4202496 8447 0 0 0 10010 1 0 0 25 0 1 0 964480976 482422784 8172 33554432000 134512640 135008748 4286659856 18446744073709551615 134527542 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/2759/statm: 117779 8172 46 122 0 116679 0
[pid=2757/tid=2772] ppid=2572 vsize=488212 CPUtime=0 cores=1,3,5,7
/proc/2757/task/2772/stat : 2772 (python) S 2572 2572 2526 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 964490990 499929088 17054 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 514 18446604471115032576 0 0 -1 1 0 0 0
[pid=2765] ppid=2572 vsize=498468 CPUtime=0.01 cores=1,3,5,7
/proc/2765/stat : 2765 (python) S 2572 2572 2526 0 -1 4202560 991 0 0 0 0 1 0 0 18 0 1 0 964490495 510431232 17021 33554432000 4194304 5683644 140733662686848 18446744073709551615 260905414214 0 0 16781312 514 4466316841634988292 0 0 17 1 0 0 0
/proc/2765/statm: 124617 17021 371 364 0 60484 0
[pid=2767] ppid=2765 vsize=471116 CPUtime=4.95 cores=1,3,5,7
/proc/2767/stat : 2767 (TNM) R 2765 2767 2767 0 -1 4202496 8444 0 0 0 494 1 0 0 25 0 1 0 964490496 482422784 8170 33554432000 134512640 135008748 4292245184 18446744073709551615 134524914 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/2767/statm: 117779 8170 46 122 0 116679 0
[pid=2769] ppid=2572 vsize=508724 CPUtime=0 cores=1,3,5,7
/proc/2769/stat : 2769 (python) S 2572 2572 2526 0 -1 4202560 861 0 0 0 0 0 0 0 15 0 1 0 964490586 520933376 17003 33554432000 4194304 5683644 140733662686848 18446744073709551615 260905414214 0 0 16781312 514 15571785054160912644 0 0 17 1 0 0 0
/proc/2769/statm: 127181 17003 349 364 0 63045 0
[pid=2771] ppid=2769 vsize=276236 CPUtime=4.17 cores=1,3,5,7
/proc/2771/stat : 2771 (hybridGM3) R 2769 2771 2771 0 -1 4202496 9522 0 0 0 416 1 0 0 25 0 1 0 964490586 282865664 9246 33554432000 134512640 135204385 4291070928 18446744073709551615 134537322 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/2771/statm: 69059 9246 64 169 0 68889 0
[pid=2773] ppid=2572 vsize=518980 CPUtime=0 cores=1,3,5,7
/proc/2773/stat : 2773 (python) S 2572 2572 2526 0 -1 4202560 831 0 0 0 0 0 0 0 19 0 1 0 964490991 531435520 17003 33554432000 4194304 5683644 140733662686848 18446744073709551615 260905414214 0 0 16781312 514 9934413841326669823 0 0 17 1 0 0 0
/proc/2773/statm: 129745 17003 345 364 0 65606 0
[pid=2776] ppid=2773 vsize=454620 CPUtime=0.12 cores=1,3,5,7
/proc/2776/stat : 2776 (TNM) R 2773 2776 2776 0 -1 4202496 1724 0 0 0 12 0 0 0 21 0 1 0 964490992 465530880 1454 33554432000 134512640 135008748 4290452432 18446744073709551615 134597214 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/2776/statm: 113655 1454 45 122 0 112555 0
[pid=2572/tid=2596] ppid=2568 vsize=529224 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2596/stat : 2596 (python) S 2568 2572 2526 0 -1 4202560 32 935 0 0 0 0 0 0 18 0 21 0 964392006 541925376 18576 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 1 0 0 0
[pid=2572/tid=2599] ppid=2568 vsize=529224 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2599/stat : 2599 (python) S 2568 2572 2526 0 -1 4202560 38 935 0 0 0 0 0 0 15 0 21 0 964392007 541925376 18576 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 7 0 0 0
[pid=2572/tid=2602] ppid=2568 vsize=529224 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2602/stat : 2602 (python) S 2568 2572 2526 0 -1 4202560 30 935 0 0 0 0 0 0 18 0 21 0 964392008 541925376 18576 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 5 0 0 0
[pid=2572/tid=2605] ppid=2568 vsize=529224 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2605/stat : 2605 (python) S 2568 2572 2526 0 -1 4202560 30 935 0 0 0 0 0 0 18 0 21 0 964392009 541925376 18576 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 3 0 0 0
[pid=2572/tid=2679] ppid=2568 vsize=529224 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2679/stat : 2679 (python) S 2568 2572 2526 0 -1 4202560 24 935 0 0 0 0 0 0 20 0 21 0 964411546 541925376 18576 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 3 0 0 0
[pid=2572/tid=2687] ppid=2568 vsize=529224 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2687/stat : 2687 (python) S 2568 2572 2526 0 -1 4202560 44 935 0 0 0 0 0 0 18 0 21 0 964421655 541925376 18576 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 7 0 0 0
[pid=2572/tid=2690] ppid=2568 vsize=529224 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2690/stat : 2690 (python) S 2568 2572 2526 0 -1 4202560 32 935 0 0 0 0 0 0 18 0 21 0 964421660 541925376 18576 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 7 0 0 0
[pid=2572/tid=2693] ppid=2568 vsize=529224 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2693/stat : 2693 (python) S 2568 2572 2526 0 -1 4202560 28 935 0 0 0 0 0 0 18 0 21 0 964421757 541925376 18576 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 7 0 0 0
[pid=2572/tid=2702] ppid=2568 vsize=529224 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2702/stat : 2702 (python) S 2568 2572 2526 0 -1 4202560 22 935 0 0 0 0 0 0 18 0 21 0 964431273 541925376 18576 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 7 0 0 0
[pid=2572/tid=2716] ppid=2568 vsize=529224 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2716/stat : 2716 (python) S 2568 2572 2526 0 -1 4202560 24 935 0 0 0 0 0 0 20 0 21 0 964451311 541925376 18576 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 7 0 0 0
[pid=2572/tid=2719] ppid=2568 vsize=529224 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2719/stat : 2719 (python) S 2568 2572 2526 0 -1 4202560 28 935 0 0 0 0 0 0 18 0 21 0 964451412 541925376 18576 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 7 0 0 0
[pid=2572/tid=2728] ppid=2568 vsize=529224 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2728/stat : 2728 (python) S 2568 2572 2526 0 -1 4202560 25 935 0 0 0 0 0 0 20 0 21 0 964460838 541925376 18576 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 7 0 0 0
[pid=2572/tid=2731] ppid=2568 vsize=529224 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2731/stat : 2731 (python) S 2568 2572 2526 0 -1 4202560 32 935 0 0 0 0 0 0 18 0 21 0 964460932 541925376 18576 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 7 0 0 0
[pid=2572/tid=2735] ppid=2568 vsize=529224 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2735/stat : 2735 (python) S 2568 2572 2526 0 -1 4202560 30 935 0 0 0 0 0 0 20 0 21 0 964461228 541925376 18576 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 7 0 0 0
[pid=2572/tid=2749] ppid=2568 vsize=529224 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2749/stat : 2749 (python) S 2568 2572 2526 0 -1 4202560 28 935 0 0 0 0 0 0 20 0 21 0 964471244 541925376 18576 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 7 0 0 0
[pid=2572/tid=2755] ppid=2568 vsize=529224 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2755/stat : 2755 (python) S 2568 2572 2526 0 -1 4202560 25 935 0 0 0 0 0 0 20 0 21 0 964480569 541925376 18576 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 7 0 0 0
[pid=2572/tid=2758] ppid=2568 vsize=529224 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2758/stat : 2758 (python) S 2568 2572 2526 0 -1 4202560 27 935 0 0 0 0 0 0 20 0 21 0 964480976 541925376 18576 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 1 0 0 0
[pid=2572/tid=2766] ppid=2568 vsize=529224 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2766/stat : 2766 (python) S 2568 2572 2526 0 -1 4202560 26 935 0 0 0 0 0 0 20 0 21 0 964490496 541925376 18576 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 1 0 0 0
[pid=2572/tid=2770] ppid=2568 vsize=529224 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2770/stat : 2770 (python) S 2568 2572 2526 0 -1 4202560 26 935 0 0 0 0 0 0 20 0 21 0 964490586 541925376 18576 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 1 0 0 0
[pid=2572/tid=2774] ppid=2568 vsize=529224 CPUtime=0 cores=1,3,5,7
/proc/2572/task/2774/stat : 2774 (python) S 2568 2572 2526 0 -1 4202560 25 935 0 0 0 0 0 0 20 0 21 0 964490992 541925376 18576 33554432000 4194304 5683644 140733662686848 18446744073709551615 260917217713 0 0 16781312 2 18446604471158751488 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3956.76
Current children cumulated vsize (KiB) 16037728

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 2572 and gives
#  childrusage.ru_utime.tv_sec=24
#  childrusage.ru_utime.tv_usec=351
#  childrusage.ru_stime.tv_sec=3
#  childrusage.ru_stime.tv_usec=63534
# CPU time returned by wait4() is 27.0639
# while last known CPU time is 3956.76
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1006.53
CPU time (s): 3956.76
CPU user time (s): 3953.59
CPU system time (s): 3.17
CPU usage (%): 393.11
Max. virtual memory (cumulated for all children) (KiB): 16037728

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 24.0004
system time used= 3.06353
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 93781
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 12745
involuntary context switches= 291

runsolver used 4.48832 second user time and 9.68453 second system time

The end

Launcher Data 

Begin job on node116 at 2011-06-10 23:33:16
IDJOB=3378196
IDBENCH=82605
IDSOLVER=1859
FILE ID=node116/3378196-1307741596
RUNJOBID= node116-1307741595-2546
PBS_JOBID= 13550793
Free space on /tmp= 74144 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v50000-c210000-S758437099-099-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378196-1307741596/watcher-3378196-1307741596 -o /tmp/evaluation-result-3378196-1307741596/solver-3378196-1307741596 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378196-1307741596.cnf -seed 960148277 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 94d10ca78c6cee5b0badfb96b777ac08
RANDOM SEED=960148277

node116.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      31781404 kB
Buffers:        251372 kB
Cached:         717708 kB
SwapCached:      13040 kB
Active:         445264 kB
Inactive:       543300 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      31781404 kB
SwapTotal:    67111528 kB
SwapFree:     67051812 kB
Dirty:           67956 kB
Writeback:           0 kB
AnonPages:       16964 kB
Mapped:           8556 kB
Slab:           115392 kB
PageTables:       4888 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   187652 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 74076 MiB
End job on node116 at 2011-06-10 23:50:05