Trace number 3317236

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (MO) 3951.51 1001.03

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v50000-c210000-S964643921-035-UNKNOWN.cnf
MD5SUMd8337de8a71a630ccf2dd02943cd7d7f
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark248.692
Satisfiable
(Un)Satisfiability was proved
Number of variables50000
Number of clauses210000
Sum of the clauses size630000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3210000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.37/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.37/0.67	c INFO: solving HOME/instance-3317236-1304268470.cnf
0.37/0.67	c INFO: computing task features
1.13/1.41	c INFO: computed variable-clause graph statistics
1.13/1.44	c INFO: computed clause balance statistics
4.83/5.19	c INFO: computed variable balance statistics
7.93/8.27	c INFO: computed small-clause counts
7.93/8.29	c INFO: computed Horn-clause counts
11.03/11.37	c INFO: computed variable graph statistics
11.03/11.37	c INFO: collected features for HOME/instance-3317236-1304268470.cnf in 10.68 s
11.13/11.40	c INFO: running hybridGM3@0 for 97 with 4789 remaining (b = 0.35)
11.13/11.43	c INFO: running hybridGM3@0 for 97 with 4789 remaining (b = 0.21)
11.13/11.46	c INFO: running hybridGM3@0 for 97 with 4789 remaining (b = 0.14)
11.13/11.49	c INFO: running hybridGM3@0 for 97 with 4789 remaining (b = 0.09)
411.39/111.72	c INFO: running hybridGM3@0 for 97 with 4688 remaining (b = 0.08)
411.39/111.75	c INFO: running hybridGM3@0 for 97 with 4688 remaining (b = 0.08)
411.39/111.78	c INFO: running hybridGM3@0 for 97 with 4688 remaining (b = 0.08)
411.73/111.81	c INFO: running hybridGM3@99 for 97 with 4688 remaining (b = 0.08)
791.49/206.92	c INFO: running hybridGM3@99 for 97 with 4593 remaining (b = 0.08)
791.49/206.94	c INFO: running TNM@0 for 97 with 4593 remaining (b = 0.08)
791.49/206.98	c INFO: running hybridGM3@99 for 97 with 4593 remaining (b = 0.07)
815.39/212.90	c INFO: running hybridGM3@100 for 97 with 4587 remaining (b = 0.07)
1191.52/307.12	c INFO: running hybridGM3@0 for 194 with 4493 remaining (b = 0.13)
1195.14/308.06	c INFO: running hybridGM3@0 for 194 with 4492 remaining (b = 0.09)
1195.14/308.10	c INFO: running hybridGM3@0 for 194 with 4492 remaining (b = 0.07)
1195.45/308.15	c INFO: running TNM@0 for 97 with 4492 remaining (b = 0.03)
1594.60/408.37	c INFO: running hybridGM3@0 for 194 with 4392 remaining (b = 0.05)
1974.20/503.65	c INFO: running hybridGM3@94 for 97 with 4297 remaining (b = 0.02)
1977.79/504.54	c INFO: running hybridGM3@100 for 97 with 4296 remaining (b = 0.02)
1978.17/504.60	c INFO: running hybridGM3@100 for 97 with 4296 remaining (b = 0.02)
2353.37/598.83	c INFO: running hybridGM3@291 for 97 with 4201 remaining (b = 0.02)
2356.95/599.72	c INFO: running hybridGM3@296 for 97 with 4200 remaining (b = 0.02)
2377.70/604.98	c INFO: running TNM@0 for 97 with 4195 remaining (b = 0.02)
2380.15/605.79	c INFO: running TNM@0 for 97 with 4194 remaining (b = 0.02)
2756.36/700.03	c INFO: running hybridGM3@386 for 97 with 4100 remaining (b = 0.02)
2759.93/700.92	c INFO: running hybridGM3@0 for 194 with 4099 remaining (b = 0.04)
2776.68/705.20	c INFO: running TNM@0 for 97 with 4095 remaining (b = 0.01)
2779.45/706.01	c INFO: running TNM@0 for 97 with 4094 remaining (b = 0.01)
3134.78/795.23	c INFO: running TNM@0 for 97 with 4005 remaining (b = 0.01)
3174.47/805.23	c INFO: running TNM@0 for 97 with 3995 remaining (b = 0.01)
3175.08/805.41	c INFO: running TNM@95 for 1940 with 3995 remaining (b = 0.20)
3549.13/899.58	c INFO: running hybridGM3@98 for 97 with 3901 remaining (b = 0.02)
3552.70/900.48	c INFO: running hybridGM3@0 for 194 with 3900 remaining (b = 0.03)
3564.61/903.48	c INFO: running hybridGM3@0 for 194 with 3897 remaining (b = 0.02)
3951.09/1000.81	c INFO: running hybridGM3@0 for 194 with 3799 remaining (b = 0.02)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317236-1304268470/watcher-3317236-1304268470 -o /tmp/evaluation-result-3317236-1304268470/solver-3317236-1304268470 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317236-1304268470.cnf -seed 809406488 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.11 7.91 5.76 5/253 2219
/proc/meminfo: memFree=29952724/32950928 swapFree=67111448/67111528
[pid=2219] ppid=2217 vsize=15184 CPUtime=0 cores=1,3,5,7
/proc/2219/stat : 2219 (python) R 2217 2219 1708 0 -1 4202496 720 0 0 0 0 0 0 0 25 0 1 0 538445734 15548416 611 33554432000 4194304 5683644 140736015292896 18446744073709551615 251814283399 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/2219/statm: 3796 615 321 364 0 283 0

[startup+0.0902379 s]
/proc/loadavg: 8.11 7.91 5.76 5/253 2219
/proc/meminfo: memFree=29952724/32950928 swapFree=67111448/67111528
[pid=2219] ppid=2217 vsize=84528 CPUtime=0.05 cores=1,3,5,7
/proc/2219/stat : 2219 (python) R 2217 2219 1708 0 -1 4202496 2192 0 0 0 4 1 0 0 25 0 1 0 538445734 86556672 1942 33554432000 4194304 5683644 140736015292896 18446744073709551615 251809272100 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/2219/statm: 21132 1945 568 364 0 1336 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 84528

[startup+0.100235 s]
/proc/loadavg: 8.11 7.91 5.76 5/253 2219
/proc/meminfo: memFree=29952724/32950928 swapFree=67111448/67111528
[pid=2219] ppid=2217 vsize=87708 CPUtime=0.05 cores=1,3,5,7
/proc/2219/stat : 2219 (python) S 2217 2219 1708 0 -1 4202496 2637 0 0 0 4 1 0 0 25 0 1 0 538445734 89812992 2354 33554432000 4194304 5683644 140736015292896 18446744073709551615 251814238208 0 2147483391 16781312 2 18446744071699215550 0 0 17 1 0 0 0
/proc/2219/statm: 21927 2354 717 364 0 1623 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 87708

[startup+0.300207 s]
/proc/loadavg: 8.11 7.91 5.76 5/253 2219
/proc/meminfo: memFree=29952724/32950928 swapFree=67111448/67111528
[pid=2219] ppid=2217 vsize=135752 CPUtime=0.19 cores=1,3,5,7
/proc/2219/stat : 2219 (python) R 2217 2219 1708 0 -1 4202496 5964 934 1 0 13 6 0 0 25 0 1 0 538445734 139010048 4151 33554432000 4194304 5683644 140736015292896 18446744073709551615 251814236165 0 2147483391 16781312 2 0 0 0 17 7 0 0 0
/proc/2219/statm: 33938 4151 1032 364 0 3124 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 135752

[startup+0.700136 s]
/proc/loadavg: 8.11 7.91 5.76 5/253 2219
/proc/meminfo: memFree=29952724/32950928 swapFree=67111448/67111528
[pid=2219] ppid=2217 vsize=284160 CPUtime=0.44 cores=1,3,5,7
/proc/2219/stat : 2219 (python) R 2217 2219 1708 0 -1 4202496 10678 934 1 0 32 12 0 0 25 0 1 0 538445734 290979840 8360 33554432000 4194304 5683644 140736015292896 18446744073709551615 4954144 0 0 16781312 2 0 0 0 17 7 0 0 0
/proc/2219/statm: 71040 8360 1761 364 0 6964 0
Current children cumulated CPU time (s) 0.44
Current children cumulated vsize (KiB) 284160

[startup+1.501 s]
/proc/loadavg: 8.11 7.91 5.76 6/254 2224
/proc/meminfo: memFree=29903984/32950928 swapFree=67111448/67111528
[pid=2219] ppid=2217 vsize=328004 CPUtime=1.23 cores=1,3,5,7
/proc/2219/stat : 2219 (python) R 2217 2219 1708 0 -1 4202496 27454 934 1 0 109 14 0 0 25 0 1 0 538445734 335876096 19385 33554432000 4194304 5683644 140736015292896 18446744073709551615 251813898419 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/2219/statm: 82001 19385 1826 364 0 17925 0
Current children cumulated CPU time (s) 1.23
Current children cumulated vsize (KiB) 328004

[startup+3.10073 s]
/proc/loadavg: 8.02 7.90 5.77 6/254 2224
/proc/meminfo: memFree=29879060/32950928 swapFree=67111448/67111528
[pid=2219] ppid=2217 vsize=328004 CPUtime=2.83 cores=1,3,5,7
/proc/2219/stat : 2219 (python) R 2217 2219 1708 0 -1 4202496 27454 934 1 0 269 14 0 0 25 0 1 0 538445734 335876096 19385 33554432000 4194304 5683644 140736015292896 18446744073709551615 251813895268 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/2219/statm: 82001 19385 1826 364 0 17925 0
Current children cumulated CPU time (s) 2.83
Current children cumulated vsize (KiB) 328004

[startup+6.30019 s]
/proc/loadavg: 8.02 7.90 5.77 6/254 2224
/proc/meminfo: memFree=29879504/32950928 swapFree=67111448/67111528
[pid=2219] ppid=2217 vsize=328004 CPUtime=6.03 cores=1,3,5,7
/proc/2219/stat : 2219 (python) R 2217 2219 1708 0 -1 4202496 27455 934 1 0 589 14 0 0 25 0 1 0 538445734 335876096 19386 33554432000 4194304 5683644 140736015292896 18446744073709551615 251817648837 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/2219/statm: 82001 19386 1827 364 0 17925 0
Current children cumulated CPU time (s) 6.03
Current children cumulated vsize (KiB) 328004

[startup+12.7011 s]
/proc/loadavg: 7.55 7.80 5.76 9/266 2236
/proc/meminfo: memFree=29806156/32950928 swapFree=67111448/67111528
[pid=2219] ppid=2217 vsize=365136 CPUtime=11.23 cores=1,3,5,7
/proc/2219/stat : 2219 (python) S 2217 2219 1708 0 -1 4202496 30445 934 1 0 1108 15 0 0 25 0 5 0 538445734 373899264 18480 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/2219/statm: 91284 18480 1881 364 0 27193 0
[pid=2225] ppid=2219 vsize=324124 CPUtime=0 cores=1,3,5,7
/proc/2225/stat : 2225 (python) S 2219 2219 1708 0 -1 4202560 738 0 0 0 0 0 0 0 18 0 1 0 538446875 331902976 16936 33554432000 4194304 5683644 140736015292896 18446744073709551615 251814260239 0 0 16781312 514 4808142912082673663 0 0 17 1 0 0 0
/proc/2225/statm: 81031 16936 353 364 0 16949 0
[pid=2227] ppid=2225 vsize=268420 CPUtime=1.27 cores=1,3,5,7
/proc/2227/stat : 2227 (hybridGM3) R 2225 2227 2227 0 -1 4202496 4727 0 0 0 127 0 0 0 25 0 1 0 538446875 274862080 4465 33554432000 134512640 135204385 4292934624 18446744073709551615 134543968 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/2227/statm: 67105 4465 57 169 0 66935 0
[pid=2228] ppid=2219 vsize=334380 CPUtime=0 cores=1,3,5,7
/proc/2228/stat : 2228 (python) S 2219 2219 1708 0 -1 4202560 738 0 0 0 0 0 0 0 18 0 1 0 538446878 342405120 16934 33554432000 4194304 5683644 140736015292896 18446744073709551615 251814260294 0 0 16781312 514 4579382645976203263 0 0 17 3 0 0 0
/proc/2228/statm: 83595 16934 347 364 0 19510 0
[pid=2230] ppid=2228 vsize=268420 CPUtime=1.25 cores=1,3,5,7
/proc/2230/stat : 2230 (hybridGM3) R 2228 2230 2230 0 -1 4202496 4716 0 0 0 125 0 0 0 25 0 1 0 538446878 274862080 4454 33554432000 134512640 135204385 4288041056 18446744073709551615 134543975 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/2230/statm: 67105 4454 57 169 0 66935 0
[pid=2231] ppid=2219 vsize=344636 CPUtime=0 cores=1,3,5,7
/proc/2231/stat : 2231 (python) S 2219 2219 1708 0 -1 4202560 744 0 0 0 0 0 0 0 18 0 1 0 538446881 352907264 16938 33554432000 4194304 5683644 140736015292896 18446744073709551615 251814260294 0 0 16781312 514 9522558226038521855 0 0 17 1 0 0 0
/proc/2231/statm: 86159 16938 347 364 0 22071 0
[pid=2233] ppid=2231 vsize=268420 CPUtime=1.22 cores=1,3,5,7
/proc/2233/stat : 2233 (hybridGM3) R 2231 2233 2233 0 -1 4202496 4697 0 0 0 121 1 0 0 25 0 1 0 538446881 274862080 4431 33554432000 134512640 135204385 4287660208 18446744073709551615 134543993 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/2233/statm: 67105 4431 57 169 0 66935 0
[pid=2234] ppid=2219 vsize=354892 CPUtime=0 cores=1,3,5,7
/proc/2234/stat : 2234 (python) S 2219 2219 1708 0 -1 4202560 759 0 0 0 0 0 0 0 18 0 1 0 538446884 363409408 16942 33554432000 4194304 5683644 140736015292896 18446744073709551615 251814260294 0 0 16781312 514 4579371101104111615 0 0 17 3 0 0 0
/proc/2234/statm: 88723 16942 347 364 0 24632 0
[pid=2236] ppid=2234 vsize=268416 CPUtime=1.18 cores=1,3,5,7
/proc/2236/stat : 2236 (hybridGM3) R 2234 2236 2236 0 -1 4202496 4672 0 0 0 118 0 0 0 25 0 1 0 538446884 274857984 4409 33554432000 134512640 135204385 4293598160 18446744073709551615 134543968 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/2236/statm: 67104 4409 57 169 0 66934 0
[pid=2219/tid=2226] ppid=2217 vsize=365136 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2226/stat : 2226 (python) S 2217 2219 1708 0 -1 4202560 26 934 0 0 0 0 0 0 25 0 5 0 538446875 373899264 18480 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 1 0 0 0
[pid=2219/tid=2229] ppid=2217 vsize=365136 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2229/stat : 2229 (python) S 2217 2219 1708 0 -1 4202560 21 934 0 0 0 0 0 0 25 0 5 0 538446878 373899264 18480 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 5 0 0 0
[pid=2219/tid=2232] ppid=2217 vsize=365136 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2232/stat : 2232 (python) S 2217 2219 1708 0 -1 4202560 19 934 0 0 0 0 0 0 25 0 5 0 538446881 373899264 18480 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 1 0 0 0
[pid=2219/tid=2235] ppid=2217 vsize=365136 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2235/stat : 2235 (python) S 2217 2219 1708 0 -1 4202560 27 934 0 0 0 0 0 0 25 0 5 0 538446884 373899264 18480 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 16.15
Current children cumulated vsize (KiB) 2796844

[startup+25.501 s]
/proc/loadavg: 7.62 7.81 5.78 9/266 2236
/proc/meminfo: memFree=29719748/32950928 swapFree=67111448/67111528
[pid=2219] ppid=2217 vsize=365136 CPUtime=11.23 cores=1,3,5,7
/proc/2219/stat : 2219 (python) S 2217 2219 1708 0 -1 4202496 30445 934 1 0 1108 15 0 0 25 0 5 0 538445734 373899264 18480 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/2219/statm: 91284 18480 1881 364 0 27193 0
[pid=2225] ppid=2219 vsize=324124 CPUtime=0.01 cores=1,3,5,7
/proc/2225/stat : 2225 (python) S 2219 2219 1708 0 -1 4202560 910 0 0 0 0 1 0 0 15 0 1 0 538446875 331902976 16959 33554432000 4194304 5683644 140736015292896 18446744073709551615 251814260239 0 0 16781312 514 16897875237801001224 0 0 17 1 0 0 0
/proc/2225/statm: 81031 16959 376 364 0 16949 0
[pid=2227] ppid=2225 vsize=276240 CPUtime=13.98 cores=1,3,5,7
/proc/2227/stat : 2227 (hybridGM3) R 2225 2227 2227 0 -1 4202496 9517 0 0 0 1397 1 0 0 25 0 1 0 538446875 282869760 9255 33554432000 134512640 135204385 4292934624 18446744073709551615 134536965 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/2227/statm: 69060 9255 64 169 0 68890 0
[pid=2228] ppid=2219 vsize=334380 CPUtime=0.01 cores=1,3,5,7
/proc/2228/stat : 2228 (python) S 2219 2219 1708 0 -1 4202560 902 0 0 0 0 1 0 0 15 0 1 0 538446878 342405120 16957 33554432000 4194304 5683644 140736015292896 18446744073709551615 251814260294 0 0 16781312 514 16897875237801001224 0 0 17 3 0 0 0
/proc/2228/statm: 83595 16957 370 364 0 19510 0
[pid=2230] ppid=2228 vsize=276240 CPUtime=14.04 cores=1,3,5,7
/proc/2230/stat : 2230 (hybridGM3) R 2228 2230 2230 0 -1 4202496 9511 0 0 0 1404 0 0 0 25 0 1 0 538446878 282869760 9249 33554432000 134512640 135204385 4288041056 18446744073709551615 134545851 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/2230/statm: 69060 9249 64 169 0 68890 0
[pid=2231] ppid=2219 vsize=344636 CPUtime=0.01 cores=1,3,5,7
/proc/2231/stat : 2231 (python) S 2219 2219 1708 0 -1 4202560 909 0 0 0 1 0 0 0 15 0 1 0 538446881 352907264 16961 33554432000 4194304 5683644 140736015292896 18446744073709551615 251814260294 0 0 16781312 514 16897875237801001224 0 0 17 1 0 0 0
/proc/2231/statm: 86159 16961 370 364 0 22071 0
[pid=2233] ppid=2231 vsize=276240 CPUtime=13.97 cores=1,3,5,7
/proc/2233/stat : 2233 (hybridGM3) R 2231 2233 2233 0 -1 4202496 9518 0 0 0 1396 1 0 0 25 0 1 0 538446881 282869760 9252 33554432000 134512640 135204385 4287660208 18446744073709551615 134537292 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/2233/statm: 69060 9252 64 169 0 68890 0
[pid=2234] ppid=2219 vsize=354892 CPUtime=0.02 cores=1,3,5,7
/proc/2234/stat : 2234 (python) S 2219 2219 1708 0 -1 4202560 939 0 0 0 1 1 0 0 15 0 1 0 538446884 363409408 16965 33554432000 4194304 5683644 140736015292896 18446744073709551615 251814260294 0 0 16781312 514 16897875237801001223 0 0 17 3 0 0 0
/proc/2234/statm: 88723 16965 370 364 0 24632 0
[pid=2236] ppid=2234 vsize=276236 CPUtime=13.95 cores=1,3,5,7
/proc/2236/stat : 2236 (hybridGM3) R 2234 2236 2236 0 -1 4202496 9516 0 0 0 1394 1 0 0 25 0 1 0 538446884 282865664 9253 33554432000 134512640 135204385 4293598160 18446744073709551615 134545837 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/2236/statm: 69059 9253 64 169 0 68889 0
[pid=2219/tid=2226] ppid=2217 vsize=365136 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2226/stat : 2226 (python) S 2217 2219 1708 0 -1 4202560 26 934 0 0 0 0 0 0 25 0 5 0 538446875 373899264 18480 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 1 0 0 0
[pid=2219/tid=2229] ppid=2217 vsize=365136 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2229/stat : 2229 (python) S 2217 2219 1708 0 -1 4202560 21 934 0 0 0 0 0 0 25 0 5 0 538446878 373899264 18480 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 5 0 0 0
[pid=2219/tid=2232] ppid=2217 vsize=365136 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2232/stat : 2232 (python) S 2217 2219 1708 0 -1 4202560 19 934 0 0 0 0 0 0 25 0 5 0 538446881 373899264 18480 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 1 0 0 0
[pid=2219/tid=2235] ppid=2217 vsize=365136 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2235/stat : 2235 (python) S 2217 2219 1708 0 -1 4202560 27 934 0 0 0 0 0 0 25 0 5 0 538446884 373899264 18480 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 67.22
Current children cumulated vsize (KiB) 2828124

[startup+51.1007 s]
/proc/loadavg: 7.75 7.82 5.84 9/267 2238
/proc/meminfo: memFree=29716240/32950928 swapFree=67111448/67111528
[pid=2219] ppid=2217 vsize=365136 CPUtime=11.23 cores=1,3,5,7
/proc/2219/stat : 2219 (python) S 2217 2219 1708 0 -1 4202496 30445 934 1 0 1108 15 0 0 25 0 5 0 538445734 373899264 18480 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/2219/statm: 91284 18480 1881 364 0 27193 0
[pid=2225] ppid=2219 vsize=324124 CPUtime=0.05 cores=1,3,5,7
/proc/2225/stat : 2225 (python) S 2219 2219 1708 0 -1 4202560 918 0 0 0 2 3 0 0 15 0 1 0 538446875 331902976 16959 33554432000 4194304 5683644 140736015292896 18446744073709551615 251814260239 0 0 16781312 514 16897875237801001224 0 0 17 1 0 0 0
/proc/2225/statm: 81031 16959 376 364 0 16949 0
[pid=2227] ppid=2225 vsize=276240 CPUtime=39.39 cores=1,3,5,7
/proc/2227/stat : 2227 (hybridGM3) R 2225 2227 2227 0 -1 4202496 9517 0 0 0 3938 1 0 0 25 0 1 0 538446875 282869760 9255 33554432000 134512640 135204385 4292934624 18446744073709551615 134546144 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/2227/statm: 69060 9255 64 169 0 68890 0
[pid=2228] ppid=2219 vsize=334380 CPUtime=0.05 cores=1,3,5,7
/proc/2228/stat : 2228 (python) S 2219 2219 1708 0 -1 4202560 911 0 0 0 2 3 0 0 15 0 1 0 538446878 342405120 16957 33554432000 4194304 5683644 140736015292896 18446744073709551615 251814260294 0 0 16781312 514 16897875237801001224 0 0 17 3 0 0 0
/proc/2228/statm: 83595 16957 370 364 0 19510 0
[pid=2230] ppid=2228 vsize=276240 CPUtime=39.64 cores=1,3,5,7
/proc/2230/stat : 2230 (hybridGM3) R 2228 2230 2230 0 -1 4202496 9513 0 0 0 3964 0 0 0 25 0 1 0 538446878 282869760 9251 33554432000 134512640 135204385 4288041056 18446744073709551615 134537510 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/2230/statm: 69060 9251 64 169 0 68890 0
[pid=2231] ppid=2219 vsize=344636 CPUtime=0.05 cores=1,3,5,7
/proc/2231/stat : 2231 (python) S 2219 2219 1708 0 -1 4202560 918 0 0 0 3 2 0 0 15 0 1 0 538446881 352907264 16961 33554432000 4194304 5683644 140736015292896 18446744073709551615 251814260294 0 0 16781312 514 16897875237801001224 0 0 17 1 0 0 0
/proc/2231/statm: 86159 16961 370 364 0 22071 0
[pid=2233] ppid=2231 vsize=276240 CPUtime=39.49 cores=1,3,5,7
/proc/2233/stat : 2233 (hybridGM3) R 2231 2233 2233 0 -1 4202496 9518 0 0 0 3948 1 0 0 25 0 1 0 538446881 282869760 9252 33554432000 134512640 135204385 4287660208 18446744073709551615 134537292 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/2233/statm: 69060 9252 64 169 0 68890 0
[pid=2234] ppid=2219 vsize=354892 CPUtime=0.06 cores=1,3,5,7
/proc/2234/stat : 2234 (python) S 2219 2219 1708 0 -1 4202560 950 0 0 0 3 3 0 0 15 0 1 0 538446884 363409408 16965 33554432000 4194304 5683644 140736015292896 18446744073709551615 251814260294 0 0 16781312 514 16897875237801001223 0 0 17 3 0 0 0
/proc/2234/statm: 88723 16965 370 364 0 24632 0

################
# More data... #
################

[pid=2365] ppid=2219 vsize=375404 CPUtime=0.81 cores=1,3,5,7
/proc/2365/stat : 2365 (python) S 2219 2219 1708 0 -1 4202560 1187 0 0 0 42 39 0 0 15 0 2 0 538476447 384413696 17003 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/2365/statm: 93851 17003 399 364 0 29754 0
[pid=2367] ppid=2365 vsize=276240 CPUtime=487.38 cores=1,3,5,7
/proc/2367/stat : 2367 (hybridGM3) T 2365 2367 2367 0 -1 4202496 9517 0 0 0 48737 1 0 0 25 0 1 0 538476447 282869760 9251 33554432000 134512640 135204385 4294693008 18446744073709551615 134547687 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/2367/statm: 69060 9251 64 169 0 68890 0
[pid=2365/tid=2387] ppid=2219 vsize=375404 CPUtime=0 cores=1,3,5,7
/proc/2365/task/2387/stat : 2387 (python) S 2219 2219 1708 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 538496096 384413696 17003 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 514 18446604454225328192 0 0 -1 1 0 0 0
[pid=2368] ppid=2219 vsize=385660 CPUtime=0.65 cores=1,3,5,7
/proc/2368/stat : 2368 (python) S 2219 2219 1708 0 -1 4202560 1228 0 0 0 32 33 0 0 16 0 2 0 538476541 394915840 17013 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/2368/statm: 96415 17013 405 364 0 32315 0
[pid=2371] ppid=2368 vsize=276240 CPUtime=392.6 cores=1,3,5,7
/proc/2371/stat : 2371 (hybridGM3) T 2368 2371 2371 0 -1 4202496 9534 0 0 0 39259 1 0 0 25 0 1 0 538476541 282869760 9265 33554432000 134512640 135204385 4294354768 18446744073709551615 134546073 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/2371/statm: 69060 9265 64 169 0 68890 0
[pid=2368/tid=2388] ppid=2219 vsize=385660 CPUtime=0 cores=1,3,5,7
/proc/2368/task/2388/stat : 2388 (python) S 2219 2219 1708 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 538496185 394915840 17013 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 514 18446604454225329088 0 0 -1 1 0 0 0
[pid=2372] ppid=2219 vsize=395916 CPUtime=0.49 cores=1,3,5,7
/proc/2372/stat : 2372 (python) S 2219 2219 1708 0 -1 4202560 1225 0 0 0 28 21 0 0 15 0 2 0 538476544 405417984 17017 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2372/statm: 98979 17017 405 364 0 34876 0
[pid=2374] ppid=2372 vsize=276240 CPUtime=295 cores=1,3,5,7
/proc/2374/stat : 2374 (hybridGM3) T 2372 2374 2374 0 -1 4202496 9530 0 0 0 29499 1 0 0 25 0 1 0 538476545 282869760 9259 33554432000 134512640 135204385 4286581744 18446744073709551615 134536887 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/2374/statm: 69060 9259 64 169 0 68890 0
[pid=2372/tid=2389] ppid=2219 vsize=395916 CPUtime=0 cores=1,3,5,7
/proc/2372/task/2389/stat : 2389 (python) S 2219 2219 1708 0 -1 4202560 50 0 0 0 0 0 0 0 15 0 2 0 538496191 405417984 17017 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 514 18446604455838808128 0 0 -1 7 0 0 0
[pid=2375] ppid=2219 vsize=406172 CPUtime=0.17 cores=1,3,5,7
/proc/2375/stat : 2375 (python) S 2219 2219 1708 0 -1 4202560 1152 0 0 0 8 9 0 0 19 0 2 0 538476550 415920128 17015 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2375/statm: 101543 17015 399 364 0 37437 0
[pid=2377] ppid=2375 vsize=471116 CPUtime=98.21 cores=1,3,5,7
/proc/2377/stat : 2377 (TNM) T 2375 2377 2377 0 -1 4202496 8438 0 0 0 9820 1 0 0 25 0 1 0 538476550 482422784 8171 33554432000 134512640 135008748 4292891488 18446744073709551615 134529409 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/2377/statm: 117779 8171 46 122 0 116679 0
[pid=2375/tid=2381] ppid=2219 vsize=406172 CPUtime=0 cores=1,3,5,7
/proc/2375/task/2381/stat : 2381 (python) S 2219 2219 1708 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 538486568 415920128 17015 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 514 18446604464341392064 0 0 -1 7 0 0 0
[pid=2382] ppid=2219 vsize=416428 CPUtime=0.32 cores=1,3,5,7
/proc/2382/stat : 2382 (python) S 2219 2219 1708 0 -1 4202560 1185 0 0 0 17 15 0 0 16 0 2 0 538486572 426422272 17019 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2382/statm: 104107 17019 399 364 0 39998 0
[pid=2384] ppid=2382 vsize=276240 CPUtime=194.66 cores=1,3,5,7
/proc/2384/stat : 2384 (hybridGM3) T 2382 2384 2384 0 -1 4202496 9526 0 0 0 19465 1 0 0 25 0 1 0 538486572 282869760 9250 33554432000 134512640 135204385 4290359792 18446744073709551615 134547392 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/2384/statm: 69060 9250 64 169 0 68890 0
[pid=2382/tid=2393] ppid=2219 vsize=416428 CPUtime=0 cores=1,3,5,7
/proc/2382/task/2393/stat : 2393 (python) S 2219 2219 1708 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 538506229 426422272 17019 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 514 18446604455838806336 0 0 -1 7 0 0 0
[pid=2394] ppid=2219 vsize=426684 CPUtime=0.17 cores=1,3,5,7
/proc/2394/stat : 2394 (python) S 2219 2219 1708 0 -1 4202560 1202 0 0 0 8 9 0 0 19 0 2 0 538506233 436924416 17023 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/2394/statm: 106671 17023 399 364 0 42559 0
[pid=2396] ppid=2394 vsize=471116 CPUtime=98.18 cores=1,3,5,7
/proc/2396/stat : 2396 (TNM) T 2394 2396 2396 0 -1 4202496 8443 0 0 0 9816 2 0 0 25 0 1 0 538506233 482422784 8171 33554432000 134512640 135008748 4288284464 18446744073709551615 134527729 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/2396/statm: 117779 8171 46 122 0 116679 0
[pid=2394/tid=2407] ppid=2219 vsize=426684 CPUtime=0 cores=1,3,5,7
/proc/2394/task/2407/stat : 2407 (python) S 2219 2219 1708 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538516250 436924416 17023 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 514 18446604458051616960 0 0 -1 7 0 0 0
[pid=2397] ppid=2219 vsize=436940 CPUtime=0.17 cores=1,3,5,7
/proc/2397/stat : 2397 (python) S 2219 2219 1708 0 -1 4202560 1177 0 0 0 8 9 0 0 19 0 2 0 538506314 447426560 17027 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2397/statm: 109235 17027 399 364 0 45120 0
[pid=2399] ppid=2397 vsize=471116 CPUtime=99.82 cores=1,3,5,7
/proc/2399/stat : 2399 (TNM) T 2397 2399 2399 0 -1 4202496 8444 0 0 0 9981 1 0 0 25 0 1 0 538506315 482422784 8172 33554432000 134512640 135008748 4292644656 18446744073709551615 134529123 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/2399/statm: 117779 8172 46 122 0 116679 0
[pid=2397/tid=2411] ppid=2219 vsize=436940 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2411/stat : 2411 (python) S 2219 2219 1708 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538516331 447426560 17027 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 514 18446604458051619648 0 0 -1 7 0 0 0
[pid=2404] ppid=2219 vsize=447196 CPUtime=0.54 cores=1,3,5,7
/proc/2404/stat : 2404 (python) S 2219 2219 1708 0 -1 4202560 1215 0 0 0 29 25 0 0 15 0 2 0 538515826 457928704 17031 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2404/statm: 111799 17031 399 364 0 47681 0
[pid=2406] ppid=2404 vsize=276244 CPUtime=297.05 cores=1,3,5,7
/proc/2406/stat : 2406 (hybridGM3) T 2404 2406 2406 0 -1 4202496 9530 0 0 0 29702 3 0 0 25 0 1 0 538515827 282873856 9258 33554432000 134512640 135204385 4287805328 18446744073709551615 134537401 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/2406/statm: 69061 9258 64 169 0 68891 0
[pid=2404/tid=2441] ppid=2219 vsize=447196 CPUtime=0 cores=1,3,5,7
/proc/2404/task/2441/stat : 2441 (python) S 2219 2219 1708 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 538535689 457928704 17031 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 514 18446604471120756992 0 0 -1 7 0 0 0
[pid=2408] ppid=2219 vsize=457452 CPUtime=0.17 cores=1,3,5,7
/proc/2408/stat : 2408 (python) S 2219 2219 1708 0 -1 4202560 1184 0 0 0 9 8 0 0 19 0 2 0 538516255 468430848 17035 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/2408/statm: 114363 17035 399 364 0 50242 0
[pid=2410] ppid=2408 vsize=471116 CPUtime=100.15 cores=1,3,5,7
/proc/2410/stat : 2410 (TNM) T 2408 2410 2410 0 -1 4202496 8447 0 0 0 10013 2 0 0 25 0 1 0 538516255 482422784 8172 33554432000 134512640 135008748 4292554256 18446744073709551615 134527693 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/2410/statm: 117779 8172 46 122 0 116679 0
[pid=2408/tid=2434] ppid=2219 vsize=457452 CPUtime=0 cores=1,3,5,7
/proc/2408/task/2434/stat : 2434 (python) S 2219 2219 1708 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 538526271 468430848 17035 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 514 18446604471100422080 0 0 -1 3 0 0 0
[pid=2412] ppid=2219 vsize=467708 CPUtime=0.16 cores=1,3,5,7
/proc/2412/stat : 2412 (python) S 2219 2219 1708 0 -1 4202560 1244 0 0 0 10 6 0 0 19 0 2 0 538516336 478932992 17047 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2412/statm: 116927 17047 407 364 0 52803 0
[pid=2414] ppid=2412 vsize=471116 CPUtime=99 cores=1,3,5,7
/proc/2414/stat : 2414 (TNM) T 2412 2414 2414 0 -1 4202496 8446 0 0 0 9899 1 0 0 25 0 1 0 538516336 482422784 8171 33554432000 134512640 135008748 4294050256 18446744073709551615 134529475 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/2414/statm: 117779 8171 46 122 0 116679 0
[pid=2412/tid=2430] ppid=2219 vsize=467708 CPUtime=0 cores=1,3,5,7
/proc/2412/task/2430/stat : 2430 (python) S 2219 2219 1708 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538526253 478932992 17047 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 514 18446604471120757888 0 0 -1 1 0 0 0
[pid=2427] ppid=2219 vsize=477964 CPUtime=0.2 cores=1,3,5,7
/proc/2427/stat : 2427 (python) S 2219 2219 1708 0 -1 4202560 1225 0 0 0 9 11 0 0 19 0 2 0 538525258 489435136 17051 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2427/statm: 119491 17051 407 364 0 55364 0
[pid=2429] ppid=2427 vsize=471116 CPUtime=101.82 cores=1,3,5,7
/proc/2429/stat : 2429 (TNM) T 2427 2429 2429 0 -1 4202496 8445 0 0 0 10181 1 0 0 25 0 1 0 538525258 482422784 8171 33554432000 134512640 135008748 4292182112 18446744073709551615 134529195 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/2429/statm: 117779 8171 46 122 0 116679 0
[pid=2427/tid=2442] ppid=2219 vsize=477964 CPUtime=0 cores=1,3,5,7
/proc/2427/task/2442/stat : 2442 (python) S 2219 2219 1708 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 538535779 489435136 17051 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 514 18446604455777227840 0 0 -1 3 0 0 0
[pid=2431] ppid=2219 vsize=488220 CPUtime=0.18 cores=1,3,5,7
/proc/2431/stat : 2431 (python) S 2219 2219 1708 0 -1 4202560 1202 0 0 0 7 11 0 0 19 0 2 0 538526258 499937280 17049 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/2431/statm: 122055 17049 401 364 0 57925 0
[pid=2433] ppid=2431 vsize=471116 CPUtime=98.19 cores=1,3,5,7
/proc/2433/stat : 2433 (TNM) T 2431 2433 2433 0 -1 4202496 8448 0 0 0 9818 1 0 0 25 0 1 0 538526258 482422784 8172 33554432000 134512640 135008748 4293550560 18446744073709551615 134529448 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/2433/statm: 117779 8172 46 122 0 116679 0
[pid=2431/tid=2446] ppid=2219 vsize=488220 CPUtime=0 cores=1,3,5,7
/proc/2431/task/2446/stat : 2446 (python) S 2219 2219 1708 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 538536078 499937280 17049 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 514 18446604455777226944 0 0 -1 7 0 0 0
[pid=2443] ppid=2219 vsize=498476 CPUtime=0.18 cores=1,3,5,7
/proc/2443/stat : 2443 (python) S 2219 2219 1708 0 -1 4202560 1067 0 0 0 7 11 0 0 15 0 1 0 538535783 510439424 17023 33554432000 4194304 5683644 140736015292896 18446744073709551615 251814260294 0 0 16781312 514 16897875237801001221 0 0 17 7 0 0 0
/proc/2443/statm: 124619 17023 372 364 0 60486 0
[pid=2445] ppid=2443 vsize=276240 CPUtime=96.69 cores=1,3,5,7
/proc/2445/stat : 2445 (hybridGM3) R 2443 2445 2445 0 -1 4202496 9530 0 0 0 9667 2 0 0 25 0 1 0 538535784 282869760 9251 33554432000 134512640 135204385 4294470064 18446744073709551615 134537383 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/2445/statm: 69060 9251 64 169 0 68890 0
[pid=2447] ppid=2219 vsize=508732 CPUtime=0.17 cores=1,3,5,7
/proc/2447/stat : 2447 (python) S 2219 2219 1708 0 -1 4202560 1025 0 0 0 10 7 0 0 15 0 1 0 538536082 520941568 17026 33554432000 4194304 5683644 140736015292896 18446744073709551615 251814260294 0 0 16781312 514 16897875237801001223 0 0 17 7 0 0 0
/proc/2447/statm: 127183 17026 371 364 0 63047 0
[pid=2449] ppid=2447 vsize=276236 CPUtime=97.42 cores=1,3,5,7
/proc/2449/stat : 2449 (hybridGM3) R 2447 2449 2449 0 -1 4202496 9530 0 0 0 9740 2 0 0 25 0 1 0 538536083 282865664 9250 33554432000 134512640 135204385 4293526192 18446744073709551615 134545968 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/2449/statm: 69059 9250 64 169 0 68889 0
[pid=2457] ppid=2219 vsize=518988 CPUtime=0 cores=1,3,5,7
/proc/2457/stat : 2457 (python) S 2219 2219 1708 0 -1 4202560 849 0 0 0 0 0 0 0 19 0 1 0 538545816 531443712 17004 33554432000 4194304 5683644 140736015292896 18446744073709551615 251814260294 0 0 16781312 514 4328728079703736319 0 0 17 7 0 0 0
/proc/2457/statm: 129747 17004 345 364 0 65608 0
[pid=2459] ppid=2457 vsize=268424 CPUtime=0.06 cores=1,3,5,7
/proc/2459/stat : 2459 (hybridGM3) R 2457 2459 2459 0 -1 4202496 1661 0 0 0 6 0 0 0 21 0 1 0 538545816 274866176 1386 33554432000 134512640 135204385 4291509792 18446744073709551615 134684316 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/2459/statm: 67106 1386 57 169 0 66936 0
[pid=2219/tid=2226] ppid=2217 vsize=529232 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2226/stat : 2226 (python) S 2217 2219 1708 0 -1 4202560 35 934 0 0 0 0 0 0 18 0 21 0 538446875 541933568 18577 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 7 0 0 0
[pid=2219/tid=2229] ppid=2217 vsize=529232 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2229/stat : 2229 (python) S 2217 2219 1708 0 -1 4202560 24 934 0 0 0 0 0 0 15 0 21 0 538446878 541933568 18577 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 1 0 0 0
[pid=2219/tid=2232] ppid=2217 vsize=529232 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2232/stat : 2232 (python) S 2217 2219 1708 0 -1 4202560 28 934 0 0 0 0 0 0 15 0 21 0 538446881 541933568 18577 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 1 0 0 0
[pid=2219/tid=2235] ppid=2217 vsize=529232 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2235/stat : 2235 (python) S 2217 2219 1708 0 -1 4202560 27 934 0 0 0 0 0 0 18 0 21 0 538446884 541933568 18577 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 3 0 0 0
[pid=2219/tid=2355] ppid=2217 vsize=529232 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2355/stat : 2355 (python) S 2217 2219 1708 0 -1 4202560 29 934 0 0 0 0 0 0 18 0 21 0 538466429 541933568 18577 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 5 0 0 0
[pid=2219/tid=2366] ppid=2217 vsize=529232 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2366/stat : 2366 (python) S 2217 2219 1708 0 -1 4202560 37 934 0 0 0 0 0 0 16 0 21 0 538476447 541933568 18577 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 1 0 0 0
[pid=2219/tid=2370] ppid=2217 vsize=529232 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2370/stat : 2370 (python) S 2217 2219 1708 0 -1 4202560 29 934 0 0 0 0 0 0 18 0 21 0 538476541 541933568 18577 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 1 0 0 0
[pid=2219/tid=2373] ppid=2217 vsize=529232 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2373/stat : 2373 (python) S 2217 2219 1708 0 -1 4202560 82 934 0 0 0 0 0 0 18 0 21 0 538476545 541933568 18577 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 1 0 0 0
[pid=2219/tid=2376] ppid=2217 vsize=529232 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2376/stat : 2376 (python) S 2217 2219 1708 0 -1 4202560 28 934 0 0 0 0 0 0 23 0 21 0 538476550 541933568 18577 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 7 0 0 0
[pid=2219/tid=2383] ppid=2217 vsize=529232 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2383/stat : 2383 (python) S 2217 2219 1708 0 -1 4202560 24 934 0 0 0 0 0 0 20 0 21 0 538486572 541933568 18577 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 7 0 0 0
[pid=2219/tid=2395] ppid=2217 vsize=529232 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2395/stat : 2395 (python) S 2217 2219 1708 0 -1 4202560 31 934 0 0 0 0 0 0 20 0 21 0 538506233 541933568 18577 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 7 0 0 0
[pid=2219/tid=2398] ppid=2217 vsize=529232 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2398/stat : 2398 (python) S 2217 2219 1708 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 21 0 538506315 541933568 18577 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 7 0 0 0
[pid=2219/tid=2405] ppid=2217 vsize=529232 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2405/stat : 2405 (python) S 2217 2219 1708 0 -1 4202560 37 934 0 0 0 0 0 0 18 0 21 0 538515827 541933568 18577 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 7 0 0 0
[pid=2219/tid=2409] ppid=2217 vsize=529232 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2409/stat : 2409 (python) S 2217 2219 1708 0 -1 4202560 30 934 0 0 0 0 0 0 20 0 21 0 538516255 541933568 18577 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 7 0 0 0
[pid=2219/tid=2413] ppid=2217 vsize=529232 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2413/stat : 2413 (python) S 2217 2219 1708 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 21 0 538516336 541933568 18577 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 7 0 0 0
[pid=2219/tid=2428] ppid=2217 vsize=529232 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2428/stat : 2428 (python) S 2217 2219 1708 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 21 0 538525258 541933568 18577 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 7 0 0 0
[pid=2219/tid=2432] ppid=2217 vsize=529232 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2432/stat : 2432 (python) S 2217 2219 1708 0 -1 4202560 28 934 0 0 0 0 0 0 20 0 21 0 538526258 541933568 18577 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 7 0 0 0
[pid=2219/tid=2444] ppid=2217 vsize=529232 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2444/stat : 2444 (python) S 2217 2219 1708 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 21 0 538535784 541933568 18577 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 7 0 0 0
[pid=2219/tid=2448] ppid=2217 vsize=529232 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2448/stat : 2448 (python) S 2217 2219 1708 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 21 0 538536083 541933568 18577 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 7 0 0 0
[pid=2219/tid=2458] ppid=2217 vsize=529232 CPUtime=0 cores=1,3,5,7
/proc/2219/task/2458/stat : 2458 (python) S 2217 2219 1708 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 21 0 538545816 541933568 18577 33554432000 4194304 5683644 140736015292896 18446744073709551615 251826063793 0 0 16781312 2 18446604455838809024 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3951.51
Current children cumulated vsize (KiB) 16046584

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 2219 and gives
#  childrusage.ru_utime.tv_sec=12
#  childrusage.ru_utime.tv_usec=218142
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=241963
# CPU time returned by wait4() is 12.4601
# while last known CPU time is 3951.51
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1001.03
CPU time (s): 3951.51
CPU user time (s): 3947.84
CPU system time (s): 3.67
CPU usage (%): 394.743
Max. virtual memory (cumulated for all children) (KiB): 16046584

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 12.2181
system time used= 0.241963
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 48607
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 3402
involuntary context switches= 128

runsolver used 5.33819 second user time and 11.3203 second system time

The end

Launcher Data 

Begin job on node126 at 2011-05-01 18:47:50
IDJOB=3317236
IDBENCH=82701
IDSOLVER=1705
FILE ID=node126/3317236-1304268470
RUNJOBID= node126-1304267283-1727
PBS_JOBID= 13173953
Free space on /tmp= 73212 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v50000-c210000-S964643921-035-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317236-1304268470/watcher-3317236-1304268470 -o /tmp/evaluation-result-3317236-1304268470/solver-3317236-1304268470 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317236-1304268470.cnf -seed 809406488 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= d8337de8a71a630ccf2dd02943cd7d7f
RANDOM SEED=809406488

node126.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.56
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      29953004 kB
Buffers:        315364 kB
Cached:        1677856 kB
SwapCached:         76 kB
Active:         981312 kB
Inactive:      1682096 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      29953004 kB
SwapTotal:    67111528 kB
SwapFree:     67111448 kB
Dirty:           36140 kB
Writeback:           0 kB
AnonPages:      670028 kB
Mapped:          22880 kB
Slab:           254924 kB
PageTables:      19628 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 15853104 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73176 MiB
End job on node126 at 2011-05-01 19:04:33