Trace number 3317112

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (MO) 4312.13 1091.23

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v50000-c210000-S746725327-020-UNKNOWN.cnf
MD5SUM0ad997eea0def3552b6d7e45aac8ca7c
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables49999
Number of clauses210000
Sum of the clauses size630000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3210000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.33/0.47	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.33/0.47	c INFO: solving HOME/instance-3317112-1304267332.cnf
0.33/0.47	c INFO: computing task features
1.12/1.20	c INFO: computed variable-clause graph statistics
1.12/1.23	c INFO: computed clause balance statistics
4.92/5.03	c INFO: computed variable balance statistics
7.82/7.97	c INFO: computed small-clause counts
7.82/7.99	c INFO: computed Horn-clause counts
10.92/11.07	c INFO: computed variable graph statistics
10.92/11.07	c INFO: collected features for HOME/instance-3317112-1304267332.cnf in 10.57 s
11.01/11.10	c INFO: running hybridGM3@0 for 97 with 4789 remaining (b = 0.33)
11.01/11.13	c INFO: running hybridGM3@0 for 97 with 4789 remaining (b = 0.21)
11.01/11.16	c INFO: running hybridGM3@0 for 97 with 4789 remaining (b = 0.14)
11.01/11.19	c INFO: running hybridGM3@0 for 97 with 4789 remaining (b = 0.09)
411.05/111.39	c INFO: running hybridGM3@0 for 97 with 4689 remaining (b = 0.07)
411.43/111.41	c INFO: running hybridGM3@0 for 97 with 4689 remaining (b = 0.07)
411.43/111.45	c INFO: running hybridGM3@0 for 97 with 4689 remaining (b = 0.07)
411.43/111.48	c INFO: running hybridGM3@99 for 97 with 4689 remaining (b = 0.07)
791.21/206.55	c INFO: running hybridGM3@99 for 97 with 4593 remaining (b = 0.08)
791.59/206.60	c INFO: running TNM@0 for 97 with 4593 remaining (b = 0.08)
791.59/206.63	c INFO: running hybridGM3@100 for 97 with 4593 remaining (b = 0.07)
811.13/211.54	c INFO: running hybridGM3@100 for 97 with 4588 remaining (b = 0.07)
1186.95/305.70	c INFO: running hybridGM3@0 for 194 with 4494 remaining (b = 0.13)
1187.36/305.78	c INFO: running hybridGM3@0 for 194 with 4494 remaining (b = 0.09)
1195.31/307.72	c INFO: running hybridGM3@0 for 194 with 4492 remaining (b = 0.07)
1195.65/307.80	c INFO: running TNM@0 for 97 with 4492 remaining (b = 0.03)
1594.45/407.98	c INFO: running hybridGM3@0 for 194 with 4392 remaining (b = 0.05)
1969.68/502.18	c INFO: running hybridGM3@99 for 97 with 4298 remaining (b = 0.02)
1970.48/502.30	c INFO: running hybridGM3@100 for 97 with 4298 remaining (b = 0.02)
1978.04/504.20	c INFO: running hybridGM3@101 for 97 with 4296 remaining (b = 0.02)
2348.95/597.34	c INFO: running hybridGM3@295 for 97 with 4203 remaining (b = 0.02)
2373.22/603.49	c INFO: running hybridGM3@294 for 97 with 4197 remaining (b = 0.02)
2377.59/604.51	c INFO: running TNM@0 for 97 with 4195 remaining (b = 0.02)
2380.77/605.39	c INFO: running TNM@0 for 97 with 4195 remaining (b = 0.02)
2752.08/698.52	c INFO: running hybridGM3@0 for 194 with 4101 remaining (b = 0.04)
2752.36/698.66	c INFO: running hybridGM3@395 for 97 with 4101 remaining (b = 0.01)
2775.91/704.58	c INFO: running TNM@0 for 97 with 4095 remaining (b = 0.01)
2776.60/704.72	c INFO: running TNM@0 for 97 with 4095 remaining (b = 0.01)
3131.03/793.85	c INFO: running TNM@0 for 97 with 4006 remaining (b = 0.01)
3174.34/804.79	c INFO: running TNM@0 for 97 with 3995 remaining (b = 0.01)
3194.94/809.95	c INFO: running TNM@94 for 1940 with 3990 remaining (b = 0.20)
3529.08/894.07	c INFO: running hybridGM3@0 for 194 with 3906 remaining (b = 0.03)
3533.45/895.13	c INFO: running hybridGM3@95 for 97 with 3905 remaining (b = 0.01)
3572.75/905.01	c INFO: running hybridGM3@0 for 194 with 3895 remaining (b = 0.02)
3935.44/996.34	c INFO: running hybridGM3@192 for 97 with 3804 remaining (b = 0.01)
4310.08/1090.67	c INFO: running hybridGM3@0 for 194 with 3709 remaining (b = 0.02)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317112-1304267332/watcher-3317112-1304267332 -o /tmp/evaluation-result-3317112-1304267332/solver-3317112-1304267332 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317112-1304267332.cnf -seed 1962411866 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.80 0.69 0.23 5/189 5773
/proc/meminfo: memFree=21684324/32951124 swapFree=67111524/67111528
[pid=5773] ppid=5771 vsize=15388 CPUtime=0 cores=1,3,5,7
/proc/5773/stat : 5773 (python) R 5771 5773 5629 0 -1 4202496 814 0 0 0 0 0 0 0 25 0 1 0 519148827 15757312 699 33554432000 4194304 5683644 140735154016144 18446744073709551615 4540219 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/5773/statm: 3847 699 334 364 0 334 0

[startup+0.0945901 s]
/proc/loadavg: 2.80 0.69 0.23 5/189 5773
/proc/meminfo: memFree=21684324/32951124 swapFree=67111524/67111528
[pid=5773] ppid=5771 vsize=99376 CPUtime=0.07 cores=1,3,5,7
/proc/5773/stat : 5773 (python) S 5771 5773 5629 0 -1 4202496 2892 0 0 0 5 2 0 0 17 0 1 0 519148827 101761024 2588 33554432000 4194304 5683644 140735154016144 18446744073709551615 264726403072 0 2147483391 16781312 2 18446744071699207358 0 0 17 7 0 0 0
/proc/5773/statm: 24844 2588 832 364 0 1695 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 99376

[startup+0.10033 s]
/proc/loadavg: 2.80 0.69 0.23 5/189 5773
/proc/meminfo: memFree=21684324/32951124 swapFree=67111524/67111528
[pid=5773] ppid=5771 vsize=104220 CPUtime=0.07 cores=1,3,5,7
/proc/5773/stat : 5773 (python) S 5771 5773 5629 0 -1 4202496 3046 0 0 0 5 2 0 0 17 0 1 0 519148827 106721280 2732 33554432000 4194304 5683644 140735154016144 18446744073709551615 264726403072 0 2147483391 16781312 2 18446744071699207358 0 0 17 7 0 0 0
/proc/5773/statm: 26055 2732 847 364 0 1803 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 104220

[startup+0.301025 s]
/proc/loadavg: 2.80 0.69 0.23 5/189 5773
/proc/meminfo: memFree=21684324/32951124 swapFree=67111524/67111528
[pid=5773] ppid=5771 vsize=168676 CPUtime=0.23 cores=1,3,5,7
/proc/5773/stat : 5773 (python) S 5771 5773 5629 0 -1 4202496 6722 932 1 0 17 6 0 0 17 0 1 0 519148827 172724224 4798 33554432000 4194304 5683644 140735154016144 18446744073709551615 264726403072 0 2147483391 16781312 2 18446744071699207358 0 0 17 7 0 0 0
/proc/5773/statm: 42169 4798 1331 364 0 3454 0
Current children cumulated CPU time (s) 0.23
Current children cumulated vsize (KiB) 168676

[startup+0.700246 s]
/proc/loadavg: 2.80 0.69 0.23 5/189 5773
/proc/meminfo: memFree=21684324/32951124 swapFree=67111524/67111528
[pid=5773] ppid=5771 vsize=296780 CPUtime=0.61 cores=1,3,5,7
/proc/5773/stat : 5773 (python) R 5771 5773 5629 0 -1 4202496 13797 932 1 0 51 10 0 0 17 0 1 0 519148827 303902720 11479 33554432000 4194304 5683644 140735154016144 18446744073709551615 264726065861 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/5773/statm: 74195 11479 1761 364 0 10119 0
Current children cumulated CPU time (s) 0.61
Current children cumulated vsize (KiB) 296780

[startup+1.50109 s]
/proc/loadavg: 2.80 0.69 0.23 6/190 5778
/proc/meminfo: memFree=21626924/32951124 swapFree=67111524/67111528
[pid=5773] ppid=5771 vsize=328000 CPUtime=1.42 cores=1,3,5,7
/proc/5773/stat : 5773 (python) R 5771 5773 5629 0 -1 4202496 27453 932 1 0 130 12 0 0 18 0 1 0 519148827 335872000 19385 33554432000 4194304 5683644 140735154016144 18446744073709551615 4539713 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/5773/statm: 82000 19385 1826 364 0 17924 0
Current children cumulated CPU time (s) 1.42
Current children cumulated vsize (KiB) 328000

[startup+3.10078 s]
/proc/loadavg: 2.80 0.69 0.23 6/190 5778
/proc/meminfo: memFree=21612664/32951124 swapFree=67111524/67111528
[pid=5773] ppid=5771 vsize=328000 CPUtime=3.02 cores=1,3,5,7
/proc/5773/stat : 5773 (python) R 5771 5773 5629 0 -1 4202496 27453 932 1 0 290 12 0 0 22 0 1 0 519148827 335872000 19385 33554432000 4194304 5683644 140735154016144 18446744073709551615 47606933432098 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/5773/statm: 82000 19385 1826 364 0 17924 0
Current children cumulated CPU time (s) 3.02
Current children cumulated vsize (KiB) 328000

[startup+6.30017 s]
/proc/loadavg: 2.97 0.76 0.25 7/190 5778
/proc/meminfo: memFree=21613004/32951124 swapFree=67111524/67111528
[pid=5773] ppid=5771 vsize=328000 CPUtime=6.22 cores=1,3,5,7
/proc/5773/stat : 5773 (python) R 5771 5773 5629 0 -1 4202496 27454 932 1 0 610 12 0 0 25 0 1 0 519148827 335872000 19386 33554432000 4194304 5683644 140735154016144 18446744073709551615 264725765888 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/5773/statm: 82000 19386 1827 364 0 17924 0
Current children cumulated CPU time (s) 6.22
Current children cumulated vsize (KiB) 328000

[startup+12.7009 s]
/proc/loadavg: 3.14 0.83 0.28 12/202 5790
/proc/meminfo: memFree=21563180/32951124 swapFree=67111524/67111528
[pid=5773] ppid=5771 vsize=365132 CPUtime=11.1 cores=1,3,5,7
/proc/5773/stat : 5773 (python) S 5771 5773 5629 0 -1 4202496 30406 932 1 0 1097 13 0 0 20 0 5 0 519148827 373895168 18480 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738231755 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/5773/statm: 91283 18480 1881 364 0 27192 0
[pid=5779] ppid=5773 vsize=324120 CPUtime=0 cores=1,3,5,7
/proc/5779/stat : 5779 (python) S 5773 5773 5629 0 -1 4202560 732 0 0 0 0 0 0 0 18 0 1 0 519149937 331898880 16936 33554432000 4194304 5683644 140735154016144 18446744073709551615 264726425103 0 0 16781312 514 4597726623096111103 0 0 17 7 0 0 0
/proc/5779/statm: 81030 16936 353 364 0 16948 0
[pid=5781] ppid=5779 vsize=276236 CPUtime=1.57 cores=1,3,5,7
/proc/5781/stat : 5781 (hybridGM3) R 5779 5781 5781 0 -1 4202496 4940 0 0 0 157 0 0 0 25 0 1 0 519149938 282865664 4679 33554432000 134512640 135204385 4289044832 18446744073709551615 134543975 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/5781/statm: 69059 4679 57 169 0 68889 0
[pid=5782] ppid=5773 vsize=334376 CPUtime=0 cores=1,3,5,7
/proc/5782/stat : 5782 (python) S 5773 5773 5629 0 -1 4202560 741 0 0 0 0 0 0 0 18 0 1 0 519149940 342401024 16934 33554432000 4194304 5683644 140735154016144 18446744073709551615 264726425158 0 0 16781312 514 4459630986304421887 0 0 17 5 0 0 0
/proc/5782/statm: 83594 16934 347 364 0 19509 0
[pid=5784] ppid=5782 vsize=276236 CPUtime=1.55 cores=1,3,5,7
/proc/5784/stat : 5784 (hybridGM3) R 5782 5784 5784 0 -1 4202496 4929 0 0 0 155 0 0 0 25 0 1 0 519149941 282865664 4666 33554432000 134512640 135204385 4287105552 18446744073709551615 134543968 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/5784/statm: 69059 4666 57 169 0 68889 0
[pid=5785] ppid=5773 vsize=344632 CPUtime=0 cores=1,3,5,7
/proc/5785/stat : 5785 (python) S 5773 5773 5629 0 -1 4202560 747 0 0 0 0 0 0 0 18 0 1 0 519149944 352903168 16938 33554432000 4194304 5683644 140735154016144 18446744073709551615 264726425158 0 0 16781312 514 4577675104418267135 0 0 17 5 0 0 0
/proc/5785/statm: 86158 16938 347 364 0 22070 0
[pid=5787] ppid=5785 vsize=276240 CPUtime=1.52 cores=1,3,5,7
/proc/5787/stat : 5787 (hybridGM3) R 5785 5787 5787 0 -1 4202496 4902 0 0 0 152 0 0 0 25 0 1 0 519149944 282869760 4640 33554432000 134512640 135204385 4287149920 18446744073709551615 134543975 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/5787/statm: 69060 4640 57 169 0 68890 0
[pid=5788] ppid=5773 vsize=354888 CPUtime=0 cores=1,3,5,7
/proc/5788/stat : 5788 (python) S 5773 5773 5629 0 -1 4202560 752 0 0 0 0 0 0 0 18 0 1 0 519149947 363405312 16942 33554432000 4194304 5683644 140735154016144 18446744073709551615 264726425158 0 0 16781312 514 4577671256127569919 0 0 17 7 0 0 0
/proc/5788/statm: 88722 16942 347 364 0 24631 0
[pid=5790] ppid=5788 vsize=276236 CPUtime=1.49 cores=1,3,5,7
/proc/5790/stat : 5790 (hybridGM3) R 5788 5790 5790 0 -1 4202496 4884 0 0 0 149 0 0 0 25 0 1 0 519149947 282865664 4623 33554432000 134512640 135204385 4288065440 18446744073709551615 134543968 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/5790/statm: 69059 4623 57 169 0 68889 0
[pid=5773/tid=5780] ppid=5771 vsize=365132 CPUtime=0 cores=1,3,5,7
/proc/5773/task/5780/stat : 5780 (python) S 5771 5773 5629 0 -1 4202560 25 932 0 0 0 0 0 0 25 0 5 0 519149938 373895168 18480 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 7 0 0 0
[pid=5773/tid=5783] ppid=5771 vsize=365132 CPUtime=0 cores=1,3,5,7
/proc/5773/task/5783/stat : 5783 (python) S 5771 5773 5629 0 -1 4202560 24 932 0 0 0 0 0 0 25 0 5 0 519149941 373895168 18480 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 5 0 0 0
[pid=5773/tid=5786] ppid=5771 vsize=365132 CPUtime=0 cores=1,3,5,7
/proc/5773/task/5786/stat : 5786 (python) S 5771 5773 5629 0 -1 4202560 21 932 0 0 0 0 0 0 25 0 5 0 519149944 373895168 18480 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 5 0 0 0
[pid=5773/tid=5789] ppid=5771 vsize=365132 CPUtime=0 cores=1,3,5,7
/proc/5773/task/5789/stat : 5789 (python) S 5771 5773 5629 0 -1 4202560 30 932 0 0 0 0 0 0 20 0 5 0 519149947 373895168 18480 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 17.23
Current children cumulated vsize (KiB) 2828096

[startup+25.5015 s]
/proc/loadavg: 4.01 1.14 0.39 9/202 5834
/proc/meminfo: memFree=21576788/32951124 swapFree=67111524/67111528
[pid=5773] ppid=5771 vsize=365132 CPUtime=11.1 cores=1,3,5,7
/proc/5773/stat : 5773 (python) S 5771 5773 5629 0 -1 4202496 30406 932 1 0 1097 13 0 0 20 0 5 0 519148827 373895168 18480 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738231755 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/5773/statm: 91283 18480 1881 364 0 27192 0
[pid=5779] ppid=5773 vsize=324120 CPUtime=0.01 cores=1,3,5,7
/proc/5779/stat : 5779 (python) S 5773 5773 5629 0 -1 4202560 894 0 0 0 1 0 0 0 15 0 1 0 519149937 331898880 16959 33554432000 4194304 5683644 140735154016144 18446744073709551615 264726425103 0 0 16781312 514 6280093758507680008 0 0 17 7 0 0 0
/proc/5779/statm: 81030 16959 376 364 0 16948 0
[pid=5781] ppid=5779 vsize=276240 CPUtime=14.26 cores=1,3,5,7
/proc/5781/stat : 5781 (hybridGM3) R 5779 5781 5781 0 -1 4202496 9511 0 0 0 1422 4 0 0 25 0 1 0 519149938 282869760 9250 33554432000 134512640 135204385 4289044832 18446744073709551615 134537455 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/5781/statm: 69060 9250 64 169 0 68890 0
[pid=5782] ppid=5773 vsize=334376 CPUtime=0.01 cores=1,3,5,7
/proc/5782/stat : 5782 (python) S 5773 5773 5629 0 -1 4202560 908 0 0 0 0 1 0 0 15 0 1 0 519149940 342401024 16957 33554432000 4194304 5683644 140735154016144 18446744073709551615 264726425158 0 0 16781312 514 6280093758507680008 0 0 17 5 0 0 0
/proc/5782/statm: 83594 16957 370 364 0 19509 0
[pid=5784] ppid=5782 vsize=276240 CPUtime=14.35 cores=1,3,5,7
/proc/5784/stat : 5784 (hybridGM3) R 5782 5784 5784 0 -1 4202496 9513 0 0 0 1434 1 0 0 25 0 1 0 519149941 282869760 9250 33554432000 134512640 135204385 4287105552 18446744073709551615 134545851 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/5784/statm: 69060 9250 64 169 0 68890 0
[pid=5785] ppid=5773 vsize=344632 CPUtime=0 cores=1,3,5,7
/proc/5785/stat : 5785 (python) S 5773 5773 5629 0 -1 4202560 921 0 0 0 0 0 0 0 15 0 1 0 519149944 352903168 16961 33554432000 4194304 5683644 140735154016144 18446744073709551615 264726425158 0 0 16781312 514 6280093758507680008 0 0 17 5 0 0 0
/proc/5785/statm: 86158 16961 370 364 0 22070 0
[pid=5787] ppid=5785 vsize=276244 CPUtime=14.29 cores=1,3,5,7
/proc/5787/stat : 5787 (hybridGM3) R 5785 5787 5787 0 -1 4202496 9518 0 0 0 1428 1 0 0 25 0 1 0 519149944 282873856 9256 33554432000 134512640 135204385 4287149920 18446744073709551615 134537272 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/5787/statm: 69061 9256 64 169 0 68891 0
[pid=5788] ppid=5773 vsize=354888 CPUtime=0.01 cores=1,3,5,7
/proc/5788/stat : 5788 (python) S 5773 5773 5629 0 -1 4202560 924 0 0 0 0 1 0 0 15 0 1 0 519149947 363405312 16965 33554432000 4194304 5683644 140735154016144 18446744073709551615 264726425158 0 0 16781312 514 6280093758507680008 0 0 17 7 0 0 0
/proc/5788/statm: 88722 16965 370 364 0 24631 0
[pid=5790] ppid=5788 vsize=276240 CPUtime=14.29 cores=1,3,5,7
/proc/5790/stat : 5790 (hybridGM3) R 5788 5790 5790 0 -1 4202496 9511 0 0 0 1428 1 0 0 25 0 1 0 519149947 282869760 9250 33554432000 134512640 135204385 4288065440 18446744073709551615 134547418 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/5790/statm: 69060 9250 64 169 0 68890 0
[pid=5773/tid=5780] ppid=5771 vsize=365132 CPUtime=0 cores=1,3,5,7
/proc/5773/task/5780/stat : 5780 (python) S 5771 5773 5629 0 -1 4202560 25 932 0 0 0 0 0 0 25 0 5 0 519149938 373895168 18480 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 7 0 0 0
[pid=5773/tid=5783] ppid=5771 vsize=365132 CPUtime=0 cores=1,3,5,7
/proc/5773/task/5783/stat : 5783 (python) S 5771 5773 5629 0 -1 4202560 24 932 0 0 0 0 0 0 25 0 5 0 519149941 373895168 18480 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 5 0 0 0
[pid=5773/tid=5786] ppid=5771 vsize=365132 CPUtime=0 cores=1,3,5,7
/proc/5773/task/5786/stat : 5786 (python) S 5771 5773 5629 0 -1 4202560 21 932 0 0 0 0 0 0 25 0 5 0 519149944 373895168 18480 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 5 0 0 0
[pid=5773/tid=5789] ppid=5771 vsize=365132 CPUtime=0 cores=1,3,5,7
/proc/5773/task/5789/stat : 5789 (python) S 5771 5773 5629 0 -1 4202560 30 932 0 0 0 0 0 0 20 0 5 0 519149947 373895168 18480 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 68.32
Current children cumulated vsize (KiB) 2828112

[startup+51.1007 s]
/proc/loadavg: 5.43 1.70 0.59 9/202 5835
/proc/meminfo: memFree=21576204/32951124 swapFree=67111524/67111528
[pid=5773] ppid=5771 vsize=365132 CPUtime=11.1 cores=1,3,5,7
/proc/5773/stat : 5773 (python) S 5771 5773 5629 0 -1 4202496 30406 932 1 0 1097 13 0 0 20 0 5 0 519148827 373895168 18480 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738231755 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/5773/statm: 91283 18480 1881 364 0 27192 0
[pid=5779] ppid=5773 vsize=324120 CPUtime=0.04 cores=1,3,5,7
/proc/5779/stat : 5779 (python) S 5773 5773 5629 0 -1 4202560 900 0 0 0 3 1 0 0 15 0 1 0 519149937 331898880 16959 33554432000 4194304 5683644 140735154016144 18446744073709551615 264726425103 0 0 16781312 514 6280093758507680008 0 0 17 7 0 0 0
/proc/5779/statm: 81030 16959 376 364 0 16948 0
[pid=5781] ppid=5779 vsize=276240 CPUtime=39.64 cores=1,3,5,7
/proc/5781/stat : 5781 (hybridGM3) R 5779 5781 5781 0 -1 4202496 9519 0 0 0 3960 4 0 0 25 0 1 0 519149938 282869760 9258 33554432000 134512640 135204385 4289044832 18446744073709551615 134536993 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/5781/statm: 69060 9258 64 169 0 68890 0
[pid=5782] ppid=5773 vsize=334376 CPUtime=0.04 cores=1,3,5,7
/proc/5782/stat : 5782 (python) S 5773 5773 5629 0 -1 4202560 916 0 0 0 1 3 0 0 15 0 1 0 519149940 342401024 16957 33554432000 4194304 5683644 140735154016144 18446744073709551615 264726425158 0 0 16781312 514 6280093758507680008 0 0 17 5 0 0 0
/proc/5782/statm: 83594 16957 370 364 0 19509 0
[pid=5784] ppid=5782 vsize=276240 CPUtime=39.95 cores=1,3,5,7
/proc/5784/stat : 5784 (hybridGM3) R 5782 5784 5784 0 -1 4202496 9513 0 0 0 3994 1 0 0 25 0 1 0 519149941 282869760 9250 33554432000 134512640 135204385 4287105552 18446744073709551615 134547654 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/5784/statm: 69060 9250 64 169 0 68890 0
[pid=5785] ppid=5773 vsize=344632 CPUtime=0.04 cores=1,3,5,7
/proc/5785/stat : 5785 (python) S 5773 5773 5629 0 -1 4202560 930 0 0 0 2 2 0 0 15 0 1 0 519149944 352903168 16961 33554432000 4194304 5683644 140735154016144 18446744073709551615 264726425158 0 0 16781312 514 6280093758507680008 0 0 17 5 0 0 0
/proc/5785/statm: 86158 16961 370 364 0 22070 0
[pid=5787] ppid=5785 vsize=276244 CPUtime=39.83 cores=1,3,5,7
/proc/5787/stat : 5787 (hybridGM3) R 5785 5787 5787 0 -1 4202496 9518 0 0 0 3982 1 0 0 25 0 1 0 519149944 282873856 9256 33554432000 134512640 135204385 4287149920 18446744073709551615 134536737 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/5787/statm: 69061 9256 64 169 0 68891 0
[pid=5788] ppid=5773 vsize=354888 CPUtime=0.04 cores=1,3,5,7
/proc/5788/stat : 5788 (python) S 5773 5773 5629 0 -1 4202560 931 0 0 0 2 2 0 0 15 0 1 0 519149947 363405312 16965 33554432000 4194304 5683644 140735154016144 18446744073709551615 264726425158 0 0 16781312 514 6280093758507680008 0 0 17 7 0 0 0
/proc/5788/statm: 88722 16965 370 364 0 24631 0

################
# More data... #
################

/proc/5882/statm: 93850 17003 399 364 0 29753 0
[pid=5884] ppid=5882 vsize=276236 CPUtime=487.14 cores=1,3,5,7
/proc/5884/stat : 5884 (hybridGM3) T 5882 5884 5884 0 -1 4202496 9523 0 0 0 48712 2 0 0 25 0 1 0 519179398 282865664 9260 33554432000 134512640 135204385 4294435296 18446744073709551615 134536663 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/5884/statm: 69059 9260 64 169 0 68889 0
[pid=5882/tid=5917] ppid=5773 vsize=375400 CPUtime=0 cores=1,3,5,7
/proc/5882/task/5917/stat : 5917 (python) S 5773 5773 5629 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 519199042 384409600 17003 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 514 18446604471159767424 0 0 -1 5 0 0 0
[pid=5886] ppid=5773 vsize=385656 CPUtime=0.61 cores=1,3,5,7
/proc/5886/stat : 5886 (python) S 5773 5773 5629 0 -1 4202560 1224 0 0 0 32 29 0 0 16 0 2 0 519179406 394911744 17013 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738231755 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5886/statm: 96414 17013 405 364 0 32314 0
[pid=5888] ppid=5886 vsize=276236 CPUtime=391.07 cores=1,3,5,7
/proc/5888/stat : 5888 (hybridGM3) T 5886 5888 5888 0 -1 4202496 9531 0 0 0 39106 1 0 0 25 0 1 0 519179407 282865664 9263 33554432000 134512640 135204385 4287378208 18446744073709551615 134537403 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/5888/statm: 69059 9263 64 169 0 68889 0
[pid=5886/tid=5918] ppid=5773 vsize=385656 CPUtime=0 cores=1,3,5,7
/proc/5886/task/5918/stat : 5918 (python) S 5773 5773 5629 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 519199054 394911744 17013 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 514 18446604471159769216 0 0 -1 5 0 0 0
[pid=5889] ppid=5773 vsize=395912 CPUtime=0.47 cores=1,3,5,7
/proc/5889/stat : 5889 (python) S 5773 5773 5629 0 -1 4202560 1203 0 0 0 22 25 0 0 15 0 2 0 519179600 405413888 17017 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738231755 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5889/statm: 98978 17017 405 364 0 34875 0
[pid=5891] ppid=5889 vsize=276240 CPUtime=294.81 cores=1,3,5,7
/proc/5891/stat : 5891 (hybridGM3) T 5889 5891 5891 0 -1 4202496 9526 0 0 0 29479 2 0 0 25 0 1 0 519179600 282869760 9256 33554432000 134512640 135204385 4293856656 18446744073709551615 134537233 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/5891/statm: 69060 9256 64 169 0 68890 0
[pid=5889/tid=5919] ppid=5773 vsize=395912 CPUtime=0 cores=1,3,5,7
/proc/5889/task/5919/stat : 5919 (python) S 5773 5773 5629 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 519199244 405413888 17017 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 514 18446604470440769536 0 0 -1 5 0 0 0
[pid=5892] ppid=5773 vsize=406168 CPUtime=0.14 cores=1,3,5,7
/proc/5892/stat : 5892 (python) S 5773 5773 5629 0 -1 4202560 1145 0 0 0 7 7 0 0 19 0 2 0 519179608 415916032 17015 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738231755 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5892/statm: 101542 17015 399 364 0 37436 0
[pid=5894] ppid=5892 vsize=471116 CPUtime=98.49 cores=1,3,5,7
/proc/5894/stat : 5894 (TNM) T 5892 5894 5894 0 -1 4202496 8438 0 0 0 9848 1 0 0 25 0 1 0 519179608 482422784 8170 33554432000 134512640 135008748 4287924800 18446744073709551615 134527693 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/5894/statm: 117779 8170 46 122 0 116679 0
[pid=5892/tid=5909] ppid=5773 vsize=406168 CPUtime=0 cores=1,3,5,7
/proc/5892/task/5909/stat : 5909 (python) S 5773 5773 5629 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 519189622 415916032 17015 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 514 18446604471135322240 0 0 -1 5 0 0 0
[pid=5910] ppid=5773 vsize=416424 CPUtime=0.3 cores=1,3,5,7
/proc/5910/stat : 5910 (python) S 5773 5773 5629 0 -1 4202560 1177 0 0 0 17 13 0 0 17 0 2 0 519189626 426418176 17019 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738231755 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5910/statm: 104106 17019 399 364 0 39997 0
[pid=5912] ppid=5910 vsize=276240 CPUtime=195.92 cores=1,3,5,7
/proc/5912/stat : 5912 (hybridGM3) T 5910 5912 5912 0 -1 4202496 9522 0 0 0 19590 2 0 0 25 0 1 0 519189627 282869760 9252 33554432000 134512640 135204385 4288109104 18446744073709551615 134537437 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/5912/statm: 69060 9252 64 169 0 68890 0
[pid=5910/tid=5928] ppid=5773 vsize=416424 CPUtime=0 cores=1,3,5,7
/proc/5910/task/5928/stat : 5928 (python) S 5773 5773 5629 0 -1 4202560 51 0 0 0 0 0 0 0 17 0 2 0 519209276 426418176 17019 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 514 18446604471135324928 0 0 -1 5 0 0 0
[pid=5929] ppid=5773 vsize=426680 CPUtime=0.15 cores=1,3,5,7
/proc/5929/stat : 5929 (python) S 5773 5773 5629 0 -1 4202560 1167 0 0 0 7 8 0 0 19 0 2 0 519209279 436920320 17023 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738231755 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5929/statm: 106670 17023 399 364 0 42558 0
[pid=5931] ppid=5929 vsize=471116 CPUtime=100.14 cores=1,3,5,7
/proc/5931/stat : 5931 (TNM) T 5929 5931 5931 0 -1 4202496 8436 0 0 0 10012 2 0 0 25 0 1 0 519209280 482422784 8169 33554432000 134512640 135008748 4291492064 18446744073709551615 134527693 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/5931/statm: 117779 8169 46 122 0 116679 0
[pid=5929/tid=5997] ppid=5773 vsize=426680 CPUtime=0 cores=1,3,5,7
/proc/5929/task/5997/stat : 5997 (python) S 5773 5773 5629 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 519219296 436920320 17023 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 514 18446604471163053184 0 0 -1 5 0 0 0
[pid=5932] ppid=5773 vsize=436936 CPUtime=0.16 cores=1,3,5,7
/proc/5932/stat : 5932 (python) S 5773 5773 5629 0 -1 4202560 1157 0 0 0 8 8 0 0 19 0 2 0 519209367 447422464 17027 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738231755 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5932/statm: 109234 17027 399 364 0 45119 0
[pid=5934] ppid=5932 vsize=471116 CPUtime=97.5 cores=1,3,5,7
/proc/5934/stat : 5934 (TNM) T 5932 5934 5934 0 -1 4202496 8447 0 0 0 9748 2 0 0 25 0 1 0 519209367 482422784 8173 33554432000 134512640 135008748 4294137184 18446744073709551615 134529300 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/5934/statm: 117779 8173 46 122 0 116679 0
[pid=5932/tid=5993] ppid=5773 vsize=436936 CPUtime=0 cores=1,3,5,7
/proc/5932/task/5993/stat : 5993 (python) S 5773 5773 5629 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 519219282 447422464 17027 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 514 18446604471138053120 0 0 -1 5 0 0 0
[pid=5990] ppid=5773 vsize=447192 CPUtime=0.69 cores=1,3,5,7
/proc/5990/stat : 5990 (python) S 5773 5773 5629 0 -1 4202560 1192 0 0 0 32 37 0 0 15 0 2 0 519218680 457924608 17031 33554432000 4194304 5683644 140735154016144 18446744073709551615 264726425158 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5990/statm: 111798 17031 399 364 0 47680 0
[pid=5992] ppid=5990 vsize=276240 CPUtime=391.26 cores=1,3,5,7
/proc/5992/stat : 5992 (hybridGM3) R 5990 5992 5992 0 -1 4202496 9526 0 0 0 39123 3 0 0 25 0 1 0 519218681 282869760 9257 33554432000 134512640 135204385 4293497536 18446744073709551615 134537049 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/5992/statm: 69060 9257 64 169 0 68890 0
[pid=5990/tid=6050] ppid=5773 vsize=447192 CPUtime=0 cores=1,3,5,7
/proc/5990/task/6050/stat : 6050 (python) S 5773 5773 5629 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 519238336 457924608 17031 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 514 18446604471142119296 0 0 -1 5 0 0 0
[pid=5994] ppid=5773 vsize=457448 CPUtime=0.16 cores=1,3,5,7
/proc/5994/stat : 5994 (python) S 5773 5773 5629 0 -1 4202560 1154 0 0 0 9 7 0 0 19 0 2 0 519219286 468426752 17035 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738231755 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5994/statm: 114362 17035 399 364 0 50241 0
[pid=5996] ppid=5994 vsize=471116 CPUtime=99.54 cores=1,3,5,7
/proc/5996/stat : 5996 (TNM) T 5994 5996 5996 0 -1 4202496 8443 0 0 0 9953 1 0 0 25 0 1 0 519219287 482422784 8171 33554432000 134512640 135008748 4293166592 18446744073709551615 134527416 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/5996/statm: 117779 8171 46 122 0 116679 0
[pid=5994/tid=6023] ppid=5773 vsize=457448 CPUtime=0 cores=1,3,5,7
/proc/5994/task/6023/stat : 6023 (python) S 5773 5773 5629 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 519229302 468426752 17035 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 514 18446604469089996992 0 0 -1 5 0 0 0
[pid=5998] ppid=5773 vsize=467704 CPUtime=0.17 cores=1,3,5,7
/proc/5998/stat : 5998 (python) S 5773 5773 5629 0 -1 4202560 1218 0 0 0 8 9 0 0 19 0 2 0 519219300 478928896 17047 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738231755 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5998/statm: 116926 17047 407 364 0 52802 0
[pid=6000] ppid=5998 vsize=471116 CPUtime=102.69 cores=1,3,5,7
/proc/6000/stat : 6000 (TNM) T 5998 6000 6000 0 -1 4202496 8441 0 0 0 10264 5 0 0 25 0 1 0 519219301 482422784 8170 33554432000 134512640 135008748 4289928608 18446744073709551615 134527327 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/6000/statm: 117779 8170 46 122 0 116679 0
[pid=5998/tid=6027] ppid=5773 vsize=467704 CPUtime=0 cores=1,3,5,7
/proc/5998/task/6027/stat : 6027 (python) S 5773 5773 5629 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 519229818 478928896 17047 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 514 18446604471135414976 0 0 -1 5 0 0 0
[pid=6020] ppid=5773 vsize=477960 CPUtime=0.18 cores=1,3,5,7
/proc/6020/stat : 6020 (python) S 5773 5773 5629 0 -1 4202560 1212 0 0 0 10 8 0 0 19 0 2 0 519228212 489431040 17051 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738231755 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/6020/statm: 119490 17051 407 364 0 55363 0
[pid=6022] ppid=6020 vsize=471116 CPUtime=100.09 cores=1,3,5,7
/proc/6022/stat : 6022 (TNM) T 6020 6022 6022 0 -1 4202496 8447 0 0 0 10008 1 0 0 25 0 1 0 519228213 482422784 8172 33554432000 134512640 135008748 4288473200 18446744073709551615 134527773 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/6022/statm: 117779 8172 46 122 0 116679 0
[pid=6020/tid=6046] ppid=5773 vsize=477960 CPUtime=0 cores=1,3,5,7
/proc/6020/task/6046/stat : 6046 (python) S 5773 5773 5629 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 519238230 489431040 17051 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 514 18446604464973898048 0 0 -1 3 0 0 0
[pid=6024] ppid=5773 vsize=488216 CPUtime=0.17 cores=1,3,5,7
/proc/6024/stat : 6024 (python) S 5773 5773 5629 0 -1 4202560 1194 0 0 0 6 11 0 0 19 0 2 0 519229307 499933184 17049 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738231755 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/6024/statm: 122054 17049 401 364 0 57924 0
[pid=6026] ppid=6024 vsize=471116 CPUtime=97.41 cores=1,3,5,7
/proc/6026/stat : 6026 (TNM) T 6024 6026 6026 0 -1 4202496 8452 0 0 0 9739 2 0 0 25 0 1 0 519229308 482422784 8172 33554432000 134512640 135008748 4292168432 18446744073709551615 134528967 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/6026/statm: 117779 8172 46 122 0 116679 0
[pid=6024/tid=6052] ppid=5773 vsize=488216 CPUtime=0 cores=1,3,5,7
/proc/6024/task/6052/stat : 6052 (python) S 5773 5773 5629 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 519239325 499933184 17049 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 514 18446604464973899840 0 0 -1 7 0 0 0
[pid=6047] ppid=5773 vsize=498472 CPUtime=0.36 cores=1,3,5,7
/proc/6047/stat : 6047 (python) S 5773 5773 5629 0 -1 4202560 1198 0 0 0 19 17 0 0 16 0 2 0 519238235 510435328 17053 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738231755 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/6047/statm: 124618 17053 401 364 0 60485 0
[pid=6049] ppid=6047 vsize=276240 CPUtime=196.17 cores=1,3,5,7
/proc/6049/stat : 6049 (hybridGM3) T 6047 6049 6049 0 -1 4202496 9531 0 0 0 19615 2 0 0 25 0 1 0 519238235 282869760 9258 33554432000 134512640 135204385 4294942064 18446744073709551615 134545975 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/6049/statm: 69060 9258 64 169 0 68890 0
[pid=6047/tid=6078] ppid=5773 vsize=498472 CPUtime=0 cores=1,3,5,7
/proc/6047/task/6078/stat : 6078 (python) S 5773 5773 5629 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 519257890 510435328 17053 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 514 18446604468302078336 0 0 -1 3 0 0 0
[pid=6053] ppid=5773 vsize=508728 CPUtime=0.33 cores=1,3,5,7
/proc/6053/stat : 6053 (python) S 5773 5773 5629 0 -1 4202560 1071 0 0 0 20 13 0 0 15 0 1 0 519239329 520937472 17026 33554432000 4194304 5683644 140735154016144 18446744073709551615 264726425158 0 0 16781312 514 6280093758507680007 0 0 17 5 0 0 0
/proc/6053/statm: 127182 17026 371 364 0 63046 0
[pid=6055] ppid=6053 vsize=276236 CPUtime=180.65 cores=1,3,5,7
/proc/6055/stat : 6055 (hybridGM3) R 6053 6055 6055 0 -1 4202496 9528 0 0 0 18063 2 0 0 25 0 1 0 519239330 282865664 9247 33554432000 134512640 135204385 4288581040 18446744073709551615 134537583 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/6055/statm: 69059 9247 64 169 0 68889 0
[pid=6079] ppid=5773 vsize=518984 CPUtime=0 cores=1,3,5,7
/proc/6079/stat : 6079 (python) S 5773 5773 5629 0 -1 4202560 849 0 0 0 0 0 0 0 19 0 1 0 519257895 531439616 17004 33554432000 4194304 5683644 140735154016144 18446744073709551615 264726425158 0 0 16781312 514 16853791148014043135 0 0 17 5 0 0 0
/proc/6079/statm: 129746 17004 345 364 0 65607 0
[pid=6081] ppid=6079 vsize=268420 CPUtime=0.43 cores=1,3,5,7
/proc/6081/stat : 6081 (hybridGM3) R 6079 6081 6081 0 -1 4202496 4127 0 0 0 42 1 0 0 21 0 1 0 519257896 274862080 3847 33554432000 134512640 135204385 4291312768 18446744073709551615 134543993 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/6081/statm: 67105 3847 57 169 0 66935 0
[pid=5773/tid=5780] ppid=5771 vsize=529228 CPUtime=0 cores=1,3,5,7
/proc/5773/task/5780/stat : 5780 (python) S 5771 5773 5629 0 -1 4202560 32 932 0 0 0 0 0 0 15 0 21 0 519149938 541929472 18577 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 5 0 0 0
[pid=5773/tid=5783] ppid=5771 vsize=529228 CPUtime=0 cores=1,3,5,7
/proc/5773/task/5783/stat : 5783 (python) S 5771 5773 5629 0 -1 4202560 28 932 0 0 0 0 0 0 15 0 21 0 519149941 541929472 18577 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 7 0 0 0
[pid=5773/tid=5786] ppid=5771 vsize=529228 CPUtime=0 cores=1,3,5,7
/proc/5773/task/5786/stat : 5786 (python) S 5771 5773 5629 0 -1 4202560 26 932 0 0 0 0 0 0 15 0 21 0 519149944 541929472 18577 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 5 0 0 0
[pid=5773/tid=5789] ppid=5771 vsize=529228 CPUtime=0 cores=1,3,5,7
/proc/5773/task/5789/stat : 5789 (python) S 5771 5773 5629 0 -1 4202560 30 932 0 0 0 0 0 0 18 0 21 0 519149947 541929472 18577 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 7 0 0 0
[pid=5773/tid=5861] ppid=5771 vsize=529228 CPUtime=0 cores=1,3,5,7
/proc/5773/task/5861/stat : 5861 (python) S 5771 5773 5629 0 -1 4202560 26 932 0 0 0 0 0 0 18 0 21 0 519169488 541929472 18577 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 7 0 0 0
[pid=5773/tid=5883] ppid=5771 vsize=529228 CPUtime=0 cores=1,3,5,7
/proc/5773/task/5883/stat : 5883 (python) S 5771 5773 5629 0 -1 4202560 38 932 0 0 0 0 0 0 18 0 21 0 519179398 541929472 18577 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 5 0 0 0
[pid=5773/tid=5887] ppid=5771 vsize=529228 CPUtime=0 cores=1,3,5,7
/proc/5773/task/5887/stat : 5887 (python) S 5771 5773 5629 0 -1 4202560 31 932 0 0 0 0 0 0 18 0 21 0 519179407 541929472 18577 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 5 0 0 0
[pid=5773/tid=5890] ppid=5771 vsize=529228 CPUtime=0 cores=1,3,5,7
/proc/5773/task/5890/stat : 5890 (python) S 5771 5773 5629 0 -1 4202560 87 932 0 0 0 0 0 0 18 0 21 0 519179600 541929472 18577 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 5 0 0 0
[pid=5773/tid=5893] ppid=5771 vsize=529228 CPUtime=0 cores=1,3,5,7
/proc/5773/task/5893/stat : 5893 (python) S 5771 5773 5629 0 -1 4202560 29 932 0 0 0 0 0 0 21 0 21 0 519179608 541929472 18577 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 5 0 0 0
[pid=5773/tid=5911] ppid=5771 vsize=529228 CPUtime=0 cores=1,3,5,7
/proc/5773/task/5911/stat : 5911 (python) S 5771 5773 5629 0 -1 4202560 25 932 0 0 0 0 0 0 20 0 21 0 519189626 541929472 18577 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 5 0 0 0
[pid=5773/tid=5930] ppid=5771 vsize=529228 CPUtime=0 cores=1,3,5,7
/proc/5773/task/5930/stat : 5930 (python) S 5771 5773 5629 0 -1 4202560 30 932 0 0 0 0 0 0 20 0 21 0 519209280 541929472 18577 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 5 0 0 0
[pid=5773/tid=5933] ppid=5771 vsize=529228 CPUtime=0 cores=1,3,5,7
/proc/5773/task/5933/stat : 5933 (python) S 5771 5773 5629 0 -1 4202560 28 932 0 0 0 0 0 0 20 0 21 0 519209367 541929472 18577 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 5 0 0 0
[pid=5773/tid=5991] ppid=5771 vsize=529228 CPUtime=0 cores=1,3,5,7
/proc/5773/task/5991/stat : 5991 (python) S 5771 5773 5629 0 -1 4202560 43 932 0 0 0 0 0 0 18 0 21 0 519218680 541929472 18577 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 5 0 0 0
[pid=5773/tid=5995] ppid=5771 vsize=529228 CPUtime=0 cores=1,3,5,7
/proc/5773/task/5995/stat : 5995 (python) S 5771 5773 5629 0 -1 4202560 27 932 0 0 0 0 0 0 20 0 21 0 519219287 541929472 18577 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 5 0 0 0
[pid=5773/tid=5999] ppid=5771 vsize=529228 CPUtime=0 cores=1,3,5,7
/proc/5773/task/5999/stat : 5999 (python) S 5771 5773 5629 0 -1 4202560 30 932 0 0 0 0 0 0 21 0 21 0 519219301 541929472 18577 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 5 0 0 0
[pid=5773/tid=6021] ppid=5771 vsize=529228 CPUtime=0 cores=1,3,5,7
/proc/5773/task/6021/stat : 6021 (python) S 5771 5773 5629 0 -1 4202560 27 932 0 0 0 0 0 0 20 0 21 0 519228213 541929472 18577 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 5 0 0 0
[pid=5773/tid=6025] ppid=5771 vsize=529228 CPUtime=0 cores=1,3,5,7
/proc/5773/task/6025/stat : 6025 (python) S 5771 5773 5629 0 -1 4202560 26 932 0 0 0 0 0 0 20 0 21 0 519229307 541929472 18577 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 5 0 0 0
[pid=5773/tid=6048] ppid=5771 vsize=529228 CPUtime=0 cores=1,3,5,7
/proc/5773/task/6048/stat : 6048 (python) S 5771 5773 5629 0 -1 4202560 29 932 0 0 0 0 0 0 20 0 21 0 519238235 541929472 18577 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 5 0 0 0
[pid=5773/tid=6054] ppid=5771 vsize=529228 CPUtime=0 cores=1,3,5,7
/proc/5773/task/6054/stat : 6054 (python) S 5771 5773 5629 0 -1 4202560 27 932 0 0 0 0 0 0 20 0 21 0 519239330 541929472 18577 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 5 0 0 0
[pid=5773/tid=6080] ppid=5771 vsize=529228 CPUtime=0 cores=1,3,5,7
/proc/5773/task/6080/stat : 6080 (python) S 5771 5773 5629 0 -1 4202560 29 932 0 0 0 0 0 0 20 0 21 0 519257895 541929472 18577 33554432000 4194304 5683644 140735154016144 18446744073709551615 264738228657 0 0 16781312 2 18446604471142117504 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4312.13
Current children cumulated vsize (KiB) 16046492

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 5773 and gives
#  childrusage.ru_utime.tv_sec=12
#  childrusage.ru_utime.tv_usec=161151
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=215967
# CPU time returned by wait4() is 12.3771
# while last known CPU time is 4312.13
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1091.23
CPU time (s): 4312.13
CPU user time (s): 4308.18
CPU system time (s): 3.95
CPU usage (%): 395.161
Max. virtual memory (cumulated for all children) (KiB): 16046492

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 12.1612
system time used= 0.215967
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 48394
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1273
involuntary context switches= 122

runsolver used 5.60215 second user time and 12.912 second system time

The end

Launcher Data 

Begin job on node141 at 2011-05-01 18:28:52
IDJOB=3317112
IDBENCH=82577
IDSOLVER=1705
FILE ID=node141/3317112-1304267332
RUNJOBID= node141-1304267280-5648
PBS_JOBID= 13173938
Free space on /tmp= 72592 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v50000-c210000-S746725327-020-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317112-1304267332/watcher-3317112-1304267332 -o /tmp/evaluation-result-3317112-1304267332/solver-3317112-1304267332 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317112-1304267332.cnf -seed 1962411866 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 0ad997eea0def3552b6d7e45aac8ca7c
RANDOM SEED=1962411866

node141.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5362.96
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      21684612 kB
Buffers:       1416528 kB
Cached:        9030692 kB
SwapCached:          4 kB
Active:        2981004 kB
Inactive:      7698212 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      21684612 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:           43928 kB
Writeback:           0 kB
AnonPages:      232096 kB
Mapped:          20724 kB
Slab:           519516 kB
PageTables:       8288 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  1614732 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72552 MiB
End job on node141 at 2011-05-01 18:47:05