Trace number 3386545

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-seq-random (fixed)? (TO) 5000.1 4980.22

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v35000-c147000-S970100151-015-UNKNOWN.cnf
MD5SUM1ba3e47e885ea3862e5726ce64a96a9c
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark152.918
Satisfiable
(Un)Satisfiability was proved
Number of variables35000
Number of clauses147000
Sum of the clauses size441000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3147000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.39/0.64	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.39/0.64	c INFO: solving HOME/instance-3386545-1305586395.cnf
0.39/0.64	c INFO: computing task features
0.88/1.15	c INFO: computed variable-clause graph statistics
0.88/1.17	c INFO: computed clause balance statistics
3.46/3.75	c INFO: computed variable balance statistics
5.56/5.84	c INFO: computed small-clause counts
5.56/5.85	c INFO: computed Horn-clause counts
7.16/7.42	c INFO: computed variable graph statistics
7.16/7.42	c INFO: collected features for HOME/instance-3386545-1305586395.cnf in 6.76 s
7.16/7.45	c INFO: running hybridGM3@0 for 97 with 4993 remaining (b = 0.36)
107.92/107.94	c INFO: running hybridGM3@0 for 97 with 4892 remaining (b = 0.22)
208.73/208.44	c INFO: running hybridGM3@0 for 97 with 4792 remaining (b = 0.14)
309.50/308.94	c INFO: running hybridGM3@100 for 97 with 4692 remaining (b = 0.11)
407.29/406.36	c INFO: running hybridGM3@100 for 97 with 4594 remaining (b = 0.11)
504.12/502.79	c INFO: running TNM@0 for 97 with 4498 remaining (b = 0.13)
607.02/605.29	c INFO: running hybridGM3@100 for 97 with 4395 remaining (b = 0.10)
704.88/702.72	c INFO: running hybridGM3@0 for 194 with 4298 remaining (b = 0.18)
900.59/897.70	c INFO: running hybridGM3@0 for 194 with 4103 remaining (b = 0.14)
1099.42/1095.64	c INFO: running hybridGM3@0 for 194 with 3905 remaining (b = 0.10)
1296.80/1292.55	c INFO: running hybridGM3@0 for 194 with 3708 remaining (b = 0.07)
1496.65/1491.52	c INFO: running TNM@0 for 97 with 3510 remaining (b = 0.03)
1595.53/1590.00	c INFO: running hybridGM3@0 for 194 with 3411 remaining (b = 0.05)
1795.22/1789.00	c INFO: running hybridGM3@0 for 194 with 3212 remaining (b = 0.04)
1990.95/1983.95	c INFO: running TNM@0 for 97 with 3017 remaining (b = 0.02)
2091.88/2084.42	c INFO: running TNM@102 for 1455 with 2917 remaining (b = 0.19)
3548.49/3534.73	c INFO: running hybridGM3@0 for 194 with 1467 remaining (b = 0.03)
3745.33/3730.71	c INFO: running hybridGM3@0 for 194 with 1271 remaining (b = 0.03)
3943.04/3927.66	c INFO: running TNM@0 for 97 with 1074 remaining (b = 0.01)
4042.98/4027.15	c INFO: running hybridGM3@0 for 194 with 974 remaining (b = 0.02)
4239.68/4223.09	c INFO: running adaptg2wsat2009++@0 for 97 with 778 remaining (b = 0.01)
4339.53/4322.56	c INFO: running TNM@0 for 97 with 679 remaining (b = 0.01)
4438.42/4421.04	c INFO: running TNM@1552 for 581 with 581 remaining (b = 0.04)
5000.10/4980.19	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3386545-1305586395/watcher-3386545-1305586395 -o /tmp/evaluation-result-3386545-1305586395/solver-3386545-1305586395 -C 5000 -W 5100 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386545-1305586395.cnf -seed 1675891500 -budget 5000 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 5030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 4.06 3.21 3.85 4/522 31371
/proc/meminfo: memFree=15804912/32951124 swapFree=45193380/67111528
[pid=31371] ppid=31369 vsize=16244 CPUtime=0.01 cores=1,3,5,7
/proc/31371/stat : 31371 (python) R 31369 31371 31236 0 -1 4202496 1040 0 0 0 1 0 0 0 25 0 1 0 748870253 16633856 912 33554432000 4194304 5683644 140736594941360 18446744073709551615 4540219 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/31371/statm: 4061 912 380 364 0 548 0

[startup+0.033622 s]
/proc/loadavg: 4.06 3.21 3.85 4/522 31371
/proc/meminfo: memFree=15804912/32951124 swapFree=45193380/67111528
[pid=31371] ppid=31369 vsize=16816 CPUtime=0.01 cores=1,3,5,7
/proc/31371/stat : 31371 (python) R 31369 31371 31236 0 -1 4202496 1213 0 0 0 1 0 0 0 25 0 1 0 748870253 17219584 1072 33554432000 4194304 5683644 140736594941360 18446744073709551615 224284923904 0 2147483391 16781312 2 0 0 0 17 3 0 0 0
/proc/31371/statm: 4204 1072 417 364 0 691 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16816

[startup+0.100611 s]
/proc/loadavg: 4.06 3.21 3.85 4/522 31371
/proc/meminfo: memFree=15804912/32951124 swapFree=45193380/67111528
[pid=31371] ppid=31369 vsize=89772 CPUtime=0.06 cores=1,3,5,7
/proc/31371/stat : 31371 (python) R 31369 31371 31236 0 -1 4202496 2672 0 0 0 5 1 0 0 17 0 1 0 748870253 91926528 2388 33554432000 4194304 5683644 140736594941360 18446744073709551615 224284923904 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/31371/statm: 22443 2388 750 364 0 1623 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 89772

[startup+0.300546 s]
/proc/loadavg: 4.06 3.21 3.85 4/522 31371
/proc/meminfo: memFree=15804912/32951124 swapFree=45193380/67111528
[pid=31371] ppid=31369 vsize=135752 CPUtime=0.19 cores=1,3,5,7
/proc/31371/stat : 31371 (python) S 31369 31371 31236 0 -1 4202496 5972 936 1 0 14 5 0 0 17 0 1 0 748870253 139010048 4149 33554432000 4194304 5683644 140736594941360 18446744073709551615 224284921861 0 2147483391 16781312 2 18446744071699199166 0 0 17 7 0 0 0
/proc/31371/statm: 33938 4149 1032 364 0 3124 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 135752

[startup+0.704573 s]
/proc/loadavg: 4.06 3.21 3.85 4/522 31371
/proc/meminfo: memFree=15804912/32951124 swapFree=45193380/67111528
[pid=31371] ppid=31369 vsize=284620 CPUtime=0.47 cores=1,3,5,7
/proc/31371/stat : 31371 (python) R 31369 31371 31236 0 -1 4202496 10791 936 1 0 35 12 0 0 15 0 1 0 748870253 291450880 8461 33554432000 4194304 5683644 140736594941360 18446744073709551615 46919832468272 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/31371/statm: 71155 8461 1761 364 0 7079 0
Current children cumulated CPU time (s) 0.47
Current children cumulated vsize (KiB) 284620

[startup+1.50038 s]
/proc/loadavg: 3.97 3.20 3.84 5/523 31376
/proc/meminfo: memFree=15754052/32951124 swapFree=45193380/67111528
[pid=31371] ppid=31369 vsize=312968 CPUtime=1.26 cores=1,3,5,7
/proc/31371/stat : 31371 (python) R 31369 31371 31236 0 -1 4202496 21978 936 1 0 113 13 0 0 16 0 1 0 748870253 320479232 15626 33554432000 4194304 5683644 140736594941360 18446744073709551615 46919664382002 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/31371/statm: 78242 15626 1826 364 0 14166 0
Current children cumulated CPU time (s) 1.26
Current children cumulated vsize (KiB) 312968

[startup+3.10011 s]
/proc/loadavg: 3.97 3.20 3.84 5/523 31376
/proc/meminfo: memFree=15746488/32951124 swapFree=45193380/67111528
[pid=31371] ppid=31369 vsize=312968 CPUtime=2.86 cores=1,3,5,7
/proc/31371/stat : 31371 (python) R 31369 31371 31236 0 -1 4202496 21978 936 1 0 273 13 0 0 18 0 1 0 748870253 320479232 15626 33554432000 4194304 5683644 140736594941360 18446744073709551615 224284584512 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/31371/statm: 78242 15626 1826 364 0 14166 0
Current children cumulated CPU time (s) 2.86
Current children cumulated vsize (KiB) 312968

[startup+6.30058 s]
/proc/loadavg: 4.14 3.25 3.86 5/523 31376
/proc/meminfo: memFree=15745000/32951124 swapFree=45193380/67111528
[pid=31371] ppid=31369 vsize=312968 CPUtime=6.06 cores=1,3,5,7
/proc/31371/stat : 31371 (python) R 31369 31371 31236 0 -1 4202496 21985 936 1 0 593 13 0 0 25 0 1 0 748870253 320479232 15633 33554432000 4194304 5683644 140736594941360 18446744073709551615 46919661607052 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/31371/statm: 78242 15633 1833 364 0 14166 0
Current children cumulated CPU time (s) 6.06
Current children cumulated vsize (KiB) 312968

[startup+12.7005 s]
/proc/loadavg: 4.04 3.25 3.85 5/515 31411
/proc/meminfo: memFree=15797128/32951124 swapFree=45193380/67111528
[pid=31371] ppid=31369 vsize=320504 CPUtime=7.23 cores=1,3,5,7
/proc/31371/stat : 31371 (python) S 31369 31371 31236 0 -1 4202496 22277 936 1 0 709 14 0 0 20 0 2 0 748870253 328196096 14996 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296752587 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/31371/statm: 80126 14996 1875 364 0 16044 0
[pid=31377] ppid=31371 vsize=310260 CPUtime=0.01 cores=1,3,5,7
/proc/31377/stat : 31377 (python) S 31371 31371 31236 0 -1 4202560 900 0 0 0 0 1 0 0 15 0 1 0 748870999 317706240 13493 33554432000 4194304 5683644 140736594941360 18446744073709551615 224284945935 0 0 16781312 514 12156718705840980229 0 0 17 5 0 0 0
/proc/31377/statm: 77565 13493 376 364 0 13483 0
[pid=31379] ppid=31377 vsize=268420 CPUtime=5.23 cores=1,3,5,7
/proc/31379/stat : 31379 (hybridGM3) R 31377 31379 31379 0 -1 4202496 8086 0 0 0 522 1 0 0 25 0 1 0 748870999 274862080 7820 33554432000 134512640 135204385 4294179920 18446744073709551615 134537510 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/31379/statm: 67105 7820 64 169 0 66935 0
[pid=31371/tid=31378] ppid=31369 vsize=320504 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31378/stat : 31378 (python) S 31369 31371 31236 0 -1 4202560 30 936 0 0 0 0 0 0 20 0 2 0 748870999 328196096 14996 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 12.47
Current children cumulated vsize (KiB) 899184

[startup+25.5004 s]
/proc/loadavg: 4.03 3.28 3.85 5/518 31414
/proc/meminfo: memFree=15798240/32951124 swapFree=45193380/67111528
[pid=31371] ppid=31369 vsize=320504 CPUtime=7.23 cores=1,3,5,7
/proc/31371/stat : 31371 (python) S 31369 31371 31236 0 -1 4202496 22277 936 1 0 709 14 0 0 20 0 2 0 748870253 328196096 14996 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296752587 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/31371/statm: 80126 14996 1875 364 0 16044 0
[pid=31377] ppid=31371 vsize=310260 CPUtime=0.06 cores=1,3,5,7
/proc/31377/stat : 31377 (python) S 31371 31371 31236 0 -1 4202560 906 0 0 0 2 4 0 0 15 0 1 0 748870999 317706240 13493 33554432000 4194304 5683644 140736594941360 18446744073709551615 224284945935 0 0 16781312 514 5961041872406151429 0 0 17 5 0 0 0
/proc/31377/statm: 77565 13493 376 364 0 13483 0
[pid=31379] ppid=31377 vsize=268420 CPUtime=18.01 cores=1,3,5,7
/proc/31379/stat : 31379 (hybridGM3) R 31377 31379 31379 0 -1 4202496 8086 0 0 0 1800 1 0 0 25 0 1 0 748870999 274862080 7820 33554432000 134512640 135204385 4294179920 18446744073709551615 134536646 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/31379/statm: 67105 7820 64 169 0 66935 0
[pid=31371/tid=31378] ppid=31369 vsize=320504 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31378/stat : 31378 (python) S 31369 31371 31236 0 -1 4202560 30 936 0 0 0 0 0 0 20 0 2 0 748870999 328196096 14996 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 25.3
Current children cumulated vsize (KiB) 899184

[startup+51.1011 s]
/proc/loadavg: 4.16 3.37 3.87 5/518 31414
/proc/meminfo: memFree=15797536/32951124 swapFree=45193380/67111528
[pid=31371] ppid=31369 vsize=320504 CPUtime=7.23 cores=1,3,5,7
/proc/31371/stat : 31371 (python) S 31369 31371 31236 0 -1 4202496 22277 936 1 0 709 14 0 0 20 0 2 0 748870253 328196096 14996 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296752587 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/31371/statm: 80126 14996 1875 364 0 16044 0
[pid=31377] ppid=31371 vsize=310260 CPUtime=0.15 cores=1,3,5,7
/proc/31377/stat : 31377 (python) S 31371 31371 31236 0 -1 4202560 918 0 0 0 7 8 0 0 15 0 1 0 748870999 317706240 13493 33554432000 4194304 5683644 140736594941360 18446744073709551615 224284945935 0 0 16781312 514 5961041872406151429 0 0 17 5 0 0 0
/proc/31377/statm: 77565 13493 376 364 0 13483 0
[pid=31379] ppid=31377 vsize=268420 CPUtime=43.58 cores=1,3,5,7
/proc/31379/stat : 31379 (hybridGM3) R 31377 31379 31379 0 -1 4202496 8086 0 0 0 4357 1 0 0 25 0 1 0 748870999 274862080 7820 33554432000 134512640 135204385 4294179920 18446744073709551615 134537403 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/31379/statm: 67105 7820 64 169 0 66935 0
[pid=31371/tid=31378] ppid=31369 vsize=320504 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31378/stat : 31378 (python) S 31369 31371 31236 0 -1 4202560 30 936 0 0 0 0 0 0 20 0 2 0 748870999 328196096 14996 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 50.96
Current children cumulated vsize (KiB) 899184

[startup+102.32 s]
/proc/loadavg: 4.07 3.47 3.87 5/518 31416
/proc/meminfo: memFree=15796152/32951124 swapFree=45193380/67111528
[pid=31371] ppid=31369 vsize=320504 CPUtime=7.23 cores=1,3,5,7
/proc/31371/stat : 31371 (python) S 31369 31371 31236 0 -1 4202496 22277 936 1 0 709 14 0 0 20 0 2 0 748870253 328196096 14996 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296752587 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/31371/statm: 80126 14996 1875 364 0 16044 0
[pid=31377] ppid=31371 vsize=310260 CPUtime=0.36 cores=1,3,5,7
/proc/31377/stat : 31377 (python) S 31371 31371 31236 0 -1 4202560 939 0 0 0 20 16 0 0 15 0 1 0 748870999 317706240 13493 33554432000 4194304 5683644 140736594941360 18446744073709551615 224284945935 0 0 16781312 514 5961041872406151429 0 0 17 5 0 0 0
/proc/31377/statm: 77565 13493 376 364 0 13483 0
[pid=31379] ppid=31377 vsize=268420 CPUtime=94.76 cores=1,3,5,7
/proc/31379/stat : 31379 (hybridGM3) R 31377 31379 31379 0 -1 4202496 8086 0 0 0 9475 1 0 0 25 0 1 0 748870999 274862080 7820 33554432000 134512640 135204385 4294179920 18446744073709551615 134536986 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/31379/statm: 67105 7820 64 169 0 66935 0
[pid=31371/tid=31378] ppid=31369 vsize=320504 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31378/stat : 31378 (python) S 31369 31371 31236 0 -1 4202560 30 936 0 0 0 0 0 0 20 0 2 0 748870999 328196096 14996 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 102.35
Current children cumulated vsize (KiB) 899184

[startup+162.301 s]
/proc/loadavg: 4.02 3.56 3.88 5/526 31426
/proc/meminfo: memFree=15752788/32951124 swapFree=45193380/67111528
[pid=31371] ppid=31369 vsize=330760 CPUtime=7.25 cores=1,3,5,7
/proc/31371/stat : 31371 (python) S 31369 31371 31236 0 -1 4202496 23205 936 1 0 711 14 0 0 19 0 3 0 748870253 338698240 15003 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296752587 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/31371/statm: 82690 15003 1878 364 0 18605 0
[pid=31377] ppid=31371 vsize=320504 CPUtime=0.39 cores=1,3,5,7
/proc/31377/stat : 31377 (python) S 31371 31371 31236 0 -1 4202560 1096 0 0 0 21 18 0 0 16 0 2 0 748870999 328196096 13531 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/31377/statm: 80126 13531 409 364 0 16044 0
[pid=31379] ppid=31377 vsize=268420 CPUtime=100.35 cores=1,3,5,7
/proc/31379/stat : 31379 (hybridGM3) T 31377 31379 31379 0 -1 4202496 8086 0 0 0 10034 1 0 0 25 0 1 0 748870999 274862080 7820 33554432000 134512640 135204385 4294179920 18446744073709551615 134536843 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/31379/statm: 67105 7820 64 169 0 66935 0
[pid=31377/tid=31417] ppid=31371 vsize=320504 CPUtime=0 cores=1,3,5,7
/proc/31377/task/31417/stat : 31417 (python) S 31371 31371 31236 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 748881045 328196096 13531 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 514 18446604455743149888 0 0 -1 5 0 0 0
[pid=31418] ppid=31371 vsize=320516 CPUtime=0.2 cores=1,3,5,7
/proc/31418/stat : 31418 (python) S 31371 31371 31236 0 -1 4202560 909 0 0 0 11 9 0 0 15 0 1 0 748881048 328208384 13491 33554432000 4194304 5683644 140736594941360 18446744073709551615 224284945990 0 0 16781312 514 5961041872406151430 0 0 17 5 0 0 0
/proc/31418/statm: 80129 13491 370 364 0 16044 0
[pid=31420] ppid=31418 vsize=268424 CPUtime=54.31 cores=1,3,5,7
/proc/31420/stat : 31420 (hybridGM3) R 31418 31420 31420 0 -1 4202496 8087 0 0 0 5430 1 0 0 25 0 1 0 748881048 274866176 7825 33554432000 134512640 135204385 4289881856 18446744073709551615 134537401 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/31420/statm: 67106 7825 64 169 0 66936 0
[pid=31371/tid=31378] ppid=31369 vsize=330760 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31378/stat : 31378 (python) S 31369 31371 31236 0 -1 4202560 30 936 0 0 0 0 0 0 20 0 3 0 748870999 338698240 15003 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 5 0 0 0
[pid=31371/tid=31419] ppid=31369 vsize=330760 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31419/stat : 31419 (python) S 31369 31371 31236 0 -1 4202560 25 936 0 0 0 0 0 0 20 0 3 0 748881048 338698240 15003 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 162.5
Current children cumulated vsize (KiB) 1508624

[startup+222.302 s]
/proc/loadavg: 4.01 3.64 3.88 5/534 31436
/proc/meminfo: memFree=15706328/32951124 swapFree=45193380/67111528
[pid=31371] ppid=31369 vsize=341016 CPUtime=7.28 cores=1,3,5,7
/proc/31371/stat : 31371 (python) S 31369 31371 31236 0 -1 4202496 24146 936 1 0 714 14 0 0 19 0 4 0 748870253 349200384 15009 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296752587 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/31371/statm: 85254 15009 1880 364 0 21166 0
[pid=31377] ppid=31371 vsize=320504 CPUtime=0.39 cores=1,3,5,7
/proc/31377/stat : 31377 (python) S 31371 31371 31236 0 -1 4202560 1096 0 0 0 21 18 0 0 16 0 2 0 748870999 328196096 13531 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/31377/statm: 80126 13531 409 364 0 16044 0
[pid=31379] ppid=31377 vsize=268420 CPUtime=100.35 cores=1,3,5,7
/proc/31379/stat : 31379 (hybridGM3) T 31377 31379 31379 0 -1 4202496 8086 0 0 0 10034 1 0 0 25 0 1 0 748870999 274862080 7820 33554432000 134512640 135204385 4294179920 18446744073709551615 134536843 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/31379/statm: 67105 7820 64 169 0 66935 0
[pid=31377/tid=31417] ppid=31371 vsize=320504 CPUtime=0 cores=1,3,5,7
/proc/31377/task/31417/stat : 31417 (python) S 31371 31371 31236 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 748881045 328196096 13531 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 514 18446604455743149888 0 0 -1 5 0 0 0

################
# More data... #
################

/proc/31461/statm: 85257 13529 399 364 0 21166 0
[pid=31463] ppid=31461 vsize=465044 CPUtime=2111.85 cores=1,3,5,7
/proc/31463/stat : 31463 (TNM) R 31461 31463 31463 0 -1 4202496 6010 0 0 0 211185 0 0 0 25 0 1 0 748920533 476205056 5749 33554432000 134512640 135008748 4292060384 18446744073709551615 134527320 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/31463/statm: 116261 5749 46 122 0 115161 0
[pid=31461/tid=31472] ppid=31371 vsize=341028 CPUtime=0 cores=1,3,5,7
/proc/31461/task/31472/stat : 31472 (python) S 31371 31371 31236 0 -1 4202560 45 0 0 0 0 0 0 0 18 0 2 0 748930780 349212672 13529 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 514 18446604468411744640 0 0 -1 7 0 0 0
[pid=31479] ppid=31371 vsize=351284 CPUtime=0.82 cores=1,3,5,7
/proc/31479/stat : 31479 (python) S 31371 31371 31236 0 -1 4202560 1164 0 0 0 42 40 0 0 17 0 2 0 748940525 359714816 13533 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31479/statm: 87821 13533 399 364 0 23727 0
[pid=31481] ppid=31479 vsize=268420 CPUtime=194.93 cores=1,3,5,7
/proc/31481/stat : 31481 (hybridGM3) T 31479 31481 31481 0 -1 4202496 8098 0 0 0 19492 1 0 0 25 0 1 0 748940526 274862080 7833 33554432000 134512640 135204385 4290498080 18446744073709551615 134537385 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/31481/statm: 67105 7833 64 169 0 66935 0
[pid=31479/tid=31496] ppid=31371 vsize=351284 CPUtime=0 cores=1,3,5,7
/proc/31479/task/31496/stat : 31496 (python) S 31371 31371 31236 0 -1 4202560 49 0 0 0 0 0 0 0 17 0 2 0 748960020 359714816 13533 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 514 18446604462584135744 0 0 -1 7 0 0 0
[pid=31497] ppid=31371 vsize=361540 CPUtime=0.83 cores=1,3,5,7
/proc/31497/stat : 31497 (python) S 31371 31371 31236 0 -1 4202560 1169 0 0 0 43 40 0 0 17 0 2 0 748960023 370216960 13537 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31497/statm: 90385 13537 399 364 0 26288 0
[pid=31499] ppid=31497 vsize=268424 CPUtime=197.9 cores=1,3,5,7
/proc/31499/stat : 31499 (hybridGM3) T 31497 31499 31499 0 -1 4202496 8086 0 0 0 19790 0 0 0 25 0 1 0 748960024 274866176 7824 33554432000 134512640 135204385 4289436992 18446744073709551615 134536656 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/31499/statm: 67106 7824 64 169 0 66936 0
[pid=31497/tid=31512] ppid=31371 vsize=361540 CPUtime=0 cores=1,3,5,7
/proc/31497/task/31512/stat : 31512 (python) S 31371 31371 31236 0 -1 4202560 48 0 0 0 0 0 0 0 17 0 2 0 748979815 370216960 13537 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 514 18446604456155511744 0 0 -1 7 0 0 0
[pid=31513] ppid=31371 vsize=371796 CPUtime=0.82 cores=1,3,5,7
/proc/31513/stat : 31513 (python) S 31371 31371 31236 0 -1 4202560 1209 0 0 0 41 41 0 0 17 0 2 0 748979818 380719104 13547 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/31513/statm: 92949 13547 405 364 0 28849 0
[pid=31515] ppid=31513 vsize=268420 CPUtime=196.55 cores=1,3,5,7
/proc/31515/stat : 31515 (hybridGM3) T 31513 31515 31515 0 -1 4202496 8092 0 0 0 19654 1 0 0 25 0 1 0 748979818 274862080 7826 33554432000 134512640 135204385 4294405584 18446744073709551615 134537383 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/31515/statm: 67105 7826 64 169 0 66935 0
[pid=31513/tid=31526] ppid=31371 vsize=371796 CPUtime=0 cores=1,3,5,7
/proc/31513/task/31526/stat : 31526 (python) S 31371 31371 31236 0 -1 4202560 95 0 0 0 0 0 0 0 17 0 2 0 748999506 380719104 13547 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 514 18446604470345609344 0 0 -1 3 0 0 0
[pid=31527] ppid=31371 vsize=382052 CPUtime=0.84 cores=1,3,5,7
/proc/31527/stat : 31527 (python) S 31371 31371 31236 0 -1 4202560 1212 0 0 0 43 41 0 0 16 0 2 0 748999509 391221248 13551 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31527/statm: 95513 13551 405 364 0 31410 0
[pid=31529] ppid=31527 vsize=268424 CPUtime=198.93 cores=1,3,5,7
/proc/31529/stat : 31529 (hybridGM3) T 31527 31529 31529 0 -1 4202496 8097 0 0 0 19892 1 0 0 25 0 1 0 748999509 274866176 7826 33554432000 134512640 135204385 4287244480 18446744073709551615 134536663 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/31529/statm: 67106 7826 64 169 0 66936 0
[pid=31527/tid=31538] ppid=31371 vsize=382052 CPUtime=0 cores=1,3,5,7
/proc/31527/task/31538/stat : 31538 (python) S 31371 31371 31236 0 -1 4202560 51 0 0 0 0 0 0 0 16 0 2 0 749019403 391221248 13551 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 514 18446604471124579328 0 0 -1 7 0 0 0
[pid=31539] ppid=31371 vsize=392308 CPUtime=0.42 cores=1,3,5,7
/proc/31539/stat : 31539 (python) S 31371 31371 31236 0 -1 4202560 1133 0 0 0 23 19 0 0 19 0 2 0 749019406 401723392 13549 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31539/statm: 98077 13549 399 364 0 33971 0
[pid=31541] ppid=31539 vsize=465044 CPUtime=98.43 cores=1,3,5,7
/proc/31541/stat : 31541 (TNM) T 31539 31541 31541 0 -1 4202496 6020 0 0 0 9842 1 0 0 25 0 1 0 749019406 476205056 5750 33554432000 134512640 135008748 4293940512 18446744073709551615 134529076 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/31541/statm: 116261 5750 46 122 0 115161 0
[pid=31539/tid=31550] ppid=31371 vsize=392308 CPUtime=0 cores=1,3,5,7
/proc/31539/task/31550/stat : 31550 (python) S 31371 31371 31236 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 749029251 401723392 13549 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 514 18446604454705079360 0 0 -1 7 0 0 0
[pid=31551] ppid=31371 vsize=402564 CPUtime=0.86 cores=1,3,5,7
/proc/31551/stat : 31551 (python) S 31371 31371 31236 0 -1 4202560 1180 0 0 0 44 42 0 0 17 0 2 0 749029254 412225536 13553 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31551/statm: 100641 13553 399 364 0 36532 0
[pid=31553] ppid=31551 vsize=268420 CPUtime=198.8 cores=1,3,5,7
/proc/31553/stat : 31553 (hybridGM3) T 31551 31553 31553 0 -1 4202496 8098 0 0 0 19879 1 0 0 25 0 1 0 749029254 274862080 7824 33554432000 134512640 135204385 4289281408 18446744073709551615 134536737 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/31553/statm: 67105 7824 64 169 0 66935 0
[pid=31551/tid=31562] ppid=31371 vsize=402564 CPUtime=0 cores=1,3,5,7
/proc/31551/task/31562/stat : 31562 (python) S 31371 31371 31236 0 -1 4202560 48 0 0 0 0 0 0 0 17 0 2 0 749049151 412225536 13553 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 514 18446604454705078464 0 0 -1 7 0 0 0
[pid=31563] ppid=31371 vsize=412820 CPUtime=0.85 cores=1,3,5,7
/proc/31563/stat : 31563 (python) S 31371 31371 31236 0 -1 4202560 1174 0 0 0 46 39 0 0 16 0 2 0 749049154 422727680 13557 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/31563/statm: 103205 13557 399 364 0 39093 0
[pid=31565] ppid=31563 vsize=268420 CPUtime=194.88 cores=1,3,5,7
/proc/31565/stat : 31565 (hybridGM3) T 31563 31565 31565 0 -1 4202496 8094 0 0 0 19487 1 0 0 25 0 1 0 749049154 274862080 7823 33554432000 134512640 135204385 4292173136 18446744073709551615 134537440 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/31565/statm: 67105 7823 64 169 0 66935 0
[pid=31563/tid=31572] ppid=31371 vsize=412820 CPUtime=0 cores=1,3,5,7
/proc/31563/task/31572/stat : 31572 (python) S 31371 31371 31236 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 749068645 422727680 13557 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 514 18446604458757907776 0 0 -1 5 0 0 0
[pid=31573] ppid=31371 vsize=423076 CPUtime=0.44 cores=1,3,5,7
/proc/31573/stat : 31573 (python) S 31371 31371 31236 0 -1 4202560 1158 0 0 0 22 22 0 0 19 0 2 0 749068648 433229824 13561 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31573/statm: 105769 13561 399 364 0 41654 0
[pid=31575] ppid=31573 vsize=465044 CPUtime=100.43 cores=1,3,5,7
/proc/31575/stat : 31575 (TNM) T 31573 31575 31575 0 -1 4202496 6020 0 0 0 10043 0 0 0 25 0 1 0 749068649 476205056 5750 33554432000 134512640 135008748 4290032496 18446744073709551615 134529468 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/31575/statm: 116261 5750 46 122 0 115161 0
[pid=31573/tid=31578] ppid=31371 vsize=423076 CPUtime=0 cores=1,3,5,7
/proc/31573/task/31578/stat : 31578 (python) S 31371 31371 31236 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 749078693 433229824 13561 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 514 18446604458757906880 0 0 -1 7 0 0 0
[pid=31705] ppid=31371 vsize=433332 CPUtime=0.85 cores=1,3,5,7
/proc/31705/stat : 31705 (python) S 31371 31371 31236 0 -1 4202560 1163 0 0 0 45 40 0 0 16 0 2 0 749223727 443731968 13565 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31705/statm: 108333 13565 399 364 0 44215 0
[pid=31707] ppid=31705 vsize=268428 CPUtime=195.93 cores=1,3,5,7
/proc/31707/stat : 31707 (hybridGM3) T 31705 31707 31707 0 -1 4202496 8094 0 0 0 19593 0 0 0 25 0 1 0 749223728 274870272 7825 33554432000 134512640 135204385 4291734144 18446744073709551615 134545851 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/31707/statm: 67107 7825 64 169 0 66937 0
[pid=31705/tid=31720] ppid=31371 vsize=433332 CPUtime=0 cores=1,3,5,7
/proc/31705/task/31720/stat : 31720 (python) S 31371 31371 31236 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 749243323 443731968 13565 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 514 18446604454800155584 0 0 -1 7 0 0 0
[pid=31721] ppid=31371 vsize=443588 CPUtime=0.85 cores=1,3,5,7
/proc/31721/stat : 31721 (python) S 31371 31371 31236 0 -1 4202560 1174 0 0 0 45 40 0 0 17 0 2 0 749243325 454234112 13569 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31721/statm: 110897 13569 399 364 0 46776 0
[pid=31723] ppid=31721 vsize=268420 CPUtime=196.92 cores=1,3,5,7
/proc/31723/stat : 31723 (hybridGM3) T 31721 31723 31723 0 -1 4202496 8096 0 0 0 19691 1 0 0 25 0 1 0 749243325 274862080 7825 33554432000 134512640 135204385 4287818880 18446744073709551615 134547392 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/31723/statm: 67105 7825 64 169 0 66935 0
[pid=31721/tid=31738] ppid=31371 vsize=443588 CPUtime=0 cores=1,3,5,7
/proc/31721/task/31738/stat : 31738 (python) S 31371 31371 31236 0 -1 4202560 45 0 0 0 0 0 0 0 17 0 2 0 749263018 454234112 13569 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 514 18446604458757908672 0 0 -1 7 0 0 0
[pid=31739] ppid=31371 vsize=453844 CPUtime=0.44 cores=1,3,5,7
/proc/31739/stat : 31739 (python) S 31371 31371 31236 0 -1 4202560 1205 0 0 0 27 17 0 0 19 0 2 0 749263020 464736256 13581 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31739/statm: 113461 13581 407 364 0 49337 0
[pid=31741] ppid=31739 vsize=465044 CPUtime=99.45 cores=1,3,5,7
/proc/31741/stat : 31741 (TNM) T 31739 31741 31741 0 -1 4202496 6023 0 0 0 9945 0 0 0 25 0 1 0 749263020 476205056 5749 33554432000 134512640 135008748 4292619920 18446744073709551615 134537440 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/31741/statm: 116261 5749 46 122 0 115161 0
[pid=31739/tid=31748] ppid=31371 vsize=453844 CPUtime=0 cores=1,3,5,7
/proc/31739/task/31748/stat : 31748 (python) S 31371 31371 31236 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 749272967 464736256 13581 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 514 18446604458449961088 0 0 -1 7 0 0 0
[pid=31749] ppid=31371 vsize=464100 CPUtime=0.85 cores=1,3,5,7
/proc/31749/stat : 31749 (python) S 31371 31371 31236 0 -1 4202560 1218 0 0 0 45 40 0 0 17 0 2 0 749272968 475238400 13584 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31749/statm: 116025 13584 406 364 0 51898 0
[pid=31751] ppid=31749 vsize=268424 CPUtime=195.89 cores=1,3,5,7
/proc/31751/stat : 31751 (hybridGM3) T 31749 31751 31751 0 -1 4202496 8101 0 0 0 19589 0 0 0 25 0 1 0 749272969 274866176 7827 33554432000 134512640 135204385 4292327888 18446744073709551615 134537272 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/31751/statm: 67106 7827 64 169 0 66936 0
[pid=31749/tid=31775] ppid=31371 vsize=464100 CPUtime=0 cores=1,3,5,7
/proc/31749/task/31775/stat : 31775 (python) S 31371 31371 31236 0 -1 4202560 46 0 0 0 0 0 0 0 17 0 2 0 749292561 475238400 13584 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 514 18446604454458275776 0 0 -1 7 0 0 0
[pid=31776] ppid=31371 vsize=474356 CPUtime=0.44 cores=1,3,5,7
/proc/31776/stat : 31776 (python) S 31371 31371 31236 0 -1 4202560 1180 0 0 0 24 20 0 0 19 0 2 0 749292562 485740544 13582 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31776/statm: 118589 13582 400 364 0 54459 0
[pid=31778] ppid=31776 vsize=431704 CPUtime=99.38 cores=1,3,5,7
/proc/31778/stat : 31778 (adaptg2wsat2009) T 31776 31778 31778 0 -1 4202496 6266 0 0 0 9938 0 0 0 25 0 1 0 749292563 442064896 5985 33554432000 134512640 135107411 4293559600 18446744073709551615 134532371 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/31778/statm: 107926 5985 59 146 0 106802 0
[pid=31776/tid=31785] ppid=31371 vsize=474356 CPUtime=0 cores=1,3,5,7
/proc/31776/task/31785/stat : 31785 (python) S 31371 31371 31236 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 749302508 485740544 13582 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 514 18446604454458273984 0 0 -1 7 0 0 0
[pid=31786] ppid=31371 vsize=484612 CPUtime=0.45 cores=1,3,5,7
/proc/31786/stat : 31786 (python) S 31371 31371 31236 0 -1 4202560 1193 0 0 0 22 23 0 0 19 0 2 0 749302510 496242688 13586 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296752587 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31786/statm: 121153 13586 400 364 0 57020 0
[pid=31788] ppid=31786 vsize=465044 CPUtime=98.4 cores=1,3,5,7
/proc/31788/stat : 31788 (TNM) T 31786 31788 31788 0 -1 4202496 6021 0 0 0 9840 0 0 0 25 0 1 0 749302510 476205056 5749 33554432000 134512640 135008748 4294706064 18446744073709551615 134528448 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/31788/statm: 116261 5749 46 122 0 115161 0
[pid=31786/tid=31797] ppid=31371 vsize=484612 CPUtime=0 cores=1,3,5,7
/proc/31786/task/31797/stat : 31797 (python) S 31371 31371 31236 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 749312356 496242688 13586 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 514 18446604459729605760 0 0 -1 7 0 0 0
[pid=31371/tid=31378] ppid=31369 vsize=494856 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31378/stat : 31378 (python) S 31369 31371 31236 0 -1 4202560 34 936 0 0 0 0 0 0 18 0 19 0 748870999 506732544 15099 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 5 0 0 0
[pid=31371/tid=31419] ppid=31369 vsize=494856 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31419/stat : 31419 (python) S 31369 31371 31236 0 -1 4202560 29 936 0 0 0 0 0 0 18 0 19 0 748881048 506732544 15099 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 5 0 0 0
[pid=31371/tid=31429] ppid=31369 vsize=494856 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31429/stat : 31429 (python) S 31369 31371 31236 0 -1 4202560 30 936 0 0 0 0 0 0 18 0 19 0 748891098 506732544 15099 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 5 0 0 0
[pid=31371/tid=31462] ppid=31369 vsize=494856 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31462/stat : 31462 (python) S 31369 31371 31236 0 -1 4202560 33 936 0 0 0 0 0 0 18 0 19 0 748920533 506732544 15099 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 5 0 0 0
[pid=31371/tid=31480] ppid=31369 vsize=494856 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31480/stat : 31480 (python) S 31369 31371 31236 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 19 0 748940526 506732544 15099 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 5 0 0 0
[pid=31371/tid=31498] ppid=31369 vsize=494856 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31498/stat : 31498 (python) S 31369 31371 31236 0 -1 4202560 28 936 0 0 0 0 0 0 20 0 19 0 748960023 506732544 15099 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 7 0 0 0
[pid=31371/tid=31514] ppid=31369 vsize=494856 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31514/stat : 31514 (python) S 31369 31371 31236 0 -1 4202560 31 936 0 0 0 0 0 0 20 0 19 0 748979818 506732544 15099 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 7 0 0 0
[pid=31371/tid=31528] ppid=31369 vsize=494856 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31528/stat : 31528 (python) S 31369 31371 31236 0 -1 4202560 33 936 0 0 0 0 0 0 20 0 19 0 748999509 506732544 15099 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 3 0 0 0
[pid=31371/tid=31540] ppid=31369 vsize=494856 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31540/stat : 31540 (python) S 31369 31371 31236 0 -1 4202560 30 936 0 0 0 0 0 0 20 0 19 0 749019406 506732544 15099 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 3 0 0 0
[pid=31371/tid=31552] ppid=31369 vsize=494856 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31552/stat : 31552 (python) S 31369 31371 31236 0 -1 4202560 28 936 0 0 0 0 0 0 20 0 19 0 749029254 506732544 15099 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 3 0 0 0
[pid=31371/tid=31564] ppid=31369 vsize=494856 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31564/stat : 31564 (python) S 31369 31371 31236 0 -1 4202560 29 936 0 0 0 0 0 0 20 0 19 0 749049154 506732544 15099 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 3 0 0 0
[pid=31371/tid=31574] ppid=31369 vsize=494856 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31574/stat : 31574 (python) S 31369 31371 31236 0 -1 4202560 28 936 0 0 0 0 0 0 20 0 19 0 749068649 506732544 15099 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 5 0 0 0
[pid=31371/tid=31706] ppid=31369 vsize=494856 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31706/stat : 31706 (python) S 31369 31371 31236 0 -1 4202560 30 936 0 0 0 0 0 0 20 0 19 0 749223727 506732544 15099 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 5 0 0 0
[pid=31371/tid=31722] ppid=31369 vsize=494856 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31722/stat : 31722 (python) S 31369 31371 31236 0 -1 4202560 30 936 0 0 0 0 0 0 20 0 19 0 749243325 506732544 15099 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 5 0 0 0
[pid=31371/tid=31740] ppid=31369 vsize=494856 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31740/stat : 31740 (python) S 31369 31371 31236 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 19 0 749263020 506732544 15099 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 3 0 0 0
[pid=31371/tid=31750] ppid=31369 vsize=494856 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31750/stat : 31750 (python) S 31369 31371 31236 0 -1 4202560 25 936 0 0 0 0 0 0 20 0 19 0 749272969 506732544 15099 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 3 0 0 0
[pid=31371/tid=31777] ppid=31369 vsize=494856 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31777/stat : 31777 (python) S 31369 31371 31236 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 19 0 749292563 506732544 15099 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 5 0 0 0
[pid=31371/tid=31787] ppid=31369 vsize=494856 CPUtime=0 cores=1,3,5,7
/proc/31371/task/31787/stat : 31787 (python) S 31369 31371 31236 0 -1 4202560 26 936 0 0 0 0 0 0 20 0 19 0 749302510 506732544 15099 33554432000 4194304 5683644 140736594941360 18446744073709551615 224296749489 0 0 16781312 2 18446604463837254016 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 5000.1
Current children cumulated vsize (KiB) 13636940

Child status: 0

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 31371 and gives
#  childrusage.ru_utime.tv_sec=4202
#  childrusage.ru_utime.tv_usec=658098
#  childrusage.ru_stime.tv_sec=10
#  childrusage.ru_stime.tv_usec=788359
# CPU time returned by wait4() is 4213.45
# while last known CPU time is 5000.1
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 4980.22
CPU time (s): 5000.1
CPU user time (s): 4989.54
CPU system time (s): 10.56
CPU usage (%): 100.399
Max. virtual memory (cumulated for all children) (KiB): 13636944

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4202.66
system time used= 10.7884
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 171443
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 9099
involuntary context switches= 12371

runsolver used 43.1074 second user time and 98.1311 second system time

The end

Launcher Data 

Begin job on node118 at 2011-05-17 00:53:16
IDJOB=3386545
IDBENCH=82610
IDSOLVER=1872
FILE ID=node118/3386545-1305586395
RUNJOBID= node118-1305586147-31255
PBS_JOBID= 13350360
Free space on /tmp= 73108 MiB

SOLVER NAME= borg-sat 11.03.02-seq-random (fixed)
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v35000-c147000-S970100151-015-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3386545-1305586395/watcher-3386545-1305586395 -o /tmp/evaluation-result-3386545-1305586395/solver-3386545-1305586395 -C 5000 -W 5100 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386545-1305586395.cnf -seed 1675891500 -budget 5000

TIME LIMIT= 5000 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 1ba3e47e885ea3862e5726ce64a96a9c
RANDOM SEED=1675891500

node118.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5374.59
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      15804944 kB
Buffers:        322356 kB
Cached:        5016004 kB
SwapCached:      22068 kB
Active:       14055584 kB
Inactive:      2746544 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      15804944 kB
SwapTotal:    67111528 kB
SwapFree:     45193380 kB
Dirty:           34796 kB
Writeback:           0 kB
AnonPages:    11454732 kB
Mapped:          33348 kB
Slab:           161160 kB
PageTables:     111968 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 163658580 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73060 MiB
End job on node118 at 2011-05-17 02:16:16