Trace number 3378246

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? 4757.76 1202.37

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v30000-c126000-S643238994-057-UNKNOWN.cnf
MD5SUM2d92a52914298e3c91073398b38892d9
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark265.737
Satisfiable
(Un)Satisfiability was proved
Number of variables30000
Number of clauses126000
Sum of the clauses size378000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3126000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.32/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.32/0.67	c INFO: solving HOME/instance-3378246-1305464132.cnf
0.32/0.67	c INFO: computing task features
0.81/1.13	c INFO: computed variable-clause graph statistics
0.81/1.15	c INFO: computed clause balance statistics
3.09/3.41	c INFO: computed variable balance statistics
4.89/5.29	c INFO: computed small-clause counts
4.99/5.30	c INFO: computed Horn-clause counts
6.09/6.51	c INFO: computed variable graph statistics
6.09/6.51	c INFO: collected features for HOME/instance-3378246-1305464132.cnf in 5.82 s
6.21/6.52	c INFO: running hybridGM3@0 for 97 with 1194 remaining (b = 0.38)
6.21/6.53	c INFO: running hybridGM3@0 for 97 with 1194 remaining (b = 0.23)
6.21/6.54	c INFO: running hybridGM3@0 for 97 with 1194 remaining (b = 0.15)
6.21/6.55	c INFO: running hybridGM3@0 for 194 with 1194 remaining (b = 0.20)
397.31/104.76	c INFO: running hybridGM3@0 for 97 with 1095 remaining (b = 0.09)
404.93/106.68	c INFO: running hybridGM3@0 for 97 with 1093 remaining (b = 0.09)
425.21/111.75	c INFO: running hybridGM3@0 for 97 with 1088 remaining (b = 0.09)
792.97/204.03	c INFO: running hybridGM3@98 for 97 with 996 remaining (b = 0.09)
793.38/204.12	c INFO: running hybridGM3@100 for 97 with 996 remaining (b = 0.09)
804.92/207.07	c INFO: running hybridGM3@0 for 194 with 993 remaining (b = 0.16)
808.89/208.01	c INFO: running TNM@0 for 97 with 992 remaining (b = 0.07)
1169.01/298.44	c INFO: running hybridGM3@103 for 97 with 902 remaining (b = 0.06)
1172.59/299.35	c INFO: running hybridGM3@0 for 97 with 901 remaining (b = 0.05)
1208.40/308.35	c INFO: running hybridGM3@0 for 194 with 892 remaining (b = 0.08)
1555.80/395.79	c INFO: running hybridGM3@0 for 194 with 804 remaining (b = 0.06)
1575.27/400.60	c INFO: running hybridGM3@100 for 97 with 800 remaining (b = 0.03)
1607.07/408.67	c INFO: running hybridGM3@99 for 97 with 792 remaining (b = 0.03)
1977.96/501.95	c INFO: running hybridGM3@299 for 97 with 698 remaining (b = 0.03)
1982.32/503.00	c INFO: running TNM@0 for 97 with 697 remaining (b = 0.03)
1993.44/505.88	c INFO: running hybridGM3@97 for 97 with 694 remaining (b = 0.03)
2356.01/597.26	c INFO: running hybridGM3@0 for 194 with 603 remaining (b = 0.05)
2364.30/599.32	c INFO: running hybridGM3@98 for 97 with 601 remaining (b = 0.02)
2371.43/601.19	c INFO: running hybridGM3@191 for 97 with 599 remaining (b = 0.02)
2376.19/602.35	c INFO: running TNM@197 for 388 with 598 remaining (b = 0.07)
2763.06/699.66	c INFO: running hybridGM3@192 for 97 with 501 remaining (b = 0.02)
2774.54/702.53	c INFO: running TNM@0 for 97 with 498 remaining (b = 0.02)
3138.26/794.01	c INFO: running hybridGM3@293 for 97 with 406 remaining (b = 0.02)
3141.05/794.79	c INFO: running hybridGM3@0 for 194 with 405 remaining (b = 0.03)
3173.22/802.87	c INFO: running TNM@0 for 97 with 397 remaining (b = 0.01)
3532.82/893.37	c INFO: running hybridGM3@0 for 194 with 307 remaining (b = 0.03)
3572.10/903.22	c INFO: running TNM@0 for 97 with 297 remaining (b = 0.01)
3925.38/992.30	c INFO: running hybridGM3@99 for 97 with 208 remaining (b = 0.01)
3950.37/998.63	c INFO: running hybridGM3@392 for 97 with 202 remaining (b = 0.01)
3993.61/1009.59	c INFO: running TNM@295 for 191 with 191 remaining (b = 0.01)
4323.52/1092.72	c INFO: running TNM@0 for 107 with 107 remaining (b = 0.01)
4332.24/1094.92	c INFO: running hybridGM3@99 for 105 with 105 remaining (b = 0.01)
4336.20/1095.97	c INFO: running TNM@0 for 104 with 104 remaining (b = 0.01)
4658.27/1202.34	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378246-1305464132/watcher-3378246-1305464132 -o /tmp/evaluation-result-3378246-1305464132/solver-3378246-1305464132 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378246-1305464132.cnf -seed 766420119 -budget 4800 -cores 4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.04 7.99 6.71 5/455 31177
/proc/meminfo: memFree=21220784/32951124 swapFree=67111348/67111528
[pid=31177] ppid=31175 vsize=16240 CPUtime=0 cores=0,2,4,6
/proc/31177/stat : 31177 (python) R 31175 31177 30709 0 -1 4202496 1027 0 0 0 0 0 0 0 25 0 1 0 658010955 16629760 899 33554432000 4194304 5683644 140735457316048 18446744073709551615 5193488 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/31177/statm: 4060 905 379 364 0 547 0

[startup+0.0576401 s]
/proc/loadavg: 8.04 7.99 6.71 5/455 31177
/proc/meminfo: memFree=21220784/32951124 swapFree=67111348/67111528
[pid=31177] ppid=31175 vsize=29792 CPUtime=0.03 cores=0,2,4,6
/proc/31177/stat : 31177 (python) S 31175 31177 30709 0 -1 4202496 1415 0 0 0 2 1 0 0 25 0 1 0 658010955 30507008 1259 33554432000 4194304 5683644 140735457316048 18446744073709551615 212687673344 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/31177/statm: 7448 1259 443 364 0 818 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 29792

[startup+0.100794 s]
/proc/loadavg: 8.04 7.99 6.71 5/455 31177
/proc/meminfo: memFree=21220784/32951124 swapFree=67111348/67111528
[pid=31177] ppid=31175 vsize=89772 CPUtime=0.05 cores=0,2,4,6
/proc/31177/stat : 31177 (python) S 31175 31177 30709 0 -1 4202496 2673 0 0 0 3 2 0 0 25 0 1 0 658010955 91926528 2389 33554432000 4194304 5683644 140735457316048 18446744073709551615 212687673344 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/31177/statm: 22443 2389 750 364 0 1623 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 89772

[startup+0.300551 s]
/proc/loadavg: 8.04 7.99 6.71 5/455 31177
/proc/meminfo: memFree=21220784/32951124 swapFree=67111348/67111528
[pid=31177] ppid=31175 vsize=135752 CPUtime=0.17 cores=0,2,4,6
/proc/31177/stat : 31177 (python) R 31175 31177 30709 0 -1 4202496 5963 937 1 0 11 6 0 0 25 0 1 0 658010955 139010048 4152 33554432000 4194304 5683644 140735457316048 18446744073709551615 4619584 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/31177/statm: 33938 4152 1032 364 0 3124 0
Current children cumulated CPU time (s) 0.17
Current children cumulated vsize (KiB) 135752

[startup+0.705536 s]
/proc/loadavg: 8.04 7.99 6.71 5/455 31177
/proc/meminfo: memFree=21220784/32951124 swapFree=67111348/67111528
[pid=31177] ppid=31175 vsize=282396 CPUtime=0.4 cores=0,2,4,6
/proc/31177/stat : 31177 (python) R 31175 31177 30709 0 -1 4202496 10235 937 1 0 29 11 0 0 17 0 1 0 658010955 289173504 7919 33554432000 4194304 5683644 140735457316048 18446744073709551615 47964061549945 0 0 16781312 2 0 0 0 17 0 0 0 0
/proc/31177/statm: 70599 7919 1761 364 0 6523 0
Current children cumulated CPU time (s) 0.4
Current children cumulated vsize (KiB) 282396

[startup+1.50037 s]
/proc/loadavg: 8.04 7.99 6.71 6/456 31182
/proc/meminfo: memFree=21176012/32951124 swapFree=67111348/67111528
[pid=31177] ppid=31175 vsize=307976 CPUtime=1.19 cores=0,2,4,6
/proc/31177/stat : 31177 (python) R 31175 31177 30709 0 -1 4202496 20140 937 1 0 106 13 0 0 18 0 1 0 658010955 315367424 14379 33554432000 4194304 5683644 140735457316048 18446744073709551615 4602375 0 0 16781312 2 0 0 0 17 0 0 0 0
/proc/31177/statm: 76994 14379 1826 364 0 12918 0
Current children cumulated CPU time (s) 1.19
Current children cumulated vsize (KiB) 307976

[startup+3.10107 s]
/proc/loadavg: 8.04 7.99 6.71 6/456 31182
/proc/meminfo: memFree=21167320/32951124 swapFree=67111348/67111528
[pid=31177] ppid=31175 vsize=307976 CPUtime=2.79 cores=0,2,4,6
/proc/31177/stat : 31177 (python) R 31175 31177 30709 0 -1 4202496 20140 937 1 0 266 13 0 0 21 0 1 0 658010955 315367424 14379 33554432000 4194304 5683644 140735457316048 18446744073709551615 212687333533 0 0 16781312 2 0 0 0 17 0 0 0 0
/proc/31177/statm: 76994 14379 1826 364 0 12918 0
Current children cumulated CPU time (s) 2.79
Current children cumulated vsize (KiB) 307976

[startup+6.30049 s]
/proc/loadavg: 7.79 7.94 6.70 6/456 31182
/proc/meminfo: memFree=21167388/32951124 swapFree=67111348/67111528
[pid=31177] ppid=31175 vsize=307976 CPUtime=5.99 cores=0,2,4,6
/proc/31177/stat : 31177 (python) R 31175 31177 30709 0 -1 4202496 20147 937 1 0 586 13 0 0 25 0 1 0 658010955 315367424 14386 33554432000 4194304 5683644 140735457316048 18446744073709551615 47963890698368 0 0 16781312 2 0 0 0 17 0 0 0 0
/proc/31177/statm: 76994 14386 1833 364 0 12918 0
Current children cumulated CPU time (s) 5.99
Current children cumulated vsize (KiB) 307976

[startup+12.7013 s]
/proc/loadavg: 7.81 7.94 6.71 9/468 31194
/proc/meminfo: memFree=21036368/32951124 swapFree=67111348/67111528
[pid=31177] ppid=31175 vsize=346664 CPUtime=6.25 cores=0,2,4,6
/proc/31177/stat : 31177 (python) S 31175 31177 30709 0 -1 4202496 22941 937 1 0 611 14 0 0 25 0 5 0 658010955 354983936 13855 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699502027 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/31177/statm: 86666 13855 1873 364 0 22575 0
[pid=31183] ppid=31177 vsize=305652 CPUtime=0.01 cores=0,2,4,6
/proc/31183/stat : 31183 (python) S 31177 31177 30709 0 -1 4202560 882 0 0 0 0 1 0 0 18 0 1 0 658011608 312987648 12342 33554432000 4194304 5683644 140735457316048 18446744073709551615 212687695375 0 0 16781312 514 426984899581280515 0 0 17 6 0 0 0
/proc/31183/statm: 76413 12342 376 364 0 12331 0
[pid=31185] ppid=31183 vsize=268420 CPUtime=6.15 cores=0,2,4,6
/proc/31185/stat : 31185 (hybridGM3) R 31183 31185 31185 0 -1 4202496 7608 0 0 0 615 0 0 0 25 0 1 0 658011608 274862080 7343 33554432000 134512640 135204385 4288627888 18446744073709551615 134545975 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/31185/statm: 67105 7343 64 169 0 66935 0
[pid=31186] ppid=31177 vsize=315908 CPUtime=0.01 cores=0,2,4,6
/proc/31186/stat : 31186 (python) S 31177 31177 30709 0 -1 4202560 910 0 0 0 0 1 0 0 18 0 1 0 658011609 323489792 12340 33554432000 4194304 5683644 140735457316048 18446744073709551615 212687695430 0 0 16781312 514 10411848728141857026 0 0 17 6 0 0 0
/proc/31186/statm: 78977 12340 370 364 0 14892 0
[pid=31188] ppid=31186 vsize=268428 CPUtime=6.15 cores=0,2,4,6
/proc/31188/stat : 31188 (hybridGM3) R 31186 31188 31188 0 -1 4202496 7606 0 0 0 614 1 0 0 25 0 1 0 658011609 274870272 7343 33554432000 134512640 135204385 4286797552 18446744073709551615 134537383 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/31188/statm: 67107 7343 64 169 0 66937 0
[pid=31189] ppid=31177 vsize=326164 CPUtime=0.02 cores=0,2,4,6
/proc/31189/stat : 31189 (python) S 31177 31177 30709 0 -1 4202560 911 0 0 0 1 1 0 0 18 0 1 0 658011610 333991936 12344 33554432000 4194304 5683644 140735457316048 18446744073709551615 212687695430 0 0 16781312 514 4255488235788009732 0 0 17 6 0 0 0
/proc/31189/statm: 81541 12344 370 364 0 17453 0
[pid=31191] ppid=31189 vsize=268420 CPUtime=5.99 cores=0,2,4,6
/proc/31191/stat : 31191 (hybridGM3) R 31189 31191 31191 0 -1 4202496 7611 0 0 0 599 0 0 0 25 0 1 0 658011610 274862080 7348 33554432000 134512640 135204385 4291026912 18446744073709551615 134537565 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/31191/statm: 67105 7348 64 169 0 66935 0
[pid=31192] ppid=31177 vsize=336420 CPUtime=0.02 cores=0,2,4,6
/proc/31192/stat : 31192 (python) S 31177 31177 30709 0 -1 4202560 912 0 0 0 1 1 0 0 18 0 1 0 658011611 344494080 12348 33554432000 4194304 5683644 140735457316048 18446744073709551615 212687695430 0 0 16781312 514 4606905071286845699 0 0 17 0 0 0 0
/proc/31192/statm: 84105 12348 370 364 0 20014 0
[pid=31194] ppid=31192 vsize=268428 CPUtime=6.11 cores=0,2,4,6
/proc/31194/stat : 31194 (hybridGM3) R 31192 31194 31194 0 -1 4202496 7603 0 0 0 610 1 0 0 25 0 1 0 658011611 274870272 7344 33554432000 134512640 135204385 4293752832 18446744073709551615 134537292 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/31194/statm: 67107 7344 64 169 0 66937 0
[pid=31177/tid=31184] ppid=31175 vsize=346664 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31184/stat : 31184 (python) S 31175 31177 30709 0 -1 4202560 24 937 0 0 0 0 0 0 25 0 5 0 658011608 354983936 13855 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 6 0 0 0
[pid=31177/tid=31187] ppid=31175 vsize=346664 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31187/stat : 31187 (python) S 31175 31177 30709 0 -1 4202560 22 937 0 0 0 0 0 0 25 0 5 0 658011609 354983936 13855 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 6 0 0 0
[pid=31177/tid=31190] ppid=31175 vsize=346664 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31190/stat : 31190 (python) S 31175 31177 30709 0 -1 4202560 21 937 0 0 0 0 0 0 25 0 5 0 658011610 354983936 13855 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 6 0 0 0
[pid=31177/tid=31193] ppid=31175 vsize=346664 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31193/stat : 31193 (python) S 31175 31177 30709 0 -1 4202560 27 937 0 0 0 0 0 0 25 0 5 0 658011611 354983936 13855 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 30.71
Current children cumulated vsize (KiB) 2704504

[startup+25.501 s]
/proc/loadavg: 7.93 7.96 6.74 9/468 31194
/proc/meminfo: memFree=21036612/32951124 swapFree=67111348/67111528
[pid=31177] ppid=31175 vsize=346664 CPUtime=6.25 cores=0,2,4,6
/proc/31177/stat : 31177 (python) S 31175 31177 30709 0 -1 4202496 22941 937 1 0 611 14 0 0 25 0 5 0 658010955 354983936 13855 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699502027 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/31177/statm: 86666 13855 1873 364 0 22575 0
[pid=31183] ppid=31177 vsize=305652 CPUtime=0.05 cores=0,2,4,6
/proc/31183/stat : 31183 (python) S 31177 31177 30709 0 -1 4202560 890 0 0 0 2 3 0 0 18 0 1 0 658011608 312987648 12342 33554432000 4194304 5683644 140735457316048 18446744073709551615 212687695375 0 0 16781312 514 8332028746542317827 0 0 17 6 0 0 0
/proc/31183/statm: 76413 12342 376 364 0 12331 0
[pid=31185] ppid=31183 vsize=268420 CPUtime=18.95 cores=0,2,4,6
/proc/31185/stat : 31185 (hybridGM3) R 31183 31185 31185 0 -1 4202496 7611 0 0 0 1895 0 0 0 25 0 1 0 658011608 274862080 7346 33554432000 134512640 135204385 4288627888 18446744073709551615 134537403 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/31185/statm: 67105 7346 64 169 0 66935 0
[pid=31186] ppid=31177 vsize=315908 CPUtime=0.05 cores=0,2,4,6
/proc/31186/stat : 31186 (python) S 31177 31177 30709 0 -1 4202560 916 0 0 0 3 2 0 0 18 0 1 0 658011609 323489792 12340 33554432000 4194304 5683644 140735457316048 18446744073709551615 212687695430 0 0 16781312 514 8332028746542317826 0 0 17 6 0 0 0
/proc/31186/statm: 78977 12340 370 364 0 14892 0
[pid=31188] ppid=31186 vsize=268428 CPUtime=18.95 cores=0,2,4,6
/proc/31188/stat : 31188 (hybridGM3) R 31186 31188 31188 0 -1 4202496 7606 0 0 0 1894 1 0 0 25 0 1 0 658011609 274870272 7343 33554432000 134512640 135204385 4286797552 18446744073709551615 134537363 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/31188/statm: 67107 7343 64 169 0 66937 0
[pid=31189] ppid=31177 vsize=326164 CPUtime=0.06 cores=0,2,4,6
/proc/31189/stat : 31189 (python) S 31177 31177 30709 0 -1 4202560 917 0 0 0 3 3 0 0 18 0 1 0 658011610 333991936 12344 33554432000 4194304 5683644 140735457316048 18446744073709551615 212687695430 0 0 16781312 514 8332028746542317828 0 0 17 6 0 0 0
/proc/31189/statm: 81541 12344 370 364 0 17453 0
[pid=31191] ppid=31189 vsize=268420 CPUtime=18.48 cores=0,2,4,6
/proc/31191/stat : 31191 (hybridGM3) R 31189 31191 31191 0 -1 4202496 7611 0 0 0 1848 0 0 0 25 0 1 0 658011610 274862080 7348 33554432000 134512640 135204385 4291026912 18446744073709551615 134537548 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/31191/statm: 67105 7348 64 169 0 66935 0
[pid=31192] ppid=31177 vsize=336420 CPUtime=0.05 cores=0,2,4,6
/proc/31192/stat : 31192 (python) S 31177 31177 30709 0 -1 4202560 919 0 0 0 3 2 0 0 18 0 1 0 658011611 344494080 12348 33554432000 4194304 5683644 140735457316048 18446744073709551615 212687695430 0 0 16781312 514 8332028746542317827 0 0 17 0 0 0 0
/proc/31192/statm: 84105 12348 370 364 0 20014 0
[pid=31194] ppid=31192 vsize=268428 CPUtime=18.87 cores=0,2,4,6
/proc/31194/stat : 31194 (hybridGM3) R 31192 31194 31194 0 -1 4202496 7605 0 0 0 1886 1 0 0 25 0 1 0 658011611 274870272 7346 33554432000 134512640 135204385 4293752832 18446744073709551615 134537272 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/31194/statm: 67107 7346 64 169 0 66937 0
[pid=31177/tid=31184] ppid=31175 vsize=346664 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31184/stat : 31184 (python) S 31175 31177 30709 0 -1 4202560 24 937 0 0 0 0 0 0 25 0 5 0 658011608 354983936 13855 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 6 0 0 0
[pid=31177/tid=31187] ppid=31175 vsize=346664 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31187/stat : 31187 (python) S 31175 31177 30709 0 -1 4202560 22 937 0 0 0 0 0 0 25 0 5 0 658011609 354983936 13855 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 6 0 0 0
[pid=31177/tid=31190] ppid=31175 vsize=346664 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31190/stat : 31190 (python) S 31175 31177 30709 0 -1 4202560 21 937 0 0 0 0 0 0 25 0 5 0 658011610 354983936 13855 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 6 0 0 0
[pid=31177/tid=31193] ppid=31175 vsize=346664 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31193/stat : 31193 (python) S 31175 31177 30709 0 -1 4202560 27 937 0 0 0 0 0 0 25 0 5 0 658011611 354983936 13855 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 81.71
Current children cumulated vsize (KiB) 2704504

[startup+51.1013 s]
/proc/loadavg: 8.02 7.98 6.77 9/476 31203
/proc/meminfo: memFree=20955916/32951124 swapFree=67111348/67111528
[pid=31177] ppid=31175 vsize=346664 CPUtime=6.25 cores=0,2,4,6
/proc/31177/stat : 31177 (python) S 31175 31177 30709 0 -1 4202496 22941 937 1 0 611 14 0 0 25 0 5 0 658010955 354983936 13855 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699502027 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/31177/statm: 86666 13855 1873 364 0 22575 0
[pid=31183] ppid=31177 vsize=305652 CPUtime=0.13 cores=0,2,4,6
/proc/31183/stat : 31183 (python) S 31177 31177 30709 0 -1 4202560 900 0 0 0 6 7 0 0 18 0 1 0 658011608 312987648 12342 33554432000 4194304 5683644 140735457316048 18446744073709551615 212687695375 0 0 16781312 514 8332028746542317827 0 0 17 0 0 0 0
/proc/31183/statm: 76413 12342 376 364 0 12331 0
[pid=31185] ppid=31183 vsize=268420 CPUtime=44.56 cores=0,2,4,6
/proc/31185/stat : 31185 (hybridGM3) R 31183 31185 31185 0 -1 4202496 7611 0 0 0 4455 1 0 0 25 0 1 0 658011608 274862080 7346 33554432000 134512640 135204385 4288627888 18446744073709551615 134537363 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/31185/statm: 67105 7346 64 169 0 66935 0
[pid=31186] ppid=31177 vsize=315908 CPUtime=0.13 cores=0,2,4,6
/proc/31186/stat : 31186 (python) S 31177 31177 30709 0 -1 4202560 926 0 0 0 7 6 0 0 18 0 1 0 658011609 323489792 12340 33554432000 4194304 5683644 140735457316048 18446744073709551615 212687695430 0 0 16781312 514 8332028746542317826 0 0 17 6 0 0 0
/proc/31186/statm: 78977 12340 370 364 0 14892 0
[pid=31188] ppid=31186 vsize=268428 CPUtime=44.54 cores=0,2,4,6
/proc/31188/stat : 31188 (hybridGM3) R 31186 31188 31188 0 -1 4202496 7606 0 0 0 4453 1 0 0 25 0 1 0 658011609 274870272 7343 33554432000 134512640 135204385 4286797552 18446744073709551615 134536698 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/31188/statm: 67107 7343 64 169 0 66937 0
[pid=31189] ppid=31177 vsize=326164 CPUtime=0.14 cores=0,2,4,6
/proc/31189/stat : 31189 (python) S 31177 31177 30709 0 -1 4202560 928 0 0 0 8 6 0 0 18 0 1 0 658011610 333991936 12344 33554432000 4194304 5683644 140735457316048 18446744073709551615 212687695430 0 0 16781312 514 8332028746542317828 0 0 17 6 0 0 0
/proc/31189/statm: 81541 12344 370 364 0 17453 0
[pid=31191] ppid=31189 vsize=268420 CPUtime=43.54 cores=0,2,4,6
/proc/31191/stat : 31191 (hybridGM3) R 31189 31191 31191 0 -1 4202496 7611 0 0 0 4354 0 0 0 25 0 1 0 658011610 274862080 7348 33554432000 134512640 135204385 4291026912 18446744073709551615 134537027 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/31191/statm: 67105 7348 64 169 0 66935 0
[pid=31192] ppid=31177 vsize=336420 CPUtime=0.13 cores=0,2,4,6
/proc/31192/stat : 31192 (python) S 31177 31177 30709 0 -1 4202560 930 0 0 0 7 6 0 0 18 0 1 0 658011611 344494080 12348 33554432000 4194304 5683644 140735457316048 18446744073709551615 212687695430 0 0 16781312 514 8332028746542317827 0 0 17 0 0 0 0
/proc/31192/statm: 84105 12348 370 364 0 20014 0

################
# More data... #
################

/proc/31569/task/31576/stat : 31576 (python) S 31177 31177 30709 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 658111913 470519808 12434 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 514 18446604442096513472 0 0 -1 6 0 0 0
[pid=31579] ppid=31177 vsize=469748 CPUtime=0.37 cores=0,2,4,6
/proc/31579/stat : 31579 (python) S 31177 31177 30709 0 -1 4202560 1045 0 0 0 19 18 0 0 18 0 1 0 658120228 481021952 12408 33554432000 4194304 5683644 140735457316048 18446744073709551615 212687695430 0 0 16781312 514 8332028746542317827 0 0 17 2 0 0 0
/proc/31579/statm: 117437 12408 378 364 0 53307 0
[pid=31581] ppid=31579 vsize=463080 CPUtime=108.79 cores=0,2,4,6
/proc/31581/stat : 31581 (TNM) R 31579 31581 31581 0 -1 4202496 5214 0 0 0 10879 0 0 0 25 0 1 0 658120228 474193920 4941 33554432000 134512640 135008748 4291889904 18446744073709551615 134528974 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/31581/statm: 115770 4941 46 122 0 114670 0
[pid=31583] ppid=31177 vsize=480004 CPUtime=0.35 cores=0,2,4,6
/proc/31583/stat : 31583 (python) S 31177 31177 30709 0 -1 4202560 1029 0 0 0 18 17 0 0 18 0 1 0 658120553 491524096 12406 33554432000 4194304 5683644 140735457316048 18446744073709551615 212687695430 0 0 16781312 514 8332028746542317827 0 0 17 4 0 0 0
/proc/31583/statm: 120001 12406 372 364 0 55868 0
[pid=31585] ppid=31583 vsize=463080 CPUtime=102.27 cores=0,2,4,6
/proc/31585/stat : 31585 (TNM) R 31583 31585 31585 0 -1 4202496 5220 0 0 0 10226 1 0 0 25 0 1 0 658120553 474193920 4943 33554432000 134512640 135008748 4287656832 18446744073709551615 134524904 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/31585/statm: 115770 4943 46 122 0 114670 0
[pid=31177/tid=31184] ppid=31175 vsize=490248 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31184/stat : 31184 (python) S 31175 31177 30709 0 -1 4202560 32 937 0 0 0 0 0 0 18 0 19 0 658011608 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 6 0 0 0
[pid=31177/tid=31187] ppid=31175 vsize=490248 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31187/stat : 31187 (python) S 31175 31177 30709 0 -1 4202560 26 937 0 0 0 0 0 0 18 0 19 0 658011609 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 6 0 0 0
[pid=31177/tid=31190] ppid=31175 vsize=490248 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31190/stat : 31190 (python) S 31175 31177 30709 0 -1 4202560 24 937 0 0 0 0 0 0 18 0 19 0 658011610 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 6 0 0 0
[pid=31177/tid=31193] ppid=31175 vsize=490248 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31193/stat : 31193 (python) S 31175 31177 30709 0 -1 4202560 28 937 0 0 0 0 0 0 18 0 19 0 658011611 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 0 0 0 0
[pid=31177/tid=31217] ppid=31175 vsize=490248 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31217/stat : 31217 (python) S 31175 31177 30709 0 -1 4202560 33 937 0 0 0 0 0 0 18 0 19 0 658031663 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 0 0 0 0
[pid=31177/tid=31220] ppid=31175 vsize=490248 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31220/stat : 31220 (python) S 31175 31177 30709 0 -1 4202560 28 937 0 0 0 0 0 0 20 0 19 0 658031757 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 0 0 0 0
[pid=31177/tid=31227] ppid=31175 vsize=490248 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31227/stat : 31227 (python) S 31175 31177 30709 0 -1 4202560 38 937 0 0 0 0 0 0 18 0 19 0 658040890 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 0 0 0 0
[pid=31177/tid=31231] ppid=31175 vsize=490248 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31231/stat : 31231 (python) S 31175 31177 30709 0 -1 4202560 37 937 0 0 0 0 0 0 18 0 19 0 658041792 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 0 0 0 0
[pid=31177/tid=31249] ppid=31175 vsize=490248 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31249/stat : 31249 (python) S 31175 31177 30709 0 -1 4202560 28 937 0 0 0 0 0 0 18 0 19 0 658050535 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 0 0 0 0
[pid=31177/tid=31520] ppid=31175 vsize=490248 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31520/stat : 31520 (python) S 31175 31177 30709 0 -1 4202560 34 937 0 0 0 0 0 0 18 0 19 0 658061256 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 4 0 0 0
[pid=31177/tid=31538] ppid=31175 vsize=490248 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31538/stat : 31538 (python) S 31175 31177 30709 0 -1 4202560 26 937 0 0 0 0 0 0 20 0 19 0 658070682 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 4 0 0 0
[pid=31177/tid=31545] ppid=31175 vsize=490248 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31545/stat : 31545 (python) S 31175 31177 30709 0 -1 4202560 28 937 0 0 0 0 0 0 20 0 19 0 658081209 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 4 0 0 0
[pid=31177/tid=31552] ppid=31175 vsize=490248 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31552/stat : 31552 (python) S 31175 31177 30709 0 -1 4202560 32 937 0 0 0 0 0 0 18 0 19 0 658090435 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 4 0 0 0
[pid=31177/tid=31556] ppid=31175 vsize=490248 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31556/stat : 31556 (python) S 31175 31177 30709 0 -1 4202560 28 937 0 0 0 0 0 0 20 0 19 0 658091243 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 4 0 0 0
[pid=31177/tid=31565] ppid=31175 vsize=490248 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31565/stat : 31565 (python) S 31175 31177 30709 0 -1 4202560 31 937 0 0 0 0 0 0 18 0 19 0 658100293 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 4 0 0 0
[pid=31177/tid=31570] ppid=31175 vsize=490248 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31570/stat : 31570 (python) S 31175 31177 30709 0 -1 4202560 24 937 0 0 0 0 0 0 20 0 19 0 658101278 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 4 0 0 0
[pid=31177/tid=31580] ppid=31175 vsize=490248 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31580/stat : 31580 (python) S 31175 31177 30709 0 -1 4202560 26 937 0 0 0 0 0 0 20 0 19 0 658120228 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 4 0 0 0
[pid=31177/tid=31584] ppid=31175 vsize=490248 CPUtime=0 cores=0,2,4,6
/proc/31177/task/31584/stat : 31584 (python) S 31175 31177 30709 0 -1 4202560 28 937 0 0 0 0 0 0 20 0 19 0 658120553 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4756.52
Current children cumulated vsize (KiB) 13779912

[startup+1202.32 s]
/proc/loadavg: 8.21 8.18 7.76 9/516 31592
/proc/meminfo: memFree=20676180/32951124 swapFree=67111348/67111528
[pid=31177] ppid=31175 vsize=490248 CPUtime=2664.71 cores=0,2,4,6
/proc/31177/stat : 31177 (python) S 31175 31177 30709 0 -1 4202496 37023 109133 1 0 641 20 265351 459 17 0 19 0 658010955 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699505471 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/31177/statm: 122562 13948 1910 364 0 58429 0
[pid=31226] ppid=31177 vsize=367188 CPUtime=1.36 cores=0,2,4,6
/proc/31226/stat : 31226 (python) S 31177 31177 30709 0 -1 4202560 1167 0 0 0 69 67 0 0 18 0 2 0 658040890 376000512 12390 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699502027 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/31226/statm: 91797 12390 399 364 0 27697 0
[pid=31228] ppid=31226 vsize=268428 CPUtime=390.79 cores=0,2,4,6
/proc/31228/stat : 31228 (hybridGM3) T 31226 31228 31228 0 -1 4202496 7610 0 0 0 39072 7 0 0 25 0 1 0 658040891 274870272 7349 33554432000 134512640 135204385 4291589888 18446744073709551615 134545853 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/31228/statm: 67107 7349 64 169 0 66937 0
[pid=31226/tid=31251] ppid=31177 vsize=367188 CPUtime=0 cores=0,2,4,6
/proc/31226/task/31251/stat : 31251 (python) S 31177 31177 30709 0 -1 4202560 49 0 0 0 0 0 0 0 18 0 2 0 658051015 376000512 12390 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 514 18446604450577279040 0 0 -1 6 0 0 0
[pid=31519] ppid=31177 vsize=397956 CPUtime=2.36 cores=0,2,4,6
/proc/31519/stat : 31519 (python) S 31177 31177 30709 0 -1 4202560 1181 0 0 0 127 109 0 0 18 0 2 0 658061256 407506944 12402 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699502027 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/31519/statm: 99489 12402 399 364 0 35380 0
[pid=31521] ppid=31519 vsize=463080 CPUtime=679.47 cores=0,2,4,6
/proc/31521/stat : 31521 (TNM) T 31519 31521 31521 0 -1 4202496 5205 0 0 0 67946 1 0 0 25 0 1 0 658061256 474193920 4941 33554432000 134512640 135008748 4286762528 18446744073709551615 134529468 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/31521/statm: 115770 4941 46 122 0 114670 0
[pid=31519/tid=31541] ppid=31177 vsize=397956 CPUtime=0 cores=0,2,4,6
/proc/31519/task/31541/stat : 31541 (python) S 31177 31177 30709 0 -1 4202560 48 0 0 0 0 0 0 0 18 0 2 0 658071190 407506944 12402 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 514 18446604453942916928 0 0 -1 2 0 0 0
[pid=31551] ppid=31177 vsize=428724 CPUtime=1 cores=0,2,4,6
/proc/31551/stat : 31551 (python) S 31177 31177 30709 0 -1 4202560 1190 0 0 0 53 47 0 0 18 0 2 0 658090435 439013376 12414 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699502027 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/31551/statm: 107181 12414 399 364 0 43063 0
[pid=31553] ppid=31551 vsize=268424 CPUtime=296.22 cores=0,2,4,6
/proc/31553/stat : 31553 (hybridGM3) T 31551 31553 31553 0 -1 4202496 7614 0 0 0 29621 1 0 0 25 0 1 0 658090436 274866176 7346 33554432000 134512640 135204385 4290555376 18446744073709551615 134537565 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/31553/statm: 67106 7346 64 169 0 66936 0
[pid=31551/tid=31574] ppid=31177 vsize=428724 CPUtime=0 cores=0,2,4,6
/proc/31551/task/31574/stat : 31574 (python) S 31177 31177 30709 0 -1 4202560 48 0 0 0 0 0 0 0 18 0 2 0 658110185 439013376 12414 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 514 18446604448662291648 0 0 -1 2 0 0 0
[pid=31564] ppid=31177 vsize=449236 CPUtime=1.05 cores=0,2,4,6
/proc/31564/stat : 31564 (python) S 31177 31177 30709 0 -1 4202560 1250 0 0 0 54 51 0 0 18 0 2 0 658100292 460017664 12439 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699502027 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/31564/statm: 112309 12439 416 364 0 48185 0
[pid=31566] ppid=31564 vsize=268424 CPUtime=305.68 cores=0,2,4,6
/proc/31566/stat : 31566 (hybridGM3) T 31564 31566 31566 0 -1 4202496 7625 0 0 0 30567 1 0 0 25 0 1 0 658100293 274866176 7351 33554432000 134512640 135204385 4290801376 18446744073709551615 134536646 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/31566/statm: 67106 7351 64 169 0 66936 0
[pid=31564/tid=31582] ppid=31177 vsize=449236 CPUtime=0 cores=0,2,4,6
/proc/31564/task/31582/stat : 31582 (python) S 31177 31177 30709 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 658120447 460017664 12439 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 514 18446604451151406016 0 0 -1 0 0 0 0
[pid=31569] ppid=31177 vsize=459492 CPUtime=0.37 cores=0,2,4,6
/proc/31569/stat : 31569 (python) S 31177 31177 30709 0 -1 4202560 1205 0 0 0 19 18 0 0 18 0 2 0 658101277 470519808 12434 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699505471 0 0 16781824 2 18446744073709551615 0 0 17 4 0 0 0
/proc/31569/statm: 114873 12434 407 364 0 50746 0
[pid=31571] ppid=31569 vsize=463080 CPUtime=102.7 cores=0,2,4,6
/proc/31571/stat : 31571 (TNM) R 31569 31571 31571 0 -1 4202496 5219 0 0 0 10269 1 0 0 18 0 1 0 658101278 474193920 4942 33554432000 134512640 135008748 4289769136 18446744073709551615 134527773 256 0 16781312 0 0 0 0 17 6 0 0 0
/proc/31571/statm: 115770 4942 46 122 0 114670 0
[pid=31569/tid=31576] ppid=31177 vsize=459492 CPUtime=0 cores=0,2,4,6
/proc/31569/task/31576/stat : 31576 (python) S 31177 31177 30709 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 658111913 470519808 12434 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781824 2 18446604442096513472 0 0 -1 6 0 0 0
[pid=31579] ppid=31177 vsize=469748 CPUtime=0.37 cores=0,2,4,6
/proc/31579/stat : 31579 (python) S 31177 31177 30709 0 -1 4202560 1045 0 0 0 19 18 0 0 18 0 1 0 658120228 481021952 12408 33554432000 4194304 5683644 140735457316048 18446744073709551615 212687695430 0 0 16781312 514 8332028746542317827 0 0 17 2 0 0 0
/proc/31579/statm: 117437 12408 378 364 0 53307 0
[pid=31581] ppid=31579 vsize=463080 CPUtime=109.19 cores=0,2,4,6
/proc/31581/stat : 31581 (TNM) R 31579 31581 31581 0 -1 4202496 5214 0 0 0 10919 0 0 0 25 0 1 0 658120228 474193920 4941 33554432000 134512640 135008748 4291889904 18446744073709551615 134527475 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/31581/statm: 115770 4941 46 122 0 114670 0
[pid=31583] ppid=31177 vsize=480004 CPUtime=0.35 cores=0,2,4,6
/proc/31583/stat : 31583 (python) S 31177 31177 30709 0 -1 4202560 1029 0 0 0 18 17 0 0 18 0 1 0 658120553 491524096 12406 33554432000 4194304 5683644 140735457316048 18446744073709551615 212687695430 0 0 16781312 514 8332028746542317827 0 0 17 4 0 0 0
/proc/31583/statm: 120001 12406 372 364 0 55868 0
[pid=31585] ppid=31583 vsize=463080 CPUtime=102.65 cores=0,2,4,6
/proc/31585/stat : 31585 (TNM) R 31583 31585 31585 0 -1 4202496 5220 0 0 0 10264 1 0 0 25 0 1 0 658120553 474193920 4943 33554432000 134512640 135008748 4287656832 18446744073709551615 134527722 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/31585/statm: 115770 4943 46 122 0 114670 0
[pid=31177/tid=31184] ppid=31175 vsize=490248 CPUtime=2758.16 cores=0,2,4,6
/proc/31177/task/31184/stat : 31184 (python) S 31175 31177 30709 0 -1 4202560 32 115529 0 0 0 0 275338 478 18 0 19 0 658011608 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 6 0 0 0
[pid=31177/tid=31187] ppid=31175 vsize=490248 CPUtime=2758.16 cores=0,2,4,6
/proc/31177/task/31187/stat : 31187 (python) S 31175 31177 30709 0 -1 4202560 26 115529 0 0 0 0 275338 478 18 0 19 0 658011609 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 6 0 0 0
[pid=31177/tid=31190] ppid=31175 vsize=490248 CPUtime=2758.16 cores=0,2,4,6
/proc/31177/task/31190/stat : 31190 (python) S 31175 31177 30709 0 -1 4202560 24 115529 0 0 0 0 275338 478 18 0 19 0 658011610 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 6 0 0 0
[pid=31177/tid=31193] ppid=31175 vsize=490248 CPUtime=2758.16 cores=0,2,4,6
/proc/31177/task/31193/stat : 31193 (python) S 31175 31177 30709 0 -1 4202560 28 115529 0 0 0 0 275338 478 18 0 19 0 658011611 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 0 0 0 0
[pid=31177/tid=31217] ppid=31175 vsize=490248 CPUtime=2758.16 cores=0,2,4,6
/proc/31177/task/31217/stat : 31217 (python) S 31175 31177 30709 0 -1 4202560 33 115529 0 0 0 0 275338 478 18 0 19 0 658031663 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 0 0 0 0
[pid=31177/tid=31220] ppid=31175 vsize=490248 CPUtime=2758.16 cores=0,2,4,6
/proc/31177/task/31220/stat : 31220 (python) S 31175 31177 30709 0 -1 4202560 28 115529 0 0 0 0 275338 478 20 0 19 0 658031757 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 0 0 0 0
[pid=31177/tid=31227] ppid=31175 vsize=490248 CPUtime=2758.16 cores=0,2,4,6
/proc/31177/task/31227/stat : 31227 (python) S 31175 31177 30709 0 -1 4202560 38 115529 0 0 0 0 275338 478 18 0 19 0 658040890 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 0 0 0 0
[pid=31177/tid=31231] ppid=31175 vsize=490248 CPUtime=2758.16 cores=0,2,4,6
/proc/31177/task/31231/stat : 31231 (python) S 31175 31177 30709 0 -1 4202560 37 115529 0 0 0 0 275338 478 18 0 19 0 658041792 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 0 0 0 0
[pid=31177/tid=31249] ppid=31175 vsize=490248 CPUtime=2758.16 cores=0,2,4,6
/proc/31177/task/31249/stat : 31249 (python) S 31175 31177 30709 0 -1 4202560 28 115529 0 0 0 0 275338 478 18 0 19 0 658050535 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 0 0 0 0
[pid=31177/tid=31520] ppid=31175 vsize=490248 CPUtime=2758.16 cores=0,2,4,6
/proc/31177/task/31520/stat : 31520 (python) S 31175 31177 30709 0 -1 4202560 34 115529 0 0 0 0 275338 478 18 0 19 0 658061256 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 4 0 0 0
[pid=31177/tid=31538] ppid=31175 vsize=490248 CPUtime=2758.16 cores=0,2,4,6
/proc/31177/task/31538/stat : 31538 (python) S 31175 31177 30709 0 -1 4202560 26 115529 0 0 0 0 275338 478 20 0 19 0 658070682 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 4 0 0 0
[pid=31177/tid=31545] ppid=31175 vsize=490248 CPUtime=2758.16 cores=0,2,4,6
/proc/31177/task/31545/stat : 31545 (python) S 31175 31177 30709 0 -1 4202560 28 115529 0 0 0 0 275338 478 20 0 19 0 658081209 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 4 0 0 0
[pid=31177/tid=31552] ppid=31175 vsize=490248 CPUtime=2758.16 cores=0,2,4,6
/proc/31177/task/31552/stat : 31552 (python) S 31175 31177 30709 0 -1 4202560 32 115529 0 0 0 0 275338 478 18 0 19 0 658090435 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 4 0 0 0
[pid=31177/tid=31556] ppid=31175 vsize=490248 CPUtime=2758.16 cores=0,2,4,6
/proc/31177/task/31556/stat : 31556 (python) S 31175 31177 30709 0 -1 4202560 28 115529 0 0 0 0 275338 478 20 0 19 0 658091243 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 4 0 0 0
[pid=31177/tid=31565] ppid=31175 vsize=490248 CPUtime=2758.16 cores=0,2,4,6
/proc/31177/task/31565/stat : 31565 (python) S 31175 31177 30709 0 -1 4202560 31 115529 0 0 0 0 275338 478 18 0 19 0 658100293 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 4 0 0 0
[pid=31177/tid=31570] ppid=31175 vsize=490248 CPUtime=2758.16 cores=0,2,4,6
/proc/31177/task/31570/stat : 31570 (python) S 31175 31177 30709 0 -1 4202560 24 115529 0 0 0 0 275338 478 20 0 19 0 658101278 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 4 0 0 0
[pid=31177/tid=31580] ppid=31175 vsize=490248 CPUtime=2758.16 cores=0,2,4,6
/proc/31177/task/31580/stat : 31580 (python) S 31175 31177 30709 0 -1 4202560 26 115529 0 0 0 0 275338 478 20 0 19 0 658120228 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 4 0 0 0
[pid=31177/tid=31584] ppid=31175 vsize=490248 CPUtime=2758.16 cores=0,2,4,6
/proc/31177/task/31584/stat : 31584 (python) S 31175 31177 30709 0 -1 4202560 28 115529 0 0 0 0 275338 478 20 0 19 0 658120553 502013952 13948 33554432000 4194304 5683644 140735457316048 18446744073709551615 212699498929 0 0 16781312 2 18446604450577278144 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4658.27
Current children cumulated vsize (KiB) 6200192

Child status: 0

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 31177 and gives
#  childrusage.ru_utime.tv_sec=4750
#  childrusage.ru_utime.tv_usec=112873
#  childrusage.ru_stime.tv_sec=8
#  childrusage.ru_stime.tv_usec=488709
# CPU time returned by wait4() is 4758.6
# while last known CPU time is 4658.27
#
# Using CPU time of the last sample as value...


# WARNING:
# The CPU time of some children was not reported to their father
# (probably because of a missing or aborted wait()).
# This 'lost CPU time' is added to the watched process CPU time.
#  lost CPU time (s): 99.4897
#  lost CPU user time (s): 99.4697
#  lost CPU system time (s): 0.0200005

Real time (s): 1202.37
CPU time (s): 4757.76
CPU user time (s): 4749.56
CPU system time (s): 8.2
CPU usage (%): 395.698
Max. virtual memory (cumulated for all children) (KiB): 13779916

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4750.11
system time used= 8.48871
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 211793
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 9140
involuntary context switches= 31393

runsolver used 8.5327 second user time and 21.2848 second system time

The end

Launcher Data 

Begin job on node139 at 2011-05-15 14:55:32
IDJOB=3378246
IDBENCH=82655
IDSOLVER=1859
FILE ID=node139/3378246-1305464132
RUNJOBID= node139-1305463021-30728
PBS_JOBID= 13324978
Free space on /tmp= 73344 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v30000-c126000-S643238994-057-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378246-1305464132/watcher-3378246-1305464132 -o /tmp/evaluation-result-3378246-1305464132/solver-3378246-1305464132 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378246-1305464132.cnf -seed 766420119 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 2d92a52914298e3c91073398b38892d9
RANDOM SEED=766420119

node139.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5321.49
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      21220940 kB
Buffers:        320092 kB
Cached:        3092268 kB
SwapCached:          0 kB
Active:        9342880 kB
Inactive:      2061220 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      21220940 kB
SwapTotal:    67111528 kB
SwapFree:     67111348 kB
Dirty:           34288 kB
Writeback:           0 kB
AnonPages:     7991636 kB
Mapped:          38880 kB
Slab:           206420 kB
PageTables:      53156 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 101544312 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73316 MiB
End job on node139 at 2011-05-15 15:15:35