Trace number 3378160

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? 3516.08 1119.41

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v25000-c105000-S5267991-057-UNKNOWN.cnf
MD5SUM1a8d3d5d5a6ad4a4b0bb6bd83ad87f53
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark312.282
Satisfiable
(Un)Satisfiability was proved
Number of variables25000
Number of clauses105000
Sum of the clauses size315000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3105000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.31/0.45	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.31/0.45	c INFO: solving HOME/instance-3378160-1305463270.cnf
0.31/0.45	c INFO: computing task features
0.70/0.81	c INFO: computed variable-clause graph statistics
0.70/0.83	c INFO: computed clause balance statistics
2.60/2.72	c INFO: computed variable balance statistics
4.10/4.27	c INFO: computed small-clause counts
4.10/4.27	c INFO: computed Horn-clause counts
5.00/5.11	c INFO: computed variable graph statistics
5.00/5.11	c INFO: collected features for HOME/instance-3378160-1305463270.cnf in 4.65 s
5.00/5.12	c INFO: running hybridGM3@0 for 97 with 1195 remaining (b = 0.40)
5.00/5.13	c INFO: running hybridGM3@0 for 97 with 1195 remaining (b = 0.24)
5.00/5.14	c INFO: running hybridGM3@0 for 97 with 1195 remaining (b = 0.17)
5.00/5.15	c INFO: running hybridGM3@0 for 194 with 1195 remaining (b = 0.23)
342.11/118.06	c INFO: running hybridGM3@0 for 97 with 1082 remaining (b = 0.10)
466.10/159.59	c INFO: running hybridGM3@0 for 97 with 1040 remaining (b = 0.10)
467.88/160.15	c INFO: running hybridGM3@0 for 97 with 1040 remaining (b = 0.10)
690.53/226.06	c INFO: running hybridGM3@97 for 97 with 974 remaining (b = 0.10)
730.03/237.39	c INFO: running hybridGM3@97 for 97 with 963 remaining (b = 0.10)
898.09/285.50	c INFO: running TNM@0 for 97 with 914 remaining (b = 0.10)
914.18/290.95	c INFO: running hybridGM3@100 for 97 with 909 remaining (b = 0.10)
1023.03/327.35	c INFO: running hybridGM3@0 for 194 with 873 remaining (b = 0.17)
1053.84/337.68	c INFO: running hybridGM3@0 for 194 with 862 remaining (b = 0.13)
1305.80/409.91	c INFO: running hybridGM3@0 for 194 with 790 remaining (b = 0.10)
1306.71/410.30	c INFO: running hybridGM3@0 for 194 with 790 remaining (b = 0.07)
1765.26/542.15	c INFO: running hybridGM3@97 for 97 with 658 remaining (b = 0.03)
1819.12/557.71	c INFO: running hybridGM3@102 for 97 with 642 remaining (b = 0.03)
2077.44/643.46	c INFO: running hybridGM3@292 for 97 with 556 remaining (b = 0.03)
2112.85/654.49	c INFO: running hybridGM3@97 for 97 with 545 remaining (b = 0.03)
2178.71/676.37	c INFO: running TNM@97 for 524 with 524 remaining (b = 0.13)
2231.73/694.16	c INFO: running TNM@0 for 97 with 506 remaining (b = 0.05)
2504.67/785.89	c INFO: running hybridGM3@292 for 97 with 414 remaining (b = 0.03)
2615.12/822.98	c INFO: running hybridGM3@0 for 194 with 377 remaining (b = 0.05)
2742.65/859.69	c INFO: running TNM@98 for 340 with 340 remaining (b = 0.07)
2821.83/886.22	c INFO: running gnovelty+2@0 for 97 with 314 remaining (b = 0.04)
3294.18/1044.94	c INFO: running TNM@0 for 155 with 155 remaining (b = 0.02)
3515.72/1119.38	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378160-1305463270/watcher-3378160-1305463270 -o /tmp/evaluation-result-3378160-1305463270/solver-3378160-1305463270 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378160-1305463270.cnf -seed 1129826996 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.68 7.80 5.90 6/413 28642
/proc/meminfo: memFree=16966304/32951124 swapFree=67082152/67111528
[pid=28642] ppid=28640 vsize=16244 CPUtime=0 cores=1,3,5,7
/proc/28642/stat : 28642 (python) R 28640 28642 28149 0 -1 4202496 1037 0 0 0 0 0 0 0 25 0 1 0 657925018 16633856 910 33554432000 4194304 5683644 140736507491872 18446744073709551615 4540219 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/28642/statm: 4061 910 380 364 0 548 0

[startup+0.0691711 s]
/proc/loadavg: 8.68 7.80 5.90 6/413 28642
/proc/meminfo: memFree=16966304/32951124 swapFree=67082152/67111528
[pid=28642] ppid=28640 vsize=54540 CPUtime=0.04 cores=1,3,5,7
/proc/28642/stat : 28642 (python) S 28640 28642 28149 0 -1 4202496 2078 0 0 0 3 1 0 0 25 0 1 0 657925018 55848960 1835 33554432000 4194304 5683644 140736507491872 18446744073709551615 228244346880 0 2147483391 16781312 2 18446744071699199166 0 0 17 5 0 0 0
/proc/28642/statm: 13635 1835 492 364 0 1285 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 54540

[startup+0.100169 s]
/proc/loadavg: 8.68 7.80 5.90 6/413 28642
/proc/meminfo: memFree=16966304/32951124 swapFree=67082152/67111528
[pid=28642] ppid=28640 vsize=104620 CPUtime=0.07 cores=1,3,5,7
/proc/28642/stat : 28642 (python) R 28640 28642 28149 0 -1 4202496 3142 0 0 0 5 2 0 0 17 0 1 0 657925018 107130880 2822 33554432000 4194304 5683644 140736507491872 18446744073709551615 5375712 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/28642/statm: 26155 2822 847 364 0 1903 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 104620

[startup+0.30013 s]
/proc/loadavg: 8.68 7.80 5.90 6/413 28642
/proc/meminfo: memFree=16966304/32951124 swapFree=67082152/67111528
[pid=28642] ppid=28640 vsize=198920 CPUtime=0.23 cores=1,3,5,7
/proc/28642/stat : 28642 (python) R 28640 28642 28149 0 -1 4202496 7020 938 1 0 16 7 0 0 17 0 1 0 657925018 203694080 5053 33554432000 4194304 5683644 140736507491872 18446744073709551615 47261353059271 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/28642/statm: 49730 5053 1466 364 0 3757 0
Current children cumulated CPU time (s) 0.23
Current children cumulated vsize (KiB) 198920

[startup+0.70106 s]
/proc/loadavg: 8.68 7.80 5.90 6/413 28642
/proc/meminfo: memFree=16966304/32951124 swapFree=67082152/67111528
[pid=28642] ppid=28640 vsize=295936 CPUtime=0.61 cores=1,3,5,7
/proc/28642/stat : 28642 (python) R 28640 28642 28149 0 -1 4202496 13525 938 1 0 52 9 0 0 17 0 1 0 657925018 303038464 11205 33554432000 4194304 5683644 140736507491872 18446744073709551615 228244009669 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/28642/statm: 73984 11205 1761 364 0 9908 0
Current children cumulated CPU time (s) 0.61
Current children cumulated vsize (KiB) 295936

[startup+1.50092 s]
/proc/loadavg: 8.68 7.80 5.90 7/414 28647
/proc/meminfo: memFree=16919796/32951124 swapFree=67082152/67111528
[pid=28642] ppid=28640 vsize=302960 CPUtime=1.4 cores=1,3,5,7
/proc/28642/stat : 28642 (python) R 28640 28642 28149 0 -1 4202496 18311 938 1 0 131 9 0 0 18 0 1 0 657925018 310231040 13124 33554432000 4194304 5683644 140736507491872 18446744073709551615 228244005044 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/28642/statm: 75740 13124 1826 364 0 11664 0
Current children cumulated CPU time (s) 1.4
Current children cumulated vsize (KiB) 302960

[startup+3.10064 s]
/proc/loadavg: 8.68 7.80 5.90 7/414 28647
/proc/meminfo: memFree=16917992/32951124 swapFree=67082152/67111528
[pid=28642] ppid=28640 vsize=302960 CPUtime=3 cores=1,3,5,7
/proc/28642/stat : 28642 (python) R 28640 28642 28149 0 -1 4202496 18312 938 1 0 291 9 0 0 22 0 1 0 657925018 310231040 13125 33554432000 4194304 5683644 140736507491872 18446744073709551615 4623672 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/28642/statm: 75740 13125 1827 364 0 11664 0
Current children cumulated CPU time (s) 3
Current children cumulated vsize (KiB) 302960

[startup+6.30105 s]
/proc/loadavg: 8.46 7.77 5.90 11/428 28661
/proc/meminfo: memFree=16860440/32951124 swapFree=67082152/67111528
[pid=28642] ppid=28640 vsize=342044 CPUtime=5.06 cores=1,3,5,7
/proc/28642/stat : 28642 (python) S 28640 28642 28149 0 -1 4202496 21135 938 1 0 496 10 0 0 25 0 5 0 657925018 350253056 12699 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256175563 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/28642/statm: 85511 12699 1873 364 0 21420 0
[pid=28650] ppid=28642 vsize=301032 CPUtime=0 cores=1,3,5,7
/proc/28650/stat : 28650 (python) S 28642 28642 28149 0 -1 4202560 730 0 0 0 0 0 0 0 19 0 1 0 657925531 308256768 11163 33554432000 4194304 5683644 140736507491872 18446744073709551615 228244368911 0 0 16781312 514 3950770957655736319 0 0 17 1 0 0 0
/proc/28650/statm: 75258 11163 353 364 0 11176 0
[pid=28652] ppid=28650 vsize=268424 CPUtime=1.14 cores=1,3,5,7
/proc/28652/stat : 28652 (hybridGM3) R 28650 28652 28652 0 -1 4202496 7133 0 0 0 114 0 0 0 25 0 1 0 657925531 274866176 6869 33554432000 134512640 135204385 4294186928 18446744073709551615 134546144 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/28652/statm: 67106 6869 64 169 0 66936 0
[pid=28653] ppid=28642 vsize=311288 CPUtime=0 cores=1,3,5,7
/proc/28653/stat : 28653 (python) S 28642 28642 28149 0 -1 4202560 739 0 0 0 0 0 0 0 17 0 1 0 657925532 318758912 11161 33554432000 4194304 5683644 140736507491872 18446744073709551615 228244368966 0 0 16781312 514 16304723903282905349 0 0 17 5 0 0 0
/proc/28653/statm: 77822 11161 347 364 0 13737 0
[pid=28655] ppid=28653 vsize=268428 CPUtime=0.63 cores=1,3,5,7
/proc/28655/stat : 28655 (hybridGM3) R 28653 28655 28655 0 -1 4202496 7024 0 0 0 62 1 0 0 25 0 1 0 657925532 274870272 6760 33554432000 134512640 135204385 4291955328 18446744073709551615 134536993 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/28655/statm: 67107 6760 64 169 0 66937 0
[pid=28656] ppid=28642 vsize=321544 CPUtime=0 cores=1,3,5,7
/proc/28656/stat : 28656 (python) S 28642 28642 28149 0 -1 4202560 741 0 0 0 0 0 0 0 20 0 1 0 657925533 329261056 11165 33554432000 4194304 5683644 140736507491872 18446744073709551615 228244368966 0 0 16781312 514 8685257031742914559 0 0 17 1 0 0 0
/proc/28656/statm: 80386 11165 347 364 0 16298 0
[pid=28660] ppid=28656 vsize=268424 CPUtime=1.12 cores=1,3,5,7
/proc/28660/stat : 28660 (hybridGM3) R 28656 28660 28660 0 -1 4202496 7128 0 0 0 112 0 0 0 25 0 1 0 657925534 274866176 6867 33554432000 134512640 135204385 4294561488 18446744073709551615 134547471 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/28660/statm: 67106 6867 64 169 0 66936 0
[pid=28658] ppid=28642 vsize=331800 CPUtime=0 cores=1,3,5,7
/proc/28658/stat : 28658 (python) S 28642 28642 28149 0 -1 4202560 745 0 0 0 0 0 0 0 18 0 1 0 657925534 339763200 11169 33554432000 4194304 5683644 140736507491872 18446744073709551615 228244368966 0 0 16781312 514 16781529594569162751 0 0 17 1 0 0 0
/proc/28658/statm: 82950 11169 347 364 0 18859 0
[pid=28661] ppid=28658 vsize=268420 CPUtime=0.51 cores=1,3,5,7
/proc/28661/stat : 28661 (hybridGM3) R 28658 28661 28661 0 -1 4202496 3689 0 0 0 51 0 0 0 25 0 1 0 657925534 274862080 3429 33554432000 134512640 135204385 4291605952 18446744073709551615 134543968 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/28661/statm: 67105 3429 57 169 0 66935 0
[pid=28642/tid=28651] ppid=28640 vsize=342044 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28651/stat : 28651 (python) S 28640 28642 28149 0 -1 4202560 24 938 0 0 0 0 0 0 25 0 5 0 657925531 350253056 12699 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 1 0 0 0
[pid=28642/tid=28654] ppid=28640 vsize=342044 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28654/stat : 28654 (python) S 28640 28642 28149 0 -1 4202560 28 938 0 0 0 0 0 0 25 0 5 0 657925532 350253056 12699 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 5 0 0 0
[pid=28642/tid=28657] ppid=28640 vsize=342044 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28657/stat : 28657 (python) S 28640 28642 28149 0 -1 4202560 36 938 0 0 0 0 0 0 25 0 5 0 657925533 350253056 12699 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 1 0 0 0
[pid=28642/tid=28659] ppid=28640 vsize=342044 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28659/stat : 28659 (python) S 28640 28642 28149 0 -1 4202560 21 938 0 0 0 0 0 0 25 0 5 0 657925534 350253056 12699 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 8.46
Current children cumulated vsize (KiB) 2681404

[startup+12.7009 s]
/proc/loadavg: 8.51 7.79 5.92 10/428 28661
/proc/meminfo: memFree=16789880/32951124 swapFree=67082152/67111528
[pid=28642] ppid=28640 vsize=342044 CPUtime=5.06 cores=1,3,5,7
/proc/28642/stat : 28642 (python) S 28640 28642 28149 0 -1 4202496 21135 938 1 0 496 10 0 0 25 0 5 0 657925018 350253056 12699 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256175563 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/28642/statm: 85511 12699 1873 364 0 21420 0
[pid=28650] ppid=28642 vsize=301032 CPUtime=0 cores=1,3,5,7
/proc/28650/stat : 28650 (python) S 28642 28642 28149 0 -1 4202560 893 0 0 0 0 0 0 0 18 0 1 0 657925531 308256768 11186 33554432000 4194304 5683644 140736507491872 18446744073709551615 228244368911 0 0 16781312 514 3950770957655703814 0 0 17 1 0 0 0
/proc/28650/statm: 75258 11186 376 364 0 11176 0
[pid=28652] ppid=28650 vsize=268424 CPUtime=5.86 cores=1,3,5,7
/proc/28652/stat : 28652 (hybridGM3) R 28650 28652 28652 0 -1 4202496 7133 0 0 0 586 0 0 0 25 0 1 0 657925531 274866176 6869 33554432000 134512640 135204385 4294186928 18446744073709551615 134537403 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/28652/statm: 67106 6869 64 169 0 66936 0
[pid=28653] ppid=28642 vsize=311288 CPUtime=0.01 cores=1,3,5,7
/proc/28653/stat : 28653 (python) S 28642 28642 28149 0 -1 4202560 912 0 0 0 0 1 0 0 15 0 1 0 657925532 318758912 11184 33554432000 4194304 5683644 140736507491872 18446744073709551615 228244368966 0 0 16781312 514 8685245486870790405 0 0 17 5 0 0 0
/proc/28653/statm: 77822 11184 370 364 0 13737 0
[pid=28655] ppid=28653 vsize=268428 CPUtime=5.62 cores=1,3,5,7
/proc/28655/stat : 28655 (hybridGM3) R 28653 28655 28655 0 -1 4202496 7134 0 0 0 561 1 0 0 25 0 1 0 657925532 274870272 6870 33554432000 134512640 135204385 4291955328 18446744073709551615 134537403 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/28655/statm: 67107 6870 64 169 0 66937 0
[pid=28656] ppid=28642 vsize=321544 CPUtime=0.01 cores=1,3,5,7
/proc/28656/stat : 28656 (python) S 28642 28642 28149 0 -1 4202560 908 0 0 0 1 0 0 0 18 0 1 0 657925533 329261056 11188 33554432000 4194304 5683644 140736507491872 18446744073709551615 228244368966 0 0 16781312 514 8685257031742882055 0 0 17 1 0 0 0
/proc/28656/statm: 80386 11188 370 364 0 16298 0
[pid=28660] ppid=28656 vsize=268424 CPUtime=5.79 cores=1,3,5,7
/proc/28660/stat : 28660 (hybridGM3) R 28656 28660 28660 0 -1 4202496 7129 0 0 0 579 0 0 0 25 0 1 0 657925534 274866176 6868 33554432000 134512640 135204385 4294561488 18446744073709551615 134545866 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/28660/statm: 67106 6868 64 169 0 66936 0
[pid=28658] ppid=28642 vsize=331800 CPUtime=0.01 cores=1,3,5,7
/proc/28658/stat : 28658 (python) S 28642 28642 28149 0 -1 4202560 915 0 0 0 0 1 0 0 15 0 1 0 657925534 339763200 11192 33554432000 4194304 5683644 140736507491872 18446744073709551615 228244368966 0 0 16781312 514 16781529594569130248 0 0 17 1 0 0 0
/proc/28658/statm: 82950 11192 370 364 0 18859 0
[pid=28661] ppid=28658 vsize=268424 CPUtime=5.19 cores=1,3,5,7
/proc/28661/stat : 28661 (hybridGM3) R 28658 28661 28661 0 -1 4202496 7132 0 0 0 519 0 0 0 25 0 1 0 657925534 274866176 6872 33554432000 134512640 135204385 4291605952 18446744073709551615 134545968 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/28661/statm: 67106 6872 64 169 0 66936 0
[pid=28642/tid=28651] ppid=28640 vsize=342044 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28651/stat : 28651 (python) S 28640 28642 28149 0 -1 4202560 24 938 0 0 0 0 0 0 25 0 5 0 657925531 350253056 12699 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 1 0 0 0
[pid=28642/tid=28654] ppid=28640 vsize=342044 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28654/stat : 28654 (python) S 28640 28642 28149 0 -1 4202560 28 938 0 0 0 0 0 0 25 0 5 0 657925532 350253056 12699 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 5 0 0 0
[pid=28642/tid=28657] ppid=28640 vsize=342044 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28657/stat : 28657 (python) S 28640 28642 28149 0 -1 4202560 36 938 0 0 0 0 0 0 25 0 5 0 657925533 350253056 12699 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 1 0 0 0
[pid=28642/tid=28659] ppid=28640 vsize=342044 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28659/stat : 28659 (python) S 28640 28642 28149 0 -1 4202560 21 938 0 0 0 0 0 0 25 0 5 0 657925534 350253056 12699 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 27.55
Current children cumulated vsize (KiB) 2681408

[startup+25.5007 s]
/proc/loadavg: 8.69 7.87 5.97 10/428 28662
/proc/meminfo: memFree=16790892/32951124 swapFree=67082152/67111528
[pid=28642] ppid=28640 vsize=342044 CPUtime=5.06 cores=1,3,5,7
/proc/28642/stat : 28642 (python) S 28640 28642 28149 0 -1 4202496 21135 938 1 0 496 10 0 0 25 0 5 0 657925018 350253056 12699 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256175563 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/28642/statm: 85511 12699 1873 364 0 21420 0
[pid=28650] ppid=28642 vsize=301032 CPUtime=0.04 cores=1,3,5,7
/proc/28650/stat : 28650 (python) S 28642 28642 28149 0 -1 4202560 899 0 0 0 2 2 0 0 18 0 1 0 657925531 308256768 11186 33554432000 4194304 5683644 140736507491872 18446744073709551615 228244368911 0 0 16781312 514 16220435337602629894 0 0 17 1 0 0 0
/proc/28650/statm: 75258 11186 376 364 0 11176 0
[pid=28652] ppid=28650 vsize=268424 CPUtime=12.19 cores=1,3,5,7
/proc/28652/stat : 28652 (hybridGM3) R 28650 28652 28652 0 -1 4202496 7135 0 0 0 1219 0 0 0 25 0 1 0 657925531 274866176 6871 33554432000 134512640 135204385 4294186928 18446744073709551615 134537403 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/28652/statm: 67106 6871 64 169 0 66936 0
[pid=28653] ppid=28642 vsize=311288 CPUtime=0.04 cores=1,3,5,7
/proc/28653/stat : 28653 (python) S 28642 28642 28149 0 -1 4202560 918 0 0 0 2 2 0 0 15 0 1 0 657925532 318758912 11184 33554432000 4194304 5683644 140736507491872 18446744073709551615 228244368966 0 0 16781312 514 16220435337602629893 0 0 17 5 0 0 0
/proc/28653/statm: 77822 11184 370 364 0 13737 0
[pid=28655] ppid=28653 vsize=268428 CPUtime=18.38 cores=1,3,5,7
/proc/28655/stat : 28655 (hybridGM3) R 28653 28655 28655 0 -1 4202496 7134 0 0 0 1837 1 0 0 25 0 1 0 657925532 274870272 6870 33554432000 134512640 135204385 4291955328 18446744073709551615 134547403 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/28655/statm: 67107 6870 64 169 0 66937 0
[pid=28656] ppid=28642 vsize=321544 CPUtime=0.04 cores=1,3,5,7
/proc/28656/stat : 28656 (python) S 28642 28642 28149 0 -1 4202560 914 0 0 0 2 2 0 0 18 0 1 0 657925533 329261056 11188 33554432000 4194304 5683644 140736507491872 18446744073709551615 228244368966 0 0 16781312 514 16220435337602629895 0 0 17 1 0 0 0
/proc/28656/statm: 80386 11188 370 364 0 16298 0
[pid=28660] ppid=28656 vsize=268424 CPUtime=12.09 cores=1,3,5,7
/proc/28660/stat : 28660 (hybridGM3) R 28656 28660 28660 0 -1 4202496 7129 0 0 0 1209 0 0 0 25 0 1 0 657925534 274866176 6868 33554432000 134512640 135204385 4294561488 18446744073709551615 134537471 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/28660/statm: 67106 6868 64 169 0 66936 0
[pid=28658] ppid=28642 vsize=331800 CPUtime=0.04 cores=1,3,5,7
/proc/28658/stat : 28658 (python) S 28642 28642 28149 0 -1 4202560 919 0 0 0 2 2 0 0 15 0 1 0 657925534 339763200 11192 33554432000 4194304 5683644 140736507491872 18446744073709551615 228244368966 0 0 16781312 514 16220435337602629896 0 0 17 1 0 0 0
/proc/28658/statm: 82950 11192 370 364 0 18859 0
[pid=28661] ppid=28658 vsize=268424 CPUtime=17.89 cores=1,3,5,7
/proc/28661/stat : 28661 (hybridGM3) R 28658 28661 28661 0 -1 4202496 7133 0 0 0 1789 0 0 0 25 0 1 0 657925534 274866176 6873 33554432000 134512640 135204385 4291605952 18446744073709551615 134536986 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/28661/statm: 67106 6873 64 169 0 66936 0
[pid=28642/tid=28651] ppid=28640 vsize=342044 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28651/stat : 28651 (python) S 28640 28642 28149 0 -1 4202560 24 938 0 0 0 0 0 0 25 0 5 0 657925531 350253056 12699 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 1 0 0 0
[pid=28642/tid=28654] ppid=28640 vsize=342044 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28654/stat : 28654 (python) S 28640 28642 28149 0 -1 4202560 28 938 0 0 0 0 0 0 25 0 5 0 657925532 350253056 12699 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 5 0 0 0
[pid=28642/tid=28657] ppid=28640 vsize=342044 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28657/stat : 28657 (python) S 28640 28642 28149 0 -1 4202560 36 938 0 0 0 0 0 0 25 0 5 0 657925533 350253056 12699 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 1 0 0 0

################
# More data... #
################

[pid=28658/tid=28740] ppid=28642 vsize=331800 CPUtime=0 cores=1,3,5,7
/proc/28658/task/28740/stat : 28740 (python) S 28642 28642 28149 0 -1 4202560 45 0 0 0 0 0 0 0 15 0 2 0 657947625 339763200 11222 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 514 18446604469704849472 0 0 -1 1 0 0 0
[pid=28743] ppid=28642 vsize=342056 CPUtime=1.77 cores=1,3,5,7
/proc/28743/stat : 28743 (python) S 28642 28642 28149 0 -1 4202560 1153 0 0 0 90 87 0 0 18 0 2 0 657953569 350265344 11226 33554432000 4194304 5683644 140736507491872 18446744073709551615 228244368966 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/28743/statm: 85514 11226 399 364 0 21420 0
[pid=28745] ppid=28743 vsize=460952 CPUtime=491.73 cores=1,3,5,7
/proc/28745/stat : 28745 (TNM) R 28743 28745 28745 0 -1 4202496 4393 0 0 0 49173 0 0 0 25 0 1 0 657953569 472014848 4133 33554432000 134512640 135008748 4287763136 18446744073709551615 134524853 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/28745/statm: 115238 4133 46 122 0 114138 0
[pid=28743/tid=28783] ppid=28642 vsize=342056 CPUtime=0 cores=1,3,5,7
/proc/28743/task/28783/stat : 28783 (python) S 28642 28642 28149 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 657966008 350265344 11226 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 514 18446604463034048576 0 0 -1 1 0 0 0
[pid=28763] ppid=28642 vsize=352312 CPUtime=1.51 cores=1,3,5,7
/proc/28763/stat : 28763 (python) S 28642 28642 28149 0 -1 4202560 1166 0 0 0 81 70 0 0 16 0 2 0 657957754 360767488 11230 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/28763/statm: 88078 11230 399 364 0 23981 0
[pid=28765] ppid=28763 vsize=268424 CPUtime=390.98 cores=1,3,5,7
/proc/28765/stat : 28765 (hybridGM3) T 28763 28765 28765 0 -1 4202496 7139 0 0 0 39097 1 0 0 25 0 1 0 657957754 274866176 6878 33554432000 134512640 135204385 4287545696 18446744073709551615 134547728 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/28765/statm: 67106 6878 64 169 0 66936 0
[pid=28763/tid=28798] ppid=28642 vsize=352312 CPUtime=0 cores=1,3,5,7
/proc/28763/task/28798/stat : 28798 (python) S 28642 28642 28149 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 657979233 360767488 11230 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 514 18446604458852391104 0 0 -1 1 0 0 0
[pid=28766] ppid=28642 vsize=362568 CPUtime=1.11 cores=1,3,5,7
/proc/28766/stat : 28766 (python) S 28642 28642 28149 0 -1 4202560 1156 0 0 0 60 51 0 0 15 0 2 0 657958787 371269632 11234 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/28766/statm: 90642 11234 399 364 0 26542 0
[pid=28768] ppid=28766 vsize=268424 CPUtime=291.9 cores=1,3,5,7
/proc/28768/stat : 28768 (hybridGM3) T 28766 28768 28768 0 -1 4202496 7134 0 0 0 29189 1 0 0 25 0 1 0 657958787 274866176 6873 33554432000 134512640 135204385 4293682384 18446744073709551615 134536663 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/28768/statm: 67106 6873 64 169 0 66936 0
[pid=28766/tid=28800] ppid=28642 vsize=362568 CPUtime=0 cores=1,3,5,7
/proc/28766/task/28800/stat : 28800 (python) S 28642 28642 28149 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 657980789 371269632 11234 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 514 18446604463858255168 0 0 -1 7 0 0 0
[pid=28784] ppid=28642 vsize=372824 CPUtime=1.49 cores=1,3,5,7
/proc/28784/stat : 28784 (python) S 28642 28642 28149 0 -1 4202560 1191 0 0 0 85 64 0 0 15 0 2 0 657966010 381771776 11244 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/28784/statm: 93206 11244 405 364 0 29103 0
[pid=28786] ppid=28784 vsize=268424 CPUtime=389.76 cores=1,3,5,7
/proc/28786/stat : 28786 (hybridGM3) T 28784 28786 28786 0 -1 4202496 7144 0 0 0 38975 1 0 0 25 0 1 0 657966010 274866176 6875 33554432000 134512640 135204385 4292308032 18446744073709551615 134546073 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/28786/statm: 67106 6875 64 169 0 66936 0
[pid=28784/tid=28809] ppid=28642 vsize=372824 CPUtime=0 cores=1,3,5,7
/proc/28784/task/28809/stat : 28809 (python) S 28642 28642 28149 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 657990467 381771776 11244 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 514 18446604461039284352 0 0 -1 7 0 0 0
[pid=28787] ppid=28642 vsize=383080 CPUtime=0.81 cores=1,3,5,7
/proc/28787/stat : 28787 (python) S 28642 28642 28149 0 -1 4202560 1183 0 0 0 41 40 0 0 17 0 2 0 657966049 392273920 11248 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/28787/statm: 95770 11248 405 364 0 31664 0
[pid=28789] ppid=28787 vsize=280980 CPUtime=195.74 cores=1,3,5,7
/proc/28789/stat : 28789 (hybridGM3) T 28787 28789 28789 0 -1 4202496 68713 0 0 0 19565 9 0 0 25 0 1 0 657966049 287723520 10524 33554432000 134512640 135204385 4292646080 18446744073709551615 134536646 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/28789/statm: 70245 10524 85 169 0 70075 0
[pid=28787/tid=28811] ppid=28642 vsize=383080 CPUtime=0 cores=1,3,5,7
/proc/28787/task/28811/stat : 28811 (python) S 28642 28642 28149 0 -1 4202560 48 0 0 0 0 0 0 0 17 0 2 0 657992656 392273920 11248 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 514 18446604461039286144 0 0 -1 7 0 0 0
[pid=28813] ppid=28642 vsize=393336 CPUtime=1.34 cores=1,3,5,7
/proc/28813/stat : 28813 (python) S 28642 28642 28149 0 -1 4202560 1166 0 0 0 71 63 0 0 18 0 2 0 657994434 402776064 11246 33554432000 4194304 5683644 140736507491872 18446744073709551615 228244368966 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/28813/statm: 98334 11246 399 364 0 34225 0
[pid=28815] ppid=28813 vsize=460952 CPUtime=323.46 cores=1,3,5,7
/proc/28815/stat : 28815 (TNM) R 28813 28815 28815 0 -1 4202496 4400 0 0 0 32346 0 0 0 25 0 1 0 657994435 472014848 4133 33554432000 134512640 135008748 4294328960 18446744073709551615 134527327 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/28815/statm: 115238 4133 46 122 0 114138 0
[pid=28813/tid=28888] ppid=28642 vsize=393336 CPUtime=0 cores=1,3,5,7
/proc/28813/task/28888/stat : 28888 (python) S 28642 28642 28149 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 658010987 402776064 11246 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 514 18446604470124787712 0 0 -1 1 0 0 0
[pid=28884] ppid=28642 vsize=403592 CPUtime=0.92 cores=1,3,5,7
/proc/28884/stat : 28884 (python) S 28642 28642 28149 0 -1 4202560 1014 0 0 0 45 47 0 0 15 0 1 0 658007317 413278208 11220 33554432000 4194304 5683644 140736507491872 18446744073709551615 228244368966 0 0 16781312 514 16220435337602629893 0 0 17 7 0 0 0
/proc/28884/statm: 100898 11220 370 364 0 36786 0
[pid=28886] ppid=28884 vsize=268428 CPUtime=196.47 cores=1,3,5,7
/proc/28886/stat : 28886 (hybridGM3) R 28884 28886 28886 0 -1 4202496 7141 0 0 0 19646 1 0 0 25 0 1 0 658007318 274870272 6871 33554432000 134512640 135204385 4289042832 18446744073709551615 134537401 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/28886/statm: 67107 6871 64 169 0 66937 0
[pid=28890] ppid=28642 vsize=413848 CPUtime=0.51 cores=1,3,5,7
/proc/28890/stat : 28890 (python) S 28642 28642 28149 0 -1 4202560 1147 0 0 0 25 26 0 0 19 0 2 0 658013641 423780352 11254 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/28890/statm: 103462 11254 399 364 0 39347 0
[pid=28892] ppid=28890 vsize=1025020 CPUtime=98.03 cores=1,3,5,7
/proc/28892/stat : 28892 (gnovelty+2) T 28890 28892 28892 0 -1 4202496 2622 0 0 0 9803 0 0 0 25 0 1 0 658013642 1049620480 2319 33554432000 134512640 135663704 4290251376 18446744073709551615 134545631 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/28892/statm: 256255 2319 113 282 0 255971 0
[pid=28890/tid=28925] ppid=28642 vsize=413848 CPUtime=0 cores=1,3,5,7
/proc/28890/task/28925/stat : 28925 (python) S 28642 28642 28149 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 658029512 423780352 11254 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 514 18446604465220815616 0 0 -1 1 0 0 0
[pid=28926] ppid=28642 vsize=424104 CPUtime=0.23 cores=1,3,5,7
/proc/28926/stat : 28926 (python) S 28642 28642 28149 0 -1 4202560 962 0 0 0 12 11 0 0 18 0 1 0 658029513 434282496 11228 33554432000 4194304 5683644 140736507491872 18446744073709551615 228244368966 0 0 16781312 514 16220435337602629893 0 0 17 7 0 0 0
/proc/28926/statm: 106026 11228 370 364 0 41908 0
[pid=28928] ppid=28926 vsize=460952 CPUtime=37.06 cores=1,3,5,7
/proc/28928/stat : 28928 (TNM) R 28926 28928 28928 0 -1 4202496 4404 0 0 0 3706 0 0 0 25 0 1 0 658029514 472014848 4134 33554432000 134512640 135008748 4290016208 18446744073709551615 134529475 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/28928/statm: 115238 4134 46 122 0 114138 0
[pid=28642/tid=28651] ppid=28640 vsize=434348 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28651/stat : 28651 (python) S 28640 28642 28149 0 -1 4202560 28 938 0 0 0 0 0 0 18 0 14 0 657925531 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 1 0 0 0
[pid=28642/tid=28654] ppid=28640 vsize=434348 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28654/stat : 28654 (python) S 28640 28642 28149 0 -1 4202560 37 938 0 0 0 0 0 0 18 0 14 0 657925532 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 5 0 0 0
[pid=28642/tid=28657] ppid=28640 vsize=434348 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28657/stat : 28657 (python) S 28640 28642 28149 0 -1 4202560 36 938 0 0 0 0 0 0 18 0 14 0 657925533 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 1 0 0 0
[pid=28642/tid=28659] ppid=28640 vsize=434348 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28659/stat : 28659 (python) S 28640 28642 28149 0 -1 4202560 22 938 0 0 0 0 0 0 18 0 14 0 657925534 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 1 0 0 0
[pid=28642/tid=28744] ppid=28640 vsize=434348 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28744/stat : 28744 (python) S 28640 28642 28149 0 -1 4202560 35 938 0 0 0 0 0 0 18 0 14 0 657953569 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 7 0 0 0
[pid=28642/tid=28764] ppid=28640 vsize=434348 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28764/stat : 28764 (python) S 28640 28642 28149 0 -1 4202560 35 938 0 0 0 0 0 0 18 0 14 0 657957754 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 1 0 0 0
[pid=28642/tid=28767] ppid=28640 vsize=434348 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28767/stat : 28767 (python) S 28640 28642 28149 0 -1 4202560 36 938 0 0 0 0 0 0 18 0 14 0 657958787 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 7 0 0 0
[pid=28642/tid=28785] ppid=28640 vsize=434348 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28785/stat : 28785 (python) S 28640 28642 28149 0 -1 4202560 33 938 0 0 0 0 0 0 18 0 14 0 657966010 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 7 0 0 0
[pid=28642/tid=28788] ppid=28640 vsize=434348 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28788/stat : 28788 (python) S 28640 28642 28149 0 -1 4202560 22 938 0 0 0 0 0 0 20 0 14 0 657966049 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 7 0 0 0
[pid=28642/tid=28814] ppid=28640 vsize=434348 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28814/stat : 28814 (python) S 28640 28642 28149 0 -1 4202560 30 938 0 0 0 0 0 0 18 0 14 0 657994435 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 7 0 0 0
[pid=28642/tid=28885] ppid=28640 vsize=434348 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28885/stat : 28885 (python) S 28640 28642 28149 0 -1 4202560 27 938 0 0 0 0 0 0 20 0 14 0 658007318 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 7 0 0 0
[pid=28642/tid=28891] ppid=28640 vsize=434348 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28891/stat : 28891 (python) S 28640 28642 28149 0 -1 4202560 25 938 0 0 0 0 0 0 20 0 14 0 658013641 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 7 0 0 0
[pid=28642/tid=28927] ppid=28640 vsize=434348 CPUtime=0 cores=1,3,5,7
/proc/28642/task/28927/stat : 28927 (python) S 28640 28642 28149 0 -1 4202560 26 938 0 0 0 0 0 0 20 0 14 0 658029514 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3514.9
Current children cumulated vsize (KiB) 9994232

[startup+1119.3 s]
/proc/loadavg: 9.21 9.15 8.16 10/493 28935
/proc/meminfo: memFree=16331264/32951124 swapFree=67082152/67111528
[pid=28642] ppid=28640 vsize=434348 CPUtime=2462.16 cores=1,3,5,7
/proc/28642/stat : 28642 (python) S 28640 28642 28149 0 -1 4202496 30120 133056 1 0 514 14 245214 474 17 0 14 0 657925018 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256179007 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/28642/statm: 108587 12762 1900 364 0 44469 0
[pid=28743] ppid=28642 vsize=342056 CPUtime=1.77 cores=1,3,5,7
/proc/28743/stat : 28743 (python) S 28642 28642 28149 0 -1 4202560 1153 0 0 0 90 87 0 0 18 0 2 0 657953569 350265344 11226 33554432000 4194304 5683644 140736507491872 18446744073709551615 228244368966 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/28743/statm: 85514 11226 399 364 0 21420 0
[pid=28745] ppid=28743 vsize=460952 CPUtime=491.86 cores=1,3,5,7
/proc/28745/stat : 28745 (TNM) R 28743 28745 28745 0 -1 4202496 4393 0 0 0 49186 0 0 0 25 0 1 0 657953569 472014848 4133 33554432000 134512640 135008748 4287763136 18446744073709551615 134527524 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/28745/statm: 115238 4133 46 122 0 114138 0
[pid=28743/tid=28783] ppid=28642 vsize=342056 CPUtime=0 cores=1,3,5,7
/proc/28743/task/28783/stat : 28783 (python) S 28642 28642 28149 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 657966008 350265344 11226 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 514 18446604463034048576 0 0 -1 1 0 0 0
[pid=28813] ppid=28642 vsize=393336 CPUtime=1.34 cores=1,3,5,7
/proc/28813/stat : 28813 (python) S 28642 28642 28149 0 -1 4202560 1166 0 0 0 71 63 0 0 18 0 2 0 657994434 402776064 11246 33554432000 4194304 5683644 140736507491872 18446744073709551615 228244368966 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/28813/statm: 98334 11246 399 364 0 34225 0
[pid=28815] ppid=28813 vsize=460952 CPUtime=323.65 cores=1,3,5,7
/proc/28815/stat : 28815 (TNM) R 28813 28815 28815 0 -1 4202496 4400 0 0 0 32365 0 0 0 25 0 1 0 657994435 472014848 4133 33554432000 134512640 135008748 4294328960 18446744073709551615 134527066 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/28815/statm: 115238 4133 46 122 0 114138 0
[pid=28813/tid=28888] ppid=28642 vsize=393336 CPUtime=0 cores=1,3,5,7
/proc/28813/task/28888/stat : 28888 (python) S 28642 28642 28149 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 658010987 402776064 11246 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 514 18446604470124787712 0 0 -1 1 0 0 0
[pid=28884] ppid=28642 vsize=403592 CPUtime=0.95 cores=1,3,5,7
/proc/28884/stat : 28884 (python) S 28642 28642 28149 0 -1 4202560 1176 0 0 0 46 49 0 0 16 0 2 0 658007317 413278208 11250 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256179007 0 0 16781824 2 18446744073709551615 0 0 17 7 0 0 0
/proc/28884/statm: 100898 11250 399 364 0 36786 0
[pid=28886] ppid=28884 vsize=268428 CPUtime=196.61 cores=1,3,5,7
/proc/28886/stat : 28886 (hybridGM3) R 28884 28886 28886 0 -1 4202496 7141 0 0 0 19660 1 0 0 25 0 1 0 658007318 274870272 6871 33554432000 134512640 135204385 4289042832 18446744073709551615 134536744 256 0 16781312 0 0 0 0 17 7 0 0 0
/proc/28886/statm: 67107 6871 64 169 0 66937 0
[pid=28926] ppid=28642 vsize=424104 CPUtime=0.23 cores=1,3,5,7
/proc/28926/stat : 28926 (python) S 28642 28642 28149 0 -1 4202560 962 0 0 0 12 11 0 0 18 0 1 0 658029513 434282496 11228 33554432000 4194304 5683644 140736507491872 18446744073709551615 228244368966 0 0 16781312 514 16220435337602629893 0 0 17 7 0 0 0
/proc/28926/statm: 106026 11228 370 364 0 41908 0
[pid=28928] ppid=28926 vsize=460952 CPUtime=37.15 cores=1,3,5,7
/proc/28928/stat : 28928 (TNM) R 28926 28928 28928 0 -1 4202496 4404 0 0 0 3715 0 0 0 25 0 1 0 658029514 472014848 4134 33554432000 134512640 135008748 4290016208 18446744073709551615 134529448 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/28928/statm: 115238 4134 46 122 0 114138 0
[pid=28642/tid=28651] ppid=28640 vsize=434348 CPUtime=2456.88 cores=1,3,5,7
/proc/28642/task/28651/stat : 28651 (python) S 28640 28642 28149 0 -1 4202560 28 133056 0 0 0 0 245214 474 18 0 14 0 657925531 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 1 0 0 0
[pid=28642/tid=28654] ppid=28640 vsize=434348 CPUtime=2456.88 cores=1,3,5,7
/proc/28642/task/28654/stat : 28654 (python) S 28640 28642 28149 0 -1 4202560 37 133056 0 0 0 0 245214 474 18 0 14 0 657925532 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 5 0 0 0
[pid=28642/tid=28657] ppid=28640 vsize=434348 CPUtime=2456.88 cores=1,3,5,7
/proc/28642/task/28657/stat : 28657 (python) S 28640 28642 28149 0 -1 4202560 36 133056 0 0 0 0 245214 474 18 0 14 0 657925533 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 1 0 0 0
[pid=28642/tid=28659] ppid=28640 vsize=434348 CPUtime=2456.88 cores=1,3,5,7
/proc/28642/task/28659/stat : 28659 (python) S 28640 28642 28149 0 -1 4202560 22 133056 0 0 0 0 245214 474 18 0 14 0 657925534 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 1 0 0 0
[pid=28642/tid=28744] ppid=28640 vsize=434348 CPUtime=2456.88 cores=1,3,5,7
/proc/28642/task/28744/stat : 28744 (python) S 28640 28642 28149 0 -1 4202560 35 133056 0 0 0 0 245214 474 18 0 14 0 657953569 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 7 0 0 0
[pid=28642/tid=28764] ppid=28640 vsize=434348 CPUtime=2456.88 cores=1,3,5,7
/proc/28642/task/28764/stat : 28764 (python) S 28640 28642 28149 0 -1 4202560 35 133056 0 0 0 0 245214 474 18 0 14 0 657957754 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 1 0 0 0
[pid=28642/tid=28767] ppid=28640 vsize=434348 CPUtime=2456.88 cores=1,3,5,7
/proc/28642/task/28767/stat : 28767 (python) S 28640 28642 28149 0 -1 4202560 36 133056 0 0 0 0 245214 474 18 0 14 0 657958787 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 7 0 0 0
[pid=28642/tid=28785] ppid=28640 vsize=434348 CPUtime=2456.88 cores=1,3,5,7
/proc/28642/task/28785/stat : 28785 (python) S 28640 28642 28149 0 -1 4202560 33 133056 0 0 0 0 245214 474 18 0 14 0 657966010 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 7 0 0 0
[pid=28642/tid=28788] ppid=28640 vsize=434348 CPUtime=2456.88 cores=1,3,5,7
/proc/28642/task/28788/stat : 28788 (python) S 28640 28642 28149 0 -1 4202560 22 133056 0 0 0 0 245214 474 20 0 14 0 657966049 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 7 0 0 0
[pid=28642/tid=28814] ppid=28640 vsize=434348 CPUtime=2456.88 cores=1,3,5,7
/proc/28642/task/28814/stat : 28814 (python) S 28640 28642 28149 0 -1 4202560 30 133056 0 0 0 0 245214 474 18 0 14 0 657994435 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 7 0 0 0
[pid=28642/tid=28885] ppid=28640 vsize=434348 CPUtime=2456.88 cores=1,3,5,7
/proc/28642/task/28885/stat : 28885 (python) S 28640 28642 28149 0 -1 4202560 27 133056 0 0 0 0 245214 474 20 0 14 0 658007318 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 7 0 0 0
[pid=28642/tid=28891] ppid=28640 vsize=434348 CPUtime=2456.88 cores=1,3,5,7
/proc/28642/task/28891/stat : 28891 (python) S 28640 28642 28149 0 -1 4202560 25 133056 0 0 0 0 245214 474 20 0 14 0 658013641 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 7 0 0 0
[pid=28642/tid=28927] ppid=28640 vsize=434348 CPUtime=2456.88 cores=1,3,5,7
/proc/28642/task/28927/stat : 28927 (python) S 28640 28642 28149 0 -1 4202560 26 133056 0 0 0 0 245214 474 20 0 14 0 658029514 444772352 12762 33554432000 4194304 5683644 140736507491872 18446744073709551615 228256172465 0 0 16781312 2 18446604458127522880 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3515.72
Current children cumulated vsize (KiB) 3648720

[startup+1119.4 s]
/proc/loadavg: 9.21 9.15 8.16 10/493 28935
/proc/meminfo: memFree=16331264/32951124 swapFree=67082152/67111528
[pid=28642] ppid=28640 vsize=331740 CPUtime=3516.06 cores=1,3,5,7
/proc/28642/stat : 28642 (python) R 28640 28642 28149 0 -1 4202496 35781 157990 1 0 515 14 350385 692 17 0 1 0 657925018 339701760 12696 33554432000 4194304 5683644 140736507491872 18446744073709551615 5258767 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/28642/statm: 82935 12696 1874 364 0 18859 0
Current children cumulated CPU time (s) 3516.06
Current children cumulated vsize (KiB) 331740

Child status: 0
Real time (s): 1119.41
CPU time (s): 3516.08
CPU user time (s): 3509.01
CPU system time (s): 7.07192
CPU usage (%): 314.101
Max. virtual memory (cumulated for all children) (KiB): 9994236

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 3509.01
system time used= 7.07192
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 194763
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 6509
involuntary context switches= 42036

runsolver used 6.36703 second user time and 15.6126 second system time

The end

Launcher Data 

Begin job on node136 at 2011-05-15 14:41:10
IDJOB=3378160
IDBENCH=82569
IDSOLVER=1859
FILE ID=node136/3378160-1305463270
RUNJOBID= node136-1305462567-28168
PBS_JOBID= 13324968
Free space on /tmp= 73012 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v25000-c105000-S5267991-057-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378160-1305463270/watcher-3378160-1305463270 -o /tmp/evaluation-result-3378160-1305463270/solver-3378160-1305463270 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378160-1305463270.cnf -seed 1129826996 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 1a8d3d5d5a6ad4a4b0bb6bd83ad87f53
RANDOM SEED=1129826996

node136.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      16966460 kB
Buffers:        314580 kB
Cached:        5903088 kB
SwapCached:       2740 kB
Active:       11224672 kB
Inactive:      4398136 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      16966460 kB
SwapTotal:    67111528 kB
SwapFree:     67082152 kB
Dirty:           33732 kB
Writeback:           0 kB
AnonPages:     9403816 kB
Mapped:          35824 kB
Slab:           241336 kB
PageTables:      53260 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 87553868 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72964 MiB
End job on node136 at 2011-05-15 14:59:50