Trace number 3321807

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (MO) 7616.9 1077.01

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v50000-c210000-S1401248878-094-UNKNOWN.cnf
MD5SUM56bd5db172bed91bc72714a8bad8639b
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark346.339
Satisfiable
(Un)Satisfiability was proved
Number of variables50000
Number of clauses210000
Sum of the clauses size630000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3210000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.36/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.36/0.67	c INFO: solving HOME/instance-3321807-1304367709.cnf
0.36/0.67	c INFO: computing task features
1.14/1.40	c INFO: computed variable-clause graph statistics
1.14/1.43	c INFO: computed clause balance statistics
4.74/5.07	c INFO: computed variable balance statistics
7.74/8.05	c INFO: computed small-clause counts
7.74/8.07	c INFO: computed Horn-clause counts
10.84/11.15	c INFO: computed variable graph statistics
10.84/11.15	c INFO: collected features for HOME/instance-3321807-1304367709.cnf in 10.45 s
10.84/11.18	c INFO: running hybridGM3@0 for 97 with 39989 remaining (b = 0.35)
10.94/11.22	c INFO: running hybridGM3@0 for 97 with 39989 remaining (b = 0.21)
10.94/11.26	c INFO: running hybridGM3@0 for 97 with 39989 remaining (b = 0.14)
11.04/11.30	c INFO: running hybridGM3@0 for 194 with 39989 remaining (b = 0.18)
11.04/11.34	c INFO: running hybridGM3@0 for 194 with 39989 remaining (b = 0.13)
11.04/11.38	c INFO: running hybridGM3@0 for 194 with 39989 remaining (b = 0.10)
11.13/11.42	c INFO: running hybridGM3@0 for 194 with 39989 remaining (b = 0.07)
11.13/11.46	c INFO: running TNM@0 for 97 with 39989 remaining (b = 0.03)
760.99/111.55	c INFO: running hybridGM3@0 for 97 with 39889 remaining (b = 0.03)
761.74/111.63	c INFO: running hybridGM3@0 for 97 with 39889 remaining (b = 0.03)
986.17/141.66	c INFO: running hybridGM3@0 for 97 with 39859 remaining (b = 0.03)
1033.03/147.98	c INFO: running hybridGM3@0 for 194 with 39852 remaining (b = 0.05)
1408.52/201.77	c INFO: running hybridGM3@0 for 194 with 39798 remaining (b = 0.04)
1444.18/206.87	c INFO: running hybridGM3@98 for 97 with 39793 remaining (b = 0.02)
1451.88/207.99	c INFO: running hybridGM3@0 for 97 with 39792 remaining (b = 0.02)
1452.50/208.03	c INFO: running hybridGM3@0 for 97 with 39792 remaining (b = 0.02)
1493.66/213.95	c INFO: running hybridGM3@0 for 97 with 39786 remaining (b = 0.02)
1661.21/238.00	c INFO: running hybridGM3@0 for 97 with 39762 remaining (b = 0.02)
1738.02/248.93	c INFO: running hybridGM3@102 for 97 with 39751 remaining (b = 0.02)
2151.28/308.12	c INFO: running hybridGM3@295 for 97 with 39692 remaining (b = 0.02)
2158.94/309.25	c INFO: running hybridGM3@0 for 194 with 39691 remaining (b = 0.03)
2200.77/315.21	c INFO: running TNM@0 for 97 with 39685 remaining (b = 0.01)
2243.37/321.31	c INFO: running hybridGM3@0 for 194 with 39679 remaining (b = 0.03)
2296.45/328.94	c INFO: running TNM@100 for 1940 with 39671 remaining (b = 0.20)
2631.73/374.24	c INFO: running hybridGM3@0 for 194 with 39626 remaining (b = 0.02)
2764.22/391.92	c INFO: running hybridGM3@88 for 97 with 39608 remaining (b = 0.01)
2849.43/403.37	c INFO: running hybridGM3@395 for 97 with 39597 remaining (b = 0.01)
2850.92/403.59	c INFO: running hybridGM3@94 for 97 with 39597 remaining (b = 0.01)
3020.25/426.54	c INFO: running hybridGM3@0 for 194 with 39574 remaining (b = 0.02)
3346.98/471.99	c INFO: running adaptg2wsat2009++@0 for 194 with 39528 remaining (b = 0.02)
3575.39/504.65	c INFO: running hybridGM3@0 for 194 with 39496 remaining (b = 0.02)
3584.54/505.92	c INFO: running hybridGM3@95 for 97 with 39494 remaining (b = 0.01)
3618.65/510.88	c INFO: running TNM@0 for 97 with 39489 remaining (b = 0.01)
3872.39/547.22	c INFO: running hybridGM3@50 for 97 with 39453 remaining (b = 0.01)
4121.29/582.97	c INFO: running hybridGM3@0 for 194 with 39417 remaining (b = 0.02)
4248.92/601.20	c INFO: running hybridGM3@0 for 194 with 39399 remaining (b = 0.02)
4564.66/646.48	c INFO: running hybridGM3@65 for 97 with 39354 remaining (b = 0.01)
4612.04/653.27	c INFO: running TNM@0 for 97 with 39347 remaining (b = 0.01)
4739.54/671.58	c INFO: running hybridGM3@0 for 194 with 39329 remaining (b = 0.01)
4962.10/701.36	c INFO: running TNM@0 for 97 with 39299 remaining (b = 0.01)
5068.94/716.63	c INFO: running adaptg2wsat2009++@100 for 97 with 39284 remaining (b = 0.01)
5296.32/747.79	c INFO: running hybridGM3@166 for 97 with 39252 remaining (b = 0.01)
5420.43/765.61	c INFO: running TNM@0 for 97 with 39235 remaining (b = 0.01)
5749.40/812.70	c INFO: running hybridGM3@0 for 194 with 39188 remaining (b = 0.01)
5785.61/817.97	c INFO: running hybridGM3@95 for 97 with 39182 remaining (b = 0.01)
5892.13/833.29	c INFO: running TNM@0 for 97 with 39167 remaining (b = 0.01)
5965.94/843.11	c INFO: running adaptg2wsat2009++@0 for 194 with 39157 remaining (b = 0.01)
6002.41/848.02	c INFO: running TNM@0 for 97 with 39152 remaining (b = 0.01)
6172.42/871.96	c INFO: running hybridGM3@195 for 291 with 39128 remaining (b = 0.02)
6313.37/891.50	c INFO: running TNM@0 for 97 with 39109 remaining (b = 0.01)
6578.98/929.34	c INFO: running hybridGM3@294 for 194 with 39071 remaining (b = 0.01)
6650.81/939.65	c INFO: running TNM@0 for 97 with 39061 remaining (b = 0.01)
6795.77/960.47	c INFO: running TNM@0 for 97 with 39040 remaining (b = 0.01)
7098.67/1003.92	c INFO: running TNM@0 for 97 with 38996 remaining (b = 0.01)
7117.43/1006.66	c INFO: running TNM@0 for 97 with 38994 remaining (b = 0.01)
7233.64/1023.34	c INFO: running TNM@0 for 97 with 38977 remaining (b = 0.01)
7432.48/1052.04	c INFO: running TNM@0 for 97 with 38948 remaining (b = 0.01)
7512.88/1062.81	c INFO: running adaptg2wsat2009++@0 for 194 with 38937 remaining (b = 0.01)
7610.45/1075.95	c INFO: running TNM@0 for 97 with 38924 remaining (b = 0.00)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3321807-1304367709/watcher-3321807-1304367709 -o /tmp/evaluation-result-3321807-1304367709/solver-3321807-1304367709 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3321807-1304367709.cnf -seed 1392468813 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.38 8.32 6.10 2/287 21911
/proc/meminfo: memFree=23280208/32950940 swapFree=67111364/67111528
[pid=21911] ppid=21909 vsize=15776 CPUtime=0 cores=0-7
/proc/21911/stat : 21911 (python) R 21909 21911 21550 0 -1 4202496 880 0 0 0 0 0 0 0 22 0 1 0 627001684 16154624 761 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988253381 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/21911/statm: 3944 761 343 364 0 431 0

[startup+0.043198 s]
/proc/loadavg: 8.38 8.32 6.10 2/287 21911
/proc/meminfo: memFree=23280208/32950940 swapFree=67111364/67111528
[pid=21911] ppid=21909 vsize=18884 CPUtime=0.01 cores=0-7
/proc/21911/stat : 21911 (python) R 21909 21911 21550 0 -1 4202496 1285 0 0 0 1 0 0 0 22 0 1 0 627001684 19337216 1137 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988253381 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/21911/statm: 4721 1137 421 364 0 690 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 18884

[startup+0.100188 s]
/proc/loadavg: 8.38 8.32 6.10 2/287 21911
/proc/meminfo: memFree=23280208/32950940 swapFree=67111364/67111528
[pid=21911] ppid=21909 vsize=87704 CPUtime=0.06 cores=0-7
/proc/21911/stat : 21911 (python) R 21909 21911 21550 0 -1 4202496 2639 0 0 0 4 2 0 0 22 0 1 0 627001684 89808896 2354 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988590592 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/21911/statm: 21926 2354 717 364 0 1622 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 87704

[startup+0.300163 s]
/proc/loadavg: 8.38 8.32 6.10 2/287 21911
/proc/meminfo: memFree=23280208/32950940 swapFree=67111364/67111528
[pid=21911] ppid=21909 vsize=135748 CPUtime=0.2 cores=0-7
/proc/21911/stat : 21911 (python) S 21909 21911 21550 0 -1 4202496 5963 935 1 0 14 6 0 0 17 0 1 0 627001684 139005952 4151 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988588549 0 2147483391 16781312 2 18446744071699199166 0 0 17 4 0 0 0
/proc/21911/statm: 33937 4151 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.2
Current children cumulated vsize (KiB) 135748

[startup+0.700084 s]
/proc/loadavg: 8.38 8.32 6.10 2/287 21911
/proc/meminfo: memFree=23280208/32950940 swapFree=67111364/67111528
[pid=21911] ppid=21909 vsize=284160 CPUtime=0.44 cores=0-7
/proc/21911/stat : 21911 (python) R 21909 21911 21550 0 -1 4202496 10678 935 1 0 32 12 0 0 16 0 1 0 627001684 290979840 8359 33554432000 4194304 5683644 140735100740224 18446744073709551615 4445115 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/21911/statm: 71040 8359 1761 364 0 6964 0
Current children cumulated CPU time (s) 0.44
Current children cumulated vsize (KiB) 284160

[startup+1.50094 s]
/proc/loadavg: 8.38 8.32 6.10 3/288 21916
/proc/meminfo: memFree=23233452/32950940 swapFree=67111364/67111528
[pid=21911] ppid=21909 vsize=328000 CPUtime=1.24 cores=0-7
/proc/21911/stat : 21911 (python) R 21909 21911 21550 0 -1 4202496 27455 935 1 0 109 15 0 0 17 0 1 0 627001684 335872000 19385 33554432000 4194304 5683644 140735100740224 18446744073709551615 4523408 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/21911/statm: 82000 19385 1826 364 0 17924 0
Current children cumulated CPU time (s) 1.24
Current children cumulated vsize (KiB) 328000

[startup+3.10066 s]
/proc/loadavg: 8.38 8.32 6.10 3/288 21916
/proc/meminfo: memFree=23208040/32950940 swapFree=67111364/67111528
[pid=21911] ppid=21909 vsize=328000 CPUtime=2.84 cores=0-7
/proc/21911/stat : 21911 (python) R 21909 21911 21550 0 -1 4202496 27455 935 1 0 269 15 0 0 19 0 1 0 627001684 335872000 19385 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988252222 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/21911/statm: 82000 19385 1826 364 0 17924 0
Current children cumulated CPU time (s) 2.84
Current children cumulated vsize (KiB) 328000

[startup+6.3001 s]
/proc/loadavg: 7.87 8.22 6.08 3/288 21916
/proc/meminfo: memFree=23209056/32950940 swapFree=67111364/67111528
[pid=21911] ppid=21909 vsize=328000 CPUtime=6.04 cores=0-7
/proc/21911/stat : 21911 (python) R 21909 21911 21550 0 -1 4202496 27456 935 1 0 589 15 0 0 25 0 1 0 627001684 335872000 19386 33554432000 4194304 5683644 140735100740224 18446744073709551615 47651232502752 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/21911/statm: 82000 19386 1827 364 0 17924 0
Current children cumulated CPU time (s) 6.04
Current children cumulated vsize (KiB) 328000

[startup+12.701 s]
/proc/loadavg: 7.40 8.12 6.06 11/312 21940
/proc/meminfo: memFree=23062452/32950940 swapFree=67111364/67111528
[pid=21911] ppid=21909 vsize=406156 CPUtime=11.21 cores=0-7
/proc/21911/stat : 21911 (python) S 21909 21911 21550 0 -1 4202496 34190 935 1 0 1103 18 0 0 25 0 9 0 627001684 415903744 18504 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000419275 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/21911/statm: 101539 18504 1889 364 0 37436 0
[pid=21917] ppid=21911 vsize=324120 CPUtime=0 cores=0-7
/proc/21917/stat : 21917 (python) S 21911 21911 21550 0 -1 4202560 738 0 0 0 0 0 0 0 18 0 1 0 627002803 331898880 16936 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988612623 0 0 16781312 514 4339648704068714495 0 0 17 0 0 0 0
/proc/21917/statm: 81030 16936 353 364 0 16948 0
[pid=21919] ppid=21917 vsize=276232 CPUtime=1.49 cores=0-7
/proc/21919/stat : 21919 (hybridGM3) R 21917 21919 21919 0 -1 4202496 4887 0 0 0 149 0 0 0 25 0 1 0 627002803 282861568 4624 33554432000 134512640 135204385 4287759536 18446744073709551615 134543975 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/21919/statm: 69058 4624 57 169 0 68888 0
[pid=21920] ppid=21911 vsize=334376 CPUtime=0 cores=0-7
/proc/21920/stat : 21920 (python) S 21911 21911 21550 0 -1 4202560 737 0 0 0 0 0 0 0 18 0 1 0 627002807 342401024 16934 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988612678 0 0 16781312 514 4331138758947635199 0 0 17 3 0 0 0
/proc/21920/statm: 83594 16934 347 364 0 19509 0
[pid=21922] ppid=21920 vsize=276236 CPUtime=1.46 cores=0-7
/proc/21922/stat : 21922 (hybridGM3) R 21920 21922 21922 0 -1 4202496 4858 0 0 0 145 1 0 0 25 0 1 0 627002807 282865664 4596 33554432000 134512640 135204385 4287998256 18446744073709551615 134543975 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/21922/statm: 69059 4596 57 169 0 68889 0
[pid=21923] ppid=21911 vsize=344632 CPUtime=0 cores=0-7
/proc/21923/stat : 21923 (python) S 21911 21911 21550 0 -1 4202560 738 0 0 0 0 0 0 0 18 0 1 0 627002811 352903168 16938 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988612678 0 0 16781312 514 2076956578442903551 0 0 17 4 0 0 0
/proc/21923/statm: 86158 16938 347 364 0 22070 0
[pid=21925] ppid=21923 vsize=276232 CPUtime=1.42 cores=0-7
/proc/21925/stat : 21925 (hybridGM3) R 21923 21925 21925 0 -1 4202496 4832 0 0 0 142 0 0 0 25 0 1 0 627002811 282861568 4569 33554432000 134512640 135204385 4287397136 18446744073709551615 134543975 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/21925/statm: 69058 4569 57 169 0 68888 0
[pid=21926] ppid=21911 vsize=354888 CPUtime=0 cores=0-7
/proc/21926/stat : 21926 (python) S 21911 21911 21550 0 -1 4202560 750 0 0 0 0 0 0 0 18 0 1 0 627002815 363405312 16942 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988612678 0 0 16781312 514 3313888193115324415 0 0 17 3 0 0 0
/proc/21926/statm: 88722 16942 347 364 0 24631 0
[pid=21928] ppid=21926 vsize=276232 CPUtime=1.36 cores=0-7
/proc/21928/stat : 21928 (hybridGM3) R 21926 21928 21928 0 -1 4202496 4789 0 0 0 136 0 0 0 25 0 1 0 627002815 282861568 4527 33554432000 134512640 135204385 4294356672 18446744073709551615 134543699 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/21928/statm: 69058 4527 57 169 0 68888 0
[pid=21929] ppid=21911 vsize=365144 CPUtime=0 cores=0-7
/proc/21929/stat : 21929 (python) S 21911 21911 21550 0 -1 4202560 756 0 0 0 0 0 0 0 18 0 1 0 627002819 373907456 16946 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988612678 0 0 16781312 514 3313884344824627199 0 0 17 6 0 0 0
/proc/21929/statm: 91286 16946 347 364 0 27192 0
[pid=21931] ppid=21929 vsize=276236 CPUtime=1.34 cores=0-7
/proc/21931/stat : 21931 (hybridGM3) R 21929 21931 21931 0 -1 4202496 4779 0 0 0 133 1 0 0 25 0 1 0 627002819 282865664 4514 33554432000 134512640 135204385 4288539344 18446744073709551615 134543993 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/21931/statm: 69059 4514 57 169 0 68889 0
[pid=21932] ppid=21911 vsize=375400 CPUtime=0 cores=0-7
/proc/21932/stat : 21932 (python) S 21911 21911 21550 0 -1 4202560 764 0 0 0 0 0 0 0 18 0 1 0 627002823 384409600 16950 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988612678 0 0 16781312 514 1391419599739682815 0 0 17 0 0 0 0
/proc/21932/statm: 93850 16950 347 364 0 29753 0
[pid=21934] ppid=21932 vsize=268416 CPUtime=1.27 cores=0-7
/proc/21934/stat : 21934 (hybridGM3) R 21932 21934 21934 0 -1 4202496 4730 0 0 0 127 0 0 0 25 0 1 0 627002823 274857984 4466 33554432000 134512640 135204385 4291368096 18446744073709551615 134543968 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/21934/statm: 67104 4466 57 169 0 66934 0
[pid=21935] ppid=21911 vsize=385656 CPUtime=0 cores=0-7
/proc/21935/stat : 21935 (python) S 21911 21911 21550 0 -1 4202560 776 0 0 0 0 0 0 0 18 0 1 0 627002827 394911744 16954 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988612678 0 0 16781312 514 3313876648243232767 0 0 17 6 0 0 0
/proc/21935/statm: 96414 16954 347 364 0 32314 0
[pid=21937] ppid=21935 vsize=268416 CPUtime=1.22 cores=0-7
/proc/21937/stat : 21937 (hybridGM3) R 21935 21937 21937 0 -1 4202496 4697 0 0 0 121 1 0 0 25 0 1 0 627002830 274857984 4430 33554432000 134512640 135204385 4291514352 18446744073709551615 134543993 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/21937/statm: 67104 4430 57 169 0 66934 0
[pid=21938] ppid=21911 vsize=395912 CPUtime=0 cores=0-7
/proc/21938/stat : 21938 (python) S 21911 21911 21550 0 -1 4202560 802 0 0 0 0 0 0 0 18 0 1 0 627002831 405413888 16964 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988612678 0 0 16781312 514 4381593148523020287 0 0 17 1 0 0 0
/proc/21938/statm: 98978 16964 353 364 0 34875 0
[pid=21940] ppid=21938 vsize=465836 CPUtime=0.5 cores=0-7
/proc/21940/stat : 21940 (TNM) R 21938 21940 21940 0 -1 4202496 5132 0 0 0 50 0 0 0 25 0 1 0 627002841 477016064 4865 33554432000 134512640 135008748 4292220640 18446744073709551615 134522310 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/21940/statm: 116459 4865 45 122 0 115359 0
[pid=21911/tid=21918] ppid=21909 vsize=406156 CPUtime=0 cores=0-7
/proc/21911/task/21918/stat : 21918 (python) S 21909 21911 21550 0 -1 4202560 26 935 0 0 0 0 0 0 25 0 9 0 627002803 415903744 18504 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 0 0 0 0
[pid=21911/tid=21921] ppid=21909 vsize=406156 CPUtime=0 cores=0-7
/proc/21911/task/21921/stat : 21921 (python) S 21909 21911 21550 0 -1 4202560 22 935 0 0 0 0 0 0 19 0 9 0 627002807 415903744 18504 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 0 0 0 0
[pid=21911/tid=21924] ppid=21909 vsize=406156 CPUtime=0 cores=0-7
/proc/21911/task/21924/stat : 21924 (python) S 21909 21911 21550 0 -1 4202560 24 935 0 0 0 0 0 0 21 0 9 0 627002811 415903744 18504 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 0 0 0 0
[pid=21911/tid=21927] ppid=21909 vsize=406156 CPUtime=0 cores=0-7
/proc/21911/task/21927/stat : 21927 (python) S 21909 21911 21550 0 -1 4202560 22 935 0 0 0 0 0 0 22 0 9 0 627002815 415903744 18504 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 0 0 0 0
[pid=21911/tid=21930] ppid=21909 vsize=406156 CPUtime=0 cores=0-7
/proc/21911/task/21930/stat : 21930 (python) S 21909 21911 21550 0 -1 4202560 27 935 0 0 0 0 0 0 24 0 9 0 627002819 415903744 18504 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=21933] ppid=21909 vsize=406156 CPUtime=0 cores=0-7
/proc/21911/task/21933/stat : 21933 (python) S 21909 21911 21550 0 -1 4202560 27 935 0 0 0 0 0 0 25 0 9 0 627002823 415903744 18504 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 0 0 0 0
[pid=21911/tid=21936] ppid=21909 vsize=406156 CPUtime=0 cores=0-7
/proc/21911/task/21936/stat : 21936 (python) S 21909 21911 21550 0 -1 4202560 25 935 0 0 0 0 0 0 25 0 9 0 627002827 415903744 18504 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=21939] ppid=21909 vsize=406156 CPUtime=0 cores=0-7
/proc/21911/task/21939/stat : 21939 (python) S 21909 21911 21550 0 -1 4202560 31 935 0 0 0 0 0 0 25 0 9 0 627002831 415903744 18504 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 21.27
Current children cumulated vsize (KiB) 5670120

[startup+25.5018 s]
/proc/loadavg: 7.75 8.16 6.11 10/312 21941
/proc/meminfo: memFree=22888780/32950940 swapFree=67111364/67111528
[pid=21911] ppid=21909 vsize=406156 CPUtime=11.21 cores=0-7
/proc/21911/stat : 21911 (python) S 21909 21911 21550 0 -1 4202496 34190 935 1 0 1103 18 0 0 25 0 9 0 627001684 415903744 18504 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000419275 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/21911/statm: 101539 18504 1889 364 0 37436 0
[pid=21917] ppid=21911 vsize=324120 CPUtime=0.02 cores=0-7
/proc/21917/stat : 21917 (python) S 21911 21911 21550 0 -1 4202560 907 0 0 0 1 1 0 0 15 0 1 0 627002803 331898880 16959 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988612623 0 0 16781312 514 12086025326560248065 0 0 17 0 0 0 0
/proc/21917/statm: 81030 16959 376 364 0 16948 0
[pid=21919] ppid=21917 vsize=276236 CPUtime=14.3 cores=0-7
/proc/21919/stat : 21919 (hybridGM3) R 21917 21919 21919 0 -1 4202496 9510 0 0 0 1429 1 0 0 25 0 1 0 627002803 282865664 9247 33554432000 134512640 135204385 4287759536 18446744073709551615 134547403 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/21919/statm: 69059 9247 64 169 0 68889 0
[pid=21920] ppid=21911 vsize=334376 CPUtime=0.02 cores=0-7
/proc/21920/stat : 21920 (python) S 21911 21911 21550 0 -1 4202560 915 0 0 0 1 1 0 0 15 0 1 0 627002807 342401024 16957 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988612678 0 0 16781312 514 12086025326560248065 0 0 17 3 0 0 0
/proc/21920/statm: 83594 16957 370 364 0 19509 0
[pid=21922] ppid=21920 vsize=276240 CPUtime=14.25 cores=0-7
/proc/21922/stat : 21922 (hybridGM3) R 21920 21922 21922 0 -1 4202496 9510 0 0 0 1424 1 0 0 25 0 1 0 627002807 282869760 9248 33554432000 134512640 135204385 4287998256 18446744073709551615 134536922 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/21922/statm: 69060 9248 64 169 0 68890 0
[pid=21923] ppid=21911 vsize=344632 CPUtime=0.02 cores=0-7
/proc/21923/stat : 21923 (python) S 21911 21911 21550 0 -1 4202560 907 0 0 0 1 1 0 0 15 0 1 0 627002811 352903168 16960 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988612678 0 0 16781312 514 12086025326560248066 0 0 17 4 0 0 0
/proc/21923/statm: 86158 16960 369 364 0 22070 0
[pid=21925] ppid=21923 vsize=276236 CPUtime=14.19 cores=0-7
/proc/21925/stat : 21925 (hybridGM3) R 21923 21925 21925 0 -1 4202496 9510 0 0 0 1419 0 0 0 25 0 1 0 627002811 282865664 9247 33554432000 134512640 135204385 4287397136 18446744073709551615 134536986 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/21925/statm: 69059 9247 64 169 0 68889 0
[pid=21926] ppid=21911 vsize=354888 CPUtime=0.02 cores=0-7
/proc/21926/stat : 21926 (python) S 21911 21911 21550 0 -1 4202560 949 0 0 0 1 1 0 0 15 0 1 0 627002815 363405312 16965 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988612678 0 0 16781312 514 12086025326560248067 0 0 17 3 0 0 0
/proc/21926/statm: 88722 16965 370 364 0 24631 0
[pid=21928] ppid=21926 vsize=276236 CPUtime=14 cores=0-7
/proc/21928/stat : 21928 (hybridGM3) R 21926 21928 21928 0 -1 4202496 9515 0 0 0 1399 1 0 0 25 0 1 0 627002815 282865664 9253 33554432000 134512640 135204385 4294356672 18446744073709551615 134536993 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/21928/statm: 69059 9253 64 169 0 68889 0
[pid=21929] ppid=21911 vsize=365144 CPUtime=0.02 cores=0-7
/proc/21929/stat : 21929 (python) S 21911 21911 21550 0 -1 4202560 955 0 0 0 1 1 0 0 15 0 1 0 627002819 373907456 16969 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988612678 0 0 16781312 514 12086025326560248066 0 0 17 6 0 0 0
/proc/21929/statm: 91286 16969 370 364 0 27192 0
[pid=21931] ppid=21929 vsize=276240 CPUtime=14.08 cores=0-7
/proc/21931/stat : 21931 (hybridGM3) R 21929 21931 21931 0 -1 4202496 9512 0 0 0 1407 1 0 0 25 0 1 0 627002819 282869760 9247 33554432000 134512640 135204385 4288539344 18446744073709551615 134536993 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/21931/statm: 69060 9247 64 169 0 68890 0

################
# More data... #
################

/proc/22036/task/22053/stat : 22053 (python) S 21911 21911 21550 0 -1 4202560 102 0 0 0 0 0 0 0 19 0 2 0 627095640 661561344 17502 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 514 18446604453985827904 0 0 -1 0 0 0 0
[pid=22039] ppid=21911 vsize=656312 CPUtime=0.59 cores=0-7
/proc/22039/stat : 22039 (python) S 21911 21911 21550 0 -1 4202560 1187 0 0 0 31 28 0 0 19 0 2 0 627085995 672063488 17462 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000419275 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/22039/statm: 164078 17462 400 364 0 99918 0
[pid=22041] ppid=22039 vsize=438604 CPUtime=196.56 cores=0-7
/proc/22041/stat : 22041 (adaptg2wsat2009) T 22039 22041 22041 0 -1 4202496 8779 0 0 0 19654 2 0 0 25 0 1 0 627085996 449130496 8498 33554432000 134512640 135107411 4288139440 18446744073709551615 134531600 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/22041/statm: 109651 8498 59 146 0 108527 0
[pid=22039/tid=22086] ppid=21911 vsize=656312 CPUtime=0 cores=0-7
/proc/22039/task/22086/stat : 22086 (python) S 21911 21911 21550 0 -1 4202560 102 0 0 0 0 0 0 0 19 0 2 0 627109268 672063488 17462 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 514 18446604453985825216 0 0 -1 5 0 0 0
[pid=22043] ppid=21911 vsize=666568 CPUtime=0.28 cores=0-7
/proc/22043/stat : 22043 (python) S 21911 21911 21550 0 -1 4202560 1267 0 0 0 15 13 0 0 19 0 2 0 627086487 682565632 17420 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000419275 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/22043/statm: 166642 17420 401 364 0 102479 0
[pid=22045] ppid=22043 vsize=471116 CPUtime=102.06 cores=0-7
/proc/22045/stat : 22045 (TNM) T 22043 22045 22045 0 -1 4202496 8458 0 0 0 10203 3 0 0 25 0 1 0 627086487 482422784 8172 33554432000 134512640 135008748 4288045664 18446744073709551615 134528967 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/22045/statm: 117779 8172 46 122 0 116679 0
[pid=22043/tid=22058] ppid=21911 vsize=666568 CPUtime=0 cores=0-7
/proc/22043/task/22058/stat : 22058 (python) S 21911 21911 21550 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 627097722 682565632 17420 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 514 18446604450785779776 0 0 -1 0 0 0 0
[pid=22049] ppid=21911 vsize=676824 CPUtime=0.28 cores=0-7
/proc/22049/stat : 22049 (python) S 21911 21911 21550 0 -1 4202560 1261 0 0 0 13 15 0 0 19 0 2 0 627090834 693067776 17429 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000419275 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/22049/statm: 169206 17429 406 364 0 105040 0
[pid=22051] ppid=22049 vsize=471116 CPUtime=101.99 cores=0-7
/proc/22051/stat : 22051 (TNM) T 22049 22051 22051 0 -1 4202496 8458 0 0 0 10197 2 0 0 25 0 1 0 627090837 482422784 8170 33554432000 134512640 135008748 4292700608 18446744073709551615 134529409 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/22051/statm: 117779 8170 46 122 0 116679 0
[pid=22049/tid=22063] ppid=21911 vsize=676824 CPUtime=0 cores=0-7
/proc/22049/task/22063/stat : 22063 (python) S 21911 21911 21550 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 627102065 693067776 17429 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 514 18446604450785781568 0 0 -1 1 0 0 0
[pid=22054] ppid=21911 vsize=687080 CPUtime=0.28 cores=0-7
/proc/22054/stat : 22054 (python) S 21911 21911 21550 0 -1 4202560 1285 0 0 0 14 14 0 0 19 0 2 0 627095650 703569920 17434 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000419275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/22054/statm: 171770 17434 407 364 0 107601 0
[pid=22056] ppid=22054 vsize=471116 CPUtime=99.5 cores=0-7
/proc/22056/stat : 22056 (TNM) T 22054 22056 22056 0 -1 4202496 8463 0 0 0 9948 2 0 0 25 0 1 0 627095650 482422784 8171 33554432000 134512640 135008748 4292604928 18446744073709551615 134527729 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/22056/statm: 117779 8171 46 122 0 116679 0
[pid=22054/tid=22077] ppid=21911 vsize=687080 CPUtime=0 cores=0-7
/proc/22054/task/22077/stat : 22077 (python) S 21911 21911 21550 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 627106878 703569920 17434 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 514 18446604450785780672 0 0 -1 4 0 0 0
[pid=22059] ppid=21911 vsize=697336 CPUtime=0.29 cores=0-7
/proc/22059/stat : 22059 (python) S 21911 21911 21550 0 -1 4202560 1166 0 0 0 14 15 0 0 18 0 1 0 627097732 714072064 17407 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988612678 0 0 16781312 514 12086025326560248066 0 0 17 3 0 0 0
/proc/22059/statm: 174334 17407 377 364 0 110162 0
[pid=22061] ppid=22059 vsize=471116 CPUtime=94.51 cores=0-7
/proc/22061/stat : 22061 (TNM) R 22059 22061 22061 0 -1 4202496 8466 0 0 0 9449 2 0 0 25 0 1 0 627097732 482422784 8172 33554432000 134512640 135008748 4291987760 18446744073709551615 134527327 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/22061/statm: 117779 8172 46 122 0 116679 0
[pid=22064] ppid=21911 vsize=707592 CPUtime=0.18 cores=0-7
/proc/22064/stat : 22064 (python) S 21911 21911 21550 0 -1 4202560 1141 0 0 0 9 9 0 0 18 0 1 0 627102076 724574208 17412 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988612678 0 0 16781312 514 12086025326560248067 0 0 17 1 0 0 0
/proc/22064/statm: 176898 17412 378 364 0 112723 0
[pid=22066] ppid=22064 vsize=471116 CPUtime=59.91 cores=0-7
/proc/22066/stat : 22066 (TNM) R 22064 22066 22066 0 -1 4202496 8464 0 0 0 5990 1 0 0 25 0 1 0 627102077 482422784 8171 33554432000 134512640 135008748 4287708176 18446744073709551615 134525023 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/22066/statm: 117779 8171 46 122 0 116679 0
[pid=22067] ppid=21911 vsize=717848 CPUtime=0.17 cores=0-7
/proc/22067/stat : 22067 (python) S 21911 21911 21550 0 -1 4202560 1130 0 0 0 10 7 0 0 18 0 1 0 627102351 735076352 17409 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988612678 0 0 16781312 514 12086025326560248066 0 0 17 1 0 0 0
/proc/22067/statm: 179462 17409 371 364 0 115284 0
[pid=22069] ppid=22067 vsize=471116 CPUtime=47.21 cores=0-7
/proc/22069/stat : 22069 (TNM) R 22067 22069 22069 0 -1 4202496 8471 0 0 0 4720 1 0 0 25 0 1 0 627102351 482422784 8172 33554432000 134512640 135008748 4293386416 18446744073709551615 134524720 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/22069/statm: 117779 8172 46 122 0 116679 0
[pid=22072] ppid=21911 vsize=728104 CPUtime=0.13 cores=0-7
/proc/22072/stat : 22072 (python) S 21911 21911 21550 0 -1 4202560 1105 0 0 0 6 7 0 0 18 0 1 0 627104018 745578496 17414 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988612678 0 0 16781312 514 12086025326560248067 0 0 17 6 0 0 0
/proc/22072/statm: 182026 17414 372 364 0 117845 0
[pid=22074] ppid=22072 vsize=471116 CPUtime=53.33 cores=0-7
/proc/22074/stat : 22074 (TNM) R 22072 22074 22074 0 -1 4202496 8464 0 0 0 5331 2 0 0 25 0 1 0 627104019 482422784 8174 33554432000 134512640 135008748 4288779152 18446744073709551615 134527320 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/22074/statm: 117779 8174 46 122 0 116679 0
[pid=22078] ppid=21911 vsize=738360 CPUtime=0.06 cores=0-7
/proc/22078/stat : 22078 (python) S 21911 21911 21550 0 -1 4202560 1131 0 0 0 3 3 0 0 18 0 1 0 627106888 756080640 17418 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988612678 0 0 16781312 514 12086025326560248066 0 0 17 1 0 0 0
/proc/22078/statm: 184590 17418 372 364 0 120406 0
[pid=22080] ppid=22078 vsize=471116 CPUtime=19.23 cores=0-7
/proc/22080/stat : 22080 (TNM) R 22078 22080 22080 0 -1 4202496 8465 0 0 0 1921 2 0 0 25 0 1 0 627106889 482422784 8171 33554432000 134512640 135008748 4286877216 18446744073709551615 134524772 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/22080/statm: 117779 8171 46 122 0 116679 0
[pid=22082] ppid=21911 vsize=748616 CPUtime=0.03 cores=0-7
/proc/22082/stat : 22082 (python) S 21911 21911 21550 0 -1 4202560 1156 0 0 0 1 2 0 0 18 0 1 0 627107966 766582784 17427 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988612678 0 0 16781312 514 12086025326560248065 0 0 17 1 0 0 0
/proc/22082/statm: 187154 17427 377 364 0 122967 0
[pid=22084] ppid=22082 vsize=438604 CPUtime=7.5 cores=0-7
/proc/22084/stat : 22084 (adaptg2wsat2009) R 22082 22084 22084 0 -1 4202496 8795 0 0 0 748 2 0 0 25 0 1 0 627107967 449130496 8498 33554432000 134512640 135107411 4293931344 18446744073709551615 134531792 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/22084/statm: 109651 8498 59 146 0 108527 0
[pid=22087] ppid=21911 vsize=758872 CPUtime=0 cores=0-7
/proc/22087/stat : 22087 (python) S 21911 21911 21550 0 -1 4202560 975 0 0 0 0 0 0 0 19 0 1 0 627109280 777084928 17405 33554432000 4194304 5683644 140735100740224 18446744073709551615 215988612678 0 0 16781312 514 4933676903404929023 0 0 17 1 0 0 0
/proc/22087/statm: 189718 17405 351 364 0 125528 0
[pid=22089] ppid=22087 vsize=465836 CPUtime=0.44 cores=0-7
/proc/22089/stat : 22089 (TNM) R 22087 22089 22089 0 -1 4202496 5163 0 0 0 43 1 0 0 21 0 1 0 627109281 477016064 4865 33554432000 134512640 135008748 4287816240 18446744073709551615 134523482 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/22089/statm: 116459 4865 45 122 0 115359 0
[pid=21911/tid=21918] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/21918/stat : 21918 (python) S 21909 21911 21550 0 -1 4202560 30 55183 0 0 0 0 150206 259 18 0 38 0 627002803 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 0 0 0 0
[pid=21911/tid=21921] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/21921/stat : 21921 (python) S 21909 21911 21550 0 -1 4202560 28 55183 0 0 0 0 150206 259 18 0 38 0 627002807 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 0 0 0 0
[pid=21911/tid=21924] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/21924/stat : 21924 (python) S 21909 21911 21550 0 -1 4202560 37 55183 0 0 0 0 150206 259 18 0 38 0 627002811 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 0 0 0 0
[pid=21911/tid=21927] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/21927/stat : 21927 (python) S 21909 21911 21550 0 -1 4202560 25 55183 0 0 0 0 150206 259 18 0 38 0 627002815 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 0 0 0 0
[pid=21911/tid=21930] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/21930/stat : 21930 (python) S 21909 21911 21550 0 -1 4202560 31 55183 0 0 0 0 150206 259 18 0 38 0 627002819 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=21933] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/21933/stat : 21933 (python) S 21909 21911 21550 0 -1 4202560 36 55183 0 0 0 0 150206 259 15 0 38 0 627002823 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 0 0 0 0
[pid=21911/tid=21936] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/21936/stat : 21936 (python) S 21909 21911 21550 0 -1 4202560 29 55183 0 0 0 0 150206 259 18 0 38 0 627002827 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=21939] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/21939/stat : 21939 (python) S 21909 21911 21550 0 -1 4202560 35 55183 0 0 0 0 150206 259 18 0 38 0 627002831 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=21950] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/21950/stat : 21950 (python) S 21909 21911 21550 0 -1 4202560 36 55183 0 0 0 0 150206 259 18 0 38 0 627016483 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=21954] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/21954/stat : 21954 (python) S 21909 21911 21550 0 -1 4202560 31 55183 0 0 0 0 150206 259 18 0 38 0 627021862 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 5 0 0 0
[pid=21911/tid=21964] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/21964/stat : 21964 (python) S 21909 21911 21550 0 -1 4202560 36 55183 0 0 0 0 150206 259 18 0 38 0 627032610 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=21967] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/21967/stat : 21967 (python) S 21909 21911 21550 0 -1 4202560 32 55183 0 0 0 0 150206 259 19 0 38 0 627033206 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=21970] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/21970/stat : 21970 (python) S 21909 21911 21550 0 -1 4202560 35 55183 0 0 0 0 150206 259 18 0 38 0 627033816 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=21975] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/21975/stat : 21975 (python) S 21909 21911 21550 0 -1 4202560 32 55183 0 0 0 0 150206 259 20 0 38 0 627039109 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=21982] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/21982/stat : 21982 (python) S 21909 21911 21550 0 -1 4202560 44 55183 0 0 0 0 150206 259 18 0 38 0 627044339 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=21987] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/21987/stat : 21987 (python) S 21909 21911 21550 0 -1 4202560 94 55183 0 0 0 0 150206 259 18 0 38 0 627048884 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=21990] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/21990/stat : 21990 (python) S 21909 21911 21550 0 -1 4202560 29 55183 0 0 0 0 150206 259 20 0 38 0 627052150 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=21995] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/21995/stat : 21995 (python) S 21909 21911 21550 0 -1 4202560 34 55183 0 0 0 0 150206 259 20 0 38 0 627052773 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=22001] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/22001/stat : 22001 (python) S 21909 21911 21550 0 -1 4202560 43 55183 0 0 0 0 150206 259 18 0 38 0 627059981 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=22005] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/22005/stat : 22005 (python) S 21909 21911 21550 0 -1 4202560 30 55183 0 0 0 0 150206 259 20 0 38 0 627061805 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 1 0 0 0
[pid=21911/tid=22011] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/22011/stat : 22011 (python) S 21909 21911 21550 0 -1 4202560 129 55183 0 0 0 0 150206 259 19 0 38 0 627067012 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=22015] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/22015/stat : 22015 (python) S 21909 21911 21550 0 -1 4202560 40 55183 0 0 0 0 150206 259 19 0 38 0 627068843 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=22020] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/22020/stat : 22020 (python) S 21909 21911 21550 0 -1 4202560 38 55183 0 0 0 0 150206 259 20 0 38 0 627071821 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=22026] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/22026/stat : 22026 (python) S 21909 21911 21550 0 -1 4202560 42 55183 0 0 0 0 150206 259 20 0 38 0 627078246 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 1 0 0 0
[pid=21911/tid=22032] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/22032/stat : 22032 (python) S 21909 21911 21550 0 -1 4202560 37 55183 0 0 0 0 150206 259 19 0 38 0 627082955 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=22037] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/22037/stat : 22037 (python) S 21909 21911 21550 0 -1 4202560 37 55183 0 0 0 0 150206 259 19 0 38 0 627085014 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=22040] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/22040/stat : 22040 (python) S 21909 21911 21550 0 -1 4202560 38 55183 0 0 0 0 150206 259 19 0 38 0 627085996 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=22044] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/22044/stat : 22044 (python) S 21909 21911 21550 0 -1 4202560 27 55183 0 0 0 0 150206 259 19 0 38 0 627086487 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=22050] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/22050/stat : 22050 (python) S 21909 21911 21550 0 -1 4202560 25 55183 0 0 0 0 150206 259 19 0 38 0 627090834 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=22055] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/22055/stat : 22055 (python) S 21909 21911 21550 0 -1 4202560 29 55183 0 0 0 0 150206 259 20 0 38 0 627095650 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=22060] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/22060/stat : 22060 (python) S 21909 21911 21550 0 -1 4202560 25 55183 0 0 0 0 150206 259 19 0 38 0 627097732 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=22065] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/22065/stat : 22065 (python) S 21909 21911 21550 0 -1 4202560 30 55183 0 0 0 0 150206 259 19 0 38 0 627102077 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=22068] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/22068/stat : 22068 (python) S 21909 21911 21550 0 -1 4202560 29 55183 0 0 0 0 150206 259 19 0 38 0 627102351 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=22073] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/22073/stat : 22073 (python) S 21909 21911 21550 0 -1 4202560 24 55183 0 0 0 0 150206 259 20 0 38 0 627104019 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=22079] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/22079/stat : 22079 (python) S 21909 21911 21550 0 -1 4202560 26 55183 0 0 0 0 150206 259 19 0 38 0 627106889 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=22083] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/22083/stat : 22083 (python) S 21909 21911 21550 0 -1 4202560 25 55183 0 0 0 0 150206 259 19 0 38 0 627107966 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
[pid=21911/tid=22088] ppid=21909 vsize=769116 CPUtime=1504.65 cores=0-7
/proc/21911/task/22088/stat : 22088 (python) S 21909 21911 21550 0 -1 4202560 26 55183 0 0 0 0 150206 259 20 0 38 0 627109280 787574784 19006 33554432000 4194304 5683644 140735100740224 18446744073709551615 216000416177 0 0 16781312 2 18446604453902894400 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 7616.9
Current children cumulated vsize (KiB) 31897256

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 21911 and gives
#  childrusage.ru_utime.tv_sec=1516
#  childrusage.ru_utime.tv_usec=596442
#  childrusage.ru_stime.tv_sec=2
#  childrusage.ru_stime.tv_usec=921555
# CPU time returned by wait4() is 1519.52
# while last known CPU time is 7616.9
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1077.01
CPU time (s): 7616.9
CPU user time (s): 7605.51
CPU system time (s): 11.39
CPU usage (%): 707.229
Max. virtual memory (cumulated for all children) (KiB): 31897256

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1516.6
system time used= 2.92156
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 127367
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 5920
involuntary context switches= 12765

runsolver used 8.24575 second user time and 21.0188 second system time

The end

Launcher Data 

Begin job on node117 at 2011-05-02 22:21:49
IDJOB=3321807
IDBENCH=82585
IDSOLVER=1712
FILE ID=node117/3321807-1304367709
RUNJOBID= node117-1304366467-21564
PBS_JOBID= 13181754
Free space on /tmp= 72660 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v50000-c210000-S1401248878-094-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3321807-1304367709/watcher-3321807-1304367709 -o /tmp/evaluation-result-3321807-1304367709/solver-3321807-1304367709 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3321807-1304367709.cnf -seed 1392468813 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 56bd5db172bed91bc72714a8bad8639b
RANDOM SEED=1392468813

node117.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5267.44
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950940 kB
MemFree:      23280372 kB
Buffers:       1022392 kB
Cached:        4518016 kB
SwapCached:        160 kB
Active:        4091316 kB
Inactive:      5079304 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950940 kB
LowFree:      23280372 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:           36128 kB
Writeback:           8 kB
AnonPages:     3630212 kB
Mapped:          24432 kB
Slab:           409808 kB
PageTables:      22808 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586996 kB
Committed_AS: 49879900 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72444 MiB
End job on node117 at 2011-05-02 22:39:48