Trace number 3321799

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (MO) 7957 1008.81

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v50000-c210000-S746725327-020-UNKNOWN.cnf
MD5SUM0ad997eea0def3552b6d7e45aac8ca7c
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark1056.42
Satisfiable
(Un)Satisfiability was proved
Number of variables49999
Number of clauses210000
Sum of the clauses size630000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3210000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.40/0.72	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.40/0.72	c INFO: solving HOME/instance-3321799-1304367544.cnf
0.40/0.72	c INFO: computing task features
1.08/1.45	c INFO: computed variable-clause graph statistics
1.08/1.48	c INFO: computed clause balance statistics
4.89/5.25	c INFO: computed variable balance statistics
7.99/8.32	c INFO: computed small-clause counts
7.99/8.34	c INFO: computed Horn-clause counts
11.09/11.42	c INFO: computed variable graph statistics
11.09/11.42	c INFO: collected features for HOME/instance-3321799-1304367544.cnf in 10.68 s
11.09/11.46	c INFO: running hybridGM3@0 for 97 with 39989 remaining (b = 0.33)
11.19/11.50	c INFO: running hybridGM3@0 for 97 with 39989 remaining (b = 0.21)
11.19/11.54	c INFO: running hybridGM3@0 for 97 with 39989 remaining (b = 0.14)
11.19/11.58	c INFO: running hybridGM3@0 for 194 with 39989 remaining (b = 0.17)
11.29/11.62	c INFO: running hybridGM3@0 for 194 with 39989 remaining (b = 0.13)
11.29/11.66	c INFO: running hybridGM3@0 for 194 with 39989 remaining (b = 0.10)
11.38/11.70	c INFO: running hybridGM3@0 for 194 with 39989 remaining (b = 0.07)
11.38/11.74	c INFO: running TNM@0 for 97 with 39989 remaining (b = 0.03)
811.55/111.85	c INFO: running hybridGM3@0 for 97 with 39888 remaining (b = 0.03)
811.55/111.89	c INFO: running hybridGM3@0 for 97 with 39888 remaining (b = 0.03)
812.28/111.93	c INFO: running hybridGM3@0 for 97 with 39888 remaining (b = 0.03)
813.08/112.01	c INFO: running hybridGM3@0 for 194 with 39888 remaining (b = 0.05)
1572.02/207.11	c INFO: running hybridGM3@100 for 97 with 39793 remaining (b = 0.02)
1572.02/207.15	c INFO: running hybridGM3@100 for 97 with 39793 remaining (b = 0.02)
1572.02/207.20	c INFO: running hybridGM3@99 for 97 with 39793 remaining (b = 0.02)
1580.71/208.24	c INFO: running hybridGM3@0 for 97 with 39792 remaining (b = 0.02)
1580.71/208.28	c INFO: running hybridGM3@0 for 97 with 39792 remaining (b = 0.02)
1581.44/208.32	c INFO: running hybridGM3@0 for 97 with 39792 remaining (b = 0.02)
1627.75/214.19	c INFO: running hybridGM3@0 for 97 with 39786 remaining (b = 0.02)
2341.30/303.56	c INFO: running hybridGM3@0 for 194 with 39697 remaining (b = 0.04)
2379.47/308.36	c INFO: running hybridGM3@0 for 194 with 39692 remaining (b = 0.03)
2380.24/308.42	c INFO: running hybridGM3@0 for 194 with 39692 remaining (b = 0.03)
2380.24/308.47	c INFO: running hybridGM3@295 for 97 with 39692 remaining (b = 0.01)
2381.43/308.69	c INFO: running TNM@0 for 97 with 39692 remaining (b = 0.01)
2388.14/309.44	c INFO: running hybridGM3@95 for 97 with 39691 remaining (b = 0.01)
2388.14/309.50	c INFO: running hybridGM3@94 for 97 with 39691 remaining (b = 0.01)
2388.89/309.55	c INFO: running TNM@100 for 1940 with 39691 remaining (b = 0.20)
3140.58/403.75	c INFO: running hybridGM3@0 for 194 with 39597 remaining (b = 0.02)
3146.87/404.71	c INFO: running hybridGM3@0 for 194 with 39596 remaining (b = 0.02)
3148.55/404.78	c INFO: running adaptg2wsat2009++@0 for 194 with 39595 remaining (b = 0.02)
3182.68/409.01	c INFO: running TNM@0 for 97 with 39591 remaining (b = 0.01)
3909.59/500.27	c INFO: running hybridGM3@100 for 97 with 39500 remaining (b = 0.01)
3948.63/505.13	c INFO: running hybridGM3@100 for 97 with 39495 remaining (b = 0.01)
3949.40/505.20	c INFO: running hybridGM3@99 for 97 with 39495 remaining (b = 0.01)
4029.86/515.33	c INFO: running hybridGM3@0 for 194 with 39485 remaining (b = 0.02)
4699.34/599.31	c INFO: running TNM@0 for 97 with 39401 remaining (b = 0.01)
4708.10/600.49	c INFO: running hybridGM3@0 for 194 with 39400 remaining (b = 0.01)
4708.88/600.57	c INFO: running hybridGM3@95 for 97 with 39400 remaining (b = 0.01)
4716.77/601.57	c INFO: running hybridGM3@295 for 194 with 39399 remaining (b = 0.02)
4755.74/606.43	c INFO: running hybridGM3@297 for 194 with 39394 remaining (b = 0.02)
4763.66/607.43	c INFO: running hybridGM3@94 for 97 with 39393 remaining (b = 0.01)
5468.21/695.86	c INFO: running TNM@0 for 97 with 39304 remaining (b = 0.01)
5498.55/699.62	c INFO: running TNM@0 for 97 with 39301 remaining (b = 0.01)
5523.14/702.73	c INFO: running hybridGM3@190 for 97 with 39298 remaining (b = 0.01)
5597.27/712.07	c INFO: running TNM@0 for 97 with 39288 remaining (b = 0.01)
6266.72/796.20	c INFO: running TNM@0 for 97 with 39204 remaining (b = 0.01)
6273.05/796.98	c INFO: running adaptg2wsat2009++@100 for 97 with 39203 remaining (b = 0.01)
6275.44/797.23	c INFO: running TNM@0 for 97 with 39203 remaining (b = 0.01)
6281.75/798.05	c INFO: running hybridGM3@385 for 194 with 39202 remaining (b = 0.02)
6290.47/799.14	c INFO: running hybridGM3@395 for 97 with 39201 remaining (b = 0.01)
6296.84/799.98	c INFO: running TNM@0 for 97 with 39200 remaining (b = 0.01)
6396.33/812.42	c INFO: running TNM@0 for 97 with 39188 remaining (b = 0.01)
7049.02/894.47	c INFO: running adaptg2wsat2009++@0 for 194 with 39106 remaining (b = 0.01)
7057.73/895.59	c INFO: running TNM@0 for 97 with 39105 remaining (b = 0.01)
7073.71/897.60	c INFO: running TNM@0 for 97 with 39103 remaining (b = 0.01)
7087.92/899.37	c INFO: running adaptg2wsat2009++@296 for 291 with 39101 remaining (b = 0.02)
7095.88/900.36	c INFO: running hybridGM3@199 for 291 with 39100 remaining (b = 0.02)
7194.47/912.78	c INFO: running TNM@0 for 97 with 39088 remaining (b = 0.01)
7855.91/995.97	c INFO: running TNM@0 for 97 with 39004 remaining (b = 0.01)
7871.81/997.97	c INFO: running TNM@0 for 97 with 39002 remaining (b = 0.01)
7948.98/1007.67	c INFO: running TNM@0 for 97 with 38993 remaining (b = 0.01)
7957.00/1008.79	c INFO: running TNM@0 for 97 with 38992 remaining (b = 0.00)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3321799-1304367544/watcher-3321799-1304367544 -o /tmp/evaluation-result-3321799-1304367544/solver-3321799-1304367544 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3321799-1304367544.cnf -seed 2123509085 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.59 8.22 7.54 1/285 29804
/proc/meminfo: memFree=26035568/32950928 swapFree=65762296/67111528
[pid=29804] ppid=29802 vsize=15776 CPUtime=0 cores=0-7
/proc/29804/stat : 29804 (python) R 29802 29804 29518 0 -1 4202496 870 0 0 0 0 0 0 0 20 0 1 0 626985113 16154624 751 33554432000 4194304 5683644 140734716449200 18446744073709551615 4540219 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/29804/statm: 3944 751 343 364 0 431 0

[startup+0.0977911 s]
/proc/loadavg: 8.59 8.22 7.54 1/285 29804
/proc/meminfo: memFree=26035568/32950928 swapFree=65762296/67111528
[pid=29804] ppid=29802 vsize=85620 CPUtime=0.05 cores=0-7
/proc/29804/stat : 29804 (python) R 29802 29804 29518 0 -1 4202496 2630 0 0 0 4 1 0 0 18 0 1 0 626985113 87674880 2348 33554432000 4194304 5683644 140734716449200 18446744073709551615 260905392128 0 2147483391 16781312 2 0 0 0 17 4 0 0 0
/proc/29804/statm: 21405 2348 713 364 0 1622 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 85620

[startup+0.100789 s]
/proc/loadavg: 8.59 8.22 7.54 1/285 29804
/proc/meminfo: memFree=26035568/32950928 swapFree=65762296/67111528
[pid=29804] ppid=29802 vsize=91828 CPUtime=0.06 cores=0-7
/proc/29804/stat : 29804 (python) R 29802 29804 29518 0 -1 4202496 2675 0 0 0 5 1 0 0 17 0 1 0 626985113 94031872 2390 33554432000 4194304 5683644 140734716449200 18446744073709551615 260900424480 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/29804/statm: 22957 2392 753 364 0 1622 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 91828

[startup+0.300824 s]
/proc/loadavg: 8.59 8.22 7.54 1/285 29804
/proc/meminfo: memFree=26035568/32950928 swapFree=65762296/67111528
[pid=29804] ppid=29802 vsize=135748 CPUtime=0.19 cores=0-7
/proc/29804/stat : 29804 (python) R 29802 29804 29518 0 -1 4202496 5965 935 1 0 14 5 0 0 17 0 1 0 626985113 139005952 4151 33554432000 4194304 5683644 140734716449200 18446744073709551615 260905390085 0 2147483391 16781312 2 0 0 0 17 4 0 0 0
/proc/29804/statm: 33937 4151 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 135748

[startup+0.700768 s]
/proc/loadavg: 8.59 8.22 7.54 1/285 29804
/proc/meminfo: memFree=26035568/32950928 swapFree=65762296/67111528
[pid=29804] ppid=29802 vsize=259464 CPUtime=0.4 cores=0-7
/proc/29804/stat : 29804 (python) R 29802 29804 29518 0 -1 4202496 8682 935 1 0 31 9 0 0 16 0 1 0 626985113 265691136 6477 33554432000 4194304 5683644 140734716449200 18446744073709551615 4439516 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/29804/statm: 64866 6477 1656 364 0 5224 0
Current children cumulated CPU time (s) 0.4
Current children cumulated vsize (KiB) 259464

[startup+1.50053 s]
/proc/loadavg: 8.59 8.22 7.54 2/286 29809
/proc/meminfo: memFree=25993588/32950928 swapFree=65762296/67111528
[pid=29804] ppid=29802 vsize=328000 CPUtime=1.19 cores=0-7
/proc/29804/stat : 29804 (python) R 29802 29804 29518 0 -1 4202496 27454 935 1 0 106 13 0 0 16 0 1 0 626985113 335872000 19385 33554432000 4194304 5683644 140734716449200 18446744073709551615 46988981977729 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/29804/statm: 82000 19385 1826 364 0 17924 0
Current children cumulated CPU time (s) 1.19
Current children cumulated vsize (KiB) 328000

[startup+3.10024 s]
/proc/loadavg: 8.14 8.13 7.52 2/286 29809
/proc/meminfo: memFree=25964572/32950928 swapFree=65762296/67111528
[pid=29804] ppid=29802 vsize=328000 CPUtime=2.79 cores=0-7
/proc/29804/stat : 29804 (python) R 29802 29804 29518 0 -1 4202496 27454 935 1 0 266 13 0 0 18 0 1 0 626985113 335872000 19385 33554432000 4194304 5683644 140734716449200 18446744073709551615 4623648 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/29804/statm: 82000 19385 1826 364 0 17924 0
Current children cumulated CPU time (s) 2.79
Current children cumulated vsize (KiB) 328000

[startup+6.30065 s]
/proc/loadavg: 8.14 8.13 7.52 2/286 29809
/proc/meminfo: memFree=25965800/32950928 swapFree=65762296/67111528
[pid=29804] ppid=29802 vsize=328000 CPUtime=5.99 cores=0-7
/proc/29804/stat : 29804 (python) R 29802 29804 29518 0 -1 4202496 27455 935 1 0 586 13 0 0 25 0 1 0 626985113 335872000 19386 33554432000 4194304 5683644 140734716449200 18446744073709551615 260905053007 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/29804/statm: 82000 19386 1827 364 0 17924 0
Current children cumulated CPU time (s) 5.99
Current children cumulated vsize (KiB) 328000

[startup+12.7015 s]
/proc/loadavg: 7.61 8.01 7.49 9/310 29833
/proc/meminfo: memFree=25809136/32950928 swapFree=65762296/67111528
[pid=29804] ppid=29802 vsize=406156 CPUtime=11.43 cores=0-7
/proc/29804/stat : 29804 (python) S 29802 29804 29518 0 -1 4202496 34192 935 1 0 1128 15 0 0 24 0 9 0 626985113 415903744 18504 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/29804/statm: 101539 18504 1889 364 0 37436 0
[pid=29810] ppid=29804 vsize=324120 CPUtime=0 cores=0-7
/proc/29810/stat : 29810 (python) S 29804 29804 29518 0 -1 4202560 731 0 0 0 0 0 0 0 18 0 1 0 626986260 331898880 16936 33554432000 4194304 5683644 140734716449200 18446744073709551615 260905414159 0 0 16781312 514 6091563802522812415 0 0 17 5 0 0 0
/proc/29810/statm: 81030 16936 353 364 0 16948 0
[pid=29812] ppid=29810 vsize=268424 CPUtime=1.22 cores=0-7
/proc/29812/stat : 29812 (hybridGM3) R 29810 29812 29812 0 -1 4202496 4695 0 0 0 122 0 0 0 25 0 1 0 626986260 274866176 4429 33554432000 134512640 135204385 4291160096 18446744073709551615 134543968 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/29812/statm: 67106 4429 57 169 0 66936 0
[pid=29813] ppid=29804 vsize=334376 CPUtime=0 cores=0-7
/proc/29813/stat : 29813 (python) S 29804 29804 29518 0 -1 4202560 736 0 0 0 0 0 0 0 18 0 1 0 626986264 342401024 16934 33554432000 4194304 5683644 140734716449200 18446744073709551615 260905414214 0 0 16781312 514 5517534345306308607 0 0 17 0 0 0 0
/proc/29813/statm: 83594 16934 347 364 0 19509 0
[pid=29815] ppid=29813 vsize=268420 CPUtime=1.18 cores=0-7
/proc/29815/stat : 29815 (hybridGM3) R 29813 29815 29815 0 -1 4202496 4667 0 0 0 117 1 0 0 25 0 1 0 626986264 274862080 4404 33554432000 134512640 135204385 4291523232 18446744073709551615 134543975 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/29815/statm: 67105 4404 57 169 0 66935 0
[pid=29816] ppid=29804 vsize=344632 CPUtime=0 cores=0-7
/proc/29816/stat : 29816 (python) S 29804 29804 29518 0 -1 4202560 741 0 0 0 0 0 0 0 18 0 1 0 626986268 352903168 16938 33554432000 4194304 5683644 140734716449200 18446744073709551615 260905414214 0 0 16781312 514 117053462432186367 0 0 17 2 0 0 0
/proc/29816/statm: 86158 16938 347 364 0 22070 0
[pid=29818] ppid=29816 vsize=268424 CPUtime=1.14 cores=0-7
/proc/29818/stat : 29818 (hybridGM3) R 29816 29818 29818 0 -1 4202496 4645 0 0 0 114 0 0 0 25 0 1 0 626986268 274866176 4377 33554432000 134512640 135204385 4294642000 18446744073709551615 134543993 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/29818/statm: 67106 4377 57 169 0 66936 0
[pid=29819] ppid=29804 vsize=354888 CPUtime=0 cores=0-7
/proc/29819/stat : 29819 (python) S 29804 29804 29518 0 -1 4202560 744 0 0 0 0 0 0 0 18 0 1 0 626986272 363405312 16942 33554432000 4194304 5683644 140734716449200 18446744073709551615 260905414214 0 0 16781312 514 117041917560094719 0 0 17 4 0 0 0
/proc/29819/statm: 88722 16942 347 364 0 24631 0
[pid=29821] ppid=29819 vsize=268424 CPUtime=1.08 cores=0-7
/proc/29821/stat : 29821 (hybridGM3) R 29819 29821 29821 0 -1 4202496 4597 0 0 0 108 0 0 0 24 0 1 0 626986272 274866176 4336 33554432000 134512640 135204385 4292233584 18446744073709551615 134543968 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/29821/statm: 67106 4336 57 169 0 66936 0
[pid=29822] ppid=29804 vsize=365144 CPUtime=0 cores=0-7
/proc/29822/stat : 29822 (python) S 29804 29804 29518 0 -1 4202560 760 0 0 0 0 0 0 0 18 0 1 0 626986276 373907456 16946 33554432000 4194304 5683644 140734716449200 18446744073709551615 260905414214 0 0 16781312 514 9674779013834342399 0 0 17 2 0 0 0
/proc/29822/statm: 91286 16946 347 364 0 27192 0
[pid=29824] ppid=29822 vsize=268420 CPUtime=1.03 cores=0-7
/proc/29824/stat : 29824 (hybridGM3) R 29822 29824 29824 0 -1 4202496 4555 0 0 0 103 0 0 0 25 0 1 0 626986276 274862080 4292 33554432000 134512640 135204385 4291047248 18446744073709551615 134543993 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/29824/statm: 67105 4292 57 169 0 66935 0
[pid=29825] ppid=29804 vsize=375400 CPUtime=0 cores=0-7
/proc/29825/stat : 29825 (python) S 29804 29804 29518 0 -1 4202560 749 0 0 0 0 0 0 0 18 0 1 0 626986280 384409600 16950 33554432000 4194304 5683644 140734716449200 18446744073709551615 260905414214 0 0 16781312 514 6141630339371499519 0 0 17 0 0 0 0
/proc/29825/statm: 93850 16950 347 364 0 29753 0
[pid=29827] ppid=29825 vsize=268420 CPUtime=1.02 cores=0-7
/proc/29827/stat : 29827 (hybridGM3) R 29825 29827 29827 0 -1 4202496 4554 0 0 0 102 0 0 0 24 0 1 0 626986280 274862080 4289 33554432000 134512640 135204385 4289883984 18446744073709551615 134543968 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/29827/statm: 67105 4289 57 169 0 66935 0
[pid=29828] ppid=29804 vsize=385656 CPUtime=0 cores=0-7
/proc/29828/stat : 29828 (python) S 29804 29804 29518 0 -1 4202560 755 0 0 0 0 0 0 0 21 0 1 0 626986284 394911744 16950 33554432000 4194304 5683644 140734716449200 18446744073709551615 260905414214 0 0 16781312 514 9674771317252947967 0 0 17 2 0 0 0
/proc/29828/statm: 96414 16950 343 364 0 32314 0
[pid=29830] ppid=29828 vsize=268416 CPUtime=0.94 cores=0-7
/proc/29830/stat : 29830 (hybridGM3) R 29828 29830 29830 0 -1 4202496 4499 0 0 0 93 1 0 0 25 0 1 0 626986285 274857984 4237 33554432000 134512640 135204385 4293791728 18446744073709551615 134543975 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/29830/statm: 67104 4237 57 169 0 66934 0
[pid=29831] ppid=29804 vsize=395912 CPUtime=0 cores=0-7
/proc/29831/stat : 29831 (python) S 29804 29804 29518 0 -1 4202560 776 0 0 0 0 0 0 0 25 0 1 0 626986288 405413888 16960 33554432000 4194304 5683644 140734716449200 18446744073709551615 260905414214 0 0 16781312 514 3229761259939299327 0 0 17 0 0 0 0
/proc/29831/statm: 98978 16960 349 364 0 34875 0
[pid=29833] ppid=29831 vsize=471116 CPUtime=0.94 cores=0-7
/proc/29833/stat : 29833 (TNM) R 29831 29833 29833 0 -1 4202496 8439 0 0 0 93 1 0 0 25 0 1 0 626986289 482422784 8173 33554432000 134512640 135008748 4287145424 18446744073709551615 134528190 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/29833/statm: 117779 8173 46 122 0 116679 0
[pid=29804/tid=29811] ppid=29802 vsize=406156 CPUtime=0 cores=0-7
/proc/29804/task/29811/stat : 29811 (python) S 29802 29804 29518 0 -1 4202560 25 935 0 0 0 0 0 0 25 0 9 0 626986260 415903744 18504 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 0 0 0 0
[pid=29804/tid=29814] ppid=29802 vsize=406156 CPUtime=0 cores=0-7
/proc/29804/task/29814/stat : 29814 (python) S 29802 29804 29518 0 -1 4202560 25 935 0 0 0 0 0 0 20 0 9 0 626986264 415903744 18504 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 0 0 0 0
[pid=29804/tid=29817] ppid=29802 vsize=406156 CPUtime=0 cores=0-7
/proc/29804/task/29817/stat : 29817 (python) S 29802 29804 29518 0 -1 4202560 19 935 0 0 0 0 0 0 20 0 9 0 626986268 415903744 18504 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 0 0 0 0
[pid=29804/tid=29820] ppid=29802 vsize=406156 CPUtime=0 cores=0-7
/proc/29804/task/29820/stat : 29820 (python) S 29802 29804 29518 0 -1 4202560 18 935 0 0 0 0 0 0 20 0 9 0 626986272 415903744 18504 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 0 0 0 0
[pid=29804/tid=29823] ppid=29802 vsize=406156 CPUtime=0 cores=0-7
/proc/29804/task/29823/stat : 29823 (python) S 29802 29804 29518 0 -1 4202560 28 935 0 0 0 0 0 0 22 0 9 0 626986276 415903744 18504 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 4 0 0 0
[pid=29804/tid=29826] ppid=29802 vsize=406156 CPUtime=0 cores=0-7
/proc/29804/task/29826/stat : 29826 (python) S 29802 29804 29518 0 -1 4202560 24 935 0 0 0 0 0 0 20 0 9 0 626986280 415903744 18504 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 2 0 0 0
[pid=29804/tid=29829] ppid=29802 vsize=406156 CPUtime=0 cores=0-7
/proc/29804/task/29829/stat : 29829 (python) S 29802 29804 29518 0 -1 4202560 26 935 0 0 0 0 0 0 22 0 9 0 626986284 415903744 18504 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 0 0 0 0
[pid=29804/tid=29832] ppid=29802 vsize=406156 CPUtime=0 cores=0-7
/proc/29804/task/29832/stat : 29832 (python) S 29802 29804 29518 0 -1 4202560 25 935 0 0 0 0 0 0 24 0 9 0 626986288 415903744 18504 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 19.98
Current children cumulated vsize (KiB) 5636348

[startup+25.5012 s]
/proc/loadavg: 7.75 8.03 7.50 9/310 29834
/proc/meminfo: memFree=25643672/32950928 swapFree=65762296/67111528
[pid=29804] ppid=29802 vsize=406156 CPUtime=11.43 cores=0-7
/proc/29804/stat : 29804 (python) S 29802 29804 29518 0 -1 4202496 34192 935 1 0 1128 15 0 0 24 0 9 0 626985113 415903744 18504 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/29804/statm: 101539 18504 1889 364 0 37436 0
[pid=29810] ppid=29804 vsize=324120 CPUtime=0.02 cores=0-7
/proc/29810/stat : 29810 (python) S 29804 29804 29518 0 -1 4202560 888 0 0 0 1 1 0 0 15 0 1 0 626986260 331898880 16959 33554432000 4194304 5683644 140734716449200 18446744073709551615 260905414159 0 0 16781312 514 8529976023914086657 0 0 17 5 0 0 0
/proc/29810/statm: 81030 16959 376 364 0 16948 0
[pid=29812] ppid=29810 vsize=276244 CPUtime=14.02 cores=0-7
/proc/29812/stat : 29812 (hybridGM3) R 29810 29812 29812 0 -1 4202496 9517 0 0 0 1402 0 0 0 25 0 1 0 626986260 282873856 9251 33554432000 134512640 135204385 4291160096 18446744073709551615 134536993 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/29812/statm: 69061 9251 64 169 0 68891 0
[pid=29813] ppid=29804 vsize=334376 CPUtime=0.02 cores=0-7
/proc/29813/stat : 29813 (python) S 29804 29804 29518 0 -1 4202560 914 0 0 0 1 1 0 0 15 0 1 0 626986264 342401024 16957 33554432000 4194304 5683644 140734716449200 18446744073709551615 260905414214 0 0 16781312 514 8529976023914086660 0 0 17 0 0 0 0
/proc/29813/statm: 83594 16957 370 364 0 19509 0
[pid=29815] ppid=29813 vsize=276240 CPUtime=13.94 cores=0-7
/proc/29815/stat : 29815 (hybridGM3) R 29813 29815 29815 0 -1 4202496 9519 0 0 0 1393 1 0 0 25 0 1 0 626986264 282869760 9256 33554432000 134512640 135204385 4291523232 18446744073709551615 134536986 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/29815/statm: 69060 9256 64 169 0 68890 0
[pid=29816] ppid=29804 vsize=344632 CPUtime=0.02 cores=0-7
/proc/29816/stat : 29816 (python) S 29804 29804 29518 0 -1 4202560 914 0 0 0 1 1 0 0 15 0 1 0 626986268 352903168 16961 33554432000 4194304 5683644 140734716449200 18446744073709551615 260905414214 0 0 16781312 514 8529976023914086660 0 0 17 2 0 0 0
/proc/29816/statm: 86158 16961 370 364 0 22070 0
[pid=29818] ppid=29816 vsize=276244 CPUtime=13.94 cores=0-7
/proc/29818/stat : 29818 (hybridGM3) R 29816 29818 29818 0 -1 4202496 9520 0 0 0 1394 0 0 0 25 0 1 0 626986268 282873856 9252 33554432000 134512640 135204385 4294642000 18446744073709551615 134537167 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/29818/statm: 69061 9252 64 169 0 68891 0
[pid=29819] ppid=29804 vsize=354888 CPUtime=0.02 cores=0-7
/proc/29819/stat : 29819 (python) S 29804 29804 29518 0 -1 4202560 925 0 0 0 1 1 0 0 15 0 1 0 626986272 363405312 16965 33554432000 4194304 5683644 140734716449200 18446744073709551615 260905414214 0 0 16781312 514 8529976023914086660 0 0 17 4 0 0 0
/proc/29819/statm: 88722 16965 370 364 0 24631 0
[pid=29821] ppid=29819 vsize=276244 CPUtime=13.74 cores=0-7
/proc/29821/stat : 29821 (hybridGM3) R 29819 29821 29821 0 -1 4202496 9512 0 0 0 1373 1 0 0 25 0 1 0 626986272 282873856 9251 33554432000 134512640 135204385 4292233584 18446744073709551615 134537020 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/29821/statm: 69061 9251 64 169 0 68891 0
[pid=29822] ppid=29804 vsize=365144 CPUtime=0.02 cores=0-7
/proc/29822/stat : 29822 (python) S 29804 29804 29518 0 -1 4202560 945 0 0 0 1 1 0 0 15 0 1 0 626986276 373907456 16969 33554432000 4194304 5683644 140734716449200 18446744073709551615 260905414214 0 0 16781312 514 8529976023914086659 0 0 17 2 0 0 0
/proc/29822/statm: 91286 16969 370 364 0 27192 0
[pid=29824] ppid=29822 vsize=276240 CPUtime=13.75 cores=0-7
/proc/29824/stat : 29824 (hybridGM3) R 29822 29824 29824 0 -1 4202496 9530 0 0 0 1374 1 0 0 25 0 1 0 626986276 282869760 9267 33554432000 134512640 135204385 4291047248 18446744073709551615 134537383 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/29824/statm: 69060 9267 64 169 0 68890 0

################
# More data... #
################

/proc/29916/statm: 117779 8170 46 122 0 116679 0
[pid=29914/tid=29932] ppid=29804 vsize=615288 CPUtime=0 cores=0-7
/proc/29914/task/29932/stat : 29932 (python) S 29804 29804 29518 0 -1 4202560 96 0 0 0 0 0 0 0 19 0 2 0 627066346 630054912 17487 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 514 18446604453984475840 0 0 -1 1 0 0 0
[pid=29920] ppid=29804 vsize=625544 CPUtime=0.25 cores=0-7
/proc/29920/stat : 29920 (python) S 29804 29804 29518 0 -1 4202560 1186 0 0 0 15 10 0 0 19 0 2 0 627064733 640557056 17445 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/29920/statm: 156386 17445 406 364 0 92235 0
[pid=29922] ppid=29920 vsize=471116 CPUtime=99.26 cores=0-7
/proc/29922/stat : 29922 (TNM) T 29920 29922 29922 0 -1 4202496 8452 0 0 0 9925 1 0 0 25 0 1 0 627064734 482422784 8170 33554432000 134512640 135008748 4293629904 18446744073709551615 134527477 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/29922/statm: 117779 8170 46 122 0 116679 0
[pid=29920/tid=29941] ppid=29804 vsize=625544 CPUtime=0 cores=0-7
/proc/29920/task/29941/stat : 29941 (python) S 29804 29804 29518 0 -1 4202560 96 0 0 0 0 0 0 0 19 0 2 0 627074662 640557056 17445 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 514 18446604453985339200 0 0 -1 4 0 0 0
[pid=29924] ppid=29804 vsize=635800 CPUtime=0.25 cores=0-7
/proc/29924/stat : 29924 (python) S 29804 29804 29518 0 -1 4202560 1274 0 0 0 12 13 0 0 19 0 2 0 627064837 651059200 17443 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/29924/statm: 158950 17443 400 364 0 94796 0
[pid=29926] ppid=29924 vsize=471116 CPUtime=99.98 cores=0-7
/proc/29926/stat : 29926 (TNM) T 29924 29926 29926 0 -1 4202496 8456 0 0 0 9996 2 0 0 25 0 1 0 627064837 482422784 8171 33554432000 134512640 135008748 4293268288 18446744073709551615 134528491 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/29926/statm: 117779 8171 46 122 0 116679 0
[pid=29924/tid=29945] ppid=29804 vsize=635800 CPUtime=0 cores=0-7
/proc/29924/task/29945/stat : 29945 (python) S 29804 29804 29518 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 627074862 651059200 17443 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 514 18446604453853798720 0 0 -1 0 0 0 0
[pid=29928] ppid=29804 vsize=646056 CPUtime=0.25 cores=0-7
/proc/29928/stat : 29928 (python) S 29804 29804 29518 0 -1 4202560 1253 0 0 0 11 14 0 0 19 0 2 0 627065111 661561344 17448 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/29928/statm: 161514 17448 401 364 0 97357 0
[pid=29930] ppid=29928 vsize=471116 CPUtime=99.83 cores=0-7
/proc/29930/stat : 29930 (TNM) T 29928 29930 29930 0 -1 4202496 8456 0 0 0 9982 1 0 0 25 0 1 0 627065112 482422784 8171 33554432000 134512640 135008748 4288787536 18446744073709551615 134529300 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/29930/statm: 117779 8171 46 122 0 116679 0
[pid=29928/tid=29949] ppid=29804 vsize=646056 CPUtime=0 cores=0-7
/proc/29928/task/29949/stat : 29949 (python) S 29804 29804 29518 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 627075138 661561344 17448 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 514 18446604453889905920 0 0 -1 1 0 0 0
[pid=29933] ppid=29804 vsize=656312 CPUtime=0.25 cores=0-7
/proc/29933/stat : 29933 (python) S 29804 29804 29518 0 -1 4202560 1245 0 0 0 15 10 0 0 19 0 2 0 627066356 672063488 17451 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/29933/statm: 164078 17451 400 364 0 99918 0
[pid=29935] ppid=29933 vsize=471116 CPUtime=99.63 cores=0-7
/proc/29935/stat : 29935 (TNM) T 29933 29935 29935 0 -1 4202496 8458 0 0 0 9962 1 0 0 25 0 1 0 627066356 482422784 8173 33554432000 134512640 135008748 4287373840 18446744073709551615 134529300 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/29935/statm: 117779 8173 46 122 0 116679 0
[pid=29933/tid=29950] ppid=29804 vsize=656312 CPUtime=0 cores=0-7
/proc/29933/task/29950/stat : 29950 (python) S 29804 29804 29518 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 627076380 672063488 17451 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 514 18446604451397039424 0 0 -1 6 0 0 0
[pid=29938] ppid=29804 vsize=666568 CPUtime=0.3 cores=0-7
/proc/29938/stat : 29938 (python) S 29804 29804 29518 0 -1 4202560 1106 0 0 0 14 16 0 0 18 0 1 0 627074560 682565632 17426 33554432000 4194304 5683644 140734716449200 18446744073709551615 260905414214 0 0 16781312 514 8529976023914086657 0 0 17 1 0 0 0
/proc/29938/statm: 166642 17426 372 364 0 102479 0
[pid=29940] ppid=29938 vsize=438604 CPUtime=113.84 cores=0-7
/proc/29940/stat : 29940 (adaptg2wsat2009) R 29938 29940 29940 0 -1 4202496 8785 0 0 0 11382 2 0 0 25 0 1 0 627074561 449130496 8498 33554432000 134512640 135107411 4293096384 18446744073709551615 134531718 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/29940/statm: 109651 8498 59 146 0 108527 0
[pid=29942] ppid=29804 vsize=676824 CPUtime=0.27 cores=0-7
/proc/29942/stat : 29942 (python) S 29804 29804 29518 0 -1 4202560 1316 0 0 0 15 12 0 0 19 0 2 0 627074673 693067776 17465 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/29942/statm: 169206 17465 406 364 0 105040 0
[pid=29944] ppid=29942 vsize=471116 CPUtime=100 cores=0-7
/proc/29944/stat : 29944 (TNM) T 29942 29944 29944 0 -1 4202496 8462 0 0 0 9999 1 0 0 25 0 1 0 627074673 482422784 8172 33554432000 134512640 135008748 4288873824 18446744073709551615 134527416 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/29944/statm: 117779 8172 46 122 0 116679 0
[pid=29942/tid=29957] ppid=29804 vsize=676824 CPUtime=0 cores=0-7
/proc/29942/task/29957/stat : 29957 (python) S 29804 29804 29518 0 -1 4202560 87 0 0 0 0 0 0 0 19 0 2 0 627084699 693067776 17465 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 514 18446604453851828480 0 0 -1 0 0 0 0
[pid=29946] ppid=29804 vsize=687080 CPUtime=0.26 cores=0-7
/proc/29946/stat : 29946 (python) S 29804 29804 29518 0 -1 4202560 1301 0 0 0 14 12 0 0 18 0 2 0 627074873 703569920 17470 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/29946/statm: 171770 17470 407 364 0 107601 0
[pid=29948] ppid=29946 vsize=471116 CPUtime=99.69 cores=0-7
/proc/29948/stat : 29948 (TNM) T 29946 29948 29948 0 -1 4202496 8462 0 0 0 9967 2 0 0 25 0 1 0 627074874 482422784 8172 33554432000 134512640 135008748 4291918832 18446744073709551615 134529552 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/29948/statm: 117779 8172 46 122 0 116679 0
[pid=29946/tid=29961] ppid=29804 vsize=687080 CPUtime=0 cores=0-7
/proc/29946/task/29961/stat : 29961 (python) S 29804 29804 29518 0 -1 4202560 51 0 0 0 0 0 0 0 19 0 2 0 627084899 703569920 17470 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 514 18446604445241746560 0 0 -1 1 0 0 0
[pid=29951] ppid=29804 vsize=697336 CPUtime=0.24 cores=0-7
/proc/29951/stat : 29951 (python) S 29804 29804 29518 0 -1 4202560 1134 0 0 0 11 13 0 0 18 0 1 0 627076391 714072064 17443 33554432000 4194304 5683644 140734716449200 18446744073709551615 260905414214 0 0 16781312 514 8529976023914086658 0 0 17 4 0 0 0
/proc/29951/statm: 174334 17443 377 364 0 110162 0
[pid=29953] ppid=29951 vsize=471116 CPUtime=95.35 cores=0-7
/proc/29953/stat : 29953 (TNM) R 29951 29953 29953 0 -1 4202496 8462 0 0 0 9533 2 0 0 25 0 1 0 627076392 482422784 8172 33554432000 134512640 135008748 4293712336 18446744073709551615 134529475 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/29953/statm: 117779 8172 46 122 0 116679 0
[pid=29958] ppid=29804 vsize=707592 CPUtime=0.03 cores=0-7
/proc/29958/stat : 29958 (python) S 29804 29804 29518 0 -1 4202560 1154 0 0 0 2 1 0 0 18 0 1 0 627084710 724574208 17447 33554432000 4194304 5683644 140734716449200 18446744073709551615 260905414214 0 0 16781312 514 8529976023914086657 0 0 17 4 0 0 0
/proc/29958/statm: 176898 17447 377 364 0 112723 0
[pid=29960] ppid=29958 vsize=471116 CPUtime=12.47 cores=0-7
/proc/29960/stat : 29960 (TNM) R 29958 29960 29960 0 -1 4202496 8465 0 0 0 1246 1 0 0 25 0 1 0 627084711 482422784 8172 33554432000 134512640 135008748 4288580464 18446744073709551615 134528967 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/29960/statm: 117779 8172 46 122 0 116679 0
[pid=29962] ppid=29804 vsize=717848 CPUtime=0.03 cores=0-7
/proc/29962/stat : 29962 (python) S 29804 29804 29518 0 -1 4202560 1136 0 0 0 1 2 0 0 18 0 1 0 627084910 735076352 17451 33554432000 4194304 5683644 140734716449200 18446744073709551615 260905414214 0 0 16781312 514 8529976023914086659 0 0 17 0 0 0 0
/proc/29962/statm: 179462 17451 377 364 0 115284 0
[pid=29964] ppid=29962 vsize=471116 CPUtime=10.61 cores=0-7
/proc/29964/stat : 29964 (TNM) R 29962 29964 29964 0 -1 4202496 8466 0 0 0 1059 2 0 0 25 0 1 0 627084911 482422784 8170 33554432000 134512640 135008748 4294090208 18446744073709551615 134528942 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/29964/statm: 117779 8170 46 122 0 116679 0
[pid=29965] ppid=29804 vsize=728104 CPUtime=0 cores=0-7
/proc/29965/stat : 29965 (python) S 29804 29804 29518 0 -1 4202560 941 0 0 0 0 0 0 0 19 0 1 0 627085881 745578496 17423 33554432000 4194304 5683644 140734716449200 18446744073709551615 260905414214 0 0 16781312 514 4049660483702095871 0 0 17 4 0 0 0
/proc/29965/statm: 182026 17423 345 364 0 117845 0
[pid=29967] ppid=29965 vsize=471116 CPUtime=0.94 cores=0-7
/proc/29967/stat : 29967 (TNM) R 29965 29967 29967 0 -1 4202496 8464 0 0 0 93 1 0 0 23 0 1 0 627085881 482422784 8172 33554432000 134512640 135008748 4287337408 18446744073709551615 134528769 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/29967/statm: 117779 8172 46 122 0 116679 0
[pid=29804/tid=29811] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29811/stat : 29811 (python) S 29802 29804 29518 0 -1 4202560 31 935 0 0 0 0 0 0 18 0 35 0 626986260 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 5 0 0 0
[pid=29804/tid=29814] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29814/stat : 29814 (python) S 29802 29804 29518 0 -1 4202560 33 935 0 0 0 0 0 0 15 0 35 0 626986264 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 6 0 0 0
[pid=29804/tid=29817] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29817/stat : 29817 (python) S 29802 29804 29518 0 -1 4202560 30 935 0 0 0 0 0 0 18 0 35 0 626986268 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 0 0 0 0
[pid=29804/tid=29820] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29820/stat : 29820 (python) S 29802 29804 29518 0 -1 4202560 27 935 0 0 0 0 0 0 18 0 35 0 626986272 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 0 0 0 0
[pid=29804/tid=29823] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29823/stat : 29823 (python) S 29802 29804 29518 0 -1 4202560 32 935 0 0 0 0 0 0 18 0 35 0 626986276 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 4 0 0 0
[pid=29804/tid=29826] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29826/stat : 29826 (python) S 29802 29804 29518 0 -1 4202560 28 935 0 0 0 0 0 0 18 0 35 0 626986280 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 2 0 0 0
[pid=29804/tid=29829] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29829/stat : 29829 (python) S 29802 29804 29518 0 -1 4202560 33 935 0 0 0 0 0 0 18 0 35 0 626986284 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 0 0 0 0
[pid=29804/tid=29832] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29832/stat : 29832 (python) S 29802 29804 29518 0 -1 4202560 31 935 0 0 0 0 0 0 18 0 35 0 626986288 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 0 0 0 0
[pid=29804/tid=29842] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29842/stat : 29842 (python) S 29802 29804 29518 0 -1 4202560 35 935 0 0 0 0 0 0 18 0 35 0 626996315 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 3 0 0 0
[pid=29804/tid=29854] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29854/stat : 29854 (python) S 29802 29804 29518 0 -1 4202560 42 935 0 0 0 0 0 0 18 0 35 0 627015470 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 3 0 0 0
[pid=29804/tid=29857] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29857/stat : 29857 (python) S 29802 29804 29518 0 -1 4202560 32 935 0 0 0 0 0 0 15 0 35 0 627015950 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 4 0 0 0
[pid=29804/tid=29860] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29860/stat : 29860 (python) S 29802 29804 29518 0 -1 4202560 30 935 0 0 0 0 0 0 18 0 35 0 627015956 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 2 0 0 0
[pid=29804/tid=29864] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29864/stat : 29864 (python) S 29802 29804 29518 0 -1 4202560 31 935 0 0 0 0 0 0 20 0 35 0 627015983 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 0 0 0 0
[pid=29804/tid=29870] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29870/stat : 29870 (python) S 29802 29804 29518 0 -1 4202560 51 935 0 0 0 0 0 0 18 0 35 0 627025489 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 0 0 0 0
[pid=29804/tid=29873] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29873/stat : 29873 (python) S 29802 29804 29518 0 -1 4202560 51 935 0 0 0 0 0 0 18 0 35 0 627025585 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 3 0 0 0
[pid=29804/tid=29876] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29876/stat : 29876 (python) S 29802 29804 29518 0 -1 4202560 44 935 0 0 0 0 0 0 18 0 35 0 627025592 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 7 0 0 0
[pid=29804/tid=29880] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29880/stat : 29880 (python) S 29802 29804 29518 0 -1 4202560 29 935 0 0 0 0 0 0 20 0 35 0 627026015 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 7 0 0 0
[pid=29804/tid=29890] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29890/stat : 29890 (python) S 29802 29804 29518 0 -1 4202560 29 935 0 0 0 0 0 0 20 0 35 0 627036647 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 1 0 0 0
[pid=29804/tid=29896] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29896/stat : 29896 (python) S 29802 29804 29518 0 -1 4202560 146 935 0 0 0 0 0 0 20 0 35 0 627045045 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 1 0 0 0
[pid=29804/tid=29900] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29900/stat : 29900 (python) S 29802 29804 29518 0 -1 4202560 31 935 0 0 0 0 0 0 20 0 35 0 627045163 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 2 0 0 0
[pid=29804/tid=29907] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29907/stat : 29907 (python) S 29802 29804 29518 0 -1 4202560 44 935 0 0 0 0 0 0 20 0 35 0 627054700 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 1 0 0 0
[pid=29804/tid=29911] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29911/stat : 29911 (python) S 29802 29804 29518 0 -1 4202560 35 935 0 0 0 0 0 0 20 0 35 0 627055077 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 4 0 0 0
[pid=29804/tid=29915] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29915/stat : 29915 (python) S 29802 29804 29518 0 -1 4202560 41 935 0 0 0 0 0 0 20 0 35 0 627056321 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 4 0 0 0
[pid=29804/tid=29921] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29921/stat : 29921 (python) S 29802 29804 29518 0 -1 4202560 35 935 0 0 0 0 0 0 20 0 35 0 627064734 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 4 0 0 0
[pid=29804/tid=29925] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29925/stat : 29925 (python) S 29802 29804 29518 0 -1 4202560 31 935 0 0 0 0 0 0 19 0 35 0 627064837 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 4 0 0 0
[pid=29804/tid=29929] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29929/stat : 29929 (python) S 29802 29804 29518 0 -1 4202560 27 935 0 0 0 0 0 0 20 0 35 0 627065112 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 4 0 0 0
[pid=29804/tid=29934] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29934/stat : 29934 (python) S 29802 29804 29518 0 -1 4202560 25 935 0 0 0 0 0 0 20 0 35 0 627066356 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 4 0 0 0
[pid=29804/tid=29939] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29939/stat : 29939 (python) S 29802 29804 29518 0 -1 4202560 25 935 0 0 0 0 0 0 20 0 35 0 627074561 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 4 0 0 0
[pid=29804/tid=29943] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29943/stat : 29943 (python) S 29802 29804 29518 0 -1 4202560 29 935 0 0 0 0 0 0 19 0 35 0 627074673 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 4 0 0 0
[pid=29804/tid=29947] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29947/stat : 29947 (python) S 29802 29804 29518 0 -1 4202560 28 935 0 0 0 0 0 0 20 0 35 0 627074874 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 4 0 0 0
[pid=29804/tid=29952] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29952/stat : 29952 (python) S 29802 29804 29518 0 -1 4202560 27 935 0 0 0 0 0 0 20 0 35 0 627076392 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 4 0 0 0
[pid=29804/tid=29959] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29959/stat : 29959 (python) S 29802 29804 29518 0 -1 4202560 56 935 0 0 0 0 0 0 20 0 35 0 627084711 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 1 0 0 0
[pid=29804/tid=29963] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29963/stat : 29963 (python) S 29802 29804 29518 0 -1 4202560 22 935 0 0 0 0 0 0 20 0 35 0 627084911 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 1 0 0 0
[pid=29804/tid=29966] ppid=29802 vsize=738348 CPUtime=0 cores=0-7
/proc/29804/task/29966/stat : 29966 (python) S 29802 29804 29518 0 -1 4202560 25 935 0 0 0 0 0 0 20 0 35 0 627085881 756068352 19024 33554432000 4194304 5683644 140734716449200 18446744073709551615 260917217713 0 0 16781312 2 18446604453876008768 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 7957
Current children cumulated vsize (KiB) 32190460

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 29804 and gives
#  childrusage.ru_utime.tv_sec=1437
#  childrusage.ru_utime.tv_usec=148520
#  childrusage.ru_stime.tv_sec=9
#  childrusage.ru_stime.tv_usec=713523
# CPU time returned by wait4() is 1446.86
# while last known CPU time is 7957
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1008.81
CPU time (s): 7957
CPU user time (s): 7947.04
CPU system time (s): 9.96001
CPU usage (%): 788.748
Max. virtual memory (cumulated for all children) (KiB): 32190460

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1437.15
system time used= 9.71352
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 143465
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 16922
involuntary context switches= 7216

runsolver used 7.78881 second user time and 21.1988 second system time

The end

Launcher Data 

Begin job on node116 at 2011-05-02 22:19:04
IDJOB=3321799
IDBENCH=82577
IDSOLVER=1712
FILE ID=node116/3321799-1304367544
RUNJOBID= node116-1304366467-29532
PBS_JOBID= 13181755
Free space on /tmp= 73736 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v50000-c210000-S746725327-020-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3321799-1304367544/watcher-3321799-1304367544 -o /tmp/evaluation-result-3321799-1304367544/solver-3321799-1304367544 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3321799-1304367544.cnf -seed 2123509085 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 0ad997eea0def3552b6d7e45aac8ca7c
RANDOM SEED=2123509085

node116.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      26035732 kB
Buffers:         63960 kB
Cached:         869388 kB
SwapCached:      25672 kB
Active:        6031380 kB
Inactive:       697756 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      26035732 kB
SwapTotal:    67111528 kB
SwapFree:     65762296 kB
Dirty:           36184 kB
Writeback:           0 kB
AnonPages:     5780516 kB
Mapped:          11960 kB
Slab:            88940 kB
PageTables:      29476 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 53002820 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73704 MiB
End job on node116 at 2011-05-02 22:35:55