Trace number 3385381

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (MO) 11781.6 1484.03

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v20000-c84000-S1298436766-097-UNKNOWN.cnf
MD5SUM28b704f4b0f31dccf5e0c8fedfb14daa
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark85.363
Satisfiable
(Un)Satisfiability was proved
Number of variables20000
Number of clauses84000
Sum of the clauses size252000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 384000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.37/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.37/0.67	c INFO: solving HOME/instance-3385381-1305566100.cnf
0.37/0.67	c INFO: computing task features
0.64/0.96	c INFO: computed variable-clause graph statistics
0.64/0.97	c INFO: computed clause balance statistics
2.15/2.45	c INFO: computed variable balance statistics
3.35/3.67	c INFO: computed small-clause counts
3.35/3.68	c INFO: computed Horn-clause counts
3.95/4.23	c INFO: computed variable graph statistics
3.95/4.23	c INFO: collected features for HOME/instance-3385381-1305566100.cnf in 3.54 s
3.95/4.26	c INFO: running hybridGM3@0 for 97 with 4996 remaining (b = 0.43)
3.95/4.29	c INFO: running hybridGM3@0 for 97 with 4996 remaining (b = 0.26)
4.08/4.32	c INFO: running hybridGM3@0 for 97 with 4996 remaining (b = 0.19)
4.08/4.35	c INFO: running hybridGM3@0 for 194 with 4996 remaining (b = 0.26)
4.08/4.38	c INFO: running hybridGM3@0 for 194 with 4996 remaining (b = 0.21)
4.40/4.42	c INFO: running hybridGM3@0 for 194 with 4996 remaining (b = 0.16)
4.40/4.45	c INFO: running TNM@0 for 97 with 4996 remaining (b = 0.07)
4.40/4.48	c INFO: running hybridGM3@0 for 194 with 4996 remaining (b = 0.12)
805.13/104.79	c INFO: running hybridGM3@0 for 194 with 4895 remaining (b = 0.09)
806.64/104.98	c INFO: running hybridGM3@0 for 97 with 4895 remaining (b = 0.04)
807.32/105.01	c INFO: running hybridGM3@0 for 97 with 4895 remaining (b = 0.04)
807.32/105.06	c INFO: running hybridGM3@0 for 97 with 4895 remaining (b = 0.04)
1567.04/200.28	c INFO: running hybridGM3@100 for 97 with 4800 remaining (b = 0.04)
1567.70/200.31	c INFO: running hybridGM3@100 for 97 with 4800 remaining (b = 0.04)
1567.70/200.36	c INFO: running hybridGM3@95 for 97 with 4800 remaining (b = 0.04)
1575.77/201.36	c INFO: running hybridGM3@0 for 97 with 4799 remaining (b = 0.04)
1575.77/201.39	c INFO: running hybridGM3@0 for 97 with 4799 remaining (b = 0.04)
1577.24/201.51	c INFO: running hybridGM3@100 for 97 with 4799 remaining (b = 0.04)
1623.52/207.37	c INFO: running hybridGM3@0 for 97 with 4793 remaining (b = 0.04)
2335.86/296.66	c INFO: running gnovelty+2@0 for 97 with 4704 remaining (b = 0.04)
2336.60/296.70	c INFO: running hybridGM3@0 for 194 with 4704 remaining (b = 0.07)
2374.89/301.60	c INFO: running hybridGM3@0 for 194 with 4699 remaining (b = 0.05)
2375.65/301.64	c INFO: running hybridGM3@0 for 194 with 4699 remaining (b = 0.04)
2376.30/301.72	c INFO: running adaptg2wsat2009++@0 for 97 with 4698 remaining (b = 0.02)
2376.86/301.83	c INFO: running hybridGM3@100 for 97 with 4698 remaining (b = 0.02)
2376.86/301.88	c INFO: running hybridGM3@291 for 97 with 4698 remaining (b = 0.02)
2383.54/302.69	c INFO: running hybridGM3@299 for 97 with 4698 remaining (b = 0.02)
3136.55/397.08	c INFO: running TNM@0 for 97 with 4603 remaining (b = 0.02)
3137.37/397.16	c INFO: running hybridGM3@95 for 97 with 4603 remaining (b = 0.02)
3137.96/397.20	c INFO: running hybridGM3@386 for 97 with 4603 remaining (b = 0.02)
3143.53/398.01	c INFO: running hybridGM3@399 for 97 with 4602 remaining (b = 0.01)
3176.25/402.06	c INFO: running TNM@0 for 97 with 4598 remaining (b = 0.02)
3897.25/492.52	c INFO: running hybridGM3@482 for 97 with 4508 remaining (b = 0.01)
3906.82/493.73	c INFO: running hybridGM3@99 for 97 with 4506 remaining (b = 0.02)
3912.39/494.49	c INFO: running hybridGM3@190 for 97 with 4506 remaining (b = 0.02)
3929.92/496.65	c INFO: running hybridGM3@101 for 97 with 4504 remaining (b = 0.02)
3936.20/497.42	c INFO: running TNM@0 for 97 with 4503 remaining (b = 0.02)
3946.52/498.70	c INFO: running hybridGM3@101 for 97 with 4502 remaining (b = 0.02)
3951.36/499.36	c INFO: running TNM@0 for 97 with 4501 remaining (b = 0.01)
4015.80/507.46	c INFO: running TNM@0 for 97 with 4493 remaining (b = 0.01)
4656.33/587.86	c INFO: running TNM@0 for 1940 with 4412 remaining (b = 0.24)
4672.29/589.81	c INFO: running hybridGM3@385 for 97 with 4410 remaining (b = 0.02)
4713.78/595.07	c INFO: running hybridGM3@0 for 194 with 4405 remaining (b = 0.03)
4729.60/597.03	c INFO: running hybridGM3@0 for 194 with 4403 remaining (b = 0.02)
4735.96/597.80	c INFO: running TNM@196 for 1843 with 4402 remaining (b = 0.18)
4751.07/599.73	c INFO: running adaptg2wsat2009++@0 for 194 with 4400 remaining (b = 0.02)
4753.16/600.05	c INFO: running hybridGM3@0 for 194 with 4400 remaining (b = 0.02)
4815.49/607.83	c INFO: running adaptg2wsat2009++@0 for 194 with 4392 remaining (b = 0.02)
5478.32/691.18	c INFO: running hybridGM3@0 for 194 with 4309 remaining (b = 0.02)
6274.76/791.22	c INFO: running hybridGM3@100 for 97 with 4209 remaining (b = 0.01)
6274.76/791.28	c INFO: running hybridGM3@99 for 97 with 4209 remaining (b = 0.01)
6315.34/796.39	c INFO: running FH@0 for 97 with 4204 remaining (b = 0.01)
6337.53/799.20	c INFO: running hybridGM3@99 for 97 with 4201 remaining (b = 0.01)
6419.49/809.51	c INFO: running adaptg2wsat2009++@97 for 97 with 4191 remaining (b = 0.01)
7046.46/888.26	c INFO: running adaptg2wsat2009++@97 for 97 with 4112 remaining (b = 0.01)
7065.63/890.66	c INFO: running hybridGM3@296 for 97 with 4110 remaining (b = 0.01)
7081.47/892.60	c INFO: running hybridGM3@296 for 97 with 4108 remaining (b = 0.01)
7096.59/894.55	c INFO: running hybridGM3@294 for 97 with 4106 remaining (b = 0.01)
7114.08/896.77	c INFO: running adaptg2wsat2009++@0 for 194 with 4103 remaining (b = 0.01)
7194.45/906.89	c INFO: running TNM@0 for 97 with 4093 remaining (b = 0.01)
7839.67/987.95	c INFO: running hybridGM3@390 for 97 with 4012 remaining (b = 0.01)
7856.50/990.05	c INFO: running hybridGM3@397 for 97 with 4010 remaining (b = 0.01)
7869.08/991.65	c INFO: running hybridGM3@196 for 291 with 4009 remaining (b = 0.02)
7903.28/995.91	c INFO: running hybridGM3@395 for 97 with 4004 remaining (b = 0.01)
7993.21/1007.27	c INFO: running TNM@94 for 1455 with 3993 remaining (b = 0.12)
8597.90/1083.30	c INFO: running hybridGM3@0 for 194 with 3917 remaining (b = 0.01)
8647.16/1089.45	c INFO: running TNM@95 for 1455 with 3911 remaining (b = 0.09)
8661.57/1091.26	c INFO: running hybridGM3@0 for 194 with 3909 remaining (b = 0.01)
8670.99/1092.44	c INFO: running FH@0 for 97 with 3908 remaining (b = 0.01)
9468.91/1192.84	c INFO: running adaptg2wsat2009++@293 for 291 with 3807 remaining (b = 0.02)
10222.29/1287.66	c INFO: running hybridGM3@297 for 194 with 3713 remaining (b = 0.01)
10227.84/1288.38	c INFO: running gnovelty+2@0 for 97 with 3712 remaining (b = 0.01)
10260.48/1292.46	c INFO: running hybridGM3@94 for 97 with 3708 remaining (b = 0.01)
11018.56/1387.84	c INFO: running hybridGM3@189 for 97 with 3612 remaining (b = 0.01)
11066.24/1393.90	c INFO: running hybridGM3@0 for 194 with 3606 remaining (b = 0.01)
11745.73/1479.32	c INFO: running FH@0 for 97 with 3521 remaining (b = 0.00)
11773.60/1482.83	c INFO: running NCVWr@0 for 485 with 3517 remaining (b = 0.02)
11781.63/1483.98	c INFO: running hybridGM3@195 for 291 with 3516 remaining (b = 0.01)
11781.63/1483.98	Traceback (most recent call last):
11781.63/1483.98	  File "HOME/solve", line 33, in <module>
11781.63/1483.98	    plac.call(main)
11781.63/1483.98	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
11781.63/1483.98	    cmd, result = parser_from(obj).consume(arglist)
11781.63/1483.98	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
11781.63/1483.98	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
11781.63/1483.98	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
11781.63/1483.98	    (_, answer, _) = solver(input_path, budget / cores, cores)
11781.63/1483.98	  File "HOME/borg/build/lib/borg/bilevel.py", line 202, in __call__
11781.63/1483.98	    solver.go(max_cost * borg.defaults.machine_speed)
11781.63/1483.98	  File "HOME/borg/build/lib/borg/solvers.py", line 179, in go
11781.63/1483.98	    self._process.start()
11781.63/1483.98	  File "/home/evaluation/evaluation/tools64/lib/python2.7/multiprocessing/process.py", line 94, in start
11781.63/1483.98	    assert self._popen is None, 'cannot start a process twice'
11781.63/1483.98	AssertionError: cannot start a process twice

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3385381-1305566100/watcher-3385381-1305566100 -o /tmp/evaluation-result-3385381-1305566100/solver-3385381-1305566100 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385381-1305566100.cnf -seed 893259170 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.98 7.90 5.69 1/374 14599
/proc/meminfo: memFree=18411096/32950928 swapFree=67042944/67111528
[pid=14599] ppid=14597 vsize=15380 CPUtime=0 cores=0-7
/proc/14599/stat : 14599 (python) S 14597 14599 14192 0 -1 4202496 817 0 0 0 0 0 0 0 17 0 1 0 668209733 15749120 703 33554432000 4194304 5683644 140736277217920 18446744073709551615 274473965568 0 2147483391 16781312 2 18446744071699215550 0 0 17 4 0 0 0
/proc/14599/statm: 3845 703 334 364 0 332 0

[startup+0.057789 s]
/proc/loadavg: 7.98 7.90 5.69 1/374 14599
/proc/meminfo: memFree=18411096/32950928 swapFree=67042944/67111528
[pid=14599] ppid=14597 vsize=29784 CPUtime=0.03 cores=0-7
/proc/14599/stat : 14599 (python) R 14597 14599 14192 0 -1 4202496 1406 0 0 0 3 0 0 0 17 0 1 0 668209733 30498816 1251 33554432000 4194304 5683644 140736277217920 18446744073709551615 4514970 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/14599/statm: 7446 1251 443 364 0 816 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 29784

[startup+0.100821 s]
/proc/loadavg: 7.98 7.90 5.69 1/374 14599
/proc/meminfo: memFree=18411096/32950928 swapFree=67042944/67111528
[pid=14599] ppid=14597 vsize=87700 CPUtime=0.07 cores=0-7
/proc/14599/stat : 14599 (python) S 14597 14599 14192 0 -1 4202496 2635 0 0 0 5 2 0 0 17 0 1 0 668209733 89804800 2353 33554432000 4194304 5683644 140736277217920 18446744073709551615 274473965568 0 2147483391 16781312 2 18446744071699215550 0 0 17 4 0 0 0
/proc/14599/statm: 21925 2353 717 364 0 1621 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 87700

[startup+0.300785 s]
/proc/loadavg: 7.98 7.90 5.69 1/374 14599
/proc/meminfo: memFree=18411096/32950928 swapFree=67042944/67111528
[pid=14599] ppid=14597 vsize=135744 CPUtime=0.19 cores=0-7
/proc/14599/stat : 14599 (python) S 14597 14599 14192 0 -1 4202496 5960 932 1 0 14 5 0 0 17 0 1 0 668209733 139001856 4150 33554432000 4194304 5683644 140736277217920 18446744073709551615 274473963525 0 2147483391 16781312 2 18446744071699215550 0 0 17 4 0 0 0
/proc/14599/statm: 33936 4150 1032 364 0 3122 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 135744

[startup+0.700675 s]
/proc/loadavg: 7.98 7.90 5.69 1/374 14599
/proc/meminfo: memFree=18411096/32950928 swapFree=67042944/67111528
[pid=14599] ppid=14597 vsize=281236 CPUtime=0.45 cores=0-7
/proc/14599/stat : 14599 (python) R 14597 14599 14192 0 -1 4202496 9946 932 1 0 33 12 0 0 16 0 1 0 668209733 287985664 7631 33554432000 4194304 5683644 140736277217920 18446744073709551615 4445115 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/14599/statm: 70309 7631 1761 364 0 6233 0
Current children cumulated CPU time (s) 0.45
Current children cumulated vsize (KiB) 281236

[startup+1.50046 s]
/proc/loadavg: 7.98 7.90 5.69 2/375 14604
/proc/meminfo: memFree=18370664/32950928 swapFree=67042944/67111528
[pid=14599] ppid=14597 vsize=297692 CPUtime=1.25 cores=0-7
/proc/14599/stat : 14599 (python) R 14597 14599 14192 0 -1 4202496 16352 932 1 0 112 13 0 0 17 0 1 0 668209733 304836608 11808 33554432000 4194304 5683644 140736277217920 18446744073709551615 5003923 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/14599/statm: 74423 11808 1826 364 0 10347 0
Current children cumulated CPU time (s) 1.25
Current children cumulated vsize (KiB) 297692

[startup+3.10018 s]
/proc/loadavg: 7.42 7.79 5.67 2/375 14604
/proc/meminfo: memFree=18369920/32950928 swapFree=67042944/67111528
[pid=14599] ppid=14597 vsize=297692 CPUtime=2.85 cores=0-7
/proc/14599/stat : 14599 (python) R 14597 14599 14192 0 -1 4202496 16353 932 1 0 272 13 0 0 19 0 1 0 668209733 304836608 11809 33554432000 4194304 5683644 140736277217920 18446744073709551615 274473616300 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/14599/statm: 74423 11809 1827 364 0 10347 0
Current children cumulated CPU time (s) 2.85
Current children cumulated vsize (KiB) 297692

[startup+6.30162 s]
/proc/loadavg: 7.42 7.79 5.67 9/399 14628
/proc/meminfo: memFree=18159536/32950928 swapFree=67042944/67111528
[pid=14599] ppid=14597 vsize=434916 CPUtime=4.22 cores=0-7
/proc/14599/stat : 14599 (python) S 14597 14599 14192 0 -1 4202496 22396 932 1 0 407 15 0 0 25 0 9 0 668209733 445353984 9457 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485794251 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/14599/statm: 108729 9457 1889 364 0 44626 0
[pid=14605] ppid=14599 vsize=287828 CPUtime=0 cores=0-7
/proc/14605/stat : 14605 (python) S 14599 14599 14192 0 -1 4202560 733 0 0 0 0 0 0 0 15 0 1 0 668210159 294735872 7610 33554432000 4194304 5683644 140736277217920 18446744073709551615 274473987599 0 0 16781312 514 6749444490374643970 0 0 17 1 0 0 0
/proc/14605/statm: 71957 7610 355 364 0 7875 0
[pid=14607] ppid=14605 vsize=268424 CPUtime=2.02 cores=0-7
/proc/14607/stat : 14607 (hybridGM3) R 14605 14607 14607 0 -1 4202496 6662 0 0 0 202 0 0 0 25 0 1 0 668210159 274866176 6397 33554432000 134512640 135204385 4290722144 18446744073709551615 134537565 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/14607/statm: 67106 6397 64 169 0 66936 0
[pid=14608] ppid=14599 vsize=297848 CPUtime=0 cores=0-7
/proc/14608/stat : 14608 (python) S 14599 14599 14192 0 -1 4202560 746 0 0 0 0 0 0 0 15 0 1 0 668210162 304996352 7743 33554432000 4194304 5683644 140736277217920 18446744073709551615 274473987654 0 0 16781312 514 6749436793793249539 0 0 17 4 0 0 0
/proc/14608/statm: 74462 7743 349 364 0 10377 0
[pid=14610] ppid=14608 vsize=268420 CPUtime=2 cores=0-7
/proc/14610/stat : 14610 (hybridGM3) R 14608 14610 14610 0 -1 4202496 6658 0 0 0 199 1 0 0 25 0 1 0 668210163 274862080 6397 33554432000 134512640 135204385 4289860288 18446744073709551615 134537483 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/14610/statm: 67105 6397 64 169 0 66935 0
[pid=14611] ppid=14599 vsize=308036 CPUtime=0 cores=0-7
/proc/14611/stat : 14611 (python) S 14599 14599 14192 0 -1 4202560 746 0 0 0 0 0 0 0 18 0 1 0 668210165 315428864 7747 33554432000 4194304 5683644 140736277217920 18446744073709551615 274473987654 0 0 16781312 514 878344593265885442 0 0 17 1 0 0 0
/proc/14611/statm: 77009 7747 349 364 0 12921 0
[pid=14613] ppid=14611 vsize=268424 CPUtime=1.96 cores=0-7
/proc/14613/stat : 14613 (hybridGM3) R 14611 14613 14613 0 -1 4202496 6662 0 0 0 195 1 0 0 25 0 1 0 668210166 274866176 6396 33554432000 134512640 135204385 4292482112 18446744073709551615 134536853 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/14613/statm: 67106 6396 64 169 0 66936 0
[pid=14614] ppid=14599 vsize=318900 CPUtime=0 cores=0-7
/proc/14614/stat : 14614 (python) S 14599 14599 14192 0 -1 4202560 756 0 0 0 0 0 0 0 18 0 1 0 668210169 326553600 7751 33554432000 4194304 5683644 140736277217920 18446744073709551615 274473987654 0 0 16781312 514 12842480259794174211 0 0 17 1 0 0 0
/proc/14614/statm: 79725 7751 349 364 0 15634 0
[pid=14616] ppid=14614 vsize=268428 CPUtime=1.93 cores=0-7
/proc/14616/stat : 14616 (hybridGM3) R 14614 14616 14616 0 -1 4202496 6653 0 0 0 193 0 0 0 25 0 1 0 668210169 274870272 6393 33554432000 134512640 135204385 4292974976 18446744073709551615 134536993 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/14616/statm: 67107 6393 64 169 0 66937 0
[pid=14617] ppid=14599 vsize=393904 CPUtime=0 cores=0-7
/proc/14617/stat : 14617 (python) S 14599 14599 14192 0 -1 4202560 725 0 0 0 0 0 0 0 18 0 1 0 668210172 403357696 7926 33554432000 4194304 5683644 140736277217920 18446744073709551615 274473987654 0 0 16781312 514 12842484108084871426 0 0 17 1 0 0 0
/proc/14617/statm: 98476 7926 348 364 0 34382 0
[pid=14619] ppid=14617 vsize=268420 CPUtime=1.88 cores=0-7
/proc/14619/stat : 14619 (hybridGM3) R 14617 14619 14619 0 -1 4202496 6650 0 0 0 188 0 0 0 25 0 1 0 668210172 274862080 6395 33554432000 134512640 135204385 4290628512 18446744073709551615 134536744 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/14619/statm: 67105 6395 64 169 0 66935 0
[pid=14620] ppid=14599 vsize=404160 CPUtime=0 cores=0-7
/proc/14620/stat : 14620 (python) S 14599 14599 14192 0 -1 4202560 733 0 0 0 0 0 0 0 18 0 1 0 668210175 413859840 7893 33554432000 4194304 5683644 140736277217920 18446744073709551615 274473987654 0 0 16781312 514 4174401727140692226 0 0 17 0 0 0 0
/proc/14620/statm: 101040 7893 348 364 0 36943 0
[pid=14622] ppid=14620 vsize=268424 CPUtime=1.87 cores=0-7
/proc/14622/stat : 14622 (hybridGM3) R 14620 14622 14622 0 -1 4202496 6655 0 0 0 186 1 0 0 25 0 1 0 668210175 274866176 6396 33554432000 134512640 135204385 4294319920 18446744073709551615 134546086 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/14622/statm: 67106 6396 64 169 0 66936 0
[pid=14623] ppid=14599 vsize=414416 CPUtime=0 cores=0-7
/proc/14623/stat : 14623 (python) S 14599 14599 14192 0 -1 4202560 735 0 0 0 0 0 0 0 18 0 1 0 668210178 424361984 7899 33554432000 4194304 5683644 140736277217920 18446744073709551615 274473987654 0 0 16781312 514 4100504100393713663 0 0 17 1 0 0 0
/proc/14623/statm: 103604 7899 348 364 0 39504 0
[pid=14625] ppid=14623 vsize=458972 CPUtime=1.81 cores=0-7
/proc/14625/stat : 14625 (TNM) R 14623 14625 14625 0 -1 4202496 3585 0 0 0 181 0 0 0 25 0 1 0 668210180 469987328 3328 33554432000 134512640 135008748 4287433152 18446744073709551615 134527847 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/14625/statm: 114743 3328 46 122 0 113643 0
[pid=14626] ppid=14599 vsize=424672 CPUtime=0 cores=0-7
/proc/14626/stat : 14626 (python) S 14599 14599 14192 0 -1 4202560 769 0 0 0 0 0 0 0 18 0 1 0 668210181 434864128 7918 33554432000 4194304 5683644 140736277217920 18446744073709551615 274473987654 0 0 16781312 514 2844662260113047810 0 0 17 1 0 0 0
/proc/14626/statm: 106168 7918 354 364 0 42065 0
[pid=14628] ppid=14626 vsize=268428 CPUtime=1.78 cores=0-7
/proc/14628/stat : 14628 (hybridGM3) R 14626 14628 14628 0 -1 4202496 6661 0 0 0 177 1 0 0 25 0 1 0 668210184 274870272 6397 33554432000 134512640 135204385 4291045872 18446744073709551615 134660101 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/14628/statm: 67107 6397 64 169 0 66937 0
[pid=14599/tid=14606] ppid=14597 vsize=434916 CPUtime=0 cores=0-7
/proc/14599/task/14606/stat : 14606 (python) S 14597 14599 14192 0 -1 4202560 25 932 0 0 0 0 0 0 23 0 9 0 668210159 445353984 9457 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 6 0 0 0
[pid=14599/tid=14609] ppid=14597 vsize=434916 CPUtime=0 cores=0-7
/proc/14599/task/14609/stat : 14609 (python) S 14597 14599 14192 0 -1 4202560 24 932 0 0 0 0 0 0 20 0 9 0 668210162 445353984 9457 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 6 0 0 0
[pid=14599/tid=14612] ppid=14597 vsize=434916 CPUtime=0 cores=0-7
/proc/14599/task/14612/stat : 14612 (python) S 14597 14599 14192 0 -1 4202560 25 932 0 0 0 0 0 0 21 0 9 0 668210166 445353984 9457 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 2 0 0 0
[pid=14599/tid=14615] ppid=14597 vsize=434916 CPUtime=0 cores=0-7
/proc/14599/task/14615/stat : 14615 (python) S 14597 14599 14192 0 -1 4202560 26 932 0 0 0 0 0 0 22 0 9 0 668210169 445353984 9457 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 3 0 0 0
[pid=14599/tid=14618] ppid=14597 vsize=434916 CPUtime=0 cores=0-7
/proc/14599/task/14618/stat : 14618 (python) S 14597 14599 14192 0 -1 4202560 86 932 0 0 0 0 0 0 23 0 9 0 668210172 445353984 9457 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 2 0 0 0
[pid=14599/tid=14621] ppid=14597 vsize=434916 CPUtime=0 cores=0-7
/proc/14599/task/14621/stat : 14621 (python) S 14597 14599 14192 0 -1 4202560 30 932 0 0 0 0 0 0 25 0 9 0 668210175 445353984 9457 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 0 0 0 0
[pid=14599/tid=14624] ppid=14597 vsize=434916 CPUtime=0 cores=0-7
/proc/14599/task/14624/stat : 14624 (python) S 14597 14599 14192 0 -1 4202560 31 932 0 0 0 0 0 0 25 0 9 0 668210178 445353984 9457 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 2 0 0 0
[pid=14599/tid=14627] ppid=14597 vsize=434916 CPUtime=0 cores=0-7
/proc/14599/task/14627/stat : 14627 (python) S 14597 14599 14192 0 -1 4202560 34 932 0 0 0 0 0 0 25 0 9 0 668210181 445353984 9457 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 19.47
Current children cumulated vsize (KiB) 5622620

[startup+12.7015 s]
/proc/loadavg: 7.51 7.79 5.69 9/399 14628
/proc/meminfo: memFree=18158180/32950928 swapFree=67042944/67111528
[pid=14599] ppid=14597 vsize=434916 CPUtime=4.22 cores=0-7
/proc/14599/stat : 14599 (python) S 14597 14599 14192 0 -1 4202496 22396 932 1 0 407 15 0 0 25 0 9 0 668209733 445353984 9457 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485794251 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/14599/statm: 108729 9457 1889 364 0 44626 0
[pid=14605] ppid=14599 vsize=287828 CPUtime=0.01 cores=0-7
/proc/14605/stat : 14605 (python) S 14599 14599 14192 0 -1 4202560 899 0 0 0 0 1 0 0 15 0 1 0 668210159 294735872 7633 33554432000 4194304 5683644 140736277217920 18446744073709551615 274473987599 0 0 16781312 514 6749444490374643970 0 0 17 1 0 0 0
/proc/14605/statm: 71957 7633 378 364 0 7875 0
[pid=14607] ppid=14605 vsize=268424 CPUtime=8.4 cores=0-7
/proc/14607/stat : 14607 (hybridGM3) R 14605 14607 14607 0 -1 4202496 6662 0 0 0 840 0 0 0 25 0 1 0 668210159 274866176 6397 33554432000 134512640 135204385 4290722144 18446744073709551615 134545979 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/14607/statm: 67106 6397 64 169 0 66936 0
[pid=14608] ppid=14599 vsize=297848 CPUtime=0.01 cores=0-7
/proc/14608/stat : 14608 (python) S 14599 14599 14192 0 -1 4202560 909 0 0 0 1 0 0 0 15 0 1 0 668210162 304996352 7765 33554432000 4194304 5683644 140736277217920 18446744073709551615 274473987654 0 0 16781312 514 6749436793793249539 0 0 17 4 0 0 0
/proc/14608/statm: 74462 7765 371 364 0 10377 0
[pid=14610] ppid=14608 vsize=268420 CPUtime=8.4 cores=0-7
/proc/14610/stat : 14610 (hybridGM3) R 14608 14610 14610 0 -1 4202496 6658 0 0 0 839 1 0 0 25 0 1 0 668210163 274862080 6397 33554432000 134512640 135204385 4289860288 18446744073709551615 134660134 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/14610/statm: 67105 6397 64 169 0 66935 0
[pid=14611] ppid=14599 vsize=308036 CPUtime=0.01 cores=0-7
/proc/14611/stat : 14611 (python) S 14599 14599 14192 0 -1 4202560 905 0 0 0 1 0 0 0 18 0 1 0 668210165 315428864 7769 33554432000 4194304 5683644 140736277217920 18446744073709551615 274473987654 0 0 16781312 514 878344593265885442 0 0 17 1 0 0 0
/proc/14611/statm: 77009 7769 371 364 0 12921 0
[pid=14613] ppid=14611 vsize=268424 CPUtime=8.36 cores=0-7
/proc/14613/stat : 14613 (hybridGM3) R 14611 14613 14613 0 -1 4202496 6664 0 0 0 835 1 0 0 25 0 1 0 668210166 274866176 6398 33554432000 134512640 135204385 4292482112 18446744073709551615 134537403 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/14613/statm: 67106 6398 64 169 0 66936 0
[pid=14614] ppid=14599 vsize=318900 CPUtime=0.01 cores=0-7
/proc/14614/stat : 14614 (python) S 14599 14599 14192 0 -1 4202560 927 0 0 0 0 1 0 0 18 0 1 0 668210169 326553600 7773 33554432000 4194304 5683644 140736277217920 18446744073709551615 274473987654 0 0 16781312 514 12842480259794174211 0 0 17 1 0 0 0
/proc/14614/statm: 79725 7773 371 364 0 15634 0
[pid=14616] ppid=14614 vsize=268428 CPUtime=8.33 cores=0-7
/proc/14616/stat : 14616 (hybridGM3) R 14614 14616 14616 0 -1 4202496 6653 0 0 0 833 0 0 0 25 0 1 0 668210169 274870272 6393 33554432000 134512640 135204385 4292974976 18446744073709551615 134536887 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/14616/statm: 67107 6393 64 169 0 66937 0
[pid=14617] ppid=14599 vsize=393904 CPUtime=0.01 cores=0-7
/proc/14617/stat : 14617 (python) S 14599 14599 14192 0 -1 4202560 802 0 0 0 0 1 0 0 18 0 1 0 668210172 403357696 7949 33554432000 4194304 5683644 140736277217920 18446744073709551615 274473987654 0 0 16781312 514 12842484108084871426 0 0 17 1 0 0 0
/proc/14617/statm: 98476 7949 371 364 0 34382 0
[pid=14619] ppid=14617 vsize=268420 CPUtime=8.17 cores=0-7
/proc/14619/stat : 14619 (hybridGM3) R 14617 14619 14619 0 -1 4202496 6652 0 0 0 817 0 0 0 25 0 1 0 668210172 274862080 6397 33554432000 134512640 135204385 4290628512 18446744073709551615 134536641 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/14619/statm: 67105 6397 64 169 0 66935 0
[pid=14620] ppid=14599 vsize=404160 CPUtime=0.01 cores=0-7
/proc/14620/stat : 14620 (python) S 14599 14599 14192 0 -1 4202560 810 0 0 0 0 1 0 0 18 0 1 0 668210175 413859840 7916 33554432000 4194304 5683644 140736277217920 18446744073709551615 274473987654 0 0 16781312 514 4174401727140692226 0 0 17 0 0 0 0
/proc/14620/statm: 101040 7916 371 364 0 36943 0
[pid=14622] ppid=14620 vsize=268424 CPUtime=8.25 cores=0-7
/proc/14622/stat : 14622 (hybridGM3) R 14620 14622 14622 0 -1 4202496 6655 0 0 0 824 1 0 0 25 0 1 0 668210175 274866176 6396 33554432000 134512640 135204385 4294319920 18446744073709551615 134536737 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/14622/statm: 67106 6396 64 169 0 66936 0
[pid=14623] ppid=14599 vsize=414416 CPUtime=0.01 cores=0-7
/proc/14623/stat : 14623 (python) S 14599 14599 14192 0 -1 4202560 819 0 0 0 0 1 0 0 18 0 1 0 668210178 424361984 7921 33554432000 4194304 5683644 140736277217920 18446744073709551615 274473987654 0 0 16781312 514 4100504100393681156 0 0 17 1 0 0 0
/proc/14623/statm: 103604 7921 370 364 0 39504 0

################
# More data... #
################

[pid=14716] ppid=14714 vsize=268424 CPUtime=195.77 cores=0-7
/proc/14716/stat : 14716 (hybridGM3) T 14714 14716 14716 0 -1 4202496 6685 0 0 0 19577 0 0 0 25 0 1 0 668278852 274866176 6398 33554432000 134512640 135204385 4293155808 18446744073709551615 134537249 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/14716/statm: 67106 6398 64 169 0 66936 0
[pid=14714/tid=14730] ppid=14599 vsize=599024 CPUtime=0 cores=0-7
/proc/14714/task/14730/stat : 14730 (python) S 14599 14599 14192 0 -1 4202560 84 0 0 0 0 0 0 0 17 0 2 0 668298554 613400576 8333 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 514 18446604471164520512 0 0 -1 1 0 0 0
[pid=14723] ppid=14599 vsize=609280 CPUtime=0.32 cores=0-7
/proc/14723/stat : 14723 (python) S 14599 14599 14192 0 -1 4202560 1190 0 0 0 17 15 0 0 19 0 2 0 668289372 623902720 8380 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485794251 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/14723/statm: 152320 8380 400 364 0 88163 0
[pid=14725] ppid=14723 vsize=254324 CPUtime=100.26 cores=0-7
/proc/14725/stat : 14725 (FH) T 14723 14725 14725 0 -1 4202496 3691 0 0 0 10026 0 0 0 25 0 1 0 668289373 260427776 3409 33554432000 134512640 134936988 4292370720 18446744073709551615 134552617 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/14725/statm: 63581 3409 41 104 0 63085 0
[pid=14723/tid=14732] ppid=14599 vsize=609280 CPUtime=0 cores=0-7
/proc/14723/task/14732/stat : 14732 (python) S 14599 14599 14192 0 -1 4202560 85 0 0 0 0 0 0 0 19 0 2 0 668299404 623902720 8380 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 514 18446604465291398016 0 0 -1 3 0 0 0
[pid=14733] ppid=14599 vsize=619536 CPUtime=0.62 cores=0-7
/proc/14733/stat : 14733 (python) S 14599 14599 14192 0 -1 4202560 1196 0 0 0 31 31 0 0 19 0 2 0 668299410 634404864 8340 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485794251 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/14733/statm: 154884 8340 400 364 0 90724 0
[pid=14735] ppid=14733 vsize=425004 CPUtime=195.03 cores=0-7
/proc/14735/stat : 14735 (adaptg2wsat2009) T 14733 14735 14735 0 -1 4202496 3749 0 0 0 19503 0 0 0 25 0 1 0 668299411 435204096 3465 33554432000 134512640 135107411 4293341232 18446744073709551615 134547568 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/14735/statm: 106251 3465 59 146 0 105127 0
[pid=14733/tid=14751] ppid=14599 vsize=619536 CPUtime=0 cores=0-7
/proc/14733/task/14751/stat : 14751 (python) S 14599 14599 14192 0 -1 4202560 88 0 0 0 0 0 0 0 19 0 2 0 668318971 634404864 8340 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 514 18446604460818139200 0 0 -1 3 0 0 0
[pid=14736] ppid=14599 vsize=629792 CPUtime=0.32 cores=0-7
/proc/14736/stat : 14736 (python) S 14599 14599 14192 0 -1 4202560 1199 0 0 0 18 14 0 0 19 0 2 0 668300423 644907008 8421 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485794251 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/14736/statm: 157448 8421 400 364 0 93285 0
[pid=14738] ppid=14736 vsize=458972 CPUtime=100.28 cores=0-7
/proc/14738/stat : 14738 (TNM) T 14736 14738 14738 0 -1 4202496 3613 0 0 0 10028 0 0 0 25 0 1 0 668300423 469987328 3328 33554432000 134512640 135008748 4290009312 18446744073709551615 134524772 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/14738/statm: 114743 3328 46 122 0 113643 0
[pid=14736/tid=14742] ppid=14599 vsize=629792 CPUtime=0 cores=0-7
/proc/14736/task/14742/stat : 14742 (python) S 14599 14599 14192 0 -1 4202560 87 0 0 0 0 0 0 0 19 0 2 0 668310454 644907008 8421 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 514 18446604460818138304 0 0 -1 7 0 0 0
[pid=14745] ppid=14599 vsize=640048 CPUtime=1.33 cores=0-7
/proc/14745/stat : 14745 (python) S 14599 14599 14192 0 -1 4202560 1272 0 0 0 63 70 0 0 15 0 2 0 668318063 655409152 8335 33554432000 4194304 5683644 140736277217920 18446744073709551615 274473987654 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/14745/statm: 160012 8335 407 364 0 95846 0
[pid=14747] ppid=14745 vsize=268420 CPUtime=383.17 cores=0-7
/proc/14747/stat : 14747 (hybridGM3) R 14745 14747 14747 0 -1 4202496 6687 0 0 0 38316 1 0 0 25 0 1 0 668318064 274862080 6403 33554432000 134512640 135204385 4288355776 18446744073709551615 134537483 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/14747/statm: 67105 6403 64 169 0 66935 0
[pid=14745/tid=14765] ppid=14599 vsize=640048 CPUtime=0 cores=0-7
/proc/14745/task/14765/stat : 14765 (python) S 14599 14599 14192 0 -1 4202560 87 0 0 0 0 0 0 0 15 0 2 0 668338973 655409152 8335 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 514 18446604459768302272 0 0 -1 5 0 0 0
[pid=14748] ppid=14599 vsize=650304 CPUtime=0.66 cores=0-7
/proc/14748/stat : 14748 (python) S 14599 14599 14192 0 -1 4202560 1243 0 0 0 34 32 0 0 17 0 2 0 668318860 665911296 8344 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485794251 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/14748/statm: 162576 8344 407 364 0 98407 0
[pid=14750] ppid=14748 vsize=268424 CPUtime=196.37 cores=0-7
/proc/14750/stat : 14750 (hybridGM3) T 14748 14750 14750 0 -1 4202496 6685 0 0 0 19636 1 0 0 25 0 1 0 668318860 274866176 6399 33554432000 134512640 135204385 4294583664 18446744073709551615 134545992 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/14750/statm: 67106 6399 64 169 0 66936 0
[pid=14748/tid=14761] ppid=14599 vsize=650304 CPUtime=0 cores=0-7
/proc/14748/task/14761/stat : 14761 (python) S 14599 14599 14192 0 -1 4202560 94 0 0 0 0 0 0 0 17 0 2 0 668338565 665911296 8344 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 514 18446604462196233472 0 0 -1 5 0 0 0
[pid=14752] ppid=14599 vsize=660560 CPUtime=0.33 cores=0-7
/proc/14752/stat : 14752 (python) S 14599 14599 14192 0 -1 4202560 1255 0 0 0 17 16 0 0 19 0 2 0 668318977 676413440 8439 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485794251 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/14752/statm: 165140 8439 406 364 0 100968 0
[pid=14754] ppid=14752 vsize=254328 CPUtime=99.97 cores=0-7
/proc/14754/stat : 14754 (FH) T 14752 14754 14754 0 -1 4202496 3699 0 0 0 9997 0 0 0 25 0 1 0 668318978 260431872 3409 33554432000 134512640 134936988 4293224752 18446744073709551615 134553498 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/14754/statm: 63582 3409 41 104 0 63086 0
[pid=14752/tid=14757] ppid=14599 vsize=660560 CPUtime=0 cores=0-7
/proc/14752/task/14757/stat : 14757 (python) S 14599 14599 14192 0 -1 4202560 93 0 0 0 0 0 0 0 19 0 2 0 668329011 676413440 8439 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 514 18446604466789847488 0 0 -1 2 0 0 0
[pid=14762] ppid=14599 vsize=670816 CPUtime=0.35 cores=0-7
/proc/14762/stat : 14762 (python) S 14599 14599 14192 0 -1 4202560 1267 0 0 0 16 19 0 0 19 0 2 0 668338571 686915584 8364 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485794251 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/14762/statm: 167704 8364 406 364 0 103529 0
[pid=14764] ppid=14762 vsize=1023176 CPUtime=99.81 cores=0-7
/proc/14764/stat : 14764 (gnovelty+2) T 14762 14764 14764 0 -1 4202496 2211 0 0 0 9981 0 0 0 25 0 1 0 668338572 1047732224 1887 33554432000 134512640 135663704 4293907200 18446744073709551615 134525302 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/14764/statm: 255794 1887 113 282 0 255510 0
[pid=14762/tid=14769] ppid=14599 vsize=670816 CPUtime=0 cores=0-7
/proc/14762/task/14769/stat : 14769 (python) S 14599 14599 14192 0 -1 4202560 90 0 0 0 0 0 0 0 19 0 2 0 668349117 686915584 8364 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 514 18446604454223472448 0 0 -1 7 0 0 0
[pid=14770] ppid=14599 vsize=681072 CPUtime=0.29 cores=0-7
/proc/14770/stat : 14770 (python) S 14599 14599 14192 0 -1 4202560 1067 0 0 0 14 15 0 0 15 0 1 0 668349124 697417728 8440 33554432000 4194304 5683644 140736277217920 18446744073709551615 274473987654 0 0 16781312 514 7662505330063933701 0 0 17 5 0 0 0
/proc/14770/statm: 170268 8440 379 364 0 106090 0
[pid=14772] ppid=14770 vsize=268420 CPUtime=89.05 cores=0-7
/proc/14772/stat : 14772 (hybridGM3) R 14770 14772 14772 0 -1 4202496 6691 0 0 0 8905 0 0 0 25 0 1 0 668349124 274862080 6399 33554432000 134512640 135204385 4292784000 18446744073709551615 134536986 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/14772/statm: 67105 6399 64 169 0 66935 0
[pid=14775] ppid=14599 vsize=691328 CPUtime=0.01 cores=0-7
/proc/14775/stat : 14775 (python) S 14599 14599 14192 0 -1 4202560 1056 0 0 0 0 1 0 0 18 0 1 0 668357666 707919872 8505 33554432000 4194304 5683644 140736277217920 18446744073709551615 274473987654 0 0 16781312 514 7045113062688260359 0 0 17 5 0 0 0
/proc/14775/statm: 172832 8505 373 364 0 108651 0
[pid=14777] ppid=14775 vsize=254328 CPUtime=4.45 cores=0-7
/proc/14777/stat : 14777 (FH) R 14775 14777 14777 0 -1 4202496 3698 0 0 0 445 0 0 0 25 0 1 0 668357666 260431872 3410 33554432000 134512640 134936988 4293252592 18446744073709551615 134554594 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/14777/statm: 63582 3410 41 104 0 63086 0
[pid=14778] ppid=14599 vsize=701584 CPUtime=0 cores=0-7
/proc/14778/stat : 14778 (python) S 14599 14599 14192 0 -1 4202560 929 0 0 0 0 0 0 0 19 0 1 0 668358016 718422016 8515 33554432000 4194304 5683644 140736277217920 18446744073709551615 274473987654 0 0 16781312 514 17336356356084465663 0 0 17 5 0 0 0
/proc/14778/statm: 175396 8515 345 364 0 111212 0
[pid=14780] ppid=14778 vsize=425800 CPUtime=0.97 cores=0-7
/proc/14780/stat : 14780 (NCVWr) R 14778 14780 14780 0 -1 4202496 3851 0 0 0 96 1 0 0 23 0 1 0 668358016 436019200 3564 33554432000 134512640 135113558 4291942352 18446744073709551615 134531393 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/14780/statm: 106450 3564 58 147 0 105325 0
[pid=14599/tid=14606] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14606/stat : 14606 (python) S 14597 14599 14192 0 -1 4202560 30 932 0 0 0 0 0 0 15 0 36 0 668210159 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 6 0 0 0
[pid=14599/tid=14609] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14609/stat : 14609 (python) S 14597 14599 14192 0 -1 4202560 30 932 0 0 0 0 0 0 18 0 36 0 668210162 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 6 0 0 0
[pid=14599/tid=14612] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14612/stat : 14612 (python) S 14597 14599 14192 0 -1 4202560 50 932 0 0 0 0 0 0 15 0 36 0 668210166 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 2 0 0 0
[pid=14599/tid=14615] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14615/stat : 14615 (python) S 14597 14599 14192 0 -1 4202560 43 932 0 0 0 0 0 0 18 0 36 0 668210169 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 3 0 0 0
[pid=14599/tid=14618] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14618/stat : 14618 (python) S 14597 14599 14192 0 -1 4202560 103 932 0 0 0 0 0 0 15 0 36 0 668210172 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 2 0 0 0
[pid=14599/tid=14621] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14621/stat : 14621 (python) S 14597 14599 14192 0 -1 4202560 37 932 0 0 0 0 0 0 18 0 36 0 668210175 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 0 0 0 0
[pid=14599/tid=14624] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14624/stat : 14624 (python) S 14597 14599 14192 0 -1 4202560 39 932 0 0 0 0 0 0 18 0 36 0 668210178 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 2 0 0 0
[pid=14599/tid=14627] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14627/stat : 14627 (python) S 14597 14599 14192 0 -1 4202560 51 932 0 0 0 0 0 0 18 0 36 0 668210181 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 1 0 0 0
[pid=14599/tid=14633] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14633/stat : 14633 (python) S 14597 14599 14192 0 -1 4202560 45 932 0 0 0 0 0 0 18 0 36 0 668220213 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 2 0 0 0
[pid=14599/tid=14649] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14649/stat : 14649 (python) S 14597 14599 14192 0 -1 4202560 36 932 0 0 0 0 0 0 20 0 36 0 668239400 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 0 0 0 0
[pid=14599/tid=14652] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14652/stat : 14652 (python) S 14597 14599 14192 0 -1 4202560 38 932 0 0 0 0 0 0 18 0 36 0 668239403 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 2 0 0 0
[pid=14599/tid=14655] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14655/stat : 14655 (python) S 14597 14599 14192 0 -1 4202560 39 932 0 0 0 0 0 0 18 0 36 0 668239893 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 7 0 0 0
[pid=14599/tid=14658] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14658/stat : 14658 (python) S 14597 14599 14192 0 -1 4202560 38 932 0 0 0 0 0 0 18 0 36 0 668239898 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 2 0 0 0
[pid=14599/tid=14661] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14661/stat : 14661 (python) S 14597 14599 14192 0 -1 4202560 37 932 0 0 0 0 0 0 22 0 36 0 668239906 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 2 0 0 0
[pid=14599/tid=14668] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14668/stat : 14668 (python) S 14597 14599 14192 0 -1 4202560 46 932 0 0 0 0 0 0 18 0 36 0 668249441 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 1 0 0 0
[pid=14599/tid=14672] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14672/stat : 14672 (python) S 14597 14599 14192 0 -1 4202560 45 932 0 0 0 0 0 0 18 0 36 0 668249940 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 7 0 0 0
[pid=14599/tid=14681] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14681/stat : 14681 (python) S 14597 14599 14192 0 -1 4202560 44 932 0 0 0 0 0 0 18 0 36 0 668259476 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 0 0 0 0
[pid=14599/tid=14685] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14685/stat : 14685 (python) S 14597 14599 14192 0 -1 4202560 35 932 0 0 0 0 0 0 20 0 36 0 668259670 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 7 0 0 0
[pid=14599/tid=14689] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14689/stat : 14689 (python) S 14597 14599 14192 0 -1 4202560 34 932 0 0 0 0 0 0 19 0 36 0 668260479 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 2 0 0 0
[pid=14599/tid=14694] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14694/stat : 14694 (python) S 14597 14599 14192 0 -1 4202560 53 932 0 0 0 0 0 0 18 0 36 0 668269240 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 5 0 0 0
[pid=14599/tid=14698] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14698/stat : 14698 (python) S 14597 14599 14192 0 -1 4202560 47 932 0 0 0 0 0 0 18 0 36 0 668269437 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 4 0 0 0
[pid=14599/tid=14703] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14703/stat : 14703 (python) S 14597 14599 14192 0 -1 4202560 51 932 0 0 0 0 0 0 18 0 36 0 668269707 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 0 0 0 0
[pid=14599/tid=14706] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14706/stat : 14706 (python) S 14597 14599 14192 0 -1 4202560 44 932 0 0 0 0 0 0 18 0 36 0 668269739 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 5 0 0 0
[pid=14599/tid=14710] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14710/stat : 14710 (python) S 14597 14599 14192 0 -1 4202560 37 932 0 0 0 0 0 0 18 0 36 0 668270516 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 5 0 0 0
[pid=14599/tid=14715] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14715/stat : 14715 (python) S 14597 14599 14192 0 -1 4202560 31 932 0 0 0 0 0 0 19 0 36 0 668278851 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 5 0 0 0
[pid=14599/tid=14724] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14724/stat : 14724 (python) S 14597 14599 14192 0 -1 4202560 35 932 0 0 0 0 0 0 19 0 36 0 668289373 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 5 0 0 0
[pid=14599/tid=14734] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14734/stat : 14734 (python) S 14597 14599 14192 0 -1 4202560 38 932 0 0 0 0 0 0 20 0 36 0 668299410 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 0 0 0 0
[pid=14599/tid=14737] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14737/stat : 14737 (python) S 14597 14599 14192 0 -1 4202560 31 932 0 0 0 0 0 0 20 0 36 0 668300423 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 5 0 0 0
[pid=14599/tid=14746] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14746/stat : 14746 (python) S 14597 14599 14192 0 -1 4202560 42 932 0 0 0 0 0 0 18 0 36 0 668318064 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 5 0 0 0
[pid=14599/tid=14749] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14749/stat : 14749 (python) S 14597 14599 14192 0 -1 4202560 35 932 0 0 0 0 0 0 20 0 36 0 668318860 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 5 0 0 0
[pid=14599/tid=14753] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14753/stat : 14753 (python) S 14597 14599 14192 0 -1 4202560 34 932 0 0 0 0 0 0 20 0 36 0 668318978 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 5 0 0 0
[pid=14599/tid=14763] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14763/stat : 14763 (python) S 14597 14599 14192 0 -1 4202560 36 932 0 0 0 0 0 0 20 0 36 0 668338572 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 0 0 0 0
[pid=14599/tid=14771] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14771/stat : 14771 (python) S 14597 14599 14192 0 -1 4202560 35 932 0 0 0 0 0 0 20 0 36 0 668349124 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 0 0 0 0
[pid=14599/tid=14776] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14776/stat : 14776 (python) S 14597 14599 14192 0 -1 4202560 32 932 0 0 0 0 0 0 20 0 36 0 668357666 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 0 0 0 0
[pid=14599/tid=14779] ppid=14597 vsize=711828 CPUtime=0 cores=0-7
/proc/14599/task/14779/stat : 14779 (python) S 14597 14599 14192 0 -1 4202560 32 932 0 0 0 0 0 0 20 0 36 0 668358016 728911872 10118 33554432000 4194304 5683644 140736277217920 18446744073709551615 274485791153 0 0 16781312 2 18446604447312879808 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 11781.6
Current children cumulated vsize (KiB) 32096404

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 14599 and gives
#  childrusage.ru_utime.tv_sec=2560
#  childrusage.ru_utime.tv_usec=615727
#  childrusage.ru_stime.tv_sec=19
#  childrusage.ru_stime.tv_usec=86098
# CPU time returned by wait4() is 2579.7
# while last known CPU time is 11781.6
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1484.03
CPU time (s): 11781.6
CPU user time (s): 11762.6
CPU system time (s): 18.99
CPU usage (%): 793.895
Max. virtual memory (cumulated for all children) (KiB): 32096404

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2560.62
system time used= 19.0861
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 134812
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 19454
involuntary context switches= 15512

runsolver used 15.6376 second user time and 41.0688 second system time

The end

Launcher Data 

Begin job on node125 at 2011-05-16 19:15:01
IDJOB=3385381
IDBENCH=82646
IDSOLVER=1870
FILE ID=node125/3385381-1305566100
RUNJOBID= node125-1305564963-14206
PBS_JOBID= 13350125
Free space on /tmp= 73244 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v20000-c84000-S1298436766-097-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3385381-1305566100/watcher-3385381-1305566100 -o /tmp/evaluation-result-3385381-1305566100/solver-3385381-1305566100 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385381-1305566100.cnf -seed 893259170 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 28b704f4b0f31dccf5e0c8fedfb14daa
RANDOM SEED=893259170

node125.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5266.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      18411500 kB
Buffers:        201008 kB
Cached:        1931968 kB
SwapCached:      55652 kB
Active:        7259656 kB
Inactive:      6945088 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      18411500 kB
SwapTotal:    67111528 kB
SwapFree:     67042944 kB
Dirty:           33368 kB
Writeback:           0 kB
AnonPages:    12016592 kB
Mapped:          35652 kB
Slab:           216992 kB
PageTables:      48416 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 89103960 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73212 MiB
End job on node125 at 2011-05-16 19:39:47