Trace number 3317124

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (TO) 4800.21 1210.85

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v35000-c147000-S448503257-098-UNKNOWN.cnf
MD5SUM0d20733a6f7d3b01f88ae7bba5f0d271
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark405.867
Satisfiable
(Un)Satisfiability was proved
Number of variables34999
Number of clauses147000
Sum of the clauses size441000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3147000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.35/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.35/0.67	c INFO: solving HOME/instance-3317124-1304267406.cnf
0.35/0.67	c INFO: computing task features
0.84/1.19	c INFO: computed variable-clause graph statistics
0.94/1.21	c INFO: computed clause balance statistics
3.44/3.76	c INFO: computed variable balance statistics
5.44/5.74	c INFO: computed small-clause counts
5.44/5.75	c INFO: computed Horn-clause counts
7.04/7.32	c INFO: computed variable graph statistics
7.04/7.32	c INFO: collected features for HOME/instance-3317124-1304267406.cnf in 6.63 s
7.04/7.35	c INFO: running hybridGM3@0 for 97 with 4793 remaining (b = 0.34)
7.04/7.38	c INFO: running hybridGM3@0 for 97 with 4793 remaining (b = 0.21)
7.12/7.41	c INFO: running hybridGM3@0 for 97 with 4793 remaining (b = 0.14)
7.12/7.44	c INFO: running hybridGM3@0 for 97 with 4793 remaining (b = 0.10)
406.87/107.59	c INFO: running hybridGM3@0 for 97 with 4693 remaining (b = 0.08)
407.27/107.63	c INFO: running hybridGM3@0 for 97 with 4693 remaining (b = 0.08)
407.27/107.67	c INFO: running hybridGM3@0 for 97 with 4693 remaining (b = 0.08)
407.65/107.70	c INFO: running hybridGM3@100 for 97 with 4692 remaining (b = 0.08)
787.04/202.73	c INFO: running hybridGM3@99 for 97 with 4597 remaining (b = 0.08)
787.04/202.77	c INFO: running hybridGM3@100 for 97 with 4597 remaining (b = 0.08)
787.43/202.81	c INFO: running TNM@0 for 97 with 4597 remaining (b = 0.10)
787.43/202.84	c INFO: running hybridGM3@99 for 97 with 4597 remaining (b = 0.07)
976.72/250.27	c INFO: running hybridGM3@0 for 194 with 4550 remaining (b = 0.13)
1187.18/302.98	c INFO: running hybridGM3@0 for 194 with 4497 remaining (b = 0.10)
1190.75/303.89	c INFO: running hybridGM3@0 for 194 with 4496 remaining (b = 0.07)
1191.44/304.01	c INFO: running hybridGM3@146 for 97 with 4496 remaining (b = 0.03)
1571.11/399.17	c INFO: running hybridGM3@0 for 194 with 4401 remaining (b = 0.05)
1760.70/446.73	c INFO: running hybridGM3@99 for 97 with 4353 remaining (b = 0.03)
1970.68/499.42	c INFO: running hybridGM3@100 for 97 with 4301 remaining (b = 0.03)
1998.14/506.36	c INFO: running hybridGM3@101 for 97 with 4294 remaining (b = 0.03)
2163.85/547.90	c INFO: running TNM@0 for 97 with 4252 remaining (b = 0.02)
2353.87/595.62	c INFO: running hybridGM3@295 for 97 with 4205 remaining (b = 0.02)
2373.39/600.59	c INFO: running hybridGM3@300 for 97 with 4200 remaining (b = 0.02)
2377.37/601.52	c INFO: running TNM@0 for 97 with 4199 remaining (b = 0.02)
2460.60/622.41	c INFO: running hybridGM3@0 for 194 with 4178 remaining (b = 0.04)
2562.57/648.08	c INFO: running TNM@0 for 97 with 4152 remaining (b = 0.02)
2752.57/695.77	c INFO: running hybridGM3@395 for 97 with 4104 remaining (b = 0.01)
2776.48/701.71	c INFO: running TNM@0 for 97 with 4098 remaining (b = 0.01)
2961.65/748.28	c INFO: running TNM@0 for 97 with 4052 remaining (b = 0.01)
3155.61/796.96	c INFO: running TNM@0 for 97 with 4003 remaining (b = 0.01)
3175.53/801.91	c INFO: running TNM@196 for 1843 with 3998 remaining (b = 0.21)
3243.21/818.90	c INFO: running hybridGM3@99 for 97 with 3981 remaining (b = 0.02)
3357.09/847.53	c INFO: running hybridGM3@0 for 194 with 3953 remaining (b = 0.03)
3550.60/896.20	c INFO: running hybridGM3@0 for 194 with 3904 remaining (b = 0.02)
3645.16/920.09	c INFO: running hybridGM3@0 for 194 with 3880 remaining (b = 0.02)
4137.90/1044.04	c INFO: running hybridGM3@98 for 97 with 3756 remaining (b = 0.01)
4331.02/1092.68	c INFO: running hybridGM3@101 for 97 with 3708 remaining (b = 0.01)
4426.02/1116.58	c INFO: running adaptg2wsat2009++@0 for 194 with 3684 remaining (b = 0.02)
4539.95/1145.24	c INFO: running hybridGM3@294 for 97 with 3655 remaining (b = 0.01)
4733.06/1193.88	c INFO: running hybridGM3@297 for 97 with 3606 remaining (b = 0.01)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317124-1304267406/watcher-3317124-1304267406 -o /tmp/evaluation-result-3317124-1304267406/solver-3317124-1304267406 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317124-1304267406.cnf -seed 1558148818 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.43 2.17 0.77 5/189 5419
/proc/meminfo: memFree=30249248/32951124 swapFree=67111416/67111528
[pid=5419] ppid=5417 vsize=15184 CPUtime=0 cores=1,3,5,7
/proc/5419/stat : 5419 (python) S 5417 5419 5229 0 -1 4202496 727 0 0 0 0 0 0 0 25 0 1 0 538342994 15548416 617 33554432000 4194304 5683644 140734995986208 18446744073709551615 212436015104 0 2147483391 16781312 2 18446744071699207358 0 0 17 3 0 0 0
/proc/5419/statm: 3796 617 322 364 0 283 0

[startup+0.0492759 s]
/proc/loadavg: 6.43 2.17 0.77 5/189 5419
/proc/meminfo: memFree=30249248/32951124 swapFree=67111416/67111528
[pid=5419] ppid=5417 vsize=18888 CPUtime=0.02 cores=1,3,5,7
/proc/5419/stat : 5419 (python) R 5417 5419 5229 0 -1 4202496 1288 0 0 0 2 0 0 0 25 0 1 0 538342994 19341312 1141 33554432000 4194304 5683644 140734995986208 18446744073709551615 212436015104 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/5419/statm: 4722 1141 421 364 0 691 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 18888

[startup+0.100281 s]
/proc/loadavg: 6.43 2.17 0.77 5/189 5419
/proc/meminfo: memFree=30249248/32951124 swapFree=67111416/67111528
[pid=5419] ppid=5417 vsize=89772 CPUtime=0.06 cores=1,3,5,7
/proc/5419/stat : 5419 (python) R 5417 5419 5229 0 -1 4202496 2641 0 0 0 4 2 0 0 25 0 1 0 538342994 91926528 2357 33554432000 4194304 5683644 140734995986208 18446744073709551615 212431054737 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/5419/statm: 22443 2359 721 364 0 1623 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 89772

[startup+0.300272 s]
/proc/loadavg: 6.43 2.17 0.77 5/189 5419
/proc/meminfo: memFree=30249248/32951124 swapFree=67111416/67111528
[pid=5419] ppid=5417 vsize=135752 CPUtime=0.19 cores=1,3,5,7
/proc/5419/stat : 5419 (python) S 5417 5419 5229 0 -1 4202496 5968 935 1 0 14 5 0 0 17 0 1 0 538342994 139010048 4151 33554432000 4194304 5683644 140734995986208 18446744073709551615 212436013061 0 2147483391 16781312 2 18446744071699207358 0 0 17 3 0 0 0
/proc/5419/statm: 33938 4151 1032 364 0 3124 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 135752

[startup+0.70018 s]
/proc/loadavg: 6.43 2.17 0.77 5/189 5419
/proc/meminfo: memFree=30249248/32951124 swapFree=67111416/67111528
[pid=5419] ppid=5417 vsize=283084 CPUtime=0.43 cores=1,3,5,7
/proc/5419/stat : 5419 (python) R 5417 5419 5229 0 -1 4202496 10413 935 1 0 33 10 0 0 15 0 1 0 538342994 289878016 8093 33554432000 4194304 5683644 140734995986208 18446744073709551615 4445115 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/5419/statm: 70771 8093 1761 364 0 6695 0
Current children cumulated CPU time (s) 0.43
Current children cumulated vsize (KiB) 283084

[startup+1.501 s]
/proc/loadavg: 6.43 2.17 0.77 6/190 5424
/proc/meminfo: memFree=30203856/32951124 swapFree=67111416/67111528
[pid=5419] ppid=5417 vsize=312976 CPUtime=1.24 cores=1,3,5,7
/proc/5419/stat : 5419 (python) R 5417 5419 5229 0 -1 4202496 21969 935 1 0 112 12 0 0 16 0 1 0 538342994 320487424 15628 33554432000 4194304 5683644 140734995986208 18446744073709551615 47098394567140 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/5419/statm: 78244 15628 1826 364 0 14168 0
Current children cumulated CPU time (s) 1.24
Current children cumulated vsize (KiB) 312976

[startup+3.1007 s]
/proc/loadavg: 6.43 2.17 0.77 6/190 5424
/proc/meminfo: memFree=30192076/32951124 swapFree=67111416/67111528
[pid=5419] ppid=5417 vsize=312976 CPUtime=2.84 cores=1,3,5,7
/proc/5419/stat : 5419 (python) R 5417 5419 5229 0 -1 4202496 21969 935 1 0 272 12 0 0 18 0 1 0 538342994 320487424 15628 33554432000 4194304 5683644 140734995986208 18446744073709551615 47098394685833 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/5419/statm: 78244 15628 1826 364 0 14168 0
Current children cumulated CPU time (s) 2.84
Current children cumulated vsize (KiB) 312976

[startup+6.30111 s]
/proc/loadavg: 6.31 2.22 0.79 6/206 5440
/proc/meminfo: memFree=30152444/32951124 swapFree=67111416/67111528
[pid=5419] ppid=5417 vsize=312976 CPUtime=6.04 cores=1,3,5,7
/proc/5419/stat : 5419 (python) R 5417 5419 5229 0 -1 4202496 21976 935 1 0 592 12 0 0 25 0 1 0 538342994 320487424 15635 33554432000 4194304 5683644 140734995986208 18446744073709551615 47098394605708 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/5419/statm: 78244 15635 1833 364 0 14168 0
Current children cumulated CPU time (s) 6.04
Current children cumulated vsize (KiB) 312976

[startup+12.7009 s]
/proc/loadavg: 6.45 2.31 0.83 9/218 5452
/proc/meminfo: memFree=30013704/32951124 swapFree=67111416/67111528
[pid=5419] ppid=5417 vsize=351280 CPUtime=7.17 cores=1,3,5,7
/proc/5419/stat : 5419 (python) S 5417 5419 5229 0 -1 4202496 24929 935 1 0 704 13 0 0 20 0 5 0 538342994 359710720 15016 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447843787 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/5419/statm: 87820 15016 1881 364 0 23729 0
[pid=5441] ppid=5419 vsize=310268 CPUtime=0 cores=1,3,5,7
/proc/5441/stat : 5441 (python) S 5419 5419 5229 0 -1 4202560 894 0 0 0 0 0 0 0 15 0 1 0 538343729 317714432 13495 33554432000 4194304 5683644 140734995986208 18446744073709551615 212436037135 0 0 16781312 514 988516757880668424 0 0 17 3 0 0 0
/proc/5441/statm: 77567 13495 376 364 0 13485 0
[pid=5443] ppid=5441 vsize=268428 CPUtime=5.33 cores=1,3,5,7
/proc/5443/stat : 5443 (hybridGM3) R 5441 5443 5443 0 -1 4202496 8086 0 0 0 532 1 0 0 25 0 1 0 538343730 274870272 7824 33554432000 134512640 135204385 4294371040 18446744073709551615 134537510 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/5443/statm: 67107 7824 64 169 0 66937 0
[pid=5444] ppid=5419 vsize=320524 CPUtime=0 cores=1,3,5,7
/proc/5444/stat : 5444 (python) S 5419 5419 5229 0 -1 4202560 910 0 0 0 0 0 0 0 15 0 1 0 538343732 328216576 13493 33554432000 4194304 5683644 140734995986208 18446744073709551615 212436037190 0 0 16781312 514 18377520750065058056 0 0 17 3 0 0 0
/proc/5444/statm: 80131 13493 370 364 0 16046 0
[pid=5446] ppid=5444 vsize=268424 CPUtime=5.27 cores=1,3,5,7
/proc/5446/stat : 5446 (hybridGM3) R 5444 5446 5446 0 -1 4202496 8087 0 0 0 526 1 0 0 25 0 1 0 538343733 274866176 7823 33554432000 134512640 135204385 4291044672 18446744073709551615 134536887 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/5446/statm: 67106 7823 64 169 0 66936 0
[pid=5447] ppid=5419 vsize=330780 CPUtime=0 cores=1,3,5,7
/proc/5447/stat : 5447 (python) S 5419 5419 5229 0 -1 4202560 903 0 0 0 0 0 0 0 15 0 1 0 538343735 338718720 13497 33554432000 4194304 5683644 140734995986208 18446744073709551615 212436037190 0 0 16781312 514 4590091339474895112 0 0 17 3 0 0 0
/proc/5447/statm: 82695 13497 370 364 0 18607 0
[pid=5449] ppid=5447 vsize=268424 CPUtime=5.27 cores=1,3,5,7
/proc/5449/stat : 5449 (hybridGM3) R 5447 5449 5449 0 -1 4202496 8090 0 0 0 527 0 0 0 25 0 1 0 538343736 274866176 7827 33554432000 134512640 135204385 4289943136 18446744073709551615 134537403 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/5449/statm: 67106 7827 64 169 0 66936 0
[pid=5450] ppid=5419 vsize=341036 CPUtime=0 cores=1,3,5,7
/proc/5450/stat : 5450 (python) S 5419 5419 5229 0 -1 4202560 928 0 0 0 0 0 0 0 15 0 1 0 538343738 349220864 13501 33554432000 4194304 5683644 140734995986208 18446744073709551615 212436037190 0 0 16781312 514 4590083642893500680 0 0 17 1 0 0 0
/proc/5450/statm: 85259 13501 370 364 0 21168 0
[pid=5452] ppid=5450 vsize=268428 CPUtime=5.22 cores=1,3,5,7
/proc/5452/stat : 5452 (hybridGM3) R 5450 5452 5452 0 -1 4202496 8087 0 0 0 521 1 0 0 25 0 1 0 538343739 274870272 7825 33554432000 134512640 135204385 4292519456 18446744073709551615 134537322 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/5452/statm: 67107 7825 64 169 0 66937 0
[pid=5419/tid=5442] ppid=5417 vsize=351280 CPUtime=0 cores=1,3,5,7
/proc/5419/task/5442/stat : 5442 (python) S 5417 5419 5229 0 -1 4202560 26 935 0 0 0 0 0 0 25 0 5 0 538343730 359710720 15016 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 3 0 0 0
[pid=5419/tid=5445] ppid=5417 vsize=351280 CPUtime=0 cores=1,3,5,7
/proc/5419/task/5445/stat : 5445 (python) S 5417 5419 5229 0 -1 4202560 23 935 0 0 0 0 0 0 25 0 5 0 538343733 359710720 15016 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 7 0 0 0
[pid=5419/tid=5448] ppid=5417 vsize=351280 CPUtime=0 cores=1,3,5,7
/proc/5419/task/5448/stat : 5448 (python) S 5417 5419 5229 0 -1 4202560 21 935 0 0 0 0 0 0 20 0 5 0 538343736 359710720 15016 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 3 0 0 0
[pid=5419/tid=5451] ppid=5417 vsize=351280 CPUtime=0 cores=1,3,5,7
/proc/5419/task/5451/stat : 5451 (python) S 5417 5419 5229 0 -1 4202560 28 935 0 0 0 0 0 0 20 0 5 0 538343739 359710720 15016 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 28.26
Current children cumulated vsize (KiB) 2727592

[startup+25.5014 s]
/proc/loadavg: 6.79 2.59 0.94 9/218 5453
/proc/meminfo: memFree=30013724/32951124 swapFree=67111416/67111528
[pid=5419] ppid=5417 vsize=351280 CPUtime=7.17 cores=1,3,5,7
/proc/5419/stat : 5419 (python) S 5417 5419 5229 0 -1 4202496 24929 935 1 0 704 13 0 0 20 0 5 0 538342994 359710720 15016 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447843787 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/5419/statm: 87820 15016 1881 364 0 23729 0
[pid=5441] ppid=5419 vsize=310268 CPUtime=0.01 cores=1,3,5,7
/proc/5441/stat : 5441 (python) S 5419 5419 5229 0 -1 4202560 898 0 0 0 1 0 0 0 15 0 1 0 538343729 317714432 13495 33554432000 4194304 5683644 140734995986208 18446744073709551615 212436037135 0 0 16781312 514 17296918793846489352 0 0 17 3 0 0 0
/proc/5441/statm: 77567 13495 376 364 0 13485 0
[pid=5443] ppid=5441 vsize=268428 CPUtime=18.11 cores=1,3,5,7
/proc/5443/stat : 5443 (hybridGM3) R 5441 5443 5443 0 -1 4202496 8086 0 0 0 1810 1 0 0 25 0 1 0 538343730 274870272 7824 33554432000 134512640 135204385 4294371040 18446744073709551615 134537224 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/5443/statm: 67107 7824 64 169 0 66937 0
[pid=5444] ppid=5419 vsize=320524 CPUtime=0.02 cores=1,3,5,7
/proc/5444/stat : 5444 (python) S 5419 5419 5229 0 -1 4202560 917 0 0 0 1 1 0 0 15 0 1 0 538343732 328216576 13493 33554432000 4194304 5683644 140734995986208 18446744073709551615 212436037190 0 0 16781312 514 17296918793846489352 0 0 17 3 0 0 0
/proc/5444/statm: 80131 13493 370 364 0 16046 0
[pid=5446] ppid=5444 vsize=268424 CPUtime=17.99 cores=1,3,5,7
/proc/5446/stat : 5446 (hybridGM3) R 5444 5446 5446 0 -1 4202496 8087 0 0 0 1798 1 0 0 25 0 1 0 538343733 274866176 7823 33554432000 134512640 135204385 4291044672 18446744073709551615 134546059 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/5446/statm: 67106 7823 64 169 0 66936 0
[pid=5447] ppid=5419 vsize=330780 CPUtime=0.02 cores=1,3,5,7
/proc/5447/stat : 5447 (python) S 5419 5419 5229 0 -1 4202560 907 0 0 0 1 1 0 0 15 0 1 0 538343735 338718720 13497 33554432000 4194304 5683644 140734995986208 18446744073709551615 212436037190 0 0 16781312 514 17296918793846489352 0 0 17 3 0 0 0
/proc/5447/statm: 82695 13497 370 364 0 18607 0
[pid=5449] ppid=5447 vsize=268424 CPUtime=18.07 cores=1,3,5,7
/proc/5449/stat : 5449 (hybridGM3) R 5447 5449 5449 0 -1 4202496 8090 0 0 0 1807 0 0 0 25 0 1 0 538343736 274866176 7827 33554432000 134512640 135204385 4289943136 18446744073709551615 134536698 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/5449/statm: 67106 7827 64 169 0 66936 0
[pid=5450] ppid=5419 vsize=341036 CPUtime=0.02 cores=1,3,5,7
/proc/5450/stat : 5450 (python) S 5419 5419 5229 0 -1 4202560 931 0 0 0 1 1 0 0 15 0 1 0 538343738 349220864 13501 33554432000 4194304 5683644 140734995986208 18446744073709551615 212436037190 0 0 16781312 514 17296918793846489352 0 0 17 1 0 0 0
/proc/5450/statm: 85259 13501 370 364 0 21168 0
[pid=5452] ppid=5450 vsize=268428 CPUtime=17.97 cores=1,3,5,7
/proc/5452/stat : 5452 (hybridGM3) R 5450 5452 5452 0 -1 4202496 8087 0 0 0 1796 1 0 0 25 0 1 0 538343739 274870272 7825 33554432000 134512640 135204385 4292519456 18446744073709551615 134536676 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/5452/statm: 67107 7825 64 169 0 66937 0
[pid=5419/tid=5442] ppid=5417 vsize=351280 CPUtime=0 cores=1,3,5,7
/proc/5419/task/5442/stat : 5442 (python) S 5417 5419 5229 0 -1 4202560 26 935 0 0 0 0 0 0 25 0 5 0 538343730 359710720 15016 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 3 0 0 0
[pid=5419/tid=5445] ppid=5417 vsize=351280 CPUtime=0 cores=1,3,5,7
/proc/5419/task/5445/stat : 5445 (python) S 5417 5419 5229 0 -1 4202560 23 935 0 0 0 0 0 0 25 0 5 0 538343733 359710720 15016 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 7 0 0 0
[pid=5419/tid=5448] ppid=5417 vsize=351280 CPUtime=0 cores=1,3,5,7
/proc/5419/task/5448/stat : 5448 (python) S 5417 5419 5229 0 -1 4202560 21 935 0 0 0 0 0 0 20 0 5 0 538343736 359710720 15016 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 3 0 0 0
[pid=5419/tid=5451] ppid=5417 vsize=351280 CPUtime=0 cores=1,3,5,7
/proc/5419/task/5451/stat : 5451 (python) S 5417 5419 5229 0 -1 4202560 28 935 0 0 0 0 0 0 20 0 5 0 538343739 359710720 15016 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 79.38
Current children cumulated vsize (KiB) 2727592

[startup+51.1006 s]
/proc/loadavg: 7.26 3.04 1.14 9/218 5453
/proc/meminfo: memFree=30013184/32951124 swapFree=67111416/67111528
[pid=5419] ppid=5417 vsize=351280 CPUtime=7.17 cores=1,3,5,7
/proc/5419/stat : 5419 (python) S 5417 5419 5229 0 -1 4202496 24929 935 1 0 704 13 0 0 20 0 5 0 538342994 359710720 15016 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447843787 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/5419/statm: 87820 15016 1881 364 0 23729 0
[pid=5441] ppid=5419 vsize=310268 CPUtime=0.04 cores=1,3,5,7
/proc/5441/stat : 5441 (python) S 5419 5419 5229 0 -1 4202560 904 0 0 0 2 2 0 0 15 0 1 0 538343729 317714432 13495 33554432000 4194304 5683644 140734995986208 18446744073709551615 212436037135 0 0 16781312 514 17296918793846489352 0 0 17 3 0 0 0
/proc/5441/statm: 77567 13495 376 364 0 13485 0
[pid=5443] ppid=5441 vsize=268428 CPUtime=43.68 cores=1,3,5,7
/proc/5443/stat : 5443 (hybridGM3) R 5441 5443 5443 0 -1 4202496 8092 0 0 0 4367 1 0 0 25 0 1 0 538343730 274870272 7830 33554432000 134512640 135204385 4294371040 18446744073709551615 134546270 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/5443/statm: 67107 7830 64 169 0 66937 0
[pid=5444] ppid=5419 vsize=320524 CPUtime=0.05 cores=1,3,5,7
/proc/5444/stat : 5444 (python) S 5419 5419 5229 0 -1 4202560 924 0 0 0 3 2 0 0 15 0 1 0 538343732 328216576 13493 33554432000 4194304 5683644 140734995986208 18446744073709551615 212436037190 0 0 16781312 514 17296918793846489352 0 0 17 3 0 0 0
/proc/5444/statm: 80131 13493 370 364 0 16046 0
[pid=5446] ppid=5444 vsize=268424 CPUtime=43.43 cores=1,3,5,7
/proc/5446/stat : 5446 (hybridGM3) R 5444 5446 5446 0 -1 4202496 8089 0 0 0 4342 1 0 0 25 0 1 0 538343733 274866176 7825 33554432000 134512640 135204385 4291044672 18446744073709551615 134537383 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/5446/statm: 67106 7825 64 169 0 66936 0
[pid=5447] ppid=5419 vsize=330780 CPUtime=0.05 cores=1,3,5,7
/proc/5447/stat : 5447 (python) S 5419 5419 5229 0 -1 4202560 915 0 0 0 2 3 0 0 15 0 1 0 538343735 338718720 13497 33554432000 4194304 5683644 140734995986208 18446744073709551615 212436037190 0 0 16781312 514 17296918793846489352 0 0 17 3 0 0 0
/proc/5447/statm: 82695 13497 370 364 0 18607 0
[pid=5449] ppid=5447 vsize=268424 CPUtime=43.67 cores=1,3,5,7
/proc/5449/stat : 5449 (hybridGM3) R 5447 5449 5449 0 -1 4202496 8090 0 0 0 4367 0 0 0 25 0 1 0 538343736 274866176 7827 33554432000 134512640 135204385 4289943136 18446744073709551615 134536663 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/5449/statm: 67106 7827 64 169 0 66936 0
[pid=5450] ppid=5419 vsize=341036 CPUtime=0.05 cores=1,3,5,7
/proc/5450/stat : 5450 (python) S 5419 5419 5229 0 -1 4202560 939 0 0 0 3 2 0 0 15 0 1 0 538343738 349220864 13501 33554432000 4194304 5683644 140734995986208 18446744073709551615 212436037190 0 0 16781312 514 17296918793846489352 0 0 17 1 0 0 0
/proc/5450/statm: 85259 13501 370 364 0 21168 0

################
# More data... #
################

/proc/5480/stat : 5480 (TNM) T 5478 5480 5480 0 -1 4202496 6014 0 0 0 9960 0 0 0 25 0 1 0 538363276 476205056 5749 33554432000 134512640 135008748 4286867952 18446744073709551615 134527553 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/5480/statm: 116261 5749 46 122 0 115161 0
[pid=5478/tid=5499] ppid=5419 vsize=351292 CPUtime=0 cores=1,3,5,7
/proc/5478/task/5499/stat : 5499 (python) S 5419 5419 5229 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 538373290 359723008 13535 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 514 18446604470076572864 0 0 -1 7 0 0 0
[pid=5500] ppid=5419 vsize=371804 CPUtime=0.77 cores=1,3,5,7
/proc/5500/stat : 5500 (python) S 5419 5419 5229 0 -1 4202560 1240 0 0 0 40 37 0 0 15 0 2 0 538373293 380727296 13549 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5500/statm: 92951 13549 405 364 0 28851 0
[pid=5502] ppid=5500 vsize=268424 CPUtime=490.48 cores=1,3,5,7
/proc/5502/stat : 5502 (hybridGM3) T 5500 5502 5502 0 -1 4202496 8089 0 0 0 49047 1 0 0 25 0 1 0 538373293 274866176 7821 33554432000 134512640 135204385 4293188336 18446744073709551615 134537292 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/5502/statm: 67106 7821 64 169 0 66936 0
[pid=5500/tid=5522] ppid=5419 vsize=371804 CPUtime=0 cores=1,3,5,7
/proc/5500/task/5522/stat : 5522 (python) S 5419 5419 5229 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 538392934 380727296 13549 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 514 18446604468639228160 0 0 -1 5 0 0 0
[pid=5503] ppid=5419 vsize=382060 CPUtime=0.45 cores=1,3,5,7
/proc/5503/stat : 5503 (python) S 5419 5419 5229 0 -1 4202560 1232 0 0 0 22 23 0 0 15 0 2 0 538373384 391229440 13553 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5503/statm: 95515 13553 405 364 0 31412 0
[pid=5505] ppid=5503 vsize=268424 CPUtime=294.54 cores=1,3,5,7
/proc/5505/stat : 5505 (hybridGM3) T 5503 5505 5505 0 -1 4202496 8099 0 0 0 29452 2 0 0 25 0 1 0 538373385 274866176 7823 33554432000 134512640 135204385 4287100928 18446744073709551615 134537452 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/5505/statm: 67106 7823 64 169 0 66936 0
[pid=5503/tid=5523] ppid=5419 vsize=382060 CPUtime=0 cores=1,3,5,7
/proc/5503/task/5523/stat : 5523 (python) S 5419 5419 5229 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 538393627 391229440 13553 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 514 18446604468639229056 0 0 -1 5 0 0 0
[pid=5514] ppid=5419 vsize=392316 CPUtime=0.29 cores=1,3,5,7
/proc/5514/stat : 5514 (python) S 5419 5419 5229 0 -1 4202560 1187 0 0 0 15 14 0 0 16 0 2 0 538382912 401731584 13551 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5514/statm: 98079 13551 399 364 0 33973 0
[pid=5516] ppid=5514 vsize=268420 CPUtime=196.11 cores=1,3,5,7
/proc/5516/stat : 5516 (hybridGM3) T 5514 5516 5516 0 -1 4202496 8089 0 0 0 19610 1 0 0 25 0 1 0 538382912 274862080 7820 33554432000 134512640 135204385 4293181440 18446744073709551615 134536698 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/5516/statm: 67105 7820 64 169 0 66935 0
[pid=5514/tid=5533] ppid=5419 vsize=392316 CPUtime=0 cores=1,3,5,7
/proc/5514/task/5533/stat : 5533 (python) S 5419 5419 5229 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 538402553 401731584 13551 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 514 18446604471165232320 0 0 -1 5 0 0 0
[pid=5526] ppid=5419 vsize=402572 CPUtime=0.84 cores=1,3,5,7
/proc/5526/stat : 5526 (python) S 5419 5419 5229 0 -1 4202560 1181 0 0 0 45 39 0 0 18 0 2 0 538397785 412233728 13555 33554432000 4194304 5683644 140734995986208 18446744073709551615 212436037190 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5526/statm: 100643 13555 399 364 0 36534 0
[pid=5528] ppid=5526 vsize=465044 CPUtime=498.32 cores=1,3,5,7
/proc/5528/stat : 5528 (TNM) R 5526 5528 5528 0 -1 4202496 6019 0 0 0 49832 0 0 0 25 0 1 0 538397785 476205056 5749 33554432000 134512640 135008748 4293370464 18446744073709551615 134529448 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/5528/statm: 116261 5749 46 122 0 115161 0
[pid=5526/tid=5541] ppid=5419 vsize=402572 CPUtime=0 cores=1,3,5,7
/proc/5526/task/5541/stat : 5541 (python) S 5419 5419 5229 0 -1 4202560 51 0 0 0 0 0 0 0 19 0 2 0 538407799 412233728 13555 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 514 18446604471138063424 0 0 -1 5 0 0 0
[pid=5534] ppid=5419 vsize=412828 CPUtime=0.16 cores=1,3,5,7
/proc/5534/stat : 5534 (python) S 5419 5419 5229 0 -1 4202560 1176 0 0 0 9 7 0 0 19 0 2 0 538403147 422735872 13559 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5534/statm: 103207 13559 399 364 0 39095 0
[pid=5536] ppid=5534 vsize=465044 CPUtime=99.69 cores=1,3,5,7
/proc/5536/stat : 5536 (TNM) T 5534 5536 5536 0 -1 4202496 6021 0 0 0 9969 0 0 0 25 0 1 0 538403147 476205056 5749 33554432000 134512640 135008748 4288146640 18446744073709551615 134529176 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/5536/statm: 116261 5749 46 122 0 115161 0
[pid=5534/tid=5548] ppid=5419 vsize=412828 CPUtime=0 cores=1,3,5,7
/proc/5534/task/5548/stat : 5548 (python) S 5419 5419 5229 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 538413162 422735872 13559 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 514 18446604471093022912 0 0 -1 5 0 0 0
[pid=5538] ppid=5419 vsize=423084 CPUtime=0.49 cores=1,3,5,7
/proc/5538/stat : 5538 (python) S 5419 5419 5229 0 -1 4202560 1194 0 0 0 25 24 0 0 15 0 2 0 538405236 433238016 13563 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/5538/statm: 105771 13563 399 364 0 41656 0
[pid=5540] ppid=5538 vsize=268424 CPUtime=295.18 cores=1,3,5,7
/proc/5540/stat : 5540 (hybridGM3) T 5538 5540 5540 0 -1 4202496 8093 0 0 0 29517 1 0 0 25 0 1 0 538405237 274866176 7823 33554432000 134512640 135204385 4287355072 18446744073709551615 134536744 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/5540/statm: 67106 7823 64 169 0 66936 0
[pid=5538/tid=5569] ppid=5419 vsize=423084 CPUtime=0 cores=1,3,5,7
/proc/5538/task/5569/stat : 5569 (python) S 5419 5419 5229 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 538424881 433238016 13563 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 514 18446604470076571968 0 0 -1 1 0 0 0
[pid=5542] ppid=5419 vsize=433340 CPUtime=0.17 cores=1,3,5,7
/proc/5542/stat : 5542 (python) S 5419 5419 5229 0 -1 4202560 1159 0 0 0 10 7 0 0 19 0 2 0 538407803 443740160 13567 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/5542/statm: 108335 13567 399 364 0 44217 0
[pid=5544] ppid=5542 vsize=465044 CPUtime=100.14 cores=1,3,5,7
/proc/5544/stat : 5544 (TNM) T 5542 5544 5544 0 -1 4202496 6019 0 0 0 10014 0 0 0 25 0 1 0 538407804 476205056 5750 33554432000 134512640 135008748 4289226128 18446744073709551615 134527327 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/5544/statm: 116261 5750 46 122 0 115161 0
[pid=5542/tid=5553] ppid=5419 vsize=433340 CPUtime=0 cores=1,3,5,7
/proc/5542/task/5553/stat : 5553 (python) S 5419 5419 5229 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 538417819 443740160 13567 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 514 18446604469499593088 0 0 -1 3 0 0 0
[pid=5549] ppid=5419 vsize=443596 CPUtime=0.17 cores=1,3,5,7
/proc/5549/stat : 5549 (python) S 5419 5419 5229 0 -1 4202560 1171 0 0 0 9 8 0 0 19 0 2 0 538413166 454242304 13571 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/5549/statm: 110899 13571 399 364 0 46778 0
[pid=5551] ppid=5549 vsize=465044 CPUtime=99.67 cores=1,3,5,7
/proc/5551/stat : 5551 (TNM) T 5549 5551 5551 0 -1 4202496 6021 0 0 0 9966 1 0 0 25 0 1 0 538413166 476205056 5749 33554432000 134512640 135008748 4287425824 18446744073709551615 134528974 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/5551/statm: 116261 5749 46 122 0 115161 0
[pid=5549/tid=5567] ppid=5419 vsize=443596 CPUtime=0 cores=1,3,5,7
/proc/5549/task/5567/stat : 5567 (python) S 5419 5419 5229 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 538423182 454242304 13571 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 514 18446604471093024704 0 0 -1 1 0 0 0
[pid=5554] ppid=5419 vsize=453852 CPUtime=0.16 cores=1,3,5,7
/proc/5554/stat : 5554 (python) S 5419 5419 5229 0 -1 4202560 1225 0 0 0 9 7 0 0 19 0 2 0 538417823 464744448 13583 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/5554/statm: 113463 13583 407 364 0 49339 0
[pid=5556] ppid=5554 vsize=465044 CPUtime=99.19 cores=1,3,5,7
/proc/5556/stat : 5556 (TNM) T 5554 5556 5556 0 -1 4202496 6030 0 0 0 9918 1 0 0 25 0 1 0 538417823 476205056 5750 33554432000 134512640 135008748 4293407792 18446744073709551615 134529367 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/5556/statm: 116261 5750 46 122 0 115161 0
[pid=5554/tid=5571] ppid=5419 vsize=453852 CPUtime=0 cores=1,3,5,7
/proc/5554/task/5571/stat : 5571 (python) S 5419 5419 5229 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 538427743 464744448 13583 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 514 18446604454224049152 0 0 -1 1 0 0 0
[pid=5564] ppid=5419 vsize=464108 CPUtime=0.17 cores=1,3,5,7
/proc/5564/stat : 5564 (python) S 5419 5419 5229 0 -1 4202560 1216 0 0 0 10 7 0 0 19 0 2 0 538422691 475246592 13587 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/5564/statm: 116027 13587 407 364 0 51900 0
[pid=5566] ppid=5564 vsize=465044 CPUtime=98.98 cores=1,3,5,7
/proc/5566/stat : 5566 (TNM) T 5564 5566 5566 0 -1 4202496 6024 0 0 0 9897 1 0 0 25 0 1 0 538422691 476205056 5749 33554432000 134512640 135008748 4290861392 18446744073709551615 134527111 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/5566/statm: 116261 5749 46 122 0 115161 0
[pid=5564/tid=5674] ppid=5419 vsize=464108 CPUtime=0 cores=1,3,5,7
/proc/5564/task/5674/stat : 5674 (python) S 5419 5419 5229 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538432610 475246592 13587 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 514 18446604467297976768 0 0 -1 5 0 0 0
[pid=5572] ppid=5419 vsize=474364 CPUtime=0.61 cores=1,3,5,7
/proc/5572/stat : 5572 (python) S 5419 5419 5229 0 -1 4202560 1234 0 0 0 32 29 0 0 15 0 2 0 538427747 485748736 13585 33554432000 4194304 5683644 140734995986208 18446744073709551615 212436037190 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/5572/statm: 118591 13585 401 364 0 54461 0
[pid=5574] ppid=5572 vsize=268424 CPUtime=361.97 cores=1,3,5,7
/proc/5574/stat : 5574 (hybridGM3) R 5572 5574 5574 0 -1 4202496 8104 0 0 0 36196 1 0 0 25 0 1 0 538427748 274866176 7823 33554432000 134512640 135204385 4289311360 18446744073709551615 134536698 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/5574/statm: 67106 7823 64 169 0 66936 0
[pid=5572/tid=5701] ppid=5419 vsize=474364 CPUtime=0 cores=1,3,5,7
/proc/5572/task/5701/stat : 5701 (python) S 5419 5419 5229 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 538447394 485748736 13585 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 514 18446604467297977664 0 0 -1 1 0 0 0
[pid=5675] ppid=5419 vsize=484620 CPUtime=0.52 cores=1,3,5,7
/proc/5675/stat : 5675 (python) S 5419 5419 5229 0 -1 4202560 1225 0 0 0 29 23 0 0 15 0 2 0 538432615 496250880 13588 33554432000 4194304 5683644 140734995986208 18446744073709551615 212436037190 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/5675/statm: 121155 13588 400 364 0 57022 0
[pid=5677] ppid=5675 vsize=268428 CPUtime=313.4 cores=1,3,5,7
/proc/5677/stat : 5677 (hybridGM3) R 5675 5677 5677 0 -1 4202496 8101 0 0 0 31339 1 0 0 25 0 1 0 538432615 274870272 7824 33554432000 134512640 135204385 4287326160 18446744073709551615 134537210 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/5677/statm: 67107 7824 64 169 0 66937 0
[pid=5675/tid=5703] ppid=5419 vsize=484620 CPUtime=0 cores=1,3,5,7
/proc/5675/task/5703/stat : 5703 (python) S 5419 5419 5229 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 538452259 496250880 13588 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 514 18446604470076574656 0 0 -1 5 0 0 0
[pid=5678] ppid=5419 vsize=494876 CPUtime=0.32 cores=1,3,5,7
/proc/5678/stat : 5678 (python) S 5419 5419 5229 0 -1 4202560 1213 0 0 0 16 16 0 0 16 0 2 0 538435004 506753024 13592 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/5678/statm: 123719 13592 400 364 0 59583 0
[pid=5680] ppid=5678 vsize=268424 CPUtime=196.43 cores=1,3,5,7
/proc/5680/stat : 5680 (hybridGM3) T 5678 5680 5680 0 -1 4202496 8101 0 0 0 19642 1 0 0 25 0 1 0 538435005 274866176 7823 33554432000 134512640 135204385 4290064624 18446744073709551615 134537435 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/5680/statm: 67106 7823 64 169 0 66936 0
[pid=5678/tid=5713] ppid=5419 vsize=494876 CPUtime=0 cores=1,3,5,7
/proc/5678/task/5713/stat : 5713 (python) S 5419 5419 5229 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 538454649 506753024 13592 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 514 18446604471162067776 0 0 -1 1 0 0 0
[pid=5714] ppid=5419 vsize=505132 CPUtime=0.16 cores=1,3,5,7
/proc/5714/stat : 5714 (python) S 5419 5419 5229 0 -1 4202560 1055 0 0 0 9 7 0 0 18 0 1 0 538454653 517255168 13566 33554432000 4194304 5683644 140734995986208 18446744073709551615 212436037190 0 0 16781312 514 17296918793846489350 0 0 17 1 0 0 0
/proc/5714/statm: 126283 13566 371 364 0 62144 0
[pid=5716] ppid=5714 vsize=431704 CPUtime=93.84 cores=1,3,5,7
/proc/5716/stat : 5716 (adaptg2wsat2009) R 5714 5716 5716 0 -1 4202496 6262 0 0 0 9384 0 0 0 25 0 1 0 538454653 442064896 5982 33554432000 134512640 135107411 4287831200 18446744073709551615 134532313 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/5716/statm: 107926 5982 59 146 0 106802 0
[pid=5419/tid=5442] ppid=5417 vsize=515376 CPUtime=566.14 cores=1,3,5,7
/proc/5419/task/5442/stat : 5442 (python) S 5417 5419 5229 0 -1 4202560 29 19664 0 0 0 0 56561 53 18 0 21 0 538343730 527745024 15113 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 3 0 0 0
[pid=5419/tid=5445] ppid=5417 vsize=515376 CPUtime=566.14 cores=1,3,5,7
/proc/5419/task/5445/stat : 5445 (python) S 5417 5419 5229 0 -1 4202560 27 19664 0 0 0 0 56561 53 15 0 21 0 538343733 527745024 15113 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 7 0 0 0
[pid=5419/tid=5448] ppid=5417 vsize=515376 CPUtime=566.14 cores=1,3,5,7
/proc/5419/task/5448/stat : 5448 (python) S 5417 5419 5229 0 -1 4202560 31 19664 0 0 0 0 56561 53 15 0 21 0 538343736 527745024 15113 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 3 0 0 0
[pid=5419/tid=5451] ppid=5417 vsize=515376 CPUtime=566.14 cores=1,3,5,7
/proc/5419/task/5451/stat : 5451 (python) S 5417 5419 5229 0 -1 4202560 28 19664 0 0 0 0 56561 53 18 0 21 0 538343739 527745024 15113 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 1 0 0 0
[pid=5419/tid=5479] ppid=5417 vsize=515376 CPUtime=566.14 cores=1,3,5,7
/proc/5419/task/5479/stat : 5479 (python) S 5417 5419 5229 0 -1 4202560 32 19664 0 0 0 0 56561 53 20 0 21 0 538363276 527745024 15113 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 7 0 0 0
[pid=5419/tid=5488] ppid=5417 vsize=515376 CPUtime=566.14 cores=1,3,5,7
/proc/5419/task/5488/stat : 5488 (python) S 5417 5419 5229 0 -1 4202560 36 19664 0 0 0 0 56561 53 18 0 21 0 538368022 527745024 15113 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 5 0 0 0
[pid=5419/tid=5501] ppid=5417 vsize=515376 CPUtime=566.14 cores=1,3,5,7
/proc/5419/task/5501/stat : 5501 (python) S 5417 5419 5229 0 -1 4202560 41 19664 0 0 0 0 56561 53 18 0 21 0 538373293 527745024 15113 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 5 0 0 0
[pid=5419/tid=5504] ppid=5417 vsize=515376 CPUtime=566.14 cores=1,3,5,7
/proc/5419/task/5504/stat : 5504 (python) S 5417 5419 5229 0 -1 4202560 95 19664 0 0 0 0 56561 53 18 0 21 0 538373385 527745024 15113 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 5 0 0 0
[pid=5419/tid=5515] ppid=5417 vsize=515376 CPUtime=566.14 cores=1,3,5,7
/proc/5419/task/5515/stat : 5515 (python) S 5417 5419 5229 0 -1 4202560 30 19664 0 0 0 0 56561 53 20 0 21 0 538382912 527745024 15113 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 5 0 0 0
[pid=5419/tid=5527] ppid=5417 vsize=515376 CPUtime=566.14 cores=1,3,5,7
/proc/5419/task/5527/stat : 5527 (python) S 5417 5419 5229 0 -1 4202560 35 19664 0 0 0 0 56561 53 18 0 21 0 538397785 527745024 15113 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 5 0 0 0
[pid=5419/tid=5535] ppid=5417 vsize=515376 CPUtime=566.14 cores=1,3,5,7
/proc/5419/task/5535/stat : 5535 (python) S 5417 5419 5229 0 -1 4202560 30 19664 0 0 0 0 56561 53 20 0 21 0 538403147 527745024 15113 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 5 0 0 0
[pid=5419/tid=5539] ppid=5417 vsize=515376 CPUtime=566.14 cores=1,3,5,7
/proc/5419/task/5539/stat : 5539 (python) S 5417 5419 5229 0 -1 4202560 33 19664 0 0 0 0 56561 53 18 0 21 0 538405236 527745024 15113 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 1 0 0 0
[pid=5419/tid=5543] ppid=5417 vsize=515376 CPUtime=566.14 cores=1,3,5,7
/proc/5419/task/5543/stat : 5543 (python) S 5417 5419 5229 0 -1 4202560 26 19664 0 0 0 0 56561 53 20 0 21 0 538407804 527745024 15113 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 1 0 0 0
[pid=5419/tid=5550] ppid=5417 vsize=515376 CPUtime=566.14 cores=1,3,5,7
/proc/5419/task/5550/stat : 5550 (python) S 5417 5419 5229 0 -1 4202560 26 19664 0 0 0 0 56561 53 20 0 21 0 538413166 527745024 15113 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 1 0 0 0
[pid=5419/tid=5555] ppid=5417 vsize=515376 CPUtime=566.14 cores=1,3,5,7
/proc/5419/task/5555/stat : 5555 (python) S 5417 5419 5229 0 -1 4202560 27 19664 0 0 0 0 56561 53 20 0 21 0 538417823 527745024 15113 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 1 0 0 0
[pid=5419/tid=5565] ppid=5417 vsize=515376 CPUtime=566.14 cores=1,3,5,7
/proc/5419/task/5565/stat : 5565 (python) S 5417 5419 5229 0 -1 4202560 27 19664 0 0 0 0 56561 53 20 0 21 0 538422691 527745024 15113 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 1 0 0 0
[pid=5419/tid=5573] ppid=5417 vsize=515376 CPUtime=566.14 cores=1,3,5,7
/proc/5419/task/5573/stat : 5573 (python) S 5417 5419 5229 0 -1 4202560 33 19664 0 0 0 0 56561 53 18 0 21 0 538427748 527745024 15113 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 1 0 0 0
[pid=5419/tid=5676] ppid=5417 vsize=515376 CPUtime=566.14 cores=1,3,5,7
/proc/5419/task/5676/stat : 5676 (python) S 5417 5419 5229 0 -1 4202560 39 19664 0 0 0 0 56561 53 18 0 21 0 538432615 527745024 15113 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 1 0 0 0
[pid=5419/tid=5679] ppid=5417 vsize=515376 CPUtime=566.14 cores=1,3,5,7
/proc/5419/task/5679/stat : 5679 (python) S 5417 5419 5229 0 -1 4202560 27 19664 0 0 0 0 56561 53 20 0 21 0 538435004 527745024 15113 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 1 0 0 0
[pid=5419/tid=5715] ppid=5417 vsize=515376 CPUtime=566.14 cores=1,3,5,7
/proc/5419/task/5715/stat : 5715 (python) S 5417 5419 5229 0 -1 4202560 27 19664 0 0 0 0 56561 53 20 0 21 0 538454653 527745024 15113 33554432000 4194304 5683644 140734995986208 18446744073709551615 212447840689 0 0 16781312 2 18446604471162066880 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.21
Current children cumulated vsize (KiB) 14368808

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 5419 and gives
#  childrusage.ru_utime.tv_sec=573
#  childrusage.ru_utime.tv_usec=935748
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=742887
# CPU time returned by wait4() is 574.679
# while last known CPU time is 4800.21
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1210.85
CPU time (s): 4800.21
CPU user time (s): 4796.21
CPU system time (s): 4
CPU usage (%): 396.434
Max. virtual memory (cumulated for all children) (KiB): 14368812

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 573.936
system time used= 0.742887
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 62158
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 3695
involuntary context switches= 5189

runsolver used 6.33704 second user time and 13.5469 second system time

The end

Launcher Data 

Begin job on node102 at 2011-05-01 18:30:06
IDJOB=3317124
IDBENCH=82589
IDSOLVER=1705
FILE ID=node102/3317124-1304267406
RUNJOBID= node102-1304267295-5248
PBS_JOBID= 13173976
Free space on /tmp= 72704 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v35000-c147000-S448503257-098-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317124-1304267406/watcher-3317124-1304267406 -o /tmp/evaluation-result-3317124-1304267406/solver-3317124-1304267406 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317124-1304267406.cnf -seed 1558148818 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 0d20733a6f7d3b01f88ae7bba5f0d271
RANDOM SEED=1558148818

node102.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30249404 kB
Buffers:        648016 kB
Cached:        1509448 kB
SwapCached:          0 kB
Active:         414148 kB
Inactive:      1863456 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30249404 kB
SwapTotal:    67111528 kB
SwapFree:     67111416 kB
Dirty:           38368 kB
Writeback:           0 kB
AnonPages:      120392 kB
Mapped:          21244 kB
Slab:           358144 kB
PageTables:       7944 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  3324972 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72600 MiB
End job on node102 at 2011-05-01 18:50:19