runsolver Copyright (C) 2010-2011 Olivier ROUSSEL
This is runsolver version 3.3.0 (svn: 977)
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3448911-1307052500/watcher-3448911-1307052500 -o /tmp/evaluation-result-3448911-1307052500/solver-3448911-1307052500 -C 1200 -W 1300 -M 7680 python portfolio_solver.py -t 1200 -s 912703200 HOME/instance-3448911-1307052500.cnf
running on 2 cores: 5,7
Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1200 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1230 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 7864320 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 7915520 KiB
Current StackSize limit: 10240 KiB
[startup+0 s]
/proc/loadavg: 0.83 0.18 0.06 7/389 22425
/proc/meminfo: memFree=23469888/32951124 swapFree=49244120/67111528
[pid=22425] ppid=22420 vsize=11796 CPUtime=0 cores=5,7
/proc/22425/stat : 22425 (runsolver) R 22420 22425 22373 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 1 0 816854425 12079104 54 33554432000 4194304 5432328 140736819756896 18446744073709551615 5055479 0 2147483391 4096 24578 0 0 0 17 7 0 0 0
/proc/22425/statm: 2949 54 34 303 0 2643 0
[startup+0.05506 s]
/proc/loadavg: 0.83 0.18 0.06 7/389 22425
/proc/meminfo: memFree=23469888/32951124 swapFree=49244120/67111528
[pid=22425] ppid=22420 vsize=15188 CPUtime=0 cores=5,7
/proc/22425/stat : 22425 (python) S 22420 22425 22373 0 -1 4202496 724 0 4 0 0 0 0 0 17 0 1 0 816854425 15552512 618 33554432000 4194304 5683644 140733259310704 18446744073709551615 218370953140 0 2147483391 16781312 2 18446744071699945619 0 0 17 7 0 0 1
/proc/22425/statm: 3797 618 322 364 0 284 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15188
[startup+0.101051 s]
/proc/loadavg: 0.83 0.18 0.06 7/389 22425
/proc/meminfo: memFree=23469888/32951124 swapFree=49244120/67111528
[pid=22425] ppid=22420 vsize=15908 CPUtime=0 cores=5,7
/proc/22425/stat : 22425 (python) S 22420 22425 22373 0 -1 4202496 940 0 4 0 0 0 0 0 17 0 1 0 816854425 16289792 821 33554432000 4194304 5683644 140733259310704 18446744073709551615 218370955264 0 2147483391 16781312 2 18446744071699945619 0 0 17 7 0 0 1
/proc/22425/statm: 3977 821 357 364 0 464 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15908
[startup+0.301019 s]
/proc/loadavg: 0.83 0.18 0.06 7/389 22425
/proc/meminfo: memFree=23469888/32951124 swapFree=49244120/67111528
[pid=22425] ppid=22420 vsize=21896 CPUtime=0.03 cores=5,7
/proc/22425/stat : 22425 (python) S 22420 22425 22373 0 -1 4202496 1512 0 5 0 1 2 0 0 17 0 1 0 816854425 22421504 1350 33554432000 4194304 5683644 140733259310704 18446744073709551615 218370957056 0 0 16781312 2 18446744071562362840 0 0 17 7 0 0 1
/proc/22425/statm: 5474 1350 443 364 0 929 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 21896
[startup+0.700946 s]
/proc/loadavg: 0.83 0.18 0.06 7/389 22425
/proc/meminfo: memFree=23469888/32951124 swapFree=49244120/67111528
[pid=22425] ppid=22420 vsize=21896 CPUtime=0.03 cores=5,7
/proc/22425/stat : 22425 (python) S 22420 22425 22373 0 -1 4202496 1512 0 5 0 1 2 0 0 17 0 1 0 816854425 22421504 1350 33554432000 4194304 5683644 140733259310704 18446744073709551615 218370957056 0 0 16781312 2 18446744071562362840 0 0 17 7 0 0 1
/proc/22425/statm: 5474 1350 443 364 0 929 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 21896
[startup+1.50084 s]
/proc/loadavg: 0.83 0.18 0.06 7/403 22439
/proc/meminfo: memFree=23414956/32951124 swapFree=49244120/67111528
[pid=22425] ppid=22420 vsize=21896 CPUtime=0.03 cores=5,7
/proc/22425/stat : 22425 (python) S 22420 22425 22373 0 -1 4202496 1512 0 5 0 1 2 0 0 17 0 1 0 816854425 22421504 1350 33554432000 4194304 5683644 140733259310704 18446744073709551615 218370957056 0 0 16781312 2 18446744071562362840 0 0 17 7 0 0 1
/proc/22425/statm: 5474 1350 443 364 0 929 0
[pid=22437] ppid=22425 vsize=8364 CPUtime=1.29 cores=5,7
/proc/22437/stat : 22437 (features) R 22425 22425 22373 0 -1 4202496 2609 0 0 0 127 2 0 0 21 0 1 0 816854444 8564736 1492 33554432000 134512640 135907751 4287453328 18446744073709551615 135392733 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22437/statm: 2091 1492 111 341 0 1747 0
Current children cumulated CPU time (s) 1.32
Current children cumulated vsize (KiB) 30260
[startup+3.10055 s]
/proc/loadavg: 0.83 0.18 0.06 6/403 22439
/proc/meminfo: memFree=23294584/32951124 swapFree=49244120/67111528
[pid=22425] ppid=22420 vsize=21896 CPUtime=0.03 cores=5,7
/proc/22425/stat : 22425 (python) S 22420 22425 22373 0 -1 4202496 1512 0 5 0 1 2 0 0 17 0 1 0 816854425 22421504 1350 33554432000 4194304 5683644 140733259310704 18446744073709551615 218370957056 0 0 16781312 2 18446744071562362840 0 0 17 7 0 0 1
/proc/22425/statm: 5474 1350 443 364 0 929 0
[pid=22437] ppid=22425 vsize=169240 CPUtime=2.89 cores=5,7
/proc/22437/stat : 22437 (features) R 22425 22425 22373 0 -1 4202496 19920 0 0 0 281 8 0 0 25 0 1 0 816854444 173301760 17898 33554432000 134512640 135907751 4287453328 18446744073709551615 135396606 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22437/statm: 42310 17898 112 341 0 41966 0
Current children cumulated CPU time (s) 2.92
Current children cumulated vsize (KiB) 191136
[startup+6.31498 s]
/proc/loadavg: 1.32 0.29 0.10 6/403 22439
/proc/meminfo: memFree=22748612/32951124 swapFree=49244120/67111528
[pid=22425] ppid=22420 vsize=21896 CPUtime=0.03 cores=5,7
/proc/22425/stat : 22425 (python) S 22420 22425 22373 0 -1 4202496 1512 0 5 0 1 2 0 0 17 0 1 0 816854425 22421504 1350 33554432000 4194304 5683644 140733259310704 18446744073709551615 218370957056 0 0 16781312 2 18446744071562362840 0 0 17 7 0 0 1
/proc/22425/statm: 5474 1350 443 364 0 929 0
[pid=22437] ppid=22425 vsize=276996 CPUtime=6.11 cores=5,7
/proc/22437/stat : 22437 (features) R 22425 22425 22373 0 -1 4202496 55915 0 0 0 597 14 0 0 25 0 1 0 816854444 283643904 53893 33554432000 134512640 135907751 4287453328 18446744073709551615 135340715 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22437/statm: 69249 53893 112 341 0 68905 0
Current children cumulated CPU time (s) 6.14
Current children cumulated vsize (KiB) 298892
[startup+12.7009 s]
/proc/loadavg: 1.54 0.35 0.12 6/403 22439
/proc/meminfo: memFree=21718168/32951124 swapFree=49244120/67111528
[pid=22425] ppid=22420 vsize=21896 CPUtime=0.03 cores=5,7
/proc/22425/stat : 22425 (python) S 22420 22425 22373 0 -1 4202496 1512 0 5 0 1 2 0 0 17 0 1 0 816854425 22421504 1350 33554432000 4194304 5683644 140733259310704 18446744073709551615 218370957056 0 0 16781312 2 18446744071562362840 0 0 17 7 0 0 1
/proc/22425/statm: 5474 1350 443 364 0 929 0
[pid=22437] ppid=22425 vsize=750400 CPUtime=12.5 cores=5,7
/proc/22437/stat : 22437 (features) R 22425 22425 22373 0 -1 4202496 180196 0 0 0 1219 31 0 0 25 0 1 0 816854444 768409600 178172 33554432000 134512640 135907751 4287453328 18446744073709551615 134534779 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22437/statm: 187600 178172 112 341 0 187256 0
Current children cumulated CPU time (s) 12.53
Current children cumulated vsize (KiB) 772296
[startup+25.5006 s]
/proc/loadavg: 2.16 0.55 0.18 6/403 22440
/proc/meminfo: memFree=21718184/32951124 swapFree=49244120/67111528
[pid=22425] ppid=22420 vsize=21896 CPUtime=0.03 cores=5,7
/proc/22425/stat : 22425 (python) S 22420 22425 22373 0 -1 4202496 1512 0 5 0 1 2 0 0 17 0 1 0 816854425 22421504 1350 33554432000 4194304 5683644 140733259310704 18446744073709551615 218370957056 0 0 16781312 2 18446744071562362840 0 0 17 7 0 0 1
/proc/22425/statm: 5474 1350 443 364 0 929 0
[pid=22437] ppid=22425 vsize=750400 CPUtime=25.3 cores=5,7
/proc/22437/stat : 22437 (features) R 22425 22425 22373 0 -1 4202496 180211 0 0 0 2499 31 0 0 25 0 1 0 816854444 768409600 178187 33554432000 134512640 135907751 4287453328 18446744073709551615 134534342 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22437/statm: 187600 178187 112 341 0 187256 0
Current children cumulated CPU time (s) 25.33
Current children cumulated vsize (KiB) 772296
[startup+51.101 s]
/proc/loadavg: 2.78 0.82 0.28 5/403 22440
/proc/meminfo: memFree=21716712/32951124 swapFree=49244120/67111528
[pid=22425] ppid=22420 vsize=21896 CPUtime=0.03 cores=5,7
/proc/22425/stat : 22425 (python) S 22420 22425 22373 0 -1 4202496 1512 0 5 0 1 2 0 0 17 0 1 0 816854425 22421504 1350 33554432000 4194304 5683644 140733259310704 18446744073709551615 218370957056 0 0 16781312 2 18446744071562362840 0 0 17 7 0 0 1
/proc/22425/statm: 5474 1350 443 364 0 929 0
[pid=22437] ppid=22425 vsize=750400 CPUtime=50.9 cores=5,7
/proc/22437/stat : 22437 (features) R 22425 22425 22373 0 -1 4202496 180241 0 0 0 5059 31 0 0 25 0 1 0 816854444 768409600 178217 33554432000 134512640 135907751 4287453328 18446744073709551615 134534779 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22437/statm: 187600 178217 112 341 0 187256 0
Current children cumulated CPU time (s) 50.93
Current children cumulated vsize (KiB) 772296
[startup+102.301 s]
/proc/loadavg: 3.47 1.31 0.47 6/415 22492
/proc/meminfo: memFree=23004360/32951124 swapFree=49244120/67111528
[pid=22425] ppid=22420 vsize=34396 CPUtime=8.08 cores=5,7
/proc/22425/stat : 22425 (python) S 22420 22425 22373 0 -1 4202496 2603 301568 5 0 2 2 751 53 18 0 1 0 816854425 35221504 1407 33554432000 4194304 5683644 140733259310704 18446744073709551615 218370957056 0 0 16781312 2 18446744071562362840 0 0 17 5 0 0 1
/proc/22425/statm: 8599 1407 468 364 0 933 0
[pid=22437] ppid=22425 vsize=0 CPUtime=89.86 cores=5,7
/proc/22437/stat : 22437 (features) Z 22425 22425 22373 0 -1 4202524 226908 0 0 0 8941 45 0 0 25 0 1 0 816854444 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 7 0 0 0
/proc/22437/statm: 0 0 0 0 0 0 0
[pid=22457] ppid=22425 vsize=0 CPUtime=0.32 cores=5,7
/proc/22457/stat : 22457 (coach) Z 22425 22425 22373 0 -1 4202524 3736 0 0 0 32 0 0 0 20 0 1 0 816863430 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 5 0 0 0
/proc/22457/statm: 0 0 0 0 0 0 0
[pid=22481] ppid=22425 vsize=8700 CPUtime=0 cores=5,7
/proc/22481/stat : 22481 (timerun) S 22425 22425 22373 0 -1 4202496 745 0 0 0 0 0 0 0 18 0 1 0 816864270 8908800 247 33554432000 4194304 4922060 140733552405696 18446744073709551615 218370776197 0 65536 16781316 65538 18446744071562232103 0 0 17 7 0 0 0
/proc/22481/statm: 2175 247 208 178 0 68 0
[pid=22482] ppid=22481 vsize=267068 CPUtime=3.84 cores=5,7
/proc/22482/stat : 22482 (march_hi) R 22481 22425 22373 0 -1 4194304 61698 0 0 0 373 11 0 0 25 0 1 0 816864270 273477632 61643 33554432000 134512640 134628092 4292116208 18446744073709551615 8287740 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22482/statm: 66767 61643 103 29 0 66348 0
Current children cumulated CPU time (s) 102.1
Current children cumulated vsize (KiB) 310164
[startup+162.3 s]
/proc/loadavg: 3.80 1.80 0.69 6/415 22535
/proc/meminfo: memFree=21512964/32951124 swapFree=49244120/67111528
[pid=22425] ppid=22420 vsize=34396 CPUtime=24.49 cores=5,7
/proc/22425/stat : 22425 (python) S 22420 22425 22373 0 -1 4202496 3633 616199 5 0 2 2 2331 114 18 0 1 0 816854425 35221504 1407 33554432000 4194304 5683644 140733259310704 18446744073709551615 218370957056 0 0 16781312 2 18446744071562362840 0 0 17 7 0 0 1
/proc/22425/statm: 8599 1407 468 364 0 933 0
[pid=22437] ppid=22425 vsize=0 CPUtime=89.86 cores=5,7
/proc/22437/stat : 22437 (features) Z 22425 22425 22373 0 -1 4202524 226908 0 0 0 8941 45 0 0 25 0 1 0 816854444 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 7 0 0 0
/proc/22437/statm: 0 0 0 0 0 0 0
[pid=22457] ppid=22425 vsize=0 CPUtime=0.32 cores=5,7
/proc/22457/stat : 22457 (coach) Z 22425 22425 22373 0 -1 4202524 3736 0 0 0 32 0 0 0 20 0 1 0 816863430 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 5 0 0 0
/proc/22457/statm: 0 0 0 0 0 0 0
[pid=22529] ppid=22425 vsize=8700 CPUtime=0 cores=5,7
/proc/22529/stat : 22529 (timerun) S 22425 22425 22373 0 -1 4202496 747 0 0 0 0 0 0 0 19 0 1 0 816865911 8908800 248 33554432000 4194304 4922060 140733448530000 18446744073709551615 218370776197 0 65536 16781316 65538 18446744071562232103 0 0 17 7 0 0 0
/proc/22529/statm: 2175 248 208 178 0 68 0
[pid=22530] ppid=22529 vsize=1263684 CPUtime=47.42 cores=5,7
/proc/22530/stat : 22530 (cryptominisat) R 22529 22425 22373 0 -1 4202496 313083 0 0 0 4698 44 0 0 25 0 1 0 816865912 1294012416 256927 33554432000 134512640 134994940 4287822960 18446744073709551615 134658267 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/22530/statm: 315921 256927 354 118 0 315106 0
Current children cumulated CPU time (s) 162.09
Current children cumulated vsize (KiB) 1306780
[startup+222.301 s]
/proc/loadavg: 3.93 2.20 0.90 6/415 22537
/proc/meminfo: memFree=21365376/32951124 swapFree=49244120/67111528
[pid=22425] ppid=22420 vsize=34396 CPUtime=24.49 cores=5,7
/proc/22425/stat : 22425 (python) S 22420 22425 22373 0 -1 4202496 3633 616199 5 0 2 2 2331 114 18 0 1 0 816854425 35221504 1407 33554432000 4194304 5683644 140733259310704 18446744073709551615 218370957056 0 0 16781312 2 18446744071562362840 0 0 17 7 0 0 1
/proc/22425/statm: 8599 1407 468 364 0 933 0
[pid=22437] ppid=22425 vsize=0 CPUtime=89.86 cores=5,7
/proc/22437/stat : 22437 (features) Z 22425 22425 22373 0 -1 4202524 226908 0 0 0 8941 45 0 0 25 0 1 0 816854444 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 7 0 0 0
/proc/22437/statm: 0 0 0 0 0 0 0
[pid=22457] ppid=22425 vsize=0 CPUtime=0.32 cores=5,7
/proc/22457/stat : 22457 (coach) Z 22425 22425 22373 0 -1 4202524 3736 0 0 0 32 0 0 0 20 0 1 0 816863430 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 5 0 0 0
/proc/22457/statm: 0 0 0 0 0 0 0
[pid=22529] ppid=22425 vsize=8700 CPUtime=0 cores=5,7
/proc/22529/stat : 22529 (timerun) S 22425 22425 22373 0 -1 4202496 747 0 0 0 0 0 0 0 19 0 1 0 816865911 8908800 248 33554432000 4194304 4922060 140733448530000 18446744073709551615 218370776197 0 65536 16781316 65538 18446744071562232103 0 0 17 7 0 0 0
/proc/22529/statm: 2175 248 208 178 0 68 0
[pid=22530] ppid=22529 vsize=1330728 CPUtime=107.43 cores=5,7
/proc/22530/stat : 22530 (cryptominisat) R 22529 22425 22373 0 -1 4202496 361130 0 0 0 10688 55 0 0 25 0 1 0 816865912 1362665472 273766 33554432000 134512640 134994940 4287822960 18446744073709551615 134658216 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/22530/statm: 332682 273766 357 118 0 331867 0
Current children cumulated CPU time (s) 222.1
Current children cumulated vsize (KiB) 1373824
[startup+282.301 s]
################
# More data... #
################
[startup+942.301 s]
/proc/loadavg: 4.07 3.93 2.65 5/415 22711
/proc/meminfo: memFree=19659908/32951124 swapFree=49244120/67111528
[pid=22425] ppid=22420 vsize=35172 CPUtime=266.94 cores=5,7
/proc/22425/stat : 22425 (python) S 22420 22425 22373 0 -1 4202496 4562 13056263 5 0 3 4 24509 2178 19 0 1 0 816854425 36016128 1602 33554432000 4194304 5683644 140733259310704 18446744073709551615 218370957056 0 0 16781312 2 18446744071562362840 0 0 17 7 0 0 1
/proc/22425/statm: 8793 1602 468 364 0 1127 0
[pid=22437] ppid=22425 vsize=0 CPUtime=89.86 cores=5,7
/proc/22437/stat : 22437 (features) Z 22425 22425 22373 0 -1 4202524 226908 0 0 0 8941 45 0 0 25 0 1 0 816854444 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 7 0 0 0
/proc/22437/statm: 0 0 0 0 0 0 0
[pid=22457] ppid=22425 vsize=0 CPUtime=0.32 cores=5,7
/proc/22457/stat : 22457 (coach) Z 22425 22425 22373 0 -1 4202524 3736 0 0 0 32 0 0 0 20 0 1 0 816863430 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 5 0 0 0
/proc/22457/statm: 0 0 0 0 0 0 0
[pid=22591] ppid=22425 vsize=8700 CPUtime=0 cores=5,7
/proc/22591/stat : 22591 (timerun) S 22425 22425 22373 0 -1 4202496 748 0 0 0 0 0 0 0 19 0 1 0 816891838 8908800 248 33554432000 4194304 4922060 140735384880544 18446744073709551615 218370776197 0 65536 16781316 65538 18446744071562232103 0 0 17 7 0 0 0
/proc/22591/statm: 2175 248 208 178 0 68 0
[pid=22592] ppid=22591 vsize=577896 CPUtime=568.18 cores=5,7
/proc/22592/stat : 22592 (mxc-sat09) R 22591 22425 22373 0 -1 4202496 170367 0 0 0 56793 25 0 0 25 0 1 0 816891839 591765504 126490 33554432000 134512640 134581694 4290585648 18446744073709551615 134530724 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22592/statm: 144474 126490 205 17 0 143821 0
Current children cumulated CPU time (s) 925.3
Current children cumulated vsize (KiB) 621768
[startup+1002.3 s]
/proc/loadavg: 4.12 3.97 2.74 5/415 22713
/proc/meminfo: memFree=19651028/32951124 swapFree=49244120/67111528
[pid=22425] ppid=22420 vsize=35172 CPUtime=266.94 cores=5,7
/proc/22425/stat : 22425 (python) S 22420 22425 22373 0 -1 4202496 4562 13056263 5 0 3 4 24509 2178 19 0 1 0 816854425 36016128 1602 33554432000 4194304 5683644 140733259310704 18446744073709551615 218370957056 0 0 16781312 2 18446744071562362840 0 0 17 7 0 0 1
/proc/22425/statm: 8793 1602 468 364 0 1127 0
[pid=22437] ppid=22425 vsize=0 CPUtime=89.86 cores=5,7
/proc/22437/stat : 22437 (features) Z 22425 22425 22373 0 -1 4202524 226908 0 0 0 8941 45 0 0 25 0 1 0 816854444 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 7 0 0 0
/proc/22437/statm: 0 0 0 0 0 0 0
[pid=22457] ppid=22425 vsize=0 CPUtime=0.32 cores=5,7
/proc/22457/stat : 22457 (coach) Z 22425 22425 22373 0 -1 4202524 3736 0 0 0 32 0 0 0 20 0 1 0 816863430 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 5 0 0 0
/proc/22457/statm: 0 0 0 0 0 0 0
[pid=22591] ppid=22425 vsize=8700 CPUtime=0 cores=5,7
/proc/22591/stat : 22591 (timerun) S 22425 22425 22373 0 -1 4202496 748 0 0 0 0 0 0 0 19 0 1 0 816891838 8908800 248 33554432000 4194304 4922060 140735384880544 18446744073709551615 218370776197 0 65536 16781316 65538 18446744071562232103 0 0 17 7 0 0 0
/proc/22591/statm: 2175 248 208 178 0 68 0
[pid=22592] ppid=22591 vsize=577896 CPUtime=628.17 cores=5,7
/proc/22592/stat : 22592 (mxc-sat09) R 22591 22425 22373 0 -1 4202496 170367 0 0 0 62792 25 0 0 25 0 1 0 816891839 591765504 126490 33554432000 134512640 134581694 4290585648 18446744073709551615 134530374 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22592/statm: 144474 126490 205 17 0 143821 0
Current children cumulated CPU time (s) 985.29
Current children cumulated vsize (KiB) 621768
[startup+1062.31 s]
/proc/loadavg: 4.04 3.97 2.82 5/415 22714
/proc/meminfo: memFree=19641520/32951124 swapFree=49244120/67111528
[pid=22425] ppid=22420 vsize=35172 CPUtime=266.94 cores=5,7
/proc/22425/stat : 22425 (python) S 22420 22425 22373 0 -1 4202496 4562 13056263 5 0 3 4 24509 2178 19 0 1 0 816854425 36016128 1602 33554432000 4194304 5683644 140733259310704 18446744073709551615 218370957056 0 0 16781312 2 18446744071562362840 0 0 17 7 0 0 1
/proc/22425/statm: 8793 1602 468 364 0 1127 0
[pid=22437] ppid=22425 vsize=0 CPUtime=89.86 cores=5,7
/proc/22437/stat : 22437 (features) Z 22425 22425 22373 0 -1 4202524 226908 0 0 0 8941 45 0 0 25 0 1 0 816854444 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 7 0 0 0
/proc/22437/statm: 0 0 0 0 0 0 0
[pid=22457] ppid=22425 vsize=0 CPUtime=0.32 cores=5,7
/proc/22457/stat : 22457 (coach) Z 22425 22425 22373 0 -1 4202524 3736 0 0 0 32 0 0 0 20 0 1 0 816863430 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 5 0 0 0
/proc/22457/statm: 0 0 0 0 0 0 0
[pid=22591] ppid=22425 vsize=8700 CPUtime=0 cores=5,7
/proc/22591/stat : 22591 (timerun) S 22425 22425 22373 0 -1 4202496 748 0 0 0 0 0 0 0 19 0 1 0 816891838 8908800 248 33554432000 4194304 4922060 140735384880544 18446744073709551615 218370776197 0 65536 16781316 65538 18446744071562232103 0 0 17 7 0 0 0
/proc/22591/statm: 2175 248 208 178 0 68 0
[pid=22592] ppid=22591 vsize=588272 CPUtime=688.18 cores=5,7
/proc/22592/stat : 22592 (mxc-sat09) R 22591 22425 22373 0 -1 4202496 172948 0 0 0 68793 25 0 0 25 0 1 0 816891839 602390528 129071 33554432000 134512640 134581694 4290585648 18446744073709551615 134530775 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22592/statm: 147068 129071 205 17 0 146415 0
Current children cumulated CPU time (s) 1045.3
Current children cumulated vsize (KiB) 632144
[startup+1122.3 s]
/proc/loadavg: 4.01 3.97 2.89 6/415 22716
/proc/meminfo: memFree=19621356/32951124 swapFree=49244120/67111528
[pid=22425] ppid=22420 vsize=35172 CPUtime=266.94 cores=5,7
/proc/22425/stat : 22425 (python) S 22420 22425 22373 0 -1 4202496 4562 13056263 5 0 3 4 24509 2178 19 0 1 0 816854425 36016128 1602 33554432000 4194304 5683644 140733259310704 18446744073709551615 218370957056 0 0 16781312 2 18446744071562362840 0 0 17 7 0 0 1
/proc/22425/statm: 8793 1602 468 364 0 1127 0
[pid=22437] ppid=22425 vsize=0 CPUtime=89.86 cores=5,7
/proc/22437/stat : 22437 (features) Z 22425 22425 22373 0 -1 4202524 226908 0 0 0 8941 45 0 0 25 0 1 0 816854444 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 7 0 0 0
/proc/22437/statm: 0 0 0 0 0 0 0
[pid=22457] ppid=22425 vsize=0 CPUtime=0.32 cores=5,7
/proc/22457/stat : 22457 (coach) Z 22425 22425 22373 0 -1 4202524 3736 0 0 0 32 0 0 0 20 0 1 0 816863430 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 5 0 0 0
/proc/22457/statm: 0 0 0 0 0 0 0
[pid=22591] ppid=22425 vsize=8700 CPUtime=0 cores=5,7
/proc/22591/stat : 22591 (timerun) S 22425 22425 22373 0 -1 4202496 748 0 0 0 0 0 0 0 19 0 1 0 816891838 8908800 248 33554432000 4194304 4922060 140735384880544 18446744073709551615 218370776197 0 65536 16781316 65538 18446744071562232103 0 0 17 7 0 0 0
/proc/22591/statm: 2175 248 208 178 0 68 0
[pid=22592] ppid=22591 vsize=608328 CPUtime=748.17 cores=5,7
/proc/22592/stat : 22592 (mxc-sat09) R 22591 22425 22373 0 -1 4202496 177971 0 0 0 74791 26 0 0 25 0 1 0 816891839 622927872 134094 33554432000 134512640 134581694 4290585648 18446744073709551615 134530758 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22592/statm: 152082 134094 205 17 0 151429 0
Current children cumulated CPU time (s) 1105.29
Current children cumulated vsize (KiB) 652200
[startup+1182.3 s]
/proc/loadavg: 4.04 3.99 2.96 5/415 22718
/proc/meminfo: memFree=19612352/32951124 swapFree=49244120/67111528
[pid=22425] ppid=22420 vsize=35172 CPUtime=266.94 cores=5,7
/proc/22425/stat : 22425 (python) S 22420 22425 22373 0 -1 4202496 4562 13056263 5 0 3 4 24509 2178 19 0 1 0 816854425 36016128 1602 33554432000 4194304 5683644 140733259310704 18446744073709551615 218370957056 0 0 16781312 2 18446744071562362840 0 0 17 7 0 0 1
/proc/22425/statm: 8793 1602 468 364 0 1127 0
[pid=22437] ppid=22425 vsize=0 CPUtime=89.86 cores=5,7
/proc/22437/stat : 22437 (features) Z 22425 22425 22373 0 -1 4202524 226908 0 0 0 8941 45 0 0 25 0 1 0 816854444 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 7 0 0 0
/proc/22437/statm: 0 0 0 0 0 0 0
[pid=22457] ppid=22425 vsize=0 CPUtime=0.32 cores=5,7
/proc/22457/stat : 22457 (coach) Z 22425 22425 22373 0 -1 4202524 3736 0 0 0 32 0 0 0 20 0 1 0 816863430 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 5 0 0 0
/proc/22457/statm: 0 0 0 0 0 0 0
[pid=22591] ppid=22425 vsize=8700 CPUtime=0 cores=5,7
/proc/22591/stat : 22591 (timerun) S 22425 22425 22373 0 -1 4202496 748 0 0 0 0 0 0 0 19 0 1 0 816891838 8908800 248 33554432000 4194304 4922060 140735384880544 18446744073709551615 218370776197 0 65536 16781316 65538 18446744071562232103 0 0 17 7 0 0 0
/proc/22591/statm: 2175 248 208 178 0 68 0
[pid=22592] ppid=22591 vsize=617040 CPUtime=808.17 cores=5,7
/proc/22592/stat : 22592 (mxc-sat09) R 22591 22425 22373 0 -1 4202496 180151 0 0 0 80791 26 0 0 25 0 1 0 816891839 631848960 136274 33554432000 134512640 134581694 4290585648 18446744073709551615 134530724 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22592/statm: 154260 136274 205 17 0 153607 0
Current children cumulated CPU time (s) 1165.29
Current children cumulated vsize (KiB) 660912
Maximum CPU time exceeded: sending SIGTERM then SIGKILL
[startup+1217.1 s]
/proc/loadavg: 3.95 3.97 2.99 5/415 22755
/proc/meminfo: memFree=19782680/32951124 swapFree=49244120/67111528
[pid=22425] ppid=22420 vsize=35172 CPUtime=266.94 cores=5,7
/proc/22425/stat : 22425 (python) S 22420 22425 22373 0 -1 4202496 4562 13056263 5 0 3 4 24509 2178 19 0 1 0 816854425 36016128 1602 33554432000 4194304 5683644 140733259310704 18446744073709551615 218370957056 0 0 16781312 2 18446744071562362840 0 0 17 7 0 0 1
/proc/22425/statm: 8793 1602 468 364 0 1127 0
[pid=22437] ppid=22425 vsize=0 CPUtime=89.86 cores=5,7
/proc/22437/stat : 22437 (features) Z 22425 22425 22373 0 -1 4202524 226908 0 0 0 8941 45 0 0 25 0 1 0 816854444 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 7 0 0 0
/proc/22437/statm: 0 0 0 0 0 0 0
[pid=22457] ppid=22425 vsize=0 CPUtime=0.32 cores=5,7
/proc/22457/stat : 22457 (coach) Z 22425 22425 22373 0 -1 4202524 3736 0 0 0 32 0 0 0 20 0 1 0 816863430 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 5 0 0 0
/proc/22457/statm: 0 0 0 0 0 0 0
[pid=22591] ppid=22425 vsize=8700 CPUtime=0 cores=5,7
/proc/22591/stat : 22591 (timerun) S 22425 22425 22373 0 -1 4202496 748 0 0 0 0 0 0 0 19 0 1 0 816891838 8908800 248 33554432000 4194304 4922060 140735384880544 18446744073709551615 218370776197 0 65536 16781316 65538 18446744071562232103 0 0 17 7 0 0 0
/proc/22591/statm: 2175 248 208 178 0 68 0
[pid=22592] ppid=22591 vsize=627408 CPUtime=842.98 cores=5,7
/proc/22592/stat : 22592 (mxc-sat09) R 22591 22425 22373 0 -1 4202496 182741 0 0 0 84271 27 0 0 25 0 1 0 816891839 642465792 138864 33554432000 134512640 134581694 4290585648 18446744073709551615 134530836 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22592/statm: 156852 138864 205 17 0 156199 0
Current children cumulated CPU time (s) 1200.1
Current children cumulated vsize (KiB) 671280
Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds
Solver just ended. Dumping a history of the last processes samples
[startup+1217.1 s]
/proc/loadavg: 3.95 3.97 2.99 5/415 22755
/proc/meminfo: memFree=19782680/32951124 swapFree=49244120/67111528
[pid=22425] ppid=22420 vsize=35172 CPUtime=266.94 cores=5,7
/proc/22425/stat : 22425 (python) S 22420 22425 22373 0 -1 4202496 4562 13056263 5 0 3 4 24509 2178 19 0 1 0 816854425 36016128 1602 33554432000 4194304 5683644 140733259310704 18446744073709551615 218370957056 0 0 16781312 2 18446744071562362840 0 0 17 7 0 0 1
/proc/22425/statm: 8793 1602 468 364 0 1127 0
[pid=22437] ppid=22425 vsize=0 CPUtime=89.86 cores=5,7
/proc/22437/stat : 22437 (features) Z 22425 22425 22373 0 -1 4202524 226908 0 0 0 8941 45 0 0 25 0 1 0 816854444 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 7 0 0 0
/proc/22437/statm: 0 0 0 0 0 0 0
[pid=22457] ppid=22425 vsize=0 CPUtime=0.32 cores=5,7
/proc/22457/stat : 22457 (coach) Z 22425 22425 22373 0 -1 4202524 3736 0 0 0 32 0 0 0 20 0 1 0 816863430 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 5 0 0 0
/proc/22457/statm: 0 0 0 0 0 0 0
[pid=22591] ppid=22425 vsize=8700 CPUtime=0 cores=5,7
/proc/22591/stat : 22591 (timerun) S 22425 22425 22373 0 -1 4202496 748 0 0 0 0 0 0 0 19 0 1 0 816891838 8908800 248 33554432000 4194304 4922060 140735384880544 18446744073709551615 218370776197 0 65536 16781316 65538 18446744071562232103 0 0 17 7 0 0 0
/proc/22591/statm: 2175 248 208 178 0 68 0
[pid=22592] ppid=22591 vsize=627408 CPUtime=842.98 cores=5,7
/proc/22592/stat : 22592 (mxc-sat09) R 22591 22425 22373 0 -1 4202496 182741 0 0 0 84271 27 0 0 25 0 1 0 816891839 642465792 138864 33554432000 134512640 134581694 4290585648 18446744073709551615 134530836 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22592/statm: 156852 138864 205 17 0 156199 0
Current children cumulated CPU time (s) 1200.1
Current children cumulated vsize (KiB) 671280
Child ended because it received signal 15 (SIGTERM)
# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 22425 and gives
# childrusage.ru_utime.tv_sec=245
# childrusage.ru_utime.tv_usec=136733
# childrusage.ru_stime.tv_sec=21
# childrusage.ru_stime.tv_usec=828681
# CPU time returned by wait4() is 266.965
# while last known CPU time is 1200.1
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...
Real time (s): 1217.13
CPU time (s): 1200.1
CPU user time (s): 1177.56
CPU system time (s): 22.54
CPU usage (%): 98.6011
Max. virtual memory (cumulated for all children) (KiB): 3864376
getrusage(RUSAGE_CHILDREN,...) data:
user time used= 245.137
system time used= 21.8287
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 13060825
page faults= 5
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 9006
involuntary context switches= 2917
runsolver used 4.77327 second user time and 10.8464 second system time
The end
Begin job on node101 at 2011-06-03 00:08:35
IDJOB=3448911
IDBENCH=82404
IDSOLVER=1947
FILE ID=node101/3448911-1307052500
RUNJOBID= node101-1307052499-22395
PBS_JOBID= 13497831
Free space on /tmp= 73500 MiB
SOLVER NAME= Coach 2011-03-18
BENCH NAME= SAT11/application/rintanen/SATPlanning/transport-transport-three-cities-sequential-14nodes-1000size-4degree-100mindistance-4trucks-14packages-2008seed.040-NOTKNOWN.cnf
COMMAND LINE= python portfolio_solver.py -t TIMELIMIT -s RANDOMSEED BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3448911-1307052500/watcher-3448911-1307052500 -o /tmp/evaluation-result-3448911-1307052500/solver-3448911-1307052500 -C 1200 -W 1300 -M 7680 python portfolio_solver.py -t 1200 -s 912703200 HOME/instance-3448911-1307052500.cnf
TIME LIMIT= 1200 seconds
MEMORY LIMIT= 7680 MiB
NBCORE= 2
MD5SUM BENCH= 6c2d25eda70423643946d76935ceb950
RANDOM SEED=912703200
node101.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009
/proc/cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5550 @ 2.67GHz
stepping : 5
cpu MHz : 2666.841
cache size : 8192 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips : 5333.68
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: [8]
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5550 @ 2.67GHz
stepping : 5
cpu MHz : 2666.841
cache size : 8192 KB
physical id : 1
siblings : 4
core id : 0
cpu cores : 4
apicid : 16
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips : 5226.07
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: [8]
processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5550 @ 2.67GHz
stepping : 5
cpu MHz : 2666.841
cache size : 8192 KB
physical id : 0
siblings : 4
core id : 1
cpu cores : 4
apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips : 5332.77
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: [8]
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5550 @ 2.67GHz
stepping : 5
cpu MHz : 2666.841
cache size : 8192 KB
physical id : 1
siblings : 4
core id : 1
cpu cores : 4
apicid : 18
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips : 5332.83
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: [8]
processor : 4
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5550 @ 2.67GHz
stepping : 5
cpu MHz : 2666.841
cache size : 8192 KB
physical id : 0
siblings : 4
core id : 2
cpu cores : 4
apicid : 4
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips : 5332.74
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: [8]
processor : 5
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5550 @ 2.67GHz
stepping : 5
cpu MHz : 2666.841
cache size : 8192 KB
physical id : 1
siblings : 4
core id : 2
cpu cores : 4
apicid : 20
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips : 5332.83
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: [8]
processor : 6
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5550 @ 2.67GHz
stepping : 5
cpu MHz : 2666.841
cache size : 8192 KB
physical id : 0
siblings : 4
core id : 3
cpu cores : 4
apicid : 6
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips : 5332.75
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: [8]
processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5550 @ 2.67GHz
stepping : 5
cpu MHz : 2666.841
cache size : 8192 KB
physical id : 1
siblings : 4
core id : 3
cpu cores : 4
apicid : 22
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips : 5332.83
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: [8]
/proc/meminfo:
MemTotal: 32951124 kB
MemFree: 23470340 kB
Buffers: 194828 kB
Cached: 869604 kB
SwapCached: 27120 kB
Active: 8907568 kB
Inactive: 293924 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 32951124 kB
LowFree: 23470340 kB
SwapTotal: 67111528 kB
SwapFree: 49244120 kB
Dirty: 548860 kB
Writeback: 0 kB
AnonPages: 8132176 kB
Mapped: 17356 kB
Slab: 136824 kB
PageTables: 75020 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 83587088 kB
Committed_AS: 78588612 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
Free space on /tmp at the end= 70436 MiB
End job on node101 at 2011-06-03 00:28:56