Trace number 3385700

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (TO) 2378.97 5100.02

General information on the benchmark

Namerandom/medium/
unif-k5-r21.3-v125-c2662-S1300400916-048.SATISFIABLE.cnf
MD5SUM2d20051b8a7a248788d1c2610fe5ea54
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.023995
Satisfiable
(Un)Satisfiability was proved
Number of variables125
Number of clauses2662
Sum of the clauses size13310
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 52662
Number of clauses of size over 50

Solver Data 

0.37/2.77	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.37/2.77	c INFO: solving HOME/instance-3385700-1308048004.cnf
0.37/2.77	c INFO: computing task features
0.37/2.79	c INFO: computed variable-clause graph statistics
0.37/2.79	c INFO: computed clause balance statistics
0.43/2.85	c INFO: computed variable balance statistics
0.43/2.89	c INFO: computed small-clause counts
0.43/2.89	c INFO: computed Horn-clause counts
0.43/2.89	c INFO: computed variable graph statistics
0.43/2.89	c INFO: collected features for HOME/instance-3385700-1308048004.cnf in 0.12 s
0.53/2.93	c INFO: running gnovelty+2@0 for 97 with 4999 remaining (b = 0.63)
0.53/2.96	c INFO: running gnovelty+2@0 for 97 with 4999 remaining (b = 0.07)
0.53/2.99	c INFO: running march_hi@0 for 388 with 4999 remaining (b = 0.30)
0.68/3.02	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.31)
0.68/3.05	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.28)
0.68/3.08	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.25)
0.78/3.11	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.22)
0.78/3.14	c INFO: running march_hi@0 for 388 with 4999 remaining (b = 0.16)
0.78/3.14	Traceback (most recent call last):
0.78/3.14	  File "HOME/solve", line 33, in <module>
0.78/3.14	    plac.call(main)
0.78/3.14	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
0.78/3.14	    cmd, result = parser_from(obj).consume(arglist)
0.78/3.14	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
0.78/3.14	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
0.78/3.14	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
0.78/3.14	    (_, answer, _) = solver(input_path, budget / cores, cores)
0.78/3.14	  File "HOME/borg/build/lib/borg/bilevel.py", line 226, in __call__
0.78/3.14	    process.die()
0.78/3.14	  File "HOME/borg/build/lib/borg/solvers.py", line 185, in die
0.78/3.14	    os.kill(self._process.pid, signal.SIGUSR1)
0.78/3.14	OSError: [Errno 3] No such process

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3385700-1308048004/watcher-3385700-1308048004 -o /tmp/evaluation-result-3385700-1308048004/solver-3385700-1308048004 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385700-1308048004.cnf -seed 777552576 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.00 0.43 1.21 1/170 24383
/proc/meminfo: memFree=31977704/32950928 swapFree=67054844/67111528
[pid=24383] ppid=24381 vsize=11804 CPUtime=0 cores=0-7
/proc/24383/stat : 24383 (runsolver) S 24381 24383 24352 0 -1 4202560 47 0 0 0 0 0 0 0 22 0 1 0 995031394 12087296 55 33554432000 4194304 5436572 140736684804992 18446744073709551615 5057063 0 2147483391 4096 24578 18446744071699215550 0 0 17 4 0 0 0
/proc/24383/statm: 2951 55 36 304 0 2643 0

[startup+0.074275 s]
/proc/loadavg: 0.00 0.43 1.21 1/170 24383
/proc/meminfo: memFree=31977704/32950928 swapFree=67054844/67111528
[pid=24383] ppid=24381 vsize=15184 CPUtime=0 cores=0-7
/proc/24383/stat : 24383 (python) S 24381 24383 24352 0 -1 4202496 729 0 0 0 0 0 0 0 22 0 1 0 995031394 15548416 617 33554432000 4194304 5683644 140734342487680 18446744073709551615 231895488512 0 2147483391 16781312 2 18446744071699215550 0 0 17 4 0 0 0
/proc/24383/statm: 3796 617 322 364 0 283 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15184

[startup+0.100261 s]
/proc/loadavg: 0.00 0.43 1.21 1/170 24383
/proc/meminfo: memFree=31977704/32950928 swapFree=67054844/67111528
[pid=24383] ppid=24381 vsize=15184 CPUtime=0 cores=0-7
/proc/24383/stat : 24383 (python) S 24381 24383 24352 0 -1 4202496 735 0 0 0 0 0 0 0 22 0 1 0 995031394 15548416 622 33554432000 4194304 5683644 140734342487680 18446744073709551615 231895488512 0 2147483391 16781312 2 18446744071699215550 0 0 17 4 0 0 0
/proc/24383/statm: 3796 622 322 364 0 283 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15184

[startup+0.300232 s]
/proc/loadavg: 0.00 0.43 1.21 1/170 24383
/proc/meminfo: memFree=31977704/32950928 swapFree=67054844/67111528
[pid=24383] ppid=24381 vsize=16240 CPUtime=0 cores=0-7
/proc/24383/stat : 24383 (python) S 24381 24383 24352 0 -1 4202496 1052 0 0 0 0 0 0 0 20 0 1 0 995031394 16629760 922 33554432000 4194304 5683644 140734342487680 18446744073709551615 231895488512 0 2147483391 16781312 2 18446744071699215550 0 0 17 4 0 0 0
/proc/24383/statm: 4060 922 381 364 0 547 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 16240

[startup+0.701146 s]
/proc/loadavg: 0.00 0.43 1.21 1/170 24383
/proc/meminfo: memFree=31977704/32950928 swapFree=67054844/67111528
[pid=24383] ppid=24381 vsize=18884 CPUtime=0.01 cores=0-7
/proc/24383/stat : 24383 (python) S 24381 24383 24352 0 -1 4202496 1240 0 0 0 1 0 0 0 16 0 1 0 995031394 19337216 1094 33554432000 4194304 5683644 140734342487680 18446744073709551615 231895488512 0 2147483391 16781312 2 18446744071699215550 0 0 17 4 0 0 0
/proc/24383/statm: 4721 1094 421 364 0 690 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 18884

[startup+1.50102 s]
/proc/loadavg: 0.00 0.43 1.21 1/171 24384
/proc/meminfo: memFree=31974488/32950928 swapFree=67054844/67111528
[pid=24383] ppid=24381 vsize=94172 CPUtime=0.06 cores=0-7
/proc/24383/stat : 24383 (python) S 24381 24383 24352 0 -1 4202496 2766 0 0 0 5 1 0 0 15 0 1 0 995031394 96432128 2474 33554432000 4194304 5683644 140734342487680 18446744073709551615 231895488512 0 2147483391 16781312 2 18446744071699215550 0 0 17 4 0 0 0
/proc/24383/statm: 23543 2474 764 364 0 1675 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 94172

[startup+3.10068 s]
/proc/loadavg: 0.00 0.43 1.21 1/171 24388
/proc/meminfo: memFree=31963328/32950928 swapFree=67054844/67111528
[pid=24383] ppid=24381 vsize=406148 CPUtime=0.78 cores=0-7
/proc/24383/stat : 24383 (python) R 24381 24383 24352 0 -1 4202496 14044 3929 1 0 61 12 4 1 18 0 7 0 995031394 415895552 7486 33554432000 4194304 5683644 140734342487680 18446744073709551615 47051957298609 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/24383/statm: 101537 7486 1884 364 0 37440 0
Current children cumulated CPU time (s) 0.78
Current children cumulated vsize (KiB) 406148
heavy processes:

[startup+6.30107 s]
/proc/loadavg: 0.48 0.52 1.23 7/183 24414
/proc/meminfo: memFree=31924548/32950928 swapFree=67054844/67111528
[pid=24383] ppid=24381 vsize=375440 CPUtime=0.83 cores=0-7
/proc/24383/stat : 24383 (python) S 24381 24383 24352 0 -1 4202496 15420 3929 1 0 66 12 4 1 20 0 1 0 995031394 384450560 7394 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907320639 0 0 16781312 2 18446744071562232103 0 0 17 0 0 0 0
/proc/24383/statm: 93860 7394 1891 364 0 29757 0
[pid=24397] ppid=24383 vsize=365136 CPUtime=0 cores=0-7
/proc/24397/stat : 24397 (python) S 24383 24383 24352 0 -1 4202560 712 0 0 0 0 0 0 0 15 0 1 0 995031693 373899264 5854 33554432000 4194304 5683644 140734342487680 18446744073709551615 231895510598 0 0 16781312 514 5633919575251714308 0 0 17 1 0 0 0
/proc/24397/statm: 91284 5854 348 364 0 27196 0
[pid=24399] ppid=24397 vsize=2812 CPUtime=3.3 cores=0-7
/proc/24399/stat : 24399 (march_hi) R 24397 24399 24399 0 -1 4202496 742 0 0 0 330 0 0 0 25 0 1 0 995031693 2879488 489 33554432000 4194304 4955518 140734218170400 18446744073709551615 4208667 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/24399/statm: 703 489 106 186 0 516 0
[pid=24400] ppid=24383 vsize=375392 CPUtime=0 cores=0-7
/proc/24400/stat : 24400 (python) S 24383 24383 24352 0 -1 4202560 731 0 0 0 0 0 0 0 15 0 1 0 995031696 384401408 5817 33554432000 4194304 5683644 140734342487680 18446744073709551615 231895510598 0 0 16781312 514 3371859119056060676 0 0 17 4 0 0 0
/proc/24400/statm: 93848 5817 348 364 0 29757 0
[pid=24402] ppid=24400 vsize=2812 CPUtime=3.27 cores=0-7
/proc/24402/stat : 24402 (march_hi) R 24400 24402 24402 0 -1 4202496 744 0 0 0 327 0 0 0 25 0 1 0 995031696 2879488 488 33554432000 4194304 4955518 140736978671072 18446744073709551615 4237810 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/24402/statm: 703 488 106 186 0 516 0
[pid=24403] ppid=24383 vsize=385648 CPUtime=0 cores=0-7
/proc/24403/stat : 24403 (python) S 24383 24383 24352 0 -1 4202560 730 0 0 0 0 0 0 0 15 0 1 0 995031699 394903552 5839 33554432000 4194304 5683644 140734342487680 18446744073709551615 231895510598 0 0 16781312 514 3806095843283403012 0 0 17 1 0 0 0
/proc/24403/statm: 96412 5839 348 364 0 32318 0
[pid=24405] ppid=24403 vsize=2812 CPUtime=3.24 cores=0-7
/proc/24405/stat : 24405 (march_hi) R 24403 24405 24405 0 -1 4202496 745 0 0 0 324 0 0 0 25 0 1 0 995031699 2879488 485 33554432000 4194304 4955518 140736367291392 18446744073709551615 4208710 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/24405/statm: 703 485 106 186 0 516 0
[pid=24406] ppid=24383 vsize=395904 CPUtime=0 cores=0-7
/proc/24406/stat : 24406 (python) S 24383 24383 24352 0 -1 4202560 741 0 0 0 0 0 0 0 15 0 1 0 995031702 405405696 5826 33554432000 4194304 5683644 140734342487680 18446744073709551615 231895510598 0 0 16781312 514 5634016332274958596 0 0 17 1 0 0 0
/proc/24406/statm: 98976 5826 348 364 0 34879 0
[pid=24408] ppid=24406 vsize=2812 CPUtime=3.21 cores=0-7
/proc/24408/stat : 24408 (march_hi) R 24406 24408 24408 0 -1 4202496 743 0 0 0 321 0 0 0 25 0 1 0 995031702 2879488 484 33554432000 4194304 4955518 140733375130496 18446744073709551615 4208676 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/24408/statm: 703 484 106 186 0 516 0
[pid=24409] ppid=24383 vsize=406160 CPUtime=0 cores=0-7
/proc/24409/stat : 24409 (python) S 24383 24383 24352 0 -1 4202560 749 0 0 0 0 0 0 0 15 0 1 0 995031705 415907840 5950 33554432000 4194304 5683644 140734342487680 18446744073709551615 231895510598 0 0 16781312 514 5627696339438502148 0 0 17 1 0 0 0
/proc/24409/statm: 101540 5950 348 364 0 37440 0
[pid=24411] ppid=24409 vsize=2812 CPUtime=3.17 cores=0-7
/proc/24411/stat : 24411 (march_hi) R 24409 24411 24411 0 -1 4202496 740 0 0 0 317 0 0 0 25 0 1 0 995031705 2879488 481 33554432000 4194304 4955518 140733625414336 18446744073709551615 4237804 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/24411/statm: 703 481 106 186 0 516 0
[pid=24412] ppid=24383 vsize=416416 CPUtime=0 cores=0-7
/proc/24412/stat : 24412 (python) S 24383 24383 24352 0 -1 4202560 785 0 0 0 0 0 0 0 15 0 1 0 995031708 426409984 5873 33554432000 4194304 5683644 140734342487680 18446744073709551615 231895510598 0 0 16781312 514 17305692900931109124 0 0 17 2 0 0 0
/proc/24412/statm: 104104 5873 354 364 0 40001 0
[pid=24414] ppid=24412 vsize=2812 CPUtime=3.15 cores=0-7
/proc/24414/stat : 24414 (march_hi) R 24412 24414 24414 0 -1 4202496 731 0 0 0 315 0 0 0 25 0 1 0 995031709 2879488 463 33554432000 4194304 4955518 140735418533456 18446744073709551615 4237704 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/24414/statm: 703 463 106 186 0 516 0
Current children cumulated CPU time (s) 20.17
Current children cumulated vsize (KiB) 2736968

[startup+12.7008 s]
/proc/loadavg: 0.92 0.62 1.26 7/183 24414
/proc/meminfo: memFree=31916752/32950928 swapFree=67054844/67111528
[pid=24383] ppid=24381 vsize=375440 CPUtime=0.83 cores=0-7
/proc/24383/stat : 24383 (python) S 24381 24383 24352 0 -1 4202496 15420 3929 1 0 66 12 4 1 20 0 1 0 995031394 384450560 7394 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907320639 0 0 16781312 2 18446744071562232103 0 0 17 0 0 0 0
/proc/24383/statm: 93860 7394 1891 364 0 29757 0
[pid=24397] ppid=24383 vsize=365136 CPUtime=0.01 cores=0-7
/proc/24397/stat : 24397 (python) R 24383 24383 24352 0 -1 4202560 787 0 0 0 0 1 0 0 15 0 1 0 995031693 373899264 5877 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907317195 0 0 16781312 514 0 0 0 17 1 0 0 0
/proc/24397/statm: 91284 5877 371 364 0 27196 0
[pid=24399] ppid=24397 vsize=4088 CPUtime=9.7 cores=0-7
/proc/24399/stat : 24399 (march_hi) R 24397 24399 24399 0 -1 4202496 1081 0 0 0 970 0 0 0 25 0 1 0 995031693 4186112 828 33554432000 4194304 4955518 140734218170400 18446744073709551615 4421280 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/24399/statm: 1022 828 106 186 0 835 0
[pid=24400] ppid=24383 vsize=375392 CPUtime=0 cores=0-7
/proc/24400/stat : 24400 (python) S 24383 24383 24352 0 -1 4202560 808 0 0 0 0 0 0 0 15 0 1 0 995031696 384401408 5840 33554432000 4194304 5683644 140734342487680 18446744073709551615 231895510598 0 0 16781312 514 3371859119056060676 0 0 17 4 0 0 0
/proc/24400/statm: 93848 5840 371 364 0 29757 0
[pid=24402] ppid=24400 vsize=4088 CPUtime=9.66 cores=0-7
/proc/24402/stat : 24402 (march_hi) R 24400 24402 24402 0 -1 4202496 1083 0 0 0 966 0 0 0 25 0 1 0 995031696 4186112 827 33554432000 4194304 4955518 140736978671072 18446744073709551615 4237818 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/24402/statm: 1022 827 106 186 0 835 0
[pid=24403] ppid=24383 vsize=385648 CPUtime=0.01 cores=0-7
/proc/24403/stat : 24403 (python) S 24383 24383 24352 0 -1 4202560 805 0 0 0 0 1 0 0 15 0 1 0 995031699 394903552 5862 33554432000 4194304 5683644 140734342487680 18446744073709551615 231895510598 0 0 16781312 514 3806095843283403012 0 0 17 1 0 0 0
/proc/24403/statm: 96412 5862 371 364 0 32318 0
[pid=24405] ppid=24403 vsize=4088 CPUtime=9.64 cores=0-7
/proc/24405/stat : 24405 (march_hi) R 24403 24405 24405 0 -1 4202496 1086 0 0 0 964 0 0 0 25 0 1 0 995031699 4186112 826 33554432000 4194304 4955518 140736367291392 18446744073709551615 4224512 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/24405/statm: 1022 826 106 186 0 835 0
[pid=24406] ppid=24383 vsize=395904 CPUtime=0.01 cores=0-7
/proc/24406/stat : 24406 (python) S 24383 24383 24352 0 -1 4202560 816 0 0 0 1 0 0 0 15 0 1 0 995031702 405405696 5849 33554432000 4194304 5683644 140734342487680 18446744073709551615 231895510598 0 0 16781312 514 5634016332274958596 0 0 17 1 0 0 0
/proc/24406/statm: 98976 5849 371 364 0 34879 0
[pid=24408] ppid=24406 vsize=4088 CPUtime=9.61 cores=0-7
/proc/24408/stat : 24408 (march_hi) R 24406 24408 24408 0 -1 4202496 1084 0 0 0 961 0 0 0 25 0 1 0 995031702 4186112 825 33554432000 4194304 4955518 140733375130496 18446744073709551615 4206496 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/24408/statm: 1022 825 106 186 0 835 0
[pid=24409] ppid=24383 vsize=406160 CPUtime=0.02 cores=0-7
/proc/24409/stat : 24409 (python) S 24383 24383 24352 0 -1 4202560 830 0 0 0 1 1 0 0 15 0 1 0 995031705 415907840 5973 33554432000 4194304 5683644 140734342487680 18446744073709551615 231895510598 0 0 16781312 514 5627696339438502148 0 0 17 1 0 0 0
/proc/24409/statm: 101540 5973 371 364 0 37440 0
[pid=24411] ppid=24409 vsize=4088 CPUtime=9.55 cores=0-7
/proc/24411/stat : 24411 (march_hi) R 24409 24411 24411 0 -1 4202496 1080 0 0 0 955 0 0 0 25 0 1 0 995031705 4186112 821 33554432000 4194304 4955518 140733625414336 18446744073709551615 4208624 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/24411/statm: 1022 821 106 186 0 835 0
[pid=24412] ppid=24383 vsize=416416 CPUtime=0 cores=0-7
/proc/24412/stat : 24412 (python) S 24383 24383 24352 0 -1 4202560 866 0 0 0 0 0 0 0 15 0 1 0 995031708 426409984 5896 33554432000 4194304 5683644 140734342487680 18446744073709551615 231895510598 0 0 16781312 514 17305692900931109124 0 0 17 2 0 0 0
/proc/24412/statm: 104104 5896 377 364 0 40001 0
[pid=24414] ppid=24412 vsize=4088 CPUtime=9.55 cores=0-7
/proc/24414/stat : 24414 (march_hi) R 24412 24414 24414 0 -1 4202496 1043 0 0 0 955 0 0 0 25 0 1 0 995031709 4186112 775 33554432000 4194304 4955518 140735418533456 18446744073709551615 4208656 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/24414/statm: 1022 775 106 186 0 835 0
Current children cumulated CPU time (s) 58.59
Current children cumulated vsize (KiB) 2744624

[startup+25.5014 s]
/proc/loadavg: 2.05 0.88 1.33 7/183 24414
/proc/meminfo: memFree=31901900/32950928 swapFree=67054844/67111528
[pid=24383] ppid=24381 vsize=375440 CPUtime=0.83 cores=0-7
/proc/24383/stat : 24383 (python) S 24381 24383 24352 0 -1 4202496 15420 3929 1 0 66 12 4 1 20 0 1 0 995031394 384450560 7394 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907320639 0 0 16781312 2 18446744071562232103 0 0 17 0 0 0 0
/proc/24383/statm: 93860 7394 1891 364 0 29757 0
[pid=24397] ppid=24383 vsize=365136 CPUtime=0.02 cores=0-7
/proc/24397/stat : 24397 (python) S 24383 24383 24352 0 -1 4202560 788 0 0 0 1 1 0 0 15 0 1 0 995031693 373899264 5877 33554432000 4194304 5683644 140734342487680 18446744073709551615 231895510598 0 0 16781312 514 5633919575251714308 0 0 17 1 0 0 0
/proc/24397/statm: 91284 5877 371 364 0 27196 0
[pid=24399] ppid=24397 vsize=7172 CPUtime=22.5 cores=0-7
/proc/24399/stat : 24399 (march_hi) R 24397 24399 24399 0 -1 4202496 1785 0 0 0 2250 0 0 0 25 0 1 0 995031693 7344128 1532 33554432000 4194304 4955518 140734218170400 18446744073709551615 4237810 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/24399/statm: 1793 1532 106 186 0 1606 0
[pid=24400] ppid=24383 vsize=375392 CPUtime=0.02 cores=0-7
/proc/24400/stat : 24400 (python) S 24383 24383 24352 0 -1 4202560 808 0 0 0 1 1 0 0 15 0 1 0 995031696 384401408 5840 33554432000 4194304 5683644 140734342487680 18446744073709551615 231895510598 0 0 16781312 514 3371859119056060676 0 0 17 4 0 0 0
/proc/24400/statm: 93848 5840 371 364 0 29757 0
[pid=24402] ppid=24400 vsize=7172 CPUtime=22.45 cores=0-7
/proc/24402/stat : 24402 (march_hi) R 24400 24402 24402 0 -1 4202496 1787 0 0 0 2245 0 0 0 25 0 1 0 995031696 7344128 1531 33554432000 4194304 4955518 140736978671072 18446744073709551615 4208753 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/24402/statm: 1793 1531 106 186 0 1606 0
[pid=24403] ppid=24383 vsize=385648 CPUtime=0.03 cores=0-7
/proc/24403/stat : 24403 (python) S 24383 24383 24352 0 -1 4202560 805 0 0 0 1 2 0 0 15 0 1 0 995031699 394903552 5862 33554432000 4194304 5683644 140734342487680 18446744073709551615 231895510598 0 0 16781312 514 3806095843283403012 0 0 17 1 0 0 0
/proc/24403/statm: 96412 5862 371 364 0 32318 0
[pid=24405] ppid=24403 vsize=7172 CPUtime=22.44 cores=0-7
/proc/24405/stat : 24405 (march_hi) R 24403 24405 24405 0 -1 4202496 1792 0 0 0 2244 0 0 0 25 0 1 0 995031699 7344128 1532 33554432000 4194304 4955518 140736367291392 18446744073709551615 4235601 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/24405/statm: 1793 1532 106 186 0 1606 0
[pid=24406] ppid=24383 vsize=395904 CPUtime=0.03 cores=0-7
/proc/24406/stat : 24406 (python) S 24383 24383 24352 0 -1 4202560 817 0 0 0 2 1 0 0 15 0 1 0 995031702 405405696 5849 33554432000 4194304 5683644 140734342487680 18446744073709551615 231895510598 0 0 16781312 514 5634016332274958596 0 0 17 1 0 0 0
/proc/24406/statm: 98976 5849 371 364 0 34879 0

################
# More data... #
################

[pid=24397/tid=24425] ppid=24383 vsize=365136 CPUtime=0 cores=0-7
/proc/24397/task/24425/stat : 24425 (python) S 24383 24383 24352 0 -1 4202560 109 0 0 0 0 0 0 0 16 0 2 0 995070929 373899264 5907 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907314097 0 0 16781312 514 18446604453840867392 0 0 -1 7 0 0 0
[pid=24400] ppid=24383 vsize=0 CPUtime=398.35 cores=0-7
/proc/24400/stat : 24400 (python) Z 24383 24383 24352 0 -1 4202588 1082 18518 0 0 29 20 39783 3 15 0 1 0 995031696 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 4 0 0 0
/proc/24400/statm: 0 0 0 0 0 0 0
[pid=24409] ppid=24383 vsize=0 CPUtime=398.8 cores=0-7
/proc/24409/stat : 24409 (python) Z 24383 24383 24352 0 -1 4202588 1140 18521 0 0 27 23 39827 3 16 0 1 0 995031705 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 1 0 0 0
/proc/24409/statm: 0 0 0 0 0 0 0
[pid=24412] ppid=24383 vsize=416416 CPUtime=0.48 cores=0-7
/proc/24412/stat : 24412 (python) S 24383 24383 24352 0 -1 4202560 1054 0 0 0 25 23 0 0 16 0 2 0 995031708 426409984 5926 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907317195 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/24412/statm: 104104 5926 406 364 0 40001 0
[pid=24414] ppid=24412 vsize=93528 CPUtime=392.62 cores=0-7
/proc/24414/stat : 24414 (march_hi) T 24412 24414 24414 0 -1 4202496 16727 0 0 0 39258 4 0 0 25 0 1 0 995031709 95772672 16459 33554432000 4194304 4955518 140735418533456 18446744073709551615 4208714 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/24414/statm: 23382 16459 106 186 0 23195 0
[pid=24412/tid=24426] ppid=24383 vsize=416416 CPUtime=0 cores=0-7
/proc/24412/task/24426/stat : 24426 (python) S 24383 24383 24352 0 -1 4202560 78 0 0 0 0 0 0 0 16 0 2 0 995070972 426409984 5926 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907314097 0 0 16781312 514 18446604453890003264 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2378.97
Current children cumulated vsize (KiB) 1344048
heavy processes:

[startup+4962.3 s]
/proc/loadavg: 0.00 0.00 0.00 1/179 24580
/proc/meminfo: memFree=31805308/32950928 swapFree=67054844/67111528
[pid=24383] ppid=24381 vsize=375440 CPUtime=795.88 cores=0-7
/proc/24383/stat : 24383 (python) S 24381 24383 24352 0 -1 4202496 15423 43134 1 0 66 12 79459 51 15 0 1 0 995031394 384450560 7394 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907320639 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/24383/statm: 93860 7394 1891 364 0 29757 0
[pid=24397] ppid=24383 vsize=365136 CPUtime=0.48 cores=0-7
/proc/24397/stat : 24397 (python) S 24383 24383 24352 0 -1 4202560 1008 0 0 0 26 22 0 0 16 0 2 0 995031693 373899264 5907 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907317195 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/24397/statm: 91284 5907 400 364 0 27196 0
[pid=24399] ppid=24397 vsize=93528 CPUtime=392.36 cores=0-7
/proc/24399/stat : 24399 (march_hi) T 24397 24399 24399 0 -1 4202496 18300 0 0 0 39232 4 0 0 25 0 1 0 995031693 95772672 18047 33554432000 4194304 4955518 140734218170400 18446744073709551615 4237810 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/24399/statm: 23382 18047 106 186 0 23195 0
[pid=24397/tid=24425] ppid=24383 vsize=365136 CPUtime=0 cores=0-7
/proc/24397/task/24425/stat : 24425 (python) S 24383 24383 24352 0 -1 4202560 109 0 0 0 0 0 0 0 16 0 2 0 995070929 373899264 5907 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907314097 0 0 16781312 514 18446604453840867392 0 0 -1 7 0 0 0
[pid=24400] ppid=24383 vsize=0 CPUtime=398.35 cores=0-7
/proc/24400/stat : 24400 (python) Z 24383 24383 24352 0 -1 4202588 1082 18518 0 0 29 20 39783 3 15 0 1 0 995031696 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 4 0 0 0
/proc/24400/statm: 0 0 0 0 0 0 0
[pid=24409] ppid=24383 vsize=0 CPUtime=398.8 cores=0-7
/proc/24409/stat : 24409 (python) Z 24383 24383 24352 0 -1 4202588 1140 18521 0 0 27 23 39827 3 16 0 1 0 995031705 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 1 0 0 0
/proc/24409/statm: 0 0 0 0 0 0 0
[pid=24412] ppid=24383 vsize=416416 CPUtime=0.48 cores=0-7
/proc/24412/stat : 24412 (python) S 24383 24383 24352 0 -1 4202560 1054 0 0 0 25 23 0 0 16 0 2 0 995031708 426409984 5926 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907317195 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/24412/statm: 104104 5926 406 364 0 40001 0
[pid=24414] ppid=24412 vsize=93528 CPUtime=392.62 cores=0-7
/proc/24414/stat : 24414 (march_hi) T 24412 24414 24414 0 -1 4202496 16727 0 0 0 39258 4 0 0 25 0 1 0 995031709 95772672 16459 33554432000 4194304 4955518 140735418533456 18446744073709551615 4208714 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/24414/statm: 23382 16459 106 186 0 23195 0
[pid=24412/tid=24426] ppid=24383 vsize=416416 CPUtime=0 cores=0-7
/proc/24412/task/24426/stat : 24426 (python) S 24383 24383 24352 0 -1 4202560 78 0 0 0 0 0 0 0 16 0 2 0 995070972 426409984 5926 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907314097 0 0 16781312 514 18446604453890003264 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2378.97
Current children cumulated vsize (KiB) 1344048
heavy processes:

[startup+5022.3 s]
/proc/loadavg: 0.00 0.00 0.00 1/179 24581
/proc/meminfo: memFree=31805404/32950928 swapFree=67054844/67111528
[pid=24383] ppid=24381 vsize=375440 CPUtime=795.88 cores=0-7
/proc/24383/stat : 24383 (python) S 24381 24383 24352 0 -1 4202496 15423 43134 1 0 66 12 79459 51 15 0 1 0 995031394 384450560 7394 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907320639 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/24383/statm: 93860 7394 1891 364 0 29757 0
[pid=24397] ppid=24383 vsize=365136 CPUtime=0.48 cores=0-7
/proc/24397/stat : 24397 (python) S 24383 24383 24352 0 -1 4202560 1008 0 0 0 26 22 0 0 16 0 2 0 995031693 373899264 5907 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907317195 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/24397/statm: 91284 5907 400 364 0 27196 0
[pid=24399] ppid=24397 vsize=93528 CPUtime=392.36 cores=0-7
/proc/24399/stat : 24399 (march_hi) T 24397 24399 24399 0 -1 4202496 18300 0 0 0 39232 4 0 0 25 0 1 0 995031693 95772672 18047 33554432000 4194304 4955518 140734218170400 18446744073709551615 4237810 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/24399/statm: 23382 18047 106 186 0 23195 0
[pid=24397/tid=24425] ppid=24383 vsize=365136 CPUtime=0 cores=0-7
/proc/24397/task/24425/stat : 24425 (python) S 24383 24383 24352 0 -1 4202560 109 0 0 0 0 0 0 0 16 0 2 0 995070929 373899264 5907 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907314097 0 0 16781312 514 18446604453840867392 0 0 -1 7 0 0 0
[pid=24400] ppid=24383 vsize=0 CPUtime=398.35 cores=0-7
/proc/24400/stat : 24400 (python) Z 24383 24383 24352 0 -1 4202588 1082 18518 0 0 29 20 39783 3 15 0 1 0 995031696 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 4 0 0 0
/proc/24400/statm: 0 0 0 0 0 0 0
[pid=24409] ppid=24383 vsize=0 CPUtime=398.8 cores=0-7
/proc/24409/stat : 24409 (python) Z 24383 24383 24352 0 -1 4202588 1140 18521 0 0 27 23 39827 3 16 0 1 0 995031705 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 1 0 0 0
/proc/24409/statm: 0 0 0 0 0 0 0
[pid=24412] ppid=24383 vsize=416416 CPUtime=0.48 cores=0-7
/proc/24412/stat : 24412 (python) S 24383 24383 24352 0 -1 4202560 1054 0 0 0 25 23 0 0 16 0 2 0 995031708 426409984 5926 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907317195 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/24412/statm: 104104 5926 406 364 0 40001 0
[pid=24414] ppid=24412 vsize=93528 CPUtime=392.62 cores=0-7
/proc/24414/stat : 24414 (march_hi) T 24412 24414 24414 0 -1 4202496 16727 0 0 0 39258 4 0 0 25 0 1 0 995031709 95772672 16459 33554432000 4194304 4955518 140735418533456 18446744073709551615 4208714 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/24414/statm: 23382 16459 106 186 0 23195 0
[pid=24412/tid=24426] ppid=24383 vsize=416416 CPUtime=0 cores=0-7
/proc/24412/task/24426/stat : 24426 (python) S 24383 24383 24352 0 -1 4202560 78 0 0 0 0 0 0 0 16 0 2 0 995070972 426409984 5926 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907314097 0 0 16781312 514 18446604453890003264 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2378.97
Current children cumulated vsize (KiB) 1344048
heavy processes:

[startup+5082.3 s]
/proc/loadavg: 0.00 0.00 0.00 1/179 24583
/proc/meminfo: memFree=31805328/32950928 swapFree=67054844/67111528
[pid=24383] ppid=24381 vsize=375440 CPUtime=795.88 cores=0-7
/proc/24383/stat : 24383 (python) S 24381 24383 24352 0 -1 4202496 15423 43134 1 0 66 12 79459 51 15 0 1 0 995031394 384450560 7394 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907320639 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/24383/statm: 93860 7394 1891 364 0 29757 0
[pid=24397] ppid=24383 vsize=365136 CPUtime=0.48 cores=0-7
/proc/24397/stat : 24397 (python) S 24383 24383 24352 0 -1 4202560 1008 0 0 0 26 22 0 0 16 0 2 0 995031693 373899264 5907 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907317195 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/24397/statm: 91284 5907 400 364 0 27196 0
[pid=24399] ppid=24397 vsize=93528 CPUtime=392.36 cores=0-7
/proc/24399/stat : 24399 (march_hi) T 24397 24399 24399 0 -1 4202496 18300 0 0 0 39232 4 0 0 25 0 1 0 995031693 95772672 18047 33554432000 4194304 4955518 140734218170400 18446744073709551615 4237810 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/24399/statm: 23382 18047 106 186 0 23195 0
[pid=24397/tid=24425] ppid=24383 vsize=365136 CPUtime=0 cores=0-7
/proc/24397/task/24425/stat : 24425 (python) S 24383 24383 24352 0 -1 4202560 109 0 0 0 0 0 0 0 16 0 2 0 995070929 373899264 5907 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907314097 0 0 16781312 514 18446604453840867392 0 0 -1 7 0 0 0
[pid=24400] ppid=24383 vsize=0 CPUtime=398.35 cores=0-7
/proc/24400/stat : 24400 (python) Z 24383 24383 24352 0 -1 4202588 1082 18518 0 0 29 20 39783 3 15 0 1 0 995031696 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 4 0 0 0
/proc/24400/statm: 0 0 0 0 0 0 0
[pid=24409] ppid=24383 vsize=0 CPUtime=398.8 cores=0-7
/proc/24409/stat : 24409 (python) Z 24383 24383 24352 0 -1 4202588 1140 18521 0 0 27 23 39827 3 16 0 1 0 995031705 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 1 0 0 0
/proc/24409/statm: 0 0 0 0 0 0 0
[pid=24412] ppid=24383 vsize=416416 CPUtime=0.48 cores=0-7
/proc/24412/stat : 24412 (python) S 24383 24383 24352 0 -1 4202560 1054 0 0 0 25 23 0 0 16 0 2 0 995031708 426409984 5926 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907317195 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/24412/statm: 104104 5926 406 364 0 40001 0
[pid=24414] ppid=24412 vsize=93528 CPUtime=392.62 cores=0-7
/proc/24414/stat : 24414 (march_hi) T 24412 24414 24414 0 -1 4202496 16727 0 0 0 39258 4 0 0 25 0 1 0 995031709 95772672 16459 33554432000 4194304 4955518 140735418533456 18446744073709551615 4208714 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/24414/statm: 23382 16459 106 186 0 23195 0
[pid=24412/tid=24426] ppid=24383 vsize=416416 CPUtime=0 cores=0-7
/proc/24412/task/24426/stat : 24426 (python) S 24383 24383 24352 0 -1 4202560 78 0 0 0 0 0 0 0 16 0 2 0 995070972 426409984 5926 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907314097 0 0 16781312 514 18446604453890003264 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2378.97
Current children cumulated vsize (KiB) 1344048
heavy processes:



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 0.00 0.00 0.00 1/179 24584
/proc/meminfo: memFree=31805188/32950928 swapFree=67054844/67111528
[pid=24383] ppid=24381 vsize=375440 CPUtime=795.88 cores=0-7
/proc/24383/stat : 24383 (python) S 24381 24383 24352 0 -1 4202496 15423 43134 1 0 66 12 79459 51 15 0 1 0 995031394 384450560 7394 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907320639 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/24383/statm: 93860 7394 1891 364 0 29757 0
[pid=24397] ppid=24383 vsize=365136 CPUtime=0.48 cores=0-7
/proc/24397/stat : 24397 (python) S 24383 24383 24352 0 -1 4202560 1008 0 0 0 26 22 0 0 16 0 2 0 995031693 373899264 5907 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907317195 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/24397/statm: 91284 5907 400 364 0 27196 0
[pid=24399] ppid=24397 vsize=93528 CPUtime=392.36 cores=0-7
/proc/24399/stat : 24399 (march_hi) T 24397 24399 24399 0 -1 4202496 18300 0 0 0 39232 4 0 0 25 0 1 0 995031693 95772672 18047 33554432000 4194304 4955518 140734218170400 18446744073709551615 4237810 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/24399/statm: 23382 18047 106 186 0 23195 0
[pid=24397/tid=24425] ppid=24383 vsize=365136 CPUtime=0 cores=0-7
/proc/24397/task/24425/stat : 24425 (python) S 24383 24383 24352 0 -1 4202560 109 0 0 0 0 0 0 0 16 0 2 0 995070929 373899264 5907 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907314097 0 0 16781312 514 18446604453840867392 0 0 -1 7 0 0 0
[pid=24400] ppid=24383 vsize=0 CPUtime=398.35 cores=0-7
/proc/24400/stat : 24400 (python) Z 24383 24383 24352 0 -1 4202588 1082 18518 0 0 29 20 39783 3 15 0 1 0 995031696 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 4 0 0 0
/proc/24400/statm: 0 0 0 0 0 0 0
[pid=24409] ppid=24383 vsize=0 CPUtime=398.8 cores=0-7
/proc/24409/stat : 24409 (python) Z 24383 24383 24352 0 -1 4202588 1140 18521 0 0 27 23 39827 3 16 0 1 0 995031705 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 1 0 0 0
/proc/24409/statm: 0 0 0 0 0 0 0
[pid=24412] ppid=24383 vsize=416416 CPUtime=0.48 cores=0-7
/proc/24412/stat : 24412 (python) S 24383 24383 24352 0 -1 4202560 1054 0 0 0 25 23 0 0 16 0 2 0 995031708 426409984 5926 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907317195 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/24412/statm: 104104 5926 406 364 0 40001 0
[pid=24414] ppid=24412 vsize=93528 CPUtime=392.62 cores=0-7
/proc/24414/stat : 24414 (march_hi) T 24412 24414 24414 0 -1 4202496 16727 0 0 0 39258 4 0 0 25 0 1 0 995031709 95772672 16459 33554432000 4194304 4955518 140735418533456 18446744073709551615 4208714 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/24414/statm: 23382 16459 106 186 0 23195 0
[pid=24412/tid=24426] ppid=24383 vsize=416416 CPUtime=0 cores=0-7
/proc/24412/task/24426/stat : 24426 (python) S 24383 24383 24352 0 -1 4202560 78 0 0 0 0 0 0 0 16 0 2 0 995070972 426409984 5926 33554432000 4194304 5683644 140734342487680 18446744073709551615 231907314097 0 0 16781312 514 18446604453890003264 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2378.97
Current children cumulated vsize (KiB) 1344048
heavy processes:

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 24383 and gives
#  childrusage.ru_utime.tv_sec=1592
#  childrusage.ru_utime.tv_usec=465908
#  childrusage.ru_stime.tv_sec=1
#  childrusage.ru_stime.tv_usec=630752
# CPU time returned by wait4() is 1594.1
# while last known CPU time is 2378.97
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5100.02
CPU time (s): 2378.97
CPU user time (s): 2377.32
CPU system time (s): 1.65
CPU usage (%): 46.6462
Max. virtual memory (cumulated for all children) (KiB): 3281268

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1592.47
system time used= 1.63075
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 101834
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 6919
involuntary context switches= 4754

runsolver used 6.69598 second user time and 16.6815 second system time

The end

Launcher Data 

Begin job on node114 at 2011-06-14 12:40:04
IDJOB=3385700
IDBENCH=82965
IDSOLVER=1870
FILE ID=node114/3385700-1308048004
RUNJOBID= node114-1308048004-24366
PBS_JOBID= 13566212
Free space on /tmp= 74108 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/medium/unif-k5-r21.3-v125-c2662-S1300400916-048.SATISFIABLE.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3385700-1308048004/watcher-3385700-1308048004 -o /tmp/evaluation-result-3385700-1308048004/solver-3385700-1308048004 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385700-1308048004.cnf -seed 777552576 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 2d20051b8a7a248788d1c2610fe5ea54
RANDOM SEED=777552576

node114.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.73
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      31978264 kB
Buffers:        158648 kB
Cached:         616092 kB
SwapCached:      10332 kB
Active:         236716 kB
Inactive:       556684 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      31978264 kB
SwapTotal:    67111528 kB
SwapFree:     67054844 kB
Dirty:           31732 kB
Writeback:           0 kB
AnonPages:       16496 kB
Mapped:          12228 kB
Slab:           113652 kB
PageTables:       4272 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   186044 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 74076 MiB
End job on node114 at 2011-06-14 14:05:09