Trace number 3385679

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (TO) 1861.95 5100.03

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v800-c3408-S1536861174-069.SATISFIABLE.cnf
MD5SUMb9f30e687453978167416852673e8ddb
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.031994
Satisfiable
(Un)Satisfiability was proved
Number of variables800
Number of clauses3408
Sum of the clauses size10224
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 33408
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.39/4.66	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.39/4.66	c INFO: solving HOME/instance-3385679-1308048003.cnf
0.39/4.66	c INFO: computing task features
0.39/4.68	c INFO: computed variable-clause graph statistics
0.39/4.68	c INFO: computed clause balance statistics
0.46/4.73	c INFO: computed variable balance statistics
0.46/4.78	c INFO: computed small-clause counts
0.46/4.78	c INFO: computed Horn-clause counts
0.46/4.79	c INFO: computed variable graph statistics
0.46/4.79	c INFO: collected features for HOME/instance-3385679-1308048003.cnf in 0.12 s
0.55/4.82	c INFO: running gnovelty+2@0 for 97 with 4999 remaining (b = 0.59)
0.55/4.85	c INFO: running gnovelty+2@0 for 97 with 4999 remaining (b = 0.08)
0.55/4.88	c INFO: running march_hi@0 for 388 with 4999 remaining (b = 0.25)
0.67/4.91	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.26)
0.67/4.94	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.23)
0.67/4.97	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.20)
0.83/5.00	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.18)
0.83/5.03	c INFO: running march_hi@0 for 388 with 4999 remaining (b = 0.13)
0.83/5.04	Traceback (most recent call last):
0.83/5.04	  File "HOME/solve", line 33, in <module>
0.83/5.04	    plac.call(main)
0.83/5.04	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
0.83/5.04	    cmd, result = parser_from(obj).consume(arglist)
0.83/5.04	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
0.83/5.04	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
0.83/5.04	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
0.83/5.04	    (_, answer, _) = solver(input_path, budget / cores, cores)
0.83/5.04	  File "HOME/borg/build/lib/borg/bilevel.py", line 226, in __call__
0.83/5.04	    process.die()
0.83/5.04	  File "HOME/borg/build/lib/borg/solvers.py", line 185, in die
0.83/5.04	    os.kill(self._process.pid, signal.SIGUSR1)
0.83/5.04	OSError: [Errno 3] No such process

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3385679-1308048003/watcher-3385679-1308048003 -o /tmp/evaluation-result-3385679-1308048003/solver-3385679-1308048003 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385679-1308048003.cnf -seed 14014297 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.00 0.31 1.02 1/170 3436
/proc/meminfo: memFree=31753468/32951124 swapFree=67054712/67111528
[pid=3436] ppid=3434 vsize=11804 CPUtime=0 cores=0-7
/proc/3436/stat : 3436 (runsolver) S 3434 3436 3404 0 -1 4202560 42 0 0 0 0 0 0 0 22 0 1 0 995031173 12087296 50 33554432000 4194304 5436572 140733772438576 18446744073709551615 4883175 0 2147483391 4096 24578 18446744071699960631 0 0 17 3 0 0 0
/proc/3436/statm: 2951 50 31 304 0 2643 0

[startup+0.0820109 s]
/proc/loadavg: 0.00 0.31 1.02 1/170 3436
/proc/meminfo: memFree=31753468/32951124 swapFree=67054712/67111528
[pid=3436] ppid=3434 vsize=14820 CPUtime=0 cores=0-7
/proc/3436/stat : 3436 (python) S 3434 3436 3404 0 -1 4202496 515 0 0 0 0 0 0 0 24 0 1 0 995031173 15175680 421 33554432000 4194304 5683644 140735081399008 18446744073709551615 216735174501 0 2147483391 4096 0 18446744071699199166 0 0 17 4 0 0 0
/proc/3436/statm: 3705 421 259 364 0 192 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 14820

[startup+0.101007 s]
/proc/loadavg: 0.00 0.31 1.02 1/170 3436
/proc/meminfo: memFree=31753468/32951124 swapFree=67054712/67111528
[pid=3436] ppid=3434 vsize=15080 CPUtime=0 cores=0-7
/proc/3436/stat : 3436 (python) S 3434 3436 3404 0 -1 4202496 618 0 0 0 0 0 0 0 24 0 1 0 995031173 15441920 513 33554432000 4194304 5683644 140735081399008 18446744073709551615 216735176704 0 2147483391 16781312 2 18446744071699199166 0 0 17 4 0 0 0
/proc/3436/statm: 3770 513 300 364 0 257 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15080

[startup+0.300961 s]
/proc/loadavg: 0.00 0.31 1.02 1/170 3436
/proc/meminfo: memFree=31753468/32951124 swapFree=67054712/67111528
[pid=3436] ppid=3434 vsize=15380 CPUtime=0 cores=0-7
/proc/3436/stat : 3436 (python) S 3434 3436 3404 0 -1 4202496 821 0 0 0 0 0 0 0 22 0 1 0 995031173 15749120 704 33554432000 4194304 5683644 140735081399008 18446744073709551615 216735176704 0 2147483391 16781312 2 18446744071699199166 0 0 17 4 0 0 0
/proc/3436/statm: 3845 704 334 364 0 332 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15380

[startup+0.700873 s]
/proc/loadavg: 0.00 0.31 1.02 1/170 3436
/proc/meminfo: memFree=31753468/32951124 swapFree=67054712/67111528
[pid=3436] ppid=3434 vsize=16396 CPUtime=0.01 cores=0-7
/proc/3436/stat : 3436 (python) R 3434 3436 3404 0 -1 4202496 1119 0 0 0 1 0 0 0 18 0 1 0 995031173 16789504 985 33554432000 4194304 5683644 140735081399008 18446744073709551615 216735174501 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/3436/statm: 4099 985 409 364 0 586 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16396

[startup+1.50073 s]
/proc/loadavg: 0.00 0.31 1.02 1/171 3437
/proc/meminfo: memFree=31750732/32951124 swapFree=67054712/67111528
[pid=3436] ppid=3434 vsize=36508 CPUtime=0.04 cores=0-7
/proc/3436/stat : 3436 (python) S 3434 3436 3404 0 -1 4202496 1578 0 0 0 3 1 0 0 15 0 1 0 995031173 37384192 1411 33554432000 4194304 5683644 140735081399008 18446744073709551615 216735176704 0 2147483391 16781312 2 18446744071699199166 0 0 17 4 0 0 0
/proc/3436/statm: 9127 1411 452 364 0 944 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 36508

[startup+3.10046 s]
/proc/loadavg: 0.00 0.31 1.02 1/171 3438
/proc/meminfo: memFree=31746068/32951124 swapFree=67054712/67111528
[pid=3436] ppid=3434 vsize=107428 CPUtime=0.08 cores=0-7
/proc/3436/stat : 3436 (python) S 3434 3436 3404 0 -1 4202496 3405 0 0 0 5 3 0 0 15 0 1 0 995031173 110006272 3058 33554432000 4194304 5683644 140735081399008 18446744073709551615 216735176704 0 2147483391 16781312 2 18446744071699199166 0 0 17 4 0 0 0
/proc/3436/statm: 26857 3058 871 364 0 2088 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 107428
heavy processes:

[startup+6.30088 s]
/proc/loadavg: 0.08 0.33 1.02 5/179 3469
/proc/meminfo: memFree=31707628/32951124 swapFree=67054712/67111528
[pid=3436] ppid=3434 vsize=375168 CPUtime=1.14 cores=0-7
/proc/3436/stat : 3436 (python) S 3434 3436 3404 0 -1 4202496 15644 7106 1 0 66 14 32 2 18 0 1 0 995031173 384172032 7358 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747008831 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/3436/statm: 93792 7358 1891 364 0 29689 0
[pid=3453] ppid=3436 vsize=375120 CPUtime=0 cores=0-7
/proc/3453/stat : 3453 (python) S 3436 3436 3404 0 -1 4202560 718 0 0 0 0 0 0 0 15 0 1 0 995031664 384122880 5822 33554432000 4194304 5683644 140735081399008 18446744073709551615 216735198790 0 0 16781312 514 4493193853620158724 0 0 17 1 0 0 0
/proc/3453/statm: 93780 5822 348 364 0 29689 0
[pid=3455] ppid=3453 vsize=3284 CPUtime=1.38 cores=0-7
/proc/3455/stat : 3455 (march_hi) R 3453 3455 3455 0 -1 4202496 856 0 0 0 138 0 0 0 25 0 1 0 995031664 3362816 605 33554432000 4194304 4955518 140733457976768 18446744073709551615 4295786 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/3455/statm: 821 605 104 186 0 634 0
[pid=3461] ppid=3436 vsize=395632 CPUtime=0 cores=0-7
/proc/3461/stat : 3461 (python) S 3436 3436 3404 0 -1 4202560 738 0 0 0 0 0 0 0 15 0 1 0 995031670 405127168 5791 33554432000 4194304 5683644 140735081399008 18446744073709551615 216735198790 0 0 16781312 514 5629147694787166468 0 0 17 1 0 0 0
/proc/3461/statm: 98908 5791 349 364 0 34811 0
[pid=3463] ppid=3461 vsize=3184 CPUtime=1.32 cores=0-7
/proc/3463/stat : 3463 (march_hi) R 3461 3463 3463 0 -1 4202496 859 0 0 0 132 0 0 0 25 0 1 0 995031670 3260416 604 33554432000 4194304 4955518 140734604731968 18446744073709551615 4227756 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/3463/statm: 796 604 104 186 0 609 0
[pid=3464] ppid=3436 vsize=405888 CPUtime=0 cores=0-7
/proc/3464/stat : 3464 (python) S 3436 3436 3404 0 -1 4202560 743 0 0 0 0 0 0 0 15 0 1 0 995031673 415629312 5812 33554432000 4194304 5683644 140735081399008 18446744073709551615 216735198790 0 0 16781312 514 4512074667292328196 0 0 17 1 0 0 0
/proc/3464/statm: 101472 5812 349 364 0 37372 0
[pid=3466] ppid=3464 vsize=3184 CPUtime=1.28 cores=0-7
/proc/3466/stat : 3466 (march_hi) R 3464 3466 3466 0 -1 4202496 857 0 0 0 128 0 0 0 25 0 1 0 995031674 3260416 601 33554432000 4194304 4955518 140734392216752 18446744073709551615 4238354 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/3466/statm: 796 601 104 186 0 609 0
[pid=3467] ppid=3436 vsize=416144 CPUtime=0 cores=0-7
/proc/3467/stat : 3467 (python) S 3436 3436 3404 0 -1 4202560 784 0 0 0 0 0 0 0 15 0 1 0 995031676 426131456 5837 33554432000 4194304 5683644 140735081399008 18446744073709551615 216735198790 0 0 16781312 514 11254794540256559364 0 0 17 2 0 0 0
/proc/3467/statm: 104036 5837 354 364 0 39933 0
[pid=3469] ppid=3467 vsize=3184 CPUtime=1.25 cores=0-7
/proc/3469/stat : 3469 (march_hi) R 3467 3469 3469 0 -1 4202496 860 0 0 0 125 0 0 0 25 0 1 0 995031677 3260416 597 33554432000 4194304 4955518 140735098272048 18446744073709551615 4227679 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/3469/statm: 796 597 104 186 0 609 0
Current children cumulated CPU time (s) 6.37
Current children cumulated vsize (KiB) 1980788

[startup+12.7008 s]
/proc/loadavg: 0.39 0.39 1.03 5/179 3469
/proc/meminfo: memFree=31703864/32951124 swapFree=67054712/67111528
[pid=3436] ppid=3434 vsize=375168 CPUtime=1.14 cores=0-7
/proc/3436/stat : 3436 (python) S 3434 3436 3404 0 -1 4202496 15644 7106 1 0 66 14 32 2 18 0 1 0 995031173 384172032 7358 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747008831 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/3436/statm: 93792 7358 1891 364 0 29689 0
[pid=3453] ppid=3436 vsize=375120 CPUtime=0 cores=0-7
/proc/3453/stat : 3453 (python) S 3436 3436 3404 0 -1 4202560 790 0 0 0 0 0 0 0 15 0 1 0 995031664 384122880 5844 33554432000 4194304 5683644 140735081399008 18446744073709551615 216735198790 0 0 16781312 514 4493193853620158724 0 0 17 1 0 0 0
/proc/3453/statm: 93780 5844 370 364 0 29689 0
[pid=3455] ppid=3453 vsize=4696 CPUtime=7.78 cores=0-7
/proc/3455/stat : 3455 (march_hi) R 3453 3455 3455 0 -1 4202496 1128 0 0 0 778 0 0 0 25 0 1 0 995031664 4808704 877 33554432000 4194304 4955518 140733457976768 18446744073709551615 4227502 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/3455/statm: 1174 877 104 186 0 987 0
[pid=3461] ppid=3436 vsize=395632 CPUtime=0 cores=0-7
/proc/3461/stat : 3461 (python) S 3436 3436 3404 0 -1 4202560 811 0 0 0 0 0 0 0 15 0 1 0 995031670 405127168 5814 33554432000 4194304 5683644 140735081399008 18446744073709551615 216735198790 0 0 16781312 514 5629147694787166468 0 0 17 1 0 0 0
/proc/3461/statm: 98908 5814 372 364 0 34811 0
[pid=3463] ppid=3461 vsize=4696 CPUtime=7.72 cores=0-7
/proc/3463/stat : 3463 (march_hi) R 3461 3463 3463 0 -1 4202496 1128 0 0 0 772 0 0 0 25 0 1 0 995031670 4808704 873 33554432000 4194304 4955518 140734604731968 18446744073709551615 4307177 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/3463/statm: 1174 873 104 186 0 987 0
[pid=3464] ppid=3436 vsize=405888 CPUtime=0 cores=0-7
/proc/3464/stat : 3464 (python) S 3436 3436 3404 0 -1 4202560 820 0 0 0 0 0 0 0 15 0 1 0 995031673 415629312 5835 33554432000 4194304 5683644 140735081399008 18446744073709551615 216735198790 0 0 16781312 514 4512074667292328196 0 0 17 1 0 0 0
/proc/3464/statm: 101472 5835 372 364 0 37372 0
[pid=3466] ppid=3464 vsize=4696 CPUtime=7.67 cores=0-7
/proc/3466/stat : 3466 (march_hi) R 3464 3466 3466 0 -1 4202496 1125 0 0 0 767 0 0 0 25 0 1 0 995031674 4808704 869 33554432000 4194304 4955518 140734392216752 18446744073709551615 4227597 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/3466/statm: 1174 869 104 186 0 987 0
[pid=3467] ppid=3436 vsize=416144 CPUtime=0 cores=0-7
/proc/3467/stat : 3467 (python) S 3436 3436 3404 0 -1 4202560 862 0 0 0 0 0 0 0 15 0 1 0 995031676 426131456 5860 33554432000 4194304 5683644 140735081399008 18446744073709551615 216735198790 0 0 16781312 514 11254794540256559364 0 0 17 2 0 0 0
/proc/3467/statm: 104036 5860 377 364 0 39933 0
[pid=3469] ppid=3467 vsize=4184 CPUtime=7.65 cores=0-7
/proc/3469/stat : 3469 (march_hi) R 3467 3469 3469 0 -1 4202496 1110 0 0 0 765 0 0 0 25 0 1 0 995031677 4284416 847 33554432000 4194304 4955518 140735098272048 18446744073709551615 4235350 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/3469/statm: 1046 847 104 186 0 859 0
Current children cumulated CPU time (s) 31.96
Current children cumulated vsize (KiB) 1986224

[startup+25.5005 s]
/proc/loadavg: 1.19 0.56 1.08 5/179 3469
/proc/meminfo: memFree=31697284/32951124 swapFree=67054712/67111528
[pid=3436] ppid=3434 vsize=375168 CPUtime=1.14 cores=0-7
/proc/3436/stat : 3436 (python) S 3434 3436 3404 0 -1 4202496 15644 7106 1 0 66 14 32 2 18 0 1 0 995031173 384172032 7358 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747008831 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/3436/statm: 93792 7358 1891 364 0 29689 0
[pid=3453] ppid=3436 vsize=375120 CPUtime=0.02 cores=0-7
/proc/3453/stat : 3453 (python) S 3436 3436 3404 0 -1 4202560 792 0 0 0 1 1 0 0 15 0 1 0 995031664 384122880 5845 33554432000 4194304 5683644 140735081399008 18446744073709551615 216735198790 0 0 16781312 514 16126633801613803780 0 0 17 1 0 0 0
/proc/3453/statm: 93780 5845 371 364 0 29689 0
[pid=3455] ppid=3453 vsize=7460 CPUtime=20.58 cores=0-7
/proc/3455/stat : 3455 (march_hi) R 3453 3455 3455 0 -1 4202496 1681 0 0 0 2058 0 0 0 25 0 1 0 995031664 7639040 1430 33554432000 4194304 4955518 140733457976768 18446744073709551615 4227831 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/3455/statm: 1865 1430 104 186 0 1678 0
[pid=3461] ppid=3436 vsize=395632 CPUtime=0.02 cores=0-7
/proc/3461/stat : 3461 (python) S 3436 3436 3404 0 -1 4202560 812 0 0 0 1 1 0 0 15 0 1 0 995031670 405127168 5814 33554432000 4194304 5683644 140735081399008 18446744073709551615 216735198790 0 0 16781312 514 16126633801613803780 0 0 17 1 0 0 0
/proc/3461/statm: 98908 5814 372 364 0 34811 0
[pid=3463] ppid=3461 vsize=7460 CPUtime=20.51 cores=0-7
/proc/3463/stat : 3463 (march_hi) R 3461 3463 3463 0 -1 4202496 1681 0 0 0 2051 0 0 0 25 0 1 0 995031670 7639040 1426 33554432000 4194304 4955518 140734604731968 18446744073709551615 4227859 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/3463/statm: 1865 1426 104 186 0 1678 0
[pid=3464] ppid=3436 vsize=405888 CPUtime=0.02 cores=0-7
/proc/3464/stat : 3464 (python) S 3436 3436 3404 0 -1 4202560 821 0 0 0 1 1 0 0 15 0 1 0 995031673 415629312 5835 33554432000 4194304 5683644 140735081399008 18446744073709551615 216735198790 0 0 16781312 514 16126633801613803780 0 0 17 1 0 0 0
/proc/3464/statm: 101472 5835 372 364 0 37372 0
[pid=3466] ppid=3464 vsize=7460 CPUtime=20.46 cores=0-7
/proc/3466/stat : 3466 (march_hi) R 3464 3466 3466 0 -1 4202496 1678 0 0 0 2046 0 0 0 25 0 1 0 995031674 7639040 1422 33554432000 4194304 4955518 140734392216752 18446744073709551615 4224687 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/3466/statm: 1865 1422 104 186 0 1678 0
[pid=3467] ppid=3436 vsize=416144 CPUtime=0.02 cores=0-7
/proc/3467/stat : 3467 (python) S 3436 3436 3404 0 -1 4202560 862 0 0 0 1 1 0 0 15 0 1 0 995031676 426131456 5860 33554432000 4194304 5683644 140735081399008 18446744073709551615 216735198790 0 0 16781312 514 16126633801613803780 0 0 17 2 0 0 0
/proc/3467/statm: 104036 5860 377 364 0 39933 0
[pid=3469] ppid=3467 vsize=7460 CPUtime=20.45 cores=0-7
/proc/3469/stat : 3469 (march_hi) R 3467 3469 3469 0 -1 4202496 1619 0 0 0 2045 0 0 0 25 0 1 0 995031677 7639040 1356 33554432000 4194304 4955518 140735098272048 18446744073709551615 4224687 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/3469/statm: 1865 1356 104 186 0 1678 0
Current children cumulated CPU time (s) 83.22
Current children cumulated vsize (KiB) 1997792

[startup+51.101 s]
/proc/loadavg: 2.15 0.84 1.16 5/179 3470
/proc/meminfo: memFree=31681972/32951124 swapFree=67054712/67111528
[pid=3436] ppid=3434 vsize=375168 CPUtime=1.14 cores=0-7
/proc/3436/stat : 3436 (python) S 3434 3436 3404 0 -1 4202496 15644 7106 1 0 66 14 32 2 18 0 1 0 995031173 384172032 7358 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747008831 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/3436/statm: 93792 7358 1891 364 0 29689 0
[pid=3453] ppid=3436 vsize=375120 CPUtime=0.05 cores=0-7
/proc/3453/stat : 3453 (python) S 3436 3436 3404 0 -1 4202560 792 0 0 0 3 2 0 0 15 0 1 0 995031664 384122880 5845 33554432000 4194304 5683644 140735081399008 18446744073709551615 216735198790 0 0 16781312 514 16126633801613803780 0 0 17 1 0 0 0
/proc/3453/statm: 93780 5845 371 364 0 29689 0
[pid=3455] ppid=3453 vsize=12484 CPUtime=46.18 cores=0-7
/proc/3455/stat : 3455 (march_hi) R 3453 3455 3455 0 -1 4202496 2644 0 0 0 4618 0 0 0 25 0 1 0 995031664 12783616 2393 33554432000 4194304 4955518 140733457976768 18446744073709551615 4244920 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/3455/statm: 3121 2393 104 186 0 2934 0
[pid=3461] ppid=3436 vsize=395632 CPUtime=0.05 cores=0-7
/proc/3461/stat : 3461 (python) S 3436 3436 3404 0 -1 4202560 812 0 0 0 3 2 0 0 15 0 1 0 995031670 405127168 5814 33554432000 4194304 5683644 140735081399008 18446744073709551615 216735198790 0 0 16781312 514 16126633801613803780 0 0 17 1 0 0 0
/proc/3461/statm: 98908 5814 372 364 0 34811 0
[pid=3463] ppid=3461 vsize=12484 CPUtime=46.12 cores=0-7
/proc/3463/stat : 3463 (march_hi) R 3461 3463 3463 0 -1 4202496 2642 0 0 0 4612 0 0 0 25 0 1 0 995031670 12783616 2387 33554432000 4194304 4955518 140734604731968 18446744073709551615 4227502 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/3463/statm: 3121 2387 104 186 0 2934 0

################
# More data... #
################

[pid=3461/tid=3488] ppid=3436 vsize=395632 CPUtime=0 cores=0-7
/proc/3461/task/3488/stat : 3488 (python) S 3436 3436 3404 0 -1 4202560 65 0 0 0 0 0 0 0 16 0 2 0 995080655 405127168 5843 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747002289 0 0 16781312 514 18446604453979277312 0 0 -1 0 0 0 0
[pid=3464] ppid=3436 vsize=405888 CPUtime=0.58 cores=0-7
/proc/3464/stat : 3464 (python) S 3436 3436 3404 0 -1 4202560 998 0 0 0 31 27 0 0 16 0 2 0 995031673 415629312 5864 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/3464/statm: 101472 5864 400 364 0 37372 0
[pid=3466] ppid=3464 vsize=85372 CPUtime=489.42 cores=0-7
/proc/3466/stat : 3466 (march_hi) T 3464 3466 3466 0 -1 4202496 18784 0 0 0 48938 4 0 0 25 0 1 0 995031674 87420928 18528 33554432000 4194304 4955518 140734392216752 18446744073709551615 4263703 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/3466/statm: 21343 18528 104 186 0 21156 0
[pid=3464/tid=3486] ppid=3436 vsize=405888 CPUtime=0 cores=0-7
/proc/3464/task/3486/stat : 3486 (python) S 3436 3436 3404 0 -1 4202560 69 0 0 0 0 0 0 0 16 0 2 0 995080623 415629312 5864 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747002289 0 0 16781312 514 18446604453979276416 0 0 -1 0 0 0 0
[pid=3467] ppid=3436 vsize=416144 CPUtime=0.47 cores=0-7
/proc/3467/stat : 3467 (python) S 3436 3436 3404 0 -1 4202560 1046 0 0 0 26 21 0 0 16 0 2 0 995031676 426131456 5890 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/3467/statm: 104036 5890 406 364 0 39933 0
[pid=3469] ppid=3467 vsize=73800 CPUtime=389.67 cores=0-7
/proc/3469/stat : 3469 (march_hi) T 3467 3469 3469 0 -1 4202496 14169 0 0 0 38963 4 0 0 25 0 1 0 995031677 75571200 13906 33554432000 4194304 4955518 140735098272048 18446744073709551615 4224615 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/3469/statm: 18450 13906 104 186 0 18263 0
[pid=3467/tid=3480] ppid=3436 vsize=416144 CPUtime=0 cores=0-7
/proc/3467/task/3480/stat : 3480 (python) S 3436 3436 3404 0 -1 4202560 75 0 0 0 0 0 0 0 16 0 2 0 995070644 426131456 5890 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747002289 0 0 16781312 514 18446604453979275520 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1861.95
Current children cumulated vsize (KiB) 2297868
heavy processes:

[startup+5022.3 s]
/proc/loadavg: 0.00 0.01 0.00 1/183 3648
/proc/meminfo: memFree=31436012/32951124 swapFree=67054712/67111528
[pid=3436] ppid=3434 vsize=375168 CPUtime=1.14 cores=0-7
/proc/3436/stat : 3436 (python) S 3434 3436 3404 0 -1 4202496 15644 7106 1 0 66 14 32 2 18 0 1 0 995031173 384172032 7358 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747008831 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/3436/statm: 93792 7358 1891 364 0 29689 0
[pid=3453] ppid=3436 vsize=375120 CPUtime=0.58 cores=0-7
/proc/3453/stat : 3453 (python) S 3436 3436 3404 0 -1 4202560 1000 0 0 0 32 26 0 0 16 0 2 0 995031664 384122880 5875 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/3453/statm: 93780 5875 400 364 0 29689 0
[pid=3455] ppid=3453 vsize=85372 CPUtime=489.66 cores=0-7
/proc/3455/stat : 3455 (march_hi) T 3453 3455 3455 0 -1 4202496 18857 0 0 0 48962 4 0 0 25 0 1 0 995031664 87420928 18606 33554432000 4194304 4955518 140733457976768 18446744073709551615 4263701 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/3455/statm: 21343 18606 104 186 0 21156 0
[pid=3453/tid=3487] ppid=3436 vsize=375120 CPUtime=0 cores=0-7
/proc/3453/task/3487/stat : 3487 (python) S 3436 3436 3404 0 -1 4202560 99 0 0 0 0 0 0 0 16 0 2 0 995080630 384122880 5875 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747002289 0 0 16781312 514 18446604453895837568 0 0 -1 0 0 0 0
[pid=3461] ppid=3436 vsize=395632 CPUtime=0.58 cores=0-7
/proc/3461/stat : 3461 (python) S 3436 3436 3404 0 -1 4202560 985 0 0 0 33 25 0 0 16 0 2 0 995031670 405127168 5843 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/3461/statm: 98908 5843 400 364 0 34811 0
[pid=3463] ppid=3461 vsize=85372 CPUtime=489.85 cores=0-7
/proc/3463/stat : 3463 (march_hi) T 3461 3463 3463 0 -1 4202496 18823 0 0 0 48982 3 0 0 25 0 1 0 995031670 87420928 18568 33554432000 4194304 4955518 140734604731968 18446744073709551615 4227502 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/3463/statm: 21343 18568 104 186 0 21156 0
[pid=3461/tid=3488] ppid=3436 vsize=395632 CPUtime=0 cores=0-7
/proc/3461/task/3488/stat : 3488 (python) S 3436 3436 3404 0 -1 4202560 65 0 0 0 0 0 0 0 16 0 2 0 995080655 405127168 5843 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747002289 0 0 16781312 514 18446604453979277312 0 0 -1 0 0 0 0
[pid=3464] ppid=3436 vsize=405888 CPUtime=0.58 cores=0-7
/proc/3464/stat : 3464 (python) S 3436 3436 3404 0 -1 4202560 998 0 0 0 31 27 0 0 16 0 2 0 995031673 415629312 5864 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/3464/statm: 101472 5864 400 364 0 37372 0
[pid=3466] ppid=3464 vsize=85372 CPUtime=489.42 cores=0-7
/proc/3466/stat : 3466 (march_hi) T 3464 3466 3466 0 -1 4202496 18784 0 0 0 48938 4 0 0 25 0 1 0 995031674 87420928 18528 33554432000 4194304 4955518 140734392216752 18446744073709551615 4263703 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/3466/statm: 21343 18528 104 186 0 21156 0
[pid=3464/tid=3486] ppid=3436 vsize=405888 CPUtime=0 cores=0-7
/proc/3464/task/3486/stat : 3486 (python) S 3436 3436 3404 0 -1 4202560 69 0 0 0 0 0 0 0 16 0 2 0 995080623 415629312 5864 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747002289 0 0 16781312 514 18446604453979276416 0 0 -1 0 0 0 0
[pid=3467] ppid=3436 vsize=416144 CPUtime=0.47 cores=0-7
/proc/3467/stat : 3467 (python) S 3436 3436 3404 0 -1 4202560 1046 0 0 0 26 21 0 0 16 0 2 0 995031676 426131456 5890 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/3467/statm: 104036 5890 406 364 0 39933 0
[pid=3469] ppid=3467 vsize=73800 CPUtime=389.67 cores=0-7
/proc/3469/stat : 3469 (march_hi) T 3467 3469 3469 0 -1 4202496 14169 0 0 0 38963 4 0 0 25 0 1 0 995031677 75571200 13906 33554432000 4194304 4955518 140735098272048 18446744073709551615 4224615 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/3469/statm: 18450 13906 104 186 0 18263 0
[pid=3467/tid=3480] ppid=3436 vsize=416144 CPUtime=0 cores=0-7
/proc/3467/task/3480/stat : 3480 (python) S 3436 3436 3404 0 -1 4202560 75 0 0 0 0 0 0 0 16 0 2 0 995070644 426131456 5890 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747002289 0 0 16781312 514 18446604453979275520 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1861.95
Current children cumulated vsize (KiB) 2297868
heavy processes:

[startup+5082.3 s]
/proc/loadavg: 0.07 0.02 0.00 1/183 3650
/proc/meminfo: memFree=31436092/32951124 swapFree=67054712/67111528
[pid=3436] ppid=3434 vsize=375168 CPUtime=1.14 cores=0-7
/proc/3436/stat : 3436 (python) S 3434 3436 3404 0 -1 4202496 15644 7106 1 0 66 14 32 2 18 0 1 0 995031173 384172032 7358 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747008831 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/3436/statm: 93792 7358 1891 364 0 29689 0
[pid=3453] ppid=3436 vsize=375120 CPUtime=0.58 cores=0-7
/proc/3453/stat : 3453 (python) S 3436 3436 3404 0 -1 4202560 1000 0 0 0 32 26 0 0 16 0 2 0 995031664 384122880 5875 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/3453/statm: 93780 5875 400 364 0 29689 0
[pid=3455] ppid=3453 vsize=85372 CPUtime=489.66 cores=0-7
/proc/3455/stat : 3455 (march_hi) T 3453 3455 3455 0 -1 4202496 18857 0 0 0 48962 4 0 0 25 0 1 0 995031664 87420928 18606 33554432000 4194304 4955518 140733457976768 18446744073709551615 4263701 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/3455/statm: 21343 18606 104 186 0 21156 0
[pid=3453/tid=3487] ppid=3436 vsize=375120 CPUtime=0 cores=0-7
/proc/3453/task/3487/stat : 3487 (python) S 3436 3436 3404 0 -1 4202560 99 0 0 0 0 0 0 0 16 0 2 0 995080630 384122880 5875 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747002289 0 0 16781312 514 18446604453895837568 0 0 -1 0 0 0 0
[pid=3461] ppid=3436 vsize=395632 CPUtime=0.58 cores=0-7
/proc/3461/stat : 3461 (python) S 3436 3436 3404 0 -1 4202560 985 0 0 0 33 25 0 0 16 0 2 0 995031670 405127168 5843 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/3461/statm: 98908 5843 400 364 0 34811 0
[pid=3463] ppid=3461 vsize=85372 CPUtime=489.85 cores=0-7
/proc/3463/stat : 3463 (march_hi) T 3461 3463 3463 0 -1 4202496 18823 0 0 0 48982 3 0 0 25 0 1 0 995031670 87420928 18568 33554432000 4194304 4955518 140734604731968 18446744073709551615 4227502 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/3463/statm: 21343 18568 104 186 0 21156 0
[pid=3461/tid=3488] ppid=3436 vsize=395632 CPUtime=0 cores=0-7
/proc/3461/task/3488/stat : 3488 (python) S 3436 3436 3404 0 -1 4202560 65 0 0 0 0 0 0 0 16 0 2 0 995080655 405127168 5843 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747002289 0 0 16781312 514 18446604453979277312 0 0 -1 0 0 0 0
[pid=3464] ppid=3436 vsize=405888 CPUtime=0.58 cores=0-7
/proc/3464/stat : 3464 (python) S 3436 3436 3404 0 -1 4202560 998 0 0 0 31 27 0 0 16 0 2 0 995031673 415629312 5864 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/3464/statm: 101472 5864 400 364 0 37372 0
[pid=3466] ppid=3464 vsize=85372 CPUtime=489.42 cores=0-7
/proc/3466/stat : 3466 (march_hi) T 3464 3466 3466 0 -1 4202496 18784 0 0 0 48938 4 0 0 25 0 1 0 995031674 87420928 18528 33554432000 4194304 4955518 140734392216752 18446744073709551615 4263703 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/3466/statm: 21343 18528 104 186 0 21156 0
[pid=3464/tid=3486] ppid=3436 vsize=405888 CPUtime=0 cores=0-7
/proc/3464/task/3486/stat : 3486 (python) S 3436 3436 3404 0 -1 4202560 69 0 0 0 0 0 0 0 16 0 2 0 995080623 415629312 5864 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747002289 0 0 16781312 514 18446604453979276416 0 0 -1 0 0 0 0
[pid=3467] ppid=3436 vsize=416144 CPUtime=0.47 cores=0-7
/proc/3467/stat : 3467 (python) S 3436 3436 3404 0 -1 4202560 1046 0 0 0 26 21 0 0 16 0 2 0 995031676 426131456 5890 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/3467/statm: 104036 5890 406 364 0 39933 0
[pid=3469] ppid=3467 vsize=73800 CPUtime=389.67 cores=0-7
/proc/3469/stat : 3469 (march_hi) T 3467 3469 3469 0 -1 4202496 14169 0 0 0 38963 4 0 0 25 0 1 0 995031677 75571200 13906 33554432000 4194304 4955518 140735098272048 18446744073709551615 4224615 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/3469/statm: 18450 13906 104 186 0 18263 0
[pid=3467/tid=3480] ppid=3436 vsize=416144 CPUtime=0 cores=0-7
/proc/3467/task/3480/stat : 3480 (python) S 3436 3436 3404 0 -1 4202560 75 0 0 0 0 0 0 0 16 0 2 0 995070644 426131456 5890 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747002289 0 0 16781312 514 18446604453979275520 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1861.95
Current children cumulated vsize (KiB) 2297868
heavy processes:



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 0.05 0.02 0.00 1/183 3650
/proc/meminfo: memFree=31436076/32951124 swapFree=67054712/67111528
[pid=3436] ppid=3434 vsize=375168 CPUtime=1.14 cores=0-7
/proc/3436/stat : 3436 (python) S 3434 3436 3404 0 -1 4202496 15644 7106 1 0 66 14 32 2 18 0 1 0 995031173 384172032 7358 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747008831 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/3436/statm: 93792 7358 1891 364 0 29689 0
[pid=3453] ppid=3436 vsize=375120 CPUtime=0.58 cores=0-7
/proc/3453/stat : 3453 (python) S 3436 3436 3404 0 -1 4202560 1000 0 0 0 32 26 0 0 16 0 2 0 995031664 384122880 5875 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/3453/statm: 93780 5875 400 364 0 29689 0
[pid=3455] ppid=3453 vsize=85372 CPUtime=489.66 cores=0-7
/proc/3455/stat : 3455 (march_hi) T 3453 3455 3455 0 -1 4202496 18857 0 0 0 48962 4 0 0 25 0 1 0 995031664 87420928 18606 33554432000 4194304 4955518 140733457976768 18446744073709551615 4263701 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/3455/statm: 21343 18606 104 186 0 21156 0
[pid=3453/tid=3487] ppid=3436 vsize=375120 CPUtime=0 cores=0-7
/proc/3453/task/3487/stat : 3487 (python) S 3436 3436 3404 0 -1 4202560 99 0 0 0 0 0 0 0 16 0 2 0 995080630 384122880 5875 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747002289 0 0 16781312 514 18446604453895837568 0 0 -1 0 0 0 0
[pid=3461] ppid=3436 vsize=395632 CPUtime=0.58 cores=0-7
/proc/3461/stat : 3461 (python) S 3436 3436 3404 0 -1 4202560 985 0 0 0 33 25 0 0 16 0 2 0 995031670 405127168 5843 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/3461/statm: 98908 5843 400 364 0 34811 0
[pid=3463] ppid=3461 vsize=85372 CPUtime=489.85 cores=0-7
/proc/3463/stat : 3463 (march_hi) T 3461 3463 3463 0 -1 4202496 18823 0 0 0 48982 3 0 0 25 0 1 0 995031670 87420928 18568 33554432000 4194304 4955518 140734604731968 18446744073709551615 4227502 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/3463/statm: 21343 18568 104 186 0 21156 0
[pid=3461/tid=3488] ppid=3436 vsize=395632 CPUtime=0 cores=0-7
/proc/3461/task/3488/stat : 3488 (python) S 3436 3436 3404 0 -1 4202560 65 0 0 0 0 0 0 0 16 0 2 0 995080655 405127168 5843 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747002289 0 0 16781312 514 18446604453979277312 0 0 -1 0 0 0 0
[pid=3464] ppid=3436 vsize=405888 CPUtime=0.58 cores=0-7
/proc/3464/stat : 3464 (python) S 3436 3436 3404 0 -1 4202560 998 0 0 0 31 27 0 0 16 0 2 0 995031673 415629312 5864 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/3464/statm: 101472 5864 400 364 0 37372 0
[pid=3466] ppid=3464 vsize=85372 CPUtime=489.42 cores=0-7
/proc/3466/stat : 3466 (march_hi) T 3464 3466 3466 0 -1 4202496 18784 0 0 0 48938 4 0 0 25 0 1 0 995031674 87420928 18528 33554432000 4194304 4955518 140734392216752 18446744073709551615 4263703 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/3466/statm: 21343 18528 104 186 0 21156 0
[pid=3464/tid=3486] ppid=3436 vsize=405888 CPUtime=0 cores=0-7
/proc/3464/task/3486/stat : 3486 (python) S 3436 3436 3404 0 -1 4202560 69 0 0 0 0 0 0 0 16 0 2 0 995080623 415629312 5864 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747002289 0 0 16781312 514 18446604453979276416 0 0 -1 0 0 0 0
[pid=3467] ppid=3436 vsize=416144 CPUtime=0.47 cores=0-7
/proc/3467/stat : 3467 (python) S 3436 3436 3404 0 -1 4202560 1046 0 0 0 26 21 0 0 16 0 2 0 995031676 426131456 5890 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/3467/statm: 104036 5890 406 364 0 39933 0
[pid=3469] ppid=3467 vsize=73800 CPUtime=389.67 cores=0-7
/proc/3469/stat : 3469 (march_hi) T 3467 3469 3469 0 -1 4202496 14169 0 0 0 38963 4 0 0 25 0 1 0 995031677 75571200 13906 33554432000 4194304 4955518 140735098272048 18446744073709551615 4224615 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/3469/statm: 18450 13906 104 186 0 18263 0
[pid=3467/tid=3480] ppid=3436 vsize=416144 CPUtime=0 cores=0-7
/proc/3467/task/3480/stat : 3480 (python) S 3436 3436 3404 0 -1 4202560 75 0 0 0 0 0 0 0 16 0 2 0 995070644 426131456 5890 33554432000 4194304 5683644 140735081399008 18446744073709551615 216747002289 0 0 16781312 514 18446604453979275520 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1861.95
Current children cumulated vsize (KiB) 2297868
heavy processes:

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 3436 and gives
#  childrusage.ru_utime.tv_sec=2
#  childrusage.ru_utime.tv_usec=228661
#  childrusage.ru_stime.tv_sec=1
#  childrusage.ru_stime.tv_usec=197817
# CPU time returned by wait4() is 3.42648
# while last known CPU time is 1861.95
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5100.03
CPU time (s): 1861.95
CPU user time (s): 1860.65
CPU system time (s): 1.3
CPU usage (%): 36.5086
Max. virtual memory (cumulated for all children) (KiB): 2297872

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2.22866
system time used= 1.19782
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 28728
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 5739
involuntary context switches= 26

runsolver used 6.43602 second user time and 17.0244 second system time

The end

Launcher Data 

Begin job on node119 at 2011-06-14 12:40:03
IDJOB=3385679
IDBENCH=82944
IDSOLVER=1870
FILE ID=node119/3385679-1308048003
RUNJOBID= node119-1308048003-3419
PBS_JOBID= 13566209
Free space on /tmp= 74028 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v800-c3408-S1536861174-069.SATISFIABLE.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3385679-1308048003/watcher-3385679-1308048003 -o /tmp/evaluation-result-3385679-1308048003/solver-3385679-1308048003 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385679-1308048003.cnf -seed 14014297 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= b9f30e687453978167416852673e8ddb
RANDOM SEED=14014297

node119.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.63
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      31754740 kB
Buffers:        199284 kB
Cached:         774804 kB
SwapCached:      11032 kB
Active:         324748 kB
Inactive:       668948 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      31754740 kB
SwapTotal:    67111528 kB
SwapFree:     67054712 kB
Dirty:           31872 kB
Writeback:           0 kB
AnonPages:       17500 kB
Mapped:          12584 kB
Slab:           137720 kB
PageTables:       4204 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   184744 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73996 MiB
End job on node119 at 2011-06-14 14:05:07