Trace number 3385651

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (exit code) 1099.62 184.288

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v600-c2556-S252359291-002.SATISFIABLE.cnf
MD5SUMe7d056c40976f07c31ebd30dc9590b89
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.022995
Satisfiable
(Un)Satisfiability was proved
Number of variables600
Number of clauses2556
Sum of the clauses size7668
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32556
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.35/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.35/0.67	c INFO: solving HOME/instance-3385651-1305571779.cnf
0.35/0.67	c INFO: computing task features
0.35/0.68	c INFO: computed variable-clause graph statistics
0.35/0.68	c INFO: computed clause balance statistics
0.43/0.73	c INFO: computed variable balance statistics
0.43/0.76	c INFO: computed small-clause counts
0.43/0.76	c INFO: computed Horn-clause counts
0.43/0.77	c INFO: computed variable graph statistics
0.43/0.77	c INFO: collected features for HOME/instance-3385651-1305571779.cnf in 0.10 s
0.53/0.80	c INFO: running gnovelty+2@0 for 97 with 4999 remaining (b = 0.58)
0.53/0.83	c INFO: running march_hi@0 for 194 with 4999 remaining (b = 0.17)
0.53/0.86	c INFO: running gnovelty+2@0 for 97 with 4999 remaining (b = 0.07)
0.53/0.89	c INFO: running march_hi@0 for 388 with 4999 remaining (b = 0.27)
0.69/0.92	c INFO: running march_hi@0 for 388 with 4999 remaining (b = 0.24)
0.69/0.95	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.25)
0.69/0.98	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.22)
1.31/1.02	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.19)
1.31/1.02	Traceback (most recent call last):
1.31/1.02	  File "HOME/solve", line 33, in <module>
1.31/1.02	    plac.call(main)
1.31/1.02	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
1.31/1.02	    cmd, result = parser_from(obj).consume(arglist)
1.31/1.02	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
1.31/1.02	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
1.31/1.02	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
1.31/1.02	    (_, answer, _) = solver(input_path, budget / cores, cores)
1.31/1.02	  File "HOME/borg/build/lib/borg/bilevel.py", line 226, in __call__
1.31/1.02	    process.die()
1.31/1.02	  File "HOME/borg/build/lib/borg/solvers.py", line 185, in die
1.31/1.02	    os.kill(self._process.pid, signal.SIGUSR1)
1.31/1.02	OSError: [Errno 3] No such process

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3385651-1305571779/watcher-3385651-1305571779 -o /tmp/evaluation-result-3385651-1305571779/solver-3385651-1305571779 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385651-1305571779.cnf -seed 1548309243 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.44 8.55 8.32 2/366 4629
/proc/meminfo: memFree=27953408/32951124 swapFree=37160400/67111528
[pid=4629] ppid=4627 vsize=15776 CPUtime=0 cores=0-7
/proc/4629/stat : 4629 (python) R 4627 4629 3708 0 -1 4202496 878 0 0 0 0 0 0 0 25 0 1 0 668778380 16154624 760 33554432000 4194304 5683644 140733197786944 18446744073709551615 4509190 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/4629/statm: 3944 761 343 364 0 431 0

[startup+0.0715251 s]
/proc/loadavg: 8.44 8.55 8.32 2/366 4629
/proc/meminfo: memFree=27953408/32951124 swapFree=37160400/67111528
[pid=4629] ppid=4627 vsize=36512 CPUtime=0.03 cores=0-7
/proc/4629/stat : 4629 (python) R 4627 4629 3708 0 -1 4202496 1615 0 0 0 3 0 0 0 25 0 1 0 668778380 37388288 1447 33554432000 4194304 5683644 140733197786944 18446744073709551615 4540219 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/4629/statm: 9128 1450 452 364 0 945 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 36512

[startup+0.10052 s]
/proc/loadavg: 8.44 8.55 8.32 2/366 4629
/proc/meminfo: memFree=27953408/32951124 swapFree=37160400/67111528
[pid=4629] ppid=4627 vsize=87704 CPUtime=0.05 cores=0-7
/proc/4629/stat : 4629 (python) R 4627 4629 3708 0 -1 4202496 2638 0 0 0 4 1 0 0 17 0 1 0 668778380 89808896 2354 33554432000 4194304 5683644 140733197786944 18446744073709551615 230804242055 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/4629/statm: 21926 2354 717 364 0 1622 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 87704

[startup+0.300493 s]
/proc/loadavg: 8.44 8.55 8.32 2/366 4629
/proc/meminfo: memFree=27953408/32951124 swapFree=37160400/67111528
[pid=4629] ppid=4627 vsize=135752 CPUtime=0.18 cores=0-7
/proc/4629/stat : 4629 (python) S 4627 4629 3708 0 -1 4202496 5963 934 1 0 13 5 0 0 17 0 1 0 668778380 139010048 4152 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809161733 0 2147483391 16781312 2 18446744071699199166 0 0 17 4 0 0 0
/proc/4629/statm: 33938 4152 1032 364 0 3124 0
Current children cumulated CPU time (s) 0.18
Current children cumulated vsize (KiB) 135752

[startup+0.700402 s]
/proc/loadavg: 8.44 8.55 8.32 2/366 4629
/proc/meminfo: memFree=27953408/32951124 swapFree=37160400/67111528
[pid=4629] ppid=4627 vsize=278452 CPUtime=0.43 cores=0-7
/proc/4629/stat : 4629 (python) R 4627 4629 3708 0 -1 4202496 9346 934 1 0 32 11 0 0 16 0 1 0 668778380 285134848 7016 33554432000 4194304 5683644 140733197786944 18446744073709551615 4623682 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/4629/statm: 69613 7016 1826 364 0 5537 0
Current children cumulated CPU time (s) 0.43
Current children cumulated vsize (KiB) 278452

[startup+1.50121 s]
/proc/loadavg: 7.84 8.43 8.28 7/384 4657
/proc/meminfo: memFree=27907508/32951124 swapFree=37160400/67111528
[pid=4629] ppid=4627 vsize=375588 CPUtime=0.93 cores=0-7
/proc/4629/stat : 4629 (python) S 4627 4629 3708 0 -1 4202496 15608 3949 1 0 62 13 17 1 20 0 1 0 668778380 384602112 7391 33554432000 4194304 5683644 140733197786944 18446744073709551615 230820995903 0 0 16781312 2 18446744071562232103 0 0 17 2 0 0 0
/proc/4629/statm: 93897 7391 1891 364 0 29794 0
[pid=4638] ppid=4629 vsize=289356 CPUtime=0 cores=0-7
/proc/4638/stat : 4638 (python) S 4629 4629 3708 0 -1 4202560 745 0 0 0 0 0 0 0 15 0 1 0 668778463 296300544 5697 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 15787843487065407746 0 0 17 4 0 0 0
/proc/4638/statm: 72339 5697 348 364 0 8254 0
[pid=4640] ppid=4638 vsize=2608 CPUtime=0.66 cores=0-7
/proc/4640/stat : 4640 (march_hi) R 4638 4640 4640 0 -1 4202496 728 0 0 0 66 0 0 0 21 0 1 0 668778464 2670592 466 33554432000 4194304 4955518 140737109454736 18446744073709551615 4310190 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/4640/statm: 652 466 104 186 0 465 0
[pid=4645] ppid=4629 vsize=310004 CPUtime=0 cores=0-7
/proc/4645/stat : 4645 (python) S 4629 4629 3708 0 -1 4202560 762 0 0 0 0 0 0 0 15 0 1 0 668778469 317444096 5722 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 14154026382794916097 0 0 17 1 0 0 0
/proc/4645/statm: 77501 5722 348 364 0 13410 0
[pid=4647] ppid=4645 vsize=2608 CPUtime=0.6 cores=0-7
/proc/4647/stat : 4647 (march_hi) R 4645 4647 4647 0 -1 4202496 728 0 0 0 60 0 0 0 23 0 1 0 668778470 2670592 465 33554432000 4194304 4955518 140736860224144 18446744073709551615 4209717 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/4647/statm: 652 465 104 186 0 465 0
[pid=4648] ppid=4629 vsize=320260 CPUtime=0 cores=0-7
/proc/4648/stat : 4648 (python) S 4629 4629 3708 0 -1 4202560 767 0 0 0 0 0 0 0 15 0 1 0 668778472 327946240 5726 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 10966331067639759107 0 0 17 1 0 0 0
/proc/4648/statm: 80065 5726 348 364 0 15971 0
[pid=4650] ppid=4648 vsize=2608 CPUtime=0.57 cores=0-7
/proc/4650/stat : 4650 (march_hi) R 4648 4650 4650 0 -1 4202496 728 0 0 0 57 0 0 0 23 0 1 0 668778473 2670592 464 33554432000 4194304 4955518 140735068924176 18446744073709551615 4224551 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/4650/statm: 652 464 104 186 0 465 0
[pid=4651] ppid=4629 vsize=396052 CPUtime=0 cores=0-7
/proc/4651/stat : 4651 (python) S 4629 4629 3708 0 -1 4202560 729 0 0 0 0 0 0 0 15 0 1 0 668778476 405557248 5903 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 9203453288500920579 0 0 17 0 0 0 0
/proc/4651/statm: 99013 5903 348 364 0 34916 0
[pid=4653] ppid=4651 vsize=2472 CPUtime=0.53 cores=0-7
/proc/4653/stat : 4653 (march_hi) R 4651 4653 4653 0 -1 4202496 695 0 0 0 53 0 0 0 22 0 1 0 668778476 2531328 439 33554432000 4194304 4955518 140733378673664 18446744073709551615 4209727 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/4653/statm: 618 439 104 186 0 431 0
[pid=4655] ppid=4629 vsize=406308 CPUtime=0 cores=0-7
/proc/4655/stat : 4655 (python) S 4629 4629 3708 0 -1 4202560 743 0 0 0 0 0 0 0 15 0 1 0 668778479 416059392 5867 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 6441119439713698049 0 0 17 1 0 0 0
/proc/4655/statm: 101577 5867 348 364 0 37477 0
[pid=4657] ppid=4655 vsize=2472 CPUtime=0.49 cores=0-7
/proc/4657/stat : 4657 (march_hi) R 4655 4657 4657 0 -1 4202496 696 0 0 0 49 0 0 0 22 0 1 0 668778479 2531328 438 33554432000 4194304 4955518 140735393192544 18446744073709551615 4235261 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/4657/statm: 618 438 104 186 0 431 0
Current children cumulated CPU time (s) 3.78
Current children cumulated vsize (KiB) 2110336

[startup+3.10095 s]
/proc/loadavg: 7.84 8.43 8.28 7/379 4660
/proc/meminfo: memFree=27899860/32951124 swapFree=37160400/67111528
[pid=4629] ppid=4627 vsize=375588 CPUtime=0.93 cores=0-7
/proc/4629/stat : 4629 (python) S 4627 4629 3708 0 -1 4202496 15608 3949 1 0 62 13 17 1 20 0 1 0 668778380 384602112 7391 33554432000 4194304 5683644 140733197786944 18446744073709551615 230820995903 0 0 16781312 2 18446744071562232103 0 0 17 2 0 0 0
/proc/4629/statm: 93897 7391 1891 364 0 29794 0
[pid=4638] ppid=4629 vsize=289356 CPUtime=0 cores=0-7
/proc/4638/stat : 4638 (python) S 4629 4629 3708 0 -1 4202560 746 0 0 0 0 0 0 0 15 0 1 0 668778463 296300544 5697 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 15787843487065407746 0 0 17 4 0 0 0
/proc/4638/statm: 72339 5697 348 364 0 8254 0
[pid=4640] ppid=4638 vsize=2928 CPUtime=2.26 cores=0-7
/proc/4640/stat : 4640 (march_hi) R 4638 4640 4640 0 -1 4202496 791 0 0 0 226 0 0 0 25 0 1 0 668778464 2998272 529 33554432000 4194304 4955518 140737109454736 18446744073709551615 4227814 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/4640/statm: 732 529 104 186 0 545 0
[pid=4645] ppid=4629 vsize=310004 CPUtime=0 cores=0-7
/proc/4645/stat : 4645 (python) S 4629 4629 3708 0 -1 4202560 766 0 0 0 0 0 0 0 15 0 1 0 668778469 317444096 5722 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 14154026382794916097 0 0 17 1 0 0 0
/proc/4645/statm: 77501 5722 348 364 0 13410 0
[pid=4647] ppid=4645 vsize=2928 CPUtime=2.2 cores=0-7
/proc/4647/stat : 4647 (march_hi) R 4645 4647 4647 0 -1 4202496 790 0 0 0 220 0 0 0 25 0 1 0 668778470 2998272 527 33554432000 4194304 4955518 140736860224144 18446744073709551615 4346362 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/4647/statm: 732 527 104 186 0 545 0
[pid=4648] ppid=4629 vsize=320260 CPUtime=0 cores=0-7
/proc/4648/stat : 4648 (python) S 4629 4629 3708 0 -1 4202560 771 0 0 0 0 0 0 0 15 0 1 0 668778472 327946240 5726 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 10966331067639759107 0 0 17 1 0 0 0
/proc/4648/statm: 80065 5726 348 364 0 15971 0
[pid=4650] ppid=4648 vsize=2928 CPUtime=2.16 cores=0-7
/proc/4650/stat : 4650 (march_hi) R 4648 4650 4650 0 -1 4202496 790 0 0 0 216 0 0 0 25 0 1 0 668778473 2998272 526 33554432000 4194304 4955518 140735068924176 18446744073709551615 4315181 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/4650/statm: 732 526 104 186 0 545 0
[pid=4651] ppid=4629 vsize=396052 CPUtime=0 cores=0-7
/proc/4651/stat : 4651 (python) S 4629 4629 3708 0 -1 4202560 729 0 0 0 0 0 0 0 15 0 1 0 668778476 405557248 5903 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 9203453288500920579 0 0 17 0 0 0 0
/proc/4651/statm: 99013 5903 348 364 0 34916 0
[pid=4653] ppid=4651 vsize=2928 CPUtime=2.13 cores=0-7
/proc/4653/stat : 4653 (march_hi) R 4651 4653 4653 0 -1 4202496 780 0 0 0 213 0 0 0 25 0 1 0 668778476 2998272 524 33554432000 4194304 4955518 140733378673664 18446744073709551615 4295560 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/4653/statm: 732 524 104 186 0 545 0
[pid=4655] ppid=4629 vsize=406308 CPUtime=0 cores=0-7
/proc/4655/stat : 4655 (python) S 4629 4629 3708 0 -1 4202560 743 0 0 0 0 0 0 0 15 0 1 0 668778479 416059392 5867 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 6441119439713698049 0 0 17 1 0 0 0
/proc/4655/statm: 101577 5867 348 364 0 37477 0
[pid=4657] ppid=4655 vsize=2776 CPUtime=2.1 cores=0-7
/proc/4657/stat : 4657 (march_hi) R 4655 4657 4657 0 -1 4202496 776 0 0 0 209 1 0 0 25 0 1 0 668778479 2842624 518 33554432000 4194304 4955518 140735393192544 18446744073709551615 4227565 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/4657/statm: 694 518 104 186 0 507 0
[pid=4658] ppid=4629 vsize=416564 CPUtime=0 cores=0-7
/proc/4658/stat : 4658 (python) S 4629 4629 3708 0 -1 4202560 781 0 0 0 0 0 0 0 15 0 1 0 668778482 426561536 5870 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 8688551993213419778 0 0 17 1 0 0 0
/proc/4658/statm: 104141 5870 354 364 0 40038 0
[pid=4660] ppid=4658 vsize=2928 CPUtime=2.06 cores=0-7
/proc/4660/stat : 4660 (march_hi) R 4658 4660 4660 0 -1 4202496 783 0 0 0 206 0 0 0 25 0 1 0 668778482 2998272 521 33554432000 4194304 4955518 140733870018128 18446744073709551615 4227970 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/4660/statm: 732 521 104 186 0 545 0
Current children cumulated CPU time (s) 13.84
Current children cumulated vsize (KiB) 2531548

[startup+6.30139 s]
/proc/loadavg: 7.84 8.43 8.28 7/379 4660
/proc/meminfo: memFree=27896024/32951124 swapFree=37160400/67111528
[pid=4629] ppid=4627 vsize=375588 CPUtime=0.93 cores=0-7
/proc/4629/stat : 4629 (python) S 4627 4629 3708 0 -1 4202496 15608 3949 1 0 62 13 17 1 20 0 1 0 668778380 384602112 7391 33554432000 4194304 5683644 140733197786944 18446744073709551615 230820995903 0 0 16781312 2 18446744071562232103 0 0 17 2 0 0 0
/proc/4629/statm: 93897 7391 1891 364 0 29794 0
[pid=4638] ppid=4629 vsize=289356 CPUtime=0.01 cores=0-7
/proc/4638/stat : 4638 (python) S 4629 4629 3708 0 -1 4202560 891 0 0 0 1 0 0 0 15 0 1 0 668778463 296300544 5720 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 15787843487065407746 0 0 17 4 0 0 0
/proc/4638/statm: 72339 5720 371 364 0 8254 0
[pid=4640] ppid=4638 vsize=3632 CPUtime=5.45 cores=0-7
/proc/4640/stat : 4640 (march_hi) R 4638 4640 4640 0 -1 4202496 938 0 0 0 545 0 0 0 25 0 1 0 668778464 3719168 676 33554432000 4194304 4955518 140737109454736 18446744073709551615 4227647 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/4640/statm: 908 676 104 186 0 721 0
[pid=4645] ppid=4629 vsize=310004 CPUtime=0.01 cores=0-7
/proc/4645/stat : 4645 (python) S 4629 4629 3708 0 -1 4202560 934 0 0 0 1 0 0 0 15 0 1 0 668778469 317444096 5745 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 14154026382794916097 0 0 17 1 0 0 0
/proc/4645/statm: 77501 5745 371 364 0 13410 0
[pid=4647] ppid=4645 vsize=3632 CPUtime=5.4 cores=0-7
/proc/4647/stat : 4647 (march_hi) R 4645 4647 4647 0 -1 4202496 937 0 0 0 540 0 0 0 25 0 1 0 668778470 3719168 674 33554432000 4194304 4955518 140736860224144 18446744073709551615 4209692 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/4647/statm: 908 674 104 186 0 721 0
[pid=4648] ppid=4629 vsize=320260 CPUtime=0.01 cores=0-7
/proc/4648/stat : 4648 (python) S 4629 4629 3708 0 -1 4202560 929 0 0 0 0 1 0 0 15 0 1 0 668778472 327946240 5749 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 10966331067639759107 0 0 17 1 0 0 0
/proc/4648/statm: 80065 5749 371 364 0 15971 0
[pid=4650] ppid=4648 vsize=3632 CPUtime=5.36 cores=0-7
/proc/4650/stat : 4650 (march_hi) R 4648 4650 4650 0 -1 4202496 935 0 0 0 536 0 0 0 25 0 1 0 668778473 3719168 671 33554432000 4194304 4955518 140735068924176 18446744073709551615 4413724 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/4650/statm: 908 671 104 186 0 721 0
[pid=4651] ppid=4629 vsize=396052 CPUtime=0.02 cores=0-7
/proc/4651/stat : 4651 (python) S 4629 4629 3708 0 -1 4202560 805 0 0 0 1 1 0 0 15 0 1 0 668778476 405557248 5926 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 9203453288500920579 0 0 17 0 0 0 0
/proc/4651/statm: 99013 5926 371 364 0 34916 0
[pid=4653] ppid=4651 vsize=3632 CPUtime=5.33 cores=0-7
/proc/4653/stat : 4653 (march_hi) R 4651 4653 4653 0 -1 4202496 926 0 0 0 533 0 0 0 25 0 1 0 668778476 3719168 670 33554432000 4194304 4955518 140733378673664 18446744073709551615 4206595 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/4653/statm: 908 670 104 186 0 721 0
[pid=4655] ppid=4629 vsize=406308 CPUtime=0.01 cores=0-7
/proc/4655/stat : 4655 (python) S 4629 4629 3708 0 -1 4202560 819 0 0 0 0 1 0 0 15 0 1 0 668778479 416059392 5890 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 6441119439713698049 0 0 17 1 0 0 0
/proc/4655/statm: 101577 5890 371 364 0 37477 0
[pid=4657] ppid=4655 vsize=3632 CPUtime=5.29 cores=0-7
/proc/4657/stat : 4657 (march_hi) R 4655 4657 4657 0 -1 4202496 915 0 0 0 528 1 0 0 25 0 1 0 668778479 3719168 657 33554432000 4194304 4955518 140735393192544 18446744073709551615 4227935 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/4657/statm: 908 657 104 186 0 721 0
[pid=4658] ppid=4629 vsize=416564 CPUtime=0.01 cores=0-7
/proc/4658/stat : 4658 (python) S 4629 4629 3708 0 -1 4202560 858 0 0 0 1 0 0 0 15 0 1 0 668778482 426561536 5893 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 8688551993213419778 0 0 17 1 0 0 0
/proc/4658/statm: 104141 5893 377 364 0 40038 0
[pid=4660] ppid=4658 vsize=3632 CPUtime=5.26 cores=0-7
/proc/4660/stat : 4660 (march_hi) R 4658 4660 4660 0 -1 4202496 928 0 0 0 526 0 0 0 25 0 1 0 668778482 3719168 666 33554432000 4194304 4955518 140733870018128 18446744073709551615 4282721 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/4660/statm: 908 666 104 186 0 721 0
Current children cumulated CPU time (s) 33.09
Current children cumulated vsize (KiB) 2535924

[startup+12.7012 s]
/proc/loadavg: 7.56 8.34 8.26 7/379 4660
/proc/meminfo: memFree=27891360/32951124 swapFree=37160400/67111528
[pid=4629] ppid=4627 vsize=375588 CPUtime=0.93 cores=0-7
/proc/4629/stat : 4629 (python) S 4627 4629 3708 0 -1 4202496 15608 3949 1 0 62 13 17 1 20 0 1 0 668778380 384602112 7391 33554432000 4194304 5683644 140733197786944 18446744073709551615 230820995903 0 0 16781312 2 18446744071562232103 0 0 17 2 0 0 0
/proc/4629/statm: 93897 7391 1891 364 0 29794 0
[pid=4638] ppid=4629 vsize=289356 CPUtime=0.03 cores=0-7

################
# More data... #
################

[pid=4660] ppid=4658 vsize=37604 CPUtime=177.8 cores=0-7
/proc/4660/stat : 4660 (march_hi) R 4658 4660 4660 0 -1 4202496 7258 0 0 0 17779 1 0 0 25 0 1 0 668778482 38506496 6996 33554432000 4194304 4955518 140733870018128 18446744073709551615 4244938 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/4660/statm: 9401 6996 104 186 0 9214 0
Current children cumulated CPU time (s) 1072.47
Current children cumulated vsize (KiB) 2739756

[startup+182.301 s]
/proc/loadavg: 6.09 7.33 7.88 7/379 4665
/proc/meminfo: memFree=27744180/32951124 swapFree=37160400/67111528
[pid=4629] ppid=4627 vsize=375588 CPUtime=0.93 cores=0-7
/proc/4629/stat : 4629 (python) S 4627 4629 3708 0 -1 4202496 15608 3949 1 0 62 13 17 1 20 0 1 0 668778380 384602112 7391 33554432000 4194304 5683644 140733197786944 18446744073709551615 230820995903 0 0 16781312 2 18446744071562232103 0 0 17 2 0 0 0
/proc/4629/statm: 93897 7391 1891 364 0 29794 0
[pid=4638] ppid=4629 vsize=289356 CPUtime=0.53 cores=0-7
/proc/4638/stat : 4638 (python) S 4629 4629 3708 0 -1 4202560 957 0 0 0 27 26 0 0 15 0 1 0 668778463 296300544 5720 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 5386340338707169538 0 0 17 4 0 0 0
/proc/4638/statm: 72339 5720 371 364 0 8254 0
[pid=4640] ppid=4638 vsize=37604 CPUtime=181.45 cores=0-7
/proc/4640/stat : 4640 (march_hi) R 4638 4640 4640 0 -1 4202496 7405 0 0 0 18144 1 0 0 25 0 1 0 668778464 38506496 7143 33554432000 4194304 4955518 140737109454736 18446744073709551615 4263648 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/4640/statm: 9401 7143 104 186 0 9214 0
[pid=4645] ppid=4629 vsize=310004 CPUtime=0.53 cores=0-7
/proc/4645/stat : 4645 (python) S 4629 4629 3708 0 -1 4202560 995 0 0 0 26 27 0 0 16 0 1 0 668778469 317444096 5745 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 5386340338707169537 0 0 17 1 0 0 0
/proc/4645/statm: 77501 5745 371 364 0 13410 0
[pid=4647] ppid=4645 vsize=37604 CPUtime=181.39 cores=0-7
/proc/4647/stat : 4647 (march_hi) R 4645 4647 4647 0 -1 4202496 7395 0 0 0 18138 1 0 0 25 0 1 0 668778470 38506496 7132 33554432000 4194304 4955518 140736860224144 18446744073709551615 4209643 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/4647/statm: 9401 7132 104 186 0 9214 0
[pid=4648] ppid=4629 vsize=320260 CPUtime=0.51 cores=0-7
/proc/4648/stat : 4648 (python) S 4629 4629 3708 0 -1 4202560 992 0 0 0 29 22 0 0 15 0 1 0 668778472 327946240 5749 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 5386340338707169539 0 0 17 1 0 0 0
/proc/4648/statm: 80065 5749 371 364 0 15971 0
[pid=4650] ppid=4648 vsize=37604 CPUtime=181.37 cores=0-7
/proc/4650/stat : 4650 (march_hi) R 4648 4650 4650 0 -1 4202496 7370 0 0 0 18136 1 0 0 25 0 1 0 668778473 38506496 7106 33554432000 4194304 4955518 140735068924176 18446744073709551615 4227831 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/4650/statm: 9401 7106 104 186 0 9214 0
[pid=4651] ppid=4629 vsize=396052 CPUtime=0.54 cores=0-7
/proc/4651/stat : 4651 (python) S 4629 4629 3708 0 -1 4202560 810 0 0 0 31 23 0 0 16 0 1 0 668778476 405557248 5926 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 5386340338707169539 0 0 17 0 0 0 0
/proc/4651/statm: 99013 5926 371 364 0 34916 0
[pid=4653] ppid=4651 vsize=37604 CPUtime=181.33 cores=0-7
/proc/4653/stat : 4653 (march_hi) R 4651 4653 4653 0 -1 4202496 7386 0 0 0 18132 1 0 0 25 0 1 0 668778476 38506496 7130 33554432000 4194304 4955518 140733378673664 18446744073709551615 4227502 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/4653/statm: 9401 7130 104 186 0 9214 0
[pid=4655] ppid=4629 vsize=406308 CPUtime=0.51 cores=0-7
/proc/4655/stat : 4655 (python) S 4629 4629 3708 0 -1 4202560 825 0 0 0 27 24 0 0 15 0 1 0 668778479 416059392 5890 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 5386340338707169537 0 0 17 1 0 0 0
/proc/4655/statm: 101577 5890 371 364 0 37477 0
[pid=4657] ppid=4655 vsize=37604 CPUtime=181.13 cores=0-7
/proc/4657/stat : 4657 (march_hi) R 4655 4657 4657 0 -1 4202496 7354 0 0 0 18111 2 0 0 25 0 1 0 668778479 38506496 7096 33554432000 4194304 4955518 140735393192544 18446744073709551615 4224648 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/4657/statm: 9401 7096 104 186 0 9214 0
[pid=4658] ppid=4629 vsize=416564 CPUtime=0.51 cores=0-7
/proc/4658/stat : 4658 (python) S 4629 4629 3708 0 -1 4202560 864 0 0 0 26 25 0 0 15 0 1 0 668778482 426561536 5893 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 5386340338707169538 0 0 17 1 0 0 0
/proc/4658/statm: 104141 5893 377 364 0 40038 0
[pid=4660] ppid=4658 vsize=37604 CPUtime=181 cores=0-7
/proc/4660/stat : 4660 (march_hi) R 4658 4660 4660 0 -1 4202496 7369 0 0 0 18099 1 0 0 25 0 1 0 668778482 38506496 7107 33554432000 4194304 4955518 140733870018128 18446744073709551615 4209776 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/4660/statm: 9401 7107 104 186 0 9214 0
Current children cumulated CPU time (s) 1091.73
Current children cumulated vsize (KiB) 2739756

[startup+183.101 s]
/proc/loadavg: 6.09 7.33 7.88 7/379 4665
/proc/meminfo: memFree=27743560/32951124 swapFree=37160400/67111528
[pid=4629] ppid=4627 vsize=375588 CPUtime=0.93 cores=0-7
/proc/4629/stat : 4629 (python) S 4627 4629 3708 0 -1 4202496 15608 3949 1 0 62 13 17 1 18 0 1 0 668778380 384602112 7391 33554432000 4194304 5683644 140733197786944 18446744073709551615 230820995903 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/4629/statm: 93897 7391 1891 364 0 29794 0
[pid=4638] ppid=4629 vsize=0 CPUtime=182.5 cores=0-7
/proc/4638/stat : 4638 (python) Z 4629 4629 3708 0 -1 4202588 1198 7429 0 0 27 26 18196 1 15 0 1 0 668778463 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 4 0 0 0
/proc/4638/statm: 0 0 0 0 0 0 0
[pid=4645] ppid=4629 vsize=310004 CPUtime=0.53 cores=0-7
/proc/4645/stat : 4645 (python) S 4629 4629 3708 0 -1 4202560 995 0 0 0 26 27 0 0 15 0 1 0 668778469 317444096 5745 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 5386340338707169537 0 0 17 1 0 0 0
/proc/4645/statm: 77501 5745 371 364 0 13410 0
[pid=4647] ppid=4645 vsize=37604 CPUtime=182.19 cores=0-7
/proc/4647/stat : 4647 (march_hi) R 4645 4647 4647 0 -1 4202496 7422 0 0 0 18218 1 0 0 25 0 1 0 668778470 38506496 7159 33554432000 4194304 4955518 140736860224144 18446744073709551615 4209628 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/4647/statm: 9401 7159 104 186 0 9214 0
[pid=4648] ppid=4629 vsize=320260 CPUtime=0.53 cores=0-7
/proc/4648/stat : 4648 (python) S 4629 4629 3708 0 -1 4202560 994 0 0 0 30 23 0 0 15 0 1 0 668778472 327946240 5749 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 5386340338707169539 0 0 17 1 0 0 0
/proc/4648/statm: 80065 5749 371 364 0 15971 0
[pid=4650] ppid=4648 vsize=37604 CPUtime=182.17 cores=0-7
/proc/4650/stat : 4650 (march_hi) R 4648 4650 4650 0 -1 4202496 7397 0 0 0 18216 1 0 0 25 0 1 0 668778473 38506496 7133 33554432000 4194304 4955518 140735068924176 18446744073709551615 4224638 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/4650/statm: 9401 7133 104 186 0 9214 0
[pid=4651] ppid=4629 vsize=396052 CPUtime=0.54 cores=0-7
/proc/4651/stat : 4651 (python) S 4629 4629 3708 0 -1 4202560 810 0 0 0 31 23 0 0 15 0 1 0 668778476 405557248 5926 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 5386340338707169539 0 0 17 0 0 0 0
/proc/4651/statm: 99013 5926 371 364 0 34916 0
[pid=4653] ppid=4651 vsize=37604 CPUtime=182.13 cores=0-7
/proc/4653/stat : 4653 (march_hi) R 4651 4653 4653 0 -1 4202496 7413 0 0 0 18212 1 0 0 25 0 1 0 668778476 38506496 7157 33554432000 4194304 4955518 140733378673664 18446744073709551615 4227583 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/4653/statm: 9401 7157 104 186 0 9214 0
[pid=4655] ppid=4629 vsize=406308 CPUtime=0.53 cores=0-7
/proc/4655/stat : 4655 (python) S 4629 4629 3708 0 -1 4202560 825 0 0 0 28 25 0 0 15 0 1 0 668778479 416059392 5890 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 5386340338707169537 0 0 17 1 0 0 0
/proc/4655/statm: 101577 5890 371 364 0 37477 0
[pid=4657] ppid=4655 vsize=37604 CPUtime=181.93 cores=0-7
/proc/4657/stat : 4657 (march_hi) R 4655 4657 4657 0 -1 4202496 7382 0 0 0 18191 2 0 0 25 0 1 0 668778479 38506496 7124 33554432000 4194304 4955518 140735393192544 18446744073709551615 4235384 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/4657/statm: 9401 7124 104 186 0 9214 0
[pid=4658] ppid=4629 vsize=416564 CPUtime=0.53 cores=0-7
/proc/4658/stat : 4658 (python) S 4629 4629 3708 0 -1 4202560 864 0 0 0 27 26 0 0 15 0 1 0 668778482 426561536 5893 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 5386340338707169538 0 0 17 1 0 0 0
/proc/4658/statm: 104141 5893 377 364 0 40038 0
[pid=4660] ppid=4658 vsize=37604 CPUtime=181.8 cores=0-7
/proc/4660/stat : 4660 (march_hi) R 4658 4660 4660 0 -1 4202496 7395 0 0 0 18179 1 0 0 25 0 1 0 668778482 38506496 7133 33554432000 4194304 4955518 140733870018128 18446744073709551615 4282647 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/4660/statm: 9401 7133 104 186 0 9214 0
Current children cumulated CPU time (s) 1096.31
Current children cumulated vsize (KiB) 2412796

[startup+183.901 s]
/proc/loadavg: 6.09 7.33 7.88 4/376 4668
/proc/meminfo: memFree=27837316/32951124 swapFree=37160400/67111528
[pid=4629] ppid=4627 vsize=375588 CPUtime=0.93 cores=0-7
/proc/4629/stat : 4629 (python) S 4627 4629 3708 0 -1 4202496 15608 3949 1 0 62 13 17 1 15 0 1 0 668778380 384602112 7391 33554432000 4194304 5683644 140733197786944 18446744073709551615 230820995903 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/4629/statm: 93897 7391 1891 364 0 29794 0
[pid=4638] ppid=4629 vsize=0 CPUtime=182.5 cores=0-7
/proc/4638/stat : 4638 (python) Z 4629 4629 3708 0 -1 4202588 1198 7429 0 0 27 26 18196 1 15 0 1 0 668778463 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 4 0 0 0
/proc/4638/statm: 0 0 0 0 0 0 0
[pid=4645] ppid=4629 vsize=0 CPUtime=182.85 cores=0-7
/proc/4645/stat : 4645 (python) Z 4629 4629 3708 0 -1 4202588 1244 7431 0 0 27 27 18230 1 16 0 1 0 668778469 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 1 0 0 0
/proc/4645/statm: 0 0 0 0 0 0 0
[pid=4648] ppid=4629 vsize=320260 CPUtime=0.53 cores=0-7
/proc/4648/stat : 4648 (python) S 4629 4629 3708 0 -1 4202560 994 0 0 0 30 23 0 0 15 0 1 0 668778472 327946240 5749 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 5386340338707169539 0 0 17 1 0 0 0
/proc/4648/statm: 80065 5749 371 364 0 15971 0
[pid=4650] ppid=4648 vsize=37604 CPUtime=182.97 cores=0-7
/proc/4650/stat : 4650 (march_hi) R 4648 4650 4650 0 -1 4202496 7424 0 0 0 18296 1 0 0 25 0 1 0 668778473 38506496 7160 33554432000 4194304 4955518 140735068924176 18446744073709551615 4227552 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/4650/statm: 9401 7160 104 186 0 9214 0
[pid=4651] ppid=4629 vsize=0 CPUtime=182.81 cores=0-7
/proc/4651/stat : 4651 (python) Z 4629 4629 3708 0 -1 4202588 1132 7423 0 0 31 23 18226 1 15 0 1 0 668778476 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 0 0 0 0
/proc/4651/statm: 0 0 0 0 0 0 0
[pid=4655] ppid=4629 vsize=406308 CPUtime=0.53 cores=0-7
/proc/4655/stat : 4655 (python) S 4629 4629 3708 0 -1 4202560 825 0 0 0 28 25 0 0 15 0 1 0 668778479 416059392 5890 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 5386340338707169537 0 0 17 1 0 0 0
/proc/4655/statm: 101577 5890 371 364 0 37477 0
[pid=4657] ppid=4655 vsize=37604 CPUtime=182.73 cores=0-7
/proc/4657/stat : 4657 (march_hi) R 4655 4657 4657 0 -1 4202496 7408 0 0 0 18271 2 0 0 25 0 1 0 668778479 38506496 7150 33554432000 4194304 4955518 140735393192544 18446744073709551615 4227679 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/4657/statm: 9401 7150 104 186 0 9214 0
[pid=4658] ppid=4629 vsize=416564 CPUtime=0.53 cores=0-7
/proc/4658/stat : 4658 (python) S 4629 4629 3708 0 -1 4202560 864 0 0 0 27 26 0 0 15 0 1 0 668778482 426561536 5893 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 5386340338707169538 0 0 17 1 0 0 0
/proc/4658/statm: 104141 5893 377 364 0 40038 0
[pid=4660] ppid=4658 vsize=37604 CPUtime=182.59 cores=0-7
/proc/4660/stat : 4660 (march_hi) R 4658 4660 4660 0 -1 4202496 7423 0 0 0 18258 1 0 0 25 0 1 0 668778482 38506496 7161 33554432000 4194304 4955518 140733870018128 18446744073709551615 4227664 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/4660/statm: 9401 7161 104 186 0 9214 0
Current children cumulated CPU time (s) 1098.97
Current children cumulated vsize (KiB) 1631532

[startup+184.101 s]
/proc/loadavg: 6.09 7.33 7.88 4/376 4668
/proc/meminfo: memFree=27837316/32951124 swapFree=37160400/67111528
[pid=4629] ppid=4627 vsize=375588 CPUtime=549.89 cores=0-7
/proc/4629/stat : 4629 (python) S 4627 4629 3708 0 -1 4202496 15611 29809 1 0 62 13 54834 80 15 0 1 0 668778380 384602112 7391 33554432000 4194304 5683644 140733197786944 18446744073709551615 230820995903 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/4629/statm: 93897 7391 1891 364 0 29794 0
[pid=4645] ppid=4629 vsize=0 CPUtime=182.85 cores=0-7
/proc/4645/stat : 4645 (python) Z 4629 4629 3708 0 -1 4202588 1244 7431 0 0 27 27 18230 1 16 0 1 0 668778469 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 1 0 0 0
/proc/4645/statm: 0 0 0 0 0 0 0
[pid=4655] ppid=4629 vsize=406308 CPUtime=0.53 cores=0-7
/proc/4655/stat : 4655 (python) S 4629 4629 3708 0 -1 4202560 825 0 0 0 28 25 0 0 15 0 1 0 668778479 416059392 5890 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 5386340338707169537 0 0 17 1 0 0 0
/proc/4655/statm: 101577 5890 371 364 0 37477 0
[pid=4657] ppid=4655 vsize=37604 CPUtime=182.92 cores=0-7
/proc/4657/stat : 4657 (march_hi) R 4655 4657 4657 0 -1 4202496 7416 0 0 0 18290 2 0 0 25 0 1 0 668778479 38506496 7158 33554432000 4194304 4955518 140735393192544 18446744073709551615 4209740 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/4657/statm: 9401 7158 104 186 0 9214 0
[pid=4658] ppid=4629 vsize=0 CPUtime=183.16 cores=0-7
/proc/4658/stat : 4658 (python) Z 4629 4629 3708 0 -1 4202588 1160 7429 0 0 27 26 18262 1 16 0 1 0 668778482 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 3 0 0 0
/proc/4658/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 1099.35
Current children cumulated vsize (KiB) 819500

[startup+184.201 s]
/proc/loadavg: 6.09 7.33 7.88 4/376 4668
/proc/meminfo: memFree=27837316/32951124 swapFree=37160400/67111528
[pid=4629] ppid=4627 vsize=375588 CPUtime=549.89 cores=0-7
/proc/4629/stat : 4629 (python) S 4627 4629 3708 0 -1 4202496 15611 29809 1 0 62 13 54834 80 15 0 1 0 668778380 384602112 7391 33554432000 4194304 5683644 140733197786944 18446744073709551615 230820995903 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/4629/statm: 93897 7391 1891 364 0 29794 0
[pid=4645] ppid=4629 vsize=0 CPUtime=182.85 cores=0-7
/proc/4645/stat : 4645 (python) Z 4629 4629 3708 0 -1 4202588 1244 7431 0 0 27 27 18230 1 16 0 1 0 668778469 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 1 0 0 0
/proc/4645/statm: 0 0 0 0 0 0 0
[pid=4655] ppid=4629 vsize=406308 CPUtime=0.53 cores=0-7
/proc/4655/stat : 4655 (python) S 4629 4629 3708 0 -1 4202560 825 0 0 0 28 25 0 0 15 0 1 0 668778479 416059392 5890 33554432000 4194304 5683644 140733197786944 18446744073709551615 230809185862 0 0 16781312 514 5386340338707169537 0 0 17 1 0 0 0
/proc/4655/statm: 101577 5890 371 364 0 37477 0
[pid=4657] ppid=4655 vsize=37604 CPUtime=183.02 cores=0-7
/proc/4657/stat : 4657 (march_hi) R 4655 4657 4657 0 -1 4202496 7418 0 0 0 18300 2 0 0 25 0 1 0 668778479 38506496 7160 33554432000 4194304 4955518 140735393192544 18446744073709551615 4227824 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/4657/statm: 9401 7160 104 186 0 9214 0
[pid=4658] ppid=4629 vsize=0 CPUtime=183.16 cores=0-7
/proc/4658/stat : 4658 (python) Z 4629 4629 3708 0 -1 4202588 1160 7429 0 0 27 26 18262 1 16 0 1 0 668778482 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 3 0 0 0
/proc/4658/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 1099.45
Current children cumulated vsize (KiB) 819500

Child status: 1
Real time (s): 184.288
CPU time (s): 1099.62
CPU user time (s): 1097.82
CPU system time (s): 1.80872
CPU usage (%): 596.688
Max. virtual memory (cumulated for all children) (KiB): 2739760

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1097.82
system time used= 1.80872
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 75502
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 5389
involuntary context switches= 3751

runsolver used 0.665898 second user time and 1.52077 second system time

The end

Launcher Data 

Begin job on node104 at 2011-05-16 20:49:39
IDJOB=3385651
IDBENCH=82916
IDSOLVER=1870
FILE ID=node104/3385651-1305571779
RUNJOBID= node104-1305564966-3722
PBS_JOBID= 13350146
Free space on /tmp= 73272 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v600-c2556-S252359291-002.SATISFIABLE.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3385651-1305571779/watcher-3385651-1305571779 -o /tmp/evaluation-result-3385651-1305571779/solver-3385651-1305571779 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385651-1305571779.cnf -seed 1548309243 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= e7d056c40976f07c31ebd30dc9590b89
RANDOM SEED=1548309243

node104.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5293.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      27953572 kB
Buffers:        297924 kB
Cached:        2638308 kB
SwapCached:     152320 kB
Active:        1253300 kB
Inactive:      3456268 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      27953572 kB
SwapTotal:    67111528 kB
SwapFree:     37160400 kB
Dirty:           32020 kB
Writeback:           0 kB
AnonPages:     1635568 kB
Mapped:          10444 kB
Slab:           129280 kB
PageTables:      88616 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 87699832 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73240 MiB
End job on node104 at 2011-05-16 20:52:44