Trace number 3317221

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (TO) 4800.24 1211.62

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v35000-c147000-S625465881-078-UNKNOWN.cnf
MD5SUM3ba69e7f651721ce4358ca0320f27486
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark497.12
Satisfiable
(Un)Satisfiability was proved
Number of variables35000
Number of clauses147000
Sum of the clauses size441000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3147000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.33/0.56	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.33/0.56	c INFO: solving HOME/instance-3317221-1304268378.cnf
0.33/0.56	c INFO: computing task features
0.84/1.08	c INFO: computed variable-clause graph statistics
0.84/1.10	c INFO: computed clause balance statistics
3.43/3.65	c INFO: computed variable balance statistics
5.53/5.75	c INFO: computed small-clause counts
5.53/5.76	c INFO: computed Horn-clause counts
7.13/7.32	c INFO: computed variable graph statistics
7.13/7.32	c INFO: collected features for HOME/instance-3317221-1304268378.cnf in 6.74 s
7.13/7.35	c INFO: running hybridGM3@0 for 97 with 4793 remaining (b = 0.36)
7.13/7.38	c INFO: running hybridGM3@0 for 97 with 4793 remaining (b = 0.21)
7.22/7.41	c INFO: running hybridGM3@0 for 97 with 4793 remaining (b = 0.14)
7.22/7.44	c INFO: running hybridGM3@0 for 97 with 4793 remaining (b = 0.10)
407.15/107.64	c INFO: running hybridGM3@0 for 97 with 4692 remaining (b = 0.08)
407.15/107.67	c INFO: running hybridGM3@0 for 97 with 4692 remaining (b = 0.08)
407.15/107.70	c INFO: running hybridGM3@0 for 97 with 4692 remaining (b = 0.08)
407.58/107.75	c INFO: running hybridGM3@100 for 97 with 4692 remaining (b = 0.08)
787.17/202.81	c INFO: running hybridGM3@100 for 97 with 4597 remaining (b = 0.08)
787.17/202.86	c INFO: running hybridGM3@99 for 97 with 4597 remaining (b = 0.08)
787.17/202.89	c INFO: running TNM@0 for 97 with 4597 remaining (b = 0.10)
787.57/202.94	c INFO: running hybridGM3@99 for 97 with 4597 remaining (b = 0.08)
1187.11/303.09	c INFO: running hybridGM3@0 for 194 with 4497 remaining (b = 0.14)
1191.07/304.01	c INFO: running hybridGM3@0 for 194 with 4496 remaining (b = 0.10)
1191.07/304.06	c INFO: running hybridGM3@0 for 194 with 4496 remaining (b = 0.07)
1191.37/304.14	c INFO: running TNM@0 for 97 with 4496 remaining (b = 0.03)
1590.83/404.41	c INFO: running hybridGM3@0 for 194 with 4396 remaining (b = 0.05)
1969.90/499.62	c INFO: running hybridGM3@100 for 97 with 4300 remaining (b = 0.02)
1973.53/500.56	c INFO: running hybridGM3@100 for 97 with 4300 remaining (b = 0.02)
1997.38/506.55	c INFO: running hybridGM3@100 for 97 with 4294 remaining (b = 0.02)
2348.99/594.82	c INFO: running hybridGM3@297 for 97 with 4205 remaining (b = 0.02)
2373.27/600.93	c INFO: running hybridGM3@0 for 194 with 4199 remaining (b = 0.04)
2376.48/601.76	c INFO: running TNM@0 for 97 with 4198 remaining (b = 0.02)
2376.77/601.80	c INFO: running hybridGM3@299 for 97 with 4198 remaining (b = 0.02)
2548.81/645.02	c INFO: running TNM@0 for 97 with 4155 remaining (b = 0.02)
2755.86/697.01	c INFO: running hybridGM3@345 for 97 with 4103 remaining (b = 0.02)
2767.40/699.99	c INFO: running TNM@0 for 97 with 4100 remaining (b = 0.01)
2947.77/745.25	c INFO: running TNM@0 for 97 with 4055 remaining (b = 0.01)
3134.81/792.23	c INFO: running hybridGM3@444 for 97 with 4008 remaining (b = 0.01)
3155.54/797.49	c INFO: running hybridGM3@94 for 97 with 4003 remaining (b = 0.02)
3166.66/800.23	c INFO: running TNM@0 for 97 with 4000 remaining (b = 0.01)
3346.19/845.49	c INFO: running TNM@94 for 1940 with 3955 remaining (b = 0.21)
3534.05/892.70	c INFO: running hybridGM3@193 for 97 with 3907 remaining (b = 0.02)
3536.83/893.46	c INFO: running hybridGM3@0 for 194 with 3907 remaining (b = 0.03)
3564.60/900.47	c INFO: running hybridGM3@0 for 194 with 3900 remaining (b = 0.02)
3935.96/993.93	c INFO: running hybridGM3@294 for 97 with 3806 remaining (b = 0.01)
4094.05/1033.74	c INFO: running hybridGM3@0 for 194 with 3766 remaining (b = 0.02)
4314.05/1089.15	c INFO: running hybridGM3@429 for 97 with 3711 remaining (b = 0.01)
4369.28/1103.07	c INFO: running hybridGM3@101 for 97 with 3697 remaining (b = 0.01)
4692.21/1184.37	c INFO: running adaptg2wsat2009++@0 for 194 with 3616 remaining (b = 0.02)
4747.39/1198.29	c INFO: running TNM@0 for 97 with 3602 remaining (b = 0.01)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317221-1304268378/watcher-3317221-1304268378 -o /tmp/evaluation-result-3317221-1304268378/solver-3317221-1304268378 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317221-1304268378.cnf -seed 1070760300 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.85 7.78 5.48 5/280 10508
/proc/meminfo: memFree=30123548/32951124 swapFree=67106388/67111528
[pid=10508] ppid=10506 vsize=15776 CPUtime=0 cores=1,3,5,7
/proc/10508/stat : 10508 (python) R 10506 10508 9843 0 -1 4202496 896 0 0 0 0 0 0 0 24 0 1 0 519254145 16154624 777 33554432000 4194304 5683644 140737264772992 18446744073709551615 212039653376 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/10508/statm: 3944 777 343 364 0 431 0

[startup+0.0565119 s]
/proc/loadavg: 7.85 7.78 5.48 5/280 10508
/proc/meminfo: memFree=30123548/32951124 swapFree=67106388/67111528
[pid=10508] ppid=10506 vsize=36512 CPUtime=0.03 cores=1,3,5,7
/proc/10508/stat : 10508 (python) R 10506 10508 9843 0 -1 4202496 1587 0 0 0 3 0 0 0 24 0 1 0 519254145 37388288 1422 33554432000 4194304 5683644 140737264772992 18446744073709551615 212039655168 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/10508/statm: 9128 1425 452 364 0 945 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 36512

[startup+0.100493 s]
/proc/loadavg: 7.85 7.78 5.48 5/280 10508
/proc/meminfo: memFree=30123548/32951124 swapFree=67106388/67111528
[pid=10508] ppid=10506 vsize=104220 CPUtime=0.07 cores=1,3,5,7
/proc/10508/stat : 10508 (python) S 10506 10508 9843 0 -1 4202496 3050 0 0 0 5 2 0 0 24 0 1 0 519254145 106721280 2733 33554432000 4194304 5683644 140737264772992 18446744073709551615 212039653376 0 2147483391 16781312 2 18446744071699199166 0 0 17 5 0 0 0
/proc/10508/statm: 26055 2733 847 364 0 1803 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 104220

[startup+0.300466 s]
/proc/loadavg: 7.85 7.78 5.48 5/280 10508
/proc/meminfo: memFree=30123548/32951124 swapFree=67106388/67111528
[pid=10508] ppid=10506 vsize=135748 CPUtime=0.2 cores=1,3,5,7
/proc/10508/stat : 10508 (python) S 10506 10508 9843 0 -1 4202496 5959 936 1 0 15 5 0 0 18 0 1 0 519254145 139005952 4151 33554432000 4194304 5683644 140737264772992 18446744073709551615 212039651333 0 2147483391 16781312 2 18446744071699199166 0 0 17 5 0 0 0
/proc/10508/statm: 33937 4151 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.2
Current children cumulated vsize (KiB) 135748

[startup+0.70038 s]
/proc/loadavg: 7.85 7.78 5.48 5/280 10508
/proc/meminfo: memFree=30123548/32951124 swapFree=67106388/67111528
[pid=10508] ppid=10506 vsize=289636 CPUtime=0.53 cores=1,3,5,7
/proc/10508/stat : 10508 (python) R 10506 10508 9843 0 -1 4202496 12015 936 1 0 43 10 0 0 17 0 1 0 519254145 296587264 9701 33554432000 4194304 5683644 140737264772992 18446744073709551615 212039316165 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/10508/statm: 72409 9701 1761 364 0 8333 0
Current children cumulated CPU time (s) 0.53
Current children cumulated vsize (KiB) 289636

[startup+1.50023 s]
/proc/loadavg: 7.85 7.78 5.48 6/281 10513
/proc/meminfo: memFree=30072080/32951124 swapFree=67106388/67111528
[pid=10508] ppid=10506 vsize=312972 CPUtime=1.33 cores=1,3,5,7
/proc/10508/stat : 10508 (python) R 10506 10508 9843 0 -1 4202496 21963 936 1 0 121 12 0 0 18 0 1 0 519254145 320483328 15628 33554432000 4194304 5683644 140737264772992 18446744073709551615 4314432 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/10508/statm: 78243 15628 1826 364 0 14167 0
Current children cumulated CPU time (s) 1.33
Current children cumulated vsize (KiB) 312972

[startup+3.10095 s]
/proc/loadavg: 7.85 7.78 5.48 6/281 10513
/proc/meminfo: memFree=30066376/32951124 swapFree=67106388/67111528
[pid=10508] ppid=10506 vsize=312972 CPUtime=2.93 cores=1,3,5,7
/proc/10508/stat : 10508 (python) R 10506 10508 9843 0 -1 4202496 21963 936 1 0 281 12 0 0 21 0 1 0 519254145 320483328 15628 33554432000 4194304 5683644 140737264772992 18446744073709551615 47552077295305 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/10508/statm: 78243 15628 1826 364 0 14167 0
Current children cumulated CPU time (s) 2.93
Current children cumulated vsize (KiB) 312972

[startup+6.30037 s]
/proc/loadavg: 7.78 7.76 5.49 6/281 10513
/proc/meminfo: memFree=30066276/32951124 swapFree=67106388/67111528
[pid=10508] ppid=10506 vsize=312972 CPUtime=6.13 cores=1,3,5,7
/proc/10508/stat : 10508 (python) R 10506 10508 9843 0 -1 4202496 21970 936 1 0 601 12 0 0 25 0 1 0 519254145 320483328 15635 33554432000 4194304 5683644 140737264772992 18446744073709551615 47552077146252 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/10508/statm: 78243 15635 1833 364 0 14167 0
Current children cumulated CPU time (s) 6.13
Current children cumulated vsize (KiB) 312972

[startup+12.7012 s]
/proc/loadavg: 7.88 7.78 5.51 9/293 10525
/proc/meminfo: memFree=29926776/32951124 swapFree=67106388/67111528
[pid=10508] ppid=10506 vsize=351276 CPUtime=7.27 cores=1,3,5,7
/proc/10508/stat : 10508 (python) S 10506 10508 9843 0 -1 4202496 24968 936 1 0 714 13 0 0 20 0 5 0 519254145 359706624 15016 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051482059 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/10508/statm: 87819 15016 1881 364 0 23728 0
[pid=10514] ppid=10508 vsize=310264 CPUtime=0 cores=1,3,5,7
/proc/10514/stat : 10514 (python) S 10508 10508 9843 0 -1 4202560 894 0 0 0 0 0 0 0 15 0 1 0 519254880 317710336 13495 33554432000 4194304 5683644 140737264772992 18446744073709551615 212039675407 0 0 16781312 514 5632221104664707336 0 0 17 3 0 0 0
/proc/10514/statm: 77566 13495 376 364 0 13484 0
[pid=10516] ppid=10514 vsize=268424 CPUtime=5.32 cores=1,3,5,7
/proc/10516/stat : 10516 (hybridGM3) R 10514 10516 10516 0 -1 4202496 8081 0 0 0 531 1 0 0 25 0 1 0 519254881 274866176 7820 33554432000 134512640 135204385 4288941632 18446744073709551615 134537290 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/10516/statm: 67106 7820 64 169 0 66936 0
[pid=10517] ppid=10508 vsize=320520 CPUtime=0 cores=1,3,5,7
/proc/10517/stat : 10517 (python) S 10508 10508 9843 0 -1 4202560 910 0 0 0 0 0 0 0 15 0 1 0 519254883 328212480 13493 33554432000 4194304 5683644 140737264772992 18446744073709551615 212039675462 0 0 16781312 514 4443369759085396232 0 0 17 1 0 0 0
/proc/10517/statm: 80130 13493 370 364 0 16045 0
[pid=10519] ppid=10517 vsize=268420 CPUtime=5.29 cores=1,3,5,7
/proc/10519/stat : 10519 (hybridGM3) R 10517 10519 10519 0 -1 4202496 8079 0 0 0 528 1 0 0 25 0 1 0 519254884 274862080 7818 33554432000 134512640 135204385 4294414176 18446744073709551615 134537401 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/10519/statm: 67105 7818 64 169 0 66935 0
[pid=10520] ppid=10508 vsize=330776 CPUtime=0 cores=1,3,5,7
/proc/10520/stat : 10520 (python) S 10508 10508 9843 0 -1 4202560 904 0 0 0 0 0 0 0 15 0 1 0 519254886 338714624 13497 33554432000 4194304 5683644 140737264772992 18446744073709551615 212039675462 0 0 16781312 514 5632217256374010120 0 0 17 1 0 0 0
/proc/10520/statm: 82694 13497 370 364 0 18606 0
[pid=10522] ppid=10520 vsize=268428 CPUtime=5.24 cores=1,3,5,7
/proc/10522/stat : 10522 (hybridGM3) R 10520 10522 10522 0 -1 4202496 8090 0 0 0 524 0 0 0 25 0 1 0 519254887 274870272 7822 33554432000 134512640 135204385 4290353696 18446744073709551615 134537292 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/10522/statm: 67107 7822 64 169 0 66937 0
[pid=10523] ppid=10508 vsize=341032 CPUtime=0 cores=1,3,5,7
/proc/10523/stat : 10523 (python) S 10508 10508 9843 0 -1 4202560 942 0 0 0 0 0 0 0 15 0 1 0 519254889 349216768 13501 33554432000 4194304 5683644 140737264772992 18446744073709551615 212039675462 0 0 16781312 514 10086903459715449093 0 0 17 5 0 0 0
/proc/10523/statm: 85258 13501 370 364 0 21167 0
[pid=10525] ppid=10523 vsize=268428 CPUtime=5.2 cores=1,3,5,7
/proc/10525/stat : 10525 (hybridGM3) R 10523 10525 10525 0 -1 4202496 8087 0 0 0 519 1 0 0 25 0 1 0 519254890 274870272 7822 33554432000 134512640 135204385 4291847776 18446744073709551615 134536737 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/10525/statm: 67107 7822 64 169 0 66937 0
[pid=10508/tid=10515] ppid=10506 vsize=351276 CPUtime=0 cores=1,3,5,7
/proc/10508/task/10515/stat : 10515 (python) S 10506 10508 9843 0 -1 4202560 25 936 0 0 0 0 0 0 25 0 5 0 519254880 359706624 15016 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 5 0 0 0
[pid=10508/tid=10518] ppid=10506 vsize=351276 CPUtime=0 cores=1,3,5,7
/proc/10508/task/10518/stat : 10518 (python) S 10506 10508 9843 0 -1 4202560 23 936 0 0 0 0 0 0 25 0 5 0 519254883 359706624 15016 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 3 0 0 0
[pid=10508/tid=10521] ppid=10506 vsize=351276 CPUtime=0 cores=1,3,5,7
/proc/10508/task/10521/stat : 10521 (python) S 10506 10508 9843 0 -1 4202560 23 936 0 0 0 0 0 0 25 0 5 0 519254887 359706624 15016 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 1 0 0 0
[pid=10508/tid=10524] ppid=10506 vsize=351276 CPUtime=0 cores=1,3,5,7
/proc/10508/task/10524/stat : 10524 (python) S 10506 10508 9843 0 -1 4202560 31 936 0 0 0 0 0 0 20 0 5 0 519254890 359706624 15016 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 28.32
Current children cumulated vsize (KiB) 2727568

[startup+25.5009 s]
/proc/loadavg: 7.97 7.81 5.55 9/293 10526
/proc/meminfo: memFree=29927296/32951124 swapFree=67106388/67111528
[pid=10508] ppid=10506 vsize=351276 CPUtime=7.27 cores=1,3,5,7
/proc/10508/stat : 10508 (python) S 10506 10508 9843 0 -1 4202496 24968 936 1 0 714 13 0 0 20 0 5 0 519254145 359706624 15016 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051482059 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/10508/statm: 87819 15016 1881 364 0 23728 0
[pid=10514] ppid=10508 vsize=310264 CPUtime=0.02 cores=1,3,5,7
/proc/10514/stat : 10514 (python) S 10508 10508 9843 0 -1 4202560 901 0 0 0 1 1 0 0 15 0 1 0 519254880 317710336 13495 33554432000 4194304 5683644 140737264772992 18446744073709551615 212039675407 0 0 16781312 514 10493721383308067080 0 0 17 3 0 0 0
/proc/10514/statm: 77566 13495 376 364 0 13484 0
[pid=10516] ppid=10514 vsize=268424 CPUtime=18.1 cores=1,3,5,7
/proc/10516/stat : 10516 (hybridGM3) R 10514 10516 10516 0 -1 4202496 8081 0 0 0 1809 1 0 0 25 0 1 0 519254881 274866176 7820 33554432000 134512640 135204385 4288941632 18446744073709551615 134547435 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/10516/statm: 67106 7820 64 169 0 66936 0
[pid=10517] ppid=10508 vsize=320520 CPUtime=0.02 cores=1,3,5,7
/proc/10517/stat : 10517 (python) S 10508 10508 9843 0 -1 4202560 918 0 0 0 1 1 0 0 15 0 1 0 519254883 328212480 13493 33554432000 4194304 5683644 140737264772992 18446744073709551615 212039675462 0 0 16781312 514 10493721383308067080 0 0 17 1 0 0 0
/proc/10517/statm: 80130 13493 370 364 0 16045 0
[pid=10519] ppid=10517 vsize=268420 CPUtime=18.07 cores=1,3,5,7
/proc/10519/stat : 10519 (hybridGM3) R 10517 10519 10519 0 -1 4202496 8079 0 0 0 1806 1 0 0 25 0 1 0 519254884 274862080 7818 33554432000 134512640 135204385 4294414176 18446744073709551615 134536835 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/10519/statm: 67105 7818 64 169 0 66935 0
[pid=10520] ppid=10508 vsize=330776 CPUtime=0.02 cores=1,3,5,7
/proc/10520/stat : 10520 (python) S 10508 10508 9843 0 -1 4202560 910 0 0 0 1 1 0 0 15 0 1 0 519254886 338714624 13497 33554432000 4194304 5683644 140737264772992 18446744073709551615 212039675462 0 0 16781312 514 10493721383308067080 0 0 17 1 0 0 0
/proc/10520/statm: 82694 13497 370 364 0 18606 0
[pid=10522] ppid=10520 vsize=268428 CPUtime=18 cores=1,3,5,7
/proc/10522/stat : 10522 (hybridGM3) R 10520 10522 10522 0 -1 4202496 8090 0 0 0 1800 0 0 0 25 0 1 0 519254887 274870272 7822 33554432000 134512640 135204385 4290353696 18446744073709551615 134547764 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/10522/statm: 67107 7822 64 169 0 66937 0
[pid=10523] ppid=10508 vsize=341032 CPUtime=0.02 cores=1,3,5,7
/proc/10523/stat : 10523 (python) S 10508 10508 9843 0 -1 4202560 947 0 0 0 0 2 0 0 15 0 1 0 519254889 349216768 13501 33554432000 4194304 5683644 140737264772992 18446744073709551615 212039675462 0 0 16781312 514 10493721383308067077 0 0 17 5 0 0 0
/proc/10523/statm: 85258 13501 370 364 0 21167 0
[pid=10525] ppid=10523 vsize=268428 CPUtime=17.89 cores=1,3,5,7
/proc/10525/stat : 10525 (hybridGM3) R 10523 10525 10525 0 -1 4202496 8087 0 0 0 1788 1 0 0 25 0 1 0 519254890 274870272 7822 33554432000 134512640 135204385 4291847776 18446744073709551615 134536843 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/10525/statm: 67107 7822 64 169 0 66937 0
[pid=10508/tid=10515] ppid=10506 vsize=351276 CPUtime=0 cores=1,3,5,7
/proc/10508/task/10515/stat : 10515 (python) S 10506 10508 9843 0 -1 4202560 25 936 0 0 0 0 0 0 25 0 5 0 519254880 359706624 15016 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 5 0 0 0
[pid=10508/tid=10518] ppid=10506 vsize=351276 CPUtime=0 cores=1,3,5,7
/proc/10508/task/10518/stat : 10518 (python) S 10506 10508 9843 0 -1 4202560 23 936 0 0 0 0 0 0 25 0 5 0 519254883 359706624 15016 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 3 0 0 0
[pid=10508/tid=10521] ppid=10506 vsize=351276 CPUtime=0 cores=1,3,5,7
/proc/10508/task/10521/stat : 10521 (python) S 10506 10508 9843 0 -1 4202560 23 936 0 0 0 0 0 0 25 0 5 0 519254887 359706624 15016 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 1 0 0 0
[pid=10508/tid=10524] ppid=10506 vsize=351276 CPUtime=0 cores=1,3,5,7
/proc/10508/task/10524/stat : 10524 (python) S 10506 10508 9843 0 -1 4202560 31 936 0 0 0 0 0 0 20 0 5 0 519254890 359706624 15016 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 79.41
Current children cumulated vsize (KiB) 2727568

[startup+51.1013 s]
/proc/loadavg: 7.98 7.82 5.62 9/293 10526
/proc/meminfo: memFree=29926668/32951124 swapFree=67106388/67111528
[pid=10508] ppid=10506 vsize=351276 CPUtime=7.27 cores=1,3,5,7
/proc/10508/stat : 10508 (python) S 10506 10508 9843 0 -1 4202496 24968 936 1 0 714 13 0 0 20 0 5 0 519254145 359706624 15016 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051482059 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/10508/statm: 87819 15016 1881 364 0 23728 0
[pid=10514] ppid=10508 vsize=310264 CPUtime=0.07 cores=1,3,5,7
/proc/10514/stat : 10514 (python) S 10508 10508 9843 0 -1 4202560 907 0 0 0 3 4 0 0 15 0 1 0 519254880 317710336 13495 33554432000 4194304 5683644 140737264772992 18446744073709551615 212039675407 0 0 16781312 514 10493721383308067080 0 0 17 3 0 0 0
/proc/10514/statm: 77566 13495 376 364 0 13484 0
[pid=10516] ppid=10514 vsize=268424 CPUtime=43.66 cores=1,3,5,7
/proc/10516/stat : 10516 (hybridGM3) R 10514 10516 10516 0 -1 4202496 8081 0 0 0 4365 1 0 0 25 0 1 0 519254881 274866176 7820 33554432000 134512640 135204385 4288941632 18446744073709551615 134537565 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/10516/statm: 67106 7820 64 169 0 66936 0
[pid=10517] ppid=10508 vsize=320520 CPUtime=0.07 cores=1,3,5,7
/proc/10517/stat : 10517 (python) S 10508 10508 9843 0 -1 4202560 927 0 0 0 3 4 0 0 15 0 1 0 519254883 328212480 13493 33554432000 4194304 5683644 140737264772992 18446744073709551615 212039675462 0 0 16781312 514 10493721383308067080 0 0 17 1 0 0 0
/proc/10517/statm: 80130 13493 370 364 0 16045 0
[pid=10519] ppid=10517 vsize=268420 CPUtime=43.63 cores=1,3,5,7
/proc/10519/stat : 10519 (hybridGM3) R 10517 10519 10519 0 -1 4202496 8079 0 0 0 4362 1 0 0 25 0 1 0 519254884 274862080 7818 33554432000 134512640 135204385 4294414176 18446744073709551615 134536663 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/10519/statm: 67105 7818 64 169 0 66935 0
[pid=10520] ppid=10508 vsize=330776 CPUtime=0.07 cores=1,3,5,7
/proc/10520/stat : 10520 (python) S 10508 10508 9843 0 -1 4202560 919 0 0 0 4 3 0 0 15 0 1 0 519254886 338714624 13497 33554432000 4194304 5683644 140737264772992 18446744073709551615 212039675462 0 0 16781312 514 10493721383308067080 0 0 17 1 0 0 0
/proc/10520/statm: 82694 13497 370 364 0 18606 0
[pid=10522] ppid=10520 vsize=268428 CPUtime=43.5 cores=1,3,5,7
/proc/10522/stat : 10522 (hybridGM3) R 10520 10522 10522 0 -1 4202496 8090 0 0 0 4350 0 0 0 25 0 1 0 519254887 274870272 7822 33554432000 134512640 135204385 4290353696 18446744073709551615 134537272 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/10522/statm: 67107 7822 64 169 0 66937 0
[pid=10523] ppid=10508 vsize=341032 CPUtime=0.07 cores=1,3,5,7
/proc/10523/stat : 10523 (python) S 10508 10508 9843 0 -1 4202560 955 0 0 0 3 4 0 0 15 0 1 0 519254889 349216768 13501 33554432000 4194304 5683644 140737264772992 18446744073709551615 212039675462 0 0 16781312 514 10493721383308067077 0 0 17 5 0 0 0
/proc/10523/statm: 85258 13501 370 364 0 21167 0

################
# More data... #
################

/proc/10605/statm: 87822 13535 399 364 0 23728 0
[pid=10607] ppid=10605 vsize=465044 CPUtime=466 cores=1,3,5,7
/proc/10607/stat : 10607 (TNM) R 10605 10607 10607 0 -1 4202496 6011 0 0 0 46600 0 0 0 25 0 1 0 519274434 476205056 5748 33554432000 134512640 135008748 4293631408 18446744073709551615 134527750 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/10607/statm: 116261 5748 46 122 0 115161 0
[pid=10605/tid=10618] ppid=10508 vsize=351288 CPUtime=0 cores=1,3,5,7
/proc/10605/task/10618/stat : 10618 (python) S 10508 10508 9843 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 519284452 359718912 13535 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 514 18446604462264079424 0 0 -1 5 0 0 0
[pid=10622] ppid=10508 vsize=371800 CPUtime=0.55 cores=1,3,5,7
/proc/10622/stat : 10622 (python) S 10508 10508 9843 0 -1 4202560 1237 0 0 0 28 27 0 0 15 0 2 0 519284547 380723200 13549 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/10622/statm: 92950 13549 405 364 0 28850 0
[pid=10624] ppid=10622 vsize=268428 CPUtime=293.4 cores=1,3,5,7
/proc/10624/stat : 10624 (hybridGM3) T 10622 10624 10624 0 -1 4202496 8102 0 0 0 29340 0 0 0 25 0 1 0 519284548 274870272 7831 33554432000 134512640 135204385 4289443984 18446744073709551615 134536663 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/10624/statm: 67107 7831 64 169 0 66937 0
[pid=10622/tid=10693] ppid=10508 vsize=371800 CPUtime=0 cores=1,3,5,7
/proc/10622/task/10693/stat : 10693 (python) S 10508 10508 9843 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 519304796 380723200 13549 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 514 18446604471135148864 0 0 -1 5 0 0 0
[pid=10625] ppid=10508 vsize=382056 CPUtime=1.13 cores=1,3,5,7
/proc/10625/stat : 10625 (python) S 10508 10508 9843 0 -1 4202560 1215 0 0 0 60 53 0 0 15 0 2 0 519284552 391225344 13553 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/10625/statm: 95514 13553 405 364 0 31411 0
[pid=10627] ppid=10625 vsize=268424 CPUtime=586.37 cores=1,3,5,7
/proc/10627/stat : 10627 (hybridGM3) T 10625 10627 10627 0 -1 4202496 8097 0 0 0 58637 0 0 0 25 0 1 0 519284552 274866176 7826 33554432000 134512640 135204385 4287612688 18446744073709551615 134537290 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/10627/statm: 67106 7826 64 169 0 66936 0
[pid=10625/tid=10692] ppid=10508 vsize=382056 CPUtime=0 cores=1,3,5,7
/proc/10625/task/10692/stat : 10692 (python) S 10508 10508 9843 0 -1 4202560 51 0 0 0 0 0 0 0 15 0 2 0 519304197 391225344 13553 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 514 18446604469056654208 0 0 -1 5 0 0 0
[pid=10628] ppid=10508 vsize=392312 CPUtime=0.18 cores=1,3,5,7
/proc/10628/stat : 10628 (python) S 10508 10508 9843 0 -1 4202560 1143 0 0 0 10 8 0 0 19 0 2 0 519284559 401727488 13551 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/10628/statm: 98078 13551 399 364 0 33972 0
[pid=10630] ppid=10628 vsize=465044 CPUtime=99.45 cores=1,3,5,7
/proc/10630/stat : 10630 (TNM) T 10628 10630 10630 0 -1 4202496 6016 0 0 0 9945 0 0 0 25 0 1 0 519284560 476205056 5749 33554432000 134512640 135008748 4286807744 18446744073709551615 134529409 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/10630/statm: 116261 5749 46 122 0 115161 0
[pid=10628/tid=10677] ppid=10508 vsize=392312 CPUtime=0 cores=1,3,5,7
/proc/10628/task/10677/stat : 10677 (python) S 10508 10508 9843 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 519294582 401727488 13551 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 514 18446604463851170752 0 0 -1 5 0 0 0
[pid=10678] ppid=10508 vsize=402568 CPUtime=0.36 cores=1,3,5,7
/proc/10678/stat : 10678 (python) S 10508 10508 9843 0 -1 4202560 1185 0 0 0 20 16 0 0 17 0 2 0 519294586 412229632 13555 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/10678/statm: 100642 13555 399 364 0 36533 0
[pid=10680] ppid=10678 vsize=268424 CPUtime=195.79 cores=1,3,5,7
/proc/10680/stat : 10680 (hybridGM3) T 10678 10680 10680 0 -1 4202496 8096 0 0 0 19578 1 0 0 25 0 1 0 519294587 274866176 7828 33554432000 134512640 135204385 4286708032 18446744073709551615 134536986 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/10680/statm: 67106 7828 64 169 0 66936 0
[pid=10678/tid=10697] ppid=10508 vsize=402568 CPUtime=0 cores=1,3,5,7
/proc/10678/task/10697/stat : 10697 (python) S 10508 10508 9843 0 -1 4202560 48 0 0 0 0 0 0 0 17 0 2 0 519314234 412229632 13555 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 514 18446604471127114176 0 0 -1 5 0 0 0
[pid=10698] ppid=10508 vsize=412824 CPUtime=1.17 cores=1,3,5,7
/proc/10698/stat : 10698 (python) S 10508 10508 9843 0 -1 4202560 1211 0 0 0 63 54 0 0 15 0 2 0 519314238 422731776 13559 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/10698/statm: 103206 13559 399 364 0 39094 0
[pid=10700] ppid=10698 vsize=268428 CPUtime=582.75 cores=1,3,5,7
/proc/10700/stat : 10700 (hybridGM3) T 10698 10700 10700 0 -1 4202496 8101 0 0 0 58274 1 0 0 25 0 1 0 519314239 274870272 7824 33554432000 134512640 135204385 4293257072 18446744073709551615 134537363 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/10700/statm: 67107 7824 64 169 0 66937 0
[pid=10698/tid=10722] ppid=10508 vsize=412824 CPUtime=0 cores=1,3,5,7
/proc/10698/task/10722/stat : 10722 (python) S 10508 10508 9843 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 519333890 422731776 13559 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 514 18446604459552649408 0 0 -1 5 0 0 0
[pid=10701] ppid=10508 vsize=423080 CPUtime=0.19 cores=1,3,5,7
/proc/10701/stat : 10701 (python) S 10508 10508 9843 0 -1 4202560 1174 0 0 0 9 10 0 0 19 0 2 0 519314321 433233920 13563 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/10701/statm: 105770 13563 399 364 0 41655 0
[pid=10703] ppid=10701 vsize=465044 CPUtime=98.18 cores=1,3,5,7
/proc/10703/stat : 10703 (TNM) T 10701 10703 10703 0 -1 4202496 6016 0 0 0 9817 1 0 0 25 0 1 0 519314321 476205056 5748 33554432000 134512640 135008748 4294278160 18446744073709551615 134527693 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/10703/statm: 116261 5748 46 122 0 115161 0
[pid=10701/tid=10710] ppid=10508 vsize=423080 CPUtime=0 cores=1,3,5,7
/proc/10701/task/10710/stat : 10710 (python) S 10508 10508 9843 0 -1 4202560 51 0 0 0 0 0 0 0 19 0 2 0 519324140 433233920 13563 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 514 18446604469674752960 0 0 -1 7 0 0 0
[pid=10705] ppid=10508 vsize=433336 CPUtime=0.19 cores=1,3,5,7
/proc/10705/stat : 10705 (python) S 10508 10508 9843 0 -1 4202560 1176 0 0 0 8 11 0 0 19 0 2 0 519318647 443736064 13567 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/10705/statm: 108334 13567 399 364 0 44216 0
[pid=10707] ppid=10705 vsize=465044 CPUtime=98.24 cores=1,3,5,7
/proc/10707/stat : 10707 (TNM) T 10705 10707 10707 0 -1 4202496 6018 0 0 0 9823 1 0 0 25 0 1 0 519318647 476205056 5748 33554432000 134512640 135008748 4288680576 18446744073709551615 134528491 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/10707/statm: 116261 5748 46 122 0 115161 0
[pid=10705/tid=10717] ppid=10508 vsize=433336 CPUtime=0 cores=1,3,5,7
/proc/10705/task/10717/stat : 10717 (python) S 10508 10508 9843 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 519328666 443736064 13567 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 514 18446604461858597568 0 0 -1 7 0 0 0
[pid=10711] ppid=10508 vsize=443592 CPUtime=0.19 cores=1,3,5,7
/proc/10711/stat : 10711 (python) S 10508 10508 9843 0 -1 4202560 1178 0 0 0 10 9 0 0 19 0 2 0 519324144 454238208 13571 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/10711/statm: 110898 13571 399 364 0 46777 0
[pid=10713] ppid=10711 vsize=465044 CPUtime=99.42 cores=1,3,5,7
/proc/10713/stat : 10713 (TNM) T 10711 10713 10713 0 -1 4202496 6021 0 0 0 9941 1 0 0 25 0 1 0 519324145 476205056 5748 33554432000 134512640 135008748 4294658096 18446744073709551615 134527416 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/10713/statm: 116261 5748 46 122 0 115161 0
[pid=10711/tid=10723] ppid=10508 vsize=443592 CPUtime=0 cores=1,3,5,7
/proc/10711/task/10723/stat : 10723 (python) S 10508 10508 9843 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 519334164 454238208 13571 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 514 18446604467074269632 0 0 -1 1 0 0 0
[pid=10718] ppid=10508 vsize=453848 CPUtime=0.19 cores=1,3,5,7
/proc/10718/stat : 10718 (python) S 10508 10508 9843 0 -1 4202560 1230 0 0 0 12 7 0 0 19 0 2 0 519328670 464740352 13583 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/10718/statm: 113462 13583 407 364 0 49338 0
[pid=10720] ppid=10718 vsize=465044 CPUtime=97.87 cores=1,3,5,7
/proc/10720/stat : 10720 (TNM) T 10718 10720 10720 0 -1 4202496 6023 0 0 0 9786 1 0 0 25 0 1 0 519328671 476205056 5749 33554432000 134512640 135008748 4294485040 18446744073709551615 134529176 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/10720/statm: 116261 5749 46 122 0 115161 0
[pid=10718/tid=10728] ppid=10508 vsize=453848 CPUtime=0 cores=1,3,5,7
/proc/10718/task/10728/stat : 10728 (python) S 10508 10508 9843 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 519338690 464740352 13583 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 514 18446604461858596672 0 0 -1 7 0 0 0
[pid=10724] ppid=10508 vsize=464104 CPUtime=0.19 cores=1,3,5,7
/proc/10724/stat : 10724 (python) S 10508 10508 9843 0 -1 4202560 1226 0 0 0 10 9 0 0 19 0 2 0 519334168 475242496 13596 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/10724/statm: 116026 13596 416 364 0 51899 0
[pid=10726] ppid=10724 vsize=465044 CPUtime=98.27 cores=1,3,5,7
/proc/10726/stat : 10726 (TNM) T 10724 10726 10726 0 -1 4202496 6028 0 0 0 9826 1 0 0 25 0 1 0 519334169 476205056 5749 33554432000 134512640 135008748 4290176416 18446744073709551615 134524772 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/10726/statm: 116261 5749 46 122 0 115161 0
[pid=10724/tid=10743] ppid=10508 vsize=464104 CPUtime=0 cores=1,3,5,7
/proc/10724/task/10743/stat : 10743 (python) S 10508 10508 9843 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 519344188 475242496 13596 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 514 18446604470476866880 0 0 -1 7 0 0 0
[pid=10744] ppid=10508 vsize=484616 CPUtime=0.59 cores=1,3,5,7
/proc/10744/stat : 10744 (python) S 10508 10508 9843 0 -1 4202560 1221 0 0 0 27 32 0 0 15 0 2 0 519344192 496246784 13588 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/10744/statm: 121154 13588 400 364 0 57021 0
[pid=10746] ppid=10744 vsize=268420 CPUtime=292.87 cores=1,3,5,7
/proc/10746/stat : 10746 (hybridGM3) T 10744 10746 10746 0 -1 4202496 8101 0 0 0 29285 2 0 0 25 0 1 0 519344193 274862080 7825 33554432000 134512640 135204385 4287181376 18446744073709551615 134537435 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/10746/statm: 67105 7825 64 169 0 66935 0
[pid=10744/tid=10765] ppid=10508 vsize=484616 CPUtime=0 cores=1,3,5,7
/proc/10744/task/10765/stat : 10765 (python) S 10508 10508 9843 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 519364448 496246784 13588 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 514 18446604459552651200 0 0 -1 5 0 0 0
[pid=10760] ppid=10508 vsize=494872 CPUtime=0.34 cores=1,3,5,7
/proc/10760/stat : 10760 (python) S 10508 10508 9843 0 -1 4202560 1076 0 0 0 16 18 0 0 15 0 1 0 519357519 506748928 13562 33554432000 4194304 5683644 140737264772992 18446744073709551615 212039675462 0 0 16781312 514 10493721383308067077 0 0 17 5 0 0 0
/proc/10760/statm: 123718 13562 371 364 0 59582 0
[pid=10762] ppid=10760 vsize=268420 CPUtime=173.03 cores=1,3,5,7
/proc/10762/stat : 10762 (hybridGM3) R 10760 10762 10762 0 -1 4202496 8103 0 0 0 17301 2 0 0 25 0 1 0 519357519 274862080 7822 33554432000 134512640 135204385 4287753072 18446744073709551615 134545851 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/10762/statm: 67105 7822 64 169 0 66935 0
[pid=10770] ppid=10508 vsize=505128 CPUtime=0.05 cores=1,3,5,7
/proc/10770/stat : 10770 (python) S 10508 10508 9843 0 -1 4202560 1030 0 0 0 3 2 0 0 18 0 1 0 519372582 517251072 13567 33554432000 4194304 5683644 140737264772992 18446744073709551615 212039675462 0 0 16781312 514 10493721383308067079 0 0 17 5 0 0 0
/proc/10770/statm: 126282 13567 372 364 0 62143 0
[pid=10772] ppid=10770 vsize=431704 CPUtime=27.01 cores=1,3,5,7
/proc/10772/stat : 10772 (adaptg2wsat2009) R 10770 10772 10772 0 -1 4202496 6262 0 0 0 2700 1 0 0 25 0 1 0 519372583 442064896 5983 33554432000 134512640 135107411 4288003744 18446744073709551615 134531508 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/10772/statm: 107926 5983 59 146 0 106802 0
[pid=10773] ppid=10508 vsize=515384 CPUtime=0.02 cores=1,3,5,7
/proc/10773/stat : 10773 (python) S 10508 10508 9843 0 -1 4202560 1047 0 0 0 1 1 0 0 18 0 1 0 519373975 527753216 13570 33554432000 4194304 5683644 140737264772992 18446744073709551615 212039675462 0 0 16781312 514 10493721383308067077 0 0 17 5 0 0 0
/proc/10773/statm: 128846 13570 371 364 0 64704 0
[pid=10775] ppid=10773 vsize=465044 CPUtime=12.81 cores=1,3,5,7
/proc/10775/stat : 10775 (TNM) R 10773 10775 10775 0 -1 4202496 6024 0 0 0 1281 0 0 0 25 0 1 0 519373975 476205056 5748 33554432000 134512640 135008748 4293471200 18446744073709551615 134528940 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/10775/statm: 116261 5748 46 122 0 115161 0
[pid=10508/tid=10515] ppid=10506 vsize=525628 CPUtime=481.18 cores=1,3,5,7
/proc/10508/task/10515/stat : 10515 (python) S 10506 10508 9843 0 -1 4202560 30 19736 0 0 0 0 48059 59 18 0 22 0 519254880 538243072 15119 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 5 0 0 0
[pid=10508/tid=10518] ppid=10506 vsize=525628 CPUtime=481.18 cores=1,3,5,7
/proc/10508/task/10518/stat : 10518 (python) S 10506 10508 9843 0 -1 4202560 26 19736 0 0 0 0 48059 59 18 0 22 0 519254883 538243072 15119 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 3 0 0 0
[pid=10508/tid=10521] ppid=10506 vsize=525628 CPUtime=481.18 cores=1,3,5,7
/proc/10508/task/10521/stat : 10521 (python) S 10506 10508 9843 0 -1 4202560 31 19736 0 0 0 0 48059 59 18 0 22 0 519254887 538243072 15119 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 1 0 0 0
[pid=10508/tid=10524] ppid=10506 vsize=525628 CPUtime=481.18 cores=1,3,5,7
/proc/10508/task/10524/stat : 10524 (python) S 10506 10508 9843 0 -1 4202560 31 19736 0 0 0 0 48059 59 18 0 22 0 519254890 538243072 15119 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 5 0 0 0
[pid=10508/tid=10606] ppid=10506 vsize=525628 CPUtime=481.18 cores=1,3,5,7
/proc/10508/task/10606/stat : 10606 (python) S 10506 10508 9843 0 -1 4202560 33 19736 0 0 0 0 48059 59 18 0 22 0 519274434 538243072 15119 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 5 0 0 0
[pid=10508/tid=10620] ppid=10506 vsize=525628 CPUtime=481.18 cores=1,3,5,7
/proc/10508/task/10620/stat : 10620 (python) S 10506 10508 9843 0 -1 4202560 33 19736 0 0 0 0 48059 59 18 0 22 0 519284455 538243072 15119 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 5 0 0 0
[pid=10508/tid=10623] ppid=10506 vsize=525628 CPUtime=481.18 cores=1,3,5,7
/proc/10508/task/10623/stat : 10623 (python) S 10506 10508 9843 0 -1 4202560 30 19736 0 0 0 0 48059 59 18 0 22 0 519284548 538243072 15119 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 5 0 0 0
[pid=10508/tid=10626] ppid=10506 vsize=525628 CPUtime=481.18 cores=1,3,5,7
/proc/10508/task/10626/stat : 10626 (python) S 10506 10508 9843 0 -1 4202560 95 19736 0 0 0 0 48059 59 18 0 22 0 519284552 538243072 15119 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 5 0 0 0
[pid=10508/tid=10629] ppid=10506 vsize=525628 CPUtime=481.18 cores=1,3,5,7
/proc/10508/task/10629/stat : 10629 (python) S 10506 10508 9843 0 -1 4202560 29 19736 0 0 0 0 48059 59 23 0 22 0 519284560 538243072 15119 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 5 0 0 0
[pid=10508/tid=10679] ppid=10506 vsize=525628 CPUtime=481.18 cores=1,3,5,7
/proc/10508/task/10679/stat : 10679 (python) S 10506 10508 9843 0 -1 4202560 27 19736 0 0 0 0 48059 59 20 0 22 0 519294587 538243072 15119 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 5 0 0 0
[pid=10508/tid=10699] ppid=10506 vsize=525628 CPUtime=481.18 cores=1,3,5,7
/proc/10508/task/10699/stat : 10699 (python) S 10506 10508 9843 0 -1 4202560 48 19736 0 0 0 0 48059 59 18 0 22 0 519314238 538243072 15119 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 5 0 0 0
[pid=10508/tid=10702] ppid=10506 vsize=525628 CPUtime=481.18 cores=1,3,5,7
/proc/10508/task/10702/stat : 10702 (python) S 10506 10508 9843 0 -1 4202560 23 19736 0 0 0 0 48059 59 20 0 22 0 519314321 538243072 15119 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 7 0 0 0
[pid=10508/tid=10706] ppid=10506 vsize=525628 CPUtime=481.18 cores=1,3,5,7
/proc/10508/task/10706/stat : 10706 (python) S 10506 10508 9843 0 -1 4202560 27 19736 0 0 0 0 48059 59 20 0 22 0 519318647 538243072 15119 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 5 0 0 0
[pid=10508/tid=10712] ppid=10506 vsize=525628 CPUtime=481.18 cores=1,3,5,7
/proc/10508/task/10712/stat : 10712 (python) S 10506 10508 9843 0 -1 4202560 31 19736 0 0 0 0 48059 59 20 0 22 0 519324144 538243072 15119 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 5 0 0 0
[pid=10508/tid=10719] ppid=10506 vsize=525628 CPUtime=481.18 cores=1,3,5,7
/proc/10508/task/10719/stat : 10719 (python) S 10506 10508 9843 0 -1 4202560 28 19736 0 0 0 0 48059 59 20 0 22 0 519328671 538243072 15119 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 5 0 0 0
[pid=10508/tid=10725] ppid=10506 vsize=525628 CPUtime=481.18 cores=1,3,5,7
/proc/10508/task/10725/stat : 10725 (python) S 10506 10508 9843 0 -1 4202560 28 19736 0 0 0 0 48059 59 20 0 22 0 519334168 538243072 15119 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 5 0 0 0
[pid=10508/tid=10741] ppid=10506 vsize=525628 CPUtime=481.18 cores=1,3,5,7
/proc/10508/task/10741/stat : 10741 (python) S 10506 10508 9843 0 -1 4202560 27 19736 0 0 0 0 48059 59 20 0 22 0 519343491 538243072 15119 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 5 0 0 0
[pid=10508/tid=10745] ppid=10506 vsize=525628 CPUtime=481.18 cores=1,3,5,7
/proc/10508/task/10745/stat : 10745 (python) S 10506 10508 9843 0 -1 4202560 34 19736 0 0 0 0 48059 59 18 0 22 0 519344192 538243072 15119 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 5 0 0 0
[pid=10508/tid=10761] ppid=10506 vsize=525628 CPUtime=481.18 cores=1,3,5,7
/proc/10508/task/10761/stat : 10761 (python) S 10506 10508 9843 0 -1 4202560 27 19736 0 0 0 0 48059 59 20 0 22 0 519357519 538243072 15119 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 5 0 0 0
[pid=10508/tid=10771] ppid=10506 vsize=525628 CPUtime=481.18 cores=1,3,5,7
/proc/10508/task/10771/stat : 10771 (python) S 10506 10508 9843 0 -1 4202560 28 19736 0 0 0 0 48059 59 20 0 22 0 519372582 538243072 15119 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 5 0 0 0
[pid=10508/tid=10774] ppid=10506 vsize=525628 CPUtime=481.18 cores=1,3,5,7
/proc/10508/task/10774/stat : 10774 (python) S 10506 10508 9843 0 -1 4202560 29 19736 0 0 0 0 48059 59 20 0 22 0 519373975 538243072 15119 33554432000 4194304 5683644 140737264772992 18446744073709551615 212051478961 0 0 16781312 2 18446604471134303232 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.24
Current children cumulated vsize (KiB) 15205572

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 10508 and gives
#  childrusage.ru_utime.tv_sec=489
#  childrusage.ru_utime.tv_usec=88647
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=797878
# CPU time returned by wait4() is 489.887
# while last known CPU time is 4800.24
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1211.62
CPU time (s): 4800.24
CPU user time (s): 4795.35
CPU system time (s): 4.89
CPU usage (%): 396.183
Max. virtual memory (cumulated for all children) (KiB): 15205576

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 489.089
system time used= 0.797878
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 63309
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2610
involuntary context switches= 2527

runsolver used 7.29989 second user time and 15.7636 second system time

The end

Launcher Data 

Begin job on node146 at 2011-05-01 18:46:19
IDJOB=3317221
IDBENCH=82686
IDSOLVER=1705
FILE ID=node146/3317221-1304268378
RUNJOBID= node146-1304267279-9862
PBS_JOBID= 13173933
Free space on /tmp= 73312 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v35000-c147000-S625465881-078-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317221-1304268378/watcher-3317221-1304268378 -o /tmp/evaluation-result-3317221-1304268378/solver-3317221-1304268378 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317221-1304268378.cnf -seed 1070760300 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 3ba69e7f651721ce4358ca0320f27486
RANDOM SEED=1070760300

node146.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.86
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30123828 kB
Buffers:        463924 kB
Cached:        1308256 kB
SwapCached:        888 kB
Active:         905260 kB
Inactive:      1519832 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30123828 kB
SwapTotal:    67111528 kB
SwapFree:     67106388 kB
Dirty:           34404 kB
Writeback:           0 kB
AnonPages:      651784 kB
Mapped:          25088 kB
Slab:           318060 kB
PageTables:      22712 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 33224104 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73212 MiB
End job on node146 at 2011-05-01 19:06:32