Trace number 3249460

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.26 1202.4

General information on the benchmark

Namecrafted/mosoi/289/
289-unsat-7x39.cnf
MD5SUMf717554c4e7f802b3078c48464dbcd9d
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables1092
Number of clauses64155
Sum of the clauses size253344
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 21638
Number of clauses of size 30
Number of clauses of size 462517
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.02/0.06	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249460-1304063077.cnf 
0.02/0.06	c Using SATTIMEOUT of 4800
0.02/0.06	c Using SATRAM of 15500
0.02/0.06	c Size of subset node = 32
0.02/0.06	c Variables = 1092
0.02/0.06	c Clauses = 64155
0.02/0.06	c Literals = 253344
0.02/0.06	c Reduced size:
0.02/0.06	c Variables = 1092
0.02/0.06	c Clauses = 64155
0.02/0.06	c Literals = 253344
0.02/0.06	c a2
0.09/0.19	c a1
0.39/0.41	c Independent sub-problems = 1
0.39/0.41	c Sub-problem with 1092 variables and 64155 clauses
0.39/0.41	c using permutation method 4
0.39/0.41	c count = 1
0.39/0.41	c Ordering finished 0.390 seconds
0.39/0.41	c mapped 0.410 seconds
0.39/0.41	c permuted 0.410 seconds
0.39/0.41	c about to sort
0.39/0.42	c sorted
0.49/0.55	c clauses now 64155
0.49/0.55	c init finished 0.430 seconds
0.49/0.55	c tree built 0.550 seconds
0.49/0.55	c 
0.49/0.55	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.49/0.55	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.49/0.55	c ................................ | ..........                    |                                  0.6         0.0             
0.49/0.55	c ................................ | ..........                  1 |                                  0.6         1.8             
0.49/0.55	c ................................ | ..........                  2 |                                  0.6         3.6             
0.49/0.55	c ................................ | ..........                  3 |                                  0.6         5.5             
0.49/0.55	c ................................ | ..........                  4 |                                  0.6         7.3             
0.49/0.55	c ................................ | ..........                  5 |                                  0.6         9.1             
0.49/0.56	c ................................ | ..........               1000 |              216                 0.6      1785.7          419
95.76/24.46	c ................................ | ..........            2632000 | 96528        600367             96.0     27416.7     12875717
191.63/48.49	c ................................ | ..........            4162000 | 157658       986771            192.0     21677.1     19053931
287.87/72.54	c ................................ | ..........            5905000 | 230639       1432632           288.0     20502.8     24159029
383.71/96.58	c ................................ | ..........            7185000 | 280295       1779586           384.0     18709.0     30166621
479.95/120.62	c ................................ | *.........            8276000 | 334381       2048170           480.1     17239.9     34125529
575.81/144.65	c ................................ | *.........            9413000 | 377961       2264845           576.0     16341.4     37383360
672.06/168.71	c ................................ | *.........           10268000 | 444393       2483487           672.1     15277.5     43484430
767.91/192.76	c ................................ | *.........           10771000 | 490362       2613194           768.2     14021.5     45051032
863.79/216.78	c ................................ | *.........           11235000 | 537278       2714889           864.1     13002.0     48007480
960.04/240.81	c ................................ | **........           11718000 | 579732       2822672           960.1     12205.2     50900833
1055.89/264.82	c ................................ | **........           12210000 | 621912       2928047          1056.0     11562.5     52488687
1151.76/288.87	c ................................ | **........           12659000 | 660690       3025514          1152.0     10988.3     55142313
1248.02/312.90	c ................................ | **........           13075000 | 698340       3124509          1248.0     10476.4     58314553
1343.89/336.94	c ................................ | **........           13489000 | 732972       3210025          1344.0     10036.1     60733118
1440.16/361.00	c ................................ | ***.......           13879000 | 767566       3306646          1440.2      9637.1     63796855
1536.04/385.02	c ................................ | ***.......           14253000 | 800553       3404519          1536.1      9278.6     65497462
1631.89/409.06	c ................................ | ***.......           14634000 | 838492       3495867          1632.2      8966.1     67353658
1728.16/433.10	c ................................ | ***.......           14990000 | 874184       3584485          1728.2      8673.9     68526154
1824.03/457.18	c ................................ | ***.......           15365000 | 909427       3661200          1824.4      8422.1     68526154
1919.90/481.15	c ................................ | ****......           15778000 | 944583       3742485          1920.1      8217.2     69018195
2015.77/505.18	c ................................ | ****......           16175000 | 982353       3818688          2016.1      8022.9     69946488
2112.02/529.24	c ................................ | ****......           16586000 | 1018279      3929371          2112.2      7852.5     71735144
2207.91/553.24	c ................................ | ****......           16946000 | 1049979      4025161          2208.1      7674.6     73811940
2303.75/577.26	c ................................ | ****......           17291000 | 1079452      4106081          2304.0      7504.7     75591493
2400.01/601.35	c ................................ | *****.....           17650000 | 1115534      4194609          2400.2      7353.5     75591493
2496.27/625.41	c ................................ | *****.....           17988000 | 1147673      4273121          2496.3      7205.8     77158834
2592.14/649.41	c ................................ | *****.....           18295000 | 1175633      4344449          2592.2      7057.7     79158694
2687.96/673.41	c ................................ | *****.....           18618000 | 1204663      4418472          2688.0      6926.3     80017639
2783.83/697.45	c ................................ | *****.....           18988000 | 1239227      4497075          2784.0      6820.4     80421970
2880.04/721.50	c ................................ | ******....           19381000 | 1276049      4568943          2880.1      6729.4     82317833
2975.84/745.55	c ................................ | ******....           19753000 | 1308057      4634714          2976.1      6637.3     83711654
3072.12/769.61	c ................................ | ******....           20101000 | 1340411      4699690          3072.2      6542.9     86663630
3167.99/793.65	c ................................ | ******....           20455000 | 1372862      4764506          3168.2      6456.3     88220665
3263.85/817.69	c ................................ | ******....           20767000 | 1404172      4837054          3264.2      6362.0     90150303
3359.71/841.67	c ................................ | *******...           21060000 | 1434205      4906366          3360.0      6267.8     92470214
3456.01/865.77	c ................................ | *******...           21355000 | 1465059      4975926          3456.3      6178.6     94104491
3552.28/889.81	c ................................ | *******...           21657000 | 1498181      5046695          3552.3      6096.6     96415965
3647.73/913.76	c ................................ | *******...           22030000 | 1531455      5117550          3648.0      6038.9     98522456
3744.00/937.81	c ................................ | *******...           22383000 | 1561324      5194092          3744.1      5978.3    101342163
3839.87/961.85	c ................................ | ********..           22722000 | 1591116      5266710          3840.1      5917.0    103396050
3936.12/985.93	c ................................ | ********..           23064000 | 1623090      5333290          3936.2      5859.4    104419903
4031.94/1010.00	c ................................ | ********..           23374000 | 1653258      5400646          4032.3      5796.6    104573676
4128.23/1034.03	c ................................ | ********..           23636000 | 1677347      5463249          4128.3      5725.3    104839977
4223.99/1058.01	c ................................ | ********..           23874000 | 1701856      5520760          4224.0      5651.9    105929327
4320.18/1082.10	c ................................ | *********.           24110000 | 1723875      5590405          4320.2      5580.8    106173211
4416.04/1106.10	c ................................ | *********.           24360000 | 1746300      5664224          4416.1      5516.2    106917843
4512.27/1130.21	c ................................ | *********.           24588000 | 1766463      5730240          4512.4      5449.0    107872198
4607.74/1154.19	c ................................ | *********.           24804000 | 1786793      5790950          4608.1      5382.7    108279616
4704.01/1178.27	c ................................ | *********.           25013000 | 1806640      5854940          4704.3      5317.0    110061294
4799.90/1202.25	c CPU limit (4800) reached
4799.90/1202.25	c Time = 4800.11 seconds
4799.90/1202.25	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3249460-1304063077/watcher-3249460-1304063077 -o /tmp/evaluation-result-3249460-1304063077/solver-3249460-1304063077 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249460-1304063077.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.18 7.81 6.87 2/177 30491
/proc/meminfo: memFree=28724036/32951124 swapFree=67111356/67111528
[pid=30491] ppid=30489 vsize=9408 CPUtime=0 cores=1,3,5,7
/proc/30491/stat : 30491 (ttsth-5-4) R 30489 30491 27559 0 -1 4202496 411 0 0 0 0 0 0 0 21 0 1 0 517908427 9633792 353 33554432000 4194304 4338100 140736394189472 18446744073709551615 222921437893 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/30491/statm: 2352 354 147 36 0 257 0

[startup+0.022655 s]
/proc/loadavg: 7.18 7.81 6.87 2/177 30491
/proc/meminfo: memFree=28724036/32951124 swapFree=67111356/67111528
[pid=30491] ppid=30489 vsize=11556 CPUtime=0.02 cores=1,3,5,7
/proc/30491/stat : 30491 (ttsth-5-4) R 30489 30491 27559 0 -1 4202496 947 0 0 0 2 0 0 0 21 0 1 0 517908427 11833344 889 33554432000 4194304 4338100 140736394189472 18446744073709551615 222921437893 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/30491/statm: 2889 890 148 36 0 794 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 11556

[startup+0.100646 s]
/proc/loadavg: 7.18 7.81 6.87 2/177 30491
/proc/meminfo: memFree=28724036/32951124 swapFree=67111356/67111528
[pid=30491] ppid=30489 vsize=34760 CPUtime=0.09 cores=1,3,5,7
/proc/30491/stat : 30491 (ttsth-5-4) R 30489 30491 27559 0 -1 4202496 6085 0 0 0 9 0 0 0 21 0 1 0 517908427 35594240 6026 33554432000 4194304 4338100 140736394189472 18446744073709551615 222921172689 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/30491/statm: 8690 6026 153 36 0 6595 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 34760

[startup+0.300605 s]
/proc/loadavg: 7.18 7.81 6.87 2/177 30491
/proc/meminfo: memFree=28724036/32951124 swapFree=67111356/67111528
[pid=30491] ppid=30489 vsize=22324 CPUtime=0.29 cores=1,3,5,7
/proc/30491/stat : 30491 (ttsth-5-4) R 30489 30491 27559 0 -1 4202496 7118 0 0 0 29 0 0 0 22 0 1 0 517908427 22859776 3606 33554432000 4194304 4338100 140736394189472 18446744073709551615 4245205 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/30491/statm: 5581 3606 155 36 0 3486 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 22324

[startup+0.700496 s]
/proc/loadavg: 7.18 7.81 6.87 2/177 30491
/proc/meminfo: memFree=28724036/32951124 swapFree=67111356/67111528
[pid=30491] ppid=30489 vsize=232596 CPUtime=1.11 cores=1,3,5,7
/proc/30491/stat : 30491 (ttsth-5-4) S 30489 30491 27559 0 -1 4202496 14227 0 0 0 110 1 0 0 23 0 5 0 517908427 238178304 10589 33554432000 4194304 4338100 140736394189472 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30491/statm: 58149 10589 176 36 0 56054 0
Current children cumulated CPU time (s) 1.11
Current children cumulated vsize (KiB) 232596

[startup+1.50035 s]
/proc/loadavg: 7.18 7.81 6.87 9/186 30500
/proc/meminfo: memFree=28650812/32951124 swapFree=67111356/67111528
[pid=30491] ppid=30489 vsize=243168 CPUtime=4.3 cores=1,3,5,7
/proc/30491/stat : 30491 (ttsth-5-4) S 30489 30491 27559 0 -1 4202496 18332 0 0 0 427 3 0 0 23 0 5 0 517908427 249004032 14694 33554432000 4194304 4338100 140736394189472 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30491/statm: 60792 14694 176 36 0 58697 0
[pid=30491/tid=30493] ppid=30489 vsize=243168 CPUtime=0.93 cores=1,3,5,7
/proc/30491/task/30493/stat : 30493 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 1117 0 0 0 92 1 0 0 19 0 5 0 517908483 249004032 14694 33554432000 4194304 4338100 140736394189472 18446744073709551615 222921473579 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30491/tid=30494] ppid=30489 vsize=243168 CPUtime=0.93 cores=1,3,5,7
/proc/30491/task/30494/stat : 30494 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 1466 0 0 0 93 0 0 0 19 0 5 0 517908483 249004032 14694 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30491/tid=30495] ppid=30489 vsize=243168 CPUtime=0.93 cores=1,3,5,7
/proc/30491/task/30495/stat : 30495 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 952 0 0 0 93 0 0 0 19 0 5 0 517908483 249004032 14694 33554432000 4194304 4338100 140736394189472 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30491/tid=30496] ppid=30489 vsize=243168 CPUtime=0.93 cores=1,3,5,7
/proc/30491/task/30496/stat : 30496 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 1415 0 0 0 93 0 0 0 19 0 5 0 517908483 249004032 14694 33554432000 4194304 4338100 140736394189472 18446744073709551615 222921172393 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4.3
Current children cumulated vsize (KiB) 243168

[startup+3.10106 s]
/proc/loadavg: 7.25 7.82 6.88 10/186 30500
/proc/meminfo: memFree=28583896/32951124 swapFree=67111356/67111528
[pid=30491] ppid=30489 vsize=291428 CPUtime=10.7 cores=1,3,5,7
/proc/30491/stat : 30491 (ttsth-5-4) S 30489 30491 27559 0 -1 4202496 29942 0 0 0 1064 6 0 0 23 0 5 0 517908427 298422272 26304 33554432000 4194304 4338100 140736394189472 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30491/statm: 72857 26304 176 36 0 70762 0
[pid=30491/tid=30493] ppid=30489 vsize=291428 CPUtime=2.52 cores=1,3,5,7
/proc/30491/task/30493/stat : 30493 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 2780 0 0 0 250 2 0 0 22 0 5 0 517908483 298422272 26304 33554432000 4194304 4338100 140736394189472 18446744073709551615 222921171984 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30491/tid=30494] ppid=30489 vsize=291428 CPUtime=2.54 cores=1,3,5,7
/proc/30491/task/30494/stat : 30494 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 6644 0 0 0 253 1 0 0 22 0 5 0 517908483 298422272 26304 33554432000 4194304 4338100 140736394189472 18446744073709551615 222921172110 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30491/tid=30495] ppid=30489 vsize=291428 CPUtime=2.53 cores=1,3,5,7
/proc/30491/task/30495/stat : 30495 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 2561 0 0 0 252 1 0 0 22 0 5 0 517908483 298422272 26304 33554432000 4194304 4338100 140736394189472 18446744073709551615 222921086824 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30491/tid=30496] ppid=30489 vsize=291428 CPUtime=2.53 cores=1,3,5,7
/proc/30491/task/30496/stat : 30496 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 4575 0 0 0 253 0 0 0 22 0 5 0 517908483 298422272 26304 33554432000 4194304 4338100 140736394189472 18446744073709551615 222921172402 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 10.7
Current children cumulated vsize (KiB) 291428

[startup+6.30047 s]
/proc/loadavg: 7.25 7.82 6.88 9/186 30501
/proc/meminfo: memFree=28414984/32951124 swapFree=67111356/67111528
[pid=30491] ppid=30489 vsize=378768 CPUtime=23.48 cores=1,3,5,7
/proc/30491/stat : 30491 (ttsth-5-4) S 30489 30491 27559 0 -1 4202496 52214 0 0 0 2329 19 0 0 23 0 5 0 517908427 387858432 48576 33554432000 4194304 4338100 140736394189472 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30491/statm: 94692 48576 176 36 0 92597 0
[pid=30491/tid=30493] ppid=30489 vsize=378768 CPUtime=5.7 cores=1,3,5,7
/proc/30491/task/30493/stat : 30493 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 4940 0 0 0 563 7 0 0 18 0 5 0 517908483 387858432 48576 33554432000 4194304 4338100 140736394189472 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30491/tid=30494] ppid=30489 vsize=378768 CPUtime=5.73 cores=1,3,5,7
/proc/30491/task/30494/stat : 30494 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 13589 0 0 0 571 2 0 0 18 0 5 0 517908483 387858432 48576 33554432000 4194304 4338100 140736394189472 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30491/tid=30495] ppid=30489 vsize=378768 CPUtime=5.73 cores=1,3,5,7
/proc/30491/task/30495/stat : 30495 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 4746 0 0 0 568 5 0 0 25 0 5 0 517908483 387858432 48576 33554432000 4194304 4338100 140736394189472 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30491/tid=30496] ppid=30489 vsize=378768 CPUtime=5.73 cores=1,3,5,7
/proc/30491/task/30496/stat : 30496 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 15557 0 0 0 571 2 0 0 25 0 5 0 517908483 387858432 48576 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 23.48
Current children cumulated vsize (KiB) 378768

[startup+12.7003 s]
/proc/loadavg: 7.43 7.84 6.89 10/186 30501
/proc/meminfo: memFree=28148792/32951124 swapFree=67111356/67111528
[pid=30491] ppid=30489 vsize=495296 CPUtime=49.03 cores=1,3,5,7
/proc/30491/stat : 30491 (ttsth-5-4) S 30489 30491 27559 0 -1 4202496 82764 0 0 0 4870 33 0 0 23 0 5 0 517908427 507183104 79126 33554432000 4194304 4338100 140736394189472 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30491/statm: 123824 79127 176 36 0 121729 0
[pid=30491/tid=30493] ppid=30489 vsize=495296 CPUtime=12.09 cores=1,3,5,7
/proc/30491/task/30493/stat : 30493 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 10444 0 0 0 1196 13 0 0 25 0 5 0 517908483 507183104 79127 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30491/tid=30494] ppid=30489 vsize=495296 CPUtime=12.13 cores=1,3,5,7
/proc/30491/task/30494/stat : 30494 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 23663 0 0 0 1209 4 0 0 25 0 5 0 517908483 507183104 79127 33554432000 4194304 4338100 140736394189472 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30491/tid=30495] ppid=30489 vsize=495296 CPUtime=12.1 cores=1,3,5,7
/proc/30491/task/30495/stat : 30495 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 8888 0 0 0 1200 10 0 0 25 0 5 0 517908483 507183104 79127 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30491/tid=30496] ppid=30489 vsize=495296 CPUtime=12.13 cores=1,3,5,7
/proc/30491/task/30496/stat : 30496 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 26388 0 0 0 1209 4 0 0 25 0 5 0 517908483 507183104 79127 33554432000 4194304 4338100 140736394189472 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 49.03
Current children cumulated vsize (KiB) 495296

[startup+25.501 s]
/proc/loadavg: 7.52 7.84 6.91 9/186 30501
/proc/meminfo: memFree=27795660/32951124 swapFree=67111356/67111528
[pid=30491] ppid=30489 vsize=648880 CPUtime=100.15 cores=1,3,5,7
/proc/30491/stat : 30491 (ttsth-5-4) S 30489 30491 27559 0 -1 4202496 120742 0 0 0 9949 66 0 0 23 0 5 0 517908427 664453120 117104 33554432000 4194304 4338100 140736394189472 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30491/statm: 162220 117104 176 36 0 160125 0
[pid=30491/tid=30493] ppid=30489 vsize=648880 CPUtime=24.88 cores=1,3,5,7
/proc/30491/task/30493/stat : 30493 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 19615 0 0 0 2460 28 0 0 25 0 5 0 517908483 664453120 117104 33554432000 4194304 4338100 140736394189472 18446744073709551615 4284160 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30491/tid=30494] ppid=30489 vsize=648880 CPUtime=24.93 cores=1,3,5,7
/proc/30491/task/30494/stat : 30494 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 36428 0 0 0 2487 6 0 0 25 0 5 0 517908483 664453120 117104 33554432000 4194304 4338100 140736394189472 18446744073709551615 4206915 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30491/tid=30495] ppid=30489 vsize=648880 CPUtime=24.82 cores=1,3,5,7
/proc/30491/task/30495/stat : 30495 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 14846 0 0 0 2460 22 0 0 25 0 5 0 517908483 664453120 117104 33554432000 4194304 4338100 140736394189472 18446744073709551615 4289126 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30491/tid=30496] ppid=30489 vsize=648880 CPUtime=24.93 cores=1,3,5,7
/proc/30491/task/30496/stat : 30496 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 36471 0 0 0 2486 7 0 0 25 0 5 0 517908483 664453120 117104 33554432000 4194304 4338100 140736394189472 18446744073709551615 222921172007 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 100.15
Current children cumulated vsize (KiB) 648880

[startup+51.1003 s]
/proc/loadavg: 7.68 7.86 6.93 9/186 30502
/proc/meminfo: memFree=27416644/32951124 swapFree=67111356/67111528
[pid=30491] ppid=30489 vsize=851452 CPUtime=202.4 cores=1,3,5,7
/proc/30491/stat : 30491 (ttsth-5-4) S 30489 30491 27559 0 -1 4202496 172076 0 0 0 20126 114 0 0 23 0 5 0 517908427 871886848 168438 33554432000 4194304 4338100 140736394189472 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30491/statm: 212863 168438 176 36 0 210768 0
[pid=30491/tid=30493] ppid=30489 vsize=851452 CPUtime=50.47 cores=1,3,5,7
/proc/30491/task/30493/stat : 30493 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 33737 0 0 0 4998 49 0 0 25 0 5 0 517908483 871886848 168438 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30491/tid=30494] ppid=30489 vsize=851452 CPUtime=50.53 cores=1,3,5,7
/proc/30491/task/30494/stat : 30494 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 48129 0 0 0 5044 9 0 0 25 0 5 0 517908483 871886848 168438 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30491/tid=30495] ppid=30489 vsize=851452 CPUtime=50.27 cores=1,3,5,7
/proc/30491/task/30495/stat : 30495 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 27630 0 0 0 4985 42 0 0 25 0 5 0 517908483 871886848 168438 33554432000 4194304 4338100 140736394189472 18446744073709551615 222921172405 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30491/tid=30496] ppid=30489 vsize=851452 CPUtime=50.53 cores=1,3,5,7
/proc/30491/task/30496/stat : 30496 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 49198 0 0 0 5042 11 0 0 25 0 5 0 517908483 871886848 168438 33554432000 4194304 4338100 140736394189472 18446744073709551615 4284177 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 202.4
Current children cumulated vsize (KiB) 851452

[startup+102.307 s]
/proc/loadavg: 7.91 7.89 7.00 10/186 30503
/proc/meminfo: memFree=26835184/32951124 swapFree=67111356/67111528
[pid=30491] ppid=30489 vsize=1218324 CPUtime=406.89 cores=1,3,5,7
/proc/30491/stat : 30491 (ttsth-5-4) S 30489 30491 27559 0 -1 4202496 265212 0 0 0 40495 194 0 0 23 0 5 0 517908427 1247563776 261574 33554432000 4194304 4338100 140736394189472 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30491/statm: 304581 261574 176 36 0 302486 0
[pid=30491/tid=30493] ppid=30489 vsize=1218324 CPUtime=101.62 cores=1,3,5,7
/proc/30491/task/30493/stat : 30493 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 53966 0 0 0 10077 85 0 0 25 0 5 0 517908483 1247563776 261574 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30491/tid=30494] ppid=30489 vsize=1218324 CPUtime=101.73 cores=1,3,5,7
/proc/30491/task/30494/stat : 30494 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 84861 0 0 0 10156 17 0 0 25 0 5 0 517908483 1247563776 261574 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30491/tid=30495] ppid=30489 vsize=1218324 CPUtime=101.22 cores=1,3,5,7
/proc/30491/task/30495/stat : 30495 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 41370 0 0 0 10048 74 0 0 25 0 5 0 517908483 1247563776 261574 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30491/tid=30496] ppid=30489 vsize=1218324 CPUtime=101.73 cores=1,3,5,7
/proc/30491/task/30496/stat : 30496 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 71633 0 0 0 10158 15 0 0 25 0 5 0 517908483 1247563776 261574 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 406.89
Current children cumulated vsize (KiB) 1218324

[startup+162.301 s]
/proc/loadavg: 7.99 7.92 7.06 10/186 30505
/proc/meminfo: memFree=26349436/32951124 swapFree=67111356/67111528
[pid=30491] ppid=30489 vsize=1572448 CPUtime=646.51 cores=1,3,5,7
/proc/30491/stat : 30491 (ttsth-5-4) S 30489 30491 27559 0 -1 4202496 354044 0 0 0 64372 279 0 0 23 0 5 0 517908427 1610186752 350406 33554432000 4194304 4338100 140736394189472 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30491/statm: 393112 350406 176 36 0 391017 0
[pid=30491/tid=30493] ppid=30489 vsize=1572448 CPUtime=161.6 cores=1,3,5,7
/proc/30491/task/30493/stat : 30493 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 79649 0 0 0 16037 123 0 0 25 0 5 0 517908483 1610186752 350406 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30491/tid=30494] ppid=30489 vsize=1572448 CPUtime=161.71 cores=1,3,5,7
/proc/30491/task/30494/stat : 30494 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 115790 0 0 0 16146 25 0 0 25 0 5 0 517908483 1610186752 350406 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30491/tid=30495] ppid=30489 vsize=1572448 CPUtime=160.88 cores=1,3,5,7
/proc/30491/task/30495/stat : 30495 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 64223 0 0 0 15979 109 0 0 25 0 5 0 517908483 1610186752 350406 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30491/tid=30496] ppid=30489 vsize=1572448 CPUtime=161.73 cores=1,3,5,7
/proc/30491/task/30496/stat : 30496 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 81000 0 0 0 16154 19 0 0 25 0 5 0 517908483 1610186752 350406 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 646.51
Current children cumulated vsize (KiB) 1572448


################
# More data... #
################

/proc/30491/task/30496/stat : 30496 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 89925 0 0 0 70086 25 0 0 25 0 5 0 517908483 2944450560 677633 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2803.39
Current children cumulated vsize (KiB) 2875440

[startup+762.306 s]
/proc/loadavg: 8.12 8.07 7.53 10/186 30521
/proc/meminfo: memFree=23425400/32951124 swapFree=67111356/67111528
[pid=30491] ppid=30489 vsize=3029768 CPUtime=3042.97 cores=1,3,5,7
/proc/30491/stat : 30491 (ttsth-5-4) S 30489 30491 27559 0 -1 4202496 721259 0 0 0 303883 414 0 0 23 0 5 0 517908427 3102482432 717621 33554432000 4194304 4338100 140736394189472 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30491/statm: 757442 717621 176 36 0 755347 0
[pid=30491/tid=30493] ppid=30489 vsize=3029768 CPUtime=759.39 cores=1,3,5,7
/proc/30491/task/30493/stat : 30493 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 197427 0 0 0 75759 180 0 0 25 0 5 0 517908483 3102482432 717621 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30491/tid=30494] ppid=30489 vsize=3029768 CPUtime=761.64 cores=1,3,5,7
/proc/30491/task/30494/stat : 30494 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 258364 0 0 0 76099 65 0 0 25 0 5 0 517908483 3102482432 717621 33554432000 4194304 4338100 140736394189472 18446744073709551615 222921172384 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30491/tid=30495] ppid=30489 vsize=3029768 CPUtime=760.29 cores=1,3,5,7
/proc/30491/task/30495/stat : 30495 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 161874 0 0 0 75888 141 0 0 25 0 5 0 517908483 3102482432 717621 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30491/tid=30496] ppid=30489 vsize=3029768 CPUtime=761.07 cores=1,3,5,7
/proc/30491/task/30496/stat : 30496 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 90212 0 0 0 76081 26 0 0 25 0 5 0 517908483 3102482432 717621 33554432000 4194304 4338100 140736394189472 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3042.97
Current children cumulated vsize (KiB) 3029768

[startup+822.301 s]
/proc/loadavg: 8.09 8.07 7.56 10/186 30522
/proc/meminfo: memFree=23123164/32951124 swapFree=67111356/67111528
[pid=30491] ppid=30489 vsize=3193612 CPUtime=3282.62 cores=1,3,5,7
/proc/30491/stat : 30491 (ttsth-5-4) S 30489 30491 27559 0 -1 4202496 762576 0 0 0 327833 429 0 0 23 0 5 0 517908427 3270258688 758938 33554432000 4194304 4338100 140736394189472 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30491/statm: 798403 758938 176 36 0 796308 0
[pid=30491/tid=30493] ppid=30489 vsize=3193612 CPUtime=819.05 cores=1,3,5,7
/proc/30491/task/30493/stat : 30493 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 219106 0 0 0 81717 188 0 0 25 0 5 0 517908483 3270258688 758938 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30491/tid=30494] ppid=30489 vsize=3193612 CPUtime=821.63 cores=1,3,5,7
/proc/30491/task/30494/stat : 30494 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 266619 0 0 0 82095 68 0 0 25 0 5 0 517908483 3270258688 758938 33554432000 4194304 4338100 140736394189472 18446744073709551615 222921172414 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30491/tid=30495] ppid=30489 vsize=3193612 CPUtime=820.28 cores=1,3,5,7
/proc/30491/task/30495/stat : 30495 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 171788 0 0 0 81885 143 0 0 25 0 5 0 517908483 3270258688 758938 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30491/tid=30496] ppid=30489 vsize=3193612 CPUtime=821.06 cores=1,3,5,7
/proc/30491/task/30496/stat : 30496 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 91681 0 0 0 82080 26 0 0 25 0 5 0 517908483 3270258688 758938 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3282.62
Current children cumulated vsize (KiB) 3193612

[startup+882.301 s]
/proc/loadavg: 8.07 8.06 7.59 10/186 30524
/proc/meminfo: memFree=22938848/32951124 swapFree=67111356/67111528
[pid=30491] ppid=30489 vsize=3351200 CPUtime=3522.3 cores=1,3,5,7
/proc/30491/stat : 30491 (ttsth-5-4) S 30489 30491 27559 0 -1 4202496 803656 0 0 0 351785 445 0 0 23 0 5 0 517908427 3431628800 800018 33554432000 4194304 4338100 140736394189472 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30491/statm: 837800 800018 176 36 0 835705 0
[pid=30491/tid=30493] ppid=30489 vsize=3351200 CPUtime=878.73 cores=1,3,5,7
/proc/30491/task/30493/stat : 30493 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 235822 0 0 0 87676 197 0 0 25 0 5 0 517908483 3431628800 800019 33554432000 4194304 4338100 140736394189472 18446744073709551615 222921172105 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30491/tid=30494] ppid=30489 vsize=3351200 CPUtime=881.64 cores=1,3,5,7
/proc/30491/task/30494/stat : 30494 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 277914 0 0 0 88092 72 0 0 25 0 5 0 517908483 3431628800 800019 33554432000 4194304 4338100 140736394189472 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30491/tid=30495] ppid=30489 vsize=3351200 CPUtime=880.28 cores=1,3,5,7
/proc/30491/task/30495/stat : 30495 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 184157 0 0 0 87881 147 0 0 25 0 5 0 517908483 3431628800 800020 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30491/tid=30496] ppid=30489 vsize=3351200 CPUtime=881.07 cores=1,3,5,7
/proc/30491/task/30496/stat : 30496 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 92382 0 0 0 88080 27 0 0 25 0 5 0 517908483 3431628800 800020 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3522.3
Current children cumulated vsize (KiB) 3351200

[startup+942.3 s]
/proc/loadavg: 8.06 8.06 7.62 9/186 30525
/proc/meminfo: memFree=22656332/32951124 swapFree=67111356/67111528
[pid=30491] ppid=30489 vsize=3536940 CPUtime=3761.97 cores=1,3,5,7
/proc/30491/stat : 30491 (ttsth-5-4) S 30489 30491 27559 0 -1 4202496 852024 0 0 0 375736 461 0 0 23 0 5 0 517908427 3621826560 848386 33554432000 4194304 4338100 140736394189472 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30491/statm: 884235 848386 176 36 0 882140 0
[pid=30491/tid=30493] ppid=30489 vsize=3536940 CPUtime=938.4 cores=1,3,5,7
/proc/30491/task/30493/stat : 30493 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 259432 0 0 0 93634 206 0 0 25 0 5 0 517908483 3621826560 848386 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30491/tid=30494] ppid=30489 vsize=3536940 CPUtime=941.63 cores=1,3,5,7
/proc/30491/task/30494/stat : 30494 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 288775 0 0 0 94089 74 0 0 25 0 5 0 517908483 3621826560 848386 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30491/tid=30495] ppid=30489 vsize=3536940 CPUtime=940.28 cores=1,3,5,7
/proc/30491/task/30495/stat : 30495 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 197704 0 0 0 93878 150 0 0 25 0 5 0 517908483 3621826560 848386 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30491/tid=30496] ppid=30489 vsize=3536940 CPUtime=941.06 cores=1,3,5,7
/proc/30491/task/30496/stat : 30496 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 92731 0 0 0 94079 27 0 0 25 0 5 0 517908483 3621826560 848386 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3761.97
Current children cumulated vsize (KiB) 3536940

[startup+1002.3 s]
/proc/loadavg: 8.09 8.08 7.65 10/186 30536
/proc/meminfo: memFree=22428140/32951124 swapFree=67111356/67111528
[pid=30491] ppid=30489 vsize=3639080 CPUtime=4001.63 cores=1,3,5,7
/proc/30491/stat : 30491 (ttsth-5-4) S 30489 30491 27559 0 -1 4202496 876381 0 0 0 399691 472 0 0 23 0 5 0 517908427 3726417920 872743 33554432000 4194304 4338100 140736394189472 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30491/statm: 909770 872743 176 36 0 907675 0
[pid=30491/tid=30493] ppid=30489 vsize=3639080 CPUtime=998.38 cores=1,3,5,7
/proc/30491/task/30493/stat : 30493 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 267560 0 0 0 99627 211 0 0 25 0 5 0 517908483 3726417920 872743 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254259 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30491/tid=30494] ppid=30489 vsize=3639080 CPUtime=1001.63 cores=1,3,5,7
/proc/30491/task/30494/stat : 30494 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 298115 0 0 0 100086 77 0 0 25 0 5 0 517908483 3726417920 872743 33554432000 4194304 4338100 140736394189472 18446744073709551615 222921172414 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30491/tid=30495] ppid=30489 vsize=3639080 CPUtime=1000.28 cores=1,3,5,7
/proc/30491/task/30495/stat : 30495 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 203161 0 0 0 99876 152 0 0 25 0 5 0 517908483 3726417920 872743 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30491/tid=30496] ppid=30489 vsize=3639080 CPUtime=1000.74 cores=1,3,5,7
/proc/30491/task/30496/stat : 30496 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 94163 0 0 0 100046 28 0 0 25 0 5 0 517908483 3726417920 872743 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4001.63
Current children cumulated vsize (KiB) 3639080

[startup+1062.3 s]
/proc/loadavg: 8.07 8.08 7.67 9/186 30538
/proc/meminfo: memFree=22312556/32951124 swapFree=67111356/67111528
[pid=30491] ppid=30489 vsize=3731012 CPUtime=4241.17 cores=1,3,5,7
/proc/30491/stat : 30491 (ttsth-5-4) S 30489 30491 27559 0 -1 4202496 889956 0 0 0 423637 480 0 0 23 0 5 0 517908427 3820556288 886318 33554432000 4194304 4338100 140736394189472 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30491/statm: 932753 886318 176 36 0 930658 0
[pid=30491/tid=30493] ppid=30489 vsize=3731012 CPUtime=1058.36 cores=1,3,5,7
/proc/30491/task/30493/stat : 30493 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 278262 0 0 0 105619 217 0 0 25 0 5 0 517908483 3820556288 886318 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30491/tid=30494] ppid=30489 vsize=3731012 CPUtime=1061.63 cores=1,3,5,7
/proc/30491/task/30494/stat : 30494 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 299770 0 0 0 106085 78 0 0 25 0 5 0 517908483 3820556288 886318 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30491/tid=30495] ppid=30489 vsize=3731012 CPUtime=1060.22 cores=1,3,5,7
/proc/30491/task/30495/stat : 30495 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 203643 0 0 0 105870 152 0 0 25 0 5 0 517908483 3820556288 886318 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30491/tid=30496] ppid=30489 vsize=3731012 CPUtime=1060.36 cores=1,3,5,7
/proc/30491/task/30496/stat : 30496 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 94899 0 0 0 106007 29 0 0 25 0 5 0 517908483 3820556288 886318 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4241.17
Current children cumulated vsize (KiB) 3731012

[startup+1122.3 s]
/proc/loadavg: 8.12 8.09 7.70 10/186 30539
/proc/meminfo: memFree=22148712/32951124 swapFree=67111356/67111528
[pid=30491] ppid=30489 vsize=3775860 CPUtime=4480.71 cores=1,3,5,7
/proc/30491/stat : 30491 (ttsth-5-4) S 30489 30491 27559 0 -1 4202496 907797 0 0 0 447586 485 0 0 23 0 5 0 517908427 3866480640 904159 33554432000 4194304 4338100 140736394189472 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30491/statm: 943965 904159 176 36 0 941870 0
[pid=30491/tid=30493] ppid=30489 vsize=3775860 CPUtime=1118.33 cores=1,3,5,7
/proc/30491/task/30493/stat : 30493 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 282732 0 0 0 111614 219 0 0 25 0 5 0 517908483 3866480640 904159 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30491/tid=30494] ppid=30489 vsize=3775860 CPUtime=1121.61 cores=1,3,5,7
/proc/30491/task/30494/stat : 30494 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 299970 0 0 0 112082 79 0 0 25 0 5 0 517908483 3866480640 904159 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30491/tid=30495] ppid=30489 vsize=3775860 CPUtime=1120.18 cores=1,3,5,7
/proc/30491/task/30495/stat : 30495 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 211918 0 0 0 111863 155 0 0 25 0 5 0 517908483 3866480640 904159 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30491/tid=30496] ppid=30489 vsize=3775860 CPUtime=1120.01 cores=1,3,5,7
/proc/30491/task/30496/stat : 30496 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 99795 0 0 0 111971 30 0 0 25 0 5 0 517908483 3866480640 904159 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4480.71
Current children cumulated vsize (KiB) 3775860

[startup+1182.3 s]
/proc/loadavg: 8.25 8.12 7.73 10/186 30544
/proc/meminfo: memFree=22000956/32951124 swapFree=67111356/67111528
[pid=30491] ppid=30489 vsize=3867760 CPUtime=4720.39 cores=1,3,5,7
/proc/30491/stat : 30491 (ttsth-5-4) S 30489 30491 27559 0 -1 4202496 931017 0 0 0 471545 494 0 0 23 0 5 0 517908427 3960586240 927379 33554432000 4194304 4338100 140736394189472 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30491/statm: 966940 927379 176 36 0 964845 0
[pid=30491/tid=30493] ppid=30489 vsize=3867760 CPUtime=1178.33 cores=1,3,5,7
/proc/30491/task/30493/stat : 30493 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 289198 0 0 0 117611 222 0 0 25 0 5 0 517908483 3960586240 927379 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30491/tid=30494] ppid=30489 vsize=3867760 CPUtime=1181.61 cores=1,3,5,7
/proc/30491/task/30494/stat : 30494 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 302857 0 0 0 118081 80 0 0 25 0 5 0 517908483 3960586240 927379 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30491/tid=30495] ppid=30489 vsize=3867760 CPUtime=1180.19 cores=1,3,5,7
/proc/30491/task/30495/stat : 30495 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 225216 0 0 0 117860 159 0 0 25 0 5 0 517908483 3960586240 927379 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30491/tid=30496] ppid=30489 vsize=3867760 CPUtime=1179.69 cores=1,3,5,7
/proc/30491/task/30496/stat : 30496 (ttsth-5-4) R 30489 30491 27559 0 -1 4202560 100364 0 0 0 117938 31 0 0 25 0 5 0 517908483 3960586240 927379 33554432000 4194304 4338100 140736394189472 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4720.39
Current children cumulated vsize (KiB) 3867760



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.31 s]
/proc/loadavg: 8.09 8.09 7.73 6/183 30544
/proc/meminfo: memFree=23148536/32951124 swapFree=67111356/67111528
[pid=30491] ppid=30489 vsize=0 CPUtime=4800.17 cores=1,3,5,7
/proc/30491/stat : 30491 (ttsth-5-4) Z 30489 30491 27559 0 -1 4203548 950822 0 0 0 479510 507 0 0 18 0 2 0 517908427 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30491/statm: 0 0 0 0 0 0 0
[pid=30491/tid=30495] ppid=30489 vsize=0 CPUtime=1200.2 cores=1,3,5,7
/proc/30491/task/30495/stat : 30495 (ttsth-5-4) R 30489 30491 27559 0 -1 4203588 233073 0 0 0 119853 167 0 0 25 0 2 0 517908483 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.17
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.31 s]
/proc/loadavg: 8.09 8.09 7.73 6/183 30544
/proc/meminfo: memFree=23148536/32951124 swapFree=67111356/67111528
[pid=30491] ppid=30489 vsize=0 CPUtime=4800.17 cores=1,3,5,7
/proc/30491/stat : 30491 (ttsth-5-4) Z 30489 30491 27559 0 -1 4203548 950822 0 0 0 479510 507 0 0 18 0 2 0 517908427 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30491/statm: 0 0 0 0 0 0 0
[pid=30491/tid=30495] ppid=30489 vsize=0 CPUtime=1200.2 cores=1,3,5,7
/proc/30491/task/30495/stat : 30495 (ttsth-5-4) R 30489 30491 27559 0 -1 4203588 233073 0 0 0 119853 167 0 0 25 0 2 0 517908483 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.17
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1202.4
CPU time (s): 4800.26
CPU user time (s): 4795.1
CPU system time (s): 5.15522
CPU usage (%): 399.224
Max. virtual memory (cumulated for all children) (KiB): 3946488

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4795.1
system time used= 5.15522
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 950822
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 4849
involuntary context switches= 29193

runsolver used 1.9587 second user time and 4.6143 second system time

The end

Launcher Data 

Begin job on node109 at 2011-04-29 09:44:37
IDJOB=3249460
IDBENCH=83170
IDSOLVER=1654
FILE ID=node109/3249460-1304063077
RUNJOBID= node109-1304063077-30472
PBS_JOBID= 13170356
Free space on /tmp= 72384 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/mosoi/289/289-unsat-7x39.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3249460-1304063077/watcher-3249460-1304063077 -o /tmp/evaluation-result-3249460-1304063077/solver-3249460-1304063077 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249460-1304063077.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= f717554c4e7f802b3078c48464dbcd9d
RANDOM SEED=1880984569

node109.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      28724316 kB
Buffers:        885612 kB
Cached:        2847640 kB
SwapCached:        168 kB
Active:         822476 kB
Inactive:      2979284 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      28724316 kB
SwapTotal:    67111528 kB
SwapFree:     67111356 kB
Dirty:            1668 kB
Writeback:           0 kB
AnonPages:       68460 kB
Mapped:          14884 kB
Slab:           360900 kB
PageTables:       4340 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   211144 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72384 MiB
End job on node109 at 2011-04-29 10:04:41